Issue #3605420: Retry a failed subprocess child N times on cron
Builds on the subprocess failure handling: re-run a failed child up to N times before giving up. Re-launch is cron-driven, never inline.
What changed
- The subprocess node gains an optional
retrycount (andretry_backoff). - When a child fails and the budget remains, the engine does not resume the parent: it arms the parked parent token with a deadline (now, or now + backoff) and an internal per-token attempt count. The cron sweep then re-launches a fresh child (
TimeoutSweepdelegates to the engine'srelaunchDueSubprocess, which clears the deadline with an atomic conditional update so two concurrent sweeps cannot both re-launch). - Once the budget is spent, the parent resumes with
__subprocess_failed__, as before, so the workflow routes the failure.
Re-launch is therefore always on cron (backoff falls out naturally; no inline retry storms). Basic retry was already expressible in the graph after !199; this is the first-class form.
Scope
Retry triggers on a child that actually fails (instance FAILED), which depends on the site on_unrecoverable_failure policy. A dead-letter that raises an incident (the default) leaves the parent waiting, unchanged.
Tests
Kernel: a poison child (policy set to fail) is re-launched twice, then the parent routes down its failed branch - three child instances. phpcs and cspell clean.