Issue #3608236: Extract a core execution service so the recovery and incident logic move out and WorkflowEngine becomes a thin facade

Decomposes the 3662-line WorkflowEngine into three collaborators behind an unchanged public contract:

  • WorkflowExecutor (orchestra.workflow_executor): the core execution service: the advance loop with its retry and dead-letter handling, signal/resume, cancel and fail teardown, join and split resolution, token machinery, locking, timeout math, variable reads and writes, subprocess coordination and the inline drain. The execution primitives the collaborators call back into (proceed(), checkCompletion(), recoverStalledInstance(), claimIncident(), failInstance(), cancelLiveTokens(), definitionFor(), resolveNode()) are public on this internal service.
  • IncidentManager (orchestra.incident_manager): the operator actions resolving open incidents (retry, resume with variables, skip, cancel branch, fail instance).
  • InstanceRecovery (orchestra.instance_recovery): the three cron recovery sweeps (reconcileStuckInstances(), recoverStalledInstances(), reconcileTerminalTokens()) and their shared bounded-batch query helpers.
  • WorkflowEngine: now a 290-line facade implementing WorkflowEngineInterface by delegation. Every public method signature (interface plus the extra publics the timeout sweep uses) is unchanged, so no caller changes.

One deliberate deviation from the cut sketched in comment 16671567, needed to keep the service graph acyclic: the loop-side failure handling (handleAdvanceFailure(), retryPolicyFor(), raiseIncident(), openIncidentCount(), resolveOpenIncidents()) and the per-instance stall recovery (isStalled(), recoverStalledInstance(), armJoinTimeouts()) stay in the executor, because the advance loop itself needs them: advanceQueued() dead-letters, checkCompletion() gates on open incidents, and cancel() recovers stalls. The collaborators hold only what sits outside the loop, and both depend on the executor, never the reverse. That direction is what breaks the circular dependency this issue exists to remove.

Extracted in three reviewable steps, each kept green against the full kernel suite: the executor first, then IncidentManager, then InstanceRecovery. The architecture docs note the split.

Merge request reports

Loading