Issue #3608215: Consolidate recovery/transition behind StatusTransitions and fix the RC-audit concurrency bugs

Frank Mablyrequested to merge
issue/orchestra-3608215:3608215-decompose-workflow-engine into 1.x

Decomposes the recovery/transition layer and fixes the concurrency bugs the release-candidate audit found. Closes #3608219 (the behavioral bugs) and delivers the core of #3608215 (the shared guarded transition primitive); the further class splits (a core execution service, InstanceRecovery/IncidentManager/JoinCoordinator, the low-risk TimeoutScheduler/VariableStore extractions) are tracked in #3608236.

Each commit is self-contained; refactor and behavioral fixes are separated.

Refactor

  • Extract StatusTransitions (a6f893d): the guarded conditional-write status flip becomes a shared orchestra.status_transitions service, injected into the engine, the timeout sweep and the incident actions, so every contended flip goes through one implementation. Behavior-preserving.

Behavioral fixes (all with the guard routed through the shared primitive)

  • RC-H2 (e6bdcb5, HIGH): TimeoutSweep::fireInner double-fired a timeout whose deadline was already cleared (a concurrent fire); it now returns instead of falling through. Regression test.
  • M1 (02e286a): fireTimerInner did unguarded load-then-save flips; add a fire claim, route the consume through the primitive, current-read the task status, guard the recurring re-arm.
  • RC-H1 + M3 + M6 (059485b, HIGH): the incident actions had no concurrency claim (two operators double-executed a branch); claim the incident open-to-resolved through the primitive, make retry/resume atomic, resolve open incidents on teardown. Regression test.
  • M2 + M4 (e4aac5b): migration re-derives live tokens (a token created after planning was stranded); the stalled-recovery sweep gains a persisted cursor so permanently-stuck instances do not starve the batch.
  • M5 (595e461): a reconcileTerminalTokens cron backstop cancels live tokens orphaned on a terminal instance by an advance that finished after teardown.

Testing

Full orchestra kernel suite green (183 tests); two new regression tests (RC-H1, RC-H2) fail on the old code and pass with the fix; phpcs and phpstan (level 1) clean. Found by a release-candidate audit of the changes since 1.0.0-alpha8.

Closes #3608219. Refs #3608215, #3608236.

Merge request reports