Incident model: recoverable dead-lettering with operator resolution (#3604207)
Implements the incident model from #3604207: instead of always failing the instance, a dead-lettered token (one whose task exhausted its advance retries) raises an incident by default and leaves the instance's other branches running for an operator to resolve.
This MR (kernel core)
orchestra_incidentcontent entity +IncidentInterface.on_unrecoverable_failurepolicy (incident|fail, defaultincident); the cutoff inhandleAdvanceFailure()branches on it.raiseIncident()records the failure without cancelling the other tokens;checkCompletion()blocks while an incident is open, so the run halts for resolution rather than completing or failing.- Recovery API on
WorkflowEngineInterface:retryIncident,resumeIncident,skipIncident,cancelIncident,failFromIncident. resolve orchestra incidentspermission;InstanceCleanupdeletes incidents.EngineIncidentTestcovers the raise plus all five resolutions;EngineDeadLetterTestnow exercises thefailpolicy; the engine kernel tests install the new schema.
Design
The instance stays RUNNING while incidents are open (incident state is
derived from open records), so healthy branches keep advancing untouched. The
token's terminal ERROR state (from the dead-letter work) is the recoverable
marker; attempts is the retry-reset target.
Still to come (follow-up commits on this branch)
Operator UI (orchestra_ui actions), an incidents Views dashboard, docs
(docs/incidents.md + nav + roadmap) and translations.
Migration (pre-1.0)
No update hook ships. The orchestra_incident entity is created on fresh
installs and via the throwaway entity-update on existing sites.