fix: #3624689 Write the one column a failed advance and a timeout re-arm own, guarded on the token still being theirs

Two writers saved the whole token after a read they could not trust, so a token another worker had already advanced could be written back into the run.

  • WorkflowExecutor::handleAdvanceFailure() ran after the advance rolled back and released the token's lock, so another worker could take the token and advance it for real in between. The whole-entity save then put active back over consumed, and the next run advanced the node a second time.
  • TimeoutSweeper::fireInner() re-armed the deadline after the timeout action, from a read its own transaction snapshot keeps showing as parked. A person completing the task while their reminder fired had that completion written over, and the run was left holding a token nothing would resume.

Both now write the single column they own, guarded on the state they decided from, which is a current read where the checks above them are not: the attempt count through a new StateTransitions::setTokenAttempts() beside clearTokenAttempts(), and the deadline inline in the sweeper the way fireTimerInner() already re-arms the ladder's own alarm. A caller whose guard matches no row has been beaten to it and stops, so a failed advance no longer counts an attempt against a token it has lost, nor asks the queue to retry work already done.

Tests, both failing without the fix:

  • TimeoutTest::testRearmDoesNotResurrectTheCompletedTask - the completed token read parked again before the fix.
  • EngineDeadLetterTest::testNoAttemptIsCountedOnceAnotherWorkerHasAdvancedTheToken - the attempt was counted and the exception rethrown before the fix.

One process cannot run two workers, so each stages the second at the moment it really can act: a new FailingLock mode consumes the token as its advance lock is released, and a test timeout action acts out a completion landing while the action runs.

No update hook: nothing stored changes shape.

Audit rounds (5, closed on a clean round). Round 2 mutated each guard to check a test actually catches it. Three held: removing the state = active condition, reporting a refused write as a win, and removing the state = parked condition each turn a test red. The fourth survived — dropping the if ($written > 0) around the re-arm's cache invalidation left every test green. RawUpdateTrait asks every caller to invalidate only on a change, and orchestra_token_list is a site-wide tag, so without that check every task answered while its reminder fires would drop every dashboard, trace and inbox render on the site for a write that changed nothing. testTheRefusedRearmInvalidatesNothing now holds it, using the same checksum instrument as EngineDeadLetterTest::testClearingAnAlreadyZeroCountInvalidatesNothing at the engine's other raw write, and fired through fireNow() so nothing else is running.

The sweep for other instances of the defect found none: the three remaining whole-entity writes of these columns are all safe — IncidentManager::requeueIncidentToken() is serialised by the incident claim in the same transaction, and the executor's two setDeadline() saves run inside advance() while it holds the token's lock.

Edited by Frank Mably

Merge request reports

Loading
Loading