feat: #3591988 Compact contiguous file-purged segments

Frank Mablyrequested to merge
issue/audit_trail-3591988:3591988-compact-contiguous-file-purged into 1.x

Summary

Closes the lifecycle's open-ended tail: file-purged audit_trail_segment rows otherwise accumulate forever (they carry the verifier's gap-bridge anchors and can't be deleted in isolation). A contiguous run of file-purged segments on the same chain folds into one consolidated bridging row whose anchor_before comes from the first of the run and anchor_after from the last -- the verifier bridges the same id range with fewer rows.

Hygiene only, opt-in everywhere.

Closes #3591988.

Why

audit_trail_segment has an unbounded tail today: every file-purged row stays in the table forever because the verifier's findArchiveBridging() reads its anchor_before / anchor_after columns to walk past the purged range. Direct deletion would break the chain. With proper retention + indexing the per-chain scans stay bounded even for large tables -- nothing's broken today -- but the lifecycle has an asymmetric open tail (transient-purge / archive / live-purge / file-purge all advance through bounded states; file-purged sits forever). Compaction closes that asymmetry for installs that care about the hygiene of a finite bookkeeping table.

What

Primitive

ChainArchiver::compactFilePurgedSegments(string $chain, int $cutoff_us): array. Walks file-purged segments whose to_created is strictly before cutoff_us, groups them in from_id order into contiguous runs (segment N+1's anchor_before == segment N's anchor_after), and for each run of 2+ replaces the originals with one consolidated row. The whole INSERT + emit + DELETE per run is wrapped in a transaction so partial failure rolls back to the pre-call state. Re-signs hmac / archive_hmac / lifecycle_hmac under the current secret over the consolidated row's actual fields. Emits one segment_compacted chain event per run whose payload's compacted_from lists every absorbed original id.

The cutoff is keyed off the segment row's to_created (the latest covered row's created stamp at archive time) so the threshold dimension and boundary operator match archive_after / live_purge_after / file_purge_after: all four read the covered row's creation time using a strict <. The covered audit_trail rows are gone by file-purge time; to_created is the surviving copy of that timestamp.

Verifier

AuditTrailVerifier::verifySegmentEventCrossReference() gains a compaction-superseded redirect. When a segment_archived / segment_live_purged / segment_file_purged / segment_transient_purged event references a segment id that no longer exists, scan the chain's segment_compacted events for one whose compacted_from includes the missing id. On a match, accept the original event as legitimate post-compaction. The chain event's own HMAC is verified by the regular row walk upstream of the redirect, so trusting the payload is safe -- a forged segment_compacted would fail the row walk first. The redirect scan is scoped to the chain being verified.

SegmentLifecycleAction::COMPACTED = 'segment_compacted' added to nonCrossCheckable() so the verifier doesn't strict-equal it against a *_event_id column the segment row doesn't have.

Operator surfaces

  • Cron pass: CronArchiveHook::compactionPass() runs after the file-purge pass when compact_after_us is non-NULL. Computes cutoff_us = now - compact_after and delegates to the primitive.
  • Drush: drush audit_trail:compact --chain=<id> [--before=<iso>]. Manual one-shot for operators who want to compact outside the cron cadence or test the flow on one chain without enabling the cron knob site-wide.
  • Config: new cron_archive.compact_after knob (default NULL = disabled) + per-chain compact_after override. Inline validation: compact_after > file_purge_after.
  • Settings form + per-chain edit form: both expose the field with the same Ajax preview wiring the other retention thresholds use. Submit handlers persist the value.

Lock handling

The compaction loop folds N runs inside the chain-write lock. ChainWriteLock::renewIfStale() (added in this MR alongside CHAIN_WRITE_LOCK_RENEW_INTERVAL_S = 10s) is called between iterations so a large backlog can't outlive the TTL and let a concurrent writer race. Same renewal pattern as ChainArchiver::renewCoverageLockIfStale().

Schema / API additions

  • AuditTrailChainInterface::getCompactAfter(): ?string.
  • AuditTrailChain::compact_after config property.
  • No table schema change. The segment_compacted chain event is the source of truth; the verifier scans those events on demand.

Tests

Six dedicated tests across ChainArchiverTest + CronArchiveHookTest:

  • testCompactFilePurgedSegmentsFoldsContiguousRunIntoOne -- happy path. 3 fully-lifecycled segments fold into 1 consolidated row; verifyChain() returns clean via the new redirect.
  • testCompactFilePurgedSegmentsSingleSegmentIsNoOp -- pins the "needs 2+ contiguous" rule.
  • testCompactFilePurgedSegmentsRespectsCutoff -- backdates to_created on two of three segments; cutoff includes only those two, third stays untouched.
  • testCompactFilePurgedSegmentsRefusesNonContiguousRun -- anchor mismatch on every inter-segment boundary leaves singleton runs the < 2 short-circuit drops. Guards the anchor-equality contiguity check.
  • testCompactedEventForgedHmacDoesNotMaskMissingSegment -- tampering the segment_compacted event hmac breaks the chain at the authentication layer, BEFORE the missing-segment redirect can fire. Pins the row-walk-runs-upstream invariant.
  • testCompactionRollsBackWhenChainedWriteThrows -- stub writer throws inside chainedWrite; the consolidated INSERT is rolled back, the originals survive, no segment_compacted event persists. Verifies the transaction wrap (commit 32af780).
  • testCompactionPassFoldsAgedFilePurgedSegmentsAfterLivePurge (CronArchiveHookTest) -- production-mirror state: rows aged 30 days, archived + live-purged + file-purged (so the audit_trail rows are DELETEd before cron runs). With compact_after = P1D cron must fold the run. This is the test that catches the cutoff-derivation bug fixed in commit 5f4d758.

Each bug-fix commit's test was verified to fail on the unpatched code (stash + rerun + stash pop).

100/100 across ChainArchiverTest + CronArchiveHookTest. phpcs clean (Drupal + DrupalPractice).

Branch shape

Six commits on top of 1.x, fast-forward mergeable:

  • 9fa8456 -- initial feature (primitive + verifier + cron + drush + forms + config + tests).
  • 1c17c3b -- CI fixes (cspell + phpstan).
  • 5f4d758 -- cron compaction cutoff fix: the original derivation queried audit_trail.created, but those rows are deleted by file-purge before compaction runs, so the cutoff lookup always returned nothing and the pass short-circuited. Re-keyed off the segment row instead.
  • 32af780 -- audit fix #1: wrapped INSERT consolidated + emit segment_compacted + DELETE originals in a transaction so partial failure rolls back. Added the three negative tests.
  • e868c96 -- audit fix #2: realigned the cutoff dimension. Compaction now keys off to_created (the latest covered row's created stamp, kept on the segment row) so the threshold dimension matches archive_after / live_purge_after / file_purge_after. Added lock renewal in the foreach loop. Dropped a dead Connection injection on the drush command.
  • 9a8e7a8 -- audit fix #3: tightened the cutoff comparison from <= to strict < for boundary consistency with the sibling cron passes.

What is NOT in this MR

  • No segment-row schema change. The verifier redirects via the chain event, not a sidecar table. If the on-demand scan ever becomes a measurable hotspot, an index can land in a follow-up.
  • No batch-size limit on the cron pass. Each pass folds whatever runs are eligible past the cutoff. If multi-chain installs hit very large backlogs on the first activation, that's the moment to add a per-tick cap. Lock TTL is renewed in the loop, so wall-clock isn't a concern.
Edited by Frank Mably

Merge request reports