>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3591988. --> Reported by: [mably](https://www.drupal.org/user/3375160) Related to !13 >>> <h2>Summary</h2> <p>File-purged <code>audit_trail_segment</code> rows accumulate indefinitely. Each row is small (~1 KB) but on a long-running install with hourly granularity the table grows linearly with chain age. The rows can't be deleted today -- they carry <code>anchor_before</code> / <code>anchor_after</code> hashes the verifier uses to bridge across the now-empty id range -- but consecutive file-purged segments on the same chain can be COMPACTED into a single bridging row without breaking verification.</p> <p>Proposed: a new <code>ChainArchiver::compactFilePurgedSegments(string $chain_id, int $cutoff_id): int</code> method, plus an opt-in cron pass that compacts past a configurable retention.</p> <h2>Motivation</h2> <p>Hygiene only. The bookkeeping rows are small (~1 KB each), carry no PII (just anchor hashes + ids + counts), and with proper retention + indexing the per-chain segment scans stay bounded even for very large tables. Nothing's broken today and nothing's projected to break. This is just a "clean lifecycle should end with a way to compact the bookkeeping rather than just stop touching it" feature -- the existing transient-purge / archive / live-purge / file-purge chain ends with a row that lives forever; compaction closes that asymmetry.</p> <p>Existing <code>deleteSegment()</code> explicitly refuses any segment with any lifecycle flag set (<code>src/Archive/ChainArchiver.php:1757</code>), with a docblock that explains why file-purged rows are load-bearing for the verifier's gap-bridge. Compaction is the design-correct way to reduce the row count without breaking that invariant: replace N consecutive file-purged segments with a single equivalent bridging row.</p> <h2>Why deletion breaks the chain (recap)</h2> <p>The verifier walks rows by id. When it sees a gap, <code>findArchiveBridging()</code> looks for a segment row covering the gap; the segment's <code>anchor_before</code> must match the running <code>expected_previous</code>, and its <code>anchor_after</code> becomes the new <code>expected_previous</code> after the bridge. Without the segment row, the verifier has no source for the hash that should sit between the last pre-gap row and the first post-gap row.</p> <p>Segments on the same chain often chain to each other: segment N+1's <code>anchor_before</code> equals segment N's <code>anchor_after</code>. Deleting segment N breaks segment N+1's chain back to genesis.</p> <h2>Why compaction is safe</h2> <p>If segments <code>S1=[1, 100]</code>, <code>S2=[101, 200]</code>, <code>S3=[201, 300]</code> are all file-purged on the same chain, they can be replaced with a single row <code>S' = [1, 300]</code> whose:</p> <ul> <li><code>anchor_before</code> = <code>S1.anchor_before</code> (the original genesis anchor).</li> <li><code>anchor_after</code> = <code>S3.anchor_after</code> (the last covered row's hash).</li> <li><code>identity_hmac</code> + <code>archive_hmac</code> + <code>lifecycle_hmac</code> re-signed under the current secret against the consolidated record.</li> <li><code>file_purged_at</code> = max of the originals; <code>file_purged_event_id</code> = the most recent.</li> <li><code>row_count</code> = sum of originals; <code>ack_count</code> = sum.</li> <li><code>file_sha256</code> = empty (the original files are gone; the consolidated row attests to a range, not a file).</li> </ul> <p>The verifier walks the chain, hits the gap <code>[1, 300]</code>, finds <code>S'</code>, bridges via <code>anchor_before -&gt; anchor_after</code>. Identical to the pre-compaction behavior, fewer rows in the table.</p> <h2>Implementation sketch</h2> <ol> <li>New public method <code>ChainArchiver::compactFilePurgedSegments(string $chain_id, int $cutoff_id): array</code>. <ul> <li>Acquires the chain-write lock via <code>ChainWriteLock::acquireWithDeadline()</code>.</li> <li>Selects all segments on <code>$chain_id</code> with <code>file_purged_at != 0</code> AND <code>to_id &amp;lt;= $cutoff_id</code>, ordered by <code>from_id</code>.</li> <li>Walks the result, accumulating CONTIGUOUS runs (segment N+1's <code>from_id</code> == segment N's <code>to_id</code> + 1, OR segment N's <code>anchor_after</code> matches segment N+1's <code>anchor_before</code> when there's an id gap caused by other-chain rows).</li> <li>For each run of 2+ segments, INSERTs the consolidated row + DELETEs the originals in one transaction.</li> <li>Returns the list of (new_segment_id, deleted_segment_ids) pairs for the audit trail.</li> </ul> </li> <li>Emit a <code>segment_compacted</code> chain event per run, payload referencing the consolidated range and the deleted segment ids. Signed normally.</li> <li>Optional cron pass <code>CronArchiveHook::compactionPass()</code> using a new config knob <code>compact_after</code> (default disabled / NULL; opt-in only). When set, runs after the existing file-purge pass and compacts segments past <code>now - compact_after</code>.</li> <li>Operator interface: a drush command <code>audit_trail:compact</code> with explicit chain + cutoff arguments. Admin form deferred to a follow-up.</li> </ol> <h2>Verifier impact</h2> <p>None of the verifier rules change. <code>findArchiveBridging()</code> already accepts any segment whose anchors satisfy the bridge predicate; a wider segment that bridges the same gap is functionally identical. The segment-event cross-reference check (<code>verifySegmentEventCrossReference</code>) operates per-event, not per-segment, so consolidating segments doesn't affect it -- the original <code>segment_archived</code> / <code>segment_live_purged</code> / <code>segment_file_purged</code> events on the chain reference segment ids that no longer exist after compaction, but those events are emitted before this MR's compaction runs.</p> <p>The <code>segment_*</code> events that reference deleted segments are a real concern: the verifier's cross-reference check looks up the segment by id and would find it missing. Two resolutions:</p> <ul> <li>(a) Add <code>compacted_into_segment_id</code> on the chain-event payload of the consolidated <code>segment_compacted</code> event, and extend the cross-reference rule to follow that pointer when the original segment is gone. Self-contained, no other rule changes.</li> <li>(b) Treat the original <code>segment_*</code> events as superseded by <code>segment_compacted</code> the same way the existing live-purge supersession rule handles a later <code>segment_restored</code>.</li> </ul> <p>Option (a) is the cleaner pattern (explicit pointer rather than re-using the supersession rule with new semantics).</p> <h2>Tests to add</h2> <ul> <li>Happy path: 3 file-purged segments + post-cutoff rows. Run <code>compactFilePurgedSegments</code>. Assert: 1 consolidated row exists, 3 originals are gone, <code>verifyChain()</code> returns ok.</li> <li>Edge: only one file-purged segment in the range. No-op (compaction requires 2+ contiguous).</li> <li>Edge: file-purged segments interleaved with non-file-purged ones (e.g., one still in live-purged state). Compaction must NOT cross the boundary.</li> <li>Cross-chain id-space interleaving (per the testing convention #3591949): the consolidated range spans ids belonging to other chains. Confirm the consolidated row's anchors are correct.</li> <li>Concurrency: chain-write lock acquired around the read-modify-INSERT-DELETE so a concurrent <code>createBareSegment</code> / <code>acknowledgeChainReset</code> can't slip a bisecting row into the consolidated range.</li> <li>Verifier acceptance of the <code>compacted_into_segment_id</code> pointer (per resolution (a)).</li> </ul> <h2>Open questions</h2> <ul> <li>Should we keep a separate <code>audit_trail_compacted_segment</code> table (or similar) as a forensic record of which segments were consolidated when, OR is the <code>segment_compacted</code> chain event sufficient? Probably the latter -- the chain event already gives operators a queryable history.</li> <li>Default value for <code>compact_after</code>: opt-in (NULL = no compaction) seems right. A long default (P30D? P90D?) would be opinionated for a feature that mostly matters on multi-year installs.</li> <li>Compaction granularity: per-run (2+ contiguous segments) vs per-bucket (e.g., one consolidated per month). Per-run is simpler and matches the existing coverage-pass bucket-aligned shape if we want.</li> </ul>