>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3591838. --> Reported by: [mably](https://www.drupal.org/user/3375160) Related to !6 >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p>Every type of segment lifecycle event (<code>segment_archived</code>, <code>segment_transient_purged</code>, <code>segment_live_purged</code>, <code>segment_file_purged</code>, <code>segment_restored</code>) accumulates indefinitely in the live <code>audit_trail</code> table. They sit in chain-id gaps between archived segment <code>[from_id, to_id]</code> ranges and are never deleted.</p> <p>Observed on a real test chain (audit_trail_entity):</p> <pre> chain action n min_id max_id audit_trail_entity segment_archived 8 1509 1870 audit_trail_entity segment_file_purged 3 1823 1872 audit_trail_entity segment_live_purged 7 1547 1871 audit_trail_entity segment_transient_purged 8 1507 1869 </pre><p>The chain's archived segments cover <code>[1441,1459] [1482,1489] [1523,1523] [1569,1663] [1718,1721] [1749,1749] [1801,1801] [1846,1846]</code>. Every lifecycle event id sits in a gap between those ranges. None is inside any segment's row range, so none is deleted by <code>live-purge</code>.</p> <h4>Root cause</h4> <p>The chain has only ONE delete path: a row leaves <code>audit_trail</code> when its id falls inside an archived segment's <code>[from_id, to_id]</code> and <code>live-purge</code> fires on that segment. So a row is cleaned up if and only if it ends up inside a segment.</p> <p>User rows end up inside segments via <code>transientPurgePass()</code>: it scans rows by <code>context_transient IS NOT NULL</code>, buckets them, slices around existing segments, mints one bare segment per uncovered gap.</p> <p>Lifecycle events are written via <code>ChainArchiver::chainedWrite()</code> with the permanent bucket only (never a transient bucket). So they have <code>context_transient IS NULL</code> and <code>context_transient_hash = ''</code> from the moment they are INSERTed -- the transient-purge content filter excludes them. They never trigger bare-segment minting.</p> <p>The only way a lifecycle event currently ends up inside a segment is INCIDENTALLY -- when it happens to sit between two transient-purge-eligible user rows in the same week-bucket envelope, the bare-segment slicer sweeps it in. Lifecycle events that sit in a gap with no transient-purge-eligible neighbors stay in <code>audit_trail</code> forever.</p> <p>The archive pass cannot help either: its scan uses <code>id &amp;gt; max_archived_to_id</code> as a watermark, which assumes every row id below the max is already inside a segment. That assumption fails for any row not picked up by transient-purge -- the watermark moves past it as later segments are archived, trapping it below.</p> <h4>The architectural gap</h4> <p>No cron pass enforces the invariant "every row past some cutoff is inside a segment" unconditionally. Both existing passes have side-conditions:</p> <ul> <li><code>transientPurgePass()</code> -- gated on transient-purge being configured AND on rows having transient data. Misses lifecycle events. Misses everything on chains with transient-purge disabled.</li> <li><code>archivePass()</code> -- watermark filter traps rows below the watermark.</li> </ul> <h3 id="summary-proposed-resolution">Proposed resolution</h3> <p>Restructure cron so that <strong>one unconditional pass establishes segment coverage</strong>, and every other pass operates on the segments coverage produced.</p> <p>New pass: <code>coveragePass()</code>. Runs ALWAYS. Cutoff = the smallest configured retention threshold:</p> <pre> $coverage_cutoff = $resolved['transient_purge_after_us'] ?? $resolved['archive_after_us']; </pre><p>Scan: <code>audit_trail</code> rows past cutoff that are NOT inside any segment (<code>NOT EXISTS</code> against <code>audit_trail_segment</code>). Group by <code>segment_granularity</code>, slice each bucket envelope around existing segments, mint one bare segment per uncovered gap. No per-row side effect.</p> <p>Refactor the existing passes to operate on segments rather than raw rows:</p> <ul> <li><code>transientPurgePass()</code> -- scan BARE segments (<code>archived_at = 0 AND transient_purged_at = 0 AND to_created &amp;lt; transient_purge_after_cutoff</code>). For each, <code>UPDATE audit_trail SET context_transient = NULL</code> with a <code>context_transient IS NOT NULL</code> filter to skip rows that already have nothing to purge (lifecycle events swept into the segment, already-purged user rows). Stamp <code>transient_purged_at</code> on the segment and write the <code>segment_transient_purged</code> chain event. No more row scanning, no more content-filter row scan, no more bucket-slicing (coverage already did it). Critically: never touches an archived segment -- once the NDJSON is sealed by <code>file_sha256</code> + <code>archive_hmac</code>, NULLing live <code>context_transient</code> would diverge live data from the signed archive.</li> <li><code>archivePass()</code> -- scan BARE segments past <code>archive_after</code> (<code>archived_at = 0 AND to_created &amp;lt; archive_after_cutoff</code>). For each, write the NDJSON file + segment hashes + <code>segment_archived</code> chain event. No more raw-row scanning, no more watermark filter. The retention ordering <code>transient_purge_after &amp;lt; archive_after</code> guarantees that any bare segment past <code>archive_after</code> has already been processed by <code>transientPurgePass()</code> (if configured); the NDJSON captures the post-purge state, live and archived stay consistent.</li> <li><code>livePurgePass()</code>, <code>filePurgePass()</code> -- unchanged.</li> </ul> <p>Cron pipeline order:</p> <pre> $this-&gt;safelyRunPass($chain, 'coverage', ...); // always if ($resolved['transient_purge_after_us'] !== NULL) { $this-&gt;safelyRunPass($chain, 'transient-purge', ...); // conditional } $this-&gt;safelyRunPass($chain, 'archive', ...); // always $this-&gt;safelyRunPass($chain, 'live-purge', ...); $this-&gt;safelyRunPass($chain, 'orphan-heal', ...); $this-&gt;safelyRunPass($chain, 'file-purge', ...); </pre><p>Effect on user rows: identical timing. Coverage mints the bare segment, transient-purge NULLs the column on it, archive promotes it to archived, live-purge deletes the rows. Where today this happens via transient-purge bucket-slicing, it now happens via coverage bucket-slicing -- same logic, moved to a clearer home.</p> <p>Effect on lifecycle events: coverage now wraps them in bare segments based on chain id alone (not content). They flow through the remaining passes like every other segment and eventually get live-purged.</p> <p>Effect on chains with transient-purge disabled: coverage uses the larger <code>archive_after</code> cutoff. Bare segments get minted later but still get minted. Both user rows and lifecycle events flow through the pipeline.</p> <h3 id="summary-remaining-tasks">Remaining tasks</h3> <ul> <li>Add <code>coveragePass()</code> in <code>src/Hook/CronArchiveHook.php</code>. Reuse the bucket-slicing helper from the existing transient-purge pass; extract to a shared method on <code>ChainArchiver</code>.</li> <li>Refactor <code>transientPurgePass()</code> to scan segments past <code>transient_purge_after</code> rather than rows; remove the bucket-slicing + bare-segment minting (now in coverage).</li> <li>Refactor <code>archivePass()</code> to scan bare segments past <code>archive_after</code> rather than rows; remove the <code>max_archived_to_id</code> watermark.</li> <li>Wire the new pipeline order in <code>processChain()</code>.</li> <li>Kernel test: chain with transient-purge disabled, emit user rows + provoke several archive cycles, assert all five lifecycle event action types eventually leave <code>audit_trail</code>. Chain still verifies clean.</li> <li>Kernel test: chain with transient-purge enabled. Same shape. Verify rows get transient-purged on schedule and lifecycle events still flow through.</li> </ul> <h3 id="summary-api-changes">API changes</h3> <p>None externally observable. Refactor lives entirely inside <code>CronArchiveHook</code> + a shared helper on <code>ChainArchiver</code>.</p> <h3 id="summary-data-model-changes">Data model changes</h3> <p>None.</p>