>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3591904. --> Reported by: [mably](https://www.drupal.org/user/3375160) Related to !10 >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p>Follow-up to #3591896. That issue narrowed the chain-write lock on <code>ChainArchiver::archiveSegment()</code>, but <code>restoreLocked()</code> stayed wide-locked with the TTL-renewer pattern from #3591878. This issue completes the narrowing for restore.</p> <p>During a long restore (1M rows, ~100s), every audit log write on the same chain currently drops because the chain-write lock is held for the full restore transaction. Issue #3591896 documented the root cause: the chain-write lock cannot be placed inside the row-replay transaction (Drupal's DatabaseLockBackend shares the DB connection, so the semaphore INSERT joins the transaction and holds an InnoDB row-level lock for the full transaction duration), and it cannot be placed before the transaction without freezing all chain writes for the full duration.</p> <h3>Proposed resolution: emit segment_restored event first</h3> <p>Restructure restore from one big transaction wrapping everything into three discrete steps. The chain-write lock holds for only ~50ms in Step 1; the long row replay in Step 2 runs without the lock; the final segment row UPDATE in Step 3 runs in autocommit without the lock.</p> <pre> Step 1 (autocommit, with chain-write lock, ~50ms): acquireChainWriteLock(chain) INSERT segment_restored event at chain head releaseChainWriteLock Step 2 (one transaction, NO chain-write lock, up to ~100s): BEGIN for each NDJSON row line: INSERT at original id for each NDJSON ack line: INSERT into audit_trail_acknowledgment COMMIT Step 3 (autocommit, ~10ms): UPDATE audit_trail_segment SET live_purged_at = 0, live_purged_event_id = <event id from step>, lifecycle_secret_id = <new>, lifecycle_hmac = <refreshed> WHERE id = :segment_id </refreshed></new></event></pre><h3>Why the chain event itself can serve as the in-progress marker</h3> <p>The chain event from Step 1 is HMAC-signed with the operator's current secret and chained via <code>previous_hash</code> to the live head. It is tamper-resistant attestation that a restore was initiated. Its presence in the chain after the segment's original <code>segment_live_purged</code> event signals "restore in progress or complete" without needing a new schema column.</p> <p>The verifier extension: for any segment with a <code>segment_restored</code> event whose id is greater than the segment's current <code>live_purged_event_id</code>, accept transitional states without flagging:</p> <ul> <li>Rows in <code>[from_id, to_id]</code> may be absent, partially present, or fully present.</li> <li>Segment row's <code>live_purged_event_id</code> may still point at the original purge event (pre-Step-3) or at the new restored event (post-Step-3).</li> </ul> <p>Per-row tamper detection (HMAC + previous_hash) is unaffected. The new rule only suppresses the structural "rows present in a purged range" flag when the chain itself attests to a restore in flight.</p> <h3>Operational benefit</h3> <p>During a 1M-row restore (~100s):</p> <ul> <li>Step 1 holds the chain-write lock for ~50ms. Concurrent logger writes briefly contend via Drupal's normal retry path.</li> <li>Step 2 holds no chain-write lock. Concurrent logger writes proceed at the chain head with no contention.</li> <li>Step 3 holds no chain-write lock; brief segment row UPDATE in autocommit.</li> </ul> <p>Total chain-write-lock-held duration during the restore: ~50ms instead of ~100s. Matches the operational improvement #3591896 delivered for archive.</p> <h3>Failure modes</h3> <ul> <li><strong>Step 1 fails</strong>: nothing committed, no chain event. Operator retries from scratch.</li> <li><strong>Step 1 succeeds, Step 2 fails</strong> (any row INSERT throws -- PK conflict, malformed envelope, transient-hash mismatch): row INSERTs roll back atomically (Step 2 is one transaction). Chain has the segment_restored event but no rows. Verifier accepts via the transitional rule. Operator retries; retry detection skips Step 1, re-runs Step 2.</li> <li><strong>Step 1 and Step 2 succeed, Step 3 fails</strong>: chain has event, rows present, segment row still points at original purge event. Verifier accepts via the transitional rule (segment_restored event id is greater than current live_purged_event_id). Operator retries; retry detection skips Steps 1 and 2, runs only Step 3.</li> <li><strong>Operator abandons mid-flight</strong>: chain has segment_restored event but segment row never UPDATEd. Verifier accepts indefinitely. Operationally weird (segment in transitional state forever) but no integrity issue.</li> </ul> <h3>Retry detection in restore()</h3> <p>On entry to <code>restore()</code>, before doing any work:</p> <pre> $existing_restored_event = lookup segment_restored event referencing this segment with id &gt; segment.live_purged_event_id; if ($existing_restored_event exists) { // Step 1 already happened. if (rows fully present in [from_id, to_id]) { // Steps 1 and 2 done; finalize Step 3. goto step_3 with $existing_restored_event; } else { // Step 2 partial or never ran. Re-run Steps 2 + 3. goto step_2 with $existing_restored_event; } } // No existing event - run Step 1. goto step_1; </pre><h3>Concurrency</h3> <ul> <li><strong>Concurrent logger writes during Step 2</strong>: proceed unblocked, land at chain head. Restored rows go to OLD ids in a disjoint window; no row-id conflict. Chain integrity preserved via <code>UNIQUE(chain, previous_hash)</code> and per-row HMAC.</li> <li><strong>Concurrent restore of the same segment</strong>: both reach Step 1, both emit segment_restored events under serialized lock. Now two segment_restored events exist for the same segment. Verifier rule: latest event id wins for supersession; older event is benign (segment row's live_purged_event_id will be set to the latest at Step 3). At Step 2 both transactions try the same row INSERTs; second hits PRIMARY KEY conflict, rolls back. First continues to Step 3. Acceptable but produces a benign extra chain event -- operators may want a "Step 0 check" rejecting restore if an in-flight event already exists.</li> <li><strong>Concurrent archive on the same chain different segment</strong>: archive's Phase B chain-event INSERT serializes with restore's Step 1 via the same chain-write lock. Both succeed sequentially.</li> </ul> <h3>Comparison with the alternatives considered</h3> <ul> <li><strong>Schema marker approach</strong> (originally proposed in this issue): per-segment <code>restore_in_progress</code> column. Cleaner from an operator-UI standpoint (admin can show "in progress" segments) but requires a schema migration, update hook, and verifier rule. This approach (Approach D) gets the same operational benefit without the schema cost.</li> <li><strong>Advisory locks</strong> (MySQL <code>GET_LOCK</code> / PostgreSQL <code>pg_advisory_lock</code>): keep restore's structure unchanged and swap the chain-write lock backend to use connection-scoped advisory locks instead of the semaphore table. Smaller code change but introduces a custom lock backend running in parallel with Drupal's standard one. Doesn't leverage the chain event mechanism that already exists.</li> <li><strong>Drop restore-as-re-insertion entirely</strong>: make archives read-only; operators view archive content without re-INSERTing into <code>audit_trail</code>. Larger product-level decision; out of scope for this technical issue.</li> </ul> <p>This proposal uses the existing chain-event mechanism as the in-progress marker. No schema change, no new lock backend, no new product model.</p> <h3>Related</h3> <ul> <li>#3591896 (parent: narrowed archiveSegment lock scope but deferred restore for this design work).</li> <li>#3591878 (grandparent: introduced the TTL-renewal workaround that this issue's resolution replaces for restore).</li> </ul> <h3>Remaining tasks</h3> <ul> <li>Refactor <code>restore()</code> and <code>restoreLocked()</code> into Step 1 / Step 2 / Step 3 with the new ordering (event first).</li> <li>Add retry-detection logic in <code>restore()</code> entry: detect existing segment_restored event and partial row presence; resume from the appropriate step.</li> <li>Verifier extension for the transitional state: accept rows present in [from_id, to_id] and stale segment.live_purged_event_id when a segment_restored event with greater id exists.</li> <li>Document the transitional state in <code>docs/architecture.md</code> and <code>docs/verification.md</code>.</li> <li>Delete <code>renewChainWriteLockIfStale()</code>, <code>CHAIN_WRITE_LOCK_TTL_S</code>, <code>CHAIN_WRITE_LOCK_RENEW_INTERVAL_S</code> -- no longer needed once restore narrows.</li> <li>Kernel tests: happy-path restore; Step 1 succeeds + Step 2 fails (retry resumes); Steps 1-2 succeed + Step 3 fails (retry finalizes); concurrent logger write during Step 2 (verifier accepts); concurrent same-segment restore (PK conflict handled, two benign events).</li> </ul> <h3>API changes</h3> <p>None for external callers.</p> <h3>Data model changes</h3> <p>None.</p>