>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3583213. --> Reported by: [alex ua](https://www.drupal.org/user/110386) Related to !11 >>> <h3>Problem/Motivation</h3> <p>Last week an agent told a contributor that cache tags are invalidated synchronously &mdash; that Drupal eagerly deletes stale items on invalidation. The contributor corrected it: invalidation is two-phase, checksum update is immediate but item eviction is lazy on next read. Today a different contributor asked the same question and got the same wrong answer from a fresh session. <strong>Expert corrections currently die with the session.</strong></p> <p>We already have excellent infrastructure for <em>measuring</em> guidance quality (the eval framework from <a href="https://www.drupal.org/project/ai_best_practices/issues/3581832">#3581832</a>) and for <em>enforcing</em> deterministic rules (hooks, per <a href="https://www.drupal.org/project/ai_best_practices/issues/3583109">#3583109</a>). What's missing is the <strong>runtime capture layer</strong> &mdash; a standardized way to turn a human expert's live correction into a durable improvement: an updated skill, a new eval case, a revised guidance section (AGENTS.md / CLAUDE.md / GEMINI.md), or a new hook.</p> <h3>Proposed Resolution</h3> <p>Create a <code>capturing-expert-corrections</code> skill (following the project's gerund-based naming convention from <a href="https://www.drupal.org/project/ai_best_practices/issues/3581836">#3581836</a>) that defines a <strong>correction-to-improvement pipeline</strong> with four steps:</p> <h4>Step 1 &mdash; Capture</h4> <p>When a knowledgeable user corrects a factual claim, the agent logs a structured entry with:</p> <ul> <li>What was claimed (incorrect)</li> <li>What is correct (the user's correction)</li> <li>Which Drupal subsystem is affected (caching, entity, config, routing, etc.)</li> <li>A failure classification (see below)</li> <li>The <strong>reasoning chain that failed</strong> &mdash; not just <em>what</em> was wrong, but <em>why</em> the agent reasoned incorrectly (e.g., "assumed invalidation means deletion because most cache systems work that way, but Drupal separates checksum marking from item eviction"). This transforms the log from a fact-correction ledger into a reasoning document that shifts model behavior in future sessions.</li> </ul> <p>Log format: append-only JSONL so corrections accumulate and can be analyzed for patterns.</p> <h4>Step 2 &mdash; Classify</h4> <p>Each correction is tagged with <em>why</em> the agent got it wrong. Proposed taxonomy:</p> <table> <thead> <tr> <th>Code</th> <th>Failure Source</th> <th>Example</th> </tr> </thead> <tbody> <tr> <td><code>SKILL_GAP</code></td> <td>Guidance missing from skill</td> <td>No skill covers cache tag invalidation model</td> </tr> <tr> <td><code>SKILL_STALE</code></td> <td>Skill has outdated info</td> <td>Skill still recommends a superseded Drupal migration pattern after the current core API changed</td> </tr> <tr> <td><code>EVAL_GAP</code></td> <td>No eval catches this error</td> <td>Agent writes Drupal 7-style code and nothing flags it</td> </tr> <tr> <td><code>HOOK_CANDIDATE</code></td> <td>Deterministic rule not enforced</td> <td>phpcs violation that should be caught by a hook</td> </tr> <tr> <td><code>CONFABULATION</code></td> <td>Model hallucinated plausible details</td> <td>Agent invented a non-existent Drupal API method</td> </tr> <tr> <td><code>ASSUMPTION_ERROR</code></td> <td>Correct facts, wrong inference</td> <td>Agent knew about lazy builders but misapplied them</td> </tr> </tbody> </table> <h4>Step 3 &mdash; Remediate</h4> <p>Based on the classification, the agent proposes a fix in priority order:</p> <ol> <li><strong>Patch the skill</strong> (highest impact &mdash; fixes all future sessions immediately)</li> <li><strong>Add an eval case</strong> (prevents regression &mdash; ties into the eval framework from #3581832)</li> <li><strong>Propose a hook</strong> (if the rule is deterministic &mdash; ties into #3583109)</li> <li><strong>File an issue</strong> (if the fix needs community discussion)</li> </ol> <h4>Step 4 &mdash; Verify</h4> <p>Run the eval framework to confirm the fix actually improves output quality. This closes the loop: correction &rarr; fix &rarr; eval passes.</p> <h3>Escalation Rule</h3> <p>When <strong>3+ corrections</strong> target the same subsystem or failure type, escalate: propose a dedicated skill section, a new eval suite, or a "Known Pitfalls" block within the relevant guidance.</p> <h3>Cross-Tool Compatibility</h3> <p>The correction log is plain JSONL &mdash; agent-agnostic. Corrections captured by a Claude user benefit a Codex or Gemini user who encounters the same subsystem later, and vice versa. <strong>The log is a community asset, not a per-tool artifact.</strong> The workflow itself is described in a <code>SKILL.md</code> that works with Claude, Codex, or local models (per <a href="https://www.drupal.org/project/ai_best_practices/issues/3581977">#3581977</a>). The only tool-specific piece is where the log lives (e.g., <code>.kittify/corrections/</code> vs a project-level <code>corrections/</code> directory).</p> <h3>Integration with Existing Issues</h3> <ul> <li><span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/ai_best_practices/-/work_items/3581832" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/ai_best_practices/-/work_items/3581832</a></span> &mdash; corrections feed directly into Scenario 3 ("catching regressions") as raw material for new eval cases</li> <li><span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/ai_best_practices/-/work_items/3583109" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/ai_best_practices/-/work_items/3583109</a></span> &mdash; deterministic corrections can be graduated into hooks</li> <li><span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/ai_best_practices/-/work_items/3582953" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/ai_best_practices/-/work_items/3582953</a></span> &mdash; the correction log provides worked examples for the "catching new problems" documentation scenario</li> <li><span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/ai_best_practices/-/work_items/3581836" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/ai_best_practices/-/work_items/3581836</a></span> &mdash; skill would live at <code>skills/capturing-expert-corrections/SKILL.md</code></li> </ul> <h3>Remaining Tasks</h3> <ol> <li>Draft the <code>capturing-expert-corrections/SKILL.md</code> with the pipeline definition</li> <li>Define the JSONL schema for the correction log</li> <li>Write at least 3 worked examples (one per failure classification)</li> <li>Create initial eval cases that validate the skill itself</li> <li>Document how corrections flow into the eval framework (Scenario 3 bridge)</li> </ol>