Create an eval framework to determine if guidance updates are making things better or worse
>>> [!note] Migrated issue
<!-- Drupal.org comment -->
<!-- Migrated from issue #3581832. -->
Reported by: [webchick](https://www.drupal.org/user/24967)
Related to !1
>>>
<h3 id="summary-problem-motivation">Problem/Motivation</h3>
<p>Create an <a href="https://hamel.dev/blog/posts/evals/">eval framework</a> (a test suite that measures whether AI outputs are getting better or worse over time) that allows us to objectively measure whether changes to guidance in this repo are making AI-assisted Drupal contribution better or worse — and to compare competing guidance approaches as raised in <span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/ai_best_practices/-/work_items/3581683" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/ai_best_practices/-/work_items/3581683</a></span>.</p>
<h3 id="summary-proposed-resolution">Proposed resolution</h3>
<p>(vigorous handwaving :D — what follows is what Claude recommends, based on Anthropic's <a href="https://github.com/anthropics/skills/blob/main/skills/skill-creator/SKILL.md">skill-creator</a> skill.)</p>
<p><strong>Structure</strong><br>
The repo will follow the conventions established by the <a href="https://skills.sh/">skills.sh</a> ecosystem, which has emerged as the de facto standard across agents (Claude Code, Codex, Gemini CLI, Cursor, and others):</p>
<pre>skills/<br> hook-implementations/<br> SKILL.md<br> references/<br> issue-creation/<br> SKILL.md<br><br>evals/<br> evals.json ← follows the anthropics/skills schema<br><br>examples/ ← canonical good/bad examples, referenced by both</pre><p><strong>Tooling</strong><br>
The framework plugs into the <a href="https://github.com/anthropics/skills">anthropics/skills</a> ecosystem tooling (MIT licensed, open source), which provides an eval runner, benchmark aggregator, and visual HTML review tool. The <code>evals.json</code> format and all Python scripts are usable by anyone locally with no model dependency — no Claude account required.</p>
<p>Contributors using Claude Code additionally get an interactive workflow via the <code>skill-creator</code> skill, which guides you conversationally through writing skills and evals, runs parallel test cases, and auto-iterates skill descriptions. Think of it as the difference between writing composer.json by hand vs. running composer require — same destination, smoother ride.</p>
<p><strong>What gets tested</strong><br>
Evals will cover the full range of contribution tasks, not just code generation — including issue writing, patch review, commit messages, documentation, and triage. Each task type uses the appropriate scorer: deterministic checks (PHPCS, PHPStan) for mechanical correctness, and LLM-as-judge for Drupal-idiomatic quality.</p>
<h3 id="summary-remaining-tasks">Remaining tasks</h3>
<ol>
<li> Read skill-creator/references/schemas.md to confirm the canonical evals.json format</li>
<li> Write first skill (e.g. hook-implementations or issue-creation)</li>
<li> Write first eval case against that skill</li>
<li> Run via skill-creator tooling and confirm the workflow end-to-end</li>
<li> Wire into CI so PRs to this repo report score deltas</li>
</ol>
issue