Explore provider-agnostic eval runner
>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3583202. --> Reported by: [webchick](https://www.drupal.org/user/24967) Related to !12 >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p><span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/ai_best_practices/-/work_items/3582953" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/ai_best_practices/-/work_items/3582953</a></span> introduces an <code>evals/compare.py</code> script that compares before/after on various agent guidance rules.</p> <p>However, the current implementation is locked to Claude Code. Given Anthropic is currently making a bunch of changes to pricing and usage limits in real-time to deal with third-party tool usage from e.g. OpenClaw: <a href="https://x.com/bcherny/status/2040206440556826908">https://x.com/bcherny/status/2040206440556826908</a> it is probably better for our tools to remain provider-agnostic so developers have the freedom to switch from Claude to Codex for example.</p> <p>(See also this Slack discussion where Drupal folk are discussing swapping back and forth based on capabilities / pricing / usage limits <a href="https://drupal.slack.com/archives/C0APH70JV18/p1775230385265339?thread_ts=1775221900.663669&amp;cid=C0APH70JV18">https://drupal.slack.com/archives/C0APH70JV18/p1775230385265339?thread_ts=1775221900.663669&amp;cid=C0APH70JV18</a> &mdash; this is a common pattern around developers using AI for development.)</p> <h4 id="summary-steps-reproduce">Steps to reproduce</h4> <h3 id="summary-proposed-resolution">Proposed resolution</h3> <h3 id="summary-remaining-tasks">Remaining tasks</h3> <h3 id="summary-ui-changes">User interface changes</h3> <h3 id="summary-api-changes">API changes</h3> <h3 id="summary-data-model-changes">Data model changes</h3>
issue