Expand eval runner to test both guidance AND models?
>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3583203. --> Reported by: [webchick](https://www.drupal.org/user/24967) >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p>Over at <span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/ai_best_practices/-/work_items/3581977" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/ai_best_practices/-/work_items/3581977</a></span> folks are discussing an optimal AI development toolchain that uses F/OSS software and models that run locally on mid-range laptops.</p> <p>It would be ideal if the eval framework allowed us to not only compare/contrast before/after guidance but also how model X vs. model Y performs in those scenarios.</p> <h4 id="summary-steps-reproduce">Steps to reproduce</h4> <h3 id="summary-proposed-resolution">Proposed resolution</h3> <h3 id="summary-remaining-tasks">Remaining tasks</h3> <h3 id="summary-ui-changes">User interface changes</h3> <h3 id="summary-api-changes">API changes</h3> <h3 id="summary-data-model-changes">Data model changes</h3>
issue