Add a way to run LLM regression test
>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3489630. --> Reported by: [marcus_johansson](https://www.drupal.org/user/385947) >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p>We have the possibility to test all the services/methods that agents uses using normal testing, but there is no way to see if changes to the prompts causes changes to a wide range of different type of prompts.</p> <p>We need someway to run a list of prompts with a list of expected results against a real provider to check if prompt changes did affect any type of prompt.</p> <p>What makes this harder is that you might want a state of the website before you run the prompts, since many prompts are editing prompts that requires the context to exist.</p> <h3 id="summary-proposed-resolution">Proposed resolution</h3> <p>Look at the module AI Agents Form Integration and the FieldTypeCreationForm. It can extract CSV and then run the prompt one after another.<br> Add a listener to the postevent to check the json against the wanted json.</p> <p>(or)</p> <p>We run the agenthelper with the runSubAgent function of a prompt, however this will be hard or unneccesary to do until #3487696 is done, since its hard coded context right now as well.<br> If that ran, one would get the decoded array back which would be fairly easy to verify.</p>
issue