Add a way to run LLM regression test
>>> [!note] Migrated issue
<!-- Drupal.org comment -->
<!-- Migrated from issue #3489630. -->
Reported by: [marcus_johansson](https://www.drupal.org/user/385947)
>>>
<h3 id="summary-problem-motivation">Problem/Motivation</h3>
<p>We have the possibility to test all the services/methods that agents uses using normal testing, but there is no way to see if changes to the prompts causes changes to a wide range of different type of prompts.</p>
<p>We need someway to run a list of prompts with a list of expected results against a real provider to check if prompt changes did affect any type of prompt.</p>
<p>What makes this harder is that you might want a state of the website before you run the prompts, since many prompts are editing prompts that requires the context to exist.</p>
<h3 id="summary-proposed-resolution">Proposed resolution</h3>
<p>Look at the module AI Agents Form Integration and the FieldTypeCreationForm. It can extract CSV and then run the prompt one after another.<br>
Add a listener to the postevent to check the json against the wanted json.</p>
<p>(or)</p>
<p>We run the agenthelper with the runSubAgent function of a prompt, however this will be hard or unneccesary to do until #3487696 is done, since its hard coded context right now as well.<br>
If that ran, one would get the decoded array back which would be fairly easy to verify.</p>
issue