>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3556378. --> Reported by: [marcus_johansson](https://www.drupal.org/user/385947) Related to !1245 >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p>Currently we have a flag for providers that offers the support of running the mix of tools and structured output, see <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-7"><a href="https://www.drupal.org/project/ai/issues/3550163" title="Status: Closed (fixed)">#3550163: Add new flag ChatCombinedToolsAndStructure</a></span>.</p> <p>However it is a known problem, that some providers or models does not allow for this. This means that an agent will never be able to return a structured output, since its using tools.</p> <p>With the Tool API and its outputted structure and the cool possibilities with tools like FlowDrop or ECA, we need to open up the possibility for the agent to return a known structure in the workflow/automation.</p> <p>We need a helper overlayed LLM for this, for the LLM's that does not handle this.</p> <p>Currently for instance when you setup an agent, you can set in the system prompt that the agent should answer the weather in a specific city and return the city name, the temperature and if its raining. You may also set a JSON Schema the agent should follow.</p> <p>With OpenAI for instance, we can just loop and use the tools to get the weather information and then the final text answer will follow the JSON schema.</p> <p>However for Anthropic this would fail, so what we will do when the flag is not set, we will let Anthropic answer in a textual format, something like "In Palermo its 30 degrees celsius and its not raining" and then we will use another LLM call that takes this as input and the JSON Schema and then returns the structured output.</p> <h3 id="summary-proposed-resolution">Proposed resolution</h3> <ul> <li>Add in the output service of the agent, that if the agent does not support structured output with tools, that it loads a separate service that takes the output as the input and the json schema and outputs it. Make sure to follow the streaming settings.</li> </ul>