Allow tool calling in streamed chat (#3528549) · Issues · project / ai

Allow tool calling in streamed chat

>>> [!note] Migrated issue   Reported by: [marcus_johansson](https://www.drupal.org/user/385947) Related to !787 >>> --- AI TRACKER METADATA --- Update Summary: Streaming chat with tools - done. Making implementation. Check-in Date: MM/DD/YYYY (US format) [When we should see progress/get an update] Due Date: MM/DD/YYYY (US format) [When the issue should be fully completed] Blocked by: [#XXXXXX] (New issues on new lines) Additional Collaborators: @username1, @username2 AI Tracker found here: <a href="https://www.drupalstarforge.ai/" title="AI Tracker">https://www.drupalstarforge.ai/</a> --- END METADATA --- <h3 id="summary-problem-motivation">Problem/Motivation</h3> Currently we only allow you to return the following data in the streaming interface: <ul> <li>Text</li> <li>Role</li> <li>Metadata</li> </ul> This is subject to change with <a href="https://www.drupal.org/project/ai/issues/3524435" title="Status: Closed (fixed)">#3524435: Add token usage to streamed chat</a> where token usage might be normalized as well. We also need to normalize the tools usage in the StreamedChatMessageInterface that can take the tools response and put it in a getTools and setTools method. Since the tool might not be formed yet and it makes no sense to call the tool or the normalizer before that and it should not be normalized at that stage. So what we should do is on the StreamedChatMessageIterator that is the abstract class, we should add a variable on the createStreamedChatMessage that adds the mixed $tool part and then adds this to a tool class. On the StreamedChatMessageIteratorInterface we should force a method called normalizeTool, where each provider has to have a method for normalizing that tool. This can be added to the abstract method, to not break providers and be forced in 2.0.0 This should also be sent to the PostStreamingResponseEvent so anyone that needs to get the tools after it streamed out the text, can get it. This is the steps of a streamed tools call, just for reference for a mental model: <pre>data: {   "choices": [     {       "delta": { "role": "assistant" },       "index": 0     }   ] } data: {   "choices": [     {       "delta": { "content": "Sure! Let me check..." },       "index": 0     }   ] } data: {   "choices": [     {       "delta": {         "tool_calls": [           {             "id": "tool_abc",             "type": "function",             "function": {               "name": "get_weather",               "arguments": "{"             }           }         ]       },       "index": 0     }   ] } data: {   "choices": [     {       "delta": {         "tool_calls": [           {             "function": {               "arguments": ""location":"Berlin"             }           }         ]       },       "index": 0     }   ] } data: {   "choices": [     {       "delta": {         "tool_calls": [           {             "function": {               "arguments": """             }           }         ]       },       "index": 0     }   ] } data: {   "choices": [     {       "delta": {},       "finish_reason": "tool_calls",       "index": 0     }   ] } data: [DONE]</pre><h4 id="summary-steps-reproduce">Steps to reproduce</h4> <h3 id="summary-proposed-resolution">Proposed resolution</h3> In StreamedChatMessageInterface/StreamedChatMessage add getTools and setTools and add it to the constructor. In StreamedChatMessageIterator add a variable called tools In StreamedChatMessageIterator add tools to the createStreamedChatMessage method and add those to the tools In StreamedChatMessageIteratorInterface add a method that is normalizeTools that each provider has to take care of to normalize the tool output according to how its already done. In StreamedChatMessageIterator add a default method that does nothing, just to not have breaking changes. This will be removed in 2.0.0. Add the tools to the PostStreamingResponseEvent, via the constructor. In the PostStreamingResponseEvent add a getTools and setTools to add the tools. > Related issue: [Issue #3528549](https://www.drupal.org/node/3528549)

issue