Add artifacts to agents (#3528726) · Issues · project / ai_agents

Add artifacts to agents

>>> [!note] Migrated issue   Reported by: [marcus_johansson](https://www.drupal.org/user/385947) Related to !170 >>> --- AI TRACKER METADATA --- Update Summary: Will provide a single thing that AI Agents can work on so they aren't passing the full context every time they do everything to the next agent. This will reduce cost and improve accuracy. Check-in Date: MM/DD/YYYY (US format) [When we should see progress/get an update] Due Date: MM/DD/YYYY (US format) [When the issue should be fully completed] Blocked by: [#XXXXXX] (New issues on new lines) Additional Collaborators: @username1, @username2 AI Tracker found here: <a href="https://www.drupalstarforge.ai/" title="AI Tracker">https://www.drupalstarforge.ai/</a> --- END METADATA --- <h3 id="summary-problem-motivation">Problem/Motivation</h3> Currently we expose any output and any parameters as text in the chat history. This works with most agents, but there are use cases where you want to work with bigger data chunks, that only matter for specific tools and not the history of the chat. Think of the following scenario: You have a tool that can scrape websites and you have a tool that can extract links from a website and then a tool that can screenshot website. Your agents task is to scrape a webpage and take a screenshot of all the links on it. In reality the final history after all loops looks something like this (system prompt is something verbose about the above): <pre>User: Can you scrape https://drupal.org and take screenshots of all the external links --------------------------------------------------------------- Assistant: I will start by scraping the website Tool Usage: scrape(https://drupal.org) with tool_id 1 --------------------------------------------------------------- Tool: <html><head><title>Drupal.org</title></head><body>[Loads of HTML with links]</body></html> Tool_id: 1 --------------------------------------------------------------- Assistant: I will now extract the links Tool Usage: extract_links(<html><head><title>Drupal.org</title></head><body>[Loads of HTML with links]</body></html>) with tool_id 2 --------------------------------------------------------------- Tool: https://wordpress.org, https://joomla.org, https://dri.es Tool_id: 2 --------------------------------------------------------------- Assistant: I will now screenshot the links: Tool Usage: screenshot(https://wordpress.org, https://joomla.org, https://dri.es) with tool_id 3 --------------------------------------------------------------- Tool: file_id 1, 2, 3 Tool_id: 2 --------------------------------------------------------------- Assistant: I have tooken screenshots here they are for https://joomla.org <img src="url to 1">.....</pre>This will work, however the problem with this is that if a website is maybe around 50k tokens, you have had 3 loops of 50k tokens twice at a price of 300k tokens - something that can be measured close to ~1USD/EUR depending on your provider and some provider doesn't handle that limit even. And in this case the actual html is not even needed to be read by the LLM. What instead if we could do: <pre>User: Can you scrape https://drupal.org and take screenshots of all the external links --------------------------------------------------------------- Assistant: I will start by scraping the website Tool Usage: scrape(https://drupal.org) with tool_id 1 --------------------------------------------------------------- Tool: !artifact:1 Tool_id: 1 --------------------------------------------------------------- Assistant: I will now extract the links Tool Usage: extract_links(!artifact:1) with tool_id 2 --------------------------------------------------------------- Tool: https://wordpress.org, https://joomla.org, https://dri.es Tool_id: 2 --------------------------------------------------------------- Assistant: I will now screenshot the links: Tool Usage: screenshot(https://wordpress.org, https://joomla.org, https://dri.es) with tool_id 3 --------------------------------------------------------------- Tool: file_id 1, 2, 3 Tool_id: 2 --------------------------------------------------------------- Assistant: I have tooken screenshots here they are for https://joomla.org <img src="url to 1">.....</pre>All of a sudden we have the same results, but we saved around 300k tokens. <h3 id="summary-proposed-resolution">Proposed resolution</h3> <ul> <li>Create an artifact interface with a get and set value and id.</li> <li>Add a class with a service for it. Keep it only available for that run for now and connect it to the session of that run for security reasons. Any considerations that can already be made for making it persist in database or file system is good, but can be added later.</li> <li>Make sure that this is serializable/stringable since the agent loops are stateless and should be able to be stored into sessions/persistants.</li> <li>In the AiAgentForm for tools, make sure that we add a setting for the tools, that the output can be treated like an artifact.</li> <li>In the AiAgentEntityWrapper make sure to store any output as an artifact after the tool is run.</li> <li>In the AiAgentEntityWrapper make sure to replace any artifact with the real value - do not use AiDataTypeConverters, since they might want to run on the actual artifact.</li> <li> </ul> > Related issue: [Issue #3544182](https://www.drupal.org/node/3544182)

issue