Standardize thread/caller correlation on AI OTel spans (map tags to gen_ai.conversation.id)
## Problem
Since project/ai_agents#3574487 landed, everything needed to reconstruct the full trail of a
conversation (user, agent, and sub-agent calls) is present on the provider events.
But the only way a consumer can get at it is by parsing magic prefixes out of the
free-form `tags` string array on the span:
| tag prefix | stamped by | meaning |
|---|---|---|
| `ai_assistant_thread_*` | `ai_assistant_api` (`AiAssistantApiRunner`) | assistant conversation, shared across turns |
| `ai_agents_runner_*` | `ai_agents` (`AgentHelper`) | this agent run |
| `ai_agents_caller_runner_*` | `ai_agents` since #3574487 (merged 2026-07-02) | the parent (caller) agent's runner |
| `ai_agents_thread_*` | `ai_agents` since #3574487 | agent thread, identical for parent and sub-agent calls |
There are also the ad-hoc span attributes `provider_request_id` and
`provider_request_parent_id` (from `requestThreadId` / `requestParentId`).
Every consumer that wants to show a trail has to parse these prefixes itself and
rebuild the same grouping logic. That applies to a Grafana dashboard, the
ai_agents_debugger UI, an OTLP importer, and any external APM. The prefixes are an
undocumented convention with no owner, so consumers drift: #3574487 shipped a new
prefix that no existing consumer reads yet.
The OTel GenAI semantic conventions already define attributes for exactly this.
`gen_ai.conversation.id` is "the unique identifier for a conversation (session,
thread), used to store and correlate messages", and `gen_ai.agent.id` /
`gen_ai.agent.name` identify the agent.
#3586479 standardized what a call was (provider, model, tokens, finish reasons).
This issue standardizes how calls relate to each other.
## Proposed approach: additive, dual-emit, same pattern as #3586479
In `AiOtelSpansEventSubscriber`, parse the well-known tag prefixes once at emission
time and emit real attributes next to the raw `tags` array. The `tags` array stays,
as documented legacy.
| source | attribute | note |
|---|---|---|
| `ai_assistant_thread_<id>` tag, else `ai_agents_thread_<id>` tag | `gen_ai.conversation.id` | precedence rule: the assistant thread wins when both are present, since the assistant orchestrates the agents; one conversation id per span |
| `ai_agents_runner_<id>` tag | `gen_ai.agent.id` | the agent run this call belongs to |
| `ai_agents_caller_runner_<id>` tag | a documented ad-hoc attribute, e.g. `drupal.ai.caller_runner_id` | no semconv equivalent exists; candidate for real OTel span parenting later (see out of scope) |
| `requestThreadId` / `requestParentId` | unchanged (`provider_request_id` / `provider_request_parent_id`) | already emitted; document them as part of this contract |
Also document the contract in one place, in the ai_observability docs: which module
stamps which prefix, what each one means, and which attribute it maps to. A new
prefix from another module then has a place to register instead of becoming the next
piece of folklore.
After this, consumers read one attribute set. Grafana can group on
`gen_ai.conversation.id` without custom parsing, ai_agents_debugger and importers
stop regexing string arrays, and the trail scotteuser reconstructed by hand in
#3574487 becomes a plain query.
## Out of scope (follow-ups, please don't expand this issue)
- What `ai_assistant_api` and `ai_agents` stamp stays untouched. This issue only adds
a parse-and-promote step at the OTel boundary.
- Real OTel span parenting (activating span context so sub-agent spans are true
children and `parent_span_id` reflects the caller). Probably the right long-term
fix, but it changes trace topology for existing dashboards, so it needs its own
issue.
- Message content capture (`gen_ai.input.messages` / `gen_ai.output.messages`) stays
deferred, same as in #3586479.
## Definition of done
- `gen_ai.conversation.id`, `gen_ai.agent.id`, and the documented caller attribute
are emitted next to the existing `tags` array when spans are enabled.
- The precedence rule (assistant over agent thread) is implemented and tested,
including the case where both tags are present.
- The tag-prefix contract table is in the ai_observability documentation.
- Kernel test coverage for the parse step: no tags, one tag, both thread tags,
caller present.
The tags themselves are aalin's and scotteuser's work in project/ai_agents#3574487. This
issue just gives them a standard shape that every surface can read.
---
AI-Generated: Yes (Used Claude Code to research the tag/attribute landscape and draft this issue; reviewed and edited by zorz.)
issue