>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3604255. --> Reported by: [codeitwisely](https://www.drupal.org/user/1210320) >>> <h3>Problem/Motivation</h3> <p> <code>ai_metering_usage</code> has three dedicated integer columns for token counts:<br> <code>input_tokens</code>, <code>output_tokens</code>, and <code>cached_tokens</code>.<br> Each time a provider introduces a new token type &mdash; reasoning tokens for extended<br> thinking, audio tokens, image tokens &mdash; adding it to the schema requires a new<br> <code>hook_update_N()</code>, a new column in <code>hook_schema()</code>, and<br> changes to every place that reads or writes usage records. </p> <p> This pattern will not scale. drupal/ai is adding providers and capabilities quickly<br> and token type proliferation is expected. Raising the schema version for each new<br> type creates upgrade friction for site owners and makes the data model increasingly<br> rigid. Marcus Johansson noted this explicitly in his review of the module. </p> <h3>Steps to reproduce</h3> <ol> <li>Enable extended thinking on an Anthropic call that supports it.</li> <li>Make an AI call and let it complete.</li> <li>Inspect the <code>ai_metering_usage</code> row &mdash; there is no <code>reasoning_tokens</code> column; the reasoning token count returned by the provider is silently discarded.</li> </ol> <h3>Environment</h3> <ul> <li>Drupal version: 11.3.11</li> <li>ai_metering version: 1.0.0-alpha1</li> <li>AI provider module + version: ai_provider_anthropic 1.2.2 (extended thinking enabled)</li> <li>AI feature module tested: N/A</li> </ul> <h3>Proposed resolution</h3> <p> Add a <code>token_details</code> column of type text (JSON serialised) to<br> <code>ai_metering_usage</code>. The three existing columns<br> (<code>input_tokens</code>, <code>output_tokens</code>, <code>cached_tokens</code>)<br> remain unchanged for backward compatibility and existing dashboard queries.<br> Any additional token type reported by the provider &mdash; for example<br> <code>reasoning_tokens</code> or <code>audio_tokens</code> &mdash; is written into<br> <code>token_details</code> without a schema change. </p> <p> <code>AiPostGenerateSubscriber</code> reads the full usage object from the<br> provider and writes any key it does not recognise into <code>token_details</code>.<br> A helper method returns all token types merged from both the columns and<br> <code>token_details</code>, so the dashboard and Drush commands can surface extras<br> without knowing column names in advance. </p> <h3>Remaining tasks</h3> <ul> <li>Add <code>token_details</code> text/JSON column to <code>ai_metering_usage</code> via <code>hook_update_N()</code> and <code>hook_schema()</code>.</li> <li>Update <code>AiPostGenerateSubscriber</code> to write unrecognised token types into <code>token_details</code>.</li> <li>Add a helper method that returns all token types merged from columns and <code>token_details</code>.</li> <li>Update dashboard and Drush output to surface extra token types when present.</li> <li>Add a unit test covering at least one extra token type round-trip.</li> <li>Review.</li> </ul> <h3>User interface changes</h3> <p>Yes &mdash; the dashboard and AI Usage Log surface additional token types (e.g. reasoning_tokens) when they are present in the data.</p> <h3>API changes</h3> <p>N/A</p> <h3>Data model changes</h3> <p>Yes, a <code>token_details</code> text column (JSON) is added to <code>ai_metering_usage</code> via <code>hook_update_N()</code>. Existing rows have NULL; no data is lost.</p> <h3>AI assistance</h3> <p>N/A</p>