Issue #3586379: Persist token counts and show budget usage in CCC UI.

Kristen Polrequested to merge
issue/ai_context-3586379:3586379-save-token-counts into 1.0.x

Description

Persists estimated token counts on context items and surfaces budget-relative usage in the CCC admin UI.

Token estimates use Drupal AI's @ai.tokenizer service (which uses yethee/tiktoken under the hood). The ai_context module does not declare a direct Composer dependency on tiktoken.

Persist token counts

  • Adds a translatable, revisionable token_count base field on ai_context_item, populated in preSave() via ai_context.token_estimator.
  • Re-estimates only when the translation's content changed (status-only or label-only saves keep the stored count).
  • Recalculates token_count for every translation on save.
  • Adds ai_context_update_10015() to install the field and batch-backfill counts for all items and translations. The batch skips items deleted mid-run.

Token estimator and model-change recount

  • Wires AiContextTokenEstimator to @ai.tokenizer (model from ai_context.settings:provider_config.model, fallback gpt-4).
  • Replaces fixed 1K/3K listing tiers with budget-relative tiers:
    • low: < 10% of budget
    • mid: 10%–19.99%
    • high: ≥ 20%
  • Tooltip percentages use floor() so they align with tier thresholds (e.g. 199/2000 → 9%, not 10%).
  • When the resolved tokenizer model changes, stored counts are backfilled via batch (web) or synchronously (CLI). Triggers include General settings form save, direct ai_context.settings config save, and configuration import (AiContextSettingsConfigSubscriber). Import defers backfill until request terminate so entity saves run after sync completes.

Context item listing

  • Tokens column reads stored getTokenCount() instead of re-estimating content on each row.
  • Badges and tooltips use budget-relative tiers and show percentage of the site token budget.
  • Compact badge labels show raw counts below 1,000 (e.g. 17, 120) and K notation from 1,000 upward (e.g. 2K, 2–12K for parent ranges).

Renderer budget enforcement

  • Enforces max_tokens against the fully assembled output (blocks, separators, trailing newline), not per-block sums.
  • Uses token-aware snippet fitting via TextChunker when a provider model is configured (replaces char/4 approximation).

Agent subscription budget summary

  • Adds AiContextSubscriptionBudgetCalculator, AiContextSubscriptionBudgetSummary, and AiContextAgentBudgetSummaryLazyBuilder.
  • Agent settings form shows a budget summary at the top (within, range for conditional subcontext, or over), loaded via a lazy builder at render time.
  • Summary totals use stored guidance content only. The form note explains that rendered context also adds metadata and separators, so runtime usage is usually higher.
  • Calculator mirrors runtime selection rules:
    • Excludes never_include children from totals.
    • When conditional subcontext AI is disabled, conditional children are excluded unless always_include.
    • Always-included conditional children count in the low bound, not only the high bound.
    • match_all with empty scope subscriptions includes all published items (respecting never_include).
    • minimal with empty subscriptions includes globals, unindexed items, target-entity matches, and non-subscription indexed scopes (parity with selector candidates).
    • Excludes inheriting children when their parent is not in the budget candidate set (MINIMAL parity).
  • Caches summaries per agent configuration, module settings, scope-plugin configuration, and request context (scope context values, target entity, and request path) for one hour. Cache tags include ai_context_item_list, config:ai_context.settings, config:ai_context.agents, and each config:ai_context.scope_settings.* object. The lazy builder varies by target entity, content language, and URL path.

Services

  • ai_context.token_estimator — tokenizer injection and budget tiers
  • ai_context.token_count_backfill_scheduler — batch/sync backfill when the tokenizer model changes
  • ai_context.settings_config_subscriber — config save/import hooks for model-change recount
  • ai_context.subscription_budget — agent budget calculator (cached)
  • ai_context.agent_budget_summary_lazy_builder — agent form budget summary lazy builder

No breaking API or hook changes.

Closes #3586379

Testing instructions

Automated

From the Drupal project root (with DDEV running):

cd web/modules/contrib/ai_context
./lint.sh
cd ../../../..
ddev phpunit web/modules/contrib/ai_context/tests/src/Kernel/AiContextTokenEstimatorTest.php
ddev phpunit web/modules/contrib/ai_context/tests/src/Kernel/AiContextUpdate10015Test.php
ddev phpunit web/modules/contrib/ai_context/tests/src/Kernel/AiContextSubscriptionBudgetCalculatorTest.php
ddev phpunit web/modules/contrib/ai_context/tests/src/Kernel/AiContextSubscriptionBudgetMinimalParityTest.php
ddev phpunit web/modules/contrib/ai_context/tests/src/Kernel/AiContextRendererTest.php
ddev phpunit web/modules/contrib/ai_context/tests/src/Kernel/AiContextSelectorEventsTest.php
ddev phpunit web/modules/contrib/ai_context/tests/src/Kernel/AiContextItemTranslationTest.php --filter TokenCount
ddev phpunit web/modules/contrib/ai_context/tests/src/Kernel/AiContextSettingsConfigValidationTest.php --filter TokenRecount
ddev phpunit web/modules/contrib/ai_context/tests/src/Kernel/AiContextSettingsConfigImportValidationTest.php --filter ImportRecalculates
ddev phpunit web/modules/contrib/ai_context/tests/src/Functional/AiContextItemListTest.php --filter testTokensColumn
ddev phpunit web/modules/contrib/ai_context/tests/src/Functional/AiContextAgentSettingsTest.php --filter testAgentContextBudgetSummary

Coverage map

Area Primary tests
Token tiers, persistence, content-unchanged save, compact labels AiContextTokenEstimatorTest
Update backfill, deleted-item skip, model in batch AiContextUpdate10015Test
Budget summary rules (within/over/range, match_all) AiContextSubscriptionBudgetCalculatorTest
MINIMAL parity, cache invalidation, lazy builder, render overhead note AiContextSubscriptionBudgetMinimalParityTest
Assembled-output budget, token-aware truncation AiContextRendererTest
TEXT_RENDERED token recount alignment AiContextSelectorEventsTest
Multilingual token_count AiContextItemTranslationTest
Config save recount AiContextSettingsConfigValidationTest
Config import recount AiContextSettingsConfigImportValidationTest
Listing badges AiContextItemListTest::testTokensColumn
Agent form budget UI AiContextAgentSettingsTest::testAgentContextBudgetSummary

Manual

  1. Check out this branch and run ddev drush updb -y && ddev drush cr so update 10015 backfills token_count.
  2. Context listing (/admin/config/ai/context/items): confirm the Tokens column shows compact counts (e.g. 17, 120, 2K, 2–12K) with color badges; hover for tooltip with token count and budget percentage.
  3. General settings (/admin/config/ai/context/settings/general): note the site Maximum tokens value (install default 1200). Change the AI provider/model and save; confirm a batch recount runs when items exist.
  4. Agent settings (/admin/config/ai/context/agents/{agent}): confirm the Context budget summary and the note that counts use stored content only:
    • Within budget for small subscriptions
    • Range message when a subscribed parent has conditional subcontext children
    • Exceeded message when subscribed tokens exceed the agent/site budget
  5. Edit and save a context item; confirm token_count updates when content changes and stays the same when only status/label changes.
  6. Optional: set Conditional subcontext off at /admin/config/ai/context/settings/items and confirm agent budget no longer treats conditional children as part of the high bound unless always-included.
  7. Optional accessibility: confirm listing badge tooltips are exposed to assistive technology and the agent budget summary is readable in the tab order.

Checklist

  • I have updated the MR title to use format: Issue #1234: My issue title.
  • I have updated the MR description to include: Closes #1234
  • I have performed a self-review of my own code
  • I have added or updated tests, or explained in the description why this change is not covered by tests
  • I have updated documentation for any new or changed functionality or explained why they aren't
  • I have written testing instructions and verified them locally
  • I have noted any required post-merge steps (config imports, cache rebuilds, manual changes)
  • This MR contains no breaking API or hook changes, or they are explicitly documented in the description

Post-merge: sites upgrading from before this MR must run drush updb (update 10015) and rebuild caches. No config import required.

If system.schema for ai_context is already 10015 but the token_count column is missing (partial upgrade), reset the schema to 10014 and re-run drush updb:

drush php:eval "\Drupal::keyValue('system.schema')->set('ai_context', 10014);"
drush updb -y
drush cr

Documentation: docs/developers/services.md documents token_estimator, subscription_budget, model-change backfill, caching, lazy builder, and content-vs-render overhead. docs/developers/api-stability.md notes the internal services.

AI Compliance

Note

Check the one that best describes your usage, or leave all unchecked if AI was not significantly used.

  • AI Assisted Code
    Mainly written by a human; AI used for autocomplete or partial generation under full human supervision.

  • AI Generated Code
    Mainly generated by AI, reviewed and approved by a human before this MR was created.

  • Vibe Coded
    Generated by AI and only functionally reviewed before this MR was created.

Edited by Kristen Pol

Merge request reports