Issue #3586383: Pre-1.0 context selection reliability and diagnostics follow-up.

Kristen Polrequested to merge
issue/ai_context-3586383:3586383-context-selection-reliability into 1.0.x

Description

Follow-up to:

Assess possible 1.0 and 1.1 features and UX/DX ... (#3586363 - closed) • Unassigned • 1.0.0-beta4

Implements packages #3, #4, and #9: clearer limit semantics in UI/docs, standardized conditional-subcontext failure logging, and a privacy-safe debug selection summary.

#3 — Truncation semantics (UI + docs)

Global, Always include, and required subcontext are described as priority, not guaranteed delivery. Help text and docs now state that item and token limits can shorten or omit content, and that these settings should not be the only control for compliance-critical instructions.

UI: general settings, context item settings, agent config (including advanced limit fields), Global scope plugin, required subcontext entity labels.

Docs: configuration.md, context_selection.md, scopes.md, agent_configuration.md, subcontext.md.

Tests: existing always-include item cap (testSelectorCapsAlwaysIncludeItemsAtMaxItems); new forced-truncation kernel test covering required + always-include token truncation.

#4 — Conditional subcontext failure visibility

AiContextSubcontextResolver emits one standardized warning per failure:

AiContextSubcontextResolver: Conditional children excluded; parent=PARENT; affected=COUNT; reason=REASON.

Reason categories: missing_provider, decision_cap_exhausted, invalid_response, unsupported_provider, provider_exception.

Warnings omit context content, task text, provider responses, and exception messages. Settings form explains fail-closed fallback. Documented in docs/developers/debugging.md.

Tests: log assertions for all failure categories in AiContextSubcontextResolverTest; test provider extended for unsupported-operation and exception scenarios.

#9 — Debug selection summary

When debug logging is enabled, AiContextSelector emits one debug summary per completed selection:

AiContextSelector: Selection summary; consumer=CONSUMER; selected=[IDS]; truncated=[IDS]; tokens=USED/MAX.

Removes prior task-text debug log and always-on info render summary. Token-limit truncation warnings remain at warning level (unchanged).

Tests: testDebugSummaryReportsForcedTruncation in AiContextSelectorEventsTest.

Out of scope

No structured diagnostics UI/API (#3586242). Broader scope-form intro copy is left to #3586348; Global scope plugin wording was updated here and may need coordination if both land close together.

Post-merge

None. Cache rebuild only if upgrading from an older checkout without ./lint.sh.

Testing instructions

Automated

From web/modules/contrib/ai_context:

./lint.sh
vendor/bin/phpunit tests/src/Kernel/AiContextSelectorEventsTest.php
vendor/bin/phpunit tests/src/Kernel/AiContextSubcontextResolverTest.php
vendor/bin/phpunit tests/src/Unit/AiContextScopeGlobalTest.php
vendor/bin/phpunit tests/src/Functional/AiContextAgentSettingsTest.php
vendor/bin/phpunit tests/src/Functional/AiContextDebugLoggingTest.php

Or with DDEV from the site root:

ddev exec vendor/bin/phpunit web/modules/contrib/ai_context/tests/src/Kernel/AiContextSelectorEventsTest.php
ddev exec vendor/bin/phpunit web/modules/contrib/ai_context/tests/src/Kernel/AiContextSubcontextResolverTest.php

Manual — limit help text

  1. Log in as a user with Administer AI Context.
  2. Open Configuration → AI → Context Control Center → General settings (/admin/config/ai/context/settings/general).
  3. Confirm Max context items, Max tokens, and Enable debug logging descriptions mention that Global, Always include, and required subcontext items can still be shortened or omitted.
  4. Open Context item settings (/admin/config/ai/context/settings/items).
  5. Confirm Enable conditional subcontext AI decisions explains fail-closed behavior (conditional excluded, required preserved; token limit may still shorten required content).
  6. Open an agent’s Configure AI Context Items tab.
  7. Confirm Global / Always include help uses “priority” language and the note warns that priority does not guarantee delivery.
  8. Expand Advanced settings and confirm Max context items and Max tokens descriptions mention always-include competition and required-subcontext truncation.

Manual — debug selection summary

  1. On general settings, enable Enable debug logging and save.
  2. Create or use published context items; set a low Max tokens for injection (e.g. 30).
  3. Trigger a context selection (agent run or drush eval against ai_context.selector).
  4. Check logs: drush watchdog:show --type=ai_context --count=20
  5. Confirm one line matching AiContextSelector: Selection summary; consumer=...; selected=[...]; truncated=[...]; tokens=.../....
  6. Confirm task text does not appear in ai_context log entries.

Manual — conditional failure warnings

  1. Enable subcontext and conditional AI decisions; configure a parent with conditional children.
  2. Clear provider/model (or use an invalid provider ID) and trigger selection.
  3. Confirm a warning: AiContextSubcontextResolver: Conditional children excluded; parent=...; affected=...; reason=missing_provider.
  4. Confirm the log contains no context body, task text, or exception message.

Checklist

  • I have updated the MR title to use format: Issue #1234: My issue title.
  • I have updated the MR description to include: Closes #1234
  • I have performed a self-review of my own code
  • I have added or updated tests, or explained in the description why this change is not covered by tests
  • I have updated documentation for any new or changed functionality
  • I have written testing instructions and verified them locally
  • I have noted any required post-merge steps (config imports, cache rebuilds, manual changes)
  • This MR contains no breaking API or hook changes, or they are explicitly documented in the description

AI Compliance

Note

Check the one that best describes your usage, or leave all unchecked if AI was not significantly used.

  • AI Assisted Code
    Mainly written by a human; AI used for autocomplete or partial generation under full human supervision.

  • AI Generated Code
    Mainly generated by AI, reviewed and approved by a human before this MR was created.

  • Vibe Coded
    Generated by AI and only functionally reviewed before this MR was created.

Closes #3586383

Edited by Kristen Pol

Merge request reports