## Summary AI Context counts tokens with the AI module tokenizer (`ai.tokenizer`, which uses yethee/tiktoken 0.5.1). On first use, the tokenizer downloads a vocab file from `openaipublic.blob.core.windows.net` and caches it in the Drupal temp directory. If it cannot load the vocab, it throws an exception. In AI Context, only the push subscribers catch it: - The item save fails with `EntityStorageException` (`AiContextItem::preSave()`, `src/Entity/AiContextItem.php:198-212`). The item form shows only "The changes could not be saved." and does not tell the editor why. - Context rendering fails. Agent tools that load context stop the agent run with an error. The push subscribers send no context, and nothing tells the user. CI job 13079129 (!358 merge train) shows the save failure (shortened trace): ```text Drupal\Core\Entity\EntityStorageException: The map of tokens and ranks has duplicates of rank Caused by: InvalidArgumentException vendor/yethee/tiktoken/src/Vocab/Vocab.php:47 vendor/yethee/tiktoken/src/Vocab/Loader/DefaultVocabLoader.php:89 web/modules/contrib/ai/src/Utility/Tokenizer.php:56 src/Service/AiContextTokenEstimator.php:372 src/Service/AiContextTokenEstimator.php:77 src/Entity/AiContextItem.php:208 ``` #3586660 fixed the CI side: the CI job now loads the vocab into the cache before the parallel test runs start. This issue is about production sites, where the same error stops editors and agents. Line numbers are for 1.0.x at commit 7089167. Later commits on 1.0.x changed some of these files, so some line numbers are now different. ### When the vocab cannot be loaded - **No outbound HTTPS** (firewall, proxy, or `allow_url_fopen = Off`): `RuntimeException: Could not open stream for URI`. The error continues until the site can load the vocab. - **CDN outage or a slow network on first use.** On container hosting, the temp directory can be empty after each deploy or restart. As a result, first use can occur often. - **The temp directory is not writable:** `RuntimeException: Directory is not writable`. - **Corrupt read from concurrent first use.** yethee/tiktoken 0.5.1 writes the cache file with `fopen($cacheFile, 'w+')` and no lock, and then reads the vocab back from that file (`DefaultVocabLoader.php:79-92`). When two requests download at the same time, one request can truncate the file while the other request writes it or reads it back. The result is `ParseError` or `InvalidArgumentException` (the CI trace above fails at `DefaultVocabLoader.php:89`). If the file on disk stays corrupt, the next request finds a wrong hash and downloads the file again. yethee/tiktoken 1.1.1 fixes this race (it writes to a temporary file and renames it), but all AI module branches, including 1.x, 1.6.x, and 2.x, require `^0.5.1`. ### Why the AI module fallback does not help `Tokenizer::setModel()` (`ai/src/Utility/Tokenizer.php:48-58`) catches the `getForModel()` exception. It then calls `get('cl100k_base')` without a catch. `gpt-4` (the AI Context default) and all models that tiktoken does not know use `cl100k_base`. For these models, the fallback loads the same file and throws again (`Tokenizer.php:56` in the trace). tiktoken does not remember a failed load. Each `setModel()` call tries the download again, up to two times, and each failed `fopen()` logs a PHP warning. If a firewall drops the packets, each attempt waits for `default_socket_timeout` (60 seconds by default). Today, the first exception stops the save or the render, so each one makes a maximum of two attempts. But one request can do many selections: the push subscribers catch the exception and continue, and an agent that is not loop-aware selects context again on each loop. A fix that only catches the exception would make this worse. The renderer calls the estimator and the chunker many times in each render, because it does a binary search for the snippet length. One render could then make many download attempts. Item 3 of the proposed resolution prevents this. ### What each caller does today when the tokenizer throws | Caller | Tokenizer use | Result today | | --- | --- | --- | | `AiContextItem::preSave()` (`AiContextItem.php:198-212`) | `estimate()` for each translation that is new, has changed content, or has a stored count of 0 | The save fails with `EntityStorageException`. The item form, the duplicate form, and the revision revert form catch it and show only "The changes could not be saved." Recipe and default content imports, scripts, and other programmatic saves fail with the exception. New translations also fail. | | `AiContextRenderer::render()` (`AiContextRenderer.php:86-88`, `112-194`, `316`) | `TextChunker::setModel()` first, then `estimate()` for budget checks and `chunkText()` for trimming | Throws. A catch in the estimator alone does not fix this, because `configureTextChunkerModel()` calls the shared tokenizer before the first `estimate()` call. | | `AiContextSelector::select()` (render at `:577`, `estimate()` at `:1710`) | The renderer, then `estimate()` on the final text | Throws. | | Agent system prompt push (`AiContextBuildSystemPromptSubscriber.php:133-139`) and chat push (`AiContextPreGenerateResponseSubscriber.php:112-124`) | `select()` | They catch `\Throwable`, log an error, and continue without context. The AI request works, but it gets no context. Each push logs one error. An agent that is not loop-aware logs one error on each loop. | | `GetRelevantAiContextItems` tool (`:140-141`) and `LoadAiContextItemById` tool (`:150`) | `select()` and `render()` | No catch. ai_agents catches only `ContextException` and `AiFunctionCallingExecutionError` around the tools that the model calls, and nothing around default information tools. The agent run stops with an error. | | `AiContextSelectionFactory::getResult()`, `getRenderedContext()`, `getResultFromSelection()` (`:453-462`) | `select()` | Throws to custom code. | | `AiContextTokenCountBackfillScheduler::backfillChunk()` (`:201`). `AiContextSettingsConfigSubscriber` starts it when the tokenizer model changes. | `estimate()` for each row | Cron queue worker: logs the error and tries again in a later cron run. Batch from the CCC or AI settings form: the batch stops with an error after the config is saved. CLI `backfillAll()` (`drush cset` or `drush cim` that changes the model): the command fails after the config is saved. Update 10023: `drush updb` fails for sites with context items that upgrade from a version before beta4. | | Listing token badges (`AiContextItemListBuilder`), `AiContextSubscriptionBudgetCalculator`, `AiContextConsumerBudgetSummaryLazyBuilder`, `ListAiContextItems` tool | No tokenizer call (only the stored `token_count`) | Not affected. They show the stored counts. | ### Steps to reproduce 1. Set `allow_url_fopen = Off` in the PHP configuration (for DDEV, add a file in `.ddev/php/`), then restart. 2. In the Drupal temp directory, delete the cached vocab file `9b5ad71b2ce5302211f9c61530b329a4922fc6a4`. This name is the SHA-1 of the `cl100k_base` URL. If the tokenizer model is a `gpt-4o` model, also delete `fb374d419588a4632f3f557e76b4b70aebbca790` (the `o200k_base` file). 3. Create a context item. 4. Result: the form shows "The changes could not be saved." and does not save the item. Recent log messages show `RuntimeException: Could not open stream for URI: https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken` (type `ai_context_item`), the warning "Saving ... was rejected: Could not open stream for URI: ..." (type `ai_context`), and PHP warnings from `fopen()`. ## Proposed resolution 1. [x] Make `AiContextTokenEstimator` the safe entry point to the tokenizer. Put `ensureTokenizerModel()` and `countTokens()` in a narrow `try` / `catch (\Throwable)` block. `\Throwable` is necessary because the AI tokenizer can also throw `\Error`: `Tokenizer::getTokens()` and `getEncodedChunks()` read the typed `$encoder` property, which is not set if no `setModel()` call has worked. The push subscribers already catch `\Throwable`. 2. [x] When the tokenizer fails, return a byte-based estimate: `(int) ceil(strlen($text) / 4)`. For ASCII text, this is the same as characters / 4, which `docs/developers/rag.md` already recommends. For multibyte text, it gives a higher count than characters / 4. It is not an upper bound: for some languages (for example CJK text), it can be lower than the real token count. 3. [x] Remember the failure for each resolved model, so that the estimator does not try the download on each call. Keep it for the rest of the request, and for 10 minutes across requests in `cache.default`, so that `drush cr` clears it at once after the problem is fixed. `resetTokenizerModelConfiguration()` also clears it. Add `isTokenizerAvailable(): bool` for callers that must not use the fallback. 4. [x] Inject `logger.channel.ai_context` and log one warning for each request, not one for each call. Include the model, the exception class, and the message. Do not log item content. Example: "Could not load the tokenizer vocabulary for model @model (@class: @message). AI Context uses an approximate token count until the vocabulary loads." 5. [x] `AiContextRenderer`: when `isTokenizerAvailable()` returns FALSE, do not call `configureTextChunkerModel()`. Trim content with `mb_strcut($content, 0, $max_tokens * 4)` instead of `TextChunker::chunkText()`. If a chunker call throws during a render, use the same fallback. This change also fixes the selector, the push subscribers, both tools, and the public selection API. 6. [x] Make sure that fallback counts do not stay in storage permanently, without a schema change: - `preSave()` stores the fallback count, so the save works and the badges do not show 0. It then tells the backfill scheduler to queue one full recount on the existing `ai_context_token_count_backfill` queue. A state flag makes sure that only one recount is in the queue until it runs. - `backfillChunk()` calls `isTokenizerAvailable()` first. If it returns FALSE, `backfillChunk()` writes nothing and throws a dedicated exception, so that a recount never replaces real counts with fallback counts. - Queue worker: change that exception to `SuspendQueueException`, so that cron releases the item and tries again on the next run. - Settings form batch: release the queue item, and show a warning that cron recalculates the counts when the vocabulary loads. - CLI `backfillAll()` and update 10023: log a warning, queue a full recount for cron, and finish. Then `drush cset`, `drush cim`, and `drush updb` do not fail. 7. [x] Add one kernel test class, for example `AiContextTokenizerFallbackTest`. Next to `AiContextTestTokenizer`, add a test tokenizer that counts its calls and throws `RuntimeException` from `setModel()` until the test turns the failure off. In `register()`, point `ai_context.token_estimator` and `ai.text_chunker` at it. Assert the following: - An item save works and stores `ceil(strlen / 4)`. Also do this for a translation with multibyte text. - Many `estimate()` calls and item saves in one request call `setModel()` one time and log exactly one warning (use a buffering test logger). - `render()` returns text inside the budget and lists the truncated items. - `GetRelevantAiContextItems` returns context and does not throw. - While the tokenizer fails, the queue worker throws `SuspendQueueException` and writes nothing. After the test turns the failure off, the queued recount replaces the fallback counts. 8. [x] Update the docs: - `docs/developers/services.md` (`ai_context.token_estimator` and `ai_context.renderer`): describe the vocab download on first use, the cache location (the Drupal temp directory, with the SHA-1 of the vocab URL as the file name), the fallback, and the warning. Correct "The model name is only used to select a local tokenizer encoding", because the encoding is downloaded on first use. - `docs/developers/configuration.md`, "Recounting after a model change": describe what occurs when the tokenizer is not available. - For sites without outbound access: allow `openaipublic.blob.core.windows.net`, or copy the vocab file into the temp directory. `TIKTOKEN_CACHE_DIR` has no effect, because the AI module sets the cache directory. Sites with a temp directory that is emptied on each deploy can set `$settings['file_temp_path']` to a persistent directory. - `tests/README.md` and `docs/contribute/testing.md`: describe the failing test tokenizer. 9. [x] `./lint.sh` passes. Not in this issue: the CI cache warm-up (#3586660). #3586544 (fixed) already moved most kernel tests to a test tokenizer. The 10 test classes that still use the real tokenizer stay as they are. ## Remaining tasks 1. [x] Decide: bytes / 4 or characters / 4 for the fallback. Decided: bytes / 4. Neither is an upper bound for all languages, but bytes / 4 is closer for multibyte text, and it uses the same unit as the `mb_strcut()` trim in item 5. 2. [x] Decide: how long to remember a failure across requests. Decided: 10 minutes in `cache.default` (item 3). 3. [x] Decide: what to store after a fallback save. Decided: the fallback count, then one full queued recount (item 6). Not chosen: store 0 (badges and budget summaries show 0 until the next save), and a `token_count_estimated` base field (an entity schema update just before 1.0.0). 4. [x] Open a project/ai issue: project/ai#3586818 (update yethee/tiktoken to ^1.1.1, and handle the failure of the `cl100k_base` fallback in `Tokenizer::setModel()`). project/ai#3586582 (provider-native token counting) is related. ## Stakeholders * **Owner:** @kepol > [!note] > Check the one that best describes your usage, or leave all unchecked if AI was not significantly used. > > * [ ] AI **Assisted Issue** \ > _Mainly written by a human; AI used for partial generation under full human supervision._ > > * [x] AI **Generated Issue** \ > _Mainly generated by AI, reviewed and approved by a human before this issue was created._