>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3582363. --> Reported by: [marcus_johansson](https://www.drupal.org/user/385947) Related to !1379 >>> <p>[Tracker]<br> <strong>Update Summary: </strong>[One-line status update for stakeholders]<br> <strong>Short Description: </strong>Add documentation for the Tokenizer service<br> <strong>Check-in Date: </strong>MM/DD/YYYY<br> [/Tracker]</p> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p>The <code>ai.tokenizer</code> service (<code>src/Utility/Tokenizer.php</code>) provides a wrapper around the Tik-Token PHP library for tokenizing text using the same tokenization schemes as OpenAI models. It is a public service that developers may need when working with token limits, calculating costs, or chunking text for embeddings.</p> <p>The service provides methods to set a model, count tokens, encode text into token arrays, chunk text by token count, and decode tokens back to text. It also includes a fallback to the <code>cl100k_base</code> encoding when a model is not supported.</p> <p>There is currently no documentation for this service.</p> <h3 id="summary-proposed-resolution">Proposed resolution</h3> <ul> <li>Create a documentation page at <code>docs/developers/tokenizer.md</code> documenting the <code>ai.tokenizer</code> service.</li> <li>Document the key methods: <code>setModel()</code>, <code>countTokens()</code>, <code>getTokens()</code>, <code>getEncodedChunks()</code>, <code>decodeChunk()</code>, and <code>getSupportedModels()</code>.</li> <li>Include a usage example showing how to inject the service and count tokens for a given text and model.</li> <li>Note the fallback behavior when a model is not supported by Tik-Token PHP.</li> <li>Add the new page to the <code>mkdocs.yml</code> nav under the Develop section (e.g. under Helper Services).</li> </ul> <h3 id="summary-ai-usage">AI usage (if applicable)</h3> <p>[x] AI Assisted Issue<br> This issue was generated with AI assistance, but was reviewed and refined by the creator.</p> <p>[ ] AI Assisted Code<br> [ ] AI Generated Code<br> [ ] Vibe Coded</p> <p>- <strong>This issue was created with the help of AI</strong></p>