The max tokens calculation is hardcoded + use tokenizer for AI CCC
>>> [!note] Migrated issue
<!-- Drupal.org comment -->
<!-- Migrated from issue #3549748. -->
Reported by: [marcus_johansson](https://www.drupal.org/user/385947)
Related to !4
>>>
<p>[Tracker]<br>
<strong>Update Summary: </strong>The max tokens calculation is hardcoded + use tokenizer for AI CCC<br>
<strong>Short Description: </strong>The max tokens calculation is hardcoded + use tokenizer for AI CCC<br>
<strong>Check-in Date: </strong>MM/DD/YYYY<br>
<em>Metadata is used by the <a href="https://www.drupalstarforge.ai/" title="AI Tracker">AI Tracker.</a> Docs and additional fields <a href="https://www.drupalstarforge.ai/ai-dashboard/docs" title="AI Issue Tracker Documentation">here</a>.</em><br>
[/Tracker]</p>
<h3 id="summary-problem-motivation">Problem/Motivation</h3>
<p>Currently there is a max tokens that is hard coded to 800 characters. It should be the maxTokens.</p>
<p>The AI module that is a dependency has a token counter service.</p>
<h4 id="summary-steps-reproduce">Steps to reproduce</h4>
<p>Try to set a super high value for max tokens.<br>
The context is still clipped.</p>
<h3 id="summary-proposed-resolution">Proposed resolution</h3>
<p>Use the actual maxTokens.<br>
Use it with the token counter service.</p>
issue