Figure out random failures of testTokenCount test
>>> [!note] Migrated issue
<!-- Drupal.org comment -->
<!-- Migrated from issue #3542958. -->
Reported by: [marcus_johansson](https://www.drupal.org/user/385947)
Related to !895
>>>
<p>--- AI TRACKER METADATA ---<br>
<strong>Update Summary: </strong>Needs help. Random failure.<br>
<strong>Check-in Date: </strong>09/03/2025 (US format) [When we should see progress/get an update]<br>
<strong>Due Date:</strong> MM/DD/YYYY (US format) [When the issue should be fully completed]<br>
<strong>Blocked by:</strong> [#XXXXXX] (New issues on new lines)<br>
<strong>Additional Collaborators:</strong> @username1, @username2<br>
AI Tracker found here: <a href="https://www.drupalstarforge.ai/" title="AI Tracker">https://www.drupalstarforge.ai/</a><br>
--- END METADATA ---</p>
<h3 id="summary-problem-motivation">Problem/Motivation</h3>
<p>The testTokenCount test fails randomly around 10-15% of the time on the first item. See an example here: <a href="https://git.drupalcode.org/issue/ai-3503947/-/jobs/6214945">https://git.drupalcode.org/issue/ai-3503947/-/jobs/6214945</a></p>
<p>Code is here: <a href="https://git.drupalcode.org/project/ai/-/blob/1.2.x/tests/src/Kernel/Utility/TextChunkerTest.php?ref_type=heads#L42">https://git.drupalcode.org/project/ai/-/blob/1.2.x/tests/src/Kernel/Utility/TextChunkerTest.php?ref_type=heads#L42</a></p>
<p>We need to figure out why and fix it.</p>
<h3 id="summary-proposed-resolution">Proposed resolution</h3>
<p>Figure out why the testTokenCount failure and fix it.</p>
issue