Remove tokenizer-x, refactor token chunking and tokenizer into services, and use those
>>> [!note] Migrated issue
<!-- Drupal.org comment -->
<!-- Migrated from issue #3462477. -->
Reported by: [scott_euser](https://www.drupal.org/user/3267594)
>>>
<h3 id="summary-problem-motivation">Problem/Motivation</h3>
<p>Tokenizer-x dependency does not support Drupal 11 at this time and does not seem well supported<br>
Tokenizer/TextChunker are not replaceable by third party modules<br>
Tokenizer count changes depending on model, make this configurable.</p>
<h4 id="summary-steps-reproduce">Steps to reproduce</h4>
<p>Look at TextChunker code</p>
<h3 id="summary-proposed-resolution">Proposed resolution</h3>
<p>Replace with services</p>
<h3 id="summary-remaining-tasks">Remaining tasks</h3>
<p>MR</p>
<h3 id="summary-ui-changes">User interface changes</h3>
<p>Must choose the chat model when setting up search api backend</p>
<h3 id="summary-api-changes">API changes</h3>
<p>N/A</p>
<h3 id="summary-data-model-changes">Data model changes</h3>
<p>N/A</p>
issue