>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3469392. --> Reported by: [seogow](https://www.drupal.org/user/3065779) Related to !258 >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p>Text chunker, <code>src/Utility/TextChunker.php</code>, breaks Byte-Pair Encoding (BPE) by chunking tokenized text. Chunking must follow the tokenization rules.</p> <h4 id="summary-steps-reproduce">Steps to reproduce</h4> <p>TextChunker returns malformed strings on many occasions. The easiest way to test is iterative chunk complex text like math or Chinese.</p> <h3 id="summary-proposed-resolution">Proposed resolution</h3> <p>We need to extend Tiktoken Encoder and add a method which will extend functionality of its <code>encodeInChunks()</code>.</p>