AI Search: Have different approaches to chunking, perhaps configurable, perhaps different chunking engines.
>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3458588. --> Reported by: [yautja_cetanu](https://www.drupal.org/user/626050) >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p>In Search API we did a very basic approach to chunking where we looked at the advice at the time and just went with that.</p> <p>Things to think about:</p> <p>- Can we find Chunking libraries elsewhere so we don't do it in Drupal.<br> - Do you have whole "Chunking methods" which includes everything all in one plugin<br> - Or do we make things configurable when you set up the index (Such as 5000 characters vs 500, try and find the end of the sentence, overlap, Do you strip HTML)<br> - If its configurable should we have sensible defaults so someone doesn't have to understand how this all works.<br> - What about adding meta data into the chunk (Such as Node Title, Author, Genre, etc)</p> <h3 id="summary-proposed-resolution">Proposed resolution</h3>
issue