AI Search: Have different approaches to chunking, perhaps configurable, perhaps different chunking engines.
>>> [!note] Migrated issue
<!-- Drupal.org comment -->
<!-- Migrated from issue #3458588. -->
Reported by: [yautja_cetanu](https://www.drupal.org/user/626050)
>>>
<h3 id="summary-problem-motivation">Problem/Motivation</h3>
<p>In Search API we did a very basic approach to chunking where we looked at the advice at the time and just went with that.</p>
<p>Things to think about:</p>
<p>- Can we find Chunking libraries elsewhere so we don't do it in Drupal.<br>
- Do you have whole "Chunking methods" which includes everything all in one plugin<br>
- Or do we make things configurable when you set up the index (Such as 5000 characters vs 500, try and find the end of the sentence, overlap, Do you strip HTML)<br>
- If its configurable should we have sensible defaults so someone doesn't have to understand how this all works.<br>
- What about adding meta data into the chunk (Such as Node Title, Author, Genre, etc)</p>
<h3 id="summary-proposed-resolution">Proposed resolution</h3>
issue