>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3453578. --> Reported by: [marcus_johansson](https://www.drupal.org/user/385947) >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p>By default Search API AI does very naive RAG. This means that it just chunks pieces into a certain size and currently the only techinque being used is overlapping sliding window and even that is hard coded.</p> <p>The possibilities when chunking are endless, so trying to build a one-size-fits-all GUI module will be impossible, but at least we should open up the possibility for the most common use cases. Furthermore it should be possible to use code to inject and manage complex chunking situations.</p> <h3 id="summary-proposed-resolution">Proposed resolution</h3> <p>1. Add the whole entity being indexed as a variable for the chunking process . Meaning that the user should be able to use any field/field value while indexing.<br> 2. Add the embeddings rule as a variable for the chunking process, since this holds important data like max input size and is also where the embeddings call is triggered.<br> 3. Add the possibility for a module to completely take over the whole chunking procedure. Should it be events or Search API preprocessor or both?</p> <h3 id="summary-api-changes">API changes</h3> <p>This needs to be set in the Embeddings Search API Data Type.</p>