>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3487487. --> Reported by: [seogow](https://www.drupal.org/user/3065779) Related to !525 !942 !283 >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p>When using the AI Search module, which leverages the Search API for indexing content, we encounter a significant issue with large items that require extensive processing. The module needs to chunk content and generate embeddings for each chunk. However, this process can be time-consuming, especially for large items that result in hundreds of chunks. As a result, the indexing process can exceed the max_execution_time limit set in PHP, causing timeouts during Cron runs or user-initiated batch processes.</p> <h4 id="summary-steps-reproduce">Steps to reproduce</h4> <p>Create a very long content item, e.g. Node and trigger indexing.</p> <h3 id="summary-proposed-resolution">Proposed resolution</h3> <ol> <li>Extend Search API tracker to count total chunks and track chunk progress</li> <li>Extend Search API batch to repeat the same entity when only partially completed chunks</li> <li>Similarly have Search API batch helper indicate progress potentially as a percentage of entity completed, e.g. 12.25 entities completed if item 13 is needing 4 batches</li> </ol> <h4 id="summary-steps-to-test">Steps to test merge request</h4> <ol> <li>Check out the Search API batch issue</li> <li>Edit Search API Server with AI Search back-end to set a lower Chunking Threshold</li> <li>Ensure small enough chunk size there so that many chunks are needed for an item</li> <li>Create a long content item with a lot of content</li> <li>Attempt to 'Index now' for the new content item</li> </ol> <h4>Follow-ups</h4> <ul> <li><span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/ai/-/work_items/3539033" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/ai/-/work_items/3539033</a></span></li> </ul> > Related issue: [Issue #3551144](https://www.drupal.org/node/3551144) > Related issue: [Issue #3553054](https://www.drupal.org/node/3553054)