>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3526390. --> Reported by: [scott_euser](https://www.drupal.org/user/3267594) Related to !717 >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p>At the moment if you follow the code in <code>Drupal\ai_search\Plugin\search_api\backend\SearchApiAiSearchBackend::$maxAccessRetries</code>, we re-attempt up to 10 times to get a specific count (limit) of results.</p> <p>For some scenarios like <a href="https://www.drupal.org/project/ai_related_content">AI Related Content</a> in a context where large Nodes have been broken into many smaller chunks, even this iteration may not be sufficient, especially if no filter on access is made and subsequent access checks also exclude many nodes (e.g. a more member content driven site).</p> <h4 id="summary-steps-reproduce">Steps to reproduce</h4> <ol> <li>Have a site with many access controlled nodes</li> <li>Have large content lengths with small chunk size</li> <li>Attempt to search and retrieve a specific number of results</li> </ol> <h3 id="summary-proposed-resolution">Proposed resolution</h3> <p>Improve the iteration by allowing Vector Databases to say they are either:</p> <ol> <li>A vector database that supports Grouping or Aggregation of some form like <a href="https://milvus.io/docs/grouping-search.md">https://milvus.io/docs/grouping-search.md</a>. We can group by <code>drupal_entity_id</code>. This seems to just be Milvus (big win for Milvus!)</li> <li>A vector database that supports filtering by NOT IN array of already found <code>drupal_entity_id</code>. Most VDB Providers (if not all) should be able to support this).</li> </ol> <p>So I think some more changes to VDB Provider interfaces probably. For (1) its pre-query change, for (2) its post query condition setting by VDB Provider on recursive ::doSearch() call</p> <h3 id="summary-remaining-tasks">Remaining tasks</h3> <ol> <li>Merge request to build in this functionality</li> <li>Decide when to implement, as it will probably be a breaking change and require coordinated release of VDB providers. I suggest 2.0.x</li> </ol> <h3 id="summary-ui-changes">User interface changes</h3> <p>N/A</p> <h3 id="summary-api-changes">API changes</h3> <p>TBD</p> <h3 id="summary-data-model-changes">Data model changes</h3> <p>N/A</p> > Related issue: [Issue #3526393](https://www.drupal.org/node/3526393)