AI Search: Make vector databases abstracted and installable for recipes
>>> [!note] Migrated issue
<!-- Drupal.org comment -->
<!-- Migrated from issue #3509315. -->
Reported by: [marcus_johansson](https://www.drupal.org/user/385947)
Related to !494
>>>
<h3 id="summary-problem-motivation">Problem/Motivation</h3>
<p>Right now we have the possibility for you to create a recipe that applies any 3rd party module for the AI providers with a default operation type model. This means that you can create recipes without necessarily having to bind them to OpenAI or Ollama or anything else.</p>
<p>We should be able to do the same for:</p>
<ol>
<li>Setting up a VDB providers keys (that might make sense to move to Key module and be generic).</li>
<li>Setting up a server and index, independent on the VDB provider and independent on the Embeddings model.</li>
</ol>
<p>This means that you should be able to add a default VDB Provider.</p>
<p>This means that you should be able to define a recipe for installing a key like:</p>
<pre> vdb_provider_milvus.settings:<br> setupVDBProvider:<br> key_value: ${milvus_api_key}<br> key_name: milvus_api_key<br> key_label: 'Milvus API Key'<br> provider: milvus<br> simpleConfigUpdate:<br> host: ${milvus_host}<br> port: ${milvus_port}</pre><p>And then in your recipe you would be able to prompt in console or Project browser for hostname, port and key.</p>
<p>If no default VDB provider exists, Milvus or whatever gets installed first can opt in to become this, but it should be settable where you change the other operation types.</p>
<p>When that is setup you should then be able to import a server and an index from config, just that you can set default on the database and embedding_engine in the server config.</p>
<p>This should also be possible to do in the form, and when the form is getting saved, if the dimensions is set to set_dimesions = false and the embedding_engine is set to default, we manually collect the dimensions. This has to happen on saving, not in the form, so it will also work on recipe.</p>
<p>The index configuration doesn't really have anything connecting to dynamic values, so nothing needed to do there.</p>
<p>This will not 100% work on all VDB Providers or Embeddings Engines, since some comes with specific configurations - for instance Image Embeddings Engine from AWS. But it will work for most cases, so you can make a fairly generic recipe for AI Search.</p>
<h3 id="summary-proposed-resolution">Proposed resolution</h3>
<h3>Step one - make having a default embeddings provider possible</h3>
<ul>
<li>Make it possible to install a VDB server, with a custom recipe plugin, that rewrites the provider, model and dimensions to whatever is set in the default embeddings model.</li>
<li>Make it so that the collection generation happens on config save of a VDB provider server, instead of SubmitForm.</li>
</ul>
<h3>Step two - make having a default vdb provider possible</h3>
<ul>
<li>Make it possible to set a default VDB provider, where you setup operation types.</li>
<li>Make a helper method in the plugin manager where VDB provider's can opt in to be a default VDB Provider (or maybe this is implied?)</li>
<li>Make a listener for uninstalling VDB Providers, that removes the default VDB Provider.</li>
<li>In the previous recipe, make it possible with to choose a default VDB Provider as database when setting up a Search API server of type VDB.</li>
<li>As before, this sets the actual provider as the value, since this can't be changed after indexing easily.</li>
<li>Add a recipe for adding keys dynamically, so we also can setup a specific VDB Provider this way, see how AI Provider custom recipe does it. It could also be solved here <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-6"><a href="https://www.drupal.org/project/key/issues/3495119" title="Status: Closed (works as designed)">#3495119: Create custom recipe action for Key module</a></span>, but its some more efforts, since this should cover all the ways you can save a key.</li>
</ul>
issue