Search API Index fields need more user control, UI should be updated to make it clearer how the embedding content is prepared
>>> [!note] Migrated issue
<!-- Drupal.org comment -->
<!-- Migrated from issue #3471759. -->
Reported by: [seogow](https://www.drupal.org/user/3065779)
Related to !119
>>>
<h3 id="summary-problem-motivation">Problem/Motivation</h3>
<p>This is not purely vector search issue. Any external search database used in Views needs to duplicate fields content, if that content should be used in filtering or sorting. One such example is 'state', for filtering published/archived entities - that field has no semantic meaning, but it needs to be included in the external database. Including such fields in vector representation of entities can cause problems, especially if the content is short.</p>
<p>The second issue is that some vector databases have hard limit on storage of additional fields (e.g. Milvus has it set to 64kB), and if they do not provide keyword search, storage all the fields used for vectorisation is superfluous.</p>
<p>The third issue is that user needs better control of usage of fields for vectorisation. Now, we do assume that short fields are metadata, but that will not always be the case.</p>
<h3 id="summary-proposed-resolution">Proposed resolution</h3>
<ol>
<li>☑ See WIP UI attached below</li>
<li>☑ Provide control over which fields are:
<ul>
<li>☑ Main content for embedding/vectorisation</li>
<li>☑ Contextual content for embedding/vectorisation</li>
<li>☑ Attributes for use as filters only</li>
</ul>
</li>
<li>☑ Update labels to remove overlapping double use of metadata in terminology</li>
<li>☑ Remove the Embeddings field type</li>
<li>☑ Update hook to switch it back to String</li>
<li>☑ Add supports options for Embedding Strategies to support multiple Main Content fields</li>
<li>☑ Validation to prevent multiple main fields while Embedding Strategy does not support it</li>
<li>☑ Move maxlength to a buried option only shown when a checkbox is checked like "Override maximum lengths for storage of string metadata"</li>
<li>☑ Update embedding strategy code to reflect the new choices</li>
<li>☑ Ensure changing these choices triggers re-queueing for indexing</li>
</ol>
<p><img src="https://www.drupal.org/files/issues/2024-09-30/2024-09-30_16-11.png" alt="Screenshot of Fields UI WIP"></p>
issue