AI Search Bug Fixes and Improvements (#3565210) · Issues · project / ai_search

AI Search Bug Fixes and Improvements

>>> [!note] Migrated issue   Reported by: [mbatterton](https://www.drupal.org/user/1574576) Related to !16 >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> During testing of AI Search 2.0.0-alpha1 with OpenAI embeddings and Milvus vector database, several critical issues were discovered: 1. **Word2Vec Processor Compatibility**: When complex query structures are passed to the search backend, the recursive array flattening was not properly handling nested arrays, causing embeddings to be generated from empty strings. 2. **Null Pointer Issues**: Missing null safety checks for server instances caused potential crashes in several processor classes. 3. **Solr Boosting Algorithm**: The hybrid search implementation using `elevateIds` was completely replacing SOLR results rather than boosting AI results, reducing search effectiveness. 4. **Developer Experience**: No warnings when developers configure fulltext search on embedding fields, leading to confusion about semantic vs keyword matching. 5. **Documentation Gaps**: Lack of troubleshooting guidance for common issues like exact phrase searches not working with vector databases. <h3 id="summary-proposed-resolution">Proposed resolution</h3> <h4>1. Fix Word2Vec/Complex Query Compatibility</h4> **File**: `src/Plugin/search_api/backend/SearchApiAiSearchBackend.php` In the `getSearchVectorInput()` method (around line 794), implement recursive array flattening that properly handles nested query structures: <div class="codeblock"> <pre><?php $flatten = function($array) use (&$flatten) {    $result = [];    foreach ($array as $key => $value) {      // Skip render array keys that start with #.      if (is_string($key) && strpos($key, '#') === 0) {        continue;      }      if (is_array($value)) {        $result = array_merge($result, $flatten($value));      }      elseif (is_string($value)) {        $result[] = $value;      }    }    return $result;  };  $search_words = implode(' ', $flatten($search_words)); ?></pre></div> This ensures that complex query structures like `{"#conjunction":"OR","0":"champagne"}` are properly converted to searchable text. <h4>2. Add Null Safety Checks</h4> **Files**: - `src/Plugin/search_api/processor/DatabaseBoostByAiSearch.php` (line 30-31) - `src/Plugin/search_api/processor/ScoreThreshold.php` (line 30-31) - `src/Plugin/search_api/processor/BoostByAiSearchBase.php` (line 177-179) Add null checks before accessing server backend: <div class="codeblock"> <pre><?php  $server = $index->getServerInstance();  if ($server && $server->getBackendId() == 'search_api_db') {    return TRUE;  }  return FALSE; ?></pre></div> And add null safety for non-string keywords: <div class="codeblock"> <pre><?php  foreach ($keywords as $keyword) {    // Skip non-string keywords (e.g., nested arrays from complex queries).    if (!is_string($keyword)) {      continue;    }    // ... rest of processing  } ?></pre></div> <h4>3. Improve Solr Boost Algorithm</h4> **File**: `src/Plugin/search_api/processor/SolrBoostByAiSearch.php` Replace the `elevateIds` approach with weighted boost queries (around line 112-126): <div class="codeblock"> <pre><?php  // Use boost queries to boost AI results rather than elevateIds which  // appears to replace results entirely. Build weighted boost queries based  // on AI search scores and position.  $boost_queries = [];  $position = count($param_parts);  foreach ($param_parts as $id) {    // Boost decreases with position: first result gets highest boost.    $boost_value = 100 * $position;    $boost_queries[] = 'id:"' . $id . '"^' . $boost_value;    $position--;  }  if (!empty($boost_queries)) {    $solarium_query->addParam('bq', $boost_queries);  } ?></pre></div> Also update the query combination logic (around line 107) to use OR: <div class="codeblock"> <pre><?php  if ($query = $solarium_query->getQuery()) {    $query = $query . ' OR id:("' . implode('" "', $param_parts) . '")';    $solarium_query->setQuery($query);  } ?></pre></div> <h4>4. Add Developer Warnings</h4> **File**: `src/Plugin/search_api/backend/SearchApiAiSearchBackend.php` Add warnings when fulltext search is used on embedding fields (around line 426): <div class="codeblock"> <pre><?php  if ($query->getKeys()) {    $fulltext_fields = $query->getOption('search_api_fulltext_fields', []);    if (!empty($fulltext_fields)) {      foreach ($fulltext_fields as $field_id) {        $field = $index->getField($field_id);        if ($field && $field->getType() === 'embedding') {          $this->logger->warning('Fulltext search is being used on embedding field "@field". This performs semantic similarity matching, not exact keyword matching.   Results may not match traditional search expectations.', [            '@field' => $field_id,          ]);          $this->messenger->addWarning($this->t('Note: Searching on AI-embedded fields performs semantic similarity matching based on meaning, not exact keyword  matching. For exact matches, use a traditional search backend or add scalar fields.'));          break;        }      }    }  } ?></pre></div>

issue