Fix Word2Vec compatibility and improve hybrid search

Matt Battertonrequested to merge
issue/ai_search-3565210:3565210-ai-search-bug into 2.0.x

Multiple bug fixes and improvements for AI Search 2.0.0-alpha1:

  1. Fix Word2Vec/Complex Query Compatibility

    • Implement recursive array flattening in SearchApiAiSearchBackend::getSearchVectorInput()
    • Properly handle nested query structures like {"#conjunction":"OR","0":"champagne"}
    • Skip render array keys that start with # to avoid processing metadata

  2. Add Null Safety Checks

    • Add null checks for server instances in DatabaseBoostByAiSearch
    • Add null checks for server instances in ScoreThreshold
    • Add null safety for non-string keywords in BoostByAiSearchBase

  3. Improve Solr Boost Algorithm

    • Replace elevateIds with weighted boost queries (bq parameter)
    • Use position-based boost values (first result gets highest boost)
    • Update query combination to use OR instead of replacing results

  4. Add Developer Warnings

    • Warn when fulltext search is used on embedding fields
    • Clarify semantic similarity vs keyword matching expectations
    • Log warnings to help developers understand vector search behavior

  5. Enhanced Documentation

    • Add comprehensive troubleshooting section to README
    • Explain why exact keyword searches don't work with vector search
    • Document solutions: hybrid search, traditional backends, scalar filtering

Tested with Drupal 10.6.1, AI 2.x, OpenAI text-embedding-3-large, and Milvus VDB.

Closes #3565210

Merge request reports