bug: #3586637 should prefer chunk text over configured indexed fields

Ann Mary Sruthyrequested to merge
issue/ai-3586637:3586637-airerankerextractitemtext-should-prefer-chunk-text-over-configured-indexed-fields into 1.x

Description

AiReranker::extractItemText() only read the processor's configured source_fields off each result item's indexed fields. In ai_search's chunk-mode retrieval (queries with the search_api_ai_get_chunks_result option set, as used by RagTool), each result item is one chunk, but its indexed fields still reflect the whole entity — the chunk's own text lives only in $item->getExtraData('content'). This meant every chunk belonging to the same entity was reranked against identical, full-entity text, degrading chunk-level rerank quality to entity-level rerank quality.

extractItemText() (src/Plugin/search_api/processor/AiReranker.php) now prefers $item->getExtraData('content') when present, falling back to the existing source_fields concatenation otherwise. This is an internal method-behavior change only; the fallback path used by whole-entity, non-chunked results is unchanged.

Closes #3586637

Testing instructions

  1. Run the module's kernel test suite and confirm the two new cases pass: vendor/bin/phpunit -c web/core --filter 'testExtractItemTextPrefersChunkContentOverSourceFields|testExtractItemTextFallsBackToSourceFieldsWithoutChunkContent' web/modules/contrib/ai/tests/src/Kernel/Plugin/Processor/AiRerankerTest.php
  2. To verify end-to-end: configure a Search API index with ai_search chunking enabled, enable the ai_reranker processor on it with a source field configured, then run a query with search_api_ai_get_chunks_result set (e.g. via ai_search's RAG search / RagTool). Confirm each returned chunk is reranked against its own chunk text rather than the shared parent entity's field content — e.g. by comparing rerank order/scores before and after this patch on an index where an entity's chunks differ significantly in relevance to the query.

Checklist

  • I have linked the related issue in the MR title or description
  • I have performed a self-review of my own code
  • I have added or updated tests, or explained in the description why this change is not covered by tests
  • I have updated documentation for any new or changed functionality
  • I have written testing instructions and verified them locally
  • I have noted any required post-merge steps (config imports, cache rebuilds, manual changes)
  • This MR contains no breaking API or hook changes, or they are explicitly documented in the description

AI Compliance

Note

Check the one that best describes your usage, or leave all unchecked if AI was not significantly used.

  • AI Assisted Code
    Mainly written by a human; AI used for autocomplete or partial generation under full human supervision.

  • AI Generated Code
    Mainly generated by AI, reviewed and approved by a human before this MR was created.

  • Vibe Coded
    Generated by AI and only functionally reviewed before this MR was created.

Closes #3586637

Merge request reports