Draft: chore: add bulk chunking

Reworked for 2.0.x: batches VDB inserts during indexing.

PostgresProvider::indexItems() previously issued one INSERT per chunk. It now builds one row per chunk and inserts them all in a single query via a new insertMultipleIntoCollection() on both the provider and PostgresPgvectorClient.

Notes for review

  • Built on top of 2.0.x's existing embedding API (getChunks() / getEmbedding() / Embedding value object) — the embedding-adaptation part of the original 1.3.x MR is already present on 2.0.x, so this MR contains only the batch-insert optimization.
  • Keeps 2.0.x's parameterized-query approach (pg_query_params); it does not fall back to string interpolation.
  • A multi-row INSERT cannot use the currval()-based relation linking that the single-row path relies on (currval only yields the last row's id). Instead this RETURNs each generated id alongside its drupal_long_id (unique per chunk) and keys relation-table inserts by that, so the mapping is correct regardless of the order Postgres returns rows in.

⚠️ Not yet verified — needs a real indexing run before merge

This has not been exercised against a live Postgres/pgvector. Please verify before un-drafting:

  • Index content that produces multiple chunks per item and confirm all chunks land (row counts match).
  • Include an item with a multi-value field and confirm its relation-table rows link to the correct chunk ids (not all to the last chunk).
  • Confirm content, embedding (vector) and extra single-value columns are populated correctly, and re-indexing (delete + re-insert) still works.

Requires drupal/ai:^2.0 and drupal/ai_search:^2.0 (already in composer.json on this branch).

Edited by Dan Lemon

Merge request reports

Loading