Draft: chore: add bulk chunking
Reworked for 2.0.x: batches VDB inserts during indexing.
PostgresProvider::indexItems() previously issued one INSERT per chunk. It now builds one row per chunk and inserts them all in a single query via a new insertMultipleIntoCollection() on both the provider and PostgresPgvectorClient.
Notes for review
- Built on top of 2.0.x's existing embedding API (
getChunks()/getEmbedding()/Embeddingvalue object) — the embedding-adaptation part of the original 1.3.x MR is already present on 2.0.x, so this MR contains only the batch-insert optimization. - Keeps 2.0.x's parameterized-query approach (
pg_query_params); it does not fall back to string interpolation. - A multi-row
INSERTcannot use thecurrval()-based relation linking that the single-row path relies on (currval only yields the last row's id). Instead thisRETURNs each generatedidalongside itsdrupal_long_id(unique per chunk) and keys relation-table inserts by that, so the mapping is correct regardless of the order Postgres returns rows in.
⚠️ Not yet verified — needs a real indexing run before merge
This has not been exercised against a live Postgres/pgvector. Please verify before un-drafting:
- Index content that produces multiple chunks per item and confirm all chunks land (row counts match).
- Include an item with a multi-value field and confirm its relation-table rows link to the correct chunk ids (not all to the last chunk).
- Confirm
content,embedding(vector) and extra single-value columns are populated correctly, and re-indexing (delete + re-insert) still works.
Requires drupal/ai:^2.0 and drupal/ai_search:^2.0 (already in composer.json on this branch).
Edited by Dan Lemon