**Background story** We stumbled upon a couple of issues over the past few months, and my spidey sense told me they were all related to the embedding strategy. For example: - We experienced unexpected rate limits on Amazee AI when indexing just 400+ nodes, a fairly modest amount of content. Flat pricing on Amazee meant we didn't run into budget issues, but indexing site content still caused unexpected pain and required workarounds at both the application and the Amazee AI level. - As a poor man's solution on one project, I desperately tried to force main content to be chunked instead of contextual content once I realized what was happening behind the scenes. That led to a WSOD, caused by the exception mentioned above being thrown from `\Drupal\ai\Utility\TextChunker::chunkText()`. Today I had the chance to run a brainstorming and investigation session with a coding agent on the embedding strategy, and I believe its claims are valid. --- ## Problem Overview In `EmbeddingBase::prepareChunks()`, contextual content and main body fields share a single chunk token budget via a proportional percentage split (`1 - contextualContentMaxPercentage`). Site builders may increase "Contextual content maximum percentage" to high values (e.g., 90%+) hoping to allocate more capacity to rich metadata, taxonomy summaries, or referenced entity fields. However, the current implementation in the "both main and context need chunking" branch suffers from two primary technical defects: 1. **Multiplicative ($M \times C$) Chunk Generation:** It cross-multiplies main chunks and contextual chunks in a nested loop. 2. **Chunk Size / Overlap Precondition Violation:** When context allocation reduces the remaining main content budget below the required minimum overlap (`chunkMinOverlap`), `TextChunker::chunkText()` throws an uncaught exception. ## Technical Analysis of the Defects ### Defect 1: Cartesian Product ($M \times C$) Growth & Retrieval Noise When both main content and contextual content exceed their respective token limits: ```php foreach ($main_chunks as $main_chunk) { foreach ($contextual_chunks as $contextual_chunk) { $chunks[] = $this->prepareChunkText($title, $main_chunk, $contextual_chunk, $index); } } ``` * **Multiplicative Inflation:** If an item produces 10 main content chunks ($M$) and 5 contextual chunks ($C$), `prepareChunks()` returns **50 rendered chunks** ($10 \times 5$). Each chunk is submitted to `getRawEmbeddings()`, scaling vector storage and API embedding calls quadratically relative to single-dimension chunking. * **Retrieval Noise:** Arbitrary pairings occur (e.g., Main Chunk 1 paired with Contextual Chunk 5). In vector similarity search, multiple vectors sharing identical body content but differing only in unrelated context fragments compete against each other, potentially diluting top-$K$ search precision. ### Defect 2: TextChunker Precondition Violation & Fatal Exception Risk Drupal AI's `TextChunker` explicitly requires that `minOverlap` be strictly less than `maxSize` (`maxSize - minOverlap > 0`). If this precondition is violated, `TextChunker::chunkText()` throws `\Exception('The minimum overlap cannot be equal to or exceed the maximum chunk size.')`. The exception occurs whenever: $$\lfloor(\text{chunkSize} - \text{titleTokens}) \times (1 - p)\rfloor \le \text{chunkMinOverlap}$$ *(where $p = \text{contextualContentMaxPercentage} / 100$)* For example, given a 400-token available budget, a 20-token overlap, and a 99% context percentage ($p=0.99$): * Main chunk size evaluates to $\lfloor 400 \times 0.01 \rfloor = 4$ tokens. * Since $4 \le 20$, `TextChunker` throws an uncaught exception, which can fail the indexing operation or halt execution depending on the caller and batch context. Additionally, generating thousands of micro-chunks in extreme edge cases substantially increases memory consumption. ## Proposed Solution A chunk's primary semantic unit is its main content. Contextual fields exist to disambiguate that main content, not to generate combinatorial pairs of entity fragments. We should use **one bounded contextual prefix per main chunk** (rather than generating every main/context combination) while strictly maintaining the budget invariant: $$\text{contextualChunkSize} + \text{mainChunkSize} \le \text{availableChunkSize}$$ ### Corrected Patch for `EmbeddingBase.php` ```php // Both contextual content and main fields need chunking. $title_tokens = !empty($title) ? $this->tokenizer->countTokens($title) : 0; $available_chunk_size = $this->chunkSize - $title_tokens; $minimum_main_chunk_size = $this->chunkMinOverlap + 1; if ($available_chunk_size <= $minimum_main_chunk_size) { throw new \LogicException( 'The configured chunk size leaves no valid capacity for main content after deducting title tokens.' ); } // Compute context size, but reserve at least $minimum_main_chunk_size for main content. $requested_contextual_chunk_size = (int) ($available_chunk_size * $max_contextual_content); $main_chunk_size = max( $minimum_main_chunk_size, $available_chunk_size - $requested_contextual_chunk_size ); $contextual_chunk_size = $available_chunk_size - $main_chunk_size; // Ensure contextual overlap remains strictly less than contextual chunk size. $contextual_min_overlap = min( max(1, (int) ($this->chunkMinOverlap * $max_contextual_content)), max(1, $contextual_chunk_size - 1) ); $contextual_chunks = $this->textChunker->chunkText( $contextual_content, $contextual_chunk_size, $contextual_min_overlap ); $main_chunks = $this->textChunker->chunkText( $main_content, $main_chunk_size, $this->chunkMinOverlap ); // Use the primary contextual prefix for all main chunks to keep chunk output at M rather than M x C. $contextual_chunk = reset($contextual_chunks) ?: ''; if (count($contextual_chunks) > 1) { $this->loggerChannelFactory->get('ai_search')->debug( 'Contextual content was truncated to the primary chunk during Search API item chunking.' ); } foreach ($main_chunks as $main_chunk) { $chunks[] = $this->prepareChunkText($title, $main_chunk, $contextual_chunk, $index); } ``` ## Benefits of this Refactoring 1. **Strict Token Budget Compliance:** Ensures $\text{contextualChunkSize} + \text{mainChunkSize} \le \text{availableChunkSize}$ under all percentage configurations. 2. **Eliminates Multiplicative Growth:** Output scales linearly with main content ($M$ chunks) rather than quadratically ($M \times C$). 3. **Prevents Precondition Exceptions:** Ensures `$main_chunk_size > $chunkMinOverlap` at all times. 4. **Deterministic Context Prepending:** Preserves primary entity metadata (Title, Category, Author) across all main chunks without introducing arbitrary combinatorial pairs. ## Suggested Regression Tests 1. **Chunk Output Bound ($M \times C \to M$):** Assert `count($chunks) === count($main_chunks)` when both main and contextual content exceed chunk limits. 2. **High Context Allocation:** Verify that setting `contextualContentMaxPercentage` to 99% executes cleanly without throwing `TextChunker` exceptions or exceeding `chunkSize`. 3. **Budget Invariant Verification:** Assert that all rendered chunk strings remain within `$this->chunkSize`. 4. **Title Capacity Guard:** Assert that a `\LogicException` is thrown if title tokens leave no valid room for main content.