_As a follow up on #3584037 I have asked the coding agent to analyze whether the current auto-uppercase-title append logic in the embedding strategy makes sense and what are the PROS and CONS._ ### **Problem Statement** In `Drupal\ai_search\Plugin\EmbeddingStrategy\EmbeddingBase::prepareChunkText()`, the title is converted to uppercase using `strtoupper($title)` before prepending it as a Markdown header (`# `) to generated text chunks: ```php if (!empty($title)) { $parts[] = '# ' . strtoupper($title); } ``` Forcing titles into ALL-CAPS causes two significant issues for Search/RAG workflows: 1. **Token Inflation:** Modern BPE tokenizers (OpenAI, Tiktoken, Llama, Cohere) do not have common ALL-CAPS words in their primary vocabulary, causing words to fragment into multiple sub-word tokens and consuming context window budget unnecessarily. When uppercase title prepended to every indexed chunk across a large site, the current implemenation consumes significant context window budget without adding any semantic value. 2. **Semantic Shift:** Embedding models are case-sensitive. ALL-CAPS text forces the model into an out-of-distribution vector space (associated with acronyms, shouting, or noise), reducing retrieval precision against natural-language user queries. 3. **i18n Issue:** Standard PHP `strtoupper()` is not multibyte-safe and corrupts non-ASCII / non-Latin title strings. --- ### **Steps to Reproduce** 1. Index any content item (e.g., node with title `"Documentation"`). 2. Inspect the generated chunk text via `prepareChunkText()` or check vector database payload metadata. 3. **Observed chunk header:** `# DOCUMENTATION` 4. **Expected chunk header:** `# Documentation` --- ### **Simple Verification / Proof of Token Inflation** Testing with OpenAI's official tokenizer (`cl100k_base` / `gpt-4` / `text-embedding-3`): * `system architecture overview` $\rightarrow$ **3 token** * `System Architecture Overview` $\rightarrow$ **3 token** * `SYSTEM ARCHITECTURE OVERVIEW` $\rightarrow$ **8 tokens** `('SYSTEM', ' ARCH', 'IT', 'ECT', 'URE', ' OVER', 'VIEW')` Forcing uppercase across all title headers systematically inflates the token count of every chunk. ### **Proposed Fix** Remove `strtoupper()` in `EmbeddingBase.php` and preserve natural title casing: ```diff - $parts[] = '# ' . strtoupper($title); + $parts[] = '# ' . $title; ``` <details> <summary><b>Appendix: Authoritative References & Academic Context</b></summary> #### Token Inflation * **OpenAI Tiktoken Docs:** BPE token vocabularies are built on standard natural-language corpora. Non-standard casing misses single-token vocabulary entries and degrades into multi-token fragments. * **[Hugging Face NLP Course (Ch. 6)](https://huggingface.co/learn/llm-course/chapter6/5):** Subword tokenization algorithms (BPE/WordPiece) treat uppercase and lowercase characters as separate tokens; rare uppercase sequences fragment into smaller sub-words. #### Semantic Shift * **[Cohere Embed v3 Best Practices](https://docs.cohere.com/docs/cohere-embed):** Dense vector models preserve casing to distinguish semantics (e.g., `IT` vs `it`). Arbitrary ALL-CAPS shifts embeddings away from natural query vector space. * **Sentence-Transformers (SBERT.net):** Modern Transformer-based encoders (`BERT`, `RoBERTa`, `MPNet`) use cased token embeddings. Altering casing alters attention weights and vector distance. </details>