**Note that I switched this from a discussion issue to a documentation. Discussion has happened. Docs are in the MR.** **IMPORTANT: If #3586388+s is merged first, this issue's MR needs fixing based on that (and vice versa).** ## Summary Before we ship 1.0, we should confirm that the current rules for choosing which context items reach an AI feature are the right set — and whether anything important is missing. This issue is a **discussion**. The goal is shared understanding of what the module does today, in plain language, so product/UX and engineering can decide together whether to add ranking factors before release. ## Background When someone uses an AI feature on the site, the module picks a small set of **context items** (brand voice, writing rules, section-specific guidance, etc.) and attaches them to the AI’s instructions. There is a size budget: only so many items and so much text can be sent. Selection is **rule-based**, not “the AI decides what’s relevant.” That makes behavior predictable, but it also means many factors compete — and when an item is not chosen, there is usually no visible explanation in the UI today. ## How selection works today (high level) Selection happens in **layers**, from strongest to weakest: 1. **Remove** items that must never be included 2. **Require** items that must always be included 3. **Filter out** items that do not fit the current situation 4. **Group** remaining items into priority tiers 5. **Rank** within the “preference” tier using scope overlap 6. **Apply limits** (item count, token budget) 7. **Attach child items** (subcontext) after parents are chosen If two items are equally strong within a tier, tie-breakers are mostly **most recently updated** (and item ID as a last resort). --- ## Current ranking and selection factors ### 1. Hard exclusions (item is never considered) | Factor | In plain language | |--------|-------------------| | **Not published** | Draft or unpublished items are ignored. | | **No view permission** | If the current user cannot view the item, it is skipped. | | **Never include** | Per AI feature/agent setting: specific items are blocked even if they would otherwise match. | | **Context filters** | If an item says it only applies in certain situations, and the current situation does not match, the item is removed. Examples: wrong language, wrong site section/URL, wrong content type, or wrong specific page/entity. | | **Broken parent link** | A child item whose parent is missing is not used on the “always include” path. | ### 2. Guaranteed inclusion tiers (these beat normal ranking) These are applied **in this order**. Earlier tiers win when the total item limit is reached. | Tier | In plain language | |------|-------------------| | **Global items** | Marked “apply everywhere.” Included for every AI feature, up to a **global item limit** (site default: 3). If there are more global items than the limit allows, **most recently updated** global items win. | | **Always include** | Per AI feature/agent setting: chosen items are forced in, regardless of topic matching. | | **Auto-included by situation** | Items tied to **what is being worked on right now** — especially **specific target content** (e.g. “this guidance is for Article #42”). These skip subscription ranking because the situation itself selects them. | | **Topic-matched items (scored)** | Everything else that survived filters and overlaps with the AI feature’s **scope subscriptions** (topics/preferences). Sorted by relevance score (see below). | ### 3. Scope subscriptions (the main “preference” ranking) Each AI feature can subscribe to topics like: - **Use case** — e.g. “Working with text,” “Working with Canvas” - **Tags** — free-form labels from the AI Context Tags vocabulary - **Language** - e.g. Spanish, Arabic - **Site section** — URL/path areas - **Entity type/bundle** — content types such as “Article” or “Basic page” **How overlap is scored:** - More overlap with the AI feature’s subscriptions → higher score → more likely to be chosen. - If the item has **no value** for a subscribed topic, it is treated as **broad/generic** for that topic and gets full credit (not penalized). - If the AI feature subscribes to several values under one topic (e.g. two tags), matching **more of them** helps more than matching just one. - Topics have different **importance weights** in the math. Roughly, from most to least influential in scoring today: **Entity type/bundle → Site section → Use case / Tag → Language**. (Exact weights are internal; the relative order matters more for this discussion.) **Important UX nuance:** Language, site section, and entity type/bundle act as **both** filters (must match current situation) **and** preferences (boost ranking when the AI feature subscribes to them). Use case and tags are **preferences only** — they do not auto-filter by current page context. ### 4. Selection mode (when the AI feature has no topic subscriptions) | Mode | In plain language | |------|-------------------| | **Minimal** (default) | Safe default: only global items, always-included items, and situation-based auto-includes. Does **not** pull in general topic-scoped items just because they exist. | | **Match all** | Broader recall: if there are no topic subscriptions, many published items can be considered, ordered by **most recently updated**, still subject to item and token limits. Intended for cases that intentionally need wide coverage. | ### 5. Limits and final cutoffs | Factor | In plain language | |--------|-------------------| | **Max context items** | Cap on how many items are selected (site default: 10; per AI feature can override). | | **Max global items** | Separate cap on global items (site default: 3). | | **Max tokens** | Cap on total rendered text sent to the AI (site default: 1200 tokens). Items lower in the final order may be dropped or shortened when the budget runs out. | | **Language translation** | Items are shown in the current content language when a translation exists. | ### 6. Subcontext (parent/child items) After the main list is built: - **Required children** are always added with their parent. - **Conditional children** may be chosen by a separate AI decision (parent content + child purpose + user task), when that feature is enabled. - Children appear **immediately after their parent** in the final output. - Subcontext resolution happens **after** the main item cap — so children can add extra items beyond the primary max-items limit. ### 7. Things that are **not** ranking factors today Worth calling out for the 1.0 discussion: | Not a factor | Notes [with KP notes in square brackets] | |--------------|-------| | **[FUTURE]** **How well the item matches the user’s actual task/prompt** | Selection does not read the user’s message for relevance (except conditional subcontext decisions). [see future Smart Selection feature: #3568679+] | | **[FUTURE]** **Item title, length, or author** | Not used in ranking. [don't think we should use author; length/token count is something we could factor in deterministically but not sure we should; item title only makes sense with Smart Selection (see above)] | | **[TBD]** **Usage/popularity** | Frequently used items are not boosted. [note that usage is captured, but we aren't saving the aggregate per context item though we could; but by boosting the frequently used items then we are enforcing the boosting every time we do that, so this needs more thought] | | **[TBD]** **Recency (except tie-breakers)** | “Recently updated” only breaks ties within global items and match-all mode — not the main subscription score. [this should be easy enough to change if we wanted to] | | **[UNCLEAR]** **Manual “assigned items” on an agent** | Mentioned in docs, but **not implemented** in code. The working equivalent today is **Always include**. [this seems like a docs wording issue that needs addressing, not a ranking behavior change] | | **[UNCLEAR]** **“Always include when relevant” on the item itself** | Discussed/planned in UX docs; **not implemented** as an item-level switch yet. [I'm unclear on this one, we have "always include", so not sure what this means] | --- ## Discussion questions for 1.0 (KP notes in square brackets) 1. **[TBD]** **Is the four-tier priority order right? [I think so, but maybe there is more "situation auto-include" that makes sense]** Global [yes] → Always include [yes] → Situation auto-include [specific entity items only right now] → Topic score [yes]. 2. **[TBD]** **Are the current subscription topics enough?** Use case, tags, language, site section, entity type/bundle — or do we need item-level priority [see #3586364+], task-aware ranking [requires smart/ai selection which is for post 1.0], recency boosts [not sure we should do this at all], etc.? 3. **[TBD]** **Should broad/generic items (no topic labels) rank higher or lower?** Today they score well on subscribed topics because they are treated as universally applicable for that topic. [this is a good one to discuss asap, though we did discuss this before] 4. **[FUTURE]** **Do we need visible “why was / wasn’t this item included?” before 1.0?** Not a ranking factor itself, but likely the biggest source of confusion when ranking excludes an item. [this can't happen for 1.0 given everything else that needs to get done, but perhaps prioritize this for 1.1 so it gets into dev soon after Rotterdam, I can make a follow-up issue if I don't have one already] 5. **[UNCLEAR]** **Should item-level “always include when filters match” ship in 1.0?** Already identified in UX planning as a way to reduce reliance on two-sided subscription matching. [hmm... this is how it works now except for if there they hit max item/token so I'm not clear about this one] 6. **[OKAY]** **Is match-all mode too risky as an escape hatch for small sites?** Minimal mode is safer but requires understanding subscriptions or always-include overrides. [match all is an advanced feature they have to choose, so I think it's fine to keep as is, we can add more info to the description if necessary though] --- ## Acceptance criteria (for this discussion issue) - [ ] Product/UX and engineering agree the list above accurately reflects current behavior - [ ] Open gaps for 1.0 are identified (must-have vs nice-to-have vs post-1.0) - [ ] Follow-up issues created for any agreed changes ## AI disclosure - [x] AI Assisted Issue