bug: #3586661 Twig autoescaping in AiPromptHelper::renderPrompt() corrupts {{...

Ann Mary Sruthyrequested to merge
issue/ai-3586661:3586661-twig-autoescaping-in-aiprompthelperrenderprompt-corrupts-context-and-raw-context-before-they-reach-the-provider into 1.x

Description

{{ context }} reaches the AI provider with HTML entities still in it, so the model receives entity soup instead of the field's text.

generateTokens() builds the context token with strip_tags(), which removes markup but leaves entities behind. Given a Body value stored by CKEditor as

Tom & Jerry's "best" episode

, the model was handed:

Tom & Jerry's "best" episode

instead of:

Tom & Jerry's "best" episode

The changes

context is now decoded with ENT_QUOTES | ENT_HTML5 after tags are stripped, at all four sites that produce it:

  • RuleBase::generateTokens() - reuses the existing RuleBase::decodeLabel() helper rather than adding a second definition of how this project decodes entities. That helper was already used on the model-output side by Options, Lists and Taxonomy; this puts it on the input side too.
  • SimpleTextCompletion::generateTokens() - a redundant override that shadows its parent, so it needed the same change or "LLM: Text (simple)" automators would stay broken. It is also the plugin used in the issue's reproduction steps.
  • ExternalBase::generateTokens() - inlines html_entity_decode() because this class deliberately does not extend RuleBase, so decodeLabel() is out of reach. Commented as such at the call site.
  • RichTextImageDescriptionTrait::appendImageDescriptionsToTokens() - the ?? fallback, kept in step with RuleBase. Inert in practice (all four callers invoke it right after parent::generateTokens(), which always sets context), changed for consistency.

NumericRule, Options, Lists, Taxonomy, Boolean and LlmModerationState all delegate to parent::generateTokens() and inherit the fix with no edit.

raw_context is deliberately NOT decoded. It is documented as the field value exactly as stored including HTML, entities are part of valid HTML, and decoding would corrupt a body that intentionally displays <div> as literal text.

Testing instructions

  1. On a content type, add a Body field, Text (formatted, long).

  2. Add a second Text (formatted, long) field and enable an automator on it.

  3. Input mode: base; base field: Body.

  4. Prompt:

    Repeat the following text back to me exactly, character for character:
{{ raw_context }}

  5. Create a node. In Body type: Tom & Jerry's "best" episode - CKEditor stores

    Tom & Jerry's "best" episode

    . Leave the automator field empty, save.

  6. Re-edit the node and inspect what the model was sent. Repeat with {{ context }}.

Checklist

  • I have linked the related issue in the MR title or description
  • I have performed a self-review of my own code
  • I have added or updated tests, or explained in the description why this change is not covered by tests
  • I have updated documentation for any new or changed functionality
  • I have written testing instructions and verified them locally
  • I have noted any required post-merge steps (config imports, cache rebuilds, manual changes)
  • This MR contains no breaking API or hook changes, or they are explicitly documented in the description

AI Compliance

Note

Check the one that best describes your usage, or leave all unchecked if AI was not significantly used.

  • AI Assisted Code
    Mainly written by a human; AI used for autocomplete or partial generation under full human supervision.
  • AI Generated Code
    Mainly generated by AI, reviewed and approved by a human before this MR was created.
  • Vibe Coded
    Generated by AI and only functionally reviewed before this MR was created.

Closes #3586661

Merge request reports