### Problem/Motivation In ai_eval, only the decision rubric judge uses a decision calibration. Other callers get the raw probability. For example, the "Decision: Boolean" automator compares the raw probability with its "True threshold" (default 0.5). A calibration curve can also make the probabilities worse when it is used on other answers. The table shows the held-out calibration error (ECE: the average gap between the probability and the observed rate; lower is better) on the D3code results from https://github.com/gkastanis/d3code-calibration: | Fit and test | Raw | Calibrated | |---|---|---| | Same model and question. Fit on one half of the messages, test on the other half. | 0.251 | 0.008 | | Same model and question. Fit on two message categories, test on the third. | 0.18 to 0.29 | 0.03 to 0.07 | | Fit on Jev answers for one half of the messages, test on Laya answers for the other half. | 0.033 | 0.119 (worse) | A curve from other content still helped, but less. A curve from another model made the probabilities worse. A curve fitted on Laya's own answers gave 0.009. Thus this issue applies a curve only to the callers that a site selects, and fits it on the reviewed answers of those callers. <details> <summary>How the numbers were made</summary> Script: https://github.com/gkastanis/d3code-calibration/blob/6306a4537a5d43779ac8a0ba0c9abef84c2906c9/transfer.py Run `python3 transfer.py results-compact.jsonl` in the study repository. No API key is necessary. The script uses isotonic regression (pool-adjacent-violators) with a step lookup, as `DecisionCalibrator` does, and 10 equal bins for ECE. The half splits are averaged over 20 seeds x 2 folds. Input: arms `jev-full` and `laya-full`, one answer for each message. </details> ### Proposed resolution Add an optional submodule. The AI module does not need a change. After each successful provider call through `ProviderProxy`, the AI module dispatches `PostGenerateResponseEvent`. Then it gives the caller the output from the event. 1. **Apply.** Subscribe to `PostGenerateResponseEvent` for the `decision` operation. Use a priority lower than 20, so that the AI module's `DecisionValidationEventSubscriber` (priority 20) validates the raw answer first. For each true/false answer (`NoulAnswer`), find the active calibration that matches all of these: * the provider; * the model that answered, as the provider reports it (`DecisionResponse::getModel()`). Do not use the model name in the request, because an alias such as `jev-latest` can change; * the question fingerprint, a hash of the question wording. Calculate it with `CalibrationFingerprint::of()`. The instructions can be a string or an array, and the criteria can be NULL. Use one mapping for these values, here and in the fit form; * the scope (item 2). If a calibration matches, rebuild the answer with the calibrated probability. Keep the raw probability and the calibration ID in the raw data of the answer. If no calibration matches, do not change the answer. Do not change score or choice answers, because the decision rubric judge calibrates its own pass probability. 2. **Scope.** A calibration lists the request tags that it applies to, for example `ai_automator:id:flag_comments`. Automators send `ai_automator:id:<id>` when the automator has an ID. They send the entity type, bundle and ID when there is an entity, and the field name when it is set (`RuleBase::getTags()`). A calibration with no scope applies to no caller. 3. **Record.** For a scope with recording on, store each true/false answer. Store the provider, the model that answered, the fingerprint, the raw probability, the request tags and the time. Store the entity reference from the tags, not the text that was sent, because that text can contain personal data. If the tags have no entity ID, store the answer without a link. An optional sample rate limits the number of rows. 4. **Review and fit.** A review page lists the recorded answers, with a link to each entity. For each answer, a reviewer records if the statement is true or false for that entity. The fit uses the rules of "From reviewed results": at least 30 reviewed answers, both true and false in the fitted answers, and answers held back to test the fit. A fit that makes the held-out Brier score worse is saved inactive. 5. **Check again.** Recording continues after a calibration becomes active. New reviews test the active curve. If the curve makes the Brier score worse than the raw probability (`CalibrationVerdict`), the Calibrations page shows a warning. If the provider reports a different model ID, no calibration matches, and the caller gets the raw probability again. ### Out of scope * Score and choice confidence. The AI module maintainers note that providers calculate confidence differently (see the comments in https://git.drupalcode.org/project/ai/-/work_items/3586798). * Sending uncertain answers to a chat model: https://git.drupalcode.org/project/ai/-/work_items/3586798 * AI 2.x, which does not have the Decision operation yet: https://git.drupalcode.org/project/ai/-/work_items/3586811 ### Remaining tasks * Agree with the AI module maintainers on documented keys for the raw probability and the calibration ID in the raw data of an answer. * Decide what to do when the provider reports no model: no correction, or use the model name from the request. * Decide if a curve that became worse is deactivated automatically or only shows a warning. * Test on a site with the Decision operation (AI 1.x): one "Decision: Boolean" automator. Record, review at least 30 answers, fit and activate. Then make sure that the field value follows the calibrated probability. AI-Generated: Yes (Claude Code drafted this description from the ai_eval and AI module code and calculated the numbers from the study results file. Codex checked the claims against the code.)