Issue #3615876: Repair double-encoded UTF-8 in extracted field values

HtmlDocument's charset-recovery heuristic can misfire on already-valid UTF-8 source, re-encoding it a second time. This produced visibly wrong punctuation (e.g. "â" in place of dashes and quotes) plus raw C1 control bytes embedded alongside it, which some downstream consumers of the extracted text (e.g. embeddings APIs) reject outright.

FieldMapperService now repairs this automatically on every extracted value, for every crawl target, right after extraction. The repair is generic (detects and reverses the double-encoding pattern itself) rather than matching a fixed list of known-bad characters, and is careful not to touch genuinely correct Latin-1-range characters, including a real non-breaking space sitting directly beside actual corruption.

Closes #3615876

Merge request reports

Loading