Issue #3576613: Truncate string field values exceeding Lucene's MAX_TERM_LENGTH to prevent max_bytes_length_exceeded_exception during indexing

Arlina Espinoza Rhotonrequested to merge
issue/search_api_opensearch-3576613:3576613-max-bytes-exceeded into 2.x

Issue #3576613: Truncate string field values exceeding Lucene's MAX_TERM_LENGTH to prevent max_bytes_length_exceeded_exception during indexing.

Issue

OpenSearch's underlying Lucene engine enforces a hard 32,766-byte limit on term length (IndexWriter.MAX_TERM_LENGTH). When a string-typed Search API field contains a value that exceeds this limit, indexing fails with an unrecoverable max_bytes_length_exceeded_exception, which stops the rest of the items in a batch from being indexed.

Solution

  • IndexParamBuilder::buildFieldValues() now passes string-typed values through a truncateToMaxBytes() guard before indexing.
  • Values within the limit are returned unchanged. Values that exceed it are truncated using mb_strcut(), which respects UTF-8 multibyte character boundaries to avoid producing malformed byte sequences.
  • A warning-level log message is emitted to alert site builders that a value was truncated, with a suggestion to switch the field type to fulltext if long text content is expected.
  • Other field types, including fulltext (stored as TextValue objects) are not affected or changed.
  • Added test coverage.

Merge request reports