Skip to content

Improve chunking of embeddings.

Andrew Belcher requested to merge embedding-chunking into 1.0.x

Chunks are now:

  • A maximum of 1536 characters
  • Get split on whitespace
  • Include an overlap of at least 64 characters based on the position of whitespace

Merge request reports