Text splitters break long documents into smaller chunks that can be retrieved individually and fit within the model context window limit.

There are several strategies for splitting documents, each with its advantages.

• Text structure-based — Text is naturally organised into hierarchical units such as paragraphs, sentences, and words. We can create a division that maintains natural-language flow, preserves semantic coherence within the partition, and adapts to varying levels of text granularity.
• Length-based — An intuitive strategy is to split documents by length. This approach ensures that each chunk doesn’t exceed a specified size limit. We can split text:
  • Based on tokens, which helps work with language models.
  • Based on characters, which is more consistent across different types of text.
• Document structure-based — If a document has an inherent structure, such as HTML, Markdown, or JSON, it is better to split it by structure, as it often naturally groups semantically related text.