benbrandt/text-splitter

Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python.

What it solves

Large language models (LLMs) have limited context windows. This project provides tools to split long documents into smaller, manageable chunks that fit within these limits while preserving as much semantic meaning as possible by avoiding arbitrary cuts.

How it works

The library uses a hierarchical approach to splitting. It defines a set of "semantic levels" (e.g., paragraphs, sentences, words, characters) and attempts to split the text at the highest possible level that still fits within the specified chunk size.

It supports three specialized splitters:

  • TextSplitter: Uses Unicode boundaries for words and sentences, and newline sequences.
  • MarkdownSplitter: Uses the CommonMark spec to recognize headings, block elements, and inline elements.
  • CodeSplitter: Uses tree-sitter parsers to split based on the depth of the syntax tree.

Chunk size can be measured by character count or by using tokenizers from Hugging Face or Tiktoken.

Who it’s for

Developers building LLM applications (such as RAG systems) who need to preprocess text data into chunks for embedding or inference, specifically those using Rust or Python.

Highlights

  • Multi-format support: Dedicated splitters for plain text, Markdown, and source code.
  • Flexible sizing: Supports character counts, token counts (via Hugging Face and Tiktoken), or a range of capacities.
  • Semantic awareness: Prioritizes splitting at natural boundaries like sentences or syntax tree nodes rather than mid-word.
  • High performance: Written in Rust for efficiency.

Related

  • Project
    hydropix/TranslateBooksWithLLMs

    A desktop application that uses LLMs to translate books, subtitles, and documents while preserving original formatting and maintaining consistency across long texts.

  • Project
    drmingler/smart-llm-loader

    A Python package that converts documents and scanned images into clean, structured markdown chunks using OCR and LLM-powered semantic chunking for RAG systems.

  • Project
    segment-any-text/wtpsplit

    A robust, multilingual sentence and paragraph segmentation library that uses AI models to split text into semantic units, even without punctuation.

  • Project
    feyninc/chonkiejs

    A lightweight and efficient TypeScript text chunking library designed for RAG applications, offering multiple strategies including semantic, recursive, and code-aware splitting.

  • Project
    Abraxas-365/langchain-rust

    A Rust implementation of the LangChain framework for building composable LLM applications with support for various models, vector stores, and agents.