feyninc/chonkiejs

What it solves

Chonkiejs is a lightweight, efficient text chunking library for TypeScript developers. It solves the problem of splitting large texts into smaller, manageable pieces (chunks) for use in Retrieval-Augmented Generation (RAG) applications, providing a more flexible and faster alternative to existing heavy libraries.

How it works

The library provides several specialized chunking strategies to handle different types of content:

• Recursive Chunking: Splits text using a hierarchy of rules (paragraphs, sentences, punctuation, words, characters) to maintain context.
• Token-based Chunking: Splits text into fixed-size token chunks with optional overlap.
• Sentence Chunking: Groups sentences into token-sized chunks while respecting sentence boundaries.
• Semantic Chunking: Uses embedding similarity between sliding sentence windows to detect natural boundaries and split at "low-similarity valleys."
• Code Chunking: Uses AST-aware splitting via tree-sitter for source code.
• Table Chunking: Splits markdown or HTML tables into sub-tables while repeating the original header.
• Fast Chunking: A high-throughput byte-based chunker powered by WASM for maximum speed.

Who it’s for

TypeScript developers building RAG applications or any application that requires fast, on-the-fly text preprocessing and chunking.

Highlights

• Diverse Chunking Methods: Supports recursive, token, sentence, semantic, and specialized code/table chunking.
• Type-Safe: A TypeScript port of the original Python chonkie library.
• Lighweight: Designed to be no-nonsense and efficient with zero dependencies in the core package.
• HuggingFace Integration: Optional support for HuggingFace tokenizers via @chonkiejs/token.
• Cloud Integration: Offers cloud-based chunkers via @chonkiejs/cloud.