ekimetrics/adaptive-chunking

What it solves

Adaptive Chunking addresses the problem where no single chunking strategy (e.g., recursive, semantic, or page-based) works optimally for every document in a Retrieval-Augmented Generation (RAG) pipeline. Instead of using a fixed method, this framework automatically selects the best chunking strategy for each individual document based on a set of intrinsic quality metrics.

How it works

The system evaluates multiple chunking strategies against five intrinsic quality metrics that do not require ground-truth answers:

• Size Compliance (SC): Checks if chunks stay within target token-count bounds.
• Intrachunk Cohesion (ICC): Measures semantic similarity between a chunk's sentences and its overall embedding.
• Contextual Coherence (DCC): Measures similarity between a chunk and its surrounding context.
• Block Integrity (BI): Ensures structural blocks like paragraphs, tables, and lists remain intact.
• Filtered Missing Reference Error (RC): Tracks whether coreference chains (entity-pronoun pairs) are broken across boundaries.

Based on these scores, the framework selects the optimal strategy. It supports modular plug-ins for new chunking methods and evaluation metrics, and integrates with various PDF/Excel parsing backends (Docling, PyMuPDF, Azure Document Intelligence).

Who it’s for

Developers and researchers building RAG pipelines who want to optimize retrieval completeness and answer correctness by improving how documents are split into chunks.

Highlights

• Automatic Strategy Selection: Dynamically chooses the best splitting method per document.
• Intrinsic Evaluation: Five metrics to score chunking quality without needing labeled data.
• Modular Architecture: Easily extendable with custom splitters and scoring functions.
• Parsing Integration: Built-in support for multiple PDF and Azure cloud parsing backends.
• Proven Performance: Demonstrated improvements in retrieval completeness and answer correctness over standard recursive or page splitting.