segment-any-text/wtpsplit

What it solves

wtpsplit is designed to solve the problem of robust and efficient sentence segmentation across a wide variety of languages and domains. Unlike traditional tools, it can handle text that lacks punctuation or has inconsistent styling, providing a universal approach to splitting text into semantic units like sentences or paragraphs.

How it works

The project implements two primary models: SaT (Segment any Text) and WtP (Where’s the Point?). SaT is the state-of-the-art approach that predicts newline probabilities to determine where natural splits occur. It supports 85 languages and can be adapted to specific styles or domains using LoRA (Low-Rank Adaptation) modules.

For precise control, it uses the Viterbi algorithm to find globally optimal segmentation points by balancing the model's predictions with user-defined length constraints (min/max length) and statistical priors (e.g., Gaussian or Log-normal distributions).

Who it’s for

This tool is for developers and researchers working with multilingual text processing, NLP pipelines, and applications that require precise control over segment lengths for downstream tasks like embedding models or storage.

Highlights

• Multilingual Support: Robust segmentation across 85 different languages.
• Adaptable: Uses LoRA modules for domain-specific adaptation (e.g., legal documents, tweets, ASR transcripts).
• Length Constraints: Ability to enforce minimum and maximum segment lengths using optimized algorithms.
• High Performance: Supports ONNX inference for faster processing and provides various model sizes (1-layer to 12-layer) to balance speed and accuracy.
• Paragraph Segmentation: Can segment text into paragraphs in addition to sentences.