google/sentencepiece

SentencePiece – Language‑Independent Subword Tokenizer

What it is – SentencePiece is a fast, lightweight library (C++ core with bindings for Python, Go, Rust, etc.) that learns and applies subword tokenization models directly from raw text. It supports the two most common subword algorithms – Byte‑Pair Encoding (BPE) and the Unigram language model – and produces a single, self‑contained .model file that can be used anywhere without extra preprocessing.

Why it matters for AI – Modern neural language models (e.g., LLMs, translation systems, speech‑to‑text pipelines) require a fixed‑size vocabulary. SentencePiece lets you create that vocabulary in a fully data‑driven way, handling whitespace, Unicode, and languages without explicit word boundaries (Chinese, Japanese, Thai, etc.) without any language‑specific tokenizers. The tokenization is lossless: the original string can be reconstructed exactly, which simplifies downstream detokenization.

Key features

• Language‑independent training – feed raw sentences; no external tokenizers needed.
• Two subword algorithms – BPE and Unigram, both with optional subword regularization (sampling) and BPE‑Dropout for data augmentation.
• Fast C++ implementation – ~50 k sentences/sec, ~6 MiB memory footprint.
• Zero‑dependency runtime (SentencePiece Lite) – a tiny C++20 library (~45 KB static lib) that can be memory‑mapped and used on‑device or in high‑throughput servers.
• Cross‑language bindings – Python (pip install sentencepiece), Go, Rust, and a pure‑C API.
• Self‑contained model files – the .model file bundles vocab, normalization rules, and segmentation logic, guaranteeing identical results across platforms.
• Benchmarked performance – outperforms Hugging Face’s fast tokenizers by 4‑10× on multi‑threaded workloads (see README tables).

Typical workflow (Python)

import sentencepiece as spm
# 1️⃣ Train from a raw text file
spm.SentencePieceTrainer.train(
    input='data.txt', model_prefix='my', vocab_size=8000)

# 2️⃣ Load the model
sp = spm.SentencePieceProcessor(model_file='my.model')

# 3️⃣ Encode / decode
ids   = sp.encode('Hello world!', return_type=int)
pieces = sp.encode('Hello world!', return_type=str)
print(sp.decode(ids))   # → "Hello world!"

The same model can be used from C++, Go, or Rust with equivalent APIs.

When to use it

• Building a new LLM, translation, or speech model that needs a custom subword vocabulary.
• Working with languages lacking clear word delimiters.
• Deploying tokenization on edge devices or high‑throughput servers where a tiny, dependency‑free runtime is required.
• Wanting deterministic, reversible tokenization across multiple programming environments.

Resources – The repository includes extensive docs (CLI guide, C++/Python API references, training‑option cheat sheets, Docker deployment instructions) and a performance benchmark suite for reproducibility.

License – Apache 2.0 (permissive, commercial‑friendly).