huggingface/tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

🤖 What is huggingface/tokenizers?

A high‑performance, open‑source tokenization library written in Rust with bindings for Python, Node.js and (planned) other languages. It implements the tokenizers used by virtually every modern LLM – BPE, Unigram, WordPiece, WordLevel, etc. – and aims to be the industry‑standard engine for turning raw text into the integer IDs a model understands, and back again.

🚀 Quick start (Python)

pip install --pre tokenizers   # pre‑release contains the new v1 API

from tokenizers import Tokenizer

# Load a pretrained tokenizer (e.g. Llama‑3.1) directly from the Hub
tokenizer = Tokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

ids = tokenizer.encode("Hello, y'all! How are you 😁 ?").ids
print(ids)
# → [128000, 9906, 11, 379, 65948, 0, 2650, 527, 499, 27623, 223, 949]

print(tokenizer.decode([15339, 1917]))   # → "hello world"

⚙️ Core design

  • Rust core – the reference implementation, compiled to a tiny (~325 KB gzipped) binary with no mandatory SIMD dependencies.
  • Modular crates – separate sub‑crates for inference (tk‑encode), pre‑tokenization (bitcannon), JSON loading (tk‑serialize), legacy conversion (tk‑convert), training (tk‑train), and a dev‑only bitmap generator.
  • Zero‑copy, batch‑oriented API – upcoming v1 features include zero‑copy views on encoded IDs and a batch‑encoding API.
  • Hardware‑aware paths – NEON on ARM, AVX‑512/VBMI on modern x86, and portable fallbacks; SIMD is optional, never required for correctness.

📦 Language bindings

Language Status
Rust (reference) ✅
Python ✅
Node.js ✅
C / C++ / Java / Go 🚧 planned
Ruby community‑maintained external repo

🛣️ Roadmap highlights (v1.0.0)

  • Python API – bring back full training, batch encoding, offset outputs, padding, and stream decoding.
  • Performance – bitnorm normalization, faster training, better decode speed, word‑cache optimisations.
  • GPU‑ready prototype – tk‑devices explores keeping vocab and IDs on‑device for massive batch decoding.
  • Additional bindings – C/C++ core, then Java, Go, Swift, etc.

📈 Performance & footprint

  • The library ships a gzipped 325 KB crate; the full Rust source (including optional SIMD kernels) stays under 1 MB.
  • Benchmarks (see the linked interactive demo) show it beating existing tokenizers on CPU for all major pre‑tokenizer regexes.

📚 Documentation & resources

🎯 Who should use it?

  • Model developers needing fast, correct tokenization for training or inference.
  • Production services that serve LLMs at scale and want a tiny, dependency‑free runtime.
  • Researchers building custom tokenizers or experimenting with new pre‑tokenization algorithms.

📜 License

MIT – see the repository’s LICENSE file.

Related

  • Dispatch
    Hugging Face tokenizers v1 release notes

    Hugging Face has released a release candidate for tokenizers v1, which achieves 3 to 30 times faster encoding speeds than v0.23 through SIMD-based splitting, a word cache, and allocation-free merge loops.

  • Project
    marcelroed/gigatoken

    A high-performance Rust-based tokenizer for language modeling that provides up to 1000x speedup over HuggingFace tokenizers by using SIMD-optimized pretokenization.

  • Dispatch
    Hugging Face Accelerated Inference API Optimization

    Hugging Face achieved a 100x speedup in transformer inference for its Accelerated Inference API by combining high-level library optimizations, Rust-based tokenization, and hardware-specific compilation.

  • Dispatch
    Transformers v5 Tokenization Update

    Hugging Face has redesigned tokenization in Transformers v5 to separate tokenizer architecture from trained vocabulary, enabling easier inspection, customization, and training from scratch.

  • Dispatch
    GigaToken: Achieving 1000x Faster Language Model Tokenization

    GigaToken is a high-performance tokenizer that leverages SIMD and optimized cache hierarchies to provide up to 1000x speedup over HuggingFace tokenizers for large-scale text processing.