marcelroed/gigatoken

What it solves

Gigatoken is designed to eliminate the bottleneck of text tokenization during the preparation of massive datasets for language modeling. It provides a high-performance alternative to standard libraries like HuggingFace's tokenizers and tiktoken, which can be orders of magnitude slower when processing gigabytes or terabytes of text.

How it works

Written in Rust, Gigatoken achieves its speed by replacing standard regex-based pretokenization with a custom SIMD-optimized implementation. It minimizes branching, reduces communication between threads, and uses a highly optimized cache hierarchy for pretoken mappings to avoid redundant encoding of common words. It can be used as a drop-in replacement via compatibility modes for HuggingFace and tiktoken, or through its own native API for maximum performance by reading data directly from files.

Who it’s for

It is primarily for researchers and engineers working with large-scale language model training who need to tokenize massive amounts of text data (e.g., Common Crawl) across various CPU architectures (x86 and ARM).

Highlights

• Extreme Throughput: Capable of reaching speeds up to 24 GB/s on high-core count CPUs, claiming up to 1000x speedup over HuggingFace tokenizers.
• Broad Compatibility: Supports a wide range of popular tokenizers including Llama 3, Qwen, DeepSeek, and GPT-2.
• Flexible API: Offers both a high-speed native API and compatibility wrappers for existing HF and tiktoken workflows.
• Hardware Optimized: Optimized for modern CPUs using SIMD strategies across AVX512, AVX2, and NEON.