GigaToken: Achieving 1000x Faster Language Model Tokenization

GigaToken delivers massive throughput gains for LLM data preparation

GigaToken is a high-performance language model tokenizer designed to process text data at gigabytes per second (GB/s). By optimizing pretokenization and caching mechanisms, it achieves throughput speeds up to 1,000x faster than HuggingFace's tokenizers and significantly outperforms tiktoken across a wide range of modern x86 and ARM CPUs.

While tokenization represents a small fraction of total inference time, GigaToken is primarily designed for offline pre-training data preparation. At its peak performance on high-core-count CPUs, the tool can tokenize the entirety of Common Crawl (approximately 130 trillion tokens) in under 6.5 hours.

Technical Optimizations for Extreme Speed

GigaToken achieves its performance gains by replacing generic implementations with hardware-aware optimizations. The author identifies three primary areas of improvement:

  • SIMD-Accelerated Pretokenization: Most tokenizers outsource pretokenization to a Regex engine. GigaToken replaces this with a heavily optimized implementation using Single Instruction, Multiple Data (SIMD) to process multiple data points simultaneously.
  • Optimized Cache Hierarchies: GigaToken implements a highly efficient caching system for pretoken mappings. This allows the system to look up encoded tokens for previously seen words quickly, overcoming the challenge of long-tailed pretoken distributions.
  • Reduced Overhead: The implementation minimizes branching, reduces interactions with Python, and eliminates communication between threads to ensure maximum parallelism.

Performance Benchmarks

GigaToken demonstrates consistent speedups across various hardware configurations and tokenizer types. The following data represents encoding throughput on the owt_train.txt (11.9 GB) dataset:

High-Core Count Server (AMD EPYC 9565, 144 Cores)

On server-grade hardware, GigaToken reaches throughputs exceeding 24 GB/s for certain models. For example, GPT-2 tokenization reaches 24.53 GB/s, compared to 24.8 MB/s for HuggingFace tokenizers (a 989x increase).

Consumer Hardware (Apple M4 Max, 16 Cores)

On ARM-based consumer hardware, GigaToken maintains high efficiency. For GPT-2, it achieves 8.79 GB/s, which is 1,268x faster than HuggingFace's 6.9 MB/s.

Desktop Hardware (AMD Ryzen 7 9800X3D, 16 Cores)

On standard desktop CPUs, GigaToken provides significant gains, such as 6.27 GB/s for GPT-2, representing a 106x speedup over HuggingFace.

Note on SentencePiece: The author notes that SentencePiece-based tokenizers are not yet as optimized as BPE tokenizers, resulting in lower relative speedups (often 7x to 22x) for models like Gemma and CodeLlama.

Integration and Usage

GigaToken provides two primary ways to integrate into existing workflows:

Gigatoken API (Maximum Performance)

For the highest throughput, the Gigatoken API allows the Rust implementation to read data directly from files, bypassing Python overhead.

import gigatoken as gt

tokenizer = gt.Tokenizer("Qwen/Qwen3-8B")
file_source = gt.TextFileSource(["owt_train.txt"], separator=b"<|endoftext|>")
tokens = tokenizer.encode_files(file_source)

Compatibility Mode (Ease of Use)

GigaToken can act as a drop-in replacement for HuggingFace or Tiktoken. While this mode incurs a performance penalty to ensure exact output matching, it remains significantly faster than the original libraries.

import gigatoken as gt

# For HuggingFace
hf_tokenizer = ...
tokenizer = gt.Tokenizer(hf_tokenizer).as_hf()
tokens = tokenizer.encode_batch(["This is a test string", "And here is another"])

Limitations and Known Issues

  • WordPiece Support: WordPiece is currently not supported.
  • SentencePiece Optimization: SentencePiece-based tokenization is a low priority and currently lacks the optimization level of BPE.
  • Windows Support: The library is primarily tested on Linux and macOS; Windows users are encouraged to use WSL.
  • Python Overhead: Current Python iteration uses ABI3, which is slower than version-specific CPython APIs. Future updates aim to specialize for each Python version to potentially double speed in overhead-bound cases.

Community Perspectives

Discussion among developers highlights the tension between micro-optimization and system-wide bottlenecks. Some users noted that tokenization is often less than 0.1% of total inference time, making the 1000x speedup negligible for real-time applications. However, others argued that for massive dataset preparation, the efficiency is critical:

"When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets."

Other contributors compared the approach to SimdJson, noting that the massive performance leap is achieved through "creative programming" and hardware-level optimization.

Sources

Related