GigaToken: Achieving 1000x Faster Language Model Tokenization

GigaToken: Achieving 1000x Faster Language Model Tokenization

GigaToken delivers massive throughput gains for LLM tokenization

GigaToken is a high-performance tokenizer designed as a drop-in replacement for HuggingFace and Tiktoken, achieving encoding speeds up to 1,000x faster than existing industry standards. By optimizing the pretokenization process and implementing a sophisticated cache hierarchy, GigaToken enables the tokenization of massive datasets—such as the entirety of Common Crawl—in a fraction of the time previously required.

Performance Benchmarks

GigaToken demonstrates significant throughput increases across various CPU architectures, including AMD EPYC, Apple M4 Max, and AMD Ryzen. The most dramatic gains are seen with Byte-Pair Encoding (BPE) tokenizers.

High-End Server Performance (AMD EPYC 9565, 144 Cores)

On a dual-socket AMD EPYC setup, GigaToken reaches throughputs exceeding 24 GB/s for several models:

  • GPT-2: 24.53 GB/s (989x faster than HF)
  • Phi-4: 24.00 GB/s (801x faster than HF)
  • Llama 3/3.1/3.2: 22.15 GB/s (457x faster than HF)
  • DeepSeek V3/R1/V4: 19.69 GB/s (750x faster than HF)

Consumer and Mobile Hardware (Apple M4 Max, 16 Cores)

Performance remains consistently high on ARM architecture, with GPT-2 reaching 8.79 GB/s (1,268x faster than HF) and OLMo 2/3 reaching 7.56 GB/s (1,299x faster than HF).

Performance Limitations

While BPE tokenizers see massive gains, SentencePiece-based tokenizers (used in some Google and BERT-style models) are not yet fully optimized in GigaToken and show more modest speedups, typically ranging from 7x to 22x.

Technical Optimization Strategies

The performance leap in GigaToken is not the result of over-optimizing for a single hardware target, but rather a systemic overhaul of the tokenization pipeline.

SIMD and Regex Replacement

Most tokenizers outsource pretokenization to a Regex engine. GigaToken replaces this with a heavily optimized implementation using SIMD (Single Instruction, Multiple Data), which minimizes branching and maximizes data throughput.

Advanced Caching

GigaToken implements a highly optimized cache for pretoken mappings. Because pretoken distributions are long-tailed and the cache can grow rapidly, the project utilizes a specialized cache hierarchy to ensure that previously seen words are encoded efficiently without incurring massive memory overhead.

Reduction of Language Overhead

To maximize the efficiency of the Rust implementation, GigaToken minimizes interactions with Python and avoids communication between threads. The fastest performance is achieved via the GigaToken API, which allows Rust to read data directly from files, bypassing Python data structure overhead.

Integration and Usage

GigaToken provides two primary ways to integrate into existing workflows:

  1. Compatibility Mode: This acts as a drop-in replacement for HuggingFace or Tiktoken. While this mode ensures exact output matching, it incurs a non-negligible performance cost compared to the native API.
  2. GigaToken API: This is the fastest method, utilizing TextFileSource to allow the Rust backend to handle file I/O and parallelism directly.

Industry Impact and Use Cases

While tokenization represents a small fraction of total inference time for a single request, its impact is massive during the data preparation phase of LLM training.

"When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets."

By reducing the time to tokenize the entire Common Crawl dataset (approximately 130 trillion tokens) to under 6.5 hours on high-end hardware, GigaToken significantly accelerates the pre-training pipeline for foundation models.

Known Limitations

  • WordPiece: Not currently supported.
  • SentencePiece: Not as optimized as BPE tokenizers.
  • Windows: Not extensively tested; WSL is recommended for Windows users.
  • Python ABI: Currently uses ABI3, which is slower than version-specific CPython APIs; future updates aim to specialize for each Python version to further reduce overhead.

Sources