huggingface/tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

What it solves

Tokenization is a critical pre-processing step for machine learning models. This library provides a high-performance implementation of the most widely used tokenization algorithms, allowing developers to convert raw text into tokens that models can understand.

How it works

The project is implemented in Rust for maximum speed, with bindings for Python, Node.js, and Ruby. It allows users to to train new vocabularies using algorithms like Byte-Pair Encoding (BPE), WordPiece, or Unigram. It handles the entire pre-processing pipeline, including normalization with alignment tracking, pre-tokenization (such as splitting by whitespace), and post-processing tasks like truncation, padding, and the addition of special tokens.

Who it’s for

It is designed for researchers and production engineers who need a fast, versatile, and scalable way to tokenize text for AI models.

Highlights

  • own high-speed Rust implementation that can tokenize a gigabyte of text in under 20 seconds on a server CPU.
  • Support for multiple tokenization models including BPE, WordPiece, and Unigram.
  • Full pre-processing suite including padding, truncation, and alignment tracking for normalization.
  • Multi-language support via bindings for Python, Node.js, and Ruby.

Related

  • Project
    marcelroed/gigatoken

    A high-performance Rust-based tokenizer for language modeling that provides up to 1000x speedup over HuggingFace tokenizers by using SIMD-optimized pretokenization.

  • Project
    lindera/lindera

    A high-performance morphological analysis library in Rust for tokenizing and Japanese text, providing bindings for Python and WebAssembly.

  • Project
    openai/tiktoken

    A fast BPE tokenizer for OpenAI models that converts text into tokens for model processing with high efficiency and lossless compression.

  • Project
    robertknight/ocrs

    A Rust-based OCR library and CLI tool that uses neural network models to extract text from images with minimal preprocessing.

  • Project
    lucasjinreal/Kokoros

    A high-performance Rust implementation of the Kokoro TTS model providing fast text-to-speech synthesis via a CLI, Rust crate, and OpenAI-compatible API server.