tiktoken-go/tokenizer

Pure Go implementation of OpenAI's tiktoken tokenizer

What it solves

This project provides a pure Go implementation of OpenAI's tokenizer, allowing Go developers to encode and decode text into tokens that are compatible with OpenAI's models. This eliminates the need to rely on external dependencies or JavaScript wrappers when processing text for LLM inputs.

How it works

The library converts text into token IDs (encoding) and converts those IDs back into original text (decoding). It embeds OpenAI's vocabularies as Go maps, which are compiled directly into the binary. This differs from the Python version of tiktoken, which downloads dictionaries at runtime; by compiling them in, the library achieves faster startup times and performance.

Who it’s for

Go developers who need to calculate token counts or prepare text for OpenAI's models without leaving the Go ecosystem.

Highlights

  • Pure Go Port: No external C or JavaScript dependencies.
  • Multiple Encoding Support: Supports cl100k_base, o200k_base, r50k_base, p50k_base, and p50k_edit encodings.
  • Integrated CLI: Includes a command-line tool for encoding, decoding, and calculating tokens.
  • Compiled Vocabularies: Vocabularies are embedded as maps for improved performance and startup speed.

Related

  • Project
    niieani/gpt-tokenizer

    A high-performance TypeScript Byte Pair Encoder/Decoder for OpenAI models that allows developers to accurately count tokens and estimate API costs in JavaScript environments.

  • Project
    openai/tiktoken

    A fast BPE tokenizer for OpenAI models that converts text into tokens for model processing with high efficiency and lossless compression.

  • Project
    sashabaranov/go-openai

    An unofficial Go client library that simplifies integration with the OpenAI API for text generation, image creation, audio processing, and more.

  • Project
    jdkato/prose

    A pure-Go natural language processing library for English text that provides tokenization, part-of-speech tagging, and named entity recognition.

  • Project
    knuddelsgmbh/jtokkit

    JTokkit is a zero‑dependency Java library that reproduces OpenAI’s tokenisation algorithms (e.g., cl100k_base) for fast, accurate token counting and conversion on the JVM. It offers a thread‑safe API, Maven/Gradle distribution, and extensibility for custom encodings, all under an MIT license.