niieani/gpt-tokenizer

What it solves

gpt-tokenizer provides a high-performance TypeScript implementation of the Byte Pair Encoding (BPE) algorithm used by OpenAI models. It allows developers to accurately count tokens, encode text into integers, and decode tokens back into text within JavaScript and TypeScript environments, avoiding the need to rely on external Python-based tools like tiktoken.

How it works

The library is a port of OpenAI's tiktoken and supports all current OpenAI model families (including GPT-4o, o-series, and legacy models). It implements various encodings such as cl100k_base and o200k_base. It features an LRU (Least Recently Used) merge cache to speed up the encoding of similar strings and provides generator functions for streaming data.

Who it’s for

Developers building LLM-powered applications in TypeScript or JavaScript who need to manage token limits, estimate API costs, or process token streams in the browser or Node.js.

Highlights

• Comprehensive Model Support: Supports all current OpenAI model families, including GPT-5, GPT-4o, and o-series reasoning models.
• Performance: Claims to be the fastest and smallest footprint tokenizer on NPM, outperforming WASM/node binding implementations.
• Chat-Specific Tooling: Includes an encodeChat function to handle the specific tokenization of chat messages.
• Cost Estimation: Built-in estimateCost function based on a full OpenAI model catalog with pricing data.
• Limit Checking: A highly performant isWithinTokenLimit function to check if text exceeds a limit without encoding the entire input.
• Environment Agnostic: Works out-of-the-box in both the browser and Node.js environments.