knuddelsgmbh/jtokkit

JTokkit – Java Tokenizer Kit

What it is – A pure‑Java library that implements the same tokenisation schemes OpenAI’s models use (e.g., cl100k_base, p50k_base, etc.). It lets Java applications count or split text into the exact tokens that the OpenAI API expects, mirroring the behaviour of OpenAI’s Python tiktoken package.

Why it matters – When you call GPT‑3.5, GPT‑4, or the embedding models you must stay within a token limit. JTokkit gives JVM‑based services a fast, zero‑dependency way to compute those limits without calling the API just for counting.

Key features

• Supports all major OpenAI encodings (r50k_base, p50k_base, p50k_edit, cl100k_base, o200k_base).
• Simple API: create an EncodingRegistry, fetch an Encoding by name or model, then encode/decode.
• Thread‑safe objects that can be shared across the application.
• No external dependencies; works on Java 8+.
• Benchmarked to be 2–3× faster than comparable Java tokenisers.

Getting it

<!-- Maven -->
<dependency>
  <groupId>com.knuddels</groupId>
  <artifactId>jtokkit</artifactId>
  <version>1.1.0</version>
</dependency>

// Gradle
implementation 'com.knuddels:jtokkit:1.1.0'

Quick usage example

EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
Encoding enc = registry.getEncoding(EncodingType.CL100K_BASE);
IntArrayList tokens = enc.encode("Hello world!");   // → [15496, 995]
String back = enc.decode(tokens);                    // → "Hello world!"

// Get the encoder that matches a specific model
Encoding modelEnc = registry.getEncodingForModel(ModelType.TEXT_EMBEDDING_ADA_002);

Extending it – If you need a custom BPE vocabulary you can either implement the Encoding interface yourself and register it, or supply a GptBytePairEncodingParams with your own regex, token maps, and special‑token map, then register that with the EncodingRegistry.

Documentation & source – Full Javadoc is hosted at https://javadoc.io/doc/com.knuddels/jtokkit and a short guide lives at https://jtokkit.knuddels.de/.

License – MIT (permissive, commercial‑friendly).