guidance-ai/llguidance

What it solves

llguidance solves the problem of "hallucinations" or formatting errors in LLM outputs by implementing constrained decoding. It ensures that a model's output strictly adheres to a specific structure—such as a JSON schema, a regular expression, or a context-free grammar—without requiring the model to be fine-tuned for that specific format.

How it works

The library computes a "token mask" in real-time during the decoding process. Given a grammar and the tokens already generated, it identifies exactly which tokens from the model's vocabulary are valid next steps to maintain the grammar's rules.

Technically, it uses Earley's algorithm for context-free grammar parsing and a lexer based on regular expression derivatives. It optimizes this process by traversing a prefix tree (trie) of all possible tokens, allowing it to calculate masks extremely quickly (averaging 50µs per token) with virtually no startup cost.

Who it’s for

It is designed for developers building production AI applications that require guaranteed structured outputs, as well as developers of LLM inference engines (like vLLM, SGLang, and llama.cpp) who want to integrate high-performance grammar enforcement.

Highlights

• Extreme Performance: Computes token masks in approximately 50µs of CPU time per token for large tokenizers.
• Broad Format Support: Supports JSON schemas, regular expressions, and Lark-style context-free grammars.
• Zero Startup Overhead: Unlike some competitors, it does not require lengthy pre-computation of automaton states.
• Wide Integration: Powering structured outputs for OpenAI models and integrated into vLLM, SGLang, llama.cpp, and Chromium.