mlc-ai/xgrammar

XGrammar – Structured Generation Engine for LLMs

What it is – XGrammar is an open‑source library that lets large language models (LLMs) produce outputs that strictly follow a given structure (JSON, regex, custom context‑free grammars, etc.). It does this with constrained decoding, guaranteeing 100 % structural correctness while adding almost no latency.

Why it matters – Many AI applications need reliable, machine‑readable output (e.g., APIs returning JSON, agents that must follow a plan template, or tools that generate code snippets). Traditional LLM decoding can produce malformed text, requiring post‑processing or retries. XGrammar moves the correctness guarantee into the generation step, saving time and reducing bugs.

Key Features

• Zero‑overhead JSON generation – benchmarked as “near‑zero” extra cost compared to plain text generation.
• General CFG support – any context‑free grammar can be supplied, enabling custom formats, regex‑based constraints, or domain‑specific languages.
• Cross‑platform & cross‑hardware – works on Linux, macOS, Windows; runs on CPU, NVIDIA/AMD GPUs, Apple Silicon, TPU, etc.
• Multi‑language APIs – Python (primary), plus official bindings for C++, JavaScript, and Swift.
• Broad model compatibility – tested with Qwen, Llama, DeepSeek, Phi, Gemma and others.
• Plug‑and‑play with major inference engines – default backend for vLLM, SGLang, TensorRT‑LLM, MLC‑LLM, and more.

Typical Use Cases

Scenario	How XGrammar helps
API services returning JSON	Guarantees the response is valid JSON without a separate validator.
Agentic LLMs that must follow a plan or script	Encode the plan as a CFG; the model can only emit tokens that keep the plan syntactically correct.
Data extraction / transformation from free‑form text	Define a regex or grammar for the desired fields; extraction becomes deterministic.
Code generation with language‑specific syntax rules	Supply the language grammar so the model never produces mismatched brackets or missing keywords.

Quick Start (Python)

pip install xgrammar            # basic install
pip install "xgrammar[metal]"   # optional Metal support for Apple Silicon

import xgrammar as xgr
# Example: generate a JSON object with a fixed schema
grammar = xgr.JSONGrammar({"name": "string", "age": "int"})
result = xgr.generate(model, grammar, prompt="Give me a user profile")
print(result)  # always valid JSON

Full docs, installation guides, and code examples are at https://xgrammar.mlc.ai/docs/.

Ecosystem Integration

• vLLM, SGLang, TensorRT‑LLM, MLC‑LLM – XGrammar is the default structured‑generation backend for these inference runtimes.
• Rust bindings – community‑maintained xgrammar-rs for Rust projects.
• Industry adoption – integrated into products from NVIDIA, Meta, Google, DeepSeek, Databricks, Modular, Perplexity, and others.

Release & Governance

• Releases are versioned on GitHub and published to PyPI (e.g., xgrammar==0.2.7).
• The project follows an open governance model with a CODE_OF_CONDUCT, CONTRIBUTING guide, and a GOVERNANCE.md that lists maintainers and decision processes.

License

• Apache License 2.0 – permissive, commercial‑friendly open‑source license.

Community & Citation

• Community – discussions on GitHub, DeepWiki badge, and a growing list of collaborators.
• Citation – if used in research, cite the 2024 ML Systems paper and the 2026 ACM conference paper (see README for BibTeX).

---

XGrammar is a practical tool for anyone building LLM‑powered systems that need reliable, structured output without sacrificing speed.