dottxt-ai/outlines-core

outlines‑core – Structured Generation in Rust

What it is – A Rust library (with Python bindings) that lets you turn a JSON schema into a deterministic finite‑state automaton (DFA) that can guide token generation from a language‑model vocabulary. It is the performance‑focused core of the original Outlines project.

Key capabilities

• Schema → regex – Convert a JSON schema into a regular expression (json_schema::regex_from_str).
• Vocabulary handling – Load a tokenizer from a pretrained model (e.g., openai-community/gpt2) or build one manually, mapping tokens to integer IDs.
• Index object – Combine the regex and a Vocabulary to produce an Index that efficiently maps token IDs to DFA state transitions.
• Guide abstraction (Python) – A thin wrapper (Guide) that tracks the current DFA state, reports allowed tokens, and advances the state as tokens are consumed.
• Cross‑language – The core is written in Rust for speed and safety; a pyo3‑based Python package (outlines_core) exposes the same API to Python users.

Typical workflow

1. Write a JSON schema describing the desired output shape.
2. Generate a regex from the schema.
3. Load or construct a Vocabulary that matches the tokenizer of the LLM you will use.
4. Build an Index with the regex and vocabulary.
5. In Python (or Rust) create a Guide from the index and repeatedly:
   • query guide.get_tokens() for the set of token IDs the model is allowed to emit next,
   • feed the chosen token back via guide.advance(token_id),
   • stop when guide.is_finished().

Why it matters – By constraining generation to tokens that keep the output valid with respect to a schema, you get structured LLM outputs (e.g., JSON objects, dates, IDs) without post‑hoc parsing or costly beam search.

Getting started

• Rust: cargo add outlines-core then follow the code snippet in the README.
• Python: Clone the repo, create a virtualenv, and run pip install -e .[test]. Use the outlines_core package as shown in the Python example.
• Build the native extension for debugging with make build-extension-debug.

Development & contribution

• Fork → clone → set up a Python venv.
• Run make test (runs both Rust cargo test and Python pytest).
• Style checks via make pcc (pre‑commit). Benchmarks with make pybench.
• Join the community on Discord or open issues/PRs on GitHub.

License – MIT (see the repository’s LICENSE file).

---

Outlines‑core is a genuine, low‑level building block for AI developers who need fast, schema‑aware token filtering when working with large language models.