Yinsongxu/LLM2Jev
Turn local language models into Jev-style structured decision models. Get results from text and images with prefill alone—no token-by-token decoding required.
LLM2Jev — What It Does
LLM2Jev is a tool that turns an existing large language model (LLM) or multimodal model (MLLM — one that also understands images) into a decision engine. Instead of just generating text, the model can now answer structured questions like "which category does this text belong to?" or "how well does this image match this instruction?" — and it does this without any training or fine-tuning. You keep your current model weights and add decision-making on top.
It is an independent open-source project, not affiliated with Jev or TypeSafe (the companies behind the "Jev-style" decision format it mimics).
Core Capabilities
- Four question types, mixable in one request:
Choice— pick one option from a set, get probabilities for each.Score— rate something against a rubric.Noul— yes/no probability.Box— locate a target in an image, returning normalized bounding-box coordinates.
- Works with text and images: you can put images and text in the
state(the situation) orinstructions(what to evaluate). - Fast, prefill-only scoring: for
Choice,Score, andNoul, the model reads the relevant logits (raw prediction scores) without generating a full answer. OnlyBoxactually generates coordinates. This makes decisions much faster than normal text generation. - Shared infrastructure: chat and decisions run on the same model and service (with the SGLang backend), and you call both through a single Python client that extends the official OpenAI SDK — so you keep familiar interfaces, authentication, timeouts, and retries.
Supported Backends
LLM2Jev works with three model-serving backends:
- SGLang — for Linux with NVIDIA GPUs (the quick-start path).
- Transformers — the standard Hugging Face pipeline.
- MLX — for Apple Silicon Macs.
All three support text and image scoring, and the SGLang and MLX backends also expose a compatible HTTP endpoint (POST /v1/systemone).
How It's Fast: Shared-Prefix Reuse
When you ask several questions about the same state (e.g., "My card was charged twice"), all candidates share that text. LLM2Jev exploits this:
- With SGLang, it first scores a real "criteria" candidate to warm the Radix Cache, then submits the remaining candidates so they reuse the cached prefix — avoiding repeated computation.
- With MLX, it explicitly prefills the shared prefix before scoring each candidate suffix.
This "staged" approach reduces work significantly for long inputs with many candidates, and both backends keep the same binary scoring interface.
Quick Start (SGLang on Linux)
- Clone the repo, install with
uv sync --extra sglang. - Start a server:
llm2jev-serve --backend sglang --model-path /path/to/model \ --served-model-name local-model --host 127.0.0.1 --port 30000 - In Python, use one client for both decisions and chat:
from llm2jev import Client, Choice client = Client(base_url="http://localhost:30000/v1", api_key="unused") decision = client.systemone.create( model="local-model", state="My card was charged twice.", questions={"department": Choice(criteria={ "billing": "Charges and billing", "shipping": "Delivery", })}, ) print(decision.choices["department"].probabilities) chat = client.chat.completions.create( model="local-model", messages=[{"role": "user", "content": "How can I request a refund?"}], ) print(chat.choices[0].message.content)
Demos
Interactive examples included in the repo:
- Web demo — submit questions and inspect probabilities in a browser.
- Snake demo — a game where the model makes movement decisions.
- MuJoCo pick-and-place — a robot-arm task driven by model decisions.
- Valen Sokoban — a visual Sokoban puzzle with single-step action evaluation.
Benchmarks (from the README)
- JevBench public accuracy: 231 public items, LLM2Jev with Qwen3.5-4B scored 76.2% accuracy.
- Latency: LLM2Jev P50 48 ms / P95 346 ms, vs. Jev 1.13.0's P50 652 ms / P95 722 ms — over 10× faster at the median.
- Valen-Eval-Game: 500 image-conditioned Sokoban questions, measuring tie-aware optimal-action accuracy using the multimodal
Choicepipeline.
Note: the README states that gains depend on input length, candidate count, and cache state.
Roadmap Status
- ✅ Interactive web demo
- ✅ Initial local-image support for Transformers and SGLang
- ✅ More multimodal tasks and demos
- ⬜ More benchmarks across model sizes, datasets, and workloads (decision quality, latency, throughput)
License
Apache License 2.0.
All facts above are taken directly from the project's README. No features or numbers were added beyond what the README states.
Related
- Project
- Project
- Dispatch
- Project
- Project