featherless-ai/simple-jev
Turn any open model into a classifier/jev endpoint
What it solves
Simple Jev allows users to perform structured classification and scoring using open-source language models without needing to train a separate classifier head. It eliminates the need for the model to generate and parse prose or JSON responses token-by-token, which reduces latency and ensures the output always follows a strict schema.
How it works
Instead of autoregressive generation, the system uses the model's next-token logits (the raw probability scores for the next possible token) to determine the answer.
- Shared Context: The server processes a common input (state or chat history) once and saves the KV cache.
- Branching: It creates separate scoring branches for each question in a request, reusing the shared prefix to avoid redundant computation.
- Logit Scoring: The server reads the logits for specific allowed answer labels and normalizes them to build a JSON response containing choices, rubric scores, or truth judgments.
Who it’s for
Developers who need fast, structured decisions (like routing, urgency scoring, or fact-checking) from LLMs and want to avoid the overhead and instability of JSON generation/parsing.
Highlights
- Prefill-only Scoring: Avoids the decode loop entirely by using next-token logits, significantly reducing token processing overhead.
- Choice, Score, and Noul: Supports three distinct question types: multiple-choice (up to 255 options), rubric-based scoring (up to 50 levels), and truth/support judgments (Noul).
- KV Cache Reuse: Optimizes performance by processing shared context once across multiple questions in a single request.
- Model Agnostic: Compatible with various open models from Hugging Face, including Qwen, Gemma, and LLaVA-family vision models.
- Evaluation Tooling: Includes a framework for prompt searching and accuracy benchmarking (JevBench).
Related
- Dispatch
- Project
- Dispatch
- Project