featherless-ai/simple-jev

Turn any open model into a classifier/jev endpoint

What it solves

Simple Jev allows users to perform structured classification and scoring using open-source language models without needing to train a separate classifier head. It eliminates the need for the model to generate and parse prose or JSON responses token-by-token, which reduces latency and ensures the output always follows a strict schema.

How it works

Instead of autoregressive generation, the system uses the model's next-token logits (the raw probability scores for the next possible token) to determine the answer.

  1. Shared Context: The server processes a common input (state or chat history) once and saves the KV cache.
  2. Branching: It creates separate scoring branches for each question in a request, reusing the shared prefix to avoid redundant computation.
  3. Logit Scoring: The server reads the logits for specific allowed answer labels and normalizes them to build a JSON response containing choices, rubric scores, or truth judgments.

Who it’s for

Developers who need fast, structured decisions (like routing, urgency scoring, or fact-checking) from LLMs and want to avoid the overhead and instability of JSON generation/parsing.

Highlights

  • Prefill-only Scoring: Avoids the decode loop entirely by using next-token logits, significantly reducing token processing overhead.
  • Choice, Score, and Noul: Supports three distinct question types: multiple-choice (up to 255 options), rubric-based scoring (up to 50 levels), and truth/support judgments (Noul).
  • KV Cache Reuse: Optimizes performance by processing shared context once across multiple questions in a single request.
  • Model Agnostic: Compatible with various open models from Hugging Face, including Qwen, Gemma, and LLaVA-family vision models.
  • Evaluation Tooling: Includes a framework for prompt searching and accuracy benchmarking (JevBench).

Related

  • Dispatch
  • Project
  • Dispatch
  • Project