ekzhang/openjev-sglang

Jev-compatible API endpoint based on open models (prefill-only)

What it solves

It provides a server implementation of the TypeSafe/Jev HTTP API, allowing users to perform high-efficiency classification and scoring tasks using the Qwen3.6-35B-A3B model. It specifically addresses the need for fast, one-token readouts for multiple questions (Noul, Choice, and Score) within a single request, avoiding the overhead of long-form generation.

How it works

The server uses SGLang as the inference backend with a Rust frontend, radix caching, and breakable prefill CUDA graphs. When a request arrives, it renders a native chat template once and splits it into a common prefix and multiple question suffixes. It warms the SGLang radix cache with the common prefix and then concurrently sends the prefix plus each question suffix to the backend. Each call requests only one token (max_new_tokens=1) and retrieves the log-probabilities of specific answer labels. These probabilities are then renormalized using a stable softmax to determine the final answer (e.g., the argmax for Choice or a weighted sum for Score).

Who it’s for

Developers who need a high-throughput, low-latency classification API compatible with the Jev API, particularly those deploying on NVIDIA B200 GPUs via Modal.

Highlights

  • High-Efficiency Inference: Uses a prefill-plus-first-token-readout approach to avoid autoregressive generation.
  • Radix Caching: Optimizes performance by reusing common prompt prefixes across multiple questions.
  • B200 Optimized: Designed for deployment on B200 GPUs with SGLang's Rust frontend.
  • Multi-Question Support: Handles up to 64 questions per request, each with multiple answer options.
  • Modal Integration: Includes built-in support for easy deployment and autoscaling on Modal.

Related

  • Project
    Yinsongxu/LLM2Jev

    A framework that converts local LLMs and VLMs into structured decision models by extracting probabilities from logits during prefill, bypassing token-by-token decoding for lower latency.

  • Project
    mode-io/vllm-jev

    vLLM Jev is a fast HTTP serving wrapper for Jev‑style decision models (choice, yes/no, score). It runs on Linux GPUs via native vLLM and on Apple Silicon via MLX/PyTorch, handling text, images, and short videos. Install with `uv pip install .`, start a server with `vllm-jev serve <model-id>`, and query `/v1/systemone` for structured answers. Benchmarks show 4‑43× latency reductions versus reference implementations.

  • Dispatch
    OpenJev: Browser-Based Decision Modeling and Logit Readout

    OpenJev is a local, browser-based experiment that demonstrates the performance difference between direct logit readout for decision-making and traditional token generation using open-weight models.

  • Project
    nokia-applied-research/AnyJev

    A library that turns any LLM into a reliable decision model by removing position and label bias to provide calibrated probabilities for typed decisions.

  • Dispatch
    Jev in 25 Lines of Python – How a Minimal Local Classifier Works

    The post shows that a 25‑line Python script can replicate Jev’s classification behavior by loading a GGUF LLM, prompting it with labeled choices, and converting the final token logits into calibrated probabilities.