mode-io/vllm-jev

What is vLLM Jev?

vLLM Jev is a thin wrapper that lets you serve Jev‑style decision‑making models (Choice, Yes/No, Score) through the high‑performance vLLM inference engine. It works on two main platforms:

Linux machines with an NVIDIA GPU (using native vLLM).
Apple Silicon Macs (using MLX or PyTorch MPS).

The tool downloads a Hugging Face checkpoint, compiles it with vLLM, and exposes a simple HTTP API (/v1/systemone). You send a question together with a list of candidate answers, and the server returns the most likely label and a probability for each option.

Why it matters

Speed: Benchmarks show 4‑43× latency reductions compared with the original PyTorch reference, especially when handling many concurrent requests.
Structured output: Instead of free‑form text, the models produce a discrete decision (choice, yes/no, or ordered score) which is easier to consume by downstream systems.
Multimodal support: Some models (e.g., Valen, Clef, RSI‑Jev) accept images and short video clips in addition to text.
Cross‑platform: First‑class support for Apple Silicon, a platform that traditionally lacks fast LLM serving.

How to get started

Install (Linux) with the lightweight uv package manager:
```
uv pip install .
```
On macOS run the helper script scripts/install_mac.sh.
Start a server for any supported checkpoint, for example:
```
vllm-jev serve ZefanCai/Open-Jev-2B   # Linux GPU
# or on Mac
vllm-jev serve Valen-Team/Valen-Preview-0923
```
The command downloads the model on first use and then launches vLLM at http://127.0.0.1:8795.

Send a request (Choice example):

curl -sS http://127.0.0.1:8795/v1/systemone \
     -H 'Content-Type: application/json' \
     -d '{"state":"I was charged twice and want a refund.",
          "questions":{"intent":{"type":"choice",
                                       "instructions":"Choose the customer intent.",
                                       "criteria":{"billing":"A payment or refund issue",
                                                    "technical":"A malfunction or setup issue",
                                                    "other":"Another request"}}}}'

The response contains answers.intent.choice and the per‑option probabilities.

What models can I run?

Model	Input types	Platforms
Open‑Jev‑2B / 9B	Text	Linux (GPU)
Laya family	Text	Linux, Mac
Valen‑Preview‑0923	Text, images, short video	Linux, Mac
vjev‑vision / vjev‑vision‑pilot	Text, images	Linux
RSI‑Jev (v3.0, v4.0‑VL)	Text, images	Linux
Clef‑Flash (9B) / Clef (27B)	Text, images, video	Linux
Tiny‑Jev, OpenJev‑0.6B, etc.	Text	Linux, Mac

All of these are launched with the same vllm-jev serve <model-id> command; the README lists the exact Hugging Face IDs.

Key features & knobs

Batching & KV‑cache – vLLM handles automatic request batching and re‑uses the KV cache for faster subsequent tokens.
CUDA BF16 precision mode – VLLM_JEV_BF16_MATMUL=no_splitk can reduce batch‑dependent rounding drift.
Online prefix cache (experimental) – caches repeated prompts for Open‑Jev‑2B on Linux.
Apple‑Silicon optimisations – adaptive batching, prefix reuse, bounded memory caching, and optional CUDA‑Graph‑style reads for multilingual Laya.
Standard vLLM flags – you can pass any vllm serve options (GPU memory utilisation, port, storage path, etc.).

Performance highlights (from the README)

Platform	Model	Input	Concurrency	Latency ↓	Throughput ↑
Linux + A800	Open‑Jev‑2B	Short text	8	4360 ms → 100 ms	1.83 → 78.8 req/s
Linux + A800	Laya English	MMLU sample	16	404 ms → 77 ms	39.5 → 203 req/s
Mac M5 (MLX)	Valen‑Preview	Image	1	231.9 ms → 77.3 ms	4.3 → 12.9 req/s

These numbers illustrate the dramatic speed‑up you get simply by serving through vLLM Jev.

Where to learn more

Full user guide: docs/guide.md (installation, serving options, request formats, video input, etc.)
Model‑specific details: sections Supported Models, Additional decision models, RSI‑Jev, Clef in the README.
Issues & discussions are hosted on the GitHub repo for community help.

TL;DR: vLLM Jev turns a growing family of decision‑oriented LLM checkpoints into a fast, batched HTTP service that works on both NVIDIA‑GPU Linux boxes and Apple Silicon Macs, supporting text, images, and short videos.