jevlike: Open-source Jev-like One-Pass Scorer for Text Options

Quick takeaway

jevlike provides a lightweight, open‑source implementation of a Jev‑style one‑pass scorer that assigns probabilities to a variable‑length list of text options, enabling fast, deterministic selection without token‑by‑token generation.

What the repository delivers

A minimal model that accepts a context string and an arbitrary list of options and returns a probability for each option in a single forward pass.
Reference implementations for two classic interactive environments (Doom and chess) that showcase the model scoring controller buttons directly from visual patches.
A synthetic data generator, training script, evaluation utilities, and a command‑line predictor for rapid experimentation.
Optional support for frozen pretrained encoders (e.g., Qwen2.5‑0.5B) via the Hugging Face transformers library.
MIT‑licensed code, with separate licensing for any downloaded datasets or pretrained weights.

Core architecture explained

Each option becomes a query vector, which is a short list of numbers representing its text. The query assigns attention weights to the context tokens. Those weights make one context vector for that option. A shared dot product turns each option and context pair into one score. A softmax, which converts scores into probabilities that sum to one, runs across the options.

Option queries – Every option is embedded (byte‑level by default or via a frozen encoder) into a fixed‑size vector that serves as a query.
Attention over context – The query attends to the token embeddings of the context, producing an option‑specific context vector.
Scoring head – A shared linear dot‑product layer computes a scalar score for each (option, context) pair.
Softmax normalization – Scores are passed through a softmax across the option dimension, yielding a probability distribution.

The design mirrors the “option‑attention head” described in TypeSafe’s Jev system but is fully disclosed and extensible.

Data format and preparation

Input files are JSONL with one object per line:

{"context":"The customer needs a refund.","options":["refund","sales","technical support"],"label":0}

label is the zero‑based index of the correct option.
The number of options may vary per row, with a minimum of two.
For custom datasets, keep all options that will appear at inference time in each row and split related records together to avoid leakage.

Getting started (synthetic demo)

# Create a virtual environment and install dev dependencies
uv venv && source .venv/bin/activate
uv pip install -e '.[dev]'

# Generate synthetic data
jevlike-data synthetic --output data/synthetic

# Train on the synthetic training split
jevlike-train data/synthetic/train.jsonl \
  --validation data/synthetic/validation.jsonl \
  --output runs/synthetic.pt

# Evaluate on the synthetic test split
jevlike-eval runs/synthetic.pt data/synthetic/test.jsonl

# Predict on a new menu
jevlike-predict runs/synthetic.pt \
  --context "Choose the exact badge amber badger. Badge: amber badger." \
  --option "azure crane" \
  --option "amber badger" \
  --option "gold heron"

The evaluation reports top‑1 and top‑3 accuracy, expected calibration error, and a shuffled‑context control. A useful model should outperform the control.

Using a frozen pretrained encoder

uv pip install -e '.[transformers]'
jevlike-train data/synthetic/train.jsonl \
  --validation data/synthetic/validation.jsonl \
  --output runs/qwen-head.pt \
  --encoder hf \
  --hf-model Qwen/Qwen2.5-0.5B \
  --rank 256 \
  --batch-size 8

The checkpoint stores only the trained scorer head and the encoder identifier; the encoder weights are loaded from Hugging Face at runtime.
--rank controls the width of the scorer head (higher rank = more parameters, higher memory).

Real‑world examples

Doom controller scoring

The repository ships a joint checkpoint that scores the seven Doom controller buttons from raw image patches.
A ten‑second demo film combines Doom combat with a chess controller moving pieces, illustrating that the same option‑attention head can handle visual and textual inputs.
The Doom checkpoint achieved an average of 0.60 kills and ‑97.50 reward over ten recorded episodes.

Chess move selection

A chess‑only checkpoint scored five keys representing chess moves, achieving 4 wins, 46 draws, 0 losses against a random mover in 50 sampled games.
Against Stockfish level 0, it scored 0 wins, 2 draws, 48 losses, indicating limited strategic competence but confirming the model can process visual board states.

Performance numbers from the author’s experiments

Synthetic menus: ~98 % top‑1 accuracy using the one‑pass scorer.
Wikispeedia next‑click task (target‑disjoint split):
- Frozen Qwen2.5‑0.5B encoder + scorer → 26 % accuracy.
- Random encoder control → ~8 % accuracy.
- Small model trained from scratch on 40 k clicks → 29 % accuracy.
Speed: With eight options, the one‑pass scorer is ~100× faster than a small decoder forced to generate 400 tokens.

These figures are local experimental results and not direct comparisons to TypeSafe’s proprietary Jev model.

Community insights from Hacker News comments

Diffusion models as Jev‑like scorers – A user linked a VLLM PR that repurposes diffusion models for one‑pass option scoring, achieving ~0.2 s per decision on a DGX Spark and high accuracy on language detection tasks.
Open‑source Qwen‑2.5‑1B‑RLCD – Another comment highlighted a recent release of a faster on‑device inference model for JSON workloads, suggesting a trend toward lightweight, type‑safe models.
Use‑case diversity – Several commenters emphasized that the value lies in obtaining calibrated probability weights rather than generated text, enabling applications such as skill‑collision detection, pre‑flight cost estimation for diffusion pipelines, and routing decisions in multimodal workflows.
Clarifying the concept – One comment noted that the original TypeSafe announcement was vague; the three‑sentence description in the README ("takes a piece of text and a list of N text options… one pass…") captures the core idea more clearly.

Limitations to keep in mind

The project is a research starter, not a full replica of TypeSafe’s Jev.
Accuracy heavily depends on data quality, split strategy, and the chosen encoder.
The default byte encoder is cheap but lacks deep linguistic understanding.
Using a frozen pretrained encoder can require large downloads and additional GPU memory.
The model requires the complete option list at inference time, which may be impractical for extremely large candidate sets.
Reported speed gains compare against a small decoder, not against large commercial models.

How to extend or adapt jevlike

Replace the byte encoder with a larger multilingual model (e.g., LLaMA‑2, Mistral) via the --encoder hf flag.
Increase --rank to improve the capacity of the scorer head for more nuanced option discrimination.
Experiment with multimodal inputs by feeding visual embeddings from a vision encoder into the same option‑attention mechanism (as demonstrated in the Doom/chess demos).
Integrate the predictor into pipelines that need probabilistic routing or confidence‑aware classification rather than deterministic text generation.
Benchmark against baseline classifiers (logistic regression, fine‑tuned BERT) to quantify the trade‑off between speed and accuracy for your specific task.

License and attribution

The code is released under the MIT License.
Datasets and pretrained models retain their original licensing terms; consult the respective sources (e.g., SNAP for Wikispeedia, Hugging Face model cards).