nokia-applied-research/AnyJev

Turn any LLM into a Jev-style decision model: typed decisions, real probabilities, no training. (continue updating, welcome any issue and PR request)

What it solves

When you ask a large language model (LLM) to pick an option from a list, the raw answer often flips if you reorder the options, and the model tends to prefer certain labels regardless of the actual content. That makes raw logit readouts unreliable for decisions in agents or automation. AnyJev fixes this by letting you ask a typed question (a choice, yes/no, or a score) and returning a probability distribution over the options, with no parsing of free text. The training-free mode (L0) removes both the order bias and the label bias without needing any human-labeled data.

How it works

You provide a state (the context) and a typed question. The library reads directly from the model's next-token distribution over the option labels, so nothing is parsed.

  • Training-free (Decider): works on any open causal LLM. Level L0 reads the choice once per cyclic rotation of the options (e.g., 18 rotations for 18 options), averages out position effects, and divides out a label prior estimated without labels. You can also set a rotation budget that stops early when the answer is confident enough, calibrated against the model's own full-cycle answer — no labels needed.
  • Self-distilled (Tacit): these are fine-tuned checkpoints (1.7B to 9B) trained by self-distillation — they learn to give, in one forward pass, the same answer they would reach after reasoning. No human labels or other models are used. With adaptive=True, a capped share of low-confidence decisions is escalated to the model's own reasoning (chain of thought), and the final answer is read as a label distribution so it always has probabilities and never fails to parse.

The library provides backends for Transformers, in-process vLLM, a remote vllm serve, and also SGLang and llama.cpp (contributed). A small HTTP gateway allows many clients to share one escalation cap.

Who it’s for

  • Developers building AI agents or automated pipelines that need reliable yes/no, multiple-choice, or scoring decisions from LLMs.
  • Teams using open LLMs without wanting to train or label data — the training-free Decider works out of the box on any causal LM.
  • Anyone who wants small, fast decision models (Tacit 1.7B–9B) that run in one forward pass and can optionally use reasoning only for the hardest few percent of decisions.

Highlights

  • Training-free L0 on Qwen3-8B (BANKING77-20): order-flip rate drops from 0.230 to 0.073, accuracy rises from 0.747 to 0.803, and the share of decisions auto-decodable at ≤5% error jumps from 7.7% to 46.3% — all without any labels.
  • Tacit self-distilled models (1.7B to 9B) are on Hugging Face; adaptive mode escalates only a capped share (default 20% of every 1000) of low-confidence decisions to reasoning, adding typically 300–600 tokens per escalated decision.
  • AnyJev Tacitus 31B (fine-tuned from Gemma-4-31B) ties 3rd on the Decision Index 0.4.1 leaderboard with a Full score of 66.47 (2026-10-21).
  • The same training-free Decider on an unchanged Gemma-4-31B is 5th of 149 entries on Decision Index 0.3 with a Full score of 60.22 (2026-10-12).

Related

  • Project
  • Project
  • Dispatch
  • Dispatch
  • Project