llm-as-a-verifier/llm-as-a-verifier

LLM-as-a-Verifier is a general-purpose framework that provides fine-grained feedback for any agent without requiring additional training. It achieves SOTA performance across coding, robotics, and medical agentic benchmarks.

What it solves

LLM-as-a-Verifier provides a unified framework for providing fine-grained feedback and scoring for AI agents across any modality. It solves the problem of coarse-grained "LLM-as-a-Judge" scoring by using logprob distributions to create a more precise reward signal, which can then be used for test-time scaling, progress tracking, and reinforcement learning.

How it works

The framework operates on three primary mechanisms:

  1. Fine-grained Reward Estimation: Instead of a single discrete score, it calculates rewards by taking the expectation over the full logprob distribution of score tokens, allowing for a more nuanced evaluation of a trajectory.
  2. Probabilistic Pivot Tournament (PPT): To efficiently rank $N$ candidate trajectories without the $O(N^2)$ cost of a round-robin tournament, PPT uses a small set of "pivots" to reduce the complexity to $O(Nk)$.
  3. Prefix-Cache Optimization: To reduce API costs and latency, the framework structures prompts so that the task and trajectories are shared prefixes, significantly increasing the cache hit rate for repeated evaluations.

Who it’s for

  • AI Agent Developers: Those looking to improve agent performance via best-of-N selection or test-time scaling.
  • ML Researchers: Those studying reward models and verification frameworks for agentic benchmarks (e.g., SWE-Bench, Terminal-Bench).
  • Robotics Engineers: Developers needing to track the progress of embodied intelligence rollouts using multimodal (image/video) inputs.

Highlights

  • Multimodal Support: Accepts images and video frames as part of the trajectory for verification.
  • Progress Tracking: Includes a ProgressTracker for online monitoring of agent attempts to stop hopeless rollouts early.
  • SOTA Performance: Achieves state-of-the-art results on benchmarks like Terminal-Bench and SWE-Bench Verified.
  • Claude Code Integration: Offers a plugin called TurboAgent that acts as an API proxy to implement best-of-N selection for Claude Code.

Written about in

Related

  • Project
    msitarzewski/agency-agents

    The repository provides a library of markdown‑based AI‑agent prompts (e.g., Front‑end Developer, AI Engineer, Prompt Engineer) and scripts to install them into various LLM‑powered coding assistants such as Claude Code, Cursor, Gemini CLI, etc. A companion desktop app automates installation. It’s a prompt‑template collection, not a model‑training framework.

  • Project
    MadsLorentzen/ai-job-search

    AI Job Search is an open‑source, Claude‑Code‑driven framework that automates the whole job‑application pipeline: profile creation, job‑board scraping, fit scoring, LaTeX CV/cover‑letter drafting, reviewer‑agent critique, ATS‑friendly PDF verification, interview‑prep packs, and outcome tracking. It runs locally, requires Claude Code, Python 3.10+, Bun, and a LaTeX distribution, and is extensible to other markets via custom portal‑search skills.

  • Project
    NeoLabHQ/context-engineering-kit

    A plug‑in marketplace for LLM‑based coding assistants (Claude Code, Gemini, Antigravity, etc.) that supplies token‑efficient, context‑engineering patterns—reflection loops, sub‑agent judges, specification‑first development, code‑review agents, and Git helpers—to make AI‑generated code more reliable and easier to integrate into CI pipelines.

  • Project
    harshaneel/humanize

    humanize is an MIT‑licensed repo that ships two static, rule‑based skills—`humanize` (rewrites AI text to sound human) and `ai-check` (produces a forensic AI‑detection report). The skills work with any LLM agent (Claude Code, Codex CLI, ChatGPT desktop, Gemini, etc.) via a simple install script or marketplace command. Benchmarked on 25 diverse inputs, both internal and external perplexity‑based detectors rate the rewritten output as human‑like. The approach is offline, zero‑dependency, but cannot evade learned classifiers like GPTZero.

  • Project
    WecoAI/aideml

    AIDE ML is an open‑source Python package that implements an LLM‑guided tree‑search agent for automatically writing, debugging, and benchmarking machine‑learning code. Users specify a dataset, a natural‑language goal, and an evaluation metric; the agent iteratively generates and refines Python scripts, pruning the search tree based on metric feedback. The repo includes a CLI, a Streamlit UI, HTML visualisation, Docker and local‑LLM support, and is designed for both research on agent architectures and practical ML pipeline generation.