aaron-for-value/VeriRun

What it solves

VeriRun provides a distributed runtime for evaluating executable code and AI agents. It addresses the challenges of running untrusted model-generated code at scale, ensuring that results are reproducible, recoverable, and safe. It specifically separates infrastructure failures (like timeouts or OOMs) from actual model failures (like incorrect code), preventing misleading benchmark scores.

How it works

VeriRun implements an evidence-first infrastructure that treats execution as a series of versioned, immutable manifests. It uses a control plane to manage intent and durable state, and an execution plane that handles retryable attempts. To ensure safety, it employs isolated execution tiers, including digest-pinned Docker containers and Kubernetes Jobs with gVisor for a secure boundary. It integrates with existing benchmarks like EvalPlus and LiveCodeBench rather than replacing them, acting as the orchestration and isolation layer underneath.

Who it’s for

It is designed for researchers and developers building LLMs or AI agents who need a trustworthy, reproducible way to run executable evaluations and compute online rewards for post-training loops.

Highlights

• Immutable Provenance: Every result is linked to the specific benchmark, prompt, model revision, and runtime policy used.
• Structured Failure Semantics: Distinctly classifies errors into compile errors, test failures, timeouts, OOMs, and infrastructure failures.
• Durable Control Plane: Uses PostgreSQL and S3-compatible storage to ensure idempotent results and recovery from failures.
• Isolated Execution: Provides a security-focused runtime using Kubernetes and gVisor to execute untrusted code safely.
• Deterministic Replay: Allows frozen candidates to be re-verified without re-calling the model, ensuring consistency.