paradigmxyz/evmbench

What it solves

evmbench provides a structured environment to benchmark and deploy AI agents designed to find and exploit vulnerabilities in smart contracts. It bridges the gap between raw LLM capabilities and the practical need for a secure, reproducible harness to evaluate how well these agents can perform security audits on untrusted code.

How it works

The system operates as a distributed architecture where a user uploads smart contract source code via a Next.js frontend. A FastAPI backend manages the job queue using RabbitMQ and PostgreSQL. An "Instancer" service then spins up isolated worker containers (via Docker or Kubernetes) to execute the audit.

Inside these workers, an LLM-driven agent (using Codex) runs in "detect-only" mode. The agent analyzes the code based on a specific prompt and model map, generates a vulnerability report in JSON format, and uploads the results to a Results Service, which the user can then view through the UI.

Who it’s for

Security researchers, smart contract auditors, and AI developers who want to evaluate the effectiveness of LLM-based agents in identifying smart contract bugs.

Highlights

• Isolated Execution: Uses Docker/K8s workers to treat the runtime environment as untrusted, protecting the host system from potentially malicious uploaded code.
• Agent Harness: Integrates a specific prompt and model mapping system to standardize how agents are tested.
• Flexible Credential Management: Supports both direct API key usage (BYOK) and a proxy-token mode to keep plaintext keys outside the worker environment.
• Full-Stack Tooling: Includes a complete pipeline from file upload and job queuing to result rendering and annotation.