vercel/next-evals-oss

Next.js Evals – What It Is

Next.js Evals is an open‑source harness for automatically testing how well AI coding agents can write or modify a small Next.js application. It ships a collection of evals (tiny Next.js projects with a task description) and a set of experiments that define which AI model (Claude, GPT, Gemini, etc.) should try to solve each task. The harness runs the agent in a Vercel‑hosted sandbox, lets it edit the project, then executes hidden assertions to see if the result is correct.

---

Why it matters

• Provides a reproducible benchmark for LLM‑based code‑generation agents on real‑world Next.js code.
• Supports multiple providers and configurable “reasoning effort” levels, letting researchers compare cost vs. quality.
• Integrates with Vercel’s sandbox and AI‑Gateway, so the same environment used by Vercel’s own CI can be used locally.

---

Core pieces (as described in the README)

Component	Role
Eval fixtures (evals/…)	Tiny Next.js apps plus a PROMPT.md (task) and EVAL.ts (hidden tests). The repo only stores a sync of these from vercel/next.js.
Experiments (experiments/*.ts)	Configuration that ties a model (e.g., claude‑opus‑5) to a set of evals, defines pass@4/1 behavior, and records pricing info.
Scripts (pnpm …)	Helper commands for installing, syncing fixtures, checking credentials, running a single eval, running all changed evals, exporting results, etc.
Sandbox credentials	Vercel sandbox access via OIDC token or a trio of VERCEL_TOKEN, VERCEL_TEAM_ID, VERCEL_PROJECT_ID.
Agent keys	Provider‑specific API keys (AI_GATEWAY_API_KEY, CLAUDE_CODE_OAUTH_TOKEN, OPENAI_API_KEY, …) used by the selected model.
Export (pnpm export-results)	Produces agent-results.json in the format consumed by the public leaderboard at nextjs.org/evals.

---

How to use it (quick‑start flow)

1. Install – pnpm install --frozen-lockfile (pnpm‑only repo).
2. Sync evals – pnpm sync-evals pulls the actual Next.js fixtures from the upstream vercel/next.js repo.
3. Provide credentials – copy .env.example → .env.local and fill in either an OIDC token or the Vercel token trio, plus any model API keys you need.
4. Check everything – pnpm preflight tells you if anything is missing.
5. Run a test – pnpm eval:smoke claude-opus-5 runs one eval with the Claude model to verify the sandbox and credentials work.
6. Run real experiments – pnpm eval:run <experiment…> (or pnpm eval for an interactive picker) runs all changed evals for the chosen experiment, caching results to avoid re‑running unchanged work.
7. Publish – after a successful run, pnpm export-results creates agent-results.json; commit that file and merge a PR to main to update the public leaderboard.

---

Who might find it useful

• LLM researchers wanting a concrete, Next‑specific coding benchmark.
• Product teams at Vercel or elsewhere evaluating new model providers for code‑completion.
• Open‑source contributors who want to add new Next.js tasks or new model configurations.

---

Limitations / scope noted in the README

• The actual eval fixtures are not stored in this repo; they are synced from vercel/next.js each time you run pnpm sync-evals.
• Running the harness requires Vercel sandbox credentials; without them the evals cannot execute.
• Cost estimates rely on token usage reported by the model APIs and a static price table; prices are a snapshot and may become outdated.
• The public leaderboard shows pass@4 scores; release automation records pass@1 separately but does not publish it.

---

Bottom line

vercel/next-evals-oss is a genuine, production‑grade tool for benchmarking AI coding agents on Next.js tasks. It ties together Vercel’s sandbox, multiple LLM providers, and a reproducible evaluation suite, and it feeds the results into the public Next.js eval leaderboard.