vercel/next-evals-oss

What it is

Next.js Evals is an open‑source framework that lets you automatically test code‑generation agents (e.g., Claude, GPT, Gemini, Cursor) on small, self‑contained Next.js projects. Each eval supplies a prompt and a hidden test suite; the agent runs inside a Vercel sandbox, modifies the project, and the framework checks whether the hidden assertions pass.

How it works

Piece	Role
`@vercel/agent-eval` (npm package)	Core library that drives the sandbox, sends prompts to the chosen model, and records the transcript.
Eval fixtures (pulled from `vercel/next.js`)	Real‑world Next.js apps plus a `PROMPT.md` and an `EVAL.ts` file containing the hidden tests.
Experiments (`experiments/*.ts`)	Configuration that ties a model (e.g., `claude-code`, `gpt‑5`) to a set of evals and specifies the reasoning‑effort level.
Scripts (`pnpm eval:*`, `pnpm preflight`, …)	CLI helpers for installing, syncing fixtures, checking credentials, running a single eval, running all changed evals, exporting results, etc.
Results (`results/`, `agent-results.json`)	JSON output that records pass/fail, token usage, and estimated cost per model. The file is published to `nextjs.org/evals`.

When you’d use it

Benchmarking LLM coding assistants – compare how well Claude, GPT, Gemini, etc., refactor or implement Next.js features.
Tracking model improvements – the framework fingerprints each eval; only changed evals are rerun, making it cheap to see the impact of a new model version.
Publishing a public leaderboard – the exported agent-results.json powers the Next.js evaluation board on the official site.
Cost awareness – token counts are extracted from the model’s transcript and multiplied by list prices, giving an average USD cost per eval.

Getting started (quick‑run)

# 1. Install (pnpm is required)
pnpm install --frozen-lockfile

# 2. Pull the Next.js eval fixtures from the upstream repo
pnpm sync-evals   # or pnpm sync-evals <git‑ref>

# 3. Provide Vercel sandbox credentials (OIDC token or token‑triple)
cp .env.example .env.local
# edit .env.local and fill the requested variables
pnpm preflight    # verifies everything is in place

# 4. Run a single smoke test (e.g., Claude Opus 5) to confirm the sandbox works
pnpm eval:smoke claude-opus-5

If the smoke test succeeds you have a working environment and can start running real experiments.

Core concepts you’ll see in the repo

Eval directory layout – each eval lives in evals/<slug>/ and contains:
- PROMPT.md – the task shown to the agent.
- EVAL.ts – hidden assertions that the framework runs after the agent finishes.
- a minimal Next.js app (package.json, app/…).
Experiment configs – TypeScript files in experiments/ that declare:
- the model identifier used with the Vercel AI Gateway.
- the reasoning‑effort rung (e.g., high).
- optional pricing overrides.
Caching – results are memoized by a fingerprint of the eval content plus the experiment config; rerunning only touches changed items unless --force is used.
Cost calculation – scripts/cost.ts reads token counts from each run’s transcript-raw.jsonl and applies the static MODEL_PRICING table.
Publishing workflow – after a PR merges, CI automatically posts to https://nextjs.org/api/evals/revalidate so the live site shows the new JSON snapshot.

Who maintains it

The repository is maintained by Vercel. It works hand‑in‑hand with the vercel/next.js repository, where the actual eval fixtures live. The OSS repo contains the tooling, scripts, and CI needed to run the evaluations and publish the results.

License

MIT – see the LICENSE file in the repo.