getsentry/vitest-evals

What is vitest‑evals?

vitest‑evals is a monorepo that adds a first‑class evaluation layer on top of the JavaScript test runner Vitest. It lets developers write explicit‑run evaluation suites for AI‑powered applications (LLMs, agents, tool‑using bots, etc.) and get structured JSON reports that can be inspected locally or posted back to GitHub Actions.

Why it matters

AI‑aware test helpers – describeEval, run, and expect(...).toSatisfyJudge let you keep ordinary Vitest assertions while also checking LLM outputs with judges (e.g., factuality, tool‑call correctness).
Plug‑in harnesses – adapters for the Sentry AI SDK, OpenAI Agents, Pi‑AI, or any custom harness make the same evaluation API work across different model providers.
CI integration – a bundled GitHub Action reads the JSON report, adds a step summary, annotates failures, and can publish a Check Run that gates PR merges based on a pass‑rate or score threshold.
Local UI – vitest‑evals serve spins up a React SPA that visualises runs, tool calls, token usage, costs, and trace spans, making debugging of LLM behaviour much easier.
Replay‑able tool calls – recordings of tool invocations are stored on disk and can be replayed automatically, ensuring deterministic test runs.

Core pieces of the repo

Package / App	Role
`packages/vitest-evals`	Core API: `describeEval`, judges, harness/session types, and the custom Vitest reporter.
`packages/core`	Shared primitives, JSON schema definitions, and helpers for aggregating full reports.
`packages/report-ui`	React single‑page app and CLI (`serve`) for browsing JSON evaluation artifacts.
`packages/harness‑ai-sdk`	Adapter that lets the core API talk to the Sentry AI SDK harness.
`packages/harness‑openai‑agents`	Adapter for @openai/agents‑style agents.
`packages/harness‑pi‑ai`	Adapter for pi‑ai with built‑in tool‑replay support.
`packages/github-reporter`	GitHub Actions action that consumes the Vitest JSON report and writes summaries/annotations/check‑runs.
`apps/demo‑pi`, `apps/demo‑ai-sdk`, `apps/demo‑openai‑agents`	Small end‑to‑end demo applications showing how to evaluate a refund‑agent built with each harness.

How you would use it

Add the packages to a monorepo (or install the needed ones from npm).

Write an evaluation suite using the familiar Vitest syntax:

import { describeEval, FactualityJudge, toolCalls } from 'vitest-evals';
import { piAiHarness, piAiJudgeHarness } from '@vitest-evals/harness-pi-ai';

const judge = FactualityJudge({ judgeHarness: piAiJudgeHarness({ model: 'claude-sonnet', temperature: 0 }) });

describeEval('refund agent', { harness: piAiHarness({ agent: createRefundAgent }) }, (it) => {
  it.for([{ name: 'approves invoice', input: 'Refund invoice 123', expectedTools: ['lookupInvoice','createRefund'] }])
    ('$name', async ({ input, expectedTools }, { run }) => {
      const result = await run(input);
      expect(toolCalls(result).map(c => c.name)).toEqual(expectedTools);
      await expect(result).toSatisfyJudge(judge);
    });
});

Run the suite with the provided CLI:

pnpm evals          # runs all packages/apps that expose an "evals" script
pnpm evals --info   # richer per‑tool metadata

Inspect results locally:
```
pnpm exec vitest-evals serve vitest-results.json
```
This opens a UI showing each test, the LLM output, tool calls, token usage, and cost.

Add CI reporting (GitHub Actions example from the README):

- run: pnpm exec vitest run ... --reporter=vitest-evals/reporter --outputFile.json=vitest-results.json
- uses: getsentry/vitest-evals@v0
  with:
    results: vitest-results.json
    publish-check: true
    min-pass-rate: 0.8

The action posts a summary, annotates failures, and can block merges if the pass‑rate falls below the threshold.

Key concepts explained

Harness – a thin wrapper that knows how to start your AI‑powered app (or agent) and invoke it with a test input. Different harnesses exist for different SDKs/providers.
Judge – a piece of logic (often LLM‑backed) that scores the model’s response. Built‑in judges include FactualityJudge, StructuredOutputJudge, and ToolCallJudge. Custom judges can be written by implementing an assess(ctx) function.
Explicit‑run – instead of letting Vitest call the test function directly, the suite calls run(input) on the harness, giving full control over the request/response lifecycle and allowing the framework to capture spans, token counts, and tool usage.
Replay mode – recordings of tool calls are saved under .vitest-evals/recordings. When VITEST_EVALS_REPLAY_MODE=auto, existing recordings are replayed; record forces a live call. This makes evaluations deterministic while still allowing easy refresh of fixtures.
JSON report – the custom reporter emits a structured file containing per‑test metadata, scores, token usage, cost in USD, and raw spans. This file fuels the UI, the GitHub Action, and any downstream analytics.

Who should consider using it?

Teams building LLM‑driven services, agents, or tool‑using bots that need automated quality checks.
Projects already using Vitest for unit/integration tests and want a low‑friction way to add AI‑specific assertions.
Organizations that want CI‑gateable evaluation (e.g., require a minimum factuality score before merging).
Anyone who wants a visual debugging UI for LLM outputs, token usage, and tool interactions.

Where to learn more

Docs site – https://vitest-evals.sentry.dev/docs (setup guide, architecture, CI details).
Package READMEs – especially packages/vitest-evals/README.md for the core API.
Demo apps – apps/demo‑pi, apps/demo‑ai-sdk, apps/demo‑openai‑agents illustrate real‑world usage.

TL;DR

vitest‑evals extends Vitest with a structured evaluation framework for AI agents, offering harness adapters, reusable judges, JSON reporting, a local UI, and first‑class GitHub Actions integration. It lets you keep normal test assertions while adding AI‑specific quality checks that are reproducible, observable, and CI‑gateable.