darkrishabh/agent-skills-eval

agent‑skills‑eval – a test runner for Agent Skills

What it is

• A TypeScript/Node.js library (CLI + SDK) that lets you evaluate whether an Agent Skill actually improves a language‑model‑based agent.
• It follows the open agentskills.io specification: a skill is a folder containing a SKILL.md and an optional evals/evals.json describing test cases.

How it works

1. Run the skill – the runner injects the SKILL.md into the prompt sent to a target model (the model you want the skill to help).
2. Run the baseline – the same prompt is sent without the skill in context.
3. Judge – a second model (the judge) grades both outputs against the assertions defined in the eval file.
4. Report – results are written to a workspace folder (iteration‑N/…) and a static HTML report is generated.

Key features

Feature	Why it matters
With‑skill / without‑skill comparison	Shows the real lift (or lack thereof) a skill provides.
Judge‑graded pass/fail	Uses a separate LLM to apply the eval’s assertions/expected_output, giving reproducible scores.
CLI for quick runs & CI	One‑liner npx agent-skills-eval … works out of the box; flags map directly to a YAML config for reproducibility.
TypeScript SDK	Lets you embed the evaluator in pipelines, dashboards, or custom tooling.
OpenAI‑compatible provider	Works with OpenAI, Anthropic (via compatibility layer), Together, Groq, local Llama servers, or any service that speaks the OpenAI chat API.
Custom providers	Implement the tiny Provider interface to hook into local servers (Ollama, vLLM, etc.) or proprietary APIs.
Tool‑call assertions	Not just raw text – you can assert that the model made the correct function calls.
Portable artifacts	All outputs, timings, token usage, and grading data are stored as JSON/JSONL, making diffing and downstream analysis easy.
Static HTML report	A self‑contained site (index.html) that visualises pass rates, per‑assertion reasoning, prompt/response pairs, and resource usage.
Full agentskills.io spec compliance	Validates SKILL.md front‑matter, eval schema, and artifact layout automatically.

Typical workflow

# 1️⃣ Install (or just use npx)
npm i agent-skills-eval

# 2️⃣ Run against a folder of skills
npx agent-skills-eval ./skills \
  --target gpt-4o-mini \
  --judge  gpt-4o-mini \
  --baseline \
  --strict

• The command creates agent-skills-workspace/iteration-1/… with JSON results and report/index.html.
• Open the HTML file in a browser to see whether each skill helped the target model.

Configuration

• A YAML file (agent-skills-eval.yaml) can store all options (root folder, workspace path, model names, API base URL, concurrency, etc.).
• CLI flags always override the YAML values, which is handy for CI pipelines.

SDK example

import { OpenAICompatibleProvider, evaluateSkills, consoleReporter } from "agent-skills-eval";

const provider = new OpenAICompatibleProvider({
  baseUrl: "https://api.openai.com/v1",
  apiKey: process.env.OPENAI_API_KEY!,
  model: "gpt-4o-mini",
  providerName: "openai",
});

await evaluateSkills({
  root: "./skills",
  workspace: "./agent-skills-workspace",
  baseline: true,
  target: { model: provider.model, provider },
  judge:  { model: provider.model, provider },
  onEvent: consoleReporter(),
});

• The SDK streams events (progress, grading, errors) that can be logged as JSONL for further analysis.

When to use it

• Skill developers who want empirical evidence that their SKILL.md actually improves model performance.
• CI/CD pipelines that need automated regression testing for a library of skills.
• Product teams evaluating multiple skills side‑by‑side to decide which to ship.
• Researchers comparing different prompting or tool‑calling strategies under a controlled baseline.

Installation & quick start

npm install agent-skills-eval   # or just use npx
npx agent-skills-eval --help    # shows all flags

A minimal skill folder looks like:

my-skill/
├─ SKILL.md          # description + optional metadata
└─ evals/
   └─ evals.json    # one or more test cases with prompt, expected_output, assertions

The runner will validate the SKILL.md (if --strict is set) and then execute each eval.

Extensibility

• Implement your own Provider (only name, model, and an async complete(prompt) that returns {output, latencyMs, inputTokens, outputTokens, costUsd}) to plug in any backend.
• The artifact contract (docs/artifact-contract.md) lets downstream tools consume the JSON results without pulling in the whole library.

Documentation & community

• Full docs: https://darkrishabh.github.io/agent-skills-eval/
• Source code, issue tracker, and contribution guidelines are on GitHub.
• The project is MIT‑licensed and encourages extensions via PRs.

---

agent‑skills‑eval fills the missing piece in the Agent Skills ecosystem: a reproducible, side‑by‑side evaluation harness that turns a simple SKILL.md into measurable performance gains.