darkrishabh/agent-skills-eval

What it solves

It provides a way to empirically prove whether a "skill" (domain knowledge provided via a SKILL.md file) actually improves an AI agent's performance on a specific task. It eliminates "vibes-based" evaluation by running side-by-side comparisons between a model with the skill loaded and a baseline model without it.

How it works

For every evaluation defined in a skill's configuration, the tool runs the same prompt twice: once with the SKILL.md content in the context (with_skill) and once without it (without_skill). A separate judge model then grades both outputs against the same set of assertions and expected outputs, producing a pass/fail result for each side. The system supports OpenAI-compatible APIs, custom providers, and can output results as portable JSON/JSONL artifacts or a static HTML report.

Who it’s for

Developers building AI agents who follow the agentskills.io specification and need a rigorous, automated way to test and iterate on the domain knowledge they provide to their models.

Highlights

• A/B Testing for Skills: Directly compares performance lift by running baseline vs. skill-enabled prompts.
• Judge-Based Grading: Uses a chat model to provide cited assertions rather than simple string matching.
• Spec Compliant: Fully implements the agentskills.io specification for SKILL.md and evals.json.
• Tool-Call Assertions: Supports deterministic checks for agents that utilize tool calls.
• Flexible Integration: Offers both a CLI for CI/CD pipelines and a TypeScript SDK for custom dashboards.