mgechev/skillgrade

What it solves

Skillgrade provides a standardized way to evaluate the capabilities of AI agents. It solves the problem of verifying whether an agent can correctly discover and utilize specific "skills" (tools or workflows) by running them through a series of automated tests and grading their performance.

How it works

Users define a skill evaluation suite in an eval.yaml file, which specifies tasks, instructions, and graders. Skillgrade then executes these tasks using a chosen AI agent (such as Gemini, Claude, or custom commands) within a sandbox environment (either Docker or local).

Performance is measured using two types of graders:

• Deterministic Graders: Run a script or command and parse a JSON response to check for specific outcomes (e.g., checking if a file was created correctly).
• LLM Rubric Graders: Use another LLM to qualitatively evaluate the agent's session transcript against a provided rubric.

Results are aggregated into a pass rate, which can be used for local reporting or integrated into CI/CD pipelines to block merges if the pass rate falls below a certain threshold.

Who it’s for

• Developers building AI agents who need to benchmark their agent's performance.
• Teams creating "skills" or toolsets for AI agents and wanting to ensure they work as expected.
• Engineers integrating AI agents into production workflows who require regression testing.

Highlights

• Flexible Agent Support: Works with built-in adapters for Gemini, Claude, and Codex, as well as ACP (Agent Client Protocol) agents and completely custom command-based agents.
• Hybrid Grading: Combines hard deterministic checks with qualitative LLM-based rubrics for a comprehensive score.
• Sandboxed Execution: Supports Docker for safe local development and local execution for faster CI runs.
• AI-Powered Initialization: Can automatically scaffold eval.yaml files using LLMs based on a SKILL.md file.
• CI-Ready: Includes a --ci mode with configurable pass-rate thresholds for automated quality gates.