mgechev/skillgrade

Skillgrade – an evaluation harness for AI coding agents

What it does – Skillgrade lets you define coding‑related tasks (e.g., fix lint errors, implement an API, review code) and automatically run any AI‑powered coding agent against them. After each run it scores the result with deterministic scripts, LLM‑based rubrics, or human reviewers, producing a pass‑rate or weighted score.

How you use it

1. Install globally (npm i -g skillgrade).
2. Run skillgrade init in a folder to create an eval.yaml template (or let it generate AI‑suggested tasks if a SKILL.md file and API key are present).
3. Edit eval.yaml to describe tasks, workspace files, context files, and graders.
4. Execute the evaluation with a single command, e.g. skillgrade --smoke. The tool auto‑detects the agent from environment variables (GEMINI_API_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY) or you can force one with --agent.
5. View results in the terminal (skillgrade preview) or in a browser UI (skillgrade preview browser).

Key concepts

• Agent – any LLM‑backed coding assistant (Gemini, Claude, Codex, OpenCode, or any ACP‑compatible CLI). You can also point it at an arbitrary command.
• Workspace vs. Context – workspace files are the mutable code the agent works on; context files are read‑only reference material.
• Grader types –
  • Deterministic: runs a script that returns a JSON score.
  • LLM rubric: sends the agent’s transcript to a LLM with a rubric and parses the returned score.
  • Human: pauses for manual review. Graders can be weighted; the final score is the weighted average.
• Trials & presets – run a task multiple times (trials) to estimate reliability. Presets like --smoke (5 trials) or --regression (30 trials) provide quick or thorough checks.
• Multi‑turn tasks – tasks can have a sequence of instructions (turns) where the workspace persists across turns.
• Dataset expansion – a single task template can be expanded over a JSONL/CSV dataset, interpolating row fields into the instruction.
• Filtering – select subsets of tasks with --filter/--not-filter based on metadata fields.
• Reporting – export results as JSON, JUnit XML, CSV, or Markdown; useful for CI pipelines.

Typical workflow example

# Install
npm i -g skillgrade

# Create an eval suite
mkdir my-evals && cd my-evals
skillgrade init   # creates eval.yaml

# Edit eval.yaml – add a task that runs superlint on a broken file
# (see README for full syntax)

# Run a quick smoke test against the default Gemini agent
GEMINI_API_KEY=… skillgrade --smoke

# See a nice web UI
skillgrade preview browser

# In CI you might use:
skillgrade --regression --ci --provider=local --threshold=0.85

Why it matters – As AI coding assistants become common, developers need systematic ways to measure their reliability, correctness, and workflow compliance. Skillgrade provides a flexible, language‑agnostic harness that works with many LLM providers and can be integrated into automated test pipelines.

---

Scope – This project is a genuine software tool for evaluating AI coding agents, fitting squarely within the AI/ML tooling ecosystem.