alibaba/skill-up

An evaluation and evolution tool for Agent Skills.

What it solves

skill-up provides a structured way to evaluate and improve the capabilities (Skills) of AI agents. It replaces ad hoc testing with a declarative framework that can measure an agent's behavior across specific cases, check for regressions, and automate the setup of environments and workspaces.

How it works

The tool uses a CLI to run declarative evaluation cases defined in YAML files. It provisions the necessary runtime environment (local, Docker, or OpenSandbox), installs the target Skill or MCP servers, and executes the agent engine. It then grades the results using rule-based, script-based, or agent-based judges and generates structured reports (JSON, HTML, JUnit).

Additionally, it includes skill-upper, an AI-driven Skill that enables an "Eval-to-Evolution" loop. This allows a user to converse with an agent to identify failures, automatically repair or expand the evaluation suite, and iterate on the Skill's implementation until the desired behaviors are achieved.

Who it’s for

It is designed for developers building AI agents and "Skills" who need a repeatable, evidence-backed way to verify that their agents can perform specific tasks and handle repository-level changes without breaking existing functionality.

Highlights

  • Eval-to-Evolution Loop: Uses skill-upper to diagnose failures and evolve the eval suite through natural language conversation.
  • Declarative Configuration: Environment, engines, and cases are defined in YAML for repeatability.
  • Multi-Engine Support: Compatible with various agent engines including Claude Code, Codex, and Qoder CLI.
  • Environment Provisioning: Automates the setup of workspaces, repository fixtures, and runtime environments.
  • CI-Ready: Includes a GitHub Action to run evaluations across multiple engines in a continuous integration pipeline.

Related

  • Project
    darkrishabh/agent-skills-eval

    agent‑skills‑eval is a TypeScript/Node.js CLI + SDK that runs an Agent Skill both with and without its `SKILL.md` context, grades the two outputs with a separate LLM judge, and produces JSON artifacts plus a static HTML report. It supports OpenAI‑compatible APIs, custom providers, tool‑call assertions, and full agentskills.io spec validation, making it suitable for skill development, CI testing, and comparative analysis.

  • Dispatch
    Measuring the Impact of Agent Skills: An Introduction to agent-skills-eval

    Explore how to objectively test whether adding specialized 'skills' to AI agents improves performance, using the agent-skills-eval framework.

  • Project
    pskoett/pskoett-ai-skills

    A collection of structured skills and plugins for AI agents that implement inner and outer feedback loops to prevent scope creep, automate error recovery, and enable cross-session learning.

  • Dispatch
    Streamlining AI Agent Evaluation with Agent-evals: A Claude Skill for Startups

    As AI agents become more prevalent, ensuring their quality through systematic evaluation is crucial yet often overlooked, especially by startups without dedicated data science teams. Agent-evals, a new Claude Skill, offers a practical solution by providing an automated baseline for agent evaluation directly within the codebase, drawing on a decade of experience in production AI systems.

  • Project
    sentient-agi/EvoSkill

    A toolkit for automatically discovering and improving AI skills and prompts for coding agents, evolving general agents into specialists using benchmark-driven feedback.