benchflow-ai/skillsbench

SkillsBench evaluates how well skills work and how effective agents are at using them.

What it solves

SkillsBench addresses the need to evaluate how effectively AI agents can use "skills"—which are modular collections of instructions, scripts, and resources—to complete specialized workflows. It specifically targets the ability of agents to compose multiple skills to solve complex tasks where current state-of-the-art models often struggle.

How it works

The project uses a gym-style benchmarking approach to measure agent behavior and skill effectiveness. Tasks are defined as task.md packages that include an environment (via Docker), an oracle for baseline solutions, and a verifier to test the agent's output. It integrates with the BenchFlow SDK and supports execution in cloud sandboxes (via Modal) or local Docker environments.

Who it’s for

It is designed for AI researchers and developers building and testing AI agents, particularly those focusing on modular skill acquisition and complex task composition.

Highlights

  • Skill Composition Focus: Specifically designs tasks that require combining two or more skills to solve.
  • Gym-style Benchmarking: Provides a structured way to evaluate agent performance and behavior.
  • Flexible Execution: Supports both cloud-based execution via Modal and local execution via Docker.
  • Extensible Task System: Allows users to create new tasks with defined environments, oracles, and verifiers.

Related

  • Project
    mgechev/skillgrade

    A framework for evaluating AI agent skills by running automated tasks in sandboxed environments and grading them with deterministic scripts or LLM rubrics.

  • Project
    davidondrej/skills

    A collection of reusable skill modules and instructions that AI agents can load to execute specific workflows like research, coding orchestration, and system operations.

  • Project
    InternLM/WildClawBench

    An end-to-end agent benchmark that tests AI agents' ability to perform complex, real-world tasks in live environments using tools like browsers and bash terminals.

  • Project
    xlang-ai/OSWorld

    A benchmark and sandboxed environment for evaluating multimodal AI agents on their ability to complete open-ended tasks in real computer operating systems.

  • Project
    harbor-framework/terminal-bench

    Terminal-Bench is a continuous benchmark for measuring the capabilities of frontier AI agents using a diverse and evolving set of high-quality, difficult tasks.