microsoft/waza

What it solves

Waza is a command-line tool designed to evaluate the skills of AI agents. It provides a structured way to scaffold evaluation suites, run benchmarks across different models, and compare the results to determine which model or agent configuration performs best for specific tasks.

How it works

Waza uses a system of skill definitions (SKILL.md) and evaluation specifications (eval.yaml). Users can create skills and then generate corresponding evaluation tasks and fixtures. The tool can execute these tasks using various models (including integration with GitHub Copilot), grade the outputs using local or remote graders, and provide detailed reports on pass rates and performance deltas.

Who it’s for

AI developers and researchers who need a rigorous, reproducible way to benchmark AI agent capabilities, test for regressions, and verify that agents adhere to their intended scope and safety boundaries.

Highlights

• Comprehensive Benchmarking: Run evaluations across multiple models and compare results side-by-side.
• Automated Scaffolding: Quickly initialize projects, create new skills, and generate evaluation tasks from prompts.
• Advanced Testing: Includes adversarial testing packs to probe for prompt injection and scope bypass.
• Determinism Checks: Capture snapshots of runs to replay and bisect divergent outputs for debugging.
• CI/CD Integration: Built-in support for GitHub Actions and JUnit reporting to automate agent evaluation in pipelines.
• Result Caching: Speeds up repeated runs by caching results unless configurations or fixtures change.