microsoft/waza

What it solves

Waza is a command-line interface (CLI) designed to evaluate the skills of AI agents. It helps developers scaffold evaluation suites, run benchmarks, and compare the performance of different models to ensure AI agent capabilities are consistent and reliable.

How it works

Waza uses a structured approach to define "skills" (via SKILL.md files) and corresponding evaluation suites (via eval.yaml files). It allows users to record actual prompt runs to generate test tasks, execute these tasks against models, and grade the outputs using a registry of graders. It also supports A/B testing (baseline vs. skill), result caching for speed, and adversarial testing to check for prompt injection and scope bypass.

Who it’s for

Developers building AI agents who need a standardized way to benchmark their agents' skills, verify coverage against requirements, and ensure safety through adversarial testing.

Highlights

• Automated Scaffolding: Quickly create skill definitions and evaluation suites from templates or existing skill documentation.
• Prompt-to-Task Recording: Generate reusable evaluation tasks by recording a live prompt run through Copilot.
• Multi-Model Comparison: Run the same evaluation suite across different models and compare the results side-by-side.
• Adversarial Testing: Built-in packs to test resistance to prompt injection and prevent out-of-scope actions.
• CI/CD Integration: Provides exit codes and JUnit XML reporting for seamless integration into GitHub Actions and other pipelines.
• Determinism Checks: Capture and replay snapshots of task runs to verify consistency and bisect divergences.