NVIDIA/SkillEvaluator

What it solves

SkillEvaluator provides a structured way to evaluate "agent skills"—instructions and supporting files that extend the capabilities of AI agents. It ensures these skills are safe, well-formed, non-redundant, and actually improve the agent's performance in real-world tasks.

How it works

The framework uses a three-tier pipeline to vet skills:

• Tier 1 (Validation): Performs static checks for safety and structure. This includes dependency checks, rubric validation, and LLM-based security analysis (using tools like SkillSpector and Gitleaks).
• Tier 2 (Deduplication): Uses embeddings to detect if the guidance within a skill is repeated or overlaps with existing skills in a catalog, preventing redundancy.
• Tier 3 (Live Evaluation): Tests the skill in a sandboxed environment (typically Docker). It runs a provider-native agent both with and without the skill to measure "Skill Lift," determining if the skill actually helps the agent complete tasks.

Who it’s for

Developers and researchers creating AI agent skills who need a rigorous, automated pipeline to validate their artifacts before publishing them to a catalog or deploying them to production.

Highlights

• Multi-tier vetting: Moves from fast static analysis to expensive live execution.
• Sandboxed execution: Integrates with the Harbor framework to run agents safely in Docker.
• Skill Lift measurement: Quantifies the actual performance gain provided by a specific skill.
• Synthetic dataset generation: Can automatically create starter evaluation cases for testing.