NVIDIA/SkillEvaluator

What it solves

SkillEvaluator provides a structured way to evaluate "agent skills"—sets of instructions and supporting files that extend the capabilities of AI agents. It ensures these skills are safe, well-formed, non-redundant, and actually effective when deployed in a live environment, preventing low-quality or insecure artifacts from reaching production.

How it works

The framework uses a three-tier pipeline to assess skills:

• Tier 1: Validation focuses on safety and structure. It uses deterministic checks (schema, PII, license, Unicode) and LLM-based checks, as well as external security scanners like Semgrep and Gitleaks to ensure the skill is safe and well-formed.
• Tier 2: Deduplication identifies overlap. It uses embeddings and chat LLMs to check for semantic similarity between skills and optimize context to avoid repeated guidance.
• Tier 3: Live Evaluation tests actual performance. It generates synthetic evaluation datasets and runs the agent in a sandboxed environment (via the Harbor framework) to grade whether the skill actually helps the agent achieve its goals.

Who it’s for

Developers and organizations creating AI agent extensions (skills) who need a rigorous, automated pipeline to verify quality, security, and utility before deployment.

Highlights

• Multi-tier approach: Separates basic validation, semantic deduplication, and live performance testing.
• Flexible Provider Support: Compatible with NVIDIA Build, OpenAI, Anthropic, Amazon Bedrock, and local OpenAI-compatible endpoints.
• Automated Dataset Generation: Can automatically create evaluation cases to test agent performance.
• Sandboxed Execution: Integrates with Harbor to run live agent evaluations safely in Docker or cloud environments.