responsibleai/ASSERT

What it solves

ASSERT addresses the gap between high-level product requirements (policies, system prompts, and launch criteria) and the actual evaluation of AI systems. Instead of relying on generic benchmarks or manual test cases that may drift from the original intent, ASSERT turns natural language specifications into executable, structured evaluations to ensure AI agents behave as intended.

How it works

ASSERT uses a pipeline that transforms natural language specifications into behavior categories, generates single-turn and multi-turn test cases, and runs them against a target AI system. An LLM judge then scores the conversations based on the provided policies.

Key technical components include:

• Trace-aware Judgment: Integrates with OpenInference and OpenTelemetry to capture spans (tool calls, routing, latency), allowing the judge to cite specific evidence from the agent's internal process rather than just the final output.
• Sandboxed Execution: Provides a Docker sandbox to safely test risky agent actions by mocking or blocking tool calls and denying direct internet access.
• Harnessing LLM Assistants: Offers a "guided" experience via MCP (Model Context Protocol) servers that allow coding assistants (like Claude Code or Cursor) to discover risks, write evaluation configs, and run pipelines.
• ACS (Agent Control Specification): Generates deployable policies from real failures to govern the agent, then re-runs evaluations to prove the failure rate has decreased.

Who it’s for

It is designed for developers and AI safety engineers building agents or multi-agent systems who need to move from a policy document to a measured, regression-tested safety gate in their CI/CD pipeline.

Highlights

• Spec-driven coverage: Generates test cases directly from product requirements rather than generic benchmarks.
• Framework-agnostic: Supports any model endpoint via LiteLLM and any agent framework (LangGraph, CrewAI, AutoGen, etc.) via OpenInference.
• Curated behavior library: Includes a catalog of atomic behavior presets for safety, bias, and fairness.
• Local-first artifacts: Writes all results to local JSON/JSONL files and includes a bundled viewer for side-by-side run comparison.