Giskard-AI/giskard-oss

What it solves

Giskard provides a structured way to test and evaluate AI agents and LLM-based systems. It addresses the challenge of verifying non-deterministic AI behavior, helping developers catch regressions, ensure safety, and validate the quality of RAG (Retrieval-Augmented Generation) systems.

How it works

The library is divided into two main modular components: giskard-checks for defining custom evaluation scenarios (like assertions or LLM-as-judge assessments) and giskard-scan for automated red-teaming. You wrap your agent—any sync or async function—and use these tools to simulate interactions, probe for vulnerabilities like prompt injection, and verify that outputs meet your specific requirements.

Who it’s for

It is designed for developers and AI engineers building agentic systems or LLM applications who need a reliable, programmatic way to perform quality assurance, security testing, and performance validation throughout their development lifecycle.

Highlights

• Automated Red Teaming: Automatically generates adversarial test suites to identify vulnerabilities like prompt injection, harmful content, and misinformation.
• LLM-as-Judge: Includes built-in evaluation checks that use LLMs to assess groundedness, conformity, and other qualitative metrics.
• Modular Architecture: Built as a lightweight, async-first framework where you only install the components you need.
• Flexible Testing: Supports testing of any callable agent or pipeline, allowing for both simple assertions and complex multi-turn conversation evaluations.