Giskard-AI/giskard-oss

What it solves

Giskard is designed to handle the non-deterministic nature of AI agents and LLM-based systems, where traditional unit tests are insufficient. It provides a framework for testing, evaluating, and red-teaming these systems to catch regressions, validate RAG (Retrieval-Augmented Generation) quality, and ensure safety and security against adversarial attacks.

How it works

The project is organized into modular packages:

• Giskard Checks: A library for creating evaluations (evals) using a "Target" (the system under test), "Scenarios" (interactions and checks), and "Checks" (assertions or LLM-as-judge assessments). It supports multi-turn conversations and built-in judges for groundedness and conformity.
• Giskard Scan: A red-teaming layer that automatically generates adversarial test suites based on a plain-language description of the agent. It targets vulnerabilities like prompt injection, harmful content, and misinformation, covering OWASP LLM Top-10 threats.
• Foundation Libraries: giskard-core, giskard-llm (provider-agnostic routing), and giskard-agents (orchestration) provide the underlying infrastructure.

Who it’s for

Developers and AI engineers building LLM-based agents, RAG pipelines, and multi-step AI workflows who need to ensure their systems are robust, safe, and reliable before deployment.

Highlights

• LLM-as-Judge: Uses LLMs to evaluate the quality and groundedness of agent responses.
• Automated Red-Teaming: Automatically generates adversarial inputs to probe for vulnerabilities and prompt injections.
• Modular Architecture: Lightweight packages that allow users to wrap any sync/async callable as a target.
• RAG Evaluation: Specialized tools for evaluating knowledge-base quality and grounding.