Giskard-AI/giskard-oss

🐢 Open-Source Evaluation & Testing library for LLM Agents

What it solves

It provides a framework for testing and evaluating agentic systems and LLM-based applications. It addresses the challenge of testing non-deterministic outputs where traditional unit tests fail, and helps developers identify vulnerabilities like prompt injection, data leakage, and hallucinations in RAG systems.

How it works

Giskard is organized as a modular set of Python packages that can wrap any LLM, black-box agent, or pipeline:

  • Giskard Checks: A library for creating evaluations (evals) using a scenario API. It supports simple assertions, regex, semantic similarity, and "LLM-as-judge" assessments (e.g., checking for groundedness or conformity).
  • Giskard Scan: An automated red-teaming tool that generates adversarial test suites based on a plain-language description of the agent to probe for security threats (OWASP LLM Top-10), stereotypes, and misinformation.
  • RAG Evaluation: Tools for generating synthetic test sets (questions, reference answers, and context) directly from a knowledge base to validate RAG quality.

Who it’s for

AI developers and QA engineers building LLM agents, RAG pipelines, and multi-turn conversational systems who need to ensure safety, reliability, and performance.

Highlights

  • Modular Architecture: Lightweight packages (giskard-checks, giskard-scan) with minimal dependencies.
  • Async-first: Designed for dynamic, multi-turn testing of AI agents.
  • Automated Red-Teaming: Automatically generates adversarial inputs to test for prompt injection and harmful content.
  • LLM-as-Judge: Built-in support for complex evaluations like groundedness and conformity.