rhesis-ai/rhesis

What it solves

Rhesis is a collaborative testing platform designed to move LLM and agentic application validation from manual, one-off audits to a continuous, team-based process. It addresses the non-deterministic nature of AI outputs by providing tools for automated test generation, adversarial red-teaming, and structured evaluation using a wide array of metrics.

How it works

The platform integrates into the AI development lifecycle through a Python SDK and a collaborative UI. It uses AI-powered synthesis to generate test scenarios from plain-language requirements and connects to context sources (via files or MCP) to ensure tests are knowledge-aware.

For execution, it establishes a persistent outbound WebSocket connection to the user's application, allowing Rhesis to trigger tests remotely without requiring a public URL. It utilizes OpenTelemetry for tracing LLM calls and latency, and leverages LiteLLM to support over 100 different model providers for both test generation and evaluation.

Who it’s for

• Engineers who need to integrate AI testing into CI/CD pipelines and debug failures via traces.
• Product Managers and Domain Experts who define behavioral requirements and review evaluation results without needing to write code.
• Security Teams looking for continuous adversarial testing (red-teaming) to detect jailbreaks and PII leaks.

Highlights

• AI-Powered Test Synthesis: Generates hundreds of edge cases and adversarial prompts from simple requirements.
• Adversarial Agents: Includes the Polyphemus agent for red-teaming and built-in Garak integration for vulnerability scanning.
• Conversation Simulation: The Penelope agent simulates realistic multi-turn dialogues to test context retention and coherence.
• Extensive Metrics: Over 60 pre-built metrics from frameworks like RAGAS and DeepEval, using LLM-as-Judge for reasoning.
• Deep Observability: OpenTelemetry-based tracing for LangChain, LangGraph, and other major agent frameworks.