rhesis-ai/rhesis

The collaboration layer for AI teams: domain experts annotate and review agent behavior, engineers improve the agent from what they find.

What it solves

Rhesis provides a structured way to collect and act on feedback from domain experts and product managers when building AI agents. It bridges the gap between the people who know the correct answers (stakeholders) and the engineers building the agent, ensuring that feedback is attached to specific test cases and agent versions rather than being lost in chat threads.

How it works

Users connect their AI agent via a Python SDK (which uses a WebSocket for local or VPC access) or a public REST endpoint. Stakeholders then use a web-based playground to interact with the live agent, turn conversations into test cases, and provide structured pass/fail verdicts and comments. This feedback is then pulled back into the development loop via the SDK, REST API, or MCP-capable clients like Cursor and Claude Code. Over time, prose feedback is converted into automated tests and metrics.

Who it’s for

  • AI Engineers: To integrate feedback and reviewed test sets into their CI/CD pipelines and development workflow.
  • Domain Experts: To test the agent and provide qualitative feedback on accuracy and behavior without writing code.
  • Product Managers: To define requirements and transform scattered feedback into measurable tests and metrics.

Highlights

  • Integrated Feedback Loop: Connects live agent interaction with structured review and automated evaluation.
  • Flexible Connectivity: Supports both SDK-based WebSocket connections (no public URL needed) and REST endpoints.
  • Automated Test Generation: Creates tests from requirements, PRDs, or connected tools like Notion, GitHub, and Jira.
  • Extensive Evaluation: Includes over 60 metrics (DeepEval, garak) and LLM-as-Judge evaluators.
  • Developer Tooling: Provides an MCP skill for direct integration with AI-powered IDEs like Cursor.

Related

  • Project
    cookiy-ai/user-research-skill

    An open-source agent skill that enables AI agents to plan, run, and synthesize qualitative and quantitative user research with real human participants.

  • Project
    Tracer-Cloud/opensre

    An open-source framework for building AI SRE agents that automate production incident response by correlating data across logs, metrics, and traces.

  • Project
    chirpz-ai/pandaprobe

    An open source agent engineering platform for collaboratively tracing, evaluating, monitoring, and debugging AI agents.

  • Project
    ApodexAI/AgentHarness

    An evaluation harness for testing AI agents on deep-research benchmarks using a standardized ReAct setup.

  • Project
    rogue-security/rogue

    An AI agent evaluator and red teaming platform that stress-tests agents against business policies and adversarial attacks to identify security vulnerabilities.