chirpz-ai/pandaprobe

open source agent engineering platform: traces, evals, and metrics to debug and improve your AI agents. Integrates with LangGraph, CrewAI, Claude Agent SDK, and more.

What it solves

PandaProbe is an agent engineering platform designed to help teams collaboratively trace, evaluate, monitor, and debug AI agents. It provides a centralized way to understand how agents are performing and where they are failing.

How it works

The platform consists of a client SDK/HTTP client that sends data to a backend architecture comprising a FastAPI server, a Celery worker for background processing, and a PostgreSQL database for persistence. It uses Redis as a message broker. For evaluations, it employs an "LLM-as-a-judge" approach via LiteLLM to score and provide verdicts on agent behavior.

Who it’s for

It is built for teams of AI engineers and developers who are building and optimizing AI agents and need professional tools for observability and evaluation.

Highlights

  • Collaborative tracing, monitoring, and debugging of AI agents.
  • Support for both managed cloud hosting and self-hosting via Docker.
  • Integrated evaluation service using LLM-as-a-judge.
  • Comprehensive dashboard for visualizing traces and sessions.

Related

  • Project
    langwatch/langwatch

    A platform for LLM evaluation and AI agent testing that provides end-to-end simulations, production observability, and an AI gateway for governance.

  • Project
    canwhite/AgentEval

    A transparent HTTP proxy that captures Agent-LLM traffic to provide automated grading, behavioral diagnosis, and source-code probing for AI agent evaluation.

  • Project
    traceroot-ai/traceroot

    An open-source observability and self-improving layer for AI agents that detects production failures, root-causes them to source code, and opens verified fix PRs.

  • Project
    Tracer-Cloud/opensre

    An open-source framework for AI SRE agents that automates production incident investigation and root-cause analysis across cloud infrastructure.

  • Project
    Kritt-ai/open-kritt

    An open-source security research platform that orchestrates AI agents to find, validate, and rank vulnerabilities in codebases using focused, parallelized workflows.