chirpz-ai/pandaprobe

open source agent engineering platform: traces, evals, and metrics to debug and improve your AI agents. Integrates with LangGraph, CrewAI, Claude Agent SDK, and more.

What it solves

PandaProbe is an agent engineering platform designed to help teams collaboratively trace, evaluate, monitor, and debug AI agents. It provides a centralized way to understand how agents are performing and where they are failing.

How it works

The platform consists of a client SDK/HTTP client that sends data to a backend architecture comprising a FastAPI server, a Celery worker for background processing, and a PostgreSQL database for persistence. It uses Redis as a message broker. For evaluations, it employs an "LLM-as-a-judge" approach via LiteLLM to score and provide verdicts on agent behavior.

Who it’s for

It is built for teams of AI engineers and developers who are building and optimizing AI agents and need professional tools for observability and evaluation.

Highlights

  • Collaborative tracing, monitoring, and debugging of AI agents.
  • Support for both managed cloud hosting and self-hosting via Docker.
  • Integrated evaluation service using LLM-as-a-judge.
  • Comprehensive dashboard for visualizing traces and sessions.

Related

  • Project
    langwatch/langwatch

    LangWatch is an open‑core platform for LLM observability, testing, routing, and governance. It lets teams trace every LLM call, run simulation tests, manage prompts, enforce budgeted virtual keys, and get usage reports across coding assistants, LangChain‑style agents, and major model providers.

  • Project
    canwhite/AgentEval

    A transparent HTTP proxy that captures Agent-LLM traffic to provide automated grading, behavioral diagnosis, and source-code probing for AI agent evaluation.

  • Project
    traceroot-ai/traceroot

    TraceRoot is an open‑source platform that records AI agent calls, flags bad behaviour with custom detectors, clusters issues into signals, and runs automated evals so a coding‑agent can suggest and verify fixes, closing a self‑improvement loop. It offers a CLI, Python/TS SDKs, and integrations with major agent frameworks and model providers, and can be used via hosted cloud or self‑hosted Docker Compose.

  • Project
    Tracer-Cloud/opensre

    OpenSRE is an open‑source framework for AI‑driven Site Reliability Engineering agents. It connects to 60+ observability, cloud, and incident‑management tools, fetches relevant context, runs a tool‑calling reasoning loop, and returns evidence‑backed diagnoses or remediation actions. Users can interact via an interactive shell, a headless CLI, or a Python API, and deploy the system via a managed cloud gateway, self‑hosted container, or EC2 AMI. The project is in public‑alpha (v0.1) and licensed under Apache 2.0.

  • Project
    Kritt-ai/open-kritt

    open·kritt is an open‑source, self‑hosted platform that orchestrates AI agents (via Codex, Claude Code, Grok Build, etc.) to run custom, prompt‑driven security workflows on code repositories. It builds focused tasks, runs them in parallel Docker containers, validates findings, de‑duplicates and ranks them, and exports a complete report bundle. Designed for bug‑bounty researchers and DevSecOps, it offers a web UI and a headless CLI, supports any model provider, and is released under AGPL‑3.0.