Jwuthri/Tracely-ai

What it solves

Tracely is a trace-native CI/CD system for AI agents that turns production failures into regression tests. It eliminates the need to hand-author datasets by promoting real failing traces from production into hermetic, replayable test cases that can block pull requests if a regression is reintroduced.

How it works

Tracely captures agent traces via OTLP/HTTP, organizing them by conversation and turn. It uses online evaluators (LLM-as-judge or structural checks) to grade runs and clusters similar failures into issues. When a failure is identified, it can be promoted to a regression case with a "fail-to-pass contract." These cases are then replayed in CI/CD pipelines using recorded fixtures (making them deterministic and free of model spend) or live calls to ensure the agent's behavior is corrected.

Who it’s for

Developers building and maintaining complex AI agent systems who want to move beyond simple observability dashboards to a rigorous, automated testing and gating process based on real-world production data.

Highlights

• Trace-to-Test Promotion: Convert failing production traces into regression tests with one click.
• Hermetic Replay: Replay tests in CI using recorded tool and LLM outputs as fixtures to avoid API costs and non-determinism.
• CI Gating: Integrates with GitHub Actions to block PRs that reintroduce known failures.
• Failure Clustering: Automatically groups similar failing runs into semantic and structural issues to reduce triage noise.
• Visual Debugging: Includes a waterfall trace view and a "Fleet" view that renders conversations as a pixel-art office for easier multi-agent system explanation.
• Automated Alerting: A visual flow builder for creating complex alert rules with LLM steps and webhooks.