jwuthri-cloud/Tracely-ai

Tracely – AI‑agent observability & regression testing

What it is – Tracely is a self‑hosted platform that watches the runs of your AI agents (LLM‑driven bots, tool‑using assistants, multi‑agent systems) in production, turns any failure it sees into a deterministic regression test, and blocks pull‑requests that re‑introduce the same bug. It does this by ingesting OpenTelemetry‑style traces, evaluating them with built‑in judges, clustering similar failures, and replaying the exact recorded inputs/outputs in CI.

Key components

• Ingest service – receives OTLP/HTTP traces from any language, stores raw spans in ClickHouse/Postgres and indexes agent‑specific fields (agent id, conversation id, turn, step).
• Evaluators / judges – lightweight checks (no model) plus LLM‑based judges that score a run, a turn, or a span. Verdicts are written back as columns on the trace table.
• Failure clustering – groups failing runs by structural signature and semantic embedding, producing a single issue with a count and suggested evaluator.
• Regression cases – one‑click promotion of a failing trace into a hermetic test case: all inputs, tool calls and LLM outputs are saved as fixtures, a “fail‑to‑pass” contract is attached, and optional assertions are defined.
• CI gate – a CLI (tracely gate/tracely simulate/tracely replay) runs the suite in GitHub Actions, exits non‑zero on failure, posts a commit status and a PR comment.
• Alert flows – visual builder for “when‑then” rules (e.g., a gate fails, a new failure type appears, a rate threshold is crossed). Actions can be Slack, email, webhook, an LLM step, or a Python expression.
• Dashboard & views – trace timeline, hierarchical waterfall, “Fleet” pixel‑art office view, trends over time, judge calibration screen, and an in‑app assistant that can create evaluators, scenarios, cases, and alerts via natural language.

How you use it

1. Instrument your agent with the tracelyai-sdk (or any OTLP exporter). The SDK auto‑captures OpenAI/Anthropic/Google/Mistral calls, tool usage, token usage, latency, etc.
2. Run your service in prod; Tracely stores each conversation as a trace.
3. Detect – evaluators grade runs live; failures are automatically clustered.
4. Promote a failure to a regression case with a single click.
5. Deploy – add the Tracely GitHub Action to your PR workflow; the gate replays the case fixtures offline (no API keys, $0 cost) and blocks the PR if the bug returns.
6. Monitor – dashboards show failure counts, gate pass‑rates, latency/cost trends, and let you calibrate judges against human review.

Installation & deployment

• One‑click deployment to Railway or Railyard via provided buttons.
• Full Docker‑Compose stack (backend, worker, frontend, ClickHouse, Postgres, Redis, MinIO) – docker compose --profile demo up spins up a demo with pre‑populated traces, clusters, cases, and gates.
• For custom infra, see deploy/railway and deploy/eks guides; the stack runs in a single Kubernetes namespace.

Typical workflow snippet

import tracely_sdk as tracely
tracely.init(
    endpoint="http://localhost:8000",
    api_key="tracely_dev_key",
    service_name="support-agent",
    env="prod",
    instrument="auto",
)

with tracely.trace(agent="support-agent", conversation="conv-1"):
    client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Where is order ORD-4471?"}],
    )

The SDK records a GENERATION span with all metadata; Tracely then evaluates it, clusters any failure, and can turn it into a regression case.

Why it matters Traditional LLM evaluation relies on hand‑crafted test sets that never exactly match production inputs. Tracely flips that model: the real failing run becomes the test, guaranteeing perfect fidelity, zero model‑call cost in CI, and automatic blocking of regressions. It bridges observability and continuous integration for AI agents.

Links

• Website: https://tracely-ai.com
• Docs: https://doc.tracely-ai.com
• SDK on PyPI: tracelyai-sdk
• GitHub Actions: .github/actions/tracely-gate