adrida/tracer

What it solves

TRACER reduces the cost and latency of LLM-based classification pipelines by routing "easy" inputs to a fast, lightweight traditional ML model (the surrogate) and only deferring "hard" or uncertain inputs to the expensive LLM. This allows developers to maintain the quality of a teacher LLM while significantly cutting API costs and inference time.

How it works

TRACER uses a pipeline of embeddings, a surrogate model, and an acceptor gate:

1. Fit: It trains a suite of candidate traditional ML models (like logistic regression or gradient-boosted trees) on existing LLM classification traces (input-label pairs).
2. Gate: It attaches a learned acceptor that predicts whether the surrogate will agree with the teacher LLM.
3. Calibrate: It adjusts the acceptor threshold to meet a specific target parity (e.g., 95% agreement with the LLM).
4. Guard: It prevents deployment if the surrogate cannot meet the quality bar on held-out data.
5. Flywheel: Every deferred input handled by the LLM creates a new trace, which is used to refit the surrogate, increasing coverage over time.

Who it’s for

Developers and ML engineers building classification systems using LLMs who want to reduce operational costs and latency without sacrificing accuracy.

Highlights

• Cost Efficiency: Can route 90%+ of traffic to traditional ML, drastically reducing LLM API calls.
• Self-Improving: Uses a continual learning flywheel where deferred calls feed back into the training set.
• Formal Parity Guarantees: Ensures the surrogate matches the teacher LLM's output within a defined threshold.
• Observability: Includes a "watch" feature to record LLM calls as OpenTelemetry GenAI spans locally or via Tracer Cloud.
• Multi-language Support: Provides a Python library and a JavaScript/Node.js integration for observability and routing.