comet-ml/opik

What it solves

Opik is an open-source platform designed to help developers move LLM applications and AI agents from prototype to production. It addresses the difficulty of observing, evaluating, and optimizing the complex, multi-step workflows typical of agentic systems, providing a centralized place to trace calls, manage prompts, and monitor performance.

How it works

Opik provides a full-stack observability suite that can be self-hosted or used via cloud. It uses a Python/TypeScript SDK and OpenTelemetry support to capture detailed traces of LLM calls, tool executions, and retrieval steps. The platform includes a web UI for visualizing trace trees, a prompt playground for experimentation, and an evaluation framework that supports datasets and "LLM-as-a-judge" metrics for automated testing.

Who it’s for

It is built for ML engineers and AI teams who need a framework-agnostic observability tool that can be run within their own infrastructure to ensure data privacy and avoid vendor lock-in.

Highlights

• Full-Platform Open Source: The entire backend and web application are Apache-2.0 licensed and free to self-host.
• Agent Tracing: Deep visibility into multi-step agent activity and tool calls with full trace trees.
• LLM-as-a-Judge: Built-in metrics for hallucination detection, moderation, and RAG assessment (e.g., Answer Relevance and Context Precision).
• CI/CD Integration: PyTest integration allows teams to run LLM pipeline evaluations on every commit.
• High Scalability: Designed to handle production volumes of over 40 million traces per day.