comet-ml/opik

What it solves

Opik is an open-source platform designed to help developers move LLM applications and AI agents from prototype to production. It addresses the difficulty of monitoring complex, multi-step agent activity, evaluating the quality of LLM outputs (such as detecting hallucinations), and optimizing prompts and agents based on real-world data.

How it works

Opik provides a full-stack observability and evaluation suite that can be self-hosted or used via cloud. It works by integrating into the LLM application via a Python or TypeScript SDK (or OpenTelemetry), allowing developers to track all LLM calls, tool executions, and retrieval steps. The platform includes a server backend and a web UI for visualizing trace trees, managing datasets, and running experiments.

Who it’s for

It is built for ML engineers, AI teams, and engineering teams who require a framework-agnostic, open-source observability tool that can be run within their own infrastructure to ensure data privacy.

Highlights

• AI Agent Tracing: Deep tracing of multi-step agent activity and tool calls with full trace trees.
• LLM-as-a-Judge: Automated evaluation metrics for hallucination detection, moderation, and RAG assessment (e.g., Answer Relevance and Context Precision).
• Self-Hostable: Fully open-source (Apache-2.0) platform, including the backend and web application.
• Production Monitoring: Scalable dashboards for tracking token usage, feedback scores, and online evaluation rules.
• CI/CD Integration: PyTest integration to test LLM pipelines on every commit.