openlit/openlit

Open-source observability & evaluation platform for AI agents and coding agents. Trace LLMs, tools, prompts, costs & agent workflows with OpenTelemetry.

What it solves

OpenLIT provides observability and evaluation for AI agents and applications. It helps developers move from simply knowing an agent produced an answer to knowing if that answer was high quality, while providing deep visibility into the complex workflows (LLM calls, tool use, retrieval, and memory) that lead to that output.

How it works

OpenLIT uses OpenTelemetry to trace and monitor AI applications. It provides an SDK for Python and TypeScript that automatically instruments supported LLM providers, vector databases, and AI frameworks. It can also be integrated into existing OpenTelemetry architectures via an OTLP endpoint. For AI coding agents like Claude Code, Cursor, and Codex, it offers a dedicated CLI for instrumentation.

Who it’s for

Developers building production AI agents and LLM-powered applications who need to debug root causes of failures, track costs across models and users, and automate the evaluation of agent outputs.

Highlights

  • OpenTelemetry-native: Built on open standards rather than proprietary formats, allowing integration with other OTel backends like Grafana or Datadog.
  • AI Evaluations: Built-in LLM-as-a-Judge evaluations for hallucination, bias, toxicity, safety, and relevance.
  • Cost Observability: Detailed tracking of token usage and costs across models, providers, and environments.
  • Prompt Hub: A centralized, versioned system for managing prompts separately from application code.
  • Rule Engine: Ability to define runtime rules to dynamically control prompts and evaluations based on trace attributes.
  • Wide Integration Support: Auto-instrumentation for over 70 AI providers and data stores.

Related

  • Project
    traceloop/openllmetry

    An open-source observability framework built on OpenTelemetry that provides complete tracing and monitoring for LLM applications, providers, and vector databases.

  • Project
    mlflow/mlflow

    An open-source AI engineering platform that enables teams to debug, evaluate, monitor, and optimize LLMs, AI agents, and ML models for production.

  • Project
    langfuse/langfuse

    An open-source LLM engineering platform for tracing, monitoring, and evaluating AI applications to streamline the development and development cycle.

  • Project
    comet-ml/opik

    An open-source LLM observability and evaluation platform for tracing AI agents, managing prompts, and monitoring production LLM applications.

  • Project
    truera/trulens

    An OpenTelemetry-native evaluation framework for AI agents and RAG apps that uses LLM judges to trace and score application performance, cost, and quality.