openlit/openlit

OpenLIT is the open-source agent harness engineering platform: trace, evaluate, guard, and improve everything around the model in your AI agents, on OTEL.

What it solves

OpenLIT provides a comprehensive engineering platform for AI agents, focusing on the "harness" (the tools, prompts, and guardrails surrounding the model). It solves the difficulty of debugging and improving production agents by allowing teams to trace every step of an agent's execution, evaluate its quality, and fix issues without needing to redeploy the entire application.

How it works

OpenLIT uses an OpenTelemetry-native approach to instrument AI applications. By adding a few lines of SDK code, it automatically captures traces and metrics for LLM calls, tool usage, and retrieval steps. This telemetry flows through the OpenTelemetry protocol (OTLP) to an OpenLIT dashboard where users can monitor costs, latency, and performance. It also includes a rule engine to dynamically control runtime behavior and a prompt hub for versioned prompt management.

Who it’s for

It is designed for developers and engineering teams building and deploying AI agents and coding agents (such as those using Claude Code, Cursor, or LangGraph) who need production-grade observability, evaluation, and guardrails.

Highlights

  • OpenTelemetry Native: Uses standard GenAI semantic conventions, allowing integration with existing OTel backends like Grafana or Datadog.
  • Agent Harness Engineering: Tools for managing prompts, context, and rules separately from application code.
  • Comprehensive Observability: Traces LLM calls, tool calls, MCP requests, and GPU utilization (NVIDIA, AMD, Intel).
  • Automated Evaluations: Built-in LLM-as-a-judge evaluations for hallucination, bias, toxicity, and safety.
  • Coding Agent Support: Specialized instrumentation for AI coding agents like Cursor and Claude Code to track file edits and shell commands.
  • Runtime Guardrails: Detects and blocks prompt injection and sensitive topics at runtime.

Related

  • Project
    comet-ml/opik

    An open-source LLM observability and evaluation platform for AI agent tracing, prompt management, and production monitoring.

  • Project
    traceroot-ai/traceroot

    TraceRoot is an open‑source platform that records AI agent calls, flags bad behaviour with custom detectors, clusters issues into signals, and runs automated evals so a coding‑agent can suggest and verify fixes, closing a self‑improvement loop. It offers a CLI, Python/TS SDKs, and integrations with major agent frameworks and model providers, and can be used via hosted cloud or self‑hosted Docker Compose.

  • Project
    truera/trulens

    An OpenTelemetry-native evaluation and observability framework that uses LLM judges to trace agent failures and optimize the cost-quality tradeoff in AI applications.

  • Project
    traceloop/openllmetry

    An open-source observability framework built on OpenTelemetry that provides complete tracing and monitoring for LLM applications, providers, and vector databases.

  • Project
    Kritt-ai/open-kritt

    open·kritt is an open‑source, self‑hosted platform that orchestrates AI agents (via Codex, Claude Code, Grok Build, etc.) to run custom, prompt‑driven security workflows on code repositories. It builds focused tasks, runs them in parallel Docker containers, validates findings, de‑duplicates and ranks them, and exports a complete report bundle. Designed for bug‑bounty researchers and DevSecOps, it offers a web UI and a headless CLI, supports any model provider, and is released under AGPL‑3.0.