monocle2ai/monocle

Monocle is a framework for tracing GenAI app code. This repo contains implementation of Monocle for GenAI apps written in Python.

What it solves

Monocle provides observability and testing for Generative AI applications and agents. It solves the "black-box" nature of AI agents by turning complex, multi-step workflows—including model calls, tool invocations, and vector lookups—into structured, step-by-step traces. It also enables developers to move beyond simple input/output testing by allowing assertions on the actual execution traces (e.g., verifying if a specific tool was called or if token costs stayed within budget).

How it works

Monocle is a GenAI-specific observability layer built on top of OpenTelemetry (OTel). It uses a standardized metamodel to map AI-specific entities (like prompts, responses, and vector operations) to OTel span attributes. It provides instrumentation adapters for popular frameworks (such as LangChain, LlamaIndex, and Haystack) to automatically create spans without manual wiring. Traces are emitted as OTLP-compatible spans, which can be exported to local JSON files, consoles, or cloud backends like Okahu.

Who it’s for

  • App developers who want deep visibility into their GenAI apps without writing bespoke tracing code.
  • Platform and infrastructure engineers who prefer using wrappers or operators to inject observability into applications.
  • Enterprises and SREs who require standardized, OTel-compliant traces and CI-friendly tests for reliability and compliance.

Highlights

  • Native OpenTelemetry Support: Emits standard OTLP spans compatible with any OTel backend.
  • Auto-Instrumentation: Built-in adapters for a wide range of agentic frameworks, LLM providers, and vector stores.
  • Trace-Based Testing: The monocle-test-tools package allows for pytest-style assertions on agent behavior, tool usage, and inference costs.
  • Flexible Deployment: Supports both in-code initialization and zero-code wrapper/operator modes for platform-level integration.
  • AI Assistant Tracing: Includes specialized hooks to trace AI coding assistants like Claude CLI, GitHub Copilot, and OpenAI Codex CLI.

Related

  • Project
    traceloop/openllmetry

    An open-source observability framework built on OpenTelemetry that provides complete tracing and monitoring for LLM applications, providers, and vector databases.

  • Project
    truera/trulens

    An OpenTelemetry-native evaluation framework for AI agents and RAG apps that uses LLM judges to trace and score application performance, cost, and quality.

  • Project
    lmnr-ai/lmnr

    An open-source observability platform for AI agents that provides tracing, monitoring, and evaluation tools to debug and optimize agentic workflows.

  • Project
    Asymptote-Labs/agent-beacon

    Agent Beacon is an open‑source telemetry layer that automatically captures, normalises, and stores activity from dozens of AI‑agent runtimes (local CLIs, browser chats, CI jobs, cloud SDKs). It writes a unified JSONL log on the endpoint, offers a local read‑only dashboard, and can forward data to any major SIEM, log‑aggregation service, or object storage. Packages for macOS, Linux, and Windows support MDM‑style fleet deployment, and an offline rule engine (`beacon scan`) enables threat detection without network access.

  • Project
    traceloop/openllmetry-js

    An open-source observability library for JavaScript/TypeScript LLM applications that uses OpenTelemetry to trace calls to LLM providers and vector databases.