langfuse/langfuse

What it solves

Langfuse is an open-source LLM engineering platform designed to help teams collaboratively develop, monitor, evaluate, and debug AI applications. It addresses the complexity of tracking LLM calls, managing prompts, and ensuring the quality of AI outputs in production.

How it works

Langfuse provides a suite of tools that integrate into an AI application via SDKs (Python, JS/TS) or direct API calls. It captures "traces" of LLM calls and other application logic—such as retrieval, embedding, or agent actions—allowing developers to inspect and debug complex logs and user sessions. It also includes a central prompt management system for versioning and iterating on prompts without adding latency, and an evaluation framework that supports LLM-as-a-judge, manual labeling, and custom pipelines.

Who it’s for

It is built for AI engineers and development teams who need a professional LLM observability and evaluation pipeline to move their AI applications from prototype to production.

Highlights

• LLM Application Observability: Track and debug LLM calls, retrieval steps, and agent actions through detailed traces.
• Prompt Management: Centrally manage, version control, and collaboratively iterate on prompts with server and client-side caching.
• Evaluations: Support for LLM-as-a-judge, code evaluators, user feedback, and manual labeling.
• Datasets: Create test sets and benchmarks for continuous improvement and pre-deployment testing.
• LLM Playground: An interactive environment to test and iterate on prompt and model configurations.
• Broad Integrations: Native support for OpenAI, LangChain, LlamaIndex, Haystack, and various agent frameworks like CrewAI and AutoGen.