Arize-ai/phoenix
AI Observability & Evaluation
What it solves
Phoenix addresses the difficulty of monitoring and optimizing LLM applications. It provides visibility into application runtime through tracing, allows for performance benchmarking via evaluation, and helps manage the iterative process of prompt engineering and experimentation.
How it works
Phoenix uses OpenTelemetry-based instrumentation to trace LLM application runtimes. It leverages LLMs to perform evaluations on responses and retrieval quality. Users can create versioned datasets for experiments and use a built-in AI engineering agent (PXI) to debug traces and iterate on prompts. It is vendor-agnostic and supports various frameworks like LangChain, LlamaIndex, and CrewAI, as well as providers like OpenAI and Anthropic.
Who it’s for
- AI engineers building and debugging LLM applications.
- Developers needing to benchmark RAG (Retrieval-Augmented Generation) performance.
- Teams requiring systematic prompt management and version control.
- Users of coding agents (like Cursor or Claude Code) who want to query traces and datasets via MCP.
Highlights
- Tracing: OpenTelemetry-based runtime instrumentation.
- Evaluation: LLM-powered benchmarking for response and retrieval.
- Experimentation: Tools to track changes to prompts, models, and retrieval.
- Prompt Management: Systematic testing with version control and tagging.
- AI Agent Integration: Remote MCP server support for querying data from coding agents.