Tracing and Evaluating smolagents with Arize Phoenix
Hugging Face has detailed a workflow for integrating Arize Phoenix with the smolagents library to provide real-time observability and systematic evaluation of AI agents. This integration allows developers to visualize the step-by-step decision-making process of an agent and measure the relevance and accuracy of its tool outputs using LLM-based judging.
Real-Time Tracing for Agent Observability
Tracing provides a granular view of an agent's internal workflow, enabling developers to debug issues and optimize performance by tracking every step from tool invocation to final response generation.
To implement tracing in smolagents, developers use OpenTelemetry and OpenInference for instrumentation, with Arize Phoenix serving as the visualization platform. The setup involves installing the smolagents[telemetry] module and registering a tracer provider that points to a Phoenix instance. Once configured, every agent invocation is automatically captured as a trace in the Phoenix dashboard, allowing for an "X-ray" view of the agent's decision-making process.
Systematic Evaluation via LLM-as-a-Judge
Evaluation (evals) is used to measure how effectively an agent retrieves, processes, and presents information. A primary method for achieving this is the "LLM-as-a-judge" approach, where a highly capable model (such as GPT-4o) is used to score the performance of another model or tool.
Evaluating Tool Relevance
In a practical implementation, the relevance of a search tool (such as DuckDuckGoSearchTool) can be evaluated by following these steps:
- Retrieve Execution Spans: Extract the specific traces where the tool was called using a
SpanQueryto isolate the input (query) and the output (result). - Apply Prompt Templates: Use a specialized prompt template, such as the
RAG_RELEVANCY_PROMPT_TEMPLATE, to guide the judge LLM. - Classify Results: The judge LLM analyzes the input and output to classify the result as either "relevant" or "unrelated."
- Log Results: The resulting binary scores (1 for relevant, 0 for unrelated) are logged back into Arize Phoenix for systematic analysis.
Versatile Evaluation Templates
Arize Phoenix provides a library of pre-written and pre-tested evaluation templates that can be adapted to different agent types and use cases. By swapping the prompt template, developers can measure various performance metrics beyond simple relevance.
| Evaluation Template | Applicable Agent Type |
|---|---|
| Hallucination Detection | RAG agents, General chatbots, Knowledge-based assistants |
| Q&A on Retrieved Data | RAG agents, Research Assistants, Document Search Tools |
| RAG Relevance | RAG agents, Search-based AI assistants |
| Summarization | Summarization tools, Document digesters, Meeting note generators |
| Code Generation | Code assistants, AI programming bots |
| Toxicity Detection | Moderation bots, Content filtering AI |
| AI vs Human (Ground Truth) | Evaluation & benchmarking tools, AI-generated content validators |
| Reference (Citation) Link | Research assistants, Citation tools, Academic writing aids |
| SQL Generation Evaluation | Database query agents, SQL automation tools |
| Agent Function Calling Evaluation | Multi-step reasoning agents, API-calling AI, Task automation bots |