Azure-Samples/ai-rag-chat-evaluator

What it solves

This project provides a framework for evaluating the quality and style of answers generated by chat applications using Retrieval-Augmented Generation (RAG). It helps developers measure how changes to system prompts, search parameters, and model settings affect the accuracy and reliability of the AI's responses.

How it works

The tool runs a set of test questions against a target chat endpoint and compares the generated answers to "ground truth" (ideal) answers. It uses a combination of metrics to score the results:

• Built-in GPT Metrics: Uses a GPT-4 model to rate answers on a scale of 1-5 for coherence, relevance, groundedness, similarity, and fluency.
• Custom Prompt Metrics: Allows users to define their own prompts for evaluation to support different languages or specific requirements.
• Code-based Metrics: Calculates objective data such as latency, answer length, and the presence of correct citations.
• "I don't know" Testing: Includes a specialized workflow to generate and test questions that cannot be answered by the data to ensure the model doesn't hallucinate.

Who it’s for

Developers building RAG-based chat applications who need a systematic way to benchmark their performance and iterate on prompts and configurations.

Highlights

• Ground Truth Generation: Includes scripts to automatically generate initial QA pairs for testing.
• Comparative Analysis: Features a diff tool to compare answers and metrics between two different evaluation runs side-by-side.
• Summary Tool: Provides an interactive CLI table to view aggregate results across multiple experiments.
• Flexible Integration: Supports any app following the AI Chat Protocol and allows custom JMESPath expressions to extract answers from non-standard responses.