EVA End-to-End Evaluation Framework for Voice Agents

TL;DR

EVA is a new end‑to‑end evaluation framework released by ServiceNow that jointly scores conversational voice agents on task accuracy (EVA‑A) and user experience (EVA‑X), providing the first benchmark that captures both dimensions together and revealing a consistent trade‑off between them.

Introduction

Voice agents must succeed at two intertwined goals: completing the user’s task correctly (accuracy) and delivering a natural, concise, and timely spoken interaction (experience). Existing benchmarks evaluate these goals in isolation, which hides failures that only appear when the full conversation is considered. EVA addresses this gap by evaluating multi‑turn spoken dialogues in a realistic bot‑to‑bot setup and producing two high‑level scores—EVA‑A for accuracy and EVA‑X for experience.

Background and Motivation

Current voice‑agent benchmarks focus on single components such as speech‑to‑text quality (AudioBench, SD‑Eval, VoxEval), perceived speech quality (EmergentTTS‑Eval, SHEET), or conversational dynamics (Talking Turns, Full‑Duplex‑Bench). A few recent works (VoiceAgentBench, CAVA) begin to assess tool‑calling and instruction‑following, but none evaluate complete, multi‑step spoken workflows that combine task orchestration with conversational fluency. This fragmented evaluation landscape makes it impossible to measure how accuracy and experience interact in real deployments, motivating a unified framework.

EVA Framework

Architecture

EVA simulates live audio conversations using five core components:

  1. User Simulator – A goal‑driven conversational AI that generates realistic spoken input via high‑quality TTS.
  2. Voice Agent – The system under test, built with Pipecat and supporting both cascade (STT → LLM → TTS) and audio‑native (S2S or LALM) pipelines.
  3. Tool Executor – Deterministic Python functions that provide reproducible tool responses and update a scenario‑specific database.
  4. Validators – Automated checks that ensure each conversation reaches a valid end state; invalid runs are regenerated, eliminating the need for human post‑hoc labeling.
  5. Metrics Suite – A collection of deterministic and LLM‑as‑judge metrics that analyze the audio recording, transcript, and tool‑call logs.

EVA architecture

Data

The initial release ships with a synthetic airline dataset containing 50 scenarios and 15 tools covering flight rebooking, cancellations, standby, and voucher issuance. Each scenario defines:

  • User Goal – The precise task the caller wants to accomplish.
  • User Persona – Speaking style, patience, and personality traits.
  • Scenario Database – The backend data queried by tools.
  • Ground Truth – The expected final state of the database after a successful interaction.

The dataset is available on Hugging Face and can be explored via the EVA demo site.

Evaluation Methodology

EVA reports two primary scores and a set of diagnostic metrics.

EVA‑A: Accuracy

Accuracy is measured along three axes:

  • Task Completion (deterministic) – Checks whether the final database state matches the ground truth.
  • Faithfulness (LLM‑as‑Judge) – Detects hallucinations, policy violations, or misrepresentations in the agent’s spoken responses.
  • Speech Fidelity (LALM‑as‑Judge) – Evaluates whether the agent’s audio output correctly renders critical entities such as confirmation codes, flight numbers, and monetary amounts.

EVA‑X: Experience

Experience is measured along three complementary axes, all using LLM‑as‑Judge:

  • Conciseness – Rewards brief, focused replies suitable for phone users who cannot skim.
  • Conversation Progression – Rewards forward‑moving dialogue that avoids repetition and maintains context.
  • Turn‑Taking – Rewards appropriate timing, i.e., no interruptions and minimal dead air.

Pass@k and Pass^k

EVA runs three trials per scenario (k = 3) and reports:

  • pass@k – Probability that at least one trial succeeds.
  • pass^k – Probability that all three trials succeed. These metrics capture both peak performance and consistency.

Key Findings

  • Accuracy‑Experience Trade‑off – Across 20 evaluated systems (both proprietary and open‑source, cascade and audio‑native), higher task‑completion scores consistently correlated with poorer experience scores, and vice versa. This trade‑off is invisible to benchmarks that only measure task success.
  • Named‑Entity Transcription Errors – Mis‑recognizing a single character in a confirmation code often caused authentication failures and full conversation breakdowns.
  • Multi‑Step Workflow Fragility – Rebooking flights while preserving ancillary services (seats, baggage) was the most common failure point across all configurations.
  • Consistency Gap – The gap between pass@3 and pass^3 was large for all systems, indicating that many agents can complete a task sporadically but lack reliable performance.

Early results scatter plot

Limitations

  • Metric Bias – LLM‑as‑Judge models inherit biases from their training data and may favor certain response styles; binary task‑completion scoring ignores partial credit.
  • Domain & Language Coverage – The current release only includes 50 English‑language airline scenarios; results may not generalize to other domains, languages, or accents.
  • Simulation Fidelity – The user simulator uses a single commercial TTS provider and may not capture real‑world disfluencies, emotions, or diverse speaking styles. The bot‑to‑bot pipeline also abstracts away production‑level latency and infrastructure variations.
  • Resource Requirements – Full reproduction requires commercial API access for TTS/ASR services, and latency measurements will vary across providers.

Future Roadmap

  • Prosodic Quality – Add metrics for pronunciation, rhythm, and expressiveness, addressing the current low alignment between LALM‑as‑Judge scores and human judgments.
  • Robustness Testing – Introduce noisy audio, varied accents, multilingual users, and affect‑aware evaluation for distressed callers.
  • Domain Expansion – Release additional datasets covering different policy structures, entity types, and longer conversational memories.
  • Error‑Analysis Tooling – Provide an application that automatically surfaces per‑metric errors, representative examples, and structured summaries of model strengths and weaknesses.
  • Leaderboard Growth – Continuously update a public leaderboard to track state‑of‑the‑art voice‑agent performance across the community.

Acknowledgements

Core contributors: Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz, Oluwanifemi Bamgbose, Hoang Nguyen, Raghav Mehndiratta, and Hari Subramani. Additional thanks to Lindsay Brin, Akshay Kalkunte, Joseph Marinier, Jishnu Nair, Aman Tiwari, Fanny Riols, Anil Madamala, Sridhar Nemala, Srinivas Sunkara, and the ServiceNow PAVA and CLAE teams.

Citation

@misc{eva-2026,
  title={A New End-to-end Framework for Evaluating Voice Agents (EVA)},
  author={Bogavelli, Tara and Gauthier Melançon, Gabrielle and Stankiewicz, Katrina and Bamgbose, Oluwanifemi and Nguyen, Hoang and Mehndiratta, Raghav and Subramani, Hari},
  year={2026},
  url={https://github.com/ServiceNow/eva}
}

Sources