Evaluating Voice Agents Against Production Interaction Data

A voice agent that clears every pre-launch benchmark can still fail in production for structural reasons, not from insufficient testing. Voice agents are sequential pipelines, turn-detection feeding automatic speech recognition, ASR feeding a language model, the language model triggering tool calls, tool calls feeding text-to-speech, where each stage can corrupt the input it hands to the next. The 2026 literature on agent failure modes names several of these patterns directly: transcript drift, premature turn cutoff, false barge-in, audio-quality degradation, hallucinated tool calls grounded in mis-heard speech, and false success, where the agent announces "I've cancelled your subscription" while the underlying tool call failed.

Pre-launch test suites cannot reproduce this compounding because they are built on curated, quiet, single-speaker audio. Production callers do not read scripts. None of that variability exists in a QA environment designed to prove a demo works.

The resulting gap does not land on one team. Developers see flaky scenarios that pass in text-only testing, but they break the moment real speech enters the loop. Site reliability engineers watch latency climb after a routine provider change. The fix for this gap is a different category of evidence: the production interaction itself, captured in enough detail to show which stage of the pipeline broke and why.

Aggregate scores, WER, and the hidden broken layer

Word error rate is a legitimate diagnostic for the ASR component in isolation, and a poor gate for the agent as a whole. A model can post a 5% WER on a benchmark like Common Voice while the live stack built on top of it carries a 32% task-failure rate, and there is no contradiction in that pairing, because the two numbers are measuring different things at different layers of the system.

If you treat WER as a stand-in for agent quality, four problems make it actively misleading. ASR is only one stage of four: a transcript that is 92% accurate at the word level can still drop the prefix "re-" from "reschedule," handing the LLM an intent that is flatly wrong and producing a fully incorrect end task despite a transcript that looks almost clean. LLM and TTS failures carry no WER penalty at all: a text-to-speech engine that mispronounces a confirmation number, or a language model that misreads a regional phrase, both score a perfect zero against this metric despite producing a real failure for the caller. Latency is invisible to WER in the same way: a transcript can be word-for-word correct but still arrive too late and break the call, even though the final text, read in isolation, looks exactly right.

Aggregate task-success dashboards suffer from the same structural flaw, just in a different guise. A team that already scores task completion and considers that sufficient is answering only whether a call failed, never where in the pipeline it failed or why, and without that attribution, every proposed fix is a guess dressed up as an engineering decision.

The three scoring layers every production voice eval needs

A production-grade evaluation framework for voice agents needs three distinct scoring layers, and each one answers a different question at a different point in the development cycle. None of the three substitutes for the others.

End-task scoring functions as the launch gate. It measures conversation resolution and task completion, the only question that ultimately matters: did the caller's goal get achieved? If a bot posts a strong ASR accuracy number alongside low conversation resolution, it is still shipping a broken product, no matter how clean its transcripts look. For this layer to produce a usable rubric rather than a guess, task statements need to be tight and specific, something closer to "confirm identity; locate booking; offer next available slot; send confirmation SMS" than a vague goal that forces an LLM judge to improvise its own definition of success.

Pipeline-stage attribution is the diagnostic layer, and it activates once end-task scoring fails. It asks which stage, ASR, LLM, tool execution, or TTS, produced the breakdown, and it exists to explain the end-task score.

Conversation-coherence scoring catches failures that turn-level rubrics miss entirely: persona drift across a call, context lost after a tool call fires, failure to recover gracefully from an interruption, contradictions between something the agent said in turn three and what it says in turn nine.

Two mistakes recur across teams building these systems: scoring only WER and treating a high-accuracy transcript as proof of a successful agent, or scoring only end-task completion and watching a dashboard that gives no signal about where to look when something breaks. The corrective is to keep all three layers running simultaneously and route each one to the decision point it actually serves.

What production traces capture beyond test logs

Pipeline-stage attribution requires a data structure rich enough to support it, and that structure is the production trace. A trace for a single voice call is built from one span per call leg, speech-to-text, the LLM's reasoning, each tool call, retrieval steps, and text-to-speech, with each span carrying its own attributes (model name, retrieved chunks, tool parameters, audio duration) alongside attributes that cut across the whole call (conversation ID, agent version, channel, turn index). A test log built from a curated script has no equivalent of this structure, because it never had to.

What a test log cannot contain is the thing that actually determines whether ASR produces a usable transcript: the raw audio stream, the background noise present on the call, the codec artifacts introduced by the telephony path, the caller's accent, the exact jitter conditions at the moment of speech.

Voice failures resist reproduction in the way text failures do not, because the same speaker saying the same sentence twice can produce two different STT outputs purely from differing background noise or network jitter between attempts. A failure in a text-based agent can usually be reproduced by replaying the same prompt. A failure in a voice agent often cannot be reproduced from logs alone, because the audio conditions that caused it were never preserved.

A monitoring stack built to account for this needs several things working together: one trace per call with child spans for STT, LLM, tool calls, retrieval, and TTS; eval scores joined directly to spans so a low rubric score points at the exact turn that triggered it; SLO tracking across turn latency, WER, intent confidence, and barge-in failure rate; failure clustering so that calls sharing a root cause appear as a single named issue rather than a flood of unrelated alerts; and session replay that shows audio, transcript, and the span tree side by side. If you skip stage-level attribution built on this kind of trace, every fix stays a guess, so production-trace-driven platforms like Moda focus on pinpointing the exact pipeline layer responsible for a given failure. When a voice agent breaks in production, the useful signal is not an aggregate score but a direct answer to whether the failure originated in ASR, the LLM's reasoning, tool execution, or TTS.

The specific failure categories that only appear in production traces

Three failure categories recur in production voice traces, and each one carries a distinct fingerprint in the trace data and needs a different fix. Synthetic test suites reliably miss all three.

Tool schema drift is the most dangerous of the three because it does not necessarily produce an immediate, visible failure. A tool can return data that fails to conform to the contract the agent was built against, wrong field types, missing required keys, invalid JSON, a truncated payload, and the agent can proceed as though nothing went wrong. In the trace, a tool call span returns a value, followed by an agent turn that proceeds as though the call succeeded, with no error span anywhere in the sequence.

Prompt ambiguity causes premature termination or divergence between the agent's plan and its execution. In the trace, turn-level scores can look entirely acceptable, but conversation-coherence scoring can still reveal that the agent quietly abandoned its own goal list partway through the call.

Workflow loops appear when the agent loses track of its original task objective and falls into repetitive, unproductive replanning. The trace fingerprint here is repeated identical tool calls, no measurable progress against the goal list across turns, and a turn count that keeps climbing without the call ever resolving.

Voice carries two amplifiers of these failures with no equivalent in text-based agents: premature turn cutoff and false barge-in. Both are turn-detection errors that end the caller's utterance too early, feeding a truncated transcript into the LLM and triggering downstream failures across all three categories above.

Root-cause attribution: turning a failed trace into a specific fix

Knowing that a call failed is not the same as knowing why, and the two are easy to confuse because voice agent failures cascade. Every stage downstream of ASR performed its job correctly. Only ASR failed, and yet the failure is only visible once the full outcome plays out at the end of the call.

This is why attribution has to treat the decisive error as a reasoning problem rather than a search for the first anomaly in the trace. You want the failure whose downstream impact best explains the degraded outcome, even if it is not the first thing that looks unusual. Fixing the symptom, a tool call that fired with the wrong parameter, while the actual root cause, a transcription error upstream, goes untouched will not improve the agent on the next call with the same acoustic conditions.

A closed debugging loop for agents follows a consistent sequence: detect the step-level error, attribute its root cause to a specific layer, propose ranked fixes targeted at that layer, then rerun the trajectory around the failure point to confirm the fix. Moda's approach follows this same sequence from the trace side: it ingests production traces, attributes each failure to the specific harness layer responsible, prompt, tool, workflow, memory, model, or product logic, and generates reviewable fixes validated against historical traces before anything ships.

Running evals continuously against live call traffic rather than periodically against a fixed test set

A fixed test suite loses relevance the moment production traffic moves past it, because caller behavior, tool schemas, and the agent's own prompts keep changing while the test set sits still.

A closed loop prevents this kind of drift: offline evaluation feeds the CI gate, the CI gate feeds production trace evaluation, trace evaluation surfaces recurring failures, those failures feed targeted improvements, and the improvements ship back through CI. Treated as a static endpoint, an eval suite lets an agent drift away from its own test coverage. Treated as a loop, the same evaluation process compounds, catching problems earlier each cycle.

A practical sampling approach makes continuous monitoring workable at scale rather than overwhelming: full eval-suite coverage runs against a meaningful fraction of calls, lightweight anomaly detection (latency spikes, early hang-ups, escalation triggers) runs across all calls, and full evaluation plus human review gets reserved for whatever gets flagged. Specific SLO thresholds give this process a trigger rather than leaving it to judgment calls: p95 turn latency past 1000ms, STT confidence below 90%, task success rate below 80%, each one a signal that should kick off a trace review cycle. Failure clustering is what keeps this actionable: fifty broken calls sharing one root cause need to appear as a single named issue pointing at a specific pipeline layer, not fifty separate alerts competing for attention. This discipline applies most directly to the harness components themselves, prompts, tool schemas, workflow definitions, memory configuration, because these are the layers that drift as production conditions shift, while the underlying model typically does not.

Validating harness changes against historical traces before shipping

Diagnosing a failure and fixing it are two different problems, and the gap between them is where regressions get introduced. Shipping a revised prompt, a corrected tool schema, or a modified workflow without replaying it against the historical traces that revealed the original failure repeats the same blind-shipping pattern that caused the problem.

This matters more acutely for voice than for text because of the non-determinism built into the audio path. Replay for a voice agent means you run the updated harness against recorded tool outputs and sandboxed side effects while you preserve the original conversational state, so the system sees the same inputs production actually encountered, not a reconstructed approximation of them.

Roark, a production call replay platform from YC's W25 batch, captures real production calls and lets teams replay them against updated agent logic, including cloning the original caller's voice for realistic testing; it has processed over 10 million minutes of calls, and its core workflow takes production failures and feeds them directly into test suites. The research benchmark MTVA-BENCH (arXiv:2609.20152) operationalizes a related form of component-isolated replay: it holds the model's production interface fixed and simulates everything surrounding it, letting engineers isolate LLM behavior from the variance introduced by ASR and TTS. Both apply the same underlying principle, isolating one variable at a time, which is what makes replay trustworthy as a validation method.

Once pipeline-stage attribution has identified which layer broke, the next step is confirming that a proposed fix, a prompt adjustment, a tool retry policy, a turn-detection tweak, resolves the failure against real production calls. The full loop, capture the trace, attribute the failure to its layer, generate a reviewable fix, validate it against history, then ship, is what turns production-trace evaluation from a diagnostic exercise into an engineering discipline.