Evaluating Web and Browser Agents on Real Navigation Traces

A web agent that scores well on a static benchmark can still fail on most of what the live web actually throws at it. Task-level success metrics treat navigation as a single pass/fail event, and in doing so they hide the specific step where production reliability is won or lost. BrowserArena, introduced by Anupam et al. in 2025, states the problem directly: current agent evaluations are constrained to sandboxed environments or artificial tasks, and the open web is not represented. StressWeb, from Bai et al. in 2026, reaches the same conclusion from a different direction, noting that existing evaluations are predominantly conducted under relatively stable and well-behaved interaction conditions, which may overestimate agent robustness. Both findings point to a structural gap between what benchmarks promise and what deployed agents experience.

The consequence is practical and expensive. An agent that performs well on a curated benchmark fails routinely in production, and no task-level metric surfaces where or why. The deeper structural issue is that task-level scores collapse everything: perception, action selection, execution, and recovery all fold into a single binary outcome. A wrong click and a wrong recovery strategy look identical in the final score. It checks if the task eventually succeeded, not if the agent navigated competently along the way.

Three failure categories that only appear at the step level in real navigation traces

When traces from real web navigation are analyzed at the step level, three failure categories appear consistently. Task-level scoring cannot see any of them because each one occurs mid-trajectory, and the trajectory still has a chance to recover or fail in ways that look identical from the outside. The three categories are environmental interruptions, semantic drift in interaction rules, and execution disruptions.

Environmental interruptions are the most visible: captchas, pop-up modals, and unexpected URL navigations that appear in the middle of a trajectory. The mechanism matters more than the symptom. These interruptions break the agent's assumed page state, so a task that was succeeding fails not because the agent's reasoning was wrong but because the environment interjected. This is a harness-layer failure: the agent's tool interactions and navigation logic do not account for mid-session state changes. BrowserArena's step-level annotations surfaced exactly these modes, with captcha resolution, pop-up banner removal, and direct URL navigation appearing as consistent failures across evaluated models.

Semantic perturbations are subtler and, in StressWeb's findings, more damaging. StressWeb introduces structured perturbations across perception, action selection, and execution stages. Among these, semantic perturbations, those that alter action semantics under explicit or implicit cues, cause the most severe performance degradation of any perturbation type. The mechanism is that agents apply rigid interaction priors when interface semantics change. The workflow logic encoded in a harness assumes a stable action-to-outcome mapping, but the live web does not guarantee one. StressWeb also finds that environmental stress significantly impairs agents' ability to correctly assess their own task success, so self-reported completion becomes unreliable precisely when the environment is unstable.

Execution disruptions complete the taxonomy. StressWeb operationalizes execution perturbations that induce action failures or disruptive interruptions, modeling asynchronous responses, delayed or failed updates, and unexpected pop-up interruptions of the kind real interfaces regularly produce. The mechanism is again a harness gap: agents fail to recover from disrupted state transitions because their workflow logic does not include recovery branches for these cases.

All three categories share a common structure, and that structure is the point. Each failure is a harness-layer gap, in navigation logic, tool behavior, or workflow recovery, not a deficit in the underlying model's knowledge. Evaluating whether a model knows how to click a button is a different question from evaluating whether the harness around that model can survive a captcha, a changed button label, or a delayed page render. Production reliability depends overwhelmingly on the second question, and task-level scores never ask it.

Why cascading errors make single-step failures hard to attribute without trace structure

Identifying that a failure belongs to one of these three categories is necessary but not sufficient. The step where a failure surfaces is often several steps away from the step where it originated. Web agent architectures that integrate planning, memory, and tool use amplify a single root-cause error into a cascade of downstream failures, and by the time the task fails, the trajectory has moved well past the cause.

The mechanism is structural. A single mis-step, a wrong DOM element selected, a modal left undismissed, a URL navigated incorrectly, alters the context for every subsequent planning decision. Each step's output becomes the next step's input, so one bad observation propagates forward and distorts an entire chain of decisions. This is not a model behavior problem in isolation. It is a structural consequence of multi-step tool-use architectures, and it means that final-state inspection can only ever show the terminal symptom, never the root cause.

Existing observability approaches that replay execution traces provide little support for identifying the root cause or translating diagnosis into recovery. Replay alone is not attribution. A team reviewing a failed trace by reading the final output, or even the last few visible steps, will misattribute the failure and apply a fix at the wrong layer. Replay does not solve the problem. What teams need is a systematic way to walk backward from the final failure state to identify which step introduced the error and which harness layer to fix. Production-trace-driven diagnostic tools address this by reconstructing the causal chain and pinpointing where the cascade began, so fixes can target the actual root cause rather than downstream symptoms. Moda's approach to ingesting production traces follows this pattern, attributing failures to the layer that originated them rather than the layer where they became visible.

Security failures compound the attribution difficulty in a specific way. Mudryi et al. (2025) demonstrate that untrusted web content you encounter during navigation can hijack agent behavior through prompt injection. A failure mode of this kind originates in the perception layer but propagates through planning and execution, so its first visible symptom may appear as an incorrect tool call or a garbled output several steps later. Attribution that only examines the final state cannot distinguish a prompt injection from a planning error, and the fix for each is entirely different.

What step-level trace analysis actually requires from an evaluation harness

Evaluating web agents at the step level requires a harness that captures the full trajectory as a structured artifact, not a single outcome span. Each browser action must be independently inspectable and independently scorable, which means the harness has to record more than just the final answer. A complete replay trace for a web agent should capture, at minimum, user input, system prompt, retrieved context, tool calls and arguments, including click, type, scroll, navigate, and screenshot, tool responses or DOM snapshots, and final output. That set of fields is what makes it possible to ask, of any given step, whether the action taken was the right one given the state at that moment. The deeper structural issue is that task-level scores collapse everything into a single pass/fail, hiding the step where failure actually originates. Meaningful web agent evaluation instead requires step-level trace analysis that surfaces harness-layer failures, environmental interruptions, semantic drift, and execution disruptions, before they compound into task failure, enabling teams to validate prompt edits, tool tweaks, and workflow changes against real navigation traces rather than static benchmarks. Tools like Moda automate this by ingesting production navigation traces and surfacing the specific harness layer, tool interactions, navigation logic, workflow recovery, responsible for each failure, rather than leaving attribution to manual review of unstructured logs.

Scoring must operate at multiple levels simultaneously. Task success asks whether the agent completed the goal. Tool correctness asks if the right tool was selected, with the right arguments, in the right sequence. Step efficiency asks how many steps were taken, how many retries occurred, and how many navigations were unnecessary. Self-assessment accuracy asks whether the agent correctly judged its own success, a failure mode StressWeb identifies as degrading under environmental stress. These are not interchangeable metrics. An agent can complete a task while systematically selecting the wrong tool, and a harness can appear healthy while its agent is confidently wrong about its own performance.

BrowserArena's methodology demonstrates step-level evaluation concretely. In that platform, users flag specific agent steps as correct or incorrect relative to the step's LLM-generated goal and the user's expectations. This is step-level human feedback against the trace itself, not a final preference vote between two completed trajectories. The difference is the difference between grading an exam and auditing the work shown.

The harness must also make tool calls deterministic for replay. APIs evolve, DOM states change, backend services deploy new versions. Without recording and replaying tool outputs, a replay is not a faithful re-execution of the original trace. It is a new execution against a different environment. Wrapping each browser action in an observable span, treating the trajectory as a queryable tree rather than a flat log, enables filtering by failure category, by agent step, and by the layer responsible. That structure is what turns a trace from a record into an instrument.

How to attribute a web agent failure to the specific harness layer that caused it

Root-cause attribution for web agent failures requires distinguishing which layer originated the failure: prompt, tool schema, navigation workflow, memory, or execution logic. The same visible symptom can arise from different causes, and each cause requires a different fix. A wrong click might mean the prompt was ambiguous about the goal, the tool schema misdescribed the action, or the navigation workflow had no branch for the pop-up that appeared. Treating all three as "the agent failed" produces unfalsifiable fixes.

The attribution challenge is not just about finding the bad step. It is about separating evidence-grounded reasoning from keyword matching and tracing the propagation path through the call topology. The three failure categories from BrowserArena and StressWeb map to distinct harness layers. Captcha and modal interruptions point to navigation workflow and tool-use logic, because the harness has no recovery branch for mid-session state changes. Semantic perturbations point to prompt and action-selection logic, because the agent's action priors are hard-coded to a stable interaction model. Execution disruptions point to retry policy and state-management logic, because the harness does not detect or handle delayed state transitions.

A team that cannot point to prompt ambiguity versus tool schema drift versus missing retry logic will iterate without convergence. Identifying these step-level failures requires traces captured with sufficient structure and then analyzed systematically. Moda approaches this by ingesting production navigation traces and attributing each failure to the specific layer responsible, whether prompt, tool, workflow, memory, model, or product logic, and generating reviewable improvements validated against historical traces before recommending what to ship. The path from a broken trace to a verified fix becomes traceable and auditable, which is the property that separates diagnosis from guesswork.

Turning attributed failures into harness changes that can be validated before shipping

Once a failure is attributed to a specific harness layer, the fix is a software engineering change: new tool logic, a tighter navigation workflow, a revised prompt structure, or an extended retry policy. And like any software change, it must be validated against historical traces before deployment to confirm that it resolves the failure without introducing regressions. Harness iteration keeps model weights fixed. All reliability gains in this paradigm come from how the scaffold mediates between the model and the live web environment. Vercel's experience illustrates the point concretely: removing the majority of their agent's tools produced better results, with fewer tools meaning fewer steps, fewer tokens, faster responses, and higher task success. That is a reliability improvement achieved entirely through subtraction of harness complexity, not through any change to the model itself.

Replay-based validation is what separates a confident ship from shipping blind. Replaying the modified harness against recorded traces with deterministic tool outputs lets a team verify that a specific failure mode no longer occurs and that the change does not break previously passing trajectories. Shipping changes that have not been validated against real traces means the team cannot distinguish a real fix from a coincidental pass on the narrow scenario they happened to test.

The improvement cycle is repeatable and continuous. Live traffic generates new traces. New failure patterns surface. Attribution identifies the layer. A fix is proposed, validated against historical traces, and shipped. The point is not that a team fixes one bug. It is that the team has a systematic process for turning production failure into verified improvement. This is harness engineering treated as a first-class engineering discipline, not a one-time tuning exercise. Moda closes this loop end-to-end for web and browser agent teams, surfacing failures from production traces, attributing them to the responsible layer, generating concrete reviewable improvements, and validating them against historical traces before recommending what to ship. The teams that treat traces as the raw material of reliability engineering, rather than as diagnostics for one-off debugging, are the ones whose agents survive contact with the open web.