Root Cause Attribution in Multi-Step Agent Runs

An agent that fails on step seven usually broke on step three. That single fact makes root cause attribution in multi-step agent runs a fundamentally different problem than debugging traditional software, and it's why most teams still solve it badly: by re-running the prompt and hoping the failure doesn't repeat.

Traditional software gives you a stack trace. A null pointer exception tells you the exact line, the exact variable, the exact moment things went wrong. Agent pipelines don't offer that courtesy. They run to completion. They produce output. The failure isn't a crash, it's a bad answer wearing the clothes of a good one, and nothing in the system raises a hand to flag it. Worse, the instinct to re-run the same prompt and see if it passes is actively dangerous here, because these systems are non-deterministic. A single passing re-run does not establish that the original failure was a fluke rather than a genuine bug. It just means the dice landed differently the second time.

Now stretch that across five steps. One bad step in a five-step chain still produces output at the end. The user complains that the result was wrong, and the team is left holding five steps and zero error signals, trying to find the one that broke everything.

How bad the attribution problem is, measured by research benchmarks

The numbers here undercut the assumption that this is a solved problem just waiting for someone to build the right dashboard.

On the Who&When benchmark, which covers 184 annotated failure tasks drawn from algorithm-generated and hand-crafted multi-agent systems built on GAIA and AssistantBench queries, the best automated method identifies which agent was responsible for a failure with 53.5% accuracy. Pinpointing the actual failure step, not just the agent, drops to 14.2%. That's the gap between knowing who broke it and knowing where.

A separate benchmark, TraceElephant (Chen et al., arXiv 2604.22708), frames the same problem with even less mercy. On Who&When, the best existing methods achieve 17.1% accuracy for locating the critical error step. On the more complex TRAIL benchmark, the top-performing models achieve a joint accuracy as low as 18.3%, a bar too low for practical use. These aren't weak models struggling with an easy task. Even strong reasoning models fail to clear a bar that would make this practically usable in production. Scaling model capability doesn't fix it, because the problem isn't reasoning capacity, it's visibility into where the reasoning went wrong.

Why failures cascade in a multi-step agent trajectory

A root-cause error in an agent trajectory doesn't stay contained. It propagates. Every downstream step builds on the corrupted state left by the step before it, and it does so with apparent coherence, no exception raised, nothing to suggest a foundation crack happened three moves ago.

The math makes this concrete. Five agents each running at 95% individual accuracy don't compound to 95% overall, they compound multiplicatively to roughly 77%. And that's the optimistic case, before accounting for coordination overhead, context loss between agents, and loops that never terminate cleanly. Adding those in, multi-agent LLM systems fail somewhere between 41% and 86% of the time in production, depending on task complexity. That range is itself a data point, showing that the harness configuration rather than the underlying model does most of the work in holding a system together.

Production observations have settled on seven recurring failure patterns: cascading errors, deadlocks, context drift, infinite loops, silent failures, misalignment, and tool corruption. Each of these originates in a different layer of the system. A single debugging instinct (re-run it, check the prompt, blame the model) can't cover all seven. Attribution requires knowing which layer you're even looking at.

The four harness layers where failures originate

The harness is everything around the model: context and memory management, constraints and guardrails, tool interfaces, orchestration logic, and whatever observability exists to watch it all. When an agent produces a weak result, the harness is almost always the more productive place to look, not the model itself.

Four layers matter here, and each one fails in its own recognizable way.

The prompt layer breaks through ambiguous instructions, unclear tool descriptions, or format drift. Its signature is a high count of self-correction loops, output schemas that shift from one run to the next, and a planner that picks the wrong tool because some input phrase happened to match a different tool's description more closely than the right one.

The tool layer breaks through schema drift, malformed arguments, or semantic drift in how a tool's purpose is described to the model. Its signature is a response that's malformed but plausible enough to pass downstream, undetected until a human notices the output is wrong. Tool calling itself fails somewhere between 3% and 15% of the time in production, depending on model size and task complexity. This layer isn't a rare edge case; it's a baseline cost of doing business with tool-using agents.

The workflow layer breaks through unbounded loops, missing step limits, and retry logic that keeps replaying the same prompt against an error that persists. Its signature is runaway compute: sub-agents that never return, retry storms that amplify the original problem instead of recovering from it.

The memory layer breaks through context drift, window exhaustion, and stale reads. Its signature is the hardest to catch of the four, because the agent keeps reasoning coherently; it's just reasoning coherently from a corrupted context state that it has no way of detecting as corrupted.

Anthropic's framing of harness design is a useful lens here: every component of a harness encodes an assumption about what the model can't do on its own. Attribution, at bottom, is the process of figuring out which assumption turned out to be wrong. That reframing also explains why "the model is broken" is such a common but weak diagnosis. It's the default answer when no trace exists to say otherwise, and it's almost never the correct one, because it forecloses the actual investigation before it starts.

The tool schema drift failure, worked end-to-end

Schema drift sounds abstract until you see how little it takes to trigger. A renamed field. A newly required property. A nested object where a flat key used to be. A changed enum value, a validator that got stricter without announcement. Any one of these, on its own, is enough to break an integration that was working fine the day before.

Upgrading n8n from version 2.4.7 to 2.6.3 caused the platform to generate invalid tool schemas in its tool calls. Upgrading n8n from version 2.4.7 to 2.6.3 caused the platform to generate invalid tool schemas in its tool calls. This broke both OpenAI and Anthropic integrations at the same time, because no mechanism existed to surface the schema change to the harnesses consuming that tool. The integrations didn't know the contract had changed underneath them.

The attribution lesson here is the whole point. The failure showed up at the downstream integration layer, where OpenAI and Anthropic calls started returning bad results. But the originating cause was three steps upstream: a version bump that altered the tool schema. A team without the discipline to trace back through the tool call span would have spent its time debugging the model integration, poking at prompts and API calls, when the actual defect was sitting in a schema definition that changed during a routine upgrade.

Conventional API testing doesn't catch this category of failure. The tool still accepted requests. It still returned responses. Nothing threw a 4xx or a 5xx. The failure lived entirely in the shape and meaning of the payload, in what the industry has started calling semantic drift, and semantic drift doesn't trip the alarms that status-code monitoring is built to catch.

How to read an execution trace for layer-level attribution

A trace worth anything captures tool selection, tool arguments, model responses, memory reads, memory writes, state transitions, and decision branches. Each of these is a potential point of failure, and each one is invisible to the kind of monitoring built for traditional web services.

OpenTelemetry's span architecture for agents gives this structure a set of typed operation spans covering agent creation, invocation, workflow steps, and tool execution. Each span carries fields for the trajectory step, tool name, model prompt and output, retrieval chunks, and a parent-trace ID linking it back to the run it belongs to. That's the raw material. What you do with it is the actual skill.

The FutureAGI workflow lays out a procedural model for using that raw material. Start at the failed trace, identified by user-session ID or task ID, not at the final bad output. Walk the full trajectory of spans, the entire tree. Attach per-step evaluators along the way: TaskCompletion as the aggregate entry point that answers "did this work at all," ToolSelectionAccuracy to identify which specific tool call broke, Groundedness to catch retrieval failures, and ReasoningQuality to catch the planner drifting off course.

Where the evaluator score collapses is where the broken step lives. Once localized, the span's own inputs and outputs tell you which of the four harness layers caused it.

The diff view does the heaviest lifting in this whole process. Comparing a failed trajectory against successful runs of the same task, step by step, reveals in the diff what was structurally different about the run that failed, not just a difference in the final output but a difference in what happened three steps before the output existed.

Counterfactual replay as the standard for confirming a root cause

Localizing a suspect step isn't the same as proving it caused the failure. A low evaluator score at step three tells you something went wrong there. It doesn't tell you that step three caused the final bad outcome rather than merely contributing to it alongside other factors. Correlation across a span tree is not causation, and treating it as such is how teams end up "fixing" steps that weren't actually the problem.

CausalFlow addresses this directly. It's an interventional framework that treats an execution trace as a sequential chain of dependent steps and computes what it calls Causal Responsibility Scores through step-level counterfactual intervention: swap out a candidate step, re-execute everything downstream of it, and check whether the final outcome actually changes. If it does, that step carries causal weight. If it doesn't, the correlation was coincidental.

Tested across multiple benchmarks, CausalFlow converted 42.7% of failed executions into validated, minimal repairs. That's not a small number for a method built on rigorous intervention rather than pattern-matching.

A related approach, Causal Agent Replay, answers the same "which step actually caused this" question by modeling the run as a structural causal model, resampling a single step under the same stochastic policy the agent used originally, and measuring how much the outcome distribution shifts as a result. Both methods share the same premise: attribution by intervention, not by inference from a trace that merely looks suspicious.

Moving from a confirmed root cause to a validated fix

The common mistake at this stage is patching the visible failure surface, the bad final output, rather than the attributed layer three steps back where the actual ambiguity or schema mismatch lived. It's an understandable mistake. The visible failure is right there. The attributed cause takes real tracing work to find. But fixing the surface leaves the underlying assumption unchanged, and the same failure mode will occur again the next time conditions line up the same way.

CausalFlow's approach offers a useful discipline here: generate the minimally edited repair, the smallest change that flips the outcome to success while introducing the least drift into everything else the agent does. Each validated repair produces a contrastive pair, the wrong step next to its corrected version, which can then serve as ongoing supervision material for the system going forward.

AgentDebug's results make the case for why this precision matters. Targeted feedback grounded in isolated root-cause failures achieved 24% higher all-correct accuracy and 17% higher step accuracy than the strongest baseline tested, with task success improving by as much as 26% relative across ALFWorld, GAIA, and WebShop. That gap, between generic feedback and feedback tied to a specific attributed layer, is the entire argument for doing this work properly instead of guessing.

None of it counts until the fix is validated against reality, though. The repaired harness needs to run against the full cohort of historical failed traces, and the fix only earns trust when the eval-fail-rate across that cohort actually drops on real production traces. A single re-run of the corrected prompt passing once proves exactly as little as the original failing re-run did. Given how central non-determinism is to why this problem exists in the first place, that standard isn't optional.