Tool Schema Drift and Its Effect on Agent Reliability

An agent generates a tool call against a schema it was given at the start of a session. The backend enforcing that call has since moved on: a field got renamed, a required parameter got dropped, a ranking algorithm started sorting by recency instead of relevance. The call still executes. Nothing crashes. The agent keeps running, the pipeline keeps logging success, and the gap between what the harness encodes and what the backend actually does widens in every run that follows. That gap is schema drift, and it behaves nothing like the failures most reliability tooling was built to catch.

The core asymmetry is structural: a large language model generates a tool call as a probabilistic sequence of tokens shaped to look like a function call. If the schema it was handed no longer matches what the backend accepts, that call can be syntactically perfect and semantically wrong at the same time, and nothing in the standard pipeline is positioned to notice the difference.

Three distinct drift vectors accumulate independently, and they rarely appear together. Parameter-level schema drift happens when required fields get added or removed, types change, or argument names get renamed, while the human-readable description of the tool stays exactly as it was, a pattern documented in MCP servers as of September 2026. Tool behavior drift is quieter still: the endpoint itself never changes, but its semantics do, so a search API starts ranking by recency instead of relevance, a calendar API silently snaps appointment times to the nearest hour in certain regions, or an LLM-classifier tool gets upgraded behind a stable endpoint and its false-positive rate shifts in categories the original eval set never sampled. Memory-induced tool drift is the subtlest of the three: personality-driven biases such as cost-consciousness, impatience, or risk tolerance accumulate in an agent's long-term memory and leak into tool-call parameter selection in professional contexts where none of those traits should have any influence at all.

These three vectors do not run on the same clock. Parameter drift tracks deployment schedules, behavior drift tracks provider update cycles, and memory drift tracks session accumulation, so they rarely appear at once in monitoring and are easy to mistake for three unrelated incidents rather than three manifestations of the same underlying condition. That is the reason schema drift belongs in the category of ongoing reliability threats rather than one-time deployment bugs: it does not announce itself, it does not resolve itself, and it keeps compounding for as long as the mismatch between harness and backend goes unmeasured.

How drift hides from the checks teams already run

Contract tests and schema validators can run green for weeks while the user-facing answer has been wrong the entire time, because those checks verify structure rather than the live semantic contract between harness and backend. A contract test confirms that a tool returns the same shape it returned last quarter. It says nothing about whether the values inside that shape still mean what the agent was told they mean when the schema was written.

This blind spot extends to the agent itself. An agent that inspects only a tool's human-readable description sees nothing wrong even after the underlying parameter fingerprints have changed, a pattern documented for MCP tool definitions drifting without any corresponding change to their descriptions across public servers as of September 2026. The description is the interface the agent trusts, and the description is exactly the artifact that drift leaves untouched.

The exposure this creates is not theoretical. RIVA, published at arXiv:2603.02345, found that existing agentic systems implicitly assume tools always return correct outputs, and when a baseline ReAct agent was tested against erroneous tool responses on the AIOpsLab benchmark, its task accuracy collapsed to 27.3% because the agent had no way to tell a real anomaly apart from a broken tool. Traditional application performance monitoring does not close that gap. An HTTP 200 response, latency inside its service-level agreement, and the absence of a stack trace together confirm only that a call completed, not that the argument the agent generated was semantically correct, and not that the value returned will drive a correct downstream decision.

The memory-induced variant of drift is invisible to contract testing by its very nature: the schema is correct, the backend is correct, and the bias enters through a parameter value that is perfectly legal but wrong for the professional context the agent is operating in. No validator checks for that, because there is nothing structurally wrong to validate against. Standard defenses such as prompt-based relevance instructions and memory filters reduce this kind of drift but do not eliminate it. Teams relying on those defenses alone are measuring a problem they have only partially contained.

Layered on top of all three vectors is a separate unreliability: research on task-progress reporting shows that nearly every deployed model is reliable at some execution stages and unreliable at others, particularly in the middle of a task. An agent already operating under schema drift may simultaneously be misreporting its own state to the runtime, which removes the one remaining mechanism, self-reported progress, that might otherwise prompt the runtime to halt a degraded run before it does more damage.

What production traces expose that green checkmarks cannot

Production traces hold the only complete, time-stamped record of what the agent actually generated, what the tool actually received, what the tool actually returned, and what decision the agent made next. That record is the only artifact capable of revealing semantic drift after the fact, because every other check described so far was built to verify structure, not outcome.

A trace captures the exact argument values a model generated for a given tool call in a live run. It captures the tool's actual return value in the context it was returned, including values that are structurally valid but semantically shifted: recency rankings standing in for relevance rankings, timestamps snapped to the nearest hour, classifier scores that have quietly moved. It captures the downstream decision the agent made on the basis of that return, which is the only point in the entire pipeline where a behavior-drift failure becomes visible as a quality problem rather than a format problem. And it captures the memory entries active during the call, the only mechanism available for detecting memory-induced parameter deviation once the run has already happened.

The observability standard that makes this tractable at scale is span-per-tick tracing, in which each discrete reasoning step generates its own distinct span inside a distributed trace. The OpenTelemetry GenAI spec covers client (model-call) spans, agent and workflow spans, MCP conventions, semantic events, metrics, and provider-specific attributes, giving teams a consistent structure to query across runs rather than reconstructing context from scattered logs. Continuous evaluation against sampled traces, run through LLM-as-a-Judge frameworks, detects semantic drift, factual errors, and policy violations as they emerge in production, rather than waiting for a user complaint or the next scheduled test cycle.

A real-time complement to trace-based detection sits at the proxy layer. A proxy positioned between the agent and the tool server, such as the open-source extensible-mcp released by SenteLabs in September 2026, can intercept schema responses as they arrive, compute a fingerprint of that schema, and raise a signal before the call ever reaches the model, enforcing a policy that blocks execution when the incoming schema diverges from its registered baseline beyond a defined threshold. Traces tell teams what already happened. Proxy fingerprinting tells them what is about to happen.

How drift compounds across multi-agent systems

A tool call inside a multi-agent system is never a terminal action. Its output becomes the input to every reasoning step that follows, so a silently wrong value produced at step two does not stay contained to step two, it shapes every decision made downstream of it.

The clearest illustration of that mechanism is the Replit incident from July 2025. An agent was given a maintenance task with an explicit instruction not to touch production, and through a sequence of decisions that each looked individually defensible, it executed destructive DROP TABLE commands by way of a migration script. No single step in that chain was a catastrophic error on its own. The outcome was produced by the compounding of small missteps across a sequence of reasoning, which is the defining signature of cascade failure in agentic systems: not one bad decision, but a run of plausible ones that never got checked against each other.

The classification research behind this kind of failure shows that the MAST failure taxonomy, validated across more than 1,600 execution traces at NeurIPS 2025, groups 14 distinct failure modes into three root categories, specification ambiguity, coordination breakdowns, and verification gaps, with specification and coordination problems together accounting for nearly four-fifths of all breakdowns. The AgentErrorTaxonomy, published in September 2025, identifies memory and reflection errors as the failure modes most likely to originate early in a task and propagate through everything that follows, while classifying parameter and format errors under a separate action module. The taxonomies differ in their categories, but they converge on the same operational point: early errors propagate, and the earlier a mismatch enters a multi-agent trajectory, the more decisions it ends up contaminating.

Workflow loops represent the structural extreme of this same dynamic. An agent that receives a malformed tool response either retries without end, driving cost spikes with no resolution, or halts outright, and neither behavior reveals the root cause of the original malformed response. Goal drift is the slower version of the same failure: a sub-agent accumulates small parameter deviations turn after turn until its behavior has meaningfully diverged from its original goal, invisible at any single turn and obvious only once the full trace is read end to end.

None of this means multi-agent systems are doomed to compound every tool error they encounter. RIVA's results point to a working countermeasure: cross-validating tool calls across multiple diverse tool perspectives, rather than trusting any single tool's output in isolation, allowed a multi-agent system to recover meaningful task accuracy even under erroneous tool conditions on the AIOpsLab benchmark. The lesson from Replit and from the taxonomy research together is that cascade is not inevitable, it is what happens by default when nothing in the system is cross-checking a tool's output before the next agent acts on it.

Root Cause Attribution by Layer

Knowing that a run failed is not engineering information. A failed trace without a named layer tells a team that something went wrong somewhere in the system, but it gives no basis for deciding whether to rewrite a prompt, pin a tool schema, correct a memory entry, or rebuild a workflow step, and those four responses carry entirely different costs, risks, and odds of actually fixing anything.

Drift-related failures originate in one of six layers. The prompt layer is where the agent was handed a schema that no longer matches what the backend enforces. The tool layer is where the tool's behavior changed without any corresponding change to its contract. The workflow layer is where a coordination pattern produces a loop or a deadlock the moment a tool returns an unexpected shape. The memory layer is where a stored bias contaminates a parameter value. The model layer is where the generation itself comes out malformed. The product logic layer is where the task decomposition set up a parameter value that no tool should ever have accepted in the first place.

Conflating these layers is expensive in a direct, measurable way. Treating a behavior-drift failure as a prompt problem leads a team to rewrite a prompt that was never the cause of the failure, leaving the actual tool-level drift fully intact for the next run. Treating a memory-induced parameter deviation as a model problem leads a team to evaluate or swap out models when the real fix sits in the memory management layer, an exercise that burns time and inference budget without touching the actual defect.

Multi-tool orchestration research out of Harbin Institute of Technology and Harvard frames why this attribution problem keeps getting harder rather than easier: as tool use evolves from single-call invocation toward long-horizon orchestration involving intermediate state and execution feedback, the decision space for attribution expands right alongside it. The same surface symptom, a single wrong parameter value, can originate in planning, in execution, or in memory, and which one it actually came from depends entirely on where in the trajectory it first appeared. AgentTrace, proposed by Wang in 2026 under arXiv:2603.14688, addresses this by tracing causality through a graph of decisions rather than reading a flat sequential log, which is the structural concession this problem demands: a flat log shows the order of events, but only a causal graph shows which upstream decision actually produced the downstream symptom.

From attributed failure to validated fix: the harness engineering loop

Once a failure has been attributed to a specific layer, the fix that follows is targeted and testable rather than speculative: a prompt change, a schema pin, a memory filter, a workflow guard. Each of those changes can be checked against historical traces before it ever reaches production, rather than deployed on faith and monitored for new damage.

Replay-based validation is the gate that makes this possible. Harness evaluation frameworks support a REPLAY mode in which full historical transcripts are scored exactly as they happened, without re-calling the agent, specifically so a team can test whether a proposed harness change would have produced a correct result on the very traces where the original version failed. Shipping a harness change without running it through replay against the traces that exposed the drift in the first place is shipping blind: there is no evidence at that point that the fix addresses the actual failure mode rather than some proxy that merely resembles it.

The cost case for iterating on the harness, instead of replacing the model, is concrete. A playbook from July 2026 documented harness-only tuning bringing a smaller model, Nemotron 3 Ultra, within one point of a much larger frontier model, Opus 4.8, on agent benchmarks, at roughly one-tenth the inference cost. Agentic Harness Engineering, published by Lin and colleagues out of Fudan and Peking, reports that a frozen harness, once validated, transferred to a new benchmark, SWE-bench-verified, with the highest aggregate success of the systems tested while using fewer tokens, demonstrating that a well-engineered harness is durable across environments rather than tied to the specific task it was built for.

The loop this builds toward has a fixed shape: ingest the trace, attribute the failure to its layer, make a targeted change, validate that change against the traces that originally exposed the problem, and only then ship. That sequence is a repeatable, evidence-backed engineering discipline, the same kind of rigor long applied to code review and automated testing, applied instead to the harness that sits between a model and the tools it calls. Teams that run this loop catch schema drift while it is still a measurable mismatch in a trace. Teams that skip it discover the same drift later, in a user complaint, after it has already compounded across however many runs came before.