Tool Call Return Signatures as a Production Signal Class in Multi-Agent Learning Loops

Silent tool failure detection in production multi-agent pipelines requires capturing tool call return signatures, including raw output, schema shape, field values, latency, and error codes. These records show whether the model reasoned poorly or the tool returned a bad result. Treating them as a first-class production signal, with explicit tool-layer attribution and replay validation before fixes ship, turns harness improvement into evidence-based engineering.

Why tool call return signatures are a different class of production signal

A return signature is the structured record of what a tool invocation returned. It combines the returned payload with its schema shape, field values, latency, and error codes. The payload supplies content, the shape establishes its contract, and timing and error information record how the invocation completed. Together, they make the tool response available for comparison across calls and runs.

That record occupies a distinct place in agent observability because it captures the external system’s response. A reasoning trace can explain why an agent selected a tool. It cannot substitute for what that tool returned. An empty result, for example, may come back without an HTTP error and still explain why the final answer was wrong.

Openlayer’s agent observability guide identifies four domains, namely reasoning traces, tool call behavior, state changes and side effects, and error handling and recovery. Return signatures are the primary record for the latter three, while model traces expose the reasoning side. The guide also distinguishes observation from enforcement. Logging what an agent said is observation, while recording what it changed in the world and blocking unsafe actions before execution moves into enforcement. Return signatures provide evidence at that boundary, although capturing a response does not itself block an action.

How multi-step agent runs turn isolated tool failures into silent cascades

Earlier tool outputs supply the arguments and context for later steps, so an isolated failure can become a dependency failure. A run that calls six tools across three model invocations produces many spans, nearly all of which may show normal status codes, normal latency, and unremarkable token counts. The decisive defect can sit in one span. Later reasoning may work as usual with that bad result, so the final answer may be the first clear sign that something went wrong.

Multi-agent systems add handoff boundaries where a malformed or semantically wrong return value can be accepted and passed forward. The MAST failure taxonomy groups 14 failure modes under three root categories, which are specification issues, inter-agent misalignment, and task verification failures. Its validation covered more than 1,600 execution traces. These categories extend beyond individual tool errors, but the return record provides evidence for diagnosing how the execution departed from the intended task.

The consequences can extend beyond wrong answers. The Replit incident in July 2025 involved production database deletion during an explicit code freeze. The PocketOS/Cursor/Claude incident in April 2026 involved deletion of a production database volume and its backups in nine seconds through a root-scoped Railway token found in an unrelated file. In both cases, the failure chain was not interrupted at the tool layer. The invocation record was the earliest place to inspect the external action.

This makes threading across agent hops as important as span nesting. A bad output on turn five may trace back to context established on turn two, which isolated run records obscure. The next diagnostic question concerns the signatures themselves. Which patterns show that a tool call that seemed to succeed has sent the workflow off course?

How do you know a tool call failed silently? Three return-signature failure modes

Schema drift occurs when a field-level contract changes but the execution keeps going. If a tool argument changes from customer_id to account_id, a runtime adapter may silently map the missing field to null. A retry layer can interpret the result as a temporary failure and resubmit, while a fallback handler switches tools. Each response adds latency without exposing the original mismatch. Happy-path staging evaluations can miss this chain because they do not exercise the adapter’s null-mapping behavior.

Behavior drift preserves the schema but changes its meaning. A search endpoint may rank by recency instead of relevance, or an LLM classifier used as a tool may change behind a stable endpoint and shift its false-positive rate. The response can still be HTTP 200. Its fields are populated and structurally valid, but their semantic content no longer supports the assumptions made by the agent. Smaller models are particularly poor at detecting this distinction because they trust the structure and reason on top of it.

Stuck-loop signatures expose repeated activity without progress. When an agent repeatedly calls the same tool, the return-signature hash sequence makes that repetition observable across the run. Non-progress calls show a strong recency pattern in tool selection, with most using tools from recent retrieval windows. An instrumentation response is to emit stuck.signature_hash when the same tool call repeats, making the pattern available for alerting.

These failures need different fixes, even when their HTTP outcomes look alike. Separating them requires the full return signature, including field values and latency. That distinction is the starting point for assigning responsibility to the correct component.

Why teams attribute tool-layer failures to the model and how to stop

When output quality regresses without a return-signature record, the model is the most visible suspect. Its answer is available, while the external evidence on which it relied is missing. Rolling back a prompt, comparing model versions, and rerunning evaluations are reasonable responses to some regressions, but they cannot repair a tool contract that silently changed.

An outcome-level signal establishes that an execution failed while leaving attribution to the model, harness, environment, or grader unresolved. Those distinctions determine where an intervention belongs. Harness failures guide harness design, while environment issues may require benchmark repair. An investigator needs the tool’s actual response to tell reliably whether the reasoning was wrong or was based on wrong input.

OpenRCA benchmark results show that hallucinated data interpretation and incomplete exploration persist across model capability tiers. That pattern points to the shared agent framework. It also explains why a stronger model is not automatically the relevant remedy. Bad tool schemas, missing retries, and unstructured context remain harness defects even when their visible symptom is an implausible final answer.

The lesson for a team investigating that symptom is to connect the reasoning chain to the return signature before choosing a fix. If the tool returned a valid result and the model misread it, the evidence supports one intervention. If the tool returned empty or semantically shifted data, it supports another. Explicit attribution prevents an output-quality investigation from becoming a sequence of unrelated prompt experiments.

What a return-signature-aware trace schema actually captures

In implementation, each tool invocation should have a structured record containing the tool name and version, exact input arguments, raw output, latency, and success or failure state. Schema shape, field values, and error codes must be kept so teams can find faults. The unit is the invocation. This makes it possible to inspect what entered the tool, what came back, and how that response entered the next step.

The established mental model is a distributed trace, with one trace per run and a span for each planning step, LLM call, tool invocation, and retrieval step. Stable IDs should thread through every hop. OpenTelemetry’s GenAI semantic conventions include explicit agent spans, events, and Model Context Protocol context propagation guidance in the dedicated open-telemetry/semantic-conventions-genai repository. These conventions remain in Development status as of 2026. Aligning fields with gen_ai.* and mcp.* provides a common direction without treating the conventions as stable.

Five layers need distinct instrumentation within that trace. The request layer records the goal, session, and agent IDs, while the reasoning and planning layer records tool selection, retry count, and stop reason. The tool layer records name, input schema, latency, and raw output. The system layer records token usage, API error rate, and cost per task, while the outcome layer records the artifact, quality rating, and whether correction was needed. A shared trace ID must cross agent-to-agent handoffs so these records remain connected.

Retention and security need equally deliberate treatment. Head-based sampling should be supplemented by tail sampling triggered by failures, retries, high cost, and stuck-loop signatures. Tool I/O logging should be allowlist-first, with redaction in both the SDK and collector, because payloads can contain credentials, customer data, and compliance-sensitive content. Full diagnostic capture does not mean unrestricted payload retention. The logging pipeline can itself become a data exfiltration channel.

Using return signatures as the input to replay-based validation before shipping harness changes

Once capture is in place, the next step is to use return signatures as the ground-truth record for testing proposed harness changes. Diagnosis establishes what went wrong in a recorded run. Replay tests whether a targeted change addresses it. Live evaluation checks for drift in the environment where production calls run now. Neither replaces the other.

AgentX operationalizes this distinction through a prompt optimization process, which turns accumulated execution traces into local subagent prompt updates and admits them solely through paired replay. Each evolution run edits only the target subagent’s instructions, validation rules, output contract, and tool-use discipline. The scope of the edit stays narrow. That constraint makes a score change attributable to the local intervention.

A recorded HTTP 200 response from an API that has since been migrated away from can keep a replay passing while production behavior changes. The stub succeeds, but its premise has expired. Versioning the trace with the tool schema hash makes that mismatch detectable. Behavioral drift still needs live evaluation because content can change even when the schema stays the same.

Golden tasks should therefore use smoke, core, and torture tiers built around observed production failures, including wrong tool choice, unsafe actions, and retry loops. Every important replay action should produce a receipt recording what the tool returned, what state changed, and whether the outcome matched the expected contract. Those receipts connect the proposed repair to the attributed failure, making the change reviewable and auditable before it ships.

Building a continuous learning loop around return signature patterns

As return signatures build up, they show which failure patterns keep coming back across incidents. MAST was built through this kind of trace analysis and presented at NeurIPS 2025. A tool description that repeatedly produces malformed arguments reveals a specific intervention in the argument record, whereas an aggregate model score only reports degraded performance.

Changes to tool descriptions and parameter schemas can have a large effect. Models can choose the wrong tool or pass malformed arguments because a schema is ambiguous. Rewriting the contract addresses that source directly. The harness becomes a first-class training target, with changes informed by the execution evidence generated by the system that actually ships.

For that loop to remain trustworthy, production and evaluation should share the same harness schema state, including tool versions and their hashes. Divergence creates false confidence in evaluation results. Continuous monitoring should track field-level schema changes by comparing the structure with the versioned schema, semantic drift by checking for changes in field-value distributions, and recency bias by following the signature hash sequence across a run. Each signal preserves the distinction among contract breakage, changed behavior, and non-progress repetition.

Return signatures are the evidence base for meaningful harness improvement decisions because they preserve the external results on which agent behavior depends. Teams that give them first-class status, assign faults explicitly, and validate repairs before shipping are doing harness engineering. Teams that do not are guessing and calling it iteration.