Architecting Trace-Driven Data Pipelines for Agent Improvement

A trace-driven data pipeline for agent improvement is an engineering system. Its value comes from connecting structured execution records to layer-specific diagnosis, reviewable changes, and replay-based validation. Getting those stages right turns evidence of failure into improvements that can be shipped, tested again, and reused when similar failures recur.

Why raw execution logs cannot locate agent failures

When a chained LLM agent returns a wrong answer or loops, the visible failure often appears downstream of its cause. Barrak et al. (2025) demonstrate how errors propagate through a Planner → Executor → Critic pipeline, corrupting later stages before an output is produced. A record of the final failure therefore does not establish which agent introduced it. Flat logs show the order of events, but that order alone does not show who caused the failure.

In the financial workflow example described in the research, an exception surfaces downstream of a malformed tool call that omitted a required argument. Taking the exception as the cause sends diagnosis to the wrong step. Without hierarchy, a slow or expensive run also remains an aggregate observation rather than a cost or latency problem tied to a specific operation. Outcome-level signals indicate failure, but they do not identify the responsible layer.

What an execution trace actually is and how it must be structured

An execution trace is a hierarchical record that mirrors the agent’s causal structure. The top-level span is an agent session or user turn, and reasoning spans record the steps in its planning or deliberation. A tool call span sits beneath the reasoning span that triggered it, and its results are stored as span attributes. The basic structure is a session span containing a reasoning span containing a tool call span. That nesting preserves the connection between a decision and the operation it caused.

OpenTelemetry GenAI semantic conventions provide a vendor-neutral schema for this record, including gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens. OpenInference semantic conventions extend OpenTelemetry with tool call correlation fields such as message.tool_calls and tool_call.*. These fields link each call to the execution context around it, so it is more than a message on its own. Parent-child relationships also make cost and latency attributable to individual pipeline steps. For multi-agent systems, the record must preserve who acted as well as what happened, because tracing an action and assigning responsibility for it are distinct requirements.

Ingestion and sampling: what to keep and what to discard

Once traces have a useful structure, ingestion feeds diagnostic evidence or mostly routine traffic to later stages. Uniform retention is the wrong default because failed and anomalous runs carry different information from successful ones. Tail-based sampling makes the retention decision after a trace completes, when its failure signals are available. Head-based sampling makes that decision too early and discards failures at the same rate as healthy runs. The practical policy is to retain every failed, expensive, or anomalous trace in full while sampling the happy path aggressively.

Failure must mean more than an exception or an unsuccessful terminal status. Invalid tool arguments can lead to a loop or a made-up result, and context truncation can drop key instructions or retrieved documents without warning. Runaway loops consume tokens without converging, and swallowed tool exceptions can leave a trace that appears successful. Sampling policy must preserve these patterns if attribution is to distinguish them later.

A hybrid approach keeps full fidelity in development and staging, then uses tail-based error and latency triggers in production with a modest head-based rate for baseline coverage. This makes sampling an engineering choice about evidence. Privacy constrains that evidence collection. Prompts and completions must be scrubbed automatically in the instrumentation wrapper before spans reach storage, rather than relying on developers to handle each call separately.

Attributing failures to the specific harness layer that caused them

Attribution becomes actionable only when it names the responsible prompt, tool, workflow, or memory component. The large majority of production failures originate in the harness. The SHE framework (2026) makes that distinction operational by separating the harness into System Prompt, Rule Bank, Safety Memory, and Tool Policy artifacts. A diagnosis can then target the smallest responsible component.

Different failure mechanisms require different attribution methods. LLM-as-judge protocols predict the responsible agent and decisive step, making them useful for single-turn or short-trajectory failures such as an incorrect tool decision. Structural causal graphs deal with failures that build on each other across agents and steps, where the final mistake may depend on several earlier decisions. Architectural failures point to how the workflow or agent roles are defined, while execution failures include tool bloat and schema drift in call arguments. Data-layer failures involve stale context, schema drift blindness, or missing business semantics, and are especially prone to being mistaken for limits in model capability.

Retraining cannot repair a harness defect left in place, and prompt revision does not resolve a data-layer schema problem. Barrak et al. (2025) also use repair and harm rates to show how agent-role behavior differs and why steady planning and high-variance critiquing each need their own attribution. TraceFix (Xia et al., CAIS 2026) extends diagnosis to coordination protocols by using TLA+ model-checking counterexamples as debugging evidence for deadlocks and related coordination failures. Specificity matters because the fix can be no more targeted than the diagnosis that produced it.

The three failure categories production pipelines encounter most often

Tool schema drift happens when an external API or platform changes the call signature it expects but the harness that calls it still uses the old one. A newly required field or a deprecated endpoint can therefore invalidate a previously acceptable call. Diagnosis requires checking whether the emitted arguments match the current interface. CI-side schema snapshot testing provides lead time before production encounters the drift.

Prompt ambiguity and prompt drift create a different mismatch. A later prompt version may be scored against a rubric written for an earlier version, leaving the agent judged by criteria it was not designed to satisfy. Ambiguity also appears as low determinism across replays, where the same task produces inconsistent behavior. That variation can flag underspecified prompts, loose tool descriptions, or model-setting drift. Determinism is therefore a property to measure.

Workflow loops concern the sequence itself. When per-step success is below certainty, success compounds multiplicatively across the trajectory, which a single-turn rubric cannot assess. Detecting a loop requires inspecting span sequences across the full trace. TraceFix demonstrates that TLC-verified coordination protocols substantially reduce deadlock and livelock rates compared with prompt-only baselines, with the largest separation under fault injection. Structural workflow failures require workflow-level fixes.

Turning attributed failures into concrete, reviewable fix candidates

Once attribution finds the layer at fault, generation must produce a change that the layer’s owner can review. Reviewability is an engineering requirement because an opaque replacement leaves the reason for the improvement unclear. The SHE framework (2026) uses diagnosed failures to choose which harness artifacts to change, leaving the rest of the harness as it is. This limits the change surface and makes the relationship between diagnosis and revision auditable.

The same principle applies to skills. Memento-Skills uses a read-write reflective loop in which a router retrieves relevant skills for execution and the agent updates them from execution feedback. The resulting artifacts are versioned and inspectable. SkillGenBench (2026) identifies structured skills that are auditable, cached, shareable, and independently updatable as a substrate for this work. It also shows why structure matters, since automatically generated skills can be unstable and induce negative transfer.

A candidate fix for tool schema should change the tool wrapper, one for prompt ambiguity should change the prompt, and one for a workflow loop should change the coordination logic. Mixing those layers makes it harder to determine which revision affected the outcome. Generation must also remain separate from validation. A plausible, readable change is still a hypothesis until evidence shows that it repairs the attributed failure without introducing regressions.

Replay-based validation before a fix is shipped

Replay tests that hypothesis against the traces that exposed the defect. Golden tasks should come from incidents with unusual data, downstream tools that are partly broken, or user input at the edge of the expected distribution, in place of product demonstrations. Full-trace evaluation can then detect wrong tool selection, incorrect arguments, retry loops, incorrect state mutation, and unsafe actions. A satisfactory final answer alone cannot establish that the execution was sound.

The evaluation harness must define golden tasks, replay tool calls deterministically, use real rubrics to score runs, and block regressions in CI. Low determinism is itself evidence that a candidate remains insufficiently specified. A revised prompt that produces different tool call sequences across replays of the same trace has not resolved the underlying ambiguity. TraceFix applies a verification-first repair loop in which exhaustive model checking challenges protocol candidates until no violation is found. For harness changes, the corresponding discipline is to iterate until historical failure replays pass.

In a deployed financial customer-service workflow, targeted reranker and preference-pair improvements at the pipeline layer produced accuracy gains that swapping the underlying LLM did not approach. Replay supplies evidence for repairing the responsible pipeline component before reaching for a more expensive and slower model-level intervention.

Handling long-horizon and multi-agent traces where attribution is harder

Long trajectories widen the gap between the step that introduces an error and the step that reveals it. LongRCA Bench (2026) tests how well diagnosis finds the roles at fault and the root causes of these long-horizon failures. Short-context attribution methods do not generalize when many intermediate spans separate cause from symptom. The pipeline therefore needs evidence that preserves responsibility across those spans. More records are useful only if the relationships needed for diagnosis remain intact.

Multi-agent handoffs require explicit annotations identifying the producing agent’s role, the state passed, and the decision that triggered the transfer. This metadata allows attribution to follow an error backward through a cascade. Concurrency adds another difficulty because races, deadlocks, and missed handshakes depend on interleavings rarely exercised in observed runs. Trace replay alone cannot expose all of those possibilities. TraceFix addresses this gap through formal verification of coordination protocols before runtime execution, complementing observed traces with evidence about coordination paths those traces may not contain.

Closing the loop: feeding validated fixes back as durable pipeline signal

Shipping a validated fix should not discard the evidence that made it possible. Memento-Skills sends execution feedback to the skill store, where how often a skill is retrieved and its runtime failure rates help decide whether to compress it, add to it, or retire it. SkillGenBench (2026) also treats the generation of correct, reusable, executable skills as a distinct problem, so skill updates need the same replay-validation discipline as prompt and tool changes.

Comparing batches of trajectories helps separate reusable patterns from task-specific noise. A single successful replay does not establish that a fix generalizes, while a recurring failure after a prior repair is evidence for renewed attribution. Production traces become evaluation datasets, and validated improvements return as skills, rubric updates, or schema snapshots that sharpen detection of the next similar defect. The full cycle connects preserved evidence, layer-specific diagnosis, reviewable generation, and replay validation. This architecture is not an occasional debugging tool, but a repeatable engineering system that prevents recurring failures from requiring the same diagnostic work each time.