Selecting High-Value Production Traces for Harness Improvement

Selecting high-value production traces for harness improvement means choosing evidence that can support a durable fix. The strongest selection sets combine attributable failures, diverse failure modes, and representative production cases. Replay-based validation then tests whether a proposed change improves the harness without breaking behavior that already works.

Why trace selection is an engineering decision, not a logging configuration

A harness determines what an agent observes, which actions it can execute, and how execution recovers after deviations. Harness-R1 shows that what an agent can do can change a great deal under different harnesses, even with identical model weights. Execution traces show how the harness affects reliability in ways that model-level benchmark scores cannot separate out. Engineers must identify which traces expose changes worth making.

AutoSaddler illustrates the stakes on Terminal Bench 2.0, where harness changes including self-verification, different-input testing, build-artifact clean-up, library preference, and environment discovery deliver gains without changing the model. AutoSaddler also reached its best result after using roughly 147 traces, many times fewer than a competing method. Its ablations show that removing generalization-aware selection substantially reduces performance and reverses the regression trend from improving to worsening. Production traffic offers far more trajectories than a team can usefully examine, so retaining everything does not resolve the central constraint. Without care in choosing traces, repeated or misleading evidence can lead optimization to chase noise at the cost of reliability.

What makes a trace carry attributable, actionable signal

A useful trace must support an intervention, and a useful selection set must support an intervention that generalizes. Three properties govern that distinction, namely attributability, diversity, and representativeness. Attributability connects the step where a failure surfaces to the harness layer that caused it, whether prompt, tool schema, workflow control, memory, or product logic. AutoSaddler uses deep debugging of failed trajectories because a long-horizon failure may originate many steps before the visible error. Without that connection, a failed trace provides evidence of a problem but no clear fix target.

Diversity concerns coverage across the selection set. A set dominated by one tool or failure mode can encourage a local patch while leaving other weaknesses untouched. RHO addresses this through coreset selection, using a determinantal point process to choose a small set of past tasks that vary in difficulty.

Representativeness keeps that coverage tied to the work that happens in production. Rare, exotic failures can deserve attention, but they should not displace common cases whose behavior a patch must preserve. Passing cases are useful traces too, because they can be used for regression checks. These properties constrain one another, because attributable examples from a single tool lack diversity, while a varied collection of exceptional cases may still misrepresent production.

The failure categories that produce attributable traces

Tool-call malformation and schema drift provide the clearest starting point for attribution. A trace can expose missing or invalid arguments together with the environment’s rejection or silent failure. The tool schema supplies a formal artifact against which the emitted call can be inspected. Harness-R1 identifies tool misuse as a systematic failure signature that its engineer model learns to patch from trajectories. The value of this category lies in the inspectable gap between the required call and the call the agent produced.

Prompt ambiguity and silent context failures require a longer causal search. An agent may repeat invalid arguments because the prompt did not constrain its action space, or critical instructions may disappear when the context window fills. The observable error can occur several steps after that loss of guidance. AutoSaddler’s emphasis on deep debugging addresses precisely this separation between the originating cause and the step where failure becomes visible. A review confined to the final error risks assigning the fix to the wrong layer.

Runaway workflow loops have a different signature, which is repeated calls with minor variation, high token consumption, and no convergence. Harness-R1 explicitly targets “repeated attempts and failed recovery” as a failure class. Such traces show where termination rules or loop-detection mechanisms are missing, so the fix belongs in the control logic. They are costly and tend to recur until that logic changes. Across all three categories, reasoning and tool execution can entangle over many turns, so attribution requires evidence from the steps leading to the final error.

Why automated attribution is still an open and partially solved problem

The Who&When benchmark introduced by Zhang et al., ICML 2025, measures how well automated failure attribution finds the agent and step at fault in LLM multi-agent systems. AgentDebugX’s multi-round approach improves attribution accuracy to 28.8%, well above the strongest single-pass method but far below what fully automated selection would require. That result leaves most attribution decisions unresolved automatically. The gap matters because identifying a failed run is much easier than identifying the agent and step responsible for its failure.

AGENTSCOPE takes a neuro-symbolic approach, abstracting behavior into a Reasoning-Action Graph with formally defined neural invariant violation conditions. Even with its best-performing model tested, it resolves only a modest share of failure attribution cases. The constraint is not specific to one design. A system must connect an error to an earlier cause across a trajectory whose reasoning and actions may have obscured that connection.

AgentDebugX addresses the problem through a closed loop of Detect, Attribute, Recover, Rerun. It escalates difficult cases to a multi-round diagnosis agent that produces an auditable root-cause report. Teams should use it to prioritize traces for examination, while retaining engineering judgment over which diagnoses justify a harness change.

Sampling strategy: which traces to capture before selection can happen

Sampling determines which evidence remains available when selection begins. A tail-based strategy should keep every failed, expensive, or anomalous trace in full while keeping only a small sample of the happy path. Once a baseline is in place, more successful traces often add little to what teams can learn about failures. Failed, slow, and anomalous runs are more likely to expose a distinct weakness, although cost and failure remain candidate signals that require root-cause diagnosis.

That strategy requires spans for every inference step and tool call, capturing arguments, outputs, step outcomes, and timing. A function name alone cannot show whether a call broke the rules of its schema. Standard APM tools cannot expose tool-call malformation, context truncation, or loop signatures without the necessary agent-level detail. Full arguments and intermediate outcomes turn a record of activity into evidence that can support attribution.

How much detail gets captured also affects whether later validation can reproduce the original failure. Coarse logs may show which run went wrong but lack the detail needed to test a fix against it. Chronicle reports boundary-level recording overhead of 23 μs per crossing, showing that complete recording can be operationally feasible.

How to build and maintain a selection set that drives durable harness fixes

The practical selection set should be organized by failure category and harness layer. Within those groups, stratify by tool name, error code, and argument structure, including argument-edge-case buckets. This prevents the latest incident from crowding out unrelated but important weaknesses. It also helps teams judge whether a trace adds coverage or repeats a pattern they have already seen. The unit of progress is a better-covered failure mode.

RHO’s coreset selection formalizes the same principle by choosing challenging tasks that differ across the trajectory space. Teams can pursue that diversity through stratification without implementing a determinantal point process. A weekly promotion cadence then keeps the set aligned with current production behavior as the harness changes. Any failing trace that reveals an unrepresented failure mode becomes a candidate for promotion into the eval set. Traces that duplicate existing patterns are sampled.

Coverage alone is insufficient if the selected evidence encourages changes that damage passing behavior. AutoSaddler’s generalization-aware selection validates candidate updates against a held-out set before accepting them. AutoSaddler’s 2026 analysis reports that with this mechanism, the regression-rate trend is -0.24 percentage points per iteration. AutoSaddler’s 2026 ablations report a trend of +0.16 percentage points per iteration without it.

The practical equivalent is to replay a proposed fix against the existing selection set before promoting it. Where ground-truth labels are unavailable, disagreement and contradiction can help identify candidates for investigation. RHO re-solves coreset tasks multiple times and uses cross-trajectory disagreement for self-consistency analysis and within-trajectory contradictions for self-validation. These signals can reveal weaknesses in the harness without external grading. They support selection in unlabeled production settings without removing the need to establish a credible fix target.

Validating harness changes against the selection set before shipping

A selection set earns its engineering value when its diagnostic archive becomes a release gate. Agent failures are hard to reproduce because inference is non-deterministic, tools read state that changes, and new runs rarely follow a multi-step trajectory exactly. Replay addresses that problem by testing changes against the captured execution evidence. Chronicle’s boundary-level recording makes this approach operationally feasible without requiring a fresh production failure for every validation attempt. The purpose is to test the proposed intervention against the conditions that motivated it.

Replay also has a place in a broader three-layer eval model. Unit evals test discrete steps, LLM-as-judge regression suites assess subjective output quality, and continuous production trace sampling detects real-world drift. Each layer covers a different regression risk, so none replaces the others. AutoSaddler’s 2026 ablations underline the danger of optimizing only for motivating examples. Omitting its generalization-aware check drops performance from 62.0% to a substantially lower held-out test score even when training-set scores improve.

Before shipping, a proposed harness change should be tested against the selection set to check that it fixes the failure that led to it. It must preserve cases that already pass. It must also generalize to at least one other trace in the same failure category.

Making trace-driven harness improvement a continuous engineering practice

Harness improvement can run on a shorter cycle than model updates because it does not require changing model weights. Harness-R1 demonstrates a learned harness-engineer model that converts batches of target-agent failures into validated executable patches. Same-batch reruns of the target agent provide outcome rewards that update the engineer. This creates an improvement loop that does not wait for labeled data or manual review of every patch. It demonstrates the practicality of continuous iteration, without establishing that all production attribution decisions can be automated.

RHO shows a related path in a label-free setting. A retrospective pass selects a diverse coreset, re-solves its tasks, and uses self-preference to choose a harness update without external grading. Its analysis shows that each diagnostic step progressively isolates signal that contributes to performance gains. For teams without labeled outcomes, sparse corrections from a stronger model on live traffic can also serve as a proxy for harness health. That signal can flag degradation before failures accumulate into a larger review backlog.

The operating discipline is a living selection set, weekly promotion of new failure modes, and a validation gate before release. Its feedback loop should surface recurring patterns left by repeated local patches as candidates for structural changes. If the same failure category keeps producing new traces despite fixes, the harness architecture becomes a more likely cause than an individual prompt or tool schema.