Review Processes for Production-Derived Agent Harness Changes

Changes to agent prompts, tools, memory, and workflows that come from production deserve the same care in review as code. A plausible fix is not sufficient evidence for release. The review process must link a production trace to a specific harness layer, check the proposed change against that layer’s failure modes, and require replay validation before shipping.

The harness is the dominant determinant of production agent performance

A production LLM agent is a model wrapped in a harness that determines what it sees and how it acts. That harness includes the prompt template, tool set, memory and retrieval layer, planning strategy, and verification policy. Each component can fail independently of the others. Prompt style, tool policy, planning depth, and step budget are therefore distinct controls that can be improved independently of model replacement.

Jung and Son’s 2026 study describes harness configuration changing end-to-end performance on a fixed model by as much as six times on the same benchmark. Their decomposition of a planning harness into four measurable layers found that declarative planning produced the largest performance gain, with zero LLM calls. LLM-backed revision activated on only a small fraction of turns. This “harness-first, LLM-residual” picture places the harness at the center of reliability engineering, with a corresponding implication for review. Changes to controls with this much influence introduce commensurate regression risk.

Why harness changes introduce regression risk that code changes alone do not

The six-times performance swing makes harness editing a powerful intervention, but it does not establish that any particular edit will help. Shao et al.’s 2026 Harness-R1 study found that a fixed Self-Refine rule lowered reward on all three benchmarks tested, while gains from frontier harness editors were unstable. A change can read as a sensible correction while degrading the behavior it was intended to improve, so approval based only on the edit’s apparent logic is insufficient.

The difficulty extends beyond the changed component because harness failures propagate through subsequent decisions. A tool defect can first become visible as an incorrect final action, far from the call that introduced it. Tool schema drift illustrates this separation. An external API adds a required field or deprecates an endpoint, but the agent continues using the old signature and malformed responses propagate without error. Reviewing only the final answer obscures the originating defect.

Prompt ambiguity and workflow loops create similar diagnostic confusion. An underspecified instruction can look like a model capability limit when there is no reference for what the prompt was meant to accomplish. Workflow failures can be inner-loop problems such as schema mismatch and bad arguments, task-loop problems such as task drift and context exhaustion, or meta-loop problems such as runaway execution and circular agent dependencies. These failures require different kinds of correction. Their shared risk is that the visible symptom points reviewers toward a plausible but incorrect explanation, allowing an unreviewed change to preserve the original failure path.

Root cause attribution must be layer-specific before any change is proposed

A review cannot assess a proposed correction until the originating defect has been localized. “The agent failed” describes an outcome. Attribution must instead search the evidence, test plausible explanations, and reject those contradicted by the execution record. This guards against “symptom-as-cause,” where a downstream error is mistaken for the event that started the failure. As execution logs become larger and more distributed, a structured requirement to examine competing explanations counters the tendency to stop at the first convincing one.

Layer-specific attribution is therefore a design requirement for the review process. OpenRCA classifies failures into twelve pitfall types across intra-agent reasoning, inter-agent communication, and agent-environment interaction. A reasoning failure, a communication failure, and an environmental mismatch require different evidence and targeted feedback. The proposed change must identify which harness component is responsible before reviewers can judge whether its scope fits the defect.

Paul’s 2026 work makes this relationship explicit by treating the harness as a small, fixed, enumerable action space. It describes configurations in which “every decision the controller makes is one of a small number of named, pre-reviewed configurations, so a regression is immediately attributable to a lever, not a diff.” Named controls give attribution a concrete destination. That approach also shows why the review checklist must have checks for each component. A general question about whether to approve a change cannot tell a memory write defect from a retrieval policy defect, even when both produce the same wrong answer.

Production traces are the only evidence base that makes attribution trustworthy

Without production evidence, layer attribution is still a guess. Synthetic tests can exercise an expected failure, but they cannot establish what caused the production run under review to fail. Agent observability needs to capture intent and process, including the reasoning trace, tools considered and invoked, arguments passed, responses returned, tokens spent at each step, and latency at each hop. These records must form a hierarchical trace that can be replayed.

This requirement follows from the recurrent nature of production agent execution. A prompt may combine conversation history, retrieval, persistent memory, tool outputs, workflow state, system instructions, and contributions from other agents. Component-level logging alone cannot show every drift problem that comes from these interactions. Inputs and outputs leave too much unexplained. Traditional logs also cannot reproduce non-deterministic LLM failures, making span-based trace replay a baseline for reproducing the incident and evaluating its proposed correction.

Tail-based sampling keeps every failed, expensive, or anomalous trace in full while aggressively sampling the happy path. This preserves the evidence needed for review. The resulting trace can support both the attribution argument and the later replay test, connecting diagnosis to validation.

What a layer-specific review checklist covers for each harness component

Engineers can adapt the following component-level checks to their harness. For the prompt layer, evidence must show which instruction was ambiguous or misread as well as the wrong action that followed. Reviewers must assess whether the edit resolves that specific underspecification or adds instructions that may conflict with existing ones. Scope is part of the review. Prompt snapshot diffing in CI can detect unintended scope changes between versions.

For the tool layer, the trace must show the schema at the time of the call and the response received. Review distinguishes between changing the tool schema, changing the agent’s call signature, and changing both. Reviewers also need to check backward compatibility with existing traces. Schema snapshot testing records tool schemas, compares them on each run, and fails on breaking changes. That check gives teams time before schema drift occurs in production.

For memory and retrieval, evidence must show what was written, what was retrieved, and how retrieved content shaped the output. Reviewers check whether the change affects the write policy, the retrieval policy, or both. The distinction determines where the correction belongs. Stale retrieval and a write that overwrote valid state may appear similar at the output, but they do not justify the same fix. Attribution must resolve that ambiguity before approval.

For workflow and orchestration, the review needs the full execution path, including the visibly failed step. The proposed change must fix the loop type found in the trace, whether an inner-loop schema mismatch, task-loop drift, or meta-loop runaway. Reviewers must also check the termination conditions. Reviewers must also check whether the change introduces a new agent-to-agent confirmation loop. A correction that closes one execution path while creating a circular dependency has not resolved the orchestration risk.

Replay validation against historical traces is the gate before any harness change ships

Review catches problems in the logic, but replay is needed to show emergent behavior when components interact. A prompt correction can be justified by its source trace and still cause a regression elsewhere. Before release, the candidate change must be tested against real historical traces because reviewers cannot be relied on to predict every downstream effect from the diff.

Harness-R1 validates edits by rerunning the frozen target agent on same-batch failures and using the outcomes as the reward signal. Validation uses the failures that prompted the edit. The Chronicle project formalizes a complementary mechanism called cut-point replay. It replays recorded non-deterministic boundaries, including LLM calls and tool outputs, while executing new code live against them. This turns a recorded incident into a regression test and reduces validation cost for long-running agents.

The triggering incident is necessary but not sufficient. A curated golden trace set, covering known failure modes with expected outputs or rubric scores, tests whether the fix preserves behavior beyond its immediate target. Replaying that set through the current production pipeline shows when model updates, prompt edits, and tool schema changes cause a regression. Scheduled replay provides an early warning of silent drift. Judge LLM evaluation with structured rubrics can agree closely with human evaluators while costing much less than manual review, so automated assessment can work at scale.

Replay results also supply the evidence for the ship decision. A reviewable change record includes the trace that surfaced the failure, the layer attribution, the proposed diff, and results showing that the fix closes the failure without regressing the golden set. The record is a production artifact in its own right. Someone who did not write the change can examine the connection between diagnosis, intervention, and observed outcome. That evidence chain makes review auditable rather than dependent on the author’s confidence or a sign-off detached from execution.

How the review loop feeds back into continual harness improvement over time

Consistent review produces more than a sequence of approved changes. It accumulates a structured corpus of failures, layer attributions, and verified corrections that can reveal recurring patterns. Harness-R1 describes agents that “continually accumulate interaction trajectories during deployment” and treats those trajectories as material for improving the harness. Collection alone does not complete that process. Attribution and replay turn recorded behavior into evidence that an improvement loop can act on reliably.

This distinction is important for automated improvement as well as human review. Harness optimization can need more replay or validation data, and the review process creates that data as part of normal engineering work. The output is not merely another execution log. It connects a failure to a specific control and records what happened when that control changed. Such a record supports more targeted improvement than an unexplained success or failure verdict.

The recurring discipline is to observe failure in traces, attribute it to a layer, propose a fix, validate by replay, then ship and monitor. Automated scorers complement review-time validation by blocking regressions from shipping or flagging live quality drops. Production feedback then returns to the same review process. Each caught regression strengthens the golden trace set, broadening the evidence available for subsequent changes and making earlier failures useful beyond their original repair.

Without this loop, defects can accumulate in the harness while teams reach for model upgrades to address the visible symptoms. Structured review steers that work toward the component at fault instead. Its payoff extends beyond the individual fix because the attribution and replay evidence remain available for future decisions. Review is the engineering discipline that converts production failures into durable improvements in agent reliability.