Running Production Traces as a Regression Suite for Prompt and Tool Changes

To test prompt and tool changes before deployment, teams need a versioned regression suite built from real production trajectories as well as synthetic input-output pairs. Curated traces give the suite its cases, cut-point replay tests changes at the boundaries they affect, and per-dimension CI gates block regressions. The essential discipline is to turn each production failure into evidence that the next candidate change must pass.

Why offline eval sets age out the moment you ship

A frozen eval set captures the traffic and failure modes present when it was assembled. Production does not stay inside that frame. New user intents and traffic shapes introduce paths absent from the seed data, while prompt revisions and tool schema changes alter what a correct trajectory looks like. Changes to retrieval indexes and workflow routing create further gaps, even when the original questions and expected answers remain reasonable.

Consequently, an offline pass is necessary but insufficient for shipping. It establishes performance on the recorded distribution. For teams relying only on static evals, user complaints become the regression detector. Those complaints arrive after the failure and usually describe its visible result.

What production traces contain that synthetic benchmarks cannot reproduce

A production trace records the ordered execution of an agent, including the system prompt, user input, recorded agent reasoning, tool calls with arguments and return values, retrieval results, intermediate LLM calls, final response, and outcome metadata. The trajectory is the unit. An answer that looks correct cannot by itself show whether the agent chose the right tool, gave it valid arguments, or took its steps in the right order.

This distinction concerns structure. Synthetic benchmarks that focus on final-output accuracy miss the intermediate tool-call failures, context truncation, and runaway loops that cause production incidents. Standard APM tools cannot expose these failures without agent-aware instrumentation. With the context lengths common in production multi-step workflows, schema compliance can also be much worse than benchmark scores suggest.

A small, well-chosen trace set can therefore expose real failure modes that a large synthetic set does not exercise. Production traffic supplies unanticipated combinations of inputs, tool results, and recovery attempts. Those combinations give traces a role in regression testing as well as a role as a secondary monitoring signal.

How to sample and curate traces worth running as regression cases

To turn that signal into a repeatable dataset, teams should begin with stratified sampling. The sample should cover every tool in the harness, argument-edge-case buckets, error codes, and failure outcomes, including successful and unsuccessful recovery. It should also reflect actual traffic shapes. Selection is a data-engineering task.

Fifty well-chosen golden traces from production traffic is a workable starting floor. A versioned JSONL dataset gives the selected trajectories a concrete form for repeated evaluation and comparison. Its strata should remain identifiable by tool, argument-edge-case bucket, and error code. Without that structure, a change in sampled cases can be mistaken for a change in agent quality.

Teams should review and add to the dataset each week. Every confirmed production failure deserves promotion into a regression case capable of detecting its recurrence. Near-misses also need a close look right away because a run that succeeds can hide fragile argument handling or weak plan coherence. Curating both classes prevents the suite from becoming either a catalog of disasters or a collection of reassuring successes.

The three failure categories that production traces expose most reliably

Tool schema drift occurs when a harness and a tool no longer agree on the tool’s input or output contract. Calls may omit required fields, supply incorrect data types, or return malformed data without producing an obvious error signal. The n8n issue documents invalid tool schemas after an upgrade from n8n v2.4.7 to v2.6.3 that both OpenAI and Anthropic APIs reject. AgentCheck includes schema drift as fault A4 within tool execution, with detection of field mismatch as the pass criterion. The assertion belongs at the schema boundary.

Prompt ambiguity and binding drift require a different view. Small errors can compound through a multi-step prompt chain until the final result no longer reflects the original task. Task drift, degeneration loops, and version drift cannot be diagnosed from a single output alone. Detection needs the trajectory’s intermediate decisions and their relationship to the task.

Workflow loops are visible through repetition across steps. Wink characterizes Reasoning Problems and related Tool Call Failures in which agents repeat actions or unsuccessfully repair self-introduced errors. One key indicator is invoking the same or similar tool calls three or more times in a row. These categories therefore require distinct assertions for schema fidelity, plan coherence, and repeated behavior. A single aggregate quality score cannot stand in for those checks.

How Chronicle-style cut-point replay turns a recorded incident into a CI regression test

Re-running a historical conversation does not necessarily reproduce its failure. Model inference can change from run to run, tools read state that changes, and a multi-step run may take a different path. Chronicle, by Tisha Chawla and Susheem Koul at Microsoft, addresses this problem at non-deterministic boundaries. It records boundary crossings as immutable envelopes and serves recorded values during replay. The record stabilizes the relevant execution context.

Consider the guarded-tool case used in Chronicle’s mutation study. A recorded run contains an unsafe action that guarded code should reject. A cut-point test serves the chosen surrounding boundaries from the record while executing the changed guard live. If the candidate guard allows the recorded unsafe action through, the assertion fails. Stubbing that guard as well would remove the changed behavior from the test.

Chronicle’s cut-point tests catch every mutant in that study that permits the recorded unsafe action, while the baseline that stubs every boundary catches none. Full replay makes zero model calls and is bit-stable when run again. These are different testing modes with different purposes. Full replay preserves the recorded execution, while cut-point replay exposes selected new code to that execution.

The choice of cut point determines what the test establishes. Different boundaries can isolate a tool change, workflow routing, or the prompt layer. A prompt-edit test cannot demonstrate the effect of the edit by replaying away every boundary affected by it. The test is repeatable because the recorded part is controlled, while the rest runs live against the candidate change.

Structuring the replay pipeline so it fits inside CI

The operational pipeline begins by scoring a versioned trajectory dataset with code-defined assertions and per-dimension rubrics. CI applies those thresholds on every PR and exits non-zero on a failing dimension. Production evaluation then attaches the same rubric scores to live spans after export, avoiding inline latency impact. Failing traces are clustered, scored, and promoted into the dataset. CI then tests the candidate fix against that larger set.

Each trajectory needs separate gates for tool selection, argument extraction, result utilization, error recovery, plan coherence, and task completion. These dimensions are not interchangeable evidence of correctness. High tool-selection scores cannot make up for argument extraction that sends the wrong values to the right tool. In a per-dimension CI configuration, including one expressed in a configuration language, each dimension therefore needs its own threshold and failure condition.

For teams starting from zero, the recommended sequence begins with deterministic assertion-based unit tests on every commit. Those tests catch known regressions before LLM-as-judge scoring runs. Judge-based evaluation of curated traces then addresses subtler quality differences across the distribution, while online evaluation detects drift and surprises the dataset has not captured. Each level supports the next. Starting with online monitoring alone leaves no gate to catch known regressions before deployment.

Attributing failures to the responsible layer before shipping a fix

A failing trace identifies a problem, but not necessarily its origin. A wrong final response may originate in a tool schema mismatch three steps earlier, an ambiguous system prompt, or a workflow loop that exhausted the context window. The visible failure is often downstream of the cause. Patching the final response without tracing it back to its cause can leave the original defect in place.

Full execution traces provide the evidence needed to distinguish those explanations. The Who&When dataset, introduced at ICML 2025, labels both the responsible agent and the decisive error step in annotated failure logs. The ACL 2026 TraceElephant benchmark reports substantially higher attribution accuracy with full execution traces than with output-only logs. Continual Search addresses a related weakness by guiding judges through successive attribution turns rather than accepting the most proximate-looking failure.

Attribute first, fix second, and validate third. Attribution should identify whether the responsible layer is the prompt, tool, workflow, memory, model, or product logic before a fix is written. The ordered trace supplies the causal context, while a selected cut point makes the proposed explanation testable. Otherwise, a change to the wrong layer can pass existing tests while leaving the same production failure available to recur.

How to test prompt and tool changes against production traces before deploying

A candidate change should run against the expanded regression set, including the newly promoted traces that motivated the fix. Testing only the original offline set omits the evidence that established its inadequacy. The candidate’s per-dimension results should be compared with the pre-change baseline. Any regression on a previously passing dimension is a blocker, even with a higher aggregate score.

Boundary-level validation supplies a separate requirement. Cut-point replay should show that the fix works where the attributed failure took place, even if other parts of the trajectory also improve. The minimum viable gate therefore asks two distinct questions. Does the candidate resolve the failure on the traces that exhibited it, and does the full regression set retain its previously passing dimensions. Both conditions must hold before deployment.

This focus favors harness iteration over speculative changes to the model. Prompts, tools, workflow, and memory are the principal sources of agent failures identified in this methodology, and changes to those non-parametric layers are faster and less costly to validate than model retraining. GEPA, by Agrawal et al., 2026, demonstrates the leverage of trajectory-based prompt improvement. It reflects on execution trajectories, mutates prompts, and maintains a Pareto frontier of candidates, outperforming reinforcement learning approaches while using up to 35× fewer rollouts.

The release gate is where the earlier work becomes operational evidence. A validated fix can then ship through CI with a specific account of what was tested.

What the regression suite looks like after the first production cycle

After one failure-attribution-fix-validation cycle, the suite contains more than historical conversations with expected answers. It has a versioned JSONL dataset stratified by tool, argument-edge-case bucket, and error code, with a confirmed production incident permanently represented. At least one cut-point test is anchored to the boundary where that incident occurred. Per-dimension CI assertions exit non-zero on the failing axis. Those artifacts make the cycle repeatable for the next prompt or tool change.

The weekly promotion ritual keeps the dataset connected to production. Teams review failing traces and near-misses in a set way and turn chosen cases from isolated observability records into permanent tests. Teams can review harness changes, fixes have evidence behind them, and improvements must pass replay validation before they ship. The suite now has a maintenance process.

Its value compounds through each subsequent incident. A static set drifts away from changing traffic, while a trace-driven suite grows toward the paths production actually takes. That growth depends on preserving the full discipline, including curation, attribution, boundary testing, and deployment gates. Each production failure promoted into a regression case gives the next release one more known way not to fail.