Cold Start Problems in Agent Improvement Flywheels

Agent improvement flywheels face a cold-start problem because useful production evidence is scarce precisely when failures are hardest to diagnose. The way through is to seed high-quality traces deliberately, identify which layer caused each failure, and validate targeted harness changes through replay. Model retraining is not the first response to a system that has not yet learned where it breaks.

Why the agent improvement flywheel stalls at launch

An improvement flywheel uses evidence from earlier runs to make the agent work better in later runs. At launch, however, there is little evidence to compound. Production traces must accumulate at sufficient volume and quality before they can expose recurring failures and support validation. Yet the agent must already do real work well enough to create those useful traces.

Two interlocking causes deepen the gap. Signal density, meaning usable training examples extracted per unit of interaction, is low because early curation heuristics are coarse. At the same time, tool-calling agents are most vulnerable at the start of a session, before completing regular agentic tasks, a condition called the cold-start safety gap. The system therefore produces its least mature evidence while operating at its most fragile point.

Multi-step execution compounds that fragility. A root-cause error can cascade through planning, memory, and tool-use modules before appearing as a failed task. By then, the result shows the effects of several decisions made after the error. The trace may be long without being informative about where the failure began.

What outcome-level failure signals cannot tell you

Knowing that execution failed is different from knowing why it failed. A dissatisfied user, an incomplete task, or a missing output establishes an outcome. Without attribution to the responsible layer, a proposed improvement remains a guess. More logs do not resolve that uncertainty if they expose only downstream symptoms.

Tool-call failures, context truncation, and runaway loops cause more agent incidents than model errors do. Standard APM tools cannot distinguish these causes without agent-aware instrumentation. Asking an LLM to read execution logs does not always solve the problem either. Such reviews risk first-plausible-cause bias, in which the reviewer settles on a convincing explanation before exploring the evidence fully.

Root-cause attribution instead identifies the first divergence from correct execution and assigns responsibility to the model, harness, environment, or grader. AgentRx (2026) synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log with failure category attribution. The AgentRx (2026) paper reports an average 75% improvement in step localization over prior work. Its relevance to cold start is the specificity of the resulting feedback, which makes even a small trace batch more actionable.

AgentDebugX (2026) connects detection and attribution to ranked fixes and reruns around the failure point, escalating hard cases to DeepDebug for an auditable root-cause report. Both approaches distinguish locating a fault from merely recognizing a bad result. Work in 2026 proposes causal graphs rather than flat logs for hierarchical attribution in those systems.

The three failure categories that dominate cold-start production traces

Early production traces cluster around three practical categories, which are tool schema drift, prompt ambiguity, and workflow loops. These categories give reviewers a vocabulary for connecting evidence to an intervention. Although each can end in a wrong answer or failed task, they require different harness changes. Treating them all as reasoning failures hides the boundary that needs repair.

Tool schema drift occurs when the harness submits a payload a tool no longer accepts or retrieves context that has become stale. Without semantic and freshness validation, the problem can remain hidden until task completion. The failure then looks like poor reasoning even though the fault came through a tool or data source. The intervention belongs at the tool boundary, using schema validation, versioned tool contracts, and canary-style comparison of tool responses before full rollout.

Prompt ambiguity occurs when instructions leave several plausible actions open without enough constraint to choose correctly. It is especially acute at cold start because prompts are often written against synthetic scenarios that diverge from the distribution of real traffic. The model can therefore produce a confident-looking answer while following an interpretation the workflow did not intend. The fix at the prompt-layer is to rewrite unclear instructions with examples of real failures as counter-examples.

Workflow loops arise when planning makes the agent keep working toward a completion goal it cannot meet. In long-horizon workflows, hooks can run predefined tests after each step and return failures to the model as error text. Without a break condition, that feedback sustains repeated tool calls and growing context until truncation, timeout, or another external stop. Explicit break conditions, step-count guards, and loop-detection instrumentation surfaced in the trace address this workflow-layer failure.

Across all three categories, drift can remain silent. Model updates, prompt edits, and tool schema changes can erode quality without producing an immediate execution error. A successful repair therefore establishes behavior that subsequent changes must preserve.

Why the harness is the right place to intervene first

Harness iteration is the correct first response to these cold-start failures. Tool schema drift, prompt ambiguity, and workflow loops originate in the harness, so retraining targets the wrong layer. The harness comprises the execution environment, agent scaffolding, and orchestration logic around the LLM. Its responsibilities include multi-step workflows, tool use, external retrieval, error recovery, and human-in-the-loop checkpoints. A clever prompt or a model update alone cannot replace all of that execution logic.

The distinction matters because different faults call for different work. Harness failures should shape how the harness is built, while model failures should shape the goals of post-training. Tianshi Xu, Huifeng Wen, and Meng Li (2026) argue for “adapting the interface, not the model” through runtime harness adaptation before invoking post-training. That case is especially strong when early evidence points to a broken contract, instruction, or continuation rule and the model already has the ability the task needs.

Harness changes are faster, cheaper, and reversible. Prompts, tools, and routing can be versioned as code with git history, then tested against a regression suite built from production failures. Retraining changes model weights and carries the risk of catastrophic forgetting, in which new training erodes prior capability. It is a higher-stakes intervention than editing the component responsible for a localized failure.

NVIDIA's 2025 adaptive data flywheel report grounds component-specific intervention in production evidence. For NVInfo AI, the team analyzed post-deployment failures, replaced a large routing model with a fine-tuned smaller one, and fine-tuned the query rephrasal model, achieving 70% latency improvement. This was not a no-training approach, but it targeted diagnosed components rather than treating full retraining as the default. The strongest objection is that self-modifying agents blur the boundary by updating their own prompts or routing logic. At cold start, however, trace volume is insufficient to make that self-modification reliable, which favors human-reviewable, evidence-backed harness changes.

How to manually seed the flywheel before production traffic arrives

Begin with task scenarios covering intended workflows and known edge cases, then run them before launch to generate the first traces. Ground those scenarios in the real task distribution. For a customer support agent, that means using real past tickets. Synthetic execution is useful only insofar as its tasks represent the work the agent must perform.

Structured feedback makes those runs more useful than unannotated examples. The AITL framework, developed at Airbnb and presented at EMNLP 2025, captures four annotation types, namely pairwise response preferences, agent adoption decisions and rationales, knowledge relevance checks, and missing knowledge identification. AITL feeds that evidence into the improvement pipeline from day one and reduces retraining cycles from months to weeks. Its production pilot also demonstrates the value of embedding feedback directly into live operations.

Next, instrument expert-curated runs in which the agent succeeds end-to-end and preserve them as golden traces. Each trace should have every user message, tool call, and tool response so it can serve as a deterministic fixture for replay. The recommended starting regression suite is roughly 20–50 “must-still-work” cases drawn from production or expert curation. This small set establishes a baseline against which later changes can be tested.

Once early traffic arrives, route a portion of live runs to human reviewers who label outcomes, correct failures, and identify missing capabilities. Capture the rationale for each correction so the evidence points to the appropriate harness layer. Then curate raw logs into clean successes, corrected failures, and low-confidence edge cases. The goal is a high-signal first batch. If signal density stays flat as heuristics are refined, diagnose the curation pipeline before collecting more data.

Replay-based validation as the gate between seeded and live improvement

A seeded fix is not ready to ship until it has been tested against existing execution history. Replay reruns the agent against recorded inputs and tool responses without making new live calls. By playing back the same tool responses as deterministic fixtures, it reproduces the conditions needed to compare behavior before and after a change.

That control matters because LLM failures are non-deterministic. Traditional logs record what happened but cannot by themselves reproduce the execution conditions. Replay can expose whether a prompt or tool change produces a different outcome on the same input, without claiming to make the model itself deterministic. Trace-level replay is therefore a baseline requirement for reliable agent evaluation.

Three evaluation layers support this gate. Unit evals test each step on its own, LLM-as-judge regression suites judge the subjective quality of outputs, and ongoing sampling of production traces catches real-world drift. The Chronicle framework adds replay from intermediate cut points, so regression tests can restart long-horizon trajectories without the cost of full replay. Together, these methods connect a localized repair to its effect on the broader workflow.

FlyRoute (2026) provides a quantitative example of the value beyond initial seeding. On an enterprise developer-support dataset, seed-only retrieval outperformed a zero-shot LLM router, but the full flywheel after replaying the training stream reached 89.83%. The result distinguishes a useful starting point from the improvement cycle built on it.

Replay must also continue after deployment. Running the golden trace set against current production on a schedule can give early warning of silent drift from model updates, prompt edits, and tool schema changes. These are the same changes that can quietly reopen the three failure categories.

How the flywheel transitions from seeded to self-sustaining

The flywheel becomes self-sustaining when production traffic supplies enough well-curated evidence for each cycle to generate higher-signal traces than the last. Signal density rises, failure rates in each category fall week-over-week, and corrected production runs add to the golden trace set without anyone having to build those traces by hand. NVIDIA's NVInfo AI deployment shows this shift: during a three-month watch after deployment, it gathered enough structured negative feedback to find the main failures and choose improvements to address them. The evidence had become specific enough to guide component-level work.

Different learning layers reach usefulness at different cadences. In-context learning adapts to each request through the prompt without changing weights, so it has the highest cold-start resilience. Retrieval-augmented generation refreshes its index on a minutes-to-hours cadence and can begin with curated knowledge before traffic arrives. LoRA/QLoRA adapters and preference optimization operate on daily-to-weekly or per-batch schedules and become appropriate only when the flywheel supplies reliable training signal. Maturity does not mean moving every improvement into model weights.

GEPA (2026) reinforces that distinction. Its Reflective Prompt Mutation uses natural-language reflection on execution trajectories to evolve prompts and outperforms GRPO by 6 percentage points on average using a fraction of the rollouts. Harness improvement therefore remains productive beyond launch. It is not merely a temporary substitute for training.

Sustaining that productivity requires a repeatable engineering cadence. Version prompts, tools, and routing as code, maintain golden traces, and schedule weekly failure reviews followed by targeted updates and replay validation before deployment. Measure improvement cadence. Without reviews on a schedule, harness configurations can drift silently and lose behavior that used to work.