Trace Sampling Strategies for Agent Observability

Agent traces don't behave like the request logs that observability tooling was built for. What you choose to sample from those traces decides which failures anyone ever sees, and which ones just vanish into the gap between a 200 status code and a wrong answer. Get the sampling strategy wrong, and you don't lose a little detail. You lose the failure entirely, with no record that it happened.

Traditional APM runs on three assumptions: the same input takes the same code path, that path is deterministic, and a 200 response means the request succeeded. Agents break all three at once. The model branches on its own output, tool selection isn't fixed ahead of time, and a 200 can wrap a confidently wrong answer with no error anywhere in sight. The unit of analysis has to shift from a single request to a task, and a task spans tool calls, reasoning steps, memory reads and writes, and handoffs between sub-agents. Most agent incidents (tool-call failures, context getting truncated, loops that never terminate) leave no trace in infrastructure metrics. So the sampling strategy isn't a cost-control knob bolted onto observability after the fact. It decides whether observability exists for a given failure.

What a production agent trace contains

A trace is the causal record of one request, made up of nested spans that share a trace_id, each one carrying a timestamp, a status, a pointer to its parent span, and a bag of metadata. Four span types cover four distinct ways things go wrong.

Tool-call spans carry the name of the tool, the arguments passed in, the raw output, duration, retry count, and error state. Reasoning spans record the plan, the action chosen, the observation that came back, and the decision about what to do next. State-transition spans hold working memory before and after each step, plus whatever payload got handed off. Memory-operation spans log reads and writes, retrieval scores, and freshness.

Scale turns this into a real problem fast. A single coding-agent session can produce a JSONL log running past ten thousand lines, and most of those lines are tool output, not decision points. The moments that actually matter, where the agent chose one path over another, sit buried inside file contents and terminal dumps. Stanford's VCC (View-oriented Conversation Compiler) research takes this on directly by compiling raw JSONL into three separate views, full, UI, and adaptive, and reports cutting reflector token use by half to two thirds while lifting task goal pass rates by 1.1 to 4.2 points across 168- and 416-task splits. Raw traces, as agents produce them, are hostile to analysis by default. That's the whole reason a compiler like VCC needs to exist.

The case for capturing everything

Keep-everything is a serious position, not a lazy default. Rare failures (hallucinations, retry storms, loops that never resolve) are long-tail events by definition, and any sampling scheme biases against exactly the thing you most need to catch. Most LLM failures also return HTTP 200, so sampling triggered by error rate misses them. And the economics favor keeping everything: a compiled trace summary can run around 1.5 kB in storage, while the inference cost behind it is orders of magnitude larger. Retention guidance generally favors keeping traces in hot, searchable storage for as long as cost allows, with cold storage held longer for building eval datasets later.

Where this argument falls apart is pricing, and it falls apart hard. For high-volume systems, the binding constraint is vendor ingest cost. It's vendor ingest cost, and that's a different bill. Observability vendors generally treat sampling as justified once trace volume grows large enough to create meaningful cost and capacity pressure. Multi-agent systems make this worse by construction, since one user request fans out into sub-agent traces, tool-call spans, and memory operations that multiply faster than the request count itself. Volume doesn't scale linearly with agent complexity. It scales super-linearly. The keep-everything argument gets weaker exactly when the system gets more interesting.

Deciding which traces are safe to thin and which ones need to be kept whole is what matters most. It's which traces are safe to thin and which ones need to be kept whole, and answering that requires knowing what each failure category actually looks like inside a trace. That's what the rest of this piece works through.

What head-based sampling protects and discards

Head-based sampling makes the keep-or-drop call at ingestion, before a single span gets written. It's stateless, cheap to run, and doesn't need to know anything about how the trace ends.

That buys a representative slice of successful runs for baseline performance work, sane cost and capacity management under volume spikes, and latency percentiles that stay statistically valid. What it can't buy is anything tied to a specific input, payload, or context window state, which happens to be exactly the condition behind tool schema drift and prompt ambiguity failures. Rare, severe events (runaway loops, cascading sub-agent failures, context truncation at some boundary case) get sampled away with the same odds as everything else. And silent failures, the 200 wrapping a wrong answer, throw off no signal that head-based sampling could even use to favor them.

The honest tradeoff: cheap and easy to run, but it needs careful per-service tuning, or it quietly erases the exact signals that matter most. Treat it as a capacity lever. Vendor guidance tends to recommend it mainly once volume passes a few hundred traces.

Tail-based sampling: catching errors and slow traces after the fact

Tail-based sampling waits. The decision doesn't get made until the trace closes, once the collector has seen every span and can judge the whole thing at once.

That deferral buys real coverage. Traces with anomalous behavior, such as runaway loops and unbounded retries, can be caught once the full span record is available for inspection. Traces carrying explicit failure signals, such as a failed tool call or malformed output, become eligible for retention once the collector can evaluate the complete trace. If there's an inline scorer attached, traces that cross an eval threshold get caught as well. The cost is infrastructure: the collector has to buffer spans until the trace closes before it can decide anything, a heavier lift than head-based sampling's stateless coin flip.

What tail-based sampling still misses is the harder category: silent quality failures. Wrong answer, wrong tool picked, an argument the model invented outright, all of these can complete cleanly, no latency breach, no error span, nothing for the collector to key on. Description-level tool schema drift lives here too. The model calls a tool it shouldn't, or skips one it should have called, and no exception fires either way.

Error-biased and eval-score-gated sampling for silent failures

Tool call failures occur somewhere between 3 and 15% of the time in production, depending on model size and task complexity, and a meaningful share of those never raise an error signal. Tool call failures, incomplete context handling, and unresolved reasoning loops fail silently in a few distinct flavors, and none of them look like the failures traditional monitoring was built to catch.

Description-level schema drift is one: the model invokes a tool in the wrong context, or skips it entirely, and neither schema type-checking nor a normal unit test suite catches it, because nothing is technically broken. Attention decay is a quieter version of the same problem. Because attention to the original system prompt fades relative to whatever tokens came in most recently as a session runs longer, the failure appears mid-session as gradual behavioral drift rather than as a discrete error anyone can point to.

Error-biased sampling handles part of this by weighting retention toward traces with specific span-level red flags: a tool-call span with retry count above some threshold, a high count of self-correction loops (usually a sign of bad schema design or an unclear prompt), or token-budget exhaustion flags. The Token Budgets catalog, documenting 63 production incidents across overrun, runaway loops, budget exhaustion, and unbounded retry, is a useful reference for how varied this failure family actually gets.

Eval-score-gated sampling picks up the rest by running an online scorer against a slice of traffic and using the resulting score, not an error flag, as the retention signal. Low scores become high-priority keeps, high scores get thinned harder, and quality regressions, top failure categories, and expensive workflow patterns all become things you can alert on directly. That turns sampling from a cost lever into a quality signal, a genuinely different job. It only works if the scorer runs fast enough at trace close to avoid adding latency to the retention pipeline itself, and it has to score at the step level rather than just the final answer, or it will miss divergence that happens mid-trace and quietly self-corrects before the end.

Step-level adaptive sampling in multi-agent and multi-step workflows

Trace-level sampling assumes the trace is the right unit to keep or drop as a whole. Multi-agent systems break that assumption. Reported failure rates for multi-agent LLM systems in production run between 41% and 86.7%, and specification ambiguity combined with unstructured coordination protocols accounts for 79% of the breakdowns behind that number.

Error propagation is the mechanism that makes trace-level sampling especially blind here. An orchestrator hallucinates a fact, hands it to three sub-agents, and all three produce answers that are individually coherent and individually wrong, with no exception thrown anywhere along the chain. Looked at from the trace root, everything looks fine. The MAST failure taxonomy, validated across more than 1,600 execution traces, maps 14 distinct failure modes across three root categories. The pattern that falls out of that labeling: these failures live in specific spans, not at the top of the trace.

Step-level adaptive sampling responds by applying the retention policy at the span level instead of the trace level. Sub-agent handoff spans, tool-call spans hitting external APIs, and memory-write spans that get read downstream all get kept whole. Repeated retrieval spans for the same entity, or intermediate reasoning spans on a workflow path that's proven stable, get thinned more freely. The adaptive part adjusts span retention rates based on the failure rate observed for that span type in recent traffic, so the policy tightens automatically around whichever span type is currently misbehaving.

Stratifying by tenant or feature matters here too. A small tenant's failure pattern can get statistically drowned out by a high-volume tenant's traffic, unless the sampling policy guarantees a retention floor per tenant, which several vendors now recommend explicitly for this reason. On the cost side, ClawTrace's approach of compiling sessions into roughly 1.5 kB TraceCards, with per-step USD cost, typed token counts, and clusters of redundant tool calls, lets a policy identify whether a span was expensive and consequential or expensive and simply wasteful.

Matching failure categories to the strategy that surfaces them

Tool schema drift splits into two very different problems depending on the kind of mismatch. A schema mismatch throws a runtime error, and tail-based error retention catches it without much trouble. A description mismatch produces behavioral drift instead, with no error to key on, so it needs eval-score-gated sampling, or it simply never gets seen. Any change to a tool's name, description, or schema is a potential breaking change, and the failure modes it produces are quiet enough that ordinary API testing walks right past them.

Prompt ambiguity and attention decay appear mid-session as plan drift or a wrong branch taken, usually with completely normal latency. Catching this needs reasoning spans, along with step-level eval scores. Neither head-based sampling nor latency-threshold tail-based sampling has any way to see it. Only scorer-gated retention does.

Workflow loops and unbounded retry are more tractable, since retry count, self-correction loop count, and token-budget exhaustion are all direct span-level signals. Tail-based latency thresholds catch the severe cases, and error-biased sampling on loop-count flags catches the earlier-stage version before it becomes a real incident.

Cascading sub-agent failures are the hardest of the four to catch, full stop. The root cause is usually one corrupted handoff payload early in the trace, and every span downstream of it looks fine in isolation. Catching this needs step-level adaptive sampling that preserves handoff spans specifically, paired with diffing across failed and successful runs. Even with that in place, automated root-cause analysis for cascading sub-agent failures remains a hard open problem.

Hybrid sampling in practice: a layered policy that covers the failure space

No single strategy covers the failure space on its own, and that's really the point of walking through them one at a time. Head-based sampling handles baseline volume and cost control. Tail-based sampling adds the errors and latency outliers that head-based sampling structurally can't see. Error-biased and eval-score-gated sampling reach past both of those into the silent failures, wrong answers, drifted behavior, tools called in the wrong context, that never trip a status code or a latency threshold. Step-level adaptive sampling is what makes any of this survivable at the scale multi-agent systems actually run at, since it lets a policy protect the specific spans, handoffs, external tool calls, memory writes, where cascading failures actually start, while thinning spans on workflow paths that have proven consistently stable.

Put together, that's a layered policy rather than a single sampling rate: broad, cheap coverage at the head, error and latency capture at the tail, quality-gated retention for the silent failures, and span-level granularity wherever multi-agent coordination is in play. None of the four strategies is wrong on its own. Each one is just blind to a different category of failure, and an agent's failure modes (tool schema drift, prompt ambiguity, cascading handoffs) refuse to confine themselves to one category. Building the sampling policy around the failure taxonomy, rather than around a flat percentage, actually decides whether an incident gets captured in the trace store or disappears without a trace.