Cost, Latency, and Quality Alerting on Agent Production Traces

Cost, latency, and quality alerts on agent traces only earn their keep when they run as one system, because agent failures almost never respect the boundary lines between those three axes. A latency spike is often a cost event wearing a different hat. A quality drop is sometimes the delayed symptom of a budget decision made three spans earlier. Treat the three as separate dashboards and the team ends up debugging the wrong layer, every time, with confidence.

Traditional application performance monitoring was built for a world of deterministic HTTP paths: request comes in, code runs the same way each time, response goes out, and any failure sits on a predictable line of the stack trace. Agents don't work that way. A single user request can trigger a planner decision, several LLM calls, a handful of tool invocations, a vector lookup or two, and a handoff to a sub-agent, and each of those steps is a place where something can go wrong invisibly to a monitor built for the old model. Industry observers consistently name the lack of trace-level visibility and quality measurement as one of the leading reasons agent rollouts stall inside organizations, a pattern that shows up across practitioner accounts and postmortems alike.

The rest of this piece works through what each axis actually measures, where the three cross wires in production, and what an alerting system looks like when it's built around that reality instead of around three separate charts.

What production traces must capture before any alert can be meaningful

A trace is the full reassembly of a run, planner decisions, tool calls, retries, sub-agent handoffs, retrievals, and LLM calls, stitched together into one causal chain using context propagation standards (W... It's the full reassembly of a run, planner decisions, tool calls, retries, sub-agent handoffs, retrievals, and LLM calls, stitched together into one causal chain using context propagation standards (W3C trace context, for instance) that survive process boundaries. Without that reassembly, an alert is just a number with no story attached.

The minimum signal set a production trace needs to carry is longer than most teams expect going in. It includes the input prompt and the rendered prompt template (these are not the same thing once variables get injected), retrieved documents and their reranker scores, the LLM request and response pair, tool call arguments and tool call outputs, guardrail decisions, token usage broken into prompt tokens, completion tokens, and total, latency at the span level, errors, evaluation scores, user feedback, and cost per request computed from model pricing rather than pulled off a monthly invoice weeks later.

Three units of analysis do the work here, and alerting that only understands one of them is alerting half-blind. A span is one model call or one tool call, and it's where per-step cost and latency actually live. A trace is one complete agent run assembled from its spans, and it's where root cause attribution happens. A session is the full conversation across many traces, and quality drift across turns appears there, often well after the turn where the real damage started. Evaluations can and should score at all three levels. A platform that only scores the final answer is going to miss every step-level failure that got quietly patched over by a later, more confident-sounding response.

A full prompt-and-response payload runs into the kilobytes per span, and plenty of observability platforms bill by span count or by gigabyte ingested, which imposes a real cost on granularity. A full prompt-and-response payload runs into the kilobytes per span, and plenty of observability platforms bill by span count or by gigabyte ingested. So teams sample. They keep 1 in 10 traces, or they truncate payloads, and the incident that only appears in the tail, the one weird retry storm at 2am, never got recorded in the first place. Any alerting design that doesn't account for the sampling tax is building on a foundation that quietly disappears exactly when it's needed most.

Infrastructure spans belong in the same store as LLM spans, not off in a separate APM tool. A support-agent trace in OpenObserve ran 8.42 seconds end to end, with 41% of that time a Postgres lock wait, nothing to do with the model at all. That attribution was only possible because the database span and the LLM spans lived in the same trace. Platforms that segregate LLM observability from traditional APM create blind spots that no amount of clever alerting logic can close after the fact, because the data needed to close it was never joined in the first place.

What the cost axis measures and where it breaks without trace-level accounting

Cost, done right, is computed per span from model pricing the team itself defines, matched to the actual billing arrangement in place, including negotiated enterprise rates or the amortized cost of a self-hosted model. That's a meaningfully different thing from reading a total off an invoice at the end of the month. Done right, the expensive step is a row in a table that can be sorted and inspected, not a surprise that arrives four weeks too late to do anything about.

The right units for cost tracking are per request, per span, per session, per agent, and, critically, per successful outcome, meaning cost per successful outcome rather than cost per token consumed. Raw token counts are close to useless as a health metric on their own, because a system can burn fewer tokens while resolving fewer problems, which is not an improvement by any definition that matters to the business paying the bill.

Cost alerting breaks down in a few specific and recurring ways once trace-level accounting is missing. Retry storms are the clearest case: a tool call fails, the harness retries silently, each retry burns its own set of tokens, and the alert that eventually fires is on aggregate spend, not on the tool failure that actually caused the spend. The team chasing that alert is looking at a symptom three steps removed from the cause. Context bloat is the slower version of the same problem: long-running conversational states inflate input token counts over time, and without span-level accounting there's no way to tell whether that growth is coming from retrieval pulling in more documents, from memory accumulation, or from something else entirely. Model routing gaps round out the list: sending every task, routine or not, to a frontier model inflates spend without improving outcomes, and without per-span model attribution, that inefficiency doesn't appear anywhere.

Cost and quality are coupled more often than teams assume. Maxim's guidance on observability points to semantic caching as one example, where caching cuts both token spend and latency while holding quality steady, but that tradeoff is only visible as a tradeoff when cost and quality scores are attached to the same trace record. Split them across two systems and the correlation just evaporates.

Budget governance, hierarchical spend caps and the like, is really a harness-layer concern dressed up as a finance concern. Spend caps only work as an enforcement mechanism if the trace can say which agent, which feature, or which user tier is driving the number up. Cost alerting without that attribution just fires alarms into a room with the lights off.

What the latency axis measures and why tail percentiles change the diagnosis

Latency needs measurement at the same three levels as everything else: session, trace, and span. An average computed at the trace level is close to worthless for diagnosis, because it flattens out exactly the information needed to find which span is the actual bottleneck.

P95 and P99 tail percentiles are what drive perceived slowness and timeout complaints in distributed systems generally, and Maxim's monitoring guidance treats controlling the tail as being just as important as reducing the mean. Alerting on mean latency leaves the worst cases, the ones users actually notice and complain about, completely undetected until someone happens to go looking.

Most teams underestimate non-model latency, which deserves its own attention. That same OpenObserve support-agent trace, the 8.42-second one, had 41% of its total time sitting in a Postgres lock wait. An alert scoped only to LLM inference time would have missed the actual cause entirely and sent an on-call engineer down a dead end investigating model performance. Tool I/O variability compounds this, since external tools bring their own latency profile that needs bounding at the span level, and without per-tool timing, there's no way to tell a slow third-party provider apart from a slow internal tool or a slow model call. Retrieval pipelines add their own layer on top, with index query time, embedding generation, ranking, and context assembly each contributing separately traceable latency, and each a candidate for where things actually went wrong.

Retry loops amplify latency in a way that's easy to miss without the right context. In a multi-agent scenario, a single agent retrying repeatedly can consume the majority of total answer time, turning what looks like a slow run into a retry-attribution problem. A latency alert without retry-count context treats that as a slow agent, when the real problem is upstream, in whatever caused the retries in the first place.

Streaming is a legitimate mitigation for perceived latency, emitting tokens to the user while background steps keep running closes the gap between when a response starts and when it fully completes, but measuring whether it's actually working requires first-token timing captured at the span level. Without that, streaming's benefit is assumed rather than shown.

None of this stays contained to latency alone. A latency spike caused by a retry storm is also, by definition, a cost event. A latency spike caused by context length growth may be pointing at a memory management problem that will eventually show up as a quality issue too. Latency alerts that don't cross-reference cost and quality signals are, at best, half a diagnosis.

What the quality axis measures and why it is the hardest to alert on

Quality is not a single score attached to the final answer. It needs metrics at three levels, and each level catches a different class of failure. At the step level: tool selection accuracy, tool argument correctness, planning quality, step-level faithfulness, reasoning coherence. At the trace level, the run either completed its stated objective or it did not, either followed policy or did not, and either held onto context correctly across the turns inside that one run or did not. At the session level, quality drift across an entire multi-turn conversation is the concern, and the failure usually surfaces around turn five, not turn one, by which point the damage has already compounded.

Semantic failures are harder to alert on than infrastructure failures for a specific, structural reason: a tool can return a server error, and the agent, instead of surfacing that failure, fabricates a plausible-sounding response built on top of it. No exception gets raised. Latency looks normal. Cost looks normal. The only thing that catches it is a faithfulness evaluator scoring the output as degraded, because everything downstream of the failure looks, on paper, like a healthy run. FutureAGI's taxonomy of failure modes classifies this specifically as a no-error-handling subtype of tool error, and it's a genuinely hard one to catch with anything short of an evaluator looking directly at the content.

LLM-as-judge evaluation is the main mechanism teams use to score faithfulness and context match, usually alongside statistical and human evaluators for cross-checking. The OpenObserve trace example attaches a faithfulness score of 0.91 directly at the span level, which is the right instinct: quality as a first-class attribute of the span, sitting alongside latency and cost, not bolted on afterward as a separate report.

Multi-agent systems introduce a cascading version of this problem. A single hallucinated fact, produced by an orchestrator and passed downstream to three specialized sub-agents, can produce three separately coherent, separately wrong answers, with no exception raised anywhere along the chain. A TUM survey covering 55 papers on trajectory analysis for failure attribution and system enhancement in LLM agents highlights the depth of open challenges in tracing quality failures across agent runs. Spotting that three answers were wrong is easy. It's tracing all three back to the one hallucinated fact at the origin, which requires span-level quality scores rather than an aggregate score computed once at the trace level.

Anomaly detection is the mechanism that catches what threshold alerts miss here: automated surfacing of quality drift, new categories of failure that hadn't been seen before, prompt-injection patterns, timeout spikes. Without it, quality degradation accumulates quietly in production long before any fixed threshold trips an alert.

And again, this axis doesn't stay isolated either. A quality drop caused by context truncation under a token budget is really a cost-driven quality failure wearing a quality-shaped costume. A quality drop caused by a schema mismatch is a tool-layer failure that shows no cost signal and no latency signal at all. Quality alerting without trace context routinely sends teams to fix the wrong layer.

How failures cross axes in production, and why single-axis alerts consistently misattribute them

Knowing that a run failed is close to useless without knowing which layer caused it. A postmortem review spanning two years at a major retailer found a persistent attribution error rate hovering around 10%, where the diagnostic model blamed technologies simply because they were mentioned somewhere in the incident thread, not because they had any actual causal role in the failure. That's not a small margin of error when it's driving engineering time.

Four cross-axis patterns show up repeatedly, and each one fools a single-axis alert in a slightly different way.

Tool schema drift presents as a pure quality failure. A field name changes on the backend, the validator on the agent side partially accepts the old name anyway, the backend returns an empty or ambiguous result, and the agent explains the outcome confidently regardless. No latency spike. No cost spike. Just a quality signal, sitting there alone, pointing everyone toward the prompt when the actual fix is schema enforcement at the tool boundary.

Retry storms present as a latency and cost spike with no quality signal attached at all. A tool call fails, the agent retries with the same argument shape it just tried, burning tokens and wall clock time on each attempt, and somewhere down the line the agent gives up and fabricates a plausible answer instead. Cost and latency alerts both fire. The actual root cause is a missing error-handling branch in the harness, not a model problem and not a pricing problem.

Memory accumulation presents as a joint cost and latency spike. The context window fills up with accumulated history, input token counts climb, retrieval starts taking longer because there's more to search through, and the model is forced to reason over a bloated context it shouldn't have to hold. Cost and latency alerts fire together, but the fix is a memory management policy change, not a model swap or a tool fix.

Prompt ambiguity presents as sporadic quality failures with nothing else attached. An instruction that's slightly under-specified works fine for most inputs and quietly fails on edge cases, often infrequently enough to stay under any fixed alert threshold indefinitely. Anomaly detection scanning quality scores over time is the only thing that can surface that pattern, because no single incident is ever loud enough to trip a threshold on its own.

Multi-agent architectures act as an amplifier on top of all four patterns: a root-cause error at one layer propagates through every subsequent decision downstream of it. Attribution means tracing the failure back to its origin step across the full execution graph, sub-agent by sub-agent, span by span, not just noticing that a run failed. It's tracing the failure back to its origin step across the full execution graph, sub-agent by sub-agent, span by span. Research presented at ICSE 2025 found that incorporating code-level knowledge into root cause localization improved accuracy by 28.3% over the previous leading method, which is a meaningful signal that better attribution is achievable with the right inputs. Most teams already have that signal in their traces. It's whether their alerting system was built to go looking for it.

What a unified three-axis alerting system looks like in practice

The organizing principle is straightforward to state and harder to build: alerts, service-level objectives, and incident management should run through one pipeline, so that an AI quality regression follows the exact same response path as a traditional production outage. Cost spikes, latency spikes, error rates, and eval-score drops all need to land in the same system, not three different ones that someone has to manually reconcile after the fact.

OpenObserve's product structure offers a workable model for what that looks like broken into views. An agent graph reassembles what actually happened during a run, planner, tool calls, retries, handoffs, as a graph where the broken node is something a person can click on directly, rather than a string someone has to go search for in a log. A quality score health view watches eval scores across every configuration continuously, not just for the one run currently under investigation. An eval jobs view tracks the scheduled and triggered jobs that produce those scores in the first place. And a scorers view exposes the individual evaluators, LLM-as-judge, statistical, human, as inspectable components rather than a black box that spits out a number nobody can argue with or verify.

A few alert design principles fall directly out of everything above. Alerts should fire on correlated signals rather than isolated thresholds: a cost spike that coincides with a rising retry count and a falling quality score demands a genuinely different alert, and a different investigation, than a cost spike showing up on its own. Every alert should carry span-level attribution baked in from the start, naming which agent, which tool, which step, which model version was involved, rather than reporting only that "the run failed" and leaving the rest to a manual dig. Tail-based alerting should take priority over average-based alerting, since P95 and P99 thresholds catch the failure cases users are actually living through, while mean-based thresholds smooth right past them. And quality alerts need to operate at session scope, not just trace scope, because scoring only at the level of one run misses the slower degradation that builds up across a full multi-turn conversation.

None of this closes the loop on its own, though. Attribution is a hypothesis until someone tests it against reality. The DoVer framework's approach is instructive here: a targeted edit made at the layer the trace implicates, a message, a plan, a specific tool call, followed by a rerun against real production traces, is what turns a plausible-sounding root cause into a confirmed one. Anything short of that rerun is still a guess, no matter how confident the alert sounded when it fired.