Behavioral Logs vs. Evaluation Scores as Improvement Inputs

An evaluation score answers one question: did the agent get the right answer? A behavioral log answers a different one entirely: what did the agent actually do to get there? These are not two versions of the same information. A pass rate can climb while the underlying process gets worse, and an engineer staring only at that number will never know it.

Why evaluation scores and behavioral logs answer different questions

The standard evaluation workflow is familiar to anyone who has shipped an agent into production. Run it against a set of held-out tasks, check whether the final answer matches expectations, report a pass rate. That number is useful. It tells you how the system performs on tasks you already know about, measured after the fact, once everything the agent did is finished and collapsed into a single verdict.

What it doesn't tell you is almost everything that happened before the verdict arrived. Before landing on the right tool, did the agent loop three times? Did it call the wrong API, notice the error, and quietly recover, burning tokens and latency along the way, even though the final transcript looks clean? Did a user get frustrated across four turns before finally getting what they needed, in a way that never appears in the record because the task technically "passed"? Tracing logs what happened, step by step. Evaluation scores whether those steps, taken together, produced the correct final decision. The two are complementary, but treating them as interchangeable is where a lot of agent debugging goes wrong.

What a behavioral log contains that a score discards

Agent observability, in the operational sense, means capturing every step an agent takes: which tool it selected, what arguments it passed into that tool, what the model returned, what it read from and wrote to memory, how its internal state shifted, and which branch it chose when more than one path was available. All of that gets structured into a single hierarchical trace, the kind that can be replayed step by step rather than read as a summary.

That structure means you can reconstruct, in order, what the agent did and why, with the actual inputs and outputs sitting at each step rather than inferred after the fact. A score can't do this. A score takes that entire path, every branch and every recovery and every wasted call, and reduces it to pass or fail. Everything in between gets thrown away.

What a trace actually lets you interrogate is more specific than "did it work." Was there a coherent plan before the agent started acting, or did it start improvising immediately? Were the right tools invoked at the right moments? Were the arguments passed into those tools sensible given the task? Did each intermediate decision follow logically from the context and the results the agent had already gathered? Did the agent's behavior stay anchored to the user's actual goal as new information came in, or did it drift? And for systems built from multiple components, did each sub-agent, retriever, or downstream call do its job the way it was supposed to?

None of these are answerable from a score. All of them are answerable from a trace.

The origin point of compounding agent failures across steps that scores miss

Failures in agentic systems don't stay where they start. A weak plan formed in step one, a wrong tool call in step two, a bad assumption baked in early, none of these remain contained. They cascade. A small misunderstanding or a broken assumption can compound across many reasoning loops before it ever produces a visible failure, and by the time it does, the failure looks nothing like its cause.

Non-determinism makes this worse. The same input, run twice, can take a different path each time depending on memory state and tool outputs. A single passing test tells you almost nothing about how reliable that path actually is. This is a meaningfully different failure mode than traditional software, where a bug is usually traceable to a specific line of code. Agent failures live inside long, language-heavy execution trajectories, and the thing that broke is often buried many steps upstream of the thing that finally went wrong.

A score sees only the endpoint. It has no mechanism for pointing back at the origin.

The three levels of evaluation requiring trace data

Evaluation for agents actually operates on three separate levels, and each one answers a different question.

End-to-end, or outcome-level evaluation, asks whether the task succeeded. It's a binary pass or fail against the final goal state. It's blunt by design: it can tell you that something broke, but it says nothing about where or why. Trajectory-level evaluation asks whether the agent took a reasonable path to get there, scoring the intermediate steps, the tool choices, the reasoning quality, whether arguments were correct, whether decisions came in a sensible order. This is where most of the real diagnostic signal actually sits. Component-level evaluation goes a layer deeper still, asking which specific retriever, tool, or sub-agent broke down. That requires scoring individual spans: was the right tool selected, was the retrieved context actually relevant, did the planning step produce a coherent strategy.

Scores, on their own, are most naturally suited to level one. Levels two and three require access to the trace itself: a platform that logs traces without scoring the individual spans inside them has built agent tracing, not agent evaluation. That difference is not cosmetic. Knowing something failed is not the same as knowing what to fix. Metrics like tool selection accuracy, planning quality, step-level faithfulness, reasoning coherence, argument correctness, plan adherence, and handoff quality between agents can only be computed with trace data in hand. There's no way to back them out of a pass rate.

Where failures originate: evidence from root-cause attribution research

Attribution turns out to be genuinely hard, precisely because failures compound. The step where a score registers "fail" is rarely the step where the actual error occurred, and research into root-cause attribution for multi-agent LLM systems backs this up directly.

AgenTracer, a framework built to automate this kind of attribution using counterfactual replay and reinforcement-learning-based training, found that existing LLMs perform below 10% accuracy on failure attribution tasks when left without this structured approach. Integrating AgenTracer produced performance gains of up to 14.2% in self-correcting agentic systems, a gap that says something important about how far off intuition is when it comes to locating the actual point of failure.

AgentDebugX's DeepDebug component takes a related approach, running multi-turn root-cause diagnosis through global trajectory understanding, structure-guided investigation, and cross-examination of the trace. On the Who&When benchmark, it reached 28.8% exact agent-and-step attribution accuracy using a 9B-parameter backbone, compared to 21.7% for the strongest single-pass baseline available. The benchmark landscape reflects how seriously this problem is being taken: alongside Who&When, there's TRAIL, TELBench, and AgentRx, and Who&When Pro extends the dataset further, to more than 12,000 labeled trajectories spanning different agent frameworks, domains, and modalities. None of this work would be possible starting from scores alone. It all depends on trace data as the raw material.

Harness layers that show up in traces and are the actual site of most failures

The harness is the runtime layer wrapped around a model, including the prompts, the tool interfaces, how context gets constructed, how state gets managed, the middleware, the recovery logic, and the evaluation hooks. It's everything surrounding the model that isn't the model itself, and increasingly, it's where the actual engineering work happens.

Harness design today reaches well past prompt templates. It covers workflow design, loop engineering, permission controls, persistent state management, and evaluation infrastructure, closer in spirit to runtime and systems design than to prompt writing. Using the identical model, identical data, and identical prompt, a programming benchmark's success rate moved from 42% to 78% with only one variable changed, the runtime environment surrounding the model, in one documented case. Nothing about the model itself was touched.

A jump of that size appears clearly in a score. What the score cannot do is say which layer of the harness caused it: the tool interface, the state management, or the recovery logic. That answer lives only in the trace.

Closing the loop between observed failure and validated fix through production traces

Catching a quality problem in production only matters if there's a mechanism to act on it, and that mechanism has to be systematic rather than ad hoc. The loop looks like this in practice: edge cases discovered in live traces get converted into new regression tests, patterns from production traffic improve the accuracy of the evaluators themselves, and behavioral signals surfaced from real usage expose failure categories that no synthetic benchmark thought to test for.

None of this works without keeping the right traces around, and that means being deliberate about sampling. Tail-based sampling, where every failed, expensive, or anomalous trace is kept in full and the happy path gets sampled aggressively, is close to a prerequisite here. Head-based sampling, applied at high volume, tends to drop exactly the traces an engineer needs most the moment an incident actually happens.

Instrumentation matters too. Building traces on OpenTelemetry's semantic conventions for generative AI provides a consistent instrumentation foundation across tooling. A team can swap out or stack evaluation platforms later without rewriting how the system is instrumented.

What to look for in a trace-based improvement platform

Most LLM evaluation tooling was built for single-turn prompt-response pairs or simple chains. Pointed at a multi-step agent, that kind of tooling logs the trace but stops there. It doesn't score whether the agent made the right call at each individual step. The diagnostic work still lands entirely on the engineer reading the transcript by hand.

Agent tracing shows what happened, and agent evaluation judges whether what happened was correct. A platform that only does the former hasn't solved how to diagnose failures. It's just made the raw material easier to look at.

A handful of criteria separate platforms that actually support trace-driven improvement from ones that just visualize logs. Span-level evaluation matters most: can individual agent steps be scored on their own, independent of whatever the final output turns out to be? Agent-specific metrics matter next: the platform needs scoring built for agentic behavior specifically, tool selection accuracy and planning quality among them, rather than metrics borrowed from single-turn chatbot evaluation and stretched to fit. Anything short of that leaves the hardest part of the job, tracing a visible failure back to the step that actually caused it, undone.