1. The real problem: knowing whether the agent got better
Enterprises are past the first wave of AI experimentation, and the serious question has changed: not whether an agent can do the task, but whether you can make it reliably better, cheaper, and safer, release after release.
That turns out to be an evaluation problem, and not the narrow kind. A single judge prompt does not survive contact with a real agent. Behavior is shaped by prompts, models, memory, tools, retrieval, runtime state, permissions, and the product workflow around all of it. Change one and the whole trajectory moves. A benchmark fixed at a point in time cannot see that, and neither can a rubric written once.
Much of what follows we learned the hard way, building and evaluating agents inside a ~$100B, Fortune 500 enterprise. In that world, a plausible-sounding answer is still wrong when the store is closed, the item is out of stock, or the cart total is invalid. A fraction of a percent of regression is measured in real dollars.
The lesson was that evaluation had to run continuously, feeding the gate rather than waiting at it. Without a cheap, repeatable way to answer whether a change helped, every release is a guess.
2. Evaluate in trace space, seeded from production
A single prompt-and-response pair hides almost everything that matters: which tool the agent called, whether the tool returned correct data, whether memory polluted the result, whether a later turn repaired an earlier mistake, whether the task actually succeeded. The unit rich enough to evaluate an agent is the trace: the full record of a session.
So the first principle is simple. Build your evals in trace space, and seed them from production. Your best eval set isn't synthetic, it's your own successful sessions, failures, abandonments, escalations, tool errors, and human overrides: the real distribution of what your users actually do.
Synthetic data still earns its place, but after trace analysis, not before. Use it to cover gaps, stress edges, and rehearse low-frequency risks. It is a poor seed distribution when real traffic exists, and a reasonable one when nothing has shipped yet. If it is, you are optimizing against a world nobody lives in. We watched this fail in numbers: at that enterprise, sixty percent of the assistant's pre-launch eval budget went to synthetic evals with almost no predictive power, while a fifth of real queries turned out to be discount and support questions nobody had designed for. A staged rollout surfaced in days what months of synthetic evaluation had missed.
Shopify's Flow team shows what the right order looks like. When it taught an agent to generate automation workflows it had no production conversations to learn from, so it worked backward from workflows merchants had already built and validated, reconstructing the requests that would have produced them. That reverse-engineered set is synthetic, and Shopify says so. What makes it defensible is the anchor: every example descends from a real artifact rather than from an imagined user. The three-hundred-example benchmark it is scored against was hand-built separately, and judges semantic correctness with a model and syntax with a program. Anchor to real artifacts wherever they exist, and be explicit when they do not.
The catch: raw traces are hostile to human reading. A single session can be dozens of tool calls, nested JSON, retries, and truncated context. Before traces can become evals, they have to become legible.
3. Condense traces into readable narratives
The highest-leverage, least glamorous investment in an eval program is turning raw traces into readable narratives.
A good condensed trace reads like a transcript a domain expert can skim in fifteen seconds: what the user wanted, what the agent decided, which tools it called and what came back, where the turn went wrong, how it ended. Tool payloads are summarized, not dumped. Redundant retries are collapsed. The decisive moment is highlighted.
This is not cosmetic. Annotation throughput is the bottleneck of every eval program, and readability is its single biggest lever. When reading a session takes fifteen seconds instead of five minutes, the same review budget covers far more of them, and labeled data is what calibrates every judge downstream. Tooling moves the same lever: one data-annotation company cut labeling time from six hours per task to two by moving annotators into an agentic IDE, and within weeks nine in ten annotators had requested licenses.
4. Explore traces like a reasoning model
Once traces are legible, the next question is how to find the failures worth caring about across thousands of sessions. Manual spot-checking does not scale, and dashboards only answer questions you already knew to ask.
The pattern that works is to explore traces the way a reasoning model explores a codebase: interactively, hypothesis-first. You ask a question in plain language: "where did the agent recommend an item that was not in the catalog?" A reasoning model then retrieves and reads the relevant traces, clusters what it finds, proposes a root cause, and pulls the exemplar sessions back for a human to confirm.
That loop (reason, retrieve, read, cluster, hypothesize, verify) turns trace exploration from manual archaeology into a directed search. You go from "something feels off this week" to "here are 43 sessions where one tool returns stale inventory, worth an estimated six figures in lost carts" in minutes rather than days.
A human stays in the loop at the end. The model narrows the candidates. A person confirms the real failure. Each confirmation is a label, and each label feeds the flywheel.
5. The human-annotation flywheel
Everything above composes into a single loop, and the loop compounds:
- Production emits traces.
- Traces are condensed into readable narratives and explored to surface failure clusters.
- Humans annotate the exemplars: the ground truth the rest of the loop calibrates against.
- Those labels calibrate judges, deterministic and model-based, and feed prompt and policy optimization.
- The improved agent version is gated and released.
- Release produces new traces, and the loop turns again.
Human annotation isn't about labeling everything forever. The goal is a calibrated, self-improving system: enough ground truth to trust your judges, enough signal to optimize your prompts. A well-run flywheel needs a shrinking share of human labels over time, because the judges get trustworthy and the failures get rare.
Two outputs matter most. The first is optimized prompts and policies: annotation shows you exactly where the agent misreads intent, and each finding becomes a targeted prompt change rather than a guess. The second is agent performance itself: the same labeled failures become regression cases, so an improvement cannot silently revert.
DoorDash's Ask DoorDash team has published what this loop looks like at scale: rubrics that define success, a transcript builder that reconstructs sessions, a simulator for repeatable offline runs, and a judge calibrated against human labels. The part worth envying is the signal it unlocked. Quality feedback went from roughly one employee-submitted report a day to two thousand auto-graded sessions a day, and that visibility drove an eight-point improvement in agent quality score. That is the loop working: more sessions graded, faster, with the score moving.
6. Agent-as-a-judge
A single model-as-judge prompt is a fine start and a poor finish. The strongest judges for agentic work are themselves agentic: they use tools to check claims against ground truth rather than eyeballing prose.
An agent-as-judge can re-run the tool the original agent called and compare results, check whether the final state of the system is correct, verify a citation against its source, or confirm that an irreversible action had the required approval. It evaluates the trajectory and the world the agent touched, not only the final message.
One version of this we've seen work end to end, at a payments company, puts a small gateway between the agent and the action: validate the agent's credential, read the delegation chain to confirm who authorized the request, check the requested amount and ownership against the live database, inspect the action in real time, then log the decision and issue a one-time credential for the downstream system. The agent never holds a real key. The judge re-derives the refund's validity from the same systems of record the agent was supposed to check, rather than taking its word for it.
Two disciplines keep judges honest. First, calibrate them against human labels and track the agreement over time with a real statistic, Cohen's kappa, treating a drifting kappa as a ship blocker. A judge whose agreement with humans decays quietly becomes a silent source of bad ship decisions. Second, prefer binary, severity-weighted verdicts over 1-to-5 scores. "Did it complete the task," "did it claim something unsupported," "did it escalate correctly": these calibrate cleanly and resist gaming in a way vague ratings never do.
This is visible in what teams have published. Netflix's writeup of its synopsis-evaluation system describes a golden set of roughly six hundred examples built by writer consensus, judges reaching eighty-five percent agreement with those writers, and judge quality that correlates with streaming metrics. The judge earned trust the way a human rater would: by agreeing with the experts, repeatedly, on record.
And the ceiling is higher than parity with humans. DoorDash runs juries of models over its food metadata whose annotation accuracy came out roughly twenty percent above typical human reviewers, and the jury's verdicts do double duty: failures feed context optimization, and the accumulated labels fine-tuned a model to frontier quality at a tenth of the inference cost. A well-built judge does more than gate: every verdict it emits is another labelled example.
7. Tie every eval to ROI
An eval disconnected from a business outcome proves nothing. Every agent needs a primary business metric, a primary quality metric, a primary cost metric, and a primary risk metric, and the eval exists to protect or move the business one.
The pattern repeats across every enterprise we work with, in different clothes:
- At that same enterprise, the ROI unit for the shopping assistant was never "good answer," it was successful task completion: cart creation, checkout conversion, basket size, and cost per successful cart. In production the gates were concrete: sessions running past forty seconds, carts created, orders completed, and an average order value lift of roughly twelve percent against each user's own baseline. For its AI code reviewer, the unit was real, severity-weighted findings per dollar, not raw comment volume. Production acceptance flatters loud reviewers and cannot see what they missed.
- At a high-growth go-to-market automation platform, offline toy tasks predicted little, because outcomes depend on live customer data and multi-step execution. End-to-end simulation predicted far more, and the metric was conversion and credit-spend efficiency, not "agent score."
- At a national home-services field operation, the hard part was never the agent, it was naming the business metric. A technician follow-up agent can execute the flow flawlessly and still be worthless if retention, routing, and throughput stay flat. That number has to come from the business owner, not the ML team.
- At an AI data-operations company, serious evals were a program, not a script: VM environments, calibrated rubrics, multi-level human audit. The ROI unit was benchmark quality and reusable eval infrastructure.
- At a global on-demand logistics platform, internal automations were justified by risk avoided, not model cost saved. One three-week build protected two to three million dollars a day of revenue at risk, an axis no per-token cost model would ever surface.
- At a real-time voice platform, containment and resolution looked fine on the surface, but the gating metric was latency and reliability. The eval had to include first-audio latency and interruption handling, not only transcript quality.
The clearest public version of this discipline is architectural choices measured in business units. DoorDash's writeup of its agent memory system does not argue that memory is good design. It reports that sessions backed by memory converted to checkout at a roughly twenty-four percent higher relative rate for grocery, fifteen percent for restaurants, and misread user intent a third less often. Memory usually gets pitched as an architecture decision, but measured properly it turns out to be a conversion lever.
The through-line: the business metric comes first, and the data work is what connects a raw trace back to it.
8. Scope the evaluation end to end
Before writing a single check, scope the evaluation the way you would scope a product. Three questions do most of the work.
What is the unit of success? Not "helpfulness," but the concrete, ideally binary outcome the business cares about: a valid cart created, the correct system updated, a real bug caught, a call contained.
What in the trace proves it? Name the exact fields (tool outputs, final state, downstream events) that let you decide success without a human re-litigating the whole session. If proving success requires data you do not capture, your first work item is instrumentation, not evaluation. One answer-and-analytics agent we've seen handles this cleanly: a rotating bank of fifty to a hundred questions with known correct values, the kind a finance team already tracks by hand, so an answer is checked against the number itself instead of judged for how confident it sounds.
What is the cost and risk budget? A cheap smoke check on every change and a deep, expensive simulation on every version are different instruments. Decide which failures you are willing to catch late.
Getting this scope right is the difference between an eval set that drives ship decisions and a growing pile of checks nobody reads.
9. From production traces to offline evals
The payoff of trace-space thinking is that production becomes the seed for a durable offline eval environment.
Golden traces are the first artifact: a curated set of real sessions with known-correct outcomes, drawn from production and labeled by humans, that a new agent version must still get right. They are your regression suite in the agent's native units.
Golden traces have a failure mode subtler than staleness: mistrust. We have watched the same version of it twice, at two unrelated companies. A team spends months building an eval suite, then stops shipping because every change trips a regression. Eventually someone discovers the regressions were artifacts of a flawed eval, not real quality loss. Both times the fix was the same: strip the eval back to nothing, ship by feel until the product is right again, then recalibrate the suite to that known-good state instead of the other way around.
The richer artifact is an offline eval environment initialized from production traces. Instead of scripting synthetic scenarios from scratch, you take the real distribution of sessions you have already captured and use it to initialize the data store the offline environment runs against: replaying real inputs, mocking the backends deterministically, and simulating users where the interaction is multi-turn. The closer that environment's initial state is to production reality, the better its offline signal predicts online behavior.
We go deep on the internals (the trace data store, the simulation environment, the eval engine, and the CI/CD wiring) in a companion piece, Offline Eval Environments.
10. Agentic CI/CD
The last piece is treating agent releases like software releases, with a gate that understands agents.
Agentic CI/CD makes the eval flywheel the release pipeline. A change proposes a new agent version. Cheap deterministic and smoke checks run first. A version that fails them never reaches the expensive stages. Surviving versions run against the golden traces and the offline environment, scored by calibrated deterministic checks, model judges, and agent-as-judges. The gate is expressed in ROI terms: not "did the average score go up" but "did severity-weighted task success improve without a cost or risk regression." Every production failure that slips through becomes a new regression case, so the same mistake cannot ship twice. Rollout stays versioned and reversible.
None of this is hypothetical anymore. The cheap-checks-first stage already runs at industrial scale: GitHub gates Copilot changes behind more than four thousand offline tests, most of them running inside its automated CI pipeline before anything reaches production. The feedback half of the loop has a public articulation too: Spotify's experimentation team describes evals and A/B tests as a funnel rather than a fork, where evals run first so that whatever reaches live testing has already earned its slot, and experiment outcomes flow back to sharpen the evals that screen the next change. Neither instrument replaces the other.
Even the vendors are formalizing it. Salesforce now describes an agent development lifecycle in which evaluation criteria and test suites exist before the agent does, and models stay swappable behind them.
One discipline makes this work: version everything that can move a metric. Prompts, tools, memory, judges, rubrics, datasets, the simulator itself. Without that, you cannot tell whether the agent improved or the ruler changed.
Done well, agentic CI/CD is the opposite of a release tax. Every change arrives with evidence, so the team ships faster.
11. Where we are taking this
The advantage does not come from the size of an eval suite. It comes from evaluation that compounds: every production failure becomes a future test, every release is a controlled experiment, and every prompt, tool, and judge change is attributable to a number the business cares about.
That is what we are building at Surface Area.