1. Why offline eval environments
Online evaluation tells you what already happened. To ship safely, you need to know what will happen before you release. That means running candidate agent versions against a faithful, offline substitute for production. That substitute is the offline eval environment, a sealed rebuild of the systems the agent meets in production, and its quality decides whether your offline signal predicts online reality.
The design goal is singular: an environment whose initial state and dynamics sit close enough to production that passing offline is strong evidence of passing online. Everything below serves that goal. It has also stopped being a private conviction: Nubank's evaluation-driven framework, published from five customer-support agent deployments serving over a hundred million users, is built on the same premise, with ideas validated offline end to end before they reach production and LLM judges held to measured agreement with human raters along the way.
2. The trace data store
Everything starts with a trace data store: production sessions parsed into a consistent schema of turns, tool calls and outputs, memory reads and writes, model calls, token cost, latency, human approvals, downstream actions, and final outcome.
Two properties make the store useful for evaluation rather than mere observability. It must be joined to business metrics: a trace you cannot connect to a cart, a resolved ticket, or a merged change cannot be tied to ROI. And it must be queryable by outcome and cluster, so you can pull "all sessions where the agent updated the wrong record" as easily as a single trace. Together, those two properties make the store the seed for everything downstream.
The failure mode we keep seeing is siloed data, not missing data. Agent behavior lives in a tracing tool, business outcomes live in an analytics warehouse, and nobody has joined them. The store earns its keep only once that join exists: a trace ID that resolves to a cart, a ticket, or a merged change, not a UUID that means something only to the tracing tool.
3. Golden traces and simulation
The store seeds two complementary artifacts.
Golden traces are curated real sessions with known-correct outcomes: a regression set in the agent's native units. They are cheap to run and unambiguous to read: a new version either still gets them right or it does not. They also go stale. The teams that keep them honest rotate the set on a thirty-to-sixty-day cycle and re-seed it from live traffic as the product drifts.
A simulation environment is the richer, more expensive instrument. It replays real inputs against deterministically mocked backends, and where interactions are multi-turn it drives them with a simulated user built from real transcripts. Because it is initialized from the production trace distribution rather than hand-written scenarios, its coverage tracks what users actually do, and its failures tend to reproduce in production.
One consumer-software team we've talked with confirmed the alternative does not work: toy and mock tasks at the offline level were either too easy or too noisy to surface real signal. The only method that produced a trustworthy read was full end-to-end simulation of real user workflows in the product itself, tracked over multiple trials.
This is the best-documented part of the stack. DoorDash's evaluation flywheel drives its simulator with an LLM that plays the customer, adapting in real time to whatever the agent says rather than following a script, and the Ask DoorDash evaluation harness shows why the investment pays for itself: a sweep of fifty scenarios at eight trials each produces four hundred scored conversations in about twenty minutes, work that previously took over six hours by hand. Amazon reports the same architecture from its own agent teams, with one addition worth stealing: the simulated user's personas are seeded from historical customer interactions, and the evaluation grades routing and orchestration decisions, not only the final answer.
The same discipline holds outside chat agents entirely. CVE-Factory's automated vulnerability-reproduction pipeline, an ICML 2026 oral, builds an isolated environment for each CVE and accepts it only once an automated test suite proves the exploit fires against the unpatched code and stops firing once the fix is applied. Checked against human expert reproductions it reaches ninety-five percent solution correctness and ninety-six percent environment fidelity; on a separate set of the newest vulnerabilities it verifies about two-thirds.
4. The eval engine
A layered eval engine scores candidate versions, cheapest signal first.
Deterministic checks come first: schema validity, policy conformance, whether the correct system was updated, whether an irreversible action had approval. They are fast, free, and unambiguous, and they should catch as much as possible before any model runs. For a code-review agent, the strongest deterministic signal we know is high-urgency bug count: critical findings block the pipeline, low-risk changes auto-approve.
Model judges handle what determinism cannot: groundedness, tone, whether the response actually answered the question. They are calibrated against human labels and tracked for drift.
Agent-as-judges handle the trajectory. They use tools to verify claims against ground truth, re-run the actions the agent took, and check the final state of the world, not only the final message.
The most rigorous version of this we've seen chains four layers together: automated capture of every step, a human reviewer, a QC pass, and a final auditor, with the automated layer alone running well over a dozen programmatic checks per task. The output that matters is not whether any single trajectory looks clean. It's how reliably the agent notices it's off course and recovers.
Spotify's background coding agents show the shape of it: deterministic verifiers run first, then an LLM judge holds veto power over the pull request. Across thousands of agent sessions the judge vetoes about a quarter, and the agent course-corrects half of those. Spotify is candid that it has not yet built evals for the judge itself, which is the honest state of most judges in production and the reason the deterministic layer ahead of it carries the load.
The same suspicion should point at the eval engine itself. DoorDash benchmarks its code review agent by replaying historical pull requests and scoring weighted precision, weighted recall, not-real findings, recall on high and critical issues, latency, and cost as independent axes, on the grounds that a system can win one axis and lose another in the same run. Their conclusion doubles as the thesis of this section: "better" means nothing until you say better at what.
5. Judge and dataset versioning
An eval result is only meaningful relative to the measurement that produced it. So the eval substrate versions its own artifacts: judge prompts, rubrics, the golden set, the simulator, and the mocked backends. When a number moves, versioning is what lets you attribute the move to the agent rather than to a changed judge or a drifted dataset. Judges are re-calibrated against fresh human labels on a schedule, and eval sets are refreshed as production drifts.
This one has burned people quietly. At one large technology company, product managers were given easy tooling to define their own eval metrics and kept rewriting the definitions. Leadership watched the numbers move and tried to diagnose agent quality, but the instrument itself had changed underneath them, and nobody could say which. Version the metric definitions, not only the scores; otherwise a trend line is comparing two different rulers.
6. The CI/CD gate
The engine feeds an agentic CI/CD gate. Cheap checks run on every change. The full offline environment runs on every candidate version. The gate is expressed in ROI terms, severity-weighted task success within a cost and risk budget, and every production failure that escapes becomes a new regression case. Rollout stays versioned and reversible, so a bad release is a rollback rather than an incident.
One version of "safe to ship" we've seen made concrete, at a payments company: every agent action passes through a small gateway that checks a threshold. Actions under a couple hundred dollars with low fraud risk auto-approve. Anything above that or riskier routes to a human, and the credential the agent uses to act is scoped to a single task and expires within the hour. Uber's own writeup of its ML deployment safety practices describes the same instinct at the infrastructure layer: continuously measuring the health of a rollout and triggering an automatic rollback the moment a failure signal appears, rather than waiting for a person to notice.
7. Governance, cost, and access
Because this infrastructure runs real prompts against real data, it needs the same governance as production: scoped credentials for the mocked and live integrations, PII controls on stored traces, audit logs on who changed which judge or dataset, and cost attribution per eval run. Two lessons worth designing in early: at least one global bank's governance bars using the same model as both judge and actor, and short-window response caching keeps repeated eval questions from being paid for twice. Without cost visibility it becomes its own source of runaway spend, and without access controls its own compliance exposure.
Built well, the offline eval environment is the substrate the whole flywheel runs on: the durable memory that turns every production session into a permanent, ROI-linked test.
For the methodology and the human-in-the-loop side, see Efficient Agent Evals.