1. The first real test is production
Most agents are changed and shipped in one motion: edit a prompt, swap a model, deploy. The first time the new behavior meets a real case, it meets a real customer.
An agent's behavior moves with every change to its model, prompt, tools, or data. If the only place that behavior plays out is the live product, every release is a gamble.
An offline environment is a sealed, replayable reconstruction of the world the agent works in, where a candidate version runs against real cases before anyone deploys it.
2. Online tells you what happened. Offline tells you what would happen.
Online evaluation is watching the live agent: dashboards, user feedback, A/B tests. The signal is real, but it's slow, it's noisy, and it only ever observes the version you deployed.
It also has a blind spot. DoorDash's team showed this while benchmarking its AI code reviewer: acceptance data records what engineers did with the comments the reviewer made, but it can never record the bug the reviewer missed, or the clean code where silence was right. Optimize on acceptance alone and you build a reviewer that is confidently loud, with no way to learn what it failed to see.
Offline inverts all of it. Any candidate version can run, not just the deployed one. The same scenario replays identically, as many times as you want. Outcomes are known in advance, so misses are visible. And failure costs nothing.
Neither replaces the other. Online is field observation; offline is a controlled experiment. A team with only the observational half can watch a number move without being able to test why.
3. What an offline environment actually is
Three reconstructions, running together:
- The data, as it was. The environment is seeded with a snapshot of the state the agent saw: inventory, account, ticket history, codebase.
- The tools, made deterministic. Real APIs and databases are mocked with stand-ins that give the same answers every run. If the world answers differently on Tuesday than Monday, you can't tell whether the agent changed or the world did.
- The user, simulated. For multi-turn work, a model plays the human's side, grounded in real transcripts, adapting to what the agent says.
The raw material is the trace: the full recording of a session. Intent, tool calls, data touched, outcome. It is the seed the world is rebuilt from.
A benchmark is an environment plus curated cases plus scoring.
The public benchmarks are built this way. SWE-bench: frozen GitHub repos, real issues, graded by the project's own tests. τ-bench: a fake airline with seeded databases, deterministic tool APIs, and an LLM playing the customer. WebArena: working replica websites where an agent can fail at checkout a thousand times without a single real order. OSWorld: a full virtual machine, reset before every attempt. The recipe is the same in each: rebuild the world, freeze it, replay it, score against known outcomes.
4. How a world gets built
Capture. You cannot rebuild what you did not record. Trace richly enough to reconstruct a full session; teams that skip this end up hand-writing scenarios, which test the product they imagine rather than the one users meet.
Curate. When DoorDash built DashBench, the benchmark behind its code reviewer, it started from about a thousand candidate pull requests and kept roughly a hundred. The mix is what does the work: cases with confirmed findings, to measure coverage; benign cases, to measure restraint ("a reviewer that's loud on clean code is its own failure mode"); and changes later reverted or hotfixed, which measure whether a candidate would have caught the regression that actually shipped.
Reconstruct. Seed the data, mock the tools, stand up the simulated user. DoorDash's chatbot simulator uses an LLM customer that adapts in real time, because a scripted user can only test the conversation that already happened.
Freeze. Containerize and version it. When a score moves, you want one candidate explanation (the agent changed), not three.
5. Deciding what "right" means
Scores are only as good as labels, and every label source is fallible. Humans miss real issues and fatigue fast. Production feedback encodes urgency and workflow, not correctness. Model judges scale but need calibration.
DashBench's answer: triangulate. Human annotations, production feedback, and a model judge are compared; disagreements get re-reviewed by hand; resolved cases calibrate the judge. No single source is treated as ground truth; the label set is something the team builds and maintains, not something the data hands over.
6. Repetition is the point
You don't run a benchmark once. Agents are stochastic: the same version, same case, same frozen world produces different runs, and a single run understates real coverage. Run repeatedly and aggregate.
And never trust a single number. DashBench scores severity-weighted precision and recall, false findings, latency, and cost as independent axes. The results show why: the staged production reviewer found 504 real findings at 53.6% weighted recall for $3.91 per review; a single-pass baseline found 164 at 30.7% for $0.75. Neither is better in the abstract. One buys coverage, the other buys economy, and which one you want depends on what the review is for.
7. How teams use one
- Swap components with the world held still. DashBench replayed identical PRs across a dozen scout/reviewer model mixes. No configuration won every axis: one led coverage, another precision, the cheap ones held the cost floor. Production could never tell you that; it runs one version at a time on traffic that never holds still.
- Catch regressions before users do. Every diagnosed production failure becomes a permanent case. The same mistake cannot ship twice, and the case set accumulates every failure mode the product has shipped.
- Gate releases. A candidate replays the bench, stands against the baseline, and ships only if quality held within budget.
- Evaluate on your world, not the leaderboard's. Every new model tops SWE-bench at launch, but SWE-bench is a dozen open-source Python repos and τ-bench's airline doesn't exist. A leaderboard ranks models on somebody else's work. Only your own benchmark ranks them on yours.
8. A worked example: an agent that touches other people's systems
Consider a workflow automation platform whose agents run automations across thousands of integrated apps on customers' behalf: close the ticket, update the CRM, send the email. Here evaluation hits a wall that has nothing to do with model quality. You can't replay production traces against real connections, because replaying replays the side effects: a test that deletes a record deletes a customer's record. And hand-written test sets miss the silent regressions born of odd accounts, unusual configurations, and third-party APIs that changed underneath you.
The fix, which platforms of exactly this shape now run internally: sample real workflows from traces (with privacy controls), rebuild the relevant slice of the world as fake accounts and faithful connector state, and let the candidate agent execute its tool calls for real against the fakes.
Grading barely needs a judge, because action-taking agents leave evidence a script can check: right app, action, and parameters; valid, correctly sequenced calls; the intended final state; nothing unnecessary or unsafe. These checks are cheap to run and return the same verdict every time. Results stand against the current production version, and new failures join the case set permanently.
Third-party APIs have rare quirks, and an agent that once found the workaround will not remember it months later. Each quirk becomes a case, which is how that knowledge survives a model swap.
9. Beyond replay: one seed, a family of tasks
The most sophisticated practice treats a case not as a recording but as a generator.
A blockchain-intelligence firm we work with runs investigations across an entity graph, internal tools, and external data vendors. Because the environment is built on that graph plus synthetic vendors, one real investigation expands into a family of variants, each probing a different failure mode:
- Baseline. All evidence present. Right tools, sensible order, supported conclusion?
- Time-travel forks. Fork a real failed investigation at the turn it went wrong; replay from there. Each becomes a pinpoint regression test.
- Evidence withheld. The right answer is "not enough evidence," qualified confidence, or escalation. Does the agent notice?
- Evidence poisoned. Stale or contradictory inputs. Does it weigh sources, resist anchoring, revise?
- Graph perturbed. Swap an entity, add a relationship, open a second plausible path. One seed becomes dozens of controlled cases.
- Alternate paths. Distinguish required outcomes from acceptable routes, wasteful detours, and unsafe actions, so a different legitimate route isn't punished.
- Tool weather. Synthetic vendors serve empty, partial, stale, and conflicting responses, timeouts, schema changes, on demand. Production will eventually serve all of these; here you schedule them.
- Long horizons. Many turns, many tools: keep the objective, compress findings, don't repeat work, recover from turn-ten mistakes.
Production supplies realistic seeds; the environment expands each into baseline, counterfactual, adversarial, and recovery variants. Replay covers what happened, and a generator covers the cases around it that have not happened yet.
10. What this buys you
Safety. Agents graduated from drafting text to doing work: refunds, records, automations, code. When that work goes wrong, the failure is a wrong action in a system of record rather than a bad paragraph. For agents with side effects, a place where failure costs nothing is not a testing preference but a precondition for changing the agent at all.
Cost. Evaluation is routinely one of the biggest line items in an agent program, because human review scales with everything you ship. Offline is a different regime. DoorDash's harness for its support assistant scores 400 conversations in about twenty minutes; the same work took over six hours by hand. Quality signal went from one employee-filed report a day to about two thousand auto-graded sessions a day, and drove an eight-point quality improvement. DashBench prices its verdicts at about $3.91 per PR and $0.82 per real finding. At that price evaluation runs on every change, rather than once at the end of a release cycle.
Confidence at the gate. DoorDash's reviewer now handles 10,000+ PRs a week at about $3 and seven minutes each, with engineers accepting roughly 60% of its high-severity findings. The benchmark came first and the adoption followed it. A new frontier model arrives every few months; with a benchmark, "should we switch?" is a replay run rather than a debate, and the failure modes a team worries about get tested on a schedule instead of discovered in production.
11. Keeping the world honest
The real world keeps moving, and a benchmark frozen at launch measures the product as it used to be. Three disciplines keep it alive: rotate cases in from recent traffic and retire stale ones; recalibrate judges against fresh human labels and watch the agreement over time; and version everything that can move a number, because the first question when a score changes is whether the agent moved or the ruler did.
12. Offline and online together
Offline and online form a funnel, not a fork. Offline is where candidates are generated, compared, and culled: cheap, safe, in volume. Online is where the survivor proves itself, and where reality supplies the next round of cases. Neither is sufficient alone.
The teams that improve agents fastest are not the ones with the largest models or the most traffic. They are the ones who can answer "did this change help?" in an afternoon, against a case set that already holds every failure the product has shipped.
For the methodology on top (seeding evals from traces, the annotation flywheel, tying scores to ROI), see Efficient Agent Evals. For the infrastructure internals, see Offline Eval Environments.