An agent needs an eval

Most agent benchmarks grade a model choosing among the benchmark's tools inside the benchmark's loop. That answers a question about the model. It says nothing about the harness we wrote around it: our sandbox, our catalog ranker, our tool schemas, our prompts. Those are the parts we change every week, and they were the parts we could not measure.

So we run the real agent against Zapier's AutomationBench, and we grade what comes out the other end. The model is one component under test, not the test.

The eval

In production the agent reaches Pipedream over the network to find and run actions across a user's connected apps. Pipedream is somebody else's service, addressed by a URL.

The eval has a world state server that plays exactly that role. It holds a simulated world state of accounts, contacts, and emails, and the agent reaches it through the same path it reaches Pipedream: run_action to Rails, Rails to the world. The only difference is a URL in an environment variable.

The world state server is a separate Python repository, not a directory in the Rails app, because it is a third-party benchmark with zero Rails coupling and it should stay that way. It vendors AutomationBench frozen at one commit, MIT licensed, and imports it rather than forking it.

world state server (Starlette, :8099)          Note8 (Rails)
  GET  /tasks     list the fixtures        Eval::Run wipes the box, boots a conversation
  POST /reset     boot one, return prompt  AgentSessionJob dispatches the turn
  POST /search    BM25 over 549 tools      the box runs the agent loop
  POST /execute   mutate the world         run_action proxies back through Rails
  POST /score     run the rubric           Eval::Report aggregates

Each run gets its own session state, keyed by conversation id. Scoring is code, not a model judging another model: judge models are great but code based evals are the way to go when a company open sources a benchmark on api calls. The rubric asserts facts about the final world state, and the assertions either hold or they do not.

Why the measurement is honest

The property that makes this trustworthy is one we built for safety and got a second use out of.

The box can only reach two hosts. It has native tools, a shell, a Python REPL, a filesystem, and it can think and compute with all of them freely. But the only channel that can change the scored world state is run_action through the Rails proxy. The agent cannot reach around the harness to make an assertion pass. It cannot fetch the answer. Egress isolation makes shortcutting a task structurally impossible rather than merely unlikely.

Swapping the world state is then a config change. Point the URL somewhere else and the same agent runs against a different world that has the search and execute .endpoints for search_actions(tools) and run_action.

What the rig taught us

Here is the entire task the agent fails hardest on:

Add Director-level contacts to the Q2 Executive Outreach campaign (ID: 701xx000001EXE1). Make sure to follow our enrollment guidelines.

Two sentences, and the guidelines are in neither of them. They sit in a Google Sheet the agent has to go find, and they layer. Skip Healthcare and Government accounts. Skip any subsidiary whose parent company is in a restricted industry, whatever the subsidiary's own industry field says. Take "Director" but reject "Associate Director" and "Director Emeritus". Read every contact record and its account record for opt-outs and outreach holds before enrolling. Four rules, decoy rows sitting beside them in the same sheet, and each rule applies to each candidate.

Six real runs in, the findings split cleanly into wins and nulls.

The dominant failure mode turned out to be behavioral rather than technical. The agent looked for referenced data in its own memory, failed to find it, and stopped to ask the user, instead of searching the connected apps and acting. One directive about working autonomously fixed most of that.

On an exclusion task the agent gets inclusion perfectly right and exclusion half right, run after run, at exactly 0.500. Three general directives moved it nowhere. It is not a gathering problem, since the thorough run read the data. It is not a bookkeeping problem, since the REPL run tracked everything. It is a rule-application limit: cross-referencing each record's linked account and parent company against layered rules, across many records, without dropping one. Prompts do not appear to reach it.

Rules we run by

Five of these cost us something to learn.

A failed run is not a zero. A rate limit or a box timeout is a failure to measure. Averaging it in as 0.0 quietly poisons the data.

One run tells you nothing. The same task scored 0.33 to 0.50 across identical runs. One run even refused the task and called no tools. Three runs minimum before any claim.

Changes must be general working methods. Wording a directive to fix one task is overfitting to the benchmark, and it invalidates the measurement it appears to improve. Every lever we keep has to read as advice you would give about any task.

Watch tool calls as a cost metric. Every directive we tested raised the call count, whether or not it raised the score. More thorough is never free.

Keep the eval path inert in production. The benchmark backend activates on one conversation type and nothing else. No global flag, no branch in the hot path, no way for a test rig to reach a real user's Gmail.

The rig cost about nine hundred lines of Ruby and a small Python service. What it buys is the ability to change one thing and know whether it helped, which is the only way to tell engineering from guessing.