Field note
Ten thousand runs a day changes the price
An agent that cost cents per run in the pilot cost far more per month at scale. Routing, caching and a budget ceiling fixed it; the ceiling mattered most.
An internal workflow agent for an enterprise client. It took requests from a shared inbox, classified them, pulled the relevant records and drafted a response for a person to approve. In the pilot it handled a few hundred requests a day and cost a few cents each. Everyone was pleased. The rollout took it to the whole company.
The first full month’s invoice from the model provider arrived and the sponsor’s first question was whether it had been a mistake. It had not. Ten thousand runs a day, each using the largest model for every step, each pulling more context than it needed, with a retry loop that ran the whole thing again when a downstream system timed out, adds up to a number that looks nothing like “a few cents”. The agent was working. It was doing more work than anyone had priced, in the most expensive way it could.
Three changes brought the bill down to something the value could justify. Routing: the classification step and most of the drafting went to a small model, with the large one reserved for the requests the small one flagged as hard, roughly one in ten. Caching: the same records were being fetched and re-summarised for every request from the same department, so the summaries were cached for a day. Limits: the retry loop got a maximum of two attempts and a per-run budget, after which the request went to a person with a note.
Together they brought the monthly bill down to roughly a quarter of that first invoice, with no measurable change in the approval rate the reviewers were seeing. Most of that came from routing. The large model had been doing work a small one does just as well, at ten times the price.
The change that mattered most was not any of those. It was the ceiling. We added a monthly budget to the operations layer, and when the budget is hit the agent stops taking new requests and the inbox goes back to the people, with an alert to the owner. That single control turned the invoice from a surprise into a decision. The sponsor can now raise the ceiling when the value justifies it, and the finance team can see it coming.
What I would do again: price a run before the rollout, not after, and build the ceiling on day one. Route by default, and let the evaluation set decide which model each step needs. What I would not do again: let a pilot’s cost per run stand in for a production cost per run. The pilot has no retries at scale, no long tail of hard cases and no month-end. Production has all three.