What "production-grade" actually means for an AI agent

“Production-grade” gets used the way “enterprise-ready” used to be used: as a feeling about a product rather than a property of it. That is a problem, because the feeling is what gets an agent into a pilot and the property is what gets it out. This essay gives the phrase a definition you can test. It is the second part of the argument that starts with why most enterprise AI agents never leave the demo.

Production-grade agent

An agent that passes five tests at once: its refusals are enforced in code, every run can be reconstructed from its trace, a named person owns it and can stop it, it runs inside a budget it cannot exceed, and its behaviour on a fixed evaluation set is measured on every change.

Test one: are the refusals enforced, or requested?

Every agent has things it must not do. Issue a refund above a threshold. Change a dosage. Email an external address. Delete anything. The question is where that rule lives.

If it lives in the prompt, it is a request. The model will honour it most of the time, and “most of the time” is the phrase that ends pilots. Prompt injection, an unusual input or a model update can all turn “never” into “usually”.

If it lives at the tool layer, it is a control. The tool that issues refunds checks the amount and returns a refusal the model cannot argue with. The email tool has an allow-list of domains. The delete tool does not exist. The model can want to do the wrong thing all it likes; it does not have the means.

The test: take the refusal list and, for each item, find the line of code that enforces it. If the answer is “the system prompt says”, the item fails. The OWASP Top 10 for LLM applications is a useful reminder of why: the first item on the list is prompt injection, and the defence is not a better prompt.

There is a second half to this test. An agent that cannot explain a refusal is not production-ready, because the person it refused will ask why, and “the model declined” is not an answer a support team can give. Refusals need a reason code and a human path. That is a product decision as much as an engineering one, and it is usually missing.

Test two: can a stranger reconstruct a run?

Pick a run from last week. Hand its trace to someone who did not build the system, and ask them to tell you what happened. They should be able to say what the input was, what context the model saw, what it decided at each step, which tools it called and with what arguments, what the tools returned, what went out, and how much it cost. If they cannot, the trace is incomplete.

This is the test regulators are actually running, in their own words. The EU AI Act’s obligations for high-risk systems include record-keeping and traceability (Regulation (EU) 2024/1689); the HIPAA Security Rule’s audit controls expect you to record and examine activity in systems that touch protected health information (45 CFR 164.312); ISO/IEC 42001, the AI management system standard, asks for the same evidence in a form an auditor can follow (ISO/IEC 42001:2023). They are different documents with one question in common: can you show what the system did and why.

Two practical notes. Retention is part of the test; a trace that is deleted after seven days does not help with a complaint filed on day thirty. And the trace has to be readable by a person, which usually means a rendered view, not a JSON dump. If a run cannot be replayed from its trace, you do not have an audit log. You have a diary.

Test three: who can stop it, and how fast?

Find the person who owns the agent. Not the team; the person. Ask them to switch it off. Time it.

Under a minute, from a place they can reach on a phone, without a deploy, is the standard. That is not a high bar for the software your company already runs; every payments system has a feature flag. It is a high bar for agent projects because the ownership was never assigned, so the kill switch was never built.

The same person should receive the alerts, sign the change that puts a new version live, and be the one who decides whether a failure is a bug or a breach. If that person does not exist, the agent is not in production. It is running.

Test four: does it run inside a budget it cannot exceed?

Ask what a run may cost, what the month may cost, and what happens at the ceiling. The answers should be numbers and a behaviour, and the behaviour should be implemented.

An agent that stops at the ceiling and escalates is production-grade. An agent that keeps going and surprises the finance team is a pilot with a credit card. The budget also covers retries and loops: a run that calls the same tool forty times because the tool keeps failing is a budget problem before it is a correctness problem, and the fix is a limit, not a better prompt.

Test five: is the behaviour measured on every change?

This is the test that separates software from a science project. Take a fixed set of real cases, with expected outcomes, and run the agent against it every time anything changes: the prompt, the model, a tool, the retrieval index. Record the score. A change that drops the score does not ship.

The evaluation set has to be real. Cases from production, including the ugly ones, including the ones where the right answer is a refusal. A set written by the team that built the agent measures the team’s imagination, not the agent. The way to know your set is real is that the agent fails some of it.

The research community has built public versions of this for agents, τ-bench being the clearest example for tool-using agents in customer-facing domains. You do not need a public benchmark. You need your own, and you need it to run in the pipeline, not in a notebook.

A note on “a human reviews everything”

The most common answer to all five tests is that a person checks every output before it goes anywhere. I understand the instinct, and for the first weeks of a rollout it is right. As a permanent design it fails both ways.

It fails as a control because reviewers stop reading. Ask anyone who has approved two hundred agent drafts in a morning what the hundred and fiftieth one said. The approval becomes a click, the click becomes an audit trail that says a person agreed, and the person did not. A control that everyone knows is not really applied is worse than no control, because it is trusted.

It fails as economics because the review costs more than the agent saves. If a run costs a few cents and the review costs three minutes of a professional’s time, the agent has not automated anything; it has added a step.

The production-grade design puts the human where their judgement matters: on the small set of actions that need approval (the tool layer decides which), on the escalations the agent refuses, and on the weekly sample of traces that keeps the evaluation set honest. Everywhere else the five tests do the work, and they do not get tired.

Why governance is an engineering problem first

The temptation is to treat these five tests as a policy document: write them down, get them signed, move on. That is backwards. Each test is a component. The refusal list is the tool layer’s permission model. The trace is the logging design. The kill switch is a feature flag with an owner. The budget is a counter with a limit. The evaluation set is a test suite that runs in CI.

Build them and the policy writes itself, because the policy is a description of what the system does. Write the policy first and you get a document that describes a system nobody built. That is why, at Ailoitte, ISO 27001 and ISO 9001 are not paperwork that follows the work. They are the shape of the delivery process, and an agent that passes the five tests passes the audit as a side effect.

If you are doing this in your company

Run the five tests on your current agent, honestly, in an afternoon. For each one, write down pass or fail and the one thing that would turn a fail into a pass. You will usually find that the model is fine and the missing pieces are ordinary engineering: a permission check, a log, a flag, a counter, a test suite.

Then do the missing pieces before the next model upgrade, not after. The upgrade will make the agent better. The five tests are what let you find out.

The reference architecture playbook shows where each test lives in the system. If you are buying rather than building, the vendor evaluation playbook turns the same five tests into questions to ask a supplier.