How to evaluate an agentic AI vendor

Most enterprise agent evaluations spend an hour on a demo and ten minutes on the contract. This playbook inverts that. It gives you seven steps, each with the artefact to ask for, the good answer, the bad answer, and what to do with a vendor who cannot answer at all. It works for platforms and for services firms, and it works on Ailoitte, which is the test I apply to any advice I give.

The principle behind every step is the same: evaluate the vendor the way you will later have to defend the agent. Your risk committee, your auditor and your customers will ask for refusals, traces, costs and an owner. Ask for those first.

Vendor evaluation for agentic AI

A structured check of whether a supplier can deliver an agent that passes the production-grade tests, judged on artefacts from real deployments rather than on a demo.

1. Ask for the refusal list and where it is enforced

Ask: “What must your agent never do, and show me the code that stops it.”

The good answer is a list, and for each item a mechanism at the tool layer: a permission check, an allow-list, a threshold, a tool that simply does not exist. The vendor should be able to trigger a refusal in front of you and show you the reason code the user sees.

The bad answer is “the system prompt instructs the model not to”. A prompt is a request. Prompt injection, an unusual input or a model update can turn “never” into “usually”, which is the phrase that ends pilots. The OWASP Top 10 for LLM applications puts prompt injection first for a reason.

If the vendor cannot show a refusal list at all, stop the evaluation. This is the cheapest artefact on the list and a vendor without one has not been to production.

2. Ask for the evaluation set and the scores

Ask: “What cases do you test against, what is the pass rate, and how has it changed?”

The good answer is a set of real cases with expected outcomes, including cases where the right outcome is a refusal or an escalation, and a history of the pass rate across model and prompt changes. The history matters more than the number: it shows the vendor runs the set on every change rather than once before the demo.

The bad answer is a benchmark score from a model provider. Public benchmarks such as τ-bench are useful for comparing models; they say nothing about whether this vendor’s agent handles your cases. The other bad answer is an evaluation set the vendor wrote themselves, with no production cases in it. That measures the vendor’s imagination.

Then ask how they will build your evaluation set. The right vendor will ask you for real cases in the first week and will want the ugly ones.

3. Ask to see a trace

Ask: “Show me one run from a real deployment, end to end.”

This is the most informative ten minutes of the evaluation. A trace should show the input, the context the model saw, each decision, each tool call with arguments and result, each guardrail verdict, the output, the cost and the latency, rendered so that you can read it without the vendor narrating. Then ask two follow-ups: how long is this kept, and who on the client side can read it.

The good answer is a trace you can follow, kept for as long as the decision matters, readable by the client’s compliance team without a login to the vendor’s tooling. Retention should be set by your regulator, not by the vendor’s storage costs.

The bad answer is a log of outputs, or a promise that tracing is on the roadmap. If a run cannot be replayed from its trace, the vendor does not have an audit log; they have a diary. In healthcare, finance and government that is disqualifying, because the EU AI Act, the HIPAA Security Rule and ISO/IEC 42001 all reduce to the same question: show what the system did and why.

4. Ask for the cost per run and the behaviour at the ceiling

Ask: “What does a run cost, what will the month cost, and what happens when we hit the ceiling?”

The good answer is a range per run that includes model tokens, tool calls, retries and the human minutes spent on escalations; a monthly ceiling; and an implemented behaviour at the ceiling, usually “stop accepting new work and escalate”. The vendor should be able to show the cost meter in the trace from step three.

The bad answer is a token price. Token price is the cost of one ingredient. It tells you nothing about how many tokens a run uses, how often the agent retries, or what the escalations cost you in people. The other bad answer is “it depends on usage” with no range. It does depend on usage; a competent vendor gives you the range anyway and tells you what drives it.

Ask about the model strategy too. A vendor that routes most cases to a small model and reserves the expensive one for the hard slice has thought about your bill. A vendor that uses the largest model for everything is spending your money on their convenience.

5. Ask about data, residency and compliance

Ask: “Where does our data go, who can train on it, and which certifications are current?”

The good answer names every system the data touches, including the model provider, with the contractual terms on training and retention for each. Certifications are current and scoped: ISO 27001 for information security, ISO 9001 for process, and for AI specifically ISO/IEC 42001 or a documented mapping to the NIST AI Risk Management Framework. For healthcare, a business associate agreement and a HIPAA mapping. For the EU, a statement of where the agent sits in the AI Act’s risk categories and what that obliges.

The bad answer is a logo wall. Certifications are a starting point; the question is whether the vendor can show the mapping from the certification to the specific agent you are buying. “We are ISO 27001 certified” is a fact about the company. “Here is how the trace retention, access control and refusal mechanism in this agent map to the controls in our ISO 27001 statement of applicability” is a fact about the agent.

6. Ask who owns the agent after launch

Ask: “On the day after launch, who is on call, who can switch it off, and what do we receive?”

The good answer is a runbook: a named owner on your side, a kill switch that person can operate from a phone, an alert channel, an on-call arrangement, and the handover of the evaluation set, the refusal list and the trace tooling as deliverables. The vendor should describe the handover as part of the scope, not as a support tier.

The bad answer is a support SLA with no mention of a client-side owner. A support contract tells you how fast the vendor responds. It does not tell you who in your organisation can stop the agent at 2 a.m. If the vendor’s model assumes they hold the kill switch, the agent is theirs, not yours.

7. Run a bounded pilot with exit criteria

By this step you will have narrowed to one or two vendors. Do not sign. Run a pilot, and write the exit criteria before it starts.

Two to four weeks. A slice of real traffic, not curated cases. Your evaluation set, which the vendor helps you build in the first week. A pass rate, a cost per run and a guardrail failure rate that you have written down as the definition of success, and a kill switch that you have tested. The pilot is paid and fixed-price; a free pilot has no deadline and no definition of done.

At the end, the decision is mechanical. The numbers met the criteria or they did not. If they did, you are buying a production system that has already run in production. If they did not, you have learned that for the price of a month, which is the cheapest lesson on this list.

What to ask the reference customers

Every vendor offers references. Most reference calls are a pleasant conversation about the sales process. Ask the reference three questions instead, in this order.

“Can you open a trace from last week while we talk?” If they can, the tracing is real and the client has access to it. If they have to ask the vendor, note that.

“What happened the first time the agent did something wrong, and who noticed?” You are listening for a named owner, an alert, a kill switch and a fix that ended up in the evaluation set. A story in which the vendor noticed first, or in which nobody is quite sure, tells you where the ownership sits.

“What did the second month cost compared with the first?” The answer should be a number they know. If the cost rose sharply, ask what the vendor did about it; if the reference does not know the number, the budget ceiling from step four is missing.

A reference who answers all three with specifics is telling you the vendor has been to production. A reference who answers with satisfaction is telling you the vendor has good account managers, which is not the same thing.

Scoring

Score each step from 0 to 3: 0 for no answer, 1 for a roadmap or a promise, 2 for an answer with an artefact from a real deployment, 3 for an artefact you were able to verify yourself. A vendor scoring under 2 on step one or step three should not proceed to the pilot regardless of the rest. The downloadable checklist has the scoring grid.

If you are doing this in your company

Send this list to the vendor before the first meeting and ask them to bring the artefacts. The meeting will be shorter and the demo will be the least important part of it. If a vendor objects that the list is unreasonable, you have your answer; every item on it is something you will be asked for by your own auditor within a year of launch.

If you are the vendor, and I say this as one: the list is also the specification for the first two weeks of any engagement. Produce the artefacts before you quote and the quote can be fixed. That is the argument in fixed price for agentic AI, and the reference architecture is how the artefacts get built.