# Agentic AI vendor evaluation checklist

From "How to evaluate an agentic AI vendor" by Sunil Kumar
https://sunilkumar.ai/playbooks/how-to-evaluate-an-agentic-ai-vendor/
Licence: CC BY 4.0. Copy it, edit it, send it to vendors.

Score each step 0 to 3:
0 = no answer  1 = roadmap or promise  2 = artefact from a real deployment  3 = artefact you verified yourself

A vendor scoring under 2 on step 1 or step 3 should not proceed to the pilot.

| Step | Score (0-3) | Notes |
|---|---|---|
| 1. Refusal list, enforced in code | | |
| 2. Evaluation set and score history | | |
| 3. Trace of a real run | | |
| 4. Cost per run and ceiling behaviour | | |
| 5. Data, residency and compliance | | |
| 6. Ownership after launch | | |
| 7. Bounded pilot with exit criteria | | |

## 1. Refusal list and where it is enforced

Ask: "What must your agent never do, and show me the code that stops it."

- [ ] Written refusal list from a real deployment (anonymised is fine)
- [ ] Each item enforced at the tool layer (permission check, allow-list, threshold, missing tool), demonstrated live
- [ ] Reason codes shown to the user; a human path for a wrong refusal
- [ ] The vendor asks for our refusal list before quoting

Disqualifying answer: "the system prompt instructs the model not to."

## 2. Evaluation set and scores

Ask: "What cases do you test against, what is the pass rate, and how has it changed?"

- [ ] Evaluation set from a real deployment, including refusals and escalations
- [ ] Pass rate with history across model and prompt changes
- [ ] Evaluation runs in the vendor's pipeline on every change (seen running)
- [ ] A plan to build our set from our traffic in the first two weeks

Weak answer: a public benchmark score, or a set the vendor wrote with no production cases.

## 3. A trace

Ask: "Show me one run from a real deployment, end to end."

- [ ] Input, context, each decision, each tool call with arguments and result, guardrail verdicts, output, cost, latency
- [ ] Readable without the vendor narrating
- [ ] Retention period stated; configurable to our regulator's requirement
- [ ] Client-side access to traces without vendor involvement

Disqualifying answer: a log of outputs, or "tracing is on the roadmap."

## 4. Cost per run and behaviour at the ceiling

Ask: "What does a run cost, what will the month cost, and what happens at the ceiling?"

- [ ] Cost per run as a range, all-in (tokens, tool calls, retries, human minutes on escalations)
- [ ] Monthly ceiling and the implemented behaviour when it is hit
- [ ] Cost visible per run in the trace
- [ ] Model routing strategy explained, with the evaluation set as the arbiter

Weak answer: a token price.

## 5. Data, residency and compliance

Ask: "Where does our data go, who can train on it, and which certifications are current?"

- [ ] Data flow diagram naming every system, including model providers
- [ ] Training and retention terms for each provider, in the contract
- [ ] Certifications current and scoped (ISO 27001, ISO 9001; ISO/IEC 42001 or NIST AI RMF mapping for AI)
- [ ] Regulator-specific mapping for the agent, not the company (HIPAA, EU AI Act, sector rules)

Weak answer: a logo wall.

## 6. Ownership after launch

Ask: "On the day after launch, who is on call, who can switch it off, and what do we receive?"

- [ ] Runbook with a named client-side owner and a kill switch they can operate from a phone
- [ ] Alert channel and on-call arrangement, with the vendor's role stated
- [ ] Evaluation set, refusal list and trace tooling handed over as deliverables
- [ ] Plan for the first model upgrade after launch: who runs the evaluation, who signs

Weak answer: a support SLA with no client-side owner.

## 7. Bounded pilot with exit criteria

- [ ] Exit criteria written and signed before the pilot: pass rate, cost per run, guardrail failure rate
- [ ] Real traffic slice, our evaluation set, our traces
- [ ] Kill switch tested by our owner during the pilot
- [ ] Paid, fixed-price, two to four weeks
- [ ] Decision made against the criteria, in writing

---
Sunil Kumar, co-founder and CEO, Ailoitte Technologies. https://sunilkumar.ai/