Field note
The approver who said yes 1,900 times
A payables agent stopped and asked a human 1,900 times in three weeks. By week three the human said yes in four seconds, once to a changed bank account.
An accounts payable agent for a logistics group, sixth week in production, an AI Velocity Pod engagement. About 9,000 supplier invoices a month. The agent matched each invoice to its purchase order and goods receipt, coded it, and prepared the payment run. Roughly seven in ten went straight through. The rest, anything with a mismatch, a new supplier, a changed bank account or an amount over the threshold, stopped and went to a human: the AP team lead, named in the runbook, with a screen that showed the invoice, the reason and two buttons.
That is the design I recommend. A named approver for anything irreversible is one of the four tests in Agentic in Production, and this agent passed it on paper.
In the first three weeks the agent stopped 1,900 times. The team lead approved 1,893. Median time from opening an item to pressing yes: 41 seconds in week one, 12 in week two, four in week three. The seven rejections were all in the first four days.
On day nineteen an invoice arrived from a real supplier, for a real delivery, with a line in the body: please use the updated remittance details below. The bank account did not match the master record. The agent did exactly what it was built to do. It stopped and asked. The team lead pressed yes, in three seconds, and $41,200 went to an account nobody at the supplier had ever heard of. The Friday reconciliation caught it two days later and the bank recalled the payment, which is not a control either. It is luck with a good calendar.
The retro was short, because the number was on the wall: 1,893 out of 1,900. Nobody blamed the team lead. She had been handed 630 decisions a week that all looked the same, and 629 of them were fine. An approval step that fires on everything does not add a human to the loop. It trains the human to leave it. WorkOS called this approval fatigue in August; security teams have called it alert fatigue for a decade. The fault was ours. We had measured the agent every day and never once measured the person.
Three rules came out of that retro, and every pod at Ailoitte runs them now.
One: route by exception. A human sees an item only when the agent cannot decide it on the evidence. Ordinary mismatches, a quantity off by one, a price inside tolerance, are resolved by the agent, checked by a second agent, and listed in a daily digest the team lead reads with her coffee. Approvals fell from about 630 a week to 40.
Two: gate on consequence, not on category. Anything irreversible, a payment to a new or changed bank account, an amount over the threshold, gets a different screen. The approver has to type the last four digits of the account from the supplier master record, not from the invoice, before the button works. Median time on those approvals is now 70 seconds, and it should be.
Three: measure the approver, not only the agent. Median time to approve and the yes rate sit on the dashboard next to the agent’s own numbers. If the median drops under ten seconds or the yes rate goes above 98 percent, the queue is too wide, and that is an alert to the pod, not a compliment to the agent.
Three weeks of saying yes, then the fix. The agent’s numbers never changed; the person’s did. The figure is free to share: download it as a PNG (1600px wide, CC BY 4.0).
What I would do again: put a named person in the loop for anything irreversible. What I would not do again: measure the agent and forget to measure the person. If the median approval takes four seconds, you do not have a human in the loop. You have a button.