---
title: "The evaluation set that measured the wrong thing"
description: "A document agent scored 94 percent on its evaluation set and failed in production. The set had been written by the team that built the agent."
type: Field note
url: https://sunilkumar.ai/notes/the-evaluation-set-that-measured-the-wrong-thing/
author: Sunil Kumar (https://sunilkumar.ai/about/)
datePublished: 2026-09-02
pillar: Field notes
tags: [evaluation, healthcare, data]
license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
---

# The evaluation set that measured the wrong thing

A document agent scored 94 percent on its evaluation set and failed in production. The set had been written by the team that built the agent.

A document agent for a healthcare provider. It read referral letters, pulled out the fields the intake team needed, and filled a form. The evaluation set had two hundred letters. The agent scored 94 percent on it. The intake team ran it for two weeks and asked us to switch it off.

The two hundred letters had been chosen by the engineers, from a sample the client had sent early in the project. They were clean: typed, recent, one referral per letter. Production was not. A third of the real letters were scanned handwriting, a good number were two referrals stapled together, and some were not referrals at all but discharge summaries that had been filed in the wrong tray. The agent handled the letters it had been tested on and confidently mangled the rest. 94 percent was a true score for a set that did not exist in the building.

The uncomfortable part is that nobody lied. The team built the set in good faith from the data they had. But an evaluation set written by the team that built the agent measures the team's assumptions, and the assumptions were that letters look like letters.

We rebuilt the set from production. Two hundred cases, sampled at random from a month of real intake, labelled by the intake team rather than by us, including the stapled pairs and the discharge summaries. The expected outcome for a discharge summary was "refuse and route to a person", which the original set had no concept of. The agent's score on the new set was 61 percent. That number was bad and honest, and it gave us a target.

Six weeks of work on the ugly cases, mostly on classifying what a document is before extracting anything from it, took the honest score above the intake team's own consistency rate, which we had measured at the same time. The agent went back on with a rule that any document classified below a confidence threshold goes to a person with the agent's draft attached.

What I would do again: sample the evaluation set from real traffic, at random, and have the domain team label it. Measure the humans on the same set, because "as good as the intake team" is a number, not a feeling. What I would not do again: accept a score above 90 percent on a set the agent's builders assembled. If the agent is not failing part of the set, the set is not real.


---
Sunil Kumar, Co-founder and CEO, Ailoitte Technologies. https://sunilkumar.ai/about/