# Agent harness decision-layer checklist

From "Agent harness playbook: let the model choose, not act" by Sunil Kumar
https://sunilkumar.ai/playbooks/agent-harness-decision-layer/
Licence: CC BY 4.0. Copy it, edit it, put it in the repository next to the code that enforces it.

The principle: a model can suggest the next move; the host still owns the move. The selector picks from a menu the host prepared, the host checks the pick against current state, permission stays on its own path, and every decision leaves a receipt.

Score each step 0 to 3:
0 = not done  1 = described in a document  2 = implemented in code  3 = implemented and covered by a test you can run

A pilot with under 2 on step 5 (permission) or step 4 (re-check) does not go near a real repository, customer or ledger.

| Step | Score (0-3) | Notes |
|---|---|---|
| 1. One decision the host can check | | |
| 2. The menu, and what is off it | | |
| 3. Typed result, abstain and fallback defined first | | |
| 4. State re-checked at the moment of acting | | |
| 5. Selection and permission kept apart | | |
| 6. A receipt per decision, no story | | |
| 7. Whole-job measurement against the plain harness | | |
| 8. Human gate on every harness change | | |

## 1. Pick one decision the host can check

- [ ] One decision named (route a fresh task, choose the next tool bundle, propose a low-risk action id)
- [ ] The host can enumerate every option, run any of them, and observe the outcome
- [ ] The decision is on the host's side of the line, not inside a provider's own loop
- [ ] Pinned choices and running sessions excluded by rule
- [ ] Written answer to "what does the outcome look like and where will we see it"

Off the table: provider-internal tool loops, permissions, anything the code cannot enforce.

## 2. Write the menu, and what is off it

- [ ] Candidate list built at call time from live state (installed, enabled, valid for this task)
- [ ] Filtered before the selector sees it
- [ ] Opaque ids only; the host keeps the details needed to run each option
- [ ] Menu rules versioned like code, with a maximum size (a hosted decision model such as Jev allows up to 255 choices; two is often enough)
- [ ] Empty-menu behaviour defined
- [ ] Bounded state only; never raw documents, tickets or secrets in the selector's input

## 3. Type the result and define abstain before the first call

- [ ] Schema: one id from the menu or abstain, optional confidence, no free text
- [ ] Malformed output, unknown ids and abstention each have a defined outcome
- [ ] Fallback policy written before the selector is wired in (original route, cheapest safe option, or a question to a person)
- [ ] Fallbacks logged as fallbacks, never counted as successful selections
- [ ] "Zero hallucinations" understood as schema validity, not correctness

## 4. Re-check the state at the moment of acting

- [ ] Pre-selection checks: candidate freshness, preconditions, authorisation
- [ ] Post-selection checks: the pick is still eligible and current (fingerprint, expiry)
- [ ] Dispatch checks: the tool bundle rebuilt from live definitions, the named tool verified again
- [ ] Maximum age per decision type
- [ ] A stale pick is rejected and the fallback runs; the selector is not re-asked with a nudge

## 5. Keep selection and permission apart

- [ ] The permission path is unchanged by the decision layer and unreachable from it
- [ ] Irreversible actions (payments, deletions, messages to customers) gated on a named person, whatever the confidence
- [ ] "Answer" style choices carry an empty tool bundle by construction
- [ ] A chosen action and an executed action are separate responsibilities in the code
- [ ] Approval volume and median time-to-approve on the dashboard, with thresholds that alert (under 10 seconds median, or a yes rate above 98 percent)

## 6. Record the decision, not the story

- [ ] One receipt per decision: candidates offered, pick or abstention, backend confidence, validation result, fallback, observed result, session metadata
- [ ] No chain-of-thought or free-text rationale stored as fact
- [ ] Retention set with the regulator's expectations; readable by compliance without engineering help
- [ ] Client data and secrets excluded by design, with a test that proves it
- [ ] Receipts mapped to record-keeping (EU AI Act Article 12) and human oversight (Article 14) where the system is in scope; to the ISO/IEC 42001 controls where certified

## 7. Measure the whole job against the plain harness

- [ ] Baseline: the plain harness on a fixed task set, scored by fixed rules
- [ ] Metrics: valid-selection rate, abstention rate, fallback rate, selector latency, total task cost including human review minutes
- [ ] Held-out task set for the final comparison
- [ ] Failures kept and read; awkward runs recorded
- [ ] Every tuning change re-measured; "seemed better" is not a result

## 8. Put a human gate on every change to the harness

- [ ] Receipts convertible into replayable scenarios (same candidates, same task flags)
- [ ] Every proposed change (menu rules, schema or prompt, fallback, selector model, tool bundles) replayed against baseline and candidate, comparison recorded
- [ ] A named person approves the new baseline; old versions kept; rollback tested
- [ ] No automatic training or policy rewrite from production traffic

## Before production

- [ ] Selector vendor reviewed like any AI vendor: region, retention, training on inputs, early-access status, price stability, exit
- [ ] A local selector for regulated workloads until the hosted contract exists
- [ ] Prompt-injection review of everything that feeds the selector's input
- [ ] Deny rules for provider loops enforced at the sandbox and permission layer, in code, not prose
- [ ] Local-first tooling assessed for the fleet: signing, notarisation, device management, where records and credentials live
- [ ] Routing cost measured: a selector that takes longer than the task is a waiting room

## The lab (a spare Mac, a throwaway repository)

- [ ] Apple silicon, macOS 15 or later; keel at the 0.2.0 snapshot, not main
- [ ] Same task in normal, local (Laya) and, if available, hosted (Jev) mode; decision records read
- [ ] Pinned route left alone by the selector
- [ ] Tool definition changed between selection and dispatch; stale bundle rejected
- [ ] Empty menu forces abstention; fallback observed
- [ ] Whole-task time recorded per mode, including your review
- [ ] The four lines written: one decision, the exact choices, what happens on abstain, how you will know it helped

---
Sunil Kumar, co-founder and CEO, Ailoitte Technologies. https://sunilkumar.ai/