Agent harness playbook: let the model choose, not act

A builder who posts as Avid published a walkthrough on 23 September of a coding harness called keel, in which a small decision model picks from a menu the host prepares and the host checks the pick before anything runs. It is the best short account I have read of a rule my pods have run for two years without a name for it: a model can suggest the next move, the host still owns the move. This playbook turns that pattern into something a centre of excellence, an R&D team or an enterprise platform team can run, with the risks to read first, the security and compliance work that goes around it, and a lab an engineer can finish in an afternoon.

Two things before the steps. The pattern is not about one vendor. keel’s selector can be a local model or a hosted one, and the playbook works with either, or with a classifier you already have. And it is not about coding agents only. Route selection, next-step selection and low-risk action selection show up in every agent that does real work; coding is simply where the failure is cheapest to study.

Agent harness

Everything around the model that turns it into an agent: the tools it may call, the state it can see, the permissions, the loop that decides what happens next, the checks on the result, and the records. If you are not the model, you are the harness.

Most of that definition is Addy Osmani’s, from April. The harness is where enterprises win or lose with agents, because it is where the four tests in Agentic in Production live: the refusal list, the named approver, the trace and the budget. A decision layer is a small addition to the harness that makes one part of the loop, the choice of what to do next, inspectable. That is worth doing. It is also easy to do badly, which is what the eight steps are for.

The pattern in one paragraph

The host, meaning your application code, builds a short list of things it is prepared to do right now, and removes anything it cannot actually run. A selector, which is a small model built for decisions rather than prose, receives that list as opaque ids and returns one id or says it cannot choose. The host checks the pick against the current state, because the world may have moved while the model was thinking, and then either acts or falls back to a rule it wrote before the call. Permission is a separate path the selector never touches. Every decision leaves a receipt: what was offered, what was chosen, whether the host accepted it, what happened next. That is the whole design, and every step below is one of those sentences made enforceable.

Step 1. Pick one decision the host can check

Start with one decision, not a decision layer. In keel there are three, and they are a good menu for anyone. Which provider and model should take a fresh, unpinned task. Which of four focuses, inspect, implement, verify or answer, the embedded loop should take next, each mapped to a tool bundle the host defines. And, on the desktop, which low-risk, reversible action id to propose. Each has a property the decisions you might be tempted by lack: the host can enumerate the options, run any of them, and see the outcome.

The decisions to keep off the table are the ones the host does not own. The coding providers a harness embeds, Claude Code, Codex, Cursor and the rest in keel’s case, run their own tool loops through the agent client protocol. The harness can route a task to them; it does not control their next tool call, and drawing an arrow on a diagram does not create an interface. A listed slash command is not an action the selector can invoke. Permissions are not a decision either. If you cannot point at the code that enforces the boundary, the selector does not get to act there.

Step 2. Write the menu, and what is off it

The selector cannot choose a good option the host forgot to offer, and it must never be able to choose one the host cannot deliver. So the menu is built at call time from what is installed, enabled and valid for this task, and filtered before the model sees it. A provider that is not enabled, a model that cannot be found, a reasoning level that is not supported: all removed first. The model receives ids that refer to those options. It cannot invent a provider, summon a missing tool or rewrite a candidate, because a label has nowhere to hide an instruction.

Write the menu rules down as code and as a document, because they are where most of the behaviour lives. Include the cardinality: a hosted decision model such as Jev allows up to 255 choices, a local classifier may allow fewer, and a menu of two is often enough. Include what the host does when the menu is empty, which is the most common failure in the first week.

Step 3. Type the result and define abstain before the first call

The result has a fixed shape: one id from the menu, or an abstention, plus whatever confidence the backend supplies. That is why the interface can be dull to integrate; every caller reads the same three fields and either accepts the pick, or does not. Jev’s pitch is exactly this, typed results with a calibrated probability at 70 to 500 milliseconds, and a schema you can validate is the useful part. Note what “zero hallucinations” means in that pitch: the output always matches the schema. It says nothing about whether the choice was right.

Abstention is a valid answer, and the host must know what to do with it before it asks. Define the fallback first: the user’s original route, the cheapest safe option, or a question to a person. Then count fallbacks separately from selections. A fallback that gets counted as a selector win makes every later measurement a lie.

Step 4. Re-check the state at the moment of acting

A decision is made against a snapshot. By the time the host acts, the provider list may have changed, the repository may have moved, the tool definitions may have been updated. keel checks freshness, a fingerprint of what the selector saw, and expiry before it applies a route, and it rebuilds the tool bundle against the current definitions before dispatch and checks the named tool again at the moment of the call. Both checks serve one purpose, to keep an old decision from acting on a changed system, and both are boring, which, as the article says, is what you want from safety checks.

This step is also where you decide how long a decision may live. Seconds for a tool choice inside a loop, minutes for a route, never longer than the state it depends on. When the check fails, the host rejects the pick and uses the fallback from step three. It does not ask the selector again with a nudge.

Step 5. Keep selection and permission apart

This is the rule most teams try to relax first, and the one that decides whether the design survives an audit. The selector can pick a candidate. It cannot approve a dangerous action. If a tool needs permission, the normal permission path applies, and where the application requires a person, a person approves. Model selection does not grant authority, and neither does a high confidence value.

In practice that means three things. The “answer” focus gets no tools at all, because that is what answering means. A chosen click and an executed click are separate responsibilities. And anything irreversible, a payment, a deletion, a message to a customer, still lands with the named approver from your runbook. It also means the approver’s queue is a design surface. A selector that routes by exception keeps that queue small; a harness that asks for approval on everything trains the approver to stop reading, which is the lesson of the approver who said yes 1,900 times.

Step 6. Record the decision, not the story

After validation and after the outcome is observed, the host writes a receipt. keel’s contains the bounded candidate summaries, the typed result and backend confidence, the validation and fallback outcomes, the observed execution result and the session metadata. It contains no reasoning, because a model’s explanation of why it chose something is not evidence of why it chose it. The receipt lets you ask ordinary engineering questions: what were the candidates, did it choose or abstain, did validation accept it, did a fallback happen, what came back from the tool.

For an enterprise the receipt is more than a debugging aid. It is the record-keeping that the EU AI Act asks of high-risk systems and that ISO/IEC 42001 asks of an AI management system, produced as a by-product of running rather than as a compliance project afterwards. Decide retention with your regulator’s expectations, keep receipts readable by compliance without an engineer in the room, and make sure client data stays out of them; the receipt records ids and outcomes, not the documents the task touched.

One receipt per decision, written after the outcome is known. Free to share: download it as a PNG (1600px wide, CC BY 4.0).

Step 7. Measure the whole job against the plain harness

An extra model call has to earn its place. The comparison is not selector accuracy; it is total task cost: selector time, worker time, retries, and the human review effort at the end, measured against the same harness with the selector switched off. Keep the task set and the scoring rules fixed, keep a held-out set for the final comparison, and record the awkward runs, because a clean demo hides the missing rule.

The wider field is not doing this. Marmelab’s audit of 246 harness repositories and 57 publications, published this week, found that about 60 percent of harnesses have no tests or evaluations at all. Success rates on identical tasks varied from 68 to 88 percent across eight harnesses. Only five of the 391 repositories it counted measured which of their own rules ever fired. keel’s own build report is honest about the same gap: 1,477 tests pass, and no latency or coding-quality gain has been measured. Treat any claim that a selector made an agent better as a hypothesis until your numbers say otherwise.

Step 8. Put a human gate on every change to the harness

“Self-improving” should mean one thing in an enterprise: receipts become scenarios, scenarios can be replayed against a proposed change, and a person decides whether the change becomes the new baseline. A change can be to the candidate rules, the selector’s schema or prompt, the fallback policy, the model behind the selector, or the tool bundle attached to a focus. For each one, keep the old version, replay the relevant scenarios, inspect the regressions, and approve or reject with a diff and a rollback path. Nothing trains itself from chat history. Nothing rewrites its own policy after a failed run.

This is the review of intent from AI-DLC applied to the harness itself, and it is the gate that keeps the harness from rotting quietly as models change. Marmelab’s audit lists self-improving harnesses and reviewer agents among the unproven ideas, and I agree. A changed rule, an old result and a new result are something I can work with. “The agent learned something” is not.

Before production: the risks, and where to look

Read this section twice if the harness will touch a real repository, a real customer or a real ledger.

The hosted selectors are new, and unproven where it counts. TypeSafe announced Jev on 15 September 2026; it is in early access behind a waitlist, hosted only, with no published weights and no self-hosting option, running from infrastructure on the US West Coast, and the announcement I read makes no statement about data handling. As of this writing there is no public case of it carrying a high-stress production workload, and we have not put one on it ourselves. Its output tokens are free and its input price is a fraction of a cent, which TypeSafe itself says it cannot prove is unsubsidised. None of that makes it a bad choice for a lab. All of it makes it a procurement question before production, and the vendor evaluation playbook applies to a decision model exactly as it applies to an agent platform. keel’s default is a local model, which is the right default for anything that sees client data.

The selector’s input is an injection surface. It receives bounded state and candidate summaries; if any of that is built from repository contents, tickets or documents, an attacker can shape the menu or the pick. OWASP’s June 2026 state-of-agentic-security report puts prompt injection at the centre of agentic failures and names the dangerous combination, private data plus untrusted content plus a way to communicate out. Keep secrets and raw documents out of the selector’s input, and keep the selector’s output unable to reach anything the host did not already offer.

The harness does not govern the providers inside it. A coding provider running its own loop will make its own tool calls under its own permissions. Your controls for those calls sit at the sandbox and the permission layer, not in the decision layer. Marmelab found that only twelve of the 391 repositories it counted enforce deny rules in code rather than in prose, and that written rules fail 84 to 96 percent of the time. Enforce, do not instruct.

The fallback can become the problem. “Ask the user” as a fallback for every uncertain pick recreates the approval-fatigue failure. Route by exception and watch the queue.

Local-first is not fleet-ready. keel is a Rust and GPUI application for Apple silicon Macs on macOS 15 or later, released under MIT, ad hoc signed, with its guide asking for a single installed copy and noting that public distribution would need a Developer ID signature and notarisation. Decision records live on the developer’s machine, in the app’s data directory, and the hosted credential lives on the machine too. That is fine for a lab and wrong for a hundred developer laptops under device management without further work.

Routing can become its own bureaucracy. If choosing a worker takes longer than the task, you have built what the article calls an elaborate waiting room. The measurement in step seven is the only defence.

Security and compliance, in the language your auditor uses

Least privilege: each focus maps to a tool bundle, and the bundle is the permission scope. Empty bundles are legitimate. Nothing outside the bundle can be called, and the check runs at dispatch, not at planning.

Segregation of duties: the selector chooses, the host validates, the approver permits, the provider executes, the record is written by the host. No component holds two of those roles.

Data residency and processing: know where the selector runs and what it receives. A local model keeps task metadata on the machine; a hosted one sends candidate summaries and bounded state to the vendor’s region, which for Jev today is the US. Contract for retention and training before any client data is in scope, and prefer a local selector for regulated workloads until the contract exists.

Logging and retention: the receipt is the log. Keep it for as long as the decision matters, in a form compliance can read, and prove that it excludes secrets and content. Article 12 of the EU AI Act, on record-keeping, and Article 14, on human oversight, are the two to have open when you design it, whether or not the system is formally high-risk.

Change control: every harness change goes through the human gate in step eight, with a diff, a replay and an approver. That is the control ISO/IEC 42001 auditors ask about first, and the one a self-training harness cannot pass.

Supply chain: pin the versions of the selector model, the worker, and the tool definitions, and verify what you install. keel’s build report records SHA-256 checks on its runtime inputs and a signed build; hold your own build to the same standard, and treat a decision model’s weights or endpoint as a dependency with an owner.

For engineers: try it this week

This is the part for the people who want to see it run, and it is safe to do on a spare Mac with a throwaway repository.

Take an Apple silicon Mac on macOS 15 or later. Clone keel at the 0.2.0 snapshot the article links to, not main, which still carries the earlier prototype, and read the decision architecture document before you build. Build with cargo build -p keel --bin keel --release, keep a single installed copy as the guide asks, and install the local selector with Keel.app/Contents/MacOS/keel laya install; it downloads about 680 megabytes. Use a repository with nothing in it you would mind losing, no production credentials, and a provider account that is not connected to client work.

Then run the four experiments that teach the pattern. Run the same small task in normal mode, in the local Laya mode, and, if you have a credential, in Jev mode, and read the decision record each leaves in chat and in the app’s data directory. Pin a route and confirm the selector leaves it alone. Change a tool definition between selection and dispatch and confirm the host rejects the stale bundle rather than running it. Force an abstention with an empty menu and watch the fallback. Time the whole task in each mode, including your own review at the end, and write the four numbers down.

Finally, write the four things the article asks for before you touch any of your own harness: the one decision it is allowed to make, the exact choices it can see, what happens when it abstains, and how you will know whether the outcome helped. If you cannot write the fourth line, you are not ready for the first three.

How I would run this in a pod

An AI Velocity Pod treats the harness the way it treats any agent: the refusal list says what the selector may never choose, the named approver holds every irreversible action, every decision is traced, and the budget covers the selector’s calls as well as the worker’s. The two-week definition phase would add one artefact, the menu rules, and one number, the total task cost on the baseline harness, before anyone turns a selector on. The client’s domain reviewer owns the menu rules the way they own the refusal list. And the exit condition includes the human gate from step eight running without us.

If you are doing this in your company

Choose one decision the host can enumerate and check, and refuse the rest for now. Write the menu rules, the schema, the fallback and the receipt before the selector is wired in, and put them in the same repository as the code that enforces them. Keep permission on its own path with a named approver, and put the approver’s numbers on the dashboard. Measure the plain harness first, then the selector, on the same tasks, including review time. Put a person and a diff between every harness change and production. And run the lab on a spare machine with nothing to lose before you run anything on a machine with something to lose.

If you want a second pair of eyes on a harness design or a decision-layer pilot before it reaches a real repository, that is the kind of review I do, and the work with me page has the shape of it.

Sources

• Avid, How to Build Agentic Harness using Jev (Builder’s Guide), X, September 2026, and the keel repository with its decision architecture, guide and build report at the 0.2.0 snapshot.
• TypeSafe AI, Introducing System One models and Jev, 15 September 2026 (latency, price, early access, cardinality, the subsidy caveat).
• Zaninotto, F., The State of AI Harness Engineering 2026, Marmelab, 24 September 2026 (246 repositories and 57 publications audited; the test, deny-rule and success-rate figures).
• Osmani, A., Agent Harness Engineering, 19 April 2026 (the definition of a harness).
• OWASP GenAI Security Project, State of Agentic AI Security and Governance, version 2.01, June 2026, as reported by Help Net Security.
• Regulation (EU) 2024/1689, the EU AI Act, Articles 12 and 14; ISO/IEC 42001.