Agentic AI reference architecture for enterprise

This playbook is the architecture I use when an enterprise asks Ailoitte to take an agent from a working demo to something that runs unattended on real work. It is not the only way to build an agent. It is a way to build one that passes the five production-grade tests, and it is organised so that each test has an obvious home.

The architecture has six layers. Each has one owner, one job and one checklist. The order matters: the first layer produces the documents that every other layer is built against.

Agent

A system in which a model decides, at run time, which of a bounded set of actions to take next in order to complete a task, and in which the consequences of those actions are real.

1. Define the work

Nothing in this layer is code. It is four documents, and they are the specification for everything below.

The task list. Each task the agent handles, with an example input, the expected outcome, and the systems involved. Keep it short at first. An agent that handles five tasks well is worth more than one that handles fifty tasks badly, and it is far easier to evaluate.

The refusal list. What the agent must never do (send money, change a dosage, delete a record, contact an external address) and what it must decline to attempt (anything outside the task list, anything where confidence is low, anything the user is not entitled to). Each item gets a reason code that the agent will show the user when it refuses.

The evaluation set. Real cases, drawn from production traffic where it exists, with expected outcomes. Include cases where the right outcome is a refusal and cases where the right outcome is an escalation to a person. Agree the pass rate. Two hundred cases is a workable minimum for a first agent; fewer than fifty and you are measuring luck.

The budget. Cost per run as a range, cost per month as a ceiling, behaviour at the ceiling. Latency is part of this: a run that takes ninety seconds is free to compute and expensive to wait for.

2. Choose the orchestration pattern

The orchestration layer decides what happens next. The mistake at this layer is reaching for the most autonomous pattern first. Do the opposite: pick the least autonomous pattern that passes the evaluation set.

A workflow is a graph you can draw. The model fills in steps (classify this, extract that, draft this) but the sequence is fixed in code. Most enterprise tasks are workflows, and a workflow is easier to test, cheaper to run and simpler to explain to an auditor. Anthropic’s note on building effective agents makes the same argument with examples; I would go further and say that if you can draw the graph, you should not build the loop.

An agent loop is for the cases where you cannot draw the graph: the model observes, decides, acts and observes again until a stopping condition. The pattern is well described in the ReAct paper. Use it for the sub-task that genuinely needs it, inside a workflow that bounds it, with a step limit and a budget.

Whichever pattern you use, the orchestration layer must not contain permissions. It decides what to try; the tool layer decides what is allowed. Mixing those two is the single most common architectural mistake I see, and it is the one that turns a prompt injection into an incident.

3. Build the tool layer as the boundary

The tool layer is where the agent touches the world, and it is the security boundary of the whole system. Every action the agent can take is a tool, and every tool is a small piece of ordinary, testable, boring software.

A tool has a typed input and a typed output, validated before anything runs. It has a permission check that uses the identity of the user on whose behalf the agent is acting, not a service account with god rights. It has a rate limit and a per-run call limit, because a model that gets an error will retry forever. Where the action has side effects, it has an idempotency key, so a retry cannot issue two refunds. And it writes an audit entry: who, what, arguments, result, cost.

The refusal list from layer one is implemented here. The refund tool checks the threshold. The email tool has an allow-list. The delete tool does not exist. When a tool refuses, it returns a structured refusal with the reason code, so the model can explain it and the trace can show it.

Two design choices that pay for themselves. First, keep read tools and write tools separate, and give the agent read tools generously and write tools sparingly. Second, make the tool interface standard across agents so a tool built for one can be reused by the next; the Model Context Protocol is a reasonable standard to adopt for that, provided the permission checks stay in your code and not in the protocol.

Human approval lives here too. A tool that needs a person’s sign-off (a payment above a threshold, a change to a customer’s record) pauses the run, notifies the approver with the trace so far, and continues or refuses on their answer. That is a tool with a slow response, not a special case in the orchestrator.

4. Design memory deliberately

“Memory” covers three different things, and treating them as one is how agents leak data.

Working context is what the model sees during a run: the task, the conversation so far, tool results. It is bounded by the context window and, more importantly, by attention; models use the middle of a long context less reliably than the ends, a finding documented in Lost in the Middle. Keep working context short and structured. Summarise tool results rather than pasting them.

Retrieval is fetching documents or records relevant to the task. Build it as a tool, with the same permission checks as any other tool, so that the agent can only retrieve what the user could read. Every retrieved passage carries a citation that survives into the output. Retrieval without citations produces confident answers nobody can check.

Long-term memory is anything the agent keeps between runs: preferences, past decisions, learned facts. Decide explicitly what may be stored, for how long, and who can read or delete it. In healthcare and finance the answer is often “nothing beyond the record of the run itself”, and that is a fine answer. An agent that remembers everything is a data-protection incident with a delay timer.

5. Put evaluation and guardrails in the request path

There are two kinds of checks, and both belong in the request path, where they can stop something, rather than in a dashboard, where they can only report it.

Guardrails run on every request. Input guardrails classify the incoming request: is it in scope, does it contain an injection attempt, does it contain data the agent should not process. Output guardrails check the result before it leaves: does it contain personal data it should not, does it make a claim the retrieved sources do not support, does it match the format the downstream system expects. A failed guardrail produces a refusal or an escalation, never a silent pass-through. The guardrail models can be small and cheap; the OWASP Top 10 for LLM applications is a good list of what they should be looking for.

Evaluation runs on every change. Take the evaluation set from layer one and run the full agent against it whenever the prompt, the model, a tool, the retrieval index or a guardrail changes. Record the pass rate, the cost and the latency. A change that drops the pass rate below the agreed threshold does not ship. This is a test suite in the pipeline, not a notebook someone runs before a demo.

Sample production runs into the evaluation set continuously. The set that was right at launch is wrong six months later, because the traffic moved. A weekly review of a random sample of traces, by a person who knows the domain, is the cheapest instrument you will ever buy.

6. Build operations before launch

Operations is the layer that makes the other five defensible, and it is the one most teams build last, which is why most teams do not launch. Build it in the first week. The definition phase already told you what it must show.

The trace. One record per run, containing input, context, each decision, each tool call with arguments and result, each guardrail verdict, the output, the cost and the latency. Rendered so that a person who did not build the system can read it. Retained for as long as the decision matters, which in regulated industries is years, not days.

The cost meter. Per-run cost computed from the trace, per-day and per-month totals, and the ceiling from layer one enforced: at the ceiling, the agent stops accepting new work and escalates.

The kill switch. A flag that a named person can flip from a phone, that stops new runs within a minute and lets in-flight runs finish or roll back cleanly.

The rollout. New versions go to a slice of traffic first, with the evaluation score and the guardrail failure rate compared against the current version, and an automatic revert if either moves the wrong way. Google’s writing on error budgets is the right mental model: agree how much failure is acceptable, spend it deliberately, and stop when it is gone.

The owner. A name, an alert channel, an on-call arrangement, and the authority to stop the system. Without this the other four are instruments nobody is reading.

How the layers map to the production-grade tests

Refusals enforced in code: layer three. Runs reconstructable: layer six, fed by layers three, four and five. A named owner with a kill switch: layer six. A budget it cannot exceed: layers one and six. Behaviour measured on every change: layer five. If you build the six layers, you pass the five tests, and you pass them in a way that an auditor can follow without a briefing.

If you are doing this in your company

Draw the six layers on a whiteboard and write the name of the person who owns each one. If the same name appears in every box, that is your first problem, and it is an organisational one. If a box is empty, that is the layer that will stop your launch.

Then take your current agent, demo or pilot, and mark which layers exist. In my experience the honest answer is usually two: orchestration and a set of tools with the permissions inside the orchestrator. Start with layer one anyway, because the four documents are what tell you how to fix the rest, and they take two weeks.

If you are choosing a vendor to build this rather than building it yourself, the vendor evaluation playbook turns each layer into questions to ask them.