AI SRE agents: Uber's Debug Assist, and how to adopt one
An AI SRE agent finds the root cause of a production issue and proposes the fix. How Uber's Debug Assist works, how Ailoitte runs it, and how to adopt one.
Tagged “evaluation”
Everything Sunil Kumar has written about evaluation in the context of agentic AI in production.
Agent harness playbook: let the model choose, not act
How to add a decision layer to an agent harness without handing the model authority. Eight steps for a CoE or R&D team, the risks, the compliance checks, a lab.
Fixed price for agentic AI: how pods make it possible
Fixed-price agentic AI works when "done" is an evaluation set, a refusal list and a budget. Why hourly billing fails for agents and how pods price the work.
What "production-grade" actually means for an AI agent
Production-grade is a checklist, not a feeling. Five tests an enterprise AI agent must pass, on refusals, traces, ownership, budget and evaluation.
Why most enterprise AI agents never leave the demo
Enterprise AI agents fail between demo and production for organisational reasons, not model reasons. The three gaps they die in, and what closes each one.
Agentic AI reference architecture for enterprise
A six-layer reference architecture for enterprise AI agents - work definition, orchestration, tools, memory, guardrails, operations - with steps and checklists.
How to evaluate an agentic AI vendor
Seven steps to evaluate an agentic AI vendor before you sign - refusals, evaluation set, traces, cost per run, compliance, ownership, pilot. With a checklist.
The evaluation set that measured the wrong thing
A document agent scored 94 percent on its evaluation set and failed in production. The set had been written by the team that built the agent.