---
title: "The AI software factory: how far should you let go?"
description: "AI software factories, explained for executives. What a dark factory is, the four levels of letting go, and why blast radius picks your level, not the vendor."
type: Essay
url: https://sunilkumar.ai/writing/ai-software-factory/
author: Sunil Kumar (https://sunilkumar.ai/about/)
datePublished: 2026-09-16
pillar: Organisation
tags: [software-factory, agentic-ai, ai-dlc, pods]
license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
---

# The AI software factory: how far should you let go?

AI software factories, explained for executives. What a dark factory is, the four levels of letting go, and why blast radius picks your level, not the vendor.

## In short

- An AI software factory is a delivery setup where agents write, test and fix the code and humans stop reading it. StrongDM's version has two rules, code must not be written by humans and code must not be reviewed by humans, and only the second one is new.
- Manufacturing answered this question decades ago. FANUC runs robot-building lines unattended for up to thirty days, and EU law still requires a named Qualified Person to certify every batch of medicine. The lights go off where being wrong is cheap, and stay on where it is not.
- There are four levels of letting go. Agents draft and humans read every line, which does not scale. Agents write and humans check each small piece does what was asked. Agents write and test while humans keep a sealed set of tests the agents never see. And the dark factory, where nobody reads anything, which is unproven.
- A test written by the agent under test proves only that the agent agrees with itself. StrongDM's real contribution is the sealed test set, scenarios kept away from the agents, and it is the answer to that problem rather than a way around it.
- You do not choose a level. Your blast radius, how far the damage spreads when a change is wrong, chooses it for you. Changes that are easy to undo and affect nobody else can run at level three. Anything that touches money, health or an auditor stays at level two, with a named human signing off each release.

A term has started arriving in board packs this year, usually from a vendor and usually without a definition: the AI software factory. The idea behind it is simple and a little frightening. AI agents write the software, AI agents test it, AI agents fix what they broke, and the people who used to do those things stop reading the code altogether. One company, StrongDM, has published its version with two rules, and the rules are worth reading slowly. Code must not be written by humans. Code must not be reviewed by humans.

I run an engineering company that has put more than three hundred AI agents into production, doing real work every day for clients in healthcare, finance and government, and I want to answer the question a chief executive actually has about this, which is not "is StrongDM right" but "how far should we go". The honest answer is that manufacturing worked this out forty years ago, and the answer was never all the lights off in the whole building. Whatever level you pick, the software still has to meet the standard I call [Agentic in Production](/ideas/agentic-in-production/): an agent that survives Monday morning, not one that survives the demo.

> **AI software factory**: A software delivery setup in which AI agents write, test and repair the code, and humans decide where to stop reading. People still define what the software must do and the scenarios it must survive; the point at which they stop reading the output is a deliberate decision rather than an accident. The maximal version, where no human reads anything, is called a dark factory.

## Manufacturing settled this argument forty years ago

FANUC, the Japanese robotics company, began switching the lights off in its factories at the foot of Mount Fuji in the early 1980s. Robots build robots there. Machine tools make the parts for other machine tools. Lines run unattended for up to thirty days at a stretch, through weekends and holidays, and the people who remain do design, programming, preventive maintenance and supervision. Nobody stands on the line. It is the most famous example of what manufacturing calls lights-out production, and it has worked for four decades.

Now walk across Europe to a pharmaceutical plant. The filling line is automated to a degree FANUC would recognise. And yet not one batch of medicine can leave that building until a specific, named, registered individual, the Qualified Person, certifies in writing that it was made and checked correctly. This is not company policy. It is Article 51 of Directive 2001/83/EC. The Qualified Person carries personal legal liability, and the duty can be passed only to another Qualified Person, never to a system and never to a committee.

Both of these are right, and they are right for the same reason. The lights go off where being wrong is cheap. A misaligned robot arm makes a bad part that a sensor catches and a bin collects. A bad batch of medicine reaches a patient. Engineers have a phrase for this, the blast radius: how far the damage spreads when something goes wrong. Manufacturing did not decide between automation and accountability. It decided, room by room, which one each room needed.

That is the whole argument about AI software factories, and the rest of this piece is just the rooms.

## What an AI software factory actually is

Strip the marketing and StrongDM's factory has three parts. People write down what the software should do and the scenarios it must handle. Agents write the code and run it against those scenarios, over and over, fixing their own failures until the scenarios pass. Nobody reads the code in between. In StrongDM's words, validation replaces code review. They have said publicly that a team should expect to spend about a thousand dollars a day on AI usage per engineer, and that number set off a long argument on Hacker News, the forum where engineers go to disagree, that I will come back to.

The term is older than the idea. The United States Department of Defense has used "software factory" since the late 2010s for organisations like Kessel Run and Platform One, and there it means a shared, standardised way of building and shipping software that many teams use. StrongDM has borrowed the name for something quite different, which is part of why executives hear it and are unsure what is being sold. If someone offers you a software factory, ask which of the two they mean before you ask the price.

The part of StrongDM's writing that deserves the attention is not the two rules. It is a detail that Simon Willison picked out: they keep a set of test scenarios that the agents are never shown. The agents build to the scenarios they can see; the sealed scenarios, what engineers call a holdout set, decide whether the work is accepted. It is the difference between a student who has seen the exam paper and one who has not.

A software factory is not a tool you buy. It is a decision about which humans stop reading.

## Nobody argues about rule one

Code must not be written by humans. Read that rule again and notice that it is not controversial. In most engineering teams I visit, the majority of new code is already drafted by an agent and has been for a year. Engineers describe, agents write, engineers fix. The productivity argument is over and the agents won it.

Rule two is the actual claim. Code must not be reviewed by humans. Everything interesting about the software factory lives in rule two, and everything that went wrong in the four hundred and fifty comments under StrongDM's announcement was about rule two. Nobody argues about rule one.

The reason rule two is hard is arithmetic, and Michael Webster of CircleCI put it plainly in a talk this year: an agent produces fifteen hundred lines of code in minutes, a careful human reads about five hundred lines an hour, and in most companies a change already waits three to fourteen hours for a person to look at it. If the agents get faster and the humans do not change what they do, the humans become the factory's bottleneck and everyone can see it. There are two ways out. You can remove the human from the loop, which is StrongDM's answer. Or you can change what the human is asked to look at, which is ours.

## The four levels of letting go

![The four levels of letting go, drawn as four rooms. Level one, agents draft and humans read every line: the person at a desk under a bright light, buried in paper, marked "does not scale". Level two, agents write and humans check each small piece: the person at a gate checking parcels one at a time, marked "Ailoitte pods, client work". Level three, agents write and test, humans hold the sealed tests: the person outside the room holding a sealed envelope, marked "StrongDM, and our own tools". Level four, the dark factory: an empty dark room with a question mark, marked "unproven".](../../assets/images/writing/ai-software-factory-levels.png)

_Four levels, four rooms. The further right you go the fewer humans read the code, and the more the decision depends on what happens when the code is wrong. The figure is free to share: [download it as a PNG](/images/social/ai-software-factory-levels.png) (1600px wide, CC BY 4.0)._

**Level one: agents draft, humans read every line.** This is where most companies are today, whether they have named it or not. It feels safe and it is not, because a person cannot read fifteen hundred lines at nine in the morning, so they skim, approve, and hope. I wrote up [one change of ours that went through exactly that way](/notes/the-pull-request-nobody-could-review/), and it ended with a client's finance team stopping a payment run. Level one does not scale, and pretending it does is more dangerous than either of the levels beside it.

**Level two: agents write, humans check each small piece.** The agent may not hand over more than a person can read in twenty minutes; if the work is bigger, the agent splits it. The reviewer does not read for style. They ask one question: does this do what was asked, and nothing else. A named person owns the decision to release, the way the Qualified Person owns the batch. This is the lifecycle I call [AI-DLC](/ideas/ai-dlc/), and it is where Ailoitte runs all client work.

**Level three: agents write and test, humans hold the sealed tests.** Nobody reads the code. A person writes the scenarios, keeps a portion of them sealed, and the agents keep trying until the sealed ones pass. This is StrongDM's factory, and it is a genuine advance, because it answers the objection in the next section rather than ignoring it. It is also where we run our own internal tools.

**Level four: the dark factory.** No human reads anything, including the tests. Dan Shapiro coined the term, and the honest position in September 2026 is that it remains a hypothesis: no company has published evidence of running one at scale, and the four hundred and fifty comments under StrongDM's announcement include a good many engineers who have watched earlier attempts to take people out of software end with people quietly put back.

You do not choose a level. Your blast radius chooses it for you.

## Why the test the agent wrote is not an approval

The sharpest objection on Hacker News was a short one: if the agent writes the code and the agent writes the tests, the tests prove nothing. I think that is correct, and it is the reason level three needs the sealed envelope.

A test suite written by the thing under test proves only that the thing agrees with itself. We learned this before agents were writing code, when [a set of test cases assembled by the team that built an agent scored 94 percent and failed in production](/notes/the-evaluation-set-that-measured-the-wrong-thing/), because the team had tested their own assumptions. An agent testing its own code is the same mistake, made faster.

The sealed test set is the fix, and it is why I take StrongDM more seriously than most of their critics do. But notice what it requires: a human who writes scenarios the agent will never see, keeps them current as the product changes, and is accountable for what they cover. That is not zero humans. It is one human, doing something harder than code review, whose name is on the outcome. The lights are off in the room, and somebody is standing outside it with the key.

A test written by the thing it is testing is a signature on your own permission slip.

## How Ailoitte runs it: level two for clients, level three for ourselves

We deliver through [AI Velocity Pods](/ideas/ai-velocity-pods/), which are small senior teams, three to five people, that take one outcome to production for a fixed price. The pod has a lead who owns the test cases and the price, one or two engineers who own the agents, a platform engineer who owns the plumbing and the spending limits, and a reviewer from the client's own business who owns the list of what the agent must refuse.

For client work, a pod runs at level two, on purpose, and we got there the hard way. The clients are hospitals, lenders and government bodies. The blast radius of a wrong change is a patient record, a settlement, or a finding in an ISO 27001 audit that we, not the client, would have to explain. The rule about small units exists because an agent once handed us a 3,100-line change, a reviewer approved it in twenty-five minutes because every automated check had passed, and an insurer's finance team found the fault twelve days later, sixty-one dollars adrift across thirty-eight thousand settlements. That was level one failing quietly. Level two is what we built in response: the agents write, the units are small enough to read, a person reviews intent, and a named person releases.

For our own internal tools, the things only we use and can undo in a minute, we run at level three. The scenarios are sealed, the agents keep trying, nobody reads the change. It is faster and it is fine, because when it is wrong the cost is an afternoon of ours and nobody else's.

The same company, two levels, chosen by the same rule. We run the lights off in our own workshop and keep them on in the client's building.

## What it costs, and what it returns

Be sceptical of any claim, including mine, that does not come with a bill. StrongDM's thousand dollars a day per engineer is the most honest number anyone has published. It is roughly what a senior engineer costs in San Francisco and several times what one costs in Bengaluru, which is why the Hacker News thread argued about it for a day. The economics of level three depend on that number falling, and it is falling. But the AI bill is not the real cost of running dark. The real cost is that the moment something goes wrong, you have no record of who decided what, and a regulator, an auditor or a customer asks.

Level two costs more in people-hours per unit and returns something the factory cannot: a trail. Every unit has a person's name on the release. When the auditor comes, the answer takes a minute rather than three weeks of reconstruction. When a customer asks why, there is a human who can say. For a business whose product is trust, that trail is the product.

## If you are doing this in your company

List every system your agents touch, and sort it by what happens when a change is wrong. Easy to undo, nobody else affected, no regulator interested: one list. Money, health, personal data, or anyone who can fine you: the other. Do this in an afternoon with the people who would take the call.

Run level three on the first list and level two on the second, and write the boundary down in one page. The failure I see most is a company that has not decided, and is quietly at level one everywhere, which is the worst of both.

Name the human who signs off each release for every system on the second list, the way a pharmaceutical company names its Qualified Person. If nobody wants their name on it, you have learned something about that system.

Ask your engineering leader two questions: how big is a unit of agent work before a person sees it, and who writes the tests the agent cannot see. If the answers are "whatever the agent produces" and "the agent", you are at level one with the lights flickering.

Measure one thing before you change anything, so you can tell whether it worked. How many faults reach customers each month, or how often a change has to be undone, will do. Vendors will quote you speed; you want to know what reached your customers.

If you want help doing this, it is the work Ailoitte's pods do: the boundary, the sign-off, the sealed tests, and the agents on top, for a fixed price. If not, the rule is yours to keep. The lights go off where being wrong is cheap.

## Sources

- StrongDM, [The StrongDM Software Factory: Building Software with AI](https://www.strongdm.com/blog/the-strongdm-software-factory-building-software-with-ai) and [Software factories and the agentic moment](https://factory.strongdm.ai/), 2026. The two rules, validation replacing review, and the thousand-dollars-a-day figure.
- Simon Willison, [Software factory](https://simonwillison.net/2026/Feb/7/software-factory/), 7 February 2026. The holdout-set observation and Dan Shapiro's "dark factory" term.
- Hacker News, [Software factories and the agentic moment](https://news.ycombinator.com/item?id=46924426), 304 points and 459 comments at the time of writing. The circular-testing and cost objections.
- Michael Webster, CircleCI, [AI Works, Pull Requests Don't](https://www.infoq.com/presentations/ai-sdlc-pull-request/), InfoQ, 2026. The review-rate arithmetic.
- Industrial Automation India, [FANUC and Lights-Out Manufacturing](https://www.industrialautomationindia.in/articles/fanuc-lights-out-manufacturing-benchmark). Lights-out production at Mount Fuji since the early 1980s, lines unattended for up to thirty days.
- European Commission, [Directive 2001/83/EC](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32001L0083), Articles 48 to 51, and [EU GMP Annex 16](https://www.gmp-compliance.org/guidelines/gmp-guideline/eu-gmp-annex-16-certification-by-a-qualified-person-and-batch-release), certification by a Qualified Person and batch release. Personal liability; delegation only to another QP.
- Defense Unicorns, [Software origins of DoD software factories](https://medium.com/defense-unicorns/software-origins-of-dod-software-factories-with-enrique-oti-a8596a6d1c92). The earlier meaning of the term, Kessel Run and Platform One.


---
Sunil Kumar, Co-founder and CEO, Ailoitte Technologies. https://sunilkumar.ai/about/