Small language models: your competitors are already training theirs

Every call your company sends to a rented model teaches it something about your business: how you price, how you classify a claim, which invoice looks wrong, what a good answer to your customer sounds like. You pay for each of those lessons and you keep none of them. The model goes back to being the same model your competitor rents, at the same price, knowing nothing that is yours.

A growing number of enterprises have stopped accepting that trade. They are moving the routine work their AI does, which is most of it, onto small language models they fine-tune on their own data and run in their own environment. AT&T’s chief data and AI officer said in February that its small models run at about a tenth of the cost of large ones and are about as accurate. Gartner expects organisations to use small, task-specific models three times as much as general-purpose large ones by 2027. And most enterprises have not started, which is exactly why the gap is opening now, and why it will be hard to close later.

Small language model (SLM)

A language model small enough to run on hardware you control, typically under ten billion parameters, and usually fine-tuned for a narrow job rather than general conversation. NVIDIA's researchers define it by the device rather than the size: it fits on common hardware and answers fast enough to be practical.

The model everyone rents cannot be your advantage

Frontier models are extraordinary, and they are a utility. Your competitor can rent the same one tomorrow, on the same price list and the same terms. Whatever it does for you, it will do for them. A model your competitor can rent tomorrow is parity, not advantage, and parity is what most enterprises are paying a premium for.

The advantage moves to the thing they cannot rent: a model that has learned from your decisions. Gartner’s analysts put it plainly in their forecast. As enterprises recognise the value of their private data and the processes behind it, they expect them to specialise models on that data, and in time to offer access to those models to others. Picture an insurer whose claims model has seen ten years of its own adjudications, or a bank whose fraud model knows its own customers. When a model like that becomes a product, the company that built it has turned its data into revenue. The company that rented a frontier model for the same job has a receipt.

Most of an agent’s work is small work

Look inside any agent that does real work and most of what it does is narrow and repetitive: classify this, extract that, pick the next tool, check the format, summarise the call. Very little of it needs a model that can also write a sonnet and explain quantum field theory.

NVIDIA’s researchers went through three open agent frameworks and estimated that 40 to 70 percent of their large-model calls could be handled by specialised small models: about 60 percent in MetaGPT, 40 in Open Operator and 70 in Cradle. They put the cost of serving a seven-billion-parameter model at 10 to 30 times less than a 70 to 175-billion one, in latency, energy and compute. Their conclusion is the one I give clients: build agents that are small by default and large by exception.

The frontier model does not go away. It stays for the open-ended question and the case nobody has seen before; it simply stops being the model that answers everything. The reference architecture puts that decision in the gateway, and I have written about what routing the cheap calls away from the expensive model did to one bill in ten thousand runs a day changes the price. Small models are where that routing ends up.

The companies that moved pay a tenth for the same decision

The public cases are no longer experiments. AT&T uses small models of four to seven billion parameters, fine-tuned on its own data, for contract analysis, network root-cause analysis, fraud detection, dispatch and the intent behind customer transcripts. Andy Markus, its chief data and AI officer, told Forbes in February that they cost about 10 percent of what large models cost and are about as accurate. Forbes reports that the return on AT&T’s AI work doubled in a year, from two times to four.

Checkr, which runs about 1.5 million background checks a month, replaced GPT-4 on its hardest classification job with a fine-tuned eight-billion-parameter Llama model. Accuracy went from 88 to 97 percent overall, and from 82 to 85 on the messiest records. Responses went from seven to fifteen seconds to half a second. The bill went from roughly $7,000 to $12,000 a month to about $800.

ITC chose Microsoft’s small Phi-3 model for Krishi Mitra, its copilot for farmers, which reaches more than a million of them. ITC’s stated goal was to improve efficiency while keeping the accuracy of a large model.

Checkr’s numbers, as reported by Computerworld. Free to share: download the figure as a PNG (1600px wide, CC BY 4.0).

Read those three together and the pattern is the uncomfortable part. Each company now pays a fraction of what a competitor on a rented frontier model pays for the same decision, gets the answer faster, and has published the same or better accuracy where it has published accuracy at all. A competitor who pays a tenth per decision can afford to put AI into ten times as many processes. That is how a cost gap becomes a capability gap.

Waiting for prices to fall is the expensive option

The obvious objection is that frontier models keep getting cheaper, so why build anything? Gartner answered it in March. It expects the cost of inference on a trillion-parameter model to fall by more than 90 percent by 2030, and in the same release says providers will not pass those savings on to enterprise customers in full. Meanwhile agentic work needs 5 to 30 times more tokens per task than a chatbot. The unit price falls; the bill rises. Gartner’s own advice is to send routine, high-frequency tasks to small and domain-specific models and keep frontier models for the complex reasoning that justifies the price.

Waiting is a decision too. It is the decision to keep paying the frontier price for every routine call, on a volume that grows each time you add an agent. Meanwhile the companies that moved pay a fraction, and spend the difference on the next process.

Your next model is sitting in your logs

This is the part that should worry anyone who has not started, because it is the part money cannot buy later. A fine-tuned small model is made from examples of your own work: the inputs your systems saw, the outputs that were right, and the ones a person corrected. NVIDIA’s researchers describe the conversion as a loop that begins with exactly that. Log every call securely, curate it and strip what is sensitive, cluster the calls into recurring tasks, pick a small model for each, fine-tune it, and keep refining as new data arrives.

After Belcak and colleagues, NVIDIA Research. Free to share: download the figure as a PNG (1600px wide, CC BY 4.0).

Every step after the first can be bought or hired. The first cannot be done in arrears. If your AI calls are not being logged with their outcomes today, the dataset your model needs does not exist, and the only way to create it is to start collecting and wait. A competitor who started keeping those logs a year ago has a year of labelled examples that no budget can buy back for you. The first model is a quarter’s work. The data behind it is a year’s work, and that year starts the day you start keeping it. The evaluation set has to come from the same production cases, not from the team that built the model, which is the lesson of the evaluation set that measured the wrong thing.

The regulator is pushing the same way

For Indian enterprises there is a date on this. India’s Digital Personal Data Protection Rules were notified on 14 November 2025 with an 18-month phased timeline that runs to May 2027, and personal data leaves the country only on the government’s terms. A small model running inside your own environment answers a whole class of the questions a regulator will ask, about where personal data goes and who processes it, in one sentence. The same logic applies under the GDPR and the EU AI Act, and in every hospital and bank that already ring-fences its data.

One of the enquiries we took last month began with a single requirement: it has to run on our premises. In regulated industries, that is becoming the first line of the brief rather than the last.

What small models are bad at

None of this means the frontier model is going away, and an essay that told you so would be selling something. Small models are worse at open-ended reasoning, at breadth, at the question nobody has asked before, and at work that happens too rarely to be worth training for. They need data you may not have yet, an evaluation set built from production, and an owner who runs them like any other system that touches your data.

It is also true that most enterprises have not moved. Menlo Ventures’ report at the end of 2025 found open models holding 11 percent of the enterprise market, down from 19 percent the year before, with fine-tuning still a niche technique behind prompting and retrieval. You can read that as a reason to wait. I read it as the window. When most of your competitors pay the frontier price for routine work, the few who do not have an advantage that compounds every quarter, and the data that comes with it is the kind you cannot buy later.

Why Ailoitte is betting on small models

I should say where I stand, because I run a company that builds this. At Ailoitte we already have fine-tuned small models in production with clients.

For a provider network in New Zealand, a small model we fine-tuned handles the triage step of a patient app, inside the client’s own environment. In the first phase more than 26,000 patients came onto the platform, and 35 percent of routine queries are handled end to end without a person.

For a US clinical-documents platform, we fine-tuned a small model to do the extraction and classification under HIPAA, again inside the client’s environment. The system around it cut review time by 74 percent and raised throughput two and a half times, with extraction accuracy at 92 percent.

We are now building that into a practice: the team, the pipeline and the tooling to take a client from logged calls to a fine-tuned, evaluated, owned model as a matter of routine rather than a research project. It runs the way our AI Velocity Pods run everything else. The evaluation set comes first, the price is fixed once the work is defined, the model runs in your environment, and the weights, the data and the evaluation set are yours when we leave. I would rather a client owned a small model than rented a large one, from us or from anyone else.

We still use frontier models for the hard minority of calls. The point is to stop using them for everything.

If you are doing this in your company

Start logging tomorrow. Every model call, with its input, its output and what happened next: accepted, corrected or escalated. Keep it in your own environment, with personal data handled the way your regulator expects. It costs almost nothing, and it is the one step you cannot do later.

Pick one step, the highest-volume and narrowest your AI performs: a classification, an extraction, a routing decision. Write its evaluation set from production cases, including the ugly ones.

Fine-tune a model of three to eight billion parameters on those logs and run it in shadow beside the frontier model for a few weeks. When it passes the evaluation set inside its budget, route that step to it and keep the frontier model as the fallback.

Decide where it runs: your cloud, your data centre or the device. Then give the model an owner, a version history and a retirement date, like any other asset that carries your data.

If you want help with the first one, it is the work Ailoitte’s pods do, and the work with me page has the shape of it.

Sources

• Belcak, P. and colleagues, Small Language Models are the Future of Agentic AI, NVIDIA Research, arXiv 2506.02153, June 2025, revised September 2026 (the SLM definition, the 10 to 30 times serving cost, the MetaGPT, Open Operator and Cradle estimates, the conversion loop).
• Gartner, Gartner Predicts by 2027, Organizations Will Use Small, Task-Specific AI Models Three Times More Than General-Purpose Large Language Models, 9 April 2025.
• Gartner, Gartner Predicts That by 2030, Performing Inference on an LLM With 1 Trillion Parameters Will Cost GenAI Providers Over 90% Less Than in 2025, 25 March 2026.
• Koetsier, J., AT&T Says SLMs Run At 10% Of The Cost Of LLMs While Being ‘About As Accurate’, Forbes, 10 February 2026.
• Mearian, L., Checkr ditches GPT-4 for a smaller genAI model, streamlines background checks, Computerworld, 30 September 2024.
• Microsoft, Introducing Phi-3: Redefining what’s possible with SLMs, Azure blog, 23 April 2024 (ITC Krishi Mitra).
• Menlo Ventures, 2025: The State of Generative AI in the Enterprise, 9 December 2025.
• Press Information Bureau, Government of India, DPDP Rules, 2025 notified, 14 November 2025.
• Ailoitte client examples are anonymised; the outcomes are from Ailoitte’s 2026 case studies.