When a smaller AI model is enough

You would not send a lorry to deliver a letter. It would get there, but it would cost more, take longer to park and use far more fuel. The right vehicle depends on the load.

AI models come in many sizes, and the large ones get most of the attention. Yet many everyday tasks run well on much smaller models. This piece explains what model size means, which tasks suit a smaller model, and how the choice changes cost and waiting time.

What "size" means for a model

A model's size usually refers to how many weights, or parameters, it has. More weights let a model hold more knowledge and handle harder reasoning.

More weights also mean more work for every token. A larger model needs more memory, more chip time and more electricity to read a prompt and write an answer. Our piece on input tokens and output tokens explains the reading and writing steps.

Tasks a smaller model often handles well

Smaller models tend to do well when the task is narrow, well defined and has a clear right answer. Common examples include:

Sorting and labelling. Routing support tickets, tagging documents or detecting a language.

Pulling out details. Extracting names, dates or amounts from forms and emails into a set format.

Short summaries. Condensing a message or a meeting note where the source is clear.

Simple rewriting. Fixing tone, grammar or format to a known style.

Answering from supplied text. Responding to a question when the relevant passage is given in the prompt.

A smaller model can also be fine-tuned on your own examples. For a narrow task, that can close much of the gap with a larger general model.

Tasks that may need a larger model

Some work benefits from more capacity. The signs are tasks that are open-ended, long, or need careful reasoning.

Multi-step reasoning. Planning, complex analysis or problems where one error spoils the result.

Broad knowledge. Questions that rely on what the model already knows, across many subjects.

Long or mixed inputs. Working across long documents, several sources, or mixed text and code.

High stakes. Cases where a wrong answer is costly and checking is hard.

Even here, a mix can work. A small model can handle the easy cases and pass the hard ones to a larger model.

What changes when you pick a smaller model

Cost. Smaller models usually cost less per token, whether you pay per use or run them yourself. At high volume, the difference adds up.

Waiting time. Fewer weights to move for each output token generally means faster answers. That matters for anything a person waits on.

Where it can run. Smaller models need less memory, so they fit on more kinds of hardware. That widens the choice of location, which can help with latency and data residency. We look at distance in How far your AI answer travels.

Energy. Less work per token generally means less electricity per answer. See Why AI uses so much electricity, in plain words.

The trade-off is quality. A smaller model may make more mistakes on harder inputs, so the saving only counts if the results stay good enough.

How to tell if a smaller model is enough

The reliable way is to test on your own work, not on public scores alone. Our piece on how to read an AI benchmark explains why. A simple approach:

1. Collect a sample of real requests, including some hard ones.
2. Decide what a good enough answer looks like, before you start.
3. Run the same sample through a small, a medium and a large model.
4. Compare quality, cost and waiting time side by side.
5. Pick the smallest model that meets your bar, and recheck it from time to time.

For dated prices and speeds across model sizes, see Model Watch.

Matching the model to the job

Many organisations will end up using several models. A small one for high-volume routine work, a larger one for difficult cases, and a rule that decides which handles each request.

That approach treats model choice as an ongoing operating decision rather than a one-off purchase. It also means the infrastructure needs to run different models in different places without much friction.

What we are building

We think the right model, in the right place, is usually a better answer than the largest model everywhere.

We're building standardized compute close to power, connectivity and demand, connected through one programmable network for AI inference. Workloads are designed to be placed by policy, on latency, data residency, model and load, and the network is designed so every workload can generate an auditable record of commitment, placement and delivery.

Matching the work to the model is part of keeping AI affordable.

If you are running high-volume AI tasks and want to right-size them, use Request capacity below.