AI training vs inference: two different jobs

Think of a medical school and a hospital. One trains doctors over years, in a few big institutions. The other sees patients every day, in every town. Nobody would build one in the shape of the other.

AI has the same split, and most buyers only need the hospital.

Two jobs that share a name

When people talk about "AI compute", they usually lump two very different jobs together.

Training is teaching a model. You feed it huge amounts of text, images or code, and it slowly adjusts billions of internal numbers until it gets good at predicting what comes next. This happens a few times per model, in long runs that can last weeks. It needs enormous numbers of chips working tightly together in one place, connected by very fast links, because they constantly share their progress.

Inference is using the model. Every time someone asks a question, summarises a file or gets a code suggestion, the finished model runs once to produce an answer. Each request is small. But requests arrive constantly, from everywhere, all day and all night.

Same chips, broadly. Same models. Completely different jobs. The electricity side of this is covered in why AI uses so much electricity.

What training needs

Training rewards concentration. The bigger the cluster, and the faster the links inside it, the sooner a model is finished. A few hundred metres of extra cable inside a building can matter. Location relative to users barely matters at all, because no user is waiting on a training run in real time.

So training gravitates to very large campuses with huge power connections. A lab training a frontier model wants as much tightly coupled capacity as it can get in one spot, and it will plan around grid connections that take years to arrive.

What answering needs

Inference rewards almost the opposite.

It runs all the time. A product with users never switches its AI off. Demand rises in the morning in one region and in the evening in another. Capacity has to be available continuously, not in occasional bursts.

It is sensitive to distance. A person waiting for a reply feels every extra round trip. A machine on a factory line or a camera watching a junction may not be able to wait at all.

It cares where data goes. Inference touches live customer data: messages, records, images. Rules and contracts often say where that data may be processed.

It comes in many sizes. Some requests need a large model. Many are well served by a smaller one. Matching each request to the right model and the right hardware saves money and power. What a megawatt buys you looks at how that matching decides how much work a site delivers.

None of these favour one giant building far away. They favour capacity spread out, close to users, and run as one system.

Why the difference matters to buyers

Most companies will never train a frontier model. They will use models other people trained, open or paid, and run them for their own customers and staff. Their bill, and their user experience, is almost entirely inference.

Yet much of the market still sells capacity shaped for training: long commitments, large blocks, a few regions. A buyer who needs steady, local, policy-aware answering ends up renting something built for a different job. They pay for scale they do not use, and they accept distance and data paths they would not choose.

This is a version of a point Starfield CEO Paweł Czech made on theCUBE: most buyers do not need hyperscale capacity, they need the right capacity, close to them, at a fair price. (We wrote about that idea in Stop buying the sun.)

A quick way to tell which job you have

Ask yourself four questions about your AI workload:

1. Are you changing the model itself, or only using it? If you are only using it, you are doing inference.
2. Does someone, or something, wait for each answer? If yes, distance matters.
3. Does the work run every day, or in occasional big batches? Daily, always-on work is answering.
4. Does it touch customer or regulated data? If yes, where it runs matters as much as how fast.

If most of your answers point to inference, shop for inference. Ask providers where the capacity sits, how close it is to your users, how they handle data residency and whether you can see what was delivered.

Fine-tuning sits in between

There is a middle ground. Fine-tuning takes an existing model and adjusts it on your own data so it fits your task better. It is a small training job: shorter, cheaper and far less demanding than training from scratch. Many companies do it occasionally, then run the tuned model as inference for months. It is a reason to keep some flexible capacity on hand, not a reason to buy a training cluster.

Building for the hospital

The AI industry spent its first big wave building medical schools. That was necessary. Models had to be trained before anyone could use them.

The next wave is about the hospitals: capacity that serves answers every day, near the people who need them. That is the job we're building Starfield Fabric for. It is designed to deploy standardized compute close to power, connectivity and demand, then connect it through one programmable network for distributed AI infrastructure. The network is designed to place workloads by policy, on latency, data residency, model and load, and designed so every workload can generate an auditable record of commitment, placement and delivery.

If your AI bill is mostly answering, buy capacity built for answering.