Latency and throughput in AI, explained

Think of a coffee shop. One measure is how long you wait for your own cup. Another is how many cups the shop can make in an hour.

A shop can be good at one and poor at the other. AI systems are the same. This piece explains the two words that describe them, latency and throughput, and why the difference matters when you buy AI capacity.

Latency: how long one answer takes

Latency is the wait for a single request. You ask a question, and latency is the time until something useful comes back. For AI, people often split it into two parts.

Time to first word. This is the pause before the answer starts to appear. It is what a person notices most in a chat window.

Time to finish. This is how long the whole answer takes to arrive. A long answer can start quickly and still take a while to complete.

Both parts depend on distance, on how busy the system is, and on the model itself. We look at distance in How far your AI answer travels.

Throughput: how much the system can serve

Throughput is volume. It is how many requests, or how much generated text, a system can produce over a period of time. It is a property of the whole system, not of one answer.

A system with high throughput can serve many people at once without falling behind. That matters for a business with lots of users, or for batch jobs that process large piles of documents overnight.

Throughput is shaped by the hardware, the model size, and how requests are grouped together.

Why vendors mix them up

The two words get blurred because both sound like speed. A quote might say a system is "fast" without saying which kind of fast it means.

The trouble is that the two often pull against each other. Grouping many requests together tends to raise throughput. It can also make each individual person wait a little longer.

So a system tuned for volume may feel slower to one user. A system tuned for quick single answers may serve fewer people per machine. Neither is wrong. They suit different jobs.

Which one matters for your work

Start with what the work actually is.

Live conversations. A support chat or a voice assistant lives or dies on the pause before the first word. Latency comes first here.

Bulk processing. Summarising an archive or tagging a catalogue does not need instant replies. Throughput, and the cost of that throughput, matter more.

Mixed use. Many businesses have both. It can make sense to send each kind of work to capacity set up for it, rather than forcing one setup to do everything.

Answering questions and training a model are also different jobs with different needs. See Training vs inference.

Questions to ask a provider

Speed claims are only useful if they are specific. These questions help pin them down.

1. When you say fast, do you mean the time to the first word, the time to finish, or total volume?
2. Under what load was that measured: a quiet system or a busy one?
3. Where was the measurement taken from, and how far is that from my users?
4. What happens to single-answer speed when the system is at its busiest?
5. Is the figure dated, and can I see how it changes over time?

The last question matters. A speed figure without a date and conditions is hard to rely on. We explain why in Why we publish dated prices. Starfield's Model Watch publishes dated speed figures for this reason. The glossary defines both words in short form.

What we are building

We think speed should be described in terms of the job, not as a single headline word. That means placing each kind of work where it fits.

We're building standardized compute close to power, connectivity and demand, connected through one programmable network for AI inference. The aim is to place workloads by policy, on latency, data residency, model and load, with an auditable record of commitment, placement and delivery for each workload.

Starfield Fabric, the control plane meant to make these placement decisions, is in development. It is designed to weigh latency and load together, rather than treating speed as one number.

If you are weighing speed claims for an AI workload, use Request capacity below.