How AI services answer many people at once

Think of a lift in an office building. It does not carry one person at a time. It waits a moment, picks up everyone heading the same way, and moves them together.

AI services work in a similar way. They group requests and keep a line for those still waiting. This piece explains those two ideas, batching and queues, and why they make a busy hour feel different from a quiet one.

Why not answer one person at a time

The hardware that runs AI models is expensive and draws a lot of power. Serving one request at a time would leave much of it idle. We cover the power side in Why AI uses so much electricity.

AI chips are good at doing many similar calculations side by side. Handling several requests together makes better use of that ability. The cost of running the machine is shared across more answers.

This is one way AI services keep prices manageable. It is also why your request is rarely handled entirely on its own.

Batching: travelling together

Batching means grouping several requests and processing them in one go. Like the lift, the system may wait a very short moment to collect a group before it moves.

Modern systems are flexible about this. New requests can often join a group already in progress, and finished ones can leave. This keeps the hardware busy without making people wait for a whole group to finish.

There is still a balance to strike. Bigger groups serve more people per machine. They can also slow each individual answer a little. We explain this trade-off in our piece on latency and throughput.

Queues: waiting your turn

When requests arrive faster than the hardware can take them, they wait in a queue. At a quiet hour, the queue is short or empty, and requests start almost straight away.

At a busy hour, the queue grows. The work itself is no slower, but each request spends longer waiting to begin. That waiting shows up as a longer pause before the first word. We describe that pause in our piece on why the first word of an AI answer is the slow one.

If demand keeps rising, a service has a few options. It can add hardware, send work elsewhere, slow down some users, or turn requests away.

Why busy hours feel different

Demand for AI tends to follow people's working days. Many users arrive at similar times, so the same service can feel quite different across a day.

Longer waits to start. Queues grow, so the first word takes longer.

Slower streaming. Larger groups can mean each answer streams out a little more slowly.

Limits and errors. Some services cap how many requests a user can send. At peak times, those limits are reached more often.

Priority tiers. Some services let certain customers skip ahead. Others wait longer as a result.

None of this is a fault. It is what shared capacity looks like when many people want it at once.

Spreading the load

One way to soften busy hours is to have capacity in more than one place. Requests can then be sent to wherever there is room, not just to one crowded site.

Another is to separate urgent work from work that can wait. Overnight summaries and bulk tagging do not need to compete with live chats. We discuss failover in Data center outage: what happens when one site goes down.

Questions to ask a provider

These questions help you understand how a service behaves under pressure.

1. How do response times change between your quietest and busiest hours?
2. Are there limits on how many requests I can send, and what happens when I reach them?
3. Can urgent requests be kept separate from bulk jobs?
4. If one location is full, can my work be sent somewhere else, and where?
5. Do some customers get priority, and where do I sit?

Ask for answers that are dated and tied to stated conditions. Behaviour on a quiet test day can look very different from a busy week.

What we are building

Busy hours are a question of where capacity is, and how work is shared across it. We think that should be handled by clear rules rather than luck.

We're building standardized compute close to power, connectivity and demand, connected through one programmable network for AI inference. The aim is to place workloads by policy, on latency, data residency, model and load, with an auditable record of commitment, placement and delivery for each workload.

Starfield Fabric, our control plane, is in development. It is designed to send each request to capacity that suits it, taking current load into account.

If your AI use has busy hours, use Request capacity below.