GPUs and CPUs: what AI needs from each

Imagine a kitchen with one head chef and a long line of prep cooks. The head chef handles many different tasks and makes the decisions. The prep cooks each do one simple job, side by side, very quickly.

Computers that run AI are built in a similar way. The ordinary processor, the CPU, is the head chef. The graphics chip, the GPU, is the line of prep cooks. This piece explains what each does, and why AI needs both.

What a CPU is good at

A CPU is the general-purpose processor found in almost every computer. It is built to handle many kinds of tasks, one after another, and to switch between them quickly.

It is good at decisions, branching logic and keeping a system organised. It runs the operating system, handles files and networks, and coordinates everything else in the machine.

A CPU has a modest number of powerful cores. Each can do complex work, but there are not many of them.

What a GPU is good at

A GPU was first designed to draw images on screens. Drawing an image means doing the same simple sum for millions of pixels at once.

To do that, a GPU has a very large number of simpler cores working in parallel. It is less flexible than a CPU, but quicker at doing the same kind of sum many times over.

It also has its own fast memory, close to those cores. That lets it feed data to all of them without long waits.

Why AI leans on GPUs

A modern AI model is, at heart, a huge set of numbers. Running it means multiplying and adding those numbers together, over and over.

Those sums are simple and highly repetitive, and many can be done at the same time. That matches what GPUs were built for. So both training a model and answering with it lean heavily on GPUs. We compare the two jobs in Training vs inference.

There are also other chips designed specifically for AI. They follow the same idea of many simple calculations in parallel. In this piece, we use "GPU" to cover this family of accelerators.

What the CPU still does

A GPU does not run on its own. Every AI server still has CPUs, and they do essential work.

Running the system. The CPU runs the operating system and the software that manages the machine.

Preparing requests. It receives incoming requests, turns text into the form the model uses, and turns the output back into text.

Coordinating the GPUs. It hands work to the GPUs, tracks what they are doing, and moves data in and out of them.

Everything around the model. Checks, filters, logging, looking things up and talking to the network often run on CPUs.

If the CPU side is too weak, expensive GPUs can sit waiting. A well-balanced server keeps both busy.

Memory and connections matter too

Chips are only part of the picture. Large models need a lot of fast memory to hold them, and that memory can limit a system as much as the chips do.

The links between chips also matter. Big models can be spread across several GPUs, and those GPUs need to share data quickly. The same is true between servers in a data centre.

Then there is power and cooling. GPUs draw a lot of electricity and produce a lot of heat. We look at this in Why AI uses so much electricity and What a megawatt buys you in AI.

Questions to ask about hardware

You do not need to be an engineer to ask useful questions.

• Which kind of accelerator will run my work, and is it suited to answering rather than training?
• Is there enough memory for the model I want to use?
• Are the servers balanced, so GPUs are not left waiting on other parts?
• Is the hardware the same across sites, or does it vary?
• How are power and cooling handled where the hardware sits?

Consistency matters more than many buyers expect. We explain why in Modular data center: why standard units matter.

What we are building

We think AI hardware should be chosen for the job, then deployed in a consistent way, so that work is designed to move between sites without surprises.

We're building standardized compute close to power, connectivity and demand, connected through one programmable network for AI inference. The aim is to place workloads by policy, on latency, data residency, model and load, with an auditable record of commitment, placement and delivery for each workload.

If you are planning hardware for AI inference, use Request capacity below.