Data center outage: what happens when one site goes down

Every data center has a bad day eventually. A power feed trips, a cooling unit fails, a fibre line gets cut by a digger two streets away. The question for anyone running AI in production isn't whether this happens. It's what your customers see when it does.

The answer depends less on how good one building is and more on how many buildings your work can use.

What an outage looks like from the outside

When a large site has a problem, the people using the AI service built on it see something simple: requests slow down, then fail. A chatbot stops answering. An image tool spins forever. An internal assistant returns an error at the moment someone needs it.

Behind that, a lot can be going on. Traffic may be queued while engineers work out what's wrong. Some of it may be sent to another region, if one is available and if the service was set up to do that. Some of it simply waits.

For buyers, the frustrating part is often the uncertainty. Is it down for everyone or just us? Will it be back in five minutes or five hours? Is anything being moved somewhere else, and if so, where?

One big site, or several

There are two broad ways to arrange AI capacity.

Concentrated. Most of the work runs in a small number of very large sites. These sites are built to a high standard, with backup power, spare cooling and several network links. Failures are rare. When they do happen, a lot of work is affected at once, and moving it elsewhere means finding room in another large site that may already be busy.

Spread out. The work runs across many smaller sites. Each site on its own is a smaller thing, and a single small site going offline affects less work. The trade-off is that a spread-out system needs good coordination. Something has to know which sites are healthy, which have room and which rules each request has to follow.

Neither approach is free of failures. They fail differently. A concentrated setup has fewer, larger incidents. A spread-out one has more, smaller ones, and depends on how well work moves between sites. Think of a single motorway compared with a grid of smaller roads. The motorway is quick until it closes. The grid takes more planning, but it rarely shuts down all at once.

What "moving the work" really involves

Sending a request somewhere else sounds easy. In practice, three things have to be true.

There has to be somewhere to go. Another site needs capacity free, the right kind of hardware and the right model ready to run.

The new place has to be allowed. If a request has to stay in a certain country, it can only move to sites in that country. If a customer has asked for a particular class of hardware, the new site needs it too.

Someone has to know it happened. If work moves, the buyer should be able to see that it moved, where to and why. Otherwise a reliability fix can quietly become a compliance problem.

This is where many systems fall short. They can fail over, but the rules about where data may go are checked loosely or not at all, and the record of what happened is thin.

Questions to ask before the bad day

You learn the most about a provider's resilience before anything breaks. A few questions help:

1. If the site running my work goes offline, where does my work go?
2. Does that backup location respect the same data residency rules as the main one?
3. How will I find out that my work was moved?
4. Can I see, afterwards, which requests ran where during the incident?
5. Is there a single place my work depends on, such as one region or one network link?
6. What happens to requests that were in flight when the problem started?

The answers don't need to be perfect. They need to be clear. A provider that can explain its plan in plain words usually has one.

Design choices that help

A few principles make a spread-out network easier to rely on.

Standard building blocks. When sites are built from the same repeatable units, a request that ran in one can run in another without surprises.

Rules that travel with the request. If each request carries its own constraints, on location, model or latency, the system can only move it to places that meet them.

A record of every move. If work changes site, that change is written down at the time, so buyers and auditors can see it later.

None of these removes failures. They make failures smaller and easier to explain.

Where Starfield fits

We're building standardized compute for many sites, connected through one programmable network for AI inference. The network is designed to place workloads by policy, on latency, data residency, model and load, which is what makes a second site an option during an incident.

That network is designed so that when a site has a problem, work can be placed on other sites that meet the same policy, and every workload can generate an auditable record of commitment, placement and delivery. If your work moves during an incident, the record shows where it went. The record itself is the subject of AI audit trail: a receipt for AI work, and AI compliance audit trail: trust is a record, not a promise covers why finance and compliance teams ask for it. Starfield Fabric is where we describe the network.

We'd rather talk about how a system behaves on a bad day than make promises about a good one.

If resilience is on your list for AI capacity, talk to us through the contact link on this page.