Edge inference vs cloud inference: does location matter?

Ask an infrastructure company whether location matters and you will hear "always". Ask a busy product team and you will hear "never thought about it". Both answers are wrong, in opposite directions.

In the debate over edge inference vs cloud inference, the honest answer is that for most AI tasks, where the request runs barely matters. For a few, it matters more than anything else. This piece sorts them, so you know which pile your work is in.

Pile one: it barely matters

Some AI work has no one waiting for it and no sensitive data riding along. For this work, the location of the model is mostly a question of price.

Overnight and batch jobs. Summarising last week's support tickets, tagging a product catalogue, translating a document library. Nobody is watching the clock. If it runs in another country and takes a little longer, the result is the same.

Drafting and brainstorming. Writing a first draft of a blog post or a list of campaign ideas from public information. A short wait is fine, and the inputs are not confidential.

Research on public material. Asking a model to explain a concept or compare public products. The data is already public, and a person reading the answer will not notice small delays.

For this pile, choose on cost, model quality and convenience. Distance is a minor factor.

Pile two: it matters somewhat

The middle pile is where most business AI now sits. A person is waiting, or the data is mildly sensitive, or the task makes many calls in a row.

Chat assistants for staff and customers. People tolerate a short pause, but they notice slow replies, and slow replies get abandoned. Every extra round trip chips away at the experience.

Coding and agent tools. As we described in Your AI answer travels further than you think, one task can make dozens of model calls. Small delays add up into minutes.

Internal documents. Summarising contracts, HR files or board papers is not public work. Your company may have rules about where that material can be processed, even if no law says so.

For this pile, location is one factor among several. It is worth asking where requests run, how busy the service gets at peak times and what happens to the data afterwards.

Pile three: it matters a lot

Then there is work where location is the whole question.

Machines that act in real time. A robot picking parts, a vision system checking a production line, a camera network watching for hazards. If the answer arrives late, the moment has passed. These systems need their AI close by.

Voice and live video. A spoken conversation feels broken when replies lag. Live video analysis produces a lot of data that is slow and costly to send far away.

Regulated or personal data. Patient records, financial transactions, citizen data and many customer records come with rules about where they may be processed. Breaking those rules is a legal problem first, whatever it does to speed. (We cover the details in Data residency in one page.)

Very high volumes of data. When the input is huge, such as hours of footage or millions of sensor readings, moving it costs real money in network fees and time. Processing it near where it is created is often the only sensible option.

For this pile, choose location first. Then pick a suitable model and price offered there. A slightly smaller model running nearby will often serve this work better than a larger one far away.

How to sort your own work

Take each AI use case you run, or plan to run, and ask four questions:

1. Is anyone or anything waiting for the answer? If a person, distance matters somewhat. If a machine, it may matter a lot.
2. How many model calls does one task make? One is forgiving. Dozens are not.
3. What data goes in? Public, internal or regulated? The further along that line, the more location matters.
4. How much data goes in? A few sentences travel easily. Video and large files do not.

Mostly "no", "one", "public" and "a little" means pile one. Any answer of "a machine", "regulated" or "a lot of data" means pile three.

The mistake to avoid

The common mistake is treating every workload the same. Some teams send everything to a large, distant service because it was easy to set up, then discover months later that their voice assistant feels sluggish or their legal team will not sign off on patient data. Others over-engineer simple batch jobs that could run anywhere.

A sorted list avoids both. Each workload goes where it fits: a low-cost suitable place for pile one, a nearby and well-understood place for pile two, and a local, policy-controlled place for pile three. The difference between these places is the heart of the edge inference vs cloud inference question, and Edge AI inference: what close to the user really means takes it further.

Where the answer runs, by design

A workable setup lets you make this choice per workload, without running three different systems. That is what Starfield is building towards. Starfield Fabric is designed to deploy standardized compute close to power, connectivity and demand, then connect it through one programmable network for AI inference. In that design, workloads are placed by policy, on latency, data residency, model and load, so a batch job and a real-time vision system can each run where they belong. Every workload can generate an auditable record of commitment, placement and delivery, so you can show where each request ran. The design is set out at Starfield Fabric.

Does it matter where your AI request runs? Sometimes not at all. Sometimes completely. Knowing which is the first step.

If some of your workloads sit in pile three, use Request capacity below.