What it takes to run an open model yourself

Buying a car outright can be cheaper than taking taxis. But you also take on insurance, servicing, parking and the time it takes to look after it. Whether it pays off depends on how much you drive.

Running an open-weight model yourself is a similar choice. The weights may cost nothing to download, but everything around them has a cost. This piece sets out the hardware, people and monitoring involved, so you can weigh it against paying per use. For the background, see our piece on open-weight and paid AI models.

The hardware

A model has to fit in the memory of the chips that run it. Most serious inference today runs on GPUs or similar accelerators, and larger models may need several working together.

Enough memory. The model's weights, plus working memory for each active request, must fit. Longer inputs and more users need more.

Enough chips for peak demand. You size for your busiest hour, not your average. Spare capacity sits idle the rest of the time.

A home for the hardware. Servers need racks, power, cooling and a network connection. That means your own facility, colocation space, or rented servers from a cloud provider. See What a megawatt buys you in AI.

Spares and redundancy. Hardware fails. If one server goes down, something else must take the load. We cover this in Data center outage: what happens when one site goes down.

The software

The model file alone does not answer requests. It needs a serving layer around it.

An inference server. Software that loads the model, accepts requests, batches them together and streams answers back. Its settings strongly affect cost and speed.

Scaling and routing. Rules for adding or removing capacity, and for spreading requests across machines.

Access and security. Authentication, network controls, logging and protection against misuse.

Updates. New model versions, security patches and driver updates all need testing before they go live.

The people

This is the cost buyers most often underestimate. Someone has to own the system day and night.

Typically that means skills in infrastructure and networking, machine learning operations, security, and on-call support. A small team can run a modest setup. A large or critical one needs cover across time zones.

These people also need to keep up with a fast-moving field. Serving software and good practice change often.

The monitoring

Once the model is live, it needs watching on several fronts.

Health. Are servers up, and are chips running hot or failing?

Speed. How long do users wait for the first word and for the full answer?

Load. How busy is the hardware, and is there headroom for a spike?

Quality. Are answers still good, or has a change made them worse?

Cost. What does each request actually cost, once hardware, power and people are counted?

Without this, problems surface as user complaints rather than alerts.

Weighing it against paying per use

The comparison is not simply "free model" versus "per-token price". It is the full cost of running versus the full cost of buying. We list the less visible costs in Hidden costs of AI capacity that aren't on the quote.

Running it yourself tends to make more sense when:

• volume is high and steady, so hardware stays busy;
• data must be processed in a particular place, and paid services cannot meet that;
• you need a fixed model version, or a fine-tuned one;
• you already have the skills and facilities.

Paying per use tends to make more sense when:

• volume is low or unpredictable;
• you want to try several models before committing;
• you do not want to hire or build an operations team.

For dated per-use prices to set against your own running costs, see Model Watch.

Questions to ask before you decide

1. What is our expected volume at peak, and on an average day?
2. Which model sizes do we need, and how much memory do they require?
3. Where must the data be processed?
4. Who will be on call, and do we have those skills today?
5. What is our plan if a server, or a whole site, fails?
6. Have we counted power, space, staff and spares, not just chips?

What we are building

There is a middle path between running everything yourself and sending everything to a single paid service. We're building towards a setup where you choose the open model and an operator runs the hardware in the places you need.

We're building standardized compute close to power, connectivity and demand, connected through one programmable network for AI inference. Workloads are designed to be placed by policy, on latency, data residency, model and load, and the network is designed so every workload can generate an auditable record of commitment, placement and delivery.

We are building so that choosing an open model does not have to mean building an operations team.

We're building towards running open models without you operating the hardware yourself. If that is what you are planning, use Request capacity below.