TL;DR
Self hosting an LLM is cheaper than an API only above a volume threshold, and that threshold is far higher than the hourly GPU rate suggests. Self hosted LLM cost is dominated by GPU hours you have reserved, not by tokens you have served. On representative mid-2026 numbers, an 8B open-weight model on a specialist-cloud H100 breaks even against a mid-tier API somewhere around 800 million output tokens a month, once you account for redundancy, realistic utilisation and the engineering hours to run it.
Three things push the break-even up, and every optimistic spreadsheet omits at least one: a GPU bills the same idle as busy, one replica is not a production answer, and somebody has to operate the thing.
One variable moves the answer more than the GPU model does: the provider. The same H100 runs around $3 an hour on a specialist GPU cloud and $11 to $13 on a hyperscaler. That 4x spread can shift the break-even by a factor of three on its own.
Deciding build versus buy? We run the comparison honestly, including the version where the answer is to stay on the API. See how we approach AI development.
Key Takeaways
- Self-hosting wins above a volume threshold, not as a general principle. Below it, an API is cheaper and simpler.
- A GPU costs the same at 9% utilisation as at 90%, so peaky traffic silently multiplies your real cost per token.
- The same H100 is roughly 4x more expensive on a hyperscaler than on a specialist GPU cloud.
- Engineering time frequently exceeds the GPU bill at small scale, and it is the line most often left out.
- A 70B model needs more GPUs each and serves far fewer tokens per second, so it often never breaks even against a mid tier.
- Data residency, predictable spend and freedom to fine-tune can justify self-hosting well below the cost break-even.
The comparison most people run, and why it is wrong
The optimistic version goes like this. An H100 costs about $3 an hour. That is roughly $2,190 a month. It can serve thousands of tokens a second. Multiply it out and you get billions of tokens for the price of a rounding error on your API bill. Self-hosting looks like an obvious win.
Three assumptions are doing all the work there, and all three are wrong in the same direction.
One: you will use all of it. You will not. Traffic is peaky, and a GPU bills identically whether it is saturated or idle. If your average utilisation is 30%, which is generous for most products, your real cost per token is more than three times what the spec-sheet calculation implied.
Two: one machine is enough. It is not. A single replica means a single point of failure, no room for a deployment, and an outage every time you update a model. Production means at least two, so your floor doubles before you serve a single token.
Three: it runs itself. It does not. Someone deploys it, monitors it, plans capacity, handles failover, updates models and carries the pager. Twenty hours a month at a loaded engineering rate is a real line item, and at modest volume it often exceeds the hardware.
Put those three back in and the picture changes completely.
Put your own volume, provider and utilisation in and the calculator will find your crossing point.
Output sets your GPU capacity need.
Affects the API side of the comparison.
Median rate across dedicated GPU providers.
A 70B model needs more GPUs each and serves far fewer tokens per second.
Refine the inputsHide the detail
A GPU bills the same idle. Peaky traffic rarely clears 40%.
One replica is not a production answer.
Per replica, batched.
Compare like for like: a mid tier, not a frontier model.
What is moving this answer
Each replica can serve 2.0B output tokens/month at this rate. The other 70% is paid for and idle.
27% of the self-host total.
No utilisation fixes this. You need more volume or cheaper GPUs.
One caveat this model cannot price for you: an open-weight model is not the same model as a frontier API. Compare against the tier you would actually replace, not the most expensive one on the menu, or the break-even will flatter self-hosting.
GPU cost per hour: the number that decides it
Before optimising anything else, look at where you would rent. LLM hosting cost is mostly this one line.
The same H100 varies by roughly 4x depending on who you rent it from. Specialist GPU clouds sit around $2.50 to $3.30 an hour, with a median near $3. The major hyperscalers charge $9 to $13 an hour for the same silicon. A100 rates run roughly $1.30 to $2.50.
| GPU and provider type | Typical on-demand $/hr | Monthly, one GPU |
|---|---|---|
| H100, specialist GPU cloud | $2.50 to $3.30 | around $2,190 |
| H100, hyperscaler | $9 to $13 | around $8,000 |
| A100 80GB | $1.30 to $2.50 | around $1,450 |
| L40S | around $1.10 | around $800 |
This single choice usually settles the build vs buy question before any other variable is considered. If your business case only works on hyperscaler pricing, it does not work. If it only works on specialist pricing, be honest about whether you can accept the operational maturity difference that comes with it.
Serverless GPU is worth a mention here, because it looks like the answer to the utilisation problem. Pay only for the seconds you compute, and idle stops costing anything. The trade is cold starts, which for a large model means loading many gigabytes of weights before the first token, and per-second rates well above dedicated equivalents. It genuinely helps spiky, latency-tolerant batch work. It rarely helps an interactive product.
Utilisation is the variable that kills business cases
Of everything on this page, this is the one that surprises teams most, so it is worth stating plainly.
A GPU bills the same when it is idle. Your traffic has a daily shape, a weekly shape, and a growth curve. Unless you are running continuous batch work, average utilisation of 20% to 40% is normal, and every point below 100% multiplies your effective cost per token.
There are real ways to raise utilisation: batching aggressively, colocating multiple workloads on the same replica, moving asynchronous jobs onto the same fleet, and scaling replicas down out of hours. All of them are engineering work, which loops back to the third hidden cost.
The line nobody puts in the spreadsheet
Serving your own model is an ongoing operational commitment, not a one-off deployment.
Somebody has to build the serving stack, typically vLLM or a managed equivalent, manage model versions and rollbacks, plan capacity ahead of growth, handle node failures, keep latency inside your product's budget, patch security issues, and be reachable when it breaks at 3am. Whether that is twenty hours a month or eighty, it is a real cost, and at modest volume it frequently exceeds the GPU bill.
This is also the cost that does not shrink when traffic does. Your API bill halves in a quiet month. Your GPU reservation and your engineer do not.
When self-hosting genuinely wins
None of this is an argument against self-hosting. It is an argument against choosing it for cost reasons that do not survive arithmetic. Above the break-even, the saving is large, durable, and gets better as you grow.
Self-hosting is usually right when:
- Volume is high and predictable. Steady, large, continuous load is the ideal shape, because it is the one that fills a GPU.
- Data residency or privacy rules out an API. If sending data to a third party is not permitted, or the deployment must be on premise, the cost comparison is not the deciding factor.
- You need to fine-tune freely. Owning the weights means iterating without per-token training economics or provider restrictions.
- You want predictable spend. A fixed monthly infrastructure line is easier to plan against than a metered bill that scales with success.
- Latency or rate limits are binding. Dedicated capacity means no shared-tenancy queueing and no throttling at the worst moment.
And it is usually wrong when volume is modest, traffic is spiky, the team has no infrastructure capacity to spare, or the honest comparison is against a cheap mid-tier API rather than a frontier model.
The comparison trap: open weights are not the frontier
One methodological point, because it inflates more business cases than any other.
Teams compare a self-hosted 8B open source LLM against frontier API pricing, which is comparing two different products. Open source LLM cost and frontier API cost are not the same axis. If the work genuinely needs frontier reasoning, an 8B model will not do it at any price, and the comparison is meaningless. If the work does not need frontier reasoning, then the honest comparison is against a cheap mid-tier API, and that is a much harder bar to beat.
Run the calculator against the tier you would actually replace. On our defaults, moving the comparison from a mid tier to frontier pricing drops the break-even by roughly four times, which tells you how much of a typical business case is really about model tier rather than about hosting.
The related question, whether a small model can do the job at all, is covered by NVIDIA's research on small language models in agentic systems, which puts small models at 10x to 30x cheaper on narrow tasks and frequently just as accurate.
Do the four cheaper things first
Self-hosting is the fifth and most committing lever in the sequence, and it is worth being blunt about the order.
Changing where inference runs is a large architectural commitment with an ongoing operational tail. Capping an agent loop, trimming carried context, caching a stable prefix and routing narrow work to a smaller model are all cheaper, faster, reversible, and they reduce the volume that the self-hosting calculation is based on. Doing them first frequently moves you from "above break-even" to "comfortably below it", which means the right answer changes.
That ordering, and what each earlier lever is worth, is in our guide to LLM cost optimization. If your workload is an agent, AI agent costs will usually find more savings than a hosting change would.
How we run this decision at HorizonLux
HorizonLux is a software development company specialising in AI, and build-versus-buy on inference is one of the questions we are asked most often. Our approach is deliberately unglamorous.
- We run the four cheaper levers first, then recalculate. The volume that goes into the hosting decision should be the optimised volume, not the current one.
- Utilisation is estimated from real traffic shape, not assumed. If nobody has the data, we instrument for a fortnight before modelling anything.
- Engineering hours go in the model at a loaded rate. A comparison that treats operational work as free is not a comparison.
- We compare against the tier we would actually replace. Benchmarking an open-weight model against frontier pricing produces a number nobody can act on.
- We say "stay on the API" when that is the answer, which it usually is below a billion output tokens a month.
What to do this week
- Find your monthly output token volume. It is the number that drives GPU capacity, and most teams know their spend but not this.
- Get a real utilisation estimate from your traffic shape rather than assuming saturation.
- Price the same GPU at a specialist cloud and at your hyperscaler. The gap is often the whole decision.
- Put engineering hours in the model at a loaded rate, including on-call.
- Run the four cheaper levers first, then recalculate. The answer changes more often than you would expect.
Self-hosting is not cheaper than an API. It is cheaper past a volume, and knowing that volume is the difference between a sound infrastructure decision and an expensive one made from a spec sheet.
Want the comparison run properly? We model utilisation, redundancy and operational load, and we will tell you when the answer is to stay on the API. See how we approach AI development, or tell us what your invoice looks like.
Frequently asked questions
Is self-hosting an LLM cheaper than using an API?
Only above a volume threshold, and the threshold is higher than an hourly GPU rate suggests. On representative mid-2026 figures, an 8B open-weight model on a specialist-cloud H100 breaks even against a mid-tier API around 800 million output tokens a month, once redundancy, 30% utilisation and engineering time are included. Below that an API is cheaper and considerably simpler.
What does it cost to run LLM locally or on your own GPUs?
The dominant line is GPU hours, not tokens. A single H100 runs roughly $2,190 a month at specialist-cloud rates and closer to $8,000 at hyperscaler rates, and production means at least two replicas. Add engineering time to operate it. The resulting number is fixed: it does not fall when your traffic does, which is the key behavioural difference from an API.
How much does a GPU cost per hour?
As of mid-2026, an H100 is roughly $2.50 to $3.30 an hour on specialist GPU clouds and $9 to $13 an hour on the major hyperscalers, which is about a 4x spread for identical hardware. An A100 80GB runs roughly $1.30 to $2.50, and an L40S around $1.10. The provider choice usually moves a self-hosting business case more than the choice of GPU does.
Why does GPU utilisation matter so much?
Because a GPU bills the same idle as saturated, so every point of unused capacity raises your effective cost per token. At 30% average utilisation, which is normal for interactive products with peaky traffic, your real cost per token is more than three times the figure calculated from a specification sheet. Utilisation, not hardware price, is what most often breaks a self-hosting business case.
Is self-hosting a 70B open source model worth it?
Usually not, unless volume is very large or a requirement rules out APIs. A 70B model needs multiple GPUs per replica and serves far fewer tokens per second than an 8B one, so cost per token rises on both axes simultaneously. Against a cheap mid-tier API it often never breaks even at all. Against frontier pricing the case is much stronger, but then check whether an 8B model would have done the job.
What are the non-cost reasons to self-host?
Data residency and privacy requirements, freedom to fine-tune on your own data without per-token training economics, predictable fixed spend rather than a metered bill, no rate limits or shared-tenancy queueing, and independence from a provider's deprecation schedule. Any of these can justify self-hosting well below the cost break-even, and if one is a hard requirement then the arithmetic is not really the deciding factor.
Should I use serverless GPUs instead?
They solve the idle problem by billing per second, which is genuinely useful for spiky or asynchronous batch work. The trade is cold starts, since a large model must load many gigabytes of weights before producing a first token, and per-second rates well above dedicated equivalents. For interactive products with latency budgets, serverless usually costs more than it saves.
