TL;DR
Every comparison tells you vLLM wins on throughput, which is true and not very useful. Two things they leave out. First, the two most-cited benchmarks disagree by roughly 2.5x on vLLM's actual numbers, because they ran different hardware, versions and quantization, so neither figure transfers to your setup. Second, there is one metric where Ollama genuinely beats vLLM under load: inter-token latency. That looks like a win and is actually the clearest evidence of the problem, because Ollama keeps it low by making most of your users wait in a queue.
Key Takeaways
- Ollama defaults to 4 parallel requests. Not a tuning oversight, a design choice for single-user work.
- Red Hat measured 793 TPS for vLLM against 41 TPS for Ollama on an A100, with P99 time-to-first-token of 80ms against 673ms at peak.
- A second benchmark on an RTX Pro 4500 put vLLM at 2,031 t/s, roughly 2.5x the Red Hat figure on nominally weaker hardware. Version and quantization differences dominate. Do not quote either number as "vLLM's speed".
- Ollama wins on inter-token latency above ~16 concurrent requests, because throttling keeps its active batch small. The cost lands in TTFT instead.
- Tuning Ollama does not fix it. At
OLLAMA_NUM_PARALLEL=32throughput still plateaus and ITL becomes erratic, which is head-of-line blocking. - Agent loops are where it compounds. A 20-step agent is roughly 60 to 80 seconds of inference on Ollama against 3 to 5 seconds on vLLM.
- Four separate Ollama instances on four GPUs is a legitimate pattern for a small team, and almost nobody mentions it.
The number everyone quotes, and why it does not transfer
Search this comparison and you will get a throughput figure presented as fact. The two most-cited published benchmarks do not agree.
| Red Hat | Exxact | |
|---|---|---|
| GPU | NVIDIA A100-PCIE-40GB | NVIDIA RTX Pro 4500, 32GB |
| Model | Llama 3.1 8B Instruct | Llama 3.1 8B |
| vLLM peak throughput | 793 TPS | 2,031 t/s (BF16), 4,870 (NVFP4) |
| Ollama | 41 TPS | 134 t/s single user |
| vLLM TTFT | 80ms P99 at peak | ~25ms (BF16), ~13ms (NVFP4) |
| Ollama TTFT | 673ms P99 at peak | ~500ms |
A workstation card reporting 2.5x the throughput of an A100 is not a contradiction and neither benchmark is wrong. They ran different vLLM versions, different quantization, and different concurrency ranges. Red Hat used original uncompressed weights; the higher Exxact figures lean on NVFP4, which needs Blackwell and is not available on Ollama at all.
The useful conclusion is the ratio, not the number. Both put vLLM roughly 15 to 20 times ahead of Ollama on aggregate throughput, and both put Ollama's TTFT under load at 500 to 700ms against vLLM's 13 to 80ms. Those directions are consistent. The absolute figures are properties of a specific rig.
Worth knowing about the sources: Red Hat sells a vLLM-based inference server and Exxact sells GPU systems. Both have a reason to like the result they published. The reason to believe them anyway is that practitioners on r/LocalLLaMA reach the same conclusion independently, with no product to sell, and that thread ranks alongside both benchmarks.
The metric where Ollama wins
Here is the part almost every comparison omits, and it is the most interesting thing in the data.
Red Hat measured inter-token latency, the gap between successive tokens once generation starts. Above roughly 16 concurrent requests, vLLM's ITL rises while Ollama's stays low and flat. On that chart, Ollama beats vLLM.
Read it without context and you would conclude Ollama generates text more smoothly under load. That is true, and it is true for a reason that should stop you using it in production.
Ollama keeps ITL low by refusing work. With four slots, it only ever has a handful of requests in flight, so each one generates cleanly. Everyone else is in a queue, not being measured. vLLM's ITL rises because it deliberately batches many requests at once to maximise total output, and a bigger batch means slightly slower per-token generation inside it.
The cost of Ollama's approach appears in a different column: P99 time-to-first-token of 673ms against vLLM's 80ms, at peak. Users are not getting smooth generation. They are waiting to start.
Tuning Ollama does not close the gap
The obvious response is to raise the parallelism limit, and Red Hat tested exactly that. They set OLLAMA_NUM_PARALLEL=32, the highest value that stayed stable on an A100.
Two things happened, and the second is the important one.
Throughput still plateaued. Ollama's curve lifted but kept its shape, never approaching vLLM's near-linear scaling at any concurrency level.
Inter-token latency became erratic, with large spikes at higher concurrency. That is the signature of head-of-line blocking, where one stalled request holds up an entire batch. So the metric Ollama won at default settings is the metric that breaks when you tune it for the workload where winning would matter.
That is the honest end of the tuning argument. Ollama is not a throttled vLLM waiting to be unlocked. The architecture is different: vLLM's PagedAttention allocates KV cache in small pages on demand, and continuous batching pulls new requests in the moment compute frees up. Neither exists in Ollama, which is built on llama.cpp for a different job.
Where llama.cpp sits, because this is really a three-way
The comparison is usually framed as two engines. It is closer to two, one of which is a wrapper.
Ollama is built on llama.cpp. It adds model management, a REST API, an OpenAI-compatible mode and the one-command install. It does not replace the inference core. So the frequent claim that llama.cpp is faster than Ollama is not describing a different engine, it is describing wrapper overhead and the configuration surface Ollama hides. On the axis that matters here, concurrency, both sit on the same side of the line, because both inherit llama.cpp's execution model rather than vLLM's.
That leaves a genuine three-way, with one axis the vendor benchmarks do not mention: memory.
| llama.cpp | Ollama | vLLM | |
|---|---|---|---|
| Role | inference engine | llama.cpp plus a server | separate engine |
| Weights | GGUF | GGUF | HuggingFace, FP16/AWQ/GPTQ/NVFP4 |
| Concurrency | largely sequential | 4 parallel by default | continuous batching |
| KV cache quantization | most aggressive, down to 4-bit | inherits llama.cpp | 8-bit, no 4-bit |
| VRAM, same quantized model | lowest | low | highest |
| Configurability | maximum, and manual | deliberately limited | high, and required |
Practitioners in the r/LocalLLaMA thread raise the point both benchmarks skip: the llama.cpp family can run a given quantized model in less VRAM than vLLM, because it exposes more aggressive KV cache quantization, and vLLM's CUDA graphs claim memory of their own. If you are fitting a large model onto one modest card, that can settle the question before throughput becomes relevant at all.
The honest three-way, then. llama.cpp for maximum control and minimum VRAM, if you will accept manual work. Ollama for that same engine with the sharp edges removed. vLLM when concurrency is the constraint and you have the memory to pay for it.
Where the gap actually bites: agent loops
For a human reading a streamed reply, 25ms versus 500ms to first token is barely perceptible. This comparison sounds academic right up to the point you put an agent in the loop.
An agent making 20 sequential model calls pays the latency 20 times:
| Stack | Per step, 256 tokens | 20-step agent, inference only |
|---|---|---|
| Ollama, single user | ~2 to 4s | ~60 to 80s |
| vLLM BF16 | ~0.13s | ~3 to 5s |
| vLLM NVFP4 | ~0.05s | ~1 to 3s |
The difference between a minute and four seconds per agent run changes what you can build. This is the same compounding problem that makes agent memory and context engineering worth doing properly: anything paid per step gets multiplied by the loop.
One documented case in the Exxact write-up: an internal knowledge assistant on Ollama went from 3 second P95 latency at 3 users to over a minute at 40. Moving to vLLM on the same hardware brought it back under 2 seconds.
When Ollama is the right answer
The framing that treats this as a winner and a loser is wrong, and there is one deployment pattern that almost never gets mentioned.
Single developer on a workstation. Ollama installs in a minute, needs no containers, and the model management is genuinely better. For local prototyping the throughput conversation is irrelevant, because concurrency is one.
A small team with a GPU each. Four independent Ollama instances across four GPUs deliver roughly 148 t/s per user with zero contention, because nobody shares a queue. If your team is four researchers who each want a dedicated lane, that beats standing up shared serving infrastructure you then have to operate.
Anything where GGUF and simplicity matter more than throughput. Ollama runs GGUF via llama.cpp; vLLM serves HuggingFace-format weights and wants more configuration.
The crossover is not gradual. It arrives the moment more than about five people or agents share one GPU. For the local-desktop end of this decision, LM Studio vs Ollama is the more relevant comparison, and self-hosting vs API covers whether to run anything yourself at all.
Which to pick
Ollama if concurrency is one, or if each user has their own GPU, or if you are prototyping and want to be running in five minutes.
vLLM if more than a handful of users or agents share hardware, if you need predictable latency under load, or if you want NVFP4 and other quantization Ollama does not support.
Both, at different stages, which is what most teams actually do. The migration is an endpoint change and a model format conversion, not a rewrite. Frameworks like LangChain, LangGraph, CrewAI and AutoGen talk to both through the same OpenAI-compatible API, so agent code moves unchanged.
Related guides
- SGLang vs vLLM: the next decision up, once you have chosen production serving over a local runtime
Frequently asked questions
Is vLLM faster than Ollama?
For aggregate throughput under concurrent load, substantially. Red Hat measured 793 tokens per second for vLLM against 41 for Ollama on an A100, and a separate benchmark on an RTX Pro 4500 measured 2,031 against 134. For a single user on a local machine the difference is far smaller and Ollama is easier to run.
Why does Ollama have better inter-token latency than vLLM?
Because it limits itself to four parallel requests by default, so its active batch stays small and each running request generates smoothly. The users it is not serving are queued and are not in that measurement. The cost appears in time-to-first-token, which Red Hat measured at 673ms P99 against vLLM's 80ms.
Can I tune Ollama to match vLLM?
No. At OLLAMA_NUM_PARALLEL=32, the highest stable setting Red Hat found on an A100, throughput still plateaued well below vLLM and inter-token latency became erratic with spikes indicating head-of-line blocking. The difference is architectural, not configuration.
How many users can Ollama handle?
Roughly four concurrent requests at default settings, which is what it was designed for. Practitioner reports put noticeable degradation from about five concurrent connections upward. Beyond that, requests queue and time-to-first-token climbs.
Does it matter for AI agents?
More than for humans. An agent making 20 sequential calls multiplies per-step latency, so the same workload is roughly 60 to 80 seconds of inference on Ollama against 3 to 5 seconds on vLLM. Human readers barely notice a 500ms first token; a 20-step loop does.
Is llama.cpp faster than Ollama?
Not meaningfully at the inference core, because Ollama is built on llama.cpp. The difference is wrapper overhead and how much configuration Ollama hides. Both inherit the same execution model, so both behave the same way under concurrency, which is where vLLM separates from them.
Which uses the least VRAM?
The llama.cpp family, including Ollama, for the same quantized model. llama.cpp exposes more aggressive KV cache quantization than vLLM, which offers 8-bit but not 4-bit, and vLLM's CUDA graphs consume additional memory. If a model only just fits on your card, that can decide the choice before throughput matters.
Should I switch from Ollama to vLLM?
Only when you are sharing a GPU across more than a few users or agents. The switch is an endpoint change plus a model format conversion from GGUF to HuggingFace format, and agent code using an OpenAI-compatible client needs no changes.
Sources
- Red Hat, Ollama vs vLLM: a deep dive into performance benchmarking: A100-PCIE-40GB, vLLM 0.9.1, Ollama 0.9.2, Llama 3.1 8B, concurrency 1 to 256, 300s per test
- Red Hat, Ollama or vLLM: how to choose the right LLM serving tool: the companion piece setting out the use-case split
- GuideLLM: the benchmarking tool used for the Red Hat measurements
- vLLM documentation: PagedAttention and continuous batching
- Exxact, Ollama vs vLLM: which fits your deployment (May 2026): RTX Pro 4500 32GB figures, BF16 and NVFP4, and the 20-step agent timings
Figures are quoted from the published benchmarks above, on the hardware and versions each specifies. They are not ours and they do not transfer to different rigs. Both publishers sell products in this space, which is noted in the article.
