Self-Hosted LLMs in 2026: Ollama vs vLLM vs llama.cpp, and When to Use Each

Two years ago, self-hosting a useful language model meant a research cluster. Today a capable open-weight model runs on a laptop, and a single GPU server can serve a team. The hard question is no longer can we? but which runtime, on which hardware, for which job, and is it actually cheaper than an API?

Why teams self-host

• Data control: prompts and documents never leave your network.
• Predictable cost at steady, high volume.
• Offline or air-gapped environments.
• Customisation: fine-tuned or domain models you own.

And the honest counterpoint: the strongest frontier models are typically only available through APIs, GPUs sitting idle still cost money, and you now own uptime, upgrades and security patches. Many teams end up hybrid: local models for private or high-volume simple tasks, APIs for the hardest reasoning.

The three runtimes at a glance

• Ollama: the easiest way to run models locally. One command downloads and serves a model, with a simple model library and an OpenAI-compatible endpoint. Best for laptops, prototyping, internal tools and low concurrency.
• vLLM: a high-throughput inference server built for GPUs. Its PagedAttention memory management and continuous batching let it serve many concurrent users efficiently. Best for production APIs on NVIDIA (and increasingly other) accelerators.
• llama.cpp: a lightweight C/C++ engine that runs quantized GGUF models on CPUs, Apple Silicon and modest GPUs. It is the engine behind many desktop tools, and includes its own OpenAI-compatible server. Best for edge devices and squeezing models onto limited hardware.

Getting started with each

Ollama

ollama pull llama3.1:8b
ollama run llama3.1:8b "Summarise this ticket in one line: ..."
# OpenAI-compatible API on http://localhost:11434/v1

vLLM

pip install vllm
vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 16384
# OpenAI-compatible API on http://localhost:8000/v1

llama.cpp

llama-server -m ./models/llama-3.1-8b-instruct-Q4_K_M.gguf -c 8192
# OpenAI-compatible API on http://localhost:8080/v1

Throughput vs simplicity

For a single user, the runtimes feel similar. The difference appears under load. vLLM is designed to batch many simultaneous requests on a GPU, so total tokens per second across users grows well as concurrency rises. Ollama and llama.cpp prioritise easy local use and small footprints; they handle a few parallel requests but are not usually the first choice for serving hundreds of users. Benchmark with your prompt lengths and concurrency, because published numbers vary widely by model, hardware and settings.

Sizing hardware: the rule of thumb

Model weights need roughly parameters × bytes per parameter of memory: about 2 bytes at 16-bit, 1 byte at 8-bit and about half a byte at 4-bit quantization. So an 8B-parameter model needs around 16 GB at 16-bit but around 5 GB at 4-bit, plus extra for the KV cache, which grows with context length and concurrent users. Our GPU guide goes deeper on matching models to cards.

• Laptop / Apple Silicon with 16 to 32 GB: small and mid models (roughly 3B to 14B) quantized, via Ollama or llama.cpp.
• Single 24 GB GPU: mid-size models quantized, or small models at full precision, with vLLM for multi-user serving.
• 80 GB-class GPUs or multi-GPU: large models and high concurrency; vLLM supports tensor parallelism across GPUs.

Is self-hosting actually cheaper?

Compare total cost per million tokens, not GPU price alone. A rented GPU costs the same whether it serves one request or ten thousand, so self-hosting tends to win only when utilisation is consistently high, or when data-control requirements rule out APIs anyway. At low or spiky volume, pay-per-token APIs are usually cheaper and far less work. Include engineering time, monitoring and on-call in the calculation.

Security checklist for self-hosted models

• Never expose the inference port directly to the internet; put it behind authentication and TLS.
• Download weights only from trusted sources and pin versions.
• Check each model's licence for commercial-use terms.
• Log usage and set per-user limits just as you would for a paid API. See our governance guide.

How to choose

• Prototyping on your machine: Ollama.
• Serving a team or product on GPUs: vLLM.
• Edge, CPU-only or tight memory: llama.cpp.
• Hardest reasoning tasks: keep a frontier API in the mix and route by task.

Learn first, then bring your own models

agentswarms.fyi lets you compare current hosted models in agents and swarms without any setup, which is a quick way to learn what quality level each task really needs. The self-hosted AgentSwarms platform runs on your own infrastructure and can connect to models wherever they run: your own Ollama or vLLM servers, plus most popular hosted providers, including OpenRouter and Groq. That makes the hybrid pattern above practical: point private or high-volume agents at a local model, and route the hardest reasoning tasks to a frontier API, all under the same permissions, budgets and audit trail.