Running LLMs Locally in 2026: Ollama vs llama.cpp vs LM Studio vs MLX

I run local models every week, mostly to test agent code against something that costs nothing per token and never leaves my desk. Running LLMs locally in 2026 is no longer a hobbyist exercise: Ollama now hands Apple Silicon models to MLX by default, llama.cpp ships semantic versions and a one-line installer, and the open-weight models that fit in 32GB are good enough to drive a coding agent for real work. What has not changed is the physics. This post explains the five runtimes people actually choose between (Ollama, llama.cpp, LM Studio, MLX and vLLM), which models fit in 8, 16, 32 and 64GB, what quantization formats like Q4_K_M really trade away, how to predict tokens per second before you download anything, and the licence and privacy questions that come up the moment a team wants to use this at work.

Ollama vs llama.cpp vs LM Studio vs MLX vs vLLM: which runtime?

All five run the same kinds of open-weight models. They differ in who they are for, and in how much they hide.

llama.cpp is the engine underneath most of the ecosystem. It reads GGUF files, runs on CPUs, CUDA, ROCm, Vulkan and Metal, and ships llama-quantize, the tool that produces most of the quantized files you download. The October release, v0.7.0, added a GPU cache for MoE experts kept in host memory (up to about 2.2x faster decode when experts are offloaded, per the notes) and changed the server's default port from 8080 to 9931, which will break a few scripts. The README now leads with a unified llama binary: llama cli -hf <repo> to chat and llama serve -hf <repo> for an OpenAI-compatible server.

Ollama wraps an engine in a model registry, a background service on port 11434, a CLI and an API that speaks Ollama's own dialect plus OpenAI- and Anthropic-compatible endpoints. The big 2026 change is v0.40.0: on Apple Silicon, architectures the MLX runtime supports now run on MLX automatically. Ollama moves fast (six point releases, v0.40.1 to v0.40.6, landed between 7 and 16 October), so pin a version on anything shared.

LM Studio is the desktop GUI. LM Studio 0.4.0 (January 2026) split the core out as a headless server called llmster, added parallel requests and a stateful REST API, and 0.4.1 added an Anthropic-compatible /v1/messages endpoint. It is closed source but has been free for use at work since July 2025.

MLX is Apple's array framework, and mlx-lm (v0.32.2) is the Python package for running and fine-tuning models on it. You can use it directly or get it implicitly through Ollama or LM Studio.

vLLM (v0.32.0) is a different class of tool: a throughput-oriented server for many concurrent users on datacentre GPUs. If "local" means a shared box in a rack serving a team, it belongs here. If it means your laptop, it does not. I covered why its batching and KV-cache design matter in LLM inference serving in 2026.

Runtime	Latest version (checked 20 Oct 2026)	Best for	Model format	Licence
Ollama	v0.40.6	Developers who want a CLI, an API and a registry	GGUF, Safetensors, MLX	MIT
llama.cpp	v0.7.0	Maximum control, odd hardware, CPU inference	GGUF	MIT
LM Studio	0.4.x	GUI users, model browsing, quick comparisons	GGUF, MLX	Proprietary, free for work
mlx-lm	v0.32.2	Apple Silicon, Python, local fine-tuning	MLX (Safetensors)	MIT
vLLM	v0.32.0	Many users on NVIDIA/AMD GPUs	Safetensors and others	Apache-2.0

My default advice: start with Ollama, drop to llama.cpp when you need a flag Ollama does not expose, use LM Studio if you prefer clicking to typing, and reach for vLLM only when concurrency is the problem.

How much RAM do you need to run an LLM locally?

The weights must fit in memory you can read quickly: VRAM on a discrete GPU, or unified memory on Apple Silicon and AMD's Ryzen AI Max parts. On top of the weights you need room for the KV cache, which grows linearly with context length, and for the operating system and whatever else you have open.

A useful rule: the file size of the quantized model plus 20 to 30 percent for a modest context. Long contexts change that sum quickly. The KV cache is roughly 2 x layers x kv_heads x head_dim x context_tokens x bytes_per_element, so doubling context doubles it. Ollama's context length docs now pick a default from available VRAM (4k below 24 GiB, 32k from 24 to 48 GiB, 256k at 48 GiB and above) and recommend at least 64k tokens for agents and coding tools. That recommendation is the single most common reason a model that "fits" suddenly spills onto the CPU.

Here is what fits, using Ollama's own library sizes for the default 4-bit builds:

Memory	Realistic picks (download size)	Notes
8GB	gemma4:e2b (4.6GB), gemma4:e4b (6.6GB), llama3.1:8b (4.9GB)	Short contexts only. Fine for summarising, classification, autocomplete.
16GB	gemma4:12b (7.7GB), gpt-oss:20b (14GB), deepseek-r1:14b (9.0GB)	gpt-oss:20b leaves little headroom on a 16GB Mac.
32GB	gemma4:26b (16GB, MoE), qwen3.6:27b (18GB), qwen3.8:27b (18GB), gemma4:31b (19GB)	The sweet spot in 2026. Room for a 32k to 64k context.
64GB	qwen3.6:35b (23GB, 35B-A3B MoE), qwen3-coder-next (52GB)	Long contexts on 27B to 35B models, or one much larger coder.
96GB+	gpt-oss:120b (65GB)	M5 Max with 128GB, RTX PRO 6000, or two big GPUs.

Two things the table hides. First, mixture-of-experts models (Gemma 4 26B-A4B, Qwen3.6 35B-A3B, gpt-oss) need memory for all their weights but only read the active experts per token, so they decode far faster than their size suggests. Second, a 16GB Mac does not give the GPU all 16GB; macOS keeps a share for itself, which is why the 14GB gpt-oss build is a squeeze there.

What is Q4_K_M? Quantization explained

A model trained in 16-bit floats stores each weight in two bytes. Quantization stores groups of weights at lower precision with a shared scale, trading a little accuracy for a lot of memory. The GGUF names encode the recipe: Q4 is roughly four bits, K means the k-quant block scheme, and the S/M/L suffix says how many sensitive tensors get extra bits. IQ formats use an importance matrix (calibration data) to decide where precision matters most, which is how 2- and 3-bit files stay usable.

The llama.cpp project publishes measurements for Llama 3.1 8B in its quantize README. They are the cleanest public illustration of the trade-off I know of:

The pattern is the important bit. Going from F16 to Q4_K_M shrinks the file from 14.96 GiB to 4.58 GiB and lifts decode from 29.2 to 71.9 tokens per second on the same hardware. Below four bits the speed curve goes flat while quality keeps falling. That is why almost every registry defaults to Q4_K_M or an equivalent, and why I only go lower when the alternative is not running the model at all. Two newer wrinkles: Google publishes quantization-aware-trained (QAT) Q4_0 builds of Gemma 4 that were trained to survive 4 bits, and gpt-oss ships natively in MXFP4, so for those models the "quantized" version is the original.

How fast will a local LLM run? The bandwidth rule

Decode speed is mostly a memory-bandwidth problem. Generating one token means reading every active weight once, so the ceiling is roughly bandwidth divided by the bytes of active weights. Prompt processing (prefill) is compute-bound and behaves differently, which is why a long prompt can take seconds before the first token even on fast hardware. I went deeper on this ratio in the on-device inference post; here it is applied to desktop hardware.

#!/usr/bin/env bash
# Rough decode ceiling: bandwidth (GB/s) / active weight bytes (GB).
# Real numbers land at roughly 50-80% of this. Prefill is a separate, compute-bound phase.
# Bandwidths: apple.com/macbook-pro/specs and nvidia.com/en-us/geforce/graphics-cards/compare
MODEL_GB=4.9   # llama3.1:8b at Q4_K_M, per ollama.com/library/llama3.1/tags
for hw in "M5:153" "M5_Pro:307" "M5_Max_40c:614" "RTX_5060_Ti:448" "RTX_5080:960" "RTX_5090:1792"; do
  name=${hw%%:*}; bw=${hw##*:}
  printf "%-12s %5s GB/s -> ceiling ~%3.0f tok/s\n" "$name" "$bw" "$(echo "$bw / $MODEL_GB" | bc -l)"
done

Running that gives ceilings of about 31 tokens per second on a base M5 (153 GB/s), 63 on an M5 Pro (307 GB/s), 125 on a 40-core M5 Max (614 GB/s) and 366 on an RTX 5090 (1,792 GB/s). The 5090 is fastest per token but holds 32GB; a 128GB M5 Max is slower per token but can hold models the 5090 cannot. That is the whole Mac-versus-GPU debate in one sentence. For MoE models, plug in active parameters rather than total size and the numbers improve dramatically, which is why 30B-class MoE models feel quick on laptops.

Partial offload is where speed falls off a cliff. When ollama ps shows something like 48%/52% CPU/GPU, the CPU half reads over system RAM at a fraction of GPU bandwidth and becomes the bottleneck. A smaller quant or a shorter context that fits entirely on the GPU is almost always faster than a bigger model split across both.

Can you use local models commercially? Licences in 2026

The licences got simpler this year, at least for the models most people run locally. On Hugging Face, Gemma 4, Qwen3.6, Qwen3.8, gpt-oss and Ministral 3 are all tagged Apache-2.0. Llama remains on Meta's own community licence, with attribution requirements and a separate-licence clause for very large services. Read the model card, not the registry page, because fine-tunes and community quantizations can carry different terms from the base. I wrote about the clauses that have crept into open-weight licences, and the EU AI Act obligations around them, in Open Weights in 2026.

The runtime licence matters less than people fear. Ollama, llama.cpp and mlx-lm are MIT, vLLM is Apache-2.0, and LM Studio's proprietary app is free for commercial use.

Is running an LLM locally actually private?

By default, yes, with three caveats that catch people.

First, the network. Ollama binds to 127.0.0.1:11434 unless you set OLLAMA_HOST, but the Docker examples publish the port on every interface, and people set 0.0.0.0 to reach the box from a laptop. SentinelLABS and Censys found about 175,000 publicly reachable Ollama hosts across 130 countries, roughly half with tool calling enabled, and described a marketplace reselling access to them. A local model server with no authentication is an open API with your GPU bill attached. Bind to localhost, or put it behind a reverse proxy with auth, exactly as you would any internal service.

Second, cloud features. Ollama now offers cloud-hosted models through the same CLI. Its FAQ says local runs are never seen by Ollama, and documents OLLAMA_NO_CLOUD=1 (or disable_ollama_cloud in ~/.ollama/server.json) to switch cloud features off entirely. If you have a compliance reason to be local, set it, so a :cloud tag cannot slip into a config.

Third, the tools around the model. A coding agent pointed at a local model can still fetch URLs, run shell commands and read your files. Local inference removes one data flow; it does not sandbox the agent. (I run PandaStack, a Firecracker microVM platform for agent code execution, so I have an obvious interest here, but the point stands whatever you use.)

FAQ

What is the best local LLM for 16GB of RAM?

For general use, gemma4:12b (7.7GB) leaves room for a useful context window. gpt-oss:20b (14GB) is stronger at reasoning but tight on a 16GB Mac, so keep the context small. On a 16GB NVIDIA card, both fit entirely in VRAM with headroom.

Is Ollama faster than llama.cpp?

Not inherently. In the v0.40.6 source, Ollama launches llama.cpp's own llama-server binary for GGUF models and an MLX runner on Apple Silicon (the RUNNER column in ollama ps shows llamacpp or mlx), so speed is usually within noise once settings match. Differences come from defaults: context length, KV cache type, GPU layer split and batch size. llama.cpp exposes every knob; Ollama picks them for you.

Is MLX faster than llama.cpp on a Mac?

Often somewhat, for models MLX supports well, which is why Ollama 0.40 switched to MLX by default on Apple Silicon. The gap varies by model and quant, so benchmark your own model with both rather than trusting a chart. Bandwidth still sets the ceiling either way.

Should I use vLLM or Ollama?

Use Ollama for one person or a handful of light users. Use vLLM when you need high concurrency on datacentre GPUs, because continuous batching and paged KV cache are what keep throughput up under load. If you are heading that way, the vLLM on Kubernetes guide covers the deployment side.

This quarter I would standardise a team on Ollama with a pinned version, OLLAMA_NO_CLOUD=1 where policy demands it, a 32GB-class model with a 64k context for coding work, and the server bound to localhost or fronted by an authenticated proxy. Then I would write down which models and licences are approved, the same way we already do for container base images. If you want the hands-on version, my step-by-step guide to running a local LLM with Ollama covers install, GPUs, Modelfiles, the API and the errors you will hit, and self-hosting LLMs vs API costs covers whether any of this saves money.

Related: On-Device Inference in 2026, LLM Inference in 2026: Prefill Is a Different Machine Now and Open Weights in 2026.