How to Run a Local LLM With Ollama: Setup, GPU and API Guide (2026)

This guide is for developers who want a language model running on their own machine, reachable over an API, without guessing which settings matter. By the end you will have Ollama installed as a service, a model confirmed to be running on your GPU, a context window sized for real work, a custom model built from a Modelfile, OpenAI-compatible API calls working from Python, and a coding agent pointed at it. Commands and flags were checked against Ollama v0.40.6 (released 16 October 2026) and the docs in the ollama/ollama repository. You need macOS 14 or newer, Windows 10 22H2 or newer, or a recent 64-bit Linux, and ideally a GPU or Apple Silicon with at least 8GB of memory. Ollama was not installed on the Mac I wrote this on, so sample output below comes from the official docs and source rather than from my terminal. If you are still choosing between Ollama, llama.cpp, LM Studio and MLX, read the companion explainer, Running LLMs Locally in 2026, first.

Step 1: Install Ollama

macOS. Download Ollama.dmg from ollama.com/download and drag it to Applications. On first launch it offers to link the ollama CLI into /usr/local/bin. The macOS docs list Apple M-series for GPU support; Intel Macs run on CPU only.

Linux. The install script sets up a systemd service and an ollama user:

# Official installer: installs to /usr, creates the ollama user and systemd unit
curl -fsSL https://ollama.com/install.sh | sh

# Pin a version instead of taking latest (useful for shared boxes and CI)
curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.40.6 sh

If you have an AMD GPU and install manually, you also need the ROCm package (ollama-linux-amd64-rocm.tar.zst), per the Linux docs.

Windows. Run OllamaSetup.exe from the download page. It installs into your home directory without admin rights and runs in the background, per the Windows docs.

Docker (Linux or WSL2 with NVIDIA). The Docker docs use -p 11434:11434, which publishes the API on every interface. Bind it to localhost unless you mean to share it:

# Requires nvidia-container-toolkit on the host. Pin the image tag.
docker run -d --gpus=all \
  -v ollama:/root/.ollama \
  -p 127.0.0.1:11434:11434 \
  --name ollama ollama/ollama:0.40.6

# AMD on Linux uses the rocm image and the kfd/dri devices
# docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama \
#   -p 127.0.0.1:11434:11434 --name ollama ollama/ollama:0.40.6-rocm

Docker Desktop on macOS has no GPU passthrough, so on a Mac use the native app.

Verify the install:

ollama -v
# ollama version is 0.40.6
curl -s http://127.0.0.1:11434/api/version
# {"version":"0.40.6"}

Step 2: Pull a model that fits your memory

Pick by memory, not by leaderboard. These download sizes come from the Ollama library for the default 4-bit builds:

Your GPU memory or unified memory	Start with	Download size
8GB	gemma4:e4b	6.6GB
16GB	gemma4:12b	7.7GB
24 to 32GB	qwen3.6:27b or gemma4:26b	18GB / 16GB
48GB+	qwen3.6:35b	23GB

ollama pull gemma4:12b
ollama run gemma4:12b "Explain a Kubernetes liveness probe in two sentences."

# Housekeeping
ollama ls                 # what is on disk
ollama show gemma4:12b    # architecture, parameters, context length, capabilities
ollama rm gemma4:12b      # free the disk space

Models live in ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models for the Linux service and C:\Users\<you>\.ollama\models on Windows. Set OLLAMA_MODELS to move them, and on Linux chown -R ollama:ollama the new directory, as the FAQ notes.

Step 3: Confirm the model is on the GPU

This is the step most guides skip and the one that explains most "Ollama is slow" complaints. With a model loaded, run:

ollama ps

NAME          ID              SIZE      PROCESSOR    CONTEXT    RUNNER     UNTIL
gemma4:12b    312246b09fab    9.1 GB    100% GPU     4096       llamacpp   4 minutes from now

Read two columns. PROCESSOR should say 100% GPU. If it says 100% CPU, no GPU was used; if it shows a split such as 48%/52% CPU/GPU, the model plus its context did not fit and part of it is running from system RAM. CONTEXT is the window actually allocated, which may be smaller than you asked for (more on that in Step 4). In v0.40 there is also a RUNNER column showing llamacpp or mlx; since v0.40.0, supported models on Apple Silicon run on MLX automatically. (Size in the sample above is illustrative; yours depends on context.)

Per-platform checks if the GPU is not used:

# NVIDIA: driver 550+ (570+ for compute capability 5.0 to 6.2), per docs/gpu.mdx
nvidia-smi                       # the ollama process should appear while generating
journalctl -u ollama --no-pager | grep -i -E "cuda|gpu|vram" | tail -20

# AMD on Linux: ROCm v7 driver; the service user needs the render/video groups
rocminfo | grep -i gfx
sudo usermod -aG render,video ollama && sudo systemctl restart ollama

# Apple Silicon: Metal and MLX are automatic; check the server log
tail -50 ~/.ollama/logs/server.log

The hardware docs list supported cards: NVIDIA compute capability 5.0 and up, AMD via ROCm v7 on Linux and Windows, Apple via Metal, and other GPUs via Vulkan, which is on by default when the backend is installed.

Step 4: Set the context length deliberately

Context length is the biggest lever on memory after model size. Ollama's context length docs say the default depends on VRAM: 4k tokens below 24 GiB, 32k from 24 to 48 GiB, 256k at 48 GiB or more. In the v0.40.6 scheduler source, an automatic default that does not fit is stepped down (256k to 32k to 4k) rather than failing. The docs recommend at least 64,000 tokens for agents, web search and coding tools, which on a 16GB machine means you must choose: a smaller model with a big window, or a bigger model with a small one.

Set it at the level that matches how you use Ollama:

# 1. Whole server, foreground
OLLAMA_CONTEXT_LENGTH=65536 ollama serve

# 2. macOS app: set for launchd, then quit and reopen Ollama
launchctl setenv OLLAMA_CONTEXT_LENGTH 65536

# 3. Linux systemd service
sudo systemctl edit ollama.service
#   [Service]
#   Environment="OLLAMA_CONTEXT_LENGTH=65536"
sudo systemctl daemon-reload && sudo systemctl restart ollama

# 4. One interactive session
ollama run gemma4:12b
# >>> /set parameter num_ctx 65536

# 5. One native API request
curl -s http://127.0.0.1:11434/api/chat -d '{
  "model": "gemma4:12b",
  "messages": [{"role": "user", "content": "hello"}],
  "options": {"num_ctx": 65536},
  "stream": false
}'

The Mac and Windows apps also have a context slider in Settings. Afterwards, check ollama ps again: the CONTEXT column should show your value and PROCESSOR should still say 100% GPU. If it now shows a CPU split, the window is too big for this model on this machine.

If memory is tight, quantize the KV cache. Per the FAQ, OLLAMA_KV_CACHE_TYPE=q8_0 uses about half the memory of the default f16 with a very small quality loss, and q4_0 about a quarter with a more noticeable loss at long contexts. It needs Flash Attention, which Ollama enables automatically where the backend supports it.

Step 5: Build a custom model with a Modelfile

A Modelfile bakes a system prompt and parameters into a named model. It is also the only way to set context length for clients that use the OpenAI-compatible API, because that API has no field for it (the OpenAI compatibility docs say so directly).

# Modelfile: an infra-review assistant with a 64k window
FROM gemma4:12b

# Lower temperature for more deterministic technical answers
PARAMETER temperature 0.2
PARAMETER num_ctx 65536
PARAMETER top_p 0.9

SYSTEM """You are a senior infrastructure engineer. Review Terraform, Kubernetes
and CI configuration. Point out security and reliability problems first,
cite the exact line, and say when you are unsure."""

ollama create infra-reviewer -f ./Modelfile
ollama run infra-reviewer < main.tf
ollama show --modelfile infra-reviewer   # confirm what was baked in

Valid parameters and defaults are in the Modelfile reference. FROM can also point at a local GGUF file (FROM ./model.gguf) or a Safetensors directory if you downloaded weights from Hugging Face.

Step 6: Call the OpenAI-compatible API

Ollama serves its native API under /api and an OpenAI-compatible one under /v1 on port 11434. Most SDKs and tools only need a base URL change. The API key is required by the client library but ignored by Ollama.

curl -s http://127.0.0.1:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "infra-reviewer", "messages": [{"role": "user", "content": "Is hostNetwork: true risky?"}]}'

# pip install "openai>=1.0"
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11434/v1/", api_key="ollama")  # key is ignored

resp = client.chat.completions.create(
    model="infra-reviewer",
    messages=[{"role": "user", "content": "Review: resources: {} on a Deployment"}],
)
print(resp.choices[0].message.content)

/v1/responses and /v1/models also work. For tools hard-coded to an OpenAI model name, ollama cp gemma4:12b gpt-4o-mini creates an alias. Two native-API controls worth knowing: keep_alive (default five minutes; -1 keeps the model loaded, 0 unloads it immediately) and preloading with an empty request, both from the FAQ:

# Preload and keep resident, so the first real request does not pay the load time
curl -s http://127.0.0.1:11434/api/generate -d '{"model": "infra-reviewer", "keep_alive": -1}'

If several services will share one Ollama box, put a gateway in front for keys, budgets and fallbacks; the LiteLLM proxy guide covers that.

Step 7: Use it with coding tools

ollama launch configures and starts supported coding agents against your Ollama server. The CLI docs list OpenCode, Claude Code, Codex, VS Code and Droid.

ollama launch                                # interactive picker
ollama launch claude --model qwen3.6:27b     # Claude Code on a local model
ollama launch droid --config                 # write config without launching

Manual setup for Claude Code, per Ollama's integration page, uses its Anthropic-compatible endpoint:

export ANTHROPIC_BASE_URL=http://localhost:11434
export ANTHROPIC_AUTH_TOKEN=ollama
export ANTHROPIC_API_KEY=""
claude --model qwen3.6:27b

Two practical notes. The model must support tool calling (filter the library by "tools"), and agents need the 64k+ context from Step 4 or they lose track of the repository mid-task. A local model also does not make the agent safe: it can still run shell commands, so keep its permission prompts on, as I argue in It's 2 AM. Do You Know What Your AI Agent Is Doing?.

Step 8: Lock it down

Ollama listens on 127.0.0.1:11434 by default and has no authentication on the local API. SentinelLABS and Censys counted about 175,000 Ollama hosts reachable from the internet, mostly from people setting OLLAMA_HOST=0.0.0.0 or publishing the Docker port. If other machines need access, keep Ollama on localhost and put an authenticating reverse proxy in front.

# Linux service: explicit localhost bind, cloud features off, longer keep-alive
sudo systemctl edit ollama.service
#   [Service]
#   Environment="OLLAMA_HOST=127.0.0.1:11434"
#   Environment="OLLAMA_NO_CLOUD=1"
#   Environment="OLLAMA_KEEP_ALIVE=30m"
sudo systemctl daemon-reload && sudo systemctl restart ollama
journalctl -u ollama --no-pager | grep -i "cloud disabled"
# expect: Ollama cloud disabled: true

OLLAMA_NO_CLOUD=1 (or "disable_ollama_cloud": true in ~/.ollama/server.json) turns off cloud models and web search, per the FAQ, so a :cloud tag cannot quietly send prompts off the machine.

Troubleshooting

Symptom	Likely cause	Fix
ollama ps shows 100% CPU on a GPU machine	GPU discovery failed or driver too old	Check server.log or journalctl -u ollama; NVIDIA driver 550+, AMD ROCm v7 driver; reboot after driver install
ollama ps shows 48%/52% CPU/GPU and generation crawls	Weights plus KV cache exceed VRAM	Lower num_ctx, use a smaller tag, or set OLLAMA_KV_CACHE_TYPE=q8_0
model runner has unexpectedly stopped, this may be due to resource limitations	Runner ran out of memory or crashed on load	Read the server log for the real error; reduce context or model size; update GPU driver
Coding agent forgets files or loops	Context window is the 4k default	Set OLLAMA_CONTEXT_LENGTH=65536 or a Modelfile num_ctx, then confirm in ollama ps
Context setting ignored by an OpenAI SDK client	/v1 API has no context field	Create a Modelfile with PARAMETER num_ctx and call that model name
GPU fell back to CPU after Linux suspend	NVIDIA UVM driver bug	sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm
AMD GPU not found, log says discovery timed out after 30s	Kernel driver older than ROCm 7	Upgrade with amdgpu-install to ROCm v7, reboot
Docker container worked on GPU, later switched to CPU	systemd cgroup driver issue	Add "exec-opts": ["native.cgroupdriver=cgroupfs"] to /etc/docker/daemon.json
First request after idle takes many seconds	Model unloaded after 5-minute keep_alive	Set OLLAMA_KEEP_ALIVE=30m or send "keep_alive": -1 to preload
503 responses under load	Queue full	Raise OLLAMA_MAX_QUEUE or OLLAMA_NUM_PARALLEL if memory allows, or put a gateway in front

Slow generation

Start with ollama ps. If the model is 100% on the GPU, compare your speed with the ceiling: memory bandwidth divided by model size. A 7.7GB model on a 153 GB/s base M5 tops out near 20 tokens per second however you tune it. Mixture-of-experts models read only their active experts per token, so gemma4:26b can decode faster than a dense 12B despite being twice the download. Also check OLLAMA_NUM_PARALLEL: per the FAQ, required memory scales with parallel slots times context length, so raising it can push a model off the GPU.

Out of memory

The fix is almost always "make the working set smaller" in this order: shorter context, quantized KV cache, smaller quantization (q4_K_M instead of q8_0), smaller model. On multi-GPU boxes Ollama splits a model across GPUs only when it will not fit on one; CUDA_VISIBLE_DEVICES or ROCR_VISIBLE_DEVICES restricts which ones it sees. Turn on OLLAMA_DEBUG=1 for more detail on GPU discovery and memory allocation in the server log.

FAQ

Why is Ollama not using my GPU?

Usually an old driver, a missing ROCm package on a manual Linux install, the service user lacking the render group for AMD, or Docker started without --gpus=all. Run ollama ps to confirm, then read the server log, which records what GPU discovery found. Docker Desktop on macOS cannot use the GPU at all.

How do I increase the context length in Ollama?

Set OLLAMA_CONTEXT_LENGTH for the server, /set parameter num_ctx in an interactive session, options.num_ctx in a native API call, or PARAMETER num_ctx in a Modelfile for OpenAI-compatible clients. Then check the CONTEXT column in ollama ps, because an automatic default that does not fit can be stepped down.

Does Ollama send my prompts to the cloud?

Not for local models: the FAQ states Ollama does not see prompts when running locally. Cloud models (tags ending in :cloud) do run remotely. Set OLLAMA_NO_CLOUD=1 to disable them entirely.

Can I use Ollama models with Claude Code or Codex?

Yes. ollama launch claude or ollama launch codex configures the tool for you, or you can point the tool's base URL at http://localhost:11434. Choose a model with tool-calling support and a 64k or larger context.

When a local model misbehaves, the first thing I check is ollama ps, every time, because the PROCESSOR and CONTEXT columns explain most problems before I open a log. On anything shared I pin the Ollama version, bind to localhost behind an authenticating proxy, set OLLAMA_NO_CLOUD=1 where policy requires it, and bake context and system prompts into Modelfiles kept in Git, so the model configuration is reviewed like any other infrastructure. Official references: the Ollama docs, FAQ, hardware support, Modelfile reference and release notes. When one machine is no longer enough, the vLLM on Kubernetes guide is the next step.

Related: On-Device Inference in 2026, LLM Inference in 2026: Prefill Is a Different Machine Now and Running LLMs Locally in 2026.