How to Fix Docker 'No Space Left on Device': Prune, Logs, data-root (2026)
This guide is for anyone whose docker build, docker pull or running container has just failed with "no space left on device" on a Linux server, a CI runner or Docker Desktop. By the end you will know which filesystem is actually full, what every prune command deletes before you run it, how to cap build cache and container logs so it does not happen again, and how to move Docker's storage to a bigger disk, including the /var/lib/containerd path that catches people on Docker Engine 29. Commands and config keys are checked against the Docker Engine 29.9 documentation (29.9.2 is current, released 20 October 2026), Compose v5.6.1 and Buildx 0.38.1. You need shell access with sudo on the Docker host and GNU coreutils.
The error is the kernel's ENOSPC, and it has three different causes that look identical in the log: the filesystem ran out of blocks, it ran out of inodes, or a kernel limit that happens to report ENOSPC (inotify watches, a tmpfs such as /dev/shm) was hit. The fix depends on which one, so check before you delete anything.
Step 1: Find which filesystem is actually full
Docker writes to more places than people expect. Ask the daemon where its data lives and which image store it uses, then check blocks and inodes on each path.
# Where this daemon keeps its data (default /var/lib/docker)
docker info --format '{{ .DockerRootDir }}'
# Which image store: [[driver-type io.containerd.snapshotter.v1]] means the containerd store
docker info --format '{{ .DriverStatus }}'
# Blocks and inodes for every path Docker writes to
df -h /var/lib/docker /var/lib/containerd /tmp
df -i /var/lib/docker /var/lib/containerd
The second command matters more in 2026. The containerd image store is the default for fresh installs of Docker Engine 29.0 and later, and with it, image content and container snapshots live in /var/lib/containerd, not /var/lib/docker. Per the daemon docs, only volumes and other daemon data stay in /var/lib/docker. If /var/lib/docker sits on a big separate disk and the host was rebuilt on Engine 29, images are now quietly filling the root filesystem. The containerd store also keeps both compressed and unpacked copies of each layer, so the same images use more disk than they did on overlay2.
If IUse% is 100% while Use% looks fine, skip to Step 8. If neither is full, the ENOSPC is coming from a kernel limit, also in Step 8.
Step 2: See what Docker is using with docker system df
docker system df summarises images, containers, local volumes and build cache, with a RECLAIMABLE column that estimates what pruning would free. The verbose form lists every image, container, volume and cache record.
docker system df
docker system df -v | less
# Cross-check against the filesystem; -x stays on one filesystem, skipping overlay mounts
sudo du -xh --max-depth=1 /var/lib/docker | sort -h | tail -n 8
sudo du -xh --max-depth=1 /var/lib/containerd | sort -h | tail -n 8
The CLI reference shows the summary in this shape (the reference's own example; current versions add a Build Cache row):
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 5 2 16.43 MB 11.63 MB (70%)
Containers 2 0 212 B 212 B (100%)
Local Volumes 2 1 36 B 0 B (0%)
If du says /var/lib/docker/containers is large but docker system df shows small containers, the space is in log files, which the containers row does not include. Go to Step 5. On the containerd store, upgrade to Engine 29.6 or later before trusting the numbers: 29.6.0 fixed docker system df image size reporting and made docker system prune count unpacked image data in its reclaimed total.
Step 3: Prune safely, and know what each command deletes
Docker does not garbage-collect images, containers, volumes or networks on its own. Every prune command asks for confirmation unless you pass --force, and each one has a precise scope:
| Command | Deletes | Does not delete |
|---|---|---|
docker container prune |
All stopped containers and their writable layers | Running containers |
docker image prune |
Dangling images (untagged, unreferenced) | Tagged images |
docker image prune -a |
Every image not used by at least one container | Images used by any container, running or stopped |
docker builder prune |
Dangling build cache; -a for all unused cache |
Cache in use by a running build |
docker volume prune |
Unused anonymous volumes; -a adds unused named volumes |
Volumes referenced by any container |
docker system prune |
Stopped containers, unused networks, dangling images, unused build cache | Volumes, unless --volumes (anonymous only); tagged images, unless -a |
That table comes straight from the system prune, image prune and volume prune references. Use the until filter so you keep anything recent:
# Stopped containers older than a day
docker container prune --force --filter "until=24h"
# Images not used by any container and created more than 7 days ago
docker image prune --all --force --filter "until=168h"
# The usual one-liner, still keeping volumes and recent objects
docker system prune --all --force --filter "until=168h"
-a removes base images too, so the next build pulls them again. Unauthenticated Docker Hub pulls are limited to 100 per 6 hours per IPv4 address or IPv6 /64, which a CI fleet behind one NAT address can burn through quickly. Log in, or use a pull-through mirror, before you make -a a habit.
Step 4: Clear and cap the BuildKit build cache
On build hosts the cache is usually the biggest bucket. Inspect it, prune it by age, or prune it down to a size:
docker buildx du --verbose | tail -n 20 # per-record sizes, last access
docker builder prune --force --filter "until=72h" # unused cache older than 3 days
docker buildx prune --force --max-used-space 20gb # trim least recently used down to 20 GB
BuildKit also garbage-collects on a schedule. For the default docker driver, its limits live in /etc/docker/daemon.json. In current Engine releases the keys are defaultReservedSpace, defaultMaxUsedSpace and defaultMinFreeSpace; the older defaultKeepStorage key is still accepted as a deprecated alias for defaultReservedSpace.
{
"builder": {
"gc": {
"enabled": true,
"defaultReservedSpace": "5GB",
"defaultMaxUsedSpace": "30GB",
"defaultMinFreeSpace": "15GB"
}
}
}
The garbage collection docs explain the default policy order: regenerable cache (local contexts, Git checkouts, cache mounts) unused for 48 hours goes first, then anything unused for 60 days, then cache over the size limit. Engine 29.6.0 also fixed the default policy so it prunes reproducible cache types as intended, which is another reason not to sit on an old 29.x build. Builders created with docker buildx create (the docker-container driver) ignore daemon.json and read their own buildkitd.toml.
Step 5: Find runaway container logs and turn on rotation
This is the classic cause on servers. Docker Engine on Linux defaults to the json-file driver and does no rotation, so one chatty container can fill a disk on its own.
# Biggest json-file logs, with the container they belong to
sudo find /var/lib/docker/containers -name '*-json.log' -size +100M -exec ls -lh {} \;
docker inspect --format '{{.Name}} {{.HostConfig.LogConfig.Type}} {{.LogPath}}' $(docker ps -aq)
# Emergency only: empty one container's log in place
sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' web)"
Truncate, never rm. The daemon holds the file open, so deleting it frees nothing until the container restarts; sudo lsof -nP +L1 lists files that are deleted but still open, which is how you find that mistake later. The json-file docs also warn that touching these files with outside tools can interfere with Docker, so treat truncation as a stopgap. The fix is rotation in /etc/docker/daemon.json, merged with the builder settings from Step 4:
{
"live-restore": true,
"log-driver": "local",
"log-opts": {
"max-size": "20m",
"max-file": "5"
},
"builder": {
"gc": {
"enabled": true,
"defaultReservedSpace": "5GB",
"defaultMaxUsedSpace": "30GB",
"defaultMinFreeSpace": "15GB"
}
}
}
The local driver rotates and compresses by default and keeps 100 MB per container (five 20 MB files); docker logs still works. If a log shipper reads *-json.log files from disk, stay on json-file and set max-size and max-file instead. Either way, values in log-opts must be strings. Apply it carefully:
sudo dockerd --validate --config-file /etc/docker/daemon.json # expect: configuration OK (non-zero exit if invalid)
sudo systemctl reload docker # SIGHUP: live-restore is reloadable, so containers survive the next step
sudo systemctl restart docker # log-driver and builder.gc need a restart
docker info --format '{{.LoggingDriver}}'
docker compose up -d --force-recreate # log settings only apply to newly created containers
Live restore keeps containers running while the daemon restarts, and it is on the list of options that reload without a restart, which is why it goes first. For Compose stacks you can also set rotation per service with a logging: block; the sibling post, Docker Compose in Production (2026), has a full example.
Step 6: Check volumes and container writable layers
Volumes hold real data, so look before you prune. Anonymous volumes are the usual leak: every container started from an image that declares VOLUME gets a new one, and it outlives the container unless the container was started with --rm.
# Writable layer size per container; a big number means the app writes inside the container
docker ps --all --size --format 'table {{.Names}}\t{{.Size}}\t{{.Status}}'
docker container diff web | head -n 30 # which files it changed
# Volumes no container references
docker volume ls --filter dangling=true
docker volume prune --force # unused anonymous volumes only
docker volume prune --all # also unused named volumes: read the prompt first
Since API 1.42 (Engine 23), docker volume prune only touches anonymous volumes unless you add --all, which is why it often "frees nothing" on hosts where stacks use named volumes. A large writable layer usually means an app is writing caches or temp files to its own filesystem; mount a named volume or a size-limited tmpfs there instead.
Step 7: Move Docker's data-root to a bigger disk
If the host is simply too small, move the storage. Stop every container and then the daemon (with live-restore enabled, stopping the daemon alone leaves containers running and writing), copy with hard links and extended attributes preserved (overlay uses xattrs for whiteouts), then point the daemon at the new path.
# Classic overlay2 store: everything is under /var/lib/docker
docker stop $(docker ps -q) # with live-restore on, stopping the daemon leaves containers running
sudo systemctl stop docker.socket docker.service
sudo mkdir -p /mnt/docker-data
sudo rsync -aHAXS --numeric-ids /var/lib/docker/ /mnt/docker-data/
# add "data-root": "/mnt/docker-data" to /etc/docker/daemon.json
sudo dockerd --validate --config-file /etc/docker/daemon.json
sudo systemctl start docker
docker info --format '{{ .DockerRootDir }}' # expect /mnt/docker-data
On the containerd image store, data-root does not move images. Move containerd's root as well, as the Docker docs describe:
sudo systemctl stop docker.socket docker.service containerd.service
sudo rsync -aHAXS --numeric-ids /var/lib/containerd/ /mnt/containerd-data/
sudoedit /etc/containerd/config.toml
# /etc/containerd/config.toml (keep any existing lines such as disabled_plugins)
version = 2
root = "/mnt/containerd-data"
sudo systemctl start containerd.service docker.service
docker image ls # your images should be listed; then remove the old copies
Mounting the new disk directly at /var/lib/docker (and /var/lib/containerd) via /etc/fstab works too and needs no config change. Two warnings from the docs: do not share one data directory between daemons, for example on NFS, and do not edit anything under /var/lib/docker by hand. On XFS, the overlay driver needs ftype=1, so format any new disk with -n ftype=1.
Step 8: Fix inode exhaustion and the ENOSPC errors that are not about disk
Images full of small files (think node_modules layers) and build cache can exhaust inodes while plenty of space remains.
df -i /var/lib/docker
sudo du --inodes -x -d 2 /var/lib/docker | sort -n | tail -n 10
# Kernel limits that also return ENOSPC
cat /proc/sys/fs/inotify/max_user_watches
echo 'fs.inotify.max_user_watches = 524288' | sudo tee /etc/sysctl.d/60-inotify.conf
sudo sysctl --system
# /dev/shm inside a container (64 MiB unless set)
docker exec db df -h /dev/shm
Pruning frees inodes as well as blocks. If that is not enough, the filesystem is the problem: on ext4 the inode ratio is fixed when the filesystem is created, while XFS allocates inodes dynamically up to maxpct (25% of space on filesystems under 1 TB, per mkfs.xfs). Moving the data root to an XFS volume (Step 7) is the durable fix.
The other two are not disk at all. inotify_add_watch returns ENOSPC when the per-user watch limit is reached, which file watchers (dev servers, docker compose watch) can hit on large source trees. And /dev/shm defaults to 64 MiB per container; Postgres parallel queries and headless Chrome outgrow it. Raise it with --shm-size or shm_size: "1gb" in Compose.
Docker Desktop and CI runners
On macOS and Windows, Docker Desktop keeps everything in one virtual disk. Its size cap is Settings > Resources > Advanced > Disk usage limit, next to Disk image location (settings docs). On a Mac that file is Docker.raw; the Mac FAQ says to move it through Settings, never in Finder, and notes that tools often report its maximum size rather than its real one. Desktop's build cache GC defaults to a 20 GB defaultKeepStorage, editable under Settings > Docker Engine. Since Desktop 4.94 (5 October 2026), new Linux containers use the local log driver by default.
GitHub-hosted Linux runners are documented with 14 GB of SSD storage, which a couple of large image builds can fill. Build smaller images (the multi-stage guide covers it), prune between steps, or move to a larger runner; the caching trade-offs are in How to Speed Up GitHub Actions CI. Self-hosted runners and build servers should prune on a schedule:
# /etc/cron.d/docker-prune: Sundays 04:17, keep the last 7 days, never touch volumes
17 4 * * 0 root /usr/bin/docker system prune --all --force --filter "until=168h" >> /var/log/docker-prune.log 2>&1
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Root filesystem full, but /var/lib/docker is on a big disk |
Engine 29 containerd image store keeps images in /var/lib/containerd |
Move containerd's root in /etc/containerd/config.toml (Step 7) |
Disk full, docker system df shows little reclaimable |
Container log files, not counted in the containers row | Find *-json.log files, truncate, set log-opts, recreate (Step 5) |
| Deleted a log file, space did not come back | The daemon still holds it open | lsof -nP +L1, restart that container; truncate next time |
df -h shows free space but writes fail |
Inodes exhausted | df -i, prune, move to XFS (Step 8) |
| New log settings had no effect | Daemon log options only apply to new containers | docker compose up -d --force-recreate |
docker volume prune freed nothing |
Since API 1.42 it only removes anonymous volumes | Check docker volume ls --filter dangling=true, then --all |
| Disk use jumped after enabling the containerd store | Old overlay2 data is hidden but still on disk | Switch back, export what you need, then delete the old data |
| Build fails with ENOSPC on a host with free space | A docker-container builder or the Desktop VM has its own disk |
docker buildx prune --builder <name>, or raise the Desktop disk limit |
| Postgres or Chrome reports no space left with free disk | /dev/shm at the 64 MiB default |
Set shm_size or --shm-size |
| Watchers fail with ENOSPC | fs.inotify.max_user_watches reached |
Raise it in /etc/sysctl.d/ (Step 8) |
FAQ
Is it safe to run docker system prune -a?
It is safe for data in volumes, which it never removes unless you add --volumes, and even then only anonymous ones. It does remove every image without a container, including base images, plus all unused build cache, so the next builds are slower and pull more. Add --filter "until=168h" to keep the last week.
Does docker system prune delete volumes?
Not by default. With --volumes it removes unused anonymous volumes. Named volumes are only removed by docker volume prune --all or by deleting them explicitly.
How do I limit Docker container log size?
Set "log-driver": "local" (100 MB per container by default) or keep json-file with "max-size" and "max-file" in /etc/docker/daemon.json, restart the daemon, and recreate containers. Compose services can set the same options in a logging: block.
Why is /var/lib/docker/overlay2 so big, and can I delete it?
It holds image layers and container writable layers on the classic store. Never delete files in it by hand; Docker's docs say not to manipulate anything under /var/lib/docker. Use the prune commands, or move the whole data root to a bigger disk.
When a Docker host fills up, I run df -h and df -i side by side before anything else, because deleting images to fix an inode or containerd-path problem just wastes a rebuild. Then docker system df -v, then log files. On every host I own, the daemon gets the local log driver, explicit build cache limits and a weekly prune with an until filter on day one, because "no space left on device" is almost always a missing default, not a mystery. For the wider picture of what changed in Docker this year, see Docker in 2026. Official references: docker system prune, Prune unused Docker objects, Configure logging drivers, Build garbage collection, containerd image store and Daemon data directory.
Related: How to Reduce Docker Image Size: Multi-Stage Builds Guide (2026), How to Troubleshoot Linux Performance in 60 Seconds (2026 Checklist) and How to Speed Up GitHub Actions CI: Caching, Sharding, Docker (2026).
I'm Ajay Kumar — I build and operate PandaStack, an open-source Firecracker microVM cloud for AI agents, and write about the infrastructure and tooling around it.
Need this kind of infrastructure work? See what I do or email hello@ajayk.sh.
Related
How to Troubleshoot DNS with dig: NXDOMAIN, SERVFAIL, DNSSEC (2026)
Step-by-step DNS troubleshooting with dig and nslookup: compare your OS resolver, read dig output, tell NXDOMAIN from NODATA and SERVFAIL, query authoritative servers, use +trace, debug DNSSEC with +cd, EDE codes and a validator, negative TTLs and CNAME-at-apex errors.
17 minOct 31, 2026How to Upgrade an EKS Cluster to 1.37 Safely: Step by Step (2026)
Upgrade Amazon EKS one minor version at a time without an outage: refresh upgrade insights, scan for removed APIs with Pluto, upgrade the control plane with eksctl or the AWS CLI, then VPC CNI, CoreDNS and kube-proxy, then node groups and Karpenter, with the 7-day rollback.
15 minOct 31, 2026How to Run Zero-Downtime Postgres Migrations: Step by Step (2026)
Run PostgreSQL schema migrations without downtime: lock_timeout and retries, CREATE INDEX CONCURRENTLY, NOT NULL via CHECK NOT VALID or the PG 18 NOT NULL NOT VALID, batched backfills, an expand/contract column rename, pgroll 0.16, Squawk in CI and pg_locks queries.
17 min