Checkpoint Restore Implementation: What It Cost Me to Build

A June 2026 analyst report compared nineteen AI-agent sandbox platforms and concluded that checkpoint/restore becomes table stakes by 2027. That report and the press coverage around it are also where the category's headline figures come from, and I pass them on as industry reporting rather than as anything I measured: every hyperscaler entered the category during 2026; Modal is reported to have raised $355M at a $4.65B valuation, and Daytona a $24M Series A; E2B publishes a figure of more than a billion sandboxes started, with pause/resume that preserves memory state; two long-tail projects are described as dormant. The same report's comparison matrix spans creation times from 0.79ms to about 2.7s across Firecracker, gVisor, Kata, Docker, V8 isolates and plain namespaces.

I have no quarrel with the prediction. I do have a note to add, which is that "table stakes" is a sentence you can write in an afternoon and a system you cannot.

I built one. PandaStack is an Apache-2.0 Firecracker microVM cloud I wrote solo over about six months — roughly 400 Go files, 55 Terraform files, and 300+ organizations signed up. Snapshot restore is not a feature there; it is the only create path. Every sandbox that has ever started on it was restored from a checkpoint. So this post is the bill: what a consistent checkpoint actually costs to take, what it costs to store, what lazy restore buys you, and the three failure modes that did not exist in my head and did exist in production.

None of the numbers below come from that report or any other. They come from my own fleet.

Part 1: taking a consistent checkpoint

A checkpoint of a running VM is three problems stacked.

Pause. You cannot dump memory that is being written. Firecracker's snapshot API takes the VM out of the running state first, which means every checkpoint has a stop-the-world window and the guest's clock stops with it. That window is the thing you are actually engineering. Everything else is bookkeeping.

Dump. You write the device and vCPU state (vm.state) and the guest RAM image (vm.mem). The state file is small and structured. The memory file is the size of the guest's configured RAM, and it is where all the cost lives.

Sparsify. A freshly booted 4 GiB guest is mostly zeroes. Storing them is pure waste, and — this matters more — fetching them later is worse than waste, because a fetch is a network round trip in a page-fault handler. So at bake time I walk the memory file at a fixed 4 MiB chunk granularity and record which chunks contain any non-zero byte at all. That map ships as a small sidecar next to the snapshot. On the write side, all-zero regions get punched back to the filesystem with fallocate(PUNCH_HOLE) — the region still reads back as zeros, but the blocks are returned. No reader can tell the difference.

Then there is the resume side of the API, which is worth stating precisely because people get it wrong in blog posts: you PUT /snapshot/load, then PATCH /vm with {"state": "Resumed"}. Load and resume are separate operations. That separation is not pedantry — it is the seam where you inject a memory backend, and Part 3 is entirely about that seam.

Part 2: the storage bill, and why a 4 GiB VM does not cost 4 GiB

The naive model is that N sleeping 4 GiB VMs cost 4N GiB of object storage. Three mechanisms break that model, in descending order of how much they save.

Zero elision. The chunk map from Part 1 means all-zero chunks are neither uploaded nor fetched — they are reconstructed by zero-filling at fault time. How much this saves depends entirely on what the guest was doing when you froze it, which is why I am not going to quote you a universal ratio.

I can quote one measured number, with a caveat attached that I think is the honest part of this post. Across a cumulative window on my fleet, roughly 85% of page faults resolved as zero-fills rather than fetches — about 15.7 million out of 18.5 million. That is a fault-path measurement, not a storage-path one, and when I went back to value a related optimisation against it I discovered the counter was measuring a code path the current fleet had stopped taking. The number was real. The conclusion I wanted to draw from it was not. If you take one methodological thing from this post: before you price a fix using a metric, confirm the code path that metric measures is the live path.

Sparse files. The local cache of a memory image is a sparse file sized to the logical image, with holes where chunks have not been fetched. du and ls -l disagree with each other, and du is the one your capacity planning should believe. Getting eviction accounting right here means counting allocated blocks, not file sizes.

Per-generation shared chunk cache. This is the mechanism I would build first if I were starting over, and it is the piece the streaming-restore internals describe in full. Chunks fetched from object storage land in a persistent, content-addressed, per-host cache keyed by the seed generation's bucket and object path. The first restore of a template on a host pays the network latency once. Every later restore of that same generation — including concurrent ones — is served from local NVMe. Re-baking the template changes the key, so a stale overlay is structurally impossible rather than something you remember to invalidate.

Crash safety here is subtle and worth spelling out. The presence bitmap is only advanced after fdatasync of the data file followed by an atomic rename of the bitmap. So a bit that claims a chunk is valid always implies the data was durable first. A crash between the two steps forgets recent chunks and re-fetches them. It can never serve a torn page. Getting that ordering backwards gives you a cache that silently hands corrupted memory to a guest, which is a failure mode you will find out about from a customer.

Net effect: the per-host warm state lives in a content-addressed cache, not in idle VMs. That is what lets me have no warm pool at all and still restore in 179ms p50.

Part 3: restoring lazily with userfaultfd

If your restore path downloads a multi-GiB memory image before the VM can start, your cold start is a function of your network, and cross-host restore is a cliff. The fix is to not download it.

The mechanism is userfaultfd. Before loading the snapshot, the agent opens a userfaultfd handler on a Unix socket and hands the file descriptor plus the region mappings to Firecracker over SCM_RIGHTS. Firecracker then runs the guest against that backend instead of an mmap'd local file. When the guest touches an unmapped page, the kernel raises a fault, the handler maps the faulting address to an offset in the snapshot, range-GETs the containing 4 MiB chunk from object storage, and installs it with UFFDIO_COPY. Chunks the header marks absent are zero-filled with no fetch at all.

Two additions turn that from "works" into "fast":

• A prefetch trace. At bake time I record the chunk indices that a representative warm-up actually faulted in, in access order. At restore, a background worker replays that list into the cache ahead of the guest. Faults that would have blocked on the network become local cache hits.
• The shared cache from Part 2, sitting between the per-restore resolver and the network.

Two things I want to be clear about, because they are commonly conflated.

userfaultfd streams memory, not disk. The rootfs stays a local file, because copy-on-write via XFS reflink needs a local block device. Streaming removes the multi-GiB memory download from the critical path. It does not remove the disk.

Choosing userfaultfd means choosing 4 KiB pages. Transparent huge pages are allocated by the native anonymous fault handler; pages arriving via UFFDIO_COPY are base pages. THP and userfaultfd do not compose — that is documented upstream, and I confirmed it the expensive way after two rounds of research looking for a way around it. You can get 2 MiB pages by backing the guest with hugetlbfs, but then hugepage-ness becomes a snapshot property: Firecracker will only restore such a snapshot through the userfaultfd backend, so every restore path has to detect the marker and force streaming whether or not you asked for it. I describe the full mechanism in the snapshot-restore internals docs.

Part 4: the three failure modes that only appear in production

One transient object-storage error killed a VM

The original fault handler did a single HTTP request with no retry, and escalated the first resolve error to a fatal state that terminated the handler. When the handler dies, the faulting guest thread is never woken. One 503 or one connection reset from object storage — the kind of thing that happens several times a day at any scale — permanently wedged a running VM.

This is the class of bug that only exists once you put storage in the page-fault path. In a normal system a transient read error is an error return. In a fault handler it is a thread that never resumes.

The fix was three layers, not one. A bounded fast retry with exponential backoff and jitter in the storage client, distinguishing retryable (transport error, 5xx, 429, 408) from permanent (4xx). Above it, a much longer durable retry budget — 120 seconds by default — so a sustained brownout stalls the touching thread rather than killing the machine. Above that, a watchdog that stamps progress on every retry so a fault that is actively retrying is not mistaken for a stall, plus counters for retries and for fatal handler exits, so when the handler does die it dies loudly.

The design principle I extracted: in a page-fault path, stalling is always better than failing. A stalled thread recovers. A dead VM does not.

Snapshots on the wrong disk, wiped by a host rebuild

A GCE host maintenance event on a nested-virt instance — which cannot live-migrate — triggered an automatic restart, which triggered the managed instance group's autohealer, which ran recreateInstance. That gave the host a fresh boot disk from a months-old image.

All hibernation snapshots lived on the boot disk. Twenty-two customer databases came back marked failed.

The durable volumes were on a separate persistent disk and survived intact — no customer data was lost — and the recovery path turned out to be a plain wake that falls back to booting from the durable volume when the snapshot is missing. I proved that on one database before batching the rest. But the lesson is not about the recovery. It is that a checkpoint is only as durable as the disk it landed on, and "the host's disk" is not durable. I had a second, smaller version of the same bug earlier: a custom template that existed only as a local rootfs on one agent, with no object-storage seed. It survived reboots, so it looked fine for months. It would not have survived a host replacement.

Both bugs have the same shape: local artifacts that look persistent because reboots are the failure you test and host replacement is the failure you do not.

The asymmetry: ~50s to hibernate, ~1.2s to wake

This one is not a bug. It is arithmetic, and it drives an entire product policy.

Operation	Measured	Why
Create (snapshot restore)	179ms p50 / 203ms p99	the only create path; no warm pool
PUT /snapshot/load alone	~80ms	lazy page-in, not a full read
First-ever spawn of a template	~3s	real cold boot, then bake
Fork, same host	400–750ms	reflink + memory restore
Fork, cross host	1.2–3.5s	fetch + restore
Wake from hibernate	~1.2s	restore path
Hibernate a 4 GiB app VM	~50s	write 4 GiB, sparsify, upload

Roughly forty to one. Hibernate is a bulk write; wake is a lazy read that touches a small fraction of the image. Everything I have built above optimises the read side, and none of it helps the write side.

The policy consequence is direct, and it is why the scale-to-zero behaviour is documented as a timeout rather than a promise. If sleeping costs 50 seconds of disk and waking costs 1.2, your idle timeout cannot be aggressive — a short window means you pay a 50-second write to save a few minutes of RAM, and if traffic is bursty you pay it repeatedly. The asymmetry, not the wake latency, sets the timeout. I worked through the full economics, including where my own wake number was wrong by a factor of ten before I measured it properly, in what scale-to-zero actually cost me to build.

What I would ask a vendor

The report is right that persistence has converged and checkpoints are becoming a checkbox. Which makes the checkbox useless as a purchasing signal — by 2027 everyone will tick it.

The question that still separates implementations is not "do you have checkpoints." It is:

What does your wake path do when the host that holds the checkpoint is gone?

Every hard problem in this post is downstream of that one. If the answer is "restores from object storage," ask what happens when a chunk fetch returns a 503 mid-fault. If the answer is "restores from the local disk," ask what happens when the instance group recreates the node. If the answer is a wake latency with no hibernate latency next to it, ask for both, because the ratio is what determines your bill.

I ask myself those three questions because I got all three wrong first. The rate card that came out the other side is $0.054 per active vCPU-hour and $0.0162 per GiB-hour, with no charge for idle — which is only possible because the checkpoint path is good enough that idle inventory is unnecessary. That pricing is a consequence of the engineering, not a marketing decision. It is worth asking any vendor which way round it is for them.

---

I'm Ajay Kumar, an infrastructure engineer with 14 years of experience and the creator of PandaStack, Riff, and PandaFlow. I take on a small number of consulting engagements around microVM platforms, snapshot/restore, and cloud infrastructure cost — get in touch on LinkedIn.