I Measured Every Stage of My 179ms Firecracker Boot Path

Sep 2, 2026 · 9 min · Ajay Kumar

I built PandaStack, an open-source Firecracker microVM cloud, solo over about six months. The number I care about most is how long a POST /v1/sandboxes takes to return a machine you can actually talk to. In production that is 179ms at p50 and 203ms at p99.

People assume that number comes from a warm pool. It does not — there is no pool of idle VMs, and every create restores a snapshot from scratch. This post is the instrumentation, not the marketing: how I define the clock, what each stage costs when I read it off the production fleet, and the two places my own numbers are less tidy than a table makes them look.

I am Ajay Kumar, an infrastructure engineer with 14 years on this kind of plumbing. Everything below came off my own fleet, not a benchmark rig. The wider story of how the platform got built is in what it took to build a Firecracker cloud solo.

How the number is measured

The agent emits a boot event per create into ClickHouse with a boot_ms field and a map of per-stage timings; p50/p99 are quantiles over that table, not a stopwatch on a happy-path demo.

The definition matters more than the percentile. boot_ms is wall time from the moment the agent accepts the create request to the moment the guest answers a TCP connection on port 22 — not to "firecracker process exists." A sandbox that has resumed but is not accepting connections is not a sandbox you can use, so I refuse to stop the clock there. That choice costs me about 40ms of the 179.

The stage breakdown

Stage	Cost	What it does
NATID slot claim	~1ms	Pop a pre-built network slot off the free list
Tap configure in netns	~6ms	Patch the tap MAC to match the baked identity
Reflink rootfs	~4ms	XFS clone of the template disk
fork+exec firecracker	~25ms	Spawn the VMM, bring up its API socket
`PUT /snapshot/load`	~80ms	Load device state, map guest memory
Resume	~6ms	Unfreeze the vCPUs
TCP probe on :22	~40ms	Wait for the guest to answer
Sandbox row insert	~6ms	Persist metadata (off the critical path)

That sums to about 168ms. The remaining ~11ms is request handling, allocation bookkeeping, and jitter.

One honest caveat about the arithmetic: these are per-stage measurements, not a strict partition of the wall clock. The TCP probe goroutine is launched before resume, so the first SYN is already in flight the moment the vCPUs unfreeze. Roughly the resume cost overlaps with the probe. I kept them as separate rows because that is how the instrumentation reports them, but do not read the table as a waterfall with no overlap.

Stage 1 and 2: networking, ~7ms because the work already happened

Cold-creating a network namespace is the most expensive thing in a naive create path. ip netns add, then ip link add for the veth pair, then a tap device, then iptables rules — roughly 100ms of netlink round trips and rule-table churn, none of it parallelising usefully.

So none of it happens during a create. The agent pre-builds slots: 16,384 pre-allocated /30 subnets carved out of 10.200.0.0/16, each one a complete (netns, veth pair, tap, iptables) unit sitting on a free list.

The subtle part is what a slot is keyed on. Firecracker cannot change a guest's network identity at restore — the guest inside the snapshot already believes it has a specific IP, MAC, and gateway, frozen at bake time. So the DNAT rules inside the netns are wired at prebuild time to that exact baked identity, and a slot can only be claimed by a restore whose snapshot carries a matching one. The pool is effectively per-template, not global.

Claiming a slot is: pop from the free list, patch the tap MAC. About 7ms combined, with a background refill after each claim.

Two things I have to keep correcting people on:

The pool depth is not a concurrency limit. The default prebuild depth is 4 slots per template identity. When it drains, allocation does not return 503 — it falls back to building a slot from scratch at roughly 500ms. You get a slow create, not a failed one.

16,384 is not the real ceiling either. It is the ceiling from the /16 address space. The binding constraint in practice is host RAM, because baked template sizes are what they are: 4 GiB for the base apps runtime, 2 GiB for the code-interpreter and agent templates, 1 GiB for Postgres. You run out of memory long before you run out of subnets. More on the namespace and DNAT layout in the networking internals doc.

Stage 3: reflink, ~4ms for a multi-gigabyte disk

Every sandbox gets its own writable rootfs. Copying a 2–10 GiB ext4 image per create would obviously be fatal, so the agent does cp --reflink on XFS.

A reflink copy is O(metadata). It duplicates the extent map and marks the extents shared; it does not move a byte of data. The new file is a full independent disk from the guest's point of view, and the first write to any block triggers copy-on-write at block granularity.

Four milliseconds for a multi-gigabyte disk clone is the best ratio in the pipeline. It is also the foundation of forking a running sandbox — same mechanism, applied to a live VM's disk instead of a template's, which is why a same-host fork lands in 400–750ms. That path is in the fork and copy-on-write internals doc, and I wrote up what a fork actually preserves and where it quietly does not after a production smoke test contradicted my own documentation.

Stage 4: fork+exec firecracker, ~25ms

Process spawn, the VMM's API socket coming up, and a private mount namespace for the VM directory.

25ms is not fast in absolute terms — it is a seventh of the budget for something that has not booted anything yet. It is fast relative to any full VMM: Firecracker has no BIOS, no PCI enumeration, no option ROMs, and a device model of about five devices. There is nothing to probe. I have not found a way to shrink it without pre-spawning VMM processes, which reintroduces the warm-pool problem I am specifically trying to avoid.

Stage 5: snapshot load, ~80ms — and why it is not 8 seconds

This is the stage that makes the whole design work, and it is the one people are most suspicious of. "You restored a 4 GiB guest in 80 milliseconds? That would need 50 GB/s of disk."

It would, if a restore read the memory file. It does not.

A Firecracker snapshot is two artifacts: a small state file with the device model and vCPU registers, and a memory file that is a raw image of guest RAM. Loading maps the memory file with MAP_PRIVATE instead of reading it. The kernel sets up page table entries and returns; nothing is faulted in. Then the guest runs, touches a page, takes a fault, and the kernel pulls exactly that page from the file. MAP_PRIVATE means any write gets a private copy and the backing file is untouched, so one snapshot file serves every concurrent restore of that template.

Three consequences fall out of that:

Restore latency is roughly independent of guest RAM size. The 4 GiB base template restores at about the same speed as the 1 GiB Postgres template. Mapping setup dominates; image size barely registers.
The cost is deferred and spread, not eliminated. Page-in is paid during execution, a fault at a time, and most of a booted guest's RAM is never touched at all.
The snapshot file is shared. Concurrent sandboxes off one template share page cache for every page none of them has written.

The full mechanism is in the snapshot restore internals doc.

There is a variant where the memory image is not on local disk at all: the agent hands Firecracker a userfaultfd handler instead of a file path, and guest page faults are served by HTTP range GETs against object storage with a local chunk cache in front. That removes the multi-gigabyte download from the "new host, cold template" path entirely. It is written up in the streaming restore doc, and it is where most of my worst bugs have lived.

Stage 6: resume, ~6ms

Unfreeze the vCPUs. The guest resumes at the exact instruction it was paused on when the snapshot was taken.

One thing bites here that is not obvious: the guest's clock resumes at the frozen value too. A guest restored from a snapshot baked hours ago believes it is hours in the past, which breaks TLS certificate validation on the first outbound HTTPS call. I had to add an explicit clock sync on restore, resume, and wake. Everyone building on snapshot restore hits this eventually.

Stage 7: the TCP probe, ~40ms of honest waiting

40ms — roughly a quarter of the total — is the agent doing nothing except waiting for sshd inside the guest to answer a SYN.

I could delete this stage and report 139ms creates. I would be lying. The gap between "vCPUs are running" and "the listener is accepting" is real, and if I return before it closes, the caller's first request fails and they retry, which costs them more than 40ms. The only mitigation is the overlap: the probe starts before resume, so the connection attempt races the guest coming back rather than queueing behind it.

The probe goes to the veth-side address, where iptables DNATs it to the baked guest IP. That indirection is what lets the guest keep the fixed identity its snapshot was baked with while the host addresses it through a per-sandbox slot.

Stage 8: the database write, ~6ms and off the critical path

Persisting the sandbox row costs about 6ms. On the fast path it does not block the response — the row is written after the VM answers. Moving it out of the critical path was worth 6ms, which is 3% of the budget for a change that took an afternoon.

Cold boot: ~3 seconds, and why the gap is the whole point

The first time a template is ever spawned on a host there is no snapshot to restore. That path is a real Linux boot: kernel init, systemd reaching its target, sshd generating host keys and starting, network configuration. It costs about 3 seconds, roughly 17x the snapshot path.

That first boot auto-bakes a snapshot, and every subsequent create on that host takes the 179ms path. Pay 3 seconds once per template per host, amortize it across every create afterward.

The gap is also a useful sanity check on the 80ms restore claim. Restoring is not booting faster. It is not booting at all. All the work that made those 3 seconds — sshd's keys generated, systemd's units settled — is frozen into the memory image and replayed by page fault.

The tradeoff: no warm pool, and what that costs

The obvious alternative to snapshot restore is a warm pool: keep N booted VMs idling and hand one out on request. Handing back an already-running VM skips almost everything in the table above, so it wins the happy path. I have not built one, so I am not going to invent a number for it.

Here is the arithmetic that killed it for me. The base template is 4 GiB. Take a host with 128 GiB of usable RAM: ten warm base VMs is 40 GiB, nearly a third of the host, pinned to do nothing. Then multiply by template count, because a warm base VM cannot serve a request for postgres-16. Pool depth times templates times template RAM, all committed before a single customer has asked for anything.

With snapshot restore, idle cost is disk for the snapshot artifacts plus whatever page cache the kernel decides to keep. No RAM is committed to a machine nobody has asked for.

What I give up:

p99 is 203ms, and a pool that hands out running VMs would beat it. Every stage in the table except the readiness probe is work a warm pool has already done.
The first restore of a template on a fresh host is slow — it either downloads the memory image or pays object-storage latency on early faults.
The prebuilt slot pool can drain, and a create that has to build a namespace from scratch costs ~500ms instead of 7ms.

What I get beyond the RAM: because there is no pool to keep warm, scale-to-zero is nearly free. Hibernating an idle app deletes the VM entirely rather than parking it, and waking it is the same restore path, about a second. A warm-pool architecture cannot do that, because the pool is the idle cost. "Nearly free" is doing some work in that sentence, though — I measured what the sleep and wake sides actually cost, and hibernation is not the cheap half.

Things that broke on the way here

Honest list, because the pipeline above reads cleaner than it was.

vsock socket collisions under concurrent restore. The snapshot bakes a vsock device at a fixed absolute path, and Firecracker recreates that listener at the same path on restore. Two concurrent restores of the same template both try to bind it: Address in use (os error 98). The fix is spawning each VMM in a private mount namespace with its own bind-mounted socket directory, so each restore's socket lands on a separate inode.

The streaming memory handler died on a single blip. One transient object-storage error and the fault handler exited, which wedges the guest — every subsequent page fault hangs forever with nobody to answer it. It took a three-layer retry to survive one bad HTTP response. A demand-paging handler must never exit; there is no "return an error to the caller" when the caller is a page fault.

Hugepage snapshots have a hard restore constraint. Backing guest memory with 2 MiB pages cuts fault counts by 512x, but Firecracker will only restore a hugepage snapshot through the userfaultfd backend — the plain file path is rejected. So hugepage-ness has to travel with the snapshot as an explicit marker, and every restore path has to check it and force the right backend regardless of configuration. Relatedly, re-baking a template invalidates every snapshot taken from it, because the baked identity no longer matches. That has to be detected up front, not discovered at restore time.

The general lesson

Nothing above is a clever trick. Every stage is fast for the same boring reason: the expensive work was moved out of the request path. Namespaces are pre-built. Disks are shared extents. Memory is mapped, not read. The boot already happened, three seconds at a time, once per host.

The only stage I could not move out is the 40ms spent waiting for the guest to answer. That is the one I refuse to hide, because it is the difference between a number and a machine.

Keep reading

Sep 2, 2026