Firecracker microVM Cloud, Built Solo in 6 Months: 179ms Boots

I spent about six months building PandaStack, an open-source (Apache-2.0) Firecracker microVM cloud. Sandboxes, git-driven app hosting, managed Postgres, serverless functions. Roughly 400 Go files across the agent and control plane, and 55 Terraform files. One person.

Sandbox create is 179ms at p50, 203ms at p99. That number is the whole reason the architecture looks the way it does, and it's the part people ask about most, so I'll show the full boot path breakdown and where every millisecond goes.

I'll also cover the parts that don't fit on a landing page: what broke in production, what a single operator genuinely cannot do, and what it actually costs to keep a Firecracker fleet alive.

Who I am: I'm Ajay Kumar, an infrastructure and DevOps engineer with 14 years running production systems. I built and operate PandaStack solo. Everything below is from the running system, not a design doc.

Decision 1: snapshot-restore on every create, not a warm pool

The obvious way to make VM creation fast is to keep idle VMs around: a warm pool of booted machines waiting to be handed out. You pay for idle RAM and you serve requests instantly.

I refused to do that, and it turned out to be the single most consequential decision in the codebase.

The reason is economics, not elegance. A warm pool means your floor cost scales with peak concurrency, not actual usage. For a solo-operated platform with 300+ orgs signed up and spiky traffic, that's a permanent bill for VMs nobody is using. Idle has to be genuinely close to zero or the unit economics never work.

So there is no warm pool. Every create restores a baked Firecracker snapshot from scratch.

The mechanism: on a template's first spawn the agent does a real cold boot (~3s), lets the guest settle, then takes a Firecracker snapshot: memory image, device state, rootfs. Every subsequent create restores that snapshot. Slow once, then never again.

This has a second-order benefit I didn't anticipate. Because restore is the only boot path, it gets exercised on literally every request. There's no rarely-used cold path quietly rotting. If restore breaks, I know within seconds, not on the next scale-out event. I wrote up the full mechanism in the snapshot-restore internals doc.

The tradeoff I accepted: snapshots are brittle in a specific way. Re-baking a template invalidates every snapshot derived from it, and guest identity (IP, MAC, gateway) is frozen at bake time. Which leads directly to the next decision.

Decision 2: pre-allocated network namespaces, because ip netns add is expensive

Here's a thing you find out fast when you're chasing a sub-200ms budget: Linux network setup is the second-biggest cost after the hypervisor itself.

Creating a fresh network namespace, a veth pair, a TAP device, and the iptables NAT rules for one sandbox costs roughly 100ms cold. That's over half my entire latency budget spent on ip and iptables invocations.

Time it yourself:

time sudo ip netns add test-ns
time sudo ip link add veth-h type veth peer name veth-g netns test-ns
time sudo ip netns exec test-ns ip tuntap add tap0 mode tap

Individually they look cheap. Together, with NAT rule programming, they aren't.

So the agent pre-builds them. It carves 16,384 /30 subnets out of 10.200.0.0/16 and keeps a pool of fully-constructed slots sitting idle and ready to claim: namespace, veth pair, TAP device, NAT rules already programmed. On create, the agent claims a slot and patches the TAP MAC address to match what the snapshot was baked with. That's ~1ms to claim, ~6ms to configure.

Roughly 100ms of work becomes roughly 7ms, because the expensive part happened minutes ago.

Two things I'd flag for anyone copying this:

The pool depth is not a concurrency cap. The warm prebuild depth defaults to 4 slots per template identity. When it drains, allocation falls back to building a slot from scratch, about 500ms. Slow, but not an error. It does not return 503. I've seen people assume a pool size is a hard limit; here it's a latency optimization with a graceful degradation path.

The real ceiling is 16,384 sandboxes per agent, from the /16 address space, and you will never hit it. Host RAM binds long before addresses do. My base template is 4 GiB, code-interpreter and agent are 2 GiB, browser is 4 GiB, postgres-16 is 1 GiB. Do the division on any real machine and you run out of memory two orders of magnitude before you run out of subnets.

Full detail in the networking internals doc.

Decision 3: XFS reflink for copy-on-write rootfs

Every sandbox needs its own writable disk. Copying a multi-gigabyte ext4 image per create is obviously off the table.

XFS reflink solves this in one syscall:

cp --reflink=always /var/lib/pandastack/templates/base/rootfs.ext4 \
                    /var/lib/pandastack/vms/$ID/clone.ext4

That's an O(metadata) operation. The extents are shared until someone writes, then the filesystem splits them. It costs about 4ms in the boot path regardless of image size.

This is also what makes forking a sandbox cheap. Same-host fork is 400–750ms: reflink the disk, restore the memory snapshot, done. Cross-host fork is 1.2–3.5s because the snapshot has to come down from object storage first.

The constraint reflink imposes is that the rootfs must be a local file. You cannot stream a block device you want to reflink. That distinction becomes important in the next section, and it's the thing I see people get wrong most often when they talk about "streaming VMs."

Decision 4: UFFD to demand-page memory from object storage

The remaining problem: a 4 GiB template snapshot has a 4 GiB memory file. On a fresh host, or after a rolling deploy, restoring means downloading 4 GiB before the VM can start. That's not a 179ms boot, that's a coffee break.

The fix is userfaultfd. Instead of handing Firecracker a memory file, the agent registers a userfaultfd handler over the guest's memory regions and passes the file descriptor to Firecracker over a Unix socket. When the guest touches a page it hasn't seen:

1. The kernel raises a page fault and delivers it to my handler.
2. The handler maps the faulting address to an offset in the snapshot's memory image.
3. It fetches the containing 4 MiB chunk from GCS with an HTTP Range GET.
4. It installs the page with UFFDIO_COPY and the guest continues.

Three optimizations make this practical rather than merely clever:

Zero-elision. A baked sidecar header records which chunks contain non-zero bytes. A freshly-booted guest's RAM is overwhelmingly zeros, and there's no reason to pay a network round trip to fetch a page of nothing. Absent chunks get zero-filled locally with no fetch at all.

Prefetch traces. At bake time I record which chunks the guest actually touches during boot. On restore, that hot set is replayed in the background so the faults that matter become cache hits.

A shared on-disk chunk cache. Chunks fetched by any restore land in a per-seed sparse cache keyed by a hash of the object path, so re-bakes self-invalidate. The first restore on a host pays GCS latency once. Every subsequent restore of that template reads from local disk. It's crash-safe: the presence bitmap only advances after fdatasync of the data file and an atomic rename, so a set bit always implies durable bytes.

To be precise about what's streamed: UFFD streams memory, not disk. The rootfs still has to be local because reflink CoW needs a local block device. Streaming removes the multi-gigabyte memory download, which is the part that actually dominates.

The 179ms boot path, step by step

Measured on the production agent, snapshot-restore path with a warm NATID slot:

Step	Cost
Claim pre-allocated NATID slot	~1ms
Configure TAP in the namespace	~6ms
Reflink rootfs clone	~4ms
Fork + exec the Firecracker process	~25ms
PUT /snapshot/load	~80ms
PATCH /vm → Resume	~6ms
Probe guest TCP :22 for readiness	~40ms
Insert the sandbox row (async)	~6ms

The synchronous path sums to about 162ms. Observed p50 is 179ms and p99 is 203ms. The gap is HTTP overhead between the control plane and the agent, plus scheduler selection. I pulled each of these stages apart separately in my stage-by-stage measurement of the boot path, including how the timings are instrumented and why the arithmetic does not partition the wall clock cleanly.

Two observations from staring at this table for months.

Snapshot load at 80ms is the largest single item and it's mostly out of my hands. That's Firecracker restoring device state and setting up memory mappings. Memory is mapped MAP_PRIVATE, so pages fault in lazily rather than being copied upfront. That's why an 80ms restore of a 4 GiB guest isn't a contradiction.

The 40ms readiness probe is the one I could most plausibly cut, and it's the one I'm least willing to. Returning a sandbox ID before the guest can accept a connection just moves the failure into the user's first API call. I'd rather eat 40ms than ship a race condition.

You can watch these numbers on any deployment:

curl -s http://localhost:9100/metrics | grep -E 'pandastack_(sandbox_boot|uffd)'

The full state machine (create, pause, snapshot, fork, hibernate, wake) is documented under sandbox lifecycle.

What actually broke

Design docs are cheap. Here's what production did to me — the short version; I wrote the full timelines up as four postmortems from this fleet.

GCP host maintenance ate a fleet. A maintenance event triggered recreateInstance on the managed instance group, which wiped boot disks. Data on separate persistent disks survived, so no customer data was lost, but every locally-baked snapshot on those hosts was gone and the databases came back dead. The lesson wasn't "back up more." It was that I had a class of artifact, locally-baked seeds, with no authoritative copy in object storage. Anything that only exists on a boot disk in a MIG does not exist.

A config knob drifted and silently disabled a subsystem. A setting that lived only in cloud-init got quietly reverted when a host was rebuilt. Database auto-suspend stopped working, the memory admission ledger filled with VMs that should have been asleep, and deploys started 503-ing. The bug was not in any code path; it was that the knob had exactly one home and that home was rebuild-scoped. Anything you can set in cloud-init and nowhere else will drift forever, and it will drift silently. Configuration needs a source of truth that survives instance replacement.

The UFFD handler died on a single GCS blip. One transient object-storage error, and the page-fault handler exited. When a UFFD handler exits, every guest whose memory it was serving hangs on its next fault. One HTTP 500 could wedge a host's worth of VMs. It took retries at three layers (chunk fetch, fault service, handler supervision) before that path was genuinely safe. If you build on userfaultfd, treat the handler as a hard availability dependency of every VM it serves, because it is.

Crypto mining. Free-tier accounts doing exactly what you'd expect. Contained by blocking Stratum protocol egress at the network layer. If you offer arbitrary code execution, this is not an edge case, it's week one.

What a solo founder can and cannot do

Can: ship a kernel-adjacent distributed system. Six months alone was enough for userfaultfd page-fault handling, copy-on-write disks, pre-allocated network namespaces, blue-green git deploys, managed Postgres with branching and point-in-time restore, two SDKs, a CLI, an OpenAPI 3.1 spec, and an MCP server. That surprised me. Tooling for infrastructure work in 2026 is genuinely good.

Cannot: be on call 24/7. I sleep. If an agent wedges at 3am, it stays wedged until I wake up. I've compensated where I can — health-check loops that auto-restart failed apps, a lease system that routes around agents with stale heartbeats, watchdogs that reconcile state rather than page a human — but automated recovery has a ceiling and I am below it. That constraint is why the platform is pre-revenue by design. I won't sell an SLA I can't personally staff.

Also cannot: parallelize incident response. During the MIG incident I was doing root cause analysis, customer communication, recovery scripting, and permanent-fix design at once. Those are four jobs. Doing them sequentially is why an incident a team resolves in two hours takes me a day.

The honest cost

I won't publish the invoice, but I'll tell you its shape, because the shape is more useful than the number.

Memory is the bill. Not CPU. My templates run 8 vCPU as burst capacity, shared fairly under contention via cgroup weights, and CPU is billed on active CPU-seconds actually burned. But committed RAM is committed whether the guest uses it or not, and RAM is what caps sandboxes per host. Every architectural decision above exists to raise the number of sandboxes I can fit on a given amount of memory, or to let idle ones cost nothing — I put real numbers on that second half in what scale to zero cost me to build.

Object storage is the second line item, and it grows with fork trees and snapshot history, not with active users. That one sneaks up on you.

The largest cost is engineering time, and it isn't close. Building the boot path took weeks. Making it survive host replacement, config drift, transient storage errors, and abuse took months. The ratio of "make it work" to "make it not fall over" was roughly 1:4, and that ratio is the real answer to how long a project like this takes.

If you're evaluating whether to build a Firecracker platform or buy one, budget for the 4, not the 1.

Related reading

• What a copy-on-write VM fork really preserves
• Why isolation was never my abuse control

---

The platform is at pandastack.ai and the internals are documented publicly. I take a limited number of infrastructure consulting engagements (Firecracker, GCP, Terraform, and the general problem of making a fleet survive its own operators) at $75–100/hr or a $6–8K/mo retainer. Find me on LinkedIn or Upwork.