microVM Production Incidents: 4 Postmortems From My Fleet

I built PandaStack alone over about six months: an open-source Firecracker microVM cloud where every sandbox create is a snapshot restore, p50 179ms, p99 203ms. Around 300 organizations have signed up. I am also the only person on call for it.

That last part is what this post is about. Building a microVM platform is a distributed-systems problem, and I wrote that half up separately in what it took to build a Firecracker cloud solo. Operating one for other people is a different problem, and every lesson I have learned about it arrived as an incident. Here are four, with the mechanism and the thing I changed afterward. Three cost customers something real.

The fleet, so the incidents make sense

Agent nodes are GCE n2 instances (nested virtualization, so they cannot live-migrate). Each runs a Go agent that owns Firecracker VMs directly: 16,384 pre-allocated /30 network namespaces, XFS reflink copy-on-write rootfs, snapshot restore on every create rather than a warm pool. Managed Postgres databases are their own microVMs, pinned to the host that created them, with a durable volume for the data directory.

The storage layout is the whole story of incident one:

Path	Backing device	Contents
/var/lib/pandastack	loop-mounted XFS image on the boot disk	template seeds, VM working dirs, hibernation snapshots
/var/lib/pandastack/volumes	separate attached persistent disk	customer database volumes, one ext4 file each

I designed that split deliberately. Customer data goes on the durable PD. Everything reconstructible goes on the boot disk. I was proud of that decision right up until I learned what "reconstructible" actually meant.

Incident 1: host maintenance ate three days of hibernation snapshots

On 2026-08-26 at 18:19 UTC, GCE performed host maintenance on one of my agent nodes. Because nested virtualization forces terminateOnHostMaintenance, the instance was terminated rather than live-migrated. automaticRestart fired, the managed instance group's autohealer decided the node needed repair, and it ran compute.instances.repair.recreateInstance.

That call does not restart your VM. It rebuilds it from the instance template, which means a fresh boot disk.

The attached PD reattached cleanly. All 22 customer database volumes survived, 5 GiB ext4 files, completely intact. No customer data was lost, and I want to be clear about that because the rest of this is bad enough.

What died was the loop-mounted XFS image holding vms/, where every hibernation snapshot lives. My platform hibernates idle databases for real scale-to-zero: the VM is paused and snapshotted, compute billing stops, and the next connection wakes it in about a second. Twenty-two databases were in exactly that state when the boot disk was replaced with a blank one.

A second detail made it worse. The instance group's template was from 11 August. The node that came back was not a copy of the node that went away, it was a copy of a node from two weeks earlier.

Recovery worked better than I feared, thanks to a fallback I had shipped days earlier for an unrelated reason: when a wake finds no hibernation snapshot, the agent cold-boots from the durable volume instead, in 117-399ms. Every database whose owner reconnected after the wipe was silently rescued by that path. I proved it deliberately on 2026-08-29 by calling the agent's wake endpoint on one failed row: running in 113ms, healthy.

The lesson. I could have drawn my storage diagram from memory and still got it wrong, because I was reasoning about durability of customer data when the real question was what is on the boot disk when the boot disk is replaced without warning. Snapshots felt like cache. They are cache in the sense that losing them does not lose data. They are not cache in the sense that losing them takes 22 workloads offline until something touches each one.

Two rules came out of this. Write down, per mount point, what happens during an instance recreate, not a reboot; those are different events and most people only ever test the reboot. And treat an instance template as versioned config that drifts from your running fleet the moment you change anything outside it, because your cloud provider can collapse that drift to zero at 18:19 on a Wednesday.

Incident 2: a knob that only existed at first boot

This is the direct sequel, and it is my favorite because the root cause is so boring.

Managed databases auto-suspend after 15 minutes idle. The window is set by one environment variable, PANDASTACK_DB_IDLE_AFTER_SECONDS=900. When I rolled it out I set it in two places: cloud-init for new nodes, and by hand on the existing ones.

Cloud-init writes the agent's env file only at first boot. The rebuilt node came from the 11 August template, whose cloud-init predates that variable. So the node came back healthy, passed every health check, served traffic, and had silently lost database auto-suspend entirely. The other agent still had it.

The failure took a few days to build. Every database that woke stayed awake forever, and committed guest memory accumulated until the agent's admission ledger was full:

507: 4096 MB requested, 30720 MB already committed,
     budget 31066 MB (host 32090 MB - reserve 1024 MB, overcommit 1.00)

I run overcommit at 1.00 deliberately. Guests fault pages in on demand but Firecracker does not give the memory back, so oversubscribing means killing someone's VM later. Once the ledger was full, admission refused everything. Customers saw 503 no compute node on app deploys, and wakes on their databases were refused.

Then the second-order effect. My app auto-deploy had no retry backoff, so a 503 produced an immediate retry, which produced another 503. In 24 hours there were 9,230 failed deployment rows, and a single application accounted for 6,483 of them. A memory bookkeeping problem on one host became a self-inflicted retry storm across the control plane.

The fix was one line moving in one file, from cloud-init into Ansible's managed env block:

# ansible/group_vars/all.yml
agent_env_managed:
  PANDASTACK_STREAM_DISK: "0"            # rootfs demand-paging: off until re-verified
  PANDASTACK_NOHUGE_TEMPLATES: "..."     # templates excluded from the hugepage pool
  PANDASTACK_DB_IDLE_AFTER_SECONDS: "900"

Cloud-init sets a value once. Ansible re-asserts it on every run. That is the entire difference, and it converts this class of bug from permanent to self-healing.

Auditing the rebuilt host turned up two more variables that had drifted the same way, including one gating a rootfs-streaming feature that had corrupted guest disks in an earlier experiment. It had been sitting at the wrong value on the host carrying every managed database, dormant purely by luck.

Then verification, because "I fixed it" is not a claim I get to make without evidence. I drained six idle databases, restarted the agent gracefully (6 of 6 hibernated in 70 seconds, zero failures), watched committed memory go from 30,720 MB to 0, watched a create that had been returning 507 return 201, then woke a database, left it alone, and waited for the sweeper to log db idle-suspend: hibernating idle database ... idle_for=15m29.9s on its own.

The lesson. Any setting whose absence is unsafe must live in something that re-asserts it, not something that applies it once. First-boot-only config does not drift slowly. It drifts absolutely, on the first rebuild, and stays wrong forever because nothing ever looks at it again. If I cannot answer "what re-asserts this?" in one sentence, the knob goes into config management before it goes near the fleet.

Incident 3: one transient 503 killed a running VM

This one makes me least comfortable, because I found it by reading my own code rather than by being paged.

Streamed restore is the feature I am proudest of. Instead of downloading a multi-gigabyte vm.mem before booting, the agent registers a userfaultfd handler with Firecracker. The guest touches a page, the kernel raises a fault, my handler maps it to an offset in the snapshot, fetches a 4 MiB chunk from object storage with an HTTP range request, and installs it with UFFDIO_COPY. A baked header records which chunks are non-zero so absent ones are zero-filled with no network call at all.

It works. It is also, structurally, a system where a remote read sits inside a page fault.

The original implementation had two defects that composed into something fatal. The object-storage source did a single HTTP request with no retry, and the handler escalated the first resolve error into a fatal flag that terminated the whole handler.

Terminating the handler does not crash the VM. It does something worse: the guest thread that faulted is never woken. It waits on a page that will never be installed. The VM hangs forever, and the process is still alive, so nothing marks it unhealthy.

One transient 503, one connection reset, one 429 during a busy minute, and a customer's running microVM becomes a process that answers nothing.

The fix shipped in three layers, because one was not enough:

1. Source retry. Bounded fast retry on the range GET: 4 attempts, exponential backoff with full jitter, retryable on transport error, 5xx, 429 and 408, permanent on 4xx. Bounded and fast matters, because this sits inside a page fault and p50 fault latency is user-visible.
2. A durable fault-retry budget. Above the source, a separate budget rides out a sustained brownout, default 120 seconds, tunable by env var. Fast retry handles a blip. The budget handles object storage having a bad five minutes.
3. Dying loudly. When the budget is exhausted the handler increments uffd_handler_fatal_total, with uffd_fault_retries_total counting every retry along the way. Both are on the agent's Prometheus endpoint, alongside a watchdog tracking time since last fault progress.

The lesson. Demand paging moves a network call into the memory subsystem. Everything you know about handling remote errors still applies, but the blast radius changed: the caller you fail is not an HTTP client that can retry, it is a kernel thread that will block until the machine is rebooted. In a hot path like this, every remote read needs retry, backoff, a bounded budget, a fallback, and a metric. Not one of those. All of them.

The original code was not careless. It was correct for the case it was written for, and that case was "the fetch works". I only found it by asking the codebase a specific question: what is the worst thing a single transient dependency failure can do here.

Incident 4: three days of dead databases and no alert

The 22 dead databases from incident one were not discovered by an alert. They were discovered because I was running a routine abuse scan and typed a query I type often:

select status, count(*) from sandboxes group by status;

Twenty-four failed. Six days earlier that number had been two.

Every layer of my observability said things were fine:

• ClickHouse had zero failed-type events for any of the 22. Several code paths set the status without emitting an event, so every transition was silent.
• The backup health check reported ok for all of them, correctly, because it checks whether backups exist and are recent, not whether the database is alive. All 22 had history: 144 backups across 19 distinct customer organizations. They had provisioned, served traffic, taken backups, and then died with a healthy-looking backup record.
• The old instance's journal died with its boot disk, because I was not shipping agent logs off-host. Per-database failure timestamps are permanently unrecoverable.

The concentration is what made the root cause findable at all:

Agent node	failed	hibernated	running
node A	22	3	3
node B	0	11	0

A 79% failure rate on one host and 0% on the other is not a product bug. That is a host, and it pointed straight at the drift.

The lesson. I had spent months on prevention: admission control, isolation, capacity gating, egress controls. I had spent almost nothing on detecting that a resource was simply dead. A count of rows in a terminal state, alerted on when it moves, would have caught this in minutes instead of three days.

Three changes came out of it. Alert on failed-state counts by resource type, on the absolute count moving rather than a rate, because 22 failures over three days is a rate you will miss. Make every transition into a terminal state emit an event and a log line, because I had to reconstruct this incident from backup catalogs and a partial journal. And ship agent logs off-host: losing the boot disk should not lose the evidence of why.

One design change too. Hibernation snapshots belong on the durable disk, or boot-disk loss must proactively re-hydrate every hibernated row instead of waiting for someone to reconnect.

What actually changed in how I operate

Know your disks, not your intentions. I knew what I meant to put on the durable volume. I had not checked what an instance recreate does to everything else.

Config applied once is config that is wrong. The only question worth asking about a setting is what re-asserts it.

Failure paths are where the bugs are. Every one of these was a defect in handling an unlikely event, not in the happy path. Users test the happy path constantly. Production tests the failure path once, at scale.

Detection is not a subset of prevention. The expensive part of the database incident was not the failure. It was the three days.

Running this solo removes the illusion that someone else is watching. There is no second pager, no team channel where somebody mentions that a graph looks odd. If a query I did not run would have found it, it does not get found.

Related reading

• What scale to zero actually cost me to build
• Why isolation was never my abuse control
• What a copy-on-write VM fork really preserves

Who I am

I am Ajay Kumar, an infrastructure and DevOps engineer with 14 years of experience. I built and operate PandaStack, an Apache-2.0 Firecracker microVM cloud: sandboxes, git-driven app hosting, managed Postgres with branching and point-in-time restore, and serverless functions. The architecture is documented publicly, including the isolation model and the database backup and recovery model these incidents stress-tested.

I take on infrastructure consulting: LinkedIn, Upwork, GitHub.