My Fork Docs Were Wrong: What VM Forking Really Preserves

I run PandaStack, an open-source Firecracker microVM cloud I built solo over about six months. One of its primitives is forking a live sandbox: take a running VM and get N copies of it, each with the parent's state, in a few hundred milliseconds.

The pitch writes itself. An agent hits a decision point, forks its environment five ways, tries five different fixes in real execution, keeps the one whose tests pass. Tree search over the world instead of over tokens.

The pitch is also where I got burned. My own internals doc claimed a fork gave you "same processes, same open file descriptors, same in-memory variables." Then I ran a full new-user smoke test against production with the published SDK, and a child fork could not see a file the parent had written to /tmp seconds earlier. The doc was describing a different code path than the one the fork() SDK method called.

So this post is the mechanism as it actually works, with the numbers I measured and the failure modes I hit.

Three fork paths, not one

The first thing to internalise is that "fork" is not one operation. There are three, and they preserve different things:

	plain fork	fork-tree	fork with mode: warm
Child inherits	disk only	disk and memory	disk and memory
Running processes survive	no	yes	yes
How the child starts	full kernel + init boot (~3s)	snapshot restore, a few hundred ms each	snapshot restore, under ~500ms
Children per call	1-16	1-16	exactly 1
Parent afterwards	paused for the copy, then resumed	paused for the snapshot, then resumed	paused and stopped
Child network identity	fresh IP/MAC	fresh IP/MAC	reuses the parent's

The plain fork pauses the parent, copies its rootfs for a consistent disk view, resumes the parent, and then cold-boots each child on that disk copy. Anything that lived only in RAM is gone: a warm Python interpreter, an imported module, an open database connection, a dev server on a port. What survives is what hit the disk.

fork-tree is the one people actually mean when they say "fork a running VM." It snapshots the parent once — pause, capture memory plus device state, resume — and restores N children from that one snapshot in parallel. Processes survive there. That distinction is now written up properly in snapshots and forks, and correcting it was the most useful thing that smoke test produced.

Layer 1: the disk, via XFS reflink

Every sandbox's disk is a single ext4 image file on the host, sitting on an XFS data partition. Cloning it is one ioctl:

cp --reflink=always /var/lib/pandastack/vms/$PARENT/rootfs.ext4 \
                    /var/lib/pandastack/vms/$CHILD/rootfs.ext4

Under the hood that is FICLONE. XFS creates a new inode that points at the same extents as the original and bumps the refcounts. No data is read, no data is written. The cost is proportional to the extent map, not the file size, so a multi-gigabyte image clones in single-digit milliseconds. The agent tries FICLONE first and falls back to a byte copy only when the filesystem can't do it — which is exactly what happens on my Mac dev loop inside Lima, where the guest filesystem has no reflink support and a "fast" clone quietly becomes a 150ms read-write of the whole image.

After the clone, both images are independent as far as either side can tell. The first write to any shared block allocates a fresh extent for the writer. The parent never sees the child's writes and the child never sees the parent's.

This is the part of the system that never surprises me. Block-level CoW is old, boring, and correct.

Layer 2: the memory, via snapshot plus MAP_PRIVATE

Memory is where the interesting engineering is, because you cannot reflink RAM.

Firecracker's snapshot writes the guest's memory out to a vm.mem file and the device state to vm.state. The pause and resume around it are a few milliseconds each; the write is the expensive part and scales with guest RAM. To restore, the agent execs a fresh Firecracker process and hands it that memory file, mmaped MAP_PRIVATE.

MAP_PRIVATE is what makes fan-out affordable. Every child's guest RAM initially points at the same physical pages backing that one snapshot file. The kernel copies a page only when a child writes to it. Ten children restored from one 2 GiB snapshot do not cost 20 GiB of RSS — they cost the shared image plus each child's own dirty set. Two children scribbling on different parts of memory never contend.

The same trick is why a normal sandbox create is fast at all. There is no warm pool on my platform. Every single create restores a baked template snapshot: p50 179ms, p99 203ms, with the snapshot restore itself around 80ms p50. Under an 8-way concurrent burst against production, server-side boot came out at p50 230ms, min 80ms, max 377ms. A fork is that same machinery pointed at a user's snapshot instead of a template's.

There is a second memory path worth knowing about, because it changes the fork story: for template restores I don't download the memory file at all. A userfaultfd handler intercepts the guest's page faults and pulls 4 MiB chunks from object storage over HTTP range requests, with a baked header recording which chunks are non-zero so all-zero pages get filled locally without a fetch. That is streaming restore, and it is also the single most fragile component I own — a one-request GCS blip killed the handler and took the VM with it until I added three layers of retry.

Why same-host is 400-750ms and cross-host is 1.2-3.5s

Same-host fork: 400-750ms per child. Cross-host: 1.2-3.5s.

The gap is entirely object storage. Same-host, the parent's snapshot tuple and rootfs are already on the local NVMe, so the work is a reflink, a Firecracker exec, an mmap, a resume, and a network namespace allocation. The netns is pre-built — I keep a pool of pre-allocated /30 subnets with the veth pair and TAP device already created, because doing ip netns add plus ip link add plus iptables rules cold costs about 100ms and doing it warm costs about 9ms.

Cross-host, the destination agent has none of that. It has to pull vm.mem, vm.state and the rootfs out of GCS before it can start. Concurrent forks of the same parent share one download under a per-snapshot mutex, so a 10-way fan-out downloads once, but the first child still pays full freight.

One number that gets left out of fork benchmarks, including mine: the parent snapshot upload. A fork-tree call snapshots the parent synchronously, and when a snapshot bucket is configured the agent mirrors that snapshot to object storage before returning — currently around 24-27 seconds. That got slower on purpose: the snapshot used to be memory-only, and it now reflinks the live rootfs inside the same pause window and ships it as a compressed sparse tarball, so a restore is a true point-in-time copy of memory and disk instead of memory over a fresh template disk. The children themselves restore in parallel in a few hundred ms each; the mirror dominates the wall clock of the call. "Five branches in 500ms" is only true if you already had the snapshot.

A single production measurement from that smoke test, for calibration: a same-host plain fork returned in 552ms.

The application: branch-and-explore

Here is the payoff. An agent with a failing test generates three candidate patches. Instead of picking one and hoping, it forks the environment three ways, applies one patch per branch, runs the test in each, and promotes whichever branch goes green.

from pandastack import Candidate

sb.filesystem.write("/work/app.py", buggy_code)

def patch(label, code):
    def body(branch):
        branch.filesystem.write("/work/app.py", code)
        return branch.exec("cd /work && pytest -q", timeout_seconds=60)
    return Candidate.call(label, body)

result = sb.explore(
    [patch("guard-zero", fix_a), patch("try-except", fix_b), patch("or-default", fix_c)],
    score_fn=lambda branch, o: 1.0 if (o.value and o.value.exit_code == 0) else 0.0,
)
winner = result.winner   # promoted to a standalone sandbox; losers reaped

That single call issues one fork-tree, runs each candidate in its own branch with bounded concurrency, scores the outcomes, promotes the best branch to an independent sandbox and deletes the siblings. The full workflow is in the branch-and-explore docs.

Three design constraints made this worth building as a platform primitive rather than SDK glue:

Losers must always die. A fork costs real RAM on a real host. If the orchestrating process crashes mid-experiment, you have orphaned VMs burning capacity. The children carry a TTL stamp so the server reaps them without the client. Fan-out is also the endpoint most likely to be missing an admission check, which is exactly the bug I found in my own quota middleware — that story is in the layers isolation never covered.

The outcome has to be agent-readable and small. An LLM cannot diff two multi-gigabyte root filesystems. Each branch reports exit code, stdout and stderr tails, files changed, and duration. That's what fits in a context window.

Fan-out is bounded by host RAM, not by cleverness. The API caps a call at 16 children, but the real ceiling is memory. My base template bakes at 4 GiB and 8 vCPU; code-interpreter and agent at 2 GiB. Memory CoW means N children cost far less than N times the baked size, but "far less" is not "free", and it only holds on one host. Spread the fan-out across hosts and you have traded 400ms forks for 1.2-3.5s ones.

Where this breaks down

Unflushed writes vanish. This is the one that cost me a corrected doc. A plain fork copies the block device; it does not flush the guest's page cache first. I wrote a file to /tmp in the parent without syncing and the child saw nothing. The same file written to /root followed by sync came through perfectly. If you are going to fan out from a prepared parent, sync before you fork, and prefer fork-tree when the parent has meaningful in-memory state.

The clock stops. A restored guest resumes believing it is whatever time the snapshot was taken. I found this the way everyone finds it: TLS handshakes started failing inside restored VMs because certificates looked not-yet-valid. The fix was explicit clock re-sync on restore, resume and wake. If you are building your own snapshot layer, assume every wall-clock assumption in the guest is wrong at restore.

Entropy and identity are shared. The kernel's CRNG re-mixes on restore, but a userspace process that seeded its own PRNG before the snapshot carries that seed into every child. Ten branches, ten identical "random" sequences. Anything that derives an ID, a nonce or a jitter value from a pre-snapshot seed will collide across branches. Re-seed after fork if it matters.

In-flight connections die. Children get a fresh IP and MAC, and open TCP connections do not survive snapshot and restore. For an idle agent sandbox that is a non-issue by definition. For a branch that had a long-lived websocket or a held database connection, the connection is gone the moment it comes back up.

State outside the VM does not fork. This is the real limit, and it is not a Firecracker problem. Forking the VM forks the disk and the memory. It does not fork your Postgres, your S3 bucket, your git remote, your Stripe account, or the third-party API you just POSTed to. Five branches running the same migration against one shared database will corrupt each other, and no amount of CoW saves you. The only honest answers are to make branches touch nothing shared, or to branch the shared thing too — which is why my managed Postgres has its own branch, clone and point-in-time restore operations. Branch-and-explore is safe exactly to the extent that your agent's side effects live inside the VM boundary.

Which mode to reach for

Prepared-disk fan-out — install dependencies and clone the repo once, then spawn workers that already have everything on disk: plain fork. You pay a ~3s cold boot per child and you don't care, because you saved a five-minute npm install per worker.

Best-of-N over a live agent session — a warm interpreter, a loaded dataset, an in-progress session you want to branch from: fork-tree. That's the mode where processes survive, and it's the one explore() is built on.

Both CoW layers ride on the same machinery as an ordinary create, which I broke down stage by stage in every millisecond of my 179ms boot path — the reflink and the MAP_PRIVATE mmap are line items there too.

Related reading

• What it took to build a Firecracker microVM cloud solo
• Four postmortems from running this fleet in production
• What scale to zero actually cost me to build

Who I am

I'm Ajay Kumar, an infrastructure and DevOps engineer with 14 years of experience. I built and operate PandaStack, an Apache-2.0 Firecracker microVM cloud with sub-second sandbox boots, snapshot and fork, managed Postgres, git-driven app hosting and serverless functions. Around 400 Go files and 300+ organizations signed up, built solo. Every number in this post came off my own production fleet, and the mistakes came off it too. You can find me on LinkedIn or GitHub.