Sandbox Creation Time Benchmarks Measure Different Things

There is a June 2026 analyst report comparing 19 AI-agent sandbox platforms. Like every comparison of this kind, it has a creation time column. The values in that column run from 0.79ms to about 2.7 seconds.

That is a spread of roughly 3,400x. No two production systems doing the same work differ by 3,400x. What differs is what each vendor decided to put a stopwatch around.

I build one of these systems. PandaStack is an Apache-2.0 Firecracker microVM cloud I wrote solo over about six months, and its published creation number is 179ms at p50. I have spent enough time arguing with my own instrumentation to know that my 179 and someone else's 0.79 are not two points on one axis. They are two different measurements wearing the same label.

This post is not an accusation. I have no visibility into anyone else's internals and I am not going to speculate about them. The problem is not the vendors; it is that "creation time" is an under-specified metric, and an under-specified metric put in a table acquires an authority it has not earned.

I am Ajay Kumar, an infrastructure engineer with 14 years on this kind of plumbing. I am not writing this as a surveyed vendor — none of my numbers come out of anyone's matrix. Every PandaStack figure below came off my own production fleet. Every industry figure is passed on as reported by others.

At least five different clocks

When a vendor says "creation time," they could be timing any of these. All five are legitimate things to measure. None of them are each other.

(a) Cold boot of a fresh VM. Kernel starts, init runs, userspace comes up. This is the honest worst case and nobody puts it in a marketing table. Mine is about 3 seconds, and it happens exactly once per template per host — the first spawn cold-boots, captures a snapshot, and every subsequent create takes the fast path.

(b) Restore from a memory snapshot. The guest kernel already booted, somewhere, at some earlier time. You are reloading device state and mapping guest memory back in. This is where the interesting engineering is, and it is where most serious platforms now compete.

(c) Checkout of an already-running instance from a warm pool. The machine existed before your request arrived. Someone paid to keep it alive. The measured "creation" is a bookkeeping operation on a pre-existing resource, and its real cost was paid earlier by idle capacity you are not being shown.

(d) API returns an identifier. The control plane wrote a row and handed you a UUID. Whether anything is running behind that UUID is a separate question with a separate latency.

(e) Time until the guest accepts a command. You can connect, and something in the guest answers. This is the only one that maps to what a developer actually feels.

A table that mixes (c) and (e) in one column is comparing a coat-check ticket to a cooked meal.

What my 179ms actually is

Precision beats advocacy here, so here is my definition in full.

179ms p50, 203ms p99 is wall time from the moment the per-host agent accepts a create request to the moment the guest answers a TCP connection on port 22. It is measured as quantiles over a boot event the agent writes to ClickHouse on every create in production, not a stopwatch on a demo. There is no warm pool anywhere in the system — every create restores a snapshot, which is a design commitment I wrote about in detail in the full stage-by-stage anatomy of that boot path.

The stages, as I read them off the fleet:

Stage	Cost	What it is
Claim a network slot	~1ms	Pop a pre-built netns/veth/tap off a free list
Configure tap in netns	~6ms	Patch the MAC to match the baked guest identity
Reflink the rootfs	~4ms	XFS copy-on-write clone of the template disk
fork+exec firecracker	~25ms	Spawn the VMM, wait for its API socket
PUT /snapshot/load	~80ms	Load device state, map guest memory
PATCH /vm → Resumed	~6ms	Unpause the vCPUs
Probe TCP :22	~40ms	Wait for the guest to answer
Insert the sandbox row	~6ms	Async, off the critical path

What that number includes: VMM process spawn, memory restore, network attach, and a readiness probe against the guest.

What it excludes, and I want this on the record:

• The ~3s cold boot on the very first spawn of a template on a host. Different population, different number, reported separately.
• Control-plane time before the agent sees the request — auth, scheduling, HTTP round trips from wherever you are.
• The ~500ms fallback when the pre-built network slot pool drains. I keep a shallow warm depth of four slots per template; when it empties, the agent builds a slot on demand rather than returning a 503. Building one cold costs about 100ms of namespace and interface work plus the rest of the setup. That path is real, it is in my p99 tail, and it is why I quote p99 at all.
• Fork, which is a different operation entirely: 400-750ms same-host, 1.2-3.5s cross-host.
• Managed database creation, which is 30-90 seconds because PostgreSQL has to actually come up and be ready.

Four bullets of exclusions for one number. That is roughly the disclosure surface any honest creation-time figure needs, and roughly none of them get it in a comparison matrix.

Why sub-millisecond means the clock started somewhere else

A figure like 0.79ms deserves a careful reading rather than a dismissive one.

You cannot restore a Linux guest's memory state, attach it to a network, resume its vCPUs, and have it answer a connection in under a millisecond. My memory restore alone is 80ms, and that is a well-optimized path on a system I have spent months tuning. Loading device state and faulting in guest pages has a floor set by physics and page tables, not by cleverness.

So a sub-millisecond number is almost certainly measuring something other than a usable Linux guest. The plausible candidates are all reasonable engineering:

• A V8 isolate, which is not a Linux guest at all. The report I mentioned lists isolation models spanning Firecracker, gVisor, Kata, Docker, V8 isolates, and plain namespaces. A V8 isolate genuinely can be created in microseconds. It is also a fundamentally different security and compatibility surface — you get JavaScript, not a filesystem and a process tree.
• Handle allocation, where the returned object is a lazily-materialized reference and the real work happens on first use.
• Checkout from a pool where the expensive part happened before your clock started.

None of those are dishonest. They are all answers to the question "how long does creation take" under a defensible definition of creation. They just cannot be subtracted from my 179 to yield a meaningful difference, any more than you can subtract a shoe size from a temperature.

The same applies in the other direction. Somebody reporting 2.7 seconds may be reporting a genuine cold boot with a container image pull, which is a far more conservative and far more honest measurement than a warm-pool checkout reported as a single-digit millisecond figure. The column punishes them for it. That is the part that actually bothers me: the metric as constructed rewards moving the start line.

Questions that make the column comparable

If you are evaluating platforms, these six questions convert a marketing number into an engineering one. They are also fair to ask — every vendor with a good answer will enjoy giving it.

1. What event starts the clock, and what event stops it? The single highest-value question. "Request received at the host agent" to "guest answers TCP" is a different span from "API handler entered" to "row committed."

2. Is the guest reachable at t=stop? If I can't run a command at the moment the clock stops, the clock stopped early. Ask them to demonstrate the stop condition, not describe it.

3. Cold or warm — and what fraction of real requests take which path? A warm-pool number is fine if the pool is deep enough that you always hit it. Ask what happens at the moment the pool is empty, because that is the number your users will feel during a traffic spike.

4. p50, p99, or best-of-N? Best-of-20 on an idle rig is a legitimate engineering datapoint and an illegitimate SLA. My p50 is 179 and my p99 is 203; the gap between those two numbers tells you more about a system than either one alone. A vendor who cannot quote a p99 is probably not measuring in production.

5. Does it include network attach? Networking is where I found the largest avoidable cost in my own system. Creating a namespace, veth pair, tap device, and firewall rules cold runs about 100ms, which is more than half my total budget. I pre-build them for exactly that reason. A creation number that starts after the network is attached is hiding my single biggest optimization.

6. What is excluded? Ask directly. Image pull, scheduling, queueing, first-spawn bake, volume attach, and control-plane round trips are all commonly and reasonably outside the measured span. You need the list.

If I could redesign the column, I would replace one number with three fields: the start event, the stop event, and the population it was sampled over. That is unglamorous, and it would make the table honest overnight.

Being fair about what the reports get right

The June 2026 report is doing something genuinely useful. Surveying 19 platforms across a category that did not exist in this form two years ago is real work, and its structural observations look right to me from the inside.

The category-level facts are worth taking seriously, and I pass them on as industry reporting rather than as anything I verified myself. As reported: every hyperscaler entered the category in 2026 — Vercel Sandbox reached GA, Cloudflare shipped a Sandbox SDK, AWS has AgentCore, Google has Agent Sandbox, NVIDIA has OpenShell. Modal is reported to have raised $355M at a $4.65B valuation, and Daytona a $24M Series A. E2B publishes a figure of more than a billion sandboxes started, with pause/resume that preserves memory state. Two long-tail projects in the survey are described as dormant. None of those are my measurements, and none of them depend on the creation-time column being coherent.

Its prediction that checkpoint/restore becomes table stakes by 2027 matches what I see. Persistence has already converged — everyone has some form of it. Snapshot-restore is where the remaining differentiation lives, which is exactly why the measurement question is about to get more urgent rather than less.

Here is the uncomfortable part: convergence will make the column look more comparable while it is still broken. Once most vendors report a restore time rather than a cold boot, the numbers will cluster in the same order of magnitude and readers will start diffing them. But a restore that stops the clock at "vCPUs resumed" and a restore that stops at "guest answered" still differ by the readiness probe — 40ms in my case, which is 22% of my total. Two systems with identical real performance can report 139 and 179 depending on one line of instrumentation code. Clustered numbers with unclustered definitions are more dangerous than an obviously absurd 3,400x spread, because the absurd spread at least warns you.

What I would actually compare instead

If creation time is not reliable, what is? The things I would rank a platform on, in order:

The shape of the tail. p99 divided by p50. Mine is 1.13. A ratio near 1 means the fast path is the only path. A large ratio means there is a slow path you will meet under load, and you should find out what it is.

Behavior when capacity runs out. My network slot pool falls back to on-demand construction and takes about 500ms instead of failing. That was a deliberate choice: degraded is better than refused. Ask what a platform does at its own edge. The answer tells you more about operational maturity than any latency figure.

What the isolation boundary actually is. Firecracker, gVisor, Kata, containers, and V8 isolates are not interchangeable, and the choice constrains everything downstream — what code you can run, what a kernel bug costs you, whether snapshot-restore is even available. I picked hardware virtualization for running untrusted AI-generated code, and I pay for it in boot milliseconds I would not pay with a process sandbox.

Idle cost. A platform with a warm pool has someone paying for idle machines, and it is either the vendor's margin or your bill. My whole snapshot-restore design exists so that idle costs approximately nothing, which is why I can restore on every create rather than keeping anything warm.

Whether the vendor will show you their instrumentation. Not a benchmark — the actual definition, the actual query, the actual population.

The creation-time column is not going away. Tables need numbers and readers want one. But it should be read the way you read a fuel-economy sticker: a real measurement, taken under conditions specified elsewhere, that will not predict your commute.

Ask what the clock starts on. Ask whether the guest answers when it stops. Everything else is downstream of those two questions, and both of them are usually a single email away.

Related reading

• Anatomy of a 179ms microVM boot
• Checkpoint restore: what it cost me to build
• Container vs microVM vs gVisor

---

I'm Ajay Kumar, an infrastructure and DevOps engineer with 14 years of experience, and the creator of PandaStack, Riff, and PandaFlow. PandaStack is an Apache-2.0 Firecracker microVM cloud I built solo. I'm on LinkedIn and take on infrastructure consulting; the code lives at github.com/pandastack-io.