Isolation Is Not an Abuse Control: Lessons From My Fleet

Sep 2, 2026 · 9 min · Ajay Kumar

I run a platform whose entire product is executing code nobody reviewed. An LLM writes a Python script, an API call ships it to my infrastructure, and it runs as root a few hundred milliseconds later. That is not a failure mode I am defending against. That is the feature.

In June, an account that was 27 minutes old used that feature to mine cryptocurrency on one of my hosts. My isolation boundary held perfectly throughout, and it made no difference whatsoever. That gap — between a boundary that works and a platform that is safe to operate — is what this post is about.

The isolation comparison comes first, because you need it to see why it was never the control that was going to save me. Then the incident, then the four layers I should have built alongside it.

Who I am: I'm Ajay Kumar, an infrastructure engineer with 14 years in DevOps. I built PandaStack, an open-source (Apache-2.0) Firecracker microVM cloud, solo over about six months. 300+ organizations have signed up. Everything below is from running it, not from reading threat models. The build story is in what it took to run a Firecracker cloud solo.

Two threat models that get conflated

When people say "untrusted code" they usually mean one of two very different things, and the confusion causes bad architecture decisions.

Model 1: unreviewed code from a cooperative user. An agent generates a pandas script. The user is not attacking me. But the code was never read by a human, might rm -rf something, might pull a compromised package, might loop forever. The risk is accident and supply chain.

Model 2: a hostile tenant who signed up specifically to abuse you. They read your docs to find the biggest template. They are actively probing your quota math. This is the one people underestimate, and it is the one I have actually dealt with.

Isolation addresses model 1 well. It barely touches model 2, for reasons I'll get to.

The three isolation models, fairly

Containers

Namespaces, cgroups, seccomp, and a shared host kernel. Every syscall the untrusted code makes is serviced by the same kernel serving every other tenant on the box.

This is not a knock on containers. For your own code, your own CI, your own services, containers are the right answer and I use them daily. Packaging, scheduling, and density are all excellent, and startup is measured in tens of milliseconds.

The problem is scope. The attack surface is the full Linux syscall interface, roughly 350 syscalls plus their combinatorial argument space, plus ioctls. seccomp profiles help, but a profile tight enough to be meaningful tends to break real workloads, and "run arbitrary code the model wrote" is about as broad a workload as exists. One kernel LPE bug reaches every tenant on the host.

Containers are a strong isolation boundary between cooperating workloads. For arbitrary untrusted code I don't think they're a security boundary on their own, and Google, AWS, and Cloudflare all independently concluded the same thing and shipped something else.

gVisor

gVisor puts a kernel written in Go between the workload and the host. It intercepts syscalls and services most of them itself, only reaching the host kernel through a much narrower filtered path.

This is a real improvement and I want to be clear about that. If you're already running Kubernetes, gVisor is close to a drop-in via RuntimeClass, which is a genuinely underrated property. You get a large reduction in host syscall surface without rebuilding your platform.

The costs are real too. Syscall compatibility is good but not complete, and the gaps show up in hard-to-predict places, especially around less common filesystem and networking behavior. I/O-heavy work pays a measurable tax because the interception sits in the data path.

And it's a different bet, not a strictly better one. You have replaced "trust the whole Linux syscall interface" with "trust a large body of new Go code that reimplements most of it, plus a narrowed host path." You are betting that a fresh, memory-safe, deliberately-narrowed kernel has fewer exploitable bugs than a 30-year-old one with vastly more eyes on it. I think that bet is usually right. It is still a bet.

KVM microVMs

Each workload gets its own Linux kernel running as a hardware-virtualized guest. The host-facing surface is KVM plus whatever the VMM exposes. Firecracker (which I use, v1.16, unmodified) is roughly 50k lines of Rust with no BIOS, no PCI passthrough, and no legacy device emulation. The guest talks to a handful of virtio devices and nothing else.

A tenant getting root in their guest is the normal supported state on my platform. To reach another tenant they need a guest-to-host escape through KVM or through that small virtio device model, not a container breakout.

Here's what I owe you in the other direction. Virtualization is not magic. The host kernel and KVM still have to be correct, and I track Firecracker and kernel updates because of it. The plumbing around the VM (tap devices, block devices, the snapshot path) is host-side code I wrote, and that code is in the trust boundary. CPU side channels get mitigated at the host level; hardware virtualization does not make them go away. The honest claim is a much smaller and much better-audited attack surface, not an absolute one.

	Containers	gVisor	KVM microVM
Kernel	Shared with all tenants	Shared, behind a user-space kernel	Own kernel per workload
Host attack surface	Full syscall interface	Narrowed syscall path	KVM + small virtio device model
Startup	~10-50ms	~100-200ms	~180ms warm, ~3s cold
Syscall compatibility	Complete	Good, with gaps	Complete
I/O overhead	None	Measurable	Small (virtio)
Best fit	Your own code	Untrusted code on existing k8s	Hostile multi-tenant, root in guest

What the microVM choice actually cost me

The classic objection is boot time and density, so here are my measured numbers.

Sandbox create is 179ms p50, 203ms p99. That is the full API call: allocate a pre-built network slot, reflink the rootfs, exec Firecracker, load the snapshot, resume, probe the guest's SSH port, return. The snapshot load itself is about 80ms p50.

That number only exists because there is no cold boot in the path. Every create restores a pre-baked Firecracker snapshot rather than booting a kernel. Cold boot, which happens on the first spawn of a template before its snapshot is baked, is around 3 seconds. Users essentially never see it. I wrote up the mechanism in the snapshot-restore internals doc, and the per-stage timings in my measurement of the 179ms boot path.

Density: the hard ceiling is 16,384 VMs per host, which comes from carving a /16 into pre-allocated /30 subnets, not from anything about virtualization. The real constraint is RAM. My templates bake 4 GiB for the apps base image, 2 GiB for code-interpreter and agent, 1 GiB for Postgres. Memory is the number that runs out.

So the tradeoff people warn about is mostly payable. What is not payable is the engineering time: the snapshot path, the demand-paged memory streaming from object storage, the pre-allocated network slot pool. That is the real cost, and it's why "just use gVisor" is often the correct answer for a team that isn't building a sandbox product.

The full boundary description is in the isolation model docs.

Now the part nobody writes about

In June, GCP's abuse detection flagged crypto-mining egress from one of my agent hosts. Stratum protocol, port 3333, five minutes of traffic.

Here is what happened. A free-tier account signed up at 11:30 UTC. By 11:57 they had a miner running. Twenty-seven minutes. They created a sandbox and ran the miner as three concurrent long-lived streaming exec calls, 432, 421, and 336 seconds. The idle reaper killed the sandbox at 12:02, which is roughly when the mining stopped.

The isolation worked perfectly. Nothing escaped. No tenant saw another tenant. The guest kernel held, KVM held, the cross-tenant firewall drop held. And it did not matter even slightly, because the attacker never needed an escape. They had legitimate compute. I gave it to them. That is the product.

This is the thing I wish someone had told me before I built a sandbox platform: isolation is a containment boundary, not an abuse control. They solve different problems. A perfectly isolated miner is still a miner.

The fix was egress, not isolation

The structural fix was a protocol-level block: a pool-wide DROP for outbound TCP to the standard Stratum control ports, inserted at the top of the host FORWARD chain and keyed on the whole VM pool CIDR rather than per-sandbox.

# shape of the rule (top of chain, before any ACCEPT)
iptables -I FORWARD 1 -s 10.200.0.0/16 -d 10.200.0.0/16 -j DROP       # no VM-to-VM
iptables -I FORWARD 1 -s 10.200.0.0/16 -d 169.254.0.0/16 -j DROP      # no cloud metadata
iptables -I FORWARD 1 -s 10.200.0.0/16 -o eth0 -p tcp --dport 3333 -j DROP

Two details mattered more than the rule itself.

Pool-wide, not per-sandbox. Because the drop is keyed on the pool CIDR, the first VM created after the deploy covers every current and future sandbox. No rebuild of the pre-allocated network slots, no rolling restart.

Ordering is pinned by a test. A drop rule that lands below an accept rule is decoration. The insert position is asserted in unit tests, because a refactor that reorders chain setup is exactly the kind of change that looks harmless in review.

The link-local drop in that same block is worth calling out separately: it kills the SSRF-to-cloud-credentials path by construction. A guest cannot query the metadata endpoint, so it has no ambient cloud identity to steal. Details are in the egress controls docs and the networking internals.

And then the denylist got evaded, twice

I am not going to pretend the port block was a solution.

Two months later, same abuse class, completely different shape. A serverless function whose handler was one line: curl an xmrig-family binary from an attacker-controlled domain over HTTPS, untar, execute. Port 443. My port denylist correctly caught nothing, because there was nothing on those ports to catch.

Two days later, a third case had zero outbound connections in any network namespace while pegging two 8-core hosts. Load average 16.49 on one of them.

A port denylist stops the default configuration of off-the-shelf miners. That is worth something. It is not a control against anyone who reads their own config file.

The layers that actually did the work

If I were starting over, this is the list I would build alongside isolation rather than after it.

Per-VM egress attribution. Every VM NATs through its own namespace, so every outbound connection maps to a VM, which maps to an org. When the abuse report arrived I could answer "which tenant" in minutes instead of guessing. This is the single highest-value thing on this list and it costs almost nothing to build early. Retrofitting attribution into a running platform is miserable.

Quota admission, and get the arithmetic right. Mine was wrong. The check compared current usage against the cap without counting the incoming workload:

if totalCPU >= maxCPU || totalMem >= maxMem { refuse }

A 10 GiB cap therefore admits three 4 GiB guests: at the third create, current usage is 8192, which is less than 10240, so it passes. A cap of C really means C + (largest single workload) - 1. That off-by-one is how a brand-new free account ended up holding 12 GiB and 24 vCPU. Separately, my quota middleware matched only POST /sandboxes, so the fork endpoint cloned workloads with no quota check at all. Both bugs were in the direction that costs money.

Abuse detection that doesn't use CPU as the signal. This surprised me most. In one investigation the highest CPU consumer on the fleet was a legitimate user running npm builds and a Node deploy at 4.91 cores. The miner was further down the list. CPU magnitude cannot distinguish xmrig from next build; both saturate every core you give them. The signal that separates them is duration, not magnitude, and even that misfires on a 40-minute training job. Alert, don't auto-kill. A false positive that silently kills a customer's long job is worse than a miner you catch twenty minutes later.

The economics. One miner burned 3,273 CPU-seconds and was billed $0.06. If free-tier credit funds years of mining, no detector wins the arms race. Tightening free-tier quota and requiring billability did more than any signature ever did.

Detection, which I had almost none of. Every layer above is prevention. When 22 managed databases died on one host I did not find out from an alert, I found out from a query I happened to type three days later — that one is postmortem four from my fleet, and it is the cheapest lesson on this page to steal.

Authorization, one layer above isolation. My worst finding had nothing to do with kernels. Managed database VMs exposed their underlying sandbox ID in the API response, and the exec endpoint had no guard for platform-managed sandboxes. A customer could get a root shell inside their own database VM. Delete was already guarded; exec, fs, pty, and fork were not, and that asymmetry is what told me it was an oversight rather than a design. Isolation was intact the entire time. The boundary that failed was authorization.

What I'd actually recommend

Running your own code, your own CI, first-party services: containers. Don't overthink it.

Running untrusted code where you can absorb some syscall compatibility risk, especially if you're already on Kubernetes: gVisor is a genuinely good answer and the integration cost is low.

Running arbitrary code with arbitrary syscalls, root in the guest, hostile signups, multi-tenant: microVMs. And then budget at least as much engineering for egress policy, quota admission, attribution, and abuse response as you spend on the isolation itself. In my case it has been more.

Isolation is necessary. It has never once been sufficient.

PandaStack is Apache-2.0, so the isolation and egress paths described here are public and auditable. If you're building something similar and want a second pair of eyes on the abuse surface, get in touch.

Keep reading

Sep 2, 2026