It's 2 AM. Do You Know What Your AI Agent Is Doing?

I operate PandaStack, a Firecracker microVM cloud. Other people's AI agents run inside it. That gives me an unusual vantage point on a question the industry mostly answers with vibes: at 2 AM, when nobody is watching, what is your agent actually doing?

Here is the first honest answer. I don't know what commands it ran, and I can't find out.

That isn't modesty. The platform never stores exec command text — not in the analytics store, not in the audit log, not in the event metadata. I went looking specifically to write this post and confirmed it four different ways. The audit table records method, path and status across 334,793 rows with an empty metadata column. The sandbox event metadata carries seven keys, and none of them is your code.

So I cannot tell you that agents mostly run pip install. I would like to. It would make a better opening. But I would be making it up, and this whole site is built on not doing that.

What I can see is everything around your agent: which image it booted, how long it lived, how much memory it held, when it died and what killed it. And the picture that assembles from those is the actual finding, which is much stranger than the one I went looking for.

Mostly, they are asleep

Across 9,640 boots on the fleet between June and now, roughly half — 47.8% — are the code-interpreter image. Another 29.4% are the general-purpose base image. So the honest summary of what AI agents do is: they run some Python, briefly, and then they stop existing.

That is the anticlimax, and I want to sit on it for a second, because the marketing register around agents has settled on a very different image. Tireless digital workers. Grinding through the night. An always-on colleague.

The telemetry says: it ran for ninety seconds and the reaper swept it up.

Which would be a boring post, except for the part I did not expect. At 2 AM, the agents are asleep. The other robots are wide awake. And in every genuinely bad night I have had running this platform, the agent was not the problem. The automation around the agent was.

The app that could not sleep because robots kept asking if it was awake

Apps on the platform scale to zero after fifteen minutes of no traffic. A user reported that apps with a custom domain attached never slept.

I went through the domain machinery exhaustively. It was innocent. The reconciler only talks to DNS. No health check targets the app port. Attaching a domain doesn't even write to the app record.

The cause was the internet. A public custom domain attracts a steady drizzle of automated traffic that an obscure default UUID URL never receives — uptime monitors, crawlers, favicon fetches, robots.txt probes. Uptime monitors poll every 60 to 300 seconds. The idle timer is 900 seconds. Three hundred is less than nine hundred, so the timer could never elapse. Not once. Not ever.

The act of publicly announcing your app is what prevented it from resting. Give it a real domain and the internet's immune system shows up: a permanent low-grade drumbeat of machines asking are you alive? — and the honest answer becomes only because you keep asking.

The same shape had already bitten me one layer down. A dashboard tab polled sandbox status every 30 seconds against a five-minute idle timeout. Anyone who left the dashboard open kept every sandbox on the platform alive indefinitely. Watching the thing is what stopped it from sleeping.

The database whose insurance policy kept it awake

A user reported that a fresh, empty database with no connections refused to auto-suspend.

The backup system fires a full base backup when accumulated write-ahead log exceeds a ratio of the base backup size. On a large database that rule is correct — it bounds how much WAL you must replay on restore. On an empty database, the base backup is about 5 MB gzipped, so the trigger arms before a single 16 MB WAL segment even fills. And an idle Postgres still produces WAL, from checkpoints alone.

Each backup holds a replication sender open for about 100 seconds. The idle probe correctly counts an active replication sender as busy, because you must never suspend a database mid-backup.

The top three databases took 46, 27 and 24 base backups in 24 hours. The correct number is 1. Roughly 250 MB of redundant tarballs each, per day, forever. The thing keeping the database awake — and therefore billing the customer — was its own insurance policy, functioning perfectly.

The diagnosis took far longer than it should have, for a reason worth stealing: the idle sweeper logs nothing when it declines to suspend something. I had to reconstruct the whole chain from backup completion timestamps. A silent veto path is how a system with otherwise excellent logging still cannot tell you why it isn't doing the obvious thing.

Four milliseconds, then all night

The second time a persistent sandbox went through idle hibernation, it wedged.

Hibernate pauses the VM, then snapshots it. The pause call had a ten-second timeout. A busy guest on its second cycle took 10.004 seconds. On timeout the function returned early without resuming — leaving the VM permanently paused, its control socket hung, while the agent cheerfully continued reporting the sandbox as running.

Then the idle sweeper, which had no backoff, no failure cap and no memory, re-selected the same wedged sandbox every 30 seconds and tried again. Each attempt overwrote the memory snapshot while leaving it inconsistent with the older header. The VM did not just stay broken; it got monotonically worse all night.

Four milliseconds. The guest was fine. The data was fine — it sat on a durable volume the entire time. What killed it was an error path that forgot to undo the thing it had just done, plus a retry loop that could not tell the difference between try again and keep hitting it.

Every VM woke up in the past

Firecracker restores a guest with its wall clock frozen at the moment the snapshot was baked. No real-time clock, no NTP in the templates. This was latent from day one and entirely harmless, because a guest a few weeks behind still validates certificates fine.

Then GitHub rotated a leaf certificate.

Seeds baked June 22. New certificate valid from July 3. Failures started July 8. Every restored VM in the fleet was now living before that certificate's validity window began, so each one correctly refused it as not-yet-valid. Every git clone in every app deploy, on every template, on every restore path, died at the same moment. Nobody had deployed anything. Nothing on my side had changed.

The error was server certificate verification failed. CAfile: none, which sent me straight at the CA bundle — present and intact in the seed, in the rootfs, and in the live guest. The CA was a red herring. The machines were not misconfigured. They were time travelers, and the internet had moved on without them.

Every old snapshot on a platform like this is a small time bomb whose fuse is lit by somebody else's routine maintenance — an event you don't control, can't schedule, and won't be told about.

The night three correct systems produced an outage

This is the one that changed how I think about the question in the title.

A cloud maintenance event forced a restart on the host carrying every managed database. The instance group's autohealer did exactly what autohealers do: it rebuilt the machine from its instance template. That template was five weeks old, and its provisioning script predated a config knob that enables database idle-suspend.

Nothing alerted, because nothing was broken. The host had simply, silently, lost the ability to put idle databases to sleep. So every database that woke stayed awake forever. Committed memory accumulated — nine databases, 22.5 GiB — until the admission ledger was full, at which point the host began refusing every new workload.

App deploys started failing platform-wide. And because the git auto-deploy path had no backoff, they retried. 9,230 failed deploys in 24 hours. One single app accounted for 6,483 of them.

Not one component malfunctioned. The maintenance handler did its job. The autohealer did its job — it rebuilt a broken machine flawlessly, from a faithful snapshot of a configuration that had been fixed weeks earlier. The retrier did its job, 6,483 times, into a wall.

The fix was one line of config moved out of first-boot provisioning into the continuously-managed set. Committed memory went from 30,720 MB to 0.

Config drift is not a thing that happens to you once. It is a thing a self-healing system will faithfully restore for you, forever, at 2 AM, from a template nobody remembers writing.

The 22 that nobody visited

The same maintenance event had an earlier consequence I did not find for three days.

The rebuilt instance came up with a fresh boot disk. The hibernation snapshots lived on that boot disk; the durable customer volumes lived on a separate persistent disk. So the memory snapshots evaporated and the data survived. 22 managed databases across 19 organisations flipped from hibernated to failed.

Silently. Zero events emitted. Zero audit rows. No alert. The backup-health checker reported OK for all 22 the entire time — because it checks whether backups exist, not whether the database is alive. I found it three days later during an unrelated scan.

The scariest number in that paragraph is not 22. It is zero. Every one of those status flips went through an error branch that marked the row failed and then returned without telling anyone. The system knew. It simply never said so.

And the ones that recovered, recovered by accident. Any customer who happened to come back and poke their database triggered a fallback that quietly rebuilt it from the durable volume in 113 milliseconds. The ones that stayed dead were the ones nobody visited.

Twenty-seven minutes

Sometimes the answer to what is running in your infrastructure at 2 AM is exactly what you'd fear.

A free-tier account signed up at 11:30:45 UTC and had a cryptocurrency miner running before 11:57. Twenty-seven minutes. My cloud provider's abuse detector noticed the outbound mining protocol before I did.

Two things about that still bother me, and neither is the miner.

The first: the thing that stopped it was not a security control. It was the idle timeout. A janitor swept the sandbox out at 12:02:36 without ever knowing what it was, coincidentally right as the mining stopped.

The second is the arithmetic. A later incident burned 3,273 CPU-seconds of mining. Total cost to the platform: six cents. Run that against a free-tier credit and it funds years of continuous mining. Egress blocklists and domain denylists are speed bumps. The only durable lever is making abuse cost more than it earns.

And the signal everyone reaches for is wrong. CPU saturation did not identify the miner. In one investigation the highest-CPU tenant on the box turned out to be a completely legitimate user whose npm builds burned more cores than the actual attacker.

The part where I made it worse

A managed database showed as hibernated but refused to start, reporting that it was already running. It was half-awake: an automatic wake had restored the hypervisor process, but the guest OS never came back. Because the driver handle was registered, the platform believed it was running and rejected every recovery path.

At which point I — the person who wrote all of this — deleted the sandbox, on the confident belief that managed deletes preserve the durable volume.

They do not. 233 MB of PGDATA, gone.

The data survived anyway, entirely because of continuous WAL archiving that had been quietly running in the background the whole time, unasked and untested. And then the restore failed too, for an unrelated reason: it staged the rebuilt volume in a temp directory on one filesystem and renamed it onto a different mount.

Three things had to go right and only one of them was designed to. The rule I now operate by is blunt: never delete a managed resource in order to recover it. And the more general one — an untested restore path is not a backup. It's a hope.

So: what is your agent doing at 2 AM?

Almost certainly nothing. It ran some Python and stopped existing.

The question is better aimed elsewhere. At 2 AM your agent is asleep and the rest of it is wide awake — the uptime monitor knocking every sixty seconds, the backup job that fires 46 times a day, the sweeper retrying a wedged VM every thirty seconds until dawn, the autohealer rebuilding a machine from a five-week-old template, the janitor that cannot distinguish doesn't exist here from doesn't exist.

None of those is an AI. Every one of them is a small, correct, unsupervised program doing precisely what it was told. The interesting failures on this platform have never been a model doing something surprising. They have been four or five boring automations composing into a behaviour that nobody designed, chose, or could see — while the health check stayed green.

We spend a lot of energy asking whether autonomous agents will do something we didn't intend. Worth remembering that we have been shipping autonomous systems for decades, they already do that constantly, and mostly we call it Tuesday.

If you want the longer versions, I've written up four production microVM postmortems and why isolation is not an abuse control.