Cloud Outages 2025-2026: What 7 Official Postmortems Actually Teach

A year ago this week, at 11:48 PM Pacific on 19 October 2025, an empty DNS record for one DynamoDB endpoint started a fourteen-and-a-half-hour AWS outage, and a lot of teams discovered dependencies on us-east-1 that had never appeared in any architecture diagram. Since then Azure, Cloudflare (three times) and Google have published their own long postmortems. This post reads seven of those cloud outage postmortems side by side: what actually broke, why it spread, how long it lasted, and which resilience patterns (static stability, cells, dependency mapping) would have changed the outcome for the people downstream. Everything here comes from the providers' own write-ups, linked inline.

The seven outages at a glance

Here is the list, with durations taken from each provider's own timeline. Where a postmortem gives two numbers, I say which one I used.

Incident	Date	Customer impact	Trigger	Why it spread	Source
Google Cloud API outage	12 Jun 2025	~3 h for most products, last service 18:18 PDT	Policy with blank fields written to Spanner	Quota metadata replicates globally in seconds; new code path had no feature flag	Google incident report
AWS us-east-1	19 to 20 Oct 2025	14 h 32 min overall, DynamoDB 2 h 52 min	Race between two DNS Enactors left an empty record	EC2 lease manager and NLB health checks depended on DynamoDB	AWS summary
Azure Front Door	29 to 30 Oct 2025	15:41 to 00:05 UTC, about 8 h 24 min	Valid customer config across two control plane versions	Crash surfaced after 5 minutes, so the bad config reached the last-known-good snapshot	Azure PIR YKYN-BWZ
Cloudflare global	18 Nov 2025	11:28 to 17:06 UTC, core traffic fixed 14:30	Database permissions change doubled a feature file	File pushed network-wide every five minutes; proxy hard-limited at 200 features	Cloudflare
Cloudflare global	5 Dec 2025	25 min, ~28% of HTTP traffic	Killswitch flipped through global config system	Instant fleet-wide propagation hit a years-old Lua bug in FL1	Cloudflare
Cloudflare BYOIP	20 Feb 2026	17:56 to 23:03 UTC by the timeline	Cleanup job sent an empty pending_delete filter	API read empty as "all"; ~1,100 of 4,306 BYOIP prefixes withdrawn	Cloudflare
Azure West US	23 Jul 2026	14:44 to 19:41 UTC, about 4 h 57 min	Blast-radius tool widened a one-device repair to a whole datacenter's WAN links	Automated rollback depended on the connectivity it had just cut	Azure PIR ZJV6-SGG

Two caveats. Cloudflare's February post states a total of 6 hours 7 minutes in its introduction, but its own timeline runs from 17:56 to 23:03, about five hours, and I used the timeline. Google's preliminary report gave a three-hour window for the main event; Vertex AI Online Prediction was the last product to recover, at 18:18 PDT.

What caused the AWS us-east-1 outage in October 2025?

A race condition in the automation that manages DynamoDB's DNS. AWS's post-event summary describes a DNS Planner that builds plans of load balancers and weights, and a DNS Enactor, running in three Availability Zones, that applies them to Route 53. One Enactor was badly delayed. Meanwhile a second Enactor applied a newer plan and ran cleanup on old ones. The delayed Enactor then wrote its stale plan over the new one (its freshness check was long out of date), and cleanup deleted that plan too, leaving dynamodb.us-east-1.amazonaws.com with an empty record the automation could not repair.

DynamoDB was back by 2:40 AM. The other twelve hours came from what depended on it. EC2's DropletWorkflow Manager checks its host leases through DynamoDB; on recovery it had so many leases to re-establish that it fell into what AWS calls congestive collapse, with no established recovery procedure. Then Network Load Balancer health checks flapped against instances whose network state had not propagated, removing healthy capacity from DNS. AWS disabled automatic NLB health-check failover at 9:36 AM and only turned it back on at 2:09 PM.

The fixes AWS lists are telling: disable the Planner and Enactor worldwide until the race is fixed, add a velocity control so a single NLB cannot remove too much capacity during health-check failover, and build a test suite for the DWFM recovery path. Two of the three are about recovery, not prevention.

Azure Front Door and Cloudflare: when config is the outage

Four of the seven incidents were configuration pushes, and they rhyme.

Microsoft's final review for Azure Front Door says a sequence of valid customer changes, made across two control plane build versions, produced incompatible metadata that crashed the data plane. The crash only appeared about five minutes after deployment, so the change passed the protection system, reached most edge sites by 15:39 UTC, and was saved as the last-known-good snapshot. Rolling back meant hand-editing the snapshot. Microsoft's repair list includes decoupling config processing from traffic-serving processes, cutting propagation from 45 to about 15 minutes, and adding "micro cell" segmentation of customer configuration.

Cloudflare's 18 November outage began with a ClickHouse permissions change. A query that built the Bot Management feature file did not filter by database, so it returned duplicate columns and doubled the file. The proxy preallocates room for 200 features (production used about 60), so the oversized file caused a panic and 5xx errors wherever it landed. Because only some database nodes produced the bad file, the network flapped between broken and healthy, which looked enough like an attack to send the first investigation the wrong way.

On 5 December, while raising the WAF body buffer from 128 KB to 1 MB to protect customers from the React Server Components vulnerability CVE-2025-55182, Cloudflare disabled an internal testing rule through its global configuration system, which reaches the whole fleet in seconds. That tripped a nil-value bug in the older FL1 proxy and returned HTTP 500 for about 28 percent of HTTP traffic for 25 minutes. The newer Rust FL2 proxy was unaffected.

Cloudflare's answer, published on 19 December as Code Orange: Fail Small, is to put configuration through the same health-mediated staged rollout as code, to make modules fail to safe defaults, and to remove circular dependencies from break-glass access. Then 20 February 2026 happened anyway: a cleanup sub-task called an internal API with pending_delete and no value, the server read that as every BYOIP prefix, and about 1,100 prefixes were withdrawn. Cloudflare's remediation list for that one adds a circuit breaker for withdrawals that are too fast or too broad. It is the same lesson again, on a different surface: any system that can change everything at once eventually will.

Google Cloud and Azure West US: safety systems that failed open the wrong way

Google's 12 June report is the cleanest example of a latent bug waiting for data. A quota-policy check added to Service Control on 29 May had a red-button kill switch but no feature flag and no error handling. On 12 June a policy with blank fields was written to Spanner, replicated globally within seconds, and every regional Service Control crashed on a null pointer. SRE had the root cause in 10 minutes and the kill switch out in about 40. Then us-central1 took up to another 2 hours 40 minutes, because restarting tasks hammered the Spanner quota table without randomised exponential backoff.

Azure's 23 July 2026 West US review shows the other failure mode of safety tooling. A blast-radius analyser, the system meant to keep maintenance small, expanded a single-device optical repair to every device connecting one datacenter to the WAN, because it validated devices one at a time rather than their combined effect. Routes were withdrawn, the datacenter dropped off the WAN, and the automated rollback kept failing because it needed the WAN to run. Investigators first chased a routing signal that was misclassified as a WAN problem.

Trackers also report physical incidents this year (power and cooling among them), but without official postmortems I could verify, they are not in the table.

Static stability: the pattern behind most of the fixes

AWS has a name for the property that the downstream victims of these outages lacked. Mike Furr's Builders' Library article on static stability defines it as a design where "the overall system keeps working even when a dependency becomes impaired." The practical rule: your recovery should use things that are already running (the data plane), not things you have to create or change during the incident (the control plane).

AWS's own fault isolation whitepaper lists where this bites. The control planes for IAM, Organizations, Route 53 public DNS, CloudFront and ACM for CloudFront live in us-east-1. Its list of failover anti-patterns includes editing Route 53 record values or weights to fail over, creating IAM roles during the failover, and provisioning load balancers or databases in the DR region during the event. Every one of those is a control plane call into the region that might be the one on fire. Route 53 itself responded in November 2025 with accelerated recovery, an opt-in mode that keeps a copy of public hosted zones in us-west-2 and targets a 60-minute recovery objective for making record changes. Opt-in and 60 minutes: plan not to need it.

The cheapest dependency audit I know is to grep your runbooks and failover scripts for control plane verbs. This is a starting point, not a complete list:

#!/usr/bin/env bash
# find-control-plane-deps.sh: flag control plane calls in a failover path.
# Usage: ./find-control-plane-deps.sh runbooks/ scripts/dr/
# Tested with BSD grep 2.6.0 (macOS) and ugrep 7.8.4.
set -euo pipefail

patterns=(
  'route53 change-resource-record-sets'    # edits DNS via us-east-1 control plane
  'iam (create|put|attach|update)-'         # IAM writes go to us-east-1
  'elbv2 create-load-balancer'              # new ELB means new Route 53 records
  'rds create-db-(cluster|instance)'        # provisioning during the event
  'cloudfront update-distribution'          # CloudFront control plane, us-east-1
  's3api (create-bucket|put-bucket-)'       # several bucket APIs depend on us-east-1
  'globalaccelerator update-endpoint-group' # traffic dial edits, us-west-2 control plane
)

for p in "${patterns[@]}"; do
  grep -rEn --include='*.sh' --include='*.md' --include='*.tf' --include='*.py' "$p" "$@" \
    && echo "  ^ control plane call in recovery path: $p" || true
done

# Also check what your resolvers will actually cache during a failover.
dig +noall +answer app.example.com A   # second column is the TTL clients will honour

When I ran the dig line against dynamodb.us-east-1.amazonaws.com this morning, the record came back with a TTL of 5 seconds. AWS's endpoint was not slow to re-resolve once fixed; the slow part was every system that had lost state while it was gone.

Cell-based architecture and the blast radius question

Static stability answers "what happens when my provider breaks". Cells answer "how much breaks when I break it", which the table suggests is the more common question. AWS's whitepaper on cell-based architecture describes splitting a workload into independent copies, each serving a slice of customers, behind a thin router, so a bad deploy or a poison request takes out one cell rather than everyone.

Cloudflare's feature file and Google's blank policy both replicated everywhere within seconds; both fixes amount to making configuration respect cell boundaries the way code deploys already did. Azure's "micro cells" for Front Door are the same idea. The control I would add for any team is a propagation budget: how fast is a config change allowed to reach 100 percent of your cells, and what health signal must be green before it moves on? If the honest answer is "seconds, and none", you have built Cloudflare's 18 November in miniature. That is also where progressive delivery and feature flags stop being a product team's toy and become infrastructure.

Disclosure: I run PandaStack, a Firecracker microVM cloud, and the rule I hold myself to there is the same one: a host that is already serving sandboxes should keep serving them if the control plane disappears, and a config change reaches one cell before it reaches all of them.

Does multi-region actually protect you from a cloud outage?

Sometimes. It helped teams whose failover was already provisioned, health-checked and rehearsed, and did not route through a global service. It did not help anyone whose DR plan started with "create the stack in us-west-2", or whose identity provider, CI or status page sat in the failed region. Even the providers tripped on this: Google's first public post was delayed to 11:46 because its status tooling was affected, and Azure's Front Door review mentions customers unable to open support cases.

It also did not help against a global configuration push, because that fails every region at once. For edge and CDN providers, the realistic hedge is a second provider for the critical path, or at least a tested way to bypass the CDN, not a second region. If you want to rehearse any of this, the step-by-step multi-region failover guide covers Route 53 health checks, ARC routing controls, Cloud DNS failover policies and database promotion, and chaos engineering is how you find out whether any of it works before the provider tests it for you.

The AI angle is real but boring: more automation acting on infrastructure means more systems that can change everything at once. The AIOps incidents where the automation was the outage are the same story with a language model in the loop.

FAQ

What caused the AWS outage on 20 October 2025?

A latent race condition between two DynamoDB DNS Enactors left the regional DynamoDB endpoint in us-east-1 with an empty DNS record. DynamoDB recovered in under three hours, but EC2's host lease manager and NLB health checks, which depend on DynamoDB, took until 2:20 PM PDT to recover, about 14.5 hours in total.

Why did Cloudflare go down twice in late 2025?

On 18 November a database permissions change doubled the size of a Bot Management feature file, exceeding a hard 200-feature limit in the proxy. On 5 December a killswitch pushed through the instant global configuration system triggered an old bug in the FL1 proxy. Both were configuration changes that reached the whole network without a staged rollout.

What is static stability in cloud architecture?

Static stability means a system keeps working when a dependency fails, because recovery uses resources that are already running rather than creating or modifying them during the incident. In practice: pre-provision capacity in the failover location, fail over with health checks or data plane controls, and avoid control plane calls such as DNS record edits or IAM changes in the recovery path.

Is us-east-1 riskier than other AWS regions?

It hosts the control planes for several global services, including IAM, Route 53 public DNS and CloudFront, according to AWS's fault isolation whitepaper, so failures there can affect configuration changes everywhere. Data planes for those services are distributed, so workloads in other regions keep running if they do not need control plane changes to recover.

This quarter, if I were running a platform team, I would do three things: run the grep above against every runbook and Terraform DR module and remove each control plane call I find; put a propagation budget on configuration changes, not just code; and schedule one real failover, in business hours, with the incident commander from the incident management process running it. None of it needs a new tool. All of it needs someone to own it.

Related: microVM Production Incidents: 4 Postmortems From My Fleet, DevOps Still Matters in 2026: AI Cut Delivery Stability 7.2%, Then Doubled Merged PRs and Added 91% to Review Time and Infrastructure as Code in 2026: Two Forks at 1.16 and 1.13, a $6.4B Owner, 912 Public State Files, and One Agent That Ran terraform destroy.