How to Build Multi-Region Failover with Route 53 and Cloud DNS (2026)

This guide is for engineers who own a service on AWS or Google Cloud and want it to survive the loss of a region without a heroic night. By the end you will have DNS failover driven by health checks (Route 53 or Cloud DNS), a manual data plane override with ARC routing controls, a replicated database you know how to promote, and a test procedure you can run in business hours. Commands and config were checked against current AWS and Google Cloud documentation in October 2026; the Terraform was validated with the hashicorp/aws provider 6.69 (latest is v6.69.1, 15 October 2026). You need a Route 53 public hosted zone or a Cloud DNS public zone, a working deployment in two regions, and AWS CLI v2 or a recent gcloud.

If you want the background on why this matters, the companion post on what the 2025 and 2026 cloud outage postmortems teach walks through seven incidents, including the 14.5-hour AWS us-east-1 event. The one rule from it that shapes everything below: fail over using things that are already running, never by creating or editing resources during the incident.

Step 1: Choose active-passive or active-active, and write down RTO and RPO

Route 53's docs draw the line cleanly. Active-active uses any routing policy except failover (weighted, latency, geolocation) with health checks, so every healthy region serves traffic. Active-passive uses the failover routing policy: Route 53 answers with the primary while it is healthy and switches to the secondary only when all primary resources are unhealthy.

Pattern	DNS policy	Typical data layer	Cost	Main risk
Active-passive (warm standby)	Route 53 failover; Cloud DNS failover policy	Aurora Global Database, async replicas	Secondary runs at reduced scale	Standby is under-scaled or stale when you need it
Active-active	Route 53 latency or weighted with health checks; Cloud DNS WRR or geo	Spanner multi-region, DynamoDB global tables	Full capacity in each region	Write conflicts, and every region must absorb the others' load
Pilot light	Failover record pointing at minimal stack	Replicated data, compute scaled to zero	Lowest	Scaling up during the event depends on control planes

Write two numbers before you touch Terraform. RTO is how long you can be down; RPO is how much data you can lose. Aurora's docs say global database RTO can be in the order of minutes and RPO is typically measured in seconds for an unplanned failover. If your business says RPO zero, you need synchronous replication (Spanner, or a planned switchover), not asynchronous.

Avoid pilot light for anything important. AWS's fault isolation whitepaper lists "provisioning disaster recovery (DR) resources, like ELBs and RDS instances during a failure event" as an anti-pattern, because new load balancers create Route 53 records through a control plane hosted in us-east-1.

Step 2: Build a health endpoint that tells the truth

DNS failover is only as good as the signal. A /healthz that returns 200 while the database is unreachable will keep traffic pinned to a broken region. A check that fails whenever any dependency hiccups will flap and move traffic for no reason. Aim for one endpoint that checks the dependencies the region cannot serve without (its database writer or local replica, its cache) and returns a small, stable body:

# healthz.py: Flask 3.1, psycopg 3.2. Deep check for DNS failover.
from flask import Flask, jsonify
import psycopg

app = Flask(__name__)
DSN = "host=app-writer.example.internal dbname=app user=health connect_timeout=1"

@app.get("/healthz/deep")
def deep():
    try:
        with psycopg.connect(DSN) as conn:          # 1 s connect timeout
            conn.execute("SELECT 1").fetchone()
    except Exception:
        return jsonify(status="degraded"), 503      # Route 53 needs 2xx/3xx to pass
    return jsonify(status="ok"), 200

Keep it fast. Route 53 requires a TCP connection within four seconds and a 2xx or 3xx status within two seconds of connecting, and for string matching the search string must appear in the first 5,120 bytes of the body. Also note that HTTPS health checks do not validate certificates, so an expired certificate will not trigger failover. Monitor certificate expiry separately.

Step 3: Create Route 53 health checks and failover records

This Terraform creates an HTTPS string-match health check on the primary and a pair of failover alias records. I ran terraform validate on it against provider 6.69.

# main.tf: Terraform >= 1.14, hashicorp/aws ~> 6.69
terraform {
  required_version = ">= 1.14"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.69"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

variable "zone_id" { type = string }
variable "primary_alb_dns" { type = string }
variable "primary_alb_zone_id" { type = string }
variable "secondary_alb_dns" { type = string }
variable "secondary_alb_zone_id" { type = string }

resource "aws_route53_health_check" "primary" {
  fqdn              = "primary.app.example.com"
  port              = 443
  type              = "HTTPS_STR_MATCH"
  resource_path     = "/healthz/deep"
  search_string     = "\"status\":\"ok\""
  request_interval  = 10   # 10 or 30 seconds
  failure_threshold = 3    # consecutive failures per checker
  regions           = ["us-east-1", "us-west-2", "eu-west-1"]  # minimum of three
  measure_latency   = true
  tags = { Name = "app-primary-use1" }
}

resource "aws_route53_record" "app_primary" {
  zone_id         = var.zone_id
  name            = "app.example.com"
  type            = "A"
  set_identifier  = "primary-use1"
  health_check_id = aws_route53_health_check.primary.id

  failover_routing_policy {
    type = "PRIMARY"
  }

  alias {
    name                   = var.primary_alb_dns
    zone_id                = var.primary_alb_zone_id
    evaluate_target_health = true
  }
}

resource "aws_route53_record" "app_secondary" {
  zone_id        = var.zone_id
  name           = "app.example.com"
  type           = "A"
  set_identifier = "secondary-usw2"

  failover_routing_policy {
    type = "SECONDARY"
  }

  alias {
    name                   = var.secondary_alb_dns
    zone_id                = var.secondary_alb_zone_id
    evaluate_target_health = true
  }
}

Three details matter here. First, Route 53 aggregates results from its checkers and treats an endpoint as healthy if more than 18 percent of checkers report it healthy, which protects you from one network partition failing you over. Second, evaluate_target_health = true on an ALB alias lets Route 53 route away from a load balancer with no healthy targets even without an explicit health check, but an explicit deep check catches the "ALB is fine, database is not" case. Third, Route 53 has a last-resort mode: if every record is unhealthy, it treats all of them as healthy and answers anyway, rather than returning nothing.

Check the status after terraform apply:

# Per-checker view of the health check (one entry per checker region/IP)
aws route53 get-health-check-status \
  --health-check-id YOUR_ID \
  --query 'HealthCheckObservations[].[Region,StatusReport.Status]' --output table

Cost is small: Route 53 health checks are $0.50 a month for an AWS endpoint plus $1.00 per optional feature (HTTPS, string matching, fast interval and latency measurement), so even a fully featured check costs a few dollars a month, and checks on ELB endpoints are free.

Step 4: Add ARC routing controls for a manual override

Health checks handle the clean failures. The messy ones (a region that is half up, a gray failure, a provider status page that says nothing) need a human decision, and that decision should not depend on editing DNS records, because Route 53 record changes go through a control plane that runs in us-east-1. Route 53 did add opt-in accelerated recovery in November 2025, with a 60-minute recovery objective for public zone changes during a us-east-1 impairment, which is useful but too slow to be your failover mechanism.

ARC routing controls are on/off switches attached to Route 53 health checks of type RECOVERY_CONTROL. The cluster exposes endpoints in five regions, and ARC says at least three of the five are always available for state changes. Save the endpoint list in your runbook now; you cannot rely on looking it up during the incident.

# Turn the primary cell off and the secondary on in one call.
# Endpoint and region come from `describe-cluster` -> ClusterEndpoints; try each in turn.
aws route53-recovery-cluster update-routing-control-states \
  --update-routing-control-state-entries '[
    {"RoutingControlArn": "arn:aws:route53-recovery-control::111122223333:controlpanel/EXAMPLE/routingcontrol/primary-use1", "RoutingControlState": "Off"},
    {"RoutingControlArn": "arn:aws:route53-recovery-control::111122223333:controlpanel/EXAMPLE/routingcontrol/secondary-usw2", "RoutingControlState": "On"}
  ]' \
  --region us-west-2 \
  --endpoint-url https://host-EXAMPLE.us-west-2.example.com/v1
# A successful call returns {}

Add a safety rule that requires at least one routing control to be on, so nobody turns off both cells at 3 AM. The cost is $2.50 per cluster per hour, roughly $1,800 a month, so many teams share one cluster across applications. For orchestrating the whole sequence (scale up, promote database, shift traffic) across accounts, ARC Region switch runs plans from a data plane in each region and costs $70 per plan per month. ARC readiness check, the third feature, stopped accepting new customers on 30 April 2026 according to the pricing page.

Step 5: Configure Cloud DNS failover on Google Cloud

Cloud DNS has a failover routing policy with an active set and a geo-based backup set. The routing policy overview is specific about scope: health checks for internal load balancers work only in private zones, and health checks for external endpoints work only in public zones. External endpoint checks support TCP, HTTP and HTTPS, probe from three source regions, and need a check interval between 30 and 300 seconds. The endpoint must be reachable from the public internet, and your firewall must allow the probes.

# gcloud beta commands per the Cloud DNS docs (October 2026). Use a current SDK:
# my local SDK 472.0.0 lacked the --health-check flag on record-sets create.

# 1. Global health check with three probe regions (30 s minimum interval)
gcloud beta compute health-checks create https app-primary-hc \
  --global \
  --port=443 \
  --request-path=/healthz/deep \
  --check-interval=30s \
  --source-regions=us-east4,us-central1,us-west1

# 2. Failover record: primary IP, geo backup, 0 trickle (no traffic to backup until failover)
gcloud beta dns record-sets create app.example.com. \
  --zone=example-public \
  --type=A \
  --ttl=30 \
  --routing-policy-type=FAILOVER \
  --routing-policy-primary-data=203.0.113.10 \
  --routing-policy-backup-data-type=GEO \
  --routing-policy-backup-data="us-west1=198.51.100.20" \
  --backup-data-trickle-ratio=0.0 \
  --health-check=app-primary-hc

Two things to verify against gcloud beta dns record-sets create --help on your SDK version: the format of primary data for external endpoints (the docs' gcloud section describes forwarding rules, while the console flow uses external IPs), and whether you want geofencing. The trickle ratio is a fraction between 0 and 1; the docs give 0.1 as typical for keeping the backup warm, and setting it to 1 forces a manual failover, which is a handy data plane switch.

Step 6: Replicate and promote the data layer

DNS moves requests in seconds. Data is where failovers actually fail.

Aurora Global Database distinguishes switchover and failover. Switchover (formerly "managed planned failover") waits for the secondary to synchronise, so RPO is zero, and on recent engine versions (Aurora MySQL 3.09, Aurora PostgreSQL 16.8, 15.12, 14.17, 13.20 or later) read/write recovery is typically under 30 seconds. Failover is for a real outage and can lose unreplicated writes.

# Planned switchover (game days, failback). Run against the CURRENT primary region.
aws rds --region us-east-1 switchover-global-cluster \
  --global-cluster-identifier app-global \
  --target-db-cluster-identifier arn:aws:rds:us-west-2:111122223333:cluster:app-usw2

# Unplanned failover. Run against the SECONDARY region you are promoting.
aws rds --region us-west-2 failover-global-cluster \
  --global-cluster-identifier app-global \
  --target-db-cluster-identifier arn:aws:rds:us-west-2:111122223333:cluster:app-usw2 \
  --allow-data-loss

# Before choosing a target, check replication lag (milliseconds) on each secondary
aws cloudwatch get-metric-statistics --region us-west-2 \
  --namespace AWS/RDS --metric-name AuroraGlobalDBRPOLag \
  --dimensions Name=DBClusterIdentifier,Value=app-usw2 \
  --start-time "$(date -u -v-10M +%FT%TZ 2>/dev/null || date -u -d '-10 min' +%FT%TZ)" \
  --end-time "$(date -u +%FT%TZ)" --period 60 --statistics Maximum

Use the global writer endpoint so applications do not need new connection strings, and follow AWS's advice to keep client DNS caching around 5 seconds for it. Promoted clusters do not inherit parameter groups, alarms or integrations, so keep those identical in both regions ahead of time. On PostgreSQL, rds.global_db_rpo (minimum 20 seconds) can enforce an upper bound on lag by blocking commits, which trades availability for RPO.

Spanner takes the other approach. A multi-region configuration has two read-write regions, each with two read-write replicas, plus a witness region, offers a 99.999 percent availability SLA, and moves leaders to the second read-write region automatically if the default leader region fails. You pay in write latency and price, and you skip the promotion step entirely.

Do not forget the rest: secrets replicated to the second region, IAM roles that already exist there, S3 buckets created with their policies, container images in a regional registry. Each of those, if created during the event, is a control plane call into a stressed provider.

Step 7: Test the failover before your provider does

An untested failover is a hypothesis. The least disruptive test on AWS is to invert the primary health check, which makes Route 53 treat it as unhealthy without touching the application:

# Simulate primary failure (Route 53 now reports the primary as unhealthy)
aws route53 update-health-check --health-check-id YOUR_ID --inverted

# Watch answers change from a few public resolvers (TTL is the second column)
for r in 1.1.1.1 8.8.8.8 9.9.9.9; do
  echo "== $r"; dig +noall +answer @"$r" app.example.com A
done

# Restore
aws route53 update-health-check --health-check-id YOUR_ID --no-inverted

For a fuller game day, do it in order: invert the check (or turn off the ARC routing control), confirm traffic in the secondary's load balancer metrics, run an Aurora switchover, run synthetic writes, then fail back with another switchover. Time each step against your RTO. Doing this regularly is what chaos engineering experiments formalise, and writing up what surprised you is what a blameless postmortem template is for, even when nothing broke.

Fix DNS TTL and caching problems during failover

Most "failover didn't work" reports are really "failover worked and clients ignored it". Things to check:

• Record TTL. For non-alias records, keep failover records short (30 to 60 seconds). For alias records you do not set a TTL; check what resolvers actually return with dig.
• Application DNS caches. The AWS SDK docs recommend a JVM DNS TTL of 5 seconds and warn that some Java configurations never refresh until restart. Set networkaddress.cache.ttl=5 in java.security or with Security.setProperty at startup; it cannot be set with -D.
• Long-lived connections. Connection pools and HTTP/2 keep-alive connections never re-resolve. Set a maximum connection lifetime so clients reconnect within your RTO.
• Negative caching. If a name returned NXDOMAIN during a botched change, resolvers cache that per the zone's SOA settings.

Troubleshooting

Symptom	Likely cause	Fix
Health check unhealthy but site works in a browser	Endpoint blocks Route 53 checker IPs, or response exceeds 2 s	Allow the Route 53 health checker ranges; make /healthz/deep cheap
String match check fails intermittently	Search string beyond first 5,120 bytes, or body arrives after 2 s	Return a tiny JSON body with the string first
Traffic still goes to primary after it fails	Client or JVM DNS cache, pooled connections	5 s JVM TTL, max connection lifetime, verify with dig
Both regions get traffic in active-passive	All records unhealthy, so Route 53 last-resort mode answers with all	Fix the secondary's health; check evaluate_target_health on both
Failover never happens on ALB alias	evaluate_target_health false and no health check attached	Set it to true, add an explicit deep health check
Cloud DNS health check always unhealthy	Firewall blocks probes, or interval under 30 s	Allow probes on the port; set --check-interval to 30 s or more
terraform apply fails on health check regions	Fewer than three regions listed	List at least three regions or omit the argument
Aurora switchover rejected	Engine major/minor versions differ between clusters	Upgrade so versions match, or use the manual failover procedure
Writes appear in old primary after failover	Write fencing is best effort; apps cached old endpoint	Take writers offline first, use the global writer endpoint, short DNS TTL
HTTPS health check fails on a TLS 1.3-only endpoint	Route 53 HTTPS checks need TLS 1.0, 1.1 or 1.2 support	Keep TLS 1.2 enabled on the health check listener
Disabled the check to test, traffic did not move	A disabled health check is always considered healthy	Use --inverted to simulate failure
Routing control update times out	That cluster endpoint is in maintenance	Retry the next of the five endpoints saved in the runbook

FAQ

How long does Route 53 DNS failover take?

Detection takes roughly the request interval multiplied by the failure threshold (30 seconds with a 10-second interval and threshold 3), plus the time for enough checkers to agree. Then clients must re-resolve, which depends on the record TTL and any caching in the application. Plan for one to a few minutes end to end, and measure it in a test.

Should I use active-active or active-passive?

Active-passive is simpler and works with asynchronous databases such as Aurora Global Database. Active-active gives faster recovery and uses all capacity, but needs a data layer that handles writes in several regions (Spanner, DynamoDB global tables) and enough headroom for each region to take the other's load. Most teams should start active-passive with a warm, pre-scaled standby.

Can I fail over by updating DNS records during an outage?

You can, but you should not depend on it. Route 53 record changes use a control plane hosted in us-east-1, which is exactly what may be impaired. Use health checks or ARC routing controls, which work through the data plane.

Does Cloud DNS support health checks for public zones?

Yes. Cloud DNS supports health checks for external endpoints in public zones and for internal load balancers in private zones, but not the other way round. External checks use TCP, HTTP or HTTPS from three source regions with intervals of 30 to 300 seconds.

When I review someone's failover setup, the first thing I check is not the DNS: it is whether the secondary region can serve real traffic right now, without anyone creating anything. Then I look at client DNS caching, then at whether anyone has run the inverted-health-check test in the last quarter. If you are writing this as code, the Terraform testing guide helps keep the DR module honest. Official references: Route 53 DNS failover, ARC developer guide, Cloud DNS routing policies, Aurora Global Database DR and Spanner instance configurations.

Related: microVM Production Incidents: 4 Postmortems From My Fleet, Infrastructure as Code in 2026: Two Forks at 1.16 and 1.13, a $6.4B Owner, 912 Public State Files, and One Agent That Ran terraform destroy and It's 2 AM. Do You Know What Your AI Agent Is Doing?.