How to Run Kubernetes on Spot Instances: Interruption Handling (2026)

This guide is for platform engineers who want to move Kubernetes workloads on EKS onto Spot capacity without turning every interruption into an incident. At the end you will have one working interruption path (Karpenter's native handler, EKS managed node groups, or the AWS Node Termination Handler in Queue Processor mode for self-managed groups), workloads with PodDisruptionBudgets, zone and capacity-type spread, a guaranteed on-demand floor, grace periods sized for a 2-minute warning, and a repeatable interruption test with AWS FIS. Versions used: Karpenter 1.15.1, Node Termination Handler (NTH) 1.25.6 with Helm chart 0.27.6, eksctl 0.231.0, and Kubernetes 1.31 or later. I rendered both Helm charts locally with helm template to check values and environment variables; I did not apply anything to a live account for this post, so run it in a non-production cluster first. Prerequisites: an EKS cluster, kubectl, Helm 3.8+ and permission to create IAM, SQS and EventBridge resources. If you want the background on discounts and interruption rates first, read Spot Instances Explained (2026).

Step 1: Pick one interruption handler for each node group

Spot handling has two layers. The node layer notices the warning and drains the node. The workload layer makes sure a drain, or a node vanishing with no drain at all, does not cause an outage. Most broken setups get the node layer wrong by running two handlers that fight, so decide this first.

The facts behind the diagram: Karpenter handles Spot interruption warnings, scheduled maintenance, and instance stopping and terminating events once you give it an SQS queue (Karpenter disruption docs). EKS managed node groups enable Capacity Rebalancing and drain Spot nodes on rebalance recommendations and interruption notices (EKS docs), and the NTH README says plainly that with managed node groups "you don't need the aws-node-termination-handler" (NTH). NTH remains the answer for self-managed Auto Scaling groups. Do the step that matches your setup and skip the other two.

Step 2 (Karpenter): Wire the interruption queue and split Spot from on-demand

If you have not installed Karpenter yet, follow my Karpenter setup guide, which creates the SQS queue and EventBridge rules through the official CloudFormation template. Then confirm the controller is actually reading the queue. Without it, Karpenter still runs, but nodes die at the 2-minute mark with no drain.

# The Helm value settings.interruptionQueue becomes the INTERRUPTION_QUEUE env var (chart 1.15.1)
kubectl -n kube-system get deploy karpenter \
  -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="INTERRUPTION_QUEUE")].value}{"\n"}'
# Expected: your queue name, usually the cluster name. Empty output means no interruption handling.

Next, give yourself a guaranteed on-demand floor. A single NodePool that allows both spot and on-demand will happily put every replica on Spot. The Karpenter scheduling docs describe a trick: two NodePools with disjoint values of a made-up capacity-spread label, and a topology spread across that label. Four Spot values and one on-demand value give roughly a 4:1 ratio. I add one deliberate change to the docs example: the Spot pool also allows on-demand, so when Spot is short Karpenter falls back inside that pool instead of leaving 80% of your replicas Pending.

# nodepools-spot-ratio.yaml (Karpenter 1.15.1, karpenter.sh/v1)
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-pref
spec:
  template:
    spec:
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: default }
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]       # Spot first; on-demand only if Spot is unavailable
        - key: capacity-spread
          operator: In
          values: ["2", "3", "4", "5"]        # four virtual domains
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: karpenter.k8s.aws/instance-family
          operator: Exists
          minValues: 5                        # refuse to launch from fewer than 5 families
      terminationGracePeriod: 1h              # cap on how long any drain may block
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: on-demand-floor
spec:
  template:
    spec:
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: default }
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: capacity-spread
          operator: In
          values: ["1"]                       # one virtual domain = about 1 in 5 replicas
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m

kubectl apply -f nodepools-spot-ratio.yaml
kubectl get nodepools            # both should show READY True

The minValues: 5 line matters more than it looks. Every instance family is another Spot capacity pool, and AWS's own guidance is to diversify; Karpenter will refuse a launch that cannot satisfy five families rather than quietly concentrating you in one. Karpenter does not drain on rebalance recommendations, only on the actual interruption warning, and it publishes a SpotRebalanceRecommendation event instead.

Step 3 (Managed node groups): Create a diversified Spot node group

With managed node groups, EKS does the node layer for you. Your job is to pass many instance types of the same size, keep system add-ons on an on-demand group, and not install NTH. This eksctl config uses the fields from eksctl's own managed Spot example.

# spot-nodegroups.yaml (eksctl 0.231.0)
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: demo            # your existing cluster
  region: us-east-1
managedNodeGroups:
  - name: system-od
    instanceTypes: ["m7i.large", "m6i.large"]
    minSize: 2
    maxSize: 4
    labels: { role: system }          # CoreDNS, ingress, controllers live here
  - name: spot-4vcpu-16gib
    spot: true
    # Same vCPU and memory so the Cluster Autoscaler's size estimates hold
    instanceTypes: ["m7i.xlarge", "m6i.xlarge", "m6a.xlarge", "m7a.xlarge", "m5.xlarge", "m5a.xlarge"]
    minSize: 0
    maxSize: 20
    labels: { role: spot-workers }

eksctl create nodegroup --config-file spot-nodegroups.yaml
kubectl get nodes -L eks.amazonaws.com/capacityType,role
# Expected: spot-workers nodes show SPOT, system nodes show ON_DEMAND

New Spot groups on Kubernetes 1.28 or later use the price-capacity-optimized allocation strategy. EKS drains a Spot node once its replacement is Ready after a rebalance recommendation, but on a best-effort basis: if the 2-minute notice lands first, the drain starts without waiting.

Step 4 (Self-managed ASGs): Install Node Termination Handler in Queue Processor mode

NTH has two modes that cannot run together. IMDS mode is a DaemonSet that polls instance metadata. Queue Processor mode is a Deployment that reads an SQS queue fed by EventBridge and also handles ASG lifecycle hooks, AZ rebalancing and instance state changes. For anything beyond a hobby cluster, use Queue Processor. The AWS side is a queue, a termination lifecycle hook on each ASG, EventBridge rules for the five event types in the NTH README, and a tag on the instances.

# Run with your own AWS credentials; shown here, not executed against an account.
QUEUE_ARN=arn:aws:sqs:us-east-1:123456789012:nth-queue
ASG=my-k8s-spot-asg

aws autoscaling put-lifecycle-hook --lifecycle-hook-name nth-term-hook \
  --auto-scaling-group-name "$ASG" \
  --lifecycle-transition autoscaling:EC2_INSTANCE_TERMINATING \
  --default-result CONTINUE --heartbeat-timeout 300

# NTH only acts on instances with this tag key (value ignored)
aws autoscaling create-or-update-tags --tags \
  "ResourceId=$ASG,ResourceType=auto-scaling-group,Key=aws-node-termination-handler/managed,Value=,PropagateAtLaunch=true"

aws events put-rule --name nth-spot --event-pattern \
  '{"source":["aws.ec2"],"detail-type":["EC2 Spot Instance Interruption Warning"]}'
aws events put-targets --rule nth-spot --targets "Id=1,Arn=$QUEUE_ARN"
# Repeat for: "EC2 Instance Rebalance Recommendation", "EC2 Instance State-change Notification",
# aws.autoscaling "EC2 Instance-terminate Lifecycle Action", and aws.health scheduledChange events.

Then install the chart with an IRSA or Pod Identity role allowing autoscaling:CompleteLifecycleAction, autoscaling:DescribeAutoScalingInstances, autoscaling:DescribeTags, ec2:DescribeInstances, sqs:DeleteMessage and sqs:ReceiveMessage.

helm upgrade --install aws-node-termination-handler \
  oci://public.ecr.aws/aws-ec2/helm/aws-node-termination-handler --version 0.27.6 \
  --namespace kube-system \
  --set enableSqsTerminationDraining=true \
  --set queueURL=https://sqs.us-east-1.amazonaws.com/123456789012/nth-queue \
  --set awsRegion=us-east-1 \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789012:role/nth

kubectl -n kube-system rollout status deploy/aws-node-termination-handler

When I rendered chart 0.27.6 with these values, the Deployment carried ENABLE_SQS_TERMINATION_DRAINING=true, CHECK_TAG_BEFORE_DRAINING=true and NODE_TERMINATION_GRACE_PERIOD=120. That tag check is the most common reason NTH logs messages and drains nothing.

Step 5: Keep the wrong pods off Spot

Stateful sets, databases, cluster add-ons and singletons should never land on Spot. Pin them with a required node affinity on the on-demand label your path uses (karpenter.sh/capacity-type: on-demand or eks.amazonaws.com/capacityType: ON_DEMAND). For Spot-tolerant services on GKE and AKS, the labels differ, and AKS taints Spot pools so pods need a toleration (AKS docs).

# Fragment for a database pod template on EKS with Karpenter
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: karpenter.sh/capacity-type
              operator: In
              values: ["on-demand"]
---
# Fragment for a Spot-tolerant pod on AKS (the Spot pool is tainted NoSchedule)
tolerations:
  - key: kubernetes.azure.com/scalesetpriority
    operator: Equal
    value: spot
    effect: NoSchedule
# On GKE the node label is cloud.google.com/gke-spot=true

Step 6: Add PodDisruptionBudgets that allow a drain

A PDB controls voluntary evictions, which is what every interruption handler uses. Set one per replicated workload, leave room for at least one eviction, and use unhealthyPodEvictionPolicy: AlwaysAllow (stable since Kubernetes 1.31, per the PDB docs) so a crash-looping pod cannot block the drain of a node about to disappear.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  maxUnavailable: 1                       # only one of minAvailable/maxUnavailable per PDB
  unhealthyPodEvictionPolicy: AlwaysAllow # don't let not-ready pods block evictions
  selector:
    matchLabels: { app: api }

kubectl get pdb api
# NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
# api    N/A             1                 1                     10s

Be clear about what a PDB cannot do. Spot interruption is forceful in Karpenter and is not delayed by disruption budgets, and a PDB that blocks eviction does not stop EC2 from reclaiming the instance two minutes later. The PDB's job is to stop the handler from evicting all your replicas from several nodes at the same moment.

Step 7: Spread replicas across zones and capacity types

Interruptions cluster by capacity pool, which means by instance type and zone. Spread across zones so a pool running dry in one zone takes out a fraction of replicas, and across the capacity-spread domains from Step 2 so some replicas always sit on on-demand.

# deployment-api.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 10
  selector:
    matchLabels: { app: api }
  template:
    metadata:
      labels: { app: api }
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector: { matchLabels: { app: api } }
        - maxSkew: 1
          topologyKey: capacity-spread          # Karpenter path only; about 2 of 10 on on-demand
          whenUnsatisfiable: DoNotSchedule
          labelSelector: { matchLabels: { app: api } }
      terminationGracePeriodSeconds: 25         # leave headroom inside the 2-minute warning
      containers:
        - name: api
          image: ghcr.io/example/api:1.4.2      # pin a tag or digest
          ports: [{ containerPort: 8080 }]
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }
            periodSeconds: 5
          lifecycle:
            preStop:
              sleep: { seconds: 5 }             # let endpoints update before SIGTERM
          resources:
            requests: { cpu: 250m, memory: 256Mi }

The preStop sleep action is built into Kubernetes and needs no shell in the image. On the managed node group path, drop the capacity-spread constraint and use a second Deployment or a preferred affinity instead.

Step 8: Size grace periods for the warning you actually get

AWS gives the instance two minutes. Your pods get less. The EKS docs state that evictions start after a delay and run sequentially, that pods can be killed without a signal during mass reclamations, and recommend grace periods of 30 seconds or less with no long preStop hooks. On GKE, regular pods get 15 seconds of the default 30-second shutdown, and a larger terminationGracePeriodSeconds does not extend it (GKE docs). On Azure the Scheduled Events notice is up to 30 seconds. So: handle SIGTERM by failing readiness, stop taking new work, finish or hand back in-flight work within 20 seconds, and make every unit of work idempotent so a SIGKILL is a retry, not data loss. For workers that run longer than that, checkpoint progress to external storage and accept that a fraction of jobs will restart. My readiness and liveness probe guide covers the probe side.

Step 9: Test an interruption with AWS FIS

Do not wait for EC2 to test this for you. AWS Fault Injection Service sends a real rebalance recommendation, then a real 2-minute notice, then interrupts the instance. The action takes a durationBeforeInterruption between 2 and 15 minutes, (FIS docs), and one experiment can target up to 100 Spot Instances chosen by tag (FIS quotas). This template follows the FIS Spot tutorial and targets one running node from the Karpenter pool.

{
  "description": "Interrupt one Spot node in the spot-pref NodePool",
  "targets": {
    "oneSpotNode": {
      "resourceType": "aws:ec2:spot-instance",
      "resourceTags": { "karpenter.sh/nodepool": "spot-pref" },
      "filters": [{ "path": "State.Name", "values": ["running"] }],
      "selectionMode": "COUNT(1)"
    }
  },
  "actions": {
    "interrupt": {
      "actionId": "aws:ec2:send-spot-instance-interruptions",
      "parameters": { "durationBeforeInterruption": "PT2M" },
      "targets": { "SpotInstances": "oneSpotNode" }
    }
  },
  "stopConditions": [{ "source": "none" }],
  "roleArn": "arn:aws:iam::123456789012:role/fis-spot-interrupt"
}

aws fis create-experiment-template --cli-input-json file://spot-interrupt.json
aws fis start-experiment --experiment-template-id EXT_ID_FROM_OUTPUT

# In another terminal, watch the node layer react
kubectl get events -A --field-selector reason=SpotInterrupted -w
kubectl get nodeclaims -w     # a replacement NodeClaim should appear while the old node drains

Karpenter emits SpotRebalanceRecommendation, then SpotInterrupted and TerminatingOnInterruption events from its interruption controller, and counts queue messages in the karpenter_interruption_received_messages_total metric. While the experiment runs, hit the service with a load generator and count errors. Zero errors with one node lost is the pass mark; then try three nodes in one zone.

Troubleshooting

Symptom	Likely cause	Fix
Spot nodes die with no drain, no SpotInterrupted events	Karpenter installed without settings.interruptionQueue, or EventBridge rules missing	Set the Helm value, check the queue receives EC2 Spot Instance Interruption Warning events
Nodes replaced every few minutes, churn with no interruptions	NTH running alongside Karpenter or managed node groups, draining on rebalance recommendations	Remove NTH from those nodes; one handler per node group
NTH logs messages but never drains	Instances lack the aws-node-termination-handler/managed tag (checkTagBeforeDraining is true by default)	Tag the ASG with PropagateAtLaunch=true, or set checkTagBeforeDraining=false if every ASG is Kubernetes
Drain stalls, pods killed at the deadline	PDB with ALLOWED DISRUPTIONS 0, often from a not-ready pod	kubectl get pdb -A; use maxUnavailable: 1 and unhealthyPodEvictionPolicy: AlwaysAllow
Pods get SIGKILL with no SIGTERM handling	Long terminationGracePeriodSeconds or preStop under mass interruption	Keep grace at 30 s or less, make work idempotent
80% of replicas Pending after a Spot shortage	Spot NodePool allows only spot with a DoNotSchedule capacity spread	Allow on-demand in the Spot pool as in Step 2, or relax to ScheduleAnyway
Many nodes interrupted at once	Too few instance types or one zone, so one capacity pool	Add families (minValues), more types per node group, more subnets
Spot group never scales up	Spot vCPU quota, or one instance type with no capacity	Check Service Quotas for Spot vCPUs and widen instanceTypes
Database pod rescheduled onto Spot	No required on-demand node affinity	Add the Step 5 affinity; audit with kubectl get pods -o wide and node labels

FAQ

Do I need AWS Node Termination Handler with Karpenter?

Usually no. Karpenter handles Spot interruption warnings, scheduled maintenance and instance state events itself once its SQS interruption queue is configured. Its docs say you can add NTH if you want rebalance-recommendation draining, but warn that this causes more node churn. Never run both on the same nodes without disabling NTH's Spot and rebalance draining.

Do EKS managed node groups handle Spot interruptions automatically?

Yes. Spot managed node groups enable Capacity Rebalancing, launch a replacement on a rebalance recommendation, and cordon and drain the old node. The NTH README states you do not need NTH with managed node groups. Draining is best effort, so the workload-layer steps still apply.

What ratio of Spot to on-demand should I use?

There is no universal number. Start with an on-demand floor large enough to serve peak traffic at reduced capacity if every Spot node vanished, often one in four or five replicas, then lower it as interruption tests pass. Keep anything stateful at 100% on-demand.

How do I test Spot interruption handling without waiting for AWS?

Use AWS FIS with the aws:ec2:send-spot-instance-interruptions action, or select a Spot request in the EC2 console and choose Actions, Initiate interruption. Both send a rebalance recommendation and a 2-minute notice, then interrupt the instance.

What I check first in production is boring: that exactly one handler owns each node group, that the interruption queue is actually receiving events, and that kubectl get pdb -A shows no budgets stuck at zero allowed disruptions. Then I run one FIS experiment per quarter per cluster, because the configuration that passed last spring is not necessarily the one running now. The official references are the Karpenter disruption docs, the EKS managed node group docs, the NTH README and the EC2 interruption notice docs.

Related: How to Set Up Karpenter on EKS: NodePools, Spot and Consolidation (2026), How to Run Chaos Experiments on Kubernetes with Chaos Mesh (2026) and Kubernetes Cost Optimization in 2026: 8% CPU Use and What Fixes It.