How to Fix Kubernetes Pods Stuck in Pending: Causes and Commands (2026)

Oct 20, 2026 · 12 min · Ajay Kumar

This guide is for anyone staring at kubectl get pods and a column of Pending that will not move. By the end you will be able to read the scheduler's FailedScheduling message, map each phrase in it to one specific cause, and fix it with a known command, including the cases where the real problem is Karpenter or the Cluster Autoscaler refusing to add a node. Commands and message strings were checked against Kubernetes 1.37 source, Karpenter 1.15 and Cluster Autoscaler 1.36; my local kubectl is v1.32.3, and everything here works from 1.30 onward unless noted. You need kubectl access to the namespace and, for node-level checks, read access to nodes.

A pod in Pending has been accepted by the API server but is not running. That is either because the scheduler cannot find a node for it (the usual case, and the subject of this guide) or because it has a node and is still pulling images, which shows up as ContainerCreating or ImagePullBackOff rather than a clean Pending. If your pods say ImagePullBackOff, you want the ImagePullBackOff guide instead.

Pending pod triage: the event message picks the branch Pod stuck in Pending Read FailedScheduling event kubectl events --for pod/NAME Insufficient cpu/memory or Too many pods untolerated taint, node affinity, topology spread unbound PVC, PV node affinity, volume zone No scheduler event at all Check requests vs allocatable, then why Karpenter/CA did not scale Compare pod spec with node labels and taints; fix spec or NodePool Check StorageClass binding mode, PV zone, CSI provisioner logs Check schedulingGates, schedulerName, quota errors on ReplicaSet One message can list several reasons, one per group of nodes. Fix the reason that covers the nodes you expected the pod to land on, not the first one printed.
Decision flow built from the reason strings in the Kubernetes 1.37 scheduler plugins and the Cluster Autoscaler and Karpenter event names.

Step 1: Find the Pending pods and read the FailedScheduling event

Start wide, then narrow to one pod. The scheduler writes its verdict as a Warning event with reason FailedScheduling, and the pod gets a PodScheduled=False condition with reason Unschedulable.

# Every Pending pod in the cluster, oldest first
kubectl get pods -A --field-selector=status.phase=Pending \
  --sort-by=.metadata.creationTimestamp

# Warnings for one pod (kubectl events is available in current kubectl releases)
kubectl events -n shop --for pod/checkout-7d9f8b6c4-x2kqp --types=Warning

# The classic way, same information plus the pod spec
kubectl describe pod -n shop checkout-7d9f8b6c4-x2kqp | sed -n '/^Events:/,$p'

# The scheduling condition, machine-readable
kubectl get pod -n shop checkout-7d9f8b6c4-x2kqp \
  -o jsonpath='{.status.conditions[?(@.type=="PodScheduled")]}{"\n"}'

A typical message looks like this (illustrative, from a three-node cluster):

Warning  FailedScheduling  12s  default-scheduler  0/3 nodes are available:
  1 node(s) had untolerated taint(s), 2 Insufficient cpu. preemption: 0/3 nodes are
  available: 1 Preemption is not helpful for scheduling, ...

Read it as a tally. The number before each phrase counts the nodes rejected for that reason; the counts add up to the total. The phrases come straight from the scheduler's filter plugins, so each one maps to one cause below. If the Events: section says <none> and the pod is older than an hour, events may have expired (the API server keeps them for one hour by default); delete one pod to get a fresh attempt.

Step 2: Fix "Insufficient cpu", "Insufficient memory" and "Too many pods"

These come from the NodeResourcesFit plugin in the scheduler source. The scheduler compares the pod's requests against each node's allocatable minus the requests of pods already there. Actual usage does not matter. A node at 5 percent CPU can still be "full" if its pods requested everything.

# What does the pending pod ask for? (init containers count too, as the max)
kubectl get pod -n shop checkout-7d9f8b6c4-x2kqp \
  -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.requests}{"\n"}{end}'

# How much of each node is already requested
kubectl describe nodes | grep -A 8 "Allocated resources"

# Allocatable per node, side by side
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory,PODS:.status.allocatable.pods

Then pick the fix that matches:

  • The request is bigger than any node's allocatable. No autoscaler can help; a 6-CPU request will never fit a 4-vCPU node. Lower the request or add a bigger instance type to the node group or NodePool.
  • The request is reasonable but the cluster is full. This is the node autoscaler's job. Go to Step 6.
  • Requests are inflated. Run the VPA in updateMode: "Off" to get recommendations, then right-size. My requests and limits explainer covers how to choose numbers.
  • Too many pods. The node hit its pods allocatable (often 110, or lower on EKS where it follows ENI limits). Use larger nodes or raise max-pods in the node configuration.

If the pod has a priorityClassName, the scheduler will also try preemption. Preemption is not helpful for scheduling means evicting lower-priority pods would not make room, usually because the blocking reason is a taint or affinity, not resources.

Step 3: Fix taints, tolerations and node affinity

Taints. In Kubernetes 1.35 and later the message is the terse node(s) had untolerated taint(s); in 1.34 and earlier it named the taint, for example node(s) had untolerated taint {dedicated: gpu}. You can see the change in the 1.37 taint plugin. On newer clusters, list the taints yourself:

kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key,EFFECT:.spec.taints[*].effect'

Then either add a matching toleration to the pod or stop targeting those nodes:

# Pod spec fragment: tolerate a dedicated GPU pool AND select it
spec:
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"
  nodeSelector:
    karpenter.sh/nodepool: gpu   # a toleration allows, it does not attract

Common culprits are control-plane taints (node-role.kubernetes.io/control-plane), node.kubernetes.io/not-ready on nodes that never finished booting, and node.kubernetes.io/unschedulable on cordoned nodes, which also shows up as node(s) were unschedulable.

Node affinity and nodeSelector. node(s) didn't match Pod's node affinity/selector means no node carries the labels the pod demands. Compare both sides:

kubectl get pod -n shop checkout-7d9f8b6c4-x2kqp -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.affinity.nodeAffinity}{"\n"}'
kubectl get nodes -L topology.kubernetes.io/zone,node.kubernetes.io/instance-type,kubernetes.io/arch

Typos in label values, a zone that has no nodes, and kubernetes.io/arch: arm64 on an amd64-only cluster account for most of these. With Karpenter, the label must also be allowed by a NodePool's requirements, or Karpenter will not create a node that has it.

Step 4: Fix unbound PVCs and volume zone conflicts

Storage messages come from the VolumeBinding and VolumeZone plugins:

Message Meaning
pod has unbound immediate PersistentVolumeClaims A PVC with Immediate binding has no PV yet
node(s) didn't match PersistentVolume's node affinity The bound PV lives in another zone or on another node (1.32 and earlier: had volume node affinity conflict; 1.33 onward uses the new wording)
node(s) had no available volume zone Legacy zone labels on the PV do not match any node
node(s) did not have enough free storage Local or capacity-tracked storage is full

The strings are from binder.go at v1.37.1.

kubectl get pvc -n shop
kubectl describe pvc -n shop data-checkout-0 | sed -n '/^Events:/,$p'

# Binding mode of the StorageClass: you almost always want WaitForFirstConsumer
kubectl get storageclass -o custom-columns=NAME:.metadata.name,PROVISIONER:.provisioner,MODE:.volumeBindingMode

# Which zone a bound PV is pinned to
kubectl get pv "$(kubectl get pvc -n shop data-checkout-0 -o jsonpath='{.spec.volumeName}')" \
  -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms}{"\n"}'

The zone conflict is the classic StatefulSet trap: a PV was created in eu-west-1a, the node died, and the only nodes left (or the only ones the autoscaler will create) are in eu-west-1b. Block volumes such as EBS and Persistent Disk cannot cross zones. Fixes: make sure the node group or NodePool can launch in every zone that holds volumes, use a WaitForFirstConsumer StorageClass for new claims so the volume follows the pod, and if the PVC never binds at all, read the CSI provisioner's logs for quota or permission errors.

Step 5: Fix topology spread, quotas and scheduling gates

Topology spread. node(s) didn't match pod topology spread constraints means placing the pod would break maxSkew for a constraint with whenUnsatisfiable: DoNotSchedule. The (missing required label) suffix means some nodes lack the topologyKey label entirely.

kubectl get pod -n shop checkout-7d9f8b6c4-x2kqp -o jsonpath='{.spec.topologySpreadConstraints}' | python3 -m json.tool
kubectl get pods -n shop -l app=checkout -o wide   # where the existing replicas sit

If you spread across three zones but your nodes exist in two, every third pod is unplaceable. Use ScheduleAnyway for soft spreading, or make capacity available in the missing zone. Karpenter 1.15 added default topology spread awareness when provisioning, but it still cannot launch into a zone its NodePool excludes.

Resource quotas. A pod that would exceed a ResourceQuota is rejected at admission, so it never appears as Pending. What you see is a Deployment with fewer ready pods than desired and a FailedCreate event on the ReplicaSet saying exceeded quota:

kubectl get events -n shop --field-selector reason=FailedCreate
kubectl describe resourcequota -n shop

Scheduling gates. If kubectl get pods shows SchedulingGated rather than Pending, the pod has spec.schedulingGates and the scheduler will not even try until a controller (often a queueing system such as Kueue) removes them. Check with kubectl get pod NAME -o jsonpath='{.spec.schedulingGates}'.

No event at all. If the pod has no FailedScheduling event, check spec.schedulerName. A pod that asks for a scheduler that is not running will sit in Pending silently.

Step 6: Fix Karpenter or Cluster Autoscaler not scaling up

When the message is Insufficient cpu and you have a node autoscaler, the question becomes why it did not add a node. Both projects leave evidence on the pod.

Cluster Autoscaler. It scans for unschedulable pods every 10 seconds, according to the FAQ. Look for TriggeredScaleUp or NotTriggerScaleUp events on the pod, and read its status ConfigMap:

kubectl events -n shop --for pod/checkout-7d9f8b6c4-x2kqp | grep -E 'TriggeredScaleUp|NotTriggerScaleUp'
kubectl -n kube-system get configmap cluster-autoscaler-status -o yaml
kubectl -n kube-system logs deploy/cluster-autoscaler --since=15m | grep -iE 'scale.?up|max|backoff'

pod didn't trigger scale-up (it wouldn't fit if a new node is added) means no node group's template satisfies the pod: wrong instance size, missing label or taint on the template, or a zone mismatch. If it did trigger but nothing joined, the cloud refused (capacity, quota, IAM) and CA will back off that group; nodes that do not register within max-node-provision-time (15 minutes by default) are treated as failed.

Karpenter. Karpenter emits its own pod events. In v1.15 source they are Nominated (it found or is creating a node), FailedScheduling with a message starting Failed to schedule pod,, and NoCompatibleInstanceTypes on the NodePool.

KARPENTER_NAMESPACE=kube-system   # wherever you installed the Helm chart
kubectl events -n shop --for pod/checkout-7d9f8b6c4-x2kqp | grep -iE 'karpenter|Nominated|Failed to schedule'
kubectl get nodepools
kubectl get nodeclaims -o wide
kubectl describe nodepool default | sed -n '/Limits/,/Events/p'
kubectl -n "$KARPENTER_NAMESPACE" logs deploy/karpenter --since=15m | grep -iE 'error|incompatible|limit'

The messages that matter most:

  • all available instance types exceed limits for nodepool means the NodePool's limits block is full. Raise it deliberately; it is your cost ceiling.
  • incompatible requirements or a list of NodePools the pod did not match means the pod's selectors, tolerations or zones are not allowed by any NodePool.
  • A NodeClaim that exists but never becomes Ready points to the cloud side: insufficient capacity for that instance type (Karpenter 1.15 now publishes a pod event for this), subnet IP exhaustion, or a broken node class (AMI, security group, IAM role).

On GKE with node auto-provisioning, AKS node auto-provisioning or EKS Auto Mode, the same logic applies with the provider's managed version. My Kubernetes autoscaling explainer covers how those layers fit together.

Troubleshooting

Symptom Likely cause Fix
N Insufficient cpu or memory Requests exceed free allocatable Right-size requests; let node autoscaler add capacity; check Step 6
Too many pods Node at its max-pods allocatable Larger nodes or higher max-pods setting
node(s) had untolerated taint(s) Pod lacks toleration for a tainted pool kubectl get nodes -o custom-columns=...TAINTS; add toleration or retarget
didn't match Pod's node affinity/selector Label typo, missing zone or arch kubectl get nodes -L ...; fix selector or NodePool requirements
pod has unbound immediate PersistentVolumeClaims PVC not provisioned kubectl describe pvc; check CSI logs; use WaitForFirstConsumer
didn't match PersistentVolume's node affinity PV pinned to a zone with no usable nodes Allow that zone in node group or NodePool
didn't match pod topology spread constraints Skew rule with a missing zone Add capacity in zone or use ScheduleAnyway
Fewer pods than replicas, no Pending pod ResourceQuota rejection kubectl get events --field-selector reason=FailedCreate; adjust quota
Status SchedulingGated spec.schedulingGates still set Check the controller that owns the gate (for example Kueue)
NotTriggerScaleUp or exceed limits for nodepool Autoscaler template or limits block it Fix node group template or raise NodePool limits

FAQ

Why is my pod Pending with no events?

Either the events expired (one hour by default) or the scheduler never looked at the pod. Check spec.schedulerName and spec.schedulingGates, and confirm the scheduler is healthy. Deleting a pod from a Deployment forces a fresh scheduling attempt and a fresh event.

Does a Pending pod mean the node is out of real CPU?

No. Scheduling uses requests, not usage. A cluster can be mostly idle and still reject pods if every node's requests are already allocated. Compare kubectl describe nodes "Allocated resources" with kubectl top nodes to see the gap.

Why doesn't Karpenter create a node for my Pending pod?

Most often the pod asks for something no NodePool allows (a label, zone, architecture or taint), or the NodePool has hit its limits. Read the pod's Karpenter events and kubectl get nodeclaims; the message says which.

How long should a pod stay Pending during scale-up?

On a healthy cloud, the time for a new node to boot and join plus the image pull, typically a minute or two. Anything beyond the Cluster Autoscaler's 15-minute provision timeout is a failure, not slowness.

When I get paged for Pending pods, I read the full FailedScheduling line before touching anything, because the reason that matters is usually the second one listed, not the first. Then I check whether the autoscaler left an event; if it did not, the autoscaler is the incident. The official references are the scheduling, preemption and eviction docs, the Karpenter disruption docs and the Cluster Autoscaler FAQ. For the wider context, see Kubernetes autoscaling explained.

Related: Kubernetes as the Agent Control Plane in 2026, Fix Kubernetes CrashLoopBackOff and Kubernetes OOMKilled Exit Code 137.

I'm Ajay Kumar — I build and operate PandaStack, an open-source Firecracker microVM cloud for AI agents, and write about the infrastructure and tooling around it.

Need this kind of infrastructure work? See what I do or email hello@ajayk.sh.


Related