How to Fix Kubernetes Pods Stuck in Pending: Causes and Commands (2026)
This guide is for anyone staring at kubectl get pods and a column of Pending that will not move. By the end you will be able to read the scheduler's FailedScheduling message, map each phrase in it to one specific cause, and fix it with a known command, including the cases where the real problem is Karpenter or the Cluster Autoscaler refusing to add a node. Commands and message strings were checked against Kubernetes 1.37 source, Karpenter 1.15 and Cluster Autoscaler 1.36; my local kubectl is v1.32.3, and everything here works from 1.30 onward unless noted. You need kubectl access to the namespace and, for node-level checks, read access to nodes.
A pod in Pending has been accepted by the API server but is not running. That is either because the scheduler cannot find a node for it (the usual case, and the subject of this guide) or because it has a node and is still pulling images, which shows up as ContainerCreating or ImagePullBackOff rather than a clean Pending. If your pods say ImagePullBackOff, you want the ImagePullBackOff guide instead.
Step 1: Find the Pending pods and read the FailedScheduling event
Start wide, then narrow to one pod. The scheduler writes its verdict as a Warning event with reason FailedScheduling, and the pod gets a PodScheduled=False condition with reason Unschedulable.
# Every Pending pod in the cluster, oldest first
kubectl get pods -A --field-selector=status.phase=Pending \
--sort-by=.metadata.creationTimestamp
# Warnings for one pod (kubectl events is available in current kubectl releases)
kubectl events -n shop --for pod/checkout-7d9f8b6c4-x2kqp --types=Warning
# The classic way, same information plus the pod spec
kubectl describe pod -n shop checkout-7d9f8b6c4-x2kqp | sed -n '/^Events:/,$p'
# The scheduling condition, machine-readable
kubectl get pod -n shop checkout-7d9f8b6c4-x2kqp \
-o jsonpath='{.status.conditions[?(@.type=="PodScheduled")]}{"\n"}'
A typical message looks like this (illustrative, from a three-node cluster):
Warning FailedScheduling 12s default-scheduler 0/3 nodes are available:
1 node(s) had untolerated taint(s), 2 Insufficient cpu. preemption: 0/3 nodes are
available: 1 Preemption is not helpful for scheduling, ...
Read it as a tally. The number before each phrase counts the nodes rejected for that reason; the counts add up to the total. The phrases come straight from the scheduler's filter plugins, so each one maps to one cause below. If the Events: section says <none> and the pod is older than an hour, events may have expired (the API server keeps them for one hour by default); delete one pod to get a fresh attempt.
Step 2: Fix "Insufficient cpu", "Insufficient memory" and "Too many pods"
These come from the NodeResourcesFit plugin in the scheduler source. The scheduler compares the pod's requests against each node's allocatable minus the requests of pods already there. Actual usage does not matter. A node at 5 percent CPU can still be "full" if its pods requested everything.
# What does the pending pod ask for? (init containers count too, as the max)
kubectl get pod -n shop checkout-7d9f8b6c4-x2kqp \
-o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.requests}{"\n"}{end}'
# How much of each node is already requested
kubectl describe nodes | grep -A 8 "Allocated resources"
# Allocatable per node, side by side
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory,PODS:.status.allocatable.pods
Then pick the fix that matches:
- The request is bigger than any node's allocatable. No autoscaler can help; a 6-CPU request will never fit a 4-vCPU node. Lower the request or add a bigger instance type to the node group or NodePool.
- The request is reasonable but the cluster is full. This is the node autoscaler's job. Go to Step 6.
- Requests are inflated. Run the VPA in
updateMode: "Off"to get recommendations, then right-size. My requests and limits explainer covers how to choose numbers. Too many pods. The node hit itspodsallocatable (often 110, or lower on EKS where it follows ENI limits). Use larger nodes or raise max-pods in the node configuration.
If the pod has a priorityClassName, the scheduler will also try preemption. Preemption is not helpful for scheduling means evicting lower-priority pods would not make room, usually because the blocking reason is a taint or affinity, not resources.
Step 3: Fix taints, tolerations and node affinity
Taints. In Kubernetes 1.35 and later the message is the terse node(s) had untolerated taint(s); in 1.34 and earlier it named the taint, for example node(s) had untolerated taint {dedicated: gpu}. You can see the change in the 1.37 taint plugin. On newer clusters, list the taints yourself:
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key,EFFECT:.spec.taints[*].effect'
Then either add a matching toleration to the pod or stop targeting those nodes:
# Pod spec fragment: tolerate a dedicated GPU pool AND select it
spec:
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
nodeSelector:
karpenter.sh/nodepool: gpu # a toleration allows, it does not attract
Common culprits are control-plane taints (node-role.kubernetes.io/control-plane), node.kubernetes.io/not-ready on nodes that never finished booting, and node.kubernetes.io/unschedulable on cordoned nodes, which also shows up as node(s) were unschedulable.
Node affinity and nodeSelector. node(s) didn't match Pod's node affinity/selector means no node carries the labels the pod demands. Compare both sides:
kubectl get pod -n shop checkout-7d9f8b6c4-x2kqp -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.affinity.nodeAffinity}{"\n"}'
kubectl get nodes -L topology.kubernetes.io/zone,node.kubernetes.io/instance-type,kubernetes.io/arch
Typos in label values, a zone that has no nodes, and kubernetes.io/arch: arm64 on an amd64-only cluster account for most of these. With Karpenter, the label must also be allowed by a NodePool's requirements, or Karpenter will not create a node that has it.
Step 4: Fix unbound PVCs and volume zone conflicts
Storage messages come from the VolumeBinding and VolumeZone plugins:
| Message | Meaning |
|---|---|
pod has unbound immediate PersistentVolumeClaims |
A PVC with Immediate binding has no PV yet |
node(s) didn't match PersistentVolume's node affinity |
The bound PV lives in another zone or on another node (1.32 and earlier: had volume node affinity conflict; 1.33 onward uses the new wording) |
node(s) had no available volume zone |
Legacy zone labels on the PV do not match any node |
node(s) did not have enough free storage |
Local or capacity-tracked storage is full |
The strings are from binder.go at v1.37.1.
kubectl get pvc -n shop
kubectl describe pvc -n shop data-checkout-0 | sed -n '/^Events:/,$p'
# Binding mode of the StorageClass: you almost always want WaitForFirstConsumer
kubectl get storageclass -o custom-columns=NAME:.metadata.name,PROVISIONER:.provisioner,MODE:.volumeBindingMode
# Which zone a bound PV is pinned to
kubectl get pv "$(kubectl get pvc -n shop data-checkout-0 -o jsonpath='{.spec.volumeName}')" \
-o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms}{"\n"}'
The zone conflict is the classic StatefulSet trap: a PV was created in eu-west-1a, the node died, and the only nodes left (or the only ones the autoscaler will create) are in eu-west-1b. Block volumes such as EBS and Persistent Disk cannot cross zones. Fixes: make sure the node group or NodePool can launch in every zone that holds volumes, use a WaitForFirstConsumer StorageClass for new claims so the volume follows the pod, and if the PVC never binds at all, read the CSI provisioner's logs for quota or permission errors.
Step 5: Fix topology spread, quotas and scheduling gates
Topology spread. node(s) didn't match pod topology spread constraints means placing the pod would break maxSkew for a constraint with whenUnsatisfiable: DoNotSchedule. The (missing required label) suffix means some nodes lack the topologyKey label entirely.
kubectl get pod -n shop checkout-7d9f8b6c4-x2kqp -o jsonpath='{.spec.topologySpreadConstraints}' | python3 -m json.tool
kubectl get pods -n shop -l app=checkout -o wide # where the existing replicas sit
If you spread across three zones but your nodes exist in two, every third pod is unplaceable. Use ScheduleAnyway for soft spreading, or make capacity available in the missing zone. Karpenter 1.15 added default topology spread awareness when provisioning, but it still cannot launch into a zone its NodePool excludes.
Resource quotas. A pod that would exceed a ResourceQuota is rejected at admission, so it never appears as Pending. What you see is a Deployment with fewer ready pods than desired and a FailedCreate event on the ReplicaSet saying exceeded quota:
kubectl get events -n shop --field-selector reason=FailedCreate
kubectl describe resourcequota -n shop
Scheduling gates. If kubectl get pods shows SchedulingGated rather than Pending, the pod has spec.schedulingGates and the scheduler will not even try until a controller (often a queueing system such as Kueue) removes them. Check with kubectl get pod NAME -o jsonpath='{.spec.schedulingGates}'.
No event at all. If the pod has no FailedScheduling event, check spec.schedulerName. A pod that asks for a scheduler that is not running will sit in Pending silently.
Step 6: Fix Karpenter or Cluster Autoscaler not scaling up
When the message is Insufficient cpu and you have a node autoscaler, the question becomes why it did not add a node. Both projects leave evidence on the pod.
Cluster Autoscaler. It scans for unschedulable pods every 10 seconds, according to the FAQ. Look for TriggeredScaleUp or NotTriggerScaleUp events on the pod, and read its status ConfigMap:
kubectl events -n shop --for pod/checkout-7d9f8b6c4-x2kqp | grep -E 'TriggeredScaleUp|NotTriggerScaleUp'
kubectl -n kube-system get configmap cluster-autoscaler-status -o yaml
kubectl -n kube-system logs deploy/cluster-autoscaler --since=15m | grep -iE 'scale.?up|max|backoff'
pod didn't trigger scale-up (it wouldn't fit if a new node is added) means no node group's template satisfies the pod: wrong instance size, missing label or taint on the template, or a zone mismatch. If it did trigger but nothing joined, the cloud refused (capacity, quota, IAM) and CA will back off that group; nodes that do not register within max-node-provision-time (15 minutes by default) are treated as failed.
Karpenter. Karpenter emits its own pod events. In v1.15 source they are Nominated (it found or is creating a node), FailedScheduling with a message starting Failed to schedule pod,, and NoCompatibleInstanceTypes on the NodePool.
KARPENTER_NAMESPACE=kube-system # wherever you installed the Helm chart
kubectl events -n shop --for pod/checkout-7d9f8b6c4-x2kqp | grep -iE 'karpenter|Nominated|Failed to schedule'
kubectl get nodepools
kubectl get nodeclaims -o wide
kubectl describe nodepool default | sed -n '/Limits/,/Events/p'
kubectl -n "$KARPENTER_NAMESPACE" logs deploy/karpenter --since=15m | grep -iE 'error|incompatible|limit'
The messages that matter most:
all available instance types exceed limits for nodepoolmeans the NodePool'slimitsblock is full. Raise it deliberately; it is your cost ceiling.incompatible requirementsor a list of NodePools the pod did not match means the pod's selectors, tolerations or zones are not allowed by any NodePool.- A NodeClaim that exists but never becomes Ready points to the cloud side: insufficient capacity for that instance type (Karpenter 1.15 now publishes a pod event for this), subnet IP exhaustion, or a broken node class (AMI, security group, IAM role).
On GKE with node auto-provisioning, AKS node auto-provisioning or EKS Auto Mode, the same logic applies with the provider's managed version. My Kubernetes autoscaling explainer covers how those layers fit together.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
N Insufficient cpu or memory |
Requests exceed free allocatable | Right-size requests; let node autoscaler add capacity; check Step 6 |
Too many pods |
Node at its max-pods allocatable | Larger nodes or higher max-pods setting |
node(s) had untolerated taint(s) |
Pod lacks toleration for a tainted pool | kubectl get nodes -o custom-columns=...TAINTS; add toleration or retarget |
didn't match Pod's node affinity/selector |
Label typo, missing zone or arch | kubectl get nodes -L ...; fix selector or NodePool requirements |
pod has unbound immediate PersistentVolumeClaims |
PVC not provisioned | kubectl describe pvc; check CSI logs; use WaitForFirstConsumer |
didn't match PersistentVolume's node affinity |
PV pinned to a zone with no usable nodes | Allow that zone in node group or NodePool |
didn't match pod topology spread constraints |
Skew rule with a missing zone | Add capacity in zone or use ScheduleAnyway |
| Fewer pods than replicas, no Pending pod | ResourceQuota rejection |
kubectl get events --field-selector reason=FailedCreate; adjust quota |
Status SchedulingGated |
spec.schedulingGates still set |
Check the controller that owns the gate (for example Kueue) |
NotTriggerScaleUp or exceed limits for nodepool |
Autoscaler template or limits block it | Fix node group template or raise NodePool limits |
FAQ
Why is my pod Pending with no events?
Either the events expired (one hour by default) or the scheduler never looked at the pod. Check spec.schedulerName and spec.schedulingGates, and confirm the scheduler is healthy. Deleting a pod from a Deployment forces a fresh scheduling attempt and a fresh event.
Does a Pending pod mean the node is out of real CPU?
No. Scheduling uses requests, not usage. A cluster can be mostly idle and still reject pods if every node's requests are already allocated. Compare kubectl describe nodes "Allocated resources" with kubectl top nodes to see the gap.
Why doesn't Karpenter create a node for my Pending pod?
Most often the pod asks for something no NodePool allows (a label, zone, architecture or taint), or the NodePool has hit its limits. Read the pod's Karpenter events and kubectl get nodeclaims; the message says which.
How long should a pod stay Pending during scale-up?
On a healthy cloud, the time for a new node to boot and join plus the image pull, typically a minute or two. Anything beyond the Cluster Autoscaler's 15-minute provision timeout is a failure, not slowness.
When I get paged for Pending pods, I read the full FailedScheduling line before touching anything, because the reason that matters is usually the second one listed, not the first. Then I check whether the autoscaler left an event; if it did not, the autoscaler is the incident. The official references are the scheduling, preemption and eviction docs, the Karpenter disruption docs and the Cluster Autoscaler FAQ. For the wider context, see Kubernetes autoscaling explained.
Related: Kubernetes as the Agent Control Plane in 2026, Fix Kubernetes CrashLoopBackOff and Kubernetes OOMKilled Exit Code 137.
I'm Ajay Kumar — I build and operate PandaStack, an open-source Firecracker microVM cloud for AI agents, and write about the infrastructure and tooling around it.
Need this kind of infrastructure work? See what I do or email hello@ajayk.sh.
Related
How to Do Canary Deployments with Argo Rollouts 1.10 (2026)
Step-by-step Argo Rollouts canary guide for 2026: install v1.10.1 with Helm and the Gateway API plugin v0.17.0, write a Rollout with weighted steps, add a Prometheus AnalysisTemplate, watch automated rollback, and fix the common errors.
14 minOct 20, 2026How to Set Up cert-manager with Let's Encrypt and Fix Errors (2026)
Install cert-manager v1.21 with Helm, create Let's Encrypt ClusterIssuers for HTTP-01 (Gateway API or Ingress) and DNS-01, then debug a Certificate stuck not ready by walking CertificateRequest, Order and Challenge, with rate limits, ARI and alerts.
14 minOct 20, 2026How to Run Chaos Experiments on Kubernetes with Chaos Mesh (2026)
Step-by-step Chaos Mesh 2.8.5 guide: install with Helm on containerd, opt namespaces in, write PodChaos, NetworkChaos and StressChaos YAML, gate experiments on a Prometheus SLO, schedule them, and run the same test with a LitmusChaos promProbe, plus a troubleshooting table.
16 min