Kubernetes Troubleshooting Interview Scenarios: 10 Labs (2026)

This guide is for anyone preparing for a DevOps, SRE or platform interview with a live Kubernetes round, or for a new on-call engineer who wants to have seen the common failures before 3 a.m. You will build a three-node practice cluster on your laptop, then break and fix it ten different ways, each with the commands to run, what to look for, and how to explain your reasoning out loud. Everything here was checked against kind v0.33.0 (released 26 August 2026), whose default node image runs Kubernetes 1.37.0, and the current Kubernetes docs. You need Docker or Podman, kubectl, and about 6 GB of free memory. The paired post, DevOps interview questions for 2026, covers the non-Kubernetes half of the interview.

How to explain your reasoning in a troubleshooting interview

Interviewers grade the path, not only the fix. Say what you are checking and why before you type. Almost every scenario below resolves through the same four questions, in this order:

1. What does the status say? kubectl get pods -o wide and kubectl get events --sort-by=.lastTimestamp.
2. What does the object say about itself? kubectl describe and its Events section.
3. What did the process say? kubectl logs, and kubectl logs --previous for a container that already restarted.
4. What does the thing it depends on say? The node, the Service's EndpointSlices, the PVC, CoreDNS, RBAC.

Step 1: Build the practice lab with kind

Install kind v0.33.0 (on macOS brew install kind, or download the binary from the kind quick start), then create a cluster with one control plane and two workers. The release notes say to pin node images by digest, so this config does.

# lab.yaml: kind v0.33.0, Kubernetes v1.37.0 node image pinned by digest
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: lab
nodes:
  - role: control-plane
    image: kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5
  - role: worker
    image: kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5
  - role: worker
    image: kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5

kind version                      # expect: kind v0.33.0 ...
kind create cluster --config lab.yaml
kubectl config current-context    # expect: kind-lab (never practise on a real context)
kubectl get nodes                 # lab-control-plane, lab-worker, lab-worker2 all Ready
kubectl create namespace lab
kubectl config set-context --current --namespace=lab

Check the context every time you open a new terminal. Breaking things on purpose is only fun on the right cluster.

Step 2: Scenario 1, CrashLoopBackOff

# s1-crash.yaml: the process exits with code 1 a second after starting
apiVersion: v1
kind: Pod
metadata:
  name: crash
spec:
  containers:
    - name: app
      image: busybox:1.37.0
      command: ["sh", "-c", "echo connecting to db at $DB_HOST; sleep 1; exit 1"]

kubectl apply -f s1-crash.yaml
kubectl get pod crash -w                 # STATUS flips Error -> CrashLoopBackOff, RESTARTS climbs
kubectl logs crash --previous            # output of the run that just died
kubectl get pod crash -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'
# expect: 1

Reasoning to say out loud: CrashLoopBackOff is not an error in itself, it is the kubelet waiting before the next restart. The pod lifecycle docs put the default delay at 10s, doubling to a cap of 300 seconds, reset after 10 minutes of clean running. Exit code 1 is the application; 137 is SIGKILL, often OOM; 143 is SIGTERM, often a liveness probe. Here the log shows an empty DB_HOST, so the fix is the missing env var. The full cause list is in my CrashLoopBackOff guide; if a liveness probe is the killer, see the probe failure guide.

Step 3: Scenario 2, Pod stuck in Pending

# s2-pending.yaml: requests more CPU than any kind node has
apiVersion: v1
kind: Pod
metadata:
  name: greedy
spec:
  containers:
    - name: app
      image: nginx:1.30.6-alpine
      resources:
        requests:
          cpu: "64"
          memory: 128Mi

kubectl apply -f s2-pending.yaml
kubectl describe pod greedy | sed -n '/Events:/,$p'
# expect a FailedScheduling event similar to:
#   0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: },
#   2 Insufficient cpu. ...
kubectl describe node lab-worker | grep -A6 Allocatable

Reasoning: Pending means the scheduler has not placed the pod, so the container has never run and there are no logs to read. The FailedScheduling event lists every node and why it was rejected. Compare requests to node allocatable, not to current usage. Lower the request or add capacity. The other causes (taints, affinity, unbound PVCs, quotas) are in Pods stuck in Pending.

Step 4: Scenario 3, OOMKilled and exit code 137

This mirrors the example in the Kubernetes memory resources task: allocate 250M under a 100Mi limit. The polinux/stress:1.0.4 image is published for amd64 only, so on an arm64 host such as Apple Silicon expect a pull or start failure instead; use an arm64-capable stress image there.

# s3-oom.yaml
apiVersion: v1
kind: Pod
metadata:
  name: oom
spec:
  containers:
    - name: stress
      image: polinux/stress:1.0.4
      command: ["stress"]
      args: ["--vm", "1", "--vm-bytes", "250M", "--vm-hang", "1"]
      resources:
        requests:
          memory: 50Mi
        limits:
          memory: 100Mi

kubectl apply -f s3-oom.yaml
kubectl get pod oom                      # STATUS OOMKilled, then CrashLoopBackOff
kubectl get pod oom -o jsonpath='{.status.containerStatuses[0].lastState.terminated}{"\n"}'
# expect: ... "exitCode":137 ... "reason":"OOMKilled" ...

Reasoning: the kernel's OOM killer acted because the container crossed its cgroup memory limit, and 137 is 128 plus signal 9. Logs often show nothing useful because the process had no chance to write. Decide whether the limit is too low or the app leaks; raising limits blindly just moves the crash. Distinguish this from node-pressure eviction, which shows Evicted instead. More in the OOMKilled guide and requests and limits explained.

Step 5: Scenario 4, cluster DNS failing

# Break it: take CoreDNS away
kubectl -n kube-system scale deployment coredns --replicas=0

# Observe from inside a pod
kubectl run dns-test --image=busybox:1.37.0 --restart=Never -it --rm -- nslookup kubernetes.default
# expect a timeout such as: ;; connection timed out; no servers could be reached

# Diagnose
kubectl run dns-test --image=busybox:1.37.0 --restart=Never -it --rm -- cat /etc/resolv.conf
kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system get endpointslices -l kubernetes.io/service-name=kube-dns

# Fix
kubectl -n kube-system scale deployment coredns --replicas=2

Reasoning: prove it is DNS before blaming DNS. resolv.conf should point at the kube-dns Service IP; if that Service has no ready endpoints, nothing answers. In real clusters the usual culprits are CoreDNS crashlooping on a forwarding loop, a NetworkPolicy blocking UDP and TCP 53, or ndots:5 making external lookups slow. The upstream DNS debugging task and my CoreDNS troubleshooting guide go further.

Step 6: Scenario 5, Service has no endpoints

# s5-svc.yaml: Deployment labelled app=web, Service selecting app=wbe, wrong targetPort too
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
        - name: nginx
          image: nginx:1.30.6-alpine
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector: { app: wbe }      # typo on purpose
  ports:
    - port: 80
      targetPort: 8080        # nginx listens on 80

kubectl apply -f s5-svc.yaml
kubectl run curl --image=busybox:1.37.0 --restart=Never -it --rm -- wget -qO- -T 3 http://web
kubectl get endpointslices -l kubernetes.io/service-name=web
# expect: ENDPOINTS <unset>, the selector matches nothing
kubectl get pods -l app=wbe            # No resources found
kubectl patch service web -p '{"spec":{"selector":{"app":"web"}}}'
kubectl get endpointslices -l kubernetes.io/service-name=web   # now two IPs, port 8080
kubectl patch service web -p '{"spec":{"ports":[{"port":80,"targetPort":80}]}}'

Reasoning: a Service is a selector plus a port mapping. Empty EndpointSlices mean the selector matches no ready pods (wrong label, or pods not passing readiness). Endpoints present but connections refused means targetPort does not match the listening port. Use EndpointSlices rather than kubectl get endpoints; the v1 Endpoints API is deprecated since 1.33. The full checklist is in the Service connection refused guide and the upstream debug Services task.

Step 7: Scenario 6, ImagePullBackOff

kubectl run badimg --image=nginx:1.99.0-doesnotexist
kubectl get pod badimg                         # ErrImagePull, then ImagePullBackOff
kubectl describe pod badimg | grep -A3 -i "failed to pull"
# expect a message saying the tag or manifest was not found
kubectl delete pod badimg

Reasoning: read the exact pull error, because "not found", "unauthorized" and "too many requests" have three different fixes: correct the tag, add imagePullSecrets (or node credentials), or use a mirror or authenticated pulls. On mixed-architecture clusters, a "no matching manifest" error means the image lacks your node's platform. See ImagePullBackOff and ErrImagePull.

Step 8: Scenario 7, node NotReady

kind nodes are containers, so stopping one simulates a dead machine.

kubectl get pods -o wide                       # note which pods run on lab-worker2
docker stop lab-worker2                        # or: podman stop lab-worker2
kubectl get nodes -w                           # lab-worker2 goes NotReady after the grace period
kubectl describe node lab-worker2 | grep -A8 Conditions
kubectl get node lab-worker2 -o jsonpath='{.spec.taints}{"\n"}'
# expect node.kubernetes.io/unreachable taints with effect NoSchedule and NoExecute
docker start lab-worker2

Reasoning: the controller manager marks a node unhealthy once it has not heard from it for --node-monitor-grace-period, which defaults to 50s. Pods are not evicted at once: Kubernetes adds a toleration for not-ready and unreachable with tolerationSeconds=300, so they stay bound for five minutes, per taints and tolerations. On a real node, check kubelet logs (journalctl -u kubelet), the container runtime, disk and memory pressure, and certificate expiry. The upstream cluster debugging page lists more.

Step 9: Scenario 8, PVC stuck in Pending

Two traps in one. kind ships a default StorageClass called standard that binds only when a pod uses the claim, and a claim naming a missing class never binds.

# s8-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-default
spec:
  accessModes: ["ReadWriteOnce"]
  resources: { requests: { storage: 1Gi } }
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-fast
spec:
  storageClassName: fast-ssd   # does not exist in kind
  accessModes: ["ReadWriteOnce"]
  resources: { requests: { storage: 1Gi } }

kubectl apply -f s8-pvc.yaml
kubectl get storageclass                       # standard (default), VOLUMEBINDINGMODE WaitForFirstConsumer
kubectl get pvc                                # both Pending
kubectl describe pvc data-default | sed -n '/Events:/,$p'
# expect: waiting for first consumer to be created before binding
kubectl describe pvc data-fast | sed -n '/Events:/,$p'
# expect an event saying storageclass "fast-ssd" was not found

Reasoning: data-default is fine, it binds as soon as a pod mounts it; saying so is the point of the exercise. data-fast needs a real class name, and since storageClassName cannot be changed on an existing claim, you delete and recreate it. In cloud clusters, add zone mismatches and a missing CSI driver to the list, plus Multi-Attach error when a ReadWriteOnce volume is still attached to an old node. The PVC Pending and Multi-Attach guide and the persistent volumes docs cover these.

Step 10: Scenario 9, RBAC forbidden

kubectl create serviceaccount reader
kubectl auth can-i list pods --as=system:serviceaccount:lab:reader
# expect: no
kubectl get pods --as=system:serviceaccount:lab:reader
# expect: Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:lab:reader"
#         cannot list resource "pods" in API group "" in the namespace "lab"

# Fix with least privilege: a namespaced Role, not cluster-admin
kubectl create role pod-reader --verb=get,list,watch --resource=pods
kubectl create rolebinding reader-pods --role=pod-reader --serviceaccount=lab:reader
kubectl auth can-i list pods --as=system:serviceaccount:lab:reader   # yes

Reasoning: the error message names the user, verb, resource, API group and namespace, which is everything you need to write the Role. kubectl auth can-i (reference) checks before and after. Mention that a ClusterRoleBinding to cluster-admin would also "fix" it, and that saying so and then not doing it is what interviewers want to hear. See the RBAC forbidden guide and the upstream RBAC docs.

Step 11: Scenario 10, rollout stuck

# Reuse the web Deployment from scenario 5, shorten the deadline, ship a bad image
kubectl patch deployment web -p '{"spec":{"progressDeadlineSeconds":60}}'
kubectl set image deployment/web nginx=nginx:1.30.6-alpine-typo
kubectl rollout status deployment/web --timeout=120s
# expect after about a minute: error: deployment "web" exceeded its progress deadline
kubectl get deployment web -o jsonpath='{.status.conditions[?(@.type=="Progressing")].reason}{"\n"}'
# expect: ProgressDeadlineExceeded
kubectl get pods -l app=web                    # old pods still Running, one new pod ImagePullBackOff
kubectl rollout history deployment/web
kubectl rollout undo deployment/web

Reasoning: with the default rolling update, old pods keep serving while the new ReplicaSet fails, so this is degraded, not down. The Deployment docs note that progressDeadlineSeconds defaults to 600 and that Kubernetes only reports ProgressDeadlineExceeded; it does not roll back for you. Roll back first, then find out why the new pods failed (image, probe, config). If the failure came through Helm, the Helm UPGRADE FAILED guide covers the release-state side.

When you are done: kind delete cluster --name lab.

Troubleshooting

Symptom	Likely cause	Fix
kind create cluster hangs or fails on node image	Image digest for another kind version, or wrong host architecture	Use the digest from the v0.33.0 release notes; images must match your host platform
kubectl talks to the wrong cluster	Current context is not kind-lab	kubectl config use-context kind-lab before every exercise
Scenario 3 pod shows Evicted, not OOMKilled	Node memory pressure, not the container limit	Give Docker more memory or lower --vm-bytes
nslookup works in scenario 4	CoreDNS pods still terminating or cached	Wait for kubectl -n kube-system get pods -l k8s-app=kube-dns to show none
Scenario 5 still refused after fixing the selector	targetPort still 8080 while nginx listens on 80	Patch targetPort to 80, then retest
Node never goes NotReady in scenario 7	Stopped the wrong container	docker ps --filter name=lab- and stop a worker
kubectl auth can-i says yes before the fix	Testing as yourself, not the service account	Pass --as=system:serviceaccount:lab:reader
rollout status returns success in scenario 10	Image tag actually exists, or deadline not patched	Confirm the tag is invalid and progressDeadlineSeconds is 60

FAQ

What are the most common Kubernetes troubleshooting interview questions?

CrashLoopBackOff, Pending pods, OOMKilled, a Service with no endpoints, DNS failures, ImagePullBackOff, a NotReady node, PVC binding, RBAC forbidden and a stuck rollout. Those are the ten scenarios above, and between them they cover most of what a live round can reasonably include.

Should I use kind, minikube or k3d to practise?

Any of them works. I use kind here because multi-node clusters are a single config file and nodes are plain containers you can stop to simulate failures. minikube and k3d can do the same with different commands.

Can I use an AI assistant in a live troubleshooting interview?

Ask first; policies vary. Even where it is allowed, interviewers watch whether you check its suggestion against describe and events before running it, so the method above still matters. Tools like K8sGPT and HolmesGPT are compared in my AI troubleshooting post.

How is this different from the CKA troubleshooting domain?

The CKA's troubleshooting domain also covers broken control plane components and kubelet configuration on real nodes, which kind makes awkward. These scenarios cover the workload side; pair them with a kubeadm cluster if you are sitting the exam.

In production, the first thing I check is the events sorted by time, because they usually name the failing object before I have to guess, and the second is what changed in the last hour, because most incidents follow a deploy. I keep the kubectl quick reference and the upstream application debugging pages bookmarked, and I run through scenarios like these whenever someone new joins the rota. For the rest of the interview, see the 36 DevOps interview questions.

Related: How to Fix Kubernetes CrashLoopBackOff: Causes and Commands (2026), How to Fix Kubernetes Pods Stuck in Pending: Causes and Commands (2026) and How to Troubleshoot Kubernetes DNS and CoreDNS: Step by Step (2026).