How to Troubleshoot Kubernetes DNS and CoreDNS: Step by Step (2026)

This guide is for anyone staring at i/o timeout, NXDOMAIN or a mysterious five-second delay inside a Kubernetes pod and wondering whether it really is DNS. (It usually is.) By the end you will have a repeatable sequence: prove the failure from a debug pod, read the pod's resolver config, check CoreDNS end to end, read its logs and metrics, and fix the common causes, including the conntrack race and the ndots:5 query storm, with NodeLocal DNSCache as the structural fix. Commands were checked against the Kubernetes docs for 1.35 to 1.37, CoreDNS v1.14.7 and the upstream NodeLocal DNSCache manifest. You need kubectl access with permission to create pods and read kube-system; node-level steps need permission to run kubectl debug node.

How Kubernetes DNS works in one paragraph

Each pod gets an /etc/resolv.conf written by the kubelet. With the default ClusterFirst policy it points at the kube-dns Service ClusterIP (for example 10.96.0.10), lists search domains for the pod's namespace, and sets options ndots:5. The Service is still called kube-dns for historical reasons, but behind it run CoreDNS pods labelled k8s-app=kube-dns. Queries for cluster names (*.cluster.local) are answered by CoreDNS's kubernetes plugin from the API server's view of Services and EndpointSlices; everything else goes through the forward plugin to the upstream resolvers in the node's /etc/resolv.conf. Every query from a pod to that ClusterIP passes through kube-proxy's DNAT and the kernel's conntrack table, which matters in Step 6.

Step 1: Reproduce the failure from a debug pod

Do not debug DNS from your application container, which may lack tools or use a resolver (Go, Java, musl) with its own quirks. Start a known-good pod in the same namespace as the failing workload. The official docs use agnhost; I prefer jessie-dnsutils because it ships dig:

# Debug pod with dig and nslookup, in the failing workload's namespace
kubectl run dnsutils -n my-app \
  --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7 \
  --restart=Never --command -- sleep 86400

kubectl wait -n my-app --for=condition=Ready pod/dnsutils --timeout=60s

Now test three kinds of name, because each fails for different reasons:

# 1. The API server Service: proves CoreDNS answers cluster names
kubectl exec -n my-app dnsutils -- nslookup kubernetes.default

# 2. Your own Service, by short name and by FQDN
kubectl exec -n my-app dnsutils -- dig +search my-svc
kubectl exec -n my-app dnsutils -- dig my-svc.my-app.svc.cluster.local

# 3. An external name: proves CoreDNS can forward upstream
kubectl exec -n my-app dnsutils -- dig example.com

A healthy nslookup kubernetes.default returns the DNS server address and the kubernetes Service ClusterIP, as shown in the Kubernetes DNS debugging docs. Read the result as a matrix. All three fail: CoreDNS is unreachable (Steps 2, 3, 5). Cluster names work and external names fail: upstream forwarding (Step 4). One Service fails with NXDOMAIN: the name, namespace or Service itself is wrong. Everything works but sometimes takes five seconds: Step 6.

If your workload pod has a shell, kubectl debug -it <pod> --image=nicolaka/netshoot:v0.16 --target=<container> attaches an ephemeral container sharing its network namespace, which tests the exact same path.

Step 2: Read the pod's resolv.conf and understand ndots:5

kubectl exec -n my-app dnsutils -- cat /etc/resolv.conf

Expected output on most clusters (search list varies by provider):

search my-app.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

Check three things. The nameserver must equal the kube-dns Service ClusterIP (kubectl get svc -n kube-system kube-dns), or the NodeLocal DNSCache address if you run it in IPVS mode. If it is the node's resolver instead, the pod has dnsPolicy: Default, or it runs with hostNetwork: true without ClusterFirstWithHostNet. The DNS for Services and Pods docs note that Default is not the default: an unset policy means ClusterFirst.

Then ndots:5. Any name with fewer than five dots is treated as relative and tried against every search domain before being tried as-is. A lookup of api.example.com (two dots) from that pod becomes, in order, api.example.com.my-app.svc.cluster.local, api.example.com.svc.cluster.local, api.example.com.cluster.local, then api.example.com, and most resolvers send A and AAAA queries for each. That is up to eight queries and three guaranteed NXDOMAINs for every external lookup, more if your cloud adds search domains. It is correct behaviour, it just multiplies load on CoreDNS and widens the window for Step 6.

Two fixes, both safe. In code or config, use fully qualified names with a trailing dot (api.example.com.) for external endpoints. Or lower ndots for workloads that mostly call external APIs:

# Pod spec fragment: fewer search-domain expansions
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
      - name: single-request-reopen   # glibc only; see Step 6

With ndots: 2, my-svc and my-svc.my-app still expand through the search list, while api.example.com goes straight out. Kubernetes caps the search list at 32 domains and 2,048 characters in total.

Step 3: Check CoreDNS pods, the Service and its endpoints

# CoreDNS pods keep the legacy kube-dns label
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

# The Service and its ClusterIP
kubectl get svc -n kube-system kube-dns

# EndpointSlices: there must be ready IPs behind port 53
kubectl get endpointslice -n kube-system -l kubernetes.io/service-name=kube-dns

Expected: two or more CoreDNS pods Running and 1/1 Ready on different nodes, a kube-dns Service with 53/UDP,53/TCP (and 9153/TCP on many distributions), and EndpointSlices listing the pod IPs. Empty endpoints means the pods exist but are not Ready, so nothing answers on the ClusterIP. Look at kubectl describe pod for failed readiness probes (the ready plugin on port 8181) and at Step 4's logs.

If the pods are in CrashLoopBackOff, the most common cause on self-managed nodes is a forwarding loop: the node's /etc/resolv.conf points at a local stub such as systemd-resolved's 127.0.0.53, CoreDNS forwards to itself, and the loop plugin halts the process on purpose. The Kubernetes docs fix is pointing the kubelet's --resolv-conf at /run/systemd/resolve/resolv.conf; kubeadm detects this automatically. My CrashLoopBackOff guide covers the general case.

Also confirm capacity. Two CoreDNS replicas on a 300-node cluster is a different problem from a broken config, and the max_concurrent rejects in Step 4 will tell you.

Step 4: Read CoreDNS logs and metrics

Start with the logs:

kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100 --prefix

Lines to recognise:

• plugin/loop: Loop (127.0.0.1:...) detected for zone ".": the forwarding loop from Step 3.
• [ERROR] plugin/errors: 2 example.com. A: read udp 10.244.1.5:41234->10.0.0.2:53: i/o timeout: CoreDNS reached its upstream and got no answer. The problem is the upstream resolver or the node's path to it, not cluster DNS.
• Kubernetes API errors (Failed to watch *v1.EndpointSlice): CoreDNS cannot reach the API server or lacks RBAC, so cluster records go stale.

By default CoreDNS logs only errors. To see every query, add the log plugin to the Corefile temporarily:

kubectl -n kube-system edit configmap coredns

.:53 {
    log            # add this line; remove it when you are done
    errors
    health
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

The reload plugin picks up the change within a minute or two, no restart needed. On managed services the ConfigMap may be reconciled by the provider (GKE runs kube-dns rather than CoreDNS by default, and EKS manages CoreDNS as an add-on), so check your provider's docs for the supported customisation path before editing.

Metrics are better than logs for anything intermittent. CoreDNS exposes Prometheus metrics on port 9153 through the prometheus plugin:

# Deployment is named coredns on kubeadm and EKS; adjust for your distribution
kubectl -n kube-system port-forward deploy/coredns 9153:9153 &

curl -s localhost:9153/metrics | grep -E \
  'coredns_dns_responses_total|coredns_forward_max_concurrent_rejects_total|coredns_cache_hits_total|coredns_panics_total' \
  | head -20

Useful PromQL, if you already scrape it:

# Share of SERVFAIL responses per CoreDNS pod over 5 minutes
sum by (pod) (rate(coredns_dns_responses_total{rcode="SERVFAIL"}[5m]))
  / sum by (pod) (rate(coredns_dns_responses_total[5m]))

# p99 latency of upstream forwarding, per upstream
histogram_quantile(0.99,
  sum by (le, to) (rate(coredns_proxy_request_duration_seconds_bucket{proxy_name="forward"}[5m])))

A rising NXDOMAIN rate is usually the ndots expansion from Step 2 and is mostly harmless. Rising SERVFAIL points upstream. Any increase in coredns_forward_max_concurrent_rejects_total means CoreDNS is shedding load: scale replicas, add caching, or reduce the query volume.

Step 5: Check network policies for blocked DNS

If DNS fails in one namespace and works in another, suspect a NetworkPolicy. A default-deny egress policy blocks port 53 to kube-system unless you allow it. The fix is an explicit egress rule, on both UDP and TCP because responses over 512 bytes fall back to TCP:

# Allow all pods in my-app to reach cluster DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: my-app
spec:
  podSelector: {}
  policyTypes: ["Egress"]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

If you run NodeLocal DNSCache, pods talk to a node-local address instead, so the rule above does not match; you need an ipBlock for the link-local address you chose. My network policies guide has the default-deny pattern in full.

Step 6: Fix intermittent five-second DNS timeouts (conntrack)

The classic symptom: CoreDNS is healthy, most lookups take milliseconds, and a few take almost exactly five seconds. Five seconds is the default glibc resolver timeout. The cause, documented in kubernetes/kubernetes#56903, is a race in the kernel's conntrack when a resolver sends the A and AAAA queries in parallel from the same UDP socket through DNAT: one packet's conntrack entry fails to insert and the packet is dropped, so the client waits out its timeout and retries.

Check the counter on an affected node:

# Node debug pod shares the host network namespace
kubectl debug node/ip-10-0-1-23.ec2.internal -it \
  --profile=sysadmin --image=nicolaka/netshoot:v0.16 -- conntrack -S

Look at insert_failed across CPUs; a value that keeps climbing while DNS lookups stall is the race. Distribution kernels have carried fixes for parts of this since 2019 (for example Ubuntu's Bionic SRU), but it still shows up on older node images.

Mitigations, from quick to structural:

1. single-request-reopen in dnsConfig.options (Step 2 snippet) makes glibc use separate sockets for A and AAAA. It does nothing for musl (Alpine) or for runtimes with their own resolvers.
2. Use Alpine 3.18 or later if you use Alpine at all; the Kubernetes docs note earlier musl versions lacked TCP fallback, so responses over 512 bytes failed.
3. Deploy NodeLocal DNSCache (Step 7), which removes DNAT from the pod-to-DNS path.

Step 7: Install NodeLocal DNSCache

NodeLocal DNSCache, stable since Kubernetes 1.18, runs a CoreDNS-based cache as a DaemonSet on every node. Pods query a local address, skipping iptables DNAT and conntrack, and cache misses go to CoreDNS over TCP, whose conntrack entries are removed on close instead of waiting out the 30-second UDP timeout. Managed providers often have their own toggle (check yours first); on a self-managed cluster with kube-proxy in iptables mode:

# Upstream manifest; image in it is registry.k8s.io/dns/k8s-dns-node-cache:1.27.1
curl -sLO https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml

kubedns=$(kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}')
domain=cluster.local
localdns=169.254.20.10   # any link-local IP not used in your cluster

# iptables mode (for IPVS mode, use the variant in the docs)
sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/__PILLAR__DNS__SERVER__/$kubedns/g" nodelocaldns.yaml

kubectl create -f nodelocaldns.yaml
kubectl rollout status ds/node-local-dns -n kube-system

(On macOS, sed -i needs sed -i ''.) In iptables mode no kubelet change is needed, because the cache listens on both the link-local address and the kube-dns ClusterIP. In IPVS mode the kubelet's --cluster-dns must point at the link-local address. Verify by re-running Step 1 and checking the cache's metrics, which the manifest annotates for scraping on port 9253.

Troubleshooting

Symptom	Likely cause	Fix
nslookup kubernetes.default times out from every pod	No ready CoreDNS endpoints, or kube-proxy broken on nodes	Step 3: check pods, EndpointSlices, kube-proxy logs
CoreDNS CrashLoopBackOff, log says Loop ... detected	Node resolv.conf points at a local stub (systemd-resolved)	Set kubelet --resolv-conf=/run/systemd/resolve/resolv.conf or forward to real upstreams
NXDOMAIN for one Service	Wrong namespace or name, or the Service does not exist	Use svc.ns.svc.cluster.local; kubectl get svc -n ns
Cluster names resolve, external names SERVFAIL	Upstream resolver failing or unreachable from nodes	Check forward errors in logs; test upstream from a node
i/o timeout in one namespace only	NetworkPolicy blocks egress to port 53	Add the Step 5 allow rule (UDP and TCP)
Random 5-second delays, CoreDNS healthy	Conntrack race on parallel A/AAAA over UDP	single-request-reopen, then NodeLocal DNSCache
Large responses fail on Alpine images	musl before Alpine 3.18 had no TCP fallback	Upgrade base image to Alpine 3.18 or later
hostNetwork pod cannot resolve Services	dnsPolicy falls back to node resolver	Set dnsPolicy: ClusterFirstWithHostNet
CoreDNS CPU high, max_concurrent rejects rising	Query volume from ndots:5 expansion, too few replicas	Lower ndots, use FQDNs, scale CoreDNS, add NodeLocal DNSCache

FAQ

Why does Kubernetes use ndots:5?

So that names like my-svc.my-ns.svc (three dots) still resolve through the search list without callers writing the full cluster.local suffix. The cost is extra queries for external names, which you can avoid with a trailing dot or a lower ndots in dnsConfig.

How do I see every DNS query CoreDNS receives?

Add the log plugin to the Corefile in the coredns ConfigMap and wait for reload to apply it, then tail the CoreDNS pod logs. Remove it afterwards, because query logging on a busy cluster is expensive.

Should I run NodeLocal DNSCache on every cluster?

On small clusters with low query rates it is optional. Once you see five-second timeouts, conntrack pressure or CoreDNS load from many pods per node, it is the standard fix, and it also reduces cross-node DNS traffic.

What is the difference between kube-dns and CoreDNS?

CoreDNS replaced the original kube-dns implementation as the default cluster DNS server, but the Service and pod label kept the kube-dns name for compatibility. Some managed offerings, such as GKE by default, still run kube-dns itself.

When DNS breaks in production, the first thing I check is whether cluster names and external names fail together or separately, because that one split rules out half the causes in under a minute. After that I look at the SERVFAIL and reject counters rather than logs, and on any cluster with dense nodes I treat NodeLocal DNSCache as default infrastructure rather than a fix, which matters more as consolidation packs more pods per node (see my Kubernetes cost optimisation post). The official references are the DNS debugging docs, DNS for Services and Pods, NodeLocal DNSCache and the CoreDNS plugin docs.

Related: Kubernetes as the Agent Control Plane in 2026, Observability in 2026 and microVM Production Incidents.