kube-proxy ipvs to nftables, Tested: What Broke on Kubernetes 1.37
On 16 November 2026 Kubernetes SIG Network set the end date for kube-proxy's ipvs mode: it stops working by default in 1.40 and the code goes in 1.43. I covered the announcement and the confusing dates in kube-proxy IPVS mode stops working by default in 1.40. This post is the part a news story cannot give you. I ran the migration, on the day, and wrote down what happened.
The blog's advice is short. For nftables, change the mode and restart kube-proxy. For iptables, change the mode and reboot each node. I wanted to know three things. What does a cluster look like on 1.40 if nobody migrates? Does a restart-only switch really leave a clean node? And are the sync time differences visible even on a small cluster?
The answers: on 1.40 kube-proxy crash-loops, and existing Services keep working while new ones do not. The switch to nftables cleaned up after itself. The switch to iptables did not, and a deleted Service kept answering until I restarted the node. And yes, sync times differ by roughly four to thirty times, depending on how you count.
The setup
Everything ran locally. Nothing here touched a real cluster: kubectl ran inside the kind control-plane container against its own admin kubeconfig, with KUBECONFIG=/dev/null set on the host.
| Item | Value |
|---|---|
| Host | Apple M3 Pro, macOS 27.0.1 |
| Container runtime | Docker Desktop, engine 28.0.4, VM kernel 6.10.14-linuxkit, 11 CPUs, 8 GB |
| Cluster | kind v0.33.0, image kindest/node:v1.37.0, one control-plane node and one worker |
| Workload | agnhost:2.53 netexec, 3 replicas, all on the worker |
| Services | 200 ClusterIP Services (echo-0 to echo-199), all selecting the same 3 Pods |
| Measurements | kube-proxy's own Prometheus metrics on the worker, and curl from the worker node |
The cluster config is four lines plus the node list:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: ipvs-test
networking:
kubeProxyMode: ipvs
nodes:
- role: control-plane
- role: worker
The cluster was up in about 42 seconds. kube-proxy's first log lines already carry the deprecation notice, with the dates KEP-5495 gives rather than the blog's shorthand:
I1117 07:42:57.974537 server_linux.go:191] "Using ipvs Proxier"
E1117 07:42:57.974553 server_linux.go:193] "The ipvs proxier has been deprecated and will be
disabled by default in Kubernetes 1.40 and removed in Kubernetes 1.43. Migrate to the
'nftables' proxier instead."
It is logged at error level. If your log pipeline alerts on kube-proxy errors, it has been firing on this line since you upgraded to 1.35. That is also the quickest way to find ipvs clusters: search your logs for Using ipvs Proxier.
To change modes I replaced config.conf in the kube-proxy ConfigMap and ran kubectl -n kube-system rollout restart ds/kube-proxy. That is the kubeadm reconfigure path, which kind uses. Field names are in the kube-proxy configuration reference.
Baseline: ipvs with 200 Services
With 200 Services, the worker had 202 addresses on the kube-ipvs0 dummy interface: one per ClusterIP, plus the API server and DNS. Only 85 lines of iptables rules remained, which is the point of ipvs. All 200 ClusterIPs answered from the worker, and requests to echo-150 were spread evenly, 10 to each Pod over 30 requests.
The first oddity came here. /proc/net/ip_vs listed only 116 TCP virtual servers, although every one of the 200 Services answered. ipvsadm is not in the kindest/node image, so I could not get an authoritative count, and I have not explained the gap. If you script IPVS checks against /proc/net/ip_vs, test them against ipvsadm -Ln first.
Preview of 1.40: turn the gate off
In 1.40 the KubeProxyIPVS feature gate turns off by default. It already exists in 1.37 with a default of true, so you can see the future today by adding this to the kube-proxy configuration:
featureGates:
KubeProxyIPVS: false
My first attempt failed for a dull reason that is worth knowing. The config.conf I pulled out with kubectl get -o jsonpath had no trailing newline, so appending the block glued featureGates: onto the last line. kube-proxy went into CrashLoopBackOff with yaml: line 66: did not find expected key. Check for that if you script this edit with cat >>.
With the YAML fixed, kube-proxy logged a warning that the gate is deprecated, then exited:
E1117 07:46:50.818523 run.go:72] "command failed" err="failed validate:
KubeProxyConfiguration.Mode.ProxyMode: Invalid value: \"ipvs\": must be iptables,
nftables or blank (blank means the best-available proxy [currently iptables])"
Both kube-proxy Pods were in Error with two restarts after 30 seconds. Then came the part that matters for an upgrade window:
| While kube-proxy was crash-looping | Result |
|---|---|
curl to an existing Service (echo-150) from the worker |
Answered. The kernel IPVS rules stayed in place |
New Service late, created after kube-proxy went down |
Timed out (curl exit code 28) |
late after switching to nftables |
Answered 5 seconds after I applied the new ConfigMap |
This is the failure mode to expect if a cluster reaches 1.40 in ipvs mode without anyone noticing. Nothing goes down at once. Every new Service, and every endpoint change for an old one, silently stops being programmed, so traffic for Pods that move would keep going to their old IPs. It would look like a slow, confusing incident, not an obvious outage.
Switch 1: ipvs to nftables, restart only
I changed mode: ipvs to mode: nftables and restarted kube-proxy. On startup the new kube-proxy cleans up the old mode's rules, and here it logged six errors:
E1117 07:47:27.196959 cleanup.go:89] "Failed to execute iptables-restore" err=<
exit status 4: iptables-restore v1.8.9 (nf_tables):
line 7: CHAIN_DEL failed (Device or resource busy): chain KUBE-MARK-MASQ
> table="nat"
E1117 07:47:29.300411 proxier.go:1285] "Failed to list existing sets"
err="receiveAckAware: netlink receive: no such file or directory" ipFamily="IPv4"
They look alarming. The end state was clean. Ten seconds later kube-ipvs0 was gone, the IPVS table was empty, and iptables-save showed only chains kubelet owns (KUBE-FIREWALL, KUBE-KUBELET-CANARY) and the KUBE-IPTABLES-HINT marker. kube-proxy's own nftables table held 1,249 lines of rules for the same 200 Services. Requests were spread across all three Pods. My reading is that the first cleanup pass hit chains that were still referenced and a later pass removed them, but that is an inference from the end state, not something the log says. Do not page anyone over these lines. Do check the end state.
Switch 2: ipvs to iptables, restart only
The blog tells you to reboot after switching to iptables, "for technical reasons". I went back to ipvs, then switched to iptables with a restart only, to see what those reasons look like.
After the restart, kube-proxy wrote 3,152 lines of iptables rules, as expected. But kube-ipvs0 was still there with 203 addresses, and the IPVS virtual servers were still in the kernel. Then I deleted echo-150 and waited eight seconds:
| From | curl to the deleted Service's ClusterIP |
|---|---|
| Worker node | Answered, from Pod echo-...-jpbr9 |
A Pod on the worker (agnhost connect) |
Connected |
| Control-plane node | Timed out |
A Service deleted from the API kept taking traffic on the worker, through IPVS state that no running process manages any more. On the control-plane node the same IP timed out, and I have not worked out why the two nodes differ. In a real cluster this is how a Service you deleted, or a ClusterIP that gets reused, sends traffic somewhere it should not.
Restarting the worker container, which is the closest kind has to a node reboot, fixed it. kube-ipvs0 was gone, the IPVS table was empty, the deleted Service timed out and live Services answered. The reboot advice is not optional.
Sync timings
To compare the three modes I applied 100 new Services in one kubectl apply and read kube-proxy's kubeproxy_sync_proxy_rules_duration_seconds histogram on the worker before and after. I ran it three times per mode, deleting the 100 Services between runs, with the 200 baseline Services always present.
The raw numbers:
| Mode | Syncs per burst | Total sync time | Mean per sync | Last new Service reachable after |
|---|---|---|---|---|
| ipvs | 33, 31, 34 | 3,695 / 3,677 / 3,678 ms | 112.0 / 118.6 / 108.2 ms | 3.96 / 4.13 / 3.99 s |
| nftables | 5, 5, 5 | 121 / 153 / 123 ms | 24.3 / 30.6 / 24.5 ms | 4.54 / 4.50 / 4.48 s |
| iptables | 4, 5, 5 | 701 / 328 / 330 ms | 175.3 / 65.6 / 66.0 ms | 5.24 / 5.85 / 4.82 s |
Three things stand out, and one thing does not.
ipvs did the most work by far. It ran six to seven times as many syncs as the other two modes and spent about 3.7 seconds of kube-proxy time on a burst that cost nftables about an eighth of a second. That matches what the blog says about the IPVS API having no batch updates. In kind's default config, ipvs.minSyncPeriod and nftables.minSyncPeriod were both 0s and iptables.minSyncPeriod was 1s. So the iptables mode's low sync count is partly that setting, but the gap between ipvs and nftables is not.
nftables was fastest per sync. About 25 ms, against 66 ms for iptables after its first run. The one slow iptables run (175 ms) was the first after switching modes, when it was probably still rewriting state.
"Reachable after" barely moved. That number includes kubectl apply, API server writes, EndpointSlice updates and my polling loop, and on a cluster this small those dominate. It came out between 4 and 6 seconds in every mode, and ipvs was even slightly quicker. Control-plane cost is what changes with mode. At 200 Services it is not user-visible. At 20,000 it would be.
Connection setup was the same in all three modes. Over 300 curl requests from the worker to one ClusterIP, median connect time was 0.088 ms for ipvs, 0.094 ms for nftables and 0.087 ms for iptables, with p99 between 0.19 and 0.25 ms. On a two-node laptop cluster that is noise. The blog's own latency chart, and the original nftables mode post, make the same point from larger tests: nftables is not slower than ipvs.
What I would do on a real cluster
Here is the order I would use, based on what broke here:
- Find every ipvs cluster from kube-proxy logs (
Using ipvs Proxier) or akube-ipvs0interface on any node. The error-level deprecation line makes this easy if your logs are centralised. - Check the target. nftables needs Kubernetes 1.36 or later and kernel 5.13 or later, per the blog. Everything else goes to iptables.
- Fix the ConfigMap with a real YAML edit, not an append. Move any
ipvs:overrides to theirnftables:equivalents. An ipvsschedulersetting has no equivalent, and per SIG Network never did much in kube-proxy anyway. - Roll one node pool at a time. Existing Services survive the restart, but changes made during the gap only apply once kube-proxy is back, so do not run deploys at the same time.
- Verify the end state on a node, not the log. Check that
kube-ipvs0is gone and that noKUBE-SVCorKUBE-SEPchains remain iniptables-save. - If the target is iptables, reboot or replace each node after the switch. My deleted-Service test is the reason.
- Do not lean on the
KubeProxyIPVSgate. Setting it back to true in 1.40 works for three releases, and the gate itself logs a deprecation warning.
On managed services, check the provider's own setting first. Azure AKS, for example, exposes kube-proxy mode as IPTABLES, IPVS or NFTABLES through its kube-proxy configuration (still in preview), and editing the ConfigMap directly there is the wrong lever.
What I could not test
- Scale. 200 Services and 3 endpoints on a laptop. The interesting ipvs problems in the blog, such as large bursts without batch updates, need thousands of Services and real nodes.
- A real reboot.
docker restartof a kind node resets its network namespace, which is close to a reboot for this purpose but not the same as rebooting a VM. - Pod-to-Pod traffic across nodes with stale IPVS state. I saw the control-plane node time out where the worker answered and did not dig into why.
- Why
/proc/net/ip_vslisted 116 virtual servers for 200 working Services. Withoutipvsadmin the node image I could not cross-check it.
The scripts and configs I used are short enough to rebuild from this post. The state of the ipvs deprecation, and every other removal with a deadline, is on the Kubernetes release and deprecation calendar. For the Service plumbing all three modes implement, see Kubernetes networking explained. If you would rather drop kube-proxy entirely, the eBPF route is covered in eBPF explained.
FAQ
How do I switch kube-proxy from ipvs to nftables?
On kubeadm-style clusters, change mode: ipvs to mode: nftables in the kube-proxy ConfigMap in kube-system and restart the DaemonSet with kubectl -n kube-system rollout restart ds/kube-proxy. In my test on Kubernetes 1.37 the new kube-proxy removed the IPVS state itself. It logged some cleanup errors but ended clean.
What happens if I upgrade to Kubernetes 1.40 still using ipvs mode?
kube-proxy exits with "must be iptables, nftables or blank" and crash-loops, unless you set the KubeProxyIPVS feature gate to true. In my preview with the gate off, existing Services kept working from leftover kernel rules, but new Services and endpoint changes were not applied.
Do I need to reboot nodes after changing kube-proxy mode?
For ipvs to nftables, a kube-proxy restart was enough in my test. For ipvs to iptables, reboot: without one, the IPVS virtual servers and the kube-ipvs0 interface stayed, and a deleted Service kept answering on that node.
Is nftables mode faster than ipvs?
For rule syncs, yes in my test. A 100-Service burst cost about 0.12 seconds of sync time in nftables mode against 3.7 seconds in ipvs mode on the same kind cluster. Connection setup latency was the same in all three modes at this size.
Related: kube-proxy IPVS mode stops working by default in Kubernetes 1.40, How to Upgrade a kubeadm Cluster and How to Fix Kubernetes Service Connection Refused.
Spot an error or something out of date? Email me and I will fix it and log it on the corrections page.
I'm Ajay Kumar — I build and operate PandaStack, an open-source Firecracker microVM cloud for AI agents, and write about the infrastructure and tooling around it.
Need this kind of infrastructure work? See what I do or email hello@ajayk.sh.
Related
kube-proxy IPVS Mode Stops Working by Default in Kubernetes 1.40
SIG Network set the end of kube-proxy's ipvs backend on 16 November 2026: off by default in 1.40, gone in 1.43. The blog and the KEP word the dates differently. What each says, who is affected, and how to check your clusters.
5 minNov 10, 2026How to Fix AWS ALB Unhealthy Targets and 502/503 Errors (2026)
Fix AWS Application Load Balancer unhealthy targets and 502, 503 and 504 errors step by step: read the health reason code, fix health check paths and success codes, security groups, ip vs instance targets, EKS readiness gates, deregistration delay, keep-alive timeouts and access logs.
13 minNov 10, 2026Kubernetes Networking Explained (2026): CNI, Services, kube-proxy in 1.37
How Kubernetes networking works in 2026: the flat pod network model, what a CNI plugin does (Cilium 1.20, Calico 3.33, Flannel 0.28), ClusterIP vs NodePort vs LoadBalancer, EndpointSlices, cluster DNS, and why kube-proxy is moving from iptables and IPVS to nftables or eBPF.
11 min