Apache Kafka in 2026: Kafka 4.4, No ZooKeeper, Queues and Diskless

I have watched teams adopt Apache Kafka for two reasons: because they had a genuine event-streaming problem, or because a conference talk made it sound like the grown-up choice. In 2026 the first group has a better product than ever and the second group has more ways than ever to regret it. Kafka 4.4.0 shipped on 22 October 2026, ZooKeeper has been gone for over a year and a half, queues are production-ready, and "diskless Kafka" went from a vendor pitch to an accepted Apache proposal. This post explains what actually changed in Apache Kafka 4.x, what KRaft means for operators, where tiered storage and diskless topics stand, how WarpStream, AutoMQ and Redpanda compare, and when you should skip Kafka entirely. If you want the hands-on version on Kubernetes, the companion guide is How to Run Kafka on Kubernetes with Strimzi.

What changed in Apache Kafka 4.x?

Five minor releases in roughly nineteen months. The headline is not any single feature but that the project finished paying down a decade of architectural debt (ZooKeeper, the old rebalance protocol, ancient protocol versions) and then started adding things people had wanted for years.

Release	Date	What matters	Source
3.9.0	6 Nov 2024	Last release with ZooKeeper mode; tiered storage production-ready; dynamic KRaft quorum (KIP-853)	Kafka 3.9 announcement
4.0.0	18 Mar 2025	ZooKeeper removed; new consumer rebalance protocol (KIP-848) GA; queues early access; brokers need Java 17	Kafka 4.0 announcement
4.1.0	2 Sep 2025	Queues in preview, "still not ready for production"	Confluent on 4.1
4.2.0	17 Feb 2026	Queues (share groups) production-ready; Streams rebalance protocol GA with a limited feature set; Java 25 support	Kafka 4.2 announcement
4.3.0	22 May 2026	25 KIPs; follower fetch from tiered offsets; OAuth client assertions	Kafka 4.3 announcement
4.4.0	22 Oct 2026	Dead-letter queues for share groups (KIP-1191); delayed tiered upload (KIP-1241); 963 commits	Confluent on 4.4

The downloads page currently lists 4.4.0, 4.3.2 and 4.2.2 as the supported releases and everything older as archived. If you are on 3.x, you are on an archived line.

A few 4.0 changes still bite people who skipped versions. Kafka 4.0 removed old protocol API versions for the first time, so brokers and clients both need to be on 2.1 or newer before you cross that line. Brokers, Connect and the tools need Java 17, while clients and Streams need Java 11. Logging moved from Log4j to Log4j2. None of this is exotic, but every one of them has broken somebody's "just bump the image tag" upgrade.

KRaft without ZooKeeper: what it changes for operators

KRaft replaces the external ZooKeeper ensemble with a Raft quorum of Kafka controllers that store cluster metadata in an internal log. The operational win is obvious: one system to deploy, secure, monitor and upgrade instead of two. The less obvious change is that you now manage two kinds of Kafka node, controllers and brokers, and they have different sizing and failure profiles. Controllers are small and latency-sensitive; brokers are big and throughput-bound. The usual production layout is three or five dedicated controllers and a separate broker pool. Combined-mode nodes (both roles in one process) are fine for development and small clusters.

Two things matter on day two. First, since KIP-853 in 3.9 the controller quorum is dynamic, so you can add or remove controllers with kafka-metadata-quorum.sh instead of rebuilding the cluster. Kafka 4.4 goes further with KIP-1312, which lets controllers be unregistered. Second, the metadata version is now a first-class upgrade step. You roll new binaries first, then bump the metadata version once every node runs them, and that second step can be hard or impossible to reverse. Treat it like a database schema migration, not a config tweak.

If you still run ZooKeeper, the road is fixed: upgrade to a bridge release (3.9 is the recommended one), migrate metadata to KRaft there, then upgrade to 4.x. The 3.9 announcement calls 3.9 the bridge. There is no shortcut, and 3.9 is no longer on the supported list, so this is overdue work rather than optional work.

On Kubernetes the same shift happened in the operators. Strimzi, the CNCF operator most self-hosters use, moved to a v1 CRD API in 1.0.0 and removed v1beta2 entirely; its 1.3.1 release of 4 November 2026 supports Kafka 4.2.0 to 4.3.2 and Kubernetes 1.32 or newer. Kafka 4.4.0 support is in the unreleased 1.4.0 section of the Strimzi changelog, which also drops Kafka 4.2.x. If you run Kafka on Kubernetes, your Kafka version is really gated by your operator version.

Queues for Kafka (KIP-932): do you still need RabbitMQ?

Classic consumer groups assign each partition to one consumer, which makes partitions the ceiling on parallelism and makes "one slow message blocks everything behind it" a familiar failure. Share groups, the feature behind KIP-932, let many consumers read from the same partition, with per-record acknowledgement, redelivery after a lock timeout, and delivery counts. That is queue semantics on top of a log.

The status history is worth knowing because blog posts from 2025 are still ranking: early access in 4.0, preview in 4.1, and in 4.2 the Apache announcement says share groups are "now production-ready". Kafka 4.2 also added a RENEW acknowledgement type (KIP-1222) for long-running work. Kafka 4.4 added dead-letter queues for share groups (KIP-1191), which only work once share.version is upgraded to 2.

The feature is gated by a cluster feature flag. On a cluster upgraded from an older release you turn it on explicitly, as the Apache upgrade notes describe:

# Kafka 4.4.0 tools, run against a test cluster first.
# Check finalized feature levels (share.version, metadata.version, etc.)
bin/kafka-features.sh --bootstrap-server localhost:9092 describe

# Enable share groups (level 1). Level 2 adds dead-letter queues in 4.4 and
# needs metadata.version 4.4-IV0 or later.
bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature share.version=2

# Two of these in parallel share one partition's records between them.
bin/kafka-console-share-consumer.sh --bootstrap-server localhost:9092 \
  --topic orders --group order-workers

So, do you still need RabbitMQ? If you already run Kafka and want work-queue consumption for some topics, share groups remove a reason to run a second broker. If you do not run Kafka, share groups are not a reason to start: you would be adopting a distributed log to get a queue. Note the trade-off too. Share groups give up per-partition ordering, because records are dispatched and redelivered independently.

Tiered storage vs diskless Kafka: what is actually shipped?

These two get conflated constantly. They solve different halves of the cost problem.

Tiered storage (KIP-405, production-ready since 3.9) keeps the active segment on broker disks and replicates it between brokers as usual, then moves closed segments to object storage. It shrinks disk bills and makes long retention affordable. It does not touch the write path, so cross-zone replication traffic is unchanged. Kafka 4.3 and 4.4 kept polishing it: followers can fetch from tiered offsets (KIP-1023) and uploads can be deliberately delayed with remote.copy.lag.bytes and remote.copy.lag.ms (KIP-1241).

Diskless topics go after the write path. The KIP-1150 page states the motivation bluntly: two of the three big clouds charge for cross-zone traffic, and the KIP cites AWS at $0.02 per GiB, so replicating every byte to two other zones is often the largest line on the bill. Diskless topics write straight to object storage, run alongside classic topics in the same cluster, and openly "will incur higher produce latency". The umbrella KIP was accepted on 2 March 2026 with 9 binding votes. The implementation KIP, KIP-1163 Diskless Core, still reads "Under Discussion" with no target release. As of November 2026, diskless is a direction for Apache Kafka, not a feature you can turn on.

That gap is exactly where the vendors live. The cross-zone networking story is covered in more depth in my AWS VPC explainer, and it is the reason "Kafka on S3" became a product category.

WarpStream vs AutoMQ vs Redpanda vs Apache Kafka

All of these speak the Kafka protocol. They differ in where bytes live, who owns the code, and what latency you accept.

Option	Architecture	License or model	Latest checked	Notable figure	Source
Apache Kafka	Brokers with local disks, optional tiered storage	Apache-2.0	4.4.0, 22 Oct 2026	Diskless accepted, not shipped	Downloads
Redpanda	Kafka API-compatible broker, not a Kafka fork	BSL 1.1, converts to Apache-2.0 four years after each release	v26.2.2, 22 Aug 2026	Licence bars offering it as a streaming service	Licence
AutoMQ	Kafka fork with storage on S3-compatible object stores	Apache-2.0	1.7.4, 29 Aug 2026	Vendor claims sub-10 ms P99	Releases
WarpStream	Stateless agents in your account, data in object storage	Commercial BYOC, owned by Confluent	n/a (SaaS control plane)	$0.01 per GiB written (first 50 TiB); P99 produce 500 ms	Pricing
Aiven Inkless	Apache Kafka fork implementing KIP-1150	Open source fork	inkless-release-0.51, 6 Nov 2026	Tracks the upstream diskless design	Repo

A few notes the table cannot carry. Confluent bought WarpStream in September 2024, and IBM completed its roughly $11 billion acquisition of Confluent on 17 March 2026, so the largest commercial Kafka vendor and the best-known diskless product now sit inside IBM. That is not a reason to avoid either, but it belongs in your vendor-risk notes. WarpStream's own pricing page puts regular produce latency at a 250 ms median and 500 ms P99, which is perfectly fine for logs, telemetry and lake ingestion and wrong for a payment authorisation path. AutoMQ's sub-10 ms figure is a vendor claim from its own site, and its own README says the open-source build running directly on S3 sees hundreds of milliseconds, with single-digit latency reserved for the enterprise version; I would benchmark before believing any number in this row, including WarpStream's.

My rule of thumb: if your traffic is high-volume, latency-tolerant and spread across zones (observability pipelines, CDC into a lake), object-storage designs can be dramatically cheaper and the latency cost is irrelevant. If you need tens of milliseconds end to end, stay on classic Kafka with tiered storage and put effort into rack-aware placement and fetch-from-follower instead.

When do you not need Kafka at all?

This is the section I wish more architecture reviews included. Kafka is the right tool when you have multiple independent consumers of the same stream, need replay, need ordering per key at high throughput, or are feeding stream processing. It is the wrong tool when you need a job queue for one service, a few hundred messages a second, or a way to decouple two microservices that a database table already decouples.

Cheaper answers, roughly in order of how often they are enough:

• Your existing Postgres. A jobs table with SELECT ... FOR UPDATE SKIP LOCKED handles a surprising amount of work, and it is transactional with your data. If you already run Postgres on Kubernetes (see my CloudNativePG guide), that is one less stateful system.
• A managed queue such as SQS or Pub/Sub when you are already on that cloud and do not need replay.
• NATS JetStream (v2.15.2, 26 October 2026) when you want lightweight streams and request-reply without Kafka's operational weight.
• Redis or Valkey streams for short-lived, memory-sized backlogs.

The tell that you have outgrown these is usually not throughput. It is the moment three teams want to consume the same events independently, at their own pace, with replay. That is the problem Kafka was built for.

Platform discipline matters more here than tool choice. A Kafka cluster without topic ownership, retention policies, schema rules and consumer-lag alerts becomes the most expensive shared drive in the company. Whether you self-host or buy, decide who owns topics and who gets paged for lag before the first producer goes live. The same reasoning I use for running databases on Kubernetes applies: the operator handles the mechanics, not the ownership.

FAQ

Does Kafka 4 still support ZooKeeper?

No. Kafka 4.0, released on 18 March 2025, was the first major release that runs entirely without ZooKeeper, and every 4.x release is KRaft-only. To migrate a ZooKeeper cluster you upgrade to 3.9, run the KRaft migration there, then move to 4.x.

Are Kafka queues (share groups) production-ready?

Yes, as of Kafka 4.2.0 (February 2026), which the Apache announcement describes as production-ready. They were early access in 4.0 and preview in 4.1. Kafka 4.4 added dead-letter queues for share groups, which need share.version set to 2.

Is diskless Kafka available in Apache Kafka?

Not yet. KIP-1150 was accepted as a direction on 2 March 2026, but the core implementation proposal, KIP-1163, is still under discussion with no target release. Today you get diskless behaviour from products such as WarpStream, AutoMQ or Aiven's Inkless fork.

What is the latest Apache Kafka version?

Apache Kafka 4.4.0, released on 22 October 2026. The supported lines are 4.4.0, 4.3.2 and 4.2.2. If you run on Kubernetes with Strimzi, check operator support first: Strimzi 1.3.1 tops out at Kafka 4.3.2.

What I would do this quarter

If I were running Kafka today, I would get every cluster onto KRaft and a supported 4.x line first, because ZooKeeper-era clusters are now on archived releases and the 3.9 bridge only gets further away. I would move the latency-tolerant, high-volume topics (logs, metrics, CDC to the lake) onto tiered storage now and price an object-storage option for them, while leaving low-latency topics on classic replication. I would try share groups for one work-queue use case before deleting any RabbitMQ. And for every new "we need Kafka" proposal, I would ask which three consumers need the same stream. If nobody can name them, the answer is a Postgres table. For the Kubernetes mechanics, the paired Strimzi setup guide walks through a KRaft cluster end to end.

Related: Should You Run Databases on Kubernetes in 2026?, AWS VPC Explained (2026): Subnets, NAT Gateways, Endpoints and Costs and Self-Hosting vs SaaS in 2026.