Queues for Apache Kafka went GA earlier this year: Confluent announced share consumers reaching general availability on Confluent Platform 8.2 and Confluent Cloud, built on KIP-932, which landed as production-ready in Apache Kafka 4.2. The one-line version: queue semantics implemented natively at the broker level, so multiple consumers can cooperatively process the same topic regardless of how many partitions it has.
I have run Kafka in production for years: event-driven pipelines processing 5M+ events per day at a crypto exchange, and today the high-throughput ingress of a multi-tenant SaaS platform serving roughly 2k requests per second. In both systems, the partition count was never just a storage detail; it was a capacity decision with long consequences. That is the lens I read this release through, and I wrote about the general version of that decision in choosing the axis you scale on.
The partition tax#
Classic consumer groups map each partition to exactly one consumer in the group. That single rule quietly shaped a decade of Kafka operations. Your maximum parallelism is decided when you create the topic, so you over-partition as insurance, and pay for it in metadata, rebalances, and broker load. One slow message blocks everything behind it in its partition, which is head-of-line blocking by design. And when you finally need more consumers than partitions, the extra consumers stand idle. Every Kafka team I have worked with has done the same ritual: guess the consumer count you will need in two years, multiply for safety, and hope.
What share groups actually do#
A share consumer does not own a partition. When it fetches records, the broker takes a time-limited acquisition lock on each record (30 seconds by default) for that consumer. The consumer then does one of four things per message: acknowledge it as processed, release it for retry, reject it as unprocessable, or renew the lock to buy more processing time. If it does nothing, the lock expires and the record becomes available to another consumer. The broker counts delivery attempts per record, up to a configurable maximum of five. This is per-message acknowledgment, the thing traditional queues always had over Kafka, now speaking the Kafka protocol:
| Consumer groups | Share groups (KIP-932) | |
|---|---|---|
| Parallelism unit | Partition: one consumer per partition, extras idle | Worker: any number of consumers on the same topic |
| Acknowledgment | Offset commit, effectively per batch | Per message: acknowledge, release, reject, or renew |
| Ordering | Strict within a partition | Sacrificed for elastic scaling |
| Retry story | Seek and reprocess from an offset | Broker-tracked redelivery, max five attempts; DLQ arrives later with KIP-1191 |
| Depth signal | Consumer lag per partition | Share lag, the queue-depth analog |
| Best for | Pipelines, ETL, stream processing, anything ordered | Work queues, job processing, command handling |
Adoption is additive rather than disruptive: share consumers can attach to existing topics, even ones a classic consumer group is reading at the same time. You do not migrate; you add a different kind of reader.
The fine print at GA#
None of these are hidden, which I respect. But they draw the boundary clearly. A payments pipeline keyed by account, an event-sourced system, anything where the sequence of events is the meaning of the data: those stay on consumer groups, full stop. The five-attempt redelivery without a DLQ also means that until KIP-1191 lands, your poison messages need a plan that you write yourself.
What this changes about capacity planning#
Here is the part that matters to me as someone who has sized these systems. With consumer groups, partition count is a day-one bet about year-two parallelism, and repartitioning later is a project. With share groups, consumers scale like a stateless worker pool: need more throughput on the queue, add workers; traffic drops, remove them. That moves Kafka consumers into the same operational model as the horizontally scaled services in front of them, and it pairs naturally with per-message acknowledgment as a backpressure mechanism, a theme I wrote about in push systems and slow consumers: the consumer that stops acknowledging is the consumer that stops receiving.
What I would do with it#
- Inventory your consumers: anything that is really a work queue wearing a consumer-group costume (job runners, command handlers, notification senders) is a share-group candidate.
- Keep every ordered pipeline on consumer groups; the tradeoff is not subtle and Confluent does not pretend it is.
- Plan the poison-message path yourself until KIP-1191 ships a real DLQ.
- Watch share lag the way you watch queue depth, because that is what it is.
- For new topics feeding work queues, stop over-partitioning as insurance; partition for storage and broker throughput, and let the share group handle parallelism.
Partition count was always two decisions wearing one config value: how to store the data and how many workers you could ever have. Share groups finally let you make them separately.