How a RabbitMQ backlog can block every publisher

A search indexer stops acknowledging events. Checkout is healthy. Billing is healthy. The broker is not.

An hour later, checkout cannot publish order.created. Not because search is on the request path. Not because the exchange waits for search to finish. The broker has run low on a resource and has blocked every publishing connection to protect itself.

That is the RabbitMQ failure worth understanding. “The consumer is slow” sounds local. Its backlog is not.

This can happen in any system that accepts work faster than it can retire it. Kafka has consumer lag. PostgreSQL replicas lag. A queue is not unusual for falling behind.

The shape of the debt is different. With a conventional RabbitMQ fan-out, each subscriber usually owns a queue and therefore its own stored copy of the outstanding work. One forgotten queue can keep accumulating state until a cluster-wide memory or disk alarm makes the problem everybody's.

The queue that stopped reading

Imagine one event routed to three durable queues:

orders exchange
  -> billing.events     900 in / 900 acked each second
  -> email.events       900 in / 900 acked each second
  -> search.events      900 in /   0 acked each second

The exchange does not wait for the search consumer. Publishing can continue while search.events grows by 900 messages every second. That is useful isolation at first. The producer stays fast and the healthy consumers keep working.

The unpaid bill is now sitting inside the broker.

For a durable replicated queue, the broker must store and replicate that queue's messages until the consumer acknowledges them or a policy removes them. Adding another subscriber means adding another queue lifecycle. RabbitMQ's own Streams guide calls this potentially inefficient for large persistent fan-outs because each subscriber needs a dedicated queue.

Turn the clock in the illustration. Search remains down. Its private queue consumes the same disk headroom every publisher depends on.

Turn the outage clock.

offline

Queues

open blocked

18%

Log

open

lag0

09:00

Search stops. The queue is empty.

The numbers are illustrative, not a capacity estimate. Message size, queue type, replication, ingress rate, acknowledgement rate, storage and version all change how quickly a real cluster reaches the boundary. The mechanism does not depend on the made-up clock: one queue owns the backlog, while the resource alarm applies beyond that queue.

The alarm crosses the queue boundary

RabbitMQ has two kinds of pressure that are easy to blur together.

Flow control throttles a publishing connection when the broker cannot pass messages through its internal pipeline quickly enough. A client may alternate between flow and running. This is backpressure along a path.

A resource alarm is broader. When a node crosses its memory watermark, RabbitMQ says the alarm is eventually propagated to the other cluster nodes and publishing connections are blocked. When one node crosses the free-disk limit, the disk alarm is cluster-wide: all nodes block incoming messages. The publisher guide is blunt about the effect. Every connection in the cluster that attempts to publish is blocked until every alarm clears.

So the incident can travel like this:

search dependency slows
  -> search consumer stops acknowledging
  -> search.events grows
  -> broker disk or memory crosses its limit
  -> resource alarm becomes cluster-wide
  -> checkout, billing and unrelated publishers block

The first arrow is local. The last one is shared infrastructure.

This is not an argument against alarms. Blocking publishers is safer than letting the broker fill the disk and terminate. It is an argument against treating a queue's capacity as private just because its consumer is private.

Kafka has lag too. It owes different state

If producers write 900 records per second and a Kafka consumer group reads none, that group falls behind by 900 records per second. Arithmetic did not disappear.

What changes is what the broker stores for the subscriber.

Kafka appends each record once to a partition log, apart from configured replicas. Consumer groups keep positions in that log. A second group mostly adds another offset, not another copy of every retained record. Kafka's current introduction describes topics as multi-subscriber and says records remain until the topic's retention policy removes them, independent of whether a consumer has read them.

A slow group therefore does not normally pin a private queue forever. It can suffer a different failure: retention keeps moving and the group can return after its unread offsets are gone. The publisher is isolated from that consumer, but the consumer is not promised infinite recovery time.

This is also why “RabbitMQ versus Kafka” is too coarse. RabbitMQ Streams is an append-only, replicated log with non-destructive reads and retention. It was built for large fan-outs, replay, high throughput and large backlogs. The conventional classic or quorum queue behavior in this article is not a description of every RabbitMQ data type.

Kafka can still run out of disk. A bad retention policy, an ingest spike or failed storage can still stop a log-based cluster. The narrower claim is about subscriber state: ordinary Kafka lag does not create one separately retained queue per group.

“Dropped messages” is not a diagnosis

The pressure incident explains why publishers block. It does not, by itself, explain a missing message.

RabbitMQ can preserve a message correctly while an application loses it at another boundary. During an incident I would ask where the evidence ends:

Boundary	What can disappear	Evidence to require
publisher to broker	a write whose outcome was unknown after timeout or disconnect	publisher confirm, stable message ID and a retry decision
exchange to queue	an unroutable AMQP 0-9-1 message published with `mandatory=false`	returned-message handler or alternate exchange, plus unroutable metrics
inside the queue	messages removed by TTL, length limit, overflow or an operator action	effective policies, dead-letter path and audit trail
broker to consumer	work sent with automatic acknowledgement before processing completed	manual acknowledgement after the effect is durable
consumer side effect	a database write or API call that succeeded before the process crashed	idempotency record and domain reconciliation

RabbitMQ documents each of the first four behaviors. A socket write is not proof that the broker accepted the message, which is why publisher confirms exist. An unroutable message is discarded by default when mandatory is false and there is no alternate exchange. A configured queue limit drops or dead-letters older messages by default, while reject-publish can reject new ones. Automatic consumer acknowledgements trade safety for throughput and can lose work when the connection closes before the consumer actually finishes.

The positive guarantee matters too. RabbitMQ says a message confirmed by a quorum queue should survive as long as a majority of its members are not permanently lost. A report that “RabbitMQ dropped it” is incomplete until it says whether the publish was confirmed, which queue accepted it, how that queue was configured, and whether the consumer acknowledged it.

That is why I would not copy the headline from the Nanit migration case study. The published account says RabbitMQ bottlenecks led to dropped messages and reliability problems, but it does not provide the RabbitMQ version, queue type, topology, confirm mode, acknowledgement mode or overflow policy. It is useful evidence that their queue-based architecture stopped fitting their needs. It is not enough evidence to assign the loss to one broker behavior.

A real queue backup rarely stays polite

Reddit Engineering's notifications migration gives the failure more texture. Their legacy pipeline handled roughly 20,000 to 25,000 messages per second. By the end of 2022, RabbitMQ backups were paging engineers once or twice a week. They responded by stopping message production to avoid an out-of-memory crash.

The triggers were not one tidy broker bug. They included slow dependencies, a slow database and input spikes. The team also saw connections enter RabbitMQ's flow-control state and propagate the slowdown upstream. In another incident shape, RabbitMQ memory grew while visible queue length did not. They had a theory involving connection management and scaling, but wrote that they never found the root cause.

That last sentence is good operational writing. They reported the observation and the theory separately.

Reddit did not replace RabbitMQ with bare Kafka and declare victory. Their workload needed long retries, tens of millions of queued messages, high parallelism and state for individual work items. Kafka's partition log gave them durable backlog storage, but not all the queue semantics they needed. They built Kafqueue on top of it, paying new complexity and latency for that state.

The senior lesson is not “Kafka is better.” It is that persistent backlog is part of the product. Choose the storage model for the worst recovery period, not the happy-path message rate.

Why logs carry this workload differently

There is no magic throughput property attached to the word log. The advantage comes from work the storage model can avoid or batch.

Kafka appends batches to partition files, reads large sequential chunks, keeps consumer position as an offset per partition and preserves compressed batches through the broker. Its current design documentation describes the practical results: larger sequential I/O, fewer network round trips, less byte copying and small per-group position state.

Conventional queues keep richer per-message lifecycle state. A message may be ready, delivered but unacknowledged, requeued, dead-lettered, expired or deleted after an acknowledgement. That state buys flexible work distribution and per-message settlement. It also means the broker does more than append a record and advance a reader's integer.

RabbitMQ itself says its persistent queue types do not match log-based systems for stream throughput and that most queues are optimized to converge toward empty. That is a workload statement, not a universal benchmark. Modern RabbitMQ queues are not frozen in an old failure profile. RabbitMQ 3.12, for example, changed classic queue storage and improved long quorum queue behavior. Hardware, replication, message size, acknowledgements and client batching can reverse a casual benchmark.

The fair comparison is not product against product. It is queue semantics against log semantics under the backlog and fan-out you actually expect.

Adding consumers may move the bottleneck

If the search consumer is CPU-bound and work is independent, more consumers can drain the queue. RabbitMQ exposes consumer capacity as a hint that additional consumers or higher prefetch might help.

But a slow downstream database does not become faster because more workers call it. More consumers can increase lock contention, exhaust its connection pool or turn a recoverable slowdown into an outage. Raising prefetch can move messages from ready to unacknowledged without increasing acknowledgements. The dashboard looks different while the debt stays.

The rate that matters is not deliveries per second. It is durable acknowledgements per second after the required effect completes.

Measure at least these together:

publish rate and confirm latency
ready and unacknowledged bytes, not only message counts
acknowledgement rate and age of the oldest ready message
memory and disk headroom on every broker node
blocked connections and time spent in flow control
downstream latency, saturation and error rate

Then load test the recovery, not just steady state. Stop one consumer for the longest outage you claim to survive. Keep realistic message sizes and replication. Bring it back while live traffic continues. Observe how long the queue takes to drain and what happens to confirm latency, disk, memory and the downstream service.

If that exercise can exhaust the cluster, the queue has no meaningful recovery window yet.

Pick the debt you are prepared to own

Use a conventional RabbitMQ queue when messages represent work that should disappear after acknowledgement, consumers compete for jobs, per-message retry and routing matter, and backlogs are bounded by design.

Use a log such as Kafka, Redpanda or RabbitMQ Streams when many independent readers need the same history, replay is normal, a consumer may be offline for a long time, and retention is easier to reason about than one queue per subscriber.

Sometimes both belong in the same system. A durable log can hold the history while a queue schedules retryable work. That does not remove coordination. It makes each storage structure pay for the semantics it is good at.

Whichever model you choose, write down the failure contract:

maximum offline time × publish bytes per second × replication overhead
  < storage headroom reserved for recovery

Then decide what happens at the boundary. Do you block upstream, reject with a retriable error, expire old work, spill elsewhere, or shed an optional subscriber? “The broker handles it” is not a policy.

One slow consumer should be allowed to become late. It should not be allowed to become an unnamed cluster-wide capacity decision.

One slow consumer can stop every RabbitMQ publisher