Delayed delivery without unbounded queues

The delayed-job API looked harmless: give it a duration and it would run the job later. The broker paid for that flexibility by creating a queue for each distinct delay.

Seven seconds needed a queue. Thirteen seconds needed another. A retry policy with jitter created more. A product feature that accepted arbitrary scheduling values quietly turned user input into RabbitMQ topology.

That is the wrong growth curve. Messages should grow with traffic. Broker objects should grow with architecture.

The replacement is a bounded binary delay ladder. Create one queue for each power of two, encode the requested delay as bits, and let per-queue message TTL plus dead-letter routing move the message through only the levels it needs. Any delay inside the chosen range becomes representable, while the number of queues never changes.

I used this pattern in a Sneakers-backed Active Job adapter. The adapter is not the interesting part. The useful idea is that an arbitrary application delay can map onto fixed broker infrastructure.

The pattern in one line: represent time as data in the routing key, not as a new queue in the broker.

The queue-per-delay design turns data into infrastructure

A queue with a fixed message TTL and a dead-letter exchange is a clean building block. Every message has the same configured residence limit. Once it has spent that long in the queue, RabbitMQ can expire and dead-letter it.¹

The trouble starts when the application creates one such queue for every requested duration.

flowchart LR
  subgraph U["Unbounded: requested values become topology"]
    D7["delay = 7s"] --> Q7["queue.7s"]
    D13["delay = 13s"] --> Q13["queue.13s"]
    D42["delay = 42s"] --> Q42["queue.42s"]
    DN["delay = n"] --> QN["queue.n"]
  end

  subgraph B["Bounded: requested values become routes"]
    A7["delay = 7s"] --> R["binary encoder"]
    A13["delay = 13s"] --> R
    A42["delay = 42s"] --> R
    AN["delay = n"] --> R
    R --> L["fixed power-of-two ladder"]
  end

  classDef bad fill:#f8ded7,stroke:#c84a43,color:#5b2825
  classDef good fill:#dce8df,stroke:#466b5c,color:#29463b
  class Q7,Q13,Q42,QN bad
  class R,L good

Figure 1: Arbitrary delay values create broker objects in the unbounded design. The binary design maps every value onto one fixed ladder.

If the system sees N distinct delay values, the queue-per-delay model can create N queues. The input domain controls operational state. Jitter makes it worse because it deliberately increases the number of distinct values.

Putting per-message TTLs into one shared queue does not fully solve the problem. RabbitMQ expires messages when they reach the head, so an expired short-delay message can sit behind a longer-lived message and continue consuming resources.² A per-queue message TTL avoids that ordering mismatch because every message in a level has the same residence time.

The binary ladder changes the relationship. For a maximum delay D, measured in a chosen base unit, the topology needs only:

levels = ceil(log2(D + 1))

That is logarithmic growth. A 20-level ladder represents 1,048,575 positive values. Adding another level doubles the range instead of adding one more supported timeout.

The pattern uses one queue for every binary place

Each level represents a power of two: 1, 2, 4, 8, 16, and so on. A queue at level k has a per-queue message TTL of 2^k base units and dead-letters into the next lower exchange.

The public lineage of this design is useful. Particular documented a 28-level RabbitMQ delayed-delivery topology for NServiceBus.³ Kombu later added the same native delayed-delivery shape.⁴ Celery now enables it for ETA and countdown tasks when RabbitMQ quorum queues are detected.⁵ Kombu's current implementation declares 28 levels, assigns each queue a power-of-two TTL, and encodes the countdown as a 28-bit routing-key prefix.⁶

This is not specific to Celery, NServiceBus, Sneakers, or Active Job. Those libraries are examples of the same pattern:

• A bounded set of levels defines the delay range.
• A base unit defines the resolution.
• A binary route says which levels a message must wait in.
• A destination suffix survives the delay path and selects the final queue.
• TTL and dead-letter routing move the message without an application timer process.

The broker becomes a small delay computer. It does not calculate time dynamically. It routes through a fixed circuit whose waiting periods add up to the requested delay.

Forty-two seconds becomes three waits

The binary form of 42 is 101010:

42 = 32 + 8 + 2
42 = 1×2⁵ + 0×2⁴ + 1×2³ + 0×2² + 1×2¹ + 0×2⁰

The message waits in the 32-second queue, bypasses 16, waits in 8, bypasses 4, waits in 2, bypasses 1, then reaches its destination. It uses three waiting queues even though the ladder contains six levels.

sequenceDiagram
  participant P as Publisher
  participant E5 as Level 5 exchange
  participant Q32 as 32s queue
  participant E4 as Level 4 exchange
  participant E3 as Level 3 exchange
  participant Q8 as 8s queue
  participant E2 as Level 2 exchange
  participant E1 as Level 1 exchange
  participant Q2 as 2s queue
  participant E0 as Level 0 exchange
  participant D as Delivery exchange

  P->>E5: 1.0.1.0.1.0.destination
  E5->>Q32: bit 5 is 1
  Note over Q32: wait 32 seconds
  Q32-->>E4: TTL expires, dead-letter
  E4->>E3: bit 4 is 0, bypass
  E3->>Q8: bit 3 is 1
  Note over Q8: wait 8 seconds
  Q8-->>E2: TTL expires, dead-letter
  E2->>E1: bit 2 is 0, bypass
  E1->>Q2: bit 1 is 1
  Note over Q2: wait 2 seconds
  Q2-->>E0: TTL expires, dead-letter
  E0->>D: bit 0 is 0, deliver

Figure 2: A 42-second message waits at the 32-, 8-, and 2-second levels. Zero bits bypass their queues.

The highest set bit is also the best entry point. Publishing 42 directly to level 5 avoids walking through unused higher levels. At every level, a topic exchange makes one decision: 1 enters the waiting queue; 0 passes directly to the next exchange.

Try the same decomposition with other values. The model below keeps the topology fixed at 12 queues while representing 4,095 positive delays.

LIVE MODEL // 12-BIT DELAY LADDER

Build a delay from fixed queues

Change the delay. The topology stays at 12 queues.

12 queues

Delay Unit seconds minutes

Try 10 42 300 1023

42 seconds = 32 + 8 + 2

0 2¹¹ 2048s

0 2¹⁰ 1024s

0 2⁹ 512s

0 2⁸ 256s

0 2⁷ 128s

0 2⁶ 64s

1 2⁵ 32s

0 2⁴ 16s

1 2³ 8s

0 2² 4s

1 2¹ 2s

0 2⁰ 1s

The message waits only at active levels

1. LEVEL 05 32s TTL, then dead-letter
2. LEVEL 03 8s TTL, then dead-letter
3. LEVEL 01 2s TTL, then dead-letter
4. FINAL delivery exchange route to destination

Representable positive delays 4,095

Waiting queues for this message 3

Queues created for the ladder 12

This teaching model uses 12 levels. A production ladder chooses its level count and base unit from the maximum supported delay and required precision.

The maximum number of waits for one message is the number of 1 bits, often called the population count. The maximum number of routing levels is fixed by the ladder. Neither depends on how many distinct delay values users request.

The routing key carries both time and destination

A useful routing key has two parts:

<delay bits>.<original destination>

For a six-level teaching ladder, a 42-second message for email.delivery could use:

1.0.1.0.1.0.email.delivery

Each level binds against the bit at its own position. The final exchange ignores the bit prefix and matches the destination suffix. This separates two concerns without losing either:

• The prefix determines when the message leaves the delay infrastructure.
• The suffix determines where the message goes afterward.

The destination binding is part of correctness. A perfect delay path that ends at an unbound final exchange still loses the message. I prefer three defenses: declare destination bindings from the consumer, ensure them before delayed publish where practical, and configure an alternate exchange that sends only unroutable messages to a visible parking queue.⁷

A normal # binding is not a parking path. It would match valid destinations too and copy ordinary delayed traffic into the parking queue.

That last queue is not another delay level. It is a fault boundary. Its normal depth is zero, and any message in it deserves investigation.

A delay ladder is not a reliability guarantee

The ladder answers one question: how can arbitrary delays use bounded broker topology? It does not answer every delivery question.

flowchart LR
  P["Publisher"] -->|"publisher confirm"| X["highest active level"]
  X -->|"bit = 1"| Q["TTL queue"]
  X -->|"bit = 0"| N["next level"]
  Q -->|"dead-letter after TTL"| N
  N --> F["final delivery exchange"]
  F -->|"destination binding"| W["worker queue"]
  F -. "alternate exchange if unroutable" .-> K["parking queue + alert"]
  W -->|"ack after success"| C["consumer"]
  C --> I["idempotent effect or durable claim"]

  classDef boundary fill:#f7f1e3,stroke:#2e4055,color:#2e4055
  classDef safe fill:#dce8df,stroke:#466b5c,color:#29463b
  classDef warn fill:#f8ded7,stroke:#c84a43,color:#5b2825
  class P,X,Q,N,F,W,C boundary
  class I safe
  class K warn

Figure 3: Bounded topology solves delay routing only. Publish, dead-letter, final-routing, acknowledgement, and side-effect boundaries still need separate guarantees.

Five boundaries still need explicit decisions.

Entry into the broker

Use publisher confirms if the producer must know that RabbitMQ accepted the message. A successful application call before a confirmed publish is still a loss window.

Movement between levels

Dead-lettering is an internal republish. RabbitMQ documents that ordinary dead-lettering does not use internal publisher confirms by default, so a target outage can lose the message after it leaves the source queue. Quorum queues support at-least-once dead-lettering with internal confirms.⁸

Exit from the ladder

The final exchange needs a binding for every valid delayed destination. Monitor unroutable messages or attach an alternate exchange backed by a parking queue. Do not infer safety from the delay queues being healthy.

Consumer acknowledgement

A late acknowledgement protects against a worker crash before completion, but it also permits redelivery. The job must tolerate the same message arriving again.

External side effects

At-least-once transport cannot create exactly-once payments, emails, or API calls. Use an idempotency key, a durable claim, an outbox, or a business-level uniqueness constraint where duplicates matter. This is the same separation I use when giving Redis claims explicit ownership and failure rules.

Time resolution and delay horizon are design inputs

Do not copy a 28-level topology because another library chose it. Pick the smallest base unit and horizon your product actually needs.

For a base unit U and L levels:

resolution = U
maximum delay = (2^L - 1) × U

Base unit	Levels	Maximum representable delay	Good fit
1 second	12	4,095 seconds, about 68 minutes	short retries and cooldowns
1 second	20	1,048,575 seconds, about 12.1 days	mixed retries and scheduled jobs
1 minute	16	65,535 minutes, about 45.5 days	coarse reminders
1 minute	20	1,048,575 minutes, about 2 years	long-horizon, minute-precision delivery
1 second	28	268,435,455 seconds, about 8.5 years	the Particular and Kombu range

If early delivery is unacceptable, convert a duration to units with ceiling, not rounding:

encoded units = ceil(requested duration / base unit)

A 1.2-second request on a one-second ladder becomes two seconds. The pattern provides a not-before boundary, not an exact appointment. Broker load, dead-letter processing, routing, and consumer backlog can all make delivery later.

Reject values beyond the supported horizon. Silently truncating the high bits turns a long delay into a much shorter one, which is one of the worst possible failure modes for a scheduler.

Operate the ladder as one subsystem

The topology is fixed, so its telemetry can stay fixed too. Useful measurements include:

• ready message count and oldest age by level
• publish-confirm failures at the entry exchange
• dead-letter forwarding failures
• messages reaching the parking queue
• final-exchange unroutable counts
• end-to-end delivery lateness, measured against the requested not-before time
• destination binding coverage
• delayed message volume by coarse delay band, not raw requested duration

Avoid a metric label for the full routing key or requested delay. Those values are unbounded. The level number, outcome, destination class, and broad error type are bounded dimensions.

The alerting model should reflect the path. A deep 32-second level may be normal during a traffic spike. A message older than its level TTL plus a reasonable forwarding allowance is not. A non-empty parking queue is immediately actionable because its expected value is zero.

Operational commands should also treat the ladder as a unit. Provision, inspect, migrate, and remove the exchanges, queues, policies, and bindings together. Half a ladder is not degraded capacity. It is incorrect routing.

Migrate without stranding delayed messages

Existing delayed messages may outlive several deployments, so a cutover cannot replace the old topology in place.

1. Measure the old horizon. Find the latest possible delivery time already stored in the existing queues.
2. Declare the full ladder first. Verify every level, TTL, dead-letter target, and final destination binding.
3. Canary new publishes. Send synthetic delays with known destinations and verify the path, timing, and final consumption.
4. Switch only new delayed publishes. Let old messages continue draining through the old queues.
5. Watch both systems for one full old horizon. Compare delayed publish counts with final deliveries and parking outcomes.
6. Remove old queues only when empty and past the horizon. Queue age matters more than deployment age.

This is one place where compatibility code is worth keeping temporarily. New and old delayed paths can coexist as long as their routing names do not collide and both end at valid destinations.

When this pattern is the wrong tool

The binary ladder is useful when the broker must hold arbitrary not-before delays using fixed, native queue primitives. It is not a universal scheduler.

flowchart TD
  A{"Need delayed work?"} -->|"No"| N["publish normally"]
  A -->|"Yes"| B{"Need cancellation, rescheduling, or queries?"}
  B -->|"Yes"| S["durable scheduler or workflow store"]
  B -->|"No"| C{"Only a few fixed retry delays?"}
  C -->|"Yes"| F["small fixed retry buckets"]
  C -->|"No"| D{"Broker has suitable native scheduled delivery?"}
  D -->|"Yes"| V["use the native feature"]
  D -->|"No"| E{"Arbitrary delay inside a fixed horizon?"}
  E -->|"Yes"| L["bounded binary delay ladder"]
  E -->|"No"| S

  classDef chosen fill:#dce8df,stroke:#466b5c,color:#29463b
  classDef store fill:#f8ded7,stroke:#c84a43,color:#5b2825
  class L chosen
  class S store

Figure 4: The binary ladder fits arbitrary not-before delays inside a fixed horizon. Queryable or editable schedules belong in a durable scheduler.

Use a database-backed scheduler or workflow engine when users need to list, cancel, or edit future work. Use a few explicit retry buckets when the policy has only three values. Use a broker's supported delayed-message feature when it already meets the durability, horizon, and operational requirements.

The ladder earns its complexity when delay values are genuinely arbitrary, plugins are unavailable or undesirable, and keeping broker topology bounded matters.

Design review checklist

Before adopting the pattern, answer these questions in writing:

• What is the base time unit?
• What is the maximum supported delay?
• Are out-of-range values rejected?
• Is duration conversion rounded up?
• Which queue type and dead-letter safety mode are used?
• How does the producer confirm entry into the broker?
• Who declares final destination bindings?
• Where do unroutable messages go?
• How are delayed messages moved during a broker migration?
• When does the consumer acknowledge?
• Which jobs require durable idempotency?
• Which bounded metrics prove every stage is moving?

If those answers are missing, binary routing will only make an unreliable system more elegant.

Where I landed

The important change was not replacing many queues with fewer queues. It was deciding that application data could no longer create broker topology.

A binary ladder gives arbitrary delays a fixed operational shape. Powers of two provide the range. Per-queue message TTLs provide the waits. Dead-letter exchanges provide movement. The routing key carries time and destination. Publisher confirms, safe dead-lettering, final bindings, acknowledgements, and idempotency provide the reliability around it.

The job API can still say “run this in 42 seconds.” RabbitMQ only needs to know about 32, 8, and 2.

That is the pattern: unbounded values, bounded infrastructure.

Sources

Footnotes

1. RabbitMQ, Dead Letter Exchanges, retrieved 2026-08-24. RabbitMQ dead-letters messages after expiry, rejection, queue overflow, or a quorum delivery limit. ↩

2. RabbitMQ, Time-To-Live and Expiration, retrieved 2026-08-24. Per-message expirations are removed at the head of a queue, so expired messages can remain behind longer-lived messages. ↩

3. Particular Software, RabbitMQ Delayed Delivery, retrieved 2026-08-24. The documented topology uses 28 binary levels, power-of-two per-queue message TTLs, topic routing, and a final delivery exchange. ↩

4. Kombu, 5.5.0 changelog, retrieved 2026-08-24. Kombu 5.5 added native delayed delivery for RabbitMQ, enabling Celery ETA tasks with quorum queues. ↩

5. Celery, Using RabbitMQ: Native Delayed Delivery, retrieved 2026-08-24. Celery enables Native Delayed Delivery for ETA and countdown tasks when quorum queues are detected and credits the design to NServiceBus. ↩

6. Kombu, native delayed delivery source, retrieved 2026-08-24. The implementation declares 28 levels and prefixes the destination routing key with a zero-padded 28-bit countdown. ↩

7. RabbitMQ, Alternate Exchanges, retrieved 2026-08-24. An alternate exchange receives messages only when the original exchange cannot route them to a matching destination. ↩

8. RabbitMQ, Dead-lettering safety, retrieved 2026-08-24. Default internal dead-letter republishing does not use publisher confirms; quorum queues can use at-least-once dead-lettering. ↩