The event was delivered. The work was not.

account.closed fans out to five subscribers. Four finish. Session revocation never runs.

Email sends. Search hides the profile. Analytics counts one closed account. The customer can still sign in.

The publish succeeded. The closure did not.

That difference is easy to lose in an event-driven design. The producer sends one event and stops knowing who handles it. Each subscriber owns a separate retry history and a separate definition of success. A dashboard can show a healthy broker while the business operation remains open.

Fan-out did not create the problem. The problem began when somebody treated a subscriber as part of the promise without recording that promise anywhere.

Event fan-out removes the caller, not the obligation

Publish-subscribe lets a producer announce a fact without knowing every receiver. Microsoft describes that separation directly: the publisher sends through an intermediary and does not need to know which consumers receive the event.¹

A slow analytics consumer should not delay an account closure request. A temporary email outage should not force the account service to keep an HTTP connection open. Pub/sub removes both waits from the request.

Fault isolation also changes what success means. Producer success covers the handoff it controls. It cannot mean that every interested service finished, because the producer no longer waits for them.

invoice.paid can notify analytics, search, and a recommendation system. Those services observe a fact that is already true. Their failure does not make the invoice unpaid.

account.close.requested carries a different meaning when session revocation and billing closure must happen before the product calls the account closed. Those consumers are not merely interested. They owe work to the operation.

The event name cannot settle that distinction. The business contract does.

A broker can be healthy while the operation is not

Brokers remember transport state. RabbitMQ records publisher confirms and consumer acknowledgements as separate facts. Its documentation calls the two mechanisms orthogonal and unaware of each other.² Kafka stores one position per partition and consumer group.³ NATS JetStream advances a consumer's acknowledgement floor after a reader acknowledges a message.⁴

Each mechanism answers a different question:

State	Question it answers
publisher acknowledgement	Did the broker accept responsibility under this publish contract?
consumer offset or acknowledgement	Where should this consumer resume?
business completion record	Which required outcomes still remain for this operation?

The third row does not appear by configuring the first two. This example assumes the broker already stored the event for durable consumers.

Pull a subscriber out of service

Pull a plug. Compare storage with completion.

Broker

event stored

close_481

complete

account.closed event close_481

Revoke sessions required now done

Close billing required done

Remove from search required done

Send confirmation optional done

Record analytics observer done

Event stored. close_481 complete.

Pulling the email plug leaves close_481 complete because email is optional here. Pulling session revocation changes the result. Broker publish state does not change. Only the completion contract knows which missing effect blocks the closure.

Transport telemetry still matters. Kafka delivery guarantees can tell us when a group may replay a record. Consumer lag can reveal that billing is falling behind. Redelivery counts can expose a handler that never settles. None of those facts tells us whether one particular account may still sign in.

First decide who is allowed to fail

I classify each subscriber by what happens if it never processes the event.

Observers read a fact for their own purpose. Analytics often fits here. The source operation remains valid when the observer is unavailable.

Email belongs to the user experience but may not define the state transition. A confirmation can fail after the account has closed. The system should retry it, but reopening the account would make no sense.

Session revocation becomes an obligation when "closed" means old credentials cannot authorize another request. The operation has not kept its promise until that work finishes.

Order creates a workflow. Data erasure might wait for billing to retain the records it must keep. One consumer now needs a fact produced by another. A flat fan-out diagram hides that dependency.

These categories do not come from Kafka, RabbitMQ, or NATS. Teams assign them when they define the product behavior. Trouble starts when the code treats every subscriber alike while people expect different outcomes from them.

A required subscriber needs durable completion state

Required work needs a record that outlives the process which requested it. A process manager, workflow engine, or small operation table can own that record. The name matters less than the evidence it keeps.

close_481 might carry this state:

operation: close_481
state: waiting

required work
  revoke_sessions     done
  close_billing       retrying
  remove_from_search  done

optional work
  send_email          failed
  record_analytics    done

Consumers report outcomes with the same operation ID. The owner moves the operation to complete only after every required outcome reaches a terminal success state. A deadline can move it to needs_attention instead. The system no longer guesses completion from an empty queue.

This record does not make five databases atomic. Each consumer still needs idempotency and a safe local commit. The database and network boundary still exists inside every handler that calls another system.

Completion state solves a different problem. It remembers what remains after the original request process disappears.

Retries preserve work, not meaning

Redelivery gives unfinished transport another attempt. NATS JetStream redelivers after its acknowledgement wait expires. RabbitMQ requeues unacknowledged deliveries after a channel or connection closes. Kafka resumes a group from its committed position.

None of those actions decides whether a failed subscriber may be ignored.

A dead-letter queue does not decide either. It moves a delivery away from the hot retry path. close_481 remains unfinished when a required session-revocation message waits there. Somebody still needs a deadline, an owner, and a way to retry or repair the effect.

Delayed retries make the wait manageable. They do not turn time into completion. The bounded delay queue design explains how to keep retry topology finite. The operation record must still say what the retry is for.

Idempotency protects the next attempt from repeating a committed effect. It cannot promote an optional consumer into a required one or decide when the whole operation is done. Delivery, effect, and completion remain separate facts.

Choreography still needs a protocol

Choreography works well when services react independently. AWS recommends it for sagas with only a few participants and warns that dependencies become harder to track as participants grow.⁵ Microsoft makes the same boundary sharper: choreography becomes difficult when later work waits for several earlier services, and no component holds the full in-flight state by default.⁶

A central orchestrator is one answer, but it is not the only answer. A completion tracker can join independent results without directing every step. A saga can record progress and issue compensation after a later failure. One service can wait for two outcome events when the dependency is small and explicit.

Independent observers can keep their own delivery and recovery rules. Parallel obligations need a correlation ID and somewhere durable to join their results. Once order matters, workflow state must record what may run next. Compensation belongs only where the business accepts a later action as an honest repair.

Events can still carry every transition between those states. Event-driven does not have to mean coordinator-free.

The completion contract needs a version

Subscriber lists change. A sixth consumer may join next month to remove closed accounts from a new recommendation index.

The current topic subscription list should not redefine old operations. Otherwise a replay or deploy can produce a strange question: did close_481 become incomplete because a service that did not exist at the time has never handled it?

I freeze the required obligation set when the operation starts. close_481 records the contract version or the exact required steps. A later version can add recommendation cleanup for new closures. Historical backfill remains a separate job with its own progress.

A security or policy change may require work on old accounts too. That decision should create a backfill or add a new obligation deliberately. A new subscription should not rewrite history by accident.

Optional subscribers do not need that coupling. They can start from "now," replay retained history, or build from a snapshot. Their owners choose based on the data they need.

A new reader may be cheap. A new condition on an existing business promise changes its state machine, recovery path, and meaning of old work.

Monitor the work that has waited longest

Queue depth and consumer lag show pressure on a transport path. They cannot show close_481 waiting for session revocation.

I want the oldest operation with unfinished required work. I want its operation ID and the specific obligation that blocks it. A deadline should make the operation visible before a customer reports the partial state.

Correlation IDs help traces connect the fan-out. The operation record keeps the answer after logs expire and processes restart. An operator can still ask what must happen next.

An empty queue can coexist with an incomplete operation. A consumer may have acknowledged too early. A bug may have marked the wrong account. A remote service may have accepted a request while the local result disappeared. Completion should come from the state the business owns, not from absence of broker work.

One question before the next subscription

I ask this before adding a consumer:

If this subscriber never runs, is the original operation still complete?

"Yes" describes an observer or an optional consequence. Give it independent retry and monitoring.

"No" describes an obligation. Name its terminal state. Record which operation owns it. Decide its deadline and recovery path before publishing the first event.

Pub/sub can remove the caller from the wait. It cannot remove work from the promise.

Footnotes

1. Microsoft Azure Architecture Center, Publisher-Subscriber pattern. The pattern separates senders from receivers and lets each subscriber process an event independently. ↩

2. RabbitMQ 4.3 documentation, Consumer acknowledgements and publisher confirms. Publisher confirms cover the publisher-to-broker boundary. Consumer acknowledgements cover a separate broker-to-consumer boundary. ↩

3. Apache Kafka 4.3 documentation, Design: consumer position. Kafka records a compact per-partition position for each consumer group and lets groups consume the same topic independently. ↩

4. NATS documentation, JetStream delivery and acknowledgment. A durable consumer tracks delivered and acknowledged positions and redelivers when the acknowledgement deadline expires. ↩

5. AWS Prescriptive Guidance, Saga patterns. The guidance compares event choreography with a coordinator that records and directs workflow steps. ↩

6. Microsoft Azure Architecture Center, Choreography pattern. The guidance describes the missing end-to-end view, correlation needs, and dependency problems that appear in larger choreographed workflows. ↩