Exactly once, where?

I don't trust a delivery guarantee until I can point at the state that makes it true.

A consumer reads Charge order 481 from Kafka. It charges the card, then dies before it commits its offset. Another consumer starts from the last committed offset and charges the card again.

Kafka still contains one record. The customer has two charges.

Which Kafka guarantee failed?

None.

That is the problem with saying Kafka provides at-least-once or exactly-once delivery. Delivery sounds like one event. It is several events owned by different systems, and each system can prove only its part.

Kafka's own design notes split the problem into publishing durability and consumer processing.¹ I think we need to split it further. The producer acknowledgement, replicated log, fetched record, committed consumer offset, and external side effect are separate facts.

This post follows those facts. The labels make sense only after the crash windows do.

Delivery is not one event

One call to producer.send() starts a chain of questions:

1. Did the producer put the request on the network?
2. Did the partition leader append the batch?
3. Did enough replicas copy it?
4. Did the producer receive the acknowledgement?
5. Did a consumer fetch the record?
6. Did the application finish its work?
7. Did the consumer group save its new position?

People often compress all seven into "Kafka delivered the message." That sentence hides the failure boundary.

Kafka can answer the middle questions because it owns the log and the consumer offsets. It cannot inspect a card processor, an email provider, or a PostgreSQL transaction and decide whether the business operation completed. When one event fans out to several services, delivery and business completion need separate state.

My rule is narrower:

Name the durable state behind the claim. If you cannot name it, you do not have a guarantee yet.

A producer acknowledgement proves less than it sounds

acks controls when the producer treats a write as successful. It does not describe consumer processing, and it does not prevent duplicate application intent.

The three values answer different questions:²

Producer setting	When the send can complete	What can still happen
acks=0	The client writes to its socket buffer	The broker may never receive the record
acks=1	The leader appends the record	The leader can fail before a follower copies it
acks=all	Every current in-sync replica acknowledges it	The guarantee still depends on the size of the ISR

acks=all is easy to misread. It means all replicas in the current in-sync replica set, not every replica assigned to the partition forever.

Imagine a partition with replication factor three. Two followers fall out of the ISR, so only the leader remains. acks=all can still succeed when min.insync.replicas=1. One machine now holds the acknowledged write.

Set min.insync.replicas=2, and the broker rejects that write instead. Availability drops, but the producer cannot receive success from a one-replica ISR. Kafka documents these two settings as one durability decision, not independent magic switches.³

An acknowledgement proves that the write crossed the configured replication boundary. It does not prove infinite retention. A topic can delete the record later because of time, size, or compaction policy. It also does not prove that the caller received the acknowledgement.

Retries begin with an unknowable result

A producer sends a batch. The connection closes before the response arrives.

Two histories now look identical from the producer:

History A: request lost before append
History B: batch appended, acknowledgement lost

Giving up avoids a duplicate in History B, but loses the record in History A. Retrying protects History A, but can append the same batch again in History B.

Kafka's idempotent producer gives the broker enough identity to tell a retry from a new batch. Kafka assigns a producer ID. Each batch carries a producer epoch and a sequence number for its topic partition. The broker remembers the expected sequence for that producer and partition.⁴

A repeated sequence is not new work. The broker rejects the duplicate and lets the client treat the retry as successful. A gap means the broker missed something, so the producer cannot quietly continue.

This is real exactly-once behavior, but the noun matters. Kafka writes one copy of that producer batch to that partition.

Kafka does not hash the payload and decide that two Charge order 481 records express the same business intention. An HTTP handler can run twice and call send() twice. Two workers can publish the same order. Kafka accepts both because both are new producer operations.

Producer idempotence removes duplicates caused by the producer protocol. Business idempotence still belongs to the application.

Committed is a log word

A Kafka partition has one leader and an ordered sequence of offsets. Followers copy the leader's log. Kafka considers a write committed after the current ISR receives it and the ISR satisfies the configured minimum.³

That definition gives failover meaning. Kafka can elect another in-sync replica and retain the committed prefix. unclean.leader.election.enable=true changes the bargain by allowing a replica outside the ISR to become leader, which can lose committed records.³

Kafka also gives order a narrow scope. A partition is totally ordered. A topic with eight partitions has eight ordered logs, not one global order.¹

Two events for the same entity usually use the same key so the producer chooses the same partition. Events in different partitions can reach consumers in either order. Wall-clock timestamps do not change that. A transaction across partitions can make records commit together, but it does not invent one global sequence between them.

Retries have an ordering edge too. Multiple batches can remain in flight on one connection. If an earlier batch fails while a later batch succeeds, retrying the first can reverse their log order when idempotence is off. Kafka's current producer rules preserve ordering with idempotence enabled and cap max.in.flight.requests.per.connection at five for that mode.²

Durability, deduplication, and ordering touch the same send path. They are still different guarantees.

An offset is a bookmark, not a receipt

A consumer fetches records after its current position. Fetching does not remove them from Kafka. It also does not tell the broker that the application finished anything.

Consumer groups store their committed positions in Kafka's compacted __consumer_offsets topic. The group coordinator writes the next offset for each partition, then returns that position when a consumer restarts or another member takes over.⁵

Suppose the group stores offset 81. The next record to process is 81.

The consumer has two obvious choices:

fetch 81 -> charge card -> commit 82
fetch 81 -> commit 82 -> charge card

The first order can charge twice. A crash after the charge leaves offset 81 in Kafka, so a restart fetches 81 again.

The second order can lose work. A crash after committing 82 makes the group resume after the record, even though nobody charged the card.

Kafka cannot choose the correct lie. One order prefers replay. The other prefers skipping. An at-least-once consumer processes first and commits later, then makes the effect safe to repeat.

Kill the consumer after the charge.

Kafka record 81  charge order_481

Group resumes at

81

Payment ledger

no charge

Process 81

1 record. 0 charges.

Kafka still stores one record. The second charge comes from running it again because the group never reached 82.

A committed offset therefore proves one thing: the group should resume from this position. It does not prove that a database row, payment, or email agrees.

Kafka transactions close one specific gap

Kafka transactions join Kafka writes with Kafka consumer progress. They do not wrap any arbitrary code that happens between beginTransaction() and commitTransaction().

A consume, transform, and produce loop can do this:

read records from input topic
begin Kafka transaction
write derived records to output topics
write the consumed offsets into the same transaction
commit Kafka transaction

Kafka commits the output records and offsets together, or aborts both. A consumer using read_committed hides records from aborted transactions. Kafka also holds that consumer at the last stable offset while an earlier transaction remains open.⁶

transactional.id gives a logical producer identity across process restarts. Kafka assigns producer epochs and fences an older producer instance when a newer instance claims that identity. The transaction coordinator records transaction state. Brokers append commit or abort markers to participating partitions. Consumers use those markers when they decide which records read_committed may return.⁴

Consumer offsets work here because Kafka stores them as records too. The producer sends the next offsets to its transaction. One Kafka commit can now cover the output records and the input positions.

That is a strong guarantee. It solves a Kafka-to-Kafka state transition.

It does not turn a transaction into an envelope that every consumer receives whole. A consumer can subscribe to only one participating partition, seek into the middle, or stop after part of the data. KIP-98 calls out those limits directly.⁴

Exactly-once processing in Kafka means that failures do not make a committed input contribute duplicate committed Kafka output. It does not mean every observer sees one indivisible multi-partition object.

A database or API sits outside the transaction

Put a PostgreSQL update or payment request in the middle of that loop and Kafka loses the ability to commit everything together.

begin Kafka transaction
charge card
write Kafka output
commit Kafka transaction

The card charge does not roll back when the Kafka transaction aborts. Kafka's coordinator cannot write an abort marker into the payment provider.

Reverse the calls and a different gap appears. Kafka can commit first, then the process can die before the charge. Code order moves the uncertainty around. It does not remove it.

The destination has to help:

• A payment API can accept order_481 as an idempotency key and return the first result on retry.
• A database consumer can insert the event ID and apply the business update in one database transaction. A unique constraint rejects the second attempt.
• A service that changes its database before publishing can write an outbox row in the same database transaction, then publish it later.
• A system with an uncertain remote result can query or reconcile that result before trying again.

A Redis claim may reduce duplicate work, but it is not durable proof that an irreversible effect happened. That is why I keep claims separate from durable idempotency.

Two-phase commit can coordinate systems that expose compatible transactional protocols. Most HTTP APIs do not. Even where it exists, the availability and operational cost deserve their own decision. I would not use it as a footnote that magically stretches a Kafka transaction.

Ask which fact you need

I start with the witness and the boundary.

Question	Mechanism that can answer it	Boundary
Did the log accept enough copies?	acks=all with an adequate ISR and min.insync.replicas	One Kafka partition write
Can a protocol retry append the same batch twice?	Idempotent producer sequence numbers	One producer identity and partition sequence
Where should this consumer group resume?	Committed offset	One group and topic partition
Can input offsets and Kafka outputs commit together?	Kafka transaction	Participating Kafka partitions and offsets
Did the card charge exactly once?	Payment provider idempotency or reconciliation	The payment system
Did a database apply an event exactly once?	Database transaction plus a unique event identity	The database
Do eight partitions have one global order?	Nothing in Kafka provides that	No global log exists

The word "delivery" does not appear in the middle column because it is too vague to configure or verify.

Exactly once, where?

A team can say "exactly once" and be correct while a customer gets charged twice. The team may mean Kafka stored one producer batch, or that one input produced one committed Kafka output. The customer observes a different state transition.

Neither statement cancels the other. They name different boundaries.

So I ask one question whenever somebody promises exactly once: exactly once, where?

An answer should name the durable states and the commit that joins them. Anything less is still a hope between two systems.

Footnotes

1. Apache Kafka documentation, Message delivery semantics and consumer position. Kafka separates publishing durability from consuming semantics and models consumer progress as a per-partition offset. ↩ ↩²

2. Apache Kafka 4.3 documentation, Producer configuration. The reference defines acks, retries, idempotence, and the ordering constraint on in-flight requests. ↩ ↩²

3. Apache Kafka 4.3 documentation, Topic configuration. The reference defines min.insync.replicas, record visibility, retention policies, and unclean leader election. ↩ ↩² ↩³

4. Apache Kafka, KIP-98: Exactly once delivery and transactional messaging. The adopted proposal describes producer IDs, epochs, sequence numbers, transaction markers, fencing, and the limits of transactional consumption. ↩ ↩² ↩³

5. Apache Kafka 4.3 documentation, Consumer offset tracking. Group coordinators append committed positions to the compacted __consumer_offsets topic. ↩

6. Apache Kafka 4.3 documentation, Consumer configuration. read_committed returns committed transactional records only and stops at the last stable offset while a transaction remains open. ↩