Message Queues in System Design: Delivery Guarantees and Dead Letters

This article is the architecture cluster’s second stop, and its vocabulary is deliberately load-bearing. The hub article weighed the style’s costs and kept pointing here (asynchronous communication is how a fleet of services stops blocking on itself) and microservices architecture is where those trade-offs were weighed. The promise list is even older: the distributed transactions article put event-driven patterns on the shortlist before two-phase commit, and this article now defines the shortlist’s leading item, an exactly-once pipeline built from a transactional producer and idempotent consumers. Delivery-guarantee terminology is anchored here on purpose: when the rest of this series says at-least-once or exactly-once, this is where the words get their meaning.

The mechanism deserves decision framing too, not admiration. Queues are not automatically an upgrade from direct calls: they trade latency you can measure for durability you must configure, and the trade pays off only when both sides genuinely benefit from decoupling; producers that should not wait, consumers that should not inherit the producer’s availability, bursts the consumer cannot absorb inline. What follows defines the pattern, then spends most of its length on the two questions that decide whether a queue is honest: what does the broker guarantee, and what happens to a message that keeps failing.

What is a message queue

A message queue is a broker that accepts messages from producers, stores them durably, and delivers each one to a consumer for processing; asynchronously: the sender’s call returns when the broker accepts the message, not when the work happens.

The cast has four roles. The producer writes a message to the queue; the broker stores and forwards it; the consumer receives, processes, and acknowledges; and the queue is the contract between them, in its classic form, first in, first out, with each message handed to one consumer even when several compete for work. Message ordering is a queue-level property and weaker than it looks: the moment a queue is partitioned for parallelism, ordering holds within a partition, not across them; the log-and-partition model that makes explicit is kafka architecture‘s subject. And durability is a configuration, not a law: most brokers ship with persistence settings that decide whether a message survives a broker crash, and defaults differ. A queue is exactly as reliable as its settings say it is, which is why the guarantees deserve their own section below.

What a queue buys, and what it costs

The buy is decoupling, in time and in failure. A producer that enqueues does not need the consumer up, available, or fast; the broker accepts the message and the work happens later, which converts a hard availability dependency into a durability question the broker answers. A consumer that crashes stops losing work, because its unacknowledged messages return to the queue and another consumer takes them: redelivery is failure recovery applied to the message, a rung of fault-tolerant systems‘ ladder operating between services. And each side scales on its own axis: consumers scale out to drain a backlog and scale back when it clears, the same scale-out arithmetic vertical vs horizontal scaling applies to machines, now applied to work itself.

The buy is also burst tolerance. A queue absorbs what the consumer cannot yet process, converting a spike from an outage into a backlog: the producer stays fast, the consumer stays alive, and the queue holds the difference. But absorption has a limit, and when producers outpace consumers persistently, the queue fills. At that point the honest answer is not a bigger queue but backpressure (signaling upstream to slow down) which is a design conversation of its own.

The cost is latency plus a new piece of critical infrastructure. Asynchronous delivery adds a broker hop and queueing delay, and the p99 consequences are latency vs throughput‘s to measure: a queue that helps throughput often hurts the latency of the single request that used to be one call. Meanwhile the broker itself is now load-bearing; every queue between two services is a component whose replication and durability settings decide the system’s worst case, and a fleet of queues is a fleet of systems to operate.

The last cost is the one this article’s title carries. Once delivery is asynchronous, failure stops being a connection error and becomes a semantic: did the message arrive, did it arrive twice, did the consumer’s own write succeed before the ack? Those are the delivery guarantees below. And a message that keeps failing needs somewhere to go, or it jams the pipeline forever; that somewhere is the dead-letter queue, and the discipline of watching it is monitoring and observability‘s; an unwatched dead letter queue is not a safety net, it is an outage with a delay.

Delivery guarantees

Delivery guarantees are the broker’s answer to one question: when something fails, what happens to the message? The vocabulary has exactly three entries, and everything else in a broker’s configuration is an implementation of one of them. At-most-once: the message may be processed zero or one times; the broker sends it and forgets it, and a crash anywhere can drop it. At-least-once: the message is processed one or more times; the broker redelivers until it is acknowledged, and duplicates are the price. Exactly-once: the message is processed exactly one time, which in practice means at-least-once delivery plus machinery that makes repeats undetectable in effect. The guarantee is not a feature you buy from a broker; it is a pipeline you build, and the strength of the guarantee sets the bill.

At least once delivery

At least once delivery is the workhorse default, and its mechanism is one rule: acknowledge after processing. The consumer receives the message, does its work, writes its own state, and only then sends the ack, so a consumer that crashes mid-processing leaves the message unacknowledged, and the broker redelivers it. The cost is structural: between “work done” and “ack sent” there is a gap, and every crash in that gap produces a duplicate. Duplicates are not a malfunction of at-least-once; they are its design; the guarantee buys “never lost” by selling “possibly twice.”

Which means the consumer must tolerate duplicates, and that tolerance has a name: idempotency; processing the same message twice must leave the system in the same state as processing it once. Sometimes it comes free: a write keyed by the message’s natural key is naturally idempotent. Usually it is built, with dedup keys, version checks, and idempotency tables. The mechanics of making consumers idempotent are idempotency‘s subject; this article’s job is the guarantee that makes the topic non-negotiable.

The sibling guarantee is worth naming while the scale is out. Acknowledge before processing instead of after, and the same pipeline becomes at-most-once: the broker hands off responsibility at delivery, and a consumer crash takes the message with it. That trade is sometimes correct (telemetry and metrics tolerate gaps better than duplication) but it should be chosen, not fallen into. The default position of this series is that lost work is the more expensive failure, and at-least-once is where honest pipelines start.

Exactly once delivery

Exactly once delivery is the guarantee everyone asks for first and should ask about last, because its fine print is the point. The corner case is the ack: a broker cannot atomically deliver a message and confirm its delivery across a crash. If it redelivers on doubt, the pipeline is at-least-once; if it does not, the pipeline is at-most-once. There is no third broker behavior, so real exactly-once is built, not delivered: at-least-once underneath, with machinery that makes the repeat indistinguishable in effect.

Two places that machinery lives. The first is the consumer: an idempotent consumer on an at-least-once queue delivers the same effective guarantee for most workloads, at commodity prices, dedup at the point of processing instead of coordination across it. The second is the transactional pipeline, and this is the case the transactions article pointed at: a consumer that consumes, transforms, and produces (reading from one queue and writing to another plus its own state) commits the source offsets and the sink writes atomically, so a crash mid-transform either happened or did not. Kafka implements this with its transactional producer, and the log-and-partition mechanics that make it work are kafka architecture‘s subject; which broker’s model fits a system is kafka vs rabbitmq‘s.

The decision rule: pay for transactional machinery when the duplicate effect is expensive and hard to dedup naturally (money movement is the canonical case) and take the idempotent-consumer route everywhere else. Exactly-once is real, but it is a purchase with a latency line and an operations line, and most pipelines get the same guarantee for less.

Dead letters

Dead letters are messages the pipeline could not process and chose not to keep retrying, and the dead letter queue is the quarantine where they wait instead of blocking traffic. The mechanism is a retry policy with a floor: a message that fails processing is retried a bounded number of times with exponential backoff (transient failures, like a timeout or a dependency blip, mostly clear on retry) and a message still failing after the budget is moved to the DLQ, where it stops touching the live pipeline. The split is triage: retry what is probably transient, quarantine what is probably permanent, and never let the distinction jam the queue.

What makes the policy load-bearing is the poison message: a payload so malformed, or a bug so reliable, that every consumer who touches the message fails. On a naive at-least-once setup, that message returns to the head of the queue forever; the redelivery loop blocks the line behind it, and the pipeline stalls on one bad byte. The DLQ exists to break exactly this loop.

The operational half of the contract matters as much as the mechanism. A dead letter queue nobody watches is silent data loss with a delay; alerting on its depth and growth is baseline maturity, not optional polish. Every dead letter is one of three things: a bug report (fix the code, replay the message), a data-quality signal (fix the producer), or a deliberate discard, and the third should be a logged decision, never a default. Handled that way, the DLQ is not where messages go to die; it is where failures go to be answered.

Common mistakes

• Using the queue as a database. A queue is transport with a retention policy, not a queryable store: fishing messages back by content, serving reads from it, or treating months of retention as the system of record all end the same way; the broker carries load it was never shaped for, and the data model lives in a component built for throughput, not queries. Move state into a store the moment it needs a second reader.
• Acknowledging on receipt. Auto-ack is the default in many client libraries, and it quietly converts at-least-once into at-most-once: the broker hands off responsibility before processing, and a consumer crash takes the message with it. If the work matters, the ack belongs after it.
• Retrying without backoff or a ceiling. Unbounded, immediate retries turn a dependency outage into a self-inflicted denial of service; every consumer hammers the sick dependency, the queue backs up with retry traffic, and recovery has to fight the storm it created. Retries need exponential backoff, a ceiling, and once the ceiling is hit, a trip that stops the hammering upstream; the pattern for that trip is the circuit breaker pattern.
• Designing on ordering the queue does not promise. FIFO is a per-queue property in the textbook and a per-partition property the moment parallelism arrives: add consumers, partition for throughput, and cross-partition ordering is gone. Business logic that needs a sequence either pins it to one lane and pays the throughput, or carries its own sequence numbers, it cannot assume them from the broker.
• Running without a dead-letter policy. A queue with no retry budget and no DLQ has only two failure behaviors: drop the message silently, or redeliver it forever, silent data loss or a poisoned pipeline. The policy is cheap to configure and cheaper than either outage, and the alerting on the DLQ that completes it is part of the same setup, not a later project.

FAQ

What is the difference between a message queue and pub/sub?
Worklist versus broadcast. A queue hands each message to one consumer; the work gets done once, and competing consumers drain the backlog faster together. Pub/sub delivers each message to every subscriber, each one hears the event and reacts independently. The models overlap in the middle, since consumer groups blur the line deliberately, and the full comparison (delivery shapes, ordering, replay) is event-driven architecture‘s subject.

Is exactly-once delivery real?
As an effect, yes; as a raw broker behavior, no. The ack corner makes pure exactly-once delivery impossible across a crash; redeliver on doubt and you are at-least-once; do not, and you are at-most-once. Real exactly-once is at-least-once plus machinery: idempotent consumers for most pipelines, or a transactional consume-transform-produce pipeline where source offsets and sink writes commit atomically. The guarantee is real; it is just built, not bought.

How many times should a message be retried before it dead-letters?
Bounded, and typically small: two to five attempts with exponential backoff covers nearly all transient failures, and what the number really encodes is a time budget, how long the pipeline will spend on one message before trading retries for quarantine. Set the budget from the downstream SLA, alert on the DLQ that catches the overruns, and revisit the number when the DLQ says it is wrong.

Do message queues guarantee ordering?
Within their unit, not globally. A single queue with a single consumer is FIFO; add partitions or competing consumers and ordering holds inside each partition, not across them. If business logic needs a sequence, either pin those messages to one lane and accept the throughput ceiling, or put sequence numbers in the payload and let the consumer reorder. Assuming broker ordering under parallelism is the classic pipeline bug.

Should I choose Kafka or RabbitMQ?
They are different models, not rival brands. Kafka is a durable log built for replay and high-throughput streaming; consumers track offsets, and history stays available; RabbitMQ is a classic broker built for flexible routing and per-message delivery; messages move on to whoever is next. Queue-as-log versus queue-as-transit leads to different answers on replay, ordering, and exactly-once machinery: the trade-off gets its own referee in kafka vs rabbitmq, with the log internals in kafka architecture.

Related articles

• Next read: event-driven architecture, the reading spine’s next stop: events, pub/sub, and choreography; the broadcast model that answers the one-consumer-or-all-subscribers question this article kept raising.
• kafka architecture; the log-and-partition model this article kept deferring to: how a durable log delivers, stores, and replays the messages whose guarantees this article just priced.
• idempotency, the consumer-side half of at-least-once: dedup keys, version checks, and the mechanics that make “possibly twice” safe.
• microservices architecture, the cluster hub: the trade-off ledger that explains why fleets of services need asynchronous backbones at all.
• distributed transactions: the shortlist this article’s transactional pipeline came from: coordination patterns across boundaries, and where exactly-once machinery overlaps them.

Last updated on 5 September 2026.