Dead Letter Queues: Designing Failure Handling for Reliable Messaging

Dead Letter Queues are where messages go after retries fail. You need them because one bad body can stall a worker or burn CPU all night. If you drop those messages with no trail, you lose data and cannot explain why. This piece shows how to park, inspect, and replay failed work without causing a second outage.

What they are and why they fail

A dead letter queue is a side buffer for messages that did not succeed. The main queue or subscription stops after a limit you set. Then the broker moves or copies the body to that side buffer. A human or a tool can read it later.

The pattern fails when nobody owns the side buffer. Messages sit for months. Disk use and cost grow while the schema drifts. When you finally look, replay is unsafe because your code no longer accepts the old shape.

In my experience, the first version is a log line and a hope. That works until a partner sends a null field. The worker throws, the message returns, and a hot partition sticks. Therefore, plan the parking spot before the first poison message, not after the page.

Dead Letter Queues also fail when they hide the real bug. A high depth can mean bad code, bad data, or a down dependency. If you only alert on depth, you will not know which cause it is. Store the error class next to the body so the page names the fault.

How the path should work

Keep one dead letter destination per source queue. A shared bin looks simple at first. However, replay then mixes orders, mail, and billing in one stream. You will be afraid to redrive anything, so name the source in the queue name.

When a message is parked

Park after a fixed number of receives or after a fixed age. Five tries is a common start, as an illustrative production range, not a law. If the handler calls a flaky vendor, you may want more tries with backoff. If the handler throws on a schema error, extra tries only waste time.

Count tries in the broker when you can. Application counters reset when the process dies. Then the same body can loop forever. The Amazon SQS dead letter queues guide uses a redrive policy and a max receive count.

Do not park on the first failure. Transient blips should retry on the main path. Read timeouts and idempotent retries before you set the limit. A short timeout plus a low max receive count will fill the side queue during a brief outage.

What to keep with the body

Store the original body, the source name, the attempt count, and the last error. Also store a timestamp and a trace id. Without those fields, replay is a guess. If the body holds secrets or card data, lock the queue down and set a short retention.

The Google Cloud Pub/Sub dead letter topics page shows how a subscription forwards failed deliveries. The RabbitMQ dead letter exchanges docs show routing on reject, expire, or length limit. Read those trigger rules before you copy a sample.

A length limit can dead letter healthy messages when consumers are slow. That is not a poison pill. That is backpressure. If you treat it as bad data, you will replay a flood into a system that is already behind.

Trade-offs you should set on purpose

Every policy chooses among delay, data loss, and operator load. If you never park, poison blocks the line. If you park too soon, good work waits on a human. Write the choice down before the first incident.

A separate queue is easy to redrive with broker tools. A database table is easier to query and to join with orders. However, a table means your app must move the message, so a bug can skip the move.

Many teams use the broker queue as the source of truth and export a copy for search. Automatic replay sounds kind. It is dangerous when the bug is still deployed. Gate replay with a feature flag so you can stop it in one change.

Also, replay into the main queue, not straight into the handler. Then the normal limits still apply. If you bypass them, a bad batch can take down the same workers you just restored.

Pitfalls and failure modes

FIFO queues make poison worse. One bad message holds the group. Later messages for that key sit behind it. After you park the bad one, the group can move again.

If you forget the group id on replay, order for that entity breaks. Users then see a refund before a charge, or a cancel after a ship. Keep the original key on the parked copy. When you redrive, use that same key.

Another failure is a replay storm. An operator redrives a million messages at noon. Workers melt, and the database times out. Then fresh messages fail and join the side queue too.

Therefore, replay in small batches and watch the error rate. Stop when errors rise. We once hit a bottleneck when a script ignored the batch cap and pushed the whole week in one hour. The main site slowed even though the bug fix was correct.

Webhooks add a special case. A sender will retry when you return a 500. If you also park your own outbound calls, you can double retry the same action. Align the two budgets.

The webhooks architecture notes cover signature checks and status codes you should mirror here. If you ack the webhook and then fail the internal job, the sender will not retry. Your side queue is the only copy left.

A third failure is schema drift. The body was valid last month. Your new code rejects it.

The side queue fills with old but valid shapes. When you replay, keep a decoder for the old shape or map it first.

Since pub/sub and message queues can both feed this path, test both sources. A topic may dead letter per subscription. A queue dead letters once for the whole group. If you mix those rules, you will replay some events twice and miss others.

Use this sequence after the depth alert fires. Do not mass replay as the first step.

1. First, group messages by error class and source.
2. Then, fix the code or the data for the top class.
3. Next, replay a small batch and watch the error rate.
4. Finally, drop or archive what you cannot process and record why.

A policy you can start from

This sketch shows a main queue, a max receive count, and a side queue. It is not a full vendor file. Change the numbers after you measure handler time. If the visibility window is shorter than the handler, you will park healthy work.

source_queue: order-jobs
visibility_timeout_seconds: 90
max_receives: 5
dead_letter_queue: order-jobs-dlq
dlq_retention_days: 14
on_park:
  capture: body, attempts, last_error, trace_id
  alert: page if depth over 100 for 10 minutes
replay:
  batch_size: 50
  require_flag: replay_order_jobs

The alert uses depth and time so one bad message does not page you at once. The flag stops a script from redriving after a bad deploy. Because the handler must be safe to run twice, replay reuses the original message id.

Do not invent a new id on the way back. A new id looks like new work. Then your unique key check will not block the second side effect. That is how a replay becomes a double charge.

Performance, scale, and cost

A side queue is cheap when it stays near empty. Cost grows when you retain large bodies for a long time. As an illustrative production range, a few hundred thousand parked messages with 10 KB bodies can mean several gigabytes plus replicas.

Encrypt those bodies and expire them. Fourteen days is often enough to debug a release. If legal rules need a longer hold, move the body to a store you control. Do not leave it in the broker for a year.

Redrive speed is the scale limit people forget. The main path may handle thousands of messages per second. Your replay tool may handle far less if it runs as one script. Pace it so the database stays healthy.

Also, cap concurrent replays so two operators do not double the load. A shared lock or a single runner is enough. Scheduled cleanup can run as a job, but two cleaners must not delete and replay at once.

That overlap is the same class of bug as cron jobs in distributed systems. One owner should hold the lease for the whole replay window. If the lease expires mid batch, the next runner should resume from a saved cursor.

Metrics that matter are depth, age of the oldest message, and park rate per error class. CPU on the worker is a weak signal. Still, a sudden park rate often means a new deploy or a bad partner payload.

Tie the chart to deploys so you can roll back fast. If park rate spikes and the error class is timeout, look at the dependency before you blame the payload. If the class is schema, stop replay until the decoder ships.

Do not use the side queue as a long term warehouse. After you decide, move facts you must keep into a store you can query. Then delete the broker copy. The broker is for action, and the store is for audit.

Key Takeaways

• Also, give each source queue its own side destination so replay stays clear.
• However, park only after real retries, not on the first transient timeout.
• Therefore, store the error class, attempt count, trace id, and original key with the body.
• Still, gate replay with a flag and a small batch so a fix cannot melt the site.
• Because handlers must be safe to run twice, reuse the original message id.
• After you decide, expire broker copies and keep long term facts in a real store.
• Finally, alert on age and park rate, not only on a raw depth number.

FAQ

When should a message enter a dead letter queue?

Park it after the broker hits your max receives or max age. Use a higher limit when the handler depends on a flaky vendor and the error is a timeout. Use a low limit when the error is a bad schema. If you are unsure, start at five receives and review the first week of parked bodies.

What happens if you replay before the fix ships?

The same handler throws again. The message returns to the side queue or loops on the main path. Workers spend their time on known bad work. Ship the fix first, then replay a small batch.

How should you treat poison messages on a FIFO key?

When one bad body blocks a group, later jobs for that key wait. Park that body and keep its group id. Then the rest of the group can flow. If you replay it, send it back with the same group id so order for that entity stays intact.

Can you skip Dead Letter Queues if you log errors?

Logs help you debug, but they are a weak place to retry from. A log line can be sampled, truncated, or dropped. If the work must not vanish, keep the full body in a side queue or a table. Then you can replay with the same id after the fix.

Next, list every queue and subscription and add a max retry plus a side destination. Write the replay steps in the runbook before you need them. Then alert on age and park rate, and test one small redrive on a staging source. If a source has no owner, assign one before you turn the alert on.

Last updated on 19 September 2026.