Webhooks Architecture: Security, Retries, Idempotency, and Observability

Webhooks Architecture is how one system tells another that something happened, using an HTTP call the receiver did not start. You need a clear design because a lost or forged call becomes a missed payment or a fake update. If you ack before you store the event, retries cannot save you. This guide covers signatures, retry budgets, duplicate delivery, and the signals you should watch.

What it is and why it fails

A webhook is a callback. The producer sends a POST to a URL you registered. The body describes an event.

Your handler checks the call, stores it, and does the work. Then you return a status code.

The pattern fails in production in a few repeat ways. First, the handler does the side effect and then crashes before the response. The producer retries, and you do the side effect again. Second, a slow handler hits the producer timeout, so both sides think the other failed.

Third, anyone who can reach the URL can post a fake body if you do not check a signature. In my experience, teams add the check late, after a bad actor or a buggy partner sends junk. Therefore, verify the sender before you parse business fields. Treat the raw body as untrusted until the signature matches.

Webhooks Architecture also fails when delivery order is assumed. Most producers do not promise global order. A refund event can arrive before the charge event. If your handler requires the charge row first, you will fail the refund and may drop it when retries end.

How to build the path

Split the work into two steps. Step one stores the event in an inbox and returns fast. Step two does the slow work from that inbox. If you mix them, your HTTP timeout becomes your business timeout.

Verify the caller

Check a signature over the raw body and a timestamp. Reject calls that are too old so a captured request cannot be replayed next week. RFC 9421 HTTP Message Signatures defines a standard way to sign HTTP messages. Your partner may use a simpler HMAC instead.

Rotate secrets without a gap. Accept the current secret and the previous one during the change. If you cut over in one instant, in flight calls fail and the partner retries a storm. Also, do not log the secret or the full signature string in plain text.

Ack only after the inbox write

Return a 2xx only after the event row is durable. Use a unique key on the event id. A retry then hits the same row and you still return success. The Amazon SNS HTTP and HTTPS endpoints guide explains how a subscriber confirms and receives posts.

Return 500 when the inbox is down so the producer retries. Return 400 when the body is invalid and will never succeed. If you return 400 for a timeout, the producer may stop and you will lose a good event. Map codes with care.

Do the slow work after the ack

A worker reads the inbox and performs the side effect. It can take longer than the HTTP timeout. If it fails, it retries locally or parks the row. The producer is no longer in the loop, so your own retry policy must be real.

This split matches timeouts and idempotent retries on any RPC. The HTTP call is only the handoff. The work can still fail after you said yes. Design for that case before you go live.

Retries, duplicates, and order

Producers retry when they see a timeout or a 5xx. Backoff should grow, and the total budget should be finite. The Amazon SNS message delivery retries page shows one vendor policy with delayed tries.

Your partner may differ. Ask for the budget and the status codes that stop it.

Assume at least once delivery. Store the event id and ignore a second insert. If the id is missing, hash the raw body only as a weak fallback.

A hash can collide or change if the partner adds a field. Prefer the id they document.

Order is not a gift. If a later event arrives first, store it and wait, or apply it in a way that stays valid. For example, a refund can sit until the charge row exists.

A sweep job can retry those waits. That sweep is a small case of cron jobs in distributed systems, so give it one owner.

After the producer gives up, the event is your problem only if you already stored it. If you never stored it, you must rely on their replay tool or a pull API. Do not assume they keep every body forever. Agree on a replay window in the contract.

Trade-offs you should weigh

Webhooks are push. Pull is a poll against their API. Push is faster and cheaper when events are rare.

Pull is easier to debug when you do not trust their retry rules. Many mature setups use push for speed and a nightly pull to repair gaps.

A direct handler is simpler for a tiny volume. It becomes a liability when a partner sends a burst. Put a queue between the HTTP tier and the worker if bursts are part of your load. That queue is the same idea as pub/sub and message queues, with the webhook as the publisher.

Do not use webhooks for a workflow that cannot tolerate delay or duplicates. A user waiting on the HTTP response of the producer will not wait for your retries. If the user needs an instant answer, handle it on the request path and use the webhook as a backup fact.

Pitfalls and failure modes

A common mistake I have seen is to verify the signature after a middleware already parsed and rebuilt the body. The bytes change, the signature fails, and every call looks forged. Sign the raw bytes. Then parse.

Another trap is a shared endpoint with no routing key. One URL receives every event type. A bug in one handler returns 500 for all types.

Then the partner retries invoices, refunds, and pings together. Split by type or catch errors per type so one bad handler does not fail the rest.

Timeouts cause a third trap. You do a card capture inside the request. The partner times out at ten seconds and retries.

You capture twice unless the processor key is unique. Keep the request under a short budget and move slow calls to the worker.

Poison events deserve a side path. After local retries fail, park the row and alert. Use the same care you would with dead letter queues.

A silent skip hides a missed payout. A hot loop burns the database.

If you are the producer, you own a different set of bugs. You must store outbound deliveries and retry them. You must sign the body. You must not follow open redirects on the customer URL, or you will call an internal host by mistake.

Allow only HTTPS endpoints you can explain. Bound the response size. Stop retrying after the budget.

If a customer endpoint is slow, isolate them so one tenant does not fill your worker pool. We once hit a bottleneck when one URL hung until our whole sender pool blocked.

Clock skew breaks timestamp checks. Allow a small window, such as a few minutes, as an illustrative production range. If you allow a day, captured traffic is easy to replay. If you allow one second, healthy calls fail when NTP is a bit off.

Readers of your data may be behind the webhook. That lag is eventual consistency in practice. Do not show a paid state until the inbox worker commits. A 2xx to the partner is not the same as a finished ledger row.

Use this order when deliveries look wrong. Fix verification before you replay a backlog.

1. First, check signature failures versus inbox failures.
2. Then, compare your status codes with the partner retry rules.
3. Next, look for duplicate side effects on one event id.
4. Finally, replay from the inbox or ask the partner for a bounded resend.

A handler sketch you can adapt

This sketch shows the checks and the inbox write. It is not a framework. Change the age window to match your partner. If you ack before the insert commits, retries will duplicate the side effect.

endpoint: POST /hooks/billing
verify:
  signature: hmac_sha256 over raw_body
  secrets: current, previous
  max_age_seconds: 300
inbox:
  table: webhook_events
  unique: partner_event_id
  on_conflict: return 200
responses:
  stored: 200
  bad_signature: 401
  bad_body: 400
  store_down: 500
worker:
  retry_limit: 8
  then: park_for_review

The unique key makes the second POST a no op that still returns 200. The partner then stops. Because the worker uses the same event id, a later crash does not create a second charge. Park for review after eight local tries so a human sees the stuck row.

Performance, scale, and cost

The HTTP tier should do little work. Verify, insert, and return. If that path stays under a small fraction of a second, the partner will not retry from timeouts. Retries are what multiply your cost during an incident.

As an illustrative production range, a burst of tens of thousands of events in a minute is common when a partner replays a backlog. Your inbox insert must absorb that. A worker pool can drain it over a longer time. If you process inside the request, that burst becomes an outage.

Index the event id and the status. An unindexed unique check will slow every insert as the table grows. Partition or archive old rows. You rarely need the raw body online after the replay window.

Watch four signals. Signature failure rate catches bad rotations and attacks. Inbox lag age catches a stuck worker.

Status code mix catches a mapping bug. Duplicate rate catches a missing unique key. Still, a low CPU number can hide a growing lag.

Cost follows retries and fan out. One slow endpoint, if you are the sender, can hold connections and force you to scale the pool. Isolate tenants.

If you are the receiver, a naive 500 on bad data makes the partner pay and makes you pay too. Return 400 when the body will never work.

Key Takeaways

• Also, verify the signature on the raw body before you trust any field.
• However, return success only after the event is stored under a unique id.
• Therefore, do slow work on a worker so the HTTP timeout stays short.
• Still, map 400 and 500 with care so partners retry only when retry can help.
• Because order is weak, design handlers that tolerate a later event arriving first.
• After local retries fail, park the event and alert instead of looping forever.
• Finally, repair gaps with a pull or a bounded replay, not with hope.

FAQ

When should you return a retryable error?

Return 500 or 503 when a transient fault stopped the inbox write. Return 400 when the body is invalid or the signature is wrong and a retry will not fix it. If you are unsure, look at the partner budget. A retry of a bad signature only adds noise.

What happens if you process work before you ack?

The partner can time out and send the same event again. You may charge, mail, or ship twice. Store the event id first and ack. Then let a worker do the side effect.

How should you handle a replay of old events?

When a partner resends a backlog, your unique key should turn known ids into a fast success. New ids should land in the inbox and drain at a pace you choose. Do not let a replay skip your queue and hit the database at full speed. Cap the worker so the live traffic still fits.

Can Webhooks Architecture replace a queue?

A webhook is a push into your edge. It is not a durable log you control. If you need many consumers or a long replay, put the event on a queue or a log after the inbox write. Use the webhook as the ingress, and use your broker as the system of record for fan out.

Next, draw your webhook path and mark where you verify, where you store, and where you act. Add a unique event id and a status code map before you accept production traffic. Then send a duplicate and a bad signature in a test and confirm the results. If the duplicate changes state twice, stop the rollout and fix the inbox.

Last updated on 22 September 2026.