Idempotency: Designing Operations That Survive Retries

Eight articles pre-appointed this one as the base layer. Message queues wrote the appointment into its delivery section: at-least-once means consumers must tolerate duplicates, and “the mechanics of making consumers idempotent are idempotency‘s subject; this article’s job is the guarantee that makes the topic non-negotiable.” Fault-tolerant systems gave the reason recovery cares: “recovery is the practical reason idempotent design exists (a property idempotency owns in full)” because every checkpointed replay re-runs work that may already have run. Client-server architecture planted the seed in the method grammar: “GET twice is the same GET; POST twice is two orders”, and the messaging pair priced the consumer-side bill: kafka architecture calls commit-after-processing at-least-once, “the reason consumers need idempotency on their side,” the broker comparison hands the consuming team “at-least-once handling and idempotency, not the broker,” and event-driven architecture multiplied it; ten subscribers is ten independent at-least-once problems. Both real-time articles, websockets and long polling vs sse, promised the same on reconnect: the gap replays, and the client must survive it. This article pays all of those debts in one place.

Idempotency is the property of an operation that performing it more than once leaves the system in the same state as performing it once: the second attempt neither duplicates the effect nor corrupts the result. The charge runs once, the append writes once, the message lands once; no matter how many times the operation is attempted under failure, retry, or redelivery. It is the answer to the one question a distributed system can never stop asking; did that go through?; answered in a way that stays true no matter how many times it is asked.

One boundary before the mechanics, because the cluster drew it in writing: delivery-guarantee terminology, at-most-once, at-least-once, exactly-once as pipeline promises: belongs to message queues, and this article links to it rather than re-deriving it. What this article owns is the operation side: the keys, the dedup records, and the API discipline that make an operation safe to repeat regardless of which guarantee the pipeline underneath made. Retry pacing belongs to its immediate sibling, retry with backoff and jitter; this article makes the retry safe to land; that one decides when it lands. Together they open the resilience series whose later articles: load shedding and backpressure, and the high availability pillar they support assume this property the way a building assumes its foundation.

What is idempotency

The term comes from mathematics, where an operation composed with itself changes nothing, and HTTP borrowed it in exactly that sense. In the request grammar client-server architecture laid out, the methods carry the promise by contract: GET is safe; it reads and changes nothing; PUT and DELETE are idempotent, PUT the same update twice and the second replaces the first with itself; DELETE the same resource twice and the second finds nothing left to remove. POST carries no such promise; post the same order twice and you have two orders. The vocabulary distinction matters: safe means “no effect,” idempotent means “same effect every time,” and only POST-heavy designs strictly need the machinery this article is about, because the contract already covers the rest.

Distributed systems take that grammar’s polite promise and make it load-bearing. A timeout is not an answer (the request died, or the response did, and the caller cannot tell which) so well-behaved clients retry. Failover re-runs work in flight: the fault tolerance playbook checkpoints, replays, and hands partitions to survivors, and the replay crosses work that may already have completed. Queues redeliver what was processed but never acknowledged, because at-least-once delivery is the default and the alternative drops messages on every crash. Every one of those mechanisms is correct precisely because it assumes duplication, and every one of them turns a non-idempotent operation into a production incident waiting for its retry.

The test for any operation is one sentence: run it twice, compare the world. If the second run is a no-op (or converges on the same state) the operation is idempotent, and the retries above cost nothing but latency. If the second run doubles, appends, or emits anything, the operation is not, and the choice is to change the operation or to wrap it in the machinery of the next section. Most systems need both: naturally idempotent operations where the data allows them (set-state writes, keyed upserts, absolute-value updates) and enforced idempotency where the side effect cannot be reshaped: charges, emails, append-only history.

Idempotency key

The idempotency key is the mechanism that enforces the property where the operation itself cannot provide it: a client-chosen identifier that names the logical operation rather than any single attempt at it. The client generates one key per operation the user performs (the checkout, the transfer, the form submit) and sends it with every attempt, including retries. The server’s contract is then simple: the first time it sees a key, it executes and records; every later time it sees the same key, it does not execute; it replays the recorded outcome. The duplicate becomes a cached response with a purpose: the caller gets the answer to “what happened?” rather than a second “this happened.”

Key generation is the client’s job, and the rules are strict in one direction only: the key must stay stable across retries and unique across operations. Generate the key once, when the logical operation begins, and reuse it on every attempt; a key regenerated on retry deduplicates nothing, which is the most common silent failure in the pattern. Uniqueness comes from scope: a random identifier per operation, generated at click-time and stored in the client’s state for the operation’s lifetime. Where the operation has a natural identity (an external reference number, a file checksum, a scheduled job’s run date) the natural key is better than a random one, because it also deduplicates the operation submitted twice by two different sessions that both believed they were first.

The server side is a record, not a flag. The standard shape is an idempotency table: a row keyed by the idempotency key (plus whatever scope the API sells, per account or per endpoint) holding the request’s fingerprint, the operation’s state (received, in progress, completed), and, crucially, the stored response: status code, headers, body. A unique constraint on the key is the gate itself: the first insert wins, a racing duplicate fails the insert and reads the row instead. Storing the response is what makes the replay honest; the duplicate caller receives the original outcome, not a generic “already done,” which is the difference between a client that can reconcile and a client that must guess.

Two hard questions shape the record’s life. Concurrency: a second request with the same key can arrive while the first is still in flight (the record exists but is not complete) and the API must choose between blocking the duplicate until the first finishes, returning an explicit “still processing,” or rejecting the race; blocking with a timeout is the client-friendly default, and whichever choice is made belongs in the API’s documentation, not in its folklore. Expiry: dedup records cannot live forever, so each carries a retention window, and the window must be strictly longer than any client’s retry horizon, because a record cleaned up before the last legitimate retry turns that late duplicate back into a fresh operation. The retention TTL is not a cache setting; it is a correctness contract with every client the API has ever shipped to.

Idempotent API design

Idempotent API design starts from the method grammar and then closes the gap POST leaves. GET, PUT, and DELETE arrive with the contract above; PATCH is idempotent when it sets fields and stops being so the moment it merges, appends, or computes: “set displayName to X” replays safely, “add 1 to cartCount” does not, and the difference is worth an API review. POST is where the idempotency key earns its keep: creation, payment, and action endpoints should accept and require the key, because they are exactly the endpoints that clients retry under timeout; the polite client retrying a possibly-lost order and the impatient user double-clicking submit are the same problem wearing different intentions.

The duplicate’s response is a design decision with a right answer: return the recorded result, status code and body, as the first attempt produced it. A duplicate POST that created a resource returns the original success with the original resource, not a second creation, not a differently-shaped “duplicate” reply, and not an error that forces the client to guess whether the first one landed. Failure paths need the same honesty: the first attempt’s failure is replayed too, because “it failed once” is an answer the retrying client needs, and a re-executed failure is a second side effect the operation may not survive. What an API must never do is half-remember; recording that a key was seen while losing what happened under it is the anti-pattern the mistakes section returns to.

HTTP has no standard idempotency header, which is why the pattern ships as a documented convention. Payment APIs made the idempotency-key header famous (Stripe’s is the canonical, publicly documented example) and the convention generalizes to any action endpoint: a header or body field naming the key, a published retention window, and documented duplicate behavior, including the awkward case the documentation must not skip: the same key arriving with a different payload. That case is a client bug, and the honest API surfaces it as a rejection rather than silently servicing the second request; dedup machinery that swallows contract violations stops being safety and starts being a corruption channel.

The discipline does not stop at the front door. Webhooks arrive with guaranteed duplicates, because providers retry anything that did not draw a 2xx; message consumers redeliver by design; cron jobs overlap themselves after a pause; RPC layers retry on timeout because that is their job. Every one of those surfaces wants the same shape (identify the logical operation, record the outcome, replay on repeat) and teams that build the machinery once, as a library or a platform primitive, stop reimplementing it per endpoint. That is how the property stops being folklore and becomes the default posture of the fleet.

At least once delivery

The delivery-guarantee vocabulary belongs to message queues: at-most-once fires and forgets, a crash drops the message; at-least-once acknowledges after processing; a crash redelivers; exactly-once promises one effect per message. The pipeline choice among those is a real engineering decision, and this article’s point is about what every realistic choice implies: at-least-once is the practical default everywhere (queue redelivery, failover replay, the resynchronization after a real-time reconnect) because it is the only guarantee that loses no work to a crash. The duplicates it produces are not an accident of poor engineering; they are the price of not losing work, paid on every recovery.

Exactly-once deserves its honest paragraph. Within a single system, pipeline machinery can narrow the window dramatically (Kafka’s idempotent producer deduplicates producer retries into the broker, and transactional machinery can pin the effect of consuming-and-producing) but the end-to-end claim across heterogeneous systems collapses to at-least-once at the first hand-off between systems nobody fully controls. The standard engineering translation is the honest one: exactly-once delivery is at-least-once delivery plus exactly-once effects, and the second half is idempotency, the machinery of this article. Pipelines that advertise the first without owning the second have not removed the duplicates; they have relocated them into someone else’s incident report.

The consumer-side mechanics are the idempotency key wearing a queue costume, and the queue article already named them: dedup keys, version checks, and idempotency tables. Dedup keys work when the message carries stable identity: trust the producer’s event id, or hash the operation’s meaning. Version checks work when the state itself carries a version: apply only if the incoming change supersedes the stored one, so the redelivery of an already-applied update fails its own precondition instead of re-applying. Idempotency tables work when neither is available: record the processed message id with or before the effect, and treat the record’s presence as the answer. Pub/sub fan-out multiplies the discipline (each subscriber is its own at-least-once problem with its own dedup state) and replay multiplies it again by history: the subscriber that re-reads last week’s events must treat its own earlier processing of them as duplicates.

Which closes the circle this article opened with: recovery, retry, and redelivery are the mechanisms the rest of the resilience series builds on, and every one of them re-executes work. The checkpoint replay in fault-tolerant systems, the retried call after a failover, the replayed gap after a reconnect; each is only safe because the operation underneath answers “already done?” the same way every time. Idempotency is not one resilience tool among many; it is the property that makes the rest of the toolkit safe to use, which is why it opens this series and why every later article links back to it.

Common mistakes

• Regenerating the key on retry. A fresh key per attempt turns the dedup table into write-only storage: every retry is a first request, duplicates execute, and the incident looks like a mystery because each individual request looks correct. The key names the logical operation; retries reuse it by definition.
• Deduplicating without remembering the response. A marker that says “processed” answers nothing the caller needs; the duplicate gets a shrug instead of the outcome. Store the status, headers, and body with the record, or the client that timed out on the first attempt can never learn what it did.
• Checking at the wrong layer. The gateway deduplicates the request, the service appends anyway, or the service checks the key, then hands the work to a downstream that emits its own emails and charges. Deduplication must wrap the entire effect, because the parts it misses are exactly the parts that duplicate.
• A retention window shorter than the retry horizon. The record expires in hours, the client retries for a day, and the late duplicate lands as a fresh operation with a clean conscience. The TTL is a contract with every client ever shipped; size it against the longest retry policy in existence, including the ones inside clients you do not control.
• Assuming natural idempotency that is not there. The database write is an upsert, and the webhook it triggers, the email it queues, and the analytics event it emits are not. Side effects are part of the operation, and the honest test remains “run it twice, compare the world,” applied to everything the operation touches, not just the row it updates.

FAQ

Who generates the idempotency key, the client or the server?
The client. The key names a logical operation that begins before any request is sent, and only the caller knows where one operation ends and the next begins: the user’s click, the batch’s row, the transaction’s intent. A server-generated identifier names the request, not the operation, and deduplicates nothing on retry.

Are PUT and DELETE really idempotent, or does that depend on the server?
The HTTP contract promises the semantics; the implementation must keep them. A PUT handler that appends instead of replacing breaks the contract the method grammar advertises, and a DELETE that cascades differently on the second call does the same. Treat method idempotency as a test suite: the verbs arrive safe and idempotent, and the handler’s job is to not void the warranty.

What is the difference between idempotency and exactly-once delivery?
Idempotency is a property of the operation: run it twice, the world is unchanged. Exactly-once is a promise about a pipeline: deliver it once, effect it once. End-to-end exactly-once is normally built as at-least-once delivery plus exactly-once effects: the delivery side owned by message queues, the effect side by idempotency. One describes the wire, the other describes the work.

Do we still need idempotency if our pipeline guarantees at-most-once?
Then you have traded duplicates for losses, and the bill moves to reconciliation: a gap detector, a replay window, and a way to re-submit what went missing. At-most-once is the right call where a duplicate is catastrophic and a gap is cheap, the rare case. Most systems run at-least-once precisely because losing work is usually harder to repair than deduplicating it.

How long should idempotency records be kept?
Longer than the longest retry horizon that can reach you: client retry policies, provider redelivery windows, batch reruns, and the operational “run it again by hand” case. Then apply the cost of storage honestly; the record is small, the incident it prevents is not, and most APIs choose windows measured in days, published in the documentation, and enforced by the same job that would otherwise clean up a cache.

Related articles

• Next read: load shedding; the next rung of the resilience series: what a fleet drops, degrades, and defers when the work that arrives cannot all be served, the overload decision that assumes this article’s foundation.
• message queues, the delivery-guarantee anchor: at-most-once, at-least-once, and exactly-once as pipeline promises, dead letter queues, and the terminology this article links to instead of re-deriving.
• retry with exponential backoff; the pacing half of the pair: backoff, jitter, and retry budgets deciding when the safely-landable retries of this article actually fire.
• kafka architecture, offset management as the delivery dial: commit timing choosing between at-least-once and at-most-once, with the consumer-side idempotency this article supplies.
• fault-tolerant systems; the recovery machinery whose checkpoints and replays re-run work, and the practical reason idempotent design exists.
• event-driven architecture, fan-out as the multiplier: every subscriber an independent at-least-once problem with its own dedup state to maintain.
• client-server architecture; the method grammar where the vocabulary starts: safe versus idempotent versus neither, and why POST is where the work is.