Networks fail in the one way that makes idempotency non-optional: a client sends a request, the server processes it and crashes before the response arrives, and the client — having no idea whether the request landed — retries. Without idempotency, that retry double-charges a card or double-ships an order. With it, the retry is a no-op.

Idempotency Keys

The standard mechanism is a client-generated unique key attached to the request, which the server uses to recognize a retry as the same logical operation rather than a new one.

python
def charge_card(request):
    idempotency_key = request.headers["Idempotency-Key"]
    existing = db.idempotency_records.find(idempotency_key)
    if existing:
        return existing.response  # replay, don't reprocess

    result = payment_gateway.charge(request.amount, request.card)
    db.idempotency_records.insert(idempotency_key, result, ttl_days=7)
    return result

The key has to be generated by the client, not the server — a server-generated key on each retry would just create a new record every time, defeating the point. UUIDs generated once per logical user action (not per HTTP attempt) are the usual choice.

The Race Between Check and Write

The naive version above has a bug: two concurrent retries of the same request can both pass the find before either finishes the insert, and both go on to charge the card. The fix is to make the check-and-reserve atomic — a unique constraint on the idempotency key column, with the second insert failing and the failing request waiting for or reading the first one's result, rather than a separate read-then-write.

sql
INSERT INTO idempotency_records (key, status)
VALUES ($1, 'in_progress')
ON CONFLICT (key) DO NOTHING
RETURNING key;
-- if no row returned, another request already owns this key

Idempotency Is Not Automatic From HTTP Verbs

A common misconception is that PUT is idempotent and POST isn't, full stop, and that's the end of the analysis. PUT /accounts/1/balance with {"balance": 500} is idempotent by the spec's definition (same request, same end state) but is almost never what you want for a financial operation — two racing clients both trying to set balance to different values will silently clobber each other. What you actually want is often expressed as a POST with an idempotency key: POST /accounts/1/transactions with {"amount": -50, "idempotency_key": "..."} is semantically an append, safe to retry, and doesn't have the last-writer-wins problem PUT does.

Where Else It Matters

  • Message consumers: at-least-once delivery (SQS, Kafka with default settings) means every consumer has to treat redelivery as normal, not exceptional — the same key-and-check pattern applies to message processing, not just HTTP.
  • Downstream calls inside a saga step: if a step itself calls an external API, that call needs its own idempotency key, independent of the saga's — a saga retry shouldn't fan out into duplicate calls to a third party.

Idempotency isn't a property you add at the edge once and forget — it has to be threaded through every hop that might retry, because the guarantee is only as strong as its weakest link.

A worked example

Client sends Idempotency-Key: uuid with POST /charges. Server stores key → response in a table with unique constraint. Retry with the same key returns the stored 200 without a second charge. Different body with the same key is 409. TTL 24h. The handler commits the key and the side effect in one transaction (or outbox).

A test: two parallel POSTs with the same key, one charge row.

Failure modes

Keys only in memory. Not storing the response, so retries 500 after success. Keying only on user id. GET treated as non-idempotent in docs but POSTs retried by gateways. Unique constraint missing under race. Side effects outside the transaction (email sent twice).

Reusing keys for different operations.

When this is the wrong tool

Pure GETs should already be idempotent without a key. Do not add keys to every internal call if a natural key exists (order_id). At-most-once with acceptable loss may be enough for metrics. Idempotency will not fix a non-deterministic handler that stores "now()" as part of the resource identity. For fully exactly-once with brokers, you still need transactional outbox plus consumer idempotency — a header alone is not a bus.

A worked failure mode

Retries use a new UUID each time. A server stores idempotency in memory. A client retries a non-idempotent side effect with the same key but a different body; the server ignores the mismatch. The failure is a key that does not bind to the request. Persist keys, hash the body, same response on replay.

Idempotency keys are the wrong tool if the downstream cannot be made idempotent and you have no compensation. Do not skip them on payments. Use them at every retry boundary that can double-apply.

A second, quieter failure is operational: the idea is copied from a talk into a path that has no rollback, no owner, and no metric that would show the invariant breaking. For "Idempotency in Distributed Systems: Making Retries Safe", that usually means a Friday deploy with production as the first realistic test. Write down the user-visible symptom, the invariant, and the revert before you scale the pattern. If revert is a data rewrite, you do not have a revert—you have a project. Practice the failure in staging with production-sized data at least once, or you will practice it on customers.