Idempotency in APIs: Why It Matters and How to Actually Implement It

Networks fail in the middle. Mobile clients drop connections at the worst times. Load balancers retry. Users mash buttons. None of these are bugs — they're the environment your API runs in. Idempotency is the design property that keeps any of them from creating two charges, two orders, or two of anything that should only exist once.

This post walks through the practical mechanics — what idempotency really means, where to apply it, and how to implement it without painting yourself into a corner.

The definition that actually matters

The textbook definition: an operation is idempotent if calling it N times produces the same result as calling it once.

The useful definition for API design: the second identical call from the same client should not create a second side effect, but should return a sensible response that lets the client believe they succeeded.

That second part is the part most teams forget. It's not enough to skip the duplicate write — the client needs to know "yes, your charge went through, here's the receipt", not "unknown error".

When you actually need it

Three categories of operations need idempotency, in roughly decreasing order of urgency:

1. Things that move money. Charges, refunds, transfers, settlements. A duplicate is real harm to a real person.
2. Things that send notifications. Emails, SMS, push notifications, webhooks. A duplicate is a confused user and a CX ticket.
3. Things that mutate critical business state. Creating an order, scheduling a job, provisioning a resource. A duplicate is a data quality problem and an ops headache.

For everything else (analytics events, logs, idempotent reads), it's nice-to-have. For the three above, it's required.

The standard pattern: idempotency keys

The most widely adopted approach (Stripe, AWS, every payment processor):

The client generates a unique key per logical operation and sends it on every retry of that operation.

POST /charges
Idempotency-Key: 6bf8e4c2-2a1c-4b9c-9d3a-9e5b1c5b8c2e
Content-Type: application/json

{ "amount": 1999, "currency": "INR", "customer": "cus_42" }

The server stores the key, the request hash, and the eventual response. On retry:

1. Look up the key.
2. If found and complete → return the cached response. Don't re-execute.
3. If found and in-flight → return 409 (or wait briefly).
4. If not found → execute, store the response, then return.

The contract with clients is: "send the same key for retries; generate a new one for genuinely new operations."

The race condition that bites everyone

Naïve implementations have a window:

T1: Request A arrives, key not found. App starts executing.
T2: Request A (retry) arrives, key not found. App starts executing again.
T3: Both inserts succeed. Two charges. Sad customer.

The fix is making "check + claim" atomic. Two ways:

Database-side claim. Before doing the work, insert the key with status in_progress using a unique constraint:

INSERT INTO idempotency_keys (key, status) VALUES ($1, 'in_progress')
ON CONFLICT (key) DO NOTHING RETURNING id;

If the insert returns no row, someone else already claimed it. Wait briefly and re-read the row to get their result.

Redis claim. A SET key value NX EX 60 is atomic — only one caller wins the claim. Use a Lua script if you also want to read the previous result in one round trip.

Either way: never proceed past the claim step without atomically owning the key.

What to do with the claim

Once you've atomically claimed the key:

1. Execute the operation in a database transaction (if possible).
2. Inside the same transaction, update the idempotency record with the response.
3. Commit.

The key insight: keep the idempotency record and the business write in the same transaction if the same database is involved. Otherwise you can write the charge but crash before recording the response, and the next retry creates a duplicate.

If the business write goes to an external system (a payment processor, a third-party API), this gets harder. The pattern there is:

1. Claim the key.
2. Call the external system.
3. Persist the external system's response next to the idempotency record.
4. The external system itself should also be idempotent.

Most modern payment processors accept idempotency keys themselves — pass yours through. That way the chain is idempotent end-to-end.

What the response should look like

When a retry hits a completed key, return what you returned the first time, including the original status code:

201 Created
Idempotent-Replayed: true
{ "id": "ch_abc", "status": "succeeded", ... }

Returning the original 201 (not a 200 or 409) lets generic client code keep working. The Idempotent-Replayed header is optional, useful for debugging.

If the request body differs from the one that originally created the key (same key, different amount), reject with 409 Conflict. The client used the same key for two different operations — that's a bug they need to know about.

What to store

For each idempotency record:

• The key itself.
• A hash of the canonical request body (to detect "same key, different body").
• The HTTP status returned.
• The response body returned.
• A timestamp.
• A status: in_progress, completed, failed.

24 hours of retention is the typical default for payment APIs. Longer if your retries have longer windows.

Common mistakes

A few that have bitten me or teams I've worked with:

• Letting the server generate the key. Defeats the purpose. The whole point is the client generates the key once and reuses it across retries; if the server generates it, the client can't deduplicate its own retries.
• Not validating the request body matches. Same key, different amount → silently re-using the cached response. Customer gets the wrong receipt.
• Treating idempotency keys as authentication. They're not secret. Anyone with a key can replay the response. Auth is a separate layer.
• Skipping the database transaction. Idempotency record committed but business write rolled back, or vice versa. The next retry sees "completed" and returns a fake success.
• Cleaning up keys too aggressively. A retry storm 30 seconds after a 5-second TTL is the worst possible time for the key to be gone.

A minimal implementation sketch

In Python with FastAPI and Redis, the core check fits in 25 lines:

async def with_idempotency(key: str, body_hash: str, fn):
    claim = await redis.set(
        f"idem:{key}",
        body_hash + ":in_progress",
        nx=True, ex=86400,
    )
    if not claim:
        cached = await redis.get(f"idem:{key}")
        if cached.endswith(":in_progress"):
            raise ConflictError("retry shortly")
        stored_hash, _, payload = cached.partition(":done:")
        if stored_hash != body_hash:
            raise ConflictError("idempotency key reused with different body")
        return json.loads(payload)

    result = await fn()
    await redis.set(
        f"idem:{key}",
        f"{body_hash}:done:{json.dumps(result)}",
        ex=86400,
    )
    return result

Wrap any side-effecting handler in this and you've eliminated 99% of duplicate-write bugs from the very first deploy.

What to read next

Idempotency is half of "your API survives bad networks". The other half is rate limiting. And if you're picking an API style at the same time, REST vs GraphQL vs gRPC covers the layer above. The notification service LLD writeup shows what idempotency keys look like at object-model depth — exactly the surface where retries and duplicates collide most often.