Idempotency keys: making retries safe in any API

You send a request to create an order. The connection drops before you get a reply. Did the order exist? You cannot tell. If you retry and the first attempt had worked, the customer now has two orders. If you do not retry and it had failed, they have none.

This is not a rare edge case. Any system that talks over a network meets it daily. The usual answer is the idempotency key, an idea Stripe described clearly in their engineering post on the subject. This article explains the idea in my own words and shows one way to build it.

What idempotent means

An operation is idempotent if doing it many times has the same effect as doing it once. Setting a value to 5 is idempotent. Adding 1 to a counter is not. In HTTP, PUT and DELETE are defined to be idempotent, but POST, which we use for "create something", is not. Payments, orders, emails and job submissions mostly live in POST territory, which is exactly where double execution hurts.

The idea of a key

The client generates a unique identifier for each logical operation, a random UUID works well, and sends it with the request, commonly in an Idempotency-Key header. The server remembers what it did for that key. When the same key arrives again, the server does not repeat the work. It returns the saved result.

Now the client can follow a simple rule: keep retrying until you get a definite answer, always with the same key for the same operation. The server guarantees the effect happens once.

Two details trip people up:

• One key means one operation. Generate the key when the user's intent is formed (they click "pay"), not on every attempt. Reuse it for every retry of that intent.
• Create a new key for a new intent. Reusing a key for a different request is a bug, and the server should say so.

A server-side design

You need a place to store keys with their outcome. A database table is the natural choice, because you want the key record and the business change to commit together.

CREATE TABLE idempotency_keys (
    key text NOT NULL,
    account_id uuid NOT NULL,
    request_hash text NOT NULL,
    status_code int,
    response jsonb,
    created_at timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (account_id, key)
);

Scoping by account stops one customer's keys from colliding with another's. The request hash lets you detect misuse. The handler then goes roughly like this:

1. Start a transaction.
2. Try to insert the key row. The primary key makes this safe against two requests racing with the same key.
3. If the insert succeeded, do the work, store the response on the key row, and commit.
4. If the key already existed with a stored response, compare the request hash. If it matches, return the stored response unchanged. If it differs, return an error (HTTP 422 is a common choice).
5. If the key exists but has no response yet, another request is still working. Return a "conflict, try again shortly" answer rather than starting the work twice.

Because the key row and the business change commit in one transaction, you never end up with work done but no record, or a record but no work.

When the work leaves the database

The tidy transaction story breaks when the work includes calling another service, say charging a card. If your process dies after the charge but before you record it, a retry will charge again unless the external call also gets an idempotency key. The robust approach is to pass your key (or one derived from it) along to the downstream service, so that safety extends along the chain.

For sending messages or events after a commit, a related pattern writes the message to an outbox table in the same transaction and has a separate worker deliver it. The worker may deliver more than once, so receivers should deduplicate by an identifier too. Idempotency is a habit that has to run through every hop.

Retrying well: backoff and jitter

Retries are only half the story. If a server is struggling and a thousand clients retry instantly, they make it worse. Two standard techniques help:

• Exponential backoff: wait longer after each failure, roughly doubling the delay.
• Jitter: add randomness to each wait, so clients that failed together do not all come back at the same moment, which is often called a thundering herd.

A small Go-flavoured sketch:

func retry(ctx context.Context, attempts int, fn func() error) error {
	delay := 200 * time.Millisecond
	var err error
	for i := 0; i < attempts; i++ {
		if err = fn(); err == nil {
			return nil
		}
		// full jitter: wait a random time between 0 and delay
		wait := time.Duration(rand.Int63n(int64(delay)))
		select {
		case <-time.After(wait):
		case <-ctx.Done():
			return ctx.Err()
		}
		delay *= 2
	}
	return fmt.Errorf("giving up after %d attempts: %w", attempts, err)
}

Add a maximum delay in real code, and only retry errors that are worth retrying: timeouts and server errors, not "your request is invalid".

What to store and for how long

You do not need to keep keys forever. A retention window of a day or so covers most client retry policies. Choose a window longer than your clients' longest retry, say so in your documentation, and clean up with a scheduled job. Do not put secrets or full card numbers in the saved response; store what the client needs to see.

Where this matters for AI agents

Automated agents make this even more relevant. An agent that calls tools may repeat a call after a timeout, or a protocol may re-send a request after a broken stream, as the recent MCP revision now explicitly expects clients to do. If a tool sends an email or creates a ticket, an idempotency key is the difference between one email and five. At vvek we require a key on every operation that changes state, for exactly this reason.

A short checklist

• Accept an idempotency key on every request that changes state.
• Scope keys by account and store a hash of the request.
• Commit the key and the business change together.
• Return the saved response on a replay; reject a mismatched request.
• Pass keys downstream to services you call.
• Retry with exponential backoff and jitter, and cap the attempts.
• Expire old keys on a schedule you document.

None of this is hard, but it is much easier to build on day one than to add after the first duplicate charge.

Sources

• Stripe: Designing robust and predictable APIs with idempotency