In distributed systems, things fail in the most inconvenient ways. Not clean failures where you know something went wrong -- partial failures where you have no idea what happened. A client sends a payment request, the server processes it successfully, but the response gets lost in the network. The client, seeing a timeout, does the sensible thing: retry. And now the customer has been charged twice. This is not a hypothetical edge case. It happens constantly in production systems. Every time a mobile app hits a flaky cell connection, every time a load balancer times out, every time a TCP connection drops mid-response -- the client needs to retry. And without idempotency, every retry is a potential duplicate. The fix is straightforward in concept: make your operations **idempotent**. An idempotent operation produces the same result whether you execute it once or ten times. GET requests are naturally idempotent -- fetching a page twice gives you the same page. But POST requests (create a payment, send an email, place an order) are not idempotent by default. You have to make them so. ## Idempotency Keys The most common approach is the **idempotency key** pattern. The client generates a unique identifier (typically a UUID) and attaches it to every request. The server uses this key to detect retries and return the previously computed result instead of re-executing the operation. Here is how it works step by step: 1. **Client generates a key** -- a UUID like abc-123 -- and sends it as a header: Idempotency-Key: abc-123 2. **Server checks the dedup store** -- has this key been seen before? If yes, return the stored response immediately 3. **Server processes the request** -- executes the business logic, stores the result keyed by abc-123, and returns the response If the client retries with the same key, the server finds the cached result and returns it. The operation executes exactly once regardless of how many times the request is sent. The dedup store entry should include the idempotency key, a hash of the request body (to detect misuse where different requests reuse the same key), the response body, the HTTP status code, and a creation timestamp. Set a TTL on entries -- 24 to 72 hours is typical -- so the store does not grow unbounded. **See it in practice:** The [Payment System](/learn/system-design/questions/payment) question makes idempotency a first-class requirement -- every charge operation carries an idempotency key to prevent double billing when network timeouts trigger retries. The [Webhook](/learn/system-design/questions/webhook) question requires the same pattern from the receiver's side: merchants must deduplicate incoming webhook notifications using the event ID. The [Bookstore](/learn/system-design/questions/bookstore) question uses idempotency keys to prevent duplicate order submissions during checkout. ## Delivery Semantics: At-Most-Once, At-Least-Once, and Exactly-Once When you move beyond simple request-response and into **message processing** -- Kafka consumers, webhook handlers, background job workers -- you encounter the three delivery semantics. Understanding them is essential because they determine what guarantees your system provides. Here is the key insight: **true exactly-once delivery across distributed systems is generally impossible**. You cannot guarantee that a message is transmitted exactly once over an unreliable network. What you can do is combine **at-least-once delivery** (the message will definitely arrive, possibly more than once) with **idempotent processing** (processing it again has no additional effect). The result is **effectively exactly-once** semantics -- every message is processed, and duplicates are harmless. Most message brokers (Kafka, SQS, RabbitMQ) provide at-least-once delivery by default. Your job as the consumer is to make processing idempotent. **See it in practice:** In [Job Scheduler](/learn/system-design/questions/job-scheduler), workers pull jobs from a queue and must handle the case where a job is delivered twice due to visibility timeout expiration. In [Slack](/learn/system-design/questions/slack), message delivery to clients uses at-least-once semantics -- the client deduplicates based on message IDs. The [Pub/Sub and Messaging](/learn/system-design/design-patterns/pub-sub-messaging) and [Stream Processing](/learn/system-design/design-patterns/stream-processing-realtime-aggregation) patterns dive deeper into consumer group semantics and offset management. ## The Dedup Flow: Redis as a Fast Lookup Layer For API-level idempotency, Redis is the natural choice for the dedup store. It is fast (sub-millisecond lookups), supports TTL natively, and can handle the throughput of a high-traffic API without breaking a sweat. The flow is simple: 1. **Request arrives** with an Idempotency-Key header 2. **Check Redis**: GET idempotency:<key>. If the key exists, deserialize the stored response and return it immediately -- no database call, no business logic executed 3. **On cache miss**: execute the business logic, then SET idempotency:<key> <serialized_response> EX 86400 (24-hour TTL) 4. **Return the response** to the client There is a subtle race condition to watch for: what if two identical requests arrive at the exact same time, both see a cache miss, and both start processing? You can handle this with a Redis SET NX (set-if-not-exists) as a distributed lock. The first request acquires the lock and processes; the second sees the lock and either waits or returns a 409 Conflict. **See it in practice:** In [Distributed Cache](/learn/system-design/questions/cache), you design a Redis-like system that would serve as the dedup store in this pattern. In [Ads Click Aggregator](/learn/system-design/questions/adds-click-aggregator), deduplication prevents the same ad click from being counted twice when the event stream replays. The [Distributed Caching](/learn/system-design/design-patterns/distributed-caching) pattern covers Redis topology, eviction, and replication strategies. The [Rate Limiting](/learn/system-design/design-patterns/rate-limiting) pattern also uses Redis as a fast shared-state layer. ## Database-Level Idempotency: Upserts and Conditional Updates Sometimes you do not need a separate dedup store at all. The database itself can enforce idempotency through its own constraint mechanisms. This approach is simpler (fewer moving parts) and naturally transactional. Three techniques: ### INSERT ON CONFLICT DO NOTHING Use the idempotency key (or the message's natural unique ID) as a database column with a unique constraint. Duplicate inserts silently fail: INSERT INTO payments (idempotency_key, amount, user_id) VALUES ('abc-123', 50, 42) ON CONFLICT (idempotency_key) DO NOTHING This is the simplest form of idempotency. No dedup table, no Redis lookup -- the database's unique index handles everything. The trade-off is that you do not get the previous response back (you just know the insert was a duplicate), so it works best when the caller only needs to know "this was processed" rather than needing the exact original response. ### Conditional Updates State-machine transitions are naturally idempotent when you include the current state in the WHERE clause: UPDATE orders SET status = 'shipped' WHERE id = 123 AND status = 'processing' The first execution updates the row. The second execution matches zero rows because the status is no longer processing -- it is already shipped. No error, no side effect, naturally idempotent. ### Dedup Table in the Same Transaction For cases where you need both dedup and business logic atomicity, insert into a dedup table within the same database transaction as your business write: 1. BEGIN TRANSACTION 2. INSERT INTO processed_messages (message_id) VALUES ('msg-456') -- fails on duplicate 3. INSERT INTO orders (id, product, quantity) VALUES (...) 4. COMMIT If the message is a duplicate, step 2 fails (unique constraint violation) and the entire transaction rolls back. If it is new, both the dedup record and the business data are written atomically. **See it in practice:** In [Google Calendar](/learn/system-design/questions/google-calendar), event creation uses upserts to handle duplicate submissions when the calendar UI retries. In [Online Chess](/learn/system-design/questions/online-chess), move submissions use conditional updates -- a move is only accepted if the game is in the expected state (correct turn, piece in expected position). In [CI/CD Pipeline](/learn/system-design/questions/cicd), pipeline step execution uses idempotency keys to prevent duplicate builds when webhooks fire multiple times. The [Dealing with Contention](/learn/system-design/design-patterns/dealing-with-contention) pattern covers optimistic concurrency control, which pairs naturally with conditional updates. The [Payment Systems](/learn/system-design/design-patterns/payment-systems-financial-consistency) pattern goes deeper into financial-grade idempotency requirements. ## Transactional Outbox: Idempotency Across Two Systems A common need is to update a database **and** publish an event to a message broker (Kafka, SQS) as part of the same logical operation. Writing to two systems without a shared transaction is the **dual-write problem**: if the database write succeeds but the Kafka publish fails (or vice versa), the systems become inconsistent. The **transactional outbox** pattern solves this by writing both the business data and the event in a single database transaction: 1. BEGIN TRANSACTION 2. Write business data (e.g., INSERT INTO orders) 3. Write event to an outbox table (e.g., INSERT INTO outbox with event payload and status = 'pending') 4. COMMIT -- both succeed or both fail atomically A separate **relay process** (polling the outbox table or using change data capture) reads pending events and publishes them to Kafka. After successful publish, it marks the event as sent. The relay itself must be idempotent -- publishing the same Kafka message twice with the same key is safe because consumers are idempotent (using the techniques above). The transactional outbox is the most widely adopted approach because it uses only the existing database -- no distributed transaction coordinator needed. It pairs naturally with idempotent consumers downstream: even if the relay publishes a message twice (crash after publish but before marking as sent), the consumer deduplicates it. **See it in practice:** In [Payment System](/learn/system-design/questions/payment), the transactional outbox ensures payment events are published reliably without dual-write inconsistencies -- critical when downstream services need to update balances or send receipts. In [Uber](/learn/system-design/questions/uber), ride state changes (requested, accepted, completed) use an outbox to reliably notify pricing, billing, and notification services. The [Event Sourcing and CQRS](/learn/system-design/design-patterns/event-sourcing-cqrs) pattern takes this further by making events the primary source of truth. The [Multi-Step Processes and Sagas](/learn/system-design/design-patterns/multi-step-processes-sagas) pattern covers coordinating idempotent steps across multiple services. ## Choosing the Right Approach Not every endpoint needs the same level of idempotency protection. Here is a practical decision framework: Start with the simplest technique that meets your requirements. A conditional UPDATE costs you nothing extra. A unique constraint on an idempotency key column is one line of DDL. Only reach for Redis-based dedup when you need to return the cached response body or when your throughput demands are too high for database lookups on every request.