Payment systems have a unique constraint that sets them apart from every other distributed system you will design: **correctness above all else**. A double charge erodes customer trust. A missed refund creates legal liability. An inconsistent ledger fails an audit. While most systems optimize for availability or performance and tolerate occasional inconsistency, financial systems must be exactly right -- every transaction must be accounted for, every state transition must be valid, and every failure must be handled without losing or duplicating money. Let us walk through the core patterns that make this possible. ## The Payment Flow Before diving into the safeguards, it helps to see the happy path. When a customer clicks "Pay," the request travels through several services before money actually moves. The Client sends a payment request to your **Payment Service**, which validates the request and records it in the ledger database. It then forwards the authorization request to a **Payment Gateway** (Stripe, Adyen, etc.), which communicates with the customer's **Bank**. Confirmations flow back through each layer until the client receives a receipt. The critical detail here is that every step is recorded. If the gateway times out after sending the charge to the bank, your system knows the payment is in an authorized state and can query the gateway later to find out what happened. Nothing is lost in the void. ## The Payment State Machine A payment is not just "paid" or "not paid." It moves through a well-defined set of states, and only specific transitions are allowed. This is the backbone of correctness in any payment system. The happy path is Created to Authorized to Captured to Settled. But from any pre-settlement state, the payment can transition to Failed (the bank declines it). And from Captured or Settled, it can transition to Refunded (the merchant returns the money). The state machine prevents invalid transitions -- you cannot refund a payment that was never captured, and you cannot capture a payment that was never authorized. Every transition is recorded as an immutable event, creating a complete audit trail of the payment's lifecycle. Why separate authorization from capture? Because many businesses need the gap. A hotel authorizes your card at check-in (reserving the funds) but only captures the final amount at checkout. An e-commerce site authorizes at order time but captures only when the item ships. This two-phase approach is not just a nice-to-have -- it is how the card networks actually work. ## Idempotency Keys: Preventing Double Charges Here is the scariest failure mode in payments: the client sends a charge request, the server processes it, but the response is lost due to a network timeout. The client does not know if the charge went through. If it retries, the server might process the payment again -- resulting in a double charge. **Idempotency keys** solve this elegantly. The client generates a unique UUID for each payment attempt and includes it in the request header. The server stores this key alongside the result. On retry (same idempotency key), the server looks up the key, finds the cached result, and returns it without reprocessing. No matter how many times the client retries, the payment is processed **exactly once**. The implementation is straightforward: before processing a payment, check if the idempotency key already exists in your store. If it does, return the stored result. If it does not, process the payment, store the result keyed by the idempotency key, and return the result. The key store can be Redis (fast lookups) or a database column with a unique constraint (strong durability). **See it in practice:** The [Idempotency and Exactly-Once Delivery](/learn/system-design/design-patterns/idempotency-exactly-once) pattern dives deeper into exactly-once semantics, covering deduplication strategies, the inbox/outbox pattern, and how to handle idempotency across distributed systems. The [Payment System](/learn/system-design/questions/payment) question has you implement idempotency keys as part of a full payment processing system. ## Double-Entry Ledger: Money Never Disappears In accounting, there is a rule that has survived 500 years because it works: every transaction must have a **debit** and a **credit** that balance to zero. This is the double-entry bookkeeping system, and it is the foundation of financial correctness. When a customer pays a merchant, your ledger records two entries: a debit of the customer's account (money leaving) and a credit to the merchant's account (money arriving). The sum is always zero. When a refund happens, you do not delete the original entries -- you create two new entries that reverse them. This approach gives you several powerful properties: - **Auditability:** You can reconstruct the balance of any account at any point in time by replaying the entries. - **Error detection:** If your ledger does not sum to zero, something is wrong and you know it immediately. - **Immutability:** Records are never modified or deleted, only appended. This makes the system tamper-evident. This is essentially [Event Sourcing](/learn/system-design/design-patterns/event-sourcing-cqrs) applied to money. The current balance is a derived value, computed by replaying all entries. You never store the balance directly -- you compute it from the log. ## Reconciliation: Trust but Verify Even with idempotency keys and double-entry ledgers, things can go wrong. A webhook from the payment gateway might be lost. A bank settlement might fail silently. Your internal ledger might say a payment was captured, but the bank never actually received the funds. **Reconciliation** is the process of comparing your internal records against the bank's records to find and resolve discrepancies. A reconciliation job runs on a schedule (hourly or daily) and performs a three-way comparison: 1. **Match transactions** by ID between your ledger and the bank statement. 2. **Flag discrepancies** -- transactions in your ledger but not in the bank statement (or vice versa), and amount mismatches. 3. **Generate alerts** for the finance team to investigate and resolve. Common discrepancies include: a payment you marked as "captured" that the bank never settled (missed webhook), a bank charge that does not appear in your ledger (orphaned transaction), and amount differences due to currency conversion rounding. This is your safety net. Even if every other safeguard fails, reconciliation will catch it -- just with a delay. ## Security: PCI Compliance Credit card data is extremely sensitive and regulated by PCI DSS (Payment Card Industry Data Security Standard). The simplest way to achieve compliance is to ensure card data **never touches your servers**. Use a **client-side iframe** served directly from the payment processor (Stripe Elements, Adyen Drop-in). The customer enters their card details in the iframe, which submits directly to the processor's servers. Your server receives only a **token** representing the card -- never the actual card number. This dramatically reduces your PCI compliance scope from hundreds of requirements to a simple self-assessment questionnaire. ## Webhook Delivery for Merchant Notifications To notify merchants of payment events (successful charge, refund, dispute), you need reliable webhook delivery. A CDC consumer reads payment events from the database's write-ahead log and publishes them to a message queue like Kafka. Delivery workers consume from the queue and send HTTP POST requests to merchant endpoints. If the merchant does not respond with HTTP 200, the worker retries with **exponential backoff** (1 minute, 5 minutes, 30 minutes, 2 hours) up to a maximum number of attempts. Each webhook includes an idempotency key so the merchant can safely handle duplicates. After all retry attempts are exhausted, the event goes to a dead letter queue for manual review. **See it in practice:** The [Payment System](/learn/system-design/questions/payment) question is the complete application of this pattern -- you design the PaymentIntent state machine, implement idempotency keys for exactly-once processing, build the double-entry ledger, and set up reconciliation. The [Webhook](/learn/system-design/questions/webhook) question focuses specifically on reliable notification delivery with retry logic and dead letter queues. For the event-sourcing approach behind the immutable ledger, see the [Event Sourcing and CQRS](/learn/system-design/design-patterns/event-sourcing-cqrs) pattern. The [Pub/Sub Messaging](/learn/system-design/design-patterns/pub-sub-messaging) pattern covers the message queue infrastructure that powers asynchronous payment processing and webhook delivery. ## Key Takeaways The common thread across all these patterns is the same: **assume failures will happen and design so that failures are detectable and recoverable**. State machines make invalid transitions impossible. Idempotency keys make retries safe. Double-entry ledgers make accounting errors visible. Reconciliation catches everything else. Together, they give you a payment system that is not just functional, but provably correct.