Imagine you are building an e-commerce checkout. A customer clicks "Buy" and your system needs to create the order, charge their credit card, reserve inventory, and schedule shipping. Each of those steps lives in a different service. In a monolith, you would wrap the whole thing in a database transaction -- if any step fails, the database rolls everything back automatically. But in a microservices world, there is no single database to coordinate this. Each service has its own data store, and you cannot wrap four independent databases in one ACID transaction. So what happens when the payment goes through but the inventory service discovers the item is out of stock? You have charged the customer for something you cannot ship. This is the **distributed transaction problem**, and it comes up constantly in system design interviews. ## The Problem: No Distributed ACID Traditional databases give you atomicity -- either all changes commit or none do. But the moment your workflow spans multiple services, that guarantee disappears. Two-phase commit (2PC) technically solves this by having a coordinator ask all participants to "prepare" and then "commit," but it has serious downsides: it blocks all participants while waiting for the coordinator, it does not scale, and if the coordinator crashes mid-protocol, participants can be left in a locked state indefinitely. In practice, almost no production system uses 2PC across microservices. Instead, we embrace the reality that distributed systems will have partial failures and design patterns that handle them gracefully. The two main coordination approaches are **choreography** and **orchestration**, and the key failure-handling pattern is the **Saga**. ## Choreography: Let Services React to Events In choreography, there is no central brain. Each service publishes events when it finishes its work, and other services listen for those events and act accordingly. The Order Service publishes an OrderCreated event; the Payment Service hears it and charges the card; it then publishes PaymentCharged; the Inventory Service hears that and reserves stock; and so on. The beauty of choreography is **loose coupling**. No service knows about the others -- it just publishes what happened and listens for what it cares about. Adding a new step (say, sending a confirmation email) is as simple as deploying a new service that subscribes to the right event. You do not need to modify any existing service. The downside is **visibility**. The overall business workflow is scattered across multiple service codebases. When something goes wrong, tracing the flow from "order created" to "shipment scheduled" means hopping between service logs and event streams. There is no single place that shows you the current state of the workflow. Choreography works best for **loosely coupled, eventually consistent flows** where you do not need strict ordering or centralized error handling -- things like sending notifications, updating analytics, or syncing search indexes. **See it in practice:** The [Pub/Sub and Messaging](/learn/system-design/design-patterns/pub-sub-messaging) pattern covers the event bus infrastructure that choreography relies on. In the [Slack](/learn/system-design/questions/slack) question, message delivery to channels uses event-driven fanout where each component reacts independently. The [Fan-Out and Feed Generation](/learn/system-design/design-patterns/fan-out-feed-generation) pattern uses choreography when a new post triggers independent updates to followers feeds. ## Orchestration: A Central Coordinator Drives the Flow In orchestration, a dedicated **orchestrator service** (or workflow engine) explicitly calls each step in sequence, waits for the result, and decides what to do next. It is like a function that calls other services: 1. Call Order Service to create order 2. Call Payment Service to charge card 3. Call Inventory Service to reserve stock 4. Call Shipping Service to schedule delivery 5. If any step fails, run compensations in reverse The orchestrator knows the entire workflow. You can look at one piece of code and understand the full business process, including error handling. Debugging is straightforward because the orchestrator logs every step and its result. The trade-off is that the orchestrator becomes a **critical piece of infrastructure**. If it goes down, no workflows make progress. In practice, this is mitigated by using battle-tested workflow engines like **Temporal**, **AWS Step Functions**, or **Cadence** that provide durable execution -- they persist the state of each workflow step so that if the orchestrator crashes, it resumes from the last successful step after recovery. Orchestration works best for **critical business workflows** where you need visibility, control, and reliable error handling -- things like payment processing, order fulfillment, and user onboarding. **See it in practice:** In the [Payment](/learn/system-design/questions/payment) question, the authorize-capture-settle pipeline is a classic orchestrated workflow where each step must succeed before the next begins, and failures trigger refunds or void authorizations. The [Job Scheduler](/learn/system-design/questions/job-scheduler) question involves orchestrating multi-step job execution with retries and failure handling. The [CI/CD](/learn/system-design/questions/cicd) question coordinates build-test-deploy pipelines where each stage depends on the previous one succeeding. ## The Saga Pattern: Handling Failure with Compensation A **Saga** is the specific pattern for managing failures in multi-step processes. The idea is simple: every forward step has a corresponding **compensating transaction** that semantically undoes its effect. If step 3 fails, you run compensations for steps 2 and 1 in reverse order. There is an important subtlety here: compensating transactions are **not true rollbacks**. When you issue a refund, the customer still sees a charge and a refund on their credit card statement -- it is not the same as never having charged them. Sagas provide **semantic rollback**, meaning the end state is logically equivalent to "nothing happened," even though intermediate states were visible. This is fundamentally different from a database ROLLBACK that truly erases all traces. ### Compensations Must Be Idempotent Because the compensation itself can fail (networks are unreliable), the system will retry it. That means your compensating transactions must be **idempotent** -- executing them multiple times must produce the same result as executing them once. For example, your refund logic should check whether a refund was already processed before issuing a new one. Without idempotency, a retried compensation could refund a customer twice. **See it in practice:** The [Idempotency and Exactly-Once Semantics](/learn/system-design/design-patterns/idempotency-exactly-once) pattern covers the techniques for making operations safe to retry -- idempotency keys, deduplication, and at-least-once delivery with idempotent handlers. The [Payment Systems and Financial Consistency](/learn/system-design/design-patterns/payment-systems-financial-consistency) pattern goes deep on compensation in financial contexts where correctness is non-negotiable. ## Workflow State Machines A well-designed saga tracks its progress through a **state machine**. At any point, the saga is in one of a handful of states, and transitions between states are triggered by step completions or failures. The orchestrator **persists the current state** to a durable store (database or workflow engine). This is what makes sagas crash-resilient: if the orchestrator process dies mid-saga, a new instance picks up the saga from the last persisted state and continues. Without state persistence, a crash during step 3 would leave the saga in limbo -- payment charged, inventory partially reserved, no one aware that compensation is needed. This is exactly what workflow engines like **Temporal** and **AWS Step Functions** give you out of the box. You write your saga as regular code (or a state machine definition), and the engine handles state persistence, retries, and crash recovery automatically. You do not need to build this infrastructure yourself. ## Practical Design Decisions ### When to Use Sagas vs. Simpler Approaches Not every multi-step process needs a saga. If all your steps happen within one service and one database, use a regular database transaction. Sagas add complexity -- only reach for them when your workflow genuinely spans multiple services with independent data stores. A good rule of thumb: if you can draw a line around the steps and they all touch the same database, use a transaction. If different steps touch different databases or external APIs, you need a saga. ### Saga + Event Sourcing Sagas pair naturally with **event sourcing**. When every state change is recorded as an immutable event, you get a complete audit trail of what happened during the saga -- which steps completed, which failed, which compensations ran. This is invaluable for debugging and compliance, especially in financial systems. **See it in practice:** The [Event Sourcing and CQRS](/learn/system-design/design-patterns/event-sourcing-cqrs) pattern covers how to store state changes as events rather than mutable rows, which pairs perfectly with saga compensation tracking. In the [Google Calendar](/learn/system-design/questions/google-calendar) question, creating a multi-attendee event involves reserving time slots, sending invitations, and updating calendars across users -- where partial failures must be handled gracefully. The [Bookstore](/learn/system-design/questions/bookstore) question involves a checkout flow spanning inventory, payment, and fulfillment services that maps directly to the saga pattern. The [Uber](/learn/system-design/questions/uber) question coordinates ride-matching, pricing, payment, and driver assignment across multiple services with compensation for cancelled rides. ## Common Pitfalls **Forgetting non-compensatable steps.** Some actions cannot be undone -- you cannot unsend an email or un-notify a customer. Place these steps last in the saga so they only execute after all compensatable steps have succeeded. Alternatively, use a "pending" state and only trigger the irreversible action after confirmation. **Not handling compensation failures.** Your refund API can fail too. Always implement retry logic with exponential backoff for compensations. If a compensation permanently fails after exhausting retries, alert an operator -- this is a case that requires manual intervention. **Mixing sync and async steps without care.** If some steps are synchronous API calls and others are async event-driven, make sure your orchestrator handles both correctly. A common bug is marking an async step as "complete" when the message is sent rather than when the downstream service confirms it processed the message.