Some operations simply take too long to fit inside a synchronous HTTP request-response cycle. Video transcoding, PDF generation, ML model inference, large data exports -- these can run for minutes or even hours. Clients cannot hold an HTTP connection open that long, and servers should not tie up threads waiting for slow work to finish. If you try, you will hit gateway timeouts, wasted resources, and frustrated users. The solution is straightforward: **decouple submission from execution**. Accept the request, hand the work off to a background worker, and let the client check back later. This is the async job pattern, and it shows up in nearly every large-scale system. ## Why Synchronous Breaks Down In a synchronous flow, the client sends a request and blocks until the server responds. That works fine for operations that complete in milliseconds -- fetching a user profile, validating a form. But when the operation takes minutes, everything falls apart. HTTP load balancers and proxies typically enforce timeouts of 30-60 seconds. Even if your server can handle a 10-minute operation, the infrastructure between the client and server will kill the connection. The client gets a 504 Gateway Timeout, retries, and now you have two copies of the same expensive operation running simultaneously. The async approach eliminates this entirely. The server accepts the request, enqueues a job, and immediately returns HTTP 202 (Accepted) with a task ID. The client uses that ID to check status later. The actual work happens in a background worker that can take as long as it needs. ## The Async Task Pattern Here is the full lifecycle of an async task, from submission to result retrieval. **Step 1 -- Submit:** The client sends a POST request to create the task. The API server validates the input, generates a unique task ID, and enqueues a job message onto a queue (Kafka, SQS, Redis, RabbitMQ). **Step 2 -- Respond immediately:** The API server returns HTTP 202 with the task ID. The client is free to navigate away, close the tab, or do other work. **Step 3 -- Process:** A worker process dequeues the job and executes it. This could take seconds or hours -- it does not matter because no one is waiting synchronously. **Step 4 -- Update status:** As the worker progresses, it writes status updates to a shared store (Redis for speed, or a database for durability). When done, it marks the task as completed and stores the result. **Step 5 -- Retrieve:** The client polls GET /tasks/task-id to check status. Once the task is completed, the response includes the result or a download URL. ### Polling vs Push Notification There are two ways for the client to learn when the task is done: Polling is the simplest and most common choice. Use exponential backoff on the polling interval (1s, 2s, 4s, 8s, capped at 30s) to avoid hammering the status endpoint on long tasks. **See it in practice:** In the [Job Scheduler](/learn/system-design/questions/job-scheduler) question, you design the full lifecycle from task submission through worker execution with lease-based fault tolerance. The [CI/CD](/learn/system-design/questions/cicd) question manages long-running build and test pipelines where each step may take minutes. The [File System](/learn/system-design/questions/file-system) question uses async processing for large file operations like virus scanning and thumbnail generation after upload. The [Pushing Realtime Updates](/learn/system-design/design-patterns/pushing-realtime-updates) pattern covers WebSocket and SSE in depth for when polling is not responsive enough. ## Heartbeat-Based Fault Detection Workers crash. Machines lose power. Network partitions happen. If a worker dies mid-task with no detection mechanism, the task sits in a "running" state forever -- no one retries it, no one notices. The solution is **heartbeats**. While processing a task, the worker sends periodic signals (every 10-30 seconds) to a central store. A supervisor process monitors these heartbeats. If a worker misses heartbeats beyond a configurable threshold (say, 60 seconds), the supervisor assumes the worker is dead and marks the task as failed, making it eligible for retry by another worker. Heartbeats serve a dual purpose. They prove the worker is alive, and they can carry progress information -- the worker can include how far along it is in each heartbeat, giving the system (and the user) visibility into long-running operations. A common implementation uses a **lease model**: the worker acquires a time-limited lease on the task (e.g., 60 seconds). It must renew the lease before it expires. If the lease expires without renewal, another worker can claim the task. This avoids the need for a separate supervisor process -- the lease expiry is the detection mechanism. **See it in practice:** In the [Job Scheduler](/learn/system-design/questions/job-scheduler) question, workers acquire leases on tasks and must renew them periodically -- if a worker crashes, the expired lease makes the task available for other workers. The [Remote DevBox](/learn/system-design/questions/remote-devbox) question uses heartbeats to detect when a cloud development environment becomes unresponsive, triggering automatic recovery. The [Online Chess](/learn/system-design/questions/online-chess) question uses heartbeat-like mechanisms to detect when a player disconnects mid-game. ## Checkpointing for Long Tasks Some tasks run for hours -- exporting millions of records, processing a massive video file, training a model. If such a task fails at 90% completion, restarting from the beginning wastes enormous time and compute. The answer is **checkpointing**: periodically saving progress so you can resume from the last known good state instead of starting over. The worker divides its work into chunks and, after completing each chunk, writes a checkpoint to durable storage (a database row, an S3 object, or a Redis key). The checkpoint captures enough state to resume: which records have been processed, which page of results you are on, the current offset in a file. When a failed task is retried, the new worker checks for an existing checkpoint. If one exists, it loads the saved state and continues from there. If not, it starts from the beginning. **What makes a good checkpoint:** - **Atomic and consistent:** The checkpoint must be written completely or not at all. A half-written checkpoint is worse than none. - **Minimal but sufficient:** Store the offset/cursor and any accumulated state needed to resume. Do not store the entire dataset. - **Periodic but not too frequent:** Checkpointing has overhead (a write to durable storage). Every 1,000 records or every 30 seconds is typical. The trade-off is between lost work on failure and checkpointing overhead during normal execution. **See it in practice:** In the [ChatGPT](/learn/system-design/questions/chatgpt) question, long inference tasks can checkpoint intermediate generation state so that a server failure does not force regeneration from scratch. The [Web Crawling and Data Pipelines](/learn/system-design/design-patterns/web-crawling-data-pipelines) pattern uses checkpointing extensively -- a web crawl processing millions of URLs saves its frontier state periodically so it can resume after failures without re-crawling already-visited pages. The [Stream Processing and Realtime Aggregation](/learn/system-design/design-patterns/stream-processing-realtime-aggregation) pattern uses Kafka consumer offsets as a form of checkpointing to track exactly which events have been processed. ## Progress Tracking For user-facing tasks, a binary "pending or done" status is not enough. Users want to know whether the task is 10% done or 90% done, and whether they should wait 30 seconds or 30 minutes. A **task status store** (typically Redis for low-latency reads) holds the current state of each task: - **status:** pending, running, completed, failed - **progress:** a percentage or item count (`processed 45,000 of 100,000 records`) - **message:** a human-readable description of what is happening now (`"Generating thumbnails..."`) - **eta:** estimated time remaining, computed from the current processing rate - **result:** the output (a download URL, a summary, etc.) once the task completes The worker updates these fields as it progresses. The client polls the status endpoint and renders a progress bar or status message. This gives users confidence that the system is working and helps them decide whether to wait or come back later. **See it in practice:** In the [CI/CD](/learn/system-design/questions/cicd) question, build pipelines show step-by-step progress as each stage (compile, test, deploy) completes, so developers know exactly which stage is running. The [Bookstore](/learn/system-design/questions/bookstore) question tracks order fulfillment progress through multiple stages from payment to shipping. The [Payment](/learn/system-design/questions/payment) question exposes transaction status so merchants and users can see whether a payment is processing, succeeded, or failed. ## Retry Strategies and Dead Letter Queues Tasks fail. Networks time out, downstream services return errors, input data is malformed. A good long-running task system handles failures gracefully rather than silently dropping work. ### Retries with Exponential Backoff When a task fails due to a transient error (network timeout, temporary service unavailability), retry it with increasing delays -- 1 second, 2 seconds, 4 seconds, 8 seconds. Add random jitter to prevent thundering herds when many tasks retry simultaneously. Set a **maximum retry count** (typically 3-5). After exhausting all retries, the task needs special handling rather than infinite retry loops. ### Dead Letter Queue (DLQ) After all retries are exhausted, move the failed task to a **Dead Letter Queue** rather than discarding it. The DLQ serves three purposes: 1. **Investigation:** Engineers can inspect failed tasks to diagnose bugs or data issues. 2. **Manual retry:** Once the root cause is fixed, tasks can be replayed from the DLQ. 3. **Poison message isolation:** Tasks that always fail (bad input, code bugs) are removed from the main queue so they do not block other work. ### Idempotency Since at-least-once delivery means a task might execute more than once (the worker crashed after completing the work but before acknowledging the message), tasks must produce the same result regardless of how many times they run. Use idempotency keys, check-before-write patterns, or database constraints to prevent duplicate side effects. **See it in practice:** The [Idempotency and Exactly-Once Semantics](/learn/system-design/design-patterns/idempotency-exactly-once) pattern covers idempotency keys and deduplication in depth. The [Payment](/learn/system-design/questions/payment) question requires strict idempotency to prevent double-charging when a payment task is retried. The [Webhook](/learn/system-design/questions/webhook) question implements retry with exponential backoff and DLQ for failed webhook deliveries. The [Pub/Sub Messaging](/learn/system-design/design-patterns/pub-sub-messaging) pattern covers queue semantics, ordering guarantees, and consumer group patterns. ## When Not to Use Async Processing Not every operation needs a queue. If a task reliably completes in under 2 seconds, return the result synchronously. The overhead of enqueuing, polling, managing job state, and handling the additional infrastructure complexity is not worth it for fast operations. Use async processing when: - The operation duration is **unpredictable** (ML inference, external API calls with variable latency) - The operation regularly **exceeds 10-30 seconds** (video encoding, large exports) - You need to **decouple producers from consumers** for scaling or reliability reasons - The work can be **deferred** without harming user experience (sending emails, generating reports) For the gray zone (2-30 seconds), consider showing a loading spinner synchronously for short operations and falling back to async with a task ID if the operation exceeds a threshold. **See it in practice:** The [Ads Click Aggregator](/learn/system-design/questions/adds-click-aggregator) question uses async processing to handle bursty click event streams that would overwhelm synchronous processing. The [Top-K](/learn/system-design/questions/topk) question batches and processes ranking updates asynchronously rather than recomputing rankings on every event. The [LeetCode](/learn/system-design/questions/leetcode) question runs code submissions as async tasks since execution time is unpredictable and untrusted code must run in sandboxed workers. The [Multi-Step Processes and Sagas](/learn/system-design/design-patterns/multi-step-processes-sagas) pattern extends async task management to coordinated workflows where multiple services must complete steps in sequence or be compensated on failure.