Many systems need to execute tasks at specific times -- sending a reminder email 24 hours before a meeting, running a daily report at midnight, or retrying a failed webhook after 30 minutes. At small scale, a simple cron job on a single server does the trick. But as your system grows, that single-server approach becomes a liability: it creates a single point of failure, can not distribute work across multiple machines, and gives you zero visibility into which jobs succeeded, which failed, and which are stuck. The solution is to treat job scheduling as a first-class distributed system. Let's walk through how production systems handle this. ## The Two-Layer Architecture The key insight behind reliable job scheduling is separating two concerns: **when** a job should run from **how** it actually runs. This gives you a clean architecture with two layers that scale independently. **Layer 1 -- The Scheduler** is a lightweight process that periodically polls a job store (typically a database or Redis sorted set) for jobs whose execution time has arrived. It does not execute jobs itself -- it simply moves them from "scheduled" to "ready for execution" by enqueuing them into a work queue. Because the scheduler is small and stateless, you can run two instances behind a leader election for high availability. **Layer 2 -- The Worker Pool** consists of stateless workers that pull jobs from the work queue and execute them. Workers are horizontally scalable: need more throughput? Add more workers. If a worker crashes mid-job, the job reappears in the queue after a timeout and another worker picks it up. This separation is powerful. The scheduler stays small and highly available while the execution layer scales based on workload. You can also evolve them independently -- swap out your queue technology, change your scheduling algorithm, or add priority logic without touching the other layer. **See it in practice:** The [Job Scheduler](/learn/system-design/questions/job-scheduler) question is the direct application of this pattern -- you design the two-layer architecture with a DB-backed scheduler, delay queues, and worker pools. The [CI/CD](/learn/system-design/questions/cicd) question also requires job scheduling for build pipelines, where dependent stages must execute in order with retry logic. The [ChatGPT](/learn/system-design/questions/chatgpt) question uses a similar pattern for queuing inference requests to GPU workers. ## Cron-Triggered vs Event-Triggered Jobs Not all jobs are triggered the same way. Understanding the two triggering models helps you pick the right approach for each use case. **Time-triggered (cron) jobs** run on a fixed schedule. Think daily report generation, hourly metrics aggregation, or periodic cleanup of expired sessions. The scheduler checks a cron expression and enqueues the job when it is due. This is the model you are probably familiar with from Unix cron, but in a distributed system you need to make sure only one scheduler instance enqueues each job (using leader election or distributed locking). **Event-triggered jobs** run in response to something happening in your system. A user signs up and you queue a welcome email. An image is uploaded and you queue thumbnail generation. An order is placed and you queue an inventory check. These jobs are typically enqueued by publishing to a message bus (Kafka, SNS/SQS, RabbitMQ) that workers consume from. Most production systems use both. Cron handles the periodic maintenance work, while events drive the reactive, user-facing work. The worker pool and retry infrastructure can be shared across both. **See it in practice:** In [Google Calendar](/learn/system-design/questions/google-calendar), event reminders are scheduled as time-triggered jobs (notify 15 minutes before the meeting). In [Webhook](/learn/system-design/questions/webhook), delivery attempts are event-triggered -- each webhook event enqueues a delivery job. The [Pub/Sub Messaging](/learn/system-design/design-patterns/pub-sub-messaging) pattern covers the event bus architecture in detail. ## Priority-Based Scheduling Not all jobs are equally urgent. A payment retry is more important than a log cleanup task. Priority queues let you express this. The simplest approach is to maintain separate queues for each priority level. Workers always check the high-priority queue first. If it is empty, they check medium, then low. This ensures critical jobs (payment retries, real-time alerts) execute before background work (analytics aggregation, data cleanup). The risk with strict priority ordering is **starvation** -- if high-priority jobs keep arriving, low-priority jobs may never run. The standard mitigation is an aging mechanism: if a low-priority job has been waiting longer than a threshold, promote it to a higher queue. Alternatively, you can reserve a percentage of worker capacity for each priority level. **See it in practice:** In [Uber](/learn/system-design/questions/uber), ride matching jobs get higher priority than ETA recalculation. In [Ads Click Aggregator](/learn/system-design/questions/adds-click-aggregator), real-time click counting must happen before daily reporting aggregation. The [Stream Processing](/learn/system-design/design-patterns/stream-processing-realtime-aggregation) pattern explores how to handle priority in streaming pipelines. ## Execution Guarantees and Fault Tolerance In a distributed system, things fail. Workers crash, networks partition, and processes hang. The job scheduling system needs to handle all of these gracefully. ### At-Least-Once Delivery The standard guarantee for job systems is **at-least-once execution**. Here is how it works: 1. **Dequeue with visibility timeout.** When a worker pulls a job from the queue, the job becomes invisible to other workers for a configurable period (the "visibility timeout" in SQS, or a "lease" in custom implementations). 2. **Heartbeat for long jobs.** If the job takes longer than the visibility timeout, the worker sends periodic heartbeats to extend its lease. This prevents the job from being re-assigned while it is still running. 3. **ACK on completion.** When the job finishes successfully, the worker acknowledges it (deletes it from the queue). If the worker crashes before acknowledging, the visibility timeout expires and the job reappears for another worker. 4. **Idempotent design.** Since a job might execute more than once (the worker crashes after completing the work but before sending the ACK), jobs must be **idempotent** -- running them twice should produce the same result as running them once. For example, "set user status to active" is idempotent; "increment counter by 1" is not (use "set counter to X" instead). **See it in practice:** The [Idempotency and Exactly-Once](/learn/system-design/design-patterns/idempotency-exactly-once) pattern goes deep into idempotency keys and deduplication. In [Payment](/learn/system-design/questions/payment), payment processing jobs use idempotency keys to prevent double-charging. In [Online Chess](/learn/system-design/questions/online-chess), game state updates must be idempotent since move-processing jobs may retry. ## Data Model: Separating Definitions from Executions A clean data model makes everything else easier. The key principle is separating **job definitions** (the template -- what to do, when, and how often) from **job executions** (a specific run -- status, start time, result, attempt count). This separation gives you several advantages. You can track the history of every execution, retry individual runs without modifying the definition, and support both one-off and recurring jobs with the same infrastructure. The JobExecution table also gives you observability -- you can query for stuck jobs (started but never completed), calculate success rates, and identify jobs that consistently fail. For recurring jobs, the scheduler creates a new JobExecution row each time the cron expression fires. The JobDefinition stays unchanged. ## Handling Failures: Retry Queues and Dead Letter Queues Jobs fail. The network was unreachable, a downstream service returned an error, or the input data was malformed. A robust system distinguishes between **transient** failures (worth retrying) and **permanent** failures (retrying won't help). **Retry with exponential backoff.** When a job fails, re-enqueue it with an increasing delay: 1 second, then 2, then 4, then 8, and so on. This gives transient issues time to resolve without hammering the failing service. Cap the backoff at a reasonable maximum (e.g., 5 minutes) and add jitter (small random delay) to prevent retry storms. **Dead letter queue (DLQ).** After a job has been retried the maximum number of times (say, 5 attempts), move it to a dead letter queue instead of retrying forever. The DLQ serves as a holding pen for jobs that need human investigation. Set up alerting on the DLQ so your on-call engineer knows when jobs are failing persistently. **What to do with DLQ jobs.** Once you fix the underlying issue (deploy a bug fix, restore a downstream service), you can replay the jobs from the DLQ back into the main queue. This is why storing the original payload in the job execution record matters -- you need all the context to replay. **See it in practice:** In [Webhook](/learn/system-design/questions/webhook), failed deliveries are retried with exponential backoff up to a maximum, then moved to a DLQ for manual replay. In [Slack](/learn/system-design/questions/slack), message delivery failures follow the same pattern. The [Managing Long-Running Tasks](/learn/system-design/design-patterns/managing-long-running-tasks) pattern covers what happens when jobs take minutes or hours instead of seconds. The [Distributed Caching](/learn/system-design/design-patterns/distributed-caching) pattern is relevant when your workers need fast access to shared state. ## Delay Queues: Scheduling for the Future Sometimes you need a job to run not now, but at a specific future time: "send this reminder in 24 hours" or "retry this webhook in 30 minutes." Delay queues solve this. **Redis Sorted Sets** are the most popular choice for custom implementations. You add the job ID as a member with the scheduled execution timestamp as the score. The scheduler periodically runs ZRANGEBYSCORE to find all jobs with scores less than or equal to the current time, pops them atomically with ZPOPMIN, and pushes them to the worker queue. This gives you millisecond precision and O(log n) insert/query performance. **SQS Delay Messages** provide a managed alternative -- messages can be sent with a delay of up to 15 minutes. For longer delays, store the job in a database with a scheduledAt timestamp and use a short-interval poller (every 10-30 seconds) to check for due jobs and enqueue them. **Database polling** is the simplest approach and works well at moderate scale. Store jobs in a table with a scheduledAt column, and have the scheduler query for rows where scheduledAt <= now() AND status = 'pending'. Add an index on (status, scheduledAt) for efficiency. **See it in practice:** In [Job Scheduler](/learn/system-design/questions/job-scheduler), Redis sorted sets power the delay queue for scheduling jobs at precise future times. In [Google Calendar](/learn/system-design/questions/google-calendar), event reminders use delay queues to fire notifications at the right moment. The [Scaling Writes](/learn/system-design/design-patterns/scaling-writes) pattern covers how to handle high write throughput into your job queue, and the [Rate Limiting](/learn/system-design/design-patterns/rate-limiting) pattern is useful when your workers call external APIs with rate limits. ## Putting It Together: Design Checklist When designing a job scheduling system (or encountering one in an interview), walk through these considerations: The beauty of the two-layer architecture is that it gives you a framework to answer all of these questions. The scheduler layer handles triggering and delay. The worker layer handles execution, retry, and scaling. And the data model (definitions plus executions) gives you the observability you need to keep the system healthy.