Job Scheduling

Many systems need to execute tasks at specific times -- sending a reminder email 24 hours before a meeting, running a daily report at midnight, or retrying a failed webhook after 30 minutes. At small scale, a simple cron job on a single server does the trick. But as your system grows, that single-server approach becomes a liability: it creates a single point of failure, can not distribute work across multiple machines, and gives you zero visibility into which jobs succeeded, which failed, and which are stuck.

The solution is to treat job scheduling as a first-class distributed system. Let's walk through how production systems handle this.

The Two-Layer Architecture#

The key insight behind reliable job scheduling is separating two concerns: when a job should run from how it actually runs. This gives you a clean architecture with two layers that scale independently.

Layer 1 -- The Scheduler is a lightweight process that periodically polls a job store (typically a database or Redis sorted set) for jobs whose execution time has arrived. It does not execute jobs itself -- it simply moves them from "scheduled" to "ready for execution" by enqueuing them into a work queue. Because the scheduler is small and stateless, you can run two instances behind a leader election for high availability.

Layer 2 -- The Worker Pool consists of stateless workers that pull jobs from the work queue and execute them. Workers are horizontally scalable: need more throughput? Add more workers. If a worker crashes mid-job, the job reappears in the queue after a timeout and another worker picks it up.

This separation is powerful. The scheduler stays small and highly available while the execution layer scales based on workload. You can also evolve them independently -- swap out your queue technology, change your scheduling algorithm, or add priority logic without touching the other layer.

Cron-Triggered vs Event-Triggered Jobs#

Not all jobs are triggered the same way. Understanding the two triggering models helps you pick the right approach for each use case.

Time-triggered (cron) jobs run on a fixed schedule. Think daily report generation, hourly metrics aggregation, or periodic cleanup of expired sessions. The scheduler checks a cron expression and enqueues the job when it is due. This is the model you are probably familiar with from Unix cron, but in a distributed system you need to make sure only one scheduler instance enqueues each job (using leader election or distributed locking).

Event-triggered jobs run in response to something happening in your system. A user signs up and you queue a welcome email. An image is uploaded and you queue thumbnail generation. An order is placed and you queue an inventory check. These jobs are typically enqueued by publishing to a message bus (Kafka, SNS/SQS, RabbitMQ) that workers consume from.

Most production systems use both. Cron handles the periodic maintenance work, while events drive the reactive, user-facing work. The worker pool and retry infrastructure can be shared across both.