min read

Scheduling and Async Processing

Introduction

A scheduler + async processing/worker is a common design pattern for handling complex workflows which involves a sequence of processing steps. At its core, it typically consists of three main components:

a scheduler that manages tasks;
(message) queue that holds pending tasks;
workers that execute tasks asynchronously;

Common Building Blocks

a {task | crawler | notification} scheduler
queue or message queue: SQS, Kafka
worker: AWS Lambda, EC2