Stream Processing & Real-Time Aggregation - System Design

Sign in to keep reading

“Stream Processing & Real-Time Aggregation” is free with an account. Your progress and bookmarks follow you across devices.

Batch vs Stream: Two Mental Models#

Before diving in, it helps to understand what stream processing is replacing and why.

Batch processing (Hadoop, Spark) works on bounded datasets. You collect a day's worth of click logs, run a MapReduce job over them, and get a report. The results are accurate because you have the full picture — but the latency is minutes to hours. If an ad is draining budget because of a click fraud spike, you will not know until the next batch run completes.

Stream processing (Flink, Kafka Streams) works on unbounded streams. Events are processed one by one (or in micro-batches) as they arrive. You get results in milliseconds to seconds. The trade-off is that you never have the full picture — new events keep arriving, and some may show up late or out of order.

Figure 1Batch processing collects hours of data then processes as a unit; stream processing handles each event as it arrives with millisecond latency

Batch vs Stream Processing
Dimension	Batch (Spark/Hadoop)	Stream (Flink/Kafka Streams)
Data model	Bounded dataset	Unbounded event stream
Latency	Minutes to hours	Milliseconds to seconds
Completeness	Full picture — accurate	Partial — events still arriving
Failure recovery	Re-run the entire job	Checkpoint and replay from offset
Use case	Daily reports, ML training	Live dashboards, fraud detection, trending

In practice, most production systems use both. Batch gives you accuracy for historical data; stream gives you speed for the present. We will see how the Lambda Architecture combines them later in this article.

See it in practice

See it in practice: The Ads Click Aggregator question is the canonical stream processing design — you build a real-time pipeline that counts ad clicks per minute as they arrive. The Job Scheduler question also contrasts batch and stream models when deciding how to process task results.

Windowing: How to Aggregate an Infinite Stream#

Since a stream never ends, you cannot just "count everything." You need a finite boundary — a window — that groups events together so you can compute an aggregation (count, sum, average) and emit a result.

There are three main window types, and choosing the right one depends on your use case.

Figure 2Overlapping intervals, sliding windows overlap, and session windows are bounded by inactivity gaps

Tumbling windows are fixed-size, non-overlapping time intervals. Every 1 minute, compute the click count and emit the result. Simple and efficient, but events near window boundaries can split related activity across two windows.

Sliding windows have a fixed size but advance continuously. A 5-minute window sliding every 1 minute produces a new result every minute, each covering the last 5 minutes of data. The overlap gives smoother results — great for moving averages and trend detection — but requires more computation since each event belongs to multiple windows.

Session windows are variable-length, bounded by periods of inactivity. A session starts with the first event and ends when no new events arrive for a configurable gap duration (say, 30 minutes of silence). This is ideal for user session analysis where activity comes in natural bursts with pauses between them.

Choosing the Right Window Type
Window Type	Size	Overlap	Best For
Tumbling	Fixed (e.g., 1 min)	None	Periodic aggregations: clicks per minute, revenue per hour
Sliding	Fixed (e.g., 5 min)	Yes — slides by a step	Smoothed metrics: moving averages, rolling top-K
Session	Variable	None

See it in practice

See it in practice: In Ads Click Aggregator, tumbling windows count clicks per minute per ad. In Top K, sliding windows maintain rolling counts of the most frequent items. In Leaderboard, tumbling windows aggregate score updates to refresh rankings at regular intervals.