How many ads were clicked in the last minute? Which topics are trending right now? Is this credit card transaction fraudulent? These questions show up constantly in system design interviews — the [Ads Click Aggregator](/learn/system-design/questions/adds-click-aggregator) question is built entirely around them, and the [Leaderboard](/learn/system-design/questions/leaderboard) and [Top K](/learn/system-design/questions/topk) questions both need real-time counting over continuous data. The problem is that traditional databases and batch jobs are not built for this. They process data that has already landed. But these questions demand answers about data that is still arriving — an unbounded, never-ending flow of events. That is the domain of **stream processing**. ## Batch vs Stream: Two Mental Models Before diving in, it helps to understand what stream processing is replacing and why. **Batch processing** (Hadoop, Spark) works on bounded datasets. You collect a day's worth of click logs, run a MapReduce job over them, and get a report. The results are accurate because you have the full picture — but the latency is minutes to hours. If an ad is draining budget because of a click fraud spike, you will not know until the next batch run completes. **Stream processing** (Flink, Kafka Streams) works on unbounded streams. Events are processed one by one (or in micro-batches) as they arrive. You get results in milliseconds to seconds. The trade-off is that you never have the full picture — new events keep arriving, and some may show up late or out of order. In practice, most production systems use both. Batch gives you accuracy for historical data; stream gives you speed for the present. We will see how the Lambda Architecture combines them later in this article. **See it in practice:** The [Ads Click Aggregator](/learn/system-design/questions/adds-click-aggregator) question is the canonical stream processing design — you build a real-time pipeline that counts ad clicks per minute as they arrive. The [Job Scheduler](/learn/system-design/questions/job-scheduler) question also contrasts batch and stream models when deciding how to process task results. ## Windowing: How to Aggregate an Infinite Stream Since a stream never ends, you cannot just "count everything." You need a finite boundary — a **window** — that groups events together so you can compute an aggregation (count, sum, average) and emit a result. There are three main window types, and choosing the right one depends on your use case. **Tumbling windows** are fixed-size, non-overlapping time intervals. Every 1 minute, compute the click count and emit the result. Simple and efficient, but events near window boundaries can split related activity across two windows. **Sliding windows** have a fixed size but advance continuously. A 5-minute window sliding every 1 minute produces a new result every minute, each covering the last 5 minutes of data. The overlap gives smoother results — great for moving averages and trend detection — but requires more computation since each event belongs to multiple windows. **Session windows** are variable-length, bounded by periods of inactivity. A session starts with the first event and ends when no new events arrive for a configurable gap duration (say, 30 minutes of silence). This is ideal for user session analysis where activity comes in natural bursts with pauses between them. **See it in practice:** In [Ads Click Aggregator](/learn/system-design/questions/adds-click-aggregator), tumbling windows count clicks per minute per ad. In [Top K](/learn/system-design/questions/topk), sliding windows maintain rolling counts of the most frequent items. In [Leaderboard](/learn/system-design/questions/leaderboard), tumbling windows aggregate score updates to refresh rankings at regular intervals. ## Event Time, Watermarks, and Late Data Events do not always arrive in the order they happened. A mobile device might buffer click events during a subway ride and send them all at once when connectivity returns. A network partition might delay events from one data center by several seconds. This is where the distinction between **event time** (when the event actually happened) and **processing time** (when the system receives it) becomes critical. If you window by processing time, those delayed subway clicks all land in the current window instead of the window they belong to. Your counts are wrong. The solution is to window by event time and use **watermarks** to track progress. A watermark of time T means "we believe all events with event time less than T have arrived." When the watermark passes the end of a window, the system considers that window complete and emits the result. But watermarks are heuristic — they can be wrong. Some events will arrive after the watermark has already passed their window. These **late-arriving events** are handled with an **allowed lateness** configuration. Late events that arrive within the allowed lateness window update the previously emitted result. Events that arrive after the allowed lateness are dropped or routed to a side output for separate handling. This is a fundamental trade-off: set the allowed lateness too short and you miss legitimate late data; set it too long and you hold window state in memory longer than necessary. **See it in practice:** In [Ads Click Aggregator](/learn/system-design/questions/adds-click-aggregator), late-arriving clicks from mobile devices must still be counted toward the correct minute-window — watermarks and allowed lateness handle this directly. In [ChatGPT](/learn/system-design/questions/chatgpt), streaming token delivery to clients faces similar ordering challenges when tokens arrive out of sequence. ## The Processing Pipeline: Kafka + Flink The most common production architecture pairs **Apache Kafka** as the event transport with **Apache Flink** as the stream processor. Here is how the pieces fit together for a click aggregation pipeline. **Kafka** acts as the durable, partitioned event log. Producers (web servers, mobile backends) write click events to a click-events topic. Kafka retains events for a configurable period (days to weeks), allowing consumers to replay from any offset. Partitioning by ad_id ensures all clicks for the same ad land on the same partition, which enables ordered processing downstream. **Flink** consumes from Kafka, keys the stream by ad_id, applies a tumbling window of 1 minute, and computes the sum. The result — "Ad #42 had 1,247 clicks between 14:00 and 14:01" — is written to a serving store (Redis, Druid, or a time-series database) where dashboards and APIs can query it. The key insight is that Kafka and Flink have complementary strengths. Kafka handles durability and decoupling (producers and consumers run independently). Flink handles stateful computation (windowing, aggregation, joins). Together, they form the backbone of most real-time data pipelines. **See it in practice:** The [Ads Click Aggregator](/learn/system-design/questions/adds-click-aggregator) question walks through this exact Kafka-to-Flink pipeline. The [Pub/Sub Messaging](/learn/system-design/design-patterns/pub-sub-messaging) pattern covers Kafka's internals — partitions, consumer groups, and offset management. The [Distributed Counters](/learn/system-design/questions/distributed-counters) question explores the downstream challenge of serving aggregated counts at scale. ## Exactly-Once Processing In a distributed stream processor, machines crash, networks partition, and containers get rescheduled. The critical question is: when a failure happens mid-computation, how do you avoid counting an event twice (or missing it entirely)? There are three **delivery semantics**: - **At-most-once**: fire and forget. Fast, but you may lose events. Rarely acceptable for aggregations. - **At-least-once**: retry on failure. No data loss, but an event might be processed more than once, inflating your counts. - **Exactly-once**: each event affects the result exactly once, even across failures. This is what you want for accurate aggregations. Flink achieves exactly-once through **distributed snapshots** based on the Chandy-Lamport algorithm. The coordinator periodically injects **barrier markers** into the input streams. When an operator receives barriers from all its input channels, it snapshots its internal state (window contents, counters, aggregation buffers) to durable storage like S3 or HDFS. On failure, the system restores from the latest snapshot and replays events from Kafka starting at the corresponding offset. Since Kafka retains events and Flink restores state to a consistent point, each event is counted exactly once in the output. The practical caveat: exactly-once applies within Flink's processing boundary. If your pipeline writes results to an external system (a database, Redis), you need the sink to be **idempotent** or support **two-phase commits** to extend the guarantee end-to-end. Flink provides exactly-once Kafka-to-Kafka guarantees natively and supports two-phase commit sinks for databases. **See it in practice:** In [Ads Click Aggregator](/learn/system-design/questions/adds-click-aggregator), exactly-once semantics prevent double-counting clicks that would inflate advertiser bills. In [Payment](/learn/system-design/questions/payment), idempotency keys solve the same "don't process twice" problem at the application layer. The [Event Sourcing and CQRS](/learn/system-design/design-patterns/event-sourcing-cqrs) pattern covers how event logs and idempotent replay provide similar guarantees in event-driven architectures. ## Lambda Architecture: Best of Both Worlds What if you need both the accuracy of batch processing and the speed of stream processing? The **Lambda Architecture** combines both into a single system. The architecture has three layers: - **Batch layer**: periodically reprocesses the complete historical dataset (every few hours) using Spark or Hadoop. Produces accurate, authoritative results — but they are always somewhat stale. - **Speed layer**: processes the real-time event stream using Flink or Storm. Produces up-to-the-second results that cover the gap since the last batch run — but they may be approximate (no late-data correction, simpler aggregation). - **Serving layer**: merges results from both layers to answer queries. Recent data comes from the speed layer; older data comes from the batch layer. When a new batch run completes, it replaces the corresponding speed-layer results. The trade-off is **operational complexity**. You maintain two separate processing pipelines (batch and stream) that must produce compatible results. The code paths are different, the failure modes are different, and keeping them in sync is an ongoing burden. The **Kappa Architecture** is a simpler alternative that uses only the stream layer. Instead of a separate batch reprocessing job, you replay the event log through the same stream processor with updated logic. This works well when your event log is durable and replayable (Kafka makes this practical) and your stream processor can handle backfill volumes. Many teams are migrating from Lambda to Kappa as stream processing engines like Flink have matured. **See it in practice:** In [Ads Click Aggregator](/learn/system-design/questions/adds-click-aggregator), the Lambda Architecture combines real-time Flink aggregations with periodic Spark batch corrections to handle late data and ensure billing accuracy. The [Time-Series and Metrics Ingestion](/learn/system-design/design-patterns/time-series-metrics-ingestion) pattern covers how the serving layer stores and queries time-bucketed aggregation results. The [Scaling Writes](/learn/system-design/design-patterns/scaling-writes) pattern explains how Kafka and LSM-tree stores handle the write throughput that both layers generate. ## When to Use Stream Processing Stream processing is not always the right tool. Use it when your application needs **continuous, low-latency results** over data that arrives as an unbounded flow. If you can wait for a nightly batch job and your data fits in a single query, keep it simple. The key questions in a system design interview are: What is your latency requirement? Can your pipeline tolerate approximate results for speed, or does it need batch-level accuracy? Do events arrive out of order? These answers determine your window type, your watermark strategy, and whether you need Lambda or Kappa.