Pub/Sub Messaging

In the Webhook question, a payment service needs to notify dozens of downstream systems when a charge succeeds. In Slack, a single message post must update the channel, notify online users, increment unread counts, and index for search. If the payment service called each downstream system directly, it would need to know about all of them — and if any one is slow or down, the whole flow breaks.

This is where Pub/Sub messaging comes in. Instead of services calling each other directly, a producer publishes an event to a message broker, and any number of consumers subscribe independently. The producer doesn't know or care who's listening. The broker handles delivery, buffering, and retry.

Think about it: this single pattern unlocks asynchronous processing, service decoupling, and horizontal scalability — three things that show up in nearly every system design interview.

Why Not Just Call Services Directly?#

Direct service-to-service communication (synchronous HTTP calls, gRPC) seems simple, but it creates tight coupling. Every new consumer means changing the producer. If the notification service is down, the payment service either blocks or drops the event. The producer becomes a coordination bottleneck.

Here's the key insight: with Pub/Sub, adding a new consumer is a zero-change operation for the producer. The fraud detection team wants payment events? They subscribe to the topic. The analytics team wants them too? Another subscription. The payment service never changes.

Kafka's Architecture: Topics, Partitions, and Offsets#

Apache Kafka is the most widely used Pub/Sub system in production. Understanding its architecture is essential for system design interviews because it explains why Kafka can handle millions of messages per second.

A topic is a named channel — like payment-events or user-signups. Producers write to topics, consumers read from them.

Each topic is divided into partitions — ordered, append-only logs. This is where the magic happens. Messages within a partition are strictly ordered and assigned a sequential offset (a number like 0, 1, 2, ...). Consumers track their position by offset, so they can resume exactly where they left off after a restart.

Key-based routing determines which partition a message lands in. If you publish with key user-123, Kafka hashes that key and consistently routes all messages for user-123 to the same partition. This guarantees ordered processing per entity — critical when you need to process a user's events in sequence.

Why partitions matter for throughput: each partition can be consumed independently. Three partitions means three consumers can work in parallel. More partitions = more parallelism = higher throughput. A single Kafka cluster can handle millions of messages per second by spreading partitions across broker nodes.