In the [Webhook](/learn/system-design/questions/webhook) question, a payment service needs to notify dozens of downstream systems when a charge succeeds. In [Slack](/learn/system-design/questions/slack), a single message post must update the channel, notify online users, increment unread counts, and index for search. If the payment service called each downstream system directly, it would need to know about all of them — and if any one is slow or down, the whole flow breaks. This is where **Pub/Sub messaging** comes in. Instead of services calling each other directly, a producer publishes an event to a **message broker**, and any number of consumers subscribe independently. The producer doesn't know or care who's listening. The broker handles delivery, buffering, and retry. Think about it: this single pattern unlocks asynchronous processing, service decoupling, and horizontal scalability — three things that show up in nearly every system design interview. ## Why Not Just Call Services Directly? Direct service-to-service communication (synchronous HTTP calls, gRPC) seems simple, but it creates tight coupling. Every new consumer means changing the producer. If the notification service is down, the payment service either blocks or drops the event. The producer becomes a coordination bottleneck. Here's the key insight: with Pub/Sub, adding a new consumer is a **zero-change operation** for the producer. The fraud detection team wants payment events? They subscribe to the topic. The analytics team wants them too? Another subscription. The payment service never changes. **See it in practice:** In [Webhook](/learn/system-design/questions/webhook), the payment service publishes events to Kafka and webhook workers consume them independently — adding a new webhook subscriber never touches the payment code. In [Slack](/learn/system-design/questions/slack), message events fan out to notification, search indexing, and unread-count services through Kafka topics. ## Kafka's Architecture: Topics, Partitions, and Offsets **Apache Kafka** is the most widely used Pub/Sub system in production. Understanding its architecture is essential for system design interviews because it explains *why* Kafka can handle millions of messages per second. A **topic** is a named channel — like payment-events or user-signups. Producers write to topics, consumers read from them. Each topic is divided into **partitions** — ordered, append-only logs. This is where the magic happens. Messages within a partition are strictly ordered and assigned a sequential **offset** (a number like 0, 1, 2, ...). Consumers track their position by offset, so they can resume exactly where they left off after a restart. **Key-based routing** determines which partition a message lands in. If you publish with key user-123, Kafka hashes that key and consistently routes all messages for user-123 to the same partition. This guarantees ordered processing per entity — critical when you need to process a user's events in sequence. **Why partitions matter for throughput:** each partition can be consumed independently. Three partitions means three consumers can work in parallel. More partitions = more parallelism = higher throughput. A single Kafka cluster can handle millions of messages per second by spreading partitions across broker nodes. **See it in practice:** In [Ad Click Aggregator](/learn/system-design/questions/adds-click-aggregator), click events are partitioned by ad ID so all clicks for the same ad are processed in order on a single consumer — enabling accurate count aggregation without coordination. In [Job Scheduler](/learn/system-design/questions/job-scheduler), job events are partitioned by job ID to ensure state transitions (PENDING to RUNNING to COMPLETED) are processed sequentially. ## Consumer Groups: Scalable Parallel Processing A **consumer group** is a set of consumers that cooperate to process a topic. Kafka assigns each partition to exactly one consumer in the group — so messages are processed once without any coordination between consumers. Here's how it scales: with 3 partitions and 3 consumers in a group, each consumer handles one partition. Add a 4th consumer? It sits idle (you can't have more consumers than partitions). Remove a consumer? Kafka **rebalances** — redistributing that consumer's partitions to the remaining members. This is why you should plan your partition count around your maximum expected parallelism. If you might need 12 consumers during peak traffic, create at least 12 partitions up front. Adding partitions later is possible but triggers a full rebalance. **See it in practice:** In [Online Chess](/learn/system-design/questions/online-chess), game-move events are consumed by a group that validates moves, updates board state, and notifies players — scaling horizontally by adding consumers. In [CICD](/learn/system-design/questions/cicd), build-trigger events are consumed by a worker group where each worker picks up one build at a time. ## At-Least-Once Delivery and Idempotency Kafka guarantees **at-least-once delivery** by default. Here's what that means in practice. When a consumer reads a message, it processes it and then **commits the offset** — telling Kafka "I'm done with this message." If the consumer crashes *after* processing but *before* committing, Kafka doesn't know the message was handled. On restart, the consumer replays from the last committed offset, processing that message again. This means your consumers **must be idempotent** — processing the same message twice should produce the same result. Common strategies: - **Deduplication key:** store a unique message ID in the database. Before processing, check if you've already seen it. Use INSERT ... ON CONFLICT DO NOTHING or equivalent. - **Idempotent writes:** design your writes so that repeating them is harmless. Setting balance = 500 is idempotent; balance += 500 is not. - **Transactional outbox:** combine the business write and offset commit in a single database transaction — if either fails, both roll back. **See it in practice:** In [Payment](/learn/system-design/questions/payment), duplicate charge prevention is critical — the consumer uses an idempotency key to ensure the same payment event is never processed twice. The [Idempotency and Exactly-Once](/learn/system-design/design-patterns/idempotency-exactly-once) pattern goes deep into deduplication strategies and the transactional outbox pattern. ## Edge Case: Replaying Messages in Kafka One of Kafka's most powerful — and sometimes dangerous — capabilities is **message replay**. Because Kafka persists messages to disk with configurable retention (days, weeks, or even forever), consumers can **reset their offset** to any point in the past and reprocess messages from scratch. Why would you want this? Imagine your analytics pipeline had a bug that miscounted payment events for the last 3 days. With a traditional queue, those messages are gone — consumed and deleted. With Kafka, you reset the consumer group's offset back 3 days and reprocess everything. The data is still there. Common replay scenarios: - **Bug fix reprocessing:** a consumer had a logic error — reset the offset and replay to recompute correct results - **New consumer backfill:** a new service (say, fraud detection) subscribes to an existing topic and needs to process all historical events, not just new ones - **Schema migration:** you changed how events are stored downstream and need to rebuild from the source of truth - **Disaster recovery:** a downstream database was corrupted — replay events to reconstruct its state Here's the catch: replay means your consumers will process messages **they've already seen**. This is where idempotency becomes critical again. If your consumer isn't idempotent, replaying can cause duplicate charges, duplicate emails, or double-counted metrics. The same strategies from at-least-once delivery apply — deduplication keys, idempotent writes, and upserts. The rule of thumb: **design every consumer as if replay could happen at any time.** Even if you don't plan to replay, a production incident might force it. Consumers that are idempotent and can handle out-of-order or duplicate messages are resilient consumers. **See it in practice:** In [Ad Click Aggregator](/learn/system-design/questions/adds-click-aggregator), if the aggregation pipeline miscounts clicks, the team can reset the consumer offset and replay click events to recompute accurate totals — but only because the aggregation uses idempotent upserts. In [Bookstore](/learn/system-design/questions/bookstore), replaying CDC events from Kafka rebuilds the Elasticsearch index from scratch after a corruption. The [Event Sourcing and CQRS](/learn/system-design/design-patterns/event-sourcing-cqrs) pattern takes this idea to the extreme — the entire system state is derived by replaying events from the log. ## Fan-Out: One Event, Many Reactions **Fan-out** is one of the most powerful Pub/Sub patterns. A single event published to a topic can be consumed by multiple independent consumer groups, each doing something completely different with it. Each consumer group maintains its own offset — so the email service can be caught up while the analytics pipeline is running 30 seconds behind, and neither affects the other. This is fundamentally different from a queue (where a message is delivered to exactly one consumer). In Kafka, every consumer group gets every message. This pattern shows up everywhere in system design interviews: - **Payment processed** → send receipt email, update analytics, check for fraud, deliver webhooks - **User signs up** → send welcome email, create default settings, notify sales team, trigger onboarding flow - **Post published** → update followers' feeds, index for search, send push notifications, log for moderation **See it in practice:** In [Fan-Out Feed Generation](/learn/system-design/design-patterns/fan-out-feed-generation), a single post-publish event fans out to update potentially millions of follower feeds. In [Yelp](/learn/system-design/questions/yelp), a new review triggers rating recalculation, search re-indexing, and notification delivery — all via separate consumer groups on the same topic. ## Change Data Capture (CDC) A common challenge is keeping multiple data stores in sync. You update a row in PostgreSQL, but you also need that change reflected in Elasticsearch (for search), Redis (for caching), and a data warehouse (for analytics). The naive approach — writing to all of them in application code (a **dual-write**) — is inherently unreliable. If the app crashes between writing to PostgreSQL and Elasticsearch, the stores diverge. **Change Data Capture** solves this by reading the database's **write-ahead log** (WAL) and publishing each change as an event to Kafka. Tools like **Debezium** connect to the database replication stream and capture every INSERT, UPDATE, and DELETE as a structured event. Since the WAL is the single source of truth, CDC is atomic with the database transaction — no changes are missed or duplicated. Downstream consumers of the CDC stream can independently maintain search indexes, populate analytics warehouses, invalidate caches, or trigger notifications — all without the original service knowing about them. **See it in practice:** In [Bookstore](/learn/system-design/questions/bookstore), CDC streams inventory changes from PostgreSQL to Elasticsearch for real-time search updates without dual-writes. The [Event Sourcing and CQRS](/learn/system-design/design-patterns/event-sourcing-cqrs) pattern extends CDC's philosophy — storing events as the primary data model and projecting read-optimized views from the event stream. ## Kafka vs Redis Pub/Sub: When to Use Which Not every messaging need requires Kafka's full machinery. **Redis Pub/Sub** is a lightweight alternative for ephemeral, real-time signaling. The rule of thumb: if you can't afford to lose the message, use Kafka. If it's a real-time notification where missing one is acceptable (like "user is typing"), Redis Pub/Sub is simpler and faster. **See it in practice:** In [Slack](/learn/system-design/questions/slack), Kafka handles durable message persistence (chat history must never be lost), while Redis Pub/Sub handles ephemeral typing indicators and presence updates. In [ChatGPT](/learn/system-design/questions/chatgpt), streaming token delivery uses lightweight Pub/Sub for real-time pushing while completed responses are durably stored. The [Pushing Realtime Updates](/learn/system-design/design-patterns/pushing-realtime-updates) pattern covers how to combine WebSockets with Pub/Sub for live delivery.