Understanding message queues, when to use them, and choosing between Kafka, RabbitMQ, and SQS.
Finished this lesson?
Sign in to save your progress across devices.
Tip: use ← and → to move between lessons.
Was this lesson helpful?
Rehearse this with a Staff+ engineer
45-minute mock with a Staff+ engineer who interviews at your target company. Premium members save $20.
Message queues enable asynchronous communication between services. They decouple producers from consumers, improve reliability, and help handle traffic spikes.
Without message queues:
With message queues:
If a user action doesn't need an immediate response, it's a candidate for async processing via a message queue.
Producer → Message Queue → Consumer
Example:
Order Service → Queue → Email Service
→ Inventory Service
→ Analytics Service
The unit of data being transmitted. Contains:
| Queue vs Topic | |
|---|---|
| Name | Description |
| Queue (Point-to-Point) | Each message consumed by exactly one consumer. Good for task distribution. |
| Topic (Pub/Sub) | Each message can be consumed by multiple subscribers. Good for event broadcasting. |
| Popular Message Queue Solutions | |
|---|---|
| Name | Description |
| Apache Kafka | Distributed streaming platform. High throughput, durable, replayable. Best for: event streaming, logs, high-volume data pipelines. |
| RabbitMQ | Traditional message broker. Rich routing, multiple protocols. Best for: complex routing, RPC patterns, lower volume. |
| Amazon SQS | Fully managed queue service. Simple, scalable, no ops. Best for: AWS-native apps, simple queuing needs. |
| Amazon SNS | Pub/sub notification service. Pairs with SQS. Best for: fan-out patterns, push notifications. |
| Redis Streams | Redis-based streaming. Good for existing Redis users. Best for: simpler streaming needs, lower latency. |
| Message Queue Comparison | |
|---|---|
| Name | Description |
| Throughput | Kafka: Millions/sec. RabbitMQ: Tens of thousands/sec. SQS: Unlimited (managed). |
| Message retention | Kafka: Configurable (days/weeks). RabbitMQ: Until consumed. SQS: Up to 14 days. |
| Replay capability | Kafka: Yes (core feature). RabbitMQ: No. SQS: No. |
| Ordering | Kafka: Per partition. RabbitMQ: Per queue. SQS: FIFO queues only. |
| Operations | Kafka: Complex. RabbitMQ: Moderate. SQS: None (managed). |
Distribute tasks across multiple workers.
┌→ Worker 1
Producer → Queue → Worker 2
└→ Worker 3
Use case: Image processing, email sending, report generation
Key considerations:
Broadcast events to multiple interested services.
┌→ Email Service
Order Service → Topic → Inventory Service
└→ Analytics Service
Use case: Event-driven architectures, notifications
Asynchronous request-response pattern.
Client → Request Queue → Server
← Reply Queue ←
Use case: Long-running operations where client needs the result
Coordinate transactions across multiple services.
Order Created → Payment Service (charge)
→ Inventory Service (reserve)
→ Shipping Service (schedule)
If any fails → Compensating transactions (refund, unreserve, cancel)
Choose the Right Pattern
You're building an e-commerce system. When a user places an order, you need to: charge their card, update inventory, send confirmation email, and update analytics. How would you design this?
Approach: Event-driven with Saga pattern
Why this works:
Saga coordination:
Message may be lost, never duplicated.
Producer sends → Network fails → Message lost
Use when: Loss is acceptable (metrics, logs that can be approximated)
Message delivered at least once, may be duplicated.
Producer sends → Consumer processes → Ack lost → Consumer reprocesses
Use when: Loss is unacceptable, consumers are idempotent
Message delivered exactly once. Hard to achieve.
Requires: Idempotent consumers OR transactional guarantees
Use when: Financial transactions, inventory updates
True exactly-once is complex and expensive. Most systems achieve "effectively exactly-once" through idempotent consumers with at-least-once delivery.
Messages that fail processing go to a separate queue for investigation.
Main Queue → Consumer → Success
↘ Failure (after N retries) → Dead Letter Queue → Manual review
# Exponential backoff with jitter
def get_retry_delay(attempt):
base_delay = 1 # seconds
max_delay = 300 # 5 minutes
jitter = random.uniform(0, 1)
delay = min(base_delay * (2 ** attempt) + jitter, max_delay)
return delay
Make consumers safe to run multiple times.
def process_order(order_id):
# Check if already processed
if db.exists(f"processed:{order_id}"):
return # Already done, skip
# Process the order
do_work(order_id)
# Mark as processed
db.set(f"processed:{order_id}", True)
| Message Queue Knowledge by Level | |
|---|---|
| Name | Description |
| Mid-Level (L4) | Know when to use async processing. Understand basic queue concepts. Can explain producer/consumer pattern. |
| Senior (L5) | Choose appropriate queue technology with reasoning. Discuss delivery guarantees. Design for failure (DLQ, retries, idempotency). |
| Staff+ (L6+) | Design event-driven architectures. Handle exactly-once semantics. Discuss operational concerns (monitoring, backpressure, scaling consumers). |
| Engineering Manager | Evaluate build vs buy for messaging infrastructure. Plan team ownership of event contracts. Understand debugging complexity in async systems. |
"I'd use a message queue when the caller doesn't need an immediate response. For example: sending emails after signup, processing uploaded images, or updating analytics. This decouples services, handles failures gracefully, and smooths out traffic spikes."
"Multiple layers: persistent storage in the queue (Kafka writes to disk), acknowledgments from consumers (only ack after successful processing), and dead letter queues for messages that fail repeatedly. Producers should retry on failure, and consumers should be idempotent."
"It depends on the technology. In Kafka, ordering is guaranteed within a partition—so I'd partition by the entity that needs ordering (e.g., user_id for user events). In SQS, I'd use FIFO queues with message group IDs. For RabbitMQ, single-consumer queues maintain order."
Message queues handle async communication. Next, we'll look at blob storage for handling large files and media content.
Technology choice: Kafka for durability and replay capability, or SNS+SQS if on AWS and prefer managed services.