Metrics Aggregator
High-throughput ingestion, Kafka buffering, and rollups
About this session
A 90-minute live deep-dive into building a metrics aggregation pipeline: agent-based collection (push vs pull), Kafka as an ingestion buffer, a purpose-built TSDB for append-heavy writes and time-range scans, and multi-resolution rollups powering both real-time dashboards and historical queries at millions of points per second. The canonical time-series interview problem.
Course Syllabus
Course Overview
- Format:
- Live Zoom, 90 minutes (75 min teaching + 15 min Q&A)
- Audience:
- Mid-level to Staff+ engineers prepping for data/infra system-design interviews
- Prerequisite:
- Comfort with message queues (Kafka), databases, and basic aggregation
What You Will Learn
- Ingest millions of time-stamped data points per second through a four-stage pipeline.
- Choose between push and pull collection models and know when to use each.
- Use Kafka as an ingestion buffer to decouple producers from the TSDB and absorb bursts.
- Pick a TSDB and explain why relational databases buckle under this workload.
- Build multi-resolution rollups that serve both real-time dashboards and historical range queries.
What This Course Is NOT
- Not an alerting/anomaly-detection ML course
- Not a vendor comparison — we use Kafka/TSDBs as representative building blocks
Pre-Class Preparation (24 hours before)
Read the problem framing only (10 min):
Design a metrics aggregation system that ingests millions of time-stamped data points per second from many sources, buffers them reliably, stores them efficiently, and serves both real-time dashboards and historical range queries via multi-resolution rollups.
Think about (don't research yet):
- Why does a relational database fall over at a million metrics per second?
- What does Kafka buy you between the agents and the database?
- How do you answer a 'last 90 days' query fast without scanning raw points?
Come to class with a rough four-stage pipeline sketched out.
Lecture: author of showoffer
Jack
ShowOffer Coach
12+ years of experience in AI/ML infrastructure and distributed aggregation systems. Tech lead for fan-out architectures serving millions of product queries.
Reviews
No reviews yet. Confirmed attendees can leave a review after the session.
Pre-session Q&A
Questions confirmed attendees asked before this session.
No questions yet.