Every system you build generates metrics. Request latency, CPU usage, error rates, business events like signups or purchases -- they all share the same shape: a value, a timestamp, and some labels describing what was measured. Monitoring, analytics, and IoT systems all revolve around ingesting these time-stamped data points at massive scale, storing them efficiently, and supporting both real-time dashboards and historical range queries. Traditional relational databases are poorly suited for this workload. They lack the write throughput for high-frequency inserts and the query optimization for time-range scans. A PostgreSQL instance that comfortably handles your application's CRUD operations will buckle under a million metrics per second. That is why purpose-built time-series databases (TSDBs) exist -- and understanding how they work will show up in many system design interviews. ## The Ingestion Pipeline A typical time-series metrics system has four stages, each solving a specific problem in the data flow: 1. **Metrics sources** emit data points. Every service, server, and device in your infrastructure produces metrics -- request latency histograms, CPU utilization gauges, business event counters. 2. **Collection agents** run on each host. An agent (like Prometheus Node Exporter, Telegraf, or a custom sidecar) buffers metrics locally, aggregates where possible, and forwards them in compressed batches. This is important: without a local agent, every service would need to handle network failures, retries, and batching itself. The agent centralizes all that delivery logic. 3. **Kafka acts as an ingestion buffer.** This is the same pattern you see in [Scaling Writes](/learn/system-design/design-patterns/scaling-writes) -- Kafka decouples the rate at which producers emit metrics from the rate at which the TSDB can consume them. During traffic spikes, Kafka absorbs the burst. If the ingestion service falls behind, messages are retained (hours or days) until the consumer catches up. Partitioning by metric name or source ensures ordered processing within each partition. 4. **The TSDB stores and indexes the data** for both real-time queries and historical analysis. InfluxDB, TimescaleDB, Prometheus, and ClickHouse are all optimized for this: append-heavy writes, time-range scans, and aggressive compression. Each database makes different architectural trade-offs -- see [Choosing a TSDB](#choosing-a-tsdb-for-your-pipeline) below for a comparison framework. **See it in practice:** The [Ad Click Aggregator](/learn/system-design/questions/adds-click-aggregator) question is the canonical time-series ingestion problem -- you design a pipeline that ingests millions of click events per second, stores them in time-series format, and produces real-time and historical aggregations. The [Webhook](/learn/system-design/questions/webhook) question also uses Kafka buffering to absorb delivery-result writes. The [Pub/Sub Messaging](/learn/system-design/design-patterns/pub-sub-messaging) pattern covers Kafka internals (topics, partitions, consumer groups) in depth. ## Push vs Pull: Two Collection Models Before data reaches your pipeline, you need to decide *how* agents collect it. There are two fundamentally different approaches, and many production systems use both. **Push** gives you flexibility -- applications can emit metrics at any frequency, and you can collect from ephemeral containers or serverless functions that might not exist long enough to be scraped. The downside is that the collector becomes a potential bottleneck and single point of failure. **Pull** gives you control -- the scraper decides the collection interval, which prevents a misbehaving service from flooding the system with metrics. It also makes service discovery straightforward: register targets, and Prometheus will find them. The downside is that pull doesn't work well for short-lived processes or when you need sub-second granularity. Most production monitoring stacks combine both: Prometheus pull-scraping for infrastructure and service metrics, plus a push gateway or OpenTelemetry pipeline for events and traces. **See it in practice:** In [Uber](/learn/system-design/questions/uber), driver location updates are pushed at high frequency since vehicles are constantly moving and you cannot wait for a scraper. In the [CI/CD Pipeline](/learn/system-design/questions/cicd) question, build metrics from short-lived CI runners must be pushed because the runners terminate after the job completes. The [Stream Processing and Real-Time Aggregation](/learn/system-design/design-patterns/stream-processing-realtime-aggregation) pattern covers how push-based event streams feed real-time computation. ## Why TSDBs Are Fast: The Write-Optimized Path The secret to a TSDB's write performance is that it never does random I/O. Every write follows an append-only path optimized for sequential disk access. Here is how a typical TSDB write works: 1. The metric arrives and is appended to a **write-ahead log** (WAL) for durability -- this is pure sequential I/O, extremely fast. 2. Simultaneously, it is written to an **in-memory memtable** -- essentially instant. 3. When the memtable fills up, it is flushed to disk as an **immutable block**, sorted by time and compressed. 4. Background **compaction** merges small blocks into larger ones, organized by time partitions (e.g., 2-hour blocks). This is the same LSM-tree approach used by Cassandra and RocksDB, which you can read about in [Scaling Writes](/learn/system-design/design-patterns/scaling-writes). The key difference is that TSDBs add **time-based partitioning** on top: data is organized into blocks by time window, so dropping old data is as simple as deleting a file -- no expensive row-by-row deletion. ### Columnar Storage: Why Compression Matters TSDBs also store data in a **column-oriented** layout rather than the row-oriented format of traditional databases. This makes a dramatic difference for both compression and query performance. In a row-oriented database, a time-range query like "give me CPU usage for the last hour" must read entire rows -- including columns you do not need (host, region, other metric fields). In a columnar layout, the query reads only the timestamp and value columns, skipping everything else. The compression benefits are even more significant. When timestamps are stored together, they form a nearly monotonic sequence that compresses extremely well with delta encoding. When float values from the same metric are stored together, they tend to be similar in magnitude, enabling techniques like XOR compression (used by Gorilla/Facebook's TSDB) that achieve 10-100x compression ratios compared to general-purpose databases. **See it in practice:** In [Top K](/learn/system-design/questions/topk), efficient storage of streaming frequency counts over time windows relies on the same columnar compression principles. In the [Leaderboard](/learn/system-design/questions/leaderboard) question, historical score data benefits from time-partitioned storage for trend analysis. The [Event Sourcing and CQRS](/learn/system-design/design-patterns/event-sourcing-cqrs) pattern uses a similar append-only log structure for event storage. ## Choosing a TSDB for Your Pipeline Understanding *how* TSDBs work is only half the battle -- you also need to articulate *which* one fits a given scenario and why. Interviewers use this as a staff-eng signal: can you reason about trade-offs beyond "just use InfluxDB"? ### Decision Framework **Query pattern** is the first axis. If the primary consumer is Grafana dashboards with PromQL, Prometheus is the natural fit. If stakeholders need ad-hoc SQL -- slicing metrics by arbitrary dimensions (see [Query Strategies](#query-strategies-raw-scans-pre-aggregation-and-olap) for a deeper treatment), running JOINs with business data -- ClickHouse or TimescaleDB are better choices. TimescaleDB wins specifically when you need to JOIN time-series data with relational tables already living in PostgreSQL. **Scale and cardinality** is the second axis. At moderate scale (fewer than 1 million active series), any of these databases will work. Once you push into high-cardinality territory -- per-user metrics, per-device IoT telemetry, per-ad analytics -- ClickHouse's columnar MergeTree engine handles it gracefully while Prometheus actively discourages it. This is why systems like the [Ad Click Aggregator](/learn/system-design/questions/adds-click-aggregator) gravitate toward ClickHouse. **Operational model** is the third axis. If you are already running Kubernetes with Prometheus for infrastructure monitoring, extending it with Thanos for long-term storage is the path of least resistance. If your team has deep PostgreSQL expertise, TimescaleDB avoids introducing a new operational surface. If you need maximum flexibility and can invest in ops, ClickHouse offers the most powerful engine. And if you just need something running in an afternoon, InfluxDB is the fastest to stand up. **See it in practice:** The [Ad Click Aggregator](/learn/system-design/questions/adds-click-aggregator) is a strong ClickHouse fit -- high-cardinality per-ad analytics with SQL aggregation queries. In [Uber](/learn/system-design/questions/uber), the push-based pipeline for driver location updates illustrates why pull-based Prometheus would be a poor fit for high-frequency ephemeral data. The [Scaling Writes](/learn/system-design/design-patterns/scaling-writes) pattern covers the LSM-tree and MergeTree foundations these engines build on. The [Event Sourcing and CQRS](/learn/system-design/design-patterns/event-sourcing-cqrs) pattern shows how append-only logs enable schema-change replay -- relevant when migrating between TSDBs. ## Cardinality: The Silent Performance Killer Every unique combination of metric name and label values creates a distinct **time series**. This sounds harmless until you do the math. Consider a metric like http_requests with labels for method, endpoint, status code, region, and instance: 5 methods x 1,000 endpoints x 50 status codes x 10 regions x 500 instances = **1.25 billion distinct series** This is a **cardinality explosion**, and it is the most common reason TSDB deployments fail. Each active series consumes memory for its index entry, and high cardinality degrades both write throughput (more index lookups) and query performance (more series to scan). The rule of thumb: if a label value can take millions of distinct values, it should not be a metric label. Use logs or traces for high-cardinality dimensions instead. **See it in practice:** In the [Ad Click Aggregator](/learn/system-design/questions/adds-click-aggregator), naive per-ad-per-user metrics would create billions of series -- aggregating by ad campaign and region keeps cardinality manageable. In [Distributed Counters](/learn/system-design/questions/distributed-counters), the challenge of tracking counts per entity directly mirrors cardinality management. In [Google Maps](/learn/system-design/questions/google-map), tile-serving metrics must avoid per-tile-coordinate labels to prevent cardinality explosion. ## Multi-Resolution Rollups Storing every raw data point forever is prohibitively expensive. A single metric emitting once per second generates 86,400 points per day, 31.5 million per year. Multiply that by thousands of metrics and hundreds of hosts, and storage costs become astronomical. The solution is **multi-resolution rollups**: progressively downsample data as it ages, keeping high-resolution data only for recent time windows. Each rollup computes **min, max, avg, sum, and count** for the window. Keeping all five aggregates (not just the average) is important -- it lets you reconstruct percentile-like information from downsampled data. A background compaction job runs periodically, reads raw data, writes aggregated data to the appropriate resolution tier, and deletes the raw data after its retention window expires. When a user queries a dashboard, the system automatically selects the appropriate resolution. Zoomed into the last hour? You get raw data. Looking at the last month? You get 5-minute aggregates. The user sees a smooth graph either way, but the query scans orders of magnitude fewer data points for longer time ranges. Rollups address the time dimension, but real queries also slice by arbitrary labels -- region, service, endpoint. See [Query Strategies](#query-strategies-raw-scans-pre-aggregation-and-olap) below for how to support multi-dimensional access patterns alongside time-based rollups. **See it in practice:** In the [Ad Click Aggregator](/learn/system-design/questions/adds-click-aggregator), multi-resolution rollups are essential -- advertisers need second-level granularity for live campaigns but only daily totals for last year's performance. In [Top K](/learn/system-design/questions/topk), streaming aggregation over different time windows (last minute, last hour, all time) follows the same multi-resolution principle. The [Distributed Caching](/learn/system-design/design-patterns/distributed-caching) pattern covers TTL-based eviction strategies that parallel retention policies. ## Query Strategies: Raw Scans, Pre-Aggregation, and OLAP Multi-resolution rollups solve the time axis -- they compress historical data so that time-range queries remain fast regardless of how far back you look. But real production queries rarely filter on time alone. A dashboard showing p99 latency might need to break down by region, service, and endpoint simultaneously. An SLA report might aggregate by customer tier and datacenter. An on-call engineer investigating a spike might slice by any combination of labels they can think of. These multi-dimensional access patterns require a deliberate query strategy. There are three fundamentally different approaches, each trading off latency, flexibility, and operational complexity. Most production systems end up using more than one. Most production systems combine all three strategies. Consider the [Ad Click Aggregator](/learn/system-design/questions/adds-click-aggregator): pre-aggregate `(ad_id, region, minute)` tuples for the real-time advertiser dashboard where latency must be sub-100ms, store raw click events in S3 for audit trails and compliance, and route ad-hoc investigative queries ("which creatives drove the most clicks in Brazil last Tuesday?") through ClickHouse. This hybrid approach is a strong interview signal -- proposing it demonstrates that you understand no single strategy covers all access patterns, and that the right answer is matching each query class to the engine best suited for it. ### Supporting Multi-Dimensional Queries Time rollups compress one axis, but real queries involve four or more dimensions simultaneously. "Show me p99 latency for the payments service in us-east-1, broken down by endpoint, for the last 6 hours" touches time, service, region, and endpoint all at once. There are two primary approaches: materialized views for known dimension combinations, and OLAP engines for flexible slicing. Each additional dimension multiplies the series space. A combination of `(service x region x endpoint x minute)` with 10 services, 5 regions, 100 endpoints, and 1440 minutes produces roughly 7.2 million buckets per day -- feasible for pre-aggregation. But adding `user_id` with 100K users explodes this to 720 billion buckets, which is clearly impractical. High-cardinality dimensions like user IDs, request IDs, or IP addresses belong in OLAP engines where columnar compression handles them efficiently, not in pre-aggregated cubes where they cause combinatorial explosion. **See it in practice:** The [Ad Click Aggregator](/learn/system-design/questions/adds-click-aggregator) combines all three query strategies -- pre-aggregation for the real-time dashboard, raw storage for audit, and OLAP for ad-hoc analysis. [Distributed Counters](/learn/system-design/questions/distributed-counters) uses a similar three-path architecture where different read patterns get different serving paths. The [Event Sourcing and CQRS](/learn/system-design/design-patterns/event-sourcing-cqrs) pattern covers materialized views and read-side projections -- the same concept as pre-aggregated cubes, applied to event-driven systems. ## Alerting: Detecting Problems in Real Time Collecting metrics is only useful if you can react to them. There are two approaches for evaluating alert rules against time-series data: **Scheduled polling** is the simpler approach. A cron-like evaluator periodically queries the TSDB (e.g., every 15 seconds) and checks conditions like "avg CPU greater than 90% for 5 minutes." This is easy to implement and reason about, but it introduces evaluation lag equal to the polling interval. Prometheus Alertmanager works this way. **Stream processing** evaluates alert rules in real-time on the Kafka stream *before* data reaches the TSDB. Tools like Flink or Kafka Streams compute sliding window aggregations and fire alerts with sub-second latency. This is more complex to operate but essential for latency-sensitive alerts like SLA breaches or error rate spikes. Most production systems combine both: stream processing for critical alerts (payment failures, error rate spikes) and scheduled polling for less urgent conditions (disk usage trending toward full, gradual latency increases). **See it in practice:** In the [Job Scheduler](/learn/system-design/questions/job-scheduler) question, alerting on stuck or failed jobs requires real-time evaluation of job-state metrics. In [Online Chess](/learn/system-design/questions/online-chess), detecting server-side latency anomalies during live games demands sub-second alert evaluation. The [Stream Processing and Real-Time Aggregation](/learn/system-design/design-patterns/stream-processing-realtime-aggregation) pattern covers the Flink/Kafka Streams architecture for real-time computation in depth. The [Job Scheduling](/learn/system-design/design-patterns/job-scheduling) pattern covers the scheduled polling approach. ## Putting It All Together When you encounter a time-series ingestion problem in an interview, the architecture follows a predictable shape: The pattern connects directly to several other design patterns. [Scaling Writes](/learn/system-design/design-patterns/scaling-writes) covers the sharding and LSM-tree foundations that TSDBs build on. [Pub/Sub Messaging](/learn/system-design/design-patterns/pub-sub-messaging) explains the Kafka layer in detail. [Stream Processing](/learn/system-design/design-patterns/stream-processing-realtime-aggregation) covers real-time aggregation for alerting. And [Distributed Caching](/learn/system-design/design-patterns/distributed-caching) is relevant when you cache frequently-queried dashboard data to avoid repeated TSDB scans.