Time-Series Metrics Ingestion

Every system you build generates metrics. Request latency, CPU usage, error rates, business events like signups or purchases -- they all share the same shape: a value, a timestamp, and some labels describing what was measured. Monitoring, analytics, and IoT systems all revolve around ingesting these time-stamped data points at massive scale, storing them efficiently, and supporting both real-time dashboards and historical range queries.

Traditional relational databases are poorly suited for this workload. They lack the write throughput for high-frequency inserts and the query optimization for time-range scans. A PostgreSQL instance that comfortably handles your application's CRUD operations will buckle under a million metrics per second. That is why purpose-built time-series databases (TSDBs) exist -- and understanding how they work will show up in many system design interviews.

The Ingestion Pipeline#

A typical time-series metrics system has four stages, each solving a specific problem in the data flow:

1. Metrics sources emit data points. Every service, server, and device in your infrastructure produces metrics -- request latency histograms, CPU utilization gauges, business event counters.

2. Collection agents run on each host. An agent (like Prometheus Node Exporter, Telegraf, or a custom sidecar) buffers metrics locally, aggregates where possible, and forwards them in compressed batches. This is important: without a local agent, every service would need to handle network failures, retries, and batching itself. The agent centralizes all that delivery logic.

3. Kafka acts as an ingestion buffer. This is the same pattern you see in Scaling Writes -- Kafka decouples the rate at which producers emit metrics from the rate at which the TSDB can consume them. During traffic spikes, Kafka absorbs the burst. If the ingestion service falls behind, messages are retained (hours or days) until the consumer catches up. Partitioning by metric name or source ensures ordered processing within each partition.

4. The TSDB stores and indexes the data for both real-time queries and historical analysis. InfluxDB, TimescaleDB, Prometheus, and ClickHouse are all optimized for this: append-heavy writes, time-range scans, and aggressive compression. Each database makes different architectural trade-offs -- see Choosing a TSDB below for a comparison framework.

Push vs Pull: Two Collection Models#

Before data reaches your pipeline, you need to decide how agents collect it. There are two fundamentally different approaches, and many production systems use both.

Push gives you flexibility -- applications can emit metrics at any frequency, and you can collect from ephemeral containers or serverless functions that might not exist long enough to be scraped. The downside is that the collector becomes a potential bottleneck and single point of failure.

Pull gives you control -- the scraper decides the collection interval, which prevents a misbehaving service from flooding the system with metrics. It also makes service discovery straightforward: register targets, and Prometheus will find them. The downside is that pull doesn't work well for short-lived processes or when you need sub-second granularity.

Most production monitoring stacks combine both: Prometheus pull-scraping for infrastructure and service metrics, plus a push gateway or OpenTelemetry pipeline for events and traces.