Sign in to keep reading
“Top-K Videos” is free with an account. Your progress and bookmarks follow you across devices.
Sign in to continueDesign Top-K Videos - This system design covers approximate counting with Count-Min Sketch, Heavy Hitters tracking, event-time bucketing, and scalable trend computation at scale.
“Top-K Videos” is free with an account. Your progress and bookmarks follow you across devices.
Sign in to continueThink Beyond the Happy Path
Could you answer these today? Tick what you already know.
Top-K and trending topic systems go beyond basic aggregations like ad click counts — they require ranking and returning only the top results. This adds complexity around ordering, windowing, and filtering by dimensions like time or location. It's a common pattern in systems like YouTube Trending, Twitter hashtags, or Spotify top charts.
FR1 – Retrieve Top-K Videos
Users should be able to retrieve the Top-K most popular videos based on predefined ranking criteria.
This is the core capability of the system — allowing end users, internal services, or recommendation engines to request and receive a ranked list of trending videos. The system should return results ordered by a consistent popularity metric (e.g., view count, engagement score, or watch time). This functionality enables features like the "Trending" tab on YouTube, homepage carousels, or content recommendation surfaces.
FR2 – Filter by Geographic Location
Users should be able to filter Top-K video queries by geographic location.
Filter conditions can be beyond location, but to keep it simple and representative, we use location filter only. This allows the system to surface region-specific trends — for example, the top 10 videos in the United States, India, or Japan. Focusing on location keeps the scope manageable while still enabling meaningful personalization and relevance. Location-based filtering is essential for global platforms where content popularity varies significantly across regions, and supports both user-facing experiences and country-specific analytics.
FR3 – Arbitrary Time Window
Users should be able to define an arbitrary time window for which the Top-K videos are computed.
This means the system must support flexible input of start and end timestamps, such as "from June 1 to June 7" or "from 2025-06-20 10:00 to 2025-06-21 15:00." Arbitrary time windowing enables a wide range of use cases — from generating daily or hourly trend reports, to supporting experiments, historical analysis, or detecting flash trends during specific events. This flexibility gives users precise control over how video popularity is evaluated.
Key Design Trade-off: Exact vs. Approximate Accuracy
A critical design decision for Top-K systems is whether to guarantee exact or approximate ranking accuracy.
For use cases like YouTube Trending Top K, where the goal is to surface broadly representative popular content at scale, approximate methods (e.g., Count-Min Sketch, Heavy Hitters, we will discuss later) offer excellent performance, cost efficiency, and near-real-time responsiveness — with minimal impact on user experience.
In contrast, exact accuracy is essential in systems like leaderboards, where even a slight ranking error may affect fairness, rewards, or reputation. These designs typically rely on Distributed Heaps or precise sorting, but at the cost of higher complexity and resource usage.
[Common Pitfall]
Assuming exact accuracy is always necessary can lead to over-engineered and inefficient systems. Instead, accuracy strategy should match the product's trust, scale, and latency needs.
[Further Read]
We'll cover leaderboard system design — where precision matters — in a follow-up chapter on our website, including distributed heaps, sorted aggregators, tie-breakers, and real-time ranking consistency.
For user-facing Top-K use cases like YouTube Trending, approximate accuracy is the preferred approach. It enables real-time responsiveness, scalability, and cost efficiency — while maintaining enough precision to reflect actual trends. Exact accuracy can be reserved for offline use cases where correctness is critical, such as creator revenue reports or audits.
Therefore, the non-functional requirement list here, and in the next chapters, we will by default discuss in approximate accuracy design strategy.
NFR1 – Low Latency
Query response time < 10secs
NFR2 – High Scalability
Handle ≥ 5M events/sec across 10B+ videos
NFR3 – Approximate Precision
≤ ±1% error with > 95% Top-K accuracy
NFR4 – Cost Efficiency
< 1MB memory per stream using sketches
NFR5 – Data Freshness
Update trends every 30–60 seconds
Requirement Summary
| Functional Requirements (FRs) | |
|---|---|
| Name | Description |
| 1. Retrieve Top-K Videos | Retrieve the Top-K most popular videos based on predefined ranking criteria. |
| 2. Filter by Geographic Location | Filter Top-K video queries by geographic location. |
| 3. Arbitrary Time Window | Define an arbitrary time window for which the Top-K videos are computed. |
| Non-Functional Requirements (NFRs) | |
|---|---|
| Name | Description |
| 1. Low Latency | Query response time < 10secs. |
| 2. High Scalability | Handle ≥ 5M events/sec across 10B+ videos. |
| 3. Approximate Precision | ≤ ±1% error with > 95% Top-K accuracy. |
| 4. Cost Efficiency | < 1MB memory per stream using sketches. |
| 5. Data Freshness | Update trends every 30–60 seconds. |
Why Define Data Flow Stages?
Clearly outlining data flow stages helps break down system responsibilities, optimize for performance at each step, and isolate concerns like ingestion, aggregation, and serving. It enables better scalability, maintainability, and fault tolerance — especially in infrastructure systems where each stage may be independently scaled, tuned, or audited. For Top-K systems, this modularity is key to supporting real-time updates and accurate trend surfacing under high load.
(location, time window) partition(location, time window)While the online system favors speed and scalability using approximate aggregations, an offline audit stage is essential for reconciliation and accuracy validation. It allows the system to recompute exact Top-K results from raw logs for use cases like billing, creator rewards, compliance, or A/B test validation. This dual-path design ensures real-time performance without sacrificing correctness where it matters.