You need to design a system to download a large ML model (500GB) and send it to all 1,000 GPU workers in a data center. The system must work quickly, handle limited internet speed, and keep working even if computers fail. The candidate must design a system that: - Downloads a 500GB ML model from the internet/cloud - Delivers it to every GPU worker in a data center (1,000 machines) - Minimizes total deployment time end-to-end - Remains resilient to worker failures mid-transfer - Verifies integrity of every chunk received, using a pre-distributed **Chunk Manifest** that describes every piece of the model (byte offsets, checksums, chunk count) Before diving in, ask yourself: - Why is a pipeline better than a tree or sequential transfer? Show the math. - How do we use 100% of the network capacity? - Why split the file into small pieces? How big should each chunk be? - What if a worker in the middle of the chain breaks? How does the system know when the job is done? - Does it still work with 10,000 workers? You don't need to answer all of these right away. A **Senior/Staff+ Engineer** doesn't stop at basic functionality — they **anticipate edge cases**, **design for resilience**, and **push for production-grade reliability**. But **great systems start with great questions.** What would *you* ask next? ## Requirements ### Constraints > **"Full-duplex"** means a worker can upload at 10 Gbps AND download at 10 Gbps simultaneously. This is the key multiplier that makes P2P distribution viable. ### Functional Requirements **FR1 — Download the model from an external source** The system must pull a 500GB model from the internet or a remote object store (e.g., S3) through the single 10 Gbps external pipe. This is the entry point and the fundamental bottleneck. **FR2 — Deliver the full model to every GPU worker** Every one of the 100–1,000 GPU workers must receive a complete, verified copy of the model before it can be used for inference. Partial delivery is not acceptable. **FR3 — Minimize total deployment time** The design must treat time-to-full-delivery as the primary optimization metric. Naive sequential approaches are disqualifying. The architecture must exploit internal bandwidth. ### Non-Functional Requirements **NFR1 — Fault tolerance:** No single non-seed worker failure should abort the transfer. Chunks must have redundant sources. **NFR2 — Bandwidth efficiency:** The system should saturate the 10 Gbps external pipe as the only bottleneck. Internal propagation should not become a ceiling. **NFR3 — Scalability:** The architecture should work for 10,000 workers with minimal design changes — O(log N) propagation, not O(N). **NFR4 — Integrity:** Each chunk must be verified (e.g., via checksum) before a worker marks it as complete or begins seeding it. **NFR5 — Observability:** The system must provide a definitive signal when all workers reach `complete` status, enabling the orchestrator to proceed with inference job scheduling. Completion is derived by the Tracker — no separate signal needed from workers. ### Follow-up Questions to Expect The question below marked **[Core Thinking]** is answered directly in the Core Thinking section. The remaining four are addressed in the Deep Dives. - **[Core Thinking] Strategy:** Why is a pipeline better than a tree or sequential transfer? Show the math. - **[DD1] Bandwidth:** How do we use 100% of the network capacity? - **[DD2] Chunking:** Why split the file into small pieces? How big should each chunk be? - **[DD3] Failures & Completion:** What if a worker in the middle of the chain breaks? How does the system know when the job is done? - **[DD4] Scaling:** Does it still work with 10,000 workers? ## Core Thinking & Design Pattern ### The Fundamental Bottleneck The single 10 Gbps external pipe is the **fundamental constraint**. Everything else is about making that pipe the *only* bottleneck — never a node inside the cluster. The key realization: once any chunk is inside the cluster, it can spread at internal speeds — effectively free. The problem reduces to: *how fast can you fill the cluster from one slow external pipe?* ### Three Strategies: Sequential vs. Tree vs. P2P Pipeline There are three plausible approaches. Understanding why each one fails or succeeds — and doing the math — is what separates a passing answer from a staff-level one. #### Strategy 1 — Sequential (Naive) The seed node receives the full 500GB from the internet (~400 sec), then pushes it to Worker 1, then Worker 2, one by one. Each worker waits for the full file before the next starts. **Result: ~111 hours for 1,000 workers.** ```jsx Time = N × file_size / bandwidth = 1,000 × 500 GB / 10 Gbps = ~111 hours ``` Every worker waits in line. The external pipe is used once per worker sequentially — catastrophically wasteful. #### Strategy 2 — Pure Binary Tree Each node that receives the full file forwards it to 2 children simultaneously. Logarithmic rounds instead of linear steps. **Result: ~67 minutes.** ```jsx Rounds = ceil(log₂(N)) = 10 rounds Time per round = file_size / bandwidth = 400 sec Total = 10 × 400 sec = ~4,000 sec (~67 min) ``` Better — logarithmic rounds. But each round must wait for the *full* 500GB file to land before the next round starts. No pipelining. The file is still treated as an atomic blob. #### Strategy 3 — Chunked Pipeline + P2P (Optimal) Split the file into chunks. The seed node begins seeding chunk 1 the moment it lands — while chunk 2 is still downloading. Workers immediately become seeders for each verified chunk, spreading it to peers in parallel. **Result: ~7 minutes.** ```jsx Download time = file_size / bandwidth = 400 sec Propagation = ceil(log₂(N)) × chunk_transfer_time = 10 × 0.8 sec = 8 sec Total = ~408 sec (~7 min) ``` The key difference: chunking eliminates per-round idle time. Each chunk is an independent unit that enters the propagation graph the moment it lands — not after the full file arrives. The external pipe is always saturated, and internal propagation overlaps with ongoing downloads. > **The staff-level answer:** Pipeline beats tree because it eliminates the per-round wait. It beats sequential because workers become distributors, not consumers. The bottleneck stays the external pipe regardless of fleet size. #### Strategy Comparison ### The P2P Propagation: Doubling Effect Each round, every seeder sends a chunk to one new worker. The count **doubles every round**: ``` Round 0: 1 seeder (seed node) Round 1: 2 seeders Round 2: 4 seeders Round 3: 8 seeders ... Round 10: 1,024 seeders → covers all 1,000 workers ✓ ``` **Rounds needed:** ``` ceil(log₂(1000)) = 10 rounds ``` **Time per round** (1 GB chunk at 10 Gbps): ``` 1 GB / 10 Gbps = 0.8 sec per round ``` **Total propagation overhead:** ``` 10 rounds × 0.8 sec = ~8 seconds ``` > **Staff+ signal:** Propagation is O(log N). At 10,000 workers: ceil(log₂(10000)) = 14 rounds × 0.8s = ~11 sec overhead. The bottleneck stays the external pipe regardless of fleet size. ### Why Chunking Is Essential A single 500GB blob must be fully received before it can be forwarded. Chunks solve three problems: **Optimal chunk size tradeoff:** **The BitTorrent Insight** This problem is **BitTorrent disguised as a systems design question.** The optimal solution turns workers from passive consumers into active distributors. The moment you recognize this, the design falls into place. With the pattern established — chunked P2P pipeline coordinated by a lightweight Tracker — the next step is to nail down exactly what the system needs to store and how components talk to each other. Core Entities & API defines the data model and API that make this pattern concrete. ## Core Entities & API ### Core Entities > The Tracker is the only centralized component. It does not move data — it only moves metadata. All actual data flows peer-to-peer between workers. **Transfer is intentionally excluded** — it is soft operational state (a metrics log or Datadog event), not a domain entity the system depends on to function. #### Model The ML artifact being distributed. One record per deployment job. #### Chunk Atomic unit of transfer. One model splits into many chunks; each is independently downloaded, verified, and seeded. > **Chunk `status` is seed-node-only state.** `downloading` means the seed node is actively pulling this chunk from the external object store — it does not exist inside the cluster yet. `available` means the seed holds it and P2P propagation can begin. Workers never write to `Chunk.status` — their per-chunk progress is tracked entirely through `ChunkAvailability` rows (one row inserted per worker per verified chunk). #### Worker A GPU machine registered in the distribution mesh. A worker **simultaneously downloads chunks it doesn't have yet and seeds chunks it has already verified** — these are not mutually exclusive phases. 10s are marked `offline` and removed from peer maps"], ["`registered_at`", "`TIMESTAMP`", "When this worker joined the current distribution session"], ]} /> **Worker status lifecycle:** 10s; removed from all peer maps"], ]} /> > **Why not separate `downloading` and `seeding` states?** Because they happen at the same time. A worker that has verified chunks 1–50 is actively seeding those to peers while pulling chunks 51–500 in parallel. Splitting them into separate states implies mutual exclusivity that doesn't exist — `active` correctly captures this concurrent reality. #### Tracker *(in-memory coordinator)* The only centralized component. Its sole job is knowing which workers hold which chunks and answering peer lookup requests. It never touches model data — all byte transfers flow directly between workers. The Tracker holds **four in-memory structures**: **WorkerMap** — `HashMap`. Every registered worker entry with IP, status, active upload count, and last heartbeat. Written on `POST /tracker/register` and `POST /tracker/heartbeat`. Read on every peer lookup to filter out offline workers and sort by upload load. 10s ago"], ]} /> **ChunkAvailability** — `HashMap>`. The hot index powering all peer lookups. One entry per (chunk, worker) pair, appended on every `POST /tracker/chunk-complete`. Read on every `GET /tracker/peers`. **HeartbeatTimer** — background loop running every ~1s. Scans WorkerMap; any worker with `last_heartbeat_at > 10s ago` is marked `offline` and purged from ChunkAvailability peer lists immediately. **Model metadata** — read-only constants set at job start: `chunk_count = 500`, `chunk_size_bytes = 1GB`. Used to derive `Worker.status = complete` when `COUNT(ChunkAvailability WHERE worker_id = X) == chunk_count`. No separate completion signal needed from the worker. > **Why all in-memory?** ChunkAvailability alone reaches 500 chunks × 1,000 workers = 500K rows at scale, written on every chunk completion and read on every peer lookup. A Redis hash or in-process dict handles this at microsecond latency. All four structures are **soft state** — fully reconstructible on Tracker restart since workers re-announce their held chunks on reconnect. #### Chunk Manifest *(read-only, distributed at job start)* A static reference file generated once when the deployment job is created. It describes every chunk of the model so that workers know exactly what to fetch and can verify each piece independently — without querying the Tracker for this information. > The Manifest is **immutable** after job creation. Workers fetch it once via `GET /manifest` immediately after registration and cache it locally for the duration of the job. It is the single source of truth for what "done" looks like for each chunk. ### Entity Relationships ### API Design The diagram maps every API call as a single-column sequence flow. Four actors run as vertical lifelines — Worker (orange), Seed Node (blue), Tracker (purple), S3/Cloud (green) — color-coded consistently. Blue solid arrows = requests. Green dashed arrows = responses. Flow 5 is red because it is the only flow where the Tracker is never in the path — data moves directly between workers. The 8 flows below correspond 1-to-1 with the diagram rows. Each includes the full request/response signature and what the Tracker does internally as a side-effect. #### Flow 0 — Worker Fetches Chunk Manifest **Actor:** Worker → Tracker ```bash GET /manifest?model_id= Response 200: { "model_id": "...", "chunk_count": 500, "chunk_size_bytes": 1073741824, "chunks": [ { "chunk_id": "chunk_001", "index": 0, "byte_offset": 0, "size_bytes": 1073741824, "checksum_sha256": "sha256:..." }, ... ] } ``` > Called **once per worker**, immediately after registration (Flow 1) and before entering the download loop. The worker caches this locally for the duration of the job. This is how workers know which chunks exist, what byte ranges to request, and what checksum to verify against after each P2P transfer. The Tracker serves this from an in-memory constant — it never changes after job start. #### Flow 1 — Worker Registration **Actor:** Worker → Tracker ```bash POST /tracker/register Body: { "worker_id": "gpu-042", "ip": "10.0.1.42", "chunks_held": [] } Response 200: { "registered": true } ``` > Tracker creates Worker record, sets `status = idle`. On reconnect after crash, worker re-announces `chunks_held` so Tracker rebuilds ChunkAvailability without a restart. Immediately after this call, the worker fetches the Chunk Manifest via `GET /manifest` (Flow 0) before entering the download loop. #### Flow 2 — Seed Node Downloads Chunk from S3 **Actor:** Seed Node → S3 / Cloud ```bash GET https:///models//chunks/ Response 200: ``` > Seed verifies `checksum_sha256` after download. On match: `Chunk.status` transitions `downloading → available`. On mismatch: retry or mark `failed`. #### Flow 3 — Seed Registers Chunk with Tracker **Actor:** Seed Node → Tracker ```bash POST /tracker/chunk-complete Body: { "worker_id": "seed-node", "chunk_id": "chunk_001", "checksum": "sha256:..." } Response 200 OK ``` > Tracker inserts `ChunkAvailability` row `{ chunk_id, worker_id: seed-node, verified_at }`. Seed now appears in peer lists for this chunk. Any worker querying Flow 4 will receive seed as an available source. #### Flow 4 — Worker Requests Peers for a Chunk **Actor:** Worker → Tracker ```bash GET /tracker/peers?chunk_id=chunk_007 Response 200: { "chunk_id": "chunk_007", "peers": [ { "worker_id": "seed-node", "ip": "10.0.0.1" }, { "worker_id": "gpu-017", "ip": "10.0.1.17" } ] } ``` > Tracker queries `ChunkAvailability WHERE chunk_id = X`, joins `Worker` for IP, sorts by `active_uploads ASC` to avoid hot peers. Returns top-N (e.g. 5) seeders. #### Flow 5 — Worker Pulls Chunk Directly from Peer *(P2P — Tracker never involved)* **Actor:** Worker A → Worker B (peer/seeder) ```bash GET /chunks/{chunk_id} # Sent directly to peer IP — Tracker is never in the path Response 200: ``` > Core data plane. No Tracker hop. Receiver verifies `checksum_sha256` against the manifest. A worker runs multiple of these in parallel for different chunks from different peers — saturating its 10 Gbps download bandwidth. #### Flow 6 — Worker Reports Chunk Verified *(becomes a seeder for this chunk immediately)* **Actor:** Worker → Tracker ```bash POST /tracker/chunk-complete Body: { "worker_id": "gpu-042", "chunk_id": "chunk_007", "checksum": "sha256:..." } # Called once per chunk — 500× total per worker Response 200 OK ``` > Same endpoint as Flow 3. Tracker inserts `ChunkAvailability` row — worker immediately appears in peer lists. `Worker.status = complete` is **derived** by Tracker when `COUNT(ChunkAvailability WHERE worker_id = X) == Model.chunk_count`. No separate signal needed. #### Flow 7 — Heartbeat *(liveness signal)* **Actor:** Worker → Tracker ```bash POST /tracker/heartbeat Body: { "worker_id": "gpu-042" } // sent every ~5 seconds Response 200 OK ``` > Tracker updates `Worker.last_heartbeat_at`. Workers silent >10s: `status = offline`, evicted from all `ChunkAvailability` peer maps. Other workers querying Flow 4 will no longer receive this worker as a candidate. ## High Level Design > The system is structured around five components: **S3/Object Store → Seed Node → Tracker → Chunk Manifest → Worker Mesh**. The external 10 Gbps pipe is the sole bottleneck. Everything inside the data center operates at 10 Gbps full-duplex per worker, making internal propagation nearly free once the first chunk arrives. ### Components **S3 / Object Store** The source of truth for the 500GB model. The Seed Node pulls chunks from here through the single 10 Gbps external pipe. Workers never contact S3 directly — only the Seed Node does. **Seed Node** The single entry point for external data. It downloads chunks from S3 one at a time (~0.8 sec/GB), verifies each chunk's `checksum_sha256` against the Chunk Manifest, registers it with the Tracker via `POST /tracker/chunk-complete`, and immediately begins serving that chunk to requesting workers. The Seed Node is simultaneously a downloader and a seeder — it never waits for the full model to land before distributing. **Tracker** *(in-memory coordinator — metadata only)* Maintains four in-memory structures: `WorkerMap` (all registered workers with IP and heartbeat), `ChunkAvailability` (chunk → worker peer index), `HeartbeatTimer` (evicts silent workers after 10s), and model metadata (`chunk_count`, `chunk_size_bytes` for completion detection). Workers query the Tracker to discover peers for each chunk via `GET /tracker/peers`. The Tracker sorts candidates by `active_uploads ASC` to avoid hot peers and returns the top-N seeders. It does not proxy any chunk data — every byte transfer flows directly between workers, bypassing the Tracker entirely. **Chunk Manifest** A read-only reference distributed to all workers at job start. Contains `chunk_count`, `chunk_size_bytes`, each chunk's `byte_offset`, and its `checksum_sha256`. Workers use this to know which chunks they need and to verify integrity after each P2P transfer, without contacting the Tracker for this information. **Worker Mesh** *(P2P data plane)* Each worker runs a local agent. Before entering the download loop, it completes a short bootstrap sequence: 1. `POST /tracker/register` — announces itself to the Tracker, receives `status = idle` 2. `GET /manifest?model_id=X` — fetches and caches the Chunk Manifest (chunk list, byte offsets, checksums) 3. Begins the continuous download loop — the Tracker derives `status = active` once the first `POST /tracker/chunk-complete` is received from this worker The download loop runs continuously until all chunks are verified: 1. Query the Tracker for a chunk it doesn't yet hold (`GET /tracker/peers?chunk_id=X`) 2. Select a peer from the returned list and pull the chunk directly (`GET /chunks/{chunk_id}`) 3. Verify the chunk's `checksum_sha256` against the locally cached Chunk Manifest 4. Register the verified chunk with the Tracker (`POST /tracker/chunk-complete`) — the worker is now a seeder for this chunk 5. Repeat for the next missing chunk, running multiple parallel transfers simultaneously A worker in `active` status is simultaneously downloading chunks it doesn't have yet and seeding chunks it has already verified. These are not sequential phases but concurrent operations, fully utilizing the 10 Gbps full-duplex internal link. ### Data Flow (Happy Path) ## Deep Dives Each deep dive below addresses one of the follow-up challenge questions from Requirements, and maps directly to one or more NFRs. The strategy question — Sequential vs. Tree vs. P2P Pipeline — is covered in full in Core Thinking & Design Pattern. ### DD1 — Bandwidth: How Do We Hit 100% Network Utilization? The HLD establishes three mechanisms for bandwidth efficiency: external pipe saturation, full-duplex worker operation, and parallel chunk pulls. But it leaves three concrete failure modes unresolved. Each one can silently cap effective utilization far below the theoretical maximum — and none of them show up until you're under real load. **Failure 1 — The Tracker becomes a hotspot** At 1,000 workers each polling `GET /tracker/peers` for every chunk they need, the Tracker receives up to 500,000 RPC calls per distribution job. A single in-process Tracker handles this comfortably at 1,000 workers. At 10,000 it becomes a serialization bottleneck — peer lookup latency rises, workers stall waiting for responses, and effective download throughput drops. And we have three options. The Tracker caches `GET /tracker/peers` responses per chunk with a short TTL (1–2s). Workers hitting the same chunk get the same response from cache — Tracker CPU drops dramatically. Staleness is fine: a slightly stale peer list means a worker tries a peer that may have gone offline, gets a connection error, and retries. The cost is negligible. Deploy multiple read-only Tracker replicas that serve peer lookups. The primary Tracker accepts all writes (`chunk-complete`, `heartbeat`). Replicas sync from primary with eventual consistency. Workers are load-balanced across replicas. Adds operational complexity but handles extreme scale (10,000+ workers). Eliminate the Tracker from the peer-lookup path entirely. Workers periodically exchange chunk availability with random neighbors. Any worker can answer "who has chunk X?" after a few gossip rounds. Convergence is O(log N). No Tracker bottleneck by design — but consistency is eventual and implementation complexity is high. For 1,000 workers: **Option A**. Cache TTL of 1–2s is invisible to users and eliminates the bottleneck with zero infrastructure change. For 10,000+ workers: **Option B** adds replicas on top of caching. **Option C** is the right long-term architecture if you expect to eventually remove the Tracker as a centralized dependency — but it requires reimplementing failure detection as well. *Ask yourself: if the Tracker went down right now, how long would workers keep distributing using cached peer maps? That gap is your Tracker SLA.* **Failure 2 — Hot peers** Workers that joined early and hold many chunks become magnets. If every worker queries the Tracker for `chunk_001` and gets back the same top seeder, that seeder receives 999 simultaneous download requests — its upload bandwidth saturates, its queue backs up, and effective distribution stalls despite abundant peers. The fix is already in the design — the Tracker sorts peer responses by `active_uploads ASC` and returns only the top-N seeders (e.g. 5). But the cap value matters: **Max 4 concurrent uploads** is the recommended default. At 10 Gbps per worker, 4 simultaneous uploads saturates the upload side of the full-duplex link (4 × 10 Gbps = 40 Gbps theoretical — in practice capped by link speed, but the point is you're not leaving capacity idle). **Failure 3 — Checksum verification stalls the pipeline** The naive flow is: download chunk → verify checksum → register with Tracker → begin next download. The verify step (~10–50ms for a 1 GB SHA-256) sits on the critical path. This creates a sequential stall: the worker is idle during verification, then idle again while waiting for the Tracker to respond — before pulling the next chunk. The fix is to parallelize: begin the next chunk download immediately after receiving the previous chunk, while verification runs on a background thread. Registration happens asynchronously when verification completes. A worker in steady state always has one download in flight and one verification running concurrently. External pipe: 10 Gbps, always saturated (Seed downloads back-to-back). Internal: each worker pulls multiple chunks in parallel from different peers, uploads up to 4 simultaneously, verifies asynchronously. The bottleneck stays the external 10 Gbps pipe — exactly where we want it. *Follow-up: what happens to utilization if chunk size drops from 1 GB to 100 MB? More Tracker calls, more ChunkAvailability entries, tighter verification windows. At what point does metadata overhead start eating into throughput?* ### DD2 — Chunking: Why Split? How Big? The HLD says chunks are 1 GB and there are 500 of them. But the design never justifies *why* that size — or what breaks if you get it wrong. Chunk size is the single biggest lever in the entire system: too small and the Tracker drowns in metadata; too large and you lose pipelining entirely and are back to blob transfer. Getting it right requires understanding the three-way tension between pipelining benefit, metadata cost, and integrity granularity. **Why chunking is necessary at all** A monolithic 500 GB file cannot be pipelined. The Seed Node must receive all 500 GB before it can forward a single byte. With 1 GB chunks, the Seed Node begins seeding chunk_001 after 0.8s — while chunks 2–500 are still downloading. Workers become seeders for each verified chunk immediately, compounding the distribution exponentially. Remove chunking and the system degrades to the binary tree strategy: ~67 minutes instead of ~7. Chunking also enables **parallel pulls**: a worker can request chunk_001 from Worker A and chunk_002 from Worker B simultaneously, saturating its full 10 Gbps download link independently of any single peer's upload capacity. **The chunk size decision** At 1 GB: 500 chunks × 1,000 workers = 500K ChunkAvailability rows. An in-memory hashmap handles this trivially. Workers see new chunks available every ~0.8s, keeping every propagation round continuously fed. This is the sweet spot. **Chunk ordering: rarest-first vs sequential** Workers must decide *which* chunk to request next. Two strategies: Workers download chunks in order: chunk_001, then chunk_002, then chunk_003. Easy to implement. But if the Seed Node falls behind, every worker stalls waiting for the next sequential chunk from the same source. The last few chunks become a single-seeder bottleneck regardless of how many peers exist. Workers query `ChunkAvailability` and prioritize chunks with the fewest seeders. This spreads demand across all chunks proportionally, preventing any single chunk from becoming the last bottleneck. Borrowed directly from BitTorrent — proven at internet scale. **Integrity: the per-chunk verification contract (NFR4)** Each chunk in the Manifest carries a `checksum_sha256`. The receiver verifies this after every P2P transfer — before calling `POST /tracker/chunk-complete`. A worker never seeds a chunk it has not verified. If verification fails (corrupted seeder, network bit-flip), the chunk is discarded and re-requested from a different peer. The bad peer is not evicted — it may have served the chunk correctly to other workers. This is the key integrity property: **verification happens at the smallest independently transferable unit**. A single corrupted chunk costs one retry (~0.8s), not a full restart. Don't just say "1 GB is the sweet spot." Derive it: at 10 Gbps, 1 GB takes 0.8s to transfer. That sets the propagation round time. 10 rounds × 0.8s = 8s overhead vs 400s download = 2% overhead. Any chunk size that keeps that ratio under ~5% and keeps ChunkAvailability under ~1M rows is acceptable. The 1 GB default hits both constraints cleanly. *Follow-up: if internal bandwidth doubled to 20 Gbps per worker, would you change the chunk size? What would change and what would stay the same?* ### DD3 — Fault Tolerance & Completion Detection The HLD shows the happy path: workers register, download chunks, become seeders, and eventually reach `complete`. It says nothing about what happens when components fail mid-distribution. This matters more here than in most systems because failures are not recoverable from the client side — there is no user to retry. The system must detect and recover automatically, and it must do so without disrupting the 999 workers that are healthy. There are three distinct failure surfaces, each requiring a different response: **Failure 1 — A worker crashes mid-transfer (most common)** Worker B is downloading chunk_042 from Worker A. Worker A crashes. Worker B's HTTP connection to A times out (~5s). Without intervention, Worker B would stall or retry forever. Recovery flow: 1. Worker A stops sending heartbeats 2. Tracker's `HeartbeatTimer` (1s scan loop) detects silence >10s → marks Worker A `offline`, purges it from all `ChunkAvailability` entries immediately 3. Worker B's connection times out → Worker B calls `GET /tracker/peers?chunk_id=042` again 4. Tracker returns Worker C, Worker D (Worker A no longer appears) 5. Worker B re-requests chunk_042 from Worker C 6. **Total delay: ~5–15 seconds per failed transfer** The P2P redundancy is the key property: once any chunk has 2+ seeders, no single worker failure can block its distribution. By the time 10 rounds of propagation have completed, every chunk has 1,024 seeders — any single failure is invisible. If a chunk fails verification after a P2P transfer (corrupted seeder, network bit-flip), it is discarded and re-requested from a different peer. The bad seeder is not evicted — it may have served other workers correctly. The cost is one retry per affected chunk (~0.8s). **Failure 2 — The Seed Node crashes** The Seed Node is the only source for chunks not yet ingested into the cluster. If it crashes, chunks not yet downloaded from S3 cannot enter the mesh. Workers with incomplete chunk sets stall waiting for chunks that will never arrive. Run two Seed Node instances sharing the same external access credentials. The standby monitors the primary via heartbeat. On failure, the standby takes over and resumes downloading from S3 at the last registered chunk offset. Failover time: <30s. Workers experience a brief stall, then resume normally. Before the distribution job starts, copy the 500 GB model from S3 to a local object store inside the data center. The Seed Node pulls from local storage instead of the internet. S3 failure is eliminated as a dependency. Adds a staging step but makes the distribution job fully self-contained once started. Run N Seed Nodes, each responsible for a non-overlapping chunk range (Seed A owns chunks 1–250, Seed B owns 251–500). Failure of one Seed only blocks its chunk range — not the full job. Workers holding those chunks continue seeding normally; only new ingestion of the affected range stalls. Most complex to coordinate but highest resilience. **Failure 3 — The Tracker crashes** The Tracker is the only centralized metadata coordinator. If it crashes, workers cannot discover new peers — but they can continue seeding chunks to workers who already know about them via cached peer maps. Run a standby Tracker that takes over on primary failure. State is fully reconstructible: all workers re-announce their `chunks_held` via `POST /tracker/register` on reconnect. Failover time: ~5s. Workers stall on new peer discovery during that window, then resume. Replicate Tracker state across 3 nodes using Raft consensus. Leader election is automatic. Strong consistency guarantees: no stale reads. Operationally heavier, but appropriate for deployments where even a 5s stall is unacceptable. Eliminate the Tracker from peer lookup entirely. Workers share chunk availability with random neighbors periodically. The Tracker handles only registration and completion detection. Worker failures and peer discovery become fully decentralized. Highest resilience, highest implementation complexity. Run the Tracker and Seed Node as active-passive pairs — they are the only two centralized components in the design. Worker failures need no configuration: P2P redundancy and heartbeat eviction handle them automatically. This covers the vast majority of failure scenarios with the least operational overhead. *Ask yourself: if both the Tracker and Seed Node failed simultaneously, what is the blast radius? Which workers are stuck, which can keep going, and how long before the job must be restarted from scratch?* ### DD4 — Scaling: Does It Work with 10,000 Workers? The HLD is designed for 1,000 workers. The math for P2P propagation scales elegantly — O(log N) means 10× more workers adds only 4 more rounds. But the Tracker does not scale the same way. It is the only centralized component in the design, and at 10,000 workers it faces three simultaneous pressures that don't appear at 1,000: query volume, peer map size, and registration storms at job start. Each needs an explicit solution. **The math first — propagation still works** ```jsx ceil(log₂(10,000)) = 14 rounds Propagation overhead = 14 × 0.8s = ~11s Total time = 400s download + 11s propagation = ~411s (~7 min) ``` **The total time barely changes.** The external 10 Gbps pipe still dominates. Seed Node behavior is unchanged — it still saturates the pipe regardless of how many workers consume from it, because workers pull from each other, not from the Seed Node. **What actually changes: the Tracker** Deploy read-only Tracker replicas that serve `GET /tracker/peers`. The primary handles all writes (`chunk-complete`, `heartbeat`, `register`). Replicas sync with eventual consistency — a slightly stale peer list is fine; a worker that tries an offline peer just retries. Alongside this: cap peer responses at top-N (e.g. 5) sorted by `active_uploads ASC`. At 10,000 workers, returning the full seeder list per chunk wastes bandwidth and processing. Workers only need a handful of peers to saturate their download link. Partition the Tracker horizontally: Tracker A owns chunks 0–249, Tracker B owns chunks 250–499. Workers hash their `chunk_id` to determine which Tracker instance to query. Each Tracker handles half the write volume and half the read volume. Add shards as fleet grows. Trade-off: routing logic now lives on the worker side. Tracker failures affect half the chunk space rather than all of it. More operationally complex than replicas, but necessary if a single primary cannot handle write throughput. Eliminate the Tracker from peer discovery entirely. Workers exchange chunk availability with random neighbors on a fixed interval. Any worker can answer "who has chunk X?" after O(log N) gossip rounds. Convergence time matches propagation time — no additional overhead. The Tracker is retained only for registration and completion detection. The hot path — peer lookup — is fully decentralized. Trade-off: implementation complexity is high. Failure detection must also be gossip-based (e.g. suspicion protocols). Staleness windows are longer than with a centralized Tracker. Appropriate when the fleet exceeds ~50,000 workers and Tracker sharding becomes unmanageable. **Manifest fetch storm at job start** At 10,000 workers all calling `GET /manifest` simultaneously at job start, the Tracker receives 10,000 identical requests in the first few seconds. The fix is simple: cache the manifest response at an edge proxy or CDN inside the data center. The manifest is immutable after job creation — TTL can be the full duration of the job. Workers hit the cache, not the Tracker primary. Don't just say "shard the Tracker." Show the math that motivates it: at 10,000 workers × 500 chunks × 1 chunk-complete call each = 5M writes per job. At 1s scan intervals for heartbeat, that's another 10,000 writes/sec continuously. A single Redis instance handles ~100K ops/sec — so sharding becomes necessary somewhere between 10K and 100K workers, not at 1K. *Follow-up: if the Tracker is replaced entirely with a gossip protocol, what happens to completion detection (NFR5)? How does the orchestrator know the job is done?* ## Final Thoughts Distributing a 500 GB model to 1,000 GPU workers looks like a data transfer problem. It isn't. It's a propagation problem — and the moment you see it that way, the entire design falls out of one insight: once a chunk is inside the cluster, it spreads at internal speeds for free. The only real constraint is how fast you can fill the cluster from the single slow external pipe. Everything else is about making sure nothing inside the system becomes a second bottleneck. What makes this question unusual in an interview context is that it rewards a specific kind of thinking: **quantitative reasoning over intuition**. Most system design questions reward breadth — naming the right components, drawing the right arrows. This one rewards depth. The candidate who says "we need chunking" is not wrong. The candidate who derives *why 1 GB is the right chunk size*, shows the 2% overhead calculation, and then explains what breaks at 100 MB — that candidate is operating at a different level. The 4 deep dives map directly to the four ways an interviewer will probe your answer: **DD1 tests whether you understand utilization as a systems property, not a component property.** Bandwidth efficiency isn't just about having fast links — it's about making sure no single component serializes the flow. The Tracker hotspot, the hot peer problem, and the verification stall are all the same failure: a bottleneck shifting from where it should be (the external pipe) to somewhere it shouldn't be. **DD2 tests whether you can derive, not just recall.** Anyone can say "chunking enables pipelining." The question is whether you can show the math that sets the chunk size, explain the three-way tension between pipelining, metadata cost, and integrity granularity, and then reason about what happens when the constraints change. **DD3 tests your failure mode intuition.** Production systems don't fail in one place — they fail in layers. Worker failure, Seed Node failure, and Tracker failure are three completely different problems with different blast radii and different recovery strategies. Candidates who give one generic answer ("we retry") haven't thought about which component failed and why that changes everything. **DD4 tests whether you know where the scaling ceiling actually is.** The propagation math scales beautifully — O(log N) is O(log N). The Tracker does not. Recognizing that the Tracker is the first thing that breaks at 10,000 workers, quantifying exactly when it breaks, and knowing which option (cache, replicas, gossip) to apply at which scale threshold is what separates a good answer from a complete one. **Derive numbers, don't assert them** When asked how long the job takes, walk through the math: 500 GB / 10 Gbps = 400s download, 10 rounds × 0.8s = 8s propagation, total ~408s. When asked about chunk size, show why 1 GB keeps propagation overhead under 2% and ChunkAvailability under 500K rows. Numbers without derivations are guesses. Numbers with derivations are engineering. **Name the tradeoff before being asked** For every design decision — 1 GB chunks vs 4 GB, active-passive vs Raft, cached peers vs read replicas — name the tradeoff yourself before the interviewer probes it. This signals you understand the design space, not just the answer you picked. The strongest candidates say: "I'm choosing X because of Y, and I'd switch to Z if the constraint changes to W." **The bottleneck question is always the right question** Every decision in this system comes back to one question: does this keep the 10 Gbps external pipe as the only bottleneck? Chunking — yes. P2P distribution — yes. Tracker peer caching — yes. Sequential verification — no. Hot peers without upload caps — no. Train yourself to evaluate every design choice through this lens and the answers become obvious. **End with the change vector** The closing signal question — *"What changes if you have 10,000 workers instead of 1,000?"* or *"What if the external pipe doubled to 20 Gbps?"* — is where strong answers separate from great ones. Great answers don't just say what changes. They identify which constraints are load-bearing (the external pipe, the Tracker write volume) and which are incidental (propagation round count, chunk size), and explain why. Showing you've internalized the design rather than memorized it is the clearest possible signal. > These aren't toy problems — they reflect real infrastructure challenges at companies deploying large-scale ML systems. If you want to walk through this design in a mock interview or get feedback on your own approach, book a session with us. We're here to help you sharpen your thinking, build your confidence, and land your next offer.