ML Model Distribution

Requirements#

Constraints#

"Full-duplex" means a worker can upload at 10 Gbps AND download at 10 Gbps simultaneously. This is the key multiplier that makes P2P distribution viable.

Functional Requirements#

FR1 — Download the model from an external source

The system must pull a 500GB model from the internet or a remote object store (e.g., S3) through the single 10 Gbps external pipe. This is the entry point and the fundamental bottleneck.

FR2 — Deliver the full model to every GPU worker

Every one of the 100–1,000 GPU workers must receive a complete, verified copy of the model before it can be used for inference. Partial delivery is not acceptable.

FR3 — Minimize total deployment time

The design must treat time-to-full-delivery as the primary optimization metric. Naive sequential approaches are disqualifying. The architecture must exploit internal bandwidth.

Non-Functional Requirements#

NFR1 — Fault tolerance: No single non-seed worker failure should abort the transfer. Chunks must have redundant sources.

NFR2 — Bandwidth efficiency: The system should saturate the 10 Gbps external pipe as the only bottleneck. Internal propagation should not become a ceiling.

NFR3 — Scalability: The architecture should work for 10,000 workers with minimal design changes — O(log N) propagation, not O(N).

NFR4 — Integrity: Each chunk must be verified (e.g., via checksum) before a worker marks it as complete or begins seeding it.

NFR5 — Observability: The system must provide a definitive signal when all workers reach complete status, enabling the orchestrator to proceed with inference job scheduling. Completion is derived by the Tracker — no separate signal needed from workers.

Follow-up Questions to Expect#

The question below marked [Core Thinking] is answered directly in the Core Thinking section. The remaining four are addressed in the Deep Dives.

• [Core Thinking] Strategy: Why is a pipeline better than a tree or sequential transfer? Show the math.
• [DD1] Bandwidth: How do we use 100% of the network capacity?
• [DD2] Chunking: Why split the file into small pieces? How big should each chunk be?
• [DD3] Failures & Completion: What if a worker in the middle of the chain breaks? How does the system know when the job is done?
• [DD4] Scaling: Does it still work with 10,000 workers?

Core Thinking & Design Pattern#

The Fundamental Bottleneck#

The single 10 Gbps external pipe is the fundamental constraint. Everything else is about making that pipe the only bottleneck — never a node inside the cluster.

The key realization: once any chunk is inside the cluster, it can spread at internal speeds — effectively free. The problem reduces to: how fast can you fill the cluster from one slow external pipe?

Three Strategies: Sequential vs. Tree vs. P2P Pipeline#

Strategy 1 — Sequential (Naive)

The seed node receives the full 500GB from the internet (~400 sec), then pushes it to Worker 1, then Worker 2, one by one. Each worker waits for the full file before the next starts.

Result: ~111 hours for 1,000 workers.

Time = N × file_size / bandwidth
     = 1,000 × 500 GB / 10 Gbps
     = ~111 hours

Every worker waits in line. The external pipe is used once per worker sequentially — catastrophically wasteful.

Strategy 2 — Pure Binary Tree

Each node that receives the full file forwards it to 2 children simultaneously. Logarithmic rounds instead of linear steps.

Result: ~67 minutes.

Rounds = ceil(log₂(N)) = 10 rounds
Time per round = file_size / bandwidth = 400 sec
Total = 10 × 400 sec = ~4,000 sec (~67 min)

Better — logarithmic rounds. But each round must wait for the full 500GB file to land before the next round starts. No pipelining. The file is still treated as an atomic blob.

Strategy 3 — Chunked Pipeline + P2P (Optimal)

Split the file into chunks. The seed node begins seeding chunk 1 the moment it lands — while chunk 2 is still downloading. Workers immediately become seeders for each verified chunk, spreading it to peers in parallel.

Result: ~7 minutes.

Download time = file_size / bandwidth = 400 sec
Propagation   = ceil(log₂(N)) × chunk_transfer_time
              = 10 × 0.8 sec = 8 sec
Total         = ~408 sec (~7 min)

The key difference: chunking eliminates per-round idle time. Each chunk is an independent unit that enters the propagation graph the moment it lands — not after the full file arrives. The external pipe is always saturated, and internal propagation overlaps with ongoing downloads.

The staff-level answer: Pipeline beats tree because it eliminates the per-round wait. It beats sequential because workers become distributors, not consumers. The bottleneck stays the external pipe regardless of fleet size.

Strategy Comparison

The P2P Propagation: Doubling Effect#

Each round, every seeder sends a chunk to one new worker. The count doubles every round:

Round 0:  1 seeder   (seed node)
Round 1:  2 seeders
Round 2:  4 seeders
Round 3:  8 seeders
...
Round 10: 1,024 seeders → covers all 1,000 workers ✓

Rounds needed:

ceil(log₂(1000)) = 10 rounds

Time per round (1 GB chunk at 10 Gbps):

1 GB / 10 Gbps = 0.8 sec per round

Total propagation overhead:

10 rounds × 0.8 sec = ~8 seconds

Staff+ signal: Propagation is O(log N). At 10,000 workers: ceil(log₂(10000)) = 14 rounds × 0.8s = ~11 sec overhead. The bottleneck stays the external pipe regardless of fleet size.

Why Chunking Is Essential#

A single 500GB blob must be fully received before it can be forwarded. Chunks solve three problems:

Optimal chunk size tradeoff:

Core Entities & API#

Core Entities#

The Tracker is the only centralized component. It does not move data — it only moves metadata. All actual data flows peer-to-peer between workers. Transfer is intentionally excluded — it is soft operational state (a metrics log or Datadog event), not a domain entity the system depends on to function.

Model

The ML artifact being distributed. One record per deployment job.

Chunk

Atomic unit of transfer. One model splits into many chunks; each is independently downloaded, verified, and seeded.

Chunk status is seed-node-only state. downloading means the seed node is actively pulling this chunk from the external object store — it does not exist inside the cluster yet. available means the seed holds it and P2P propagation can begin. Workers never write to Chunk.status — their per-chunk progress is tracked entirely through ChunkAvailability rows (one row inserted per worker per verified chunk).

Worker

A GPU machine registered in the distribution mesh. A worker simultaneously downloads chunks it doesn't have yet and seeds chunks it has already verified — these are not mutually exclusive phases.

Worker status lifecycle:

Why not separate downloading and seeding states? Because they happen at the same time. A worker that has verified chunks 1–50 is actively seeding those to peers while pulling chunks 51–500 in parallel. Splitting them into separate states implies mutual exclusivity that doesn't exist — active correctly captures this concurrent reality.

Tracker (in-memory coordinator)

The only centralized component. Its sole job is knowing which workers hold which chunks and answering peer lookup requests. It never touches model data — all byte transfers flow directly between workers.

The Tracker holds four in-memory structures:

WorkerMap — HashMap<worker_id, WorkerState>. Every registered worker entry with IP, status, active upload count, and last heartbeat. Written on POST /tracker/register and POST /tracker/heartbeat. Read on every peer lookup to filter out offline workers and sort by upload load.

ChunkAvailability — HashMap<chunk_id, Vec<worker_id>>. The hot index powering all peer lookups. One entry per (chunk, worker) pair, appended on every POST /tracker/chunk-complete. Read on every GET /tracker/peers.

HeartbeatTimer — background loop running every ~1s. Scans WorkerMap; any worker with last_heartbeat_at > 10s ago is marked offline and purged from ChunkAvailability peer lists immediately.

Model metadata — read-only constants set at job start: chunk_count = 500, chunk_size_bytes = 1GB. Used to derive Worker.status = complete when COUNT(ChunkAvailability WHERE worker_id = X) == chunk_count. No separate completion signal needed from the worker.

Why all in-memory? ChunkAvailability alone reaches 500 chunks × 1,000 workers = 500K rows at scale, written on every chunk completion and read on every peer lookup. A Redis hash or in-process dict handles this at microsecond latency. All four structures are soft state — fully reconstructible on Tracker restart since workers re-announce their held chunks on reconnect.

Chunk Manifest (read-only, distributed at job start)

A static reference file generated once when the deployment job is created. It describes every chunk of the model so that workers know exactly what to fetch and can verify each piece independently — without querying the Tracker for this information.

The Manifest is immutable after job creation. Workers fetch it once via GET /manifest immediately after registration and cache it locally for the duration of the job. It is the single source of truth for what "done" looks like for each chunk.

With the pattern established — chunked P2P pipeline coordinated by a lightweight Tracker — the next step is to nail down exactly what the system needs to store and how components talk to each other. Core Entities & API defines the data model and API that make this pattern concrete.