GPU Scheduling Platform

Requirements#

Clarify the scope first#

The question as written contains traps — a running-job cap that contradicts the utilization target, a 5-second start objective next to a 30-second preemption grace period. Clear them before drawing anything. The answers below are the ones an OpenAI-style interviewer gives when asked:

Out of scope (say this out loud): gang scheduling across nodes, the training framework inside the container, billing, and autoscaling the physical fleet. Each is a good extension question; none should eat the first ten minutes.

Functional Requirements#

Four FRs, each mapping to one flow in the High Level Design.

FR1 — Submit, cancel, and track GPU jobs

A client sends POST /v1/jobs with an image, GPU requirements, a priority, and an idempotency key, and gets back a durable job_id in well under 200 ms. Status, logs, and results are read asynchronously; cancellation is accepted asynchronously too.

FR2 — Schedule with priority and resource awareness, reserving GPUs atomically

The scheduler picks the next eligible job across teams and tiers, finds a node that satisfies GPU type, count, and memory, and claims the GPUs in one conditional transaction. Two schedulers can never both succeed for the same capacity.

FR3 — Execute jobs on node agents and expose logs and results

A node agent on every worker starts the container exactly once per attempt, confirms the start, heartbeats, streams logs and checkpoints to object storage, and reports completion through a protocol that tolerates a lost acknowledgment.

FR4 — Preempt lower-priority jobs and retry failed attempts

When a higher-priority job cannot be placed, the platform gracefully preempts strictly lower-priority work and resumes it later from its checkpoints. Failed or lost attempts are retried as new attempts with new identities.

at-least-once attempts     !=  exactly-once execution
single ACCEPTED result     !=  single running container
lease expired              !=  worker stopped
critical priority          !=  another team's guaranteed quota

Non-Functional Requirements#

NFR1 — Scale. 1,000 nodes, ~8,000 GPUs, ~10,000 queued jobs at peak, no fixed cap on concurrent jobs — concurrency is bounded by GPU capacity, and the target is more than 80% GPU utilization.

NFR2 — Latency. Submission p95 under 200 ms. A high-priority job starts within 5 seconds when sufficient capacity is immediately available; a normal-priority job within 2 minutes. The clause matters: waiting for a preemption grace period is not a start-latency violation.

NFR3 — Scheduling correctness. No double allocation of a GPU, ever. Durable job state. Retries that cannot create two accepted results for one job.

NFR4 — Availability and recovery. 99.9% for the control plane. Scheduler crashes, node crashes, and partitions are recovered automatically; node failure is detected within about 30 seconds; nothing stays stuck or permanently reserved.

NFR5 — Fairness and isolation. Per-team guaranteed quotas, no starvation of lower tiers, and noisy-neighbor isolation. Isolation (one team's job cannot hurt another's) and fairness (who gets the next free GPUs) are different requirements.

Scale & Back-of-Envelope Estimations#

The number that identifies the real system is not 8,000 — it is how little scheduling throughput the cluster needs:

cluster frees capacity at   ~1 job / s        (steady state)
one leader can place        ~100 jobs / s     (one transaction each)

The scheduler is not the bottleneck, so "add scheduler replicas" is almost never the answer. The constraints that actually decide utilization are feasibility — an 8-GPU job needs eight free GPUs on the same node, and a cluster that is 35% free can fit zero of them (DD7) — and correctness: one conditional transaction per node row is what prevents double allocation (DD1). Every number above is an illustrative assumption to be replaced by measured queue age and free-GPUs-per-node histograms.

Interviewer signal: candidates who multiply 1,000 by 8 and stop have done arithmetic. Candidates who notice that a single leader out-paces the cluster by two orders of magnitude, and then name fragmentation and the per-node transaction as the real limits, have found the system.

Follow-up Questions to Expect#

These nine questions give the rest of the design its purpose. Raise them before drawing the final architecture — every component below exists because one of them forces it.

• [DD1] Atomic reservation: two schedulers see four free GPUs on the same node. What prevents both from allocating them?
• [DD2] Lost acknowledgment: the scheduler reserves, the container starts, the scheduler crashes before recording RUNNING. How do you recover without a duplicate or a false release?
• [DD3] Partition and fencing: a node is silent for 30 seconds but still training. Do you reschedule, and what stops the old attempt from publishing?
• [DD4] Quota, fairness, and starvation: who gets the next GPUs — Team A's critical jobs over quota, or Team B's normal jobs under quota? And why have low-priority jobs waited 20 hours?
• [DD5] Preemption: a critical job needs all 8 GPUs of a node running four 2-GPU jobs. Walk through the protocol — including the job that ignores SIGTERM.
• [DD6] Scheduler failover: three replicas, the leader crashes mid-assignment; later, an old leader keeps running after losing its lease. How do you prevent two leaders from scheduling?
• [DD7] Utilization and fragmentation: 65% utilization with 3,000 jobs queued. Prove why, before touching the scheduler.
• [DD8] Idempotent completion: a job uploaded its artifacts and the node died before reporting SUCCEEDED. Do you re-run it?
• [DD9] Start latency: the 5-second start SLO is missed because an image pull takes 8 seconds. What changes?

Core Thinking & Design Pattern#

Before drawing boxes, orient around three tensions that shape everything:

1. Separate the logical job from the execution attempt#

A Job is what the user asked for. A JobRun is one attempt to do it — on one node, with one reservation, under one fencing token. Retries, preemptions, and failovers create new attempts with new identities; they never resurrect an old one. This is what lets the platform promise at-least-once attempts and one accepted result at the same time, without pretending a user's training container is idempotent.

2. GPU capacity is a correctness problem, not a counter#

A cached free-GPU count tells you where to look; it must never decide who gets the GPUs. The allocator is one conditional transaction in the source of truth — decrement available capacity only if enough remains, insert the reservation, commit — and the per-node invariant reserved + available = total is checked, not assumed. Redis orders work; PostgreSQL decides capacity.

3. A lease expiring is a question, not an answer#

Missed heartbeats and expired leases tell you the control plane lost contact, not that a container stopped. The working design treats expiry as "reconcile now": ask the agent, read its last heartbeat, look for a completion manifest, and only then release or retry. Because reconciliation can still be wrong under a partition, every commit point — reservation, assignment, heartbeat, completion, and leadership itself — is fenced by a monotonic token that rejects stale actors.

Core Entities & API#

Core Entities#

Five tables carry the whole design. Spend your explanation on Reservation, NodeCapacity, and JobRun — their consistency is the architectural problem. NodeCapacity could be folded into WorkerNode; keeping it separate makes the allocation transaction easier to explain.

Job

The logical job: what the user asked for and where it stands.

JobRun

One execution attempt. A job may have several; at most one is authoritative.

Reservation

Ownership of specific GPUs by one attempt, for a bounded time.

NodeCapacity and WorkerNode

Three state machines — deliberately small:

Job          PENDING -> SCHEDULED -> RUNNING -> SUCCEEDED | FAILED | CANCELLED
             FAILED (retries left) / PREEMPTED  -> PENDING   (new attempt)

JobRun       ASSIGNED -> RUNNING -> SUCCEEDED | FAILED
             RUNNING  -> PREEMPTING -> PREEMPTED

Reservation  RESERVED -> CONFIRMED -> RELEASED
             RESERVED/CONFIRMED -> EXPIRED   only after reconciliation says the GPUs are free

There is no transition that frees capacity on a timer. Every path to RELEASED or EXPIRED passes through a confirmation from the node agent or an explicit fence (DD2, DD3, DD5).

Storage split

API Design#

External (FR1)

POST /v1/jobs
Authorization: Bearer <token>            # team comes from here
Idempotency-Key: 7f3c-...                # required for safe client retries

{
  "image": "registry/train:1.4",
  "gpu_count": 4, "gpu_type": "H100", "memory_mb": 65536,
  "priority": "HIGH",
  "command": ["python", "train.py"],
  "max_retries": 2
}

Response 201: { "job_id": "J1", "status": "PENDING" }
Response 409: { "error": "idempotency_key_reused_with_different_payload" }

GET    /v1/jobs/{id}          -> status, current attempt, timestamps, result_uri
GET    /v1/jobs/{id}/logs     -> log stream (running) or log_uri (finished)
DELETE /v1/jobs/{id}          -> 202 Accepted; cancellation is asynchronous

Submission is asynchronous by nature: the call persists and enqueues, then answers. Nothing on this path touches a GPU node, which is how the 200 ms objective survives a 10,000-job backlog.

Internal: scheduler ↔ node agent (FR2–FR4)

ReserveGPU         (run_id, node_id, gpu_count, gpu_type, leader_epoch)
                   -> reservation_id, gpu_ids, lease_expires_at, fencing_token   # a DB transaction, not an RPC to the node
AssignJob          (run_id, reservation_id, fencing_token, image, command)       # idempotent per run_id
ConfirmAssignment  (reservation_id, fencing_token)      -> CONFIRMED / REJECTED   # agent -> control plane
Heartbeat          (node_id, agent_epoch, running: [{run_id, fencing_token}], free_gpus, health)
Preempt            (run_id, fencing_token, grace_seconds)
ReportCompletion   (run_id, fencing_token, manifest_uri) -> SUCCEEDED / STALE    # accepted once
ReleaseGPU         (reservation_id, fencing_token)       -> RELEASED / ALREADY_RELEASED

ReserveGPU is a transaction against PostgreSQL, not a message to the node. Sending "reserve" over RPC provides no atomicity; the database does. Internal calls go over gRPC; every one of them carries the fencing token and is rejected when it is stale.

High Level Design#

The best way to present this design is not to draw the final architecture immediately. Start with the simplest control plane that could work, add one component only when a real failure forces it, and keep every box's position stable. Narrate it in four phases, each in its own colour on the whiteboard: Phase 1 submit (black), Phase 2 schedule (blue), Phase 3 execute with results (green), Phase 4 fetch status (purple). Every phase changes a few rows, so the schema blocks next to the database carry example values and show exactly what each phase writes. Each phase below is an interactive walkthrough: it plays on its own as it comes into view, and you can pause, step with the arrows or keyboard, click any component for its role, switch scenarios, and expand it.

Walk the interviewer through the picture phase by phase:

Phase 1  submit    User A -> API-GW -> job submission service -> PostgreSQL (Jobs: J1, PENDING)
                   -> initial queue entry in Redis by priority -> 201 {job_id}
Phase 2  schedule  scheduler pulls by priority -> evaluates candidate workers (GPU count, memory, fairness / quota)
                   -> ONE transaction: JobRun r1 + Reservation R1 (lease, token 7) + NodeCapacity W1 8 -> 4
                   -> assign to W1 (r1, R1, token 7), idempotent per run_id
Phase 3  execute   W1 confirms the start -> RUNNING; heartbeat every 10 s to the Node Registry;
                   logs / checkpoints / artifacts to Blob Storage; report completion -> SUCCEEDED,
                   R1 RELEASED, NodeCapacity W1 4 -> 8
Phase 4  track     User A -> API-GW -> job status service -> PostgreSQL (status), Blob Storage (logs)

Preemption and retry (FR4) do not add a fifth phase: a retry is Phase 2 and Phase 3 again with a new attempt. They are covered at the end of this section and in DD3 and DD5.

Phase 1 — Submit (FR1): a 200 ms control plane for jobs that run for hours#

POST /v1/jobs  ->  validate  ->  INSERT Jobs(PENDING)  [unique (team_id, idempotency_key)]
               ->  after commit: enqueue (job_id, priority, enqueued_at) into Redis
               ->  201 { job_id, status: PENDING }                      # p95 well under 200 ms

DELETE /v1/jobs/{id}      ->  202; cancel_requested = true
                              PENDING -> skipped by the scheduler -> CANCELLED
                              RUNNING -> agent SIGTERMs the container (same path as preemption)

Two design points worth narrating here:

First, PostgreSQL first, Redis second. The Jobs row is the source of truth; the queue entry is a hint written after the commit. If the enqueue fails, a sweeper re-enqueues PENDING rows; if Redis is lost, the queues are rebuilt from the table (DD6). The reverse order — enqueue, then persist — creates jobs that exist only in a cache.

Second, idempotent submission is not optional. A client retry after a timeout must return the same job_id, not a second job that burns 8 GPUs for 10 hours. Scope the key to the team and bind it to a payload hash; the same key with a different payload is a 409, never a silent reuse.

Phase 2 — Schedule (FR2): pick a job, pick a node, claim GPUs in one transaction#

loop (leader only):
  1  quota check    team under its guaranteed share?  else within its borrow limit?
  2  pick team      DRF: the most under-served dominant share (GPU or memory)
  3  pick job       highest priority in that team, plus aging credits for long waits
  4  pick node      filter: gpu_type, free >= gpu_count, memory, health = HEALTHY
                    best fit: the node with the LEAST free GPUs that still fits
  5  reserve        BEGIN
                      UPDATE node_capacity SET available_gpus = available_gpus - 4, version = version + 1
                       WHERE node_id = 'W1' AND available_gpus >= 4;      -- 0 rows -> abort, back to step 4
                      INSERT job_runs     (r1, J1, attempt 1, W1, ASSIGNED)
                      INSERT reservations (R1, r1, W1, gpu_ids [4..7], RESERVED, lease now()+30s, token 7, epoch e)
                      UPDATE jobs SET status = 'SCHEDULED' WHERE job_id = 'J1' AND status = 'PENDING'
                    COMMIT

Redis orders, PostgreSQL decides. Popping by priority and age from a sorted set is cheap and good enough to choose what to try next. It must never decide who gets the GPUs: two schedulers reading the same cached counter would both "see four free". The conditional UPDATE with its row lock is the allocator; a stale cache only costs a retry (DD1).

A reservation is an ownership record, not a counter. It says which attempt holds which GPUs, until when, under which token. Its expiry does not free capacity by itself — a recovery step must first learn whether the node agent is still using those GPUs (DD2, DD3). Reserve gpu_ids, not just a count, when nodes mix GPU types or jobs pin devices.

Best fit, not emptiest node. Spreading small jobs across empty nodes feels balanced and quietly destroys the cluster's ability to place 8-GPU jobs. Placement is a bin-packing decision from the first day (DD7).

Phase 3 — Execute with results (FR3): the node agent owns the container, the platform owns the state transitions#

1  Scheduler  -> Agent(W1):  AssignJob(r1, R1, token 7, image, command)
2  Agent:        append to durable assignment log  ->  start container ONCE per run_id  ->  pin GPUs 4..7
3  Agent      -> Control plane:  ConfirmAssignment(R1, token 7)
                 Reservation RESERVED -> CONFIRMED,  JobRun ASSIGNED -> RUNNING,  Job -> RUNNING
4  Container  -> Object Storage:  logs, checkpoint every N minutes, artifacts under attempts/r1/
5  Agent      -> Node Registry:   heartbeat every 10 s { health, free_gpus, running: [(r1, token 7)] }
6  exit 0     -> Agent: ReportCompletion(r1, token 7, manifest_uri)
7  Completion Service: verify manifest + token -> one transaction: r1 SUCCEEDED, J1 SUCCEEDED,
                 R1 RELEASED, available_gpus += 4, result_uri = manifest

Accept is not start, and start is not confirm. The agent records the assignment durably before starting anything, so an agent restart replays it instead of forgetting it. Starting is idempotent per run_id — a second AssignJob(r1) answers "already running". Confirming is a third, separate step, and it is the one that can be lost without the work being lost (DD2).

The heartbeat carries the running attempts. Node health and free GPUs are not enough. Each heartbeat lists the run_ids and tokens actually executing on the node — that list is what makes reconciliation after a scheduler crash cheap and what lets the failure detector distinguish a dead node from a dead container.

Where the token is enforced. AssignJob, Heartbeat, Preempt, and ReportCompletion all carry the fencing token. The agent refuses an older token; the database accepts a transition only from the current attempt. A token stored in a row protects nothing by itself — it protects at the points where it is checked (DD3).

Phase 4 — Fetch status and logs (FR1): a row read and a URI#

GET  /v1/jobs/{id}        ->  job status service -> PostgreSQL: status, attempt, timestamps, result_uri
GET  /v1/jobs/{id}/logs   ->  log_uri -> stream from the agent (running) or object in Blob Storage (finished)

Status and logs are different access patterns. Status is a cheap row read. Logs are a stream from the agent while running and an object afterwards — keep them out of the database, and serve them by URI.

The read path never touches the scheduler. The job status service reads the same Jobs and JobRuns rows the control plane writes, so a status call is one indexed read even while the scheduler is busy, and a burst of polling clients cannot slow down placement.

Preempt and retry (FR4): always a new attempt, never a resurrected one#

A. node failure            3 missed heartbeats (30 s) -> W1 SUSPECT (no new placements, no releases)
                           reconcile: agent reachable?  r1 still in its running list?  completion manifest?
                             r1 RUNNING          -> repair the DB state; no new attempt
                             stopped / never ran -> release R1; retry if retries remain
                             unreachable         -> quarantine W1's GPUs until fenced (DD3)
                           retry = JobRun attempt 2 (r2), Reservation R2 on W2, token 8, resume from last checkpoint

B. priority preemption     critical J9 needs 8 GPUs on one node; W1 runs four LOW 2-GPU jobs
                           PREEMPTING (durable, GPUs stay reserved) -> SIGTERM + 30 s grace -> confirmed exit
                           -> release 0 -> 8 (one transaction, only now) -> reserve + assign J9 -> requeue victims

A retry is a new identity. New run_id, new reservation, higher fencing token; only job_id is shared. Reusing r1 would let a late report from the old node overwrite the state of the real attempt. The job-level fencing_epoch advances in the same transaction that creates the new attempt, so there is never a moment with two authoritative attempts.

Preemption is a protocol with one invariant: capacity is offered to a new reservation only after the previous containers have verifiably stopped. The full sequence, and what to do when a job ignores SIGTERM, is DD5.

Say what the platform does and does not promise. It guarantees graceful termination and a resume URI; it does not guarantee zero lost work. A SIGKILL after the grace period loses everything since the last checkpoint. Checkpoints belong to the workload — the platform's job is to make the loss bounded and the resume reliable.

Deep Dives#

DD1Two schedulers see four free GPUs on the same node. What prevents both from allocating them?#

BEGIN;
UPDATE node_capacity
   SET available_gpus = available_gpus - 4, version = version + 1
 WHERE node_id = 'W1' AND available_gpus >= 4;          -- row lock, condition re-checked at commit
-- 0 rows updated  ->  ROLLBACK; pick another node
INSERT INTO reservations (reservation_id, run_id, node_id, gpu_count, gpu_ids, status, lease_expires_at, fencing_token)
VALUES ('R1', 'r1', 'W1', 4, '{4,5,6,7}', 'RESERVED', now() + interval '30 s', 7);
COMMIT;

One conditional update decides. PostgreSQL locks the row for S1's UPDATE; S2's UPDATE blocks; when S1 commits, S2 re-evaluates the WHERE clause against the new value, finds 0 >= 4 false, updates zero rows, and rolls back. A version-based compare-and-swap (WHERE version = 7) is the same property in different clothes. Either way, the reservation insert and the capacity change are in one transaction or neither happens.

Redis is a hint; PostgreSQL is the allocator. Cached free-GPU counts make candidate selection cheap and are allowed to be stale — a stale read costs a retry. Used as the allocator, they cost double allocation, and an 8-GPU job landing on four live devices fails in a way that looks like a hardware bug.

Reserve identities when they matter. A count is enough when every GPU on a node is interchangeable. With mixed GPU types or device pinning, reserve gpu_ids and keep the invariant per node: reserved + available = total. Any violation is a bug — surface it as an alert, not as a quietly corrected number.

what the transaction solves     two HEALTHY schedulers cannot claim the same GPUs
what it does NOT solve          an expired RESERVED row does not free the GPUs (DD2)
                                two leaders should not both be scheduling in the first place (DD6)

DD2The scheduler crashed after the container started. How do you recover without a duplicate or a false release?#

Scheduler A:  1 reserve (txn)  ->  2 AssignJob(r1, R1, token 7)  ->  [crash]
Agent W1:     3 append to assignment log  ->  start container r1  ->  4 confirm RUNNING ... lost

Scheduler B (new leader):
  5  scan reservations in RESERVED past the confirm timeout
  6  ask: last heartbeat from W1 lists r1?   or   query the agent directly
       'r1 is RUNNING'     -> write RUNNING + CONFIRMED            (lost ACK != lost work)
       'never saw r1'      -> release R1, capacity += 4, retry      (safe: start is keyed by run_id)
       unreachable         -> W1 SUSPECT, GPUs stay quarantined     (lease expiry != stopped, DD3)

Reconcile, do not guess. The new leader's rule is ask before you release. The agent's heartbeat already carries its list of running attempts, so in the common case reconciliation is a read of the registry, not a round-trip to the node. Only the "unreachable" outcome is genuinely uncertain, and it resolves into quarantine rather than into a release.

Accept, start, and confirm are three different facts. The agent appends the assignment to a local durable log before starting, so an agent restart replays the assignment instead of forgetting it. Starting is idempotent per run_id: a replayed or duplicated AssignJob(r1) answers "already running". Confirming is the step that was lost — and because the other two facts are durable, it can be reconstructed.

Why "expire and reschedule" is the wrong default. If the new leader releases R1 at lease expiry and places J1 on W2, W1 is still training J1: two containers, double GPU spend, and two attempts racing to publish. Expiry is the trigger for reconciliation, never the conclusion.

DD3A node is silent for 30 seconds but still training. Do you reschedule?#

T+30 s   failure detector: W1 SUSPECT      -> no new placements on W1, no releases of W1's GPUs
         scheduler (txn): job epoch 1 -> 2, JobRun r2, Reservation R2 on W2, token 8
         AssignJob(r2, R2, token 8) -> W2

commit rule (control plane):  accept a transition only if token == current token AND run not terminal
         W2: heartbeat / complete with token 8  -> accepted
         W1: heartbeat / complete with token 7  -> rejected, even if it arrives first
         W1's GPUs: reserved 4 / available 4 until the agent reports or an operator fences the node

Fence the attempt; quarantine the GPUs. Advancing the job's fencing epoch in the same transaction that creates the new attempt revokes the old attempt's authority everywhere the token is checked: confirm, heartbeat, preempt, complete. W1 can keep computing and can even write temporary files under its attempt path, but it cannot change the job's state or publish a result. Meanwhile W1's capacity row is untouched — releasing those four GPUs would make DD1's invariant lie.

Object storage does not check tokens. Fencing protects controlled writes. A bucket will happily accept W1's files, which is why publication goes through a commit service that verifies the token and why artifacts live under attempt-scoped paths so r1 and r2 cannot overwrite each other (DD8).

Name the guarantee you are giving. Fencing provides a single accepted result. It does not provide a single active execution. If the workload is not safe to run twice — external side effects, a licensed resource — then before starting r2 you must confirm W1 stopped or fence it at the infrastructure level: power it off, isolate it on the network, or rely on an agent that kills any container whose epoch is stale the moment it reconnects. That costs availability; say so.

lease expired    ->  the control plane lost contact          (paused VM, GC pause, partition all look identical)
                     treat as "reconcile now", never as "free now"

DD4Who gets the next free GPUs: Team A's critical jobs over quota, or Team B's normal jobs under quota?#

guaranteed shares first:  B is under its guarantee  ->  the next free GPUs go to B's NORMAL jobs
                          A is at its guarantee     ->  A's CRITICAL jobs use BORROWED idle capacity only,
                                                        reclaimable when the owner needs it back
hierarchy:  1 guaranteed shares  ->  2 DRF across teams  ->  3 priority + aging within a team
            ->  4 best-fit placement  ->  5 atomic reserve

Priority lives inside a share; fairness lives between shares. A critical label says "schedule this before my team's other work". It does not say "take another team's guaranteed GPUs". Unused guaranteed capacity should be lent out — leaving 1,500 GPUs idle to protect a quota is the wrong kind of fairness — but a loan is reclaimable, by natural completion or by preemption, when the owner returns.

Pick the team with Dominant Resource Fairness, then the job by priority. DRF compares each team's dominant share (GPUs for most jobs, memory for some) against its allocation and serves the most under-served team next. Within that team, the highest-priority job goes first. Weighted random selection is not a fairness guarantee; weighted fair queuing or deficit round robin is.

Starvation needs three fixes, and preemption is not one of them. Reserve a critical share sized from its latency SLO, guarantee minimum shares for normal and low so they always make progress, and add aging: scheduling credits that grow with wait time, within each tier's rules. Charge in GPU-hours, not job count, so an 8-GPU job consumes eight times the credit of a 1-GPU job for the same slot. Then tune from wait-time p95/p99 per tier — never from a fixed 70/20/10 split. And never preempt a running critical job to run a low one: policy only allows higher to preempt strictly lower.

quota       how much is each team GUARANTEED?
fairness    who gets the NEXT free GPUs?           (DRF / weighted fair queuing)
priority    which of MY jobs runs first?           (+ aging)
preemption  may RUNNING work be displaced?         (DD5 — a separate policy)

DD5A critical job needs all 8 GPUs of a node running four 2-GPU jobs. Walk through the preemption.#

1  select victims       four LOWER-priority, preemptible jobs on W1 freeing exactly 8 GPUs
                        (weigh checkpoint age, progress, team quota — the cheapest set, not the first four)
2  PREEMPTING           JobRun RUNNING -> PREEMPTING, durable; reservations stay CONFIRMED; capacity untouched
3  SIGTERM + grace      agent signals each container; the workloads checkpoint to object storage (30 s)
4  confirm exit         agent verifies the container exited AND its GPU processes are gone
                        job 4 ignores SIGTERM  ->  SIGKILL via the container runtime  ->  verify again
5  release              one transaction: reservations RELEASED, JobRuns PREEMPTED, available 0 -> 8   (only now)
6  reserve + assign J9  new reservation, new token, AssignJob, agent confirms -> RUNNING
7  requeue victims      PREEMPTED -> PENDING as new attempts, resuming from the last durable checkpoint

The one invariant: release after stop. Capacity becomes available to a new reservation only when the previous containers have verifiably stopped. Releasing at step 2 — "they will stop in thirty seconds" — is the double-allocation bug in a different costume. Confirmation is physical: the cgroup is empty and no process holds the devices, not merely "the container runtime returned".

PREEMPTING is a real state, not a comment. It tells the failure detector that a missed heartbeat from these runs is expected, tells the status API the job did not fail, and tells a scheduler that fails over mid-preemption (DD6) which reservations are in the middle of a release.

When termination cannot be confirmed, do not place. If SIGKILL does not produce a verified exit — a wedged driver, an unreachable agent — W1 is quarantined and J9 goes elsewhere. The critical job waits a little longer; the alternative is sharing GPUs with a zombie.

Count the cost. Four jobs lose up to one checkpoint interval each and requeue behind other work; job 4, killed without a final checkpoint, loses more. The critical job waits the grace period — which is why the 5-second start SLO is defined when capacity is immediately available and excludes preemption waits.

DD6Three scheduler replicas, the leader crashes. Later, an old leader keeps running after losing its lease.#

election     PostgreSQL advisory lock or etcd lease; the coordinator issues a monotonic leader_epoch
             S1 = 101  ->  lease expires  ->  S2 acquires, epoch 102

every scheduling write carries the epoch and is validated INSIDE the transaction:
  INSERT INTO reservations (..., leader_epoch = :e) WHERE :e = (SELECT epoch FROM leader FOR SHARE);
  S1 (101): 0 rows -> rejected          S2 (102): commits
node agents also reject AssignJob with an epoch below the highest they have seen

takeover    1 acquire lease + epoch      2 rebuild Redis queues from PENDING rows
            3 scan RESERVED / PREEMPTING reservations      4 reconcile with agent heartbeats (DD2)
            5 retry incomplete assignments idempotently     (expire reservations only after reconciliation)

A single leader, fenced by an epoch at the commit point. Election gives you one leader most of the time; the epoch check inside the transaction is what makes the exception harmless. S1 may be paused, partitioned, or just slow, and may keep executing code written before it noticed — the only thing that stops its writes is a check that happens inside the write, against authoritative leadership state. A cached "am I leader?" boolean read before the transaction still permits the race.

Agents validate too, but the database is the backstop. Node agents reject assignments with an older epoch, which stops most stale commands early. But an agent that has not yet heard of epoch 102 could accept a command from 101; that is why the reservation itself must fail in the database, and why the agent's epoch knowledge is a fast path, not the guarantee.

Recovery is reconciliation, not replay. Pending jobs are not lost because they live in PostgreSQL; the Redis queues are rebuilt from PENDING rows. In-flight reservations are not blindly reassigned; each is reconciled with the agent's reported running set (DD2). An outbox owned by one scheduler is the wrong shape — shared durable state is what the next leader reads.

Two leases, two purposes. The leadership lease says who may make scheduling decisions. The reservation lease says which attempt holds which GPUs. Different lifetimes, different owners, different recovery paths; reusing one as the other conflates a scheduling failure with an execution failure.

DD765% utilization with 3,000 jobs queued. Prove why before touching the scheduler.#

Node A  5 used / 3 free
Node B  5 used / 3 free        free total = 8      8-GPU jobs that fit = 0      utilization "looks" 67%
Node C  6 used / 2 free

best fit:  place a small job on the node with the LEAST free GPUs that still fits   ->   8 / 8 / 0 used
           same 16 GPUs in use, one whole node free, the 8-GPU job starts

Separate the causes before changing anything:

Prevention: best fit, not spreading. Placing each small job on the node with the least free GPUs that still fits keeps whole nodes free for large jobs. Spreading — the emptiest node first — feels balanced and is exactly how a cluster ends up with a thousand nodes that each have two free GPUs. Add demand-driven headroom for large jobs when the queue shows them coming, rather than twenty nodes kept permanently idle (160 GPUs of utilization paid every hour).

Recovery: defragment with controlled preemption. With 100 nodes — 80 running 4-GPU jobs, 20 full, 50 critical 8-GPU jobs waiting — 320 GPUs are free and nothing can start. Freeing a whole node means preempting or migrating the small jobs on it (DD5): choose the minimum-cost set by priority, checkpoint age, progress, and team quota; consolidate first where two half-nodes can become one full node; and quantify the displacement — fifty half-full nodes freed means 200 GPUs of running work requeued.

Prefer tail metrics. Average queue wait hides the eight-GPU jobs that have waited a day behind thousands of one-GPU jobs that started instantly. Track queue age p95/p99 per job size and per tier, and the free-GPU histogram, before anything else.

DD8A job uploaded its artifacts and the node died before reporting SUCCEEDED. Do you run it again?#

1  upload artifacts        jobs/J1/attempts/r1/part-000.bin, part-001.bin, ...
2  write MANIFEST last     jobs/J1/attempts/r1/MANIFEST.json  { uris, sha256, token 7, completed_at }
3  ReportCompletion        (r1, token 7, manifest_uri)        <- lost: the node crashed here
4  commit once             Completion Service verifies every URI exists and every checksum matches,
                           checks token 7 == current, then ONE transaction:
                           r1 SUCCEEDED, J1 SUCCEEDED, R1 RELEASED, capacity += 4, result_uri = manifest

reconciler (after lease expiry on W1):
  already committed?            -> nothing to do; a late duplicate report is ignored
  valid MANIFEST for r1?        -> finalize r1 as SUCCEEDED through the same commit path — no re-execution
  parts but no manifest?        -> retry as attempt r2 (token 8); r1's partial output is garbage-collected

Attempt-scoped artifacts plus an idempotent completion protocol. The manifest is written last, after every part has landed, and carries checksums — so its existence is the proof of completeness. The protocol makes the proof; a directory listing does not. Attempt-scoped paths keep r1 and r2 from overwriting each other when a DD3 overlap happens, and the job-level result_uri points only at the accepted manifest.

Commit is conditional and fenced. The completion service accepts a report only from the currently authorized attempt and only if the job is not already terminal. If r2 is already running under token 8 when r1's late manifest appears, the commit rejects it — or, if the policy prefers, adopts r1's verified result and cancels r2. Either is defensible; choose explicitly.

Say what is exactly-once. Accepted completion is exactly-once: one terminal state per job, written by one conditional transaction. Execution is not, and cannot be promised for arbitrary user code. Attempts are at-least-once; results are accepted once.

DD9The 5-second start SLO is missed because the image pull takes 8 seconds. What changes?#

measured       100 + 500 + 200 + 100 + 8,000 + 2,000   =  10,900 ms      SLO: 5,000 ms
redesigned     100 +  50 + 200 + 100 +   300 + 1,500   ≈   2,250 ms

1  image-aware placement        prefer nodes that already hold the image layers (a term in the best-fit score)
2  node-local cache + pre-pull  top-N images per GPU type pre-pulled on every node; a per-rack registry mirror
3  warm pool for critical       a few pre-initialized sandboxes per node type; idle capacity bought for latency
4  event-driven scheduling      wake on enqueue and on release instead of a 500 ms poll
5  lazy / streaming layers      start before the full image is local; keep base images small

Fix the two big terms. Image pull and container initialization are ten of the eleven seconds. Image-aware placement and a node-local cache turn the pull into a few hundred milliseconds for the common images; a warm pool for the critical tier removes most of the initialization. The 500 ms queue wait is a polling interval and disappears when the scheduler is woken by events.

Warm capacity is a utilization trade. Every prewarmed slot is a GPU not running work. Size it per tier from measured arrival rates, reclaim it under pressure, and treat it with the same headroom reasoning as DD7 — applied to latency instead of to large-job placement.

Say what the SLO excludes. "Five seconds when capacity is immediately available" does not cover waiting for a preemption grace period or a cold image on a brand-new node type. Tag start-latency measurements with those conditions so the p95 you report is the one the SLO describes.

Common Pitfalls#

Each of these reveals a different misunderstanding of the problem. Avoiding them is not about memorizing the "correct architecture" — it is about showing where distributed-system guarantees stop and explicit protocols must begin.

Final Thoughts#

The shape of a strong answer: clear the contradictions in the prompt, separate the logical job from its attempts, size the cluster and notice that scheduling throughput is not the constraint, then start from the simplest control plane and let each failure force exactly one mechanism into the design — a conditional transaction for capacity, a node agent that starts once and confirms separately, reconciliation before release, fencing tokens on every commit point, a hierarchical scheduler that keeps quota, fairness, priority, and preemption apart, a release-after-stop preemption protocol, a leader epoch validated inside the transaction, best-fit placement, and a manifest-based completion. The FR→flow mapping keeps the HLD legible; the nine deep dives are the same nine questions you raised at the start, now answered.

Common failure modes to avoid: letting a counter allocate, treating expiry as death, reusing identities across attempts, releasing before stop, and collapsing four scheduling policies into "critical wins".

These aren't toy problems — they reflect real infrastructure challenges at companies running shared GPU fleets for many teams. If you want to walk through this design in a mock interview or get feedback on your own approach, book a session with us. We're here to help you sharpen your thinking, build your confidence, and land your next offer.