Worked Example: Reviewing an LLM Inference Doc

This is the doc Anthropic candidates report most often in the review round. The abridged version below keeps every number and every step that matters; the full text, with the Part 2 placement puzzle, is on the practice page.

The doc, abridged#

Background. The chat product serves one model behind a static load balancer. The inference engine is a black box. Prompts and responses are about the same size, so batch processing time is assumed constant regardless of batch size. Clients send synchronous HTTP requests and wait.

• Peak QPS 10k, average 1k. About 500 tokens per request.
• A worker processes one batch at a time; a batch holds up to 10k input tokens; a batch takes 1.5 seconds for either model.

Goals. Serve 10k QPS at peak. Keep low latency.

Components. A stateless API gateway (auth, rate limit, forward to the dispatcher for the requested model). One active dispatcher per model with a hot standby, holding an in-memory queue, forming batches, sending them to workers. GPU hosts with 10 units each; a small-model worker uses 2 units, a big-model worker 5; each worker exposes POST /batch and runs one batch at a time.

Request flow. Gateway authenticates. Gateway forwards to "the queue". Dispatcher takes from the queue, closes a batch, forwards it to "the queue in GPU workers". That queue sends the batch to the next worker in round-robin order. Responses return to the gateway and then to clients.

Part 2. Three hosts of 10 units. Small worker needs 2 adjacent units on one host, big needs 5. Place the workers.

Pass 1: Restate#

Two chat models behind a gateway. Synchronous clients. A per-model dispatcher batches requests by input tokens and hands batches to GPU workers that each process one batch at a time in about 1.5 seconds. The goals are 10k QPS at peak and low latency.

Explicit assumptions worth writing down: batch time is constant regardless of batch size; prompts and responses are about the same size; the engine is a black box. The first two are the ones to challenge; the third is a scope boundary to respect.

Already visible from restating: "low latency" next to a 1.5 second processing floor. Whatever "low" means, it cannot mean sub-second. Worth a clarifying question.

Pass 2: Numbers#

The doc gives four numbers that combine:

The conclusion writes itself: the stated hardware is about 50 times too small for the stated peak goal, and 5 times too small for the average. Either the budget in Part 2 is a toy, or the goal is wrong, or the author never divided. This is the headline and the first clarifying question.

The latency floor: a request waits for its batch to fill, waits for a free worker, then spends 1.5 seconds in the worker. Under load, batches fill in milliseconds (at 1k QPS a 20-request batch fills in 20 ms), so the floor is about 1.5 seconds plus queueing. At low traffic for one model, a 20-request batch could take seconds to fill, so the dispatcher needs a maximum-wait rule. The doc never says when a batch closes.

Overload: if 1k QPS arrives and 200 QPS can be served, the queue grows by 800 requests per second. The doc has nothing that stops it.

Pass 3: Trace#

One request, every hop.

• Gateway. Authenticates, rate limits. Rate limiting on a "stateless" gateway needs a shared counter somewhere; minor, note it.
• Gateway to dispatcher. Step 2 says "forwards to the queue". The overview says "forwards to the dispatcher for the requested model". Probably the same thing described twice, but ask: is the queue inside the dispatcher, or a separate component? Whoever owns it owns the failure mode.
• Dispatcher. Holds the request in memory. Closes a batch by an unstated rule. One active instance per model; the standby has no state.
• Dispatcher to worker. Step 4 says "forwards to the queue in GPU workers" and step 5 has "the queue" send batches round-robin. But workers expose a synchronous endpoint and process one batch at a time. There is no worker queue in the overview. Either there is a second queue the overview forgot, or round-robin pushes batches directly to workers, busy or not.
• Worker. Processes the batch in 1.5 seconds. Returns 20 responses.
• The way back. "Responses are returned to the gateway and then to clients." How? Twenty responses must each find the one client connection waiting for them, on the specific gateway instance holding that socket. The doc has no request ids, no mapping, no mechanism.

Findings from the trace: two undefined queues, a dispatch policy that ignores worker state, a single active dispatcher holding the only copy of the queue, and an unspecified response path.

Pass 4: Break#

Under load. The dispatcher's in-memory queue saturates first, or rather it does not saturate: it grows until the process runs out of memory, taking every queued request with it. Nothing sheds load. Per-key rate limits do not help when the aggregate exceeds capacity.

Under failure. The active dispatcher dies. Every queued request is gone. Every in-flight batch's mapping back to client connections is gone, so even responses that come back from workers cannot be delivered. The standby starts empty. The "hot standby" protects availability of the dispatcher process and nothing the user cares about. Whether this is acceptable depends on whether clients retry and whether requests are idempotent; the doc does not say.

A second failure: round-robin hands a batch to a worker that is still 1.4 seconds into its previous batch. That batch waits. The worker after it in rotation sits idle. Under load every worker accumulates a private backlog and tail latency becomes unpredictable.

Pass 5: Rank and recommend#

What the doc gets right: per-model dispatchers are the correct decomposition, batching by tokens is the correct unit, and keeping the engine as a black box keeps the scope honest. Say this before the first concern.

The opening, as it would be said#

"Let me restate it. Two chat models behind a gateway, synchronous clients, a per-model dispatcher that batches by input tokens and hands batches to GPU workers that each take about 1.5 seconds per batch. Goals are 10k QPS at peak and low latency.

The numbers first. A batch is 10k tokens at 500 per request, so about 20 requests. One worker does 20 per 1.5 seconds, about 13 QPS. Peak needs about 750 workers; average needs 75. Part 2 gives 30 GPU units, at most 15 small workers, about 200 QPS. So goal one is off by a factor of fifty with this hardware. Is the hardware fixed or is the target fixed? That decides whether this is a fleet-sizing conversation or a scoping one.

Setting that aside and assuming the fleet gets sized: the per-model dispatcher split is right and batching by tokens is right. The three things I would change before shipping are the round-robin push to busy workers, which should be pull-based; the single active dispatcher holding the only copy of the queue, where I want an explicit decision about what we lose on failover; and the response path, which the doc does not describe. After those I have smaller notes on batch closing, backpressure and request size. Where would you like to start?"

The extension: placing workers on three hosts#

Each host holds one of: 5 small workers (10 units), 2 big (10 units), or 1 big plus 2 small (9 units, 1 wasted). Per-worker throughput is the same for both models, so worker counts should follow the traffic split, which the doc does not give. Ask; if no answer, state an assumption.

For a 50/50 split, "2 big / 2 big / 5 small" uses every unit and gives 4 big and 5 small workers, but puts every small worker on one host: lose that host and the small model is down, and the dispatcher's hot standby does nothing about it. "1 big plus 2 small on all three hosts" wastes 3 units but survives any single host loss. Place the big workers first on each host, because two small workers placed carelessly can fragment a host so that no contiguous 5-unit block remains.

Whichever layout: total capacity is about 100 to 120 QPS. Say it again. Placement does not fix provisioning.

What to take from this#

• The headline came from pass 2 in under three minutes, from numbers the doc handed over.
• The next three findings came from pass 3, from walking one request through boxes the doc described loosely.
• Pass 5 is where the review becomes useful: nine findings became three to fix now, one to clarify, and a list to mention.
• The extension was answered with the same discipline: constraints, a missing number to ask for, an assumption stated out loud, a recommendation with its failure-mode reasoning.