You hit enter. Roughly 300 milliseconds later the first word appears, and then tokens arrive at about 30 per second. Between those two moments sits one of the more interesting systems running in production today. **Scope contract.** This page goes from the moment you hit enter down to the GPU's memory system. It does **not** go down to CUDA kernels. If you want the system-design layer above this — capacity planning, multi-region, 10k RPS — that's the [Batch Inference System](/learn/system-design/questions/batch-inference-system) question. If you want kernel-level work, that's a different document and a different job description. **Who this is for.** Engineers preparing for senior/staff interviews at AI labs, where "design an LLM API" is now a standard question and the deep-dive round goes exactly one level below the API. That level is what's written down here. ## Why This Matters Most engineers can draw the boxes: client → gateway → router → GPU fleet. That gets you through the first fifteen minutes of an interview. What separates candidates is what happens when the interviewer asks the follow-up: - *Why is your time-to-first-token high while GPU utilization looks low?* - *What do you autoscale on?* - *What breaks first as context lengths grow?* Every one of those questions is answered by the same small set of facts about how a transformer actually executes at inference time. There are maybe five of them. This page is those five facts, plus the vocabulary to state them cleanly under pressure. One honesty note, because it matters for how you use this: understanding these mechanics is not the same as having built vLLM. Interviewers know the difference, and they reward first-principles reasoning over recited internals. The goal here is that when someone pushes one level below your architecture diagram, you have somewhere to stand. ## One Request, End to End The single most useful thing to internalize is that **your data changes shape four times** on the way through: ``` text → token IDs → K,V tensors → text deltas → SSE frames ``` Each arrow is a component boundary, and almost every production problem lives at one of them. ### The Ten Steps ### The Punchline > **Steps 1–7 happen once per request. Steps 8–10 — the green path — run once per output token.** A 500-token response runs that green loop 500 times. Everything about inference performance follows from the fact that one part of the system runs once and the other part runs hundreds of times, and that these two parts are bottlenecked on completely different hardware resources. That's the physics section below. First, some concrete numbers. ## Worked Example: "I got an offer from OpenAI" ### Words Are Not Tokens Six words. Seven tokens: Two things worth saying out loud, because they surprise people and they have real consequences: **Leading spaces belong to the token.** `␣got` and `got` are different token IDs. This is why prompt formatting — a trailing space in a template, say — can measurably change model behavior. **Rare and compound words fragment.** BPE builds a fixed vocabulary (~100k–200k entries) from frequent byte sequences. "OpenAI" isn't frequent enough to earn its own entry, so it becomes `␣Open` + `AI`. The practical consequence: **you cannot bill, budget, or capacity-plan in words.** Non-English text and code fragment much harder — a rule of thumb of ~1.3 tokens/word for English prose can go to 2–3× for languages that don't use Latin script. ### The KV Grid Now the part that matters for systems. Picture a grid: - **Columns = tokens.** Seven of them here. - **Rows = layers.** For a 70B-class model, 80 of them. Each cell holds the K and V tensors for that (layer, token) pair. Prefill fills the entire grid in one pass. Every subsequent decode step appends exactly **one new column**, all 80 rows. For our running example model — 80 layers, 8 KV heads under GQA, head dimension 128, fp16: ``` per layer, per token: 2 × 8 × 128 × 2 bytes = 4 KB per token (80 layers): 4 KB × 80 = 320 KB this sentence: 320 KB × 7 tokens = 2.24 MB ``` **2.24 MB of GPU memory for one seven-token sentence.** Hold that number; the physics and continuous batching sections are both consequences of it. > Being able to produce this arithmetic live is a real differentiator in an interview. It's the difference between a candidate who read a blog post and one who can do napkin math on the whiteboard. The full formula, with more worked cases, is in Appendix A below. ## Prefill, Decode, and the KV Cache — The Physics ### Q, K, V at the Intuition Level You don't need the attention equation to reason about inference systems. You need this: - **Q (query)** — the question the current token is asking. *"I'm the word 'offer' — what context should I be paying attention to?"* - **K (key)** — each past token's label, advertising what it can answer. - **V (value)** — each past token's payload, the content that actually gets mixed in. Every token compares its Q against every earlier token's K, and pulls a weighted blend of their V. Now the one consequence that justifies the whole section: **Past tokens never ask questions again.** Token 3's Q was used once, when token 3 was generated, and is never needed again. But its K and V are consulted by every single future token. So K and V get cached; Q does not. **That is why it is called a KV cache and not a QKV cache.** If you say only one thing about attention in an interview, say that. ### Two Phases, Two Different Bottlenecks This is the central fact of LLM inference. **Prefill** processes all *N* prompt tokens in one forward pass. The GPU loads each weight matrix once and multiplies it against all *N* tokens. Lots of arithmetic per byte of weights loaded. **Prefill is compute-bound.** It saturates the tensor cores. **Decode** processes exactly one token. It still has to load *every single weight in the model* to do it — all 140 GB of them, for a 70B model in fp16 — and then does a trivial amount of arithmetic with them. **Decode is memory-bandwidth-bound.** The tensor cores sit mostly idle while HBM streams weights. ### Why This Matters **1. Latency is two numbers, not one.** - **TTFT** (time to first token) is dominated by queueing + prefill. It scales with *prompt* length. - **ITL** (inter-token latency) is dominated by decode. It's roughly constant per step and scales with *model size*, not prompt size. A system can have excellent TTFT and terrible ITL, or the reverse. Reporting "p99 latency" for an LLM endpoint is a tell that you haven't thought about it. **2. GPU utilization is a misleading metric.** A fleet deep in decode can show high "utilization" while doing almost no useful arithmetic — it's busy waiting on memory. This is why "utilization looks fine but everything is slow" is such a common production symptom, and why the Scaling section says never to autoscale on it. **3. Batching decode is nearly free.** The GPU reads all 140 GB of weights whether it's serving 1 sequence or 64. Serve 64 and you get 64 tokens for the same memory traffic. This single fact is the entire economic argument for continuous batching. **4. KV memory is the real capacity constraint.** Weights are a fixed cost. KV cache is a per-sequence, per-token cost that grows for the whole life of a request. It's what actually runs out. Two mitigations are worth knowing by name: - **GQA** (grouped-query attention): multiple query heads share one KV head. Our example model has 64 query heads but only 8 KV heads — an 8× reduction in KV cache size, from 2.6 MB/token to 320 KB/token. This is why modern models use it. - **PagedAttention**: allocate KV in fixed-size blocks like OS virtual memory pages instead of one contiguous reservation per sequence. Kills internal fragmentation from over-provisioning for `max_tokens`, and makes copy-on-write sharing of common prefixes possible. ## Continuous Batching If your interviewer only probes one thing below the API layer, it will probably be this. ### Static Batching, and Why It Fails The obvious design: collect *N* requests, run them together, return all *N*, repeat. Two failure modes, both severe: **Stragglers.** Requests in a batch finish at wildly different times — one generates 20 tokens, another generates 2,000. With static batching, the 20-token request's slot is held hostage until the 2,000-token request finishes. Ninety-nine sequences idle while one keeps going. **Padding waste.** Prompts have different lengths, so short ones get padded to the longest. You spend compute and memory on padding tokens. In practice static batching leaves most of your fleet's throughput on the floor. ### Iteration-Level Scheduling The fix: **stop treating the batch as a fixed group and treat it as a rolling roster.** The scheduler runs between every decode iteration — roughly every 30 ms: 1. Run one decode step for every sequence currently in the batch. 2. **Evict** any sequence that just emitted EOS or hit `max_tokens`. Free its KV blocks immediately. 3. **Admit** waiting requests into the freed slots, if there's enough free KV memory to hold them. 4. Repeat. A finished sequence's slot is reclaimed within ~30 ms instead of within seconds. In the diagram: user C hits EOS, is evicted mid-flight, and user D is admitted into the same iteration. Nobody waits for the 2,000-token straggler. **Admission is a KV memory decision.** The scheduler isn't asking "is there compute headroom?" — decode has compute headroom almost by definition (see the physics section). It's asking "do I have enough free KV blocks for this sequence's prompt, plus room to grow as it generates?" That's the gate. If a running batch grows past available KV, the scheduler must **preempt** — evict a sequence and either swap its KV to host memory or drop it and recompute later. ### The One-Liner Say this verbatim in an interview: **"Decode reads all the weights whether it's serving 1 sequence or 64, so batching decode is nearly free throughput. That's why continuous batching exists."** It's correct, it's compact, and it demonstrates you understand the *why* rather than the *what*. ### Chunked Prefill, in One Breath There's a tension: prefill is a big compute-heavy pass, and running one blocks the decode steps of everyone already in the batch. So a new arrival with a long prompt causes an ITL hiccup for every active user. **Chunked prefill** splits a long prefill into pieces and interleaves them with decode steps, trading a slightly worse TTFT for the new request against protected ITL for existing ones. Mention it as the TTFT-vs-ITL interference fix and move on unless asked. ## Scaling the Whole System Now back up to the architecture and go layer by layer. Each tier has a scaling signal, and each one is different from what you'd use for a normal web service. ### Gateway — Scale on Open Connections, Not QPS Streaming responses hold connections open for the entire generation: seconds to minutes, not milliseconds. A gateway's binding constraint is **concurrent open connections**, not requests per second. A box that handles 10k RPS of short REST calls might handle 2k concurrent streams. Size on connections and memory per connection. ### Router — Needs a Live Load View, Plus Prefix Affinity Round-robin is actively wrong here. Requests have wildly different costs (a 200-token prompt vs. a 100k-token one), and engine replicas have wildly different states. The router needs **live per-replica load**: current batch size, queue depth, and free KV blocks. Then the optimization that surprises people: **prefix affinity.** Requests sharing a long prefix — the same system prompt, the same few-shot block, the same document — should be routed to the same replica, because that replica may already have the prefix's KV cached and can skip prefilling it. Getting this right can cut TTFT dramatically on workloads with heavy prompt reuse. It's a genuinely strong thing to raise unprompted in an interview. ### Engine Fleet — Autoscale on Queue Depth and KV Pressure. Never on CPU. CPU is meaningless here; the work is on the GPU. GPU utilization is nearly as bad, for the reason covered in the physics section — a memory-bandwidth-bound decode loop looks "busy" while doing very little. The signals that actually correlate with user-visible pain: ### Scale-Up Is Slow — So Carry Headroom Adding an engine replica is not adding a stateless pod. You have to schedule a multi-GPU node, pull tens of gigabytes of weights, load and shard them across devices, and warm up. **Minutes, not seconds.** Traffic spikes arrive faster than that. Two consequences: 1. **Carry real headroom.** Target something like 60–70% steady-state KV utilization, not 95%. The buffer is what absorbs a spike while new capacity boots. 2. **Run a preemptible batch tier.** Fill the slack with latency-insensitive work — evals, offline generation, backfills — that the scheduler can evict instantly when interactive traffic arrives. You get high average utilization *and* burst headroom, which otherwise trade off against each other. > This is where this page hands off to the [Batch Inference System](/learn/system-design/questions/batch-inference-system) question, which designs this exact system end to end at scale. ## How This Shows Up in Your Interview ### The Four Follow-Ups This Page Is Built to Answer **"Your TTFT is high but GPU utilization looks low. What's happening?"** Almost certainly queueing, not compute. Requests are waiting for admission because there aren't free KV blocks, and queue time doesn't appear in GPU metrics. Check queue depth and KV utilization before you touch anything else. Secondary suspect: long prefills from other users blocking the batch — which is what chunked prefill addresses. **"What metric do you autoscale on?"** Queue wait time and KV cache utilization. Not CPU, not GPU utilization — a decode-heavy fleet is memory-bandwidth-bound and looks busy while doing almost no arithmetic. And because scale-up takes minutes, you scale on leading indicators and carry headroom rather than reacting at the threshold. **"What breaks first as context lengths grow?"** KV cache memory. It's linear in tokens and lives for the whole request — 320 KB/token means a single 100k-token context is ~32 GB. Longer contexts mean fewer concurrent sequences, smaller batches, worse throughput per GPU, and eventually preemption. Prefill compute is quadratic in sequence length too, so TTFT degrades in parallel — but memory is what falls over first. **"Why not just use bigger static batches?"** Because of stragglers. One 2,000-token generation holds every other slot in the batch hostage until it finishes. Bigger static batches make that worse, not better. Iteration-level scheduling reclaims each slot within ~30 ms of that sequence finishing, which is what actually gets you throughput. ### The Three Mistakes That Cost Candidates Points ## Appendix A: Napkin Math ### The KV Cache Formula ``` bytes per token = 2 × L × H_kv × d_head × precision_bytes ↑ ↑ ↑ ↑ ↑ │ │ │ │ └── 2 for fp16/bf16, 1 for fp8 │ │ │ └───────────── head dimension │ │ └────────────────────── number of KV heads (NOT query heads — GQA) │ └──────────────────────────── number of layers └──────────────────────────────── one for K, one for V ``` ### Worked: 70B-Class Model, fp16 ``` per token = 2 × 80 × 8 × 128 × 2 = 327,680 bytes ≈ 320 KB ``` **Without GQA** (64 KV heads instead of 8): 2.6 MB per token. An 8× difference — and the reason a 100k context would be 256 GB instead of 32 GB. ### Worked: How Many Concurrent Requests Fit? On a 4×H100 node (320 GB total HBM): ``` total HBM 320 GB − model weights −140 GB − activations, workspace −20 GB (rough) = KV budget 160 GB 160 GB ÷ 320 KB/token ≈ 500,000 tokens of KV 500,000 ÷ 8,000 ≈ 60 concurrent sequences at 8k context ``` Then apply the headroom rule from the Scaling section: target ~60–70% of that, so plan for roughly **40 concurrent 8k-context sequences per node**, not 60. ### Worked: The Decode Speed Ceiling Decode must read every weight once per step, so: ``` step time ≥ model bytes ÷ aggregate HBM bandwidth 4×H100 ≈ 13.4 TB/s aggregate 140 GB ÷ 13.4 TB/s ≈ 10.4 ms → ~96 tokens/s ceiling ``` Real-world lands well below that — kernel launch overhead, inter-GPU communication, attention over the KV cache itself, and imperfect bandwidth utilization typically leave you in the 30–50 tokens/s range. Which is what users see. ### Worked: When Does Decode Stop Being Memory-Bound? An H100 does roughly 1,000 TFLOPS bf16 against roughly 3.35 TB/s of bandwidth — an arithmetic intensity of ~300 FLOPs per byte to saturate compute. Decode does roughly *B* FLOPs per weight byte at batch size *B*. So decode stays memory-bound until batch sizes in the hundreds — which is why batching decode is nearly free, and why you will run out of KV memory long before you run out of compute. > All figures are illustrative and rounded. Substitute your own model's config; the point is the shape of the reasoning, not these specific numbers. ## Appendix B: Interview Q&A with Model Answers Gateway does auth and validation and holds the connection open. The router picks an engine replica using live load and, ideally, prefix affinity. At the engine the request queues for admission. Tokenizer converts text to token IDs. The scheduler admits it once KV blocks are available. Prefill runs one forward pass over all prompt tokens, fills the KV cache, and emits the first token. That token gets detokenized and streamed as an SSE frame. Everything up to and including prefill happens once; from there it's a per-token loop. Prefill loads each weight once and multiplies it against all *N* prompt tokens — high arithmetic intensity, saturates the tensor cores. Decode loads the identical 140 GB of weights to produce a single token. Almost no arithmetic per byte loaded, so HBM bandwidth is the ceiling and the compute units idle. K and V are each token's advertised label and payload; every future token attends against them, so they're reused indefinitely and worth caching. Q is the question the *current* token asks — used once, at generation time, never needed again. Caching it would be dead weight. Weights are fixed: parameters × precision bytes. Everything left over is the KV budget. KV per token is `2 × layers × KV_heads × head_dim × precision`. Divide the budget by (KV per token × target context length) for max concurrency, then take 60–70% of that as your plan number, because scale-up takes minutes and you need spike headroom. Static batching fixes membership for the batch's lifetime, so one long generation holds every other slot hostage and short prompts get padded. Continuous batching reschedules between every decode iteration — roughly every 30 ms. Finished sequences are evicted and their KV freed immediately; waiting requests are admitted into the gap. The batch is a rolling roster, not a fixed group. Admission is gated on free KV memory, not compute. First, refuse the single number: split into TTFT and ITL, because they have different causes. TTFT regression points at queueing (check queue depth and KV utilization), longer prompts, or prefill interference from other users' long prompts. ITL regression points at larger batches, KV pressure and preemption, or a change in model/parallelism config. Confirm which one moved before touching capacity. KV memory is linear in context length and held for the whole request, so doubling context roughly halves how many sequences fit. Fewer concurrent sequences means smaller decode batches, and since decode reads all weights regardless of batch size, small batches mean much worse tokens-per-second per GPU. Long contexts hurt throughput through the *memory* channel, not primarily the compute one. It helps enormously when many requests share a long prefix — a fixed system prompt, a shared few-shot block, a document being asked many questions about — because the shared prefill can be skipped entirely. It requires the router to send those requests to the replica holding the cache (prefix affinity), and it does nothing for workloads with high prompt diversity or where the varying content sits early in the prompt. The shared portion must be a genuine prefix; a shared *suffix* buys you nothing. Run batch work as a preemptible tier that the scheduler evicts the moment interactive traffic needs the slots. Interactive gets admission priority and a KV reservation; batch fills the slack. This is what lets you carry the headroom the Scaling section requires without paying for idle GPUs, since scale-up is too slow to react to spikes. ## Appendix C: Glossary > Ready to go one level up? The [Batch Inference System](/learn/system-design/questions/batch-inference-system) question takes everything on this page and turns it into the end-to-end design interview: capacity planning, multi-region, and serving at 10k RPS.