How LLM Inference Actually Works

You hit enter. Roughly 300 milliseconds later the first word appears, and then tokens arrive at about 30 per second. Between those two moments sits one of the more interesting systems running in production today.

Why This Matters#

Most engineers can draw the boxes: client → gateway → router → GPU fleet. That gets you through the first fifteen minutes of an interview. What separates candidates is what happens when the interviewer asks the follow-up:

• Why is your time-to-first-token high while GPU utilization looks low?
• What do you autoscale on?
• What breaks first as context lengths grow?

Every one of those questions is answered by the same small set of facts about how a transformer actually executes at inference time. There are maybe five of them. This page is those five facts, plus the vocabulary to state them cleanly under pressure.

One honesty note, because it matters for how you use this: understanding these mechanics is not the same as having built vLLM. Interviewers know the difference, and they reward first-principles reasoning over recited internals. The goal here is that when someone pushes one level below your architecture diagram, you have somewhere to stand.

One Request, End to End#

The single most useful thing to internalize is that your data changes shape four times on the way through:

text  →  token IDs  →  K,V tensors  →  text deltas  →  SSE frames

Each arrow is a component boundary, and almost every production problem lives at one of them.

The Ten Steps#

The Punchline#

Steps 1–7 happen once per request. Steps 8–10 — the green path — run once per output token.

A 500-token response runs that green loop 500 times. Everything about inference performance follows from the fact that one part of the system runs once and the other part runs hundreds of times, and that these two parts are bottlenecked on completely different hardware resources.

That's the physics section below. First, some concrete numbers.

Worked Example: "I got an offer from OpenAI"#

Words Are Not Tokens#

Six words. Seven tokens:

Two things worth saying out loud, because they surprise people and they have real consequences:

Leading spaces belong to the token. ␣got and got are different token IDs. This is why prompt formatting — a trailing space in a template, say — can measurably change model behavior.

Rare and compound words fragment. BPE builds a fixed vocabulary (~100k–200k entries) from frequent byte sequences. "OpenAI" isn't frequent enough to earn its own entry, so it becomes ␣Open + AI. The practical consequence: you cannot bill, budget, or capacity-plan in words. Non-English text and code fragment much harder — a rule of thumb of ~1.3 tokens/word for English prose can go to 2–3× for languages that don't use Latin script.

The KV Grid#

Now the part that matters for systems. Picture a grid:

• Columns = tokens. Seven of them here.
• Rows = layers. For a 70B-class model, 80 of them.

Each cell holds the K and V tensors for that (layer, token) pair. Prefill fills the entire grid in one pass. Every subsequent decode step appends exactly one new column, all 80 rows.

For our running example model — 80 layers, 8 KV heads under GQA, head dimension 128, fp16:

per layer, per token:  2 × 8 × 128 × 2 bytes  =  4 KB
per token (80 layers): 4 KB × 80              =  320 KB
this sentence:         320 KB × 7 tokens      =  2.24 MB

2.24 MB of GPU memory for one seven-token sentence. Hold that number; the physics and continuous batching sections are both consequences of it.

Being able to produce this arithmetic live is a real differentiator in an interview. It's the difference between a candidate who read a blog post and one who can do napkin math on the whiteboard. The full formula, with more worked cases, is in Appendix A below.