Batch Inference System

(We suggest you think and read about the next and come back to this answer!)

One of the sharpest interview signals is breaking down a latency budget rather than just saying "it should be fast."

For this system, a well-reasoned budget looks like:

• Queue wait time (tier-based): ~20ms average
• Batch assembly window (timeout floor): ~40ms max
• PCIe transfer (CPU to GPU RAM): ~5ms
• GPU inference (fixed per batch): ~100ms
• PCIe transfer (GPU to CPU RAM): ~5ms
• Response demux + network: ~10ms

Total: ~180ms — well within 500ms, with headroom for spikes.

Staff+ signal: candidates who derive this budget, identify the GPU inference as the dominant term, and then ask "what changes if the inference time doubles?" are showing they understand the system's load-bearing constraints.

This is a common interview trap. Running GPUs at 100% means any spike in traffic immediately creates a queue. Since GPU cold start takes 5 minutes, you have no capacity buffer during that window. Users experience dramatically degraded latency until the new GPU is warmed up.

The 70% target gives you 30% headroom to absorb spikes while new capacity comes online. It costs slightly more in GPU spend but buys you predictable latency under bursty traffic — which is almost always worth it.

Authentication and identity management happen upstream at the API Gateway layer. By the time a request reaches the batching system, all the system needs to know is the tier (Free, Paid, Enterprise) — not who the user is. Keeping the inference system tier-aware but identity-agnostic makes it simpler, faster, and easier to reason about.

This is a deliberate scope boundary. In a real interview, calling it out shows product thinking — you're not over-designing the inference layer with concerns that belong elsewhere.

Unlike a chat or online chess game application, inference is a classic request-response interaction — one input, one output, done. WebSocket adds connection management overhead and bi-directional complexity that adds no value here.

HTTP connection (holding the connection open while the batch processes) is the right primitive. It's simple, well-understood, and fits the synchronous contract clients expect. WebSocket would only make sense if we were streaming tokens back word-by-word — which is a valid follow-up scope extension but out of scope for the core design.

CUDA stands for Compute Unified Device Architecture — it's NVIDIA's programming platform that lets code run directly on GPU cores.

When batchstring([s1, s2, s3]) executes:

1. cudaMemcpy (Host to Device): Input array copies from CPU RAM to GPU RAM across the PCIe bus
2. Kernel launch: CUDA assigns one thread per input. Thread 0 processes s1, thread 1 processes s2, all simultaneously
3. cudaMemcpy (Device to Host): Results copy from GPU RAM to CPU RAM across PCIe

The PCIe bus is the physical highway between CPU and GPU on the motherboard. At PCIe 4.0 speeds (~32 GB/s), this transfer costs roughly 1-5ms per batch — regardless of batch size. This is the fixed transfer overhead in our latency budget. You pay it whether you send 1 string or 100.

This is exactly why the problem statement says "fixed latency regardless of batch size" — the PCIe transfer + kernel launch overhead dominates, and it's constant.

Logically yes, physically no. There are three levels people often conflate, and understanding the difference matters for reasoning about GPU performance.

CUDA Core is the physical arithmetic unit — it executes one floating point or integer operation per clock cycle. An A100 has 6,912 of them.

Thread is the logical unit of work. For our purposes, 1 request = 1 thread: thread i reads inputs[i], runs inference, writes outputs[i]. But a thread is a software abstraction — it does not map one-to-one to a physical core.

Warp is the real physical execution unit. CUDA groups 32 threads into a warp and runs all 32 simultaneously in lockstep — same instruction, different data slots. This is called SIMT (Single Instruction, Multiple Threads). It is why positional memory access works so well: all 32 threads compute addr = base + i * stride in a single clock cycle, reading sequential addresses in parallel.

SM (Streaming Multiprocessor) is the physical chip that executes warps. An A100 has 108 SMs, each with 64 cores. The CUDA scheduler assigns warps to SMs automatically — you have no control over this. When a warp stalls waiting on memory, the SM instantly switches to another ready warp at zero cost. This is called , and it is the core reason GPUs maintain high throughput despite memory access delays.

Most people expect that each GPU output must be "tagged" with an ID to know which request it belongs to. CUDA doesn't work that way, and the reason is a fundamental hardware optimization called coalesced memory access.

GPU threads execute in groups of 32 called warps. A warp is most efficient when all 32 threads read or write sequential memory addresses simultaneously — one clock cycle per thread. The address formula is pure arithmetic:

thread i reads:   base_address + i * stride
thread i writes:  out_base     + i * stride

This is what CUDA does. Each thread knows its slot by its own index (threadIdx.x = i), reads from that slot, computes, and writes the result back to the same slot in the output array. No parsing. No lookup. No coordination between threads.

Why ID-based routing would break this:

If each input carried a tag like {id: "req_abc", data: "..."}, the GPU would need to parse the struct, hash or scan to find the right output slot, and write to a non-sequential address. Different threads would jump to different memory locations — called random (non-coalesced) access — which can be 10-20x slower. You would destroy the parallelism advantage entirely.