Design a service that accepts individual synchronous HTTP requests from users, aggregates them into batches internally, routes batches to available GPU workers, maps responses back to the original requesters, and maintains low latency while maximizing throughput at 10,000 RPS. You are given a fixed backend function that you cannot modify: `def batchstring(inputs: list[str]) -> list[str]` Which processes a batch of string inputs and returns string outputs to client users. **Constraints:** - Input size: 1-100 strings per batch - Output size: 1-100 strings (one per input) - Latency: ~100 ms per batch (fixed, regardless of batch size within limits) - Concurrency: Each GPU instance can only process ONE batch at a time Before diving in, ask yourself: - How do you collect N concurrent HTTP connections and merge them into one GPU call, then fan responses back out to the exact waiting connections? - When should you fire a batch — when it's full, when time runs out, or something smarter? - What happens at 3 am when traffic drops to 5 RPS? Does your batcher still work correctly? - If a GPU crashes mid-batch, what happens to the users waiting for that batch's response? - Your batcher is now a bottleneck — how do you scale it horizontally without double-sending a batch to the GPU? You don't need to answer all of these right away. A **Senior/Staff+ Engineer** doesn't stop at basic functionality — they **anticipate edge cases**, **design for resilience**, and **push for production-grade reliability**. But **great systems start with great questions.** What would *you* ask next? ## Functional Requirements **FR1 — Clients can submit inference requests and receive responses synchronously** A client sends a single HTTP POST request with a text input and waits on the same connection for the result. From the client's perspective, this is a normal synchronous call — they have no visibility into the batching happening internally. **Why this matters:** The synchronous contract is non-negotiable. The system must **hold the connection open internally** and resolve it when the batch completes. **FR2 — The system batches requests internally for GPU efficiency** Individual requests must be aggregated into batches before being dispatched to GPU workers. This is completely invisible to the client. The batching logic is the core of the entire system. **Why this matters:** The GPU function has **fixed latency regardless of batch size**. Sending one request at a time wastes 99% of the GPU's parallel capacity. The entire economic argument for this system is: **pay the same fixed cost, process as many inputs as possible**. **FR3 — Requests are prioritized by user tier** The system must support at least 3 tiers — for example, Free, Paid, and Enterprise — and ensure higher-tier users experience lower queue wait times, especially under load. **Why this matters:** Under normal conditions, all users feel fast. Under high load, you shed Free traffic first to protect the Paid experience. This is where business requirements directly shape infrastructure design. ## Non-Functional Requirements **NFR1 — Low Latency: P95 under 500ms end-to-end** Clients should receive responses in under 500ms at the 95th percentile. This is the "near real-time" constraint from the problem statement. *(We suggest you think and read about the next and come back to this answer!)* One of the sharpest interview signals is breaking down a latency budget rather than just saying "it should be fast." For this system, a well-reasoned budget looks like: - **Queue wait time (tier-based):** ~20ms average - **Batch assembly window (timeout floor):** ~40ms max - **PCIe transfer (CPU to GPU RAM):** ~5ms - **GPU inference (fixed per batch):** ~100ms - **PCIe transfer (GPU to CPU RAM):** ~5ms - **Response demux + network:** ~10ms **Total: ~180ms** — well within 500ms, with headroom for spikes. Staff+ signal: candidates who derive this budget, identify the GPU inference as the dominant term, and then ask "what changes if the inference time doubles?" are showing they understand the system's load-bearing constraints. **NFR2 — High Throughput: 1,000 RPS baseline, designed for 10,000 RPS** The system must handle 1,000 requests per second at launch with the architecture ready to scale to 10x without fundamental redesign. **NFR3 — High Availability: 99.9% uptime** GPU worker failures must not cascade into full system outages. The system must degrade gracefully — shedding load, returning clean errors — rather than crashing silently. **NFR4 — GPU Utilization: Target 70-80%** GPUs are expensive. The goal is not 100% utilization — that leaves no headroom for traffic spikes during the 5-minute GPU cold start window. 70-80% is the sweet spot. This is a common interview trap. Running GPUs at 100% means any spike in traffic immediately creates a queue. Since GPU cold start takes 5 minutes, you have no capacity buffer during that window. Users experience dramatically degraded latency until the new GPU is warmed up. The 70% target gives you 30% headroom to absorb spikes while new capacity comes online. It costs slightly more in GPU spend but buys you predictable latency under bursty traffic — which is almost always worth it. **NFR5 — Backpressure: The system must degrade gracefully, never silently** When queues fill up, the system must actively reject new requests with meaningful errors (HTTP 429) rather than accepting them and letting latency balloon indefinitely. **Requirement Summary** ## Core Entities Authentication and identity management happen upstream at the API Gateway layer. By the time a request reaches the batching system, all the system needs to know is the **tier** (Free, Paid, Enterprise) — not who the user is. Keeping the inference system tier-aware but identity-agnostic makes it simpler, faster, and easier to reason about. This is a deliberate scope boundary. In a real interview, calling it out shows product thinking — you're not over-designing the inference layer with concerns that belong elsewhere. ## API Design ### Client-Facing REST API ```jsx Client sends: POST /v1/inference Authorization: Bearer Request body: { "model": "llm-v1", "input": "What is the capital of France?", "parameters": { "max_tokens": 100, "temperature": 0.7 } } Response (200 OK): { "request_id": "req_abc123", "output": "The capital of France is Paris.", "metadata": { "latency_ms": 312, "tokens_used": 14, "queue_wait_ms": 28 } } ``` ### Internal API: Batcher to GPU Worker ```jsx POST /internal/batch Request body: { "batch_id": "batch_xyz789", "requests": [ {"request_id": "req_1", "input": "..."}, {"request_id": "req_2", "input": "..."}, ... ] } Response: { "batch_id": "batch_xyz789", "results": [ {"request_id": "req_1", "output": "...", "status": "success"}, {"request_id": "req_2", "output": "...", "status": "success"} ] } ``` Unlike a chat or online chess game application, inference is a classic request-response interaction — one input, one output, done. WebSocket adds connection management overhead and bi-directional complexity that adds no value here. HTTP connection (holding the connection open while the batch processes) is the right primitive. It's simple, well-understood, and fits the synchronous contract clients expect. WebSocket would only make sense if we were streaming tokens back word-by-word — which is a valid follow-up scope extension but out of scope for the core design. ## GPU Knowledge Share Most engineers are comfortable designing around databases and web servers. GPUs behave differently in ways that directly shape every decision in this system. This section gives you just enough context to make the design feel *inevitable* rather than arbitrary. **Three Facts That Drive the Entire System Design** **Fact 1 — One batch at a time per GPU instance** A single GPU instance can only process one batch at a time. While it's running `batchstring()`, nothing else can enter that GPU. The next batch waits outside in a queue. This is the fundamental constraint that makes a batcher necessary at all — without it, you'd be firing individual requests one-by-one and wasting massive GPU capacity. **Fact 2 — Inside a batch, everything is parallel** GPUs have thousands of small cores (vs a CPU's handful of large ones). When a batch of 100 strings arrives, all 100 are processed simultaneously — one core per input item. Sending a batch of 1 wastes 99% of that parallel capacity. Sending a batch of 100 uses it fully, for the same fixed time cost. This is the entire economic argument for batching. **Fact 3 — Order is preserved by memory address, not logic** Let's take NVIDIA's A100 as example: CUDA (NVIDIA's GPU programming platform) pins each input to a fixed memory slot when a batch loads. `inputs[0]` goes to address `0x00`, `inputs[1]` to `0x01`, and so on. When compute finishes, results are written back to those same slots and read out in order. This means `outputs[i]` always maps back to `inputs[i]` — not because of any routing logic, but because of how GPU memory is laid out physically. **CUDA** stands for **Compute Unified Device Architecture** — it's NVIDIA's programming platform that lets code run directly on GPU cores. When `batchstring([s1, s2, s3])` executes: 1. **cudaMemcpy (Host to Device):** Input array copies from CPU RAM to GPU RAM across the PCIe bus 2. **Kernel launch:** CUDA assigns one thread per input. Thread 0 processes `s1`, thread 1 processes `s2`, all simultaneously 3. **cudaMemcpy (Device to Host):** Results copy from GPU RAM to CPU RAM across PCIe The PCIe bus is the physical highway between CPU and GPU on the motherboard. At PCIe 4.0 speeds (~32 GB/s), this transfer costs roughly **1-5ms per batch** — regardless of batch size. This is the fixed transfer overhead in our latency budget. You pay it whether you send 1 string or 100. This is exactly why the problem statement says "fixed latency regardless of batch size" — the PCIe transfer + kernel launch overhead dominates, and it's constant. Logically yes, physically no. There are three levels people often conflate, and understanding the difference matters for reasoning about GPU performance. **CUDA Core** is the physical arithmetic unit — it executes one floating point or integer operation per clock cycle. An A100 has 6,912 of them. **Thread** is the logical unit of work. For our purposes, 1 request = 1 thread: thread `i` reads `inputs[i]`, runs inference, writes `outputs[i]`. But a thread is a software abstraction — it does not map one-to-one to a physical core. **Warp** is the real physical execution unit. CUDA groups 32 threads into a warp and runs all 32 simultaneously in lockstep — same instruction, different data slots. This is called **SIMT (Single Instruction, Multiple Threads)**. It is why positional memory access works so well: all 32 threads compute `addr = base + i * stride` in a single clock cycle, reading sequential addresses in parallel. **SM (Streaming Multiprocessor)** is the physical chip that executes warps. An A100 has 108 SMs, each with 64 cores. The CUDA scheduler assigns warps to SMs automatically — you have no control over this. When a warp stalls waiting on memory, the SM instantly switches to another ready warp at zero cost. This is called **latency hiding**, and it is the core reason GPUs maintain high throughput despite memory access delays. What this means for `batchstring()` in practice: You only ever need to reason at the index level. Everything below that is CUDA's job — and it is very good at it. Most people expect that each GPU output must be "tagged" with an ID to know which request it belongs to. CUDA doesn't work that way, and the reason is a fundamental hardware optimization called **coalesced memory access**. GPU threads execute in groups of 32 called **warps**. A warp is most efficient when all 32 threads read or write **sequential memory addresses simultaneously** — one clock cycle per thread. The address formula is pure arithmetic: ```python thread i reads: base_address + i * stride thread i writes: out_base + i * stride ``` This is what CUDA does. Each thread knows its slot by its own index (`threadIdx.x = i`), reads from that slot, computes, and writes the result back to the same slot in the output array. No parsing. No lookup. No coordination between threads. **Why ID-based routing would break this:** If each input carried a tag like `{id: "req_abc", data: "..."}`, the GPU would need to parse the struct, hash or scan to find the right output slot, and write to a non-sequential address. Different threads would jump to different memory locations — called **random (non-coalesced) access** — which can be 10-20x slower. You would destroy the parallelism advantage entirely. **The elegant division of responsibility:** - GPU: pure compute engine. Sees only arrays and indices. Extremely fast. - Batcher (CPU): maintains the ID-to-index mapping (`index_map = {0: f_UserA, 1: f_UserB, ...}`). Resolves futures after the GPU returns. This is why `batchstring(inputs)[i]` is guaranteed to be the result for `inputs[i]` — not by coincidence, but by intentional hardware design. Position is identity because that is what makes GPUs fast. Hence the API Data Flow could be illustrated below: ## High Level Design **Be aware in this section, we assume:** - All users are active and requests are arriving normally. - Edge cases — what happens when the batcher scales horizontally, when GPUs fail, or when traffic spikes — are covered in the Deep Dives. ### FR1 + FR2 + FR3 — End-to-End Request Flow Because this system's three functional requirements are tightly coupled through a single linear path, we cover all of them in one unified flow. The diagram below shows the complete journey of a request from client to GPU and back. **1. Client sends POST, connection held open (FR1)** A user sends `POST /v1/inference` with their input string. The HTTP connection does not close after the request arrives — it stays open, parked, waiting for a response. This is the key mechanic of FR1. From the client's perspective nothing unusual is happening. From the system's perspective, this open connection is a Future that will be resolved once the batch completes. **2. Load Balancer distributes across Gateway instances** The request is routed to one of several API Gateway instances by the Load Balancer. This is standard horizontal scaling — no inference-specific logic lives here. Round-robin or least-connections both work fine. **3. API Gateway: auth, tier detection, rate limiting** The Gateway validates the API key, identifies the user's tier (Free, Paid, or Enterprise), and checks the current rate limit for that tier. If the system is healthy the request passes through. If the queue is already full the Gateway returns HTTP 429 immediately — backpressure at the earliest possible point, before the request consumes any downstream resources. **4. Request enqueued to tier-specific queue (FR3)** The Gateway writes the request into one of three separate queues based on tier. This is how FR3 is implemented — not through a single shared queue with priority fields, but through physically separate queues. A single priority queue requires the consumer to scan and skip items on every poll. Under high load, low-priority items pile up and the consumer wastes cycles evaluating them. Separate queues let the batcher implement strict priority by simply checking: "Is the Enterprise queue non-empty? If yes, take from there. Otherwise check Paid. Otherwise Free." O(1) per decision, no scanning — and Free users naturally shed first when load is high, without any special logic. **5. Batcher assembles the batch (FR2)** The Batcher continuously polls queues in priority order — Enterprise first, then Paid, then Free. It accumulates requests into a pending buffer and fires when either condition is met: - Buffer reaches `max_size = 100` — full batch, GPU fully utilized - Timer hits `max_wait = 40ms` — timeout floor, no user waits forever Before dispatching, the batcher snapshots the index map: `{0: req_1.future, 1: req_2.future, ...}`. This is the demux contract — the positional mapping that makes response routing deterministic. **6. Batch dispatched to an available GPU worker** The batcher selects a free GPU from the pool and calls `batchstring([s1, s2, ..., sN])`. As established in the GPU Primer, all N inputs run simultaneously across GPU cores. The GPU is locked for this batch until it completes, then releases for the next. **7. Response Handler demuxes outputs back to waiting connections (FR1)** When `batchstring()` returns `[r1, r2, ..., rN]`, the Response Handler iterates the output array positionally: ```python for i, output in enumerate(outputs): index_map[i].resolve(output) # HTTP responses back to the waiting client ``` Each `resolve()` call sends the HTTP response down the open connection to the waiting client. From the client's perspective, their synchronous POST just returned — they have no idea a batch of up to 100 requests was processed in the same GPU call. ### End-to-End Latency Recap Short answer: **NO, not for the core design** The reason is the request lifecycle. A request arrives, gets batched, the GPU processes it, the response flies back, and the connection closes — all in ~180ms. There is nothing that needs to outlive a single request by default. Redis handles everything on the critical path: the queue, the routing hints (`route:{req_id} -> Node-42`), and the GPU capacity signal. A database becomes necessary only when you need something that Redis cannot provide: **The key rule:** if you do add a DB, it must **never sit on the critical path**. Write to it asynchronously after the response is sent — never before. A synchronous DB write on the request path would add 5-20ms per request and break your latency budget. In an interview, the right answer is: *"The core design needs no DB. If requirements expand to include billing, auditing, or long-window rate limiting, I'd add an async write to a time-series store or lightweight OLTP DB off the critical path."* That answer shows you know when to add complexity — and more importantly, when not to. ## Deep Dives While the High Level Design shows the full request path end-to-end, three hard problems remain unresolved. The batcher is deceptively simple to describe but complex to implement correctly under concurrency. Scaling it horizontally introduces coordination problems. And GPU fleet management under failure is where most production systems actually break. These deep dives address each in turn. 1. **The Batcher: How It Actually Works** 2. **Scaling the Batcher Horizontally without Double-Sending** 3. **GPU Pool: Auto-Scaling and Failure Handling** ### DD1 — The Batcher: How It Actually Works **Where the problem lies** The HLD says "the batcher accumulates requests and fires when full or timed out." That sentence hides the hardest implementation detail in the entire system: hundreds of concurrent HTTP connections arrive simultaneously, each one waiting on a response. The batcher must collect their inputs, fire one GPU call, and route each output back to the exact connection that submitted it — all without any of those connections timing out or receiving the wrong response. **Step 1 — Requests park and wait** When a request arrives at the batcher, it does two things atomically: - Appends its input string to a shared pending buffer - Creates a `Future` (or `Promise`) and suspends — the HTTP connection stays open, yielding the thread The connection does not close. It does not poll. It simply parks on its Future until something resolves it. This is what makes synchronous-feeling responses possible despite internal async batching. A `Future` is the right primitive here because it lets the thread yield without closing the connection. The HTTP server framework holds the socket open, and the Future is the handle that lets the batcher resolve it later from a completely different code path — the flush loop. The client sees none of this — from their perspective the POST is just slow to respond. **Step 2 — The flush trigger** A background loop watches the pending buffer and fires when either condition is met: ```python if len(pending) >= MAX_BATCH_SIZE: # 100 — GPU fully utilized flush() elif time_since_first_request >= MAX_WAIT_MS: # 40ms — timeout floor flush() ``` At 5 RPS with max batch size 100, a size-only trigger means the first user waits 20 seconds for 99 others to arrive. That is completely unacceptable. The timeout floor guarantees every user gets a response within `max_wait + inference_time` even if they're the only person using the system. But how low should the timeout be? - **Too low (5ms):** Small batches, wasted GPU capacity, higher cost per request - **Too high (200ms):** Better GPU utilization, but users feel the delay at low traffic 40ms is a reasonable default — it keeps P95 latency well under 500ms while allowing batches of reasonable size at moderate traffic. At 1,000 RPS and batch size 100, you fill a batch in ~100ms anyway, so the timeout rarely fires under load. **Step 3 — Dispatch and the demux contract** Before calling the GPU, the batcher snapshots its state atomically: ```python def flush(): # Snapshot and clear atomically — new requests go into next batch batch_inputs = pending.copy() batch_futures = futures.copy() pending.clear() futures.clear() # Fire the GPU call outputs = batchstring(batch_inputs) # Demux: outputs[i] maps to batch_futures[i] by index for i, output in enumerate(outputs): batch_futures[i].resolve(output) # Each resolve() sends the HTTP response to the waiting connection ``` The correctness guarantee is positional: CUDA preserves input order in its output array, like we discussed in How GPU works section. `outputs[0]` always corresponds to `inputs[0]` — not because of any routing logic, but because GPU memory slots are fixed at batch load time. Position is identity. The demux requires zero coordination: iterate the output array, resolve the matching Future by index, done. If the interviewer signals durability isn't a concern — "all components can be memory-based" — this in-memory implementation is complete. No Redis, no Kafka, no persistence layer. **The entire batcher is a shared buffer, a background flush loop, and a dict of Futures**. It is both the minimum viable answer and a correct one. For an interview where durability is deprioritized: in-memory wins. For production: queue-backed is safer. The architecture shape is identical — the queue is just the buffer between Gateway and Batcher. The Deep Dive on scaling (DD2) explores what happens when you need to run multiple batcher instances. ### DD2 — Scaling the Batcher Horizontally without Double-Sending **Where the problem lies** At 1,000 RPS a single batcher instance handles the load comfortably. At 10,000 RPS it becomes the bottleneck — one process accumulating and flushing hundreds of requests per second hits CPU and memory limits. The natural answer is to run multiple batcher instances. But this immediately creates the **double-send problem**. **When two batcher instances poll the same queue simultaneously, they can race to claim the same requests.** Both flush the same batch to the GPU. The GPU processes it twice. Users receive duplicate responses, or worse — the Futures get crossed and users receive someone else's output. With a single batcher, the flush is atomic by definition — one process owns the buffer entirely. There is no race. With multiple batchers polling a shared queue, you need distributed coordination to ensure each request is owned by exactly one instance at the moment of flush. This is the challenge most candidates don't anticipate when asked: *"Your batcher is now a bottleneck — how do you scale it horizontally without double-sending?"* **Option A — Kafka consumer groups (recommended for production)** Partition the queue by `request_id` hash. Each batcher instance is assigned a subset of partitions by Kafka's consumer group protocol. Within any given partition, only one consumer ever reads — exclusive ownership is guaranteed by Kafka itself. No distributed locking needed. ```python Partition 0 → Batcher instance 1 # owns exclusively Partition 1 → Batcher instance 2 # owns exclusively Partition 2 → Batcher instance 3 # owns exclusively ``` When a batcher instance fails, Kafka detects the missing heartbeat and rebalances — its partitions are reassigned to surviving instances within seconds. Scaling up is equally clean: add a new instance, Kafka rebalances and gives it a partition share. **Trade-off:** Requires Kafka infrastructure. Adds ~5-20ms queue hop latency vs in-memory. Rebalancing during scaling events causes a brief pause in consumption. Worth it for production durability requirements. **Option B — Redis distributed lock per batch window** Each batcher instance races to acquire a short-lived Redis lock at the start of each flush window. Only the winner flushes. Losers skip their window and retry on the next cycle. ```python if redis.set('batch_lock', instance_id, nx=True, ex=0.05): # 50ms TTL flush() # winner else: pass # loser — skip this window, retry next cycle ``` **Trade-off:** Simple to implement without Kafka. But under high load, contention for the lock causes missed flush windows — requests sit in the buffer longer than expected, tail latency spikes. The problem gets worse at exactly the moment you need it to get better. Not recommended above moderate scale. **Option C — Partitioned gateway routing (recommended for simplicity)** Eliminate the shared state problem entirely: route each request at the API Gateway layer to a specific batcher instance using consistent hashing. Each batcher owns a non-overlapping slice of the request space and flushes its own buffer independently. No shared queue, no distributed locking, no Kafka. ```python batcher_instance = hash(request_id) % NUM_BATCHERS forward_to(batcher_instance, request) ``` The GPU pool remains shared — any batcher can dispatch to any available GPU. Only the accumulation step is partitioned. **Trade-off:** If traffic is unevenly distributed (e.g. one user ID generates 80% of requests), one batcher instance becomes a hot spot while others sit idle. Works well for uniform traffic patterns, which is typical for API workloads. **Which to recommend in an interview** - Interviewer signals in-memory / simplicity priority → **Option C** — no infrastructure dependencies, clean separation, easy to reason about - Interviewer asks about production durability or fault tolerance → **Option A** — Kafka consumer groups give you ownership guarantees and automatic failover - Never recommend **Option B** as a primary solution — Redis lock contention degrades under high load, which is exactly when you need the batcher to be most reliable. It's a band-aid, not a design. Note: Option C and A are not mutually exclusive. You can use gateway-level routing (Option C) to shard traffic across Kafka consumer groups (Option A) for both simplicity and durability at scale. ### DD3 — GPU Pool: Auto-Scaling and Failure Handling **Where the problem lies** Before diving in, it helps to name every failure point in this system and understand which ones this deep dive actually handles. The first three failures are either out of scope for this problem (the problem statement explicitly says in-memory is acceptable) or addressed in DD1. This deep dive focuses exclusively on **GPU-layer failures** — because that is where the most expensive and most irreversible damage happens in a production inference system. The GPU pool has two hard operational problems that don't appear in the HLD, and both stem from the same underlying tension: GPU capacity is expensive, slow to provision, and cannot be scaled reactively in response to real-time traffic. 1. **Cold start lag:** Spinning up a new GPU instance takes ~5 minutes. Traffic can spike in seconds. You cannot reactively scale fast enough to absorb sudden load. 2. **Mid-batch failure:** A GPU crashes while processing a batch. The pending `batchstring()` call never returns. What happens to every user whose request was in that batch? **Part 1 — Auto-scaling: the headroom principle** The answer to cold start lag is not faster provisioning — it's building in enough headroom that you rarely need emergency provisioning at all. Target 70% GPU utilization at steady state. The 30% buffer absorbs traffic spikes while new capacity comes online. Scale-up and scale-down triggers are deliberately asymmetric: ```python # queue_depth = total requests waiting across ALL tier queues # queue_depth = len(queue:enterprise) + len(queue:paid) + len(queue:free) # monitored continuously by polling Redis every few seconds # Scale UP — fast trigger, 2 minute window if gpu_utilization > 0.80 for 2 consecutive minutes: provision_new_gpu() elif queue_depth > 200: # ~0.3s of backlog — early warning, act ahead provision_new_gpu() # Scale DOWN — slow trigger, 10 minute window if gpu_utilization < 0.50 for 10 consecutive minutes: drain_and_remove_one_gpu() # one at a time, never all at once ``` The asymmetry is intentional. Scale up fast because a growing queue compounds — every second of delay makes it worse. Scale down slow because an aggressive scale-down followed by a traffic spike immediately puts you back in the red, with 5 more minutes of cold start lag ahead of you. This is the hardest interview question in this problem. The answer is not "provision more GPUs" — that's too slow to matter. The answer is **shed load immediately, before the queue overflows**: 1. Queue depth crosses 500 → Red zone kicks in automatically — Free tier requests return 429 immediately 2. Paid tier gets throttled proportional to remaining capacity 3. Enterprise tier stays fully protected By the time new GPUs are warm (5 minutes), the queue has already drained because you stopped accepting load you couldn't serve. Users get a clean 429 rather than a hung connection or a 30-second timeout. That is graceful degradation — predictable failure is always better than unpredictable slowdown. The capacity formula that drives this: `max_rps = num_healthy_gpus x requests_per_gpu_per_second`. When a GPU goes offline, recalculate immediately and push the new limit to the API Gateway via a shared Redis config key. The rate limiter tightens within milliseconds. **Dynamic rate limiting: Green / Yellow / Red zones** Tie rate limits directly to observed queue depth, not GPU utilization alone — queue depth reacts faster and more accurately reflects the user-facing experience. The monitoring service polls queue depth every few seconds and updates a Redis config key that all API Gateway instances read independently. No central coordinator needed. **Part 2 — Mid-batch GPU failure** When a GPU crashes mid-batch, `batchstring()` never returns. The batcher must not hang indefinitely — it detects failure via a timeout and handles each request individually: ```python try: outputs = await asyncio.wait_for( batchstring(batch_inputs), timeout=INFERENCE_TIMEOUT_MS / 1000 # e.g. 0.5 seconds ) except asyncio.TimeoutError: # Mark this GPU as offline gpu_pool.mark_unhealthy(gpu_worker_id) # Re-queue each request individually — not as a batch for req in batch_requests: req.retry_count += 1 if req.retry_count <= MAX_RETRIES: # MAX_RETRIES = 1 queue.requeue(req) # routes to a healthy GPU next cycle else: req.future.resolve_error(503) # clean error to client ``` Three key design decisions: **Re-queue individually, not as a batch.** Each request gets its own retry slot. This distributes the retry load across multiple GPU cycles rather than slamming one GPU with a full batch of re-runs simultaneously. **Max 1 retry per request.** One retry is enough to survive a single GPU failure. More than one means requests are bouncing around a degraded fleet, consuming healthy GPU capacity that other users need. **Client gets 503 on second failure, not a hang.** A clean error is always better than a timeout. The client can retry or surface the error immediately rather than waiting for a connection timeout. Imagine 3 GPUs and one fails mid-batch with 100 requests. Those 100 requests re-queue and land on the remaining 2 GPUs. Each now has 50 extra requests on top of normal load. If those GPUs also start timing out under the extra pressure and retries are set to 3, you now have 300 requests bouncing around 1 remaining GPU. The retry amplification collapses the fleet. One retry caps the amplification: 100 requests re-queued once. If the retry also fails, those requests exit the system as 503 errors. The fleet stays stable. In interviews, showing you know when *not* to retry is as important as knowing how to implement retries. ## Final Thoughts Designing a batch inference system looks deceptively simple on the surface — you have a function, you call it with a list, you get a list back. But as this article has shown, every interesting engineering problem lives in the gap between that simple description and the reality of concurrent users, expensive GPU hardware, cold starts, and failure modes that cascade in surprising ways. What makes this question unusual in a system design interview is that it rewards **depth over breadth**. Most system design questions reward you for covering many components. This one rewards you for going deep on one — the batcher. An interviewer who asks this question is not impressed by a diagram with ten boxes. They are impressed by a candidate who can explain, precisely, how a Future works, why CUDA doesn't need request IDs, what happens at 3am when traffic drops to five requests per second, and why retrying a failed batch more than once is worse than not retrying at all. The 3 deep dives in this article map directly to the three questions an Anthropic-level interviewer will use to separate strong answers from great ones: **DD1 tests your concurrency intuition.** Can you explain the parking mechanic? Can you derive the timeout floor from first principles? Can you trace a request from HTTP POST to GPU output to HTTP response without losing the thread? **DD2 tests whether you know when to add complexity.** The right answer to "scale the batcher" is not immediately "use Kafka." It is "here is when you need no coordination, here is when you need partition ownership, and here is the one option that looks simple but degrades under the exact pressure you're trying to handle." **DD3 tests your production instincts.** The GPU cold start problem is not a scaling problem — it is a load shedding problem. The candidate who says "provision more GPUs" has not operated production systems. The candidate who says "shed Free tier load immediately and let the queue drain" has. ### Interview Tips — What Separates Good from Great **Derive numbers, don't assert them** When asked how many GPUs you need, walk through the batch cycle math. When asked about queue thresholds, explain why 200 and 500 are the right calibration points for this throughput and cold start time. Numbers without reasoning are just guesses. Numbers with derivations are engineering. **Name the tradeoff before being asked** For every design decision — Redis vs Kafka, in-memory vs durable, 40ms timeout vs 20ms — name the tradeoff yourself before the interviewer probes it. This signals you understand the design space, not just the answer you picked. Interviewers probe tradeoffs to see if you got lucky or if you reasoned your way there. **Show you know when NOT to add complexity** The strongest answer to "do you need a database?" is "no, and here is exactly why, and here is what would change that answer." The strongest answer to "do you need Kafka?" is "not yet, and here is the specific condition that would make it necessary." Knowing when to keep the design simple is rarer and more valuable than knowing how to add components. **The GPU primer is your differentiator** Most candidates treat the GPU as a black box labeled "GPU Worker." Explaining coalesced memory access, why position is identity, the warp and SM hierarchy, and why CUDA doesn't need request IDs puts you in a category that almost no one else reaches in this interview. It signals you understand *why* the system is shaped the way it is, not just what the shape is. The closing signal question — *"What changes if the latency requirement drops to 50ms?"* — is where great answers separate from good ones. At 50ms, you eliminate the queue hop entirely, the timeout floor drops to single digits, and the batcher must live in-process with the gateway. Showing you understand which constraints are load-bearing and which are incidental is the clearest possible signal that you have internalized the design rather than memorized it. **How ShowOffer can help you?** We've included **callouts** and **open-ended design prompts** throughout this write-up — perfect for self-practice or interview discussion. If you'd like feedback or want to walk through this system design in a mock interview setting, **book a session with us at [ShowOffer.io](https://showoffer.io/)**. Moreover, we just launch a new function to book a session with the author as well. We're here to help you sharpen your skills, gain confidence, and land your next big offer.