Without rate limiting, a single misbehaving client can take down your entire service. Think about it: in the [Webhook](/learn/system-design/questions/webhook) question, a buggy merchant endpoint retries thousands of times per second. In [Slack](/learn/system-design/questions/slack), a rogue bot floods a channel with messages. In any [API Gateway](/learn/system-design/questions/uber) scenario, a scraper hammers your endpoints faster than your servers can handle. **Rate limiting** caps how many requests a client can make within a time window. It protects backend systems from overload, enforces fair usage across clients, and buys time for autoscaling to kick in. The question is not *whether* to rate limit, but *which algorithm* to use and *where* to enforce it. ## Token Bucket (The Default Choice) The **token bucket** is the most widely used algorithm, and the one you should reach for first in an interview. It works like this: - A bucket holds up to **N tokens** (the burst capacity). - Tokens are added at a fixed **refill rate** (e.g., 10 tokens per second). - Each request consumes one token. If the bucket has tokens, the request is allowed. If the bucket is empty, the request is rejected. The key property is that token bucket **allows controlled bursts**. If a client has been idle for a while, tokens accumulate up to the bucket capacity. That client can then send a short burst of requests above the steady-state rate. This matches real-world usage patterns well -- users tend to be bursty, not perfectly uniform. In practice, you store two values per client: the **current token count** and the **last refill timestamp**. On each request, you calculate how many tokens to add based on elapsed time, then check if there are enough to proceed. This is sometimes called a "lazy refill" because you only compute the refill when a request arrives. **See it in practice:** AWS API Gateway, Stripe, and GitHub all use token bucket for their public API rate limits. In the [Distributed Cache](/learn/system-design/questions/cache) question, you would use token bucket to limit how many reads/writes a single client can issue per second. ## Leaky Bucket (When You Need Uniform Output) The **leaky bucket** is the mirror image of the token bucket. Instead of controlling *admission* with tokens, it controls *output* with a queue: - Requests enter a fixed-size queue (the "bucket"). - Requests are processed at a **constant drain rate**, regardless of how fast they arrive. - If the queue is full, new requests are dropped. The key difference from token bucket: leaky bucket **smooths bursts into a uniform output rate**. No matter how spiky the input, the downstream service sees a perfectly steady stream. This makes it ideal when the service behind the rate limiter cannot handle any burst at all -- for example, a legacy payment processor that fails under variable load. The trade-off is that leaky bucket adds **latency**. Requests sit in the queue waiting to be processed, even when the system is not overloaded. Token bucket processes requests immediately if tokens are available. For most API rate limiting scenarios, token bucket is the better fit. **See it in practice:** In the [Webhook](/learn/system-design/questions/webhook) question, leaky bucket is useful for outbound delivery -- you want to send webhooks to merchant endpoints at a steady rate to avoid overwhelming them, even if your internal event stream is bursty. ## Fixed Window and Sliding Window Counters ### Fixed Window (Simple but Flawed) The simplest approach: divide time into fixed intervals (e.g., 1-minute windows) and keep a counter per window. When the counter exceeds the limit, reject. Reset the counter when a new window starts. The problem? **Boundary attacks.** A client can send the full limit at the end of one window and the full limit at the start of the next, effectively doubling their rate in a short burst that spans the boundary. ### Sliding Window Counter (The Fix) The **sliding window counter** eliminates boundary attacks by blending the previous and current windows. Instead of a hard reset at the window boundary, it uses a weighted average: The formula: take the previous window's count, multiply by the percentage of the previous window that overlaps with the current sliding window, and add the current window's count. If the limit is 100 per minute, the previous window had 80 requests, and we are 30% into the current window with 20 requests, the effective count is `80 x 0.7 + 20 = 76`. Under the limit -- request allowed. This gives you the accuracy of a true sliding window (tracking every individual timestamp) with the memory efficiency of fixed windows (just two counters). **See it in practice:** In the [Job Scheduler](/learn/system-design/questions/job-scheduler) question, sliding window counters can limit how many jobs a tenant submits per minute. In [Online Chess](/learn/system-design/questions/online-chess), fixed windows are adequate for move-rate limits since the stakes of a boundary attack are low. ## Distributed Rate Limiting with Redis On a single server, rate limiting is trivial -- just an in-memory counter. But in a multi-server deployment, every app server needs to see the same counter. If Server A allows 50 requests and Server B independently allows another 50, a client with a limit of 50 just got 100 through. **Redis** is the standard solution. It is single-threaded, so operations are naturally serialized, and it supports **Lua scripting** for atomic multi-step operations. The critical requirement is an **atomic read-modify-write**: check the current count, increment it, and set expiry -- all in one operation with no race window. The flow: 1. Client sends a request to any app server (via load balancer). 2. The app server executes a Lua script on Redis: GET the current count for this client key, check if it is under the limit, INCR the counter, and PEXPIRE the key with the window duration. 3. Redis returns whether the request is allowed or denied. 4. The app server either processes the request or returns **429 Too Many Requests**. Because the Lua script executes atomically inside Redis, there are no race conditions -- even when dozens of app servers check the same key simultaneously. ### Fail-Open vs Fail-Closed What happens when Redis is unreachable? - **Fail-closed** (reject all requests): protects your backend but kills availability. A Redis outage becomes a full service outage. - **Fail-open** (allow all requests): preserves availability but removes protection temporarily. Most production systems **fail-open with aggressive alerting**. A brief Redis hiccup should not take down the entire service. The risk of a few seconds without rate limiting is almost always less than the risk of rejecting all traffic. **See it in practice:** In the [Distributed Cache](/learn/system-design/questions/cache) question, you design a Redis-like system from scratch -- understanding Lua script atomicity and single-threaded execution is essential. In the [CI/CD Pipeline](/learn/system-design/questions/cicd) question, distributed rate limiting prevents a single team from monopolizing build runners. ## Where to Place Rate Limiting Rate limiting can live at multiple layers. Each has trade-offs: In practice, most production systems use **layered rate limiting**: a coarse global limit at the API gateway and fine-grained per-user limits in application middleware. The gateway catches obvious abuse (IP-level floods, unauthenticated spam) while the middleware enforces business rules (per-user quotas, per-endpoint throttling). This layered approach means that even if the gateway passes a request, the middleware can still reject it based on more specific rules. And if the middleware's Redis goes down (fail-open), the gateway still provides baseline protection. **See it in practice:** In [Uber](/learn/system-design/questions/uber), the API gateway rate-limits by rider/driver API key globally, while individual services enforce per-user limits on ride requests. In [Google Calendar](/learn/system-design/questions/google-calendar), the gateway limits total API calls per OAuth client, while the app layer limits event creation per user to prevent calendar spam. ## Response Headers (Tell Clients What Happened) When you rate limit a client, do not just return a bare 429. Include **response headers** so well-behaved clients can back off gracefully: - **X-RateLimit-Limit**: maximum requests allowed in the window - **X-RateLimit-Remaining**: requests remaining in the current window - **X-RateLimit-Reset**: Unix timestamp when the window resets - **Retry-After**: seconds until the client should retry (included with 429 responses) Good clients use these headers to implement **exponential backoff** -- waiting longer between retries each time they are throttled. This is a cooperative system: you enforce limits, and clients respect them. Without informative headers, clients resort to blind retries, which only makes the problem worse. ## Key Takeaways for Interviews When rate limiting comes up in a system design interview, here is what interviewers want to hear: 1. **Start with the algorithm.** Token bucket is the default. Mention leaky bucket if the downstream service needs uniform throughput. Mention sliding window if boundary attacks matter. 2. **Address distribution immediately.** A single-server counter is not enough. Redis with Lua scripts gives you atomic, shared state across all app servers. 3. **Layer your limits.** Gateway for coarse IP-level protection, middleware for fine-grained user-level rules. Explain why you need both. 4. **Discuss failure modes.** Fail-open vs fail-closed is a real production decision. Know the trade-offs and have an opinion. 5. **Mention response headers.** This shows you think about the client experience, not just the server-side enforcement. **See it in practice:** Rate limiting appears in nearly every system design question. The [Bookstore](/learn/system-design/questions/bookstore) question needs it to prevent checkout abuse. The [ChatGPT](/learn/system-design/questions/chatgpt) question needs per-user token limits on expensive LLM calls. The [Payment](/learn/system-design/questions/payment) question needs it to prevent duplicate charge attempts. The [Ads Click Aggregator](/learn/system-design/questions/adds-click-aggregator) question needs it to filter bot traffic from real clicks. Whenever you see an API in a design question, rate limiting should be part of your answer.