Rate Limiting

Without rate limiting, a single misbehaving client can take down your entire service. Think about it: in the Webhook question, a buggy merchant endpoint retries thousands of times per second. In Slack, a rogue bot floods a channel with messages. In any API Gateway scenario, a scraper hammers your endpoints faster than your servers can handle.

Rate limiting caps how many requests a client can make within a time window. It protects backend systems from overload, enforces fair usage across clients, and buys time for autoscaling to kick in. The question is not whether to rate limit, but which algorithm to use and where to enforce it.

Token Bucket (The Default Choice)#

The token bucket is the most widely used algorithm, and the one you should reach for first in an interview. It works like this:

• A bucket holds up to N tokens (the burst capacity).
• Tokens are added at a fixed refill rate (e.g., 10 tokens per second).
• Each request consumes one token. If the bucket has tokens, the request is allowed. If the bucket is empty, the request is rejected.

The key property is that token bucket allows controlled bursts. If a client has been idle for a while, tokens accumulate up to the bucket capacity. That client can then send a short burst of requests above the steady-state rate. This matches real-world usage patterns well -- users tend to be bursty, not perfectly uniform.

In practice, you store two values per client: the current token count and the last refill timestamp. On each request, you calculate how many tokens to add based on elapsed time, then check if there are enough to proceed. This is sometimes called a "lazy refill" because you only compute the refill when a request arrives.

Leaky Bucket (When You Need Uniform Output)#

The leaky bucket is the mirror image of the token bucket. Instead of controlling admission with tokens, it controls output with a queue:

• Requests enter a fixed-size queue (the "bucket").
• Requests are processed at a constant drain rate, regardless of how fast they arrive.
• If the queue is full, new requests are dropped.

The key difference from token bucket: leaky bucket smooths bursts into a uniform output rate. No matter how spiky the input, the downstream service sees a perfectly steady stream. This makes it ideal when the service behind the rate limiter cannot handle any burst at all -- for example, a legacy payment processor that fails under variable load.

The trade-off is that leaky bucket adds latency. Requests sit in the queue waiting to be processed, even when the system is not overloaded. Token bucket processes requests immediately if tokens are available. For most API rate limiting scenarios, token bucket is the better fit.