Rate limiting

A public API without rate limiting is a public API waiting to be flooded. The skill is picking the right algorithm (token bucket is the most flexible), the right dimension to throttle on (user, key, IP), and the right error signal so clients can back off politely.

The big idea

Rate limiting answers two questions:

Pick the dimension

Throttle on something stable and meaningful:

By user / API key

The most useful — protects your service and is fair to legitimate users sharing an IP.

By IP

Crude; corporate NATs and shared mobile gateways look like one client.

By endpoint

A separate budget for expensive endpoints (search, export) prevents one heavy call from starving the rest.

Globally

A backstop for total system load (queue depth, CPU).

Usually you stack two: a generous per-user limit + a tighter per-endpoint one.

Algorithms

Token bucket in Redis

The canonical implementation. Each user gets a bucket; refills happen lazily on each check.

-- Lua script — atomic in Redis
local key = KEYS[1]
local now      = tonumber(ARGV[1])
local capacity = tonumber(ARGV[2])
local refillPerSec = tonumber(ARGV[3])
 
local data = redis.call('HMGET', key, 'tokens', 'updated')
local tokens = tonumber(data[1]) or capacity
local updated = tonumber(data[2]) or now
 
local elapsed = math.max(0, now - updated)
tokens = math.min(capacity, tokens + elapsed * refillPerSec)
 
if tokens < 1 then
  redis.call('HMSET', key, 'tokens', tokens, 'updated', now)
  redis.call('EXPIRE', key, 60)
  return 0                              -- rate-limited
end
 
tokens = tokens - 1
redis.call('HMSET', key, 'tokens', tokens, 'updated', now)
redis.call('EXPIRE', key, 60)
return 1                                -- allowed

Call it on every request keyed by user/key/IP. The atomicity in Redis means you don't need a distributed lock.

What to return when you reject

429 Too Many Requests with informative headers:

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1733000060
Retry-After: 30
 
{
  "error": {
    "code": "rate_limited",
    "message": "Too many requests. Retry in 30 seconds."
  }
}

The headers let well-behaved clients back off. The Retry-After header is honoured by many HTTP clients automatically.

Distributed limit, single counter

A single counter on one Redis node is fine to hundreds of thousands of requests per second. For higher throughput, shard the counter by user ID or use a probabilistic algorithm. Trying to coordinate counters across nodes synchronously defeats the point.

Where to enforce

1. 1

   Edge / CDN

   Cheap blanket limits for crawlers, suspicious IPs. Cloudflare WAF, AWS WAF.

2. 2

   API gateway

   Per-API-key, per-endpoint limits — applies to all backends uniformly.

3. 3

   Application

   Domain-aware limits ("don't let one user create 1000 invoices in a minute").

4. 4

   Downstream / DB

   Connection pool limits, query timeouts — the last line of defence.

In practice

The hardest part isn't the algorithm; it's choosing limits that are tight enough to protect the service but loose enough that legitimate bursts (a deploy, a marketing spike) don't break. Start permissive, watch the 99th percentile, tighten gradually. And ship the limit with the documentation that explains it — a 429 with no docs is just a frustration.