Rate Limiting and Load Shedding

Capacity is finite, and at a million requests some of that traffic will be retries, scrapers, buggy clients and bursts you didn't plan for. A system that tries to serve everything under overload ends up serving nothing. Rate limiting decides who gets capacity; load shedding decides what to drop when there isn't enough to go around.

Choosing an algorithm

• —Fixed window: count requests per minute. Simple, but it allows double bursts at window edges.
• —Sliding window: smooths the edges by weighting the previous window. A good default for per-user API quotas.
• —Token bucket: tokens refill at a steady rate up to a capacity, and each request spends one. It allows short bursts while enforcing a long-term average, which matches how real clients behave.

A Redis token bucket

With many API instances, the limit has to live somewhere shared. Redis is the usual choice, and a Lua script keeps the read-refill-spend sequence atomic, so two concurrent requests can never spend the same token.

-- KEYS[1] = bucket key
-- ARGV: capacity, refill rate (tokens/sec), now (ms), cost
local capacity = tonumber(ARGV[1])
local rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local cost = tonumber(ARGV[4])

local bucket = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(bucket[1]) or capacity
local ts = tonumber(bucket[2]) or now

tokens = math.min(capacity, tokens + ((now - ts) / 1000) * rate)
local allowed = 0
if tokens >= cost then
  tokens = tokens - cost
  allowed = 1
end

redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil((capacity / rate) * 1000))
return { allowed, math.floor(tokens) }

import Redis from 'ioredis';
import { readFileSync } from 'node:fs';

const redis = new Redis(process.env.REDIS_URL!);
redis.defineCommand('takeToken', {
  numberOfKeys: 1,
  lua: readFileSync('./token-bucket.lua', 'utf8'),
});

const CAPACITY = 100; // burst size
const RATE = 10; // tokens per second

export async function rateLimit(req, res, next) {
  const key = 'rl:' + (req.user?.id ?? req.ip);
  const [allowed, remaining] = await (redis as any).takeToken(
    key, CAPACITY, RATE, Date.now(), 1,
  );

  res.set('RateLimit-Limit', String(CAPACITY));
  res.set('RateLimit-Remaining', String(remaining));

  if (!allowed) {
    res.set('Retry-After', String(Math.ceil(1 / RATE)));
    return res.status(429).json({ error: 'rate_limited' });
  }
  next();
}

Honest responses

Return 429 Too Many Requests with a Retry-After header, and make your own clients (web, React Native, Flutter) honour it with backoff. Silent drops and generic 500s cause clients to retry immediately, which multiplies the load you were trying to avoid.

Load shedding

Rate limits protect you from individual clients. Load shedding protects you from everyone at once. When the service is already saturated, rejecting some requests quickly beats serving all of them slowly. In Node.js, event loop delay is an excellent saturation signal:

import { monitorEventLoopDelay } from 'node:perf_hooks';

const loop = monitorEventLoopDelay({ resolution: 20 });
loop.enable();
setInterval(() => loop.reset(), 10_000); // look at recent behaviour only

app.use((req, res, next) => {
  const p99ms = loop.percentile(99) / 1e6; // nanoseconds to ms
  if (p99ms > 200 && req.path !== '/health') {
    res.set('Retry-After', '5');
    return res.status(503).json({ error: 'overloaded' });
  }
  next();
});

Shed the cheap-to-reject, low-priority traffic first: background syncs, prefetches and analytics beacons. Protect checkout and login until the very end.

Circuit breakers, timeouts and bulkheads

• —Timeouts on every outbound call. fetch(url, { signal: AbortSignal.timeout(2000) }) means a slow dependency costs you two seconds, not a stuck connection.
• —Circuit breakers stop calling a dependency after repeated failures, fail fast for a cool-down period, then let a trial request through. Libraries like opossum implement this for Node.js.
• —Bulkheads give each dependency its own connection pool or concurrency limit, so a slow payments API cannot exhaust the pool your product pages need.

A fast failure is a feature. A slow failure is an outage that spreads.

Key takeaways

• —Use a token bucket in Redis, updated atomically with a Lua script.
• —Key limits by user, API key or IP, with stricter buckets for expensive endpoints.
• —Answer with 429 plus Retry-After, and make clients back off.
• —Shed low-priority load early, and put timeouts and circuit breakers on every dependency.

Next we look at a platform that scales for you, up to a point: Serverless at Scale: When AWS Lambda Fits.