The Anatomy of a Million Requests

"We need to handle a million requests" is one of the most common requirements in a kickoff meeting, and one of the least precise. A million requests a day and a million requests a minute are completely different engineering problems. This series walks through how to get from one to the other, one layer at a time, starting with the arithmetic.

Turn the headline number into requests per second

Every capacity conversation should start with requests per second (RPS), because that is the unit your servers, load balancers and databases actually experience. Divide the total by the number of seconds in the window: a day has 86,400 seconds, an hour has 3,600 and a minute has 60.

Averages hide the real problem. Traffic is never flat: it follows time zones, marketing emails, push notifications and lunch breaks. A useful rule of thumb is to plan for a peak of 5 to 10 times the average. So 1M requests per day is not 11.6 RPS to design for, it is closer to 60 to 120 RPS at peak. That is comfortably served by a couple of well-tuned instances. 1M per minute, on the other hand, is a fleet.

const SECONDS = { day: 86_400, hour: 3_600, minute: 60 };

function peakRps(total: number, window: keyof typeof SECONDS, peakFactor = 8) {
  const avg = total / SECONDS[window];
  return { avg: Math.round(avg * 10) / 10, peak: Math.ceil(avg * peakFactor) };
}

peakRps(1_000_000, 'day');    // { avg: 11.6, peak: 93 }
peakRps(1_000_000, 'minute'); // { avg: 16666.7, peak: 133334 }

Little's Law: how many requests are in flight

RPS tells you the arrival rate. What actually exhausts resources is concurrency: how many requests are being processed at the same moment. Little's Law connects the two: L = λ × W, where λ is the arrival rate and W is the average time each request spends in the system.

At 100 RPS with a 200 ms average response time, you have about 20 requests in flight. Let a slow database query push latency to 2 seconds and the same traffic now means 200 concurrent requests, each holding a connection, a socket and some memory. Latency is a capacity problem, which is why making things faster is often the cheapest way to scale.

Where systems break first

Most applications do not fall over because the CPU is maxed out. They fail at a handful of predictable choke points, usually in this order:

1. 01Database connections. Each app instance opens a pool, and the database has a hard limit. Add instances and you run out of connections before you run out of CPU.
2. 02Synchronous slow work. Sending email, resizing images or calling a third-party API inside the request path makes every request as slow as the slowest dependency.
3. 03A single instance. One server means one point of failure and no room to absorb a spike.
4. 04No cache. Repeating the same expensive query thousands of times a minute is the fastest way to melt a database.
5. 05Unbounded work. Endpoints without pagination, timeouts or rate limits let one bad client consume everything.

A capacity-planning checklist

Before writing any scaling code, answer these questions. They turn a vague requirement into a design you can test:

• —What is the expected peak RPS, and what is the read/write ratio?
• —What p95 latency target do users and the business actually need?
• —Which endpoints are hot, and which of them hit the database on every call?
• —How many database connections can we afford across all instances at peak?
• —What work can move off the request path into a queue?
• —What happens when a dependency is slow or down: do we fail fast or pile up?
• —How will we know we are close to the limit: which metrics and alerts?

# 30 seconds, 50 concurrent connections against a staging endpoint
npx autocannon -c 50 -d 30 https://staging.example.com/api/products

# Record: requests/sec, p50, p97.5 and p99 latency, error count.
# That number times your instance count is your ceiling (minus headroom).

What this series covers

The next nine parts take the choke points above one at a time: horizontal scaling on EC2, caching at every layer, database indexes and pooling, Node.js under load, queues, rate limiting, serverless, load testing and observability. Each part is practical and stack-specific, with code you can adapt.

Scaling is not a feature you add at the end. It is the sum of many small decisions about where work happens and how long it takes.

Key takeaways

• —Convert every traffic goal into peak requests per second, using a 5 to 10 times peak factor.
• —Use Little's Law to estimate concurrency; latency directly multiplies the resources you need.
• —Expect to hit database connection limits and slow synchronous work before CPU limits.
• —Measure a single instance first, then scale out with real numbers.

Next up, part 2: Horizontal Scaling with EC2 Auto Scaling Groups, where we make app servers stateless and let AWS add and remove them automatically.