Observability for High-Traffic Systems

At a million requests, something is always a little bit broken. A slow query for one tenant, a flaky third party in one region, a memory leak that takes six hours to matter. You can't fix what you can't see, and with that much traffic you can't see anything by reading logs line by line. Observability is how you ask new questions of a running system and get answers in minutes.

Two methods for what to measure

Most useful dashboards come from two simple checklists:

• —RED, for services: Rate (requests per second), Errors (failed requests per second) and Duration (latency distribution). Track these for every endpoint and every dependency.
• —USE, for resources: Utilisation, Saturation and Errors for CPU, memory, disk, network, connection pools and queues. Saturation is the early warning: a queue that is growing means trouble before latency shows it.

Percentiles, not averages

Averages flatten exactly the pain you are looking for. Record latency as a histogram and look at p50, p95 and p99. At a million requests a day, a 1% tail is ten thousand slow experiences, and they are often your heaviest users with the largest accounts and the most data.

Structured logs with request IDs

Log JSON, not sentences, so you can filter and aggregate by field. Give every request an ID, pass it to downstream services in a header, and attach it to every log line. When a user reports a problem, one ID pulls the whole story together.

import pino from 'pino';
import { randomUUID } from 'node:crypto';

export const logger = pino({
  level: process.env.LOG_LEVEL ?? 'info',
  redact: ['req.headers.authorization', 'password'],
});

app.use((req, res, next) => {
  const requestId = req.get('x-request-id') ?? randomUUID();
  req.log = logger.child({ requestId });
  res.set('x-request-id', requestId);

  const start = process.hrtime.bigint();
  res.on('finish', () => {
    req.log.info(
      {
        method: req.method,
        route: req.route?.path,
        status: res.statusCode,
        durationMs: Number(process.hrtime.bigint() - start) / 1e6,
      },
      'request completed',
    );
  });
  next();
});

Distributed tracing with OpenTelemetry

Once a request passes through an API, a queue, a worker and three databases, logs alone cannot tell you where the time went. A trace follows one request across every hop as a tree of timed spans. OpenTelemetry is the vendor-neutral standard, and its Node.js auto-instrumentation covers HTTP, Express, popular database drivers and Redis without code changes:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

const sdk = new NodeSDK({
  serviceName: 'orders-api',
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4318/v1/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

Export to a collector and send the data to whichever backend you prefer, such as Grafana Tempo, Jaeger, Datadog or AWS X-Ray. Sample traces at high volume, but always keep traces for errors and slow requests.

SLOs and error budgets

A service level objective turns "fast and reliable" into a number, for example: 99.9% of checkout requests succeed in under 800 ms over 30 days. The 0.1% left over is your error budget, about 43 minutes of full downtime a month, or the equivalent in slow and failed requests. While budget remains, ship features. When it is burning fast, slow down and fix reliability. It turns endless arguments into a simple rule.

Alert on symptoms, not causes

• —Page a human for user-facing symptoms: error rate, latency SLO burn rate, checkout failures.
• —Send a ticket for causes that are not hurting users yet: high CPU, a filling disk, a growing DLQ.
• —Every page must be actionable and link to a runbook. Alerts nobody acts on train the team to ignore alerts.
• —Build a small set of dashboards: one overview with RED metrics per service, then drill-downs per dependency. If nobody opens a dashboard during incidents, delete it.

Key takeaways

• —Use RED for services and USE for resources, and watch saturation closely.
• —Record latency as histograms and look at p95 and p99.
• —Log structured JSON with request IDs, and trace requests end to end with OpenTelemetry.
• —Define SLOs, spend error budgets deliberately, and page only on symptoms users feel.

Series wrap-up

Over ten parts we took a system from "it works on my machine" to "it works at a million requests":

1. 01Understanding the traffic math and where applications break first.
2. 02Horizontal scaling with stateless services and EC2 Auto Scaling.
3. 03Caching at the CDN, Redis and in-memory layers.
4. 04Database scaling with indexes, read replicas and connection pooling.
5. 05Keeping Node.js fast under load: the event loop, clustering and backpressure.
6. 06Queues and async work to keep slow tasks off the request path.
7. 07Rate limiting and load shedding to protect the front door.
8. 08Serverless at scale, and when AWS Lambda fits.
9. 09Load testing with k6 to find the breaking point first.
10. 10Observability to see problems as they happen.

None of these ideas is exotic. Scale comes from applying the simple ones consistently, measuring everything, and fixing the next bottleneck before your users find it. If you are planning a launch, or your application is starting to feel the strain, get in touch. I'd be glad to help you scale it.