# Observability for High-Traffic Systems

> Metrics, structured logs and distributed traces for high traffic: RED and USE methods, percentiles, OpenTelemetry, SLOs and alerts that matter.

- Author: Syed Wajeeh Haider
- Published: Sep 22, 2026
- Series: Scaling to 1M Requests, part 10 of 10
- Category: Reliability
- Tags: Observability, OpenTelemetry, SLOs, Logging
- Reading time: 4 min
- URL: https://syedwajeehhaider.com/observability-for-high-traffic

At a million requests, something is always a little bit broken. A slow query for one tenant, a flaky third party in one region, a memory leak that takes six hours to matter. You can't fix what you can't see, and with that much traffic you can't see anything by reading logs line by line. Observability is how you ask new questions of a running system and get answers in minutes.

- **43.2 min**: Monthly downtime allowed at 99.9%
- **3**: Signals: metrics, logs and traces
- **p99**: What your unluckiest users feel

## Two methods for what to measure

Most useful dashboards come from two simple checklists:

- **RED, for services**: **R**ate (requests per second), **E**rrors (failed requests per second) and **D**uration (latency distribution). Track these for every endpoint and every dependency.
- **USE, for resources**: **U**tilisation, **S**aturation and **E**rrors for CPU, memory, disk, network, connection pools and queues. Saturation is the early warning: a queue that is growing means trouble before latency shows it.

## Percentiles, not averages

Averages flatten exactly the pain you are looking for. Record latency as a histogram and look at **p50, p95 and p99**. At a million requests a day, a 1% tail is ten thousand slow experiences, and they are often your heaviest users with the largest accounts and the most data.

## Structured logs with request IDs

Log JSON, not sentences, so you can filter and aggregate by field. Give every request an ID, pass it to downstream services in a header, and attach it to every log line. When a user reports a problem, one ID pulls the whole story together.

*Request-scoped JSON logging with pino*

```ts
import pino from 'pino';
import { randomUUID } from 'node:crypto';

export const logger = pino({
  level: process.env.LOG_LEVEL ?? 'info',
  redact: ['req.headers.authorization', 'password'],
});

app.use((req, res, next) => {
  const requestId = req.get('x-request-id') ?? randomUUID();
  req.log = logger.child({ requestId });
  res.set('x-request-id', requestId);

  const start = process.hrtime.bigint();
  res.on('finish', () => {
    req.log.info(
      {
        method: req.method,
        route: req.route?.path,
        status: res.statusCode,
        durationMs: Number(process.hrtime.bigint() - start) / 1e6,
      },
      'request completed',
    );
  });
  next();
});
```

> **Log volume costs money:** At high traffic, logging every successful request at full detail can cost more than the servers. Log errors and slow requests in full, sample the healthy ones, and rely on metrics for the aggregate view.

## Distributed tracing with OpenTelemetry

Once a request passes through an API, a queue, a worker and three databases, logs alone cannot tell you where the time went. A **trace** follows one request across every hop as a tree of timed spans. OpenTelemetry is the vendor-neutral standard, and its Node.js auto-instrumentation covers HTTP, Express, popular database drivers and Redis without code changes:

*instrumentation.ts: load before your app starts*

```ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

const sdk = new NodeSDK({
  serviceName: 'orders-api',
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4318/v1/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();
```

Export to a collector and send the data to whichever backend you prefer, such as Grafana Tempo, Jaeger, Datadog or AWS X-Ray. Sample traces at high volume, but always keep traces for errors and slow requests.

## SLOs and error budgets

A **service level objective** turns "fast and reliable" into a number, for example: 99.9% of checkout requests succeed in under 800 ms over 30 days. The 0.1% left over is your **error budget**, about 43 minutes of full downtime a month, or the equivalent in slow and failed requests. While budget remains, ship features. When it is burning fast, slow down and fix reliability. It turns endless arguments into a simple rule.

## Alert on symptoms, not causes

- **Page a human** for user-facing symptoms: error rate, latency SLO burn rate, checkout failures.
- **Send a ticket** for causes that are not hurting users yet: high CPU, a filling disk, a growing DLQ.
- Every page must be **actionable** and link to a runbook. Alerts nobody acts on train the team to ignore alerts.
- Build a small set of dashboards: one overview with RED metrics per service, then drill-downs per dependency. If nobody opens a dashboard during incidents, delete it.

## Key takeaways

- Use RED for services and USE for resources, and watch saturation closely.
- Record latency as histograms and look at p95 and p99.
- Log structured JSON with request IDs, and trace requests end to end with OpenTelemetry.
- Define SLOs, spend error budgets deliberately, and page only on symptoms users feel.

## Series wrap-up

Over ten parts we took a system from "it works on my machine" to "it works at a million requests":

1. Understanding the traffic math and where applications break first.
2. Horizontal scaling with stateless services and EC2 Auto Scaling.
3. Caching at the CDN, Redis and in-memory layers.
4. Database scaling with indexes, read replicas and connection pooling.
5. Keeping Node.js fast under load: the event loop, clustering and backpressure.
6. Queues and async work to keep slow tasks off the request path.
7. Rate limiting and load shedding to protect the front door.
8. Serverless at scale, and when AWS Lambda fits.
9. Load testing with k6 to find the breaking point first.
10. Observability to see problems as they happen.

None of these ideas is exotic. Scale comes from applying the simple ones consistently, measuring everything, and fixing the next bottleneck before your users find it. If you are planning a launch, or your application is starting to feel the strain, **get in touch**. I'd be glad to help you scale it.