Learning Objectives

By the end of this chapter, you will understand the three pillars of observability—logs, metrics, and traces—and be able to implement structured logging, set up health checks, and answer the three critical questions in production: What happened? What is happening? Why is it happening?

Why Does This Exist?

A system crashed at 3 AM. Thousands of users got errors. The on-call engineer woke up to an alert. They SSH into the server. The logs say Error: connect ETIMEDOUT. Nothing else. No context. No trace of which request failed. No history of what changed before it broke.

This is what happens when you ship code without observability. Observability is the ability to understand the internal state of a system from its external outputs. You cannot fix what you cannot see.

Mental Model

Analogy: Think of a hospital ICU. The patient (your application) has monitors attached everywhere: heart rate, blood pressure, oxygen saturation. Nurses watch dashboards. Alarms trigger when thresholds are crossed. Every reading is timestamped and correlated to an event. You can trace exactly when the patient's condition changed and why.

Reality: Your production service needs the same continuous instrumentation. Logs are the events. Metrics are the vital signs. Traces are the patient journey through the system.

The Three Pillars

┌─────────────────────────────────────────────────────┐
│                  OBSERVABILITY                       │
├─────────────┬─────────────────┬─────────────────────┤
│    LOGS     │     METRICS     │       TRACES        │
│             │                 │                     │
│ "What       │ "What is the    │ "Where did this     │
│ happened?"  │ current state?" │ request go?"        │
│             │                 │                     │
│ Timestamped │ Numerical       │ Distributed         │
│ records of  │ measurements    │ call graph          │
│ events      │ over time       │ across services     │
└─────────────┴─────────────────┴─────────────────────┘

Logs

A log is a timestamped record of a discrete event. The most important upgrade you can make to your logging is switching from unstructured to structured logs.

// BAD: Unstructured log
console.log("User 123 logged in from 192.168.1.1 at 10:30");

// GOOD: Structured log (JSON)
logger.info({
  event: "user.login",
  userId: "123",
  ip: "192.168.1.1",
  timestamp: new Date().toISOString(),
  requestId: "req-abc-xyz"
});

Structured logs are machine-parseable. You can query them: "Show me all login failures for userId 123 in the last hour." You cannot do that with string logs.

Correlation IDs

Every request should carry a unique ID that threads through all logs, services, and downstream calls. This is how you trace one user's journey through a distributed system.

// middleware/requestId.ts
import { v4 as uuid } from 'uuid';

export function requestIdMiddleware(req: Request, res: Response, next: NextFunction) {
  const requestId = req.headers['x-request-id'] as string || uuid();
  req.requestId = requestId;
  res.setHeader('x-request-id', requestId);
  next();
}

// In every log call
logger.info({ requestId: req.requestId, event: "payment.initiated", amount: 100 });

Metrics

Metrics are numerical measurements sampled over time. They answer: "What is the current state of the system?"

Key metric types:
┌─────────────────┬─────────────────────────────────────────┐
│ Counter         │ Always increases. Total requests, errors │
│ Gauge           │ Can go up/down. Memory usage, queue size │
│ Histogram       │ Distribution of values. Latency buckets  │
│ Summary         │ Similar to histogram, with quantiles     │
└─────────────────┴─────────────────────────────────────────┘

The Golden Signals (Google SRE):
1. Latency   - How long requests take
2. Traffic   - How many requests per second
3. Errors    - How many requests fail
4. Saturation - How full the system is (CPU, memory, queues)

Traces

A trace is a complete record of a request as it flows through multiple services. It is composed of spans: individual units of work with start time, duration, and metadata.

Request trace for POST /checkout:
│
├─ [0ms] API Gateway (span: 1ms)
│
├─ [1ms] Auth Service: verify token (span: 5ms)
│
├─ [6ms] Cart Service: fetch cart (span: 12ms)
│   └─ [8ms] PostgreSQL: SELECT cart_items (span: 8ms)
│
├─ [18ms] Payment Service: charge (span: 340ms)
│   └─ [20ms] Stripe API call (span: 330ms) ← bottleneck!
│
└─ [360ms] Order Service: create order (span: 22ms)

Total: 382ms

Without distributed tracing, you would never know that 89% of your checkout latency is a single Stripe API call.

Health Checks

Every service must expose health endpoints. Load balancers and orchestrators (Kubernetes, Docker Swarm) use these to route traffic only to healthy instances.

// Health check endpoint
app.get('/health', async (req, res) => {
  try {
    // Check DB connection
    await db.query('SELECT 1');
    
    // Check Redis
    await redis.ping();
    
    res.json({
      status: 'healthy',
      timestamp: new Date().toISOString(),
      uptime: process.uptime(),
      version: process.env.APP_VERSION
    });
  } catch (error) {
    res.status(503).json({
      status: 'unhealthy',
      error: error.message
    });
  }
});

// Readiness vs Liveness:
// /health/live   - Is the process running? (if no, restart it)
// /health/ready  - Can it serve traffic? (if no, stop sending requests)

Structured Logging in Production

// lib/logger.ts
import pino from 'pino';

export const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  base: {
    service: process.env.SERVICE_NAME,
    env: process.env.NODE_ENV,
    version: process.env.APP_VERSION
  },
  timestamp: pino.stdTimeFunctions.isoTime,
  // In prod, output JSON. In dev, prettify.
  transport: process.env.NODE_ENV !== 'production'
    ? { target: 'pino-pretty' }
    : undefined
});

// Usage
logger.info({ userId, orderId }, 'Order created successfully');
logger.error({ err, requestId }, 'Payment failed');

Alerting

Alerts should be actionable. A page at 3 AM must require immediate human action. If it does not, it is noise — and noise burns out engineers.

Alert design principles:
✓ Alert on symptoms, not causes
  - GOOD: "Error rate > 1% for 5 minutes"
  - BAD: "CPU > 80%" (might be fine)

✓ Every alert must have a runbook
  - What does this alert mean?
  - What are the most common causes?
  - How do I fix each cause?

✓ Use severity levels:
  - P1: Service down, immediate response
  - P2: Degraded, respond within 30 min
  - P3: Warning, fix during business hours

Break It

// BAD: No context in errors
app.post('/payment', async (req, res) => {
  try {
    await processPayment(req.body);
    res.json({ success: true });
  } catch (e) {
    console.log('Error'); // What error? Which request? Which user?
    res.status(500).json({ error: 'Failed' });
  }
});

Debug It

// GOOD: Full observability
app.post('/payment', async (req, res) => {
  const log = logger.child({ requestId: req.requestId, userId: req.user.id });
  
  log.info({ amount: req.body.amount }, 'Payment initiated');
  const start = Date.now();
  
  try {
    const result = await processPayment(req.body);
    const duration = Date.now() - start;
    
    log.info({ paymentId: result.id, durationMs: duration }, 'Payment succeeded');
    metrics.increment('payments.success');
    metrics.timing('payments.duration', duration);
    
    res.json({ success: true, paymentId: result.id });
  } catch (error) {
    log.error({ err: error, amount: req.body.amount }, 'Payment failed');
    metrics.increment('payments.failure', { reason: error.code });
    res.status(500).json({ error: 'Payment failed', requestId: req.requestId });
  }
});

Mini Project (20-30 min)

Set up a simple Express server that emits structured logs using Winston and exposes a /metrics endpoint for Prometheus.

import express from 'express';
import winston from 'winston';
import promClient from 'prom-client';

const app = express();
const logger = winston.createLogger({
  format: winston.format.json(),
  transports: [new winston.transports.Console()]
});

const counter = new promClient.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests'
});

app.get('/', (req, res) => {
  logger.info('Handling request', { path: req.path });
  counter.inc();
  res.send('Hello World');
});

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', promClient.register.contentType);
  res.end(await promClient.register.metrics());
});

app.listen(3000, () => logger.info('Server running on port 3000'));

Bigger Project (1-2 hours)

Build a comprehensive Observability Dashboard. Use Docker Compose to spin up a Node.js microservice, Prometheus, and Grafana. Expose custom business metrics (e.g., items in cart) and create a Grafana dashboard that visualizes them alongside RED (Rate, Errors, Duration) metrics.

version: "3.8"
services:
  app:
    build: .
    ports: ["3000:3000"]
  prometheus:
    image: prom/prometheus
    ports: ["9090:9090"]
    volumes: ["./prometheus.yml:/etc/prometheus/prometheus.yml"]
  grafana:
    image: grafana/grafana
    ports: ["3001:3000"]

Interview Questions

Revision Sheet

• Logs: Structured JSON, not strings. Include requestId, userId, event name.
• Metrics: Golden Signals — Latency, Traffic, Errors, Saturation.
• Traces: Spans form a distributed call graph. Find your bottlenecks.
• Health: /health/live (restart?) vs /health/ready (route traffic?)
• Alerts: Actionable, symptom-based, with runbooks. No noise.