Learning Objectives

You will learn to design systems that remain reliable under load, failure, and the unexpected. You will understand SLOs, error budgets, circuit breakers, graceful degradation, and disaster recovery.

Why Does This Exist?

Software will fail. Servers will crash. Networks will partition. Disks will corrupt. The question is not "will this fail?" but "when it fails, what does the user experience?" Reliability engineering is the discipline of making systems that fail gracefully, recover quickly, and maintain acceptable service during degraded conditions.

SLO, SLA, SLI, Error Budgets

SLI (Service Level Indicator)
  A quantitative measure of service behavior.
  Examples:
    - Availability: (successful requests / total requests) × 100
    - Latency: p99 request duration < 500ms
    - Error rate: HTTP 5xx / total requests

SLO (Service Level Objective)
  Target value for an SLI.
  Examples:
    - Availability: 99.9% over 30 days
    - p99 latency < 300ms
    - Error rate < 0.1%

SLA (Service Level Agreement)
  Legal contract with a customer.
  Usually weaker than your internal SLO (safety margin).
  Violation has financial consequences.

Error Budget:
  How much failure is permitted in the SLO period.
  99.9% availability = 0.1% failure = 43.8 minutes/month downtime budget

Why error budgets matter:
  - If budget is healthy: ship fast, take risks
  - If budget is depleted: freeze deploys, focus on reliability
  - Aligns engineering with business risk tolerance

Circuit Breakers

A circuit breaker prevents cascading failures. When a downstream service is failing, you stop sending requests to it instead of piling up failures that exhaust your thread pool or connection pool.

Circuit breaker states:
┌─────────────────────────────────────────────────────────┐
│  CLOSED (normal)                                        │
│  Requests flow through. Track failure rate.             │
│  → If failures > threshold: move to OPEN               │
├─────────────────────────────────────────────────────────┤
│  OPEN (failing)                                         │
│  Reject all requests immediately. Return fallback.      │
│  → After timeout period: move to HALF-OPEN             │
├─────────────────────────────────────────────────────────┤
│  HALF-OPEN (testing)                                    │
│  Allow one test request through.                        │
│  → If success: move to CLOSED                          │
│  → If failure: return to OPEN                          │
└─────────────────────────────────────────────────────────┘

// Simplified circuit breaker
class CircuitBreaker {
  private failures = 0;
  private state: 'closed' | 'open' | 'half-open' = 'closed';
  private lastFailureTime?: number;
  
  async call<T>(fn: () => Promise<T>, fallback?: () => T): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.lastFailureTime! > 30000) {
        this.state = 'half-open';
      } else {
        if (fallback) return fallback();
        throw new Error('Circuit is open');
      }
    }
    
    try {
      const result = await fn();
      if (this.state === 'half-open') this.reset();
      return result;
    } catch (error) {
      this.recordFailure();
      throw error;
    }
  }
  
  private recordFailure() {
    this.failures++;
    this.lastFailureTime = Date.now();
    if (this.failures >= 5) this.state = 'open';
  }
  
  private reset() { this.failures = 0; this.state = 'closed'; }
}

Graceful Degradation

When parts of your system fail, the rest should continue providing value, even if reduced. Design for partial functionality.

Examples of graceful degradation:

E-commerce site:
  Recommendation engine down? → Show popular items instead of personalized
  Search service down?        → Show category browsing instead
  Image CDN degraded?         → Show text descriptions with placeholder

API strategy:
  // Instead of failing completely, return a degraded response
  async function getProductPage(productId: string) {
    const product = await productService.get(productId); // required
    
    const [reviews, recommendations] = await Promise.allSettled([
      reviewService.get(productId),     // optional
      recommendationService.get(productId)  // optional
    ]);
    
    return {
      product,
      reviews: reviews.status === 'fulfilled' ? reviews.value : [],
      recommendations: recommendations.status === 'fulfilled' 
        ? recommendations.value 
        : [],
    };
  }

Backpressure

Problem: Your API receives 10,000 requests/sec but your database
         can only handle 1,000 queries/sec.

Without backpressure:
  Queue fills up → memory exhausted → entire service crashes

With backpressure:
  When queue is full, reject new requests immediately with 429 Too Many Requests.
  Callers get a clear signal to retry later (with exponential backoff).
  System remains stable under overload.

// Express rate limiting with backpressure signal
import rateLimit from 'express-rate-limit';

const limiter = rateLimit({
  windowMs: 1 * 60 * 1000, // 1 minute
  max: 100,
  standardHeaders: true, // includes Retry-After header
  message: {
    error: 'Too many requests',
    retryAfter: 60  // client should wait 60 seconds
  }
});

Backup and Restore Strategy

Backup types:
  Full backup:        Complete copy of all data
  Incremental backup: Only changes since last backup
  Point-in-time:      Can restore to any second in a window (WAL)

The 3-2-1 rule:
  3 copies of data
  2 different storage media
  1 offsite/different region

PostgreSQL backup:
  pg_dump mydb -Fc -f backup-$(date +%Y%m%d).dump

Point-in-time recovery (PITR):
  PostgreSQL WAL archiving gives you second-level restore granularity.
  Critical for financial data and compliance.

CRITICAL: Test your restores regularly.
  A backup that has never been restored is not a backup.
  Monthly restore drills are not optional for production systems.

Mini Project (20-30 min)

Write an error budget calculator that triggers an alert when the burn rate is too high.

function checkBurnRate(totalRequests: number, errorRequests: number, sloTarget: number = 0.999) {
  const errorRate = errorRequests / totalRequests;
  const errorBudget = 1 - sloTarget;
  const burnRate = errorRate / errorBudget;
  
  if (burnRate > 10) {
    return 'CRITICAL: Fast burn rate detected!';
  } else if (burnRate > 1) {
    return 'WARNING: Budget will be exhausted early.';
  }
  return 'OK: Burn rate is within limits.';
}

Bigger Project (1-2 hours)

Build a Chaos Engineering tool that randomly kills pods or introduces network latency in a local Kubernetes cluster (Minikube/Kind). Write an incident runbook and verify that your system recovers gracefully within the defined SLO.

# Chaos Mesh simplified experiment
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: delay-experiment
spec:
  action: delay
  mode: one
  delay:
    latency: "200ms"

Interview Questions

Revision Sheet

• SLI/SLO/SLA: Measure → Set targets → Make agreements. Error budget = allowed failure.
• Circuit breaker: Open → reject immediately, Half-open → test recovery.
• Graceful degradation: Partial service beats total failure. Use Promise.allSettled.
• Backpressure: Reject early (429) vs crash late (OOM). Rejection is the better failure.
• Backups: 3-2-1 rule. Test restores monthly. PITR for financial data.