Learning Objectives

You will understand why distributed systems are fundamentally harder than single-machine systems. You will be able to reason about clocks, failure modes, idempotency, consistency, and the delivery guarantees that systems can realistically provide.

The Fundamental Problem

In a single process on a single machine, if you call a function, it either runs or it doesn't. You can always check the return value. In a distributed system, when you send a message over the network, the three possible outcomes are: the message arrived and was processed, the message never arrived, or the message arrived but the response was lost. From the sender's perspective, outcomes 2 and 3 are indistinguishable.

This is not a solvable problem. It is a fundamental property of networks. Your job as an engineer is to design systems that handle this reality gracefully.

Networks Fail

The Eight Fallacies of Distributed Computing (Peter Deutsch, 1994):
1. The network is reliable              → It isn't
2. Latency is zero                      → It isn't
3. Bandwidth is infinite                → It isn't
4. The network is secure                → It isn't
5. Topology doesn't change              → It does
6. There is one administrator           → There isn't
7. Transport cost is zero               → It isn't
8. The network is homogeneous           → It isn't

Engineers who believe these fallacies build brittle systems.

Clocks in Distributed Systems

You cannot use wall-clock time to order events across different machines. Clocks drift. NTP synchronization has latency. Two events that appear to happen at the "same time" may not have a clear causal relationship.

Two clock solutions:

LOGICAL CLOCKS (Lamport timestamps)
  Each event increments a counter.
  Send counter with every message.
  Receiver takes max(local, received) + 1.
  ✓ Establishes causal ordering
  ✗ Can't compare unrelated events

VECTOR CLOCKS
  N-dimensional counter (one per node).
  Can determine: A happened before B, B before A, or concurrent.
  Used by systems like DynamoDB, Riak.

Practical implication:
  Never rely on timestamps to determine event ordering.
  Use logical sequencing (event IDs, sequence numbers).

Idempotency: The Most Important Pattern

An operation is idempotent if performing it multiple times has the same effect as performing it once. This is the most important pattern in distributed systems because it makes retries safe.

Problem: Did my payment go through?
  Client sends charge request → network timeout → client doesn't know!
  If client retries → double charge!

Solution: Idempotency keys
  Client generates unique key before sending: "idem-key-abc-123"
  Server stores (idempotency_key, result) in database.
  
  Client: POST /payments { amount: 100, idempotencyKey: "idem-key-abc-123" }
  Server: 
    SELECT * FROM idempotency_keys WHERE key = 'idem-key-abc-123'
    → Not found: process payment, store result with key
    → Found: return cached result (no double charge!)

  Client can safely retry with same key.

// Implementation
async function createPayment(req: Request, res: Response) {
  const { idempotencyKey, amount, currency } = req.body;
  
  // Check for existing result
  const existing = await db.idempotencyKeys.findUnique({
    where: { key: idempotencyKey }
  });
  if (existing) return res.json(existing.result);
  
  // Process and store
  const payment = await stripe.charge({ amount, currency });
  await db.idempotencyKeys.create({
    data: { key: idempotencyKey, result: payment, expiresAt: ... }
  });
  return res.json(payment);
}

Delivery Guarantees

AT-MOST-ONCE DELIVERY
  Send once, never retry. Message may be lost.
  ✓ No duplicates
  ✗ Data loss possible
  Use case: Metrics, telemetry (losing a few points is fine)

AT-LEAST-ONCE DELIVERY
  Retry until acknowledged. Message will arrive eventually.
  ✓ No data loss
  ✗ Duplicates are possible
  Use case: Most messaging systems (Kafka, SQS default)
  Requires: Idempotent consumers!

EXACTLY-ONCE DELIVERY
  The holy grail. Often claimed, rarely truly achievable.
  ✓ No duplicates, no loss
  ✗ Extremely expensive to implement correctly
  Reality: Usually "at-least-once with deduplication" at the application layer

Retries and Timeouts

Every network call must have a timeout. Always.

// Bad: No timeout
const response = await fetch('https://api.stripe.com/charge');

// Good: Explicit timeout
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 5000);
const response = await fetch('https://api.stripe.com/charge', {
  signal: controller.signal
});
clearTimeout(timeout);

Retry strategy: Exponential backoff with jitter
  Attempt 1: wait 100ms
  Attempt 2: wait 200ms + random(0-100ms)
  Attempt 3: wait 400ms + random(0-200ms)
  Attempt 4: wait 800ms + random(0-400ms)
  Max attempts: 5

Why jitter?
  Without jitter, all retrying clients spike simultaneously.
  Jitter spreads the load, preventing "thundering herd."

Consistency Models

STRONG CONSISTENCY
  Every read sees the most recent write.
  Requires coordination between nodes.
  ✓ Simplest to reason about
  ✗ Higher latency, reduced availability
  Example: Single-leader PostgreSQL with synchronous replication

EVENTUAL CONSISTENCY
  Given no new writes, all replicas will eventually converge.
  Reads may return stale data temporarily.
  ✓ Higher availability, lower latency
  ✗ Application must handle stale reads
  Example: DNS, DynamoDB in eventual mode, CDN cache

CAUSAL CONSISTENCY
  If A caused B, every node sees A before B.
  Middle ground between strong and eventual.
  Example: Facebook's TAO, some Cassandra configurations

Distributed Locks

Problem: Two servers try to send the same email simultaneously.

Solution: Distributed lock
  Server A: acquire lock("send-email-order-123", ttl=30s)
  → Success: send email, release lock
  Server B: acquire lock("send-email-order-123")
  → Fail: lock held, skip or wait

Redis-based lock (Redlock algorithm):
  SET lock:order:123 server-a-uuid NX PX 30000
  → NX: only set if not exists
  → PX 30000: expire in 30 seconds (safety TTL)
  
  Release: only if we still own it
  if GET lock:order:123 == server-a-uuid:
    DEL lock:order:123

Caveats:
  - TTL must be longer than operation time
  - If process dies before release, TTL ensures eventual release
  - Redlock across multiple Redis nodes is controversial for high stakes

Mini Project (20-30 min)

Implement a basic consistent hashing ring.

import crypto from 'crypto';

class ConsistentHash {
  nodes: Map = new Map();
  keys: number[] = [];

  addNode(node: string) {
    const hash = this.hash(node);
    this.nodes.set(hash, node);
    this.keys.push(hash);
    this.keys.sort((a, b) => a - b);
  }

  getNode(key: string) {
    if (this.keys.length === 0) return null;
    const hash = this.hash(key);
    const target = this.keys.find(k => k >= hash) || this.keys[0];
    return this.nodes.get(target);
  }

  private hash(key: string) {
    return parseInt(crypto.createHash('md5').update(key).digest('hex').substring(0, 8), 16);
  }
}

Bigger Project (1-2 hours)

Implement a rudimentary distributed consensus algorithm (like simplified Raft or Paxos) in Node.js. Create 3 instances of your app that elect a leader and replicate a key-value store state across the nodes.

# Simplified Vector Clock
class VectorClock:
    def __init__(self, node_id):
        self.node = node_id
        self.clock = {node_id: 0}
    
    def increment(self):
        self.clock[self.node] += 1

Interview Questions

Revision Sheet

• Networks fail: Design for failure, not uptime. Timeouts on everything.
• Idempotency: Use idempotency keys. Make all operations safe to retry.
• At-least-once: Most systems provide this. Make consumers idempotent.
• Clocks lie: Use logical ordering, not timestamps, for event sequencing.
• Consistency: Strong → more coordination. Eventual → more availability. Pick based on your domain.