History

Early batch processing used simple cron jobs reading rows from a SQL database where `status = 'pending'`. As scale grew, polling the database became a bottleneck (the "thundering herd" problem and database lock contention). This led to the birth of dedicated Message Brokers (like RabbitMQ and Apache Kafka) and in-memory queue systems (like Redis/Resque) designed specifically for high-throughput, low-latency task distribution.

Mental Model (Analogy -> Reality)

The Analogy: A Busy Restaurant Kitchen

Imagine a restaurant where the waiter (API) takes your order and then goes into the kitchen to cook the food themselves, making you wait at the table without confirming they got the order. This is the naive synchronous approach.

In a proper restaurant, the waiter writes the order on a ticket (Job), places it on a rotating ticket wheel (Queue), and immediately returns to you saying "Your order is in!" (Acknowledgment). Meanwhile, specialized cooks (Workers) pull tickets off the wheel one by one and cook the food asynchronously.

The Reality

• Job: A JSON payload describing the work (e.g., { "type": "encode", "videoId": 123 }).
• Queue: A data structure (usually a FIFO queue stored in Redis or RabbitMQ) holding pending jobs.
• Worker: A separate Node.js process (or fleet of servers) whose only job is to pop items from the Queue and execute them.

Internal Working (Memory, stack, process, network, etc.)

When you enqueue a job using Redis (e.g., via BullMQ), the API process sends a network command (LPUSH or an atomic Lua script) to the Redis server. The API process then immediately returns the HTTP response.

In a completely different process—perhaps on a different server—a Worker uses a blocking pop network command (like BRPOP) to wait for new items in Redis. When a job arrives, Redis instantly sends the payload over the network to the Worker. The Worker parses the JSON, pushes function calls onto its call stack, and processes the job. If the Worker crashes, Redis (via mechanisms like BullMQ's active sets and locks) notices the lock expired and puts the job back in the queue for another worker.

Visual Explanation (ASCII diagrams)

Client       API Server (Producer)         Redis (Queue)           Worker Server (Consumer)
  |                 |                            |                            |
  |-- 1. HTTP POST->|                            |                            |
  |                 |-- 2. Push Job JSON ------->|                            |
  |<- 3. 202 OK ----|                            |-- 4. Pops Job JSON ------->|
  |                 |                            |                            |
  |                 |                            |<- 5. Acknowledges start ---|
  |                 |                            |                            |
  |                 |                            |    [ Processing... ]       |
  |                 |                            |                            |
  |                 |                            |<- 6. Marks completed ------|
  

Syntax

Using BullMQ with TypeScript and Redis:

import { Queue, Worker } from 'bullmq';

// 1. Create a Queue
const videoQueue = new Queue('videoProcessing', { connection: redisOptions });

// 2. Add a Job (Producer)
await videoQueue.add('encode', { videoId: 123, format: 'mp4' });

// 3. Process the Job (Consumer/Worker)
const worker = new Worker('videoProcessing', async job => {
  const { videoId, format } = job.data;
  await encodeVideo(videoId, format);
}, { connection: redisOptions });

Tiny Example

Let's build a reliable email sender.

import { Queue, Worker } from 'bullmq';

const emailQueue = new Queue('emails');

// API Endpoint
app.post('/signup', async (req, res) => {
  const user = await db.users.create(req.body);
  
  // Enqueue job, don't wait for email to send!
  await emailQueue.add('welcome-email', { email: user.email });
  
  res.status(202).send({ message: 'Signed up!' });
});

// Worker Process (run in a separate terminal)
new Worker('emails', async job => {
  console.log(`Sending email to ${job.data.email}...`);
  await sendGrid.send({ to: job.data.email, template: 'welcome' });
});

Walkthrough

In our Tiny Example, when a user hits /signup, the DB record is created fast (milliseconds). The email sending via a 3rd party API (SendGrid) might take 500ms to 2 seconds, and it might fail due to network blips.

By putting the job in the emails queue, the HTTP response goes back instantly. The user sees a snappy UI. The Worker process, which runs independently, picks up the job. If SendGrid is down, the Worker can fail the job, and the Queue system will automatically retry it later.

Break It

What happens if our worker function looks like this, and sendGrid.send fails?

new Worker('emails', async job => {
  await sendGrid.send({ to: job.data.email, template: 'welcome' }); // <-- THROWS ERROR
});

If there is no error handling, the worker crashes. By default, BullMQ catches the exception, marks the job as failed, and moves it to a failed set. But the email is never sent. The user never gets their welcome email.

Debug It

To fix this, we need Retries and Exponential Backoff.

// Producer side
await emailQueue.add('welcome-email', { email: user.email }, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 1000 // 1s, 2s, 4s, 8s, 16s
  }
});

Now, if the first attempt fails, it retries in 1 second. If that fails, it waits 2 seconds, then 4, etc. If it fails 5 times, it is permanently moved to a Dead-Letter Queue (DLQ)—a special holding area for jobs that have chronically failed, allowing human engineers to inspect them and figure out why.

Mini Project (20-30 min)

Build a Webhook Dispatcher

Design a system where users configure webhooks. When an event happens, you must send an HTTP POST to their configured URL. If their server is down, you must retry with backoff. If it fails after 10 tries, mark the webhook as "disabled" in the database.

Real Application Feature

Idempotent Payments Processing

When a background job retries, you have a massive risk: what if the job actually succeeded, but the network failed during the success acknowledgment, causing it to retry? For email, they get two emails (annoying). For payments, they get charged twice (catastrophic).

We solve this with Idempotency. An operation is idempotent if doing it multiple times has the same result as doing it once.

// Implementation for Async Backend Systems
console.log("Bigger project solution");

Production Implementation

In production, you don't run Workers in the same container as your API. You have:

• An Auto-Scaling Group of API web servers (handling HTTP).
• A managed Redis instance (ElastiCache) or RabbitMQ cluster.
• An Auto-Scaling Group of Worker instances. You scale these based on Queue Length (e.g., if >1000 jobs pending, spin up more workers).

History

Early batch processing used simple cron jobs reading rows from a SQL database where `status = 'pending'`. As scale grew, polling the database became a bottleneck (the "thundering herd" problem and database lock contention). This led to the birth of dedicated Message Brokers (like RabbitMQ and Apache Kafka) and in-memory queue systems (like Redis/Resque) designed specifically for high-throughput, low-latency task distribution.

Mental Model (Analogy -> Reality)

The Analogy: A Busy Restaurant Kitchen

Imagine a restaurant where the waiter (API) takes your order and then goes into the kitchen to cook the food themselves, making you wait at the table without confirming they got the order. This is the naive synchronous approach.

In a proper restaurant, the waiter writes the order on a ticket (Job), places it on a rotating ticket wheel (Queue), and immediately returns to you saying "Your order is in!" (Acknowledgment). Meanwhile, specialized cooks (Workers) pull tickets off the wheel one by one and cook the food asynchronously.

The Reality

• Job: A JSON payload describing the work (e.g., { "type": "encode", "videoId": 123 }).
• Queue: A data structure (usually a FIFO queue stored in Redis or RabbitMQ) holding pending jobs.
• Worker: A separate Node.js process (or fleet of servers) whose only job is to pop items from the Queue and execute them.

Internal Working (Memory, stack, process, network, etc.)

When you enqueue a job using Redis (e.g., via BullMQ), the API process sends a network command (LPUSH or an atomic Lua script) to the Redis server. The API process then immediately returns the HTTP response.

In a completely different process—perhaps on a different server—a Worker uses a blocking pop network command (like BRPOP) to wait for new items in Redis. When a job arrives, Redis instantly sends the payload over the network to the Worker. The Worker parses the JSON, pushes function calls onto its call stack, and processes the job. If the Worker crashes, Redis (via mechanisms like BullMQ's active sets and locks) notices the lock expired and puts the job back in the queue for another worker.

Visual Explanation (ASCII diagrams)

Client       API Server (Producer)         Redis (Queue)           Worker Server (Consumer)
  |                 |                            |                            |
  |-- 1. HTTP POST->|                            |                            |
  |                 |-- 2. Push Job JSON ------->|                            |
  |<- 3. 202 OK ----|                            |-- 4. Pops Job JSON ------->|
  |                 |                            |                            |
  |                 |                            |<- 5. Acknowledges start ---|
  |                 |                            |                            |
  |                 |                            |    [ Processing... ]       |
  |                 |                            |                            |
  |                 |                            |<- 6. Marks completed ------|
  

Syntax

Using BullMQ with TypeScript and Redis:

import { Queue, Worker } from 'bullmq';

// 1. Create a Queue
const videoQueue = new Queue('videoProcessing', { connection: redisOptions });

// 2. Add a Job (Producer)
await videoQueue.add('encode', { videoId: 123, format: 'mp4' });

// 3. Process the Job (Consumer/Worker)
const worker = new Worker('videoProcessing', async job => {
  const { videoId, format } = job.data;
  await encodeVideo(videoId, format);
}, { connection: redisOptions });

Tiny Example

Let's build a reliable email sender.

import { Queue, Worker } from 'bullmq';

const emailQueue = new Queue('emails');

// API Endpoint
app.post('/signup', async (req, res) => {
  const user = await db.users.create(req.body);
  
  // Enqueue job, don't wait for email to send!
  await emailQueue.add('welcome-email', { email: user.email });
  
  res.status(202).send({ message: 'Signed up!' });
});

// Worker Process (run in a separate terminal)
new Worker('emails', async job => {
  console.log(`Sending email to ${job.data.email}...`);
  await sendGrid.send({ to: job.data.email, template: 'welcome' });
});

Walkthrough

In our Tiny Example, when a user hits /signup, the DB record is created fast (milliseconds). The email sending via a 3rd party API (SendGrid) might take 500ms to 2 seconds, and it might fail due to network blips.

By putting the job in the emails queue, the HTTP response goes back instantly. The user sees a snappy UI. The Worker process, which runs independently, picks up the job. If SendGrid is down, the Worker can fail the job, and the Queue system will automatically retry it later.

Break It

What happens if our worker function looks like this, and sendGrid.send fails?

new Worker('emails', async job => {
  await sendGrid.send({ to: job.data.email, template: 'welcome' }); // <-- THROWS ERROR
});

If there is no error handling, the worker crashes. By default, BullMQ catches the exception, marks the job as failed, and moves it to a failed set. But the email is never sent. The user never gets their welcome email.

Debug It

To fix this, we need Retries and Exponential Backoff.

// Producer side
await emailQueue.add('welcome-email', { email: user.email }, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 1000 // 1s, 2s, 4s, 8s, 16s
  }
});

Now, if the first attempt fails, it retries in 1 second. If that fails, it waits 2 seconds, then 4, etc. If it fails 5 times, it is permanently moved to a Dead-Letter Queue (DLQ)—a special holding area for jobs that have chronically failed, allowing human engineers to inspect them and figure out why.

Mini Project (20-30 min)

Build a Webhook Dispatcher

Design a system where users configure webhooks. When an event happens, you must send an HTTP POST to their configured URL. If their server is down, you must retry with backoff. If it fails after 10 tries, mark the webhook as "disabled" in the database.

Real Application Feature

Idempotent Payments Processing

When a background job retries, you have a massive risk: what if the job actually succeeded, but the network failed during the success acknowledgment, causing it to retry? For email, they get two emails (annoying). For payments, they get charged twice (catastrophic).

We solve this with Idempotency. An operation is idempotent if doing it multiple times has the same result as doing it once.

// Implementation for Async Backend Systems
console.log("Bigger project solution");

Production Usage

We use tools like Bull-Board (a UI for BullMQ) to monitor queues. Engineers look at the DLQ daily to find systemic bugs. We also use Event-Driven Architecture (like Kafka) when we need multiple different systems to react to one event (Pub/Sub), rather than a single queue.

History

Early batch processing used simple cron jobs reading rows from a SQL database where `status = 'pending'`. As scale grew, polling the database became a bottleneck (the "thundering herd" problem and database lock contention). This led to the birth of dedicated Message Brokers (like RabbitMQ and Apache Kafka) and in-memory queue systems (like Redis/Resque) designed specifically for high-throughput, low-latency task distribution.

Mental Model (Analogy -> Reality)

The Analogy: A Busy Restaurant Kitchen

Imagine a restaurant where the waiter (API) takes your order and then goes into the kitchen to cook the food themselves, making you wait at the table without confirming they got the order. This is the naive synchronous approach.

In a proper restaurant, the waiter writes the order on a ticket (Job), places it on a rotating ticket wheel (Queue), and immediately returns to you saying "Your order is in!" (Acknowledgment). Meanwhile, specialized cooks (Workers) pull tickets off the wheel one by one and cook the food asynchronously.

The Reality

• Job: A JSON payload describing the work (e.g., { "type": "encode", "videoId": 123 }).
• Queue: A data structure (usually a FIFO queue stored in Redis or RabbitMQ) holding pending jobs.
• Worker: A separate Node.js process (or fleet of servers) whose only job is to pop items from the Queue and execute them.

Internal Working (Memory, stack, process, network, etc.)

When you enqueue a job using Redis (e.g., via BullMQ), the API process sends a network command (LPUSH or an atomic Lua script) to the Redis server. The API process then immediately returns the HTTP response.

In a completely different process—perhaps on a different server—a Worker uses a blocking pop network command (like BRPOP) to wait for new items in Redis. When a job arrives, Redis instantly sends the payload over the network to the Worker. The Worker parses the JSON, pushes function calls onto its call stack, and processes the job. If the Worker crashes, Redis (via mechanisms like BullMQ's active sets and locks) notices the lock expired and puts the job back in the queue for another worker.

Visual Explanation (ASCII diagrams)

Client       API Server (Producer)         Redis (Queue)           Worker Server (Consumer)
  |                 |                            |                            |
  |-- 1. HTTP POST->|                            |                            |
  |                 |-- 2. Push Job JSON ------->|                            |
  |<- 3. 202 OK ----|                            |-- 4. Pops Job JSON ------->|
  |                 |                            |                            |
  |                 |                            |<- 5. Acknowledges start ---|
  |                 |                            |                            |
  |                 |                            |    [ Processing... ]       |
  |                 |                            |                            |
  |                 |                            |<- 6. Marks completed ------|
  

Syntax

Using BullMQ with TypeScript and Redis:

import { Queue, Worker } from 'bullmq';

// 1. Create a Queue
const videoQueue = new Queue('videoProcessing', { connection: redisOptions });

// 2. Add a Job (Producer)
await videoQueue.add('encode', { videoId: 123, format: 'mp4' });

// 3. Process the Job (Consumer/Worker)
const worker = new Worker('videoProcessing', async job => {
  const { videoId, format } = job.data;
  await encodeVideo(videoId, format);
}, { connection: redisOptions });

Tiny Example

Let's build a reliable email sender.

import { Queue, Worker } from 'bullmq';

const emailQueue = new Queue('emails');

// API Endpoint
app.post('/signup', async (req, res) => {
  const user = await db.users.create(req.body);
  
  // Enqueue job, don't wait for email to send!
  await emailQueue.add('welcome-email', { email: user.email });
  
  res.status(202).send({ message: 'Signed up!' });
});

// Worker Process (run in a separate terminal)
new Worker('emails', async job => {
  console.log(`Sending email to ${job.data.email}...`);
  await sendGrid.send({ to: job.data.email, template: 'welcome' });
});

Walkthrough

In our Tiny Example, when a user hits /signup, the DB record is created fast (milliseconds). The email sending via a 3rd party API (SendGrid) might take 500ms to 2 seconds, and it might fail due to network blips.

By putting the job in the emails queue, the HTTP response goes back instantly. The user sees a snappy UI. The Worker process, which runs independently, picks up the job. If SendGrid is down, the Worker can fail the job, and the Queue system will automatically retry it later.

Break It

What happens if our worker function looks like this, and sendGrid.send fails?

new Worker('emails', async job => {
  await sendGrid.send({ to: job.data.email, template: 'welcome' }); // <-- THROWS ERROR
});

If there is no error handling, the worker crashes. By default, BullMQ catches the exception, marks the job as failed, and moves it to a failed set. But the email is never sent. The user never gets their welcome email.

Debug It

To fix this, we need Retries and Exponential Backoff.

// Producer side
await emailQueue.add('welcome-email', { email: user.email }, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 1000 // 1s, 2s, 4s, 8s, 16s
  }
});

Now, if the first attempt fails, it retries in 1 second. If that fails, it waits 2 seconds, then 4, etc. If it fails 5 times, it is permanently moved to a Dead-Letter Queue (DLQ)—a special holding area for jobs that have chronically failed, allowing human engineers to inspect them and figure out why.

Mini Project (20-30 min)

Build a Webhook Dispatcher

Design a system where users configure webhooks. When an event happens, you must send an HTTP POST to their configured URL. If their server is down, you must retry with backoff. If it fails after 10 tries, mark the webhook as "disabled" in the database.

Real Application Feature

Idempotent Payments Processing

When a background job retries, you have a massive risk: what if the job actually succeeded, but the network failed during the success acknowledgment, causing it to retry? For email, they get two emails (annoying). For payments, they get charged twice (catastrophic).

We solve this with Idempotency. An operation is idempotent if doing it multiple times has the same result as doing it once.

// Implementation for Async Backend Systems
console.log("Bigger project solution");

Performance

Queues are extremely fast. Redis can handle hundreds of thousands of queue operations per second. The bottleneck is always your Worker's execution speed. To improve throughput, increase worker concurrency (new Worker(..., { concurrency: 50 })), but ensure you don't exhaust your database connection pool.

History

Early batch processing used simple cron jobs reading rows from a SQL database where `status = 'pending'`. As scale grew, polling the database became a bottleneck (the "thundering herd" problem and database lock contention). This led to the birth of dedicated Message Brokers (like RabbitMQ and Apache Kafka) and in-memory queue systems (like Redis/Resque) designed specifically for high-throughput, low-latency task distribution.

Mental Model (Analogy -> Reality)

The Analogy: A Busy Restaurant Kitchen

Imagine a restaurant where the waiter (API) takes your order and then goes into the kitchen to cook the food themselves, making you wait at the table without confirming they got the order. This is the naive synchronous approach.

In a proper restaurant, the waiter writes the order on a ticket (Job), places it on a rotating ticket wheel (Queue), and immediately returns to you saying "Your order is in!" (Acknowledgment). Meanwhile, specialized cooks (Workers) pull tickets off the wheel one by one and cook the food asynchronously.

The Reality

• Job: A JSON payload describing the work (e.g., { "type": "encode", "videoId": 123 }).
• Queue: A data structure (usually a FIFO queue stored in Redis or RabbitMQ) holding pending jobs.
• Worker: A separate Node.js process (or fleet of servers) whose only job is to pop items from the Queue and execute them.

Internal Working (Memory, stack, process, network, etc.)

When you enqueue a job using Redis (e.g., via BullMQ), the API process sends a network command (LPUSH or an atomic Lua script) to the Redis server. The API process then immediately returns the HTTP response.

In a completely different process—perhaps on a different server—a Worker uses a blocking pop network command (like BRPOP) to wait for new items in Redis. When a job arrives, Redis instantly sends the payload over the network to the Worker. The Worker parses the JSON, pushes function calls onto its call stack, and processes the job. If the Worker crashes, Redis (via mechanisms like BullMQ's active sets and locks) notices the lock expired and puts the job back in the queue for another worker.

Visual Explanation (ASCII diagrams)

Client       API Server (Producer)         Redis (Queue)           Worker Server (Consumer)
  |                 |                            |                            |
  |-- 1. HTTP POST->|                            |                            |
  |                 |-- 2. Push Job JSON ------->|                            |
  |<- 3. 202 OK ----|                            |-- 4. Pops Job JSON ------->|
  |                 |                            |                            |
  |                 |                            |<- 5. Acknowledges start ---|
  |                 |                            |                            |
  |                 |                            |    [ Processing... ]       |
  |                 |                            |                            |
  |                 |                            |<- 6. Marks completed ------|
  

Syntax

Using BullMQ with TypeScript and Redis:

import { Queue, Worker } from 'bullmq';

// 1. Create a Queue
const videoQueue = new Queue('videoProcessing', { connection: redisOptions });

// 2. Add a Job (Producer)
await videoQueue.add('encode', { videoId: 123, format: 'mp4' });

// 3. Process the Job (Consumer/Worker)
const worker = new Worker('videoProcessing', async job => {
  const { videoId, format } = job.data;
  await encodeVideo(videoId, format);
}, { connection: redisOptions });

Tiny Example

Let's build a reliable email sender.

import { Queue, Worker } from 'bullmq';

const emailQueue = new Queue('emails');

// API Endpoint
app.post('/signup', async (req, res) => {
  const user = await db.users.create(req.body);
  
  // Enqueue job, don't wait for email to send!
  await emailQueue.add('welcome-email', { email: user.email });
  
  res.status(202).send({ message: 'Signed up!' });
});

// Worker Process (run in a separate terminal)
new Worker('emails', async job => {
  console.log(`Sending email to ${job.data.email}...`);
  await sendGrid.send({ to: job.data.email, template: 'welcome' });
});

Walkthrough

In our Tiny Example, when a user hits /signup, the DB record is created fast (milliseconds). The email sending via a 3rd party API (SendGrid) might take 500ms to 2 seconds, and it might fail due to network blips.

By putting the job in the emails queue, the HTTP response goes back instantly. The user sees a snappy UI. The Worker process, which runs independently, picks up the job. If SendGrid is down, the Worker can fail the job, and the Queue system will automatically retry it later.

Break It

What happens if our worker function looks like this, and sendGrid.send fails?

new Worker('emails', async job => {
  await sendGrid.send({ to: job.data.email, template: 'welcome' }); // <-- THROWS ERROR
});

If there is no error handling, the worker crashes. By default, BullMQ catches the exception, marks the job as failed, and moves it to a failed set. But the email is never sent. The user never gets their welcome email.

Debug It

To fix this, we need Retries and Exponential Backoff.

// Producer side
await emailQueue.add('welcome-email', { email: user.email }, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 1000 // 1s, 2s, 4s, 8s, 16s
  }
});

Now, if the first attempt fails, it retries in 1 second. If that fails, it waits 2 seconds, then 4, etc. If it fails 5 times, it is permanently moved to a Dead-Letter Queue (DLQ)—a special holding area for jobs that have chronically failed, allowing human engineers to inspect them and figure out why.

Mini Project (20-30 min)

Build a Webhook Dispatcher

Design a system where users configure webhooks. When an event happens, you must send an HTTP POST to their configured URL. If their server is down, you must retry with backoff. If it fails after 10 tries, mark the webhook as "disabled" in the database.

Real Application Feature

Idempotent Payments Processing

When a background job retries, you have a massive risk: what if the job actually succeeded, but the network failed during the success acknowledgment, causing it to retry? For email, they get two emails (annoying). For payments, they get charged twice (catastrophic).

We solve this with Idempotency. An operation is idempotent if doing it multiple times has the same result as doing it once.

// Implementation for Async Backend Systems
console.log("Bigger project solution");

Best Practices

• Keep Jobs Small: Pass IDs in job data, not massive objects. Let the worker fetch the latest data from the DB.
• Idempotency Everywhere: Always assume jobs will be run at least twice.
• Graceful Shutdown: Listen for SIGTERM and tell workers to finish current jobs before exiting so they aren't killed mid-processing.

History

Early batch processing used simple cron jobs reading rows from a SQL database where `status = 'pending'`. As scale grew, polling the database became a bottleneck (the "thundering herd" problem and database lock contention). This led to the birth of dedicated Message Brokers (like RabbitMQ and Apache Kafka) and in-memory queue systems (like Redis/Resque) designed specifically for high-throughput, low-latency task distribution.

Mental Model (Analogy -> Reality)

The Analogy: A Busy Restaurant Kitchen

Imagine a restaurant where the waiter (API) takes your order and then goes into the kitchen to cook the food themselves, making you wait at the table without confirming they got the order. This is the naive synchronous approach.

In a proper restaurant, the waiter writes the order on a ticket (Job), places it on a rotating ticket wheel (Queue), and immediately returns to you saying "Your order is in!" (Acknowledgment). Meanwhile, specialized cooks (Workers) pull tickets off the wheel one by one and cook the food asynchronously.

The Reality

• Job: A JSON payload describing the work (e.g., { "type": "encode", "videoId": 123 }).
• Queue: A data structure (usually a FIFO queue stored in Redis or RabbitMQ) holding pending jobs.
• Worker: A separate Node.js process (or fleet of servers) whose only job is to pop items from the Queue and execute them.

Internal Working (Memory, stack, process, network, etc.)

When you enqueue a job using Redis (e.g., via BullMQ), the API process sends a network command (LPUSH or an atomic Lua script) to the Redis server. The API process then immediately returns the HTTP response.

In a completely different process—perhaps on a different server—a Worker uses a blocking pop network command (like BRPOP) to wait for new items in Redis. When a job arrives, Redis instantly sends the payload over the network to the Worker. The Worker parses the JSON, pushes function calls onto its call stack, and processes the job. If the Worker crashes, Redis (via mechanisms like BullMQ's active sets and locks) notices the lock expired and puts the job back in the queue for another worker.

Visual Explanation (ASCII diagrams)

Client       API Server (Producer)         Redis (Queue)           Worker Server (Consumer)
  |                 |                            |                            |
  |-- 1. HTTP POST->|                            |                            |
  |                 |-- 2. Push Job JSON ------->|                            |
  |<- 3. 202 OK ----|                            |-- 4. Pops Job JSON ------->|
  |                 |                            |                            |
  |                 |                            |<- 5. Acknowledges start ---|
  |                 |                            |                            |
  |                 |                            |    [ Processing... ]       |
  |                 |                            |                            |
  |                 |                            |<- 6. Marks completed ------|
  

Syntax

Using BullMQ with TypeScript and Redis:

import { Queue, Worker } from 'bullmq';

// 1. Create a Queue
const videoQueue = new Queue('videoProcessing', { connection: redisOptions });

// 2. Add a Job (Producer)
await videoQueue.add('encode', { videoId: 123, format: 'mp4' });

// 3. Process the Job (Consumer/Worker)
const worker = new Worker('videoProcessing', async job => {
  const { videoId, format } = job.data;
  await encodeVideo(videoId, format);
}, { connection: redisOptions });

Tiny Example

Let's build a reliable email sender.

import { Queue, Worker } from 'bullmq';

const emailQueue = new Queue('emails');

// API Endpoint
app.post('/signup', async (req, res) => {
  const user = await db.users.create(req.body);
  
  // Enqueue job, don't wait for email to send!
  await emailQueue.add('welcome-email', { email: user.email });
  
  res.status(202).send({ message: 'Signed up!' });
});

// Worker Process (run in a separate terminal)
new Worker('emails', async job => {
  console.log(`Sending email to ${job.data.email}...`);
  await sendGrid.send({ to: job.data.email, template: 'welcome' });
});

Walkthrough

In our Tiny Example, when a user hits /signup, the DB record is created fast (milliseconds). The email sending via a 3rd party API (SendGrid) might take 500ms to 2 seconds, and it might fail due to network blips.

By putting the job in the emails queue, the HTTP response goes back instantly. The user sees a snappy UI. The Worker process, which runs independently, picks up the job. If SendGrid is down, the Worker can fail the job, and the Queue system will automatically retry it later.

Break It

What happens if our worker function looks like this, and sendGrid.send fails?

new Worker('emails', async job => {
  await sendGrid.send({ to: job.data.email, template: 'welcome' }); // <-- THROWS ERROR
});

If there is no error handling, the worker crashes. By default, BullMQ catches the exception, marks the job as failed, and moves it to a failed set. But the email is never sent. The user never gets their welcome email.

Debug It

To fix this, we need Retries and Exponential Backoff.

// Producer side
await emailQueue.add('welcome-email', { email: user.email }, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 1000 // 1s, 2s, 4s, 8s, 16s
  }
});

Now, if the first attempt fails, it retries in 1 second. If that fails, it waits 2 seconds, then 4, etc. If it fails 5 times, it is permanently moved to a Dead-Letter Queue (DLQ)—a special holding area for jobs that have chronically failed, allowing human engineers to inspect them and figure out why.

Mini Project (20-30 min)

Build a Webhook Dispatcher

Design a system where users configure webhooks. When an event happens, you must send an HTTP POST to their configured URL. If their server is down, you must retry with backoff. If it fails after 10 tries, mark the webhook as "disabled" in the database.

Real Application Feature

Idempotent Payments Processing

When a background job retries, you have a massive risk: what if the job actually succeeded, but the network failed during the success acknowledgment, causing it to retry? For email, they get two emails (annoying). For payments, they get charged twice (catastrophic).

We solve this with Idempotency. An operation is idempotent if doing it multiple times has the same result as doing it once.

// Implementation for Async Backend Systems
console.log("Bigger project solution");

Interview Questions (Easy, Medium, Hard, Senior)

History

Early batch processing used simple cron jobs reading rows from a SQL database where `status = 'pending'`. As scale grew, polling the database became a bottleneck (the "thundering herd" problem and database lock contention). This led to the birth of dedicated Message Brokers (like RabbitMQ and Apache Kafka) and in-memory queue systems (like Redis/Resque) designed specifically for high-throughput, low-latency task distribution.

Mental Model (Analogy -> Reality)

The Analogy: A Busy Restaurant Kitchen

Imagine a restaurant where the waiter (API) takes your order and then goes into the kitchen to cook the food themselves, making you wait at the table without confirming they got the order. This is the naive synchronous approach.

In a proper restaurant, the waiter writes the order on a ticket (Job), places it on a rotating ticket wheel (Queue), and immediately returns to you saying "Your order is in!" (Acknowledgment). Meanwhile, specialized cooks (Workers) pull tickets off the wheel one by one and cook the food asynchronously.

The Reality

• Job: A JSON payload describing the work (e.g., { "type": "encode", "videoId": 123 }).
• Queue: A data structure (usually a FIFO queue stored in Redis or RabbitMQ) holding pending jobs.
• Worker: A separate Node.js process (or fleet of servers) whose only job is to pop items from the Queue and execute them.

Internal Working (Memory, stack, process, network, etc.)

When you enqueue a job using Redis (e.g., via BullMQ), the API process sends a network command (LPUSH or an atomic Lua script) to the Redis server. The API process then immediately returns the HTTP response.

In a completely different process—perhaps on a different server—a Worker uses a blocking pop network command (like BRPOP) to wait for new items in Redis. When a job arrives, Redis instantly sends the payload over the network to the Worker. The Worker parses the JSON, pushes function calls onto its call stack, and processes the job. If the Worker crashes, Redis (via mechanisms like BullMQ's active sets and locks) notices the lock expired and puts the job back in the queue for another worker.

Visual Explanation (ASCII diagrams)

Client       API Server (Producer)         Redis (Queue)           Worker Server (Consumer)
  |                 |                            |                            |
  |-- 1. HTTP POST->|                            |                            |
  |                 |-- 2. Push Job JSON ------->|                            |
  |<- 3. 202 OK ----|                            |-- 4. Pops Job JSON ------->|
  |                 |                            |                            |
  |                 |                            |<- 5. Acknowledges start ---|
  |                 |                            |                            |
  |                 |                            |    [ Processing... ]       |
  |                 |                            |                            |
  |                 |                            |<- 6. Marks completed ------|
  

Syntax

Using BullMQ with TypeScript and Redis:

import { Queue, Worker } from 'bullmq';

// 1. Create a Queue
const videoQueue = new Queue('videoProcessing', { connection: redisOptions });

// 2. Add a Job (Producer)
await videoQueue.add('encode', { videoId: 123, format: 'mp4' });

// 3. Process the Job (Consumer/Worker)
const worker = new Worker('videoProcessing', async job => {
  const { videoId, format } = job.data;
  await encodeVideo(videoId, format);
}, { connection: redisOptions });

Tiny Example

Let's build a reliable email sender.

import { Queue, Worker } from 'bullmq';

const emailQueue = new Queue('emails');

// API Endpoint
app.post('/signup', async (req, res) => {
  const user = await db.users.create(req.body);
  
  // Enqueue job, don't wait for email to send!
  await emailQueue.add('welcome-email', { email: user.email });
  
  res.status(202).send({ message: 'Signed up!' });
});

// Worker Process (run in a separate terminal)
new Worker('emails', async job => {
  console.log(`Sending email to ${job.data.email}...`);
  await sendGrid.send({ to: job.data.email, template: 'welcome' });
});

Walkthrough

In our Tiny Example, when a user hits /signup, the DB record is created fast (milliseconds). The email sending via a 3rd party API (SendGrid) might take 500ms to 2 seconds, and it might fail due to network blips.

By putting the job in the emails queue, the HTTP response goes back instantly. The user sees a snappy UI. The Worker process, which runs independently, picks up the job. If SendGrid is down, the Worker can fail the job, and the Queue system will automatically retry it later.

Break It

What happens if our worker function looks like this, and sendGrid.send fails?

new Worker('emails', async job => {
  await sendGrid.send({ to: job.data.email, template: 'welcome' }); // <-- THROWS ERROR
});

If there is no error handling, the worker crashes. By default, BullMQ catches the exception, marks the job as failed, and moves it to a failed set. But the email is never sent. The user never gets their welcome email.

Debug It

To fix this, we need Retries and Exponential Backoff.

// Producer side
await emailQueue.add('welcome-email', { email: user.email }, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 1000 // 1s, 2s, 4s, 8s, 16s
  }
});

Now, if the first attempt fails, it retries in 1 second. If that fails, it waits 2 seconds, then 4, etc. If it fails 5 times, it is permanently moved to a Dead-Letter Queue (DLQ)—a special holding area for jobs that have chronically failed, allowing human engineers to inspect them and figure out why.

Mini Project (20-30 min)

Build a Webhook Dispatcher

Design a system where users configure webhooks. When an event happens, you must send an HTTP POST to their configured URL. If their server is down, you must retry with backoff. If it fails after 10 tries, mark the webhook as "disabled" in the database.

Real Application Feature

Idempotent Payments Processing

When a background job retries, you have a massive risk: what if the job actually succeeded, but the network failed during the success acknowledgment, causing it to retry? For email, they get two emails (annoying). For payments, they get charged twice (catastrophic).

We solve this with Idempotency. An operation is idempotent if doing it multiple times has the same result as doing it once.

// Implementation for Async Backend Systems
console.log("Bigger project solution");

Engineering Challenge

Implement a Scheduled Job System. A user wants an email sent exactly 30 days after they sign up. Design a system that doesn't just sleep for 30 days in memory, but reliably triggers a job 30 days from now, surviving server restarts.

History

Early batch processing used simple cron jobs reading rows from a SQL database where `status = 'pending'`. As scale grew, polling the database became a bottleneck (the "thundering herd" problem and database lock contention). This led to the birth of dedicated Message Brokers (like RabbitMQ and Apache Kafka) and in-memory queue systems (like Redis/Resque) designed specifically for high-throughput, low-latency task distribution.

Mental Model (Analogy -> Reality)

The Analogy: A Busy Restaurant Kitchen

Imagine a restaurant where the waiter (API) takes your order and then goes into the kitchen to cook the food themselves, making you wait at the table without confirming they got the order. This is the naive synchronous approach.

In a proper restaurant, the waiter writes the order on a ticket (Job), places it on a rotating ticket wheel (Queue), and immediately returns to you saying "Your order is in!" (Acknowledgment). Meanwhile, specialized cooks (Workers) pull tickets off the wheel one by one and cook the food asynchronously.

The Reality

• Job: A JSON payload describing the work (e.g., { "type": "encode", "videoId": 123 }).
• Queue: A data structure (usually a FIFO queue stored in Redis or RabbitMQ) holding pending jobs.
• Worker: A separate Node.js process (or fleet of servers) whose only job is to pop items from the Queue and execute them.

Internal Working (Memory, stack, process, network, etc.)

When you enqueue a job using Redis (e.g., via BullMQ), the API process sends a network command (LPUSH or an atomic Lua script) to the Redis server. The API process then immediately returns the HTTP response.

In a completely different process—perhaps on a different server—a Worker uses a blocking pop network command (like BRPOP) to wait for new items in Redis. When a job arrives, Redis instantly sends the payload over the network to the Worker. The Worker parses the JSON, pushes function calls onto its call stack, and processes the job. If the Worker crashes, Redis (via mechanisms like BullMQ's active sets and locks) notices the lock expired and puts the job back in the queue for another worker.

Visual Explanation (ASCII diagrams)

Client       API Server (Producer)         Redis (Queue)           Worker Server (Consumer)
  |                 |                            |                            |
  |-- 1. HTTP POST->|                            |                            |
  |                 |-- 2. Push Job JSON ------->|                            |
  |<- 3. 202 OK ----|                            |-- 4. Pops Job JSON ------->|
  |                 |                            |                            |
  |                 |                            |<- 5. Acknowledges start ---|
  |                 |                            |                            |
  |                 |                            |    [ Processing... ]       |
  |                 |                            |                            |
  |                 |                            |<- 6. Marks completed ------|
  

Syntax

Using BullMQ with TypeScript and Redis:

import { Queue, Worker } from 'bullmq';

// 1. Create a Queue
const videoQueue = new Queue('videoProcessing', { connection: redisOptions });

// 2. Add a Job (Producer)
await videoQueue.add('encode', { videoId: 123, format: 'mp4' });

// 3. Process the Job (Consumer/Worker)
const worker = new Worker('videoProcessing', async job => {
  const { videoId, format } = job.data;
  await encodeVideo(videoId, format);
}, { connection: redisOptions });

Tiny Example

Let's build a reliable email sender.

import { Queue, Worker } from 'bullmq';

const emailQueue = new Queue('emails');

// API Endpoint
app.post('/signup', async (req, res) => {
  const user = await db.users.create(req.body);
  
  // Enqueue job, don't wait for email to send!
  await emailQueue.add('welcome-email', { email: user.email });
  
  res.status(202).send({ message: 'Signed up!' });
});

// Worker Process (run in a separate terminal)
new Worker('emails', async job => {
  console.log(`Sending email to ${job.data.email}...`);
  await sendGrid.send({ to: job.data.email, template: 'welcome' });
});

Walkthrough

In our Tiny Example, when a user hits /signup, the DB record is created fast (milliseconds). The email sending via a 3rd party API (SendGrid) might take 500ms to 2 seconds, and it might fail due to network blips.

By putting the job in the emails queue, the HTTP response goes back instantly. The user sees a snappy UI. The Worker process, which runs independently, picks up the job. If SendGrid is down, the Worker can fail the job, and the Queue system will automatically retry it later.

Break It

What happens if our worker function looks like this, and sendGrid.send fails?

new Worker('emails', async job => {
  await sendGrid.send({ to: job.data.email, template: 'welcome' }); // <-- THROWS ERROR
});

If there is no error handling, the worker crashes. By default, BullMQ catches the exception, marks the job as failed, and moves it to a failed set. But the email is never sent. The user never gets their welcome email.

Debug It

To fix this, we need Retries and Exponential Backoff.

// Producer side
await emailQueue.add('welcome-email', { email: user.email }, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 1000 // 1s, 2s, 4s, 8s, 16s
  }
});

Now, if the first attempt fails, it retries in 1 second. If that fails, it waits 2 seconds, then 4, etc. If it fails 5 times, it is permanently moved to a Dead-Letter Queue (DLQ)—a special holding area for jobs that have chronically failed, allowing human engineers to inspect them and figure out why.

Mini Project (20-30 min)

Build a Webhook Dispatcher

Design a system where users configure webhooks. When an event happens, you must send an HTTP POST to their configured URL. If their server is down, you must retry with backoff. If it fails after 10 tries, mark the webhook as "disabled" in the database.

Real Application Feature

Idempotent Payments Processing

When a background job retries, you have a massive risk: what if the job actually succeeded, but the network failed during the success acknowledgment, causing it to retry? For email, they get two emails (annoying). For payments, they get charged twice (catastrophic).

We solve this with Idempotency. An operation is idempotent if doing it multiple times has the same result as doing it once.

// Implementation for Async Backend Systems
console.log("Bigger project solution");

Revision Sheet

• Queue: Holding area for tasks.
• Worker: Process executing tasks.
• Job: Data payload defining the task.
• Backoff: Increasing delay between retries.
• DLQ: Graveyard for failed jobs.
• Idempotency: Safe to run multiple times.
• Eventual Consistency: Data will eventually be accurate across all systems, but might not be immediately after a write.

History

Early batch processing used simple cron jobs reading rows from a SQL database where `status = 'pending'`. As scale grew, polling the database became a bottleneck (the "thundering herd" problem and database lock contention). This led to the birth of dedicated Message Brokers (like RabbitMQ and Apache Kafka) and in-memory queue systems (like Redis/Resque) designed specifically for high-throughput, low-latency task distribution.

Mental Model (Analogy -> Reality)

The Analogy: A Busy Restaurant Kitchen

Imagine a restaurant where the waiter (API) takes your order and then goes into the kitchen to cook the food themselves, making you wait at the table without confirming they got the order. This is the naive synchronous approach.

In a proper restaurant, the waiter writes the order on a ticket (Job), places it on a rotating ticket wheel (Queue), and immediately returns to you saying "Your order is in!" (Acknowledgment). Meanwhile, specialized cooks (Workers) pull tickets off the wheel one by one and cook the food asynchronously.

The Reality

• Job: A JSON payload describing the work (e.g., { "type": "encode", "videoId": 123 }).
• Queue: A data structure (usually a FIFO queue stored in Redis or RabbitMQ) holding pending jobs.
• Worker: A separate Node.js process (or fleet of servers) whose only job is to pop items from the Queue and execute them.

Internal Working (Memory, stack, process, network, etc.)

When you enqueue a job using Redis (e.g., via BullMQ), the API process sends a network command (LPUSH or an atomic Lua script) to the Redis server. The API process then immediately returns the HTTP response.

In a completely different process—perhaps on a different server—a Worker uses a blocking pop network command (like BRPOP) to wait for new items in Redis. When a job arrives, Redis instantly sends the payload over the network to the Worker. The Worker parses the JSON, pushes function calls onto its call stack, and processes the job. If the Worker crashes, Redis (via mechanisms like BullMQ's active sets and locks) notices the lock expired and puts the job back in the queue for another worker.

Visual Explanation (ASCII diagrams)

Client       API Server (Producer)         Redis (Queue)           Worker Server (Consumer)
  |                 |                            |                            |
  |-- 1. HTTP POST->|                            |                            |
  |                 |-- 2. Push Job JSON ------->|                            |
  |<- 3. 202 OK ----|                            |-- 4. Pops Job JSON ------->|
  |                 |                            |                            |
  |                 |                            |<- 5. Acknowledges start ---|
  |                 |                            |                            |
  |                 |                            |    [ Processing... ]       |
  |                 |                            |                            |
  |                 |                            |<- 6. Marks completed ------|
  

Syntax

Using BullMQ with TypeScript and Redis:

import { Queue, Worker } from 'bullmq';

// 1. Create a Queue
const videoQueue = new Queue('videoProcessing', { connection: redisOptions });

// 2. Add a Job (Producer)
await videoQueue.add('encode', { videoId: 123, format: 'mp4' });

// 3. Process the Job (Consumer/Worker)
const worker = new Worker('videoProcessing', async job => {
  const { videoId, format } = job.data;
  await encodeVideo(videoId, format);
}, { connection: redisOptions });

Tiny Example

Let's build a reliable email sender.

import { Queue, Worker } from 'bullmq';

const emailQueue = new Queue('emails');

// API Endpoint
app.post('/signup', async (req, res) => {
  const user = await db.users.create(req.body);
  
  // Enqueue job, don't wait for email to send!
  await emailQueue.add('welcome-email', { email: user.email });
  
  res.status(202).send({ message: 'Signed up!' });
});

// Worker Process (run in a separate terminal)
new Worker('emails', async job => {
  console.log(`Sending email to ${job.data.email}...`);
  await sendGrid.send({ to: job.data.email, template: 'welcome' });
});

Walkthrough

In our Tiny Example, when a user hits /signup, the DB record is created fast (milliseconds). The email sending via a 3rd party API (SendGrid) might take 500ms to 2 seconds, and it might fail due to network blips.

By putting the job in the emails queue, the HTTP response goes back instantly. The user sees a snappy UI. The Worker process, which runs independently, picks up the job. If SendGrid is down, the Worker can fail the job, and the Queue system will automatically retry it later.

Break It

What happens if our worker function looks like this, and sendGrid.send fails?

new Worker('emails', async job => {
  await sendGrid.send({ to: job.data.email, template: 'welcome' }); // <-- THROWS ERROR
});

If there is no error handling, the worker crashes. By default, BullMQ catches the exception, marks the job as failed, and moves it to a failed set. But the email is never sent. The user never gets their welcome email.

Debug It

To fix this, we need Retries and Exponential Backoff.

// Producer side
await emailQueue.add('welcome-email', { email: user.email }, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 1000 // 1s, 2s, 4s, 8s, 16s
  }
});

Now, if the first attempt fails, it retries in 1 second. If that fails, it waits 2 seconds, then 4, etc. If it fails 5 times, it is permanently moved to a Dead-Letter Queue (DLQ)—a special holding area for jobs that have chronically failed, allowing human engineers to inspect them and figure out why.

Mini Project (20-30 min)

Build a Webhook Dispatcher

Design a system where users configure webhooks. When an event happens, you must send an HTTP POST to their configured URL. If their server is down, you must retry with backoff. If it fails after 10 tries, mark the webhook as "disabled" in the database.

Real Application Feature

Idempotent Payments Processing

When a background job retries, you have a massive risk: what if the job actually succeeded, but the network failed during the success acknowledgment, causing it to retry? For email, they get two emails (annoying). For payments, they get charged twice (catastrophic).

We solve this with Idempotency. An operation is idempotent if doing it multiple times has the same result as doing it once.

// Implementation for Async Backend Systems
console.log("Bigger project solution");

Connections

This chapter connects deeply to the Database chapter (handling DB connections in workers), the Microservices chapter (using event buses to communicate), and the DevOps chapter (deploying and scaling worker fleets independently of web servers).

History

Early batch processing used simple cron jobs reading rows from a SQL database where `status = 'pending'`. As scale grew, polling the database became a bottleneck (the "thundering herd" problem and database lock contention). This led to the birth of dedicated Message Brokers (like RabbitMQ and Apache Kafka) and in-memory queue systems (like Redis/Resque) designed specifically for high-throughput, low-latency task distribution.

Mental Model (Analogy -> Reality)

The Analogy: A Busy Restaurant Kitchen

Imagine a restaurant where the waiter (API) takes your order and then goes into the kitchen to cook the food themselves, making you wait at the table without confirming they got the order. This is the naive synchronous approach.

In a proper restaurant, the waiter writes the order on a ticket (Job), places it on a rotating ticket wheel (Queue), and immediately returns to you saying "Your order is in!" (Acknowledgment). Meanwhile, specialized cooks (Workers) pull tickets off the wheel one by one and cook the food asynchronously.

The Reality

• Job: A JSON payload describing the work (e.g., { "type": "encode", "videoId": 123 }).
• Queue: A data structure (usually a FIFO queue stored in Redis or RabbitMQ) holding pending jobs.
• Worker: A separate Node.js process (or fleet of servers) whose only job is to pop items from the Queue and execute them.

Internal Working (Memory, stack, process, network, etc.)

When you enqueue a job using Redis (e.g., via BullMQ), the API process sends a network command (LPUSH or an atomic Lua script) to the Redis server. The API process then immediately returns the HTTP response.

In a completely different process—perhaps on a different server—a Worker uses a blocking pop network command (like BRPOP) to wait for new items in Redis. When a job arrives, Redis instantly sends the payload over the network to the Worker. The Worker parses the JSON, pushes function calls onto its call stack, and processes the job. If the Worker crashes, Redis (via mechanisms like BullMQ's active sets and locks) notices the lock expired and puts the job back in the queue for another worker.

Visual Explanation (ASCII diagrams)

Client       API Server (Producer)         Redis (Queue)           Worker Server (Consumer)
  |                 |                            |                            |
  |-- 1. HTTP POST->|                            |                            |
  |                 |-- 2. Push Job JSON ------->|                            |
  |<- 3. 202 OK ----|                            |-- 4. Pops Job JSON ------->|
  |                 |                            |                            |
  |                 |                            |<- 5. Acknowledges start ---|
  |                 |                            |                            |
  |                 |                            |    [ Processing... ]       |
  |                 |                            |                            |
  |                 |                            |<- 6. Marks completed ------|
  

Syntax

Using BullMQ with TypeScript and Redis:

import { Queue, Worker } from 'bullmq';

// 1. Create a Queue
const videoQueue = new Queue('videoProcessing', { connection: redisOptions });

// 2. Add a Job (Producer)
await videoQueue.add('encode', { videoId: 123, format: 'mp4' });

// 3. Process the Job (Consumer/Worker)
const worker = new Worker('videoProcessing', async job => {
  const { videoId, format } = job.data;
  await encodeVideo(videoId, format);
}, { connection: redisOptions });

Tiny Example

Let's build a reliable email sender.

import { Queue, Worker } from 'bullmq';

const emailQueue = new Queue('emails');

// API Endpoint
app.post('/signup', async (req, res) => {
  const user = await db.users.create(req.body);
  
  // Enqueue job, don't wait for email to send!
  await emailQueue.add('welcome-email', { email: user.email });
  
  res.status(202).send({ message: 'Signed up!' });
});

// Worker Process (run in a separate terminal)
new Worker('emails', async job => {
  console.log(`Sending email to ${job.data.email}...`);
  await sendGrid.send({ to: job.data.email, template: 'welcome' });
});

Walkthrough

In our Tiny Example, when a user hits /signup, the DB record is created fast (milliseconds). The email sending via a 3rd party API (SendGrid) might take 500ms to 2 seconds, and it might fail due to network blips.

By putting the job in the emails queue, the HTTP response goes back instantly. The user sees a snappy UI. The Worker process, which runs independently, picks up the job. If SendGrid is down, the Worker can fail the job, and the Queue system will automatically retry it later.

Break It

What happens if our worker function looks like this, and sendGrid.send fails?

new Worker('emails', async job => {
  await sendGrid.send({ to: job.data.email, template: 'welcome' }); // <-- THROWS ERROR
});

If there is no error handling, the worker crashes. By default, BullMQ catches the exception, marks the job as failed, and moves it to a failed set. But the email is never sent. The user never gets their welcome email.

Debug It

To fix this, we need Retries and Exponential Backoff.

// Producer side
await emailQueue.add('welcome-email', { email: user.email }, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 1000 // 1s, 2s, 4s, 8s, 16s
  }
});

Now, if the first attempt fails, it retries in 1 second. If that fails, it waits 2 seconds, then 4, etc. If it fails 5 times, it is permanently moved to a Dead-Letter Queue (DLQ)—a special holding area for jobs that have chronically failed, allowing human engineers to inspect them and figure out why.

Mini Project (20-30 min)

Build a Webhook Dispatcher

Design a system where users configure webhooks. When an event happens, you must send an HTTP POST to their configured URL. If their server is down, you must retry with backoff. If it fails after 10 tries, mark the webhook as "disabled" in the database.

Real Application Feature

Idempotent Payments Processing

When a background job retries, you have a massive risk: what if the job actually succeeded, but the network failed during the success acknowledgment, causing it to retry? For email, they get two emails (annoying). For payments, they get charged twice (catastrophic).

We solve this with Idempotency. An operation is idempotent if doing it multiple times has the same result as doing it once.

// Implementation for Async Backend Systems
console.log("Bigger project solution");