🌙
☀️ Dark
PART 18

Files & Storage

Filesystem, multipart, object storage, CDN.

Intermediate 45 min read

PART 18 — FILES & STORAGE

Files, Blobs, and Object Storage Systems

Analogy: Imagine you are a receptionist at a busy office (the Application Server). A courier arrives with a massive truckload of furniture (a large file upload). If you try to bring all the furniture into your small reception area (Server RAM), you'll be crushed.

Reality (Streaming): Instead, you direct the workers to carry the furniture piece by piece directly to the warehouse. You never hold all the furniture at once. This is Streaming.

Reality (Object Storage & Signed URLs): Even better, what if you just gave the courier a special security pass (a Signed URL) and told them to drive straight to the massive off-site warehouse (S3 Bucket) themselves? You never even touch the furniture. Your reception stays completely clear for other business.

Internal Working (Memory, stack, process, network, etc.)

When a client uploads a file via multipart/form-data, the HTTP body is divided into "parts", separated by a boundary string defined in the Content-Type header. The server reads the TCP socket stream. A parser (like Busboy or Multer in Node.js) looks for these boundaries. Instead of accumulating the data in a Buffer (RAM), it emits data events. These chunks can be piped directly into a fs.createWriteStream or a cloud storage upload stream.

When using Signed URLs, the server uses a secret key to cryptographically sign a URL granting temporary write access to a specific S3 path. The client receives this URL and makes a direct PUT request to S3. The application server's network and CPU are completely bypassed for the actual file transfer.

Visual Explanation (ASCII diagrams)

TRADITIONAL UPLOAD (BAD FOR SCALE):
Client ---(File 50MB)---> App Server (Parses, buffers) ---(Saves)---> Local Disk
                                      |
                                  CRASH (Out of Memory)

STREAMING UPLOAD:
Client ---(Chunk 1)---> App Server --(Pipe)--> S3/Disk
       ---(Chunk 2)---> App Server --(Pipe)--> S3/Disk
       ---(Chunk 3)---> App Server --(Pipe)--> S3/Disk

SIGNED URL DIRECT UPLOAD (BEST):
1. Client -> App Server: "I want to upload avatar.png"
2. App Server -> Client: "Here is a Signed URL for S3"
3. Client ---(File 50MB)---> S3 Bucket (Direct HTTP PUT)
4. Client -> App Server: "Upload done, here is the object key"

Syntax

Creating a Signed URL (AWS SDK v3):

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const client = new S3Client({ region: "us-east-1" });
const command = new PutObjectCommand({ Bucket: "my-bucket", Key: "user-123/avatar.png" });
const url = await getSignedUrl(client, command, { expiresIn: 3600 });

Tiny Example

Uploading directly to S3 from the browser using the signed URL:

// Frontend JavaScript
const fileInput = document.querySelector('input[type="file"]');
const file = fileInput.files[0];

// 1. Get signed URL from our backend
const response = await fetch('/api/upload-url?filename=' + file.name);
const { uploadUrl } = await response.json();

// 2. Upload directly to S3
await fetch(uploadUrl, {
  method: 'PUT',
  body: file,
  headers: { 'Content-Type': file.type }
});

Walkthrough

Let's build a multipart form upload handler using multer to stream a file to disk, perform basic validation, and return the file path.

import express from 'express';
import multer from 'multer';
import path from 'path';
import fs from 'fs';

const app = express();

// Configure storage
const storage = multer.diskStorage({
  destination: (req, file, cb) => cb(null, 'uploads/'),
  filename: (req, file, cb) => {
    const uniqueSuffix = Date.now() + '-' + Math.round(Math.random() * 1E9);
    cb(null, file.fieldname + '-' + uniqueSuffix + path.extname(file.originalname));
  }
});

// File validation
const fileFilter = (req, file, cb) => {
  if (file.mimetype === 'image/jpeg' || file.mimetype === 'image/png') {
    cb(null, true);
  } else {
    cb(new Error('Invalid file type'), false);
  }
};

const upload = multer({ 
  storage, 
  fileFilter,
  limits: { fileSize: 5 * 1024 * 1024 } // 5MB limit
});

app.post('/upload', upload.single('avatar'), (req, res) => {
  if (!req.file) return res.status(400).send('No file uploaded.');
  res.json({ message: 'File uploaded', path: req.file.path });
});

Break It

What happens if a user uploads a 10GB file but your multer limits are not set? The server will spend significant CPU time and disk I/O saving this massive file, potentially filling up the disk completely, leading to a Denial of Service (DoS).

What if a user uploads a file named ../../../etc/passwd? If you blindly use file.originalname without sanitization (which multer protects against, but raw streams might not), you might overwrite critical system files (Path Traversal).

Debug It

If uploads are failing, check:

  1. Is the frontend sending Content-Type: multipart/form-data?
  2. Is Nginx/Reverse Proxy configured with a high enough client_max_body_size? (Nginx defaults to 1MB and will return 413 Payload Too Large).
  3. Are the directory permissions correct for the Node process to write to the uploads/ folder?

Mini Project (20-30 min)

Build an image upload microservice. It should take a raw image upload, use the sharp library to resize it to a 256x256 thumbnail, convert it to WebP format, and stream the result to an AWS S3 bucket. Finally, it should return the public URL of the resized image.

Real Application Feature

In a production system like a social network, you don't serve user avatars directly from S3. You configure a Content Delivery Network (CDN) like CloudFront or Cloudflare in front of S3. The CDN caches the image at edge locations worldwide. When a user in Tokyo requests the image, it's served from a Tokyo data center, not your primary S3 bucket in Virginia.

Production Implementation

A true production pipeline handles files asynchronously:

  1. Client uploads direct to S3 using a Signed URL.
  2. S3 emits an ObjectCreated event.
  3. This triggers a serverless function (AWS Lambda) or pushes to an SQS queue.
  4. A worker picks up the job, downloads the image, runs virus scanning (e.g., ClamAV), generates thumbnails, and strips EXIF metadata (which can contain GPS coordinates).
  5. The worker moves the clean, processed image to a "Public" bucket and updates the database record.

Production Usage

For file downloads, never pipe files through your Node server. Always generate a Signed GET URL, or if the file is public, serve it via the CDN. If you must control access, use CDN Signed Cookies or URL signatures at the CDN edge.

Performance

  • Streaming: Keeps memory footprint O(1) regardless of file size.
  • Direct to Cloud: Removes the application server from the data path, freeing up connection pools.
  • CDN: Drastically reduces latency and offloads egress bandwidth costs from your primary infrastructure.

Best Practices

  • Never trust user input: Validate file extensions, check Magic Numbers (file signatures), and scan for viruses.
  • Always use limits: Enforce strict file size limits on the proxy layer (Nginx) and application layer.
  • Sanitize filenames: Generate your own UUIDs for filenames; never use the user-provided filename on disk or object storage keys.
  • Strip Metadata: Remove EXIF data from images to protect user privacy.

Interview Questions

Easy: Why shouldn't you store user uploads in the same repository as your code?

Because it makes the repository huge, it gets wiped out on fresh deployments if not volume-mounted, and it prevents horizontal scaling since servers don't share local disks.

Medium: What is the difference between Base64 JSON uploads and Multipart Form Data?

Base64 encoding inflates the payload size by about 33% and usually requires loading the entire string into memory. Multipart streams binary data directly with boundary markers, allowing efficient chunking and piping.

Hard: How do you handle resumable uploads for a 10GB video file over a flaky connection?

You use a protocol like Tus or S3 Multipart Upload. The client chunks the file (e.g., 5MB parts) and uploads them in parallel or sequentially. The server keeps track of received chunks. If the connection drops, the client only retries the failed chunks. Once all chunks are received, the server stitches them together.

Senior: Describe the architecture of an image processing pipeline that scales to millions of uploads.

Direct client-to-S3 uploads via presigned URLs -> S3 Event Notifications -> SQS Queue -> Auto-scaling worker cluster (or Lambda). Workers pull jobs, stream the object to memory, use sharp/libvips for processing, stream back to a public S3 bucket, and update the Postgres DB. A CDN fronts the public bucket. The API server is never involved in the binary transfer.

Engineering Challenge

Implement an endpoint that accepts a large CSV file upload, streams the file line-by-line using csv-parser, transforms each row, and streams the transformed data directly to S3 without ever writing the file to the local disk or holding the whole array in memory.

View Conceptual Solution

Use Busboy to capture the file stream. Pipe the file stream into the csv-parser stream. Pipe the output of the parser into a transform stream. Finally, pass this transform stream to the @aws-sdk/lib-storage Upload class, which can read from a Node stream and execute an S3 Multipart upload automatically.

Revision Sheet

  • Local Storage: Bad for scale. Needs NAS/SAN.
  • Object Storage (S3): Infinite, flat, scalable storage via HTTP.
  • Multipart Form: The HTTP standard for sending binary files.
  • Streaming: Processing data in chunks to save RAM.
  • Signed URLs: Cryptographic tickets for clients to interact directly with S3.
  • CDN: Edge caching for fast global delivery.

Connections

File handling heavily relies on understanding Streams and Buffers (Part 4). Offloading image processing to workers is a prime example of Message Queues and Background Jobs (Part 22). Securing those uploads requires the concepts learned in Web Security (Part 15).

PreviousVolume 17 🏠 Curriculum NextVolume 19

Bigger Project (1-2 hours)

Create an Express endpoint with Multer that accepts a file upload, validates the file type, and saves it to a local directory simulating S3.

▶ View Solution
typescript 
// Implementation for Files & Storage
console.log("Bigger project solution");

Interview Questions

Learning Objectives

  • Understand how filesystems handle binary and text data under the hood.
  • Process multipart/form-data for file uploads in a Node.js environment.
  • Stream large files efficiently without exhausting server memory.
  • Implement cloud object storage (e.g., AWS S3, GCS) for scalable file hosting.
  • Generate and utilize signed URLs for secure, direct-to-cloud uploads and downloads.
  • Perform image processing, file validation, and virus scanning on uploads.
  • Serve static assets and user-uploaded media through a Content Delivery Network (CDN).

Analogy: Imagine you are a receptionist at a busy office (the Application Server). A courier arrives with a massive truckload of furniture (a large file upload). If you try to bring all the furniture into your small reception area (Server RAM), you'll be crushed.

Reality (Streaming): Instead, you direct the workers to carry the furniture piece by piece directly to the warehouse. You never hold all the furniture at once. This is Streaming.

Reality (Object Storage & Signed URLs): Even better, what if you just gave the courier a special security pass (a Signed URL) and told them to drive straight to the massive off-site warehouse (S3 Bucket) themselves? You never even touch the furniture. Your reception stays completely clear for other business.

Internal Working (Memory, stack, process, network, etc.)

When a client uploads a file via multipart/form-data, the HTTP body is divided into "parts", separated by a boundary string defined in the Content-Type header. The server reads the TCP socket stream. A parser (like Busboy or Multer in Node.js) looks for these boundaries. Instead of accumulating the data in a Buffer (RAM), it emits data events. These chunks can be piped directly into a fs.createWriteStream or a cloud storage upload stream.

When using Signed URLs, the server uses a secret key to cryptographically sign a URL granting temporary write access to a specific S3 path. The client receives this URL and makes a direct PUT request to S3. The application server's network and CPU are completely bypassed for the actual file transfer.

Visual Explanation (ASCII diagrams)

TRADITIONAL UPLOAD (BAD FOR SCALE):
Client ---(File 50MB)---> App Server (Parses, buffers) ---(Saves)---> Local Disk
                                      |
                                  CRASH (Out of Memory)

STREAMING UPLOAD:
Client ---(Chunk 1)---> App Server --(Pipe)--> S3/Disk
       ---(Chunk 2)---> App Server --(Pipe)--> S3/Disk
       ---(Chunk 3)---> App Server --(Pipe)--> S3/Disk

SIGNED URL DIRECT UPLOAD (BEST):
1. Client -> App Server: "I want to upload avatar.png"
2. App Server -> Client: "Here is a Signed URL for S3"
3. Client ---(File 50MB)---> S3 Bucket (Direct HTTP PUT)
4. Client -> App Server: "Upload done, here is the object key"

Syntax

Creating a Signed URL (AWS SDK v3):

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const client = new S3Client({ region: "us-east-1" });
const command = new PutObjectCommand({ Bucket: "my-bucket", Key: "user-123/avatar.png" });
const url = await getSignedUrl(client, command, { expiresIn: 3600 });

Tiny Example

Uploading directly to S3 from the browser using the signed URL:

// Frontend JavaScript
const fileInput = document.querySelector('input[type="file"]');
const file = fileInput.files[0];

// 1. Get signed URL from our backend
const response = await fetch('/api/upload-url?filename=' + file.name);
const { uploadUrl } = await response.json();

// 2. Upload directly to S3
await fetch(uploadUrl, {
  method: 'PUT',
  body: file,
  headers: { 'Content-Type': file.type }
});

Walkthrough

Let's build a multipart form upload handler using multer to stream a file to disk, perform basic validation, and return the file path.

import express from 'express';
import multer from 'multer';
import path from 'path';
import fs from 'fs';

const app = express();

// Configure storage
const storage = multer.diskStorage({
  destination: (req, file, cb) => cb(null, 'uploads/'),
  filename: (req, file, cb) => {
    const uniqueSuffix = Date.now() + '-' + Math.round(Math.random() * 1E9);
    cb(null, file.fieldname + '-' + uniqueSuffix + path.extname(file.originalname));
  }
});

// File validation
const fileFilter = (req, file, cb) => {
  if (file.mimetype === 'image/jpeg' || file.mimetype === 'image/png') {
    cb(null, true);
  } else {
    cb(new Error('Invalid file type'), false);
  }
};

const upload = multer({ 
  storage, 
  fileFilter,
  limits: { fileSize: 5 * 1024 * 1024 } // 5MB limit
});

app.post('/upload', upload.single('avatar'), (req, res) => {
  if (!req.file) return res.status(400).send('No file uploaded.');
  res.json({ message: 'File uploaded', path: req.file.path });
});

Break It

What happens if a user uploads a 10GB file but your multer limits are not set? The server will spend significant CPU time and disk I/O saving this massive file, potentially filling up the disk completely, leading to a Denial of Service (DoS).

What if a user uploads a file named ../../../etc/passwd? If you blindly use file.originalname without sanitization (which multer protects against, but raw streams might not), you might overwrite critical system files (Path Traversal).

Debug It

If uploads are failing, check:

  1. Is the frontend sending Content-Type: multipart/form-data?
  2. Is Nginx/Reverse Proxy configured with a high enough client_max_body_size? (Nginx defaults to 1MB and will return 413 Payload Too Large).
  3. Are the directory permissions correct for the Node process to write to the uploads/ folder?

Mini Project (20-30 min)

Build an image upload microservice. It should take a raw image upload, use the sharp library to resize it to a 256x256 thumbnail, convert it to WebP format, and stream the result to an AWS S3 bucket. Finally, it should return the public URL of the resized image.

Real Application Feature

In a production system like a social network, you don't serve user avatars directly from S3. You configure a Content Delivery Network (CDN) like CloudFront or Cloudflare in front of S3. The CDN caches the image at edge locations worldwide. When a user in Tokyo requests the image, it's served from a Tokyo data center, not your primary S3 bucket in Virginia.

Production Implementation

A true production pipeline handles files asynchronously:

  1. Client uploads direct to S3 using a Signed URL.
  2. S3 emits an ObjectCreated event.
  3. This triggers a serverless function (AWS Lambda) or pushes to an SQS queue.
  4. A worker picks up the job, downloads the image, runs virus scanning (e.g., ClamAV), generates thumbnails, and strips EXIF metadata (which can contain GPS coordinates).
  5. The worker moves the clean, processed image to a "Public" bucket and updates the database record.

Production Usage

For file downloads, never pipe files through your Node server. Always generate a Signed GET URL, or if the file is public, serve it via the CDN. If you must control access, use CDN Signed Cookies or URL signatures at the CDN edge.

Performance

  • Streaming: Keeps memory footprint O(1) regardless of file size.
  • Direct to Cloud: Removes the application server from the data path, freeing up connection pools.
  • CDN: Drastically reduces latency and offloads egress bandwidth costs from your primary infrastructure.

Best Practices

  • Never trust user input: Validate file extensions, check Magic Numbers (file signatures), and scan for viruses.
  • Always use limits: Enforce strict file size limits on the proxy layer (Nginx) and application layer.
  • Sanitize filenames: Generate your own UUIDs for filenames; never use the user-provided filename on disk or object storage keys.
  • Strip Metadata: Remove EXIF data from images to protect user privacy.

Interview Questions

Easy: Why shouldn't you store user uploads in the same repository as your code?

Because it makes the repository huge, it gets wiped out on fresh deployments if not volume-mounted, and it prevents horizontal scaling since servers don't share local disks.

Medium: What is the difference between Base64 JSON uploads and Multipart Form Data?

Base64 encoding inflates the payload size by about 33% and usually requires loading the entire string into memory. Multipart streams binary data directly with boundary markers, allowing efficient chunking and piping.

Hard: How do you handle resumable uploads for a 10GB video file over a flaky connection?

You use a protocol like Tus or S3 Multipart Upload. The client chunks the file (e.g., 5MB parts) and uploads them in parallel or sequentially. The server keeps track of received chunks. If the connection drops, the client only retries the failed chunks. Once all chunks are received, the server stitches them together.

Senior: Describe the architecture of an image processing pipeline that scales to millions of uploads.

Direct client-to-S3 uploads via presigned URLs -> S3 Event Notifications -> SQS Queue -> Auto-scaling worker cluster (or Lambda). Workers pull jobs, stream the object to memory, use sharp/libvips for processing, stream back to a public S3 bucket, and update the Postgres DB. A CDN fronts the public bucket. The API server is never involved in the binary transfer.

Engineering Challenge

Implement an endpoint that accepts a large CSV file upload, streams the file line-by-line using csv-parser, transforms each row, and streams the transformed data directly to S3 without ever writing the file to the local disk or holding the whole array in memory.

View Conceptual Solution

Use Busboy to capture the file stream. Pipe the file stream into the csv-parser stream. Pipe the output of the parser into a transform stream. Finally, pass this transform stream to the @aws-sdk/lib-storage Upload class, which can read from a Node stream and execute an S3 Multipart upload automatically.

Revision Sheet

  • Local Storage: Bad for scale. Needs NAS/SAN.
  • Object Storage (S3): Infinite, flat, scalable storage via HTTP.
  • Multipart Form: The HTTP standard for sending binary files.
  • Streaming: Processing data in chunks to save RAM.
  • Signed URLs: Cryptographic tickets for clients to interact directly with S3.
  • CDN: Edge caching for fast global delivery.

Connections

File handling heavily relies on understanding Streams and Buffers (Part 4). Offloading image processing to workers is a prime example of Message Queues and Background Jobs (Part 22). Securing those uploads requires the concepts learned in Web Security (Part 15).

PreviousVolume 17 🏠 Curriculum NextVolume 19

Bigger Project (1-2 hours)

Create an Express endpoint with Multer that accepts a file upload, validates the file type, and saves it to a local directory simulating S3.

▶ View Solution
typescript 
// Implementation for Files & Storage
console.log("Bigger project solution");

Interview Questions

Prerequisites

  • Solid understanding of HTTP methods, headers, and body formats.
  • Familiarity with Node.js streams and buffers.
  • Basic knowledge of cloud architecture and identity access management (IAM).
  • Experience with frontend JavaScript FormData API.

Analogy: Imagine you are a receptionist at a busy office (the Application Server). A courier arrives with a massive truckload of furniture (a large file upload). If you try to bring all the furniture into your small reception area (Server RAM), you'll be crushed.

Reality (Streaming): Instead, you direct the workers to carry the furniture piece by piece directly to the warehouse. You never hold all the furniture at once. This is Streaming.

Reality (Object Storage & Signed URLs): Even better, what if you just gave the courier a special security pass (a Signed URL) and told them to drive straight to the massive off-site warehouse (S3 Bucket) themselves? You never even touch the furniture. Your reception stays completely clear for other business.

Internal Working (Memory, stack, process, network, etc.)

When a client uploads a file via multipart/form-data, the HTTP body is divided into "parts", separated by a boundary string defined in the Content-Type header. The server reads the TCP socket stream. A parser (like Busboy or Multer in Node.js) looks for these boundaries. Instead of accumulating the data in a Buffer (RAM), it emits data events. These chunks can be piped directly into a fs.createWriteStream or a cloud storage upload stream.

When using Signed URLs, the server uses a secret key to cryptographically sign a URL granting temporary write access to a specific S3 path. The client receives this URL and makes a direct PUT request to S3. The application server's network and CPU are completely bypassed for the actual file transfer.

Visual Explanation (ASCII diagrams)

TRADITIONAL UPLOAD (BAD FOR SCALE):
Client ---(File 50MB)---> App Server (Parses, buffers) ---(Saves)---> Local Disk
                                      |
                                  CRASH (Out of Memory)

STREAMING UPLOAD:
Client ---(Chunk 1)---> App Server --(Pipe)--> S3/Disk
       ---(Chunk 2)---> App Server --(Pipe)--> S3/Disk
       ---(Chunk 3)---> App Server --(Pipe)--> S3/Disk

SIGNED URL DIRECT UPLOAD (BEST):
1. Client -> App Server: "I want to upload avatar.png"
2. App Server -> Client: "Here is a Signed URL for S3"
3. Client ---(File 50MB)---> S3 Bucket (Direct HTTP PUT)
4. Client -> App Server: "Upload done, here is the object key"

Syntax

Creating a Signed URL (AWS SDK v3):

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const client = new S3Client({ region: "us-east-1" });
const command = new PutObjectCommand({ Bucket: "my-bucket", Key: "user-123/avatar.png" });
const url = await getSignedUrl(client, command, { expiresIn: 3600 });

Tiny Example

Uploading directly to S3 from the browser using the signed URL:

// Frontend JavaScript
const fileInput = document.querySelector('input[type="file"]');
const file = fileInput.files[0];

// 1. Get signed URL from our backend
const response = await fetch('/api/upload-url?filename=' + file.name);
const { uploadUrl } = await response.json();

// 2. Upload directly to S3
await fetch(uploadUrl, {
  method: 'PUT',
  body: file,
  headers: { 'Content-Type': file.type }
});

Walkthrough

Let's build a multipart form upload handler using multer to stream a file to disk, perform basic validation, and return the file path.

import express from 'express';
import multer from 'multer';
import path from 'path';
import fs from 'fs';

const app = express();

// Configure storage
const storage = multer.diskStorage({
  destination: (req, file, cb) => cb(null, 'uploads/'),
  filename: (req, file, cb) => {
    const uniqueSuffix = Date.now() + '-' + Math.round(Math.random() * 1E9);
    cb(null, file.fieldname + '-' + uniqueSuffix + path.extname(file.originalname));
  }
});

// File validation
const fileFilter = (req, file, cb) => {
  if (file.mimetype === 'image/jpeg' || file.mimetype === 'image/png') {
    cb(null, true);
  } else {
    cb(new Error('Invalid file type'), false);
  }
};

const upload = multer({ 
  storage, 
  fileFilter,
  limits: { fileSize: 5 * 1024 * 1024 } // 5MB limit
});

app.post('/upload', upload.single('avatar'), (req, res) => {
  if (!req.file) return res.status(400).send('No file uploaded.');
  res.json({ message: 'File uploaded', path: req.file.path });
});

Break It

What happens if a user uploads a 10GB file but your multer limits are not set? The server will spend significant CPU time and disk I/O saving this massive file, potentially filling up the disk completely, leading to a Denial of Service (DoS).

What if a user uploads a file named ../../../etc/passwd? If you blindly use file.originalname without sanitization (which multer protects against, but raw streams might not), you might overwrite critical system files (Path Traversal).

Debug It

If uploads are failing, check:

  1. Is the frontend sending Content-Type: multipart/form-data?
  2. Is Nginx/Reverse Proxy configured with a high enough client_max_body_size? (Nginx defaults to 1MB and will return 413 Payload Too Large).
  3. Are the directory permissions correct for the Node process to write to the uploads/ folder?

Mini Project (20-30 min)

Build an image upload microservice. It should take a raw image upload, use the sharp library to resize it to a 256x256 thumbnail, convert it to WebP format, and stream the result to an AWS S3 bucket. Finally, it should return the public URL of the resized image.

Real Application Feature

In a production system like a social network, you don't serve user avatars directly from S3. You configure a Content Delivery Network (CDN) like CloudFront or Cloudflare in front of S3. The CDN caches the image at edge locations worldwide. When a user in Tokyo requests the image, it's served from a Tokyo data center, not your primary S3 bucket in Virginia.

Production Implementation

A true production pipeline handles files asynchronously:

  1. Client uploads direct to S3 using a Signed URL.
  2. S3 emits an ObjectCreated event.
  3. This triggers a serverless function (AWS Lambda) or pushes to an SQS queue.
  4. A worker picks up the job, downloads the image, runs virus scanning (e.g., ClamAV), generates thumbnails, and strips EXIF metadata (which can contain GPS coordinates).
  5. The worker moves the clean, processed image to a "Public" bucket and updates the database record.

Production Usage

For file downloads, never pipe files through your Node server. Always generate a Signed GET URL, or if the file is public, serve it via the CDN. If you must control access, use CDN Signed Cookies or URL signatures at the CDN edge.

Performance

  • Streaming: Keeps memory footprint O(1) regardless of file size.
  • Direct to Cloud: Removes the application server from the data path, freeing up connection pools.
  • CDN: Drastically reduces latency and offloads egress bandwidth costs from your primary infrastructure.

Best Practices

  • Never trust user input: Validate file extensions, check Magic Numbers (file signatures), and scan for viruses.
  • Always use limits: Enforce strict file size limits on the proxy layer (Nginx) and application layer.
  • Sanitize filenames: Generate your own UUIDs for filenames; never use the user-provided filename on disk or object storage keys.
  • Strip Metadata: Remove EXIF data from images to protect user privacy.

Interview Questions

Easy: Why shouldn't you store user uploads in the same repository as your code?

Because it makes the repository huge, it gets wiped out on fresh deployments if not volume-mounted, and it prevents horizontal scaling since servers don't share local disks.

Medium: What is the difference between Base64 JSON uploads and Multipart Form Data?

Base64 encoding inflates the payload size by about 33% and usually requires loading the entire string into memory. Multipart streams binary data directly with boundary markers, allowing efficient chunking and piping.

Hard: How do you handle resumable uploads for a 10GB video file over a flaky connection?

You use a protocol like Tus or S3 Multipart Upload. The client chunks the file (e.g., 5MB parts) and uploads them in parallel or sequentially. The server keeps track of received chunks. If the connection drops, the client only retries the failed chunks. Once all chunks are received, the server stitches them together.

Senior: Describe the architecture of an image processing pipeline that scales to millions of uploads.

Direct client-to-S3 uploads via presigned URLs -> S3 Event Notifications -> SQS Queue -> Auto-scaling worker cluster (or Lambda). Workers pull jobs, stream the object to memory, use sharp/libvips for processing, stream back to a public S3 bucket, and update the Postgres DB. A CDN fronts the public bucket. The API server is never involved in the binary transfer.

Engineering Challenge

Implement an endpoint that accepts a large CSV file upload, streams the file line-by-line using csv-parser, transforms each row, and streams the transformed data directly to S3 without ever writing the file to the local disk or holding the whole array in memory.

View Conceptual Solution

Use Busboy to capture the file stream. Pipe the file stream into the csv-parser stream. Pipe the output of the parser into a transform stream. Finally, pass this transform stream to the @aws-sdk/lib-storage Upload class, which can read from a Node stream and execute an S3 Multipart upload automatically.

Revision Sheet

  • Local Storage: Bad for scale. Needs NAS/SAN.
  • Object Storage (S3): Infinite, flat, scalable storage via HTTP.
  • Multipart Form: The HTTP standard for sending binary files.
  • Streaming: Processing data in chunks to save RAM.
  • Signed URLs: Cryptographic tickets for clients to interact directly with S3.
  • CDN: Edge caching for fast global delivery.

Connections

File handling heavily relies on understanding Streams and Buffers (Part 4). Offloading image processing to workers is a prime example of Message Queues and Background Jobs (Part 22). Securing those uploads requires the concepts learned in Web Security (Part 15).

PreviousVolume 17 🏠 Curriculum NextVolume 19

Bigger Project (1-2 hours)

Create an Express endpoint with Multer that accepts a file upload, validates the file type, and saves it to a local directory simulating S3.

▶ View Solution
typescript 
// Implementation for Files & Storage
console.log("Bigger project solution");

Interview Questions

Why Does This Exist?

Virtually every meaningful application requires file handling: avatars, documents, attachments, videos, and exports. Handling text data in a JSON payload is trivial, but handling a 5GB video file requires a completely different architectural approach. If you read a 5GB file into RAM on a server with 2GB of RAM, your server crashes. If you store user uploads directly on your application server's local disk, your application cannot scale horizontally, and you will lose data when the server restarts or dies.

Analogy: Imagine you are a receptionist at a busy office (the Application Server). A courier arrives with a massive truckload of furniture (a large file upload). If you try to bring all the furniture into your small reception area (Server RAM), you'll be crushed.

Reality (Streaming): Instead, you direct the workers to carry the furniture piece by piece directly to the warehouse. You never hold all the furniture at once. This is Streaming.

Reality (Object Storage & Signed URLs): Even better, what if you just gave the courier a special security pass (a Signed URL) and told them to drive straight to the massive off-site warehouse (S3 Bucket) themselves? You never even touch the furniture. Your reception stays completely clear for other business.

Internal Working (Memory, stack, process, network, etc.)

When a client uploads a file via multipart/form-data, the HTTP body is divided into "parts", separated by a boundary string defined in the Content-Type header. The server reads the TCP socket stream. A parser (like Busboy or Multer in Node.js) looks for these boundaries. Instead of accumulating the data in a Buffer (RAM), it emits data events. These chunks can be piped directly into a fs.createWriteStream or a cloud storage upload stream.

When using Signed URLs, the server uses a secret key to cryptographically sign a URL granting temporary write access to a specific S3 path. The client receives this URL and makes a direct PUT request to S3. The application server's network and CPU are completely bypassed for the actual file transfer.

Visual Explanation (ASCII diagrams)

TRADITIONAL UPLOAD (BAD FOR SCALE):
Client ---(File 50MB)---> App Server (Parses, buffers) ---(Saves)---> Local Disk
                                      |
                                  CRASH (Out of Memory)

STREAMING UPLOAD:
Client ---(Chunk 1)---> App Server --(Pipe)--> S3/Disk
       ---(Chunk 2)---> App Server --(Pipe)--> S3/Disk
       ---(Chunk 3)---> App Server --(Pipe)--> S3/Disk

SIGNED URL DIRECT UPLOAD (BEST):
1. Client -> App Server: "I want to upload avatar.png"
2. App Server -> Client: "Here is a Signed URL for S3"
3. Client ---(File 50MB)---> S3 Bucket (Direct HTTP PUT)
4. Client -> App Server: "Upload done, here is the object key"

Syntax

Creating a Signed URL (AWS SDK v3):

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const client = new S3Client({ region: "us-east-1" });
const command = new PutObjectCommand({ Bucket: "my-bucket", Key: "user-123/avatar.png" });
const url = await getSignedUrl(client, command, { expiresIn: 3600 });

Tiny Example

Uploading directly to S3 from the browser using the signed URL:

// Frontend JavaScript
const fileInput = document.querySelector('input[type="file"]');
const file = fileInput.files[0];

// 1. Get signed URL from our backend
const response = await fetch('/api/upload-url?filename=' + file.name);
const { uploadUrl } = await response.json();

// 2. Upload directly to S3
await fetch(uploadUrl, {
  method: 'PUT',
  body: file,
  headers: { 'Content-Type': file.type }
});

Walkthrough

Let's build a multipart form upload handler using multer to stream a file to disk, perform basic validation, and return the file path.

import express from 'express';
import multer from 'multer';
import path from 'path';
import fs from 'fs';

const app = express();

// Configure storage
const storage = multer.diskStorage({
  destination: (req, file, cb) => cb(null, 'uploads/'),
  filename: (req, file, cb) => {
    const uniqueSuffix = Date.now() + '-' + Math.round(Math.random() * 1E9);
    cb(null, file.fieldname + '-' + uniqueSuffix + path.extname(file.originalname));
  }
});

// File validation
const fileFilter = (req, file, cb) => {
  if (file.mimetype === 'image/jpeg' || file.mimetype === 'image/png') {
    cb(null, true);
  } else {
    cb(new Error('Invalid file type'), false);
  }
};

const upload = multer({ 
  storage, 
  fileFilter,
  limits: { fileSize: 5 * 1024 * 1024 } // 5MB limit
});

app.post('/upload', upload.single('avatar'), (req, res) => {
  if (!req.file) return res.status(400).send('No file uploaded.');
  res.json({ message: 'File uploaded', path: req.file.path });
});

Break It

What happens if a user uploads a 10GB file but your multer limits are not set? The server will spend significant CPU time and disk I/O saving this massive file, potentially filling up the disk completely, leading to a Denial of Service (DoS).

What if a user uploads a file named ../../../etc/passwd? If you blindly use file.originalname without sanitization (which multer protects against, but raw streams might not), you might overwrite critical system files (Path Traversal).

Debug It

If uploads are failing, check:

  1. Is the frontend sending Content-Type: multipart/form-data?
  2. Is Nginx/Reverse Proxy configured with a high enough client_max_body_size? (Nginx defaults to 1MB and will return 413 Payload Too Large).
  3. Are the directory permissions correct for the Node process to write to the uploads/ folder?

Mini Project (20-30 min)

Build an image upload microservice. It should take a raw image upload, use the sharp library to resize it to a 256x256 thumbnail, convert it to WebP format, and stream the result to an AWS S3 bucket. Finally, it should return the public URL of the resized image.

Real Application Feature

In a production system like a social network, you don't serve user avatars directly from S3. You configure a Content Delivery Network (CDN) like CloudFront or Cloudflare in front of S3. The CDN caches the image at edge locations worldwide. When a user in Tokyo requests the image, it's served from a Tokyo data center, not your primary S3 bucket in Virginia.

Production Implementation

A true production pipeline handles files asynchronously:

  1. Client uploads direct to S3 using a Signed URL.
  2. S3 emits an ObjectCreated event.
  3. This triggers a serverless function (AWS Lambda) or pushes to an SQS queue.
  4. A worker picks up the job, downloads the image, runs virus scanning (e.g., ClamAV), generates thumbnails, and strips EXIF metadata (which can contain GPS coordinates).
  5. The worker moves the clean, processed image to a "Public" bucket and updates the database record.

Production Usage

For file downloads, never pipe files through your Node server. Always generate a Signed GET URL, or if the file is public, serve it via the CDN. If you must control access, use CDN Signed Cookies or URL signatures at the CDN edge.

Performance

  • Streaming: Keeps memory footprint O(1) regardless of file size.
  • Direct to Cloud: Removes the application server from the data path, freeing up connection pools.
  • CDN: Drastically reduces latency and offloads egress bandwidth costs from your primary infrastructure.

Best Practices

  • Never trust user input: Validate file extensions, check Magic Numbers (file signatures), and scan for viruses.
  • Always use limits: Enforce strict file size limits on the proxy layer (Nginx) and application layer.
  • Sanitize filenames: Generate your own UUIDs for filenames; never use the user-provided filename on disk or object storage keys.
  • Strip Metadata: Remove EXIF data from images to protect user privacy.

Interview Questions

Easy: Why shouldn't you store user uploads in the same repository as your code?

Because it makes the repository huge, it gets wiped out on fresh deployments if not volume-mounted, and it prevents horizontal scaling since servers don't share local disks.

Medium: What is the difference between Base64 JSON uploads and Multipart Form Data?

Base64 encoding inflates the payload size by about 33% and usually requires loading the entire string into memory. Multipart streams binary data directly with boundary markers, allowing efficient chunking and piping.

Hard: How do you handle resumable uploads for a 10GB video file over a flaky connection?

You use a protocol like Tus or S3 Multipart Upload. The client chunks the file (e.g., 5MB parts) and uploads them in parallel or sequentially. The server keeps track of received chunks. If the connection drops, the client only retries the failed chunks. Once all chunks are received, the server stitches them together.

Senior: Describe the architecture of an image processing pipeline that scales to millions of uploads.

Direct client-to-S3 uploads via presigned URLs -> S3 Event Notifications -> SQS Queue -> Auto-scaling worker cluster (or Lambda). Workers pull jobs, stream the object to memory, use sharp/libvips for processing, stream back to a public S3 bucket, and update the Postgres DB. A CDN fronts the public bucket. The API server is never involved in the binary transfer.

Engineering Challenge

Implement an endpoint that accepts a large CSV file upload, streams the file line-by-line using csv-parser, transforms each row, and streams the transformed data directly to S3 without ever writing the file to the local disk or holding the whole array in memory.

View Conceptual Solution

Use Busboy to capture the file stream. Pipe the file stream into the csv-parser stream. Pipe the output of the parser into a transform stream. Finally, pass this transform stream to the @aws-sdk/lib-storage Upload class, which can read from a Node stream and execute an S3 Multipart upload automatically.

Revision Sheet

  • Local Storage: Bad for scale. Needs NAS/SAN.
  • Object Storage (S3): Infinite, flat, scalable storage via HTTP.
  • Multipart Form: The HTTP standard for sending binary files.
  • Streaming: Processing data in chunks to save RAM.
  • Signed URLs: Cryptographic tickets for clients to interact directly with S3.
  • CDN: Edge caching for fast global delivery.

Connections

File handling heavily relies on understanding Streams and Buffers (Part 4). Offloading image processing to workers is a prime example of Message Queues and Background Jobs (Part 22). Securing those uploads requires the concepts learned in Web Security (Part 15).

PreviousVolume 17 🏠 Curriculum NextVolume 19

Bigger Project (1-2 hours)

Create an Express endpoint with Multer that accepts a file upload, validates the file type, and saves it to a local directory simulating S3.

▶ View Solution
typescript 
// Implementation for Files & Storage
console.log("Bigger project solution");

Interview Questions

The Problem Before the Solution

Early web applications allowed users to upload files, and the server would simply save those files to a local directory (e.g., /var/www/html/uploads). The server would then serve these files directly to users. Uploads were often sent as base64-encoded strings within JSON, or as standard URL-encoded form data, which heavily bloated the payload size and consumed excessive memory.

Analogy: Imagine you are a receptionist at a busy office (the Application Server). A courier arrives with a massive truckload of furniture (a large file upload). If you try to bring all the furniture into your small reception area (Server RAM), you'll be crushed.

Reality (Streaming): Instead, you direct the workers to carry the furniture piece by piece directly to the warehouse. You never hold all the furniture at once. This is Streaming.

Reality (Object Storage & Signed URLs): Even better, what if you just gave the courier a special security pass (a Signed URL) and told them to drive straight to the massive off-site warehouse (S3 Bucket) themselves? You never even touch the furniture. Your reception stays completely clear for other business.

Internal Working (Memory, stack, process, network, etc.)

When a client uploads a file via multipart/form-data, the HTTP body is divided into "parts", separated by a boundary string defined in the Content-Type header. The server reads the TCP socket stream. A parser (like Busboy or Multer in Node.js) looks for these boundaries. Instead of accumulating the data in a Buffer (RAM), it emits data events. These chunks can be piped directly into a fs.createWriteStream or a cloud storage upload stream.

When using Signed URLs, the server uses a secret key to cryptographically sign a URL granting temporary write access to a specific S3 path. The client receives this URL and makes a direct PUT request to S3. The application server's network and CPU are completely bypassed for the actual file transfer.

Visual Explanation (ASCII diagrams)

TRADITIONAL UPLOAD (BAD FOR SCALE):
Client ---(File 50MB)---> App Server (Parses, buffers) ---(Saves)---> Local Disk
                                      |
                                  CRASH (Out of Memory)

STREAMING UPLOAD:
Client ---(Chunk 1)---> App Server --(Pipe)--> S3/Disk
       ---(Chunk 2)---> App Server --(Pipe)--> S3/Disk
       ---(Chunk 3)---> App Server --(Pipe)--> S3/Disk

SIGNED URL DIRECT UPLOAD (BEST):
1. Client -> App Server: "I want to upload avatar.png"
2. App Server -> Client: "Here is a Signed URL for S3"
3. Client ---(File 50MB)---> S3 Bucket (Direct HTTP PUT)
4. Client -> App Server: "Upload done, here is the object key"

Syntax

Creating a Signed URL (AWS SDK v3):

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const client = new S3Client({ region: "us-east-1" });
const command = new PutObjectCommand({ Bucket: "my-bucket", Key: "user-123/avatar.png" });
const url = await getSignedUrl(client, command, { expiresIn: 3600 });

Tiny Example

Uploading directly to S3 from the browser using the signed URL:

// Frontend JavaScript
const fileInput = document.querySelector('input[type="file"]');
const file = fileInput.files[0];

// 1. Get signed URL from our backend
const response = await fetch('/api/upload-url?filename=' + file.name);
const { uploadUrl } = await response.json();

// 2. Upload directly to S3
await fetch(uploadUrl, {
  method: 'PUT',
  body: file,
  headers: { 'Content-Type': file.type }
});

Walkthrough

Let's build a multipart form upload handler using multer to stream a file to disk, perform basic validation, and return the file path.

import express from 'express';
import multer from 'multer';
import path from 'path';
import fs from 'fs';

const app = express();

// Configure storage
const storage = multer.diskStorage({
  destination: (req, file, cb) => cb(null, 'uploads/'),
  filename: (req, file, cb) => {
    const uniqueSuffix = Date.now() + '-' + Math.round(Math.random() * 1E9);
    cb(null, file.fieldname + '-' + uniqueSuffix + path.extname(file.originalname));
  }
});

// File validation
const fileFilter = (req, file, cb) => {
  if (file.mimetype === 'image/jpeg' || file.mimetype === 'image/png') {
    cb(null, true);
  } else {
    cb(new Error('Invalid file type'), false);
  }
};

const upload = multer({ 
  storage, 
  fileFilter,
  limits: { fileSize: 5 * 1024 * 1024 } // 5MB limit
});

app.post('/upload', upload.single('avatar'), (req, res) => {
  if (!req.file) return res.status(400).send('No file uploaded.');
  res.json({ message: 'File uploaded', path: req.file.path });
});

Break It

What happens if a user uploads a 10GB file but your multer limits are not set? The server will spend significant CPU time and disk I/O saving this massive file, potentially filling up the disk completely, leading to a Denial of Service (DoS).

What if a user uploads a file named ../../../etc/passwd? If you blindly use file.originalname without sanitization (which multer protects against, but raw streams might not), you might overwrite critical system files (Path Traversal).

Debug It

If uploads are failing, check:

  1. Is the frontend sending Content-Type: multipart/form-data?
  2. Is Nginx/Reverse Proxy configured with a high enough client_max_body_size? (Nginx defaults to 1MB and will return 413 Payload Too Large).
  3. Are the directory permissions correct for the Node process to write to the uploads/ folder?

Mini Project (20-30 min)

Build an image upload microservice. It should take a raw image upload, use the sharp library to resize it to a 256x256 thumbnail, convert it to WebP format, and stream the result to an AWS S3 bucket. Finally, it should return the public URL of the resized image.

Real Application Feature

In a production system like a social network, you don't serve user avatars directly from S3. You configure a Content Delivery Network (CDN) like CloudFront or Cloudflare in front of S3. The CDN caches the image at edge locations worldwide. When a user in Tokyo requests the image, it's served from a Tokyo data center, not your primary S3 bucket in Virginia.

Production Implementation

A true production pipeline handles files asynchronously:

  1. Client uploads direct to S3 using a Signed URL.
  2. S3 emits an ObjectCreated event.
  3. This triggers a serverless function (AWS Lambda) or pushes to an SQS queue.
  4. A worker picks up the job, downloads the image, runs virus scanning (e.g., ClamAV), generates thumbnails, and strips EXIF metadata (which can contain GPS coordinates).
  5. The worker moves the clean, processed image to a "Public" bucket and updates the database record.

Production Usage

For file downloads, never pipe files through your Node server. Always generate a Signed GET URL, or if the file is public, serve it via the CDN. If you must control access, use CDN Signed Cookies or URL signatures at the CDN edge.

Performance

  • Streaming: Keeps memory footprint O(1) regardless of file size.
  • Direct to Cloud: Removes the application server from the data path, freeing up connection pools.
  • CDN: Drastically reduces latency and offloads egress bandwidth costs from your primary infrastructure.

Best Practices

  • Never trust user input: Validate file extensions, check Magic Numbers (file signatures), and scan for viruses.
  • Always use limits: Enforce strict file size limits on the proxy layer (Nginx) and application layer.
  • Sanitize filenames: Generate your own UUIDs for filenames; never use the user-provided filename on disk or object storage keys.
  • Strip Metadata: Remove EXIF data from images to protect user privacy.

Interview Questions

Easy: Why shouldn't you store user uploads in the same repository as your code?

Because it makes the repository huge, it gets wiped out on fresh deployments if not volume-mounted, and it prevents horizontal scaling since servers don't share local disks.

Medium: What is the difference between Base64 JSON uploads and Multipart Form Data?

Base64 encoding inflates the payload size by about 33% and usually requires loading the entire string into memory. Multipart streams binary data directly with boundary markers, allowing efficient chunking and piping.

Hard: How do you handle resumable uploads for a 10GB video file over a flaky connection?

You use a protocol like Tus or S3 Multipart Upload. The client chunks the file (e.g., 5MB parts) and uploads them in parallel or sequentially. The server keeps track of received chunks. If the connection drops, the client only retries the failed chunks. Once all chunks are received, the server stitches them together.

Senior: Describe the architecture of an image processing pipeline that scales to millions of uploads.

Direct client-to-S3 uploads via presigned URLs -> S3 Event Notifications -> SQS Queue -> Auto-scaling worker cluster (or Lambda). Workers pull jobs, stream the object to memory, use sharp/libvips for processing, stream back to a public S3 bucket, and update the Postgres DB. A CDN fronts the public bucket. The API server is never involved in the binary transfer.

Engineering Challenge

Implement an endpoint that accepts a large CSV file upload, streams the file line-by-line using csv-parser, transforms each row, and streams the transformed data directly to S3 without ever writing the file to the local disk or holding the whole array in memory.

View Conceptual Solution

Use Busboy to capture the file stream. Pipe the file stream into the csv-parser stream. Pipe the output of the parser into a transform stream. Finally, pass this transform stream to the @aws-sdk/lib-storage Upload class, which can read from a Node stream and execute an S3 Multipart upload automatically.

Revision Sheet

  • Local Storage: Bad for scale. Needs NAS/SAN.
  • Object Storage (S3): Infinite, flat, scalable storage via HTTP.
  • Multipart Form: The HTTP standard for sending binary files.
  • Streaming: Processing data in chunks to save RAM.
  • Signed URLs: Cryptographic tickets for clients to interact directly with S3.
  • CDN: Edge caching for fast global delivery.

Connections

File handling heavily relies on understanding Streams and Buffers (Part 4). Offloading image processing to workers is a prime example of Message Queues and Background Jobs (Part 22). Securing those uploads requires the concepts learned in Web Security (Part 15).

PreviousVolume 17 🏠 Curriculum NextVolume 19

Bigger Project (1-2 hours)

Create an Express endpoint with Multer that accepts a file upload, validates the file type, and saves it to a local directory simulating S3.

▶ View Solution
typescript 
// Implementation for Files & Storage
console.log("Bigger project solution");

Interview Questions

Why the Old Approach Breaks

  • Memory Exhaustion: Buffering large files entirely into RAM before saving them crashes the Node process (V8 memory limit).
  • Horizontal Scaling: If Server A saves an image to its local disk, and the next user request hits Server B, Server B won't find the image. This breaks load balancing.
  • Bandwidth Bottlenecks: Serving large media files from the application server blocks the event loop and eats up bandwidth meant for API requests.
  • Security Risks: Storing user uploads on the local filesystem makes it trivial for an attacker to upload a malicious executable (like a PHP script or a shell script) and execute it.

Analogy: Imagine you are a receptionist at a busy office (the Application Server). A courier arrives with a massive truckload of furniture (a large file upload). If you try to bring all the furniture into your small reception area (Server RAM), you'll be crushed.

Reality (Streaming): Instead, you direct the workers to carry the furniture piece by piece directly to the warehouse. You never hold all the furniture at once. This is Streaming.

Reality (Object Storage & Signed URLs): Even better, what if you just gave the courier a special security pass (a Signed URL) and told them to drive straight to the massive off-site warehouse (S3 Bucket) themselves? You never even touch the furniture. Your reception stays completely clear for other business.

Internal Working (Memory, stack, process, network, etc.)

When a client uploads a file via multipart/form-data, the HTTP body is divided into "parts", separated by a boundary string defined in the Content-Type header. The server reads the TCP socket stream. A parser (like Busboy or Multer in Node.js) looks for these boundaries. Instead of accumulating the data in a Buffer (RAM), it emits data events. These chunks can be piped directly into a fs.createWriteStream or a cloud storage upload stream.

When using Signed URLs, the server uses a secret key to cryptographically sign a URL granting temporary write access to a specific S3 path. The client receives this URL and makes a direct PUT request to S3. The application server's network and CPU are completely bypassed for the actual file transfer.

Visual Explanation (ASCII diagrams)

TRADITIONAL UPLOAD (BAD FOR SCALE):
Client ---(File 50MB)---> App Server (Parses, buffers) ---(Saves)---> Local Disk
                                      |
                                  CRASH (Out of Memory)

STREAMING UPLOAD:
Client ---(Chunk 1)---> App Server --(Pipe)--> S3/Disk
       ---(Chunk 2)---> App Server --(Pipe)--> S3/Disk
       ---(Chunk 3)---> App Server --(Pipe)--> S3/Disk

SIGNED URL DIRECT UPLOAD (BEST):
1. Client -> App Server: "I want to upload avatar.png"
2. App Server -> Client: "Here is a Signed URL for S3"
3. Client ---(File 50MB)---> S3 Bucket (Direct HTTP PUT)
4. Client -> App Server: "Upload done, here is the object key"

Syntax

Creating a Signed URL (AWS SDK v3):

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const client = new S3Client({ region: "us-east-1" });
const command = new PutObjectCommand({ Bucket: "my-bucket", Key: "user-123/avatar.png" });
const url = await getSignedUrl(client, command, { expiresIn: 3600 });

Tiny Example

Uploading directly to S3 from the browser using the signed URL:

// Frontend JavaScript
const fileInput = document.querySelector('input[type="file"]');
const file = fileInput.files[0];

// 1. Get signed URL from our backend
const response = await fetch('/api/upload-url?filename=' + file.name);
const { uploadUrl } = await response.json();

// 2. Upload directly to S3
await fetch(uploadUrl, {
  method: 'PUT',
  body: file,
  headers: { 'Content-Type': file.type }
});

Walkthrough

Let's build a multipart form upload handler using multer to stream a file to disk, perform basic validation, and return the file path.

import express from 'express';
import multer from 'multer';
import path from 'path';
import fs from 'fs';

const app = express();

// Configure storage
const storage = multer.diskStorage({
  destination: (req, file, cb) => cb(null, 'uploads/'),
  filename: (req, file, cb) => {
    const uniqueSuffix = Date.now() + '-' + Math.round(Math.random() * 1E9);
    cb(null, file.fieldname + '-' + uniqueSuffix + path.extname(file.originalname));
  }
});

// File validation
const fileFilter = (req, file, cb) => {
  if (file.mimetype === 'image/jpeg' || file.mimetype === 'image/png') {
    cb(null, true);
  } else {
    cb(new Error('Invalid file type'), false);
  }
};

const upload = multer({ 
  storage, 
  fileFilter,
  limits: { fileSize: 5 * 1024 * 1024 } // 5MB limit
});

app.post('/upload', upload.single('avatar'), (req, res) => {
  if (!req.file) return res.status(400).send('No file uploaded.');
  res.json({ message: 'File uploaded', path: req.file.path });
});

Break It

What happens if a user uploads a 10GB file but your multer limits are not set? The server will spend significant CPU time and disk I/O saving this massive file, potentially filling up the disk completely, leading to a Denial of Service (DoS).

What if a user uploads a file named ../../../etc/passwd? If you blindly use file.originalname without sanitization (which multer protects against, but raw streams might not), you might overwrite critical system files (Path Traversal).

Debug It

If uploads are failing, check:

  1. Is the frontend sending Content-Type: multipart/form-data?
  2. Is Nginx/Reverse Proxy configured with a high enough client_max_body_size? (Nginx defaults to 1MB and will return 413 Payload Too Large).
  3. Are the directory permissions correct for the Node process to write to the uploads/ folder?

Mini Project (20-30 min)

Build an image upload microservice. It should take a raw image upload, use the sharp library to resize it to a 256x256 thumbnail, convert it to WebP format, and stream the result to an AWS S3 bucket. Finally, it should return the public URL of the resized image.

Real Application Feature

In a production system like a social network, you don't serve user avatars directly from S3. You configure a Content Delivery Network (CDN) like CloudFront or Cloudflare in front of S3. The CDN caches the image at edge locations worldwide. When a user in Tokyo requests the image, it's served from a Tokyo data center, not your primary S3 bucket in Virginia.

Production Implementation

A true production pipeline handles files asynchronously:

  1. Client uploads direct to S3 using a Signed URL.
  2. S3 emits an ObjectCreated event.
  3. This triggers a serverless function (AWS Lambda) or pushes to an SQS queue.
  4. A worker picks up the job, downloads the image, runs virus scanning (e.g., ClamAV), generates thumbnails, and strips EXIF metadata (which can contain GPS coordinates).
  5. The worker moves the clean, processed image to a "Public" bucket and updates the database record.

Production Usage

For file downloads, never pipe files through your Node server. Always generate a Signed GET URL, or if the file is public, serve it via the CDN. If you must control access, use CDN Signed Cookies or URL signatures at the CDN edge.

Performance

  • Streaming: Keeps memory footprint O(1) regardless of file size.
  • Direct to Cloud: Removes the application server from the data path, freeing up connection pools.
  • CDN: Drastically reduces latency and offloads egress bandwidth costs from your primary infrastructure.

Best Practices

  • Never trust user input: Validate file extensions, check Magic Numbers (file signatures), and scan for viruses.
  • Always use limits: Enforce strict file size limits on the proxy layer (Nginx) and application layer.
  • Sanitize filenames: Generate your own UUIDs for filenames; never use the user-provided filename on disk or object storage keys.
  • Strip Metadata: Remove EXIF data from images to protect user privacy.

Interview Questions

Easy: Why shouldn't you store user uploads in the same repository as your code?

Because it makes the repository huge, it gets wiped out on fresh deployments if not volume-mounted, and it prevents horizontal scaling since servers don't share local disks.

Medium: What is the difference between Base64 JSON uploads and Multipart Form Data?

Base64 encoding inflates the payload size by about 33% and usually requires loading the entire string into memory. Multipart streams binary data directly with boundary markers, allowing efficient chunking and piping.

Hard: How do you handle resumable uploads for a 10GB video file over a flaky connection?

You use a protocol like Tus or S3 Multipart Upload. The client chunks the file (e.g., 5MB parts) and uploads them in parallel or sequentially. The server keeps track of received chunks. If the connection drops, the client only retries the failed chunks. Once all chunks are received, the server stitches them together.

Senior: Describe the architecture of an image processing pipeline that scales to millions of uploads.

Direct client-to-S3 uploads via presigned URLs -> S3 Event Notifications -> SQS Queue -> Auto-scaling worker cluster (or Lambda). Workers pull jobs, stream the object to memory, use sharp/libvips for processing, stream back to a public S3 bucket, and update the Postgres DB. A CDN fronts the public bucket. The API server is never involved in the binary transfer.

Engineering Challenge

Implement an endpoint that accepts a large CSV file upload, streams the file line-by-line using csv-parser, transforms each row, and streams the transformed data directly to S3 without ever writing the file to the local disk or holding the whole array in memory.

View Conceptual Solution

Use Busboy to capture the file stream. Pipe the file stream into the csv-parser stream. Pipe the output of the parser into a transform stream. Finally, pass this transform stream to the @aws-sdk/lib-storage Upload class, which can read from a Node stream and execute an S3 Multipart upload automatically.

Revision Sheet

  • Local Storage: Bad for scale. Needs NAS/SAN.
  • Object Storage (S3): Infinite, flat, scalable storage via HTTP.
  • Multipart Form: The HTTP standard for sending binary files.
  • Streaming: Processing data in chunks to save RAM.
  • Signed URLs: Cryptographic tickets for clients to interact directly with S3.
  • CDN: Edge caching for fast global delivery.

Connections

File handling heavily relies on understanding Streams and Buffers (Part 4). Offloading image processing to workers is a prime example of Message Queues and Background Jobs (Part 22). Securing those uploads requires the concepts learned in Web Security (Part 15).

PreviousVolume 17 🏠 Curriculum NextVolume 19

Bigger Project (1-2 hours)

Create an Express endpoint with Multer that accepts a file upload, validates the file type, and saves it to a local directory simulating S3.

▶ View Solution
typescript 
// Implementation for Files & Storage
console.log("Bigger project solution");

Interview Questions

History

To solve the HTTP binary transmission problem, multipart/form-data was introduced in RFC 1867. For scaling storage, companies moved from local disks to Network Attached Storage (NAS) and Storage Area Networks (SAN). But these were expensive and complex to maintain. In 2006, Amazon launched Simple Storage Service (S3), popularizing Object Storage. Instead of directories and files, we now have flat "buckets" and "objects" accessed via HTTP APIs. CDNs evolved alongside to cache these objects globally.

Analogy: Imagine you are a receptionist at a busy office (the Application Server). A courier arrives with a massive truckload of furniture (a large file upload). If you try to bring all the furniture into your small reception area (Server RAM), you'll be crushed.

Reality (Streaming): Instead, you direct the workers to carry the furniture piece by piece directly to the warehouse. You never hold all the furniture at once. This is Streaming.

Reality (Object Storage & Signed URLs): Even better, what if you just gave the courier a special security pass (a Signed URL) and told them to drive straight to the massive off-site warehouse (S3 Bucket) themselves? You never even touch the furniture. Your reception stays completely clear for other business.

Internal Working (Memory, stack, process, network, etc.)

When a client uploads a file via multipart/form-data, the HTTP body is divided into "parts", separated by a boundary string defined in the Content-Type header. The server reads the TCP socket stream. A parser (like Busboy or Multer in Node.js) looks for these boundaries. Instead of accumulating the data in a Buffer (RAM), it emits data events. These chunks can be piped directly into a fs.createWriteStream or a cloud storage upload stream.

When using Signed URLs, the server uses a secret key to cryptographically sign a URL granting temporary write access to a specific S3 path. The client receives this URL and makes a direct PUT request to S3. The application server's network and CPU are completely bypassed for the actual file transfer.

Visual Explanation (ASCII diagrams)

TRADITIONAL UPLOAD (BAD FOR SCALE):
Client ---(File 50MB)---> App Server (Parses, buffers) ---(Saves)---> Local Disk
                                      |
                                  CRASH (Out of Memory)

STREAMING UPLOAD:
Client ---(Chunk 1)---> App Server --(Pipe)--> S3/Disk
       ---(Chunk 2)---> App Server --(Pipe)--> S3/Disk
       ---(Chunk 3)---> App Server --(Pipe)--> S3/Disk

SIGNED URL DIRECT UPLOAD (BEST):
1. Client -> App Server: "I want to upload avatar.png"
2. App Server -> Client: "Here is a Signed URL for S3"
3. Client ---(File 50MB)---> S3 Bucket (Direct HTTP PUT)
4. Client -> App Server: "Upload done, here is the object key"

Syntax

Creating a Signed URL (AWS SDK v3):

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const client = new S3Client({ region: "us-east-1" });
const command = new PutObjectCommand({ Bucket: "my-bucket", Key: "user-123/avatar.png" });
const url = await getSignedUrl(client, command, { expiresIn: 3600 });

Tiny Example

Uploading directly to S3 from the browser using the signed URL:

// Frontend JavaScript
const fileInput = document.querySelector('input[type="file"]');
const file = fileInput.files[0];

// 1. Get signed URL from our backend
const response = await fetch('/api/upload-url?filename=' + file.name);
const { uploadUrl } = await response.json();

// 2. Upload directly to S3
await fetch(uploadUrl, {
  method: 'PUT',
  body: file,
  headers: { 'Content-Type': file.type }
});

Walkthrough

Let's build a multipart form upload handler using multer to stream a file to disk, perform basic validation, and return the file path.

import express from 'express';
import multer from 'multer';
import path from 'path';
import fs from 'fs';

const app = express();

// Configure storage
const storage = multer.diskStorage({
  destination: (req, file, cb) => cb(null, 'uploads/'),
  filename: (req, file, cb) => {
    const uniqueSuffix = Date.now() + '-' + Math.round(Math.random() * 1E9);
    cb(null, file.fieldname + '-' + uniqueSuffix + path.extname(file.originalname));
  }
});

// File validation
const fileFilter = (req, file, cb) => {
  if (file.mimetype === 'image/jpeg' || file.mimetype === 'image/png') {
    cb(null, true);
  } else {
    cb(new Error('Invalid file type'), false);
  }
};

const upload = multer({ 
  storage, 
  fileFilter,
  limits: { fileSize: 5 * 1024 * 1024 } // 5MB limit
});

app.post('/upload', upload.single('avatar'), (req, res) => {
  if (!req.file) return res.status(400).send('No file uploaded.');
  res.json({ message: 'File uploaded', path: req.file.path });
});

Break It

What happens if a user uploads a 10GB file but your multer limits are not set? The server will spend significant CPU time and disk I/O saving this massive file, potentially filling up the disk completely, leading to a Denial of Service (DoS).

What if a user uploads a file named ../../../etc/passwd? If you blindly use file.originalname without sanitization (which multer protects against, but raw streams might not), you might overwrite critical system files (Path Traversal).

Debug It

If uploads are failing, check:

  1. Is the frontend sending Content-Type: multipart/form-data?
  2. Is Nginx/Reverse Proxy configured with a high enough client_max_body_size? (Nginx defaults to 1MB and will return 413 Payload Too Large).
  3. Are the directory permissions correct for the Node process to write to the uploads/ folder?

Mini Project (20-30 min)

Build an image upload microservice. It should take a raw image upload, use the sharp library to resize it to a 256x256 thumbnail, convert it to WebP format, and stream the result to an AWS S3 bucket. Finally, it should return the public URL of the resized image.

Real Application Feature

In a production system like a social network, you don't serve user avatars directly from S3. You configure a Content Delivery Network (CDN) like CloudFront or Cloudflare in front of S3. The CDN caches the image at edge locations worldwide. When a user in Tokyo requests the image, it's served from a Tokyo data center, not your primary S3 bucket in Virginia.

Production Implementation

A true production pipeline handles files asynchronously:

  1. Client uploads direct to S3 using a Signed URL.
  2. S3 emits an ObjectCreated event.
  3. This triggers a serverless function (AWS Lambda) or pushes to an SQS queue.
  4. A worker picks up the job, downloads the image, runs virus scanning (e.g., ClamAV), generates thumbnails, and strips EXIF metadata (which can contain GPS coordinates).
  5. The worker moves the clean, processed image to a "Public" bucket and updates the database record.

Production Usage

For file downloads, never pipe files through your Node server. Always generate a Signed GET URL, or if the file is public, serve it via the CDN. If you must control access, use CDN Signed Cookies or URL signatures at the CDN edge.

Performance

  • Streaming: Keeps memory footprint O(1) regardless of file size.
  • Direct to Cloud: Removes the application server from the data path, freeing up connection pools.
  • CDN: Drastically reduces latency and offloads egress bandwidth costs from your primary infrastructure.

Best Practices

  • Never trust user input: Validate file extensions, check Magic Numbers (file signatures), and scan for viruses.
  • Always use limits: Enforce strict file size limits on the proxy layer (Nginx) and application layer.
  • Sanitize filenames: Generate your own UUIDs for filenames; never use the user-provided filename on disk or object storage keys.
  • Strip Metadata: Remove EXIF data from images to protect user privacy.

Interview Questions

Easy: Why shouldn't you store user uploads in the same repository as your code?

Because it makes the repository huge, it gets wiped out on fresh deployments if not volume-mounted, and it prevents horizontal scaling since servers don't share local disks.

Medium: What is the difference between Base64 JSON uploads and Multipart Form Data?

Base64 encoding inflates the payload size by about 33% and usually requires loading the entire string into memory. Multipart streams binary data directly with boundary markers, allowing efficient chunking and piping.

Hard: How do you handle resumable uploads for a 10GB video file over a flaky connection?

You use a protocol like Tus or S3 Multipart Upload. The client chunks the file (e.g., 5MB parts) and uploads them in parallel or sequentially. The server keeps track of received chunks. If the connection drops, the client only retries the failed chunks. Once all chunks are received, the server stitches them together.

Senior: Describe the architecture of an image processing pipeline that scales to millions of uploads.

Direct client-to-S3 uploads via presigned URLs -> S3 Event Notifications -> SQS Queue -> Auto-scaling worker cluster (or Lambda). Workers pull jobs, stream the object to memory, use sharp/libvips for processing, stream back to a public S3 bucket, and update the Postgres DB. A CDN fronts the public bucket. The API server is never involved in the binary transfer.

Engineering Challenge

Implement an endpoint that accepts a large CSV file upload, streams the file line-by-line using csv-parser, transforms each row, and streams the transformed data directly to S3 without ever writing the file to the local disk or holding the whole array in memory.

View Conceptual Solution

Use Busboy to capture the file stream. Pipe the file stream into the csv-parser stream. Pipe the output of the parser into a transform stream. Finally, pass this transform stream to the @aws-sdk/lib-storage Upload class, which can read from a Node stream and execute an S3 Multipart upload automatically.

Revision Sheet

  • Local Storage: Bad for scale. Needs NAS/SAN.
  • Object Storage (S3): Infinite, flat, scalable storage via HTTP.
  • Multipart Form: The HTTP standard for sending binary files.
  • Streaming: Processing data in chunks to save RAM.
  • Signed URLs: Cryptographic tickets for clients to interact directly with S3.
  • CDN: Edge caching for fast global delivery.

Connections

File handling heavily relies on understanding Streams and Buffers (Part 4). Offloading image processing to workers is a prime example of Message Queues and Background Jobs (Part 22). Securing those uploads requires the concepts learned in Web Security (Part 15).

PreviousVolume 17 🏠 Curriculum NextVolume 19

Bigger Project (1-2 hours)

Create an Express endpoint with Multer that accepts a file upload, validates the file type, and saves it to a local directory simulating S3.

▶ View Solution
typescript 
// Implementation for Files & Storage
console.log("Bigger project solution");

Interview Questions

Mental Model (Analogy -> Reality)

🏠 Curriculum NextVolume 2