🌙
☀️ Dark
PART 37

Real Product Development

Requirements, estimation, technical debt, ADRs.

Advanced 45 min read

Part 37: Real Product Development

Learning Objectives

You will learn the complete lifecycle of building a real product — from a vague idea to a deployed, monitored, production system. This chapter synthesizes everything in the curriculum into a single, coherent engineering workflow.

Why Does This Exist?

Technical skills without product judgment produce engineers who build the wrong thing, correctly. The ability to translate a product idea into a deployable system — with clear requirements, sensible architecture, and a realistic plan — is what separates engineers who ship from engineers who rebuild.

The Product Development Lifecycle

Idea
  ↓
Requirements (What are we actually building?)
  ↓
User Flows (Who does what, and in what order?)
  ↓
Architecture (How does it work technically?)
  ↓
Database Design (What data do we need, and how is it related?)
  ↓
API Design (What are the interfaces between frontend and backend?)
  ↓
MVP Scope (What is the minimum thing worth shipping?)
  ↓
Implementation (Build it. Test it. Review it.)
  ↓
Deployment (Docker, CI/CD, cloud)
  ↓
Monitoring (Logs, metrics, alerts)
  ↓
Iteration (Measure → Learn → Improve)

Requirements Gathering

Requirements that seem obvious are often ambiguous. The best engineers ask clarifying questions before writing a line of code.

Functional Requirements (What the system does):
  ✓ Users can register with email and password
  ✓ Users can create projects
  ✓ Team members can be invited to projects
  ✓ Files can be uploaded to projects (max 50MB, images/PDFs)
  ✓ Admins can view all projects

Non-Functional Requirements (How the system behaves):
  ✓ Page load time under 2 seconds (p95)
  ✓ 99.9% availability
  ✓ File upload within 10 seconds for 50MB
  ✓ Support 10,000 concurrent users at launch

Questions you must always ask:
  - Who are the users? (One type? Multiple roles?)
  - What is the scale? (10 users? 10,000? 10M?)
  - What is the data retention policy?
  - What are the compliance requirements? (GDPR? HIPAA?)
  - What is the rollout plan? (Big bang? Feature flags? Beta?)

Technical Planning

Before sprinting, spend time on a minimal architecture design. An hour of planning saves days of refactoring.

Architecture Decision Records (ADRs):
  Document every significant technical decision.
  
  docs/adr/
  ├── 001-use-postgresql-not-mongodb.md
  ├── 002-use-redis-for-sessions.md
  ├── 003-use-s3-for-file-storage.md
  └── 004-modular-monolith-over-microservices.md

ADR format:
  # ADR-003: Use S3 for File Storage
  
  ## Status: Accepted
  
  ## Context
  Users upload files. Files need to be stored durably, served fast,
  and scale to TB of storage.
  
  ## Decision
  Use AWS S3 with CloudFront CDN for file storage and serving.
  
  ## Consequences
  ✓ Virtually unlimited storage, 99.999999999% durability
  ✓ CDN integration for fast global access
  ✗ Requires presigned URL generation (adds backend step)
  ✗ Additional AWS dependency
  
  ## Alternatives Considered
  - Local disk: doesn't scale, single point of failure
  - Database BLOBs: terrible performance, hard to migrate

Feature Decomposition

Breaking large features into shippable increments:

Feature: "Users can collaborate on documents"
Too big to ship as one unit. Break it down:

Sprint 1: Document creation
  ✓ Create a document (title, content)
  ✓ View your documents
  ✓ Delete a document

Sprint 2: Document editing
  ✓ Edit document content (markdown)
  ✓ Auto-save draft every 30 seconds
  ✓ View edit history (version list)

Sprint 3: Sharing
  ✓ Share document via link (read-only)
  ✓ Invite specific users by email
  ✓ Set permissions (view/edit)

Sprint 4: Real-time collaboration
  ✓ See who is viewing the document
  ✓ Real-time cursor presence
  ✓ Conflict resolution (OT or CRDT)

Each sprint is independently shippable and testable.
Users get value from Sprint 1 without waiting for Sprint 4.

Estimation

Engineering estimation is hard. Here is a framework:

1. Break work into tasks under 2 days each
2. Estimate each task with 3 numbers:
   - Optimistic (everything goes right)
   - Most likely (some things go wrong)
   - Pessimistic (many things go wrong)
3. Use PERT formula: (O + 4M + P) / 6

Common estimation mistakes:
  ✗ Estimating only the "happy path" (no bugs, no review cycles)
  ✗ Forgetting integration work, testing, and documentation
  ✗ Ignoring onboarding a new technology
  ✗ No buffer for unknowns (always add 20-30%)

The 90% problem:
  "I'm 90% done" often means "I've written the code, now I need
  to test it, handle edge cases, review it, fix review feedback,
  write tests, deploy it, and fix the production bugs."
  
  Tasks that are "90% done" are often 50% done.

MVP Design

MVP (Minimum Viable Product) ≠ Minimal Viable Product

MVP means: the smallest thing that:
  1. Solves the core user problem
  2. Can be used by real users
  3. Provides enough signal to validate or invalidate your hypothesis

What to cut from an MVP:
  ✗ Advanced search and filtering (use simple list first)
  ✗ Notifications (manual until users need it)
  ✗ Mobile app (responsive web first)
  ✗ Admin dashboard (use database directly)
  ✗ Analytics (use simple event logs)
  ✗ Advanced permissions (everyone is admin until proven otherwise)

What must be in an MVP:
  ✓ Core user journey (the one thing users come for)
  ✓ Authentication (or you can't measure individual behavior)
  ✓ Basic error handling (crashes kill trust immediately)
  ✓ Data persistence (data loss is catastrophic even in beta)
  ✓ HTTPS (non-negotiable)

Technical Debt Management

Technical debt is not always bad. Intentional shortcuts to ship faster
are a business decision — not an engineering failure.

Good technical debt:
  "We'll use a SQL query without pagination for now.
   We'll add pagination when we have more than 1000 records."
  → Deliberate, documented, has a trigger for payback

Bad technical debt:
  "We don't have time for error handling."
  → No trigger. Will cause production incidents.

Debt management:
  ✓ Document shortcuts in code comments and ADRs
  ✓ Create tickets for every known debt item
  ✓ Allocate 20% of each sprint to debt reduction
  ✓ Never let debt block critical security or reliability fixes
  ✓ Treat debt as a first-class engineering concern, not guilt

Code Review as Engineering Practice

Good code reviews are not gatekeeping — they are knowledge sharing.

As a reviewer:
  ✓ Review the design before the implementation details
  ✓ Ask clarifying questions rather than demanding changes
  ✓ Distinguish: "This must change" vs "This is my preference"
  ✓ Acknowledge good work explicitly
  ✓ Review for security and edge cases, not just style
  ✗ Don't nitpick formatting (automate that with Prettier/ESLint)
  ✗ Don't block PRs on minor style preferences

As an author:
  ✓ Small PRs are reviewed faster and better (< 400 lines)
  ✓ Provide context in the PR description: why, not just what
  ✓ Self-review your PR before requesting review
  ✓ Respond to every comment (even just "Done" or "Noted")
  ✓ Don't take feedback personally — the code is not you

PR description template:
  ## What
  Brief description of what changed.
  
  ## Why
  Business or engineering reason for this change.
  
  ## How
  Key implementation decisions (especially non-obvious ones).
  
  ## Testing
  How was this tested? What edge cases were covered?
  
  ## Screenshots (for UI changes)

Incident Response

When production is on fire:

1. DETECT: Alert fires (or user reports)
2. COMMUNICATE: Post in #incidents: "Investigating reports of X"
3. MITIGATE: Fix the symptom, not the root cause
   → Roll back the last deployment
   → Scale up the affected service
   → Enable maintenance mode
4. RESOLVE: Production is healthy again. Communicate resolution.
5. INVESTIGATE: RCA (Root Cause Analysis) within 24-48 hours
6. PREVENT: Fix the root cause + add monitoring to detect earlier

Postmortem template (Blameless):
  ## Incident Summary
  What happened? When? Impact?
  
  ## Timeline
  Minute-by-minute account of events.
  
  ## Root Cause
  The technical explanation of why it happened.
  
  ## Contributing Factors
  What conditions made this incident possible?
  
  ## Action Items
  | Item | Owner | Due Date | Status |
  | ---- | ----- | -------- | ------ |
  | Add alert for X | @engineer | 2025-02-01 | Open |
  
  ## What Went Well
  ## What Could Be Improved

The Capstone Checklist

Before calling any production system "done," verify each layer:

FUNCTIONAL
  ✓ Core user journey works end-to-end
  ✓ Error states are handled (not just happy path)
  ✓ Input validation on all API endpoints

SECURITY
  ✓ Authentication implemented and tested
  ✓ Authorization enforced (users can't access other users' data)
  ✓ HTTPS enforced everywhere
  ✓ Secrets in environment variables, not code
  ✓ Dependencies scanned for vulnerabilities

RELIABILITY
  ✓ Health check endpoints implemented
  ✓ Graceful shutdown implemented
  ✓ Database connection pooling configured
  ✓ Rate limiting on public endpoints

OBSERVABILITY
  ✓ Structured JSON logging
  ✓ Error tracking (Sentry or equivalent)
  ✓ Uptime monitoring with alerts
  ✓ Key metrics tracked (error rate, latency, traffic)

OPERATIONS
  ✓ Docker containerized
  ✓ CI/CD pipeline running on every push
  ✓ Database migrations in version control
  ✓ Rollback plan documented
  ✓ Backup and restore tested

Mini Project (20-30 min)

Create a simple launch checklist generator that validates production readiness.

▶ View Solution
typescript
function validateLaunchReadiness(checklist: Record) {
  const mandatory = ['ci_cd_passing', 'load_tested', 'monitoring_active', 'rollback_plan'];
  const missing = mandatory.filter(item => !checklist[item]);
  
  if (missing.length > 0) {
    return { ready: false, blockers: missing };
  }
  return { ready: true, message: 'Go for launch 🚀' };
}

Bigger Project (1-2 hours)

Simulate a full product lifecycle: Draft an RFC (Request for Comments) for a new feature, set up a CI/CD pipeline with automated testing and linting, and deploy a blue-green release strategy to ensure zero-downtime deployment.

\n
▶ View Solution
yaml
name: CI/CD Pipeline
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - run: npm ci
      - run: npm test

Interview Questions

\n
Core: Why might a team choose to use Feature Flags?

Feature flags decouple deployment from release. This allows teams to push incomplete code to production without exposing it to users, perform canary rollouts, A/B testing, and quickly rollback a broken feature by simply toggling the flag without a new deployment.

Hard: As a tech lead, how do you balance shipping features quickly against managing technical debt?

I advocate for a strategic allocation model, dedicating roughly 70% of capacity to new features, 20% to refactoring/technical debt, and 10% to innovation or operational improvements. Technical debt is tracked in the backlog just like features, quantified by its impact on velocity and system stability, ensuring it gets prioritized before it becomes a bottleneck.

Senior: Describe how you would resolve a fundamental architectural disagreement between two senior engineers on your team.

I would facilitate an architecture review focused on objective criteria. We would define the system requirements, constraints, and success metrics first. Then, both engineers would present their solutions using Design Docs (RFCs) evaluating trade-offs (e.g., complexity vs. scalability). If consensus isn't reached, we test assumptions with a time-boxed proof-of-concept, letting data dictate the final decision.

Final Words: The Engineer's Mindset

You have reached the end of The Engineer's Bible — Full-Stack Engineering. Here is what should be different about how you think now.

You no longer think: "I know React."
You think: "I understand the browser rendering model, the virtual DOM,
            React's reconciliation algorithm, and when React is the
            wrong tool entirely."

You no longer think: "I built an API."
You think: "I designed a resource-oriented API with proper error formats,
            idempotency, rate limiting, versioning, and documentation."

You no longer think: "I deployed something."
You think: "I containerized it, ran it through CI, deployed it to a
            load-balanced cluster, and set up alerts so I'll know
            when it fails before users do."

The question you should ask about every system you build:
  What happens when 10x more users arrive tomorrow?
  What fails first? How do I know when it's failing?
  How do I fix it without taking the service down?
  How do I roll back safely if something goes wrong?

Engineering is not about knowing every technology.
It is about being able to reason about systems you have never seen before.
It is about making trade-offs you can defend.
It is about building things that work reliably for real people.

That is what you are now capable of.

Revision Sheet

  • Requirements: Functional + Non-functional. Ask the clarifying questions others skip.
  • ADRs: Document every significant decision, including the alternatives you rejected.
  • MVP: The smallest thing that validates your core assumption. Cut everything else.
  • Estimation: Optimistic / Most Likely / Pessimistic. Always add buffer for unknowns.
  • Debt: Intentional debt is a business decision. Untracked debt is a liability.
  • Incidents: Blameless. Fix symptoms first. Root cause next. Postmortem always.
🏠 Curriculum NextVolume 2