Runbook Generator

Tier: POWERFUL
Category: Engineering
Domain: DevOps / Site Reliability Engineering

Overview

Analyze a codebase and generate production-grade operational runbooks. Detects your stack (CI/CD, database, hosting, containers), then produces step-by-step runbooks with copy-paste commands, verification checks, rollback procedures, escalation paths, and time estimates. Keeps runbooks fresh with staleness detection linked to config file modification dates.

Core Capabilities

• - Stack detection — auto-identify CI/CD, database, hosting, orchestration from repo files
• Runbook types — deployment, incident response, database maintenance, scaling, monitoring setup
• Format discipline — numbered steps, copy-paste commands, ✅ verification checks, time estimates
• Escalation paths — L1 → L2 → L3 with contact info and decision criteria
• Rollback procedures — every deployment step has a corresponding undo
• Staleness detection — runbook sections reference config files; flag when source changes
• Testing methodology — dry-run framework for staging validation, quarterly review cadence

When to Use

Use when:

• - A codebase has no runbooks and you need to bootstrap them fast
• Existing runbooks are outdated or incomplete (point at the repo, regenerate)
• Onboarding a new engineer who needs clear operational procedures
• Preparing for an incident response drill or audit
• Setting up monitoring and on-call rotation from scratch

Skip when:

• - The system is too early-stage to have stable operational patterns
• Runbooks already exist and only need minor updates (edit directly)

Stack Detection

When given a repo, scan for these signals before writing a single runbook line:

CODEBLOCK0

Map detected stack → runbook templates. A Next.js + PostgreSQL + Vercel + GitHub Actions repo needs:

• - Deployment runbook (Vercel + GitHub Actions)
• Database runbook (PostgreSQL backup, migration, vacuum)
• Incident response (with Vercel logs + pg query debugging)
• Monitoring setup (Vercel Analytics, pg_stat, alerting)

Runbook Types

1. Deployment Runbook

CODEBLOCK1bash
pnpm test
pnpm lint
pnpm build

✅ Expected: All pass with 0 errors. Build output in `.next/`

### Step 2 — Apply database migrations (5 min)

Staging first

✅ Expected: `All migrations have been successfully applied.`

Verify migration applied

✅ Expected: Migration table shows new entry with today's date

### Step 3 — Deploy to production (5 min)

OR trigger manually:

✅ Expected: Vercel dashboard shows deployment in progress. URL format:
`https://app-name-<hash>-team.vercel.app`

### Step 4 — Smoke test production (5 min)

Health check

Critical path

✅ Expected: health returns `{"status":"ok","db":"connected"}`. Users API returns valid ID.

### Step 5 — Monitor for 10 min
- Check Vercel Functions log for errors: `vercel logs --since=10m`
- Check error rate in Vercel Analytics: < 1% 5xx
- Check DB connection pool: `SELECT count(*) FROM pg_stat_activity;` (< 80% of max_connections)

---

## Rollback

If smoke tests fail or error rate spikes:

Instant rollback via Vercel (preferred — < 30 sec)

Database rollback (only if migration was applied)

WARNING: This resets to previous migration. Confirm data impact first.

✅ Expected after rollback: Previous deployment URL becomes active. Verify with smoke test.

---

## Escalation
- **L1 (on-call engineer):** Check Vercel logs, run smoke tests, attempt rollback
- **L2 (platform lead):** DB issues, data loss risk, rollback failed — Slack: @platform-lead
- **L3 (CTO):** Production down > 30 min, data breach — PagerDuty: #critical-incidents

2. Incident Response Runbook

CODEBLOCK8bash

Is the app responding?

Check Vercel function errors (last 15 min)

✅ 200 = app up. 5xx or timeout = incident confirmed.

Declare severity:
- Site completely down → P1 — page L2/L3 immediately
- Partial degradation / slow responses → P2 — notify team channel
- Single feature broken → P3 — create ticket, fix in business hours

---

## Phase 2 — Diagnose (10–15 min)

Recent deployments — did something just ship?

Database health

Long-running queries (> 30 sec)

Connection pool saturation

Diagnostic decision tree:
- Recent deploy + new errors → rollback (see Deployment Runbook)
- DB query timeout / pool saturation → kill long queries, scale connections
- External dependency failing → check status pages, add circuit breaker
- Memory/CPU spike → check Vercel function logs for infinite loops

---

## Phase 3 — Mitigate (variable)

Kill a runaway DB query

---

## Phase 4 — Resolve & Postmortem

After incident is resolved, within 24 hours:

1. Write incident timeline (what happened, when, who noticed, what fixed it)
2. Identify root cause (5-Whys)
3. Define action items with owners and due dates
4. Update this runbook if a step was missing or wrong
5. Add monitoring/alert that would have caught this earlier

**Postmortem template:** `docs/postmortems/YYYY-MM-DD-incident-title.md`

---

## Escalation Path

| Level | Who | When | Contact |
|-------|-----|------|---------|
| L1 | On-call engineer | Always first | PagerDuty rotation |
| L2 | Platform lead | DB issues, rollback needed | Slack @platform-lead |
| L3 | CTO/VP Eng | P1 > 30 min, data loss | Phone + PagerDuty |

✅ Expected: File created, size > 0. `pg_restore --list backup.dump | head -20` shows tables.

Verify backup is restorable (test monthly):

✅ Expected: Row count matches production.

## Migration

✅ Expected: `All migrations have been successfully applied.`

⚠️ For large table migrations (> 1M rows), use `pg_repack` or add column with DEFAULT separately to avoid table locks.

## Vacuum & Reindex

✅ Expected: dead_ratio drops below 5% after vacuum.