The production readiness checklist

This is the list we work through before a service takes production traffic. It is deliberately not exhaustive — an exhaustive checklist gets skimmed. Everything here has caused a real incident on a real project.

Can you tell when it breaks?

• A dashboard showing request rate, error rate and latency percentiles — p50, p95 and p99, not the mean
• Alerts on customer-visible symptoms rather than on CPU or memory
• Structured logs with a request identifier that traverses service boundaries
• Distributed tracing enabled, at least sampled

Can you undo it?

• Rollback tested, not assumed — someone has actually run it this month
• Schema migrations that are backward compatible with the previous release
• Feature flags for anything user-visible, each with an owner and a removal date
• A documented, rehearsed data-restore procedure

Does it degrade gracefully?

• Timeouts set on every outbound call, including the ones to your own services
• Retries with backoff and jitter, and a cap that prevents amplification
• Circuit breakers on dependencies that can fail independently
• A defined behaviour for each dependency being unavailable — including 'refuse clearly'

The last one is skipped most often. Teams handle the failure technically but never decide what the user sees, and the default is a spinner that never resolves.

Is it safe?

• Secrets in a secret manager, never in environment files committed to a repository
• Dependencies scanned, with a named owner for the findings
• Authentication and authorisation tested, including the negative cases
• Personal data classified, with retention actually implemented rather than documented

Can someone else run it?

• A runbook for each alert, written so someone unfamiliar can act on it at 3am
• An on-call rotation with a real escalation path, tested at least once
• Architecture decision records explaining why, not just what
• A named owner — a team, not an individual

Does it hold under load?

• Load tested at expected peak, then at two to three times peak
• Known bottleneck identified — there always is one; not knowing it is the problem
• Autoscaling tested with a real traffic ramp, including scale-down
• Cost per unit of traffic understood before the first bill arrives