Operations reliability

Doctrine

• Runbooks are the single source of truth for emergency execution.
• Every rollback path must be pre-approved, tested, and owned.
• Capacity warning thresholds trigger controlled traffic control actions.
• Each handoff captures findings exports and visible owner updates.

Decision tree

1. Are customers currently impacted?
   • Yes → open incident playbook and escalate priority.
   • No → run controlled monitoring and classify as preventive action.
2. Is rollback safe and reversible now?
   • Yes → execute rollback sequence with evidence capture.
   • No → apply containment and isolate blast radius first.
3. Did capacity exceed pre-warning thresholds?
   • Yes → enforce rate limits and queue controls.
   • No → continue with mitigations and post-incident review prep.

Diagnostics pathway

1. Validate telemetry burst and alert posture:

   rg -n "critical|warning|page" /var/log/alerts/*.log

2. Inspect system saturation:

   uptime; vmstat 1 5; df -h; free -m

3. Collect dependency failure sequence:

   systemctl status nginx myservice; journalctl -u nginx -n 50

4. Verify rollback and handoff readiness:

   git log --oneline -- operations/runbooks/ | head

Operational checks

• Runbook audit with failure signature simulation at least monthly.
• Track median time to rollback and median time to recover separately.
• Confirm each closed incident has evidence bundle and handoff confirmation.

Related routes