Operations observability

Doctrine

• Use shared naming for metrics and tags across services.
• Correlate alerts with user-facing impact, not only error counts.
• Keep dashboards tied to owners, not only technical completeness.
• Set explicit suppression and escalation in each alert definition.

Decision tree

1. Is alert tied to user-facing impact?
   • Yes → run priority-1 triage and on-call escalation immediately.
   • No → move to batch triage and monitor for amplification.
2. Can metrics and logs be joined in under 60 seconds?
   • Yes → estimate blast radius and continue with runbook.
   • No → switch to structured fallback telemetry capture and notify operator.
3. Is alert volume still high after suppression?
   • Yes → pause noisy source and create temporary suppression ticket.
   • No → keep policy and document confidence score.

Diagnostics pathway

1. Validate signal pipeline integrity:

   rg -i "ingest|buffer|drop" /var/log/observability/* | tail

2. Check error bursts vs latency growth:

   for k in p95_latency error_rate cpu_usage; do echo "$k"; done

3. Inspect trace continuity around incident window:

   rg -n "trace_id|span_id|error" /var/log/traces/*.log | head

4. Validate alert policy state:

   cat /etc/ops/alert-config.yml | rg -n "repeat_interval|for:|group_wait"

Operational checks

• Run weekly fire-drill with synthetic load to validate alert-to-resolution timing.
• Verify top noise alerts have documented owners and review dates.
• Cross-check dashboards for stale fields and missing labels each sprint.

Related routes