Infrastructure scaling

Doctrine

• Every service has explicit capacity envelope and saturation thresholds.
• Scaling actions must include guardrails, cooldowns, and rollback deadlines.
• Dependency chains are part of every scale decision, not an afterthought.
• Load increases require evidence of sustained need, not one-shot spikes.

Decision tree

1. Is this sustained demand or burst behavior?
   • Sustained → evaluate horizontal and capacity-first growth.
   • Burst only → optimize cache and queue, then revisit auto scale triggers.
2. Can dependencies absorb increased throughput?
   • Yes → scale the caller and verify throughput gain.
   • No → scale dependencies or reduce upstream fan-out first.
3. Did post-change errors reduce?
   • Yes → keep scaled state and monitor long-term.
   • No → revert within rollback budget and escalate scaling owner.

Diagnostics pathway

1. Collect saturation and load data:

   top -b -n 3 | head; vmstat 1 5; iostat -xz 1 3

2. Inspect queue depth and network pressure:

   ss -s; ss -tanp | rg -i "SYN|TIME-WAIT"

3. Measure latency/throughput before and after scale action:

   wrk -t8 -c400 -d20s https://app.example.com/health

4. Validate dependency health and cascading signals:

   rg -n "upstream|dependency|timeout|reset" /var/log/system/*.log

Operational checks

• Keep a scaling runbook with pre/post commands and rollback script.
• Run capacity simulation on controlled test traffic before production scale windows.
• Review alerting noise after changes to avoid masking true incidents.

Related routes