Jonessl Knowledge

Linux

Linux service reliability

Reliability comes from explicit service state transitions, dependency ordering, and recovery criteria that do not depend on guesswork.

On this page

  1. Doctrine
  2. Decision tree
  3. Diagnostics pathway
  4. Operational checks
  5. Related routes

Doctrine

  • Define startup dependencies with `After=`, `Requires=`, and documented critical ordering.
  • Use readiness checks that verify service function, not only process liveness.
  • Set explicit restart, backoff, and timeout values for each unit.
  • Link failure codes to escalation owners and playbook references.

Decision tree

  1. Does the service fail fast or enter loop?
    • Fast fail → disable autorestart first and capture logs.
    • Loop behavior → adjust `StartLimitIntervalSec` / `StartLimitBurst` boundaries.
  2. Can dependencies prove ready?
    • Yes → promote to healthy after dependency checks pass.
    • No → hold at dependency wait with explicit warning event.
  3. Are recovery actions within SLO?
    • Yes → execute normal restart sequence.
    • No → initiate manual operator path and prevent automatic churn.

Diagnostics pathway

  1. Inspect recent state transitions: 
    systemctl status myservice
  2. Reconstruct restart history and timing: 
    journalctl -u myservice --since "2h ago" --no-pager
  3. Evaluate dependency readiness: 
    systemctl list-dependencies myservice --required
  4. Track system resource pressure around failures: 
    systemd-cgtop -b 1 5

Operational checks

  • Document and test a manual recovery command for each service family.
  • Ensure readiness checks are idempotent and tolerant during startup warm-up.
  • Validate failure-path alerts include the owning team and current severity tier.