On-call rotations that don't burn people out

The fastest way to hollow out senior engineering talent is a toxic on-call rotation. When an engineer is woken at 3:15am by an alert that requires no human action—or worse, an alert that requires manual remediation because a flaky worker pod leaked memory—you are converting technical debt directly into employee sleep deprivation.

Sustainable operations are not built by hiring more resilient engineers. They are built through disciplined alerting rules, SLO burn rate paging, and an absolute organizational commitment to eliminate non-actionable alarms.

The golden rule of paging: Human action required

An alert that triggers a phone pager must satisfy two strict criteria:

1. User-impacting degradation: Real customers are currently experiencing broken transactions or intolerable latency.
2. Immediate human intervention possible: There is a documented action the on-call engineer can take right now to mitigate the impact.

If an alert informs you that disk usage reached 82% on a non-critical log volume, it belongs in an asynchronous ticket backlog, not a 3:00am siren.

# Prometheus multi-window multi-burn-rate alert rule for 99.9% availability
groups:
  - name: service_slo_alerts
    rules:
      - alert: ErrorBudgetFastBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) > (14.4 * (1 - 0.999))
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > (14.4 * (1 - 0.999))
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Consuming >2% of 30-day error budget in 1 hour"

Every page should represent an operational emergency. If an engineer wakes up, looks at a graph, and goes back to sleep without typing a command, that alert rule is broken and must be deleted.

Multi-window burn rate alerts

Traditional threshold alerts (e.g., "Error rate > 1% for 5 minutes") suffer from high false alarm rates during minor traffic blips and fail to catch slow, persistent degradations that drain error budgets over 24 hours.

Adopting Google SRE-style multi-window multi-burn-rate alerts pages on mathematical budget consumption: paging immediately when consuming 2% of the monthly error budget in one hour (catastrophic outages), while sending gentle Slack notifications when consuming 5% over a 6-hour window.

Operational hygiene and the on-call shield

To prevent burnout, the engineer on primary rotation should be 100% shielded from regular sprint feature commitments. Their entire work week is dedicated to handling incoming support requests, triaging bugs, and writing automation to permanently eliminate the root causes of pages received during the shift.

Restructuring your team's on-call rotation or designing SLO alert policies? Get in touch.