PeoplePractitioner2 min read
Too many alerts: is on-call sustainable?
Alert volume, noise share, night wake-ups, rotation size: estimate the real load of an on-call week, its cost and whether it can last.
For: engineers and SREs · team managersPrerequisites: Basic notions of on-call and alerting.
An illustrative scenario: an on-call engineer is woken up three times in one night. A nearly full disk that clears itself, a service that restarts and comes back within a minute, a traffic spike with no consequences. The fourth alert, around six in the morning, flags a real outage. He sees it twenty minutes late: by then he had stopped jumping out of bed.
Everyone knows the car alarm that goes off at every gust of wind: after a week, nobody turns their head any more. Technical alerts have the same effect. An alert that requires no action is not free: it interrupts, it wakes people up and above all it teaches the team to stop reacting. That is what is called alert fatigue.
It can be measured. The simulator below helps settle a simple question: can your team’s on-call rotation last? It counts interruptions for the on-call person, broken nights, hours lost and the share of all that which was only noise, meaning alerts that required no action.
Parameters
Sustainability thresholds (adjustable)
Result
| Criterion | Value | Threshold | Status |
|---|
- Actionable
- Noise (non-actionable)
- Threshold
- Outside business hours (shaded)
Model based on averages: weekly rotation, one on-call person receives every alert of their week; business-hours alerts spread over the 5 weekdays, the others over the remaining 9 periods of 12 h (nights and weekend). Each alert counts as an interruption, even when several alerts belong to the same incident. The guideline of 2 incidents per 12 h shift and that of a rotation of at least 8 people are commonly cited from Google’s Site Reliability Engineering book; they are guidelines, not standards.
Things to try
Section titled “Things to try”- Look at the starting point: 40 alerts per week, of which 30% really require action, for a rotation of 6 people (the people who take on-call in turn). All four criteria are exceeded. Noise costs around €36,000 a year, not counting what it causes the team to miss.
- Remove the noise: go to 12 alerts per week, all of them useful, as if you kept only the alerts that require action. The load per 12 h period drops below the threshold while the cost of noise falls to zero. Yet 2.4 wake-ups per week remain.
- Grow the rotation to 8 people. The on-call week is no lighter, but everyone goes through it less often: 6.5 weeks a year instead of 8.7. It is an organizational lever, not a cure for noise.
- Defer what can wait. Bring the share of off-hours alerts down to 25%, for example by turning non-urgent alerts into tickets handled the next day. Wake-ups drop below the threshold and on-call becomes sustainable.
The rule to remember
Section titled “The rule to remember”Alert volume is fixed at the source: an alert should only wake someone up if it demands immediate human action. Google’s Site Reliability Engineering book is often cited for two guidelines: at most two incidents per 12-hour on-call shift, a rotation of at least eight people for a single-site team. They are orders of magnitude to adapt, not standards. In my view, what matters is to choose your thresholds explicitly, measure them and review them with the team. On Monday morning, start with last week’s list of alerts and ask, for each one, what was actually done when it came in.
Going further: the people dimension, burn rate alerts that cut needless wake-ups, organization and alert ownership and the bridges between dimensions.