Skip to content

PeoplePractitioner8 min read

Sustainable on-call

Size an on-call rotation, measure how often alerts wake people up, compensate them and only page a human when users are really affected.

For: engineers and SREs · team managersPrerequisites: Knowing what an alert is will do. SLOs and error budgets are explained on the page.

Reading mode

It is three in the morning. Lea’s phone buzzes: a server disk is 80% full. She gets up, opens her laptop and checks. There is nothing to do, since the disk will be cleaned up automatically later that morning. At half past four the same alert fires again. The next day Lea works at half speed. It is her third on-call week this quarter, because the team only has four people. A few months later she takes a job elsewhere. With her goes the person who knew the payment service best.

This scenario is made up, but many teams will recognize it. On-call is the doctor on duty applied to IT: one person stays reachable outside office hours to step in if a service goes down. It does not hold because the team is brave. It holds when four things are in place: a rotation large enough that each person is rarely on duty, few alerts that are all useful, recognized compensation and regular measurement of the real load. When one is missing, the others eventually give way.

This page gathers the published benchmarks, a way to measure the load and the rules I recommend. It complements lesson 5 of the CIO path, which looks at the topic from the leadership side.

Google’s Site Reliability Engineering book has a chapter on on-call (Being On-Call). SRE, or site reliability engineering, is the discipline that treats running a service as an engineering problem. The chapter gives orders of magnitude that are widely quoted.

According to the book, an engineer should spend no more than 25% of their time on call. At least 50% of their time should go to engineering, meaning improving the system rather than keeping it running. A 12-hour on-call shift should see no more than two incidents. Finally, a rotation on a single site should have at least eight people; spread over two sites, at least six people per site.

These figures describe how one specific organization works. I recommend treating them as reference points, not as standards. Set your own thresholds, write them down and measure them.

A rotation is the duty roster: the list of people who take turns holding the on-call phone, often week by week. A rotation that is too small wears people out mechanically, however good the alerts are.

Start by counting. With four people taking turns each week, each one is on call one week in four, about three weeks per quarter (illustrative example). With eight people, it is one week in eight.

Then ask who is secondary. A designated second level, who knows they may be paged, keeps the person on call from being left alone with a long-running incident. The chapter cited above describes this primary and secondary arrangement.

Finally, look at what happens during holidays and sick leave. If the rotation only holds when everyone is present, it is too small.

A team spread across time zones can “follow the sun”: on-call moves from one country to the next over the day, so each person holds it during their own daytime. Night pages disappear. In return, you need a written handover at every time-zone change.

I recommend that handover everywhere, at the end of every shift. Think of the change of shift between two teams in a hospital: ongoing incidents, noisy alerts spotted, planned changes, things to watch. Five lines in a shared channel are enough. Those notes also feed the monthly review described below.

You cannot reduce what you do not measure. The indicators below come from your alerting tool’s history and from one question asked to the on-call person at the end of the week. An “actionable” alert is one that really required a human to do something; the ones that required nothing are noise.

IndicatorHow to compute itWhat it reveals
Pages per shiftalerts that notified a human, per 12-hour periodraw load, to compare with the two-incident benchmark
Out-of-hours pagessame count, restricted to nights and weekendssleep disruption, the most direct burnout factor
Share of actionable alertsalerts that led to a human action, over the totalnoise; an alert with no action is a candidate for removal
Recurring alertsalerts fired more than three times in the month with the same causewhat a lasting fix would remove
Interrupt timehours spent on alerts and requests during the shiftthe time not spent on engineering
Perceived loada 1 to 5 score given by the person at the end of the shiftdrift, before it shows in the numbers

The alert fatigue simulator estimates this load from a few parameters and says whether a typical week stays sustainable.

I also recommend a thirty-minute review every month on the ten most frequent alerts. For each one, the team decides: delete it, fix what triggers it or turn it into a non-urgent ticket. An alert that comes back three months in a row without a decision is a governance issue more than a technical one: nobody felt it was their job to decide.

A car’s warning light comes on when the engine overheats, not every time the revs climb on a hill. Alerts should follow the same logic. A symptom is what the user experiences: a page that will not load, a declined payment. A cause is what is happening inside: a busy processor, a disk filling up.

In my view, the main source of noise is alerting on a cause rather than on a symptom. CPU at 90%, a disk at 80% or a restarted pod (a pod is the unit an application runs in on Kubernetes) do not tell you whether users are suffering. The Monitoring Distributed Systems chapter of the same book recommends alerting on user-visible symptoms and keeping causes for dashboards and diagnosis. It also states that every page should call for an action and warrant an urgent response.

SLOs give a precise criterion for deciding. An SLO (service level objective) is a promise with a number on it, for example “99.9% of payments succeed over thirty days”. The error budget is the margin left over: the 0.1% of failures you allow yourself over the period. Think of a monthly grocery budget. If half of it is gone by the 3rd, you act right away. If you are slightly ahead of your usual pace, it can wait for the weekly catch-up.

The Alerting on SLOs chapter of The Site Reliability Workbook applies this reasoning. It describes alerts on how fast the error budget is being spent (the burn rate), evaluated over several time windows. Fast, sustained consumption that would exhaust the budget within days pages a human. Slow consumption that would exhaust it before the end of the period, with no urgency, creates a ticket handled during working hours. Everything else feeds dashboards, without notification.

The workbook proposes specific thresholds for these windows. The error budget simulator shows how this scheme reduces pages without delaying the detection of a real incident.

flowchart LR
  S["Signal"] --> Q1{"Symptom visible<br/>to users?"}
  Q1 -->|"no"| D["Dashboard,<br/>diagnosis"]
  Q1 -->|"yes"| Q2{"Error budget<br/>burning fast?"}
  Q2 -->|"yes"| P["Page on-call"]
  Q2 -->|"slowly"| T["Ticket during<br/>working hours"]
  Q2 -->|"no"| D

One last rule, the simplest way to keep noise from creeping back: every alert that wakes someone up has an owning team and a runbook. A runbook is a sheet, however short, that says what to check and what to do, like a pilot’s emergency checklist. I recommend rejecting in review any new paging alert that has neither.

In France, article L3121-9 of the Labor Code (Code du travail) defines on-call duty and requires it to be compensated, either financially or with time off. Time spent intervening while on call counts as actual working time. The following articles of the same code leave the arrangements to a collective agreement or, failing that, to the employer after consulting employee representatives. The text is available on Légifrance. Other countries have their own rules. I recommend involving HR from the design of the rotation, not after the first complaints.

After a night of intervention, I recommend recovery time granted without justification. It is written into the on-call policy rather than left to each manager’s discretion.

Someone who is not on call does not get paged. Exceptions are named, written down and compensated.

Finally, the on-call person must have the right to hand back an alert: temporarily silence an obviously noisy alert, provided they flag it in the handover. Without that right, noise settles in because nobody feels entitled to cut it.

  • The rotation has enough people to hold during holidays
  • A second level is designated and knows it
  • Acceptable load thresholds are written down (pages per shift, night pages)
  • Alert load is measured every month with the indicators above
  • Every paging alert targets a symptom, has an owner and a runbook
  • Paging alerts are derived from SLOs where an SLO exists
  • Compensation is defined with HR, in line with applicable law
  • Recovery after a night intervention is written into the policy
  • End-of-shift handover is systematic
  • The on-call person can silence a noisy alert by flagging it