Skip to content

TechnicalBeginner7 min read (+1 technical)

SLOs and alerting

When does a slow payment deserve to wake someone up? SLI, SLO and SLA in plain words, the error budget and how to compute it, alerting on what users experience and on how fast the budget burns, then how to choose a first SLO.

For: engineers and SREs · architects · team managers · business and productPrerequisites: Having read Observability in 10 minutes, or knowing what a metric is. The collapsed section on burn rate assumes you are comfortable with percentages.

Reading mode

Monday, 3 a.m. The on-call engineer’s phone rings: a server in the payment service is running at 90% CPU. She gets up, opens her laptop, checks the dashboards. Payments are going through normally. She goes back to bed. The following Thursday, in the middle of the day, payments turn slow because the bank is responding badly. No alert fires: no server is busy. Customers are the ones who tell support.

This scenario is made up, but many teams will recognize it. The alerts watch the machines while the question that matters lies elsewhere: from what point is a slow payment a problem worth waking someone up for? SLOs give a quantified answer.

The vocabulary and method on this page come from Google’s two books on SRE (Site Reliability Engineering), cited in the sources. SRE is a way of running systems that treats reliability as an engineering problem, with measurements and objectives.

Think of a train company. It counts the trains that arrive on time: that is its measurement. Internally it sets itself a goal, say 95% of trains on time over the month: that is its promise. And it may have signed a contract with the region that sets penalties below 90%: that is its commitment. Three different things that people often mix up.

SRE gives them three names.

The SLI (service level indicator) measures what the user experiences. It is most often written as a ratio: good events divided by all events. In our example, it is the share of payments that succeed in under 2 seconds.

The SLO (service level objective) is the target set for that SLI over a period: 99.9% of payments succeed in under 2 seconds, over a rolling 30 days. It is an internal objective, the promise the team makes to itself.

The SLA (service level agreement) is a contractual commitment to a customer, with consequences if it is missed, such as penalties: say 99.5% per month (an illustrative figure). The SLA is usually less demanding than the SLO. The gap leaves time to react before breaching a contract.

A good SLI is measured as close to the user as possible. The success rate seen by the checkout service, the one that receives the order, beats the health status of a server.

Aiming for 99.9% means accepting 0.1% of failures. This tolerated share is called the error budget. You can think of it as an allowance: a number of minutes of outage, or of failures, that the team is allowed to “spend” over the period.

The arithmetic is simple. A 30-day period has 30 × 24 × 60 = 43,200 minutes. 0.1% of 43,200 minutes is 43.2 minutes. A 99.9% SLO over 30 days therefore leaves about 43 minutes of full outage, or the equivalent in partial outage.

SLO over 30 daysError budgetTime equivalent
99%1%7 h 12 min
99.5%0.5%3 h 36 min
99.9%0.1%43 min
99.95%0.05%21 min 36 s
99.99%0.01%4 min 19 s

These values are computed. Each extra “9” divides the budget by ten.

When the SLI counts requests rather than time, the budget is read in requests. Out of one million payments, 99.9% allows 1,000 failures.

The budget is above all a decision tool. As long as some remains, the team can ship and take risks. When it is spent, reliability comes first. This rule must be written down and agreed in advance by the team, product and leadership: it is the error budget policy.

Back to Monday night. The CPU at 90% is a possible cause of trouble: a technical state, like a nearly full disk or a restart. Thursday’s slow payments are a symptom: what the user experiences.

The Site Reliability Engineering book recommends alerting on symptoms and keeping causes for diagnosis (Monitoring Distributed Systems). A CPU at 90% that slows no payment does not deserve to wake anyone. Conversely, slow payments deserve an alert, even if every server looks healthy.

In our example, the bank is responding slowly and no machine is saturated. A CPU alert would never have fired. An alert on the payment SLO would have.

Alerting as soon as an error appears would wake on-call constantly. Waiting for the budget to run out would be too late. The answer is to watch how fast the budget is being consumed, the way you watch the fuel gauge on a long drive: the level is not what worries you, the rate at which it drops is.

That rate is called the burn rate. At 1, the budget runs out exactly at the end of the period. At 2, it runs out in half the time, 15 days over a 30-day period. The higher the burn rate, the more urgent the alert.

The Alerting on SLOs chapter of the Site Reliability Workbook compares several alerting methods and ends up with what it calls multi-window, multi-burn-rate alerts. The idea goes like this. Very fast consumption pages on-call, while slow but sustained consumption opens a ticket handled during business hours. For each speed, the burn rate is measured over a time window long enough to ignore an isolated spike. It is also checked over a shorter window, so the alert stops quickly once the problem is fixed.

flowchart TD
  E["SLI error rate"] --> R["Burn rate over<br/>long and short windows"]
  R --> Q{"Both windows<br/>above threshold?"}
  Q -->|"high burn rate"| P["Page on-call"]
  Q -->|"low but sustained burn rate"| T["Ticket"]
  Q -->|"no"| D["Dashboard only"]

The error budget simulator applies these rules to an incident you configure yourself. It shows that a sharp outage triggers a page within minutes and that a slow degradation only opens a ticket.

For technical teams: the parameters recommended by the workbook

For a 99.9% SLO over 30 days, the workbook proposes these parameters as a starting point. The short window is about one twelfth of the long one.

ActionLong windowShort windowBurn rateBudget consumed when alerting
Page1 hour5 minutes14.42%
Page6 hours30 minutes65%
Ticket3 days6 hours110%

To read the first row: at a burn rate of 14.4, one hour consumes 14.4 / 720 = 2% of the 30-day budget (720 hours). At that pace, the whole budget lasts about two days.

Here is the approach I recommend for a first SLO. It is deliberately modest.

  1. Pick a single journey that matters to users and to the business. Payment is a good candidate. The home page of an internal tool is less so.
  2. Write the SLI in plain words: “the share of payments that succeed in under 2 seconds”. If the sentence is not clear to the business, the SLI is not either.
  3. Measure before aiming. Look at the actual reliability over the last few weeks. A target picked at random will be either unreachable or pointless.
  4. Set a target slightly below what you observe. If the service holds 99.95%, aiming for 99.9% leaves a margin. Aiming for 100% makes no sense: no budget, so no possible release.
  5. Write the budget policy. Who decides to slow down releases when the budget is spent? Settle it before the incident, not during it.
  6. Wire burn rate alerts to this SLO, then remove the cause-based alerts that duplicate them, like the one that woke on-call on Monday.
  7. Review after a quarter. The first SLO is rarely the right one and that is normal.
  • Beyer, Jones, Petoff and Murphy (eds.), Site Reliability Engineering, O’Reilly, 2016: chapter 4, Service Level Objectives and chapter 6, Monitoring Distributed Systems.
  • Beyer, Murphy, Rensin, Kawahara and Thorne (eds.), The Site Reliability Workbook, O’Reilly, 2018: chapter 2, Implementing SLOs and chapter 5, Alerting on SLOs (table of recommended parameters for a 99.9% SLO over 30 days).
  • The budgets in the “SLO over 30 days” table are computed from a 43,200-minute period.