TechnicalPractitioner2 min read
How much downtime can you afford?
Set a reliability target (SLO), simulate an incident, watch the error budget being used up and alerts firing, or not, depending on how fast it burns.
For: engineers and SREs · team managersPrerequisites: Basic notions of SLOs. The simulator recaps the essentials.
An illustrative scenario: on a Thursday, the product manager wants to release a new feature. Operations hesitates: the service has already had two incidents this month. Ship or wait? Without a shared rule, the discussion turns into a standoff. With an error budget, it is settled by looking at a number.
The starting point is a promise with a number attached, the SLO (service level objective). For example: 99.9% of requests succeed over 30 days. That promise therefore allows 0.1% failures over the period. That is the error budget. Think of a phone data plan: while there is data left, you use it freely to ship and experiment. Once the plan runs out, you stop new features and harden the service.
What remains is how fast you are using up that plan. That is the burn rate. At 1, the budget runs out exactly at the end of the period. At 14.4, it runs out in about two days on a 30-day period. The simulator helps settle two questions. Is our target realistic? Do our alerts page on-call at the right moment?
Parameters
Result
Burn rate alerts
| Alert | Rule | Fired |
|---|
Rules for a 30-day SLO. Three come from the recommended table in Google's SRE Workbook, chapter "Alerting on SLOs" (14.4 over 1 h, 6 over 6 h, 1 over 3 days); the 3 over 24 h rule comes from the default windows of tools such as Sloth (10% of the budget in one day); it appears in the Workbook only in a configuration example. Simplification: only each rule's long window is evaluated, hourly; in production a short window confirms the alert and clears it once the incident is over.
Things to try
Section titled “Things to try”- Simulate a hard outage: 20% errors for 1 hour. The fast alert fires within the hour. This is the case where someone must be woken up.
- Simulate a slow degradation: 0.3% errors for 5 days. No fast alert goes off, yet the budget melts anyway. Only the slow alerts see it. That is by design: nobody needs waking up, but someone must deal with it during the day.
- Try background noise that is too high: set the incident duration to 0 and the background error rate to 0.08%. With no incident at all, 80% of the budget is already gone. Either the target is badly chosen or the service needs hardening.
- Try a stricter target: switch to 99.99%. The budget is divided by ten and background noise alone is enough to exhaust it. Before promising one more “nine”, check what the service delivers today.
Why several windows?
Section titled “Why several windows?”A single rule such as “error rate above X” is either too sensitive or too slow. Too sensitive, it wakes someone up for a five-minute spike. Too slow, it misses a degradation that lasts for days.
Multi-window rules solve this the way a smoke detector paired with a meter reading would. Each rule pairs a burn speed (the burn rate factor) with an observation period. The faster the budget burns, the shorter the window and the more urgent the alert. A slow burn, on the other hand, is measured over a long window and opens a ticket instead of waking anyone.
In my view, the most useful part is not fine-tuning thresholds but the written policy that goes with them: who decides to freeze releases and from what level of remaining budget.
Going further: the alert fatigue simulator, which puts a cost on needless wake-ups, plus the page SLOs and alerting.