Skip to content

People2 min read

People dimension

The people who keep observability alive. On-call, alert fatigue, blameless post-mortems, skills and adoption.

A car alarm that goes off every night because of a cat ends up waking nobody. On the night of a real theft, everyone just rolls over. On-call, the rota where an engineer can be called at night when something breaks, goes through the same thing when it gets too many alerts for nothing.

A well-built observability platform can also end up looked at by nobody. So the people dimension is about the people who keep it alive, their reflexes and their limits.

The chapters follow the learning loop of an incident: a useful alert, the response, recovery, then the post-mortem (the analysis done afterwards) whose actions improve the next alerts.

flowchart TB
  A["Actionable<br/>alert"] --> R["Response<br/>and diagnosis"] --> RS["Recovery"] --> PM["Blameless<br/>post-mortem"]
  PM --> AC["Actions,<br/>not culprits"]
  AC -->|"thresholds, dashboards,<br/>runbooks"| A
  PM -.->|"real cost of the incident"| MET["Business lines<br/>concerned"]
ChapterWhat you learn to doStatus
Alert fatiguemeasure noise, remove alerts nobody can act onplanned
Sustainable on-callsize rotations, protect sleep, tool escalationpublished; the leadership view in the CIO path, lesson 5; quality on-call for AI in the GenAI method
Blameless post-mortemsrun an incident review that produces actions, not culpritspublished; four cultural rules in the CIO path, lesson 5
Troubleshooting under pressureknow the cognitive biases at play and design dashboards that limit themplanned
Profileschoose the profile that matches the problem, define skills by level, the staff, principal and management ladders, neighbouring jobspublished; archetypes and career paths in the CIO path, lesson 5
The job description, the interview, the first 90 dayshire on the problem to solve, assess hands-on, onboard without burning people outpublished; the job description generator puts it all together
Driving adoptionbring development teams on board, measure real usagethe ADKAR change management method applied in the CIO path, lesson 6; dedicated chapter planned

The alert fatigue simulator estimates the real load of an on-call week and says whether it is sustainable. The error budget simulator shows how alerts that look at several time windows at once wake people up less often for nothing.