Skip to content

PeoplePractitioner7 min read

Blameless post-mortems

Trigger, prepare and facilitate a post-mortem that produces actions rather than culprits, then follow those actions through and measure what the organization learns.

For: engineers and SREs · team managers · business and product · executives and CIOsPrerequisites: Having been through an incident, closely or from a distance.

Reading mode

Friday evening, a release takes payments down for forty minutes. On Monday, the meeting opens with a question from the manager: “Who ran the deployment?” The person concerned says the bare minimum. The others keep quiet. The conclusion is “human error” and everyone is reminded to be careful. People go back to work. Three months later, the same incident happens again with someone else.

This scenario is made up, but it is common. The blameless post-mortem is a way out: it looks for what made the error possible rather than for the person who made it, then it produces a few actions that are followed through to the end.

Lesson 5 of the CIO path sets out the cultural rules. This page is the practical guide: when to start a post-mortem, how to run the session, which document to use, how to track the actions and what to measure.

The argument is practical before it is moral. Someone who risks a sanction says less, or says it differently. The analysis then stops at “human error” and the flaw that made the error possible stays in place for the next person. John Allspaw put this idea into words for Etsy in a text that became a reference (Blameless PostMortems and a Just Culture). The Postmortem Culture chapter of the Site Reliability Engineering book later built on it.

Blameless does not mean without accountability. Actions have owners and dates. What disappears is the hunt for a culprit.

A post-mortem takes several people a few hours. So that it depends neither on the mood of the moment nor on how serious the incident felt, the criteria must be written down and known before the incident. As a baseline I use those proposed in the Google chapter cited above: an outage or slowdown visible to users beyond a threshold set in advance, data loss of any size, a manual on-call intervention to restore service (rollback, traffic rerouting), a recovery that takes longer than an agreed delay, or an incident discovered by something other than monitoring, which reveals a blind spot.

Anyone involved can also ask for a post-mortem outside these criteria. I add one recommendation: an incident that uses up a significant share of an SLO’s error budget (the margin of error you allow yourself on a service quality target, a share you set yourself) deserves a post-mortem even if nobody complained.

When the incident is over, the incident lead appoints an author and a facilitator, ideally people who were not part of the response. In the following days, the author gathers the facts: the time of each alert and each decision, dashboard screenshots, the messages exchanged during the incident. They prepare a draft, reviewed by those who responded, leaving the actions section empty.

The session lasts about an hour. It brings together those who responded, the team that owns the service and, when the incident hit it, the business line concerned. It ends with shared causes and assigned actions. The document is then shared internally and the actions join the team’s work list, where they are tracked until they are closed.

I recommend setting a delay between the end of the incident and the session, short enough for memories to be fresh. The right delay depends on your pace; what matters is that it is written down and kept.

The facilitator guards the rule. When someone says “they should have”, the facilitator rephrases: “what made that choice reasonable at that moment?”. When the discussion drifts towards people, the facilitator brings it back to conditions: a tool, a procedure, a missing signal, time pressure. They also make sure the most junior people speak first.

I recommend spending the first third of the session agreeing on the timeline. Many disagreements about causes actually come from unspoken disagreements about what happened.

A document that is too long rarely gets filled in: this one fits on two pages. Two sections matter more than the others.

The “Detection” section links the post-mortem to observability: if a customer reported the incident rather than an alert, there is a measurement to add. The “Where we got lucky” section brings out the risks that did not materialize this time. They are often the most instructive.

The template to copy
# Post-mortem: <factual title, no person's name>
Status: draft | reviewed | published
Incident date: YYYY-MM-DD Duration: from HH:MM to HH:MM
Author: Facilitator:
Services affected: SLOs concerned:
## Summary
Three sentences: what happened, the impact, how service was restored.
## Impact
- Users or customers affected, over which period
- Error budget consumed
- Business impact established with the affected business line (orders, delays, reputation)
## Timeline
HH:MM event (source: alert, channel, dashboard)
Detection, first diagnosis, decisions, recovery.
## Detection
How the incident was detected. How long after it started.
Did monitoring see it? If not, which signal would have been enough?
## Causes and contributing factors
What triggered the incident. What made it possible.
What made it worse or longer.
## What went well
## Where we got lucky
## Actions
| Action | Type (prevent, detect, mitigate) | Owner | Due date | Done when |
## Lessons
What other teams should know.

A post-mortem whose actions are never carried out is worse than no post-mortem at all: it teaches teams that the ritual is useless.

In my view, three completed actions beat twelve pending ones. The ideas you do not keep can stay in the document, in a “leads” section, with no owner. Each action you keep has an owner, a date and a criterion that says when it is done. “Improve monitoring” is not an action; “add an alert on the payment service error rate, tied to its SLO” is one.

The actions live in the team’s work list, not in the document: they are prioritized like everything else. Once a month, a short review goes through the actions still open to close them, chase them or drop them explicitly. A drop that is decided and explained is better than an action left to rot.

The Postmortem Culture chapter of The Site Reliability Workbook compares a weak post-mortem with an improved version. I recommend it to anyone who will facilitate sessions.

These indicators tell you whether the ritual works. They are not for ranking teams: a high number of post-mortems may point to a team that reports its incidents honestly rather than to a fragile system.

IndicatorWhat it reveals
Share of eligible incidents with a published post-mortemwhether the trigger criteria are applied
Delay between the end of the incident and publicationwhether the ritual is kept or postponed
Share of actions closed by their due datewhether the post-mortem actually changes the system
Age of open actionswhat has been learned but not yet put into practice
Recurring incidents (same cause as an incident already analyzed)whether the actions addressed the right cause
Share of incidents detected by monitoring rather than by a userwhat the post-mortem brings to observability
Reads or shares of published post-mortemswhether learning reaches beyond the team concerned

Recurring incidents and the share detected by monitoring belong on the executive dashboard: they turn learning into numbers a leadership committee understands.

The most destructive is the session that turns into a trial: a manager asking “who did this?” cancels the rule for a long time. The facilitator must be able to stop it. Next comes the search for a single cause. In a system made of many services, an incident rarely has just one cause; I prefer to talk about contributing factors.

“Human error” is never a conclusion. It is a starting point: what made the error easy to make and hard to spot? Finally, a document nobody reads is useless. A three-sentence summary, widely shared, does more than a full report filed away in a folder.

  • The trigger criteria are written down and known before the incident
  • The facilitator did not take part in the response
  • The timeline is agreed before any discussion of causes
  • Neither the title nor the document names anyone as a culprit
  • The Detection section says whether monitoring saw the incident
  • Every action has an owner, a due date and a completion criterion
  • The actions are in the team’s work list
  • Open actions are reviewed every month
  • The affected business line is invited when the incident hit it
  • Post-mortems depend on sustainable on-call: an exhausted team no longer has the energy to analyze.
  • The MTTR business case links detection and recovery time to the cost of an incident.
  • CIO path, lesson 5 suggests replaying a post-mortem to gauge the maturity of the team’s culture.