How we run blameless postmortems

Someone asked me at a meetup last week how we do postmortems, and I gave a bad, rambling answer, so this is the better version. We’ve been doing them for about three years now. The first year was mostly theater, and the last two have been genuinely useful.

The idea of a blameless postmortem isn’t ours. The write-up I point people at is John Allspaw’s Blameless PostMortems and a Just Culture from Etsy, which I read in 2012 and which changed how I ran these. His argument is that if people get punished for being near a failure, they stop telling you what actually happened, and then you can’t fix anything. An engineer who ran the command that took the site down knows more about why that was a reasonable thing to do at the time than anyone else in the company. You want that knowledge, and you only get it if they aren’t afraid.

What I’d add, having done this for a while, is that “blameless” is easy to say and hard to do, because blame leaks in through the language. “Why didn’t you check the replica lag before failing over?” is a blaming question even if you ask it nicely. “What did the dashboard show you when you decided to fail over?” gets you the same information and the person answering it doesn’t have to defend themselves. Most of my job in these meetings is rephrasing questions like that, sometimes my own.

The document

Whoever was the incident lead writes a draft within two working days, while people still remember. It’s a shared doc, and anyone involved can edit it before the meeting. The sections are:

Summary, two or three sentences a customer could read.
Impact: who was affected, for how long, and how we know (numbers, not “some users”).
Timeline, in UTC, from the first signal to all-clear, including what people were thinking at each point and not just what they did.
What went well. We have this section because people skip it otherwise, and things that went well are worth keeping on purpose.
Contributing factors. Plural, deliberately. We don’t do a “root cause” section anymore.
Action items, each with one owner and a ticket.

The timeline is the most important part and the one people are worst at at first. The temptation is to write it as “14:02 on-call restarted the queue workers”, which is true and useless. “14:02 on-call restarted the queue workers, because the last two times this alert fired that’s what fixed it” tells you the alert is pointing at a symptom and the runbook never said otherwise, and neither of those is the on-call engineer’s fault.

We dropped “root cause” after about a year because it kept producing a single sentence that ended in a person. Real outages have several things going wrong at once (when our cloud region went dark the AWS outage was the trigger, but going quiet for 45 minutes and never having tested a restore were all ours), and picking one of them as “the” cause just means you fix that one and leave the rest. We tried the “five whys” for a while too and found it had the same problem. You get one chain, and it tends to go wherever the person asking wants it to go.

The meeting

Thirty to sixty minutes, within a week of the incident. Everyone who was involved is expected, and anyone else who wants to come can. I’ve had salespeople sit in, and it’s done more for their trust in the engineering team than any status page ever did.

Someone who wasn’t in the incident runs it. That’s usually me or one of the leads, and it’s the single change that made the biggest difference. If the incident lead also runs the meeting, they end up narrating and defending at the same time.

We walk the timeline first, and only then talk about fixes. If you jump to fixes early, the first idea anyone has becomes the plan, and it’s usually something like “add an alert”. We’ve added a lot of alerts that way. Some of them are still firing, unread, into a channel nobody watches.

The follow-through

This is the part we got wrong for a year, and from talking to other people, most teams get it wrong too. We had great meetings, wrote great documents, and then the action items went into a spreadsheet, and when I finally went back and checked, fewer than half of them had ever been done. Some of the same incidents happened again with the same action items still open.

What changed it was unglamorous. Action items go into the same tracker as product work, not a separate list, so they compete for the same sprint and show up in the same planning meetings. Each one has an owner and a due date. Anything that isn’t done in thirty days comes up in our weekly engineering review, with the owner explaining why, which turns out to be a strong motivator. We also got stricter about what counts as an action item, so “be more careful with migrations” gets sent back and rewritten as something like “make the migration tool refuse to run without a recent backup”.

The other thing that helped was tracking the action items themselves. Our completion rate went from under half to about 85% in two quarters, mostly by writing fewer and better ones. We used to come out of a postmortem with ten, now it’s usually three or four.

It still isn’t perfect. People are still a bit nervous in the meeting the first time, and some fixes take a quarter because they’re real projects. But we’ve had the same outage twice only once in the last year, which is a lot better than it was.