How to Run an Incident Postmortem That Prevents Repeats

There is a specific tell that a postmortem process has stopped working: someone opens an incident channel, and within ten minutes a person in it says “didn’t we have this exact thing in March?”. Somebody digs out the old document. It has a root cause section, a timeline, and four action items, none of which were ever done.

The document was not the problem. It was thorough. It just had no mechanism attached to it, and a postmortem without a mechanism is a diary entry.

The Two Failure Modes

Postmortem processes fail in one of two directions, and they need opposite fixes.

The first is the blame version. Someone gets named as the cause, the meeting becomes defensive, and the real lesson stays unspoken because admitting it costs you something. The Google SRE book’s chapter on postmortem culture ↗ makes the argument that has since become standard practice: you cannot get an honest account of what happened from people who expect consequences for giving it.

The second is subtler and much more common in teams that have already read that chapter. The postmortem is scrupulously blameless, well written, thoroughly reviewed, and produces nothing. Everyone leaves the meeting agreeing it was valuable. Nothing changes. The same incident recurs in five months.

Blamelessness is table stakes. It buys you an accurate account. What you do with the account is a separate problem, and most teams have only solved the first one.

Write the Timeline Before You Write Anything Else

Start with a timestamped sequence of events and nothing else. No analysis, no causes, no opinions. Just what happened, when, and what the person doing it could see at that moment.

That last clause is the whole trick. Compare:

14:12 Engineer restarted the primary database, causing 8 minutes of write failures.

with:

14:12 Dashboard showed connection pool at 100% and query latency climbing. Runbook step 3 says restart the primary. Engineer restarted the primary. Writes failed for 8 minutes.

The first version has smuggled in a judgement. The second version has identified a runbook that gives dangerous advice under a condition its author did not anticipate, which is an actionable finding. Same event, same person, completely different outcome for the team.

Pull the timeline from the incident channel while it still exists. Slack scrollback, alert timestamps, deploy logs and dashboard screenshots are the raw material, and some of them have a retention window measured in days.

Stop Looking for The Root Cause

The phrase implies there is one, singular, and that finding it completes the job. Real incidents rarely oblige.

A typical outage looks more like this: a config change was valid but unusual, a validation step only ran in CI and not on the manual path, an alert existed but routed to a channel nobody watches at weekends, and the runbook was written before the service was sharded. Remove any one of those four and there is no incident. Which one is the root cause?

The useful question is not “what caused this” but “what would have stopped this, and what would have made it shorter”. Those are two distinct lists and both are worth having.

Question	Produces
What made it possible?	Prevention work: validation, guardrails, config review
What made it take so long to notice?	Detection work: alerts, SLOs, monitoring gaps
What made it take so long to fix?	Response work: runbooks, access, rollback capability
What made it worse than it needed to be?	Blast radius work: isolation, rate limits, feature flags

Four buckets, filled honestly, will give you a better action list than any amount of arguing about which single factor was the true cause. Detection and response are where the biggest wins usually hide, because a 40 minute outage that could have been a 4 minute outage is the same bug with a different amount of pain attached.

Detection findings almost always trace back to the same two gaps, and both have their own fix: observability versus monitoring covers whether you can ask new questions of a live system, and service level objectives covers whether your alerts fire on things users actually feel. Response findings tend to land on rollback, which is its own discipline: see how to roll back a bad deploy.

The Meeting Has One Job

The writeup happens before the meeting, not during it. Circulate the draft, ask people to read it, and use the meeting for the parts that need a room: the moments where people disagree about what happened, and turning findings into tickets.

Run it in this order:

1. Walk the timeline. Anyone who was there corrects it. This usually surfaces one or two things nobody else knew.
2. Find the surprises. Ask directly: at what point did the system do something you did not expect? Surprise is where the gap between the mental model and reality lives, and that gap is what will bite you again.
3. Fill the four buckets. Prevention, detection, response, blast radius.
4. Write the tickets. In the meeting, in the real backlog, with a named owner and a date on each one.

If that shape looks familiar, it should: it is the same machinery as a developer retrospective that drives change, pointed at a single event rather than a fortnight.

Point four is the mechanism. Everything before it is analysis, and analysis has never fixed a system on its own. PagerDuty’s incident response documentation ↗ is blunt about this: action items without an owner and a due date do not get completed.

Cap the list. Five well-owned actions that get done beat fifteen that decorate a document. If the fifteen all feel essential, that is a signal about the state of the service, and the honest action item is a larger piece of planned work rather than fourteen small ones that will rot.

What to Track Afterwards

Two numbers tell you whether the process is real:

• Open postmortem actions by age. If the oldest is nine months, the process is decorative. Review this in the same place you review everything else.
• Repeat incidents. Tag incidents by the system and failure mode. A repeat is not a failure of the engineers who responded; it is a failure of the postmortem that preceded it.

Neither number needs tooling. A saved search on a ticket label will do it, and you can add the fancy version later if the simple one proves useful.

Make the Archive Searchable

Postmortems are only worth writing if someone finds them at 3am eighteen months later. That means consistent titles that name the service and the symptom, a tag or label per system, and a location that is not a personal folder or a thread.

The convention that works: title the document with the service, the symptom and the date, so that searching the service name during an incident surfaces every previous time it broke. Dedicated postmortem tools exist, Etsy’s Morgue being the best known of the open source ones, but a wiki page with a naming convention gets you most of the value for none of the maintenance.

The Test

After your next postmortem, wait six weeks and ask one question: which of the actions shipped? If the answer is most of them, the process is working and the documents are a by-product. If the answer is none, you do not have a postmortem problem, you have a prioritisation problem, and writing better documents will not touch it.

Start by pulling up your last three postmortems and counting the open actions. That number is the honest state of the process, and it usually takes about five minutes to find out.