Run incident reviews that lead to useful changes

An incident review is useful when it changes how the system or team behaves next time. A timeline alone can preserve the story without reducing future risk. The aim is to understand the conditions that allowed the problem, how it was detected, and what made recovery easier or harder.

Build a shared account of events

Collect the sequence of changes, signals, decisions, and actions from the people involved. Distinguish what was known at the time from what became obvious afterward. Include the user impact and its duration, while marking uncertainty where records are incomplete. This creates a stronger basis for learning than an explanation built around a single mistaken action.

Look for contributing conditions

Ask why the action seemed reasonable, which checks were missing, and what information would have changed the decision. Consider deployment design, permissions, documentation, alert quality, and workload. A request to “be more careful” rarely changes the environment that produced the incident. Prefer improvements that make the safer action easier or the failure visible sooner.

Choose a small number of accountable improvements

Separate immediate repair from follow-up prevention and detection work. Give each action an owner, a completion condition, and a review date. Check whether the proposed change addresses the observed cause instead of using the incident to justify unrelated projects. Share the lessons in a form other teams can apply.

Construct a timeline from observations

Start with what people and systems could actually observe at each point. Separate a recorded event from an interpretation made later. In a fictional deployment incident, “the queue stopped progressing at 14:10” is a useful observation; “the team should have known” is a retrospective judgment that explains little about the information available at the time.

Invite the people involved to describe their decisions and constraints. Ask which signal they used, what they expected to happen, and what made the next action difficult. A missing dashboard, an ambiguous procedure, and a delayed approval can combine even when each person behaves reasonably. The review should help reveal those conditions.

Choose follow-up work that changes the next response

Replace “be more careful” with a specific improvement: make the release state visible, add a rehearsal for the failure mode, or clarify who can pause a deployment. Assign an owner and a way to tell whether the change has been completed. A long list of vague actions often creates less progress than a small set tied directly to the incident.

Return to the actions after the review. Check whether the procedure was usable and whether a rehearsal produced the expected result. Share the lesson with teams that face a similar condition, while keeping personal and sensitive incident details within the appropriate audience. The value of the review comes from a changed system of work, not the length of its report.

A practical next step

For the next review, select one prevention improvement and one recovery improvement. Define how you will verify each, and revisit the actions after implementation to see whether they changed the operating process.