How to Write a Blameless Postmortem: Template and Examples (2026)
This guide is for the engineer who has just been told "you own the postmortem" for an incident that ended an hour ago, and for the lead who wants every team writing them the same way. By the end you will have a repeatable process: when a postmortem is required, how to build the timeline from evidence rather than memory, how to write contributing factors instead of hunting for one root cause, how to turn findings into action items that actually close, how to run a 30-minute review, and a copy-paste markdown template you can drop into a repo. The commands were checked against git 2.54, GitHub CLI 2.102 and kubectl 1.37; you need nothing more than a Git repository and read access to your deploy history, logs and paging tool. If you want the wider context on tools and on-call in 2026, I wrote that up separately in incident management in 2026.
Step 1: Decide when a postmortem is required
Write the triggers down before the next incident, not during it, otherwise every severity call turns into a negotiation about paperwork. Google's SRE book chapter on postmortem culture lists sensible defaults: user-visible downtime or degradation beyond a threshold, any data loss, any on-call intervention such as a rollback or traffic reroute, resolution time beyond a threshold, a monitoring failure (a human found it before the alerts did), and any stakeholder asking for one. Atlassian's incident handbook is stricter and simpler: severity 1 and 2 incidents always get one, everything else is optional. PagerDuty's public response docs say the same for SEV-1 and SEV-2.
Put your version in the repo next to the template so nobody has to remember it:
# postmortems/policy.yaml: read by humans, and by the bot that opens the doc
required_when:
- severity: [SEV1, SEV2]
- data_loss: true
- customer_visible_minutes: ">= 15"
- detected_by: human # monitoring missed it
- rollback_or_failover: true
optional_when:
- severity: [SEV3]
- near_miss: true # nothing broke, but only by luck
review_meeting:
SEV1: "within 3 calendar days"
SEV2: "within 5 business days"
priority_action_slo_weeks: 4 # Atlassian uses 4 or 8 depending on the service
Near misses are worth including. They are the cheapest incidents you will ever learn from.
Step 2: Assign one owner and set the clock
One person owns the document; many people contribute. Google's SRE workbook lists "several owners" as an anti-pattern because it means nobody drives it. In PagerDuty's process the incident commander names the owner at the end of the call, and the review meeting is booked within three calendar days for a SEV-1 and five business days for a SEV-2, before the writing starts. Booking first creates the deadline.
The owner's job is to gather evidence, draft, chase reviewers and file tickets. It is not to fix everything; PagerDuty is explicit that the owner creates follow-up tickets but is not responsible for resolving them. Atlassian adds named approvers per service, usually the managers who own the affected systems, whose approval means they agree with the findings and accept the proposed priority actions.
Scaffold the file so every postmortem starts from the same structure. This script creates a dated file from the template in Step 9:
#!/usr/bin/env bash
# new-postmortem.sh: ./new-postmortem.sh INC-2026-0142 "Checkout API 5xx after config push"
set -euo pipefail
id="$1"; title="$2"
day="$(date -u +%Y-%m-%d)"
slug="$(printf '%s' "$title" | tr '[:upper:]' '[:lower:]' | tr -cs 'a-z0-9' '-' | sed 's/^-//; s/-$//')"
out="postmortems/${day}-${slug}.md"
[ -e "$out" ] && { echo "exists: $out" >&2; exit 1; }
sed -e "s|{{ID}}|${id}|g" -e "s|{{TITLE}}|${title}|g" -e "s|{{DATE}}|${day}|g" \
postmortems/TEMPLATE.md > "$out"
echo "created $out"
Running it in my scratch repo:
$ ./new-postmortem.sh INC-2026-0142 "Checkout API 5xx after config push"
created postmortems/2026-10-20-checkout-api-5xx-after-config-push.md
Step 3: Build the timeline from evidence, not memory
The timeline is the spine of the document, and memory is the worst source for it. People compress, reorder and rationalise within hours. Pull timestamps from systems first, then interview people to explain the gaps. Use UTC everywhere; mixed time zones are the most common timeline bug I see.
# Incident window in UTC
FROM="2026-10-14T09:00:00Z"; TO="2026-10-14T13:00:00Z"
# What merged to main in the window
git log origin/main --since="$FROM" --until="$TO" \
--date=iso-strict --pretty=format:'%ad %h %an %s'
# What CI/CD ran (GitHub CLI 2.102)
gh run list --created "2026-10-14" --limit 200 \
--json createdAt,displayTitle,conclusion,headSha,event
# Kubernetes warnings (events expire after 1h by default, so capture them DURING the incident)
kubectl events -A --types=Warning > "incident-events-$(date -u +%H%M).txt"
# Host logs for a unit, using a UTC-suffixed time range
journalctl -u nginx --since "2026-10-14 09:00:00 UTC" --until "2026-10-14 13:00:00 UTC"
The kube-apiserver keeps events for one hour unless --event-ttl is raised, which is why the incident channel should have a "capture events" step in the runbook. Then export the incident channel and the paging tool's log entries (acknowledge, escalate, resolve) and merge everything into one table: time, source, what happened, and a link. Cloudflare's 18 November 2025 postmortem is a good model: a table of UTC times with a status and description per row, running from the 11:05 access change to full recovery at 17:06.
Write timeline entries as facts with sources, not interpretations. "11:32 Alice restarted the bot service (Slack link)" is useful. "11:32 Alice mistakenly restarted the wrong service" is a judgement, and it belongs nowhere in the document.
Step 4: Write contributing factors, not a single root cause
The trigger is what set the incident in motion. The contributing factors are the conditions that let a routine trigger become an outage, slowed detection, or slowed recovery. Google's workbook example separates the trigger (a manual re-run of a workflow) from the underlying flaws (a non-idempotent operation with no safety check) and lists other issues that worsened impact under "things that went poorly".
Two public postmortems show why one root cause is not enough:
- Cloudflare, 18 November 2025. A ClickHouse permissions change made a query return duplicate rows, which roughly doubled a Bot Management feature file. The proxy had a hard limit of 200 features (about 60 were in use), so the oversized file caused a panic and 5xx errors across the network. The permissions change was the trigger. The contributing factors include treating internally generated config as trusted, a hard limit that failed closed, and fast global propagation. Each one gets its own fix.
- GitLab, 31 January 2017. In the GitLab postmortem, an engineer removed roughly 300 GB from the primary database instead of the secondary while rebuilding replication. The deletion was the trigger. The outage lasted about 18 hours and lost about six hours of data because none of four recovery methods worked: pg_dump backups had been silently failing (9.2 tools against a 9.6 database, with failure emails rejected for lack of DMARC), Azure disk snapshots were not enabled for the database servers, and nobody owned testing restores.
Group factors under neutral headings so the analysis covers more than the code: detection, change process, tooling and limits, recovery, people and load (who was on call, how tired, how many concurrent incidents), and communication.
Step 5: Use the 5 Whys carefully, or ask "how" instead
The 5 Whys is still the default in many templates, including Atlassian's, and GitLab used it in 2017. It works for a linear mechanical fault. It struggles with real incidents for the reasons John Allspaw set out in The Infinite Hows: "why" questions drift towards who, they force a single chain through what was really a web of conditions, and where you stop asking is a choice that decides what you "find".
You can see the limit in GitLab's own analysis. It needed two separate chains, one for why the site went down and one for why recovery took so long, and the second chain is the one that produced most of the valuable fixes. A single 5 Whys would have stopped at the deletion.
Better interview questions are about how the situation looked from inside it:
- What were you seeing at that moment, and where?
- What did you expect to happen when you ran that?
- What options did you consider, and what made this one look right?
- What would have helped you notice sooner?
- Has this procedure worked before? What was different this time?
If someone's action made sense to them at the time, the fix is in whatever made it make sense: the tooling, the runbook, the naming, the dashboard. That is the practical meaning of blameless.
Step 6: Write action items with an owner, a ticket and a due date
Google's workbook lists the failure modes: no owner, vague verbs like "improve" or "make better", every item at the same priority, items that only rely on people being more careful, and items never filed as tracked bugs. Atlassian splits actions into "priority actions" (address the cause; SLO of four or eight weeks depending on the service) and "improvement actions", and tracks both in the owning team's backlog.
A good action item has a verifiable end state. "Improve backups" is not one. "Alert when the latest successful restore test is older than 7 days" is. File them from the document so the links exist from day one:
# File a priority action in the owning team's repo (GitHub CLI 2.102)
gh issue create \
--repo acme/payments-api \
--title "PM INC-2026-0142: validate generated feature file size before publish" \
--label "postmortem,priority-action" \
--assignee alice \
--milestone "2026-11" \
--body "Postmortem: https://github.com/acme/postmortems/blob/main/postmortems/2026-10-20-checkout-api-5xx-after-config-push.md
Done when: publish pipeline rejects files over the consumer's limit, with a test.
Due: 2026-11-17 (4-week priority action SLO)."
Keep the list short. PagerDuty's guidance is that only P0 and P1 items should normally become tickets, and that accepting some risk is fine when the fix costs more than the problem.
Step 7: Run the postmortem review meeting
Share the draft 24 hours ahead so people read it before the meeting, not during. PagerDuty's agenda fits in 15 to 30 minutes, with the incident commander chairing and the owner doing most of the talking:
- Walk the timeline and agree it is accurate.
- Review the key findings and anything surprising.
- Ask how this could have been caught earlier (canary, tests, load testing, alerts).
- Discuss customer impact and any customer feedback.
- Review action items: are they the right ones, owned and dated?
The chair's main job is language. When someone says "Bob should have checked", rephrase it as "what would have made the check obvious or automatic?" and move on. One correction in the first meeting sets the tone for the next fifty.
Step 8: Publish widely and track to closure
Google's chapter puts it bluntly: "an unreviewed postmortem might as well never have existed". Publish to the widest audience that can learn from it, strip anything that identifies an end user, and keep all postmortems in one searchable place. A Git repo of markdown files works well because it is greppable and versioned, and pull requests give you review and approval for free.
The workbook also flags delayed publication as an anti-pattern; its bad example came out four months after the incident. Aim for days, not weeks.
Step 9: Copy the blameless postmortem template
Save this as postmortems/TEMPLATE.md. The {{ID}}, {{TITLE}} and {{DATE}} placeholders are filled by the script in Step 2. The sections follow Google's guidance on context, impact numbers, ownership and tracked actions.
# Postmortem: {{TITLE}}
| Field | Value |
|---|---|
| Incident ID | {{ID}} |
| Date | {{DATE}} |
| Severity | SEV? |
| Status | Draft / In review / Approved / Closed |
| Owner | @name (one person) |
| Incident commander | @name |
| Approvers | @service-owner-manager |
| Review meeting | YYYY-MM-DD HH:MM UTC |
## Summary
Two or three sentences a non-engineer can follow: what users saw, for how long, and the current state.
## Impact
- Duration: start HH:MM UTC to end HH:MM UTC (N minutes)
- Users/customers affected: number or best estimate, and how it was measured
- Error budget consumed: N% of the 30-day budget
- Data loss: none / describe
- Revenue, SLA credits, support tickets: numbers or "unknown"
## Background
What the reader needs to know about the system. Define any internal names.
## Timeline (UTC)
| Time | Source | Event | Link |
|---|---|---|---|
| 09:41 | deploy log | Config change X rolled out to 5% | link |
| 09:44 | alert | p99 latency alert fired for checkout | link |
| 09:52 | pager | On-call acknowledged | link |
| 10:20 | chat | Rollback started | link |
| 10:31 | dashboard | Error rate back to baseline | link |
## Detection
How did we find out? How long from start of impact to first alert, and to a human acknowledging it?
## Trigger
The event that started the incident, stated neutrally.
## Contributing factors
- Detection:
- Change process:
- Tooling, limits and defaults:
- Recovery and rollback:
- People, load and on-call:
- Communication:
## What went well
## What went poorly
## Where we got lucky
## Action items
| Type | Action (verifiable end state) | Owner | Ticket | Due |
|---|---|---|---|---|
| Priority | Alert when restore test is older than 7 days | @name | link | YYYY-MM-DD |
| Improvement | Add runbook step to capture k8s events | @name | link | YYYY-MM-DD |
## Supporting material
Queries and commands used, dashboards, call recording, customer communication.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Postmortems arrive weeks late or never | No owner named at incident close, meeting not booked | IC names one owner on the call; book the review before drafting |
| Draft reads like a list of who did what wrong | Timeline written from memory with judgement words | Rebuild from system timestamps; strike adjectives and "should have" |
| Every incident has the same "human error" root cause | Analysis stops at the first person in the chain | Ask how the action made sense; list conditions under each category |
| Action items never close | No tracked tickets, no due dates, no approver | File tickets from the doc, set a 4 or 8 week SLO, report overdue items to approvers |
| Timeline has gaps or impossible ordering | Mixed time zones, Kubernetes events expired after 1h | Normalise to UTC; capture kubectl events during the incident |
| Nobody reads published postmortems | Shared only within the team, buried in a wiki | Publish to one searchable repo, announce in an engineering channel, review monthly |
| Review meeting runs over an hour | Draft not read in advance, debate about fixes | Share 24h ahead; take design debates offline as their own action item |
| Same incident repeats months later | Only mitigation actions, nothing preventive | Require at least one priority action that changes a system, not a behaviour |
FAQ
What is the difference between a root cause and contributing factors?
The trigger or "root cause" is the single event that started the incident. Contributing factors are the conditions that let it cause damage, slowed detection or slowed recovery. Most lasting fixes come from the contributing factors, as both the Cloudflare and GitLab postmortems show.
How long should a postmortem take to write?
Aim to have a reviewed draft within the meeting window: three calendar days for a SEV-1 and five business days for a SEV-2 in PagerDuty's process. A typical document takes a few hours of the owner's time plus short interviews. Publishing months later loses most of the detail.
Should postmortems name the people involved?
Name roles and actions in the timeline where it helps the reader follow events, but never assign blame to a person or team. Google's workbook lists blaming individuals as an anti-pattern. If someone made a reasonable choice that turned out badly, the finding is about what made that choice reasonable.
Can AI write the postmortem for me?
AI tools are good at drafting a timeline from the incident channel and call transcript, which saves real time. They are not good at the contributing-factor analysis or the interviews, and they will happily produce a confident single root cause. Use them for the first draft of the timeline and check every timestamp against the source.
What I check first in any postmortem, mine included, is the action items table: does each row have one owner, a ticket link and a date, and is at least one of them a change to a system rather than a reminder to people? If that table is right, the rest of the document usually is too. When I run reviews I keep the meeting short, the language neutral and the approver in the room. Start from the Google SRE postmortem chapter, the SRE workbook's examples, PagerDuty's response docs and Atlassian's handbook. For the alerting side that feeds the timeline, see the SLO burn rate alerts guide.
Related: AIOps in 2026: 47% on ITBench, 10% on Hard On-Call, $6.50 an Investigation, and Four Outages Where the Automation Was the Incident, microVM Production Incidents: 4 Postmortems From My Fleet and Incident Management in 2026: Opsgenie Ends April 2027, AI SREs Arrive.
I'm Ajay Kumar — I build and operate PandaStack, an open-source Firecracker microVM cloud for AI agents, and write about the infrastructure and tooling around it.
Need this kind of infrastructure work? See what I do or email hello@ajayk.sh.
Related
How to Run Chaos Experiments on Kubernetes with Chaos Mesh (2026)
Step-by-step Chaos Mesh 2.8.5 guide: install with Helm on containerd, opt namespaces in, write PodChaos, NetworkChaos and StressChaos YAML, gate experiments on a Prometheus SLO, schedule them, and run the same test with a LitmusChaos promProbe, plus a troubleshooting table.
16 minOct 20, 2026How to Fix Kubernetes CrashLoopBackOff: Causes and Commands (2026)
Fix Kubernetes CrashLoopBackOff step by step: what the state means, the 10s to 5 minute backoff and the new per-node max in 1.35+, kubectl describe and logs --previous, exit codes 1, 127, 137 and 143, probe and OOM causes, and kubectl debug for crashing containers.
13 minOct 20, 2026How to Set Up Grafana Loki on Kubernetes with Alloy (2026)
Step-by-step Grafana Loki 3.7 setup on Kubernetes: the new grafana-community Helm chart, deployment modes, S3 storage, Grafana Alloy replacing end-of-life Promtail, labels vs structured metadata, LogQL, retention, and fixes for 429 and too many outstanding requests.
12 min