It's 2:47 on an ordinary Tuesday. The error rate dashboard goes from 0.2% to 18% in four minutes, and PagerDuty alerts start piling up. Nobody knows yet which deploy caused it, but someone in the incident channel has already written the phrase that decides whether the rest of the month is going to be productive or not: "let's see who touched what."
That phrase is the exact moment where most teams ruin their incident response. Not because finding the change that broke everything is wrong, that has to happen fast, but because framing it as a witch hunt changes how people behave during the crisis. An engineer who's afraid to admit "I shipped that deploy twenty minutes ago" takes longer to say it, and those twenty minutes of silence are what separate a fifteen-minute incident from a two-hour one.
What to do in the first few minutes
The first decision isn't finding the root cause, it's containing the damage. Roll back if the change is recent and reversible, kill the feature flag if the new code sits behind one, redirect traffic to a healthy region if it's an infrastructure problem. Root cause gets investigated afterward, once the system is stable. Mixing diagnosis with mitigation is the most common mistake in incidents that drag on: people start reading logs instead of hitting the button that stops the bleeding.
A real example of this kind of problem: a database migration that adds an index without CONCURRENTLY in Postgres. Locally it runs in 200 milliseconds because the test table has 40 rows. In production, with 80 million rows, the migration takes an exclusive lock on the table for 40 minutes and brings down every request that needs to write there. Nobody touched business logic, the bug isn't in any carefully reviewed pull request, it's in an infrastructure command that nobody ran against a realistic data volume.
Why blameless culture doesn't mean no consequences
Blameless doesn't mean nobody is accountable, it means the analysis focuses on the system that allowed the error, not on the person who pressed the button. If an engineer was able to run that migration without CONCURRENTLY against production, the useful question isn't "why were they so careless," it's "why didn't our CI pipeline block a risky migration before it got to that point." The first question ends in an uncomfortable HR conversation. The second ends in an automated check that prevents the next instance of the same problem, no matter who's on call that day.
The postmortem that's actually useful, not the one that gets filed away
A useful postmortem has four non-negotiable parts: a timeline with real timestamps pulled from logs and metrics (not from memory, memory during a two-hour incident is terrible), impact measured in concrete numbers (affected users, minutes of downtime, lost revenue if applicable, not "it was bad"), the technical root cause without names in the narrative, and a list of action items with an owner and a date, not a list of good intentions.
The part that gets skipped most often is following up on those action items. It's common to see excellent postmortems with five follow-up tasks, three of which are still open six months later because nobody prioritized them against the product roadmap. If your incident process doesn't have a real mechanism for those items to compete for engineering time, the postmortem is theater: it documents the problem but doesn't fix it, and the same kind of incident comes back.
What changes in the system, not in the person
Teams that reduce the frequency of serious incidents don't do it by hiring more careful people, they do it by adding layers that make an individual mistake less costly: canary deployments that expose a change to 5% of traffic before 100%, feature flags that let you kill new code without a full rollback, SLO-based alerts with an error budget instead of arbitrary thresholds, and runbooks written for the typical incident instead of improvising every time. None of these things stop someone from making a mistake. All of them reduce the blast radius when it happens, which is the only variable you can actually control in practice.
A serious incident isn't anyone's moral failure, it's information about where your system is fragile that you didn't have until that moment. Senior teams aren't the ones who never break production, they're the ones with a process that turns every incident into a concrete improvement to the system, instead of spending the team's energy figuring out who to blame while the MTTR clock keeps running.
