The forensic method: six phases for production incidents

Before anything else, capture what rots: exact error text, timestamps, log windows, process state, versions deployed, recent changes, who is affected and since when. Restarting a service is destroying a crime scene; sometimes you must (mitigation outranks diagnosis when users are bleeding), but do it knowingly and screenshot first. The instinct this phase fights: the reflexive restart that "fixes" it and guarantees a rerun next week.

Convert "it's broken" into a statement precise enough to be wrong: expected X, observed Y, scope Z, first seen T. Half of all incidents change shape in this phase — the report says "the API is down" and the symptom turns out to be "one endpoint times out for accounts with more than 10k rows." If you cannot state when it last worked, finding that out becomes the first task.

Now the data: logs, query plans, traces, diffs of everything that changed near T-zero (deploys, config, data, dependencies, infrastructure). Build a timeline where every entry cites its source. Facts only — "response time tripled at 14:02" qualifies, "the new release broke it" does not, because that is a theory wearing a fact's clothes.

Enumerate every explanation consistent with the timeline, including the unflattering ones. For each, design the cheapest experiment that would disprove it, then run them cheapest-first. The pristine-baseline test belongs here: reproduce on a clean environment before suspecting your own code. Record kills in the ledger — a refuted hypothesis is paid-for knowledge, and the ledger is what stops the team from re-investigating it at 2am.

The surviving hypothesis still has to earn a conviction: reintroduce the cause and the failure must return; remove it and the failure must vanish. Where you can, get a second witness — a different tool observing the same fault tells you which parts of the story are real. In the capital-letter incident, webpack was the second witness that made Turbopack's vague invariant confess. If bidirectional proof is impossible (it happens: unreproducible data races, third-party black boxes), say so in the report instead of upgrading suspicion to fact.

The fix should be as small as the conviction allows — large "while we're in here" fixes smuggle new suspects into the next incident. Add the guard that makes the regression loud (test, alert, invariant, lint rule), then write the autopsy: symptom, timeline, ledger with kills, conviction evidence, fix, guard. Twenty minutes of writing converts the incident from tribal memory into infrastructure. It is also, not incidentally, how this blog gets its material.