Incident runbooks and a postmortem librarywhat to do at 3 a.m., and what you learned

Method

1. 1

   Have your assistant draft a runbook per service

   Start claude in the repository that holds your alert rules, with the know.sh plugin installed, and export the old wiki pages into the same folder. Ask it to make two shelves, Runbooks and Postmortems, and on Runbooks a document for each service that pages: ledger-worker, billing-api, auth-gateway, Postgres primary. Start with the five that paged most last quarter.

   Ask for the same overview on every service: what it does, what it depends on, which dashboards to open (by name), which team owns it, and the escalation path. If you use Claude or ChatGPT instead, attach the rules and pages to the chat; the result is the same shape.

2. 2

   Get a finding per alert, named as it pages

   Ask for a finding per alert, titled with the alert’s exact name from the rule: LedgerQueueLagHigh. At 3 a.m. the engineer opens the service’s link from the page and finds the alert by the name they were paged with; on your own on-call weeks, Look up (⌘K) takes you straight to it. Under the title, use four bold headings every time: What it means, First checks, Commands, Escalation.

   Commands go in code blocks, read-only first, with a plain warning before any command that changes something: “Changes state: only after the checks above.” Then press Edit and check every command yourself against the live system; the assistant drafted them from rules and old pages, not from experience. Mark the alerts that page at night as Key.

3. 3

   Keep secrets and customer data out of every command

   A runbook is read by whoever holds the link, so no command in it carries a credential. Refer to connection strings through the environment or the secret store (psql "$LEDGER_RO_URL"), and write where the value lives, not what it is.

   Use placeholders for anything about a customer: <account-id>, never a real account. The same applies to example output in a finding: trim the log lines of email addresses, names and invoice numbers before they go into a finding.

4. 4

   Put the link where the page lands

   Press Share on each runbook document, copy the link when it appears (it is shown once), and paste it into the alert’s description in your monitoring, so the page carries it. Leave it with no expiry (the default). Set a password on every runbook link and keep it in the on-call password manager entry; leave off the link altogether any runbook that describes a security weakness.

   Links are made per document, which is the right size here: one per service. If a link ends up somewhere it should not, replace it, and paste the new one into the alert rules.

5. 5

   Hand the incident notes to Claude Code

   After an incident, save the incident channel’s notes as a text file and remove customer names, emails and account numbers first. Start claude in that folder and ask it to draft a postmortem on the Postmortems shelf, titled with the incident number and a plain description.

   Ask for a fixed shape: an overview with the summary and impact, a Timeline finding in UTC, each contributing factor as a Problem, and each proposed action as a Suggestion. Ask it to link the postmortem to the runbook finding for the alert, add a “Seen in” line to that finding, and file what it cannot tell from the notes as a Question.

6. 6

   Review it blamelessly, then fix the runbook

   Read the draft against the logs before the review. An assistant can misread a timestamp or a sequence; the logs win. Write about roles and systems, never people: “the on-call engineer restarted the consumer”, not a name.

   In the review, agree the actions. In the editor, change each agreed Suggestion to a Decision, and delete the ones you dropped. Then go back to the runbook: if the first checks were wrong, rewrite them now. Revisions show what the assistant wrote, what you changed and when, which is a record in itself.

7. 7

   Lay the follow-ups out as a campaign

   Ask Claude Code to turn the postmortem’s Decision findings into a campaign: one operation set per incident, an operation per action, each linking back to its finding. Dependencies say what waits on what: Add a consumer lag alert blocks Tune autoscaling. The graph view shows the order.

   Campaigns are a plan for one person and have no assignees; put the owning team in each operation’s description, and keep the tickets in your tracker. Mark operations done as the tickets close.

8. 8

   Read across the postmortems

   Because every contributing factor is a finding title, you can search Look up (⌘K) for retry or certificate to list every incident where it recurs; the index gathers names such as Postgres and LedgerQueueLagHigh, with the incidents they appear in. Once a quarter, ask your assistant which factors recur across the Postmortems shelf and which runbooks still say nothing about them. That list is your reliability plan for the next quarter.