Incident runbooks and a postmortem librarywhat to do at 3 a.m., and what you learned

Recipe No. 32 · Work and teams

For: The SRE lead who keeps the on-call runbooks

You get: A runbook per service, a section per alert, postmortems linked back

Time: An afternoon, then an hour per incident

See it in know.sh

A runbook entry named exactly as it pages: first checks as a flowchart, read-only commands as steps that copy, and a note added after the last incident.

Runbooksledger-worker

3of 9

LedgerQueueLagHigh

Observation, key section, 1 highlight, 1 note, 2 sources, 412 words

What it means. The ledger consumer is more than 5,000 messages behind. Invoices are still accepted, but invoice emails and ledger totals are delayed until it catches up.

First checks. Open the ledger-worker dashboard. The pool comes first: panel “DB pool in use”.

```element flowchart
{
  "caption": "First checks for LedgerQueueLagHigh, in order",
  "description": "Check the database pool first: at 100 per cent, escalate without restarting. Otherwise check the consumer is running and restart it if not. If a deploy went out in the last hour, ask its owner to roll back. If not, watch the lag for 20 minutes and escalate if it is still rising.",
  "direction": "TD",
  "nodes": [
    {
      "id": "page",
      "label": "LedgerQueueLagHigh pages",
      "shape": "stadium"
    },
    {
      "id": "pool",
      "label": "Pool at 100%?",
      "shape": "diamond"
    },
    {
      "id": "running",
      "label": "Consumer up?",
      "shape": "diamond"
    },
    {
      "id": "restart",
      "label": "Restart the consumer",
      "shape": "rectangle"
    },
    {
      "id": "deploy",
      "label": "Deploy this hour?",
      "shape": "diamond"
    },
    {
      "id": "rollback",
      "label": "Ask for a rollback",
      "shape": "rectangle"
    },
    {
      "id": "watch",
      "label": "Watch for 20 minutes",
      "shape": "rectangle"
    },
    {
      "id": "escalate",
      "label": "Escalate",
      "shape": "rectangle",
      "emphasis": "key"
    }
  ],
  "links": [
    {
      "from": "page",
      "to": "pool",
      "line": "solid",
      "head": "arrow"
    },
    {
      "from": "pool",
      "to": "escalate",
      "label": "yes",
      "line": "solid",
      "head": "arrow"
    },
    {
      "from": "pool",
      "to": "running",
      "label": "no",
      "line": "solid",
      "head": "arrow"
    },
    {
      "from": "running",
      "to": "restart",
      "label": "no",
      "line": "solid",
      "head": "arrow"
    },
    {
      "from": "running",
      "to": "deploy",
      "label": "yes",
      "line": "solid",
      "head": "arrow"
    },
    {
      "from": "deploy",
      "to": "rollback",
      "label": "yes",
      "line": "solid",
      "head": "arrow"
    },
    {
      "from": "deploy",
      "to": "watch",
      "label": "no",
      "line": "solid",
      "head": "arrow"
    },
    {
      "from": "watch",
      "to": "escalate",
      "label": "still rising",
      "line": "solid",
      "head": "arrow"
    }
  ]
}
```

Commands. Read-only; each copies as it stands.

Read-only commands

List the consumer pods
Every pod should be Running with no recent restarts.
```
kubectl -n ledger get pods -l app=ledger-worker
```
Count retries in the last 15 minutes
More than a few hundred means the consumer is retrying, not stuck.
```
kubectl -n ledger logs deploy/ledger-worker --since=15m | grep -c "retry"
```

```element steps
{
  "label": "Read-only commands",
  "steps": [
    {
      "title": "List the consumer pods",
      "body": "Every pod should be *Running* with no recent restarts.",
      "code": "kubectl -n ledger get pods -l app=ledger-worker"
    },
    {
      "title": "Count retries in the last 15 minutes",
      "body": "More than a few hundred means the consumer is retrying, not stuck.",
      "code": "kubectl -n ledger logs deploy/ledger-worker --since=15m | grep -c \"retry\""
    }
  ]
}
```

Escalation. Page the Payments platform secondary if lag is still rising after 20 minutes. Customer-facing impact goes to the incident lead.

Seen in. INC-2026-031, INC-2026-019.

The postmortem’s contributing factors as a fishbone, each cause a Problem section below it. Blameless: every bone names a system, never a person.

```element fishbone
{
  "caption": "INC-2026-031: why invoice emails were delayed by three hours",
  "description": "A fishbone diagram with the effect, invoice emails delayed three hours, at its head. Four categories feed the spine: the consumer (retries without backoff, one pool shared by all workers), alerting (lag alert fired after 38 minutes, threshold set in 2023), the runbook (restart was the first step, pool check missing) and the deploy (retry change shipped on a Friday, no load test).",
  "effect": "Invoice emails\ndelayed three hours",
  "causes": [
    {
      "id": "consumer",
      "label": "Consumer"
    },
    {
      "label": "Retries without backoff",
      "parent": "consumer"
    },
    {
      "label": "One pool for all workers",
      "parent": "consumer"
    },
    {
      "id": "alerting",
      "label": "Alerting"
    },
    {
      "label": "Lag alert after 38 minutes",
      "parent": "alerting"
    },
    {
      "label": "Threshold set in 2023",
      "parent": "alerting"
    },
    {
      "id": "runbook",
      "label": "Runbook"
    },
    {
      "label": "Restart was step one",
      "parent": "runbook"
    },
    {
      "label": "No pool check",
      "parent": "runbook"
    },
    {
      "id": "deploy",
      "label": "Deploy"
    },
    {
      "label": "Retry change on a Friday",
      "parent": "deploy"
    },
    {
      "label": "No load test",
      "parent": "deploy"
    }
  ]
}
```

Claude Code drafting a postmortem from local notes and linking it to the runbook. Every write shows in Revisions as made by an assistant.

Claude Code, connected to know.sh

You

Read incident-notes-2026-09-18.txt (customer details already removed). Draft a blameless postmortem for INC-2026-031 on the Postmortems shelf: summary and impact in the overview, a UTC timeline, contributing factors as Problem sections, actions as Suggestion. Link it to the LedgerQueueLagHigh runbook and add a Seen in line there.

search Found ledger-worker, 3 “LedgerQueueLagHigh”

document_create Created INC-2026-031 — Ledger queue backlog delayed invoice emails on Postmortems, No. 64

section_add Added “Timeline, 18 September (UTC)” to INC-2026-031, 1

section_add Added “Connection pool exhausted by retrying consumers” (Problem), 2

section_add Added “Lag alert fired 38 minutes after the backlog began” (Problem), 3

section_add Added “Cap consumer retries with backoff” (Suggestion), 5

section_update Added “Seen in: INC-2026-031” with a link to ledger-worker, 3

Claude Code

Filed No. 64 with seven sections. Two things were not in the notes: when the backlog finished draining, and whether the delayed invoice emails were resent. I filed both as Question sections rather than guessing. Check the timeline against the consumer logs before the review. Once the actions are agreed, I can lay them out as a campaign.

How it works

When LedgerQueueLagHigh pages at 3 a.m., the link in the alert opens the service’s runbook: first checks as a flowchart, read-only commands that copy whole, and every past incident it paged for. Claude Code drafts the runbooks from your alert rules and each postmortem from your notes, and links the two.

Draft a runbook per service
Start Claude Code in the repository that holds your alert rules. Ask for a document per service and a section per alert, titled exactly as it pages.
Draw the first checks, list the commands
Ask for the first checks as a flowchart element and the read-only commands as a steps element. Then check every command yourself against the running system.
Put the link where the page lands
Share each runbook with a password and paste the link into the alert’s description, so the page carries it. Keep every credential out of every command.
Turn incidents into postmortems and follow-ups
After an incident, give Claude Code your notes with customer details removed. It drafts a blameless postmortem, links it to the runbook, and lays the agreed actions out as a campaign.

Try this prompt

A coding assistant with the know.sh plugin, such as Claude Code:

Read incident-notes.txt in this folder. Draft a blameless postmortem on my know.sh Postmortems shelf titled with the incident number: summary and impact as the overview, a UTC timeline section, each contributing factor as a Problem, each proposed action as a Suggestion. Use roles, not names. Link it to the runbook section for the alert that fired, and file anything the notes do not settle as a Question.

Your assistant, connected to know.sh (Claude, ChatGPT or a local model):

Using know.sh, read the LedgerQueueLagHigh section in my ledger-worker runbook. Rewrite its First checks as a flowchart element, top to bottom, one diamond per question, the database pool first, ending in Escalate; and its Commands as a steps element, one read-only command per step with a line on what a healthy result looks like. Keep every word of the escalation path as it is.

Made with

Document Your AI assistant Elements Links between documents Public link Campaigns

See it in know.sh

LedgerQueueLagHigh

Read-only commands

List the consumer pods

Count retries in the last 15 minutes

Read-only commands

How it works

Draft a runbook per service

Draw the first checks, list the commands

Put the link where the page lands

Turn incidents into postmortems and follow-ups

Try this prompt

Made with

See also