vvek

A calmer on-call: page only for what needs a human, and keep a short playbook for every alert

Practical on-call habits from Google's SRE workbook: workload limits, response times, actionable alerts, playbooks, escalation and handoffs.

Being on call is a promise: if something breaks, you will notice and act. How heavy that promise feels depends far less on the person than on how the system around them is set up. A rotation with three noisy alerts a night burns people out. A rotation with one clear page a week is a normal part of the job.

This post collects practical habits from the on-call chapter of Google's SRE Workbook, scaled down for a small team. Nothing here needs special tools; it needs a few decisions written down.

Start with a budget, not a rota

Before you decide who is on call, decide how much load is acceptable. The workbook suggests aiming for at most two incidents per 12-hour shift, where one incident means one underlying problem even if it triggers several alerts. It also advises keeping shifts to 12 hours, because long shifts breed tiredness and mistakes, and keeping at least half of an engineer's time for project work so that fixing the causes is never squeezed out.

Then measure. Track pages over a trailing window (the book mentions 21 days) and set a warning threshold. If the average goes over budget, that is a signal to fix the system, not to ask people to cope.

Note

Small teams cannot copy Google's staffing numbers. The principle still holds: a rotation of two people has no room for illness or holidays, so say so early and plan for cover.

Page only when a human must act now

A page is an interruption of someone's evening, sleep or family. Spend it carefully. Every paging alert should pass three checks:

  1. Is a customer or the business affected, or about to be?
  2. Is there something a person can do about it right now?
  3. Would waiting until morning make it meaningfully worse?

If the answer to the last is no, make it a ticket. A stuck batch job might deserve a response in about half an hour, and a failing backup on a service that has not launched can wait for working hours. The workbook suggests reserving a response in minutes, around five in its example, for severe, customer-facing outages.

Where a fix is always the same, automate it and stop paging for it. When you add a new paging alert, run it as non-paging first for a while and see how often it fires and whether it was useful.

If you are setting up targets for what "affected" means, Service level objectives in one afternoon is a good companion.

One alert, one playbook entry

Every alert should link to a short page that answers four questions:

  • Severity: how bad is it likely to be?
  • Impact: who or what does it affect?
  • Checks: what to look at first, with the exact dashboard or command.
  • Mitigation: what to do to reduce harm, even if the real fix comes later.
Alert: checkout error rate above 2% for 5 minutes
Impact: customers cannot pay
First checks: payments dashboard; last deploy time; provider status page
Mitigation: roll back the last deploy; if provider is down, enable the queue-and-retry flag
Escalate to: payments lead

Keep it brief. A runbook nobody reads in the middle of the night is worse than none. The workbook's advice is to let the person on call update the entry whenever an alert fires, while the details are fresh. And if an entry turns into a fixed list of commands, that is a hint to turn it into a script, or better, to fix the underlying problem.

You are never alone

Escalation is part of the design, not a sign of failure. Write down who is the secondary, who owns each service, and who can make a decision to roll back or switch a feature off. Make it normal to ask. A person who hesitates for an hour because they don't want to wake a colleague costs more than the colleague's sleep.

During a serious incident, the roles in a simple incident command routine help: one person leads, one communicates, one fixes, one writes things down.

Handoffs

At the end of a shift, write a short note: what happened, what is still open, what to watch. At the start, read the previous note before anything else. Three lines are enough:

Open: slow queries on reports DB, ticket 482, no customer impact yet
Changed: raised pool size from 20 to 30 at 16:10
Watch: nightly import at 02:00, failed twice this week

Close the loop

Two habits keep the pager quiet over time:

  • Give every page a tracked cause. "Unknown" and "transient" are placeholders, not conclusions; each hides a future repeat.
  • Write a postmortem for serious incidents, and keep it free of blame so people report problems early. Blameless postmortems covers the format.

Then tie pages to bugs and fix the ones that disturb people most. The aim is a system where being on call is dull.

A first step for this week

Open last month's pages and sort them into three piles: needed a human now, could have waited, and should have been automated. Delete or downgrade the second, automate the third, and write a four-line playbook for each alert left in the first. That alone makes next month calmer.

Sources

Keep reading

A short email when we publish. Confirm once, unsubscribe any time.