When something breaks: a simple incident command routine for small teams
Four roles, one shared document and clean handoffs: how a small team can run an incident calmly, drawing on Google's SRE guidance on managing incidents.

The first twenty minutes of an outage are rarely about the fix. They are about people: three engineers investigating the same log, nobody telling the customer-facing team anything, two people changing the same setting at once.
Google's SRE book has a chapter on managing incidents, built on the idea of an incident command system. You do not need a large organisation to use it. A team of five can run the same routine with a few habits. This post sets those habits out in plain terms.
Declare early
The first decision is whether you are having an incident at all. People hesitate because it feels dramatic. The SRE chapter says to declare early rather than hours into a worsening problem, and it gives three simple triggers. Treat it as an incident if any one is true:
- A second team has to be involved to fix it.
- Customers can see the outage.
- The problem is still unsolved after an hour of focused analysis.
Write those down for your team. A rule you agreed on in a calm week is easier to follow than a judgment call at 2 a.m.
Four roles
Under pressure, everyone wants to debug. An incident needs a few people who do something else. The chapter describes four roles:
| Role | What they do |
|---|---|
| Incident commander | Holds the overall state, assigns the other roles, removes roadblocks. Holds any role they have not delegated. |
| Operations | Uses the tools to work on the problem. Only this group changes the system during the incident. |
| Communication | Is the public face of the response: sends regular updates to responders and stakeholders and keeps the incident document current. |
| Planning | Handles the longer-term work: filing bugs, arranging handoffs, tracking how the system differs from normal so it can be put back. |
On a small team one person may hold two roles, and that is fine. What matters is that the roles exist. In particular, "only operations changes the system" removes a whole class of confusion: when two people are changing things, nobody knows which change helped.
Note
The commander does not need to be the most senior engineer or the best debugger. They need to keep the picture straight and make decisions. Often the best debugger is more useful in the operations role.
One live document
The chapter calls keeping a live incident state document the commander's most important responsibility. It is simpler than it sounds. Use a shared document that several people can edit at once, start from a template, and put the most important information at the top. A starter template:
# Incident: <short name>
Status: investigating | mitigating | monitoring | resolved
Commander: <name> Ops: <name> Comms: <name> Planning: <name>
Impact: <who is affected and how>
Current theory: <one or two lines>
## Timeline
- 14:05 Alert fired for checkout errors
- 14:12 Declared incident; Priya is commander
- 14:20 Rolled back release 4812; error rate unchanged
## Open questions
- Is the payment provider degraded?
## Actions to follow up
- (Planning fills this in)
It can be messy, but it has to stay useful. Keep it afterwards: it is the raw material for the blameless postmortem.
Handoffs
Incidents outlast shifts. The chapter is specific about handing over command, especially at the end of the working day:
- Brief the incoming commander fully, by phone or video if they are remote.
- The outgoing commander explicitly transfers command.
- They stay on until the new commander clearly acknowledges it.
- Everyone involved is told who is leading.
The point is that command is never ambiguous. A message like "I'm signing off, someone should look at this" is not a handoff. "Sam, you have command; are you ready?" followed by "I have command" is.
Communication that does not interrupt
The communication role exists so that responders are not answering questions while debugging. Agree on a rhythm: for example, an update every thirty minutes, even if the update is "no change". People stop asking when they know the next update is coming.
Keep updates short and structured: what we know, what we are doing, when the next update is. Avoid guessing at causes in messages that leave the team.
Habits that help
The chapter's list of best practices is a good checklist for a team retrospective:
- Prioritise. Stop the damage, restore service, and preserve evidence for root-cause analysis, in that order.
- Prepare. Write down incident procedures in advance, with input from the people who will use them.
- Trust. Give participants full autonomy within their assigned roles.
- Introspect. Notice when you feel panicky or overwhelmed, and ask for more support.
- Reconsider. Ask regularly whether the current approach still makes sense.
- Practise. Run the process often, including recovery exercises, until it is second nature.
- Rotate. Let team members take different roles over time so everyone learns each one.
Practice is the one most teams skip. A thirty-minute game day, where someone invents a small failure and the team runs the routine, shows the gaps in your template and your alerts without any customer being affected. Pair it with the targets from service level objectives in one afternoon, so the team agrees in advance on what counts as serious.
A minimum version for this week
- Write the three declaration triggers on your team page.
- Copy the document template above into your wiki.
- Decide who can be commander, and make the list longer than one name.
- Agree on one channel for the incident and one update rhythm.
- Schedule a game day for next month.
None of this prevents outages. What it does is make them shorter and less stressful, and it leaves a clear record for learning afterwards.


