vvek

Service level objectives in one afternoon: pick two indicators, set a loose target, spend the error budget

How to define SLIs and SLOs, why percentiles beat averages, and how an error budget turns reliability into a release decision. Based on Google's SRE book.

"Is the service reliable?" is a question that gets different answers from different people. An engineer checks a dashboard, a product manager recalls last week's outage, and a customer remembers one slow page. A service level objective (SLO) is how a team replaces those impressions with one agreed number.

This post follows the vocabulary and advice in the Service Level Objectives chapter of Google's SRE book, and turns it into steps you can do in an afternoon.

Three terms, kept apart

  • SLI (service level indicator): a measurement of some aspect of the service, such as request latency, error rate or throughput.
  • SLO (service level objective): a target value or range for an SLI. For example, the SLI stays at or below some number.
  • SLA (service level agreement): a contract with users that includes consequences for missing the SLOs in it.

The book gives a simple test to tell an SLO from an SLA: ask what happens if the target isn't met. If there is no explicit consequence, you are looking at an SLO. Most internal teams need SLOs first and can leave SLAs to the people who write contracts.

Step 1: choose two indicators, from the user's side

The book says to use a handful of representative SLIs: too many dilute attention, too few leave behaviour unexamined. It also says to start from what users care about rather than what is easy to measure.

For a user-facing service, availability, latency, and throughput are typical. For storage it adds durability, and for any system, correctness. Pick two to start. For a web API that might be:

  1. Availability: the share of requests that succeed (not 5xx).
  2. Latency: the share of requests that finish under a threshold.

Count requests at a point that matches the user's experience, like the load balancer, not deep inside a single service. The book also notes that some systems need client-side measurement, because server-side numbers can miss problems users actually see.

Step 2: use percentiles, not averages

The book points out that most metrics are better understood as distributions than as averages. Its example, in the chapter's Figure 4-1, has a median request of about 50 ms while 5% of requests are about 20 times slower. An alert on the average would miss that tail entirely.

So write latency as a percentile statement. The book's own example is: "99% (averaged over 1 minute) of Get RPC calls will complete in less than 100 ms." Note how it states the percentage, the window, the operation and the threshold. Anyone can check it.

You can use several targets to describe the shape of the curve, as the book does with 90% under 1 ms, 99% under 10 ms and 99.9% under 100 ms. Start with one.

Step 3: set the target loose

The book's advice here is direct. Do not simply copy current performance as the target, because that can lock in effort you cannot sustain. Avoid absolutes such as "always available". Keep the number of SLOs small. And it is better to start with a loose target that you tighten than to pick an overly strict one.

A target is about users, so ask: at what level would they notice and complain? Then pick something a bit easier and add the measurement window.

Step 4: turn the gap into an error budget

The book states that insisting on meeting SLOs 100% of the time is both unrealistic and undesirable. The allowed miss rate is the error budget.

A worked example, with numbers I chose for illustration. Say your availability SLO is 99.5% over 30 days. The budget is the other 0.5%. Thirty days is 720 hours, so that is 3.6 hours of full downtime, or the equivalent in failed requests.

Now the budget does a job. The book says to compare the SLO violation rate with the budget, and let the remaining gap inform release decisions:

Budget state A sensible response
Plenty left Ship features, run risky experiments
Nearly spent Slow rollouts, prioritize reliability work
Overspent Pause risky changes until the window recovers

Agree on this table before an incident, not during one. If you already use staged rollouts, our post on feature flags shows a way to make the "slow down" response cheap.

Tip

The book's phrase is worth remembering: an error budget is just an SLO for meeting other SLOs. Track it daily or weekly for the team, and expect managers to want a monthly or quarterly view.

Step 5: close the loop

The chapter describes a simple control loop:

  1. Monitor the SLIs.
  2. Compare them with the SLOs and decide whether action is needed.
  3. If so, work out what must happen to meet the target.
  4. Take that action.

When a miss turns into an incident, a blameless postmortem is the natural place to decide what changes.

A one-page template

Write this down for each service:

  • User journey: what the user is trying to do.
  • SLI 1 and SLI 2: exactly how each is measured and where.
  • SLO: the target, the percentile and the window.
  • Budget policy: what you do at "plenty", "nearly spent" and "overspent".
  • Review date: when you will look again and tighten or loosen the target.

The book also suggests making an internal SLO a bit tighter than what you publish, which gives you a safety margin. And it warns against overachieving: if you are far better than the target for long, users come to depend on that actual performance.

Sources

Keep reading

A short email when we publish. Confirm once, unsubscribe any time.