Why one slow service takes down the rest: deadlines, retry budgets and load shedding
How overload spreads between services, and four habits that stop it: deadlines that propagate, retry budgets, small queues and load shedding.

A service gets slow. Its callers wait longer, so they hold more connections, so they get slow too. Some of them time out and try again, which adds load to the service that was already struggling. A few minutes later, half the system is down and the original problem, a single slow database query, is no longer the interesting part.
Google's SRE book calls this a cascading failure, and its chapter on the topic is one of the most useful things you can read before your first big outage. This post turns its advice into four habits you can add to an ordinary web backend.
How a small problem becomes a big one
flowchart LR U[Users] --> A[API] A --> B[Service B] B --> C[(Database)] C -. slow .-> B B -. waiting .-> A A -. retries .-> B B -. retries .-> C
Each hop does something reasonable on its own: wait for the answer, and if there is none, try again. Together they multiply. The SRE book gives a vivid example: if three layers each make up to four attempts, one user request can turn into 64 attempts (four to the power of three) against the database at the bottom. The database was slow; now it is buried.
The habits below each cut one part of that loop.
1. Always set a deadline, and pass it down
A request with no deadline can wait forever, and a request with a very long one is nearly as bad. The book warns against deadlines orders of magnitude above normal latency, because stuck work holds threads and memory while nobody is waiting for the answer anymore.
The more subtle part is that a deadline belongs to the whole request, not to each call. If a user-facing request has 30 seconds and has spent 7 seconds already, the next service should be told it has about 23 seconds, minus a little for the network. If there is no time left, do not start the work.
Go's context makes this natural:
func handle(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
defer cancel()
// The same ctx flows into every downstream call.
rows, err := store.Search(ctx, r.URL.Query().Get("q"))
if err != nil {
http.Error(w, "try again shortly", http.StatusServiceUnavailable)
return
}
_ = rows
}
Every call that accepts the context stops when the time is up or the client leaves. That second part matters: cancellations should travel downward too, so abandoned requests stop using resources across the whole stack.
Tip
Pick a deadline near the slowest normal response, with some room, not near your patience. If a call usually takes 80 ms, two seconds is generous. Five minutes is a bug.
2. Retry less, and with a budget
Retries are good for brief failures and dangerous during real overload. The book's rules are simple enough to write on a sticky note:
- Use randomized exponential backoff. Waiting longer each time, with jitter, stops retries from arriving in synchronized waves. Our post on rate limits from both sides has a code sample.
- Limit attempts per request. Their naive example allowed 10 backend attempts per logical request, which made an overload far worse.
- Add a budget per process. Their example is 60 retries per minute for a whole server process. When it is used up, failures are returned instead of retried.
- Retry in one layer only. Decide where retrying happens and keep the others plain. This alone removes the 4 x 4 x 4 problem.
- Do not retry permanent errors. A malformed request or "not found" will not improve with time.
A retry budget can be tiny:
type Budget struct {
tokens atomic.Int64 // refilled once a minute by a ticker
}
func (b *Budget) Allow() bool {
for {
n := b.tokens.Load()
if n <= 0 {
return false
}
if b.tokens.CompareAndSwap(n, n-1) {
return true
}
}
}
Before retrying, ask Allow(). If it says no, fail fast. A service that fails quickly during an incident is helping its dependency recover.
Retries also assume the operation is safe to repeat, which is what idempotency keys are for.
3. Keep queues short
A big queue feels safe: absorb the spike, work through it later. During overload it does the opposite. The SRE chapter gives the arithmetic: with a queue ten times the size of the thread pool, a request that normally takes 100 ms can spend about 1.1 seconds in the system when the queue is full. By then the client has probably given up, and you are doing work nobody wants.
Their guidance:
- Reject new requests when the queue is full rather than letting them wait.
- For steady traffic, keep queues small, for example half the thread pool size or less.
- For bursty traffic, size the queue from thread count, request time, and how big and how frequent the bursts are.
When under pressure, it can also help to serve the newest requests first. Requests that have waited a long time are the most likely to have been abandoned already.
4. Shed load before you fall over
Load shedding means deliberately refusing some work so the rest can succeed. A simple version: count requests in flight, and once the number passes a limit, answer new ones with HTTP 503 immediately.
var inflight atomic.Int64
func shed(next http.Handler, limit int64) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if inflight.Add(1) > limit {
inflight.Add(-1)
w.Header().Set("Retry-After", "2")
http.Error(w, "busy", http.StatusServiceUnavailable)
return
}
defer inflight.Add(-1)
next.ServeHTTP(w, r)
})
}
A specific overload response matters: clients and upstream layers can tell "busy, back off" apart from "broken, try again". The book also offers practical cautions that I find easy to forget:
- Keep an off switch. Shedding logic is complex and can misfire or create feedback loops.
- Test the shedding path. Rarely used code tends to be broken. Running a few servers close to their limit now and then shows whether it works.
- Alert when many servers are shedding. It is normally a sign of a capacity or traffic problem worth knowing about.
A gentler cousin is graceful degradation: return a cheaper answer instead of none, for example search a smaller subset or skip a nonessential panel. It should trigger rarely, and when it does, it is a hint that capacity planning needs a look.
Check it with your health signals
None of this helps if the platform keeps sending traffic to a pod that is busy, or restarts a pod that is only slow. Revisit your Kubernetes probes: a readiness check that fails during overload can push all the load onto fewer pods, and a liveness check that does the same can restart them in a loop.
And decide what "healthy" means before you need it. A service level objective with a latency target tells you how much shedding or degrading is acceptable and when to stop and fix things.
A short plan for this week
- List every outgoing call in one service and note its timeout. Anything missing or over 10 seconds gets a deadline.
- Draw which layers retry. Remove retries from all but one.
- Add a per-process retry budget to that layer.
- Look at your queue sizes and set a limit that rejects when full.
- Add a simple in-flight limit with a 503 and a
Retry-Afterheader, behind a flag you can switch off.
None of these needs new infrastructure, and each one makes a bad day a little smaller.


