A service can fail because its clients are trying to help

Retries turn a slow service into a dead one, and turning them off does not bring it back. Six simulations you can take the controls of.

A queue is not a buffer

Here is a service with eight servers, taking requests at a rate you control. It can handle twelve thousand a second. Nothing times out and nothing retries — the only thing happening is arrival and service.

Push the load toward capacity and watch the queue. It does not grow in proportion. At three quarters of capacity it is barely there; at ninety-five percent it is most of the way to the limit. The wait a request sees goes as 1/(1 − utilisation), so the last few percent of headroom cost more than all the rest put together.

Clients do not wait forever

Now give the client a deadline. A request that has waited longer than the timeout is abandoned — but the server does not know that, and goes on to serve it anyway. That work is spent on an answer nobody is waiting for.

This figure opens past capacity, because that is the only way to get there without retries: while arrivals are slower than service, the queue stays short and nothing waits long enough to give up. Push arrivals over twelve thousand and the queue fills instead.

Then the blocks turn red. Red is work that is already lost though nothing has told it yet — everything ahead of it must be served first, and that will take longer than the client is willing to wait. Remember that shape. In the next figure we will get there without ever raising the load.

A retry is new work

A client that times out tries again. That is the correct thing for it to do, and it is also the whole problem: the retry is a new request, arriving at a service that is already behind. Failure has become a source of load.

Move the retry control off zero. Demand is no longer the load you offered — it is the load you offered plus everything the failures manufactured, and the fraction of your own traffic that is self-inflicted is the number to watch.

There is no gentle version of this

You might expect the damage to arrive in proportion — a few more retries, a little less goodput. It does not. Move the control slowly through the middle of its range and the system holds, holds, holds, and then is gone.

At 1.6 retries per failure this service does about eleven thousand a second. At 1.7 it does under a thousand. Same offered load, same capacity, same timeout; the only thing that moved was how hard the clients try. Every seed we ran puts the edge in the same place.

And it does not come back

Now the part that makes this a problem rather than a curiosity. Drive the retry control up until the service is gone — then bring it back down to where it started. The clients are behaving exactly as they were when everything was fine.

The service stays dead. By now every request in the queue has already waited longer than its client will tolerate, so all of them fail no matter how quickly they are served, and each failure buys another retry. The queue is being fed by its own contents. The system has stopped being a function of its input and become a function of its history.

What actually works

You cannot fix this by asking clients to retry less, because the retry rate is not what is holding the queue open. You fix it by making the amplification impossible to reach — a budget, expressed as a fraction of the client’s own traffic rather than as a per-request rule.

This figure opens collapsed, at 2.5 retries per failure. Pull the budget down. Once a client may only retry a tenth of what it sends, no amount of failure can manufacture the load that keeps the queue full, and the queue drains in about two seconds — because most of what was in it was never real work.

What to take back to your own system

Retries are not free, and their cost is not paid by the client that sends them. Cap them as a share of traffic rather than per request; make the cap live where the load arrives, not where it originates; and if you are already collapsed, expect to shed load to get out, because turning the cause off will not do it.

The mechanism here is documented in the Google SRE book. The particular numbers — twelve thousand a second, a 120ms timeout, forty milliseconds of service — are chosen to put the threshold somewhere you can see it, not measured from a real service. The simulation is deterministic and seeded: the same settings give the same run on any machine.