mirror of
https://github.com/hanzoai/ingress.git
synced 2026-10-26 02:47:15 +00:00
zeekayandHanzo Dev
489fe874b0
An exhausted pool is transient back-pressure, but we were reporting it as a
bare `http.Error` 503 with no retry signal and no cache directive, and we were
doing it in four separate places.
WHY this matters operationally: *.hanzo.app routes 163 sites through a single
k8s Service. Every rolling deploy briefly drains that Service to zero ready
endpoints, and with `allowEmptyServices=true` the router still exists, so the
balancer answers. In the last 24h that produced 191 such responses on the
hanzo-app wildcard router (access log: `503 20 ... "-" 0ms` — no upstream, 0ms).
Those responses carried:
- no Retry-After, so clients and probes could not tell "come back in 2s" from
"this is broken", and retried immediately against an already-empty pool;
- no Cache-Control, so any intermediary was free to retain the error. A
cached capacity error outlives the outage that produced it and turns a
two-second deploy gap into a sticky one.
The deeper problem was that "what to say when there is no server" lived in four
balancers (wrr, p2c, hrw, leasttime), each with its own copy of the error value
and its own copy of the write. Four copies is four chances to drift, and a
capacity error that drifts into a success status is exactly how a broken fleet
reports itself healthy.
So: one error value and one writer in the parent loadbalancer package, and all
four strategies funnel through it. Status is written before the body so a
saturated pool can never be mistaken for a healthy one — a status-code monitor
must see 503.
Tests assert 503 + Retry-After + no-store at the writer and through all four
strategies' ServeHTTP; they fail against the previous http.Error behaviour.
Co-authored-by: Hanzo Dev <[email protected]>