mirror of
https://github.com/hanzoai/ingress.git
synced 2026-10-26 02:47:15 +00:00
zeekayandHanzo Dev
489fe874b0
An exhausted pool is transient back-pressure, but we were reporting it as a
bare `http.Error` 503 with no retry signal and no cache directive, and we were
doing it in four separate places.
WHY this matters operationally: *.hanzo.app routes 163 sites through a single
k8s Service. Every rolling deploy briefly drains that Service to zero ready
endpoints, and with `allowEmptyServices=true` the router still exists, so the
balancer answers. In the last 24h that produced 191 such responses on the
hanzo-app wildcard router (access log: `503 20 ... "-" 0ms` — no upstream, 0ms).
Those responses carried:
- no Retry-After, so clients and probes could not tell "come back in 2s" from
"this is broken", and retried immediately against an already-empty pool;
- no Cache-Control, so any intermediary was free to retain the error. A
cached capacity error outlives the outage that produced it and turns a
two-second deploy gap into a sticky one.
The deeper problem was that "what to say when there is no server" lived in four
balancers (wrr, p2c, hrw, leasttime), each with its own copy of the error value
and its own copy of the write. Four copies is four chances to drift, and a
capacity error that drifts into a success status is exactly how a broken fleet
reports itself healthy.
So: one error value and one writer in the parent loadbalancer package, and all
four strategies funnel through it. Status is written before the body so a
saturated pool can never be mistaken for a healthy one — a status-code monitor
must see 503.
Tests assert 503 + Retry-After + no-store at the writer and through all four
strategies' ServeHTTP; they fail against the previous http.Error behaviour.
Co-authored-by: Hanzo Dev <[email protected]>
47 lines
1.9 KiB
Go
47 lines
1.9 KiB
Go
package loadbalancer
|
|
|
|
import (
|
|
"errors"
|
|
"net/http"
|
|
"strconv"
|
|
)
|
|
|
|
// ErrNoAvailableServer is the single canonical "the pool cannot take this
|
|
// request" error, shared by every balancing strategy (wrr, p2c, hrw,
|
|
// leasttime). It means the pool is empty, every server is fenced, or health
|
|
// checks have marked them all down.
|
|
var ErrNoAvailableServer = errors.New("no available server")
|
|
|
|
// NoAvailableServerRetryAfter is the Retry-After (seconds) advertised with a
|
|
// saturated/empty pool. Kept short: the common cause is a rolling deploy or a
|
|
// brief health-check flap, where the pool is back within a second or two.
|
|
const NoAvailableServerRetryAfter = 2
|
|
|
|
// WriteNoAvailableServer writes the one canonical response for
|
|
// ErrNoAvailableServer: 503 Service Unavailable with Retry-After.
|
|
//
|
|
// Every strategy funnels through here so capacity exhaustion is reported
|
|
// identically no matter which balancer is configured. Two properties matter
|
|
// operationally and are the reason this is not just http.Error:
|
|
//
|
|
// - Retry-After tells clients and probes this is transient back-pressure
|
|
// rather than a hard failure, so they back off instead of hammering an
|
|
// already-empty pool.
|
|
// - Cache-Control: no-store stops any intermediary (CDN, browser) from
|
|
// retaining the error. A cached capacity error outlives the outage that
|
|
// produced it and turns a two-second deploy gap into a sticky one.
|
|
//
|
|
// The status is written before the body precisely so a saturated pool can
|
|
// never be mistaken for a healthy one: a status-code monitor must see 503.
|
|
func WriteNoAvailableServer(rw http.ResponseWriter) {
|
|
h := rw.Header()
|
|
h.Del("Content-Length")
|
|
h.Set("Content-Type", "text/plain; charset=utf-8")
|
|
h.Set("X-Content-Type-Options", "nosniff")
|
|
h.Set("Retry-After", strconv.Itoa(NoAvailableServerRetryAfter))
|
|
h.Set("Cache-Control", "no-store")
|
|
|
|
rw.WriteHeader(http.StatusServiceUnavailable)
|
|
_, _ = rw.Write([]byte(ErrNoAvailableServer.Error() + "\n"))
|
|
}
|