Files
ai/object/auth_config_test.go
zeekayandHanzo Dev 7ef733dacc auth: an unreachable IAM costs the requests, not the process
The signing cert every bearer is validated against was fetched in a package
init(), over the network, and the process panicked when it could not be read.
That was the right answer to the fail-open it replaced — warn-and-serve left the
service Running, green, and answering 401 to everything including free routes for
a day — and the wrong answer to everything else.

It has now taken production down twice. 2026-07-27: a one-word manifest typo
killed IAM's pods. 2026-08-02: an IAM build answered no /healthz, so its Service
held zero endpoints for 25 minutes. Both times the blast radius was every route in
the process, and both times it could not recover once IAM came back — the dial had
already happened, before any subsystem mounted, and the only way forward was a
restart the crashloop kept undoing.

Three behaviours are possible and only one is right. Serving without a cert is
fail-open. Dying without a cert makes identity a hard, boot-ordered dependency of
everything. So: resolve LAZILY, retry the failure, and fail CLOSED at the request.

Unresolved is NOT "no principal" — that is precisely the 401-everything fail-open,
and at a token-parsing call site the two are indistinguishable. It is 503 at the
door, in ONE filter ahead of every filter that reads a token, so no downstream code
ever has to tell them apart. 503 rather than 401 because 401 blames the caller for
our outage, invites a client to discard a good token, and reads as ordinary traffic
on every dashboard.

The retry is rate-limited to one attempt per 5s per process. Without that, every
request during an IAM outage becomes an IAM request, and the recovering service is
met with the full inbound rate the moment it starts answering.

The security property is unchanged: a request is never served with authentication
silently switched off. What changes is the cost of an IAM blip — the requests
during it, and the first request after IAM returns re-establishes the cert with no
restart.

The fail-closed tests move with the code rather than being deleted; their fixtures
are responses captured from the live IAM and are still the point. The one asserting
a panic is replaced by one asserting the failure is REPORTED and stays reported —
what must hold is that it is never swallowed, not that it is fatal.

Mutation-checked: failing open, dropping the rate limit, re-fetching a resolved
cert, and answering 401 instead of 503 each fail a test. The memoization test did
NOT fail at first — the retry window was answering for it — so it now asserts on
the far side of that window.

Co-authored-by: Hanzo Dev <[email protected]>
2026-08-01 23:13:24 -07:00

127 lines
4.7 KiB
Go

// Copyright 2026 Hanzo AI Inc. All Rights Reserved.
package object
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
// A deployment that names no IAM is deliberately running without auth, and
// resolves cleanly. It is the ONLY path that leaves authentication unconfigured,
// so it is pinned separately from every failure below.
func TestAuthReady_NoIAMConfigured(t *testing.T) {
ResetAuthReady()
t.Setenv("IAM_URL", "")
if err := AuthReady(); err != nil {
t.Fatalf("a deployment with no IAM_URL must resolve cleanly, got %v", err)
}
}
// IAM unreachable does NOT end the process and does NOT resolve. It reports the
// failure, which the door turns into 503 — never into "no principal", which is the
// 401-everything fail-open this replaced.
func TestAuthReady_UnreachableIAMFailsClosedWithoutPanicking(t *testing.T) {
ResetAuthReady()
// A port nothing listens on: the same connection-refused the two production
// outages produced.
t.Setenv("IAM_URL", "http://127.0.0.1:1")
t.Setenv("IAM_APP_NAME", "hanzo-cloud")
err := AuthReady()
if err == nil {
t.Fatal("an unreachable IAM resolved successfully — auth would be silently off")
}
if !strings.Contains(err.Error(), "hanzo-cloud") {
t.Errorf("the failure must name the application it could not read: %v", err)
}
}
// The retry window holds: a second call inside it reuses the first failure instead
// of dialling again, so an IAM outage does not turn every inbound request into an
// IAM request and meet the recovering service with the full inbound rate.
func TestAuthReady_RetryIsRateLimited(t *testing.T) {
ResetAuthReady()
t.Setenv("IAM_URL", "http://127.0.0.1:1")
t.Setenv("IAM_APP_NAME", "hanzo-cloud")
first := AuthReady()
if first == nil {
t.Fatal("expected a failure")
}
got := AuthAttempts()
for range 50 {
if err := AuthReady(); err == nil {
t.Fatal("a failed resolution must keep failing until it succeeds")
}
}
if AuthAttempts() != got {
t.Fatalf("attempts went %d → %d inside the retry window — 50 requests became %d IAM calls",
got, AuthAttempts(), AuthAttempts()-got)
}
}
// A resolvable IAM establishes the cert, and once established it is never fetched
// again — the hot auth path must not carry an IAM round trip per request.
//
// It also proves RECOVERY, which is the whole point of resolving lazily: this
// process began life unable to authenticate anything and needs no restart to start.
func TestAuthReady_ResolvesAndThenStopsFetching(t *testing.T) {
ResetAuthReady()
var calls int
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls++
w.Header().Set("Content-Type", "application/json")
switch {
case strings.Contains(r.URL.Path, "get-application"):
_, _ = w.Write([]byte(`{"status":"ok","data":{"name":"hanzo-cloud","cert":"cert-hanzo"}}`))
case strings.Contains(r.URL.Path, "get-cert"):
_, _ = w.Write([]byte(`{"status":"ok","data":{"name":"cert-hanzo","certificate":"-----BEGIN CERTIFICATE-----\nnot-a-real-cert\n-----END CERTIFICATE-----"}}`))
default:
_, _ = w.Write([]byte(`{"status":"error","msg":"unexpected path ` + r.URL.Path + `"}`))
}
}))
defer srv.Close()
// Start from the failure the outages produced, so what is measured is a process
// that could not authenticate and then could — with no restart in between.
t.Setenv("IAM_URL", "http://127.0.0.1:1")
t.Setenv("IAM_APP_NAME", "hanzo-cloud")
if AuthReady() == nil {
t.Fatal("expected the unreachable endpoint to fail")
}
t.Setenv("IAM_URL", srv.URL)
// Past the retry window, so the next call really re-attempts.
waitOutRetryWindow()
if err := AuthReady(); err != nil {
t.Fatalf("IAM came back and the process did not recover: %v", err)
}
after := calls
for range 20 {
if err := AuthReady(); err != nil {
t.Fatalf("a resolved cert stopped being resolved: %v", err)
}
}
// Past the retry window, because inside it EVERY answer is cached — a resolved
// cert and an unresolved one are indistinguishable there. Only a call on the far
// side of the window can tell "established, never fetch again" from "rate-limited
// for now", and without that this asserted the rate limiter twice and the
// memoization not at all.
waitOutRetryWindow()
if err := AuthReady(); err != nil {
t.Fatalf("a resolved cert stopped being resolved after the retry window: %v", err)
}
if calls != after {
t.Fatalf("a resolved cert was re-fetched %d time(s) — the hot auth path must carry no IAM round trip", calls-after)
}
}
// waitOutRetryWindow sleeps past retryAfter. It is the one place the test suite
// spends real time, so it is named rather than repeated.
func waitOutRetryWindow() { time.Sleep(retryAfter + 50*time.Millisecond) }