mirror of
https://github.com/hanzoai/ai.git
synced 2026-11-21 02:05:26 +00:00
The signing cert every bearer is validated against was fetched in a package init(), over the network, and the process panicked when it could not be read. That was the right answer to the fail-open it replaced — warn-and-serve left the service Running, green, and answering 401 to everything including free routes for a day — and the wrong answer to everything else. It has now taken production down twice. 2026-07-27: a one-word manifest typo killed IAM's pods. 2026-08-02: an IAM build answered no /healthz, so its Service held zero endpoints for 25 minutes. Both times the blast radius was every route in the process, and both times it could not recover once IAM came back — the dial had already happened, before any subsystem mounted, and the only way forward was a restart the crashloop kept undoing. Three behaviours are possible and only one is right. Serving without a cert is fail-open. Dying without a cert makes identity a hard, boot-ordered dependency of everything. So: resolve LAZILY, retry the failure, and fail CLOSED at the request. Unresolved is NOT "no principal" — that is precisely the 401-everything fail-open, and at a token-parsing call site the two are indistinguishable. It is 503 at the door, in ONE filter ahead of every filter that reads a token, so no downstream code ever has to tell them apart. 503 rather than 401 because 401 blames the caller for our outage, invites a client to discard a good token, and reads as ordinary traffic on every dashboard. The retry is rate-limited to one attempt per 5s per process. Without that, every request during an IAM outage becomes an IAM request, and the recovering service is met with the full inbound rate the moment it starts answering. The security property is unchanged: a request is never served with authentication silently switched off. What changes is the cost of an IAM blip — the requests during it, and the first request after IAM returns re-establishes the cert with no restart. The fail-closed tests move with the code rather than being deleted; their fixtures are responses captured from the live IAM and are still the point. The one asserting a panic is replaced by one asserting the failure is REPORTED and stays reported — what must hold is that it is never swallowed, not that it is fatal. Mutation-checked: failing open, dropping the rate limit, re-fetching a resolved cert, and answering 401 instead of 503 each fail a test. The memoization test did NOT fail at first — the retry window was answering for it — so it now asserts on the far side of that window. Co-authored-by: Hanzo Dev <[email protected]>
141 lines
6.2 KiB
Go
141 lines
6.2 KiB
Go
package object
|
|
|
|
import (
|
|
"bytes"
|
|
"io"
|
|
"net/http"
|
|
"strings"
|
|
"testing"
|
|
|
|
iam "github.com/hanzoai/ai/internal/iam"
|
|
)
|
|
|
|
// This file proves the auth boundary fails CLOSED, against responses the live IAM
|
|
// actually emitted.
|
|
//
|
|
// THE BUG THIS LOCKS OUT. Resolution used to warn and return on any failure to read
|
|
// the IAM application or its signing cert. With no cert the process still served,
|
|
// but every bearer failed to validate, so every authenticated route answered 401 —
|
|
// including free ones like /v1/models. The pod stayed Running and passed its
|
|
// probes, so no crashloop, no failed rollout, nothing caught it.
|
|
//
|
|
// The cure was a panic in an init(), which traded that for an outage of everything
|
|
// in the process whenever IAM blinked (twice in production — see auth_config.go).
|
|
// Resolution is now lazy and retrying, and the door answers 503. What must NOT
|
|
// change, and is what this file holds, is that none of these bodies ever resolves.
|
|
//
|
|
// WHY THESE FIXTURES ARE REAL. Each body below is the VERBATIM response the live
|
|
// IAM emitted, captured 2026-07-27 against the in-cluster service:
|
|
//
|
|
// kubectl -n hanzo port-forward svc/iam 18090:80
|
|
// curl -u "hanzo-cloud:$IAM_CLIENT_SECRET" \
|
|
// 'http://127.0.0.1:18090/v1/iam/get-application?id=admin/hanzo-cloud'
|
|
// -> {"status":403,"error":"forbidden"}
|
|
// curl 'http://127.0.0.1:18090/v1/iam/get-application?id=admin/hanzo-cloud'
|
|
// -> {"status":401,"error":"authentication required"}
|
|
// curl 'http://127.0.0.1:18090/v1/iam/get-account'
|
|
// -> {"status":"error","msg":"please sign in first"}
|
|
//
|
|
// They are recorded ground truth, not our idea of what IAM returns — which matters,
|
|
// because our idea of what IAM returns is exactly what was wrong. Note the first two
|
|
// carry `status` as a NUMBER while the third carries it as a STRING: IAM serves two
|
|
// different envelopes on one surface (the zip framework's error body from the authz
|
|
// Guard, and the string-status body from the handler), and the client's
|
|
// Response.Status is a string. That type flip is what produced
|
|
//
|
|
// json: cannot unmarshal number into Go struct field Response.status of type string
|
|
//
|
|
// If IAM's envelope is unified later, these fixtures should be re-captured from the
|
|
// live service rather than edited to match a new assumption.
|
|
|
|
// liveIAMBodies are the captured responses, keyed by what produced them.
|
|
var liveIAMBodies = map[string]string{
|
|
"authz Guard forbade the read (the outage)": `{"status":403,"error":"forbidden"}`,
|
|
"unauthenticated read": `{"status":401,"error":"authentication required"}`,
|
|
"string-status handler error": `{"status":"error","msg":"please sign in first"}`,
|
|
"empty body": ``,
|
|
"HTML error page from a proxy": `<html><body>502 Bad Gateway</body></html>`,
|
|
}
|
|
|
|
// replay is an iam.HttpClient that answers every request with one fixed body.
|
|
type replay struct{ body string }
|
|
|
|
func (r replay) Do(*http.Request) (*http.Response, error) {
|
|
return &http.Response{
|
|
StatusCode: http.StatusOK, // the envelope, not the transport, carries IAM's verdict
|
|
Body: io.NopCloser(bytes.NewReader([]byte(r.body))),
|
|
Header: http.Header{"Content-Type": []string{"application/json"}},
|
|
}, nil
|
|
}
|
|
|
|
// TestInitAuthConfigFailsClosed proves that every way the IAM application read can
|
|
// fail produces an ERROR rather than a silently unconfigured auth boundary. Before
|
|
// the fix each of these returned cleanly and the process served with auth off.
|
|
func TestResolveAuthConfigFailsClosed(t *testing.T) {
|
|
t.Setenv("IAM_URL", "http://iam.hanzo.svc")
|
|
t.Setenv("IAM_CLIENT_ID", "hanzo-cloud")
|
|
t.Setenv("IAM_CLIENT_SECRET", "test-secret")
|
|
t.Setenv("IAM_ORG", "hanzo")
|
|
t.Setenv("IAM_APP_NAME", "hanzo-cloud")
|
|
|
|
for name, body := range liveIAMBodies {
|
|
t.Run(name, func(t *testing.T) {
|
|
iam.SetHttpClient(replay{body: body})
|
|
t.Cleanup(func() { iam.SetHttpClient(&http.Client{}) })
|
|
|
|
ResetAuthReady()
|
|
err := resolveAuthConfig()
|
|
if err == nil {
|
|
t.Fatalf("resolveAuthConfig returned nil for %s (body %q) — "+
|
|
"auth would be silently disabled and every bearer would 401", name, body)
|
|
}
|
|
// The operator has to be able to act on this without a debugger.
|
|
if !strings.Contains(err.Error(), "hanzo-cloud") {
|
|
t.Errorf("error must name the application it could not establish, got: %v", err)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
// AuthReady REPORTS the failure rather than ending the process, and keeps
|
|
// reporting it — so the caller (the door, which answers 503) decides what an
|
|
// unreachable identity costs.
|
|
//
|
|
// This replaces a test that asserted a panic. The panic was the right answer to the
|
|
// fail-open and the wrong answer to everything else: it ran in an init(), before any
|
|
// subsystem mounted, so a momentary IAM outage killed the whole process and it did
|
|
// not come back on its own once IAM did. What must hold is that the failure is never
|
|
// swallowed — not that it is fatal.
|
|
func TestAuthReadyReportsRatherThanEndingTheProcess(t *testing.T) {
|
|
ResetAuthReady()
|
|
t.Setenv("IAM_URL", "http://iam.hanzo.svc")
|
|
t.Setenv("IAM_APP_NAME", "hanzo-cloud")
|
|
iam.SetHttpClient(replay{body: liveIAMBodies["authz Guard forbade the read (the outage)"]})
|
|
t.Cleanup(func() { iam.SetHttpClient(&http.Client{}) })
|
|
|
|
err := AuthReady()
|
|
if err == nil {
|
|
t.Fatal("AuthReady resolved on the outage body — the process would serve with auth off")
|
|
}
|
|
if !strings.Contains(err.Error(), "hanzo-cloud") {
|
|
t.Errorf("the failure must name the application it could not establish, got: %v", err)
|
|
}
|
|
// And it stays unresolved: a later caller must not be told everything is fine
|
|
// because an earlier one already asked.
|
|
if AuthReady() == nil {
|
|
t.Fatal("a second caller was told auth was ready while the cert is still unestablished")
|
|
}
|
|
}
|
|
|
|
// TestInitAuthConfigAllowsNoIAM keeps the legitimate no-auth deployment (dev, tests,
|
|
// any build with no IAM_URL) booting. This is the ONLY path that may leave auth
|
|
// unconfigured, and it is chosen explicitly by configuration rather than reached by
|
|
// swallowing an error.
|
|
func TestResolveAuthConfigAllowsNoIAM(t *testing.T) {
|
|
ResetAuthReady()
|
|
t.Setenv("IAM_URL", "")
|
|
if err := resolveAuthConfig(); err != nil {
|
|
t.Fatalf("no IAM_URL must boot cleanly (deliberate no-auth deployment), got: %v", err)
|
|
}
|
|
}
|