gateway/LLM.md

LLM.md - Hanzo Gateway

Overview

Go module: github.com/hanzoai/gateway

Tech Stack

• Language: Go

Build & Run

go build ./...                    # default — Lura-free (NOT what ships)
go build -tags legacy ./...       # legacy  — full Lura engine; SHIPS
go test ./...                     # unit packages only
go test -tags legacy ./...        # + the tests/ integration harness

The legacy build is what ships. Makefile sets BUILD_TAGS ?= legacy and the Dockerfile runs make build with no override, so ghcr.io/hanzoai/gateway is the Lura engine, and CMD ["run", "-c", ...] is the legacy cobra subcommand that only exists under that tag. The default build currently serves only /healthz and 404s model calls until the HIP-0110 ZAP relay backends are live — see the rationale above BUILD_TAGS in the Makefile. Upstream Lura security patches therefore still matter to production; they are not quarantined away.

Local verification must mirror the Dockerfile (CGO_ENABLED=0) and bypass the repo-adjacent ~/work/hanzo/go.work, which does not list this module:

GOWORK=off CGO_ENABLED=0 go build ./... && GOWORK=off CGO_ENABLED=0 go build -tags legacy ./...

No host in the import graph — MountDeps, not cloud.Deps

Mount took cloud.Deps and read three of its 24 fields (Logger, Brand, Domain). It now takes MountDeps, declared in mount.go with exactly those three, and github.com/hanzoai/cloud is gone from go.mod — not merely unimported, absent.

The rule, and it is general: a subsystem imports zip, never its host. hanzoai/cloud ships two editions under one module path, so cloud.Deps denotes a different type in each; a package that names it can be composed by exactly one of them, decided by whichever edition is in the build. Naming the three values instead turns the arrow around — the host knows the gateway, the gateway knows zip and nothing of whoever runs it. MountDeps joins RouterDeps (BuildApp) and RelayDeps (RegisterRelay): one narrow shape per entrypoint, all declared here.

Mount stays a mount — do not "convert" it to App() (*zip.App, error). That conversion is right for a subsystem that owns a prefix and contributes routes (see hanzoai/ai). This one contributes a global middleware chain, and a child App's Use-chain is scoped to that child's own subtree under zip's snapshot semantics — verified, not assumed. A gateway that returned its own App would guard its two probes and leave every sibling subsystem on the host unguarded: the boundary still present, still passing its tests, and no longer on the path. TestMount_GatesAreOnTheChainMountInstalls is what catches the half of that a compiler will not.

Dropped alongside: five inert replace directives (thrift, jaeger-client-go, abbot/go-http-auth, mailgun/minheap, mattn/go-sqlite3) whose comment attributed them to cloud's replace set. None appeared in go.sum before or after; deleting them leaves go.sum byte-identical.

Framework Ownership — Lura is legacy-gated (#29)

The gateway owns its module path (github.com/hanzoai/gateway/v2) and its default build is Lura-free. The upstream Lura framework (github.com/luraproject/lura/v2 + github.com/krakend/*) is quarantined behind the legacy build tag and never reaches go build ./....

It DOES reach the shipping ghcr.io/hanzoai/gateway image, because that image is built -tags legacy (BUILD_TAGS ?= legacy). The quarantine is a source-layering boundary, not a production-exposure boundary — do not read it as "the legacy engine is not in prod". Verify:

go list -deps ./cmd/gateway              | grep -c 'krakend\|lura'   # -> 0
go list -tags legacy -deps ./cmd/gateway | grep -c 'krakend\|lura'   # -> 105

Build	Command	Lura pkgs	Contents	Ships?
default	go build ./...	0	HIP-0110 ZAP relay + pure reverse-proxy router (routes.go, mount.go, build_app.go, gate.go, auth_middleware.go)	no — /healthz only until Phase C
legacy	go build -tags legacy ./...	105	full Lura gin engine (legacy_engine.go + factories)	yes — this is the image

Ownership boundary:

Dependency	Module path	Ownership	Reaches default build?
gateway	github.com/hanzoai/gateway/v2	fork — ours, own path	yes (whole module)
Lura core	github.com/luraproject/lura/v2	upstream, pinned	no — legacy-only
krakend-* (24 pkgs)	github.com/krakend/*	upstream, pinned	no — legacy-only
ZAP edge	github.com/zap-proto/http, github.com/luxfi/zap	ours/fork	yes (default relay)

legacy-gated files (every lura/krakend importer): backend_factory, base_ha_backend, base_network_backend, encoding, executor, handler_factory, plugin, proxy_factory, rebrand, sd, zap_backend, zaphttp_listener, legacy_engine (+ their _test.go), the tests/ Lura integration harness, cmd/gateway-integration, and cmd/gateway/{main_legacy,rebrand}.go. The default routing/auth/mount/ZAP-relay surface already imports zero krakend/lura.

Branding: user-visible upstream-brand strings — HTTP (X-KRAKEND→X-Powered-By, backend User-Agent→hanzoai/gateway) in rebrand.go, and the cobra CLI tree in cmd/gateway/rebrand.go — are scrubbed only in the legacy engine. The default build emits no upstream-brand strings because it contains none of that code.

Phase-C end state: when the cloud binary's gateway.Mount is the sole edge, the entire legacy set is deleted and the upstream Lura dependency drops out of go.mod. Until then it stays compilable under -tags legacy for rollout safety. Forwards-only: never add a lura/krakend import to a non-legacy file.

Upstream sync — krakend-ce v2.12.x → v2.13.8 (2026-07-25)

This fork has no shared git history with krakend/krakend-ce (it began as a squashed import), so there is no merge-base and git merge is not available. Syncing is a deliberate content-level port: diff our copies of the seven derived service-assembly files against an upstream tag, take the delta, keep ours.

Base identified by content-diffing every upstream tag: our derived sources matched v2.12.x (137 residual lines = our own deltas). Now at v2.13.8.

krakend-ce is a thin assembly repo — the v2.12.1→v2.13.8 source delta is only backend_factory.go (17 lines), executor.go (3), Makefile, and tests/. The substance of an upgrade is the dependency graph.

Taken:

Change	Why
krakend-xml v2.2.0 → v2.2.2	security — exponential-cost XML rendering (algorithmic-complexity DoS); drops clbanning/mxj
go-jose/v3 v3.0.4 → v3.0.5 (transitive)	security — JOSE lib under krakend-jose
krakend-jose v2.10.0 → v2.12.3	JWT validation path
lura v2.12.1 → v2.14.1	core framework; real tag, not upstream's v2.14.2-0.2026… pseudo-version
backend_factory.go ctx propagation	bug — app context never reached client plugins, so they missed cancellation/shutdown (upstream 16a23d2)
krakend-circuitbreaker v2 → v3	import path change; pulls sony/gobreaker/v2
krakend-pubsub v2.2.0 → v2.3.1 + drop pubsub.OpenCensusViews	forced: gocloud.dev v0.45.0 removed that symbol, breaking pubsub v2.2.0
tests/integration.go → santhosh-tekuri/jsonschema/v6	drops unmaintained xeipuuv/gojsonschema
expected.Body != "" → != nil	bug — Body is interface{}, so a nil body compared true against "" and wrongly asserted. Fixture integration_no_body.json added as the guard
botdetector, cel, cobra, cors, flexibleconfig, httpsecure, jsonschema, lua, martian, metrics, opencensus, ratelimit, go-auth0	routine patch/minor bumps

Deliberately NOT taken (re-adding any of these is a regression):

• krakend-usage — the upstream phone-home telemetry. startReporter is a no-op here on purpose. krakend-audit stays indirect (it exists only to feed it).
• krakend-otel + the jaeger / ocagent / stackdriver OpenCensus exporters — all ship telemetry over gRPC; Hanzo speaks ZAP/HTTP/WS, never gRPC.
• go-contrib/uuid — only used by the usage reporter.
• Upstream's cors.NewRunServerWithLogger wrapper — this fork serves its own CORS preflight via hostProxyMiddleware; taking it double-sets headers.
• router_engine.go, krakend.json, upstream Makefile/README/SECURITY.md branding — superseded by legacy_engine.go + configs/{example,lux}.

Load-bearing upstream literals — never rebrand these. Everything else in this repo is de-branded; these four are behaviour, not branding:

Literal	Where	Why it is pinned
core.KrakendVersion, core.KrakendUserAgent	rebrand.go, Makefile ldflags	exported lura symbols; renaming breaks the build. rebrand.go overwrites their values with the Hanzo brand — that is the correct seam
KRAKEND_ env prefix (e.g. KRAKEND_PORT)	config_loader_test.go	the koanf parser hardcodes the prefix as a const; not configurable
X-KRAKEND, X-Krakend-Completed	rebrand.go, production_headers_test.go	compile-time consts in lura/core. rebrand.go deletes them off every response and the tests assert they never leak — the literal exists here in order to be removed
github.com/devopsfaith/krakend* extra_config keys	the aliases map in cmd/gateway/main_legacy.go	these are the canonical namespaces each component registers under; aliases maps the friendly name (qos/ratelimit/router) onto them via config.ExtraConfigAlias. Rename the canonical side and the component stops finding its config — silently. Note configs/*/gateway.json uses only the friendly names, so the configs themselves are already brand-free. auth/validator and auth/signer are deliberately NOT in the table — see "Edge Route Auth Classes"

Two entries that used to be on this list are gone because they were not actually pinned: the auth realm equivalent (zaphttp_listener.go's listener env var) is the fork's own knob read by direct os.Getenv, not koanf, so it is now GATEWAY_ZAP_LISTEN in line with GATEWAY_LISTEN / GATEWAY_SHUTDOWN_TIMEOUT; and the negative-assertion brand lists in brand_test.go / production_headers_test.go keep the upstream token on purpose — they are the regression guard that fails if the brand ever leaks into a Server header.

Module paths — conversion BLOCKED on missing fork repos

The standing rule is that a hanzoai fork declares its OWN module path and is required directly, never via replace. The ingress fork now satisfies this (github.com/hanzoai/yaegi, github.com/hanzoai/grpc-web). The gateway does not, and cannot yet: the 33 github.com/krakend/* modules in go.mod resolve to upstream, not to hanzoai forks, and the corresponding fork repos do not exist. Only hanzoai/krakend-otel exists — and its replace was dead (go mod why reports the main module does not need it), so it has been deleted rather than repointed.

Converting requires, per module, in dependency order: fork the upstream repo at the pinned version, rewrite its module path, fix its self-imports and its cross-references to the other forks, tag a patch bump, then repoint go.mod here. Until those repos exist the github.com/krakend/* import lines and the go.mod/go.sum require lines stay — a partial conversion would not build. The full list is the require block at the top of go.mod.

github.com/vulcand/oxy/v2 was repointed: it used to replace to the upstream edge-router fork and now resolves to github.com/hanzoai/oxy/v2 at the identical commit ingress pins.

Module path stays github.com/hanzoai/gateway/v2 — already v2 from the krakend-ce v2 lineage, predating the "stay v1.x" rule. Do not "fix" it; a path change would break every consumer for zero gain.

Structure

gateway/
  Dockerfile
  LICENSE
  LLM.md
  Makefile
  README.md
  SECURITY.md
  auth_middleware.go
  auth_middleware_test.go
  backend_factory.go
  builder/
  cmd/
  configs/
  deps.sh
  encoding.go
  executor.go

Key Files

• README.md -- Project documentation
• go.mod -- Go module definition
• Makefile -- Build automation
• Dockerfile -- Container build

JWT Audience Allowlist (edge auth)

edge.Verifier.VerifyRaw validates iss (strict, env AUTH_ISSUER, prod https://iam.hanzo.ai) and aud against an allowlist (OR semantics — a token passes if its aud matches ANY entry). IAM stamps user tokens with aud = <client_id> (the seeded app name: hanzo-app, hanzo-console, hanzo-chat, hanzo-id, …), never the gateway origin — so the prior single fixed audience (https://api.hanzo.ai) rejected EVERY normal user JWT (cowork AI 401, user billing 401).

• Single source of truth: hanzoai/authz/edge.DefaultAudiences(the known user-facing client_ids +https://api.hanzo.ai) and token.AudiencesFromEnv(). Shared by the gin/Lura middleware (auth_middleware.go), the relay gate (gate.go) and the unified-binary mount (mount.go`). One implementation, three callers.
• Override entirely with GATEWAY_ALLOWED_AUDIENCES (comma-separated). Legacy AUTH_AUDIENCE / JWT_AUDIENCE, when set, are folded IN (widen, never narrow — a live env pinned to AUTH_AUDIENCE=https://api.hanzo.ai keeps that value in an already-inclusive set rather than collapsing to one entry).
• The allowlist is never empty by construction, so the audience check is ALWAYS enforced. aud outside the set fails; a missing/empty iss fails.
• Forwards-only: append new client_ids to DefaultAudiences, never remove.
• Regression: token/token.go (aud=hanzo-app/hanzo-chat PASS, aud=evil FAIL, wrong issuer FAIL) + TestJWTAuth_RejectsWrongAudience.

Identity Headers (Trust Boundary)

Gateway is the only authority that may emit identity headers downstream. On every request the middleware unconditionally Header.Dels every header it writes (see stripIdentityHeaders in auth_middleware.go) BEFORE any bypass path runs (public hosts, public paths, API keys, no-token pass-through). After JWT validation, the gateway re-injects the headers from the validated claims.

Header	Source claim	Type
X-User-Id	sub	string
X-Org-Id	owner	string (org slug)
X-Roles	roles	comma-joined role names
X-User-Email	email	string
X-Phone-Number	phone_number/phone	E.164 string
X-User-IsAdmin	orgs[0] (Claims.Sudo)	"true" iff SuperAdmin
X-User-Permissions	permissions+isAdmin	base-10 int64 bit.Field

X-User-Permissions is gateway-derived from the JWT permissions claim (any of: numeric bit-field, []string of names, []{"name":"..."} upstream-IAM permission objects). Names are looked up in permissionBits (see auth_middleware.go); unknown names are dropped silently (forwards-compatible with new IAM permissions). A SuperAdmin (Claims.Sudo) also gets permission.Admin, and isAdmin (an org's own admin) gets permission.Live only; together 16 | 4 = 20. The header is omitted when the resulting bit-field is zero — commerce treats absent and "0" identically (bit.Field(0), fail-closed, see commerce/middleware/iammiddleware).

The single source of truth for bit positions is commerce/util/permission/permission.go — permissionBits here mirrors the iota order there. Forwards-only: never re-number, only append.

Strip-list ⊇ write-list contract: every header in authz.Headers MUST be stripped on ingress. The contract is enforced by TestStripList_CoversAllWrittenHeaders and TestStripIdentityHeaders_AllVariants — these MUST pass before merge.

Red P0-1 closed (2026-04-27): prior to this fix, the gateway neither stripped nor wrote X-User-Permissions, allowing curl -H "X-User-Permissions: 16" to grant Admin in commerce. See auth_middleware_security_test.go Test 21.

Edge Route Auth Classes + Billing (must-gate hardening, 2026-06-27)

ONE validator (HIP-0110 Phase C, edge de-braid). The endpoint pipeline used to carry 216 per-endpoint auth/validator blocks — a SECOND JWT validation, in a second crypto library (krakend-jose → go-jose v3), against a second key source (http://iam.hanzo.svc.cluster.local/..., cleartext, disable_jwk_security: true), running after NewAuthMiddleware had already validated the same token against https://hanzo.id over TLS. They are gone. What replaced them:

• The one validator is the trust boundary the engine installs ahead of routing (authpolicy.go → authGate.admit, shared by the gin edge, the zip mount and the ZAP relay). It verifies the signature, the issuer and the ANY-of audience allowlist, strips what the client claimed and writes what the token proved.
• An endpoint says whether it NEEDS an identity with "auth/public": true in its extra_config. Absent means required. requireIdentity (handler_factory.go) reads the identity the gate wrote (X-User-Id) and 401s when there is none — no key, no parse, no second JWKS.
• The polarity is fail-secure: an endpoint added to gateway.json without thinking about auth is gated. Under the old shape, forgetting the validator block silently opened the route.
• The embedded schema (cmd/gateway/schema/schema.json) REFUSES auth/validator and auth/signer at an endpoint, and the alias table in main_legacy.go no longer maps them — so a re-introduced second validator fails validation rather than being silently ignored.
• Pins: TestShippingConfig_NoEndpointValidator, TestHanzoConfig_PublicSurfaceUnchanged (the 21 public endpoints, by name), TestLuxConfig_AllPublic, TestRequireIdentity_EndToEnd.

Three divergences the second validator had, all resolved by deleting it: propagate_claims wrote X-Org-Id from the raw owner claim (the APPLICATION's org) over the effective org the gate had resolved against the token's signed membership; it 401'd hk-/sk- API keys the gate deliberately passes to the backend that owns them; and it could never carry an audience at all (ALL-of vs the edge's ANY-of). X-Roles was its last writer — a RETIRED header (authz.Retired) the gate strips and nothing writes — so it is also gone from every input_headers list.

configs/lux/gateway.json states all 14 of its endpoints: 11 public and 3 gated. The Lux edge fronts blockchain RPC and gated nothing before, so the RPC routes are public. The three that reach the node itself — POST /v1/admin, POST /v1/keystore, GET /v1/metrics — were open by omission and now require an identity.

The Hanzo route table (universe infra/k8s/gateway/gateway.json) falls in three auth classes:

• PUBLIC (13): health/discovery/catalog (/, /health, /v1/*/health, GET /v1/ai/providers, GET /v1/ai/providers/{owner}/{name}, /v1/pricing-policy, /v1/models, /v1/analytics/heartbeat, /v1/pubsub/healthz, /bot/health). The two provider reads self-protect at the backend (session auth); GET /v1/ai/providers/global does NOT and is must-gate. These carry "auth/public": true.
• AI / API-KEY (8): /v1/chat, /v1/chat/completions, /v1/completions, /v1/messages, /ai/{path}, /v1/ai/{path}. NO IAM-JWT validator — these use opaque API keys (hk-/sk-) and are billed per-token by cloud. These carry "auth/public": true: the key is verified at the backend, so an IAM-identity requirement at the edge would reject every API-key call.
• MUST-GATE (29, IAM-JWT): /cloud/{path}, /v1/cloud/{path} (both GET/POST/PATCH/DELETE), /v1/commerce/{path}, /v1/tasks/{path}, /v1/insights/{path}, /v1/o11y/{path}, /v1/mpc/{path}, /v1/evals/{path}, /v1/licensing/{path}, /v1/product/{path}, /v1/provisioning/{path}, /v1/ml/*, /v1/train/*, GET /v1/ai/providers/global, POST /v1/ai/providers, PATCH /v1/ai/providers/{owner}/{name}. These carry NO auth/public key, which is the whole declaration: requireIdentity refuses them without the identity the one gate wrote. Result: 401 at the edge without a valid JWT.

GOTCHA — hostProxyMiddleware reaches no endpoint at all: api.hanzo.ai and api.cloud.hanzo.ai are in apiCloudHosts (routes.go), so hostProxyMiddleware reverse-proxies their WHOLE surface to cloud and aborts — the gin router never runs, and no endpoint in gateway.json is reached on the canonical API host. NewAuthMiddleware still runs (it is registered ahead of the host proxy), so identity is stripped and rewritten for those hosts; what does not run is the endpoint pipeline, including requireIdentity. Same for every host in the routes table whose prefix matches. The endpoint table is therefore the FALLBACK path — direct-to-service traffic and hosts the table does not claim — and the only place it decides anything. This is also why the 216 auth/validator blocks were never the API's auth: on the one host that carries the API, they never executed.

Billing — path-scoped balance gate (auth_middleware.go): the global NewAuthMiddleware enforces a commerce balance check ONLY when BILLING_ENABLED=true AND the request path matches a BILLING_PATHS prefix. This keeps billing OFF for the 171 AI/validated routes (AI bills per-token downstream) and public routes, ON for the metered must-gate surface. Key is the verified commerce org/sub user identity; per-org billing needs a commerce-side balance-model change first. Fail-closed: 402 on zero balance, 503 if commerce is unreachable. The config schema cannot express this in gateway.json (no-op encoding forbids a sequential balance pre-check), so it lives in the Go middleware.

Activation (NOT enabled by default — live is BILLING_ENABLED=false):

1. Provision COMMERCE_SERVICE_TOKEN from KMS (gateway-secrets). AuthConfig.Validate() makes it AND AUTH_BILLING_URL REQUIRED when BILLING_ENABLED=true: gateway.Mount refuses to start without them, and NewAuthMiddleware fails the metered surface closed (503) — never open. mount.go and DefaultAuthConfig read the SAME vars (AUTH_BILLING_URL / COMMERCE_SERVICE_TOKEN); no BILLING_URL/ BILLING_TOKEN drift.
2. Set BILLING_ENABLED=true and BILLING_PATHS=/v1/cloud/,/cloud/,/v1/tasks/,/v1/insights/,/v1/o11y/,/v1/mpc/,/v1/evals/,/v1/licensing/,/v1/product/,/v1/provisioning/,/v1/ai/providers (/v1/commerce is hard-excluded in code — billingPathMatch, segment boundary — so it is NEVER balance-gated regardless of BILLING_PATHS; a 402 on the funding surface would lock users out of adding funds). billingPathMatch keys on PATH ONLY, so /v1/ai/providers meters the ungated GET reads too — the provider mutations are now distinguished by METHOD (POST/PATCH), which the prefix list cannot express. Drop the entry if read-metering is unacceptable.
3. Confirm commerce balance granularity before enabling.

Deploy = update ConfigMap gateway-config from this file + rollout restart deploy gateway. NEVER make apply-hanzo (its manifest pins an older image tag and downgrades the running gateway — live is ahead of the manifest).

Edge auth — landmines

Audience is enforced ONLY at the Go edge, and there is nowhere else it could be. NewAuthMiddleware validates aud against an ANY-of allowlist (hanzoai/authz/edge.DefaultAudiences, go-jose v4 AnyAudience): a token passes when its single aud=<client_id>matches ANY entry. The config-declaredauth/validator(krakend-jose → go-auth0 → go-jose **v3**) validated audience with ALL-semantics — the token had to carry EVERY configuredaud, and IAM stamps ONE per token, so a multi-entry "audience"there 401'd EVERY user JWT. That is the landmine that made the second validator unfixable rather than merely redundant, and it is why the config could never state the audience it was supposedly enforcing. The validator is gone;TestShippingConfig_NoEndpointValidatorkeeps both it and any"audience"` key out of every config this repo carries.

X-Project-Id is stripped at the edge (edge.Strip): it is forgeable and written from no claim, so it can never reach a backend (cross-project IDOR). Write it here from a validated claim if/when IAM carries one — same pattern as X-Org-Id.

GET /v1/ai/providers/global is must-gate. It dumps the admin provider inventory (names, base URLs, masked secrets) and the backend does NOT self-protect it, so it carries no auth/public key and needs an IAM identity. Its read siblings GET /v1/ai/providers / GET /v1/ai/providers/{owner}/{name} self-protect at the backend ("Please sign in first") via session auth and are intentionally left ungated — a Bearer-only edge validator would break their cookie-auth callers. /v1/ai/providers/global is 4 segments and the member route is 5, so the ungated param route can never shadow it.

Boot refuses a config that leaves a route unsaid

authpolicy.go holds both halves of the policy: what a request must bring, and what the config declares. The config half is read once, in NewCmdExecutor, BEFORE anything listens.

An endpoint requires an IAM identity unless it declares "auth/public": true. Absent means required, so the config is where the OPEN surface is named — and the requirement is that EVERY route says which it is:

routes == open + gated     // serve
otherwise                  // log CRITICAL, exit 1

gated is any route that states a requirement: "auth/public": false, or any other key in the auth/ namespace. A config with no endpoints passes, because 0 == 0+0.

Completeness, not presence. Asking only whether the file mentions auth somewhere is satisfied by a config written against the older contract, which states a requirement on the routes it gated and says NOTHING on the routes it left open. That config boots, reports healthy, and every one of those open routes demands an identity nothing sends. Measured on canon's own hanzo config — 216 auth/validator keys and 21 routes unsaid — the presence check passes it and the 21 unsaid routes are the entire AI and catalog surface. The completeness check refuses it and names them:

[SERVICE: Auth] 21 of 237 endpoints state no credential policy: every route says
whether it needs an IAM identity, with "auth/public" true or false, and this
config leaves that unsaid on 21 of them.

One classified route out of sixty-three satisfied the presence check too.

The order is the whole control. The readiness probe on the deployed gateway is a TCP connect to :8080 — it reads no status and no body, so a process that has bound the port is Ready whatever it answers. Checked before the bind, a config the binary refuses stalls the rollout at one surged pod while the old ones keep serving. Checked after, the same config passes readiness and the rollout completes. TestBootBindsNothingWithoutAPolicy runs the shipping binary and POLLS the port while the process is alive — asking once, after it is gone, asks about a socket that is closed either way and answers "nothing is listening" for a binary that bound the port and then failed. TestWatchCatchesATransientBind holds the watcher to that.

tests/fixtures/policy/unstated.json is a gateway.json in the shape that is mounted in production — KrakenD v2.7 schema, endpoints carrying validation/cel and qos/ratelimit/router extra_config, nothing in the auth namespace. The accept cases are derived from that same file, so the only difference between a refused config and a served one is the statement itself.

The configs say what canon's validators said

The Hanzo route table is 21 open + 216 gated. The 216 are exactly the routes canon gated with auth/validator; the migration moved where that is written down, not which routes are which. That table is deployment config and lives in universe beside the deployment; this repo carries configs/example/gateway.json, held to the same policy check (TestPolicyAdmitsRepoConfigs, TestExampleConfig_PublicSurface). gateway check -c <file> runs the same check against any config before it is mounted.

configs/lux/gateway.json is 11 open + 3 gated. The gated three all reach the node rather than the chain — POST /v1/admin (admin API), POST /v1/keystore (key management), GET /v1/metrics (node internals) — and each was open by omission on canon. Writing a route down turns an omission into a decision, and none of the three was a decision anyone made: keystore is absent from every live ConfigMap, and nothing scrapes the gateway's routed /v1/metrics.

Watch the polarity while classifying. true is the answer that both makes the check pass and preserves whatever a route did before, so it is the answer the work pulls toward. It is also the answer that needs justifying: false costs a route nothing but a credential, true states that the route is meant to be reachable without one. Decide each route on what it reaches.

Deploy pre-flight — the ConfigMap and the image go together

The image carries no config. Every deployment mounts its route table at /etc/gateway/gateway.json, so one image serves every brand. hanzo's ConfigMap gateway-config is declared in universe (infra/k8s/gateway/gateway.json); it classifies all 237 routes, so it boots.

lux is the one that must be sequenced by hand. Its ConfigMap is declared in the universe repo and the image tag is pinned there too, and the repo config is NOT a drop-in for it: configs/lux/gateway.json has 14 endpoints to one backend, the declared ConfigMap has 63 to seven, and applying the repo copy would delete the exchange, explorer and indexer routes.

Before the image moves, a human classifies every endpoint in the declared ConfigMap, per namespace:

lux-mainnet   63 endpoints   currently 0 carrying any auth/ key
lux-testnet   61 endpoints   currently 0
lux-devnet    36 endpoints   currently 0

Each route gets "auth/public" — true when it is meant to be reachable without an IAM identity, false when it is not. With today's declarations the new binary REFUSES to boot, so the classification is a prerequisite and not a follow-up.

One apply, one namespace at a time. The old binary ignores auth/public entirely, so a ConfigMap applied ahead of the image is inert and safe, and rolling the image back without rolling the ConfigMap back is safe for the same reason. An image applied ahead of its ConfigMap stalls the rollout — verified against both shapes: an unclassified config and a partially classified one both exit 1 before binding. Verify by reading the pod log for [SERVICE: Auth] before the rollout is allowed to proceed to the next namespace.

One ingress (cmd/ingress deleted)

This repo used to ship a second reverse proxy: cmd/ingress, a hand-rolled host-based proxy built into the same image as a second binary, with its own 94-host table (configs/hanzo/ingress.json) and its own IAM gate on one host. It has been removed, along with k8s/hanzo-ingress/, the build-ingress and apply-ingress-hanzo Make targets, and the Dockerfile lines that baked it in.

Why, in order of how much it mattered:

• It collided with the real edge by name. hanzoai/ingress (the Traefik fork) declares hanzo-ingress in namespace hanzo; so did this repo, twice, with two different selectors and a DigitalOcean load balancer of the same name. make apply-ingress-hanzo also rewrote the ConfigMap ingress-config — which in declared state holds the Traefik static config — so running it replaced the live edge's configuration with a route table for a process that was not running.
• It was never deployed. Declared state runs Deployment/ingress on ghcr.io/hanzoai/ingress (digest-pinned). Nothing in universe or operator referenced configs/hanzo/ingress.json, /usr/bin/ingress or the hanzo-ingress workload from this repo, and no CI built it.
• Two edges meant two auth surfaces. Its gate was a second caller of hanzoai/authz/edge with its own env, on a host nothing else knew about.

k8s/hanzo/service.yaml also lost its hanzo-ingress LoadBalancer: it selected these gateway pods on :80/:443 and forwarded :443 to a plaintext :8080. HTTP reaches this gateway through the ingress (k8s/hanzo/ingress.yaml, ingressClassName: ingress). zap-ingress stays — different protocol, its own port, not a second door to the same one.

Before merging, the routes that lived only in configs/hanzo/ingress.json have to land in universe/infra/k8s/ingress/routes.yaml. The set is small and specific: the blockscout explorer hosts (explorer{,.testnet,.devnet}.lux.network, api-explore{,-test,-dev}.lux.network, explore-{test,dev}.lux.network), lux.exchange with its /api/v1 split, mpc.lux.network (a cert exists, no router), luxstat.us, megastore.lol + www., auth.lux.network, auth.pars.ai, and {oauth,sync,webhook}.platform.hanzo.ai. goproxy.hanzo.ai was the one host it gated; declared state routes it nowhere and the standing decision for the Go module proxy is no public ingress at all, so that is a decision to confirm rather than a route to port.

Paths are /v1, never /api

apiprefix_test.go (untagged, so every lane runs it) walks every Go string literal and fails on /api as a bare path segment or on one of our hosts; third-party URLs (gitlab.com/api/v4, a Jaeger :14268/api/traces) pass. routes.yaml still maps /api/ on hanzo.bot and hanzo.team — those backends serve /api themselves and move with their own repos. tests/fixtures/policy/unstated.json /api/v2/stats is the Lux indexer's own protocol.

Upstream Kinds

• default — legacy HTTP passthrough.

• zap — binary RPC transport (github.com/hanzoai/gateway/zap).

• base-network — shard-aware routing over hanzoai/base network services (ATS, BD, TA, IAM, KMS, AML). github.com/hanzoai/gateway/base-network.

• base_ha — leader-pin routing over hanzoai/base-ha clusters (BaseApp CRD). github.com/hanzoai/gateway/base_ha. Polls GET /_ha/leader on the service DNS every leader_poll_interval (default 1s), pins write methods (POST/PUT/PATCH/DELETE or X-Base-Writer: required) to the elected writer pod, round-robins reads via the enclosing host (K8s ClusterIP). 5s read-your-writes pin per client (X-Forwarded-For + X-Org-Id) after a write. One-retry cap on writer 5xx/connect-refused (no retry storm on OOM). Stale-cache fail-secure: lease-expired + poll-stale → 503 instead of targeting a dead pod.

  Minimal per-backend config (add to an endpoint in gateway.json):

  "backend": [{
    "url_pattern": "/v1/app/foo",
    "host": ["http://foo.hanzo.svc.cluster.local:8090"],
    "extra_config": {
      "github.com/hanzoai/gateway/base_ha": {
        "service_dns": "foo.hanzo.svc.cluster.local",
        "port": 8090,
        "leader_poll_interval": "1s",
        "read_your_writes_ttl": "5s"
      }
    }
  }]
  

  Prometheus metrics: gateway_base_ha_leader_polls_total, gateway_base_ha_leader_poll_errors_total, gateway_base_ha_leader_changes_total, gateway_base_ha_writer_failures_total, gateway_base_ha_writer_failures_fatal_total, gateway_base_ha_no_writer_total.

SuperAdmin — one predicate, authz.Claims.Sudo

Platform authority is a PERSON whose home org — orgs[0], the org IAM keeps the account in — is the reserved admin org, compared verbatim. Both transports ask exactly that: the HTTP edge renders X-User-IsAdmin through edge.Apply and the money Admin bit through moneyBits, and the ZAP relay sets Forward.IsAdmin and the same bits in gate.go. cmd/admin-api asks Claims.Sudo of a bearer and the server's isSuperAdmin verdict of a session (cloud /v1/ai/account).

None of these is SuperAdmin: an admin membership further down the set (authz's retired PlatformSudo granted on that, and so did this gateway until authz v1.10.47), the owner claim (the minting application's org), isAdmin (an org's own admin), an admin-org program or a token with no membership set, and an org named Admin/ADMIN. sudo_test.go and cmd/admin-api/sudo_test.go hold them. There is no IAM_ADMIN_ORG knob: the reserved org is IAM's constant (authz.AdminOrg).

cmd/admin-guard is deleted (a9188d8c). A Deployment admin-guard (ghcr.io/hanzoai/admin-guard:v0.1.10) still runs in ns hanzo, built from the deleted source, which judged SuperAdmin on the owner claim; the middlewares that call it (ci-guard, o11y-guard, both /__guard/verify-authn) are attached to no IngressRoute as of 2026-10-19.

waitlist-guard — approval forward-auth gate (cmd/waitlist-guard)

Same forward-auth mechanism as admin-guard but the OPPOSITE surface: it fronts the CONSUMER product hosts (console/billing/app/chat/team + gate api.hanzo.ai) and lets through only APPROVED identities (IAM User.IsApproved: fail-open on an absent approvalStatus property, gated only on an explicit "pending"; admins always approved). Verdicts: 204 allow · 302→WAITLIST_URL (unapproved browser) · 403 (unapproved API) · 302→IAM PKCE login (anonymous browser). Image ghcr.io/hanzoai/waitlist-guard. Approval is stamped at signup in iam object.AddUser (EVERY route: password/SSO/social/web3/email-code), so the gate is not bypassable via a non-password signup.

Hardening vs the admin-guard clone (Red rework):

• NO org pin. startLogin must NOT set organization=admin (admin-guard does — it needs the admin identity). Consumer users log in under their own org; pinning admin would break their login. SuperAdmins still resolve through Claims.Sudo.
• Fail-OPEN on IAM 5xx (GUARD_FAIL_OPEN_ON_IAM_ERROR, default true). iamGet is tri-state — iamOK / iamUnavailable (transport err or 5xx) / iamDenied (4xx). An authenticated caller (valid JWT or IAM session) whose approval lookup hits iamUnavailable is let THROUGH so an IAM blip never blackholes the money path; a 4xx never fails open; anonymous never fails open. The durable fast path is the HMAC-signed guard cookie (8h) — it short-circuits IAM entirely.
• Inbound identity strip. handleVerify runs edge.Strip (+ stripWaitlistHeaders) up front so a forged X-Org-Id/X-Waitlist-* can't be read by the guard. The AUTHORITATIVE upstream strip is the ingress headers middleware (must strip these before forwardAuth; the guard rewrites X-Org-Id via authResponseHeaders on allow).
• Forwarded-Host allowlist. safeHost/hostAllowed honor X-Forwarded-Host only for the cookie-domain suffix (or explicit GUARD_ALLOWED_HOSTS), else fall back to r.Host — a spoofed forwarded host can't steer redirect_uri/returnTo.

Config adds over admin-guard: WAITLIST_URL, GUARD_FAIL_OPEN_ON_IAM_ERROR, GUARD_ALLOWED_HOSTS, IAM_CLIENT_ID=hanzo-waitlist-guard. Secrets: waitlist-guard-secrets (GUARD_HMAC_KEY + IAM_CLIENT_SECRET). NetworkPolicy must pin the gated services (paas:3000 etc) to ingress-only so the guard can't be bypassed by hitting a Service ClusterIP directly. Tests: cmd/waitlist-guard/main_test.go (fail-open tri-state, inbound strip, host allowlist, no-org-pin, session forgery, approval predicate). Build via arcd BuildKit Job (cmd/waitlist-guard/Dockerfile), NOT GHA. Not wired to prod — cutover is a supervised one-host canary (see universe waitlist-guard docs).

Test workflow green (fix/gateway-test-ci)

.github/workflows/test.yml was red on every commit while Build and Deploy stayed green, because the test jobs ran go tooling directly on the hanzo-build-linux-amd64 runner without the environment the Dockerfile builder provides. Root causes and fixes:

• Module resolution: the test jobs mirror the Dockerfile module env (GONOSUMDB=github.com/hanzoai/*, GOPROXY=proxy,direct) plus a per-job git config --global url."https://x-access-token:${GH_PAT}@github.com/".insteadOf step — same GH_PAT the deploy Dockerfile secret uses, same pattern as hanzoai/ingress CI. github.com/hanzoai/* is the one namespace the public checksum database cannot cover; zap-proto/* and luxfi/* are public, so they resolve through the proxy and stay verified against sum.golang.org.
• make: command not found (127): the runner ships go (setup-go) but not make/wget. make build is replaced with a direct go build -tags legacy, reading VERSION from the Makefile and stamping lura/v2/core.KrakendVersion (the User-Agent the integration fixtures assert). The schema wget is dropped — gateway check runs fine with the empty embedded schema (main_legacy.go already falls back when schema/schema.json is absent).

Two tests were compile-/assert-broken and had been masked because go-tests never ran (it needs: go-vet, which died at module fetch):

• zaphttp_listener_test.go called the removed net/http Transport.RoundTrip; zap-proto/[email protected] is now a fasthttp-style client. Round-trip rewritten to Transport.Do(*fasthttp.Request, *fasthttp.Response). fasthttp promoted to a direct require in go.mod (it already backed fasthttpadaptor in zaphttp_listener.go); go.sum untouched.
• config_loader_test.go asserted a GATEWAY_PORT koanf override, but upstream krakend-koanf hardcodes the KRAKEND_ env prefix (a const, not configurable). Corrected to KRAKEND_PORT. The fork's own service knobs (GATEWAY_LISTEN, GATEWAY_SHUTDOWN_TIMEOUT) use GATEWAY_ via direct os.Getenv — a separate mechanism from koanf config-file key overrides.

Integration divergences (tracked, not fixed here): 12 of 59 lura fixtures in tests/ exercise upstream behaviour this edge fork diverges from and are -skip'd in CI — cors_1..5 (fork ships its own CORS preflight, not security/cors), backend_404, cel-1, cel-2, lua_2, router_redirect, integration_jsonschema, negotitate_plain. 47 integration cases + every unit package gate CI. Re-enable per fixture as the edge binary grows the matching component.

The harness needs -tags legacy (fixed 2026-07-25). Every file in tests/ is //go:build legacy, so the previous untagged go test ./... resolved ./tests to zero Go files and ran no integration fixture at all — the -skip list was inert and the "46 cases gate CI" claim above was false for as long as it stood. test.yml now runs two steps: unit packages (go list -tags legacy ./... | grep -v '/tests$') and the harness (./tests -args -gateway_startup_wait=30s). They are separate because go test ./... -args forwards the flag to every package's test binary and the unit packages abort with flag provided but not defined.

-gateway_startup_wait matters: the harness execs the ~190MB edge binary and waits a fixed 1500ms before probing :8080. On a cold or loaded host that is not enough and every fixture fails with connection refused — a startup race, not a behavioural regression. Diagnose with go test -tags legacy ./tests -args -gateway_startup_wait=15s before believing a mass failure. The harness runs whatever binary is at ../gateway, so rebuild it first or you are testing a stale artifact:

VERSION=$(sed -n 's/^VERSION := *//p' Makefile)
CGO_ENABLED=0 go build -tags legacy \
  -ldflags="-X github.com/luraproject/lura/v2/core.KrakendVersion=${VERSION}" \
  -o gateway ./cmd/gateway