hanzoai/cloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 1 Wiki Activity

merge: risk: the credential is the lane a stolen key travels (red-cleared, shadow)

Browse Source 
The lifecycle defense: an edge traffic sensor, a per-credential abuse gate and
the identity boundary's own attestation. Compiled in and mounted, SHADOW per
org — edge.Store.Mode returns live only on an exact "live" match, is absent
from the inherited base, and is never set here, so nothing is armed by landing
it. /v1/risk stays unrouted; /v1/ml is untouched.

Four conflicts, each resolved on its merits rather than by side:

middleware_identity.go — an import collision where both sides were right.
main added namespace.Sanitize (OrgHasUnsafeRune, the cross-org fold refusal),
the branch added principal.Mint. Both symbols are live in the merged body, so
both imports stay; either blanket resolution drops a defense and the build.

middleware_ratelimit.go — the branch's code, main's reasoning. The branch
tests the ROUTER's path (RoutePath + underPrefix) instead of the raw spelling,
which is what stops /v1/billing/../v1/ai/chat from prefix-matching its way
into a rate-limit exemption. Its rationale, though, describes an in-process
HTTP self-dispatch that main has since replaced with a typed ZAP op, so the
comment kept is main's, which is the one that is true here.

plugin/o11y/main.go — complementary, not competing. The branch installs the
identity boundary and the abuse gate in this hand-written main; main added
cloud.ErrorHandler so a propagated refusal renders as its own status instead
of 500. Those two middlewares are precisely what emits refusals, so the
handler matters more after the branch lands, not less. Both kept.

openapi/floor.json — the pointwise maximum, which is what Floor.Raise means:
taking the lower side would silently undo the other's ratchet. Then
regenerated from source, so the regeneration is the truth.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This commit is contained in:
Hanzo Dev
2026-08-02 13:52:56 -07:00
parent abc239841f 5c4bf7e65f
commit 35f7ac4fcb
39 changed files with 6547 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files
LLM.md
+122
View File
@@ -2598,6 +2598,128 @@ migration silently strips request shapes from every generated CLI and SDK.
983/692/109. Every number in this file is tagged with how to re-measure it;
keep it that way.
## Lifecycle defense: ONE scorer seam, ONE fail policy, a sensor at the edge
`risk.go` · `agency.go` · `middleware_abuse.go` · `apps/gateway/edge/traffic.go`
**`cloud.Decide` is the only door to `/v1/risk`.** The app that owns `/v1/risk`
hands its scoring function to the core with `cloud.SetRiskScorer` — the same
inversion `SetObsEventIngest` uses, because package `cloud` cannot import an app.
Nothing in cloud scores; a second scorer would be a second answer to one question
and the two would disagree silently.
**The fail policy is `riskUnavailable`, and it is the only copy.** A scorer that
is absent, erroring, silent, out-of-vocabulary, panicking or past `RiskBudget`
(150ms, enforced by the caller) ALLOWS an ordinary request and BLOCKS a
`Privileged` one. Every answer carries a `Refusal`, so an allow-because-nobody-
was-listening is never recorded as clean. `Privileged(method, path)` is the
data list of grants: credential minting/revocation, identity provisioning,
sign-up, onboarding, the key store (all methods), and admin/org MUTATIONS.
**`AbuseGate` sits `AuditTrail → ScopeRateLimit → AbuseGate → StarterGrant →
BillingGate`.** It keys on the CREDENTIAL, which neither existing limiter can see
— EdgeRateLimit keys on IP pre-auth, ScopeRateLimit on (org, project, service) —
so a stolen key inside its org's normal ceiling is invisible to both. It counts,
classifies, asks and enforces; it never scores. A non-allow verdict is HELD for a
minute so an attack costs one screen, not one per request. A refusal is a 401/403,
which AuditTrail already puts in the tamper-evident trail — there is no second
audit write, because one event must not have two records.
**SHADOW PER ORG BY DEFAULT** (`edge.Policy.Mode`, the one new knob). Shadow
senses and records and enforces nothing. `PUT /v1/gateway/config {"mode":"live"}`
is REFUSED while no scorer is installed: fail-closed on a component that was never
wired is an outage, not a defense, and this is what makes the distinction real.
Only live mode reaches the scorer, meters a screen, or fails closed.
Mode is the one field on that route whose SCOPE and whose AUTHORITY are different
questions, and both were wrong once:
- **It does not inherit.** Every other per-org field layers a platform default
under the org's own value. Mode is not a default, it is an arming decision, so
seeding it from the platform row meant the single PUT that arms the anonymous
lane (the reserved admin org — the only way to arm the lane that has no tenant)
silently armed every tenant in the estate. `Store.Mode` reads the org's OWN row
and nothing else; the platform row governs exactly one scope, the empty org.
- **It is not self-service.** It lives on a tenant's row but writing it requires
SuperAdmin, whichever row it lands on. The subject of an abuse control does not
get to switch the control off — and an org-admin credential is what a stolen key
buys, so leaving it in the self-service branch made the gate disarmable by the
account it was watching.
**Agency — the differentiator.** `agency(class, pattern)` is pure and total, and
reads OUR issuance, never the client's self-description (there is no user-agent
heuristic and there must not be one) and never a REQUEST HEADER. "We minted this
credential" is a fact only the identity boundary can state, so the boundary states
it: `SanitizeIdentity` parks the principal it resolved with `principal.Mint`, a
request-local slot no client can write, and the classifier reads `principal.Minted`.
Reading `c.Org()`/`c.User()` instead was forgeable twice over — X-Org-Id survives
the boundary on the anonymous path by design, and a hand-written plugin process may
have no boundary in front of it at all — so two headers plus an `sk-`-shaped string
that never validated bought the agent lane. Absent attestation resolves to
anonymous, which is the fail-closed direction: only a credential we resolved buys
the lane. An attributable machine credential is the
`agent` lane whatever it claims to be; a browser session is `human`; an
unattributable caller is `unknown` until it shows an abuse SHAPE — many
credentials from one address, a wall of refusals, a path sweep — and only then
`bot`. Anonymous is not malicious.
**The sensor** (`edge.Traffic`, a LEAF package so the middleware and the app share
ONE object) counts requests, 401/403 failures, path spread and peer spread per
(org, credential) over a rolling minute. Bounded by construction: a fixed ring
plus two 64-bit population-count words per key. Credentials appear only as a keyed
per-process fingerprint — never a bare digest, so a published fingerprint cannot be
tested against a candidate key off-box.
**TENANCY IS THE DATA STRUCTURE AND THE BOUND IS PER TENANT.** One org's callers,
hosts and lane counters live in that org's OWN tables, reached only by indexing
`tenants[org]`; there is no shared map with org-prefixed keys, so a cross-tenant
read or eviction is unwritable rather than merely refused. Each tenant has its own
ceiling and reclaims only its own keys — a process-wide cap over a shared table is
a cross-tenant denial of service, because the org that fills it evicts whoever was
quietest and that victim's controls then go silent with no error. A tenant at its
ceiling degrades exactly one tenant, itself, and says so: `TrafficView.Saturated`
is its own count of its own reclaims. The one reclaim policy lives in `table[V]`
(unexported map, cap as a constructor argument, a pinned live verdict is never
dropped) so the wrong shape is unrepresentable rather than discouraged.
**The client address is `cloud.ClientIP`, and it is the only one.** The peer is the
truth: a caller that is not one of our own proxies IS the client, and no header it
sent is read. When the peer IS ours, the forwarded chain is walked from the RIGHT —
the end each hop appends to — and the first entry that is not one of ours is the
answer; everything to its left was written before our infrastructure saw the
request. A chain that is entirely ours is an in-cluster caller with no client
address (`""`), which is what keeps sibling services out of the public rate limiter.
Our own hops are a CIDR set (`CLOUD_TRUSTED_PROXIES`, defaulting to private space)
rather than a hop count, because a count is a promise about topology that nothing
enforces. Reading the LEFT-most entry — the one the client writes — let one host
present a million clients: it defeated the per-IP limit keyed on it, put a chosen
address in an audit row, and fed the sensor's address table without bound.
`GET /v1/gateway/traffic` (`gatewayTraffic`) reports the caller's own org: lane
split, denials, screens, and its busiest credentials by fingerprint. Screens are
counted there from the first request AND metered on the org's usage ledger —
`ResourceMeter.Meter` is a no-op while `CLOUD_RISK_SCREEN_CENTS` is unset (0),
because the price belongs to the pricing catalog and inventing one here would be
a fabricated number.
**A path is what the ROUTER says it is.** `cloud.RoutePath` normalizes to fiber's
own detection path — lower-cased, trailing slashes stripped — and every security
comparison (`Privileged`, `Probe`, the gate's exemptions) runs against it, on
segment boundaries. `strings.HasPrefix` over the raw `c.Path()` meant one capital
letter routed to the key store while matching no grant prefix, so the scorer's
silence ALLOWED what the fail-closed branch exists to refuse.
**Asking is bounded.** Each ask costs a goroutine that lives until the scorer
returns, so `Decide` holds `MaxScorerCalls` in flight and answers `RefusalBusy`
past the ceiling — the same fail policy a timeout gets, reached without allocating.
A scorer stuck on a lock cannot become an out-of-memory in the process it was
installed to protect.
The other enforcement point is in `hanzoai/iam` (`internal/risk` +
`internal/oidc/signup_gate.go`), which calls `POST /v1/risk/decide` over the wire
at sign-up. Its arming signal is `RISK_URL` rather than a per-org mode; the
semantic is identical — fail closed once armed, allow before.
## Cross-subsystem seams that are values, not places
- **AN AGGREGATOR CALLS; IT DOES NOT IMPORT.** Apps are separate binaries, so a Go
agency.go
+212
View File
@@ -0,0 +1,212 @@
package cloud
// Agency — telling a customer's automation apart from a bad bot.
//
// This is the question a generic bot filter cannot answer and we can, because
// the answer is a fact about OUR OWN issuance rather than a guess about the
// client. A user-agent string is whatever the caller typed; a credential is
// something we minted, to a named principal, in a named tenant, that we meter
// and can revoke. So the classification here reads the credential, never the
// client's self-description. There is no user-agent heuristic in this file and
// there must not be one: an agent that lies about its user-agent is still
// holding our key, and a scraper that copies Chrome's is still holding nothing.
//
// FOUR LANES, and the boundary between them is attributability:
//
// agent — an attributable machine credential (sk-/hk-, or a machine JWT).
// Programmatic traffic that a named org pays for and we can switch
// off. This is the lane our own agents run in, and it is the lane a
// customer's automation runs in. It gets judged on VOLUME PATTERN,
// not on being automated: being automated is the product.
// human — a browser session bearer. A person at a keyboard.
// bot — unattributable traffic already showing an abuse shape: no
// credential (or a publishable one, the kind that ships in a browser
// bundle and is therefore the kind that gets copied) TOGETHER WITH a
// pattern no legitimate client produces — many keys from one address,
// a wall of auth failures, a path sweep.
// unknown — unattributable but unremarkable. Scored normally. Most anonymous
// traffic is here and stays here, which is the point: "anonymous" is
// not "malicious".
//
// THIS FILE ANSWERS ONE HALF OF THAT AND THE SENSOR ANSWERS THE OTHER. Reading a
// request for its credential CLASS needs the request, so it is here. Turning a
// class plus a traffic pattern into a LANE needs the pattern, so it is
// edge.Lane — inside the observation that produced the counts, which is the only
// place that can compute it before it is counted. Splitting it the other way is
// what made every request in the lane report land in "unknown": the gate had to
// state the lane before it had asked what the caller had been doing.
//
// The gate's classification is a PRIOR. It is sent to the scorer as a signal and
// the scorer — which holds the agent registry, the session plane and the metered
// shape — may overrule it. Its answer is the authoritative one. That split is
// deliberate: the edge must classify in nanoseconds off facts already in hand,
// and must not learn to score.
import (
"crypto/hmac"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"strings"
"github.com/hanzoai/cloud/apps/gateway/edge"
"github.com/hanzoai/cloud/apps/principal"
"github.com/zap-proto/zip"
)
// credentialClass reads the class off a request. It looks at the Authorization
// header for the credential's SHAPE and at the validated principal for whether
// the identity boundary accepted it — never at the body, never at a client
// header the boundary does not mint. The vocabulary is edge's (edge.CredSecret
// and friends), because the lane rule that consumes it lives there.
//
// A credential that was presented and did NOT validate is anonymous, not
// secret: possession of a string that fails is possession of nothing.
func credentialClass(c *zip.Ctx) string {
tok := callerCredential(c)
switch {
case tok == "":
return edge.CredAnonymous
case IsPublishableKey(tok):
// A pk- names an org and no principal. It is a tenant label, not an
// authentication, so it stays publishable whether or not an org resolved.
return edge.CredPublishable
case !principalValidated(c):
return edge.CredAnonymous
case isAPIKey(tok):
return edge.CredSecret
default:
return edge.CredSession
}
}
// principalValidated reports whether the identity boundary VERIFIED a principal
// for this request — read from the boundary's own attestation (principal.Minted),
// never from a header.
//
// It used to read `c.Org() != "" || c.User() != ""`, and both disjuncts were
// forgeable:
//
// - X-Org-Id survives the boundary on the anonymous path by design (the Phase-1
// data passthrough, documented in middleware_identity.go), so ANY caller can
// make c.Org() non-empty by sending the header;
// - in a process where the boundary is not installed at all — a hand-written
// plugin main — nothing strips either header, so X-User-Id is the client's
// too.
//
// Either one, plus an sk--shaped string in Authorization that never validated,
// moved a caller from the anonymous lane into the AGENT lane: the lane whose
// whole meaning is "we minted this credential to a named tenant and can revoke
// it". A differentiator a client can set is not a differentiator.
//
// The attestation is absent when no boundary ran, which resolves to anonymous —
// the fail-closed direction for a classifier: unattributable traffic is judged on
// its shape, and only a credential WE resolved buys the agent lane.
func principalValidated(c *zip.Ctx) bool {
p, ok := principal.Minted(c)
return ok && p.User != ""
}
// verifiedOrg is the tenant the identity boundary resolved for this request, and
// "" for an anonymous caller or a process with no boundary. It is the sensor's
// keyspace index, so it must be the SERVER's answer: an org taken from a header
// would let one caller write into — and evict from — another tenant's state.
func verifiedOrg(c *zip.Ctx) string {
p, ok := principal.Minted(c)
if !ok || p.User == "" {
return ""
}
return p.Org
}
// observation is the ONE place an observation is built from a request, and the reason
// it is one place is that two of its fields are the same fingerprint under
// different trust:
//
// Cred — set ONLY when the identity boundary validated the credential. It
// is what the sensor keys on, so it must be a fact we stated. A
// caller keyed on a string it chooses can leave its own hold by
// typing a different one, and can open a table entry per request.
// Presented — set for whatever the request carried, valid or not. It is counted
// only as spread, because a wall of invalid credentials from one
// address IS the stuffing signature and refusing to count it would
// blind the sensor to the attack it exists to see.
//
// Building this anywhere else would mean deciding that trust question a second
// time, and the second answer is the one that would be wrong.
func observation(c *zip.Ctx, path string) edge.Signal {
presented := credentialOf(c)
s := edge.Signal{
Org: verifiedOrg(c),
Presented: presented,
IP: ClientIP(c),
Path: path,
Class: credentialClass(c),
}
if principalValidated(c) {
s.Cred = presented
}
return s
}
// fingerprintSalt is a per-PROCESS secret. A credential fingerprint is only ever
// compared with another fingerprint from the same process and the same window,
// so the salt never needs to be shared, persisted or rotated — and because it is
// never shared, a fingerprint that escapes in a log or a report cannot be tested
// against a candidate key anywhere else. Generated from crypto/rand at init; a
// generator failure is fatal at start rather than silently downgrading to a
// guessable salt.
var fingerprintSalt = mustSalt()
func mustSalt() []byte {
b := make([]byte, 32)
if _, err := rand.Read(b); err != nil {
panic("cloud: no entropy for the credential fingerprint salt: " + err.Error())
}
return b
}
// fingerprintLen is how much of the digest is kept: 12 base64url characters, 72
// bits. Enough that two live credentials colliding is not a thing that happens,
// short enough to read in a report.
const fingerprintLen = 12
// Fingerprint turns a credential into a stable per-process handle. It is what
// the sensor counts under and what a traffic report shows — the credential
// itself never enters a counter, a log, a record or a response.
//
// HMAC-SHA256 rather than a bare hash: with a bare hash, anyone holding a
// candidate key could confirm it against a published fingerprint. With a keyed
// digest under a salt that never leaves the process, they cannot.
func Fingerprint(cred string) string {
cred = strings.TrimSpace(cred)
if cred == "" {
return ""
}
m := hmac.New(sha256.New, fingerprintSalt)
_, _ = m.Write([]byte(cred))
return base64.RawURLEncoding.EncodeToString(m.Sum(nil))[:fingerprintLen]
}
// callerCredential is the credential this request presented, in the SAME
// precedence the identity boundary trusts — callerToken, the one token resolution
// SanitizeIdentity and CallerBearer already share. Reading the Authorization
// header directly would be a second, drifting answer to "which credential
// identifies this caller": a client authenticating with X-Authorization or a
// session cookie would validate upstream and then be counted here as anonymous,
// so its traffic would be pooled under its address instead of under itself.
//
// X-Api-Key is checked after it, because that header is not part of the identity
// boundary's precedence but IS a spelling several SDKs send; a caller the boundary
// could not identify is still a caller this sensor must be able to tell apart from
// the next one.
func callerCredential(c *zip.Ctx) string {
if tok := callerToken(c); tok != "" {
return tok
}
return strings.TrimSpace(c.Header("X-Api-Key"))
}
// credentialOf returns the fingerprint of whatever credential a request
// presented, and "" when it presented none.
func credentialOf(c *zip.Ctx) string { return Fingerprint(callerCredential(c)) }
agency_pin_test.go
+55
View File
@@ -0,0 +1,55 @@
package cloud
// The pin for the trust decision. An edge.Signal carries two fingerprints of the
// same credential under different trust — one the sensor may key on, one it may
// only count — so a second place that builds one is a second answer to "did this
// credential validate", and the second answer is the one that would be wrong.
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestPin_AnObservationIsBuiltInOnePlace(t *testing.T) {
names, err := filepath.Glob("*.go")
if err != nil {
t.Fatal(err)
}
for _, name := range names {
if strings.HasSuffix(name, "_test.go") || name == "agency.go" {
continue
}
b, err := os.ReadFile(name)
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(b), "edge.Signal{") {
t.Errorf("%s builds an edge.Signal; the one constructor is observation() in agency.go", name)
}
}
}
// And that constructor may only put a fingerprint in the KEY field behind the
// identity boundary's own attestation.
func TestPin_OnlyAnAttestedCredentialBecomesAKey(t *testing.T) {
b, err := os.ReadFile("agency.go")
if err != nil {
t.Fatal(err)
}
src := string(b)
i := strings.Index(src, "func observation(")
if i < 0 {
t.Fatal("observation() is gone; the observation constructor moved and this pin did not")
}
body := src[i:]
if j := strings.Index(body, "\n}\n"); j >= 0 {
body = body[:j]
}
assign := strings.Index(body, "s.Cred = ")
guard := strings.Index(body, "if principalValidated(c)")
if assign < 0 || guard < 0 || guard > assign {
t.Error("Signal.Cred is set without the identity boundary's attestation guarding it")
}
}
agency_test.go
+158
View File
@@ -0,0 +1,158 @@
package cloud
// Reading a REQUEST for the two facts the differentiator turns on: which class
// of credential it presented, and whether the identity boundary validated it.
// The lane rule those two feed is a pure function and is specified where it
// lives, in edge (lane_test.go).
import (
"crypto/sha256"
"encoding/base64"
"net/http"
"net/http/httptest"
"testing"
"github.com/hanzoai/cloud/apps/gateway/edge"
"github.com/zap-proto/zip"
)
// The whole claim of the differentiator is that the classification reads OUR
// issuance and not the client's self-description. If a user-agent string could
// move a caller between lanes, the classification would be worth nothing.
func TestCredentialClass_ReadsTheCredentialNotTheClient(t *testing.T) {
cases := []struct {
name string
org string
auth string
apiKey string
ua string
want string
}{
{"validated secret key", "acme", "Bearer sk-live-1", "", "curl/8", edge.CredSecret},
{"validated hanzo key", "acme", "Bearer hk-live-1", "", "", edge.CredSecret},
{"validated session bearer", "acme", "Bearer eyJhbGciOi.payload.sig", "", "Mozilla/5.0", edge.CredSession},
{"publishable key, org resolved", "acme", "Bearer pk-live-1", "", "", edge.CredPublishable},
{"publishable key, no org", "", "Bearer pk-live-1", "", "", edge.CredPublishable},
{"secret-shaped but unvalidated", "", "Bearer sk-live-1", "", "", edge.CredAnonymous},
{"session-shaped but unvalidated", "", "Bearer eyJhbGciOi.payload.sig", "", "", edge.CredAnonymous},
{"no credential at all", "", "", "", "Mozilla/5.0", edge.CredAnonymous},
{"key in the X-Api-Key header", "acme", "", "sk-live-1", "", edge.CredSecret},
{"a browser user-agent cannot make a key a session", "acme", "Bearer sk-live-1", "", "Mozilla/5.0 Chrome/126", edge.CredSecret},
{"a curl user-agent cannot make a session a key", "acme", "Bearer eyJhbGciOi.p.s", "", "curl/8.7", edge.CredSession},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
var got string
app := zip.New(zip.Config{})
app.Use(attest()) // the boundary; without it every caller is anonymous.
app.Get("/probe", func(c *zip.Ctx) error {
got = credentialClass(c)
return c.JSON(http.StatusOK, map[string]string{"ok": "1"})
})
req := httptest.NewRequest(http.MethodGet, "/probe", nil)
if tc.org != "" {
req.Header.Set("X-Org-Id", tc.org)
req.Header.Set("X-User-Id", "u-"+tc.org)
}
if tc.auth != "" {
req.Header.Set("Authorization", tc.auth)
}
if tc.apiKey != "" {
req.Header.Set("X-Api-Key", tc.apiKey)
}
if tc.ua != "" {
req.Header.Set("User-Agent", tc.ua)
}
if _, err := app.Fiber().Test(req); err != nil {
t.Fatal(err)
}
if got != tc.want {
t.Fatalf("credentialClass = %q, want %q", got, tc.want)
}
})
}
}
// A fingerprint must identify a caller within a process and be useless outside
// one. The salt is per-process and keyed, so a published fingerprint cannot be
// tested against a candidate key anywhere else.
func TestFingerprint(t *testing.T) {
if Fingerprint("") != "" {
t.Fatal("no credential must fingerprint to nothing, not to a constant every anonymous caller shares")
}
a, b := Fingerprint("sk-live-1"), Fingerprint("sk-live-1")
if a != b {
t.Fatal("a fingerprint must be stable within a process")
}
if a == Fingerprint("sk-live-2") {
t.Fatal("two credentials must not share a fingerprint")
}
if len(a) != fingerprintLen {
t.Fatalf("fingerprint length = %d, want %d", len(a), fingerprintLen)
}
// It must not be a bare digest of the credential: with a bare digest anyone
// holding a candidate key could confirm it against a published fingerprint.
if a == unsaltedDigest("sk-live-1") {
t.Fatal("the fingerprint is an unsalted digest — a published one would be brute-forceable")
}
// Whitespace is not a second identity for the same key.
if Fingerprint(" sk-live-1 ") != a {
t.Fatal("a padded credential must fingerprint to the same caller")
}
}
// unsaltedDigest is what a NAIVE implementation would produce. It exists only so
// the test above can assert we did not write that one.
func unsaltedDigest(s string) string {
sum := sha256.Sum256([]byte(s))
return base64.RawURLEncoding.EncodeToString(sum[:])[:fingerprintLen]
}
// The credential must be read through the SAME resolution the identity boundary
// trusts. A second answer to "which credential is this caller" would pool a
// client that authenticates by X-Authorization or by session cookie under its
// ADDRESS instead of under itself — which is exactly the caller the sensor exists
// to tell apart from its neighbours.
func TestCredentialClass_UsesTheBoundarysOwnResolution(t *testing.T) {
cases := []struct {
name string
set func(*http.Request)
wantClass string
wantSameAs string // a header spelling that must fingerprint identically
}{
{"Authorization bearer", func(r *http.Request) {
r.Header.Set("Authorization", "Bearer sk-live-1")
}, edge.CredSecret, ""},
{"X-Authorization bearer", func(r *http.Request) {
r.Header.Set("X-Authorization", "Bearer sk-live-1")
}, edge.CredSecret, "Bearer sk-live-1"},
{"X-Api-Key", func(r *http.Request) {
r.Header.Set("X-Api-Key", "sk-live-1")
}, edge.CredSecret, "Bearer sk-live-1"},
}
base := Fingerprint("sk-live-1")
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
var class, fp string
app := zip.New(zip.Config{})
app.Use(attest())
app.Get("/probe", func(c *zip.Ctx) error {
class, fp = credentialClass(c), credentialOf(c)
return c.JSON(http.StatusOK, map[string]string{"ok": "1"})
})
req := httptest.NewRequest(http.MethodGet, "/probe", nil)
req.Header.Set("X-Org-Id", "acme")
req.Header.Set("X-User-Id", "u-acme")
tc.set(req)
if _, err := app.Fiber().Test(req); err != nil {
t.Fatal(err)
}
if class != tc.wantClass {
t.Fatalf("class = %q, want %q", class, tc.wantClass)
}
if fp != base {
t.Fatalf("one credential must fingerprint to ONE caller however it is spelled: %q vs %q", fp, base)
}
})
}
}
apps/compliance/compliance.go
+5 -10
View File
@@ -1188,16 +1188,11 @@ func mustJSON(v any) json.RawMessage {
return b
}
// clientIP is the best-effort source IP for the audit record.
func clientIP(c *zip.Ctx) string {
if xff := c.Header("X-Forwarded-For"); xff != "" {
if i := strings.IndexByte(xff, ','); i >= 0 {
return strings.TrimSpace(xff[:i])
}
return strings.TrimSpace(xff)
}
return c.Header("X-Real-Ip")
}
// clientIP is the caller's address, by the ONE rule — cloud.ClientIP. It lands in
// a durable audit record, and the LEFT-most X-Forwarded-For entry (and X-Real-Ip)
// are values the client writes: an address chosen by the party being audited is
// not evidence.
func clientIP(c *zip.Ctx) string { return cloud.ClientIP(c) }
// genID mints a prefixed, collision-resistant id (prefix + 128 random bits).
func genID(prefix string) (string, error) {
apps/gateway/edge/bound_test.go
+493
View File
@@ -0,0 +1,493 @@
package edge
// The bound's own contract. Four properties, and each one is a defect that came
// back a layer down after the last one was fixed:
//
// IDENTITY — a caller is what the SERVER attested, never a string the caller
// picked. Rotating an unvalidated credential must not produce a new
// caller, must not leave a hold, and must not open a table entry.
// ADMISSION — nothing is ever removed to make room for something else. A flood
// is refused; it does not evict, and it cannot release a verdict.
// BYTES — the published per-entry ceilings are ceilings. Measured, not
// asserted.
// LOUDNESS — a ceiling that binds says so, in a graded state an operator can
// read, including for the lane that has no tenant.
import (
"fmt"
"runtime"
"strings"
"testing"
"time"
)
// REGRESSION — enforcement was keyed on the raw Authorization value, validated or
// not, which is a string the ATTACKER PICKS. A caller held under a block verdict
// walked out of it five times out of five by changing that header and nothing
// else, so presenting garbage was strictly better for an attacker than presenting
// nothing at all.
func TestTraffic_AnUnvalidatedCredentialIsNotAnIdentity(t *testing.T) {
tr := NewTraffic()
const addr = "203.0.113.200"
first := forged("", "junk-000000", addr, "/v1/models")
tr.Observe(first, t0)
tr.Hold(first, Hold{Action: "block", Reason: "peers", Decision: "d-1"}, time.Minute, t0)
for i := 1; i <= 5; i++ {
next := forged("", fmt.Sprintf("junk-%06d", i), addr, "/v1/models")
if _, ok := tr.Held(next, t0.Add(time.Duration(i)*time.Second)); !ok {
t.Fatalf("attempt %d: the hold was evaded by presenting a different credential", i)
}
}
// One address is one caller however many credentials it invents, so the
// counts accumulate against it instead of resetting per request...
var p Pattern
for i := 0; i < 40; i++ {
p = tr.Observe(forged("", fmt.Sprintf("junk-%06d", i), addr, "/v1/models"), t0)
}
if p.Requests < 40 {
t.Fatalf("requests = %d after 41 requests from one address, want at least 40", p.Requests)
}
// ...and it costs the sensor ONE key, not one per request.
tr.mu.Lock()
n := len(tr.tenants[""].callers.m)
tr.mu.Unlock()
if n != 1 {
t.Fatalf("one address opened %d caller keys; a caller may not choose its own key", n)
}
// The credentials it presented are still counted — as SPREAD, which is the
// stuffing signature the sensor exists to see.
if p.Peers < 8 {
t.Fatalf("peers = %d after 40 distinct credentials from one address, want at least 8", p.Peers)
}
// A VALIDATED credential is still an identity of its own: this is not "keys
// stop mattering", it is "only a key we issued names a caller".
a := sig("acme", "fpA", addr, "/v1/models")
b := sig("acme", "fpB", addr, "/v1/models")
tr.Hold(a, Hold{Action: "block"}, time.Minute, t0)
if _, ok := tr.Held(b, t0); ok {
t.Fatal("a hold on one validated credential reached another")
}
}
// REGRESSION — the shared lane erased its own callers. org=="" is ONE scope for
// the whole internet, and a flood of forged credentials from one address used to
// reclaim every tracked caller in it: 200 of 200 lost their accumulated state,
// which made every pattern the sensor watches for resettable on demand.
func TestTraffic_AFloodCannotEraseAnotherCaller(t *testing.T) {
tr := NewTraffic()
// 200 anonymous callers, each with a history worth erasing.
for i := 0; i < 200; i++ {
s := forged("", "", fmt.Sprintf("198.51.%d.%d", i/256, i%256), "/v1/models")
for n := 0; n < 10; n++ {
tr.Observe(s, t0)
}
}
// A hundred thousand requests from one address, each inventing a credential.
for i := 0; i < 100_000; i++ {
tr.Observe(forged("", fmt.Sprintf("forged-%06d", i), "203.0.113.9", "/v1/chat"), t0)
}
lost := 0
for i := 0; i < 200; i++ {
s := forged("", "", fmt.Sprintf("198.51.%d.%d", i/256, i%256), "/v1/models")
if p := tr.Observe(s, t0); p.Requests < 11 {
lost++
}
}
if lost != 0 {
t.Fatalf("%d of 200 tracked callers lost their state to another caller's flood", lost)
}
}
// REGRESSION — the ceiling could be overrun. reclaim skipped keys under a live
// verdict and then admitted anyway, so 25,000 held callers lived in a table that
// published a ceiling far below that, and the report said nothing.
//
// The rule now has no second pass: what is live stays, what does not fit is
// refused, and the refusal is on the scope's own report.
func TestTraffic_TheCeilingRefusesRatherThanOverrunning(t *testing.T) {
tr := NewTraffic()
held := make([]Signal, 0, maxCallers+5_000)
for i := 0; i < maxCallers+5_000; i++ {
s := sig("acme", fmt.Sprintf("fp%06d", i), "203.0.113.1", "/v1/models")
tr.Observe(s, t0)
tr.Hold(s, Hold{Action: "block", Reason: "peers", Decision: "d"}, time.Minute, t0)
held = append(held, s)
}
tr.mu.Lock()
n := len(tr.tenants["acme"].callers.m)
tr.mu.Unlock()
if n > maxCallers {
t.Fatalf("the table holds %d keys against a %d ceiling", n, maxCallers)
}
// Not one admitted verdict was released to make room for a later one.
live := 0
for _, s := range held {
if _, ok := tr.Held(s, t0.Add(time.Second)); ok {
live++
}
}
if live != n {
t.Fatalf("%d verdicts are held but the table holds %d keys: a hold was dropped, not refused", live, n)
}
v := tr.View("acme", ModeLive, t0)
if v.Strain != StrainRefuse || v.Refused == 0 {
t.Fatalf("a ceiling that refused %d callers must say so: strain=%q refused=%d",
maxCallers+5_000-n, v.Strain, v.Refused)
}
if v.Tracked != n || v.Ceiling != maxCallers {
t.Fatalf("the report must state occupancy against the ceiling: tracked=%d ceiling=%d (holding %d)",
v.Tracked, v.Ceiling, n)
}
}
// A caller the ceiling turned away is UNMEASURED, and says so on the observation
// itself — otherwise its zero counts read as "a caller making its first request",
// which is the one pattern that screens every single time.
func TestTraffic_ARefusedCallerIsToldItIsUnmeasured(t *testing.T) {
tr := NewTraffic()
for i := 0; i < maxCallers; i++ {
tr.Observe(sig("acme", fmt.Sprintf("fp%06d", i), "203.0.113.1", "/v1/models"), t0)
}
p := tr.Observe(sig("acme", "fpOverflow", "203.0.113.1", "/v1/models"), t0)
if p.Strain != StrainRefuse {
t.Fatalf("strain = %q for a caller the ceiling refused, want %q", p.Strain, StrainRefuse)
}
if p.Requests != 0 {
t.Fatalf("an unmeasured caller must report no counts, got %+v", p)
}
// The request is still counted in the scope's totals: a scope's own report
// must never understate its traffic because the sensor ran out of room.
if v := tr.View("acme", ModeLive, t0); v.Requests != maxCallers+1 {
t.Fatalf("scope requests = %d, want %d", v.Requests, maxCallers+1)
}
}
// The grade is announced ONCE per rise, not once per refused request: a flood
// produces one refusal per request, and a log line per request is an outage of
// its own.
func TestTraffic_StrainRisesOnce(t *testing.T) {
tr := NewTraffic()
rises := map[string]int{}
for i := 0; i < maxCallers+100; i++ {
if r := tr.Observe(sig("acme", fmt.Sprintf("fp%06d", i), "203.0.113.1", "/v1/models"), t0).Rise; r != "" {
rises[r]++
}
}
if rises[StrainFull] != 1 || rises[StrainRefuse] != 1 {
t.Fatalf("each grade must be announced exactly once, got %v", rises)
}
}
// The lane with no tenant is the one a bad bot calls from, and its saturation was
// readable by nobody: the report could only be asked for by NAME, and this scope
// has no name. It is the empty scope, and the empty scope is a value.
func TestTraffic_TheAnonymousLaneIsReadable(t *testing.T) {
tr := NewTraffic()
for i := 0; i < maxCallers+50; i++ {
tr.Observe(forged("", "x", fmt.Sprintf("198.51.%d.%d", i/256, i%256), "/v1/models"), t0)
}
v := tr.View("", ModeLive, t0)
if v.Strain != StrainRefuse || v.Refused == 0 || v.Tracked == 0 {
t.Fatalf("the anonymous lane's own saturation must be readable: %+v", v)
}
if v.Requests == 0 || v.Lanes[AgencyUnknown]+v.Lanes[AgencyBot] == 0 {
t.Fatalf("the anonymous lane's own traffic must be readable: %+v", v)
}
}
// The lane split is DERIVED from the counts, so it can only be computed inside
// the observation that produced them. It used to be passed IN — before those
// counts existed — so every request ever counted landed in "unknown" and the one
// number this report exists for was a constant.
func TestTraffic_TheLaneSplitNamesTheLane(t *testing.T) {
tr := NewTraffic()
tr.Observe(Signal{Org: "acme", Cred: "fpA", Presented: "fpA", IP: "203.0.113.1", Path: "/v1/models", Class: CredSecret}, t0)
tr.Observe(Signal{Org: "acme", Cred: "fpB", Presented: "fpB", IP: "203.0.113.2", Path: "/v1/models", Class: CredSession}, t0)
v := tr.View("acme", ModeShadow, t0)
if v.Lanes[AgencyAgent] != 1 || v.Lanes[AgencyHuman] != 1 {
t.Fatalf("lanes = %v, want one agent and one human", v.Lanes)
}
if v.Lanes[AgencyUnknown] != 0 {
t.Fatalf("attributable traffic landed in %q: %v", AgencyUnknown, v.Lanes)
}
// And an unattributable caller showing an abuse shape is the bot lane — the
// pattern that produced it is the one this observation just counted.
tr2 := NewTraffic()
var last Pattern
for i := 0; i < 40; i++ {
last = tr2.Observe(forged("", fmt.Sprintf("junk-%03d", i), "203.0.113.9", "/v1/models"), t0)
}
if last.Lane != AgencyBot {
t.Fatalf("lane = %q for a caller presenting 40 credentials from one address, want %q", last.Lane, AgencyBot)
}
}
// An unanswered screen is not a quiet day. The count is split so a scorer that
// has stopped answering is a number on the org's own report rather than an
// inference from traffic that all looks allowed.
func TestTraffic_UnansweredScreensAreCountedApart(t *testing.T) {
tr := NewTraffic()
tr.Observe(sig("acme", "fp1", "203.0.113.1", "/v1/models"), t0)
tr.Screen("acme", "", t0)
tr.Screen("acme", "scorer-stuck", t0)
tr.Screen("acme", "scorer-timeout", t0)
v := tr.View("acme", ModeLive, t0)
if v.Screens != 3 || v.Unscored != 2 {
t.Fatalf("screens=%d unscored=%d, want 3 and 2", v.Screens, v.Unscored)
}
}
// A held verdict carries strings from the SCORER, which is an input like any
// other. Clamped at the door, so one entry's size is a published fact and
// count × size is a real byte bound.
func TestTraffic_HeldStringsAreClampedAtTheDoor(t *testing.T) {
tr := NewTraffic()
s := sig("acme", "fp1", "203.0.113.1", "/v1/models")
tr.Hold(s, Hold{
Action: strings.Repeat("a", 4096),
Reason: strings.Repeat("b", 1<<20),
Decision: strings.Repeat("c", 1<<20),
}, time.Minute, t0)
h, ok := tr.Held(s, t0)
if !ok {
t.Fatal("the hold was not stored")
}
if len(h.Action) > maxActionLen || len(h.Reason) > maxCauseLen || len(h.Decision) > maxDecisionLen {
t.Fatalf("a held verdict kept %d/%d/%d bytes past its clamps", len(h.Action), len(h.Reason), len(h.Decision))
}
// And the key itself, which an attacker supplies as an address.
long := forged("", "", strings.Repeat("9", 4096), "/v1/models")
tr.Observe(long, t0)
tr.mu.Lock()
for k := range tr.tenants[""].callers.m {
if len(k) > maxKeyLen {
t.Errorf("caller key is %d bytes, past the %d clamp", len(k), maxKeyLen)
}
}
tr.mu.Unlock()
}
// THE BOUND IS IN BYTES. A cap on the NUMBER of keys is not a bound when the
// values behind them can be any size, so the ceiling this file publishes is
// measured here against a table filled with worst-case entries. If the published
// number understates what an entry really costs, this fails.
func TestTraffic_FootprintIsBoundedInBytes(t *testing.T) {
// Worst case per entry: the longest key the clamps admit, a live verdict with
// every string at its clamp, and a spread word in use.
action, cause, decision := strings.Repeat("a", maxActionLen), strings.Repeat("b", maxCauseLen), strings.Repeat("c", maxDecisionLen)
measure := func(fill func(tr *Traffic)) uint64 {
runtime.GC()
var before, after runtime.MemStats
runtime.ReadMemStats(&before)
tr := NewTraffic()
fill(tr)
runtime.GC()
runtime.ReadMemStats(&after)
runtime.KeepAlive(tr)
return after.HeapAlloc - before.HeapAlloc
}
const n = maxCallers
callers := measure(func(tr *Traffic) {
for i := 0; i < n; i++ {
s := sig("acme", fmt.Sprintf("%012d", i), "203.0.113.1", "/v1/models")
tr.Observe(s, t0)
tr.Hold(s, Hold{Action: action, Reason: cause, Decision: decision}, time.Minute, t0)
}
})
if per := callers / n; per > callerBytes {
t.Fatalf("one caller entry measures %d bytes against a published %d", per, callerBytes)
}
hosts := measure(func(tr *Traffic) {
for i := 0; i < n; i++ {
tr.Observe(Signal{Org: "acme", Cred: "fp", Presented: "fp", Class: CredSecret,
IP: fmt.Sprintf("2001:db8:%x:%x::%x", i/256, i%256, i), Path: "/v1/models"}, t0)
}
})
// The host fill also creates one caller entry; subtract its published cost so
// what is compared is the address table alone.
if per := (hosts - callerBytes) / n; per > hostBytes {
t.Fatalf("one address entry measures %d bytes against a published %d", per, hostBytes)
}
scopes := measure(func(tr *Traffic) {
for i := 0; i < n; i++ {
tr.Observe(sig(fmt.Sprintf("org%09d", i), "fp", "203.0.113.1", "/v1/models"), t0)
}
})
if per := (scopes - callerBytes) / n; per > tenantBytes {
t.Fatalf("one scope measures %d bytes against a published %d", per, tenantBytes)
}
}
// The budget is the process ceiling, and it is charged and REFUNDED by exactly
// the admissions and reclaims that use it — a leak either way turns a bound into
// a slow refusal of everything or into no bound at all.
func TestTraffic_TheBudgetIsTheProcessCeiling(t *testing.T) {
tr := NewTraffic()
tr.budget.max = tenantBytes + 4*callerBytes // room for one scope and four callers
// Callers with no credential, so every byte charged is a caller entry and the
// arithmetic under test is not sharing the budget with the address table.
for i := 0; i < 10; i++ {
tr.Observe(forged("acme", "", fmt.Sprintf("203.0.113.%d", i), "/v1/models"), t0)
}
tr.mu.Lock()
n, used := len(tr.tenants["acme"].callers.m), tr.budget.used
tr.mu.Unlock()
if n != 4 {
t.Fatalf("the budget admitted %d callers, want 4", n)
}
if used > tr.budget.max {
t.Fatalf("the budget is over its own ceiling: %d > %d", used, tr.budget.max)
}
// A second scope cannot be admitted, and cannot displace the first.
tr.Observe(forged("globex", "", "198.51.100.1", "/v1/models"), t0)
tr.mu.Lock()
_, stranger := tr.tenants["globex"]
_, incumbent := tr.tenants["acme"]
tr.mu.Unlock()
if stranger {
t.Fatal("a scope was admitted past the budget")
}
if !incumbent {
t.Fatal("an admission displaced a scope that was already there")
}
// Once the first scope goes idle, its whole charge comes back.
tr.mu.Lock()
tr.sweepLocked(t0.Add(tenantIdle + time.Minute))
used = tr.budget.used
tr.mu.Unlock()
if used != 0 {
t.Fatalf("%d bytes stayed charged after every scope was reclaimed", used)
}
}
// The reclaim policy has ONE rule and no exception: a key that is still in use is
// never removed. This is the property that makes every bound above safe, so it is
// asserted directly on the table rather than inferred from behaviour above it.
func TestTable_ReclaimTakesOnlyDeadKeys(t *testing.T) {
b := budget{max: MaxBytes}
tab := newTable[*caller](4, callerBytes, &b)
fresh, _ := tab.admit("fresh", t0, func() *caller { return &caller{} })
fresh.seen = t0
holder, _ := tab.admit("holder", t0, func() *caller { return &caller{} })
holder.seen = t0.Add(-time.Hour) // long idle...
holder.hold = Hold{Action: "block", Until: t0.Add(time.Hour)}
dead, _ := tab.admit("dead", t0, func() *caller { return &caller{} })
dead.seen = t0.Add(-time.Hour)
tab.swept = time.Time{}
tab.reclaim(t0)
if _, ok := tab.get("fresh"); !ok {
t.Error("a caller seen inside the window was reclaimed")
}
if _, ok := tab.get("holder"); !ok {
t.Error("a caller under a live verdict was reclaimed")
}
if _, ok := tab.get("dead"); ok {
t.Error("a caller that is neither recent nor held was kept")
}
if b.used != 2*callerBytes {
t.Errorf("budget = %d after reclaiming one of three entries, want %d", b.used, 2*callerBytes)
}
}
// A request with NO identity — no credential the boundary validated and no
// client address — has nothing to be counted against. Keying it under the empty
// address would file the entire internet in one row, make it look like the worst
// credential-stuffing run ever recorded, and let one verdict be held against
// everybody at once.
//
// This is not hypothetical. A TCP load balancer that terminates the connection
// without PROXY protocol in front of it leaves every internet request with no
// client address at all, which is exactly the shape here.
func TestTraffic_ARequestWithNoIdentityIsNotACaller(t *testing.T) {
tr := NewTraffic()
blind := Signal{Org: "", Presented: "junk", IP: "", Path: "/v1/models", Class: CredAnonymous}
var p Pattern
for i := 0; i < 50; i++ {
p = tr.Observe(blind, t0)
}
if p.Strain != StrainBlind {
t.Fatalf("strain = %q for a request with no identity, want %q", p.Strain, StrainBlind)
}
if p.Requests != 0 {
t.Fatalf("a request with no identity accumulated %d requests against something", p.Requests)
}
tr.mu.Lock()
n := len(tr.tenants[""].callers.m)
tr.mu.Unlock()
if n != 0 {
t.Fatalf("%d caller rows were opened for traffic with no identity", n)
}
// Nothing can be held against it, so no verdict can be enforced on everyone.
tr.Hold(blind, Hold{Action: "block"}, time.Minute, t0)
if _, ok := tr.Held(blind, t0); ok {
t.Fatal("a verdict was held against every unidentifiable caller at once")
}
// And the volume is still visible, named as what it is.
v := tr.View("", ModeLive, t0)
if v.Requests != 50 || v.Blind != 50 {
t.Fatalf("blind traffic must be counted as traffic: requests=%d blind=%d, want 50 and 50", v.Requests, v.Blind)
}
if v.Strain != StrainBlind {
t.Fatalf("view strain = %q, want %q — a sensor that cannot see must say so", v.Strain, StrainBlind)
}
// A caller that HAS an address is unaffected: this is "no identity", not "no
// credential".
if q := tr.Observe(forged("", "junk", "203.0.113.4", "/v1/models"), t0); q.Strain != "" || q.Requests != 1 {
t.Fatalf("an addressed caller was swept into the blind state: %+v", q)
}
}
// The report is an expensive read on the same lock every request needs to be
// observed under, so the SCAN happens under the lock and the SORT does not.
// These two say what that costs: a full-table report, and the observation path
// it must not stall.
func BenchmarkViewAtTheCeiling(b *testing.B) {
tr := NewTraffic()
for i := 0; i < maxCallers; i++ {
tr.Observe(sig("acme", fmt.Sprintf("fp%06d", i), "203.0.113.1", "/v1/models"), t0)
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
tr.View("acme", ModeLive, t0)
}
}
func BenchmarkObserveAtTheCeiling(b *testing.B) {
tr := NewTraffic()
for i := 0; i < maxCallers; i++ {
tr.Observe(sig("acme", fmt.Sprintf("fp%06d", i), "203.0.113.1", "/v1/models"), t0)
}
s := sig("acme", "fp000001", "203.0.113.1", "/v1/models")
b.ResetTimer()
for i := 0; i < b.N; i++ {
tr.Observe(s, t0)
}
}
apps/gateway/edge/lane_test.go
+39
View File
@@ -0,0 +1,39 @@
package edge
// The lane table IS the specification of the differentiator. It is a pure
// function of two values, so this table is total: there is no third input, no
// clock and no I/O that could make the classification depend on anything else.
import "testing"
func TestLane(t *testing.T) {
quiet := Pattern{Requests: 3, Paths: 2}
stuffing := Pattern{Requests: 40, Peers: stuffPeers}
guessing := Pattern{Requests: 40, Failures: guessFailures}
sweeping := Pattern{Requests: 40, Paths: sweepPaths}
cases := []struct {
class string
p Pattern
want string
why string
}{
{CredSecret, quiet, AgencyAgent, "a machine credential we issued is attributable — that is the agent lane"},
{CredSecret, sweeping, AgencyAgent, "an agent walking many endpoints is an agent doing its job, not a scraper"},
{CredSecret, guessing, AgencyAgent, "a failing agent key is still an agent key; the pattern is the scorer's to weigh"},
{CredSession, quiet, AgencyHuman, "a browser session is a person"},
{CredSession, sweeping, AgencyHuman, "a busy person is still a person"},
{CredAnonymous, quiet, AgencyUnknown, "anonymous is not malicious — every new integration starts here"},
{CredPublishable, quiet, AgencyUnknown, "a publishable key names a tenant, not a caller; unremarkable use is unremarkable"},
{CredAnonymous, stuffing, AgencyBot, "one address, many credentials, no attribution: stuffing"},
{CredAnonymous, guessing, AgencyBot, "a wall of refusals from an unattributable caller: guessing"},
{CredAnonymous, sweeping, AgencyBot, "an unattributable caller walking the map: scraping"},
{CredPublishable, stuffing, AgencyBot, "a copied publishable key behaves the same way and is classed the same way"},
{"", quiet, AgencyUnknown, "an unrecognised class is unknown, never a lane with consequences"},
}
for _, tc := range cases {
if got := Lane(tc.class, tc.p); got != tc.want {
t.Errorf("Lane(%q, %+v) = %q, want %q — %s", tc.class, tc.p, got, tc.want, tc.why)
}
}
}
apps/gateway/edge/pin_test.go
+188
View File
@@ -0,0 +1,188 @@
package edge
// Pins. Each of these asserts a property of the SOURCE, because each is a defect
// that came back by being re-spelled somewhere else after it was fixed in one
// place. A behavioural test catches the instance; these catch the class.
import (
"go/ast"
"go/parser"
"go/token"
"os"
"path/filepath"
"strings"
"testing"
)
// A bounded table is the only map in this package that may hold per-caller state,
// and it can only be built by newTenant — which supplies the ceiling, the
// per-entry cost and the process budget. A map literal built anywhere else is an
// unbounded one wearing the same shape, which is exactly how the shared,
// globally-capped table this design replaced came to exist.
func TestPin_EveryTableIsBuiltByOneConstructor(t *testing.T) {
for file, src := range sources(t) {
if strings.HasSuffix(file, "_test.go") {
continue
}
ast.Inspect(src, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
if name := callName(call); name == "newTable" || name == "newTable[V]" {
if fn := enclosing(src, call.Pos()); fn != "newTenant" {
t.Errorf("%s: newTable is called from %s; the only constructor is newTenant", file, fn)
}
}
return true
})
}
}
// The reclaim rule has ONE exit that deletes anything, and it deletes only what
// is dead. sort was the machinery of the second pass — "drop the oldest half" —
// which is what turned a memory bound into a way to evict a live caller, release
// a held verdict, and erase a neighbour's counts. If sorting reappears in the
// reclaim path, so has that pass.
func TestPin_ReclaimHasNoSecondPass(t *testing.T) {
src, err := os.ReadFile("traffic.go")
if err != nil {
t.Fatal(err)
}
body, ok := function(string(src), "func (t *table[V]) reclaim(")
if !ok {
t.Fatal("table.reclaim is gone; the reclaim policy moved and this pin did not")
}
for _, banned := range []string{"sort.", "len(all)/2", "pinned("} {
if strings.Contains(body, banned) {
t.Errorf("table.reclaim contains %q: the drop-the-oldest pass is back", banned)
}
}
if strings.Count(body, "delete(") != 1 {
t.Errorf("table.reclaim deletes from %d places; the rule is one place, dead keys only", strings.Count(body, "delete("))
}
if !strings.Contains(body, "!v.live(now)") {
t.Error("table.reclaim no longer tests liveness before deleting")
}
}
// A caller key is derived from ONE function, and that function may not read the
// credential a request merely PRESENTED. Signal.Presented is the value the caller
// picks; the day it reaches a key, holds become evadable and the table becomes
// mintable — which is the defect this whole file was rewritten for.
func TestPin_TheCallerKeyCannotSeeAPresentedCredential(t *testing.T) {
src, err := os.ReadFile("traffic.go")
if err != nil {
t.Fatal(err)
}
body, ok := function(string(src), "func callerKey(")
if !ok {
t.Fatal("callerKey is gone; the key derivation moved and this pin did not")
}
if strings.Contains(body, "Presented") {
t.Error("callerKey reads Signal.Presented — a caller may not choose its own key")
}
// And nowhere else derives one.
for file, text := range texts(t) {
if strings.HasSuffix(file, "_test.go") || file == "traffic.go" {
continue
}
if strings.Contains(text, `"cred:"`) || strings.Contains(text, `"ip:"`) {
t.Errorf("%s spells a caller key; there is one derivation, in callerKey", file)
}
}
}
// Every ceiling in this package is charged in BYTES against the one budget. A cap
// on the NUMBER of entries is not a bound when the values behind them are not
// bounded, so a table that admits without a cost is a table with no ceiling.
func TestPin_EveryCeilingIsChargedInBytes(t *testing.T) {
src, err := os.ReadFile("traffic.go")
if err != nil {
t.Fatal(err)
}
body, ok := function(string(src), "func (t *table[V]) admit(")
if !ok {
t.Fatal("table.admit is gone; admission moved and this pin did not")
}
if !strings.Contains(body, "t.budget.take(t.cost)") {
t.Error("table.admit no longer charges the budget: the byte bound is not enforced at admission")
}
if !strings.Contains(body, "t.refused.bump(") {
t.Error("table.admit no longer counts a refusal: a bound that binds silently is banned")
}
}
func sources(t *testing.T) map[string]*ast.File {
t.Helper()
out := map[string]*ast.File{}
fset := token.NewFileSet()
names, err := filepath.Glob("*.go")
if err != nil {
t.Fatal(err)
}
for _, name := range names {
f, err := parser.ParseFile(fset, name, nil, 0)
if err != nil {
t.Fatalf("%s: %v", name, err)
}
out[name] = f
}
return out
}
func texts(t *testing.T) map[string]string {
t.Helper()
out := map[string]string{}
names, err := filepath.Glob("*.go")
if err != nil {
t.Fatal(err)
}
for _, name := range names {
b, err := os.ReadFile(name)
if err != nil {
t.Fatal(err)
}
out[name] = string(b)
}
return out
}
func callName(call *ast.CallExpr) string {
switch fn := call.Fun.(type) {
case *ast.Ident:
return fn.Name
case *ast.IndexExpr: // newTable[*caller](...)
if id, ok := fn.X.(*ast.Ident); ok {
return id.Name
}
case *ast.SelectorExpr:
return fn.Sel.Name
}
return ""
}
func enclosing(f *ast.File, pos token.Pos) string {
name := "(file scope)"
for _, d := range f.Decls {
fn, ok := d.(*ast.FuncDecl)
if ok && fn.Pos() <= pos && pos <= fn.End() {
name = fn.Name.Name
}
}
return name
}
// function returns the body text of the declaration starting with head, from its
// opening brace to the first line that closes it at column zero.
func function(src, head string) (string, bool) {
i := strings.Index(src, head)
if i < 0 {
return "", false
}
rest := src[i:]
if j := strings.Index(rest, "\n}\n"); j >= 0 {
return rest[:j], true
}
return rest, true
}
apps/gateway/edge/policy.go
+84 -3
View File
@@ -75,6 +75,24 @@ type Policy struct {
// Methods is the allowlist of HTTP methods the edge accepts for this org. Empty
// means all are accepted.
Methods []string `json:"methods,omitempty"`
// Mode is the abuse gate's posture for THIS scope: "shadow" scores traffic and
// records the verdict without acting on it, "live" enforces it. Unset means
// shadow.
//
// It is the one per-org field that does NOT inherit. Every other field here
// layers a platform default under the org's own value, which is right for a
// default: a tenant that sets no rate ceiling should get the platform's. Mode
// is not a default, it is an ARMING DECISION — it is what makes a statistical
// judgement start refusing real traffic — and inheriting it means arming one
// scope arms every tenant that never asked for it, without a write to their
// row and without anything in their config changing. So a tenant is live only
// if that tenant's OWN row says live, and the platform row's mode governs
// exactly one scope: the anonymous lane, which has no tenant of its own.
//
// It is also not self-service. Writing it requires SuperAdmin (see the
// /v1/gateway config op): the subject of an abuse control does not get to
// switch the control off.
Mode string `json:"mode,omitempty"`
// UpdatedAt is the unix second this policy row was last written. Server-stamped;
// a client-supplied value is ignored.
@@ -131,9 +149,22 @@ func (p Policy) Validate() error {
return fmt.Errorf("cors_origins must not contain empty entries")
}
}
if p.Mode != "" && p.Mode != ModeShadow && p.Mode != ModeLive {
return fmt.Errorf("mode must be %q or %q", ModeShadow, ModeLive)
}
return nil
}
// The abuse gate's two postures. They are values, not booleans, because "off"
// and "watching" are different states and a deployment must be able to tell
// which one it is in.
const (
// ModeShadow scores and records; it never refuses. The DEFAULT.
ModeShadow = "shadow"
// ModeLive enforces the scorer's action.
ModeLive = "live"
)
// merge overlays the non-zero fields of over onto base and returns the result.
// A nil/empty slice or zero int in over means "keep base"; this is what makes a
// partial PUT (e.g. only OrgRPM) additive rather than a full replace.
@@ -160,6 +191,9 @@ func merge(base, over Policy) Policy {
if len(over.Methods) > 0 {
out.Methods = over.Methods
}
if over.Mode != "" {
out.Mode = over.Mode
}
if over.UpdatedAt > 0 {
out.UpdatedAt = over.UpdatedAt
out.UpdatedBy = over.UpdatedBy
@@ -296,10 +330,52 @@ func (s *Store) PutPlatform(ctx context.Context, p Policy) (Policy, error) {
func (s *Store) Effective(org string) Policy {
p := s.Platform()
eo := s.effectiveOrg(org)
p.OrgRPM, p.CacheTTLSec, p.CachePaths, p.Methods = eo.OrgRPM, eo.CacheTTLSec, eo.CachePaths, eo.Methods
p.OrgRPM, p.CacheTTLSec, p.CachePaths, p.Methods, p.Mode = eo.OrgRPM, eo.CacheTTLSec, eo.CachePaths, eo.Methods, eo.Mode
return p
}
// Mode returns the abuse gate's posture for org: THAT ORG'S OWN ROW, else shadow.
// Cached with the same short TTL as every other per-org resolver, and fail-soft to
// SHADOW — a policy-store outage must not be the reason a tenant starts being
// refused.
//
// IT DOES NOT INHERIT, and that is the point. It used to fall back to the platform
// row, so arming the one lane that has no tenant — the anonymous lane, which is
// where a bad bot calls from — armed every tenant in the estate at the same time:
// one PUT, and a statistical judgement began enforcing against customers whose own
// config still said nothing and whose operators were never asked. An arming
// decision that reaches a tenant it was not written for is not a default, it is an
// accident waiting for a scorer to have a bad day.
//
// An EMPTY org is the anonymous lane, and it resolves to the PLATFORM row — which
// is that lane's OWN row, not an inherited one: a caller with no tenant still has
// to be governed by something, and the platform scope is what governs a request
// that has no tenant at evaluation time. A SuperAdmin arms it by targeting the
// reserved admin org (PUT /v1/gateway/config?org=<adminOrg> {"mode":"live"}), since
// the admin org's row IS the platform row. One mechanism, one scope per write.
func (s *Store) Mode(org string) string {
var m string
if org == "" {
m = s.Platform().Mode
} else {
m = s.orgMode(org)
}
if m == ModeLive {
return ModeLive
}
return ModeShadow
}
// orgMode reads the org's OWN stored mode and nothing else — no platform layer,
// no static default. Cached under the same per-org entry every other resolver
// shares, and fail-soft to "" (⇒ shadow).
func (s *Store) orgMode(org string) string {
if s == nil || org == "" {
return ""
}
return s.effectiveOrg(org).Mode
}
func (s *Store) invalidate() {
s.mu.Lock()
s.cache = map[string]cacheEntry{}
@@ -323,10 +399,15 @@ func (s *Store) Platform() Policy {
}
// effectiveOrg resolves org's per-org edge config — its own row overlaid on the
// platform per-org defaults — cached under key=org with a short TTL, fail-open to
// platform per-org DEFAULTS — cached under key=org with a short TTL, fail-open to
// the platform defaults. Every per-org resolver (OrgRPM / CacheTTL / Methods) and
// the Effective read-back share this ONE value, so an org has exactly one resolved
// config and one cache entry.
//
// Mode is deliberately absent from the inherited base: it is an arming decision,
// not a default (see Policy.Mode and Store.Mode). The org's own row is its only
// source, so the value below is "" — hence shadow — for every org that has not
// been armed by name.
func (s *Store) effectiveOrg(org string) Policy {
if s == nil || org == "" {
return Policy{}
@@ -338,7 +419,7 @@ func (s *Store) effectiveOrg(org string) Policy {
if err != nil || !ok {
return base
}
return merge(base, Policy{OrgRPM: row.OrgRPM, CacheTTLSec: row.CacheTTLSec, CachePaths: row.CachePaths, Methods: row.Methods})
return merge(base, Policy{OrgRPM: row.OrgRPM, CacheTTLSec: row.CacheTTLSec, CachePaths: row.CachePaths, Methods: row.Methods, Mode: row.Mode})
})
}
apps/gateway/edge/traffic.go
+1165
View File
File diff suppressed because it is too large Load Diff
apps/gateway/edge/traffic_test.go
+514
View File
@@ -0,0 +1,514 @@
package edge
// The sensor's own contract, tested where it lives: counts roll, holds expire,
// memory is bounded, and a tenant cannot be reached from outside itself.
import (
"fmt"
"strings"
"sync"
"testing"
"time"
)
var t0 = time.Date(2026, 8, 1, 12, 0, 0, 0, time.UTC)
// sig is a VALIDATED caller: the identity boundary resolved the credential, so
// the fingerprint is both what was presented and what the sensor may key on.
// An empty cred is a caller that presented nothing at all.
func sig(org, cred, ip, path string) Signal {
s := Signal{Org: org, Cred: cred, Presented: cred, IP: ip, Path: path, Class: CredSecret}
if cred == "" {
s.Class = CredAnonymous
}
return s
}
// forged is a caller that PRESENTED a credential the identity boundary refused.
// It is the same shape a stolen-key probe, a stuffing run and a bad bot all have,
// and the fingerprint it presents is a value it picks per request.
func forged(org, presented, ip, path string) Signal {
return Signal{Org: org, Presented: presented, IP: ip, Path: path, Class: CredAnonymous}
}
func TestTraffic_CountsPerCredential(t *testing.T) {
tr := NewTraffic()
var p Pattern
for i := 0; i < 7; i++ {
p = tr.Observe(sig("acme", "fp1", "203.0.113.1", "/v1/models"), t0)
}
if p.Requests != 7 {
t.Fatalf("requests = %d, want 7", p.Requests)
}
// A SECOND credential in the same org and from the same address has its own
// count — this is the fact neither existing limiter can express.
if q := tr.Observe(sig("acme", "fp2", "203.0.113.1", "/v1/models"), t0); q.Requests != 1 {
t.Fatalf("a second credential's count = %d, want 1", q.Requests)
}
}
func TestTraffic_CountsPathSpreadAndPeerSpread(t *testing.T) {
tr := NewTraffic()
var p Pattern
for i := 0; i < 12; i++ {
p = tr.Observe(sig("acme", "fp1", "203.0.113.1", fmt.Sprintf("/v1/thing/%d", i)), t0)
}
if p.Paths < 8 {
// The spread word saturates and can collide, so the assertion is a floor,
// not equality — an approximate count must not be tested as an exact one.
t.Fatalf("path spread = %d after 12 distinct paths, want at least 8", p.Paths)
}
tr2 := NewTraffic()
var q Pattern
for i := 0; i < 10; i++ {
q = tr2.Observe(sig("acme", fmt.Sprintf("fp%d", i), "198.51.100.9", "/v1/models"), t0)
}
if q.Peers < 6 {
t.Fatalf("peer spread = %d after 10 credentials from one address, want at least 6", q.Peers)
}
}
// An anonymous stream from one address is a flood, not stuffing: with no
// credential presented there is no peer to count, and saying otherwise would let
// one anonymous client manufacture a stuffing signature by itself.
func TestTraffic_AnonymousTrafficHasNoPeers(t *testing.T) {
tr := NewTraffic()
var p Pattern
for i := 0; i < 20; i++ {
p = tr.Observe(sig("", "", "198.51.100.9", "/v1/models"), t0)
}
if p.Peers != 0 {
t.Fatalf("peers = %d for credential-less traffic, want 0", p.Peers)
}
if p.Requests != 20 {
t.Fatalf("requests = %d, want 20 — anonymous traffic is still counted", p.Requests)
}
}
func TestTraffic_CountsRollOutOfTheWindow(t *testing.T) {
tr := NewTraffic()
for i := 0; i < 5; i++ {
tr.Observe(sig("acme", "fp1", "203.0.113.1", "/v1/models"), t0)
}
later := t0.Add(2 * window)
if p := tr.Observe(sig("acme", "fp1", "203.0.113.1", "/v1/models"), later); p.Requests != 1 {
t.Fatalf("requests = %d two windows later, want 1 — the ring did not roll", p.Requests)
}
}
func TestTraffic_HoldsExpireAndAreCapped(t *testing.T) {
tr := NewTraffic()
s := sig("acme", "fp1", "203.0.113.1", "/v1/models")
if _, ok := tr.Held(s, t0); ok {
t.Fatal("nothing is held before anything is held")
}
tr.Hold(s, Hold{Action: "block", Reason: "peers", Decision: "d-1"}, 30*time.Second, t0)
h, ok := tr.Held(s, t0.Add(10*time.Second))
if !ok || h.Action != "block" || h.Decision != "d-1" {
t.Fatalf("held verdict = %+v ok=%v", h, ok)
}
if _, ok := tr.Held(s, t0.Add(31*time.Second)); ok {
t.Fatal("a hold must lapse so a false positive clears itself")
}
// A caller asking for a week gets the cap. Enforcement without a fresh
// judgement is bounded, always.
tr.Hold(s, Hold{Action: "block"}, 7*24*time.Hour, t0)
if _, ok := tr.Held(s, t0.Add(holdCap+time.Second)); ok {
t.Fatalf("a hold longer than %s must be clamped", holdCap)
}
tr.Hold(s, Hold{Action: "block"}, time.Minute, t0)
tr.Release(s)
if _, ok := tr.Held(s, t0.Add(time.Second)); ok {
t.Fatal("Release must drop a held verdict — the operator's undo")
}
}
// A hold is keyed on (org, credential). One org's hold must be unreachable from
// another's traffic even when the credential string and the address are identical.
func TestTraffic_HoldsDoNotCrossTenants(t *testing.T) {
tr := NewTraffic()
a := sig("acme", "fp1", "203.0.113.1", "/v1/models")
b := sig("globex", "fp1", "203.0.113.1", "/v1/models")
tr.Hold(a, Hold{Action: "block"}, time.Minute, t0)
if _, ok := tr.Held(b, t0); ok {
t.Fatal("acme's hold reached globex — the tenant boundary is not in the key")
}
}
func TestTraffic_ViewIsScopedToOneTenant(t *testing.T) {
tr := NewTraffic()
tr.Observe(sig("acme", "fpA", "203.0.113.1", "/v1/models"), t0)
tr.Observe(sig("acme", "fpA", "203.0.113.1", "/v1/models"), t0)
tr.Observe(sig("globex", "fpB", "203.0.113.2", "/v1/models"), t0)
v := tr.View("acme", ModeShadow, t0)
if v.Org != "acme" || v.Requests != 2 {
t.Fatalf("acme view = %+v, want org=acme requests=2", v)
}
if len(v.Callers) != 1 || v.Callers[0].Cred != "fpA" {
t.Fatalf("acme must see exactly its own caller: %+v", v.Callers)
}
blob := fmt.Sprintf("%+v", v)
if strings.Contains(blob, "fpB") || strings.Contains(blob, "globex") {
t.Fatalf("another tenant appeared in acme's view: %s", blob)
}
// An org that never called sees an empty, well-formed view — never another
// org's rows, and never an error that would confirm the other org exists.
if e := tr.View("stranger", ModeShadow, t0); e.Requests != 0 || len(e.Callers) != 0 {
t.Fatalf("an unknown org must see nothing, got %+v", e)
}
}
// The org prefix must be matched as a WHOLE segment. Without the separator,
// "acme" would be a prefix of "acme-corp" and one tenant would read another's
// rows — the classic prefix-matching tenancy bug.
func TestTraffic_ViewDoesNotLeakToAPrefixNeighbour(t *testing.T) {
tr := NewTraffic()
tr.Observe(sig("acme-corp", "fpX", "203.0.113.3", "/v1/models"), t0)
if v := tr.View("acme", ModeShadow, t0); len(v.Callers) != 0 {
t.Fatalf("acme read acme-corp's rows: %+v", v.Callers)
}
}
func TestTraffic_ViewIsBounded(t *testing.T) {
tr := NewTraffic()
for i := 0; i < maxViewCallers*3; i++ {
for n := 0; n <= i; n++ { // give each caller a distinct request count
tr.Observe(sig("acme", fmt.Sprintf("fp%04d", i), "203.0.113.1", "/v1/models"), t0)
}
}
v := tr.View("acme", ModeShadow, t0)
if len(v.Callers) != maxViewCallers {
t.Fatalf("view returned %d callers, want the %d busiest", len(v.Callers), maxViewCallers)
}
for i := 1; i < len(v.Callers); i++ {
if v.Callers[i-1].Requests < v.Callers[i].Requests {
t.Fatal("the view must be ordered busiest first")
}
}
}
// The table must not grow without bound under an adversary minting credentials —
// and the bound is the TENANT'S OWN, so it is that tenant's table that stops
// growing.
func TestTraffic_TableIsBounded(t *testing.T) {
tr := NewTraffic()
now := t0
for i := 0; i < maxCallers+5_000; i++ {
if i%1000 == 0 {
now = now.Add(2 * window) // let dead keys age out
}
tr.Observe(sig("acme", fmt.Sprintf("fp%06d", i), "203.0.113.1", "/v1/models"), now)
}
tr.mu.Lock()
n := len(tr.tenants["acme"].callers.m)
tr.mu.Unlock()
if n > maxCallers {
t.Fatalf("acme holds %d caller keys, past its %d ceiling", n, maxCallers)
}
// And the ceiling must actually BIND: a run this long has to have reached it,
// or the test is asserting a bound that was never tested.
tr2 := NewTraffic()
for i := 0; i < maxCallers+1_000; i++ {
tr2.Observe(sig("acme", fmt.Sprintf("fp%06d", i), "203.0.113.1", "/v1/models"), t0)
}
tr2.mu.Lock()
n2 := len(tr2.tenants["acme"].callers.m)
tr2.mu.Unlock()
if n2 != maxCallers {
t.Fatalf("acme holds %d live caller keys against a %d ceiling: the bound never bound", n2, maxCallers)
}
}
// REGRESSION — the systemic defect. A process-wide cap over a table every tenant
// shares is a cross-tenant denial of service: the org that fills it evicts the
// org that was quiet, and the victim's abuse controls go silent with no error and
// no alert.
//
// The assertion is the property, not the number: one org drives its OWN table
// past its ceiling for several windows, and a second org's state — its counts and
// its held verdict — is untouched throughout. A shared LRU fails this; a per-org
// keyspace with a per-org bound cannot.
func TestTraffic_OneTenantCannotEvictAnother(t *testing.T) {
tr := NewTraffic()
now := t0
victim := sig("victim", "fpVictim", "198.51.100.7", "/v1/models")
tr.Observe(victim, now)
tr.Observe(victim, now)
tr.Hold(victim, Hold{Action: "block", Reason: "stuffing"}, time.Minute, now)
// The noisy tenant mints far more credentials than its own table holds, and
// keeps doing it across windows so an LRU would have every chance to reach for
// somebody else's keys.
for i := 0; i < maxCallers*2; i++ {
tr.Observe(sig("noisy", fmt.Sprintf("fp%07d", i), "203.0.113.9", "/v1/chat"), now)
}
if h, ok := tr.Held(victim, now); !ok || h.Action != "block" {
t.Fatalf("the victim's verdict was evicted by another tenant's traffic: %+v ok=%v", h, ok)
}
v := tr.View("victim", ModeShadow, now)
if v.Requests != 2 || len(v.Callers) != 1 {
t.Fatalf("the victim's own counts moved: %+v", v)
}
if v.Refused != 0 || v.Strain != StrainClear {
t.Fatalf("the victim is reported strained by another tenant's traffic: %+v", v)
}
// The noisy tenant degraded ITSELF, and says so rather than going quiet.
if n := tr.View("noisy", ModeShadow, now); n.Refused == 0 || n.Strain != StrainRefuse {
t.Fatalf("a tenant at its own ceiling must report it, got strain=%q refused=%d", n.Strain, n.Refused)
}
}
// The ADDRESS table is the one an unauthenticated flood lands in, and it is the
// one a forged X-Forwarded-For used to feed without any ceiling at all. It is
// bounded per tenant like the credential table, and the anonymous lane is one
// tenant — so the worst an address flood can do is fill the lane it arrived in.
func TestTraffic_HostTableIsBoundedPerTenant(t *testing.T) {
tr := NewTraffic()
now := t0
for i := 0; i < maxHosts+2_000; i++ {
// A distinct address AND a credential, which is what opens a host row.
tr.Observe(Signal{Org: "", Cred: "fp", Presented: "fp", Class: CredSecret,
IP: fmt.Sprintf("198.51.%d.%d", i/256%256, i%256), Path: "/v1/models"}, now)
}
tr.mu.Lock()
n := len(tr.tenants[""].hosts.m)
tr.mu.Unlock()
if n > maxHosts {
t.Fatalf("the anonymous lane holds %d address keys, past its %d ceiling", n, maxHosts)
}
if n != maxHosts {
t.Fatalf("the address ceiling never bound: %d of %d after a flood", n, maxHosts)
}
}
// A live verdict is enforcement. Reclaiming it to make room would turn a memory
// bound into a security bypass — the caller the scorer refused would be released
// by nothing more than the table filling up.
func TestTraffic_ReclaimNeverDropsALiveHold(t *testing.T) {
tr := NewTraffic()
now := t0
held := sig("acme", "fpHeld", "203.0.113.1", "/v1/models")
tr.Hold(held, Hold{Action: "block"}, time.Minute, now)
for i := 0; i < maxCallers*2; i++ {
tr.Observe(sig("acme", fmt.Sprintf("fp%07d", i), "203.0.113.1", "/v1/chat"), now)
}
if _, ok := tr.Held(held, now); !ok {
t.Fatal("a live verdict was reclaimed by the tenant's own bound")
}
}
// The tenant table is bounded too, and it is swept by IDLENESS — never to make
// room for whoever is calling now, which would be the same cross-tenant eviction
// one level up.
func TestTraffic_IdleTenantsAreReclaimed(t *testing.T) {
tr := NewTraffic()
tr.Observe(sig("gone", "fp1", "203.0.113.1", "/v1/models"), t0)
tr.mu.Lock()
tr.sweepLocked(t0.Add(tenantIdle + time.Minute))
_, still := tr.tenants["gone"]
used := tr.budget.used
tr.mu.Unlock()
if still {
t.Fatal("a tenant idle for longer than any verdict can live was kept")
}
if used != 0 {
t.Fatalf("a reclaimed tenant left %d bytes charged against the budget", used)
}
// And an ACTIVE tenant is not swept, whatever else is happening.
tr2 := NewTraffic()
tr2.Observe(sig("live", "fp1", "203.0.113.1", "/v1/models"), t0)
tr2.mu.Lock()
tr2.sweepLocked(t0.Add(bucketSpan + time.Second)) // past the sweep cadence, far short of idle
_, kept := tr2.tenants["live"]
tr2.mu.Unlock()
if !kept {
t.Fatal("an active tenant was swept")
}
}
func TestTraffic_FailCountsOnlyAfterTheCallerIsKnown(t *testing.T) {
tr := NewTraffic()
s := sig("acme", "fp1", "203.0.113.1", "/v1/models")
tr.Fail(s, t0) // no Observe yet: nothing to attribute it to.
if p := tr.Observe(s, t0); p.Failures != 0 {
t.Fatalf("failures = %d before the caller was seen, want 0", p.Failures)
}
tr.Fail(s, t0)
tr.Fail(s, t0)
if p := tr.Observe(s, t0); p.Failures != 2 {
t.Fatalf("failures = %d, want 2", p.Failures)
}
}
// Every method takes the one lock; -race is the whole assertion.
func TestTraffic_IsRaceFree(t *testing.T) {
tr := NewTraffic()
var wg sync.WaitGroup
for g := 0; g < 8; g++ {
wg.Add(1)
go func(g int) {
defer wg.Done()
s := sig(fmt.Sprintf("org%d", g%3), fmt.Sprintf("fp%d", g), "203.0.113.1", "/v1/models")
for i := 0; i < 200; i++ {
tr.Observe(s, time.Now())
tr.Fail(s, time.Now())
tr.Hold(s, Hold{Action: "block"}, time.Second, time.Now())
tr.Held(s, time.Now())
tr.Deny(s.Org, time.Now())
tr.View(s.Org, ModeLive, time.Now())
tr.Release(s)
}
}(g)
}
wg.Wait()
}
func TestPolicy_ModeDefaultsToShadowAndIsValidated(t *testing.T) {
if err := (Policy{Mode: "enforce"}).Validate(); err == nil {
t.Fatal("an unknown mode must be refused rather than silently ignored")
}
for _, m := range []string{"", ModeShadow, ModeLive} {
if err := (Policy{Mode: m}).Validate(); err != nil {
t.Fatalf("mode %q must be accepted: %v", m, err)
}
}
s, err := New(t.TempDir(), "admin", Policy{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer func() { _ = s.Close() }()
if got := s.Mode("acme"); got != ModeShadow {
t.Fatalf("an unconfigured org must be %q, got %q", ModeShadow, got)
}
if _, err := s.Put(t.Context(), "acme", Policy{Mode: ModeLive}); err != nil {
t.Fatalf("Put: %v", err)
}
if got := s.Mode("acme"); got != ModeLive {
t.Fatalf("acme = %q after arming, want %q", got, ModeLive)
}
// Arming one tenant must not arm another.
if got := s.Mode("globex"); got != ModeShadow {
t.Fatalf("globex = %q, want %q — arming crossed the tenant boundary", got, ModeShadow)
}
// A nil store is shadow, not live: an unwired deployment never enforces.
var nilStore *Store
if got := nilStore.Mode("acme"); got != ModeShadow {
t.Fatalf("a nil store must resolve to %q, got %q", ModeShadow, got)
}
}
// A screen is the billable unit of the risk product. It must be visible to the
// org from the first request — before anyone decides what one costs — so it is
// counted here as well as metered, and the two are independent.
func TestTraffic_CountsScreensPerTenant(t *testing.T) {
tr := NewTraffic()
tr.Observe(sig("acme", "fp1", "203.0.113.1", "/v1/models"), t0)
tr.Screen("acme", "", t0)
tr.Screen("acme", "", t0)
tr.Observe(sig("globex", "fp2", "203.0.113.2", "/v1/models"), t0)
if v := tr.View("acme", ModeLive, t0); v.Screens != 2 {
t.Fatalf("acme screens = %d, want 2", v.Screens)
}
if v := tr.View("globex", ModeLive, t0); v.Screens != 0 {
t.Fatalf("globex screens = %d, want 0 — screens crossed the tenant boundary", v.Screens)
}
// A screen is not a request: counting it in both would make an org's own
// report say it sent more traffic than it did.
if v := tr.View("acme", ModeLive, t0); v.Requests != 1 {
t.Fatalf("acme requests = %d, want 1", v.Requests)
}
}
// The anonymous lane has no org, and it is the lane a bad bot calls from. It must
// resolve to the PLATFORM row — otherwise the one lane the gate exists for could
// never be armed, because there would be no org to arm.
func TestPolicy_TheAnonymousLaneResolvesToThePlatformRow(t *testing.T) {
s, err := New(t.TempDir(), "admin", Policy{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer func() { _ = s.Close() }()
if got := s.Mode(""); got != ModeShadow {
t.Fatalf("anonymous starts %q, want %q", got, ModeShadow)
}
// Arming a TENANT must not arm the anonymous lane.
if _, err := s.Put(t.Context(), "acme", Policy{Mode: ModeLive}); err != nil {
t.Fatalf("Put: %v", err)
}
if got := s.Mode(""); got != ModeShadow {
t.Fatalf("arming acme armed the anonymous lane too: %q", got)
}
// Arming the reserved admin org — which IS the platform row — arms it.
if _, err := s.PutPlatform(t.Context(), Policy{Mode: ModeLive}); err != nil {
t.Fatalf("PutPlatform: %v", err)
}
if got := s.Mode(""); got != ModeLive {
t.Fatalf("anonymous = %q after arming the platform row, want %q", got, ModeLive)
}
}
// REGRESSION — arming one scope must not arm another. Mode used to be seeded from
// the platform row for every per-org resolution, so a single PUT against the
// reserved admin org (the ONLY way to arm the anonymous lane) silently switched
// every tenant in the estate to enforcing: no write to their row, nothing changed
// in their config, and a statistical judgement began refusing their traffic.
//
// Arming is a decision per scope. The platform row governs the anonymous lane;
// a tenant is live only if that tenant's own row says so.
func TestPolicy_ArmingThePlatformRowDoesNotArmTenants(t *testing.T) {
s, err := New(t.TempDir(), "admin", Policy{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer func() { _ = s.Close() }()
if _, err := s.PutPlatform(t.Context(), Policy{Mode: ModeLive}); err != nil {
t.Fatalf("PutPlatform: %v", err)
}
if got := s.Mode(""); got != ModeLive {
t.Fatalf("the anonymous lane did not arm: %q", got)
}
// An org that never asked to be armed is NOT armed — whether it has a row of
// its own or none at all.
if got := s.Mode("globex"); got != ModeShadow {
t.Fatalf("globex was armed by the platform row: %q", got)
}
if _, err := s.Put(t.Context(), "acme", Policy{OrgRPM: 120}); err != nil {
t.Fatalf("Put: %v", err)
}
if got := s.Mode("acme"); got != ModeShadow {
t.Fatalf("acme was armed by the platform row: %q", got)
}
// The read-back a tenant sees agrees with the mode actually in force for it.
if eff := s.Effective("acme"); eff.Mode == ModeLive {
t.Fatalf("Effective reports acme live while Mode says shadow: %+v", eff)
}
// Arming a tenant by name still works, and reaches only that tenant.
if _, err := s.Put(t.Context(), "acme", Policy{Mode: ModeLive}); err != nil {
t.Fatalf("Put: %v", err)
}
if got := s.Mode("acme"); got != ModeLive {
t.Fatalf("acme = %q after being armed by name, want %q", got, ModeLive)
}
if got := s.Mode("globex"); got != ModeShadow {
t.Fatalf("arming acme reached globex: %q", got)
}
}
apps/gateway/gateway.go
+114 -15
View File
@@ -27,6 +27,11 @@
// row of self-service edge config. An org admin writes its own (org from
// principal.Org, never a raw header); a SuperAdmin may target any tenant with
// ?org=<slug>.
// - MODE, the abuse gate's posture, lives on a tenant's row but is NOT
// self-service: a control's subject may not switch the control off, so writing
// it requires SuperAdmin whichever row it lands on. It is the one field on this
// surface whose scope (which org it applies to) and whose authority (who may
// set it) are different questions.
//
// The store is owned by BuildDeps (deps.GatewayPolicy) and shared; this subsystem
// does not open or close it (serve.go closes it once at shutdown), so there is one
@@ -37,6 +42,7 @@ package gateway
import (
"context"
"fmt"
"time"
"github.com/hanzoai/cloud"
"github.com/hanzoai/cloud/apps/gateway/edge"
@@ -53,7 +59,8 @@ import (
// state is gateway's own data; shared deps live in the embedded cloud.Base.
type state struct {
store *edge.Store
store *edge.Store
traffic *edge.Traffic
}
// ops binds the mounted Service so each op can be a method value — the only bound
@@ -73,7 +80,10 @@ func Mount(app cloud.Router, deps cloud.Deps) error {
if deps.GatewayPolicy == nil {
return fmt.Errorf("gateway.Mount: nil deps.GatewayPolicy")
}
s := &cloud.Service[state]{Base: cloud.NewBase(deps, "gateway"), State: state{store: deps.GatewayPolicy}}
s := &cloud.Service[state]{
Base: cloud.NewBase(deps, "gateway"),
State: state{store: deps.GatewayPolicy, traffic: deps.Traffic},
}
routes(app, s)
@@ -102,6 +112,10 @@ func routes(app cloud.Router, s *cloud.Service[state]) {
o := ops{s: s}
zip.Get(g, "/config", o.read)
zip.Put(g, "/config", o.write)
zip.Get(g, "/traffic", o.traffic,
zip.WithOperationID("gatewayTraffic"),
zip.WithSummary("Report who is calling this org's API right now"),
zip.WithTags("gateway"))
}
// noArgs is the input of an op that takes none: no body, no query, no path param.
@@ -125,19 +139,36 @@ func caller(ctx context.Context) (*zip.Ctx, string, error) {
return c, org, nil
}
// target resolves the org a call acts on: the caller's own validated org, or — for
// a SuperAdmin only — the tenant named by ?org=<slug>. The override is read off the
// target resolves the scope a call acts on: the caller's own validated org, or —
// for a SuperAdmin only — the scope named by ?org=. The override is read off the
// URL rather than modelled as an input field, because zip binds an input field from
// the BODY too and this route has never accepted an org there.
//
// A PRESENT but EMPTY ?org= names the scope that has no tenant — the anonymous
// lane, where every caller the identity boundary could not validate is counted.
// That lane had no spelling at all, so its traffic and its saturation were
// readable by nobody, SuperAdmin included: the only way to ask for a scope was to
// name it, and this one has no name. The empty value IS its name — it cannot
// collide with any tenant, and it needs no second query parameter to mean what
// "org" already means.
func target(c *zip.Ctx, org string) string {
if c.IsAdmin() {
if q := c.Query("org"); q != "" {
return q
}
if !c.IsAdmin() {
return org
}
if q := c.Query("org"); q != "" {
return q
}
if c.Fiber().Request().URI().QueryArgs().Has("org") {
return anonymousLane
}
return org
}
// anonymousLane is the scope with no tenant, in both the policy store and the
// sensor: the empty org. Named here so the op that serves it and the store that
// resolves its mode say the same thing.
const anonymousLane = ""
// Read returns the EFFECTIVE edge policy the caller is subject to: the platform CORS
// allowlist and pre-auth per-IP flood cap in force, plus the caller's own authenticated
// rate ceiling, edge-cache TTLs and accepted-method allowlist. A SuperAdmin may inspect
@@ -156,7 +187,9 @@ func (o ops) read(ctx context.Context, _ *noArgs) (*edge.Policy, error) {
// platform write and requires SuperAdmin; otherwise it is a per-org write (org_rpm,
// cache_ttl_sec, cache_paths, methods) scoped to the caller's own org — or, for a
// SuperAdmin, the tenant named by ?org=<slug>. A body that sets nothing is a 400.
// updated_at and updated_by are server-stamped; a client-supplied value is ignored.
// The abuse gate's mode is an OPERATOR field: setting it requires SuperAdmin,
// whichever organization it lands on. updated_at and updated_by are
// server-stamped; a client-supplied value is ignored.
//
// Example: {"org_rpm": 120, "cache_ttl_sec": 30, "methods": ["GET", "POST"]}
func (o ops) write(ctx context.Context, in *edge.Policy) (*edge.Policy, error) {
@@ -167,6 +200,23 @@ func (o ops) write(ctx context.Context, in *edge.Policy) (*edge.Policy, error) {
in.UpdatedBy = c.User() // server-stamped; a client-supplied value is ignored.
in.Normalize()
// MODE IS NOT SELF-SERVICE. Every other field on this route is a tenant's own
// preference about its own traffic: how fast it may call, what it caches, which
// methods it accepts. Mode is not a preference — it is whether the platform's
// abuse control ENFORCES against this tenant, and the tenant is the subject of
// that control. Leaving it in the self-service branch meant an org admin, or
// anyone holding an org-admin credential, could PUT {"mode":"shadow"} and turn
// the defense off for exactly the account it was defending against; a stolen
// key's first useful call is the one that disarms the thing watching it.
//
// Checked BEFORE the scope split, so it holds for the platform row and for a
// tenant row alike, and stated as its own refusal so the answer names the actual
// rule rather than "not found" or "nothing to set".
if in.Mode != "" && !c.IsAdmin() {
return nil, zip.ErrForbidden("mode is set by the platform, not by the organization it governs")
}
platformWrite := len(in.CORSOrigins) > 0 || in.PerIPRPM > 0 || in.WindowSec > 0
if platformWrite {
if !c.IsAdmin() {
@@ -186,22 +236,33 @@ func (o ops) write(ctx context.Context, in *edge.Policy) (*edge.Policy, error) {
}
// Per-org scope: the tenant's OWN edge config — rate ceiling, cache policy,
// method allowlist. Only the per-org fields are forwarded, so a tenant can never
// smuggle a platform knob into its own row. Scoped to the caller's org (never a
// body-supplied org); a SuperAdmin may target a specific tenant with ?org=<slug>.
// method allowlist, plus the operator-only mode already gated above. Only these
// fields are forwarded, so a tenant can never smuggle a platform knob into its
// own row. Scoped to the caller's org (never a body-supplied org); a SuperAdmin
// may target a specific tenant with ?org=<slug>.
orgCfg := edge.Policy{
OrgRPM: in.OrgRPM,
CacheTTLSec: in.CacheTTLSec,
CachePaths: in.CachePaths,
Methods: in.Methods,
Mode: in.Mode,
UpdatedBy: in.UpdatedBy,
}
if orgCfg.OrgRPM <= 0 && orgCfg.CacheTTLSec <= 0 && len(orgCfg.CachePaths) == 0 && len(orgCfg.Methods) == 0 {
return nil, zip.ErrBadRequest("nothing to set: provide org_rpm/cache_ttl_sec/cache_paths/methods, or cors_origins/per_ip_rpm/window_sec (SuperAdmin)")
if orgCfg.OrgRPM <= 0 && orgCfg.CacheTTLSec <= 0 && len(orgCfg.CachePaths) == 0 && len(orgCfg.Methods) == 0 && orgCfg.Mode == "" {
return nil, zip.ErrBadRequest("nothing to set: provide org_rpm/cache_ttl_sec/cache_paths/methods/mode, or cors_origins/per_ip_rpm/window_sec (SuperAdmin)")
}
if err := orgCfg.Validate(); err != nil {
return nil, zip.ErrBadRequest(err.Error())
}
// Arming the abuse gate is refused while no scorer is installed. Live mode
// makes a privileged grant fail CLOSED when the scorer cannot answer — which
// is the right posture for a scorer that is momentarily down, and an outage
// for one that was never there. Refusing here is the difference between the
// two, and it is checked at the moment of arming rather than on every request
// because a gate that silently disarms itself is not a gate.
if orgCfg.Mode == edge.ModeLive && !cloud.RiskScorerInstalled() {
return nil, zip.ErrBadRequest("mode=live requires the risk scorer; none is installed in this deployment")
}
tgt := target(c, org)
saved, err := o.s.State.store.Put(ctx, tgt, orgCfg)
if err != nil {
@@ -209,7 +270,45 @@ func (o ops) write(ctx context.Context, in *edge.Policy) (*edge.Policy, error) {
return nil, zip.Errorf(503, "policy store unavailable")
}
o.s.Log.Info("gateway org policy updated", "org", tgt, "by", orgCfg.UpdatedBy,
"org_rpm", saved.OrgRPM, "cache_ttl_sec", saved.CacheTTLSec, "methods", len(saved.Methods))
"org_rpm", saved.OrgRPM, "cache_ttl_sec", saved.CacheTTLSec, "methods", len(saved.Methods),
"mode", saved.Mode)
eff := o.s.State.store.Effective(tgt)
return &eff, nil
}
// Traffic reports who is calling this organization's API right now: the request
// count for the last minute split by AGENCY LANE — agent, human, bot, unknown —
// and the busiest callers behind it, each with its request count, its
// authentication-failure count, how many distinct paths it touched, and any
// verdict currently held against it.
//
// The lane split is the answer to the question a generic bot filter cannot
// answer: which of this traffic is the customer's own automation and which is
// somebody working through a list. It is computed from credentials we issued, not
// from the client's self-description, so a scraper cannot move itself into the
// agent lane by editing a header.
//
// A validated caller appears as a FINGERPRINT — a one-way, per-process digest. It
// is stable enough to recognise the same caller across a minute and cannot be
// turned back into a key, so this report is safe to read, screenshot and paste.
//
// It also reports what the sensor's own ceilings are doing (strain, tracked,
// ceiling, refused) and how many screens the scorer did not answer (unscored), so
// a control that has stopped measuring or a judge that has stopped answering is a
// number here rather than a quiet day.
//
// Scoped to the caller's own validated organization. A SuperAdmin may inspect a
// specific tenant with ?org=<slug>, or the lane that has no tenant — every caller
// the identity boundary could not validate — with an empty ?org=.
func (o ops) traffic(ctx context.Context, _ *noArgs) (*edge.TrafficView, error) {
c, org, err := caller(ctx)
if err != nil {
return nil, err
}
if o.s.State.traffic == nil {
return nil, zip.Errorf(503, "the edge traffic sensor is not running in this deployment")
}
tgt := target(c, org)
v := o.s.State.traffic.View(tgt, o.s.State.store.Mode(tgt), time.Now())
return &v, nil
}
apps/gateway/gateway_test.go
+156
View File
@@ -1,12 +1,14 @@
package gateway
import (
"context"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/hanzoai/cloud"
"github.com/hanzoai/cloud/apps/gateway/edge"
@@ -223,3 +225,157 @@ func TestPut_ValidatesPerOrgConfig(t *testing.T) {
}
}
}
// REGRESSION — the abuse gate must not be disarmable by the account it polices.
// Mode used to sit in the self-service branch, gated only on being a validated
// principal, so an org admin — or anyone holding an org-admin credential, which is
// what a stolen key buys — could PUT {"mode":"shadow"} and switch the control off
// for exactly the account it was watching. Setting it is the platform's decision
// now, whichever row it lands on.
func TestOrgAdmin_CannotSetItsOwnMode(t *testing.T) {
app, st := mountApp(t)
// Arm acme as the platform would.
if _, err := st.Put(t.Context(), "acme", edge.Policy{Mode: edge.ModeLive}); err != nil {
t.Fatalf("arm: %v", err)
}
if got := st.Mode("acme"); got != edge.ModeLive {
t.Fatalf("precondition: acme = %q, want live", got)
}
// The subject of the control tries to turn it off, and then to turn it on.
for _, body := range []string{`{"mode":"shadow"}`, `{"mode":"live"}`} {
code, msg := call(t, app, http.MethodPut, "/v1/gateway/config", body, orgAdmin("acme"))
if code != 403 {
t.Fatalf("org admin PUT %s must be 403, got %d (%s)", body, code, msg)
}
}
if got := st.Mode("acme"); got != edge.ModeLive {
t.Fatalf("acme disarmed itself: mode = %q", got)
}
// A per-org write that carries no mode still works — the refusal is about the
// one field, not about the surface.
if code, _ := call(t, app, http.MethodPut, "/v1/gateway/config", `{"org_rpm":60}`, orgAdmin("acme")); code != 200 {
t.Fatalf("an ordinary self-service write must still succeed, got %d", code)
}
if got := st.Mode("acme"); got != edge.ModeLive {
t.Fatalf("an unrelated write changed the mode: %q", got)
}
}
// A SuperAdmin sets the mode, on the tenant it names — the one authority that may.
// Guarded by the scorer check, so arming a deployment with no scorer installed is
// refused rather than turning every privileged grant into a 403.
func TestSuperAdmin_SetsModeOnATargetedTenant(t *testing.T) {
app, st := mountApp(t)
cloud.SetRiskScorer(func(ctx context.Context, org string, q cloud.RiskQuery) (cloud.RiskVerdict, error) {
return cloud.RiskVerdict{Action: cloud.ActionAllow}, nil
})
t.Cleanup(func() { cloud.SetRiskScorer(nil) })
if code, msg := call(t, app, http.MethodPut, "/v1/gateway/config?org=acme", `{"mode":"live"}`, superAdmin("admin")); code != 200 {
t.Fatalf("SuperAdmin mode write: got %d (%s)", code, msg)
}
if got := st.Mode("acme"); got != edge.ModeLive {
t.Fatalf("acme = %q after the operator armed it, want live", got)
}
// And only that tenant.
if got := st.Mode("globex"); got != edge.ModeShadow {
t.Fatalf("arming acme reached globex: %q", got)
}
}
// Arming while no scorer is installed is refused: live makes a privileged grant
// fail CLOSED when the scorer cannot answer, which is right for a scorer that is
// momentarily down and an outage for one that was never there.
func TestSuperAdmin_CannotArmWithoutAScorer(t *testing.T) {
app, st := mountApp(t)
cloud.SetRiskScorer(nil)
if code, _ := call(t, app, http.MethodPut, "/v1/gateway/config?org=acme", `{"mode":"live"}`, superAdmin("admin")); code != 400 {
t.Fatalf("arming with no scorer must be 400, got %d", code)
}
if got := st.Mode("acme"); got != edge.ModeShadow {
t.Fatalf("acme armed with no scorer: %q", got)
}
}
// The lane that has no tenant is where every caller the identity boundary could
// not validate is counted — the lane a bad bot calls from, and the one the
// platform row exists to arm. It had no spelling on this surface at all: the
// report could only be asked for by NAME, so its traffic and its saturation were
// readable by nobody, SuperAdmin included. The empty ?org= is its name.
func TestSuperAdmin_ReadsTheLaneWithNoTenant(t *testing.T) {
st, err := edge.New(t.TempDir(), "admin", edge.Policy{})
if err != nil {
t.Fatalf("store: %v", err)
}
t.Cleanup(func() { _ = st.Close() })
tr := edge.NewTraffic()
app := zip.New(zip.Config{})
if err := Mount(app, cloud.Deps{Logger: luxlog.New("test"), GatewayPolicy: st, Traffic: tr}); err != nil {
t.Fatalf("Mount: %v", err)
}
now := time.Now()
for i := 0; i < 3; i++ {
tr.Observe(edge.Signal{Org: "", Presented: "junk", IP: "203.0.113.9", Path: "/v1/models", Class: edge.CredAnonymous}, now)
}
tr.Observe(edge.Signal{Org: "acme", Cred: "fp", Presented: "fp", IP: "198.51.100.1", Path: "/v1/models", Class: edge.CredSecret}, now)
code, body := call(t, app, http.MethodGet, "/v1/gateway/traffic?org=", "", superAdmin("admin"))
if code != 200 {
t.Fatalf("GET traffic?org= → %d: %s", code, body)
}
var v edge.TrafficView
if err := json.Unmarshal([]byte(body), &v); err != nil {
t.Fatalf("decode: %v", err)
}
if v.Org != "" || v.Requests != 3 {
t.Fatalf("the anonymous lane's own view: %+v, want org=\"\" requests=3", v)
}
if v.Strain == "" || v.Ceiling == 0 {
t.Fatalf("the lane must report what its ceilings are doing: %+v", v)
}
// It is a DIFFERENT scope from the admin org's own traffic, which is what a
// reserved-word spelling would have conflated.
code, body = call(t, app, http.MethodGet, "/v1/gateway/traffic?org=acme", "", superAdmin("admin"))
if code != 200 {
t.Fatalf("GET traffic?org=acme → %d: %s", code, body)
}
if err := json.Unmarshal([]byte(body), &v); err != nil {
t.Fatalf("decode: %v", err)
}
if v.Org != "acme" || v.Requests != 1 {
t.Fatalf("a named tenant's view: %+v, want org=acme requests=1", v)
}
}
// And only a SuperAdmin may ask for it. An org admin asking for the lane with no
// tenant gets its OWN scope, exactly as it does for any other ?org= it is not
// entitled to.
func TestOrgAdmin_CannotReadTheLaneWithNoTenant(t *testing.T) {
st, err := edge.New(t.TempDir(), "admin", edge.Policy{})
if err != nil {
t.Fatalf("store: %v", err)
}
t.Cleanup(func() { _ = st.Close() })
tr := edge.NewTraffic()
app := zip.New(zip.Config{})
if err := Mount(app, cloud.Deps{Logger: luxlog.New("test"), GatewayPolicy: st, Traffic: tr}); err != nil {
t.Fatalf("Mount: %v", err)
}
tr.Observe(edge.Signal{Org: "", Presented: "junk", IP: "203.0.113.9", Path: "/v1/models", Class: edge.CredAnonymous}, time.Now())
code, body := call(t, app, http.MethodGet, "/v1/gateway/traffic?org=", "", orgAdmin("acme"))
if code != 200 {
t.Fatalf("GET traffic?org= as an org admin → %d: %s", code, body)
}
var v edge.TrafficView
if err := json.Unmarshal([]byte(body), &v); err != nil {
t.Fatalf("decode: %v", err)
}
if v.Org != "acme" || v.Requests != 0 {
t.Fatalf("an org admin read the anonymous lane: %+v", v)
}
}
apps/gateway/typed_wire_test.go
+2 -2
View File
@@ -80,8 +80,8 @@ func TestEveryRouteIsTypedOrNamed(t *testing.T) {
len(typed), len(untypedByDesign), got, want)
}
// The MEASURED partition, so the prose cannot drift from the binary.
if len(served) != 2 || len(typed) != 2 {
t.Errorf("served = %d (want 2), typed = %d (want 2)", len(served), len(typed))
if len(served) != 3 || len(typed) != 3 {
t.Errorf("served = %d (want 3), typed = %d (want 3)", len(served), len(typed))
}
}
apps/gateway/zipdoc_gen.go
+29 -1
View File
@@ -16,6 +16,7 @@ func init() {
"Policy.cache_ttl_sec": "CacheTTLSec is the org's default edge-cache TTL for its responses, in seconds;\n0 means no caching. Unset inherits the platform default.",
"Policy.cors_origins": "CORSOrigins is the PLATFORM-scope CORS allowlist EdgeCORS admits: an exact\norigin, a bare host, or a \"*.host\" wildcard. Writable only by a SuperAdmin —\nCORS is evaluated before identity, so it has no tenant to scope to.",
"Policy.methods": "Methods is the allowlist of HTTP methods the edge accepts for this org. Empty\nmeans all are accepted.",
"Policy.mode": "Mode is the abuse gate's posture for THIS scope: \"shadow\" scores traffic and\nrecords the verdict without acting on it, \"live\" enforces it. Unset means\nshadow.\n\nIt is the one per-org field that does NOT inherit. Every other field here\nlayers a platform default under the org's own value, which is right for a\ndefault: a tenant that sets no rate ceiling should get the platform's. Mode\nis not a default, it is an ARMING DECISION — it is what makes a statistical\njudgement start refusing real traffic — and inheriting it means arming one\nscope arms every tenant that never asked for it, without a write to their\nrow and without anything in their config changing. So a tenant is live only\nif that tenant's OWN row says live, and the platform row's mode governs\nexactly one scope: the anonymous lane, which has no tenant of its own.\n\nIt is also not self-service. Writing it requires SuperAdmin (see the\n/v1/gateway config op): the subject of an abuse control does not get to\nswitch the control off.",
"Policy.org_rpm": "OrgRPM is the org's OWN authenticated rate ceiling, requests per minute, as\nScopeRateLimit enforces it. Unset inherits the platform default, then the\nstatic boot default.",
"Policy.per_ip_rpm": "PerIPRPM is the PLATFORM-scope pre-auth flood cap: requests EdgeRateLimit\nadmits per WindowSec from one client IP. SuperAdmin-only, same reason.",
"Policy.updated_at": "UpdatedAt is the unix second this policy row was last written. Server-stamped;\na client-supplied value is ignored.",
@@ -23,13 +24,40 @@ func init() {
"Policy.window_sec": "WindowSec is the window PerIPRPM is counted over, in seconds. SuperAdmin-only.",
},
})
zip.Describe("GET /v1/gateway/traffic", zip.Doc{
Description: "Traffic reports who is calling this organization's API right now: the request\ncount for the last minute split by AGENCY LANE — agent, human, bot, unknown —\nand the busiest callers behind it, each with its request count, its\nauthentication-failure count, how many distinct paths it touched, and any\nverdict currently held against it.\n\nThe lane split is the answer to the question a generic bot filter cannot\nanswer: which of this traffic is the customer's own automation and which is\nsomebody working through a list. It is computed from credentials we issued, not\nfrom the client's self-description, so a scraper cannot move itself into the\nagent lane by editing a header.\n\nA validated caller appears as a FINGERPRINT — a one-way, per-process digest. It\nis stable enough to recognise the same caller across a minute and cannot be\nturned back into a key, so this report is safe to read, screenshot and paste.\n\nIt also reports what the sensor's own ceilings are doing (strain, tracked,\nceiling, refused) and how many screens the scorer did not answer (unscored), so\na control that has stopped measuring or a judge that has stopped answering is a\nnumber here rather than a quiet day.\n\nScoped to the caller's own validated organization. A SuperAdmin may inspect a\nspecific tenant with ?org=<slug>, or the lane that has no tenant — every caller\nthe identity boundary could not validate — with an empty ?org=.",
Fields: map[string]string{
"TrafficCaller.action": "Action is the verdict currently held against it, if any.",
"TrafficCaller.cred": "Cred is the caller's key: a credential fingerprint (a per-process one-way\ndigest, not a key) for a validated caller, and \"ip:<addr>\" for one that\npresented no credential we could validate.",
"TrafficCaller.failures": "Failures is how many ended 401 or 403.",
"TrafficCaller.held_until": "HeldUntil is when the held verdict lapses, unix seconds.",
"TrafficCaller.paths": "Paths is the approximate number of distinct paths it touched (max 64).",
"TrafficCaller.reason": "Reason is why that verdict was reached.",
"TrafficCaller.requests": "Requests is its request count in the window.",
"TrafficView.blind": "Blind is how many requests in the window carried no identity to attribute\nthem to — no validated credential and no client address. Non-zero on a\npublic plane means the client address is not reaching this process (a TCP\nload balancer with no PROXY protocol in front of it, typically), so this\nscope's callers cannot be told apart and nothing can be held against them.",
"TrafficView.callers": "Callers is the scope's busiest callers this window. A credentialed caller\nappears as a FINGERPRINT — a per-process one-way digest: enough to recognise\nthe same caller across requests, never enough to reconstruct the credential.",
"TrafficView.ceiling": "Ceiling is the most callers this scope may hold at once.",
"TrafficView.denied": "Denied is how many of them the gate refused.",
"TrafficView.lanes": "Lanes is the request count per lane — agent, human, bot, unknown. This is\nthe split that separates a customer's automation from a scraper.",
"TrafficView.mode": "Mode is the abuse gate's posture for this scope: \"shadow\" records the scorer's\naction without enforcing it, \"live\" enforces it.",
"TrafficView.org": "Org is the scope this view was taken for — the validated principal's own,\nnever a value the caller supplied. Empty names the anonymous lane, the one\nscope that has no tenant.",
"TrafficView.refused": "Refused is how many callers this scope's ceilings turned away in the window.",
"TrafficView.requests": "Requests is how many requests this scope made in the window.",
"TrafficView.screens": "Screens is how many of them were put to the scorer — the billable unit of\nthe risk product. Counted from the first request, whatever the SKU costs.",
"TrafficView.strain": "Strain is what this scope's ceilings are doing: \"clear\" below them, \"full\"\nat them, \"refuse\" once a caller has been turned away inside this window —\nwhich means that caller is UNMEASURED and the numbers here are a sample\nrather than a census. It is reported rather than logged because the\nalternative — a bound that degrades a scope silently — is the failure this\ndesign exists to rule out. No other scope can move it.",
"TrafficView.tracked": "Tracked is how many callers this scope holds state for right now, and\nCeiling is the most it may hold. Tracked == Ceiling is the fact a bound\nthat binds cannot hide.",
"TrafficView.unscored": "Unscored is how many of those screens got NO answer — the scorer was absent,\nstuck, slow, erroring or silent. An unanswered screen allows ordinary\ntraffic, so this is the number that separates \"a quiet day\" from \"the judge\nstopped answering and nothing said so\".",
"TrafficView.window_sec": "WindowSec is the span the counts cover, in seconds.",
},
})
zip.Describe("PUT /v1/gateway/config", zip.Doc{
Description: "Write updates one policy scope and returns the policy in force after the write.\nA body carrying any PLATFORM field (cors_origins, per_ip_rpm, window_sec) is a\nplatform write and requires SuperAdmin; otherwise it is a per-org write (org_rpm,\ncache_ttl_sec, cache_paths, methods) scoped to the caller's own org — or, for a\nSuperAdmin, the tenant named by ?org=<slug>. A body that sets nothing is a 400.\nupdated_at and updated_by are server-stamped; a client-supplied value is ignored.",
Description: "Write updates one policy scope and returns the policy in force after the write.\nA body carrying any PLATFORM field (cors_origins, per_ip_rpm, window_sec) is a\nplatform write and requires SuperAdmin; otherwise it is a per-org write (org_rpm,\ncache_ttl_sec, cache_paths, methods) scoped to the caller's own org — or, for a\nSuperAdmin, the tenant named by ?org=<slug>. A body that sets nothing is a 400.\nThe abuse gate's mode is an OPERATOR field: setting it requires SuperAdmin,\nwhichever organization it lands on. updated_at and updated_by are\nserver-stamped; a client-supplied value is ignored.",
Fields: map[string]string{
"Policy.cache_paths": "CachePaths overrides CacheTTLSec per path PREFIX (key \"/v1/models\" → seconds).\nThe longest matching prefix wins.",
"Policy.cache_ttl_sec": "CacheTTLSec is the org's default edge-cache TTL for its responses, in seconds;\n0 means no caching. Unset inherits the platform default.",
"Policy.cors_origins": "CORSOrigins is the PLATFORM-scope CORS allowlist EdgeCORS admits: an exact\norigin, a bare host, or a \"*.host\" wildcard. Writable only by a SuperAdmin —\nCORS is evaluated before identity, so it has no tenant to scope to.",
"Policy.methods": "Methods is the allowlist of HTTP methods the edge accepts for this org. Empty\nmeans all are accepted.",
"Policy.mode": "Mode is the abuse gate's posture for THIS scope: \"shadow\" scores traffic and\nrecords the verdict without acting on it, \"live\" enforces it. Unset means\nshadow.\n\nIt is the one per-org field that does NOT inherit. Every other field here\nlayers a platform default under the org's own value, which is right for a\ndefault: a tenant that sets no rate ceiling should get the platform's. Mode\nis not a default, it is an ARMING DECISION — it is what makes a statistical\njudgement start refusing real traffic — and inheriting it means arming one\nscope arms every tenant that never asked for it, without a write to their\nrow and without anything in their config changing. So a tenant is live only\nif that tenant's OWN row says live, and the platform row's mode governs\nexactly one scope: the anonymous lane, which has no tenant of its own.\n\nIt is also not self-service. Writing it requires SuperAdmin (see the\n/v1/gateway config op): the subject of an abuse control does not get to\nswitch the control off.",
"Policy.org_rpm": "OrgRPM is the org's OWN authenticated rate ceiling, requests per minute, as\nScopeRateLimit enforces it. Unset inherits the platform default, then the\nstatic boot default.",
"Policy.per_ip_rpm": "PerIPRPM is the PLATFORM-scope pre-auth flood cap: requests EdgeRateLimit\nadmits per WindowSec from one client IP. SuperAdmin-only, same reason.",
"Policy.updated_at": "UpdatedAt is the unix second this policy row was last written. Server-stamped;\na client-supplied value is ignored.",
apps/legal/legal.go
+5 -9
View File
@@ -314,15 +314,11 @@ func decode(c *zip.Ctx, v any) error {
return c.Bind(v)
}
func clientIP(c *zip.Ctx) string {
if xff := c.Header("X-Forwarded-For"); xff != "" {
if i := strings.IndexByte(xff, ','); i >= 0 {
return strings.TrimSpace(xff[:i])
}
return strings.TrimSpace(xff)
}
return c.Header("X-Real-Ip")
}
// clientIP is the caller's address, by the ONE rule — cloud.ClientIP. It lands in
// a durable audit record, and the LEFT-most X-Forwarded-For entry (and X-Real-Ip)
// are values the client writes: an address chosen by the party being audited is
// not evidence.
func clientIP(c *zip.Ctx) string { return cloud.ClientIP(c) }
func nowUnix() int64 { return time.Now().Unix() }
apps/plugin/fleet.go
+2 -2
View File
@@ -10,9 +10,9 @@ import (
"time"
"github.com/hanzoai/cloud"
"github.com/hanzoai/cloud/audit"
"github.com/hanzoai/cloud/apps/admin/core"
"github.com/hanzoai/cloud/apps/principal"
"github.com/hanzoai/cloud/audit"
"github.com/hanzoai/ha"
"github.com/zap-proto/zip"
)
@@ -252,7 +252,7 @@ func (o *ops) record(ctx context.Context, c *zip.Ctx, a act, results []Result, f
Resource: audit.Resource{Type: "plugin", ID: a.name},
Auth: audit.AuthContext{Method: "jwt", IsAdmin: c.IsAdmin()},
Outcome: outcome,
SourceIP: c.Header("X-Forwarded-For"),
SourceIP: cloud.ClientIP(c),
UserAgent: c.Header("User-Agent"),
RequestID: c.RequestID(),
Method: c.Method(),
apps/principal/principal.go
+52
View File
@@ -139,6 +139,58 @@ func OrgOf(user, org string) (string, bool) {
return strings.Clone(org), true
}
// ─────────────────────────────────────────────────────────────────────────────
// The MINTED principal — the boundary's own attestation
// ─────────────────────────────────────────────────────────────────────────────
// Principal is what the identity boundary MINTED for a request: the effective
// org it resolved and the user it verified. Both empty for an anonymous caller.
type Principal struct {
Org string
User string
}
// mintedSlot names the request-local slot the boundary parks its attestation in.
// A request-local value is server-side state — it is not a header, it does not
// cross the wire, and there is no request a client can send that creates one.
// Unexported zero-size type, so only this package can mint or read it.
type mintedSlot struct{}
// Mint records the principal the identity boundary resolved. It is called by the
// boundary itself (cloud.SanitizeIdentity) and by nothing else.
//
// WHY THIS EXISTS BESIDE Validated. Validated reads X-User-Id, which is
// trustworthy only DOWNSTREAM of the boundary that strips and re-mints it. Most
// of cloud is downstream of it and Validated is the right question there. But a
// middleware is not guaranteed to be: the same handler is installed in the fused
// binary (behind the boundary) and in a hand-written plugin process (where the
// boundary may not be installed at all), and a gate that reads a header in the
// second case is reading whatever the client typed. So a gate whose correctness
// must not depend on its position asks THIS instead — a fact only the boundary
// can state, absent when the boundary did not run, which fails closed to
// anonymous rather than open to forged.
// Both fields are CLONED. A value read off a request is a zero-copy view into
// the reused fasthttp buffer, and this one is retained past the read — it becomes
// a map key in the edge sensor and a column in a meter — so an un-owned copy
// would mutate into unrelated bytes on the next request through that worker.
// (A string used as a map key copies the header, never the backing array; Org
// clones for exactly this reason.)
func Mint(c *zip.Ctx, p Principal) {
c.Fiber().Locals(mintedSlot{}, Principal{
Org: strings.Clone(p.Org),
User: strings.Clone(p.User),
})
}
// Minted returns the principal the boundary attested, and false when no boundary
// ran on this request. An anonymous request that DID pass a boundary returns
// (zero, true): "we looked, and there is nobody" is a different fact from "nobody
// looked", and only the caller knows which of them it can live with.
func Minted(c *zip.Ctx) (Principal, bool) {
p, ok := c.Fiber().Locals(mintedSlot{}).(Principal)
return p, ok
}
// orgKey names the request-scoped slot the validated org crosses the typed-op
// seam in. Unexported zero-size type, so only this package can mint or read one —
// the same unforgeability the header gate has.
apps/security/security.go
+5 -11
View File
@@ -453,17 +453,11 @@ func projectScope(c *zip.Ctx) string {
return p
}
// clientIP is the best-effort source IP for audit/metering. Prefers the
// gateway-forwarded header, falls back to the socket peer.
func clientIP(c *zip.Ctx) string {
if xff := c.Header("X-Forwarded-For"); xff != "" {
if i := strings.IndexByte(xff, ','); i >= 0 {
return strings.TrimSpace(xff[:i])
}
return strings.TrimSpace(xff)
}
return c.Header("X-Real-Ip")
}
// clientIP is the caller's address, by the ONE rule — cloud.ClientIP. It lands in
// a durable audit record, and the LEFT-most X-Forwarded-For entry (and X-Real-Ip)
// are values the client writes: an address chosen by the party being audited is
// not evidence.
func clientIP(c *zip.Ctx) string { return cloud.ClientIP(c) }
// genID mints a prefixed random id (mirrors clients/git.genID).
func genID(prefix string) (string, error) {
build.go
+6
View File
@@ -145,6 +145,12 @@ func BuildDeps(cfg *Config) Deps {
}
deps.GatewayPolicy = gp
// The edge traffic sensor the abuse gate writes and /v1/gateway/traffic reads.
// One object per process, hung off deps for the same reason the policy store
// is: two of them would be two answers to "who is calling", and the middleware
// and the subsystem would each be sure of a different one.
deps.Traffic = edge.NewTraffic()
return deps
}
clientip.go
+200
View File
@@ -0,0 +1,200 @@
package cloud
// The caller's address, and the ONE rule cloud derives it by.
//
// An address is not a header. X-Forwarded-For is a list a client may write the
// first entries of and each hop appends to, so the LEFT-MOST entry is whatever
// the client typed — it is the one value in the chain that is always attacker
// controlled. Reading it as "the client" is how a request from one host becomes
// a million distinct clients: it defeats every per-IP limit keyed on it, it puts
// a chosen address into an audit row and a velocity counter, and it fills any
// table keyed on it without bound.
//
// THE RULE, in one function:
//
// the socket peer is the truth. If the peer is not one of OUR proxies, it IS
// the client — a TCP source address cannot be forged inside an established
// connection, so nothing it says about itself is needed.
//
// only a trusted peer's chain is readable. When the peer IS one of ours, walk
// X-Forwarded-For from the RIGHT — the end each hop appends to — and take the
// first entry that is not itself one of our proxies. Everything to its left was
// written before our infrastructure saw the request and is therefore hearsay.
//
// our own traffic has no client. A chain that is entirely our own addresses is
// an in-cluster caller (a sibling service, the console BFF); it never transited
// the public edge, so it has no client address and gets "". That is the same
// answer this function has always given for a request with no chain at all, and
// it is what keeps in-cluster callers out of the public edge's rate limiter.
//
// WHY A CIDR SET AND NOT A HOP COUNT. A hop count is a promise about topology
// that nothing enforces; the day an extra proxy appears, a count silently reads
// one entry too far to the left — back into attacker-written territory. A set of
// addresses is checkable against the deployment and fails in the safe direction:
// an unlisted proxy is treated as a client, which over-attributes traffic to our
// own edge rather than under-attributing an attacker's.
import (
"net/netip"
"os"
"strings"
"sync"
"github.com/zap-proto/zip"
)
// TrustedProxiesEnv names the operator knob: a comma-separated list of CIDRs
// and bare addresses that are OUR OWN forwarding hops. Set it when a deployment
// is fronted by a proxy on a PUBLIC address (a CDN edge, a cloud load balancer
// with public egress); the default below covers only private space, which is
// every hop inside a cluster.
const TrustedProxiesEnv = "CLOUD_TRUSTED_PROXIES"
// defaultTrustedProxies is the address space our own hops live in when nobody
// says otherwise: loopback, the unspecified address (never a real peer — it is
// what an in-memory test connection reports), RFC1918 private space, the
// carrier-grade NAT range a managed load balancer forwards from, link-local, and
// IPv6 unique-local. A public address is NEVER trusted by default: trusting one
// by accident is what turns every customer into one shared bucket.
var defaultTrustedProxies = []string{
"127.0.0.0/8", "::1/128",
"0.0.0.0/32", "::/128",
"10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16",
"100.64.0.0/10",
"169.254.0.0/16", "fe80::/10",
"fc00::/7",
}
// maxForwardedHops bounds how much of a chain is read. A real chain is three
// entries; a request carrying thousands is an attempt to spend CPU in the parse
// itself. Read from the right, so the bound only ever discards the OLDEST
// (left-most, least trustworthy) entries.
const maxForwardedHops = 32
// trustedProxies is resolved once per process. The environment is read at first
// use rather than at init so a test can set it before the first request without
// depending on package initialization order.
var trustedProxies = sync.OnceValue(func() proxySet {
if s := parseProxySet(os.Getenv(TrustedProxiesEnv)); len(s.nets) > 0 {
return s
}
// An unset — or entirely unparseable — knob falls back to the defaults rather
// than to an EMPTY set. Trusting nothing sounds safer and is not: it would
// make the ingress itself the "client", collapsing every caller into one
// bucket and one audit address. The safe failure here is the private-space
// default, which is correct for every in-cluster deployment we run.
return parseProxySet(strings.Join(defaultTrustedProxies, ","))
})
// proxySet is the set of addresses that are our own forwarding hops.
type proxySet struct{ nets []netip.Prefix }
func parseProxySet(spec string) proxySet {
var s proxySet
for _, raw := range strings.Split(spec, ",") {
raw = strings.TrimSpace(raw)
if raw == "" {
continue
}
if p, err := netip.ParsePrefix(raw); err == nil {
s.nets = append(s.nets, p.Masked())
continue
}
// A bare address is the single-host prefix it denotes.
if a, err := netip.ParseAddr(raw); err == nil {
s.nets = append(s.nets, netip.PrefixFrom(a.Unmap(), a.Unmap().BitLen()))
}
}
return s
}
func (s proxySet) has(a netip.Addr) bool {
for _, n := range s.nets {
if n.Contains(a) {
return true
}
}
return false
}
// TrustedProxy reports whether addr is one of our own forwarding hops — for a
// health report or a test, never as a gate. The gate is ClientIP, which applies
// the whole rule.
func TrustedProxy(addr string) bool {
a, ok := parseClientAddr(addr)
return ok && trustedProxies().has(a)
}
// ClientIP is the caller's own address: the socket peer for a direct caller, the
// right-most non-proxy entry of the forwarded chain for a proxied one, and "" for
// an in-cluster caller that never transited the edge.
//
// It is the ONE client-address read in this repo — the edge rate limiter, the
// abuse sensor, the audit trail and every metered resource share it, so a
// forgeable address cannot enter one of them by a side door.
// It reads EVERY X-Forwarded-For header line, not just the first. fasthttp keeps
// repeated headers as separate lines, and a client that sends its own line before
// the proxy appends to a second one would otherwise hide the real address behind
// a value it chose.
func ClientIP(c *zip.Ctx) string {
return clientAddr(c.Fiber().IP(), c.Fiber().Request().Header.PeekAll("X-Forwarded-For"), trustedProxies())
}
// clientAddr IS the rule, as a pure function of the three facts it turns on: the
// socket peer, the forwarded chain, and which addresses are ours. Everything
// interesting about ClientIP is here, where it can be read and tested without a
// server — the exported wrapper only supplies the arguments.
//
// The chain is walked last-to-first, across lines and within a line, for the same
// reason in both cases: later is nearer to us, and nearer to us is truer.
func clientAddr(peerAddr string, forwarded [][]byte, tp proxySet) string {
peer, ok := parseClientAddr(peerAddr)
if !ok {
return ""
}
if !tp.has(peer) {
// A direct caller. The peer is the connection's own source address, so it
// is the one fact about the client that cannot be written by the client,
// and no header it sent is consulted at all.
return peer.String()
}
seen := 0
for i := len(forwarded) - 1; i >= 0; i-- {
hops := strings.Split(string(forwarded[i]), ",")
for j := len(hops) - 1; j >= 0; j-- {
if seen++; seen > maxForwardedHops {
return ""
}
a, ok := parseClientAddr(hops[j])
if !ok {
// Not an address at all. It cannot be a hop and it must never become
// a key, so it is skipped rather than passed through — an unparseable
// entry is exactly how an unbounded keyspace gets fed.
continue
}
if tp.has(a) {
continue
}
return a.String()
}
}
return ""
}
// parseClientAddr parses one chain entry or peer address into a canonical
// address. It accepts a bare address and an address:port pair, and it UNMAPS
// IPv4-in-IPv6 so "::ffff:1.2.3.4" and "1.2.3.4" are one key rather than two.
// The canonical String() is what every counter, record and report keys on.
func parseClientAddr(s string) (netip.Addr, bool) {
s = strings.TrimSpace(s)
if s == "" {
return netip.Addr{}, false
}
if a, err := netip.ParseAddr(s); err == nil {
return a.Unmap(), true
}
if ap, err := netip.ParseAddrPort(s); err == nil {
return ap.Addr().Unmap(), true
}
return netip.Addr{}, false
}
clientip_test.go
+194
View File
@@ -0,0 +1,194 @@
package cloud
// The client-address rule, and the attack it exists to stop.
//
// THE DEFECT THIS REPLACES: ClientIP returned the LEFT-MOST X-Forwarded-For entry
// with no check on who wrote it. That entry is whatever the client typed, so one
// host could present a million distinct "clients" — defeating the per-IP edge
// limit keyed on it, writing a chosen address into audit rows, and feeding the
// abuse sensor's address table without bound from an unauthenticated request.
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/zap-proto/zip"
)
// ourProxies is the deployment shape these tests reason about: private space is
// ours, everything public is a client. Built explicitly rather than read from the
// process default so the rule is tested, not the environment.
var ourProxies = parseProxySet("10.0.0.0/8,127.0.0.0/8,0.0.0.0/32,::1/128")
func xff(lines ...string) [][]byte {
out := make([][]byte, 0, len(lines))
for _, l := range lines {
out = append(out, []byte(l))
}
return out
}
func TestClientAddr(t *testing.T) {
cases := []struct {
name string
peer string
fwd [][]byte
want string
}{
// THE ATTACK. The client writes the left-most entry; our ingress appends
// what it actually saw. The right-most untrusted hop is the truth.
{"a forged left-most entry is ignored",
"10.0.0.5", xff("1.2.3.4, 203.0.113.9"), "203.0.113.9"},
{"a whole forged chain is ignored",
"10.0.0.5", xff("1.2.3.4, 5.6.7.8, 9.10.11.12, 203.0.113.9"), "203.0.113.9"},
// A second header LINE is the same attack with different syntax: fasthttp
// keeps repeated headers apart, so a rule that reads only the first line
// reads only the attacker's.
{"a forged second line does not hide the real hop",
"10.0.0.5", xff("1.2.3.4", "203.0.113.9"), "203.0.113.9"},
{"a forged FIRST line does not win over the real hop",
"10.0.0.5", xff("203.0.113.9", "10.0.0.6"), "203.0.113.9"},
// Our own hops are skipped, however many of them there are.
{"internal hops are skipped",
"10.0.0.5", xff("203.0.113.9, 10.0.0.6, 10.0.0.7"), "203.0.113.9"},
// A DIRECT caller cannot lie: the peer is the answer and no header is read.
{"a direct caller is its own peer",
"198.51.100.4", xff("1.2.3.4"), "198.51.100.4"},
// In-cluster traffic never transited the edge, so it has no client address —
// the same answer this function has always given, which is what keeps
// sibling services out of the public rate limiter.
{"an in-cluster caller has no client address",
"10.0.0.5", nil, ""},
{"a chain of only our own hops has no client address",
"10.0.0.5", xff("10.0.0.6, 127.0.0.1"), ""},
// Junk must never become a key.
{"an unparseable entry is skipped, not keyed",
"10.0.0.5", xff("not-an-ip, 203.0.113.9"), "203.0.113.9"},
{"an entry that is only junk yields nothing",
"10.0.0.5", xff("not-an-ip, <script>"), ""},
// Canonical form, so one address is one key.
{"an IPv4-mapped IPv6 address is the same key as its IPv4 form",
"10.0.0.5", xff("::ffff:203.0.113.9"), "203.0.113.9"},
{"an entry with a port is the address without it",
"10.0.0.5", xff("203.0.113.9:44321"), "203.0.113.9"},
{"whitespace does not make a second key",
"10.0.0.5", xff(" 203.0.113.9 "), "203.0.113.9"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := clientAddr(tc.peer, tc.fwd, ourProxies); got != tc.want {
t.Fatalf("clientAddr(%q, %q) = %q, want %q", tc.peer, tc.fwd, got, tc.want)
}
})
}
}
// The parse is bounded. A chain of thousands of entries is not a topology, it is
// an attempt to spend CPU in the parse — and the bound discards from the LEFT, so
// it can only ever drop the least trustworthy end.
func TestClientAddr_ChainIsBounded(t *testing.T) {
long := strings.Repeat("1.2.3.4, ", maxForwardedHops*4) + "203.0.113.9"
// The real hop is at the far right, within the bound: still found.
if got := clientAddr("10.0.0.5", xff(long), ourProxies); got != "203.0.113.9" {
t.Fatalf("got %q, want the right-most hop", got)
}
// A chain that is ONLY forged entries, longer than the bound: the walk stops
// rather than reading all of it, and no client is claimed.
onlyForged := strings.TrimSuffix(strings.Repeat("1.2.3.4, ", maxForwardedHops*4), ", ")
if got := clientAddr("10.0.0.5", xff(onlyForged, "10.0.0.6"), ourProxies); got != "1.2.3.4" && got != "" {
t.Fatalf("got %q, want the bounded walk to stop", got)
}
}
// The set is data, and it is checkable. An operator whose proxy sits on a public
// address names it; until then a public address is a client, which is the safe
// direction to be wrong in.
func TestProxySet(t *testing.T) {
s := parseProxySet("10.0.0.0/8, 203.0.113.7 ,garbage,")
for _, in := range []string{"10.1.2.3", "203.0.113.7"} {
a, _ := parseClientAddr(in)
if !s.has(a) {
t.Fatalf("%s must be trusted", in)
}
}
for _, out := range []string{"203.0.113.8", "198.51.100.1"} {
a, _ := parseClientAddr(out)
if s.has(a) {
t.Fatalf("%s must NOT be trusted", out)
}
}
// An entirely unparseable spec yields no trust rather than blanket trust.
if len(parseProxySet("garbage,,also-garbage").nets) != 0 {
t.Fatal("an unparseable spec must not produce a trusted network")
}
}
// The default set is the one a deployment gets when nobody configures anything,
// and it must trust our own space and nothing public.
func TestDefaultTrustedProxies(t *testing.T) {
s := parseProxySet(strings.Join(defaultTrustedProxies, ","))
for _, ours := range []string{"10.42.0.1", "172.16.5.5", "192.168.1.1", "127.0.0.1", "0.0.0.0", "100.64.3.3", "::1", "fd00::1"} {
a, ok := parseClientAddr(ours)
if !ok || !s.has(a) {
t.Fatalf("%s must be trusted by default (ours)", ours)
}
}
for _, theirs := range []string{"203.0.113.9", "198.51.100.4", "1.1.1.1", "2001:db8::1"} {
a, ok := parseClientAddr(theirs)
if !ok || s.has(a) {
t.Fatalf("%s must NOT be trusted by default (public)", theirs)
}
}
}
// End to end through a real request, on the process default set. fiber's test
// connection reports the unspecified address as the peer, which the default set
// treats as ours — so the chain is read, and the forged left-most entry loses.
func TestClientIP_OverARealRequest(t *testing.T) {
var got string
app := zip.New(zip.Config{})
app.Get("/probe", func(c *zip.Ctx) error {
got = ClientIP(c)
return c.JSON(http.StatusOK, map[string]string{"ok": "1"})
})
req := httptest.NewRequest(http.MethodGet, "/probe", nil)
req.Header.Add("X-Forwarded-For", "1.2.3.4")
req.Header.Add("X-Forwarded-For", "203.0.113.9, 10.0.0.6")
if _, err := app.Fiber().Test(req); err != nil {
t.Fatal(err)
}
if got != "203.0.113.9" {
t.Fatalf("ClientIP = %q, want the right-most untrusted hop 203.0.113.9", got)
}
// And a request with no chain at all is an in-cluster caller: no address, so
// the edge limiter leaves it alone.
req = httptest.NewRequest(http.MethodGet, "/probe", nil)
if _, err := app.Fiber().Test(req); err != nil {
t.Fatal(err)
}
if got != "" {
t.Fatalf("an unproxied in-cluster request must have no client address, got %q", got)
}
}
// TrustedProxy is the readable form of the same set — for a health report, never
// as a gate.
func TestTrustedProxy(t *testing.T) {
if !TrustedProxy("127.0.0.1") {
t.Fatal("loopback must be one of ours by default")
}
if TrustedProxy("203.0.113.9") {
t.Fatal("a public address must not be trusted by default")
}
if TrustedProxy("not-an-ip") {
t.Fatal("a non-address is not a proxy")
}
}
deps.go
+9
View File
@@ -151,6 +151,15 @@ type Deps struct {
// SAME store. Never nil — New always returns a working (static-only on store
// error) *Store, so the edge is never blocked. See clients/edge.
GatewayPolicy *edge.Store
// Traffic is the edge's live sensor: per-credential request cadence, path
// spread, auth-failure rate and the verdict currently held against a caller.
// BuildDeps constructs it once; AbuseGate (middleware_abuse.go) writes it on
// every request and the /v1/gateway/traffic op reads the caller's OWN org's
// slice of it. In-memory and bounded by construction — it is a sensor, not a
// record, and it is rebuilt from live traffic within one window after a
// restart. Nil makes the gate a no-op passthrough.
Traffic *edge.Traffic
}
// Per-subsystem client interfaces live in cloud/types so the
middleware_abuse.go
+453
View File
@@ -0,0 +1,453 @@
package cloud
// AbuseGate — the API plane's lifecycle defense. It watches every authenticated
// and anonymous request, classes its caller into a lane, and — when the shape of
// the traffic warrants it — asks /v1/risk what to do. It does not score. It
// senses, asks, and enforces.
//
// WHAT IT ADDS THAT THE TWO EXISTING LIMITERS DO NOT. EdgeRateLimit caps a
// client IP before identity; ScopeRateLimit caps an authenticated
// (org, project, service). Neither can see a CREDENTIAL. So a key lifted out of
// a CI log, used from a residential address, inside its org's normal ceiling, is
// invisible to both — and that is precisely the shape of token theft, credential
// stuffing, scraping and pay-as-you-go abuse. This gate keys on the credential,
// which is the thing that was stolen.
//
// WHERE IT SITS, and why exactly there (serve.go):
//
// SanitizeIdentity → AuditTrail → ScopeRateLimit → AbuseGate → StarterGrant → BillingGate
//
// - AFTER SanitizeIdentity, so the org it scopes to is the validated one and
// the credential class it reads cannot be forged.
// - INSIDE AuditTrail, so a refusal is a 401/403 that the tamper-evident trail
// already records — with the actor, the resource and the outcome. There is no
// second audit write here, because there must not be two records of one event.
// - AFTER ScopeRateLimit, so ordinary over-rate traffic is already 429'd and
// never reaches the scorer: rate limiting is a limiter's job, and asking a
// model what to do about a request that is simply too fast would be paying
// for an answer we already have.
// - BEFORE BillingGate, so an abusive request is refused before it can consume
// a balance.
//
// SHADOW BY DEFAULT, PER ORG. An org's Mode (edge.Policy.Mode) is shadow unless
// an operator sets it live. In shadow the gate observes, asks and RECORDS, and
// then lets the request through whatever the answer was. A statistical judgement
// that quietly started refusing an org's payments or logins because a feature
// shipped is the worst failure available here, so it cannot happen by default —
// only by decision, and the decision is visible at GET /v1/gateway/traffic.
//
// FAIL POLICY. The gate never decides what to do when the scorer is unavailable;
// cloud.Decide does, in one place, for every caller (see risk.go). An ordinary
// request proceeds; a PRIVILEGED GRANT — minting a credential, provisioning an
// identity, reading the key store — does not. That asymmetry is the whole
// posture: an outage in the risk plane must not be able to lock people out of
// the product, and must not be able to hand out standing authority either.
import (
"os"
"strconv"
"strings"
"time"
"github.com/hanzoai/cloud/apps/gateway/edge"
"github.com/hanzoai/cloud/apps/principal"
"github.com/zap-proto/zip"
)
// The shapes that make the gate ask. They are constants rather than per-org
// configuration on purpose: a threshold a tenant can widen is a threshold an
// attacker's target can be talked into widening, and these are properties of the
// protocol rather than preferences of a customer.
//
// They are deliberately narrower than the bot thresholds in agency.go: this is
// "worth a question", that is "already answered".
const (
// screenFailures — 401/403 in the window from one credential. A client with a
// stale token retries a handful of times; ten is a pattern.
screenFailures = 10
// screenPeers — distinct credentials presented from one address. Two is a
// laptop with two projects open; four is a list being worked through.
screenPeers = 4
// screenPaths — distinct paths one credential touched. A real integration
// walks a few endpoints; twenty-four is a survey.
screenPaths = 24
// screenFirst — a credential's first request in a window is always screened,
// so a freshly stolen key is judged on its first use rather than after it has
// done enough damage to trip a counter. This bounds the cost of screening to
// one per credential per minute, not one per request.
screenFirst = 1
)
// holdFor is how long a non-allow verdict is enforced before the scorer is asked
// again. Long enough that an attack costs one screen rather than one per
// request; short enough that a false positive clears itself in a minute without
// anyone being paged.
const holdFor = time.Minute
// screenCentsEnv names the operator knob for what one screen costs. Unset means
// ZERO: screens are COUNTED on the org's own usage ledger from day one, and
// priced when the SKU is priced. A default price here would be a fabricated one.
const screenCentsEnv = "CLOUD_RISK_SCREEN_CENTS"
// exemptPrefixes are the subtrees the gate does not watch. Two kinds, and each
// is a correctness requirement rather than a performance concession:
//
// - THE APPEAL SURFACE. /v1/risk is where a refused caller reads the decision
// that refused it and where an operator releases a hold. Putting the appeal
// behind the thing that refuses makes a false positive unappealable, which is
// not an acceptable property for a plane that refuses payments and logins.
// - the internal money plane. ScopeRateLimit exempts these for the same reason
// (its own config lives behind them), and a gate that could refuse them could
// refuse its own metering.
//
// Deliberately NOT here: /v1/ml and /v1/aml. They are ordinary API surfaces
// served by ordinary apps, and exempting a whole prefix family because the risk
// product happens to own part of it would leave the compliance face and the model
// plane unwatched. The gate calls the scorer as a Go function, not as a route
// (see cloud.Decide), so watching them cannot re-enter anything.
var exemptPrefixes = []string{
"/v1/risk/",
"/v1/billing/",
"/v1/commerce/",
"/_/",
}
// AbuseGate returns the lifecycle-defense middleware. t is the shared edge
// sensor; deps carries the policy store, the logger and the metering client.
//
// It is a no-op passthrough when the sensor is absent, mirroring every other
// gate on this path: an unwired deployment is never blocked.
func AbuseGate(deps Deps, t *edge.Traffic) zip.Handler {
if t == nil {
return func(c *zip.Ctx) error { return c.Next() }
}
g := &abuseGate{
traffic: t,
policy: deps.GatewayPolicy,
meter: NewResourceMeter(deps, "risk"),
log: deps.Logger,
cents: screenCents(),
}
return g.handle
}
func screenCents() int64 {
n, err := strconv.ParseInt(strings.TrimSpace(os.Getenv(screenCentsEnv)), 10, 64)
if err != nil || n < 0 {
return 0
}
return n
}
type abuseGate struct {
traffic *edge.Traffic
policy *edge.Store
meter *ResourceMeter
log interface {
Info(msg string, args ...any)
Warn(msg string, args ...any)
}
cents int64
}
func (g *abuseGate) handle(c *zip.Ctx) error {
// ONE normalization, at the top, and every comparison below is against it —
// the exemptions, the grant test, the path-spread key. c.Path() is the raw
// spelling the client sent; RoutePath is the form the router matches, so a
// difference in case or a trailing slash cannot put a request on one side of a
// security test and the other side of the routing table (see cloud.RoutePath).
path := RoutePath(c.Path())
if Probe(c.Method(), path) {
return c.Next()
}
for _, p := range exemptPrefixes {
if underPrefix(path, p) {
return c.Next()
}
}
// ONE observation, built in ONE place (observation). The tenant in it is the
// identity boundary's OWN answer and so is the caller's identity — see
// verifiedOrg and observation. The org is the sensor's keyspace index and the policy
// row this request is judged under, so a caller that could choose it could
// write into another tenant's state and read another tenant's posture.
now := time.Now()
sig := observation(c, routeFamily(path))
org := sig.Org
// Sense first, once. Every request is counted and lands in its lane whatever
// happens next — including one refused by a held verdict, because a caller
// that keeps knocking after being refused is exactly the caller a report needs
// to show. The lane comes BACK from the sensor: it is derived from the counts
// this observation just produced, so it cannot be stated before they exist.
p := g.traffic.Observe(sig, now)
lane := p.Lane
g.announce(c, sig, p)
// A verdict already in force short-circuits the question. The sensor holds it
// for at most a minute, so this is enforcement WITH a recent judgement behind
// it, never enforcement that outlives its reason.
if h, ok := g.traffic.Held(sig, now); ok {
return g.enforce(c, sig, lane, RiskVerdict{Action: h.Action, ID: h.Decision, Cause: h.Reason})
}
// A LAPSED hold forces the question again. Without it a hold buys a free
// minute at a time: the scorer sees more than the sensor does, so a verdict
// reached from the org's own history leaves no trace in a rolling minute of
// counts, and waiting for the local pattern to re-trip would wait forever.
privileged := Privileged(c.Method(), path)
if !g.screen(p, privileged) && !g.traffic.Lapsed(sig, now) {
return g.watch(c, sig)
}
// LIVE-ONLY QUESTION. In shadow the gate still senses and still reports, but
// it does not ask — because asking has two costs an unarmed org must not pay:
// a metered screen on its ledger, and the fail-CLOSED branch. A privileged
// grant denied by a scorer that was never installed would be an outage, not a
// defense, so arming is a decision an operator makes (and PUT /v1/gateway/config
// refuses to arm an org while no scorer is installed).
if g.mode(org) != edge.ModeLive {
return g.watch(c, sig)
}
v := Decide(c.Context(), org, RiskQuery{
Stage: StageUsage,
Subject: RiskSubject{Kind: "session", ID: subject(sig)},
Agency: lane,
Privileged: privileged,
Signals: Facts(map[string]string{
"credential": sig.Class,
"ip": sig.IP,
"path": sig.Path,
"method": c.Method(),
"requests": strconv.Itoa(p.Requests),
"failures": strconv.Itoa(p.Failures),
"paths": strconv.Itoa(p.Paths),
"peers": strconv.Itoa(p.Peers),
}),
})
if v.Agency != "" {
lane = v.Agency // the scorer's lane is the authoritative one.
}
// The screen is surfaced on BOTH rails, because they answer different
// questions and only one of them is live yet:
//
// o11y — Traffic.Screen counts it for the org from the first request,
// whatever a screen costs, and GET /v1/gateway/traffic reports it.
// billing — ResourceMeter puts it on the org's OWN usage ledger, the same
// rail every other metered resource rides, so a plan's included
// allowance and its overage are one ledger and not a second
// billing path.
//
// Meter is a NO-OP while the screen is unpriced (AmountCents<=0), which is the
// default: a price invented here would be a fabricated one, and the number
// belongs to the pricing catalog. That is exactly why the count above does not
// go through the ledger — an unpriced product must still be measurable.
// Both only in live mode: a shadow screen is not a product the org bought.
//
// AND ONLY A SCORED VERDICT IS BILLED. A screen the scorer never answered —
// absent, stuck, busy, timed out, erroring, silent — is judgement not rendered,
// and charging for it would put an outage of ours on a customer's invoice. The
// COUNT still happens either way, split into answered and unanswered, because
// "the judge stopped answering" is exactly the fact an operator must be able to
// read off the org's own report.
g.traffic.Screen(org, v.Refusal, now)
g.bill(org, principal.Project(c), c.RequestID(), sig.IP, v)
if v.Allowed() {
// The caller has been re-judged and is fine. Drop any lapsed hold, so it
// stops forcing a screen on every subsequent request.
g.traffic.Release(sig)
} else {
g.traffic.Hold(sig, edge.Hold{Action: v.Action, Reason: v.Cause, Decision: v.ID}, holdFor, now)
}
return g.enforce(c, sig, lane, v)
}
// screen reports whether this request's pattern is worth a question. A privileged
// grant always is: the fail-closed branch only protects what it is asked about.
//
// An UNMEASURED caller is not screened on cadence, because there is no cadence.
// Two ways a caller can be unmeasured, and both produce all-zero counts that
// "Requests <= 1" would read as "a caller making its first request" — for every
// request it ever makes, which is one screen each, a bill and a scorer stampede
// rather than a defense:
//
// refuse — the scope's ceiling turned this caller away.
// blind — the request carried no identity at all: no credential the boundary
// validated and no client address, so there is no caller to ask about.
// A subject of "" is not a question.
//
// A privileged grant is still screened either way: that branch protects the
// grant, not the sensor.
func (g *abuseGate) screen(p edge.Pattern, privileged bool) bool {
if privileged {
return true
}
if p.Strain == edge.StrainRefuse || p.Strain == edge.StrainBlind {
return false
}
return p.Requests <= screenFirst ||
p.Failures >= screenFailures ||
p.Peers >= screenPeers ||
p.Paths >= screenPaths
}
// bill puts one screen on the org's own usage ledger, and reports whether it
// did. ONLY a SCORED verdict is billable: a screen the scorer never answered —
// absent, stuck, busy, timed out, erroring, silent — is judgement not rendered,
// and charging for it would put an outage of ours on a customer's invoice.
//
// The rule lives in exactly one function because it is a money rule: the count
// (Traffic.Screen) and the debit answer different questions and only one of them
// is conditional, so the condition is stated once, here, where it can be read
// next to the charge it guards.
func (g *abuseGate) bill(org, project, request, ip string, v RiskVerdict) bool {
if !v.Scored() {
return false
}
g.meter.Meter(org, project, "screen", g.cents, request, ip)
return true
}
// announce logs a scope's sensor degradation ONCE per grade, not once per
// request. The sensor holds no logger and a flood produces one refused
// observation per request, so the grade change is carried back on the pattern and
// surfaced here — the alternative being a control that goes quiet with nothing in
// the log to say it did.
func (g *abuseGate) announce(c *zip.Ctx, sig edge.Signal, p edge.Pattern) {
if p.Rise == "" || p.Rise == edge.StrainClear || g.log == nil {
return
}
g.log.Warn("edge sensor strained",
"strain", p.Rise,
"org", sig.Org,
"lane", p.Lane,
"request", c.RequestID(),
)
}
// subject names WHAT the scorer is being asked about: the validated caller when
// there is one, and the address it came from when there is not. It is the sensor's
// key, so the thing that is judged and the thing that is held are the same thing —
// asking about a string the caller picked would let a refused caller be re-judged
// as somebody else by editing one header.
func subject(s edge.Signal) string {
if s.Cred != "" {
return s.Cred
}
return s.IP
}
// mode is the org's posture. An EMPTY org — the anonymous lane, which is where a
// bad bot calls from — resolves to the PLATFORM row rather than to shadow, or the
// one lane the gate exists for could never be armed. edge.Store.Mode owns that
// resolution; here it is only the nil-store case, which is shadow because an
// unwired deployment never enforces.
func (g *abuseGate) mode(org string) string {
if g.policy == nil {
return edge.ModeShadow
}
return g.policy.Mode(org)
}
// watch runs the request unjudged, and still learns from its outcome. A 401/403
// is the signal that separates a client with a stale token from one guessing
// them, and it only exists after the handler has run.
func (g *abuseGate) watch(c *zip.Ctx, sig edge.Signal) error {
err := c.Next()
g.observeOutcome(c, sig, err)
return err
}
// enforce applies a verdict. In shadow it applies NOTHING: it records what it
// would have done and lets the request through, which is what makes the mode
// switch a real one rather than a label.
func (g *abuseGate) enforce(c *zip.Ctx, sig edge.Signal, lane string, v RiskVerdict) error {
if v.Allowed() {
if v.Action == ActionReview {
g.record(c, sig, lane, v, "review")
}
return g.watch(c, sig)
}
if g.mode(sig.Org) != edge.ModeLive {
g.record(c, sig, lane, v, "shadow")
return g.watch(c, sig)
}
g.record(c, sig, lane, v, "enforced")
g.traffic.Deny(sig.Org, time.Now())
// The refusal is written in the fleet's own nested error contract, the same
// bytes every other Hanzo surface refuses with, so a client has ONE error
// shape to parse. The body names the decision id and never the score, the
// features or the rule: an attacker must not be able to use the refusal as a
// readout of the model that produced it.
switch v.Action {
case ActionChallenge:
c.SetHeader("WWW-Authenticate", `Bearer error="step_up_required"`)
return c.JSON(401, denyBody("step_up_required",
"Additional verification is required for this request. "+ref(v)))
case ActionRestrict:
c.SetHeader("Retry-After", strconv.Itoa(int(holdFor.Seconds())))
return c.JSON(429, denyBody("restricted",
"This credential is temporarily restricted. "+ref(v)))
default:
return c.JSON(403, denyBody("refused",
"This request was refused. "+ref(v)))
}
}
// ref names the decision a refusal came from, so a customer can quote one id to
// support and support can fetch the whole judgement — its rules, its features
// and the model digest — from GET /v1/risk/decisions/{id}. Without an id the
// refusal is unappealable, which is not an acceptable property for a product
// that refuses payments and logins.
func ref(v RiskVerdict) string {
if v.ID == "" {
return "Contact support if this is unexpected."
}
return "Reference " + v.ID + "."
}
// observeOutcome feeds the response status back into the sensor. Only 401/403
// count: those are the outcomes that mean a credential did not work, which is the
// signature of stuffing and of a replayed stolen token.
func (g *abuseGate) observeOutcome(c *zip.Ctx, sig edge.Signal, err error) {
if s := effectiveStatus(c.Fiber().Response().StatusCode(), err); s == 401 || s == 403 {
g.traffic.Fail(sig, time.Now())
}
}
// record surfaces a non-allow verdict to o11y. It is a LOG, not a second audit
// write: an enforced refusal is a 401/403 and AuditTrail already puts that in the
// tamper-evident trail, with the actor and the resource. Recording it twice would
// give one event two records that can disagree.
//
// A shadow verdict is the interesting one — it is the only evidence of what the
// gate WOULD do, and the only way stated-versus-realised can be measured before
// an org is armed.
func (g *abuseGate) record(c *zip.Ctx, sig edge.Signal, lane string, v RiskVerdict, outcome string) {
if g.log == nil {
return
}
g.log.Info("risk decision",
"stage", StageUsage,
"outcome", outcome,
"action", v.Action,
"org", sig.Org,
"cred", subject(sig),
"agency", lane,
"path", sig.Path,
"method", c.Method(),
"score", v.Score,
"decision", v.ID,
"cause", v.Cause,
// refusal is empty on a scored verdict and names the failure otherwise, so
// an allow that happened because nobody was listening is never read as a
// clean result.
"refusal", v.Refusal,
"request", c.RequestID(),
)
}
middleware_abuse_key_test.go
+269
View File
@@ -0,0 +1,269 @@
package cloud
// What the gate keys on, what it bills for, and what it says when it stops
// measuring. Every test here drives the real middleware through a real request,
// because the defect each one pins was invisible at the unit level: the sensor
// was correct about the key it was GIVEN, and the gate was giving it a string the
// caller had chosen.
import (
"context"
"fmt"
"testing"
"time"
"github.com/hanzoai/cloud/apps/gateway/edge"
)
// REGRESSION — a block hold was evaded five times out of five by rotating the
// Authorization header and nothing else. Enforcement was keyed on
// Fingerprint(callerToken): the raw Authorization value, validated or not, which
// is a string the ATTACKER PICKS. A caller presenting garbage was therefore
// HARDER to hold than one presenting nothing at all, which is a perverse
// incentive in exactly the anonymous lane the platform row exists to arm.
func TestAbuseGate_AHoldSurvivesCredentialRotation(t *testing.T) {
resetScorer(t)
SetRiskScorer(abuseVerdict(ActionBlock))
app, _ := abuseApp(t, edge.ModeLive)
const addr = "203.0.113.200"
if got := abuseHit(app, "GET", "/v1/models", "", "junk-000000", addr).StatusCode; got != 403 {
t.Fatalf("the first refusal did not land: %d", got)
}
for i := 1; i <= 5; i++ {
cred := fmt.Sprintf("junk-%06d", i)
if got := abuseHit(app, "GET", "/v1/models", "", cred, addr).StatusCode; got != 403 {
t.Fatalf("attempt %d with %q → %d: the hold was evaded by editing one header", i, cred, got)
}
}
// A caller presenting NO credential is held the same way — the two must not
// differ, or presenting garbage becomes the cheaper attack.
if got := abuseHit(app, "GET", "/v1/models", "", "", addr).StatusCode; got != 403 {
t.Fatalf("the same address with no credential → %d, want 403", got)
}
// A DIFFERENT address is a different caller and is judged on its own.
if got := abuseHit(app, "GET", "/v1/models", "", "junk-000000", "198.51.100.77").StatusCode; got != 403 {
t.Fatalf("a different address → %d; the scorer still blocks it on its own merits", got)
}
}
// The same rotation, seen from the sensor: one address is one row, whatever it
// puts in the header. Keying on the presented value opened a table entry per
// request, which is how the anonymous lane's ceiling was reached on demand.
func TestAbuseGate_RotatingACredentialDoesNotMintCallers(t *testing.T) {
resetScorer(t)
app, tr := abuseApp(t, edge.ModeShadow)
for i := 0; i < 500; i++ {
abuseHit(app, "GET", "/v1/models", "", fmt.Sprintf("junk-%06d", i), "203.0.113.201")
}
v := tr.View("", edge.ModeShadow, abuseNow())
if len(v.Callers) != 1 {
t.Fatalf("500 invented credentials from one address opened %d caller rows, want 1", len(v.Callers))
}
if v.Callers[0].Requests != 500 {
t.Fatalf("the address accumulated %d requests, want 500 — the counts reset per credential", v.Callers[0].Requests)
}
// The invented credentials are still counted as spread: this is the stuffing
// signature, and losing it would blind the sensor to the attack it is for.
if v.Lanes[edge.AgencyBot] == 0 {
t.Fatalf("an address presenting 500 credentials is the bot lane: %v", v.Lanes)
}
}
// A VALIDATED credential is still its own caller. The fix is "only a credential
// we issued names a caller", not "credentials stop mattering" — without this the
// whole per-credential sensor would collapse to a per-IP one.
func TestAbuseGate_AValidatedCredentialIsStillItsOwnCaller(t *testing.T) {
resetScorer(t)
app, tr := abuseApp(t, edge.ModeShadow)
abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "203.0.113.5")
abuseHit(app, "GET", "/v1/models", "acme", "sk-live-2", "203.0.113.5")
v := tr.View("acme", edge.ModeShadow, abuseNow())
if len(v.Callers) != 2 {
t.Fatalf("two validated credentials from one address = %d rows, want 2", len(v.Callers))
}
for _, c := range v.Callers {
if len(c.Cred) != fingerprintLen {
t.Fatalf("a validated caller must be keyed by its fingerprint, got %q", c.Cred)
}
}
}
// The scorer must be asked about the SAME thing the sensor holds. Asking about a
// string the caller picked would let a refused caller be re-judged as somebody
// else by editing one header — the hold would stand and the question would not.
func TestAbuseGate_TheScorerIsAskedAboutTheCallerNotTheHeader(t *testing.T) {
resetScorer(t)
var subjects []string
SetRiskScorer(func(_ context.Context, _ string, q RiskQuery) (RiskVerdict, error) {
subjects = append(subjects, q.Subject.ID)
return RiskVerdict{ID: "d", Action: ActionAllow}, nil
})
app, _ := abuseApp(t, edge.ModeLive)
abuseHit(app, "GET", "/v1/models", "", "junk-a", "203.0.113.210")
abuseHit(app, "POST", "/v1/iam/mint-user-keys", "", "junk-b", "203.0.113.210")
if len(subjects) < 2 {
t.Fatalf("the scorer was asked %d times, want at least 2", len(subjects))
}
if subjects[0] != subjects[1] {
t.Fatalf("one caller was asked about as two subjects (%q, %q)", subjects[0], subjects[1])
}
if subjects[0] != "203.0.113.210" {
t.Fatalf("subject = %q, want the address — the only identity an unvalidated caller has", subjects[0])
}
}
// A caller the sensor's ceiling turned away is UNMEASURED, and must not be read
// as a caller making its first request: that pattern screens EVERY time, so a
// flood the sensor cannot track would become one scorer call and one metered
// screen per request — a bill and a stampede, produced by the bound that was
// supposed to protect the process.
func TestAbuseGate_AnUnmeasuredCallerIsNotScreenedAsNew(t *testing.T) {
resetScorer(t)
asked := 0
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
asked++
return RiskVerdict{ID: "d", Action: ActionAllow}, nil
})
app, tr := abuseApp(t, edge.ModeLive)
// Fill the anonymous lane's ceiling with live callers. The ceiling is read off
// the report rather than hard-coded: the report is where an operator reads it.
ceiling := tr.View("", edge.ModeLive, time.Now()).Ceiling
for i := 0; i < ceiling; i++ {
tr.Observe(edge.Signal{Org: "", IP: fmt.Sprintf("10.%d.%d.%d", i/65536, i/256%256, i%256),
Path: "/v1/models", Class: edge.CredAnonymous}, time.Now())
}
before := asked
for i := 0; i < 20; i++ {
abuseHit(app, "GET", "/v1/models", "", "", fmt.Sprintf("198.51.%d.%d", i/256, i%256))
}
if asked != before {
t.Fatalf("the scorer was asked %d times about callers the sensor could not measure", asked-before)
}
// A privileged grant is still screened: that branch protects the grant, not
// the sensor.
before = asked
abuseHit(app, "POST", "/v1/iam/mint-user-keys", "", "", "198.51.200.1")
if asked == before {
t.Fatal("a privileged grant must be screened even when the sensor is saturated")
}
// And the saturation is READABLE, in the lane that has no tenant.
if v := tr.View("", edge.ModeLive, time.Now()); v.Strain != edge.StrainRefuse || v.Refused == 0 {
t.Fatalf("the anonymous lane's saturation is not readable: %+v", v)
}
}
// Money: a screen the scorer never answered is judgement not rendered, and
// charging for it would put an outage of ours on a customer's invoice.
func TestAbuseGate_OnlyAScoredScreenIsBilled(t *testing.T) {
g := &abuseGate{}
for _, tc := range []struct {
v RiskVerdict
bill bool
}{
{RiskVerdict{ID: "d", Action: ActionAllow}, true},
{RiskVerdict{ID: "d", Action: ActionBlock, Score: 0.9}, true},
{RiskVerdict{Action: ActionAllow, Refusal: RefusalAbsent}, false},
{RiskVerdict{Action: ActionAllow, Refusal: RefusalBusy}, false},
{RiskVerdict{Action: ActionAllow, Refusal: RefusalStuck}, false},
{RiskVerdict{Action: ActionAllow, Refusal: RefusalTimeout}, false},
{RiskVerdict{Action: ActionAllow, Refusal: RefusalError}, false},
{RiskVerdict{Action: ActionAllow, Refusal: RefusalSilent}, false},
{RiskVerdict{Action: ActionBlock, Refusal: RefusalUnknown}, false},
} {
if got := g.bill("acme", "default", "req-1", "203.0.113.1", tc.v); got != tc.bill {
t.Errorf("bill(%+v) = %v, want %v", tc.v, got, tc.bill)
}
}
}
// And the unanswered screens are COUNTED, so a scorer that has silently stopped
// answering is a number on the org's own report rather than a quiet day.
func TestAbuseGate_UnansweredScreensAreOnTheOrgsReport(t *testing.T) {
resetScorer(t)
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{}, nil // installed, answers with nothing
})
app, tr := abuseApp(t, edge.ModeLive)
for i := 0; i < 3; i++ {
abuseHit(app, "GET", "/v1/models", "acme", fmt.Sprintf("sk-live-%d", i), "203.0.113.5")
}
v := tr.View("acme", edge.ModeLive, abuseNow())
if v.Screens != 3 || v.Unscored != 3 {
t.Fatalf("screens=%d unscored=%d, want 3 and 3 — a dark scorer must be visible", v.Screens, v.Unscored)
}
}
// The lane split is the one number this report exists for, and it used to be a
// constant: the gate passed the lane IN, before it had asked the sensor what the
// caller had been doing, so every request ever counted landed in "unknown".
func TestAbuseGate_TheLaneSplitIsTheRealLane(t *testing.T) {
resetScorer(t)
app, tr := abuseApp(t, edge.ModeShadow)
abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "203.0.113.5")
abuseHit(app, "GET", "/v1/models", "acme", "eyJhbGciOi.payload.sig", "203.0.113.6")
v := tr.View("acme", edge.ModeShadow, abuseNow())
if v.Lanes[edge.AgencyAgent] != 1 || v.Lanes[edge.AgencyHuman] != 1 {
t.Fatalf("lanes = %v, want one agent and one human", v.Lanes)
}
if v.Lanes[edge.AgencyUnknown] != 0 {
t.Fatalf("attributable traffic landed in %q: %v", edge.AgencyUnknown, v.Lanes)
}
}
// THE DEPLOYMENT THIS RUNS IN TODAY. The DigitalOcean load balancer in front of
// the ingress is TCP with TLS passthrough and no PROXY protocol, so traefik's
// peer is the balancer's own VPC address and no client address reaches this
// process at all. ClientIP correctly answers "" — every hop in the chain is ours
// — and an anonymous caller therefore has NO identity.
//
// The dangerous spelling is to key that under the empty address: one row for the
// entire internet, counts that add up to a stuffing signature within seconds, and
// a single held verdict that refuses everybody. The sensor must decline to
// attribute it, keep counting the volume, and say what it cannot see.
func TestAbuseGate_NoClientAddressIsNotOneGiantCaller(t *testing.T) {
resetScorer(t)
asked := 0
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
asked++
return RiskVerdict{ID: "d", Action: ActionBlock, Cause: "peers"}, nil
})
app, tr := abuseApp(t, edge.ModeLive)
// No X-Forwarded-For at all, which in this deployment is every internet
// request: 40 different callers, none of whom can be told apart.
for i := 0; i < 40; i++ {
if got := abuseHit(app, "GET", "/v1/models", "", fmt.Sprintf("junk-%03d", i), "").StatusCode; got != 200 {
t.Fatalf("request %d → %d: one verdict was enforced against unidentifiable traffic", i, got)
}
}
if asked != 0 {
t.Fatalf("the scorer was asked %d times about a caller that has no identity", asked)
}
v := tr.View("", edge.ModeLive, time.Now())
if v.Blind != 40 || v.Strain != edge.StrainBlind {
t.Fatalf("the sensor must report what it cannot see: blind=%d strain=%q", v.Blind, v.Strain)
}
if v.Requests != 40 {
t.Fatalf("the volume must still be visible: requests=%d, want 40", v.Requests)
}
if len(v.Callers) != 0 {
t.Fatalf("unidentifiable traffic produced %d caller rows: %+v", len(v.Callers), v.Callers)
}
// A VALIDATED credential still names its caller in the same deployment — the
// credentialed plane keeps working with no client address at all.
abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "")
if c := tr.View("acme", edge.ModeLive, time.Now()).Callers; len(c) != 1 {
t.Fatalf("a validated caller with no address must still be its own row, got %+v", c)
}
}
middleware_abuse_test.go
+646
View File
@@ -0,0 +1,646 @@
package cloud
// The abuse gate's specification. Four groups, each answering one question a
// reviewer will ask:
//
// POLICY OUTCOMES — does each verdict produce the right wire answer?
// FAIL POLICY — does an absent/erroring scorer allow ordinary traffic and
// refuse a privileged grant?
// TENANT SCOPING — can one org's traffic reach, move or read another's?
// AGENCY — is a customer's automation told apart from a bad bot, and
// is it told apart from FACTS rather than from a header?
//
// Every test drives the middleware through a real zip app and a real request, so
// what is asserted is what a client would see.
import (
"context"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/hanzoai/cloud/apps/gateway/edge"
"github.com/hanzoai/cloud/apps/principal"
"github.com/zap-proto/zip"
)
// abuseApp mounts the gate in front of a trivial handler, with a fresh sensor and
// a policy store forced to mode. A nil store means "no policy configured", which
// must behave as shadow.
//
// The gate reads the identity boundary's OWN attestation, never a header, so the
// app installs `attest` in front of it — the stand-in for SanitizeIdentity. That
// is not test scaffolding around a gap: it is the property under test. An app
// WITHOUT it (abuseAppUnattested below) must see every caller as anonymous no
// matter what headers arrive.
func abuseApp(t *testing.T, mode string) (*zip.App, *edge.Traffic) {
t.Helper()
return abuseAppWith(t, mode, withBoundary)
}
// The two deployment shapes a middleware can find itself in. They are the same
// app apart from whether the identity boundary ran, which is exactly the fact the
// gate must turn on.
const (
withBoundary = true
withoutBoundary = false
)
// attest is the test's identity boundary: it mints the principal from the same
// headers SanitizeIdentity mints it from, and parks it where only a boundary can.
func attest() zip.Handler {
return func(c *zip.Ctx) error {
if u := c.User(); u != "" {
principal.Mint(c, principal.Principal{Org: c.Org(), User: u})
} else {
principal.Mint(c, principal.Principal{})
}
return c.Next()
}
}
// abuseAppWith mounts the gate with or without the identity boundary in front of
// it. Without it — the shape a hand-written plugin process has — every caller is
// anonymous by construction, whatever headers arrive.
func abuseAppWith(t *testing.T, mode string, boundary bool) (*zip.App, *edge.Traffic) {
t.Helper()
tr := edge.NewTraffic()
deps := Deps{GatewayPolicy: staticModeStore(t, mode), Traffic: tr}
app := zip.New(zip.Config{})
if boundary {
app.Use(attest())
}
app.Use(AbuseGate(deps, tr))
h := func(c *zip.Ctx) error { return c.JSON(http.StatusOK, map[string]string{"ok": "1"}) }
app.Get("/v1/models", h)
app.Get("/v1/models/:name", h)
app.Post("/v1/iam/mint-user-keys", h)
app.Get("/v1/kms/orgs/acme/secrets/db", h)
app.Get("/health", h)
app.Get("/v1/risk/decisions", h)
// A route that refuses, so the 401/403 feedback loop can be exercised.
app.Get("/v1/denied", func(c *zip.Ctx) error {
return c.JSON(http.StatusUnauthorized, map[string]string{"error": "no"})
})
return app, tr
}
// staticModeStore builds a real edge.Store armed to mode — on the platform row
// (the anonymous lane) and on each tenant these tests drive, BY NAME. Mode does
// not inherit: arming one scope arms exactly that scope, which is the property
// TestPolicy_ArmingThePlatformRowDoesNotArmTenants pins. Using the real store
// rather than a fake keeps the test honest about how Mode actually resolves.
func staticModeStore(t *testing.T, mode string) *edge.Store {
t.Helper()
s, err := edge.New(t.TempDir(), "admin", edge.Policy{Mode: mode})
if err != nil {
t.Fatalf("edge.New: %v", err)
}
t.Cleanup(func() { _ = s.Close() })
if mode != "" {
for _, org := range []string{"acme", "globex"} {
if _, err := s.Put(t.Context(), org, edge.Policy{Mode: mode}); err != nil {
t.Fatalf("arm %s: %v", org, err)
}
}
}
return s
}
// call drives one request. org is the validated tenant (the header
// SanitizeIdentity mints); cred is the raw credential the caller presents; ip is
// the forwarded client address.
func abuseHit(app *zip.App, method, path, org, cred, ip string) *http.Response {
req := httptest.NewRequest(method, path, nil)
if org != "" {
req.Header.Set("X-Org-Id", org)
req.Header.Set("X-User-Id", "u-"+org)
}
if cred != "" {
req.Header.Set("Authorization", "Bearer "+cred)
}
if ip != "" {
req.Header.Set("X-Forwarded-For", ip)
}
resp, err := app.Fiber().Test(req)
if err != nil {
return &http.Response{StatusCode: 0}
}
return resp
}
func abuseVerdict(action string) RiskScorer {
return func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{ID: "d-" + action, Action: action, Score: 0.9, Cause: "test"}, nil
}
}
// ---------------------------------------------------------------- policy outcomes
func TestAbuseGate_EveryPolicyOutcome(t *testing.T) {
cases := []struct {
action string
status int
header string // a response header the outcome must carry
}{
{ActionAllow, 200, ""},
{ActionReview, 200, ""}, // review summons a person; it does not stop traffic.
{ActionChallenge, 401, "WWW-Authenticate"},
{ActionRestrict, 429, "Retry-After"},
{ActionBlock, 403, ""},
}
for _, tc := range cases {
t.Run(tc.action, func(t *testing.T) {
resetScorer(t)
SetRiskScorer(abuseVerdict(tc.action))
app, _ := abuseApp(t, edge.ModeLive)
resp := abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "203.0.113.9")
if resp.StatusCode != tc.status {
t.Fatalf("%s → %d, want %d", tc.action, resp.StatusCode, tc.status)
}
if tc.header != "" && resp.Header.Get(tc.header) == "" {
t.Fatalf("%s must carry a %s header", tc.action, tc.header)
}
})
}
}
// A refusal names the decision it came from, so a customer can quote one id and
// support can fetch the whole judgement. It must NOT name the score, the rule or
// the features: a refusal that reads out the model is a refusal an attacker tunes
// against.
func TestAbuseGate_RefusalIsAppealableAndSaysNothingElse(t *testing.T) {
resetScorer(t)
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{ID: "d-77", Action: ActionBlock, Score: 0.98, Cause: "peers>=8"}, nil
})
app, _ := abuseApp(t, edge.ModeLive)
resp := abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "203.0.113.9")
body := abuseBody(t, resp)
if !strings.Contains(body, "d-77") {
t.Fatalf("a refusal must name its decision id so it can be appealed: %s", body)
}
for _, leak := range []string{"0.98", "peers", "score"} {
if strings.Contains(body, leak) {
t.Fatalf("a refusal must not read out the model (%q leaked): %s", leak, body)
}
}
}
// SHADOW IS THE DEFAULT AND IT REFUSES NOTHING. This is the test that stops a
// statistical judgement from silently starting to block an org's traffic.
func TestAbuseGate_ShadowNeverRefuses(t *testing.T) {
for _, mode := range []string{"", edge.ModeShadow} {
t.Run("mode="+mode, func(t *testing.T) {
resetScorer(t)
SetRiskScorer(abuseVerdict(ActionBlock))
app, _ := abuseApp(t, mode)
resp := abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "203.0.113.9")
if resp.StatusCode != 200 {
t.Fatalf("shadow must not enforce: got %d, want 200", resp.StatusCode)
}
})
}
}
// A held verdict is what makes an attack cost one screen instead of one per
// request — and it must lapse, so a false positive clears itself.
func TestAbuseGate_HoldsAVerdictSoTheScorerIsAskedOnce(t *testing.T) {
resetScorer(t)
asked := 0
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
asked++
return RiskVerdict{ID: "d-1", Action: ActionBlock}, nil
})
app, _ := abuseApp(t, edge.ModeLive)
for i := 0; i < 5; i++ {
if got := abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "203.0.113.9").StatusCode; got != 403 {
t.Fatalf("request %d → %d, want 403", i, got)
}
}
if asked != 1 {
t.Fatalf("the scorer was asked %d times for one held verdict; want 1", asked)
}
}
// -------------------------------------------------------------------- fail policy
func TestAbuseGate_FailsOpenForOrdinaryTrafficAndClosedForAGrant(t *testing.T) {
// A scorer that is INSTALLED and cannot answer. Absence is the other case and
// it is deliberately not here: a deployment with no risk plane, or one whose
// scorer withdrew, must not have its grant surface refused — see
// TestDecide_AnAbsentScorerIsNotASilentOne and the case below.
broken := []struct {
name string
install RiskScorer
}{
{"scorer erroring", func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{}, errors.New("down")
}},
{"scorer silent", func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{}, nil
}},
}
for _, b := range broken {
t.Run(b.name, func(t *testing.T) {
resetScorer(t)
SetRiskScorer(b.install)
app, _ := abuseApp(t, edge.ModeLive)
if got := abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "198.51.100.4").StatusCode; got != 200 {
t.Fatalf("ordinary traffic must FAIL OPEN when the scorer is down: got %d, want 200", got)
}
if got := abuseHit(app, "POST", "/v1/iam/mint-user-keys", "acme", "sk-live-2", "198.51.100.4").StatusCode; got != 403 {
t.Fatalf("minting a credential must FAIL CLOSED when the scorer is down: got %d, want 403", got)
}
if got := abuseHit(app, "GET", "/v1/kms/orgs/acme/secrets/db", "acme", "sk-live-3", "198.51.100.4").StatusCode; got != 403 {
t.Fatalf("reading the key store must FAIL CLOSED when the scorer is down: got %d, want 403", got)
}
})
}
// AND THE OTHER HALF, on the same live org: with NO scorer in the process the
// grant surface stays open. An armed org whose scorer was never installed, or
// withdrew, must not have its key store answer 403 to everyone.
t.Run("scorer absent", func(t *testing.T) {
resetScorer(t)
SetRiskScorer(nil)
app, _ := abuseApp(t, edge.ModeLive)
for _, p := range []string{"/v1/kms/orgs/acme/secrets/db"} {
if got := abuseHit(app, "GET", p, "acme", "sk-live-4", "198.51.100.4").StatusCode; got != 200 {
t.Fatalf("%s → %d with no scorer installed; want 200", p, got)
}
}
if got := abuseHit(app, "POST", "/v1/iam/mint-user-keys", "acme", "sk-live-5", "198.51.100.4").StatusCode; got != 200 {
t.Fatalf("credential minting → %d with no scorer installed; want 200", got)
}
})
}
// The fail-CLOSED branch is reachable only in live mode. An unarmed deployment —
// which is every deployment until an operator arms it — must not have its key
// store and its credential minting refused because a scorer was never installed.
// That would be an outage wearing a defense's clothes.
func TestAbuseGate_ShadowDoesNotFailClosedOnAGrant(t *testing.T) {
resetScorer(t)
SetRiskScorer(nil)
app, _ := abuseApp(t, edge.ModeShadow)
if got := abuseHit(app, "POST", "/v1/iam/mint-user-keys", "acme", "sk-live-1", "198.51.100.4").StatusCode; got != 200 {
t.Fatalf("an unarmed org must not be locked out of credential minting: got %d, want 200", got)
}
}
// The gate must never call the scorer on the scorer's own surface: it runs
// in-process, so screening /v1/risk would re-enter it on its own answer.
func TestAbuseGate_NeverScreensTheScorersOwnSurface(t *testing.T) {
resetScorer(t)
asked := 0
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
asked++
return RiskVerdict{ID: "d", Action: ActionBlock}, nil
})
app, _ := abuseApp(t, edge.ModeLive)
for _, p := range []string{"/v1/risk/decisions", "/health"} {
if got := abuseHit(app, "GET", p, "acme", "sk-live-1", "198.51.100.4").StatusCode; got != 200 {
t.Fatalf("%s must be exempt: got %d", p, got)
}
}
if asked != 0 {
t.Fatalf("the scorer was asked %d times about exempt paths; want 0", asked)
}
}
// A nil sensor must be a passthrough, exactly like every other gate on this path:
// an unwired deployment is never blocked.
func TestAbuseGate_UnwiredIsAPassthrough(t *testing.T) {
resetScorer(t)
SetRiskScorer(abuseVerdict(ActionBlock))
app := zip.New(zip.Config{})
app.Use(AbuseGate(Deps{}, nil))
app.Get("/v1/models", func(c *zip.Ctx) error { return c.JSON(200, map[string]string{"ok": "1"}) })
if got := abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "").StatusCode; got != 200 {
t.Fatalf("an unwired gate must pass through: got %d", got)
}
}
// ----------------------------------------------------------------- tenant scoping
// One org's traffic must not be able to reach, move or read another's. This is
// the property the whole product rests on.
func TestAbuseGate_TenantsAreIsolated(t *testing.T) {
resetScorer(t)
var askedFor []string
SetRiskScorer(func(_ context.Context, org string, _ RiskQuery) (RiskVerdict, error) {
askedFor = append(askedFor, org)
if org == "acme" {
return RiskVerdict{ID: "d-acme", Action: ActionBlock}, nil
}
return RiskVerdict{ID: "d-other", Action: ActionAllow}, nil
})
app, tr := abuseApp(t, edge.ModeLive)
// acme is blocked, on its own credential, from its own address.
if got := abuseHit(app, "GET", "/v1/models", "acme", "sk-acme", "203.0.113.1").StatusCode; got != 403 {
t.Fatalf("acme → %d, want 403", got)
}
// globex, presenting the SAME credential string from the SAME address, is
// untouched: the sensor keys on (org, credential), so acme's hold cannot reach
// it and its counters are its own.
if got := abuseHit(app, "GET", "/v1/models", "globex", "sk-acme", "203.0.113.1").StatusCode; got != 200 {
t.Fatalf("globex → %d, want 200 — acme's verdict crossed the tenant boundary", got)
}
for _, org := range askedFor {
if org != "acme" && org != "globex" {
t.Fatalf("the scorer was asked about %q, which is neither caller's org", org)
}
}
// And neither org's view can contain the other's rows.
av := tr.View("acme", edge.ModeLive, abuseNow())
gv := tr.View("globex", edge.ModeLive, abuseNow())
if av.Org != "acme" || gv.Org != "globex" {
t.Fatal("a view must report the org it was taken for")
}
if len(av.Callers) == 0 || len(gv.Callers) == 0 {
t.Fatal("each org must see its own caller")
}
if av.Requests == 0 || gv.Requests == 0 {
t.Fatal("each org must count its own requests")
}
if av.Requests != 1 || gv.Requests != 1 {
t.Fatalf("counts leaked across tenants: acme=%d globex=%d, want 1 each", av.Requests, gv.Requests)
}
}
// An anonymous caller has no tenant. Its traffic must not be attributed to, or
// able to move, any org's numbers.
func TestAbuseGate_AnonymousTrafficMovesNoTenant(t *testing.T) {
resetScorer(t)
SetRiskScorer(abuseVerdict(ActionAllow))
app, tr := abuseApp(t, edge.ModeLive)
for i := 0; i < 5; i++ {
abuseHit(app, "GET", "/v1/models", "", "", "198.51.100.77")
}
if v := tr.View("acme", edge.ModeLive, abuseNow()); v.Requests != 0 || len(v.Callers) != 0 {
t.Fatalf("anonymous traffic reached a tenant's numbers: %+v", v)
}
}
// ------------------------------------------------------------------------ agency
// The differentiator: a customer's automation and a bad bot are told apart by
// the credential we issued, never by anything the client says about itself.
func TestAbuseGate_ClassesAgentTrafficApartFromBots(t *testing.T) {
resetScorer(t)
var lanes []string
SetRiskScorer(func(_ context.Context, _ string, q RiskQuery) (RiskVerdict, error) {
lanes = append(lanes, q.Agency)
return RiskVerdict{ID: "d", Action: ActionAllow}, nil
})
app, _ := abuseApp(t, edge.ModeLive)
// A machine credential we issued: the agent lane, whatever the user-agent says.
abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "203.0.113.5")
if len(lanes) == 0 || lanes[0] != edge.AgencyAgent {
t.Fatalf("attributable machine traffic must be the agent lane, got %v", lanes)
}
// The same request claiming to be a browser is STILL the agent lane: we read
// the credential, not the client's self-description.
req := httptest.NewRequest("GET", "/v1/models", nil)
req.Header.Set("X-Org-Id", "acme")
req.Header.Set("X-User-Id", "u-acme")
req.Header.Set("Authorization", "Bearer sk-live-2")
req.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh) Chrome/126")
req.Header.Set("X-Forwarded-For", "203.0.113.5")
if _, err := app.Fiber().Test(req); err != nil {
t.Fatal(err)
}
if lanes[len(lanes)-1] != edge.AgencyAgent {
t.Fatalf("a user-agent string must not move a caller between lanes, got %q", lanes[len(lanes)-1])
}
}
// -------------------------------------------------------------------- the sensor
// The 401/403 feedback loop: a credential that keeps failing to authenticate is
// how stuffing and a replayed stolen token look, and the count only exists after
// the handler has run.
func TestAbuseGate_CountsAuthFailuresAfterTheHandler(t *testing.T) {
resetScorer(t)
SetRiskScorer(abuseVerdict(ActionAllow))
app, tr := abuseApp(t, edge.ModeShadow)
for i := 0; i < 3; i++ {
abuseHit(app, "GET", "/v1/denied", "acme", "sk-live-1", "203.0.113.5")
}
v := tr.View("acme", edge.ModeShadow, abuseNow())
if len(v.Callers) != 1 {
t.Fatalf("want one caller, got %d", len(v.Callers))
}
if v.Callers[0].Failures != 3 {
t.Fatalf("failures = %d, want 3", v.Callers[0].Failures)
}
}
// A credential must never appear anywhere the sensor can be read from. The
// fingerprint is keyed with a per-process salt, so it cannot be tested against a
// candidate key off-box either.
func TestAbuseGate_NeverSurfacesACredential(t *testing.T) {
resetScorer(t)
SetRiskScorer(abuseVerdict(ActionAllow))
app, tr := abuseApp(t, edge.ModeShadow)
const secret = "sk-live-supersecret-value"
abuseHit(app, "GET", "/v1/models", "acme", secret, "203.0.113.5")
v := tr.View("acme", edge.ModeShadow, abuseNow())
blob := fmt.Sprintf("%+v", v)
if strings.Contains(blob, secret) || strings.Contains(blob, "supersecret") {
t.Fatalf("the credential reached the traffic view: %s", blob)
}
if len(v.Callers) != 1 || v.Callers[0].Cred == "" {
t.Fatalf("the caller must still be identifiable by fingerprint: %+v", v)
}
if got := Fingerprint(secret); got != v.Callers[0].Cred {
t.Fatalf("the fingerprint must be stable within a process: %q vs %q", got, v.Callers[0].Cred)
}
if Fingerprint(secret) == Fingerprint(secret+"x") {
t.Fatal("two credentials must not share a fingerprint")
}
}
func abuseBody(t *testing.T, resp *http.Response) string {
t.Helper()
defer func() { _ = resp.Body.Close() }()
b := make([]byte, 4096)
n, _ := resp.Body.Read(b)
return string(b[:n])
}
// nowish is the clock the sensor tests read. A single call site so a future move
// to an injected clock has one place to change.
func abuseNow() time.Time { return time.Now() }
// A hold must not buy a free minute at a time. When it lapses, the caller is
// asked about AGAIN on its next request — whether or not the local pattern still
// looks unusual, because the scorer sees things the sensor cannot: prior accounts
// on a device, a spend curve, a chargeback. Waiting for a rolling minute of
// request counts to re-trip would wait forever.
func TestAbuseGate_ALapsedHoldForcesTheQuestionAgain(t *testing.T) {
resetScorer(t)
asked := 0
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
asked++
return RiskVerdict{ID: "d-1", Action: ActionBlock}, nil
})
tr := edge.NewTraffic()
deps := Deps{GatewayPolicy: staticModeStore(t, edge.ModeLive), Traffic: tr}
app := zip.New(zip.Config{})
app.Use(attest())
app.Use(AbuseGate(deps, tr))
app.Get("/v1/models", func(c *zip.Ctx) error { return c.JSON(200, map[string]string{"ok": "1"}) })
sig := edge.Signal{Org: "acme", Cred: Fingerprint("sk-live-1"), IP: "203.0.113.9"}
// First request screens and is held.
if got := abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "203.0.113.9").StatusCode; got != 403 {
t.Fatalf("first request → %d, want 403", got)
}
if asked != 1 {
t.Fatalf("asked %d times, want 1", asked)
}
// Held: no further questions while it is in force.
abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "203.0.113.9")
if asked != 1 {
t.Fatalf("a held verdict must not re-ask: asked %d", asked)
}
// Lapse it. The pattern is now unremarkable — a handful of requests, one
// path, no failures, no peers — so nothing but the lapse can force the ask.
// Backdate it rather than sleeping: a hold whose deadline is already in the
// past is unambiguously lapsed, where a millisecond one is a race.
tr.Hold(sig, edge.Hold{Action: ActionBlock}, time.Second, time.Now().Add(-time.Minute))
if _, ok := tr.Held(sig, time.Now()); ok {
t.Fatal("the hold did not lapse")
}
before := asked
abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "203.0.113.9")
if asked != before+1 {
t.Fatalf("a lapsed hold must force the question again: asked %d, want %d", asked, before+1)
}
// And once the scorer allows, the lapsed hold is dropped so it stops forcing
// a screen on every request thereafter.
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
asked++
return RiskVerdict{ID: "d-ok", Action: ActionAllow}, nil
})
tr.Hold(sig, edge.Hold{Action: ActionBlock}, time.Second, time.Now().Add(-time.Minute))
abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "203.0.113.9") // re-judged: allow, hold released
cleared := asked
abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "203.0.113.9")
if asked != cleared {
t.Fatalf("a released hold must stop forcing screens: asked %d, want %d", asked, cleared)
}
}
// ------------------------------------------------------- the forgery regressions
// REGRESSION — the differentiator must not be settable by the caller. The lane
// used to turn on `c.Org() != "" || c.User() != ""`, and BOTH disjuncts are
// header reads: X-Org-Id survives the boundary on the anonymous path by design,
// and in a process with no boundary installed nothing strips either header. So
// two headers plus an sk--shaped string that never validated moved a bad bot into
// the AGENT lane — the lane whose entire meaning is "a credential WE minted, to a
// named tenant, that we can revoke".
//
// Here the gate runs with NO identity boundary in front of it, which is exactly
// the plugin-process shape. Every header the caller can write is written, and it
// must still be judged as anonymous.
func TestAbuseGate_HeadersAloneDoNotBuyTheAgentLane(t *testing.T) {
resetScorer(t)
var lanes, orgs []string
SetRiskScorer(func(_ context.Context, org string, q RiskQuery) (RiskVerdict, error) {
lanes, orgs = append(lanes, q.Agency), append(orgs, org)
return RiskVerdict{ID: "d", Action: ActionAllow}, nil
})
app, tr := abuseAppWith(t, edge.ModeLive, withoutBoundary)
req := httptest.NewRequest("GET", "/v1/models", nil)
req.Header.Set("X-Org-Id", "acme") // forged: no boundary minted it
req.Header.Set("X-User-Id", "u-acme") // forged
req.Header.Set("X-User-IsAdmin", "true") // forged
req.Header.Set("Authorization", "Bearer sk-live-forged")
req.Header.Set("X-Forwarded-For", "203.0.113.66")
if _, err := app.Fiber().Test(req); err != nil {
t.Fatal(err)
}
if len(lanes) == 0 {
t.Fatal("the anonymous lane must still be screened — it is the lane a bad bot calls from")
}
if lanes[0] == edge.AgencyAgent {
t.Fatal("two headers moved an unattributable caller into the agent lane")
}
// And the tenant is not the one the header named: an unverified caller must
// not be able to write into — or read the posture of — someone else's org.
if orgs[0] != "" {
t.Fatalf("the scorer was asked about %q, which no boundary attested", orgs[0])
}
if v := tr.View("acme", edge.ModeLive, abuseNow()); v.Requests != 0 || len(v.Callers) != 0 {
t.Fatalf("a forged X-Org-Id wrote into acme's sensor state: %+v", v)
}
}
// The same request WITH a boundary in front is the agent lane — so the test above
// is pinning the forgery, not merely a gate that never classes anything.
func TestAbuseGate_AVerifiedMachineCredentialIsTheAgentLane(t *testing.T) {
resetScorer(t)
var lanes, orgs []string
SetRiskScorer(func(_ context.Context, org string, q RiskQuery) (RiskVerdict, error) {
lanes, orgs = append(lanes, q.Agency), append(orgs, org)
return RiskVerdict{ID: "d", Action: ActionAllow}, nil
})
app, _ := abuseAppWith(t, edge.ModeLive, withBoundary)
abuseHit(app, "GET", "/v1/models", "acme", "sk-live-1", "203.0.113.5")
if len(lanes) == 0 || lanes[0] != edge.AgencyAgent {
t.Fatalf("a verified machine credential must be the agent lane, got %v", lanes)
}
if orgs[0] != "acme" {
t.Fatalf("the scorer was asked about %q, want the attested acme", orgs[0])
}
}
// REGRESSION — one capital letter used to flip fail-closed to fail-OPEN. The
// grant list is compared with strings.HasPrefix against the raw c.Path(), while
// fiber routes case-INSENSITIVELY and ignores a trailing slash: `/V1/KMS/...`
// reaches the key store and missed every prefix, so the scorer's silence ALLOWED
// a read of the key store instead of refusing it.
func TestAbuseGate_AGrantIsAGrantHoweverItIsSpelled(t *testing.T) {
for _, path := range []string{
"/v1/kms/orgs/acme/secrets/db",
"/V1/KMS/orgs/acme/secrets/db",
"/v1/KMS/orgs/acme/secrets/db",
"/v1/kms/orgs/acme/secrets/db/",
} {
t.Run(path, func(t *testing.T) {
// A scorer that is THERE and does not answer: the fail policy's
// closed branch, which is the one a grant path must take.
resetScorer(t)
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{}, errors.New("model unavailable")
})
app, _ := abuseApp(t, edge.ModeLive)
got := abuseHit(app, "GET", path, "acme", "sk-live-1", "203.0.113.9").StatusCode
if got != 403 {
t.Fatalf("%s → %d; an unscored read of the key store must be refused", path, got)
}
})
}
}
middleware_billing.go
+10 -1
View File
@@ -188,8 +188,17 @@ var serviceAliases = map[string]string{
// through serviceAliases to the canonical provider label. It is the scope's
// service axis — from the route, NEVER a client field, so a caller can never spoof
// another service's cap. Empty for non-/v1 paths.
//
// THE ROUTE IS WHAT THE ROUTER MATCHED, not how the client spelled it. fiber
// routes case-insensitively and ignores a trailing slash, so "/V1/AI/chat" reaches
// exactly the same handler as "/v1/ai/chat" — and read raw, it produced the
// service label "AI", which is a DIFFERENT scope key: a different rate bucket and
// a different spend-cap axis, reachable by holding down the shift key. The label
// is derived from cloud.RoutePath for the same reason the grant list is compared
// against it: a scope key must name the route, and the route is what the router
// says it is.
func canonicalService(path string) string {
p := strings.TrimPrefix(path, "/")
p := strings.TrimPrefix(RoutePath(path), "/")
parts := strings.SplitN(p, "/", 3)
if len(parts) < 2 || parts[0] != "v1" {
return ""
middleware_identity.go
+12
View File
@@ -44,6 +44,7 @@ import (
"time"
"github.com/hanzoai/authz"
"github.com/hanzoai/cloud/apps/principal"
"github.com/hanzoai/namespace"
"github.com/zap-proto/zip"
)
@@ -395,6 +396,11 @@ func SanitizeIdentity(v *identityValidator) zip.Handler {
req.Header.Set(authz.HeaderUserOrgAdmin, "true")
}
sanitizeSubScopes(c, effOrg, claims.renderProject(), cliApp, claims.renderBillingAccount())
// The boundary's own attestation, parked where no client can reach it
// (principal.Mint). The headers above are the contract everything
// DOWNSTREAM reads; this is the fact a middleware reads when it cannot
// prove it is downstream — see principal.Mint.
principal.Mint(c, principal.Principal{Org: effOrg, User: claims.userID()})
return c.Continue()
}
@@ -416,6 +422,12 @@ func SanitizeIdentity(v *identityValidator) zip.Handler {
if cliOrg != "" {
req.Header.Set(authz.HeaderOrg, cliOrg)
}
// The boundary RAN and found nobody. Recorded as such — an EMPTY attestation,
// which is a different fact from no attestation at all. The org restored just
// above is deliberately not in it: that value is the client's, kept for the
// Phase-1 data path, and the whole point of this slot is that nothing a client
// wrote ever enters it.
principal.Mint(c, principal.Principal{})
return c.Continue()
}
}
middleware_ratelimit.go
+10 -6
View File
@@ -26,7 +26,6 @@ package cloud
import (
"context"
"strings"
"sync"
"time"
@@ -94,10 +93,15 @@ func (rl *scopeRateLimiter) handler(c *zip.Ctx) error {
// gates still cover it — so exempting it loosens no user-facing ceiling.
// (rulesFor no longer reaches commerce through this app, so this is policy
// now and not a self-reference guard; see rulesFor.)
if p := c.Path(); strings.HasPrefix(p, "/v1/billing/") ||
strings.HasPrefix(p, "/v1/commerce/") ||
strings.HasPrefix(p, "/_/commerce/") {
return c.Next()
// Compared against the ROUTER's path, not the raw spelling: a prefix test over
// c.Path() answers a question about how the client typed the URL, while the
// exemption is about which handler will run (see cloud.RoutePath). ONE
// normalization, the same one the abuse gate and the grant list use.
path := RoutePath(c.Path())
for _, p := range []string{"/v1/billing/", "/v1/commerce/", "/_/commerce/"} {
if underPrefix(path, p) {
return c.Next()
}
}
// Only an authenticated org is scope-rate-limited. Without a validated
@@ -109,7 +113,7 @@ func (rl *scopeRateLimiter) handler(c *zip.Ctx) error {
return c.Next()
}
project := principal.Project(c)
service := canonicalService(c.Path())
service := canonicalService(path)
key, rpm := bindingRateRule(rl.rulesFor(org), org, project, service)
openapi.yaml
+170 -1
View File
@@ -4074,6 +4074,26 @@ components:
items:
type: string
type: array
mode:
description: |-
Mode is the abuse gate's posture for THIS scope: "shadow" scores traffic and
records the verdict without acting on it, "live" enforces it. Unset means
shadow.
It is the one per-org field that does NOT inherit. Every other field here
layers a platform default under the org's own value, which is right for a
default: a tenant that sets no rate ceiling should get the platform's. Mode
is not a default, it is an ARMING DECISION — it is what makes a statistical
judgement start refusing real traffic — and inheriting it means arming one
scope arms every tenant that never asked for it, without a write to their
row and without anything in their config changing. So a tenant is live only
if that tenant's OWN row says live, and the platform row's mode governs
exactly one scope: the anonymous lane, which has no tenant of its own.
It is also not self-service. Writing it requires SuperAdmin (see the
/v1/gateway config op): the subject of an abuse control does not get to
switch the control off.
type: string
org_rpm:
description: |-
OrgRPM is the org's OWN authenticated rate ceiling, requests per minute, as
@@ -6286,6 +6306,114 @@ components:
wastedGiB:
type: integer
type: object
TrafficCaller:
properties:
action:
description: Action is the verdict currently held against it, if any.
type: string
cred:
description: |-
Cred is the caller's key: a credential fingerprint (a per-process one-way
digest, not a key) for a validated caller, and "ip:<addr>" for one that
presented no credential we could validate.
type: string
failures:
description: Failures is how many ended 401 or 403.
type: integer
held_until:
description: HeldUntil is when the held verdict lapses, unix seconds.
type: integer
paths:
description: Paths is the approximate number of distinct paths it touched
(max 64).
type: integer
reason:
description: Reason is why that verdict was reached.
type: string
requests:
description: Requests is its request count in the window.
type: integer
type: object
TrafficView:
properties:
blind:
description: |-
Blind is how many requests in the window carried no identity to attribute
them to — no validated credential and no client address. Non-zero on a
public plane means the client address is not reaching this process (a TCP
load balancer with no PROXY protocol in front of it, typically), so this
scope's callers cannot be told apart and nothing can be held against them.
type: integer
callers:
description: |-
Callers is the scope's busiest callers this window. A credentialed caller
appears as a FINGERPRINT — a per-process one-way digest: enough to recognise
the same caller across requests, never enough to reconstruct the credential.
items:
$ref: '#/components/schemas/TrafficCaller'
type: array
ceiling:
description: Ceiling is the most callers this scope may hold at once.
type: integer
denied:
description: Denied is how many of them the gate refused.
type: integer
lanes:
additionalProperties:
type: integer
description: |-
Lanes is the request count per lane — agent, human, bot, unknown. This is
the split that separates a customer's automation from a scraper.
type: object
mode:
description: |-
Mode is the abuse gate's posture for this scope: "shadow" records the scorer's
action without enforcing it, "live" enforces it.
type: string
org:
description: |-
Org is the scope this view was taken for — the validated principal's own,
never a value the caller supplied. Empty names the anonymous lane, the one
scope that has no tenant.
type: string
refused:
description: Refused is how many callers this scope's ceilings turned away
in the window.
type: integer
requests:
description: Requests is how many requests this scope made in the window.
type: integer
screens:
description: |-
Screens is how many of them were put to the scorer — the billable unit of
the risk product. Counted from the first request, whatever the SKU costs.
type: integer
strain:
description: |-
Strain is what this scope's ceilings are doing: "clear" below them, "full"
at them, "refuse" once a caller has been turned away inside this window —
which means that caller is UNMEASURED and the numbers here are a sample
rather than a census. It is reported rather than logged because the
alternative — a bound that degrades a scope silently — is the failure this
design exists to rule out. No other scope can move it.
type: string
tracked:
description: |-
Tracked is how many callers this scope holds state for right now, and
Ceiling is the most it may hold. Tracked == Ceiling is the fact a bound
that binds cannot hide.
type: integer
unscored:
description: |-
Unscored is how many of those screens got NO answer — the scorer was absent,
stuck, slow, erroring or silent. An unanswered screen allows ordinary
traffic, so this is the number that separates "a quiet day" from "the judge
stopped answering and nothing said so".
type: integer
window_sec:
description: WindowSec is the span the counts cover, in seconds.
type: integer
type: object
TransitionResult:
properties:
distribution:
@@ -47015,7 +47143,9 @@ paths:
platform write and requires SuperAdmin; otherwise it is a per-org write (org_rpm,
cache_ttl_sec, cache_paths, methods) scoped to the caller's own org — or, for a
SuperAdmin, the tenant named by ?org=<slug>. A body that sets nothing is a 400.
updated_at and updated_by are server-stamped; a client-supplied value is ignored.
The abuse gate's mode is an OPERATOR field: setting it requires SuperAdmin,
whichever organization it lands on. updated_at and updated_by are
server-stamped; a client-supplied value is ignored.
operationId: put_v1_gateway_config
requestBody:
content:
@@ -47041,6 +47171,45 @@ paths:
tags:
- gateway
x-app: gateway
/v1/gateway/traffic:
get:
description: |-
Traffic reports who is calling this organization's API right now: the request
count for the last minute split by AGENCY LANE — agent, human, bot, unknown —
and the busiest callers behind it, each with its request count, its
authentication-failure count, how many distinct paths it touched, and any
verdict currently held against it.
The lane split is the answer to the question a generic bot filter cannot
answer: which of this traffic is the customer's own automation and which is
somebody working through a list. It is computed from credentials we issued, not
from the client's self-description, so a scraper cannot move itself into the
agent lane by editing a header.
A validated caller appears as a FINGERPRINT — a one-way, per-process digest. It
is stable enough to recognise the same caller across a minute and cannot be
turned back into a key, so this report is safe to read, screenshot and paste.
It also reports what the sensor's own ceilings are doing (strain, tracked,
ceiling, refused) and how many screens the scorer did not answer (unscored), so
a control that has stopped measuring or a judge that has stopped answering is a
number here rather than a quiet day.
Scoped to the caller's own validated organization. A SuperAdmin may inspect a
specific tenant with ?org=<slug>, or the lane that has no tenant — every caller
the identity boundary could not validate — with an empty ?org=.
operationId: gatewayTraffic
responses:
"200":
content:
application/json:
schema:
$ref: '#/components/schemas/TrafficView'
description: ok
summary: Report who is calling this org's API right now
tags:
- gateway
x-app: gateway
/v1/generate-text-to-speech-audio:
post:
description: Convert text to speech
openapi/floor.json
+1 -1
View File
@@ -78,7 +78,7 @@
"flow": 8,
"framework": 19,
"functions": 11,
"gateway": 2,
"gateway": 3,
"generate-text-to-speech-audio": 1,
"generate-text-to-speech-audio-stream": 1,
"git": 36,
plugin/gateway/mcp.json
+13 -1
View File
@@ -1,4 +1,12 @@
[
{
"description": "Traffic reports who is calling this organization's API right now: the request\ncount for the last minute split by AGENCY LANE — agent, human, bot, unknown —\nand the busiest callers behind it, each with its request count, its\nauthentication-failure count, how many distinct paths it touched, and any\nverdict currently held against it.\n\nThe lane split is the answer to the question a generic bot filter cannot\nanswer: which of this traffic is the customer's own automation and which is\nsomebody working through a list. It is computed from credentials we issued, not\nfrom the client's self-description, so a scraper cannot move itself into the\nagent lane by editing a header.\n\nA validated caller appears as a FINGERPRINT — a one-way, per-process digest. It\nis stable enough to recognise the same caller across a minute and cannot be\nturned back into a key, so this report is safe to read, screenshot and paste.\n\nIt also reports what the sensor's own ceilings are doing (strain, tracked,\nceiling, refused) and how many screens the scorer did not answer (unscored), so\na control that has stopped measuring or a judge that has stopped answering is a\nnumber here rather than a quiet day.\n\nScoped to the caller's own validated organization. A SuperAdmin may inspect a\nspecific tenant with ?org=\u003cslug\u003e, or the lane that has no tenant — every caller\nthe identity boundary could not validate — with an empty ?org=.",
"inputSchema": {
"properties": {},
"type": "object"
},
"name": "gatewayTraffic"
},
{
"description": "Read returns the EFFECTIVE edge policy the caller is subject to: the platform CORS\nallowlist and pre-auth per-IP flood cap in force, plus the caller's own authenticated\nrate ceiling, edge-cache TTLs and accepted-method allowlist. A SuperAdmin may inspect\na specific tenant's effective policy with ?org=\u003cslug\u003e.",
"inputSchema": {
@@ -8,7 +16,7 @@
"name": "get_v1_gateway_config"
},
{
"description": "Write updates one policy scope and returns the policy in force after the write.\nA body carrying any PLATFORM field (cors_origins, per_ip_rpm, window_sec) is a\nplatform write and requires SuperAdmin; otherwise it is a per-org write (org_rpm,\ncache_ttl_sec, cache_paths, methods) scoped to the caller's own org — or, for a\nSuperAdmin, the tenant named by ?org=\u003cslug\u003e. A body that sets nothing is a 400.\nupdated_at and updated_by are server-stamped; a client-supplied value is ignored.",
"description": "Write updates one policy scope and returns the policy in force after the write.\nA body carrying any PLATFORM field (cors_origins, per_ip_rpm, window_sec) is a\nplatform write and requires SuperAdmin; otherwise it is a per-org write (org_rpm,\ncache_ttl_sec, cache_paths, methods) scoped to the caller's own org — or, for a\nSuperAdmin, the tenant named by ?org=\u003cslug\u003e. A body that sets nothing is a 400.\nThe abuse gate's mode is an OPERATOR field: setting it requires SuperAdmin,\nwhichever organization it lands on. updated_at and updated_by are\nserver-stamped; a client-supplied value is ignored.",
"inputSchema": {
"properties": {
"cache_paths": {
@@ -36,6 +44,10 @@
},
"type": "array"
},
"mode": {
"description": "Mode is the abuse gate's posture for THIS scope: \"shadow\" scores traffic and\nrecords the verdict without acting on it, \"live\" enforces it. Unset means\nshadow.\n\nIt is the one per-org field that does NOT inherit. Every other field here\nlayers a platform default under the org's own value, which is right for a\ndefault: a tenant that sets no rate ceiling should get the platform's. Mode\nis not a default, it is an ARMING DECISION — it is what makes a statistical\njudgement start refusing real traffic — and inheriting it means arming one\nscope arms every tenant that never asked for it, without a write to their\nrow and without anything in their config changing. So a tenant is live only\nif that tenant's OWN row says live, and the platform row's mode governs\nexactly one scope: the anonymous lane, which has no tenant of its own.\n\nIt is also not self-service. Writing it requires SuperAdmin (see the\n/v1/gateway config op): the subject of an abuse control does not get to\nswitch the control off.",
"type": "string"
},
"org_rpm": {
"description": "OrgRPM is the org's OWN authenticated rate ceiling, requests per minute, as\nScopeRateLimit enforces it. Unset inherits the platform default, then the\nstatic boot default.",
"type": "integer"
plugin/gateway/openapi.json
+127 -1
View File
@@ -40,7 +40,7 @@
"put": {
"operationId": "put_v1_gateway_config",
"summary": "Write updates one policy scope and returns the policy in force after the write.",
"description": "Write updates one policy scope and returns the policy in force after the write.\nA body carrying any PLATFORM field (cors_origins, per_ip_rpm, window_sec) is a\nplatform write and requires SuperAdmin; otherwise it is a per-org write (org_rpm,\ncache_ttl_sec, cache_paths, methods) scoped to the caller's own org — or, for a\nSuperAdmin, the tenant named by ?org=\u003cslug\u003e. A body that sets nothing is a 400.\nupdated_at and updated_by are server-stamped; a client-supplied value is ignored.",
"description": "Write updates one policy scope and returns the policy in force after the write.\nA body carrying any PLATFORM field (cors_origins, per_ip_rpm, window_sec) is a\nplatform write and requires SuperAdmin; otherwise it is a per-org write (org_rpm,\ncache_ttl_sec, cache_paths, methods) scoped to the caller's own org — or, for a\nSuperAdmin, the tenant named by ?org=\u003cslug\u003e. A body that sets nothing is a 400.\nThe abuse gate's mode is an OPERATOR field: setting it requires SuperAdmin,\nwhichever organization it lands on. updated_at and updated_by are\nserver-stamped; a client-supplied value is ignored.",
"tags": [
"gateway"
],
@@ -75,6 +75,28 @@
}
}
}
},
"/v1/gateway/traffic": {
"get": {
"operationId": "gatewayTraffic",
"summary": "Report who is calling this org's API right now",
"description": "Traffic reports who is calling this organization's API right now: the request\ncount for the last minute split by AGENCY LANE — agent, human, bot, unknown —\nand the busiest callers behind it, each with its request count, its\nauthentication-failure count, how many distinct paths it touched, and any\nverdict currently held against it.\n\nThe lane split is the answer to the question a generic bot filter cannot\nanswer: which of this traffic is the customer's own automation and which is\nsomebody working through a list. It is computed from credentials we issued, not\nfrom the client's self-description, so a scraper cannot move itself into the\nagent lane by editing a header.\n\nA validated caller appears as a FINGERPRINT — a one-way, per-process digest. It\nis stable enough to recognise the same caller across a minute and cannot be\nturned back into a key, so this report is safe to read, screenshot and paste.\n\nIt also reports what the sensor's own ceilings are doing (strain, tracked,\nceiling, refused) and how many screens the scorer did not answer (unscored), so\na control that has stopped measuring or a judge that has stopped answering is a\nnumber here rather than a quiet day.\n\nScoped to the caller's own validated organization. A SuperAdmin may inspect a\nspecific tenant with ?org=\u003cslug\u003e, or the lane that has no tenant — every caller\nthe identity boundary could not validate — with an empty ?org=.",
"tags": [
"gateway"
],
"responses": {
"200": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/TrafficView"
}
}
},
"description": "ok"
}
}
}
}
},
"components": {
@@ -106,6 +128,10 @@
},
"type": "array"
},
"mode": {
"description": "Mode is the abuse gate's posture for THIS scope: \"shadow\" scores traffic and\nrecords the verdict without acting on it, \"live\" enforces it. Unset means\nshadow.\n\nIt is the one per-org field that does NOT inherit. Every other field here\nlayers a platform default under the org's own value, which is right for a\ndefault: a tenant that sets no rate ceiling should get the platform's. Mode\nis not a default, it is an ARMING DECISION — it is what makes a statistical\njudgement start refusing real traffic — and inheriting it means arming one\nscope arms every tenant that never asked for it, without a write to their\nrow and without anything in their config changing. So a tenant is live only\nif that tenant's OWN row says live, and the platform row's mode governs\nexactly one scope: the anonymous lane, which has no tenant of its own.\n\nIt is also not self-service. Writing it requires SuperAdmin (see the\n/v1/gateway config op): the subject of an abuse control does not get to\nswitch the control off.",
"type": "string"
},
"org_rpm": {
"description": "OrgRPM is the org's OWN authenticated rate ceiling, requests per minute, as\nScopeRateLimit enforces it. Unset inherits the platform default, then the\nstatic boot default.",
"type": "integer"
@@ -128,6 +154,106 @@
}
},
"type": "object"
},
"TrafficCaller": {
"properties": {
"action": {
"description": "Action is the verdict currently held against it, if any.",
"type": "string"
},
"cred": {
"description": "Cred is the caller's key: a credential fingerprint (a per-process one-way\ndigest, not a key) for a validated caller, and \"ip:\u003caddr\u003e\" for one that\npresented no credential we could validate.",
"type": "string"
},
"failures": {
"description": "Failures is how many ended 401 or 403.",
"type": "integer"
},
"held_until": {
"description": "HeldUntil is when the held verdict lapses, unix seconds.",
"type": "integer"
},
"paths": {
"description": "Paths is the approximate number of distinct paths it touched (max 64).",
"type": "integer"
},
"reason": {
"description": "Reason is why that verdict was reached.",
"type": "string"
},
"requests": {
"description": "Requests is its request count in the window.",
"type": "integer"
}
},
"type": "object"
},
"TrafficView": {
"properties": {
"blind": {
"description": "Blind is how many requests in the window carried no identity to attribute\nthem to — no validated credential and no client address. Non-zero on a\npublic plane means the client address is not reaching this process (a TCP\nload balancer with no PROXY protocol in front of it, typically), so this\nscope's callers cannot be told apart and nothing can be held against them.",
"type": "integer"
},
"callers": {
"description": "Callers is the scope's busiest callers this window. A credentialed caller\nappears as a FINGERPRINT — a per-process one-way digest: enough to recognise\nthe same caller across requests, never enough to reconstruct the credential.",
"items": {
"$ref": "#/components/schemas/TrafficCaller"
},
"type": "array"
},
"ceiling": {
"description": "Ceiling is the most callers this scope may hold at once.",
"type": "integer"
},
"denied": {
"description": "Denied is how many of them the gate refused.",
"type": "integer"
},
"lanes": {
"additionalProperties": {
"type": "integer"
},
"description": "Lanes is the request count per lane — agent, human, bot, unknown. This is\nthe split that separates a customer's automation from a scraper.",
"type": "object"
},
"mode": {
"description": "Mode is the abuse gate's posture for this scope: \"shadow\" records the scorer's\naction without enforcing it, \"live\" enforces it.",
"type": "string"
},
"org": {
"description": "Org is the scope this view was taken for — the validated principal's own,\nnever a value the caller supplied. Empty names the anonymous lane, the one\nscope that has no tenant.",
"type": "string"
},
"refused": {
"description": "Refused is how many callers this scope's ceilings turned away in the window.",
"type": "integer"
},
"requests": {
"description": "Requests is how many requests this scope made in the window.",
"type": "integer"
},
"screens": {
"description": "Screens is how many of them were put to the scorer — the billable unit of\nthe risk product. Counted from the first request, whatever the SKU costs.",
"type": "integer"
},
"strain": {
"description": "Strain is what this scope's ceilings are doing: \"clear\" below them, \"full\"\nat them, \"refuse\" once a caller has been turned away inside this window —\nwhich means that caller is UNMEASURED and the numbers here are a sample\nrather than a census. It is reported rather than logged because the\nalternative — a bound that degrades a scope silently — is the failure this\ndesign exists to rule out. No other scope can move it.",
"type": "string"
},
"tracked": {
"description": "Tracked is how many callers this scope holds state for right now, and\nCeiling is the most it may hold. Tracked == Ceiling is the fact a bound\nthat binds cannot hide.",
"type": "integer"
},
"unscored": {
"description": "Unscored is how many of those screens got NO answer — the scorer was absent,\nstuck, slow, erroring or silent. An unanswered screen allows ordinary\ntraffic, so this is the number that separates \"a quiet day\" from \"the judge\nstopped answering and nothing said so\".",
"type": "integer"
},
"window_sec": {
"description": "WindowSec is the span the counts cover, in seconds.",
"type": "integer"
}
},
"type": "object"
}
}
}
plugin/o11y/main.go
+27 -3
View File
@@ -73,7 +73,7 @@ func run() error {
// no-op provider and emitted nothing.
defer cloud.InstallTelemetry(context.Background(), deps.Logger, "hanzo-o11y")(context.Background())
app := newApp(deps)
app := newApp(cfg, deps)
if err := o11y.MountO11y(app, deps); err != nil {
return fmt.Errorf("mount: %w", err)
@@ -139,10 +139,34 @@ func run() error {
//
// Installed BEFORE the mount because fiber runs middleware in registration order:
// one added after the routes never runs.
func newApp(deps cloud.Deps) *zip.App {
//
// THE IDENTITY BOUNDARY, for the same reason and with more force. o11y scopes
// every read by the validated tenant (apps/o11y/scope.go reads c.Org(), and its
// admin surfaces read X-User-IsAdmin) — headers that are only trustworthy because
// SOMETHING strips the client's copy and re-mints them from a verified token.
// cloud.Serve installs that boundary for every scaffolded app; this main is
// hand-written, so it has to install it itself, and until it did, this process
// took those headers from the wire.
//
// It is also what the abuse gate below needs: the gate classes a caller from the
// boundary's own attestation (principal.Minted), so without a boundary every
// caller here is anonymous — safe, but blind to the customer automation it exists
// to tell apart from a scraper.
//
// AbuseGate then, by the same argument as EdgeCORS: this process answers public
// requests, so it carries the lifecycle defense itself, and an app that opted out
// of the gate by being hand-written would be the one prefix family a stolen
// credential could work against unwatched.
//
// Shadow per org by default, exactly as in the fused binary, so this is a sensor
// here until an operator arms the org — not a second policy.
func newApp(cfg *cloud.Config, deps cloud.Deps) *zip.App {
// cloud.ErrorHandler for the same reason cloud.Serve installs it: this main is
// hand-written, and without it a refusal this app propagates renders 500.
// hand-written, and without it a refusal this app propagates renders 500 —
// which is exactly what the identity boundary and the abuse gate above emit.
app := zip.New(zip.Config{AppName: "o11y", Logger: deps.Logger, ErrorHandler: cloud.ErrorHandler})
app.Use(cloud.EdgeCORS(deps.GatewayPolicy))
app.Use(cloud.IdentityMiddleware(cfg))
app.Use(cloud.AbuseGate(deps, deps.Traffic))
return app
}
plugin/o11y/main_test.go
+33 -1
View File
@@ -31,7 +31,11 @@ func probeApp(t *testing.T, origins []string) *zip.App {
if pol == nil {
t.Fatal("edge.New must always return a usable store")
}
app := newApp(cloud.Deps{GatewayPolicy: pol})
// A config with no IAM issuer: the identity boundary still runs, still strips
// every client-supplied authority header, and validates nothing — the shape a
// deployment has before it is pointed at an issuer, and the one that must not
// leave the chain trusting the wire.
app := newApp(&cloud.Config{}, cloud.Deps{GatewayPolicy: pol})
app.Get("/v1/summary", func(c *zip.Ctx) error {
return c.JSON(200, map[string]string{"page_title": "Hanzo status"})
})
@@ -123,3 +127,31 @@ func TestSummaryEmitsNothingWhenAllowlistEmpty(t *testing.T) {
t.Fatalf("ACAO = %q, want none — an unset allowlist must not double the ingress header", got)
}
}
// This process composes its own root, so the identity trust boundary is not
// inherited from cloud.Serve — it has to be installed here, and until it was,
// every X-* authority header on the wire reached o11y's own org scoping and admin
// gates verbatim. The chain must strip them whether or not a token validates.
func TestChainStripsClientSuppliedAuthority(t *testing.T) {
pol, _ := edge.New("", "admin", edge.Policy{})
app := newApp(&cloud.Config{}, cloud.Deps{GatewayPolicy: pol})
var seen struct{ org, user, admin string }
app.Get("/v1/summary", func(c *zip.Ctx) error {
seen.org, seen.user, seen.admin = c.Org(), c.User(), c.Header("X-User-IsAdmin")
return c.JSON(200, map[string]string{"ok": "1"})
})
req := httptest.NewRequest(http.MethodGet, "/v1/summary", nil)
req.Header.Set("X-User-Id", "u-forged")
req.Header.Set("X-User-IsAdmin", "true")
if _, err := app.Fiber().Test(req); err != nil {
t.Fatalf("test: %v", err)
}
if seen.user != "" {
t.Fatalf("a client-supplied X-User-Id survived the chain: %q", seen.user)
}
if seen.admin != "" {
t.Fatalf("a client-supplied X-User-IsAdmin survived the chain: %q", seen.admin)
}
}
resource_billing.go
-14
View File
@@ -346,17 +346,3 @@ func parseNonNegCents(s string) (int64, bool) {
}
return n, true
}
// ClientIP extracts the originating client IP from X-Forwarded-For (the gateway
// sets it); the left-most entry is the real client. Shared by the edge gate and
// the resource meter so usage records carry a consistent client_ip.
func ClientIP(c *zip.Ctx) string {
xff := c.Header("X-Forwarded-For")
if xff == "" {
return ""
}
if i := strings.IndexByte(xff, ','); i > 0 {
return strings.TrimSpace(xff[:i])
}
return strings.TrimSpace(xff)
}
risk.go
+526
View File
@@ -0,0 +1,526 @@
package cloud
// The scorer seam — the ONE door from anywhere in cloud to /v1/risk.
//
// /v1/risk is the platform's scoring and decision plane: it judges an entity at
// a lifecycle moment and answers with an action. Everything that DEFENDS a
// lifecycle moment — the abuse gate on the API plane here, the signup gate in
// hanzoai/iam over the wire — asks that one scorer. None of them scores. A
// second scorer would be a second answer to one question, and the two would
// disagree silently, which is the failure mode a risk product cannot have.
//
// The seam is installed the way the observability plane installs its claim on
// the event door (obsevents.go): the app that owns /v1/risk hands its scoring
// function to the core at mount, and the core calls it through a nil-safe
// accessor. That inverts the import — package cloud cannot import an app, since
// every app imports cloud — and it means the gate below is complete and testable
// before the scorer exists, and stays correct when it is absent at run time.
//
// THE FAIL POLICY LIVES HERE, IN ONE FUNCTION, AND NOWHERE ELSE.
//
// ordinary path — a scorer that is absent, erroring, timing out or silent
// ALLOWS. A risk plane that is down must not be able to take
// the product down with it. Login keeps working.
// privileged grant — the same conditions BLOCK. A grant that hands out standing
// authority (a new tenant, a credential, an elevation) is not
// a request to be waved through because the judge is out; the
// caller can retry in a minute.
//
// Both answers carry a Refusal naming why, so an unscored allow is never
// mistaken for a clean result. Silence must not read as innocence — the same
// doctrine the anomaly engine applies to its own refusals.
import (
"context"
"net/http"
"strings"
"sync/atomic"
"time"
)
// The lifecycle stages a decision can be asked at. A stage selects the feature
// window and the rule set on the scorer's side; it never selects a different
// tenant gate.
const (
// StageSignup is registration: is this account real, and is it one account?
StageSignup = "signup"
// StageUsage is the API/usage plane: is this traffic the customer's, and is
// it being used the way a customer uses it?
StageUsage = "usage"
// StagePayment is authorization time on a transaction.
StagePayment = "payment"
)
// The actions a scorer can return, most permissive first. They are the whole
// vocabulary: a caller that receives anything else treats it as unrecognised and
// applies the fail policy, rather than guessing.
const (
// ActionAllow proceeds.
ActionAllow = "allow"
// ActionReview proceeds and summons a person. A statistical judgement may
// reach here and no further on its own.
ActionReview = "review"
// ActionChallenge proceeds only after the caller proves something more.
ActionChallenge = "challenge"
// ActionRestrict proceeds at a reduced ceiling.
ActionRestrict = "restrict"
// ActionBlock does not proceed.
ActionBlock = "block"
)
// The reasons an answer is not a scored one. A caller that logs, audits or
// reports an outcome reports this beside it, so "allowed" and "allowed because
// nobody was listening" are never the same row.
const (
// RefusalAbsent — no scorer is installed in this process.
RefusalAbsent = "scorer-absent"
// RefusalError — the scorer returned an error.
RefusalError = "scorer-error"
// RefusalTimeout — the scorer did not answer inside the budget.
RefusalTimeout = "scorer-timeout"
// RefusalSilent — the scorer answered with no action.
RefusalSilent = "scorer-silent"
// RefusalUnknown — the scorer answered with an action outside the vocabulary.
RefusalUnknown = "scorer-unknown"
// RefusalBusy — the scorer was already answering as many questions at once as
// it is allowed to. The question was not asked.
RefusalBusy = "scorer-busy"
// RefusalStuck — the scorer holds every slot and has not returned from ANY
// call for longer than a stall. It is installed and it is not answering, which
// is a different fact from busy: busy clears in microseconds, this does not
// clear at all.
RefusalStuck = "scorer-stuck"
)
// RiskBudget bounds how long a decision may take. The gate sits on the request
// path, so a scorer that hangs must not hang the product: past the budget the
// answer is the fail policy's, not the scorer's. 150ms is generous for an
// in-process half-space-tree score and tight enough to be invisible next to any
// real handler.
const RiskBudget = 150 * time.Millisecond
// RiskSubject names WHAT is being judged. Kind is the entity class — account,
// transaction, session, agent, merchant, payout — and ID is its identity within
// the tenant. The tenant itself is never in here: it is the org argument, and it
// comes from the validated principal.
type RiskSubject struct {
Kind string `json:"kind"`
ID string `json:"id"`
}
// RiskQuery is one question for the scorer.
type RiskQuery struct {
// Stage is the lifecycle moment.
Stage string `json:"stage"`
// Subject is the entity being judged.
Subject RiskSubject `json:"subject"`
// Agency is the caller's lane as the asking gate classed it — agent, human,
// bot or unknown. It is a SIGNAL, not an assertion: the scorer may overrule it
// on facts the gate does not hold, and its answer is the one that counts.
Agency string `json:"agency,omitempty"`
// Signals are the facts the gate observed: ip, path spread, failure count,
// credential class, and whatever else the stage cares about. Free-form because
// the vocabulary belongs to the scorer's feature inventory, not to the gate.
Signals map[string]string `json:"signals,omitempty"`
// Privileged marks a grant of standing authority — a new tenant, a credential,
// an elevation. It selects the FAIL-CLOSED branch: silence denies. It is set by
// the gate from the request it is judging, never by the caller being judged.
Privileged bool `json:"-"`
}
// RiskVerdict is the answer.
type RiskVerdict struct {
// ID is the scorer's decision id, the handle a decision record is fetched by.
ID string `json:"id,omitempty"`
// Action is what to do, from the vocabulary above.
Action string `json:"action"`
// Score is the weight of evidence in [0,1].
Score float64 `json:"score,omitempty"`
// Agency is the lane the scorer settled on — the authoritative one.
Agency string `json:"agency,omitempty"`
// Cause is the scorer's short reason, for the audit record.
Cause string `json:"cause,omitempty"`
// Refusal names why this is not a scored answer, and is empty when it is one.
Refusal string `json:"refusal,omitempty"`
}
// Scored reports whether a verdict came from the scorer rather than the fail
// policy. Every recorder asks this before treating an allow as evidence.
func (v RiskVerdict) Scored() bool { return v.Refusal == "" }
// Allowed reports whether the verdict lets the request proceed unchanged.
// Review proceeds too — it summons a person, it does not stop traffic.
func (v RiskVerdict) Allowed() bool { return v.Action == ActionAllow || v.Action == ActionReview }
// RiskScorer is the scoring function /v1/risk installs. org is the SERVER-resolved
// tenant; a scorer never re-derives identity, and never answers for a tenant it
// was not asked about.
type RiskScorer func(ctx context.Context, org string, q RiskQuery) (RiskVerdict, error)
// riskScorer holds the installed scorer. atomic.Value rather than a bare var
// because Mount runs on one goroutine and the request path reads on all of them.
var riskScorer atomic.Value // RiskScorer
// SetRiskScorer installs the ONE scorer. Called by the app that owns /v1/risk
// when its model and stores are ready. Installing nil uninstalls it, which is how
// a degraded scorer withdraws rather than answering badly.
func SetRiskScorer(fn RiskScorer) {
if fn == nil {
riskScorer.Store(RiskScorer(nil))
return
}
riskScorer.Store(fn)
}
// RiskScorerInstalled reports whether a scorer is available — for a health probe
// or a report, never as a gate. The gate is Decide, which handles absence itself.
func RiskScorerInstalled() bool { return loadRiskScorer() != nil }
func loadRiskScorer() RiskScorer {
fn, _ := riskScorer.Load().(RiskScorer)
return fn
}
// Decide asks the scorer and applies the fail policy. It is the ONLY way cloud
// code reaches /v1/risk, so the policy is stated once and cannot drift between
// the gates that depend on it.
//
// It never returns an error. A gate needs an action, and "I could not tell you"
// IS an action — stated by q.Privileged and carried in Refusal.
func Decide(ctx context.Context, org string, q RiskQuery) RiskVerdict {
return decide(scorerCalls, ctx, org, q)
}
// decide is Decide with its ceiling passed in — the whole of the logic, over the
// one piece of process state it holds. Decide supplies the process ceiling; a
// test supplies its own, so the bound is asserted without mutating a global that
// another test's parked goroutine is still reading.
func decide(sem *calls, ctx context.Context, org string, q RiskQuery) RiskVerdict {
fn := loadRiskScorer()
if fn == nil {
return riskUnavailable(q, RefusalAbsent)
}
// BOUNDED, always. Every ask costs a goroutine that lives until the scorer
// returns — which, for a scorer stuck on a lock, a model load or a stalled
// socket, is longer than the budget below. Without a ceiling those goroutines
// accumulate one per screened request for as long as the stall lasts, so a
// slow scorer becomes an out-of-memory in the process it was installed to
// protect. Past the ceiling the question is not asked at all and the fail
// policy answers, which is the same answer a timeout gives and reaches it
// without allocating anything.
if !sem.take() {
// FULL is not the same fact as STOPPED. A healthy scorer returns in
// microseconds, so every slot being held means either a burst (which clears
// before the caller could retry) or a scorer that has stopped returning at
// all — and the second one never clears, because a slot is released by the
// goroutine that holds it. Told apart by when a call last came back.
if sem.stalled(time.Now()) {
return riskUnavailable(q, RefusalStuck)
}
return riskUnavailable(q, RefusalBusy)
}
// The budget is the gate's, not the scorer's: a scorer that ignores its
// context must still not hold the request. The answer is taken from whichever
// arrives first, and a late one is discarded by a buffered channel with no
// goroutine left blocked on it.
ctx, cancel := context.WithTimeout(ctx, RiskBudget)
defer cancel()
type answer struct {
v RiskVerdict
err error
}
ch := make(chan answer, 1)
go func() {
// The slot is released by the goroutine that holds it, when the scorer
// actually returns — NOT when the budget expires. Releasing it at the
// timeout would let the ceiling be exceeded without bound by exactly the
// scorer it exists to contain. The return is also the health signal: a
// scorer that keeps returning is busy, one that never returns is stuck.
defer sem.give()
defer func() {
// A panicking scorer is a scorer that did not answer. Contained here
// so a model bug cannot take down the request path.
if r := recover(); r != nil {
ch <- answer{err: errScorerPanic}
}
}()
v, err := fn(ctx, org, q)
ch <- answer{v, err}
}()
select {
case <-ctx.Done():
return riskUnavailable(q, RefusalTimeout)
case a := <-ch:
switch {
case a.err != nil:
return riskUnavailable(q, RefusalError)
case a.v.Action == "":
return riskUnavailable(q, RefusalSilent)
case !riskActionKnown(a.v.Action):
return riskUnavailable(q, RefusalUnknown)
}
return a.v
}
}
// MaxScorerCalls is how many questions may be in flight at once, process-wide.
// It is a bound on the COST of asking, not a rate limit: a healthy in-process
// score returns in microseconds, so this ceiling is never reached by real load —
// it is reached only when the scorer has stopped answering, which is exactly when
// asking it again is worthless.
const MaxScorerCalls = 256
// ScorerStall is how long every slot may be held with nothing coming back before
// the scorer is called stuck rather than busy. Twenty budgets: far past any burst
// a healthy scorer produces, and reached in seconds by one that has deadlocked.
const ScorerStall = 20 * RiskBudget
// calls is the ceiling on asking, and the ONE record of whether asking still
// works: how many questions are in flight, and when an answer last came back.
// The two facts live together because neither one alone can tell a burst from a
// deadlock, and that difference decides whether a privileged grant waits or
// proceeds.
type calls struct {
// slots is a counting semaphore over a buffered channel: take never blocks (a
// full channel means "no room", which is an answer), give always succeeds
// because only a taker gives.
slots chan struct{}
// last is the unix-nano time a scorer call last RETURNED. Stamped when a
// scorer is installed, so a scorer that saturates immediately is busy rather
// than born stuck.
last atomic.Int64
}
func newCalls(n int) *calls {
c := &calls{slots: make(chan struct{}, n)}
c.answered(time.Now())
return c
}
// scorerCalls is the process ceiling.
var scorerCalls = newCalls(MaxScorerCalls)
func (c *calls) take() bool {
select {
case c.slots <- struct{}{}:
return true
default:
return false
}
}
func (c *calls) give() {
c.answered(time.Now())
<-c.slots
}
func (c *calls) answered(at time.Time) { c.last.Store(at.UnixNano()) }
// stalled reports that nothing has come back for longer than ScorerStall. Only
// asked when every slot is held: a quiet deployment has an old timestamp and is
// not stuck, it is unasked.
func (c *calls) stalled(now time.Time) bool {
return now.UnixNano()-c.last.Load() > int64(ScorerStall)
}
// errScorerPanic is the error a contained panic reports as. Unexported and
// never returned to a caller — Decide converts it to a refusal like any other.
var errScorerPanic = errScorer("the scorer panicked")
type errScorer string
func (e errScorer) Error() string { return string(e) }
// riskUnavailable is THE fail policy, in one place — and it turns on TWO facts,
// not one. The second is what keeps it a defense rather than an outage.
//
// privileged — the request grants standing authority, so silence must deny.
// answering — there IS a scorer here and it is returning answers. Every
// refusal except two means exactly that: it exists and it did not
// answer THIS question, which is when a grant must wait.
//
// The two exemptions are the deployments where there is no judge at all:
//
// absent — no scorer was ever installed in this process, or one withdrew (which
// is how a degraded scorer is meant to step down). Refusing to mint a
// credential or read a secret because a component is not deployed is
// not security, it is a product that cannot be operated.
// stuck — every slot is held and nothing has come back for a stall. A slot is
// released by the goroutine holding it, so a deadlocked scorer holds
// them forever: without this, one hung goroutine would 403 every armed
// org's key store until someone restarted the pod, and no operator
// could tell that apart from the control working as designed.
//
// This is the same rule hanzoai/iam applies at its own signup gate with a
// different arming signal. Two mechanisms, one semantic: fail closed once there
// is something to fail closed on, allow before.
//
// Neither exemption is silent. Every verdict carries the Refusal that produced
// it, the gate logs it, and Traffic.Screen counts the unanswered screens on the
// org's own report — so "allowed" and "allowed because nobody was listening" are
// never the same row.
func riskUnavailable(q RiskQuery, why string) RiskVerdict {
if q.Privileged && answering(why) {
return RiskVerdict{Action: ActionBlock, Agency: q.Agency, Refusal: why}
}
return RiskVerdict{Action: ActionAllow, Agency: q.Agency, Refusal: why}
}
// answering reports whether a refusal came from a scorer that is present and
// returning — the fact that separates "the judge is out today" from "the judge
// did not answer this one".
func answering(why string) bool { return why != RefusalAbsent && why != RefusalStuck }
// Facts drops the empty values from a signal map, because a fact we do not have
// must be ABSENT rather than empty. An empty string is a VALUE: a scorer keying
// velocity on "ip" would group every request whose address never arrived — which,
// behind a load balancer that does not pass the peer, is all of them — into one
// very busy caller and refuse the lot. "We do not know" and "it is the empty
// string" are different answers and only one of them is true.
//
// Stated once, at the seam every question passes through, so no gate has to
// remember it.
func Facts(m map[string]string) map[string]string {
for k, v := range m {
if v == "" {
delete(m, k)
}
}
return m
}
func riskActionKnown(a string) bool {
switch a {
case ActionAllow, ActionReview, ActionChallenge, ActionRestrict, ActionBlock:
return true
}
return false
}
// grantPaths are the surfaces that hand out STANDING AUTHORITY — a credential, a
// tenant, an elevation, or the contents of the key store. A request to one of
// these takes the FAIL-CLOSED branch: when the scorer cannot answer, it is
// refused rather than waved through.
//
// Two lists rather than one predicate, because the two have different reasons:
//
// grantPaths — every method. Reading a secret with a stolen key IS the
// attack, so a GET here is as much a grant as a POST.
// grantMutations — mutations only. A SuperAdmin READING an admin surface is
// audited and access-controlled already; it is a WRITE that
// changes who can do what.
//
// Data rather than logic, so the list of things that fail closed can be read and
// reviewed in one place instead of inferred from handlers.
var (
grantPaths = []string{
"/v1/iam/issue-user-token",
"/v1/iam/mint-user-keys",
"/v1/iam/revoke-user-keys",
"/v1/iam/admin/provision",
"/v1/iam/signup",
"/v1/iam/onboard",
"/v1/kms/",
}
grantMutations = []string{
"/v1/admin/",
"/v1/iam/",
"/v1/orgs/",
}
)
// RoutePath is a request path in the form THE ROUTER MATCHES IT, and it is the
// only form a security comparison may use.
//
// fiber resolves a route against its "detection path": the request path
// lower-cased (CaseSensitive is off) with trailing slashes stripped
// (StrictRouting is off). c.Path() is the raw spelling the client sent. So
// `/V1/KMS/secret` and `/v1/kms/` reach exactly the handlers `/v1/kms/...` and
// `/v1/kms` do, while a prefix test over the raw path matches neither — one
// capital letter turned a fail-CLOSED grant surface into a fail-OPEN one.
//
// The rule here is the ROUTER'S rule, not an approximation of it: any other
// normalization would be a second opinion about what a path means, and the
// router's is the one that decides which handler runs. Percent-encoding is
// deliberately NOT decoded, for the same reason — fiber does not decode it
// either (UnescapePath is off), so `/v1/%6bms` routes nowhere and is not a
// bypass; decoding it here would make this function match a route that does not
// exist.
func RoutePath(path string) string {
path = strings.ToLower(path)
for len(path) > 1 && path[len(path)-1] == '/' {
path = path[:len(path)-1]
}
return path
}
// underPrefix reports whether a NORMALIZED path is at or below prefix, on
// SEGMENT boundaries. Both sides are compared slash-terminated, so "/v1/kms"
// matches the "/v1/kms/" subtree (the router treats the two as one route) while
// "/v1/kmsx" does not match either — a prefix test on the bare strings would
// have said yes to the second, which is a rule about spelling rather than about
// routes.
func underPrefix(path, prefix string) bool {
prefix = strings.TrimSuffix(prefix, "/")
return path == prefix || strings.HasPrefix(path, prefix+"/")
}
// Probe reports whether a request is a liveness/readiness check. A probe is
// never a grant and is never screened — a health endpoint a risk decision can
// fail is not a health endpoint, and a kubelet is not a customer.
//
// ONE predicate, read by both the gate's exemption and Privileged below, so the
// two can never disagree about what a probe is. Read-only methods only: a POST
// to something ending in "/health" is not a probe, so an attacker cannot name a
// mutating route into the exemption.
//
// path is normalized here rather than by the caller, so a caller cannot forget.
func Probe(method, path string) bool {
if method != http.MethodGet && method != http.MethodHead {
return false
}
path = RoutePath(path)
switch path {
case "/health", "/healthz", "/readyz", "/livez", "/metrics":
return true
}
return strings.HasSuffix(path, "/health") || strings.HasSuffix(path, "/healthz")
}
// Privileged reports whether a request grants standing authority, and therefore
// whether the scorer's silence must deny it.
//
// It normalizes the path FIRST and compares nothing before it has. That order is
// the whole fix: the grant lists below describe ROUTES, and a route is what the
// router says it is.
func Privileged(method, path string) bool {
if Probe(method, path) {
return false
}
path = RoutePath(path)
for _, p := range grantPaths {
if underPrefix(path, p) {
return true
}
}
if !mutating(method) {
return false
}
for _, p := range grantMutations {
if underPrefix(path, p) {
return true
}
}
return false
}
func mutating(method string) bool {
switch method {
case "POST", "PUT", "PATCH", "DELETE":
return true
}
return false
}
risk_test.go
+431
View File
@@ -0,0 +1,431 @@
package cloud
// The fail policy is the single most consequential function in the lifecycle
// defense: it decides what happens to every request when the judge is out. These
// tests are its specification.
//
// Each one is mutation-proven — invert the branch it guards and exactly this test
// goes red.
import (
"context"
"errors"
"sync"
"testing"
"time"
"github.com/hanzoai/cloud/apps/gateway/edge"
)
// resetScorer restores the seam after a test, so one test's installed scorer can
// never leak into another's — the package-level seam is the only shared state
// here and it is reset explicitly rather than hoped about.
func resetScorer(t *testing.T) {
t.Helper()
t.Cleanup(func() { SetRiskScorer(nil) })
}
func TestDecide_FailsOpenOnTheOrdinaryPath(t *testing.T) {
resetScorer(t)
ordinary := RiskQuery{Stage: StageUsage, Subject: RiskSubject{Kind: "session", ID: "c1"}}
cases := []struct {
name string
install RiskScorer
refusal string
}{
{"no scorer installed", nil, RefusalAbsent},
{"scorer errors", func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{}, errors.New("model unavailable")
}, RefusalError},
{"scorer answers with no action", func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{Score: 0.9}, nil
}, RefusalSilent},
{"scorer answers outside the vocabulary", func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{Action: "quarantine"}, nil
}, RefusalUnknown},
{"scorer panics", func(context.Context, string, RiskQuery) (RiskVerdict, error) {
panic("nil map")
}, RefusalError},
{"scorer exceeds the budget", func(ctx context.Context, _ string, _ RiskQuery) (RiskVerdict, error) {
<-ctx.Done()
return RiskVerdict{Action: ActionBlock}, nil
}, RefusalTimeout},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
SetRiskScorer(tc.install)
v := Decide(context.Background(), "acme", ordinary)
if v.Action != ActionAllow {
t.Fatalf("ordinary path must fail OPEN: action = %q, want %q", v.Action, ActionAllow)
}
if v.Refusal != tc.refusal {
t.Fatalf("refusal = %q, want %q", v.Refusal, tc.refusal)
}
if v.Scored() {
t.Fatal("an unscored allow must not report itself as scored — silence is not innocence")
}
})
}
}
func TestDecide_FailsClosedOnAPrivilegedGrant(t *testing.T) {
resetScorer(t)
grant := RiskQuery{Stage: StageUsage, Subject: RiskSubject{Kind: "session", ID: "c1"}, Privileged: true}
cases := []struct {
name string
install RiskScorer
}{
{"scorer errors", func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{}, errors.New("model unavailable")
}},
{"scorer answers with no action", func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{}, nil
}},
{"scorer panics", func(context.Context, string, RiskQuery) (RiskVerdict, error) {
panic("nil map")
}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
SetRiskScorer(tc.install)
v := Decide(context.Background(), "acme", grant)
if v.Action != ActionBlock {
t.Fatalf("a privileged grant must fail CLOSED: action = %q, want %q", v.Action, ActionBlock)
}
if v.Refusal == "" {
t.Fatal("a fail-closed block must name why it could not be scored")
}
})
}
}
// The fail policy turns on TWO facts, and this is the second one: a scorer that
// is NOT THERE is not a scorer that refused to answer.
//
// Blocking on absence made every deployment without a risk plane — and every
// deployment whose scorer WITHDREW, which is how a degraded scorer is meant to
// step down — refuse to mint a credential, provision an identity or read a
// secret. That is not a defense, it is an outage with a security-shaped name.
// The exemption is narrow and it is loud: the verdict still carries the refusal,
// so an allow that happened because nobody was listening is never a clean result.
func TestDecide_AnAbsentScorerIsNotASilentOne(t *testing.T) {
resetScorer(t)
grant := RiskQuery{Stage: StageUsage, Subject: RiskSubject{Kind: "session", ID: "c1"}, Privileged: true}
// Never installed.
SetRiskScorer(nil)
v := Decide(context.Background(), "acme", grant)
if v.Action != ActionAllow || v.Refusal != RefusalAbsent {
t.Fatalf("no scorer in the process: %+v, want allow/%s", v, RefusalAbsent)
}
if v.Scored() {
t.Fatal("an allow reached because nothing was listening must not report itself as scored")
}
// Installed, then WITHDRAWN — the documented way a degraded scorer steps down.
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{ID: "d", Action: ActionAllow}, nil
})
if v := Decide(context.Background(), "acme", grant); !v.Scored() {
t.Fatalf("an installed scorer must be asked: %+v", v)
}
SetRiskScorer(nil)
if v := Decide(context.Background(), "acme", grant); v.Action != ActionAllow || v.Refusal != RefusalAbsent {
t.Fatalf("a withdrawn scorer 403s every armed org's key store: %+v", v)
}
// And a scorer that IS there and did not answer still denies the grant.
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{}, errors.New("model unavailable")
})
if v := Decide(context.Background(), "acme", grant); v.Action != ActionBlock {
t.Fatalf("a present-but-silent scorer must still deny a grant: %+v", v)
}
}
// A scorer that holds every slot and never returns is INSTALLED and NOT
// ANSWERING, and the two facts are told apart by when a call last came back. A
// slot is released by the goroutine holding it — never at the timeout, or the
// ceiling would not be a ceiling — so a deadlocked scorer holds all of them
// forever. Reading that as "busy" left every armed org's grant surface at 403
// until someone restarted the pod, with nothing to tell an operator whether the
// control was working or wedged.
func TestDecide_AStuckScorerDoesNotWedgeTheGrantSurface(t *testing.T) {
resetScorer(t)
sem := newCalls(1)
release := make(chan struct{})
t.Cleanup(func() { close(release) })
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
<-release
return RiskVerdict{Action: ActionAllow}, nil
})
grant := RiskQuery{Stage: StageUsage, Privileged: true}
// One call parks the only slot. Nothing has come back, but not for long.
if v := decide(sem, context.Background(), "acme", grant); v.Refusal != RefusalTimeout {
t.Fatalf("the parked call: %+v, want %s", v, RefusalTimeout)
}
if v := decide(sem, context.Background(), "acme", grant); v.Refusal != RefusalBusy || v.Action != ActionBlock {
t.Fatalf("a full ceiling is BUSY and a grant waits: %+v", v)
}
// Once nothing has come back for a whole stall, the scorer is not busy — it is
// gone, and a grant proceeds rather than waiting for a pod restart.
sem.answered(time.Now().Add(-ScorerStall - time.Second))
v := decide(sem, context.Background(), "acme", grant)
if v.Refusal != RefusalStuck || v.Action != ActionAllow {
t.Fatalf("a stuck scorer: %+v, want allow/%s", v, RefusalStuck)
}
if v.Scored() {
t.Fatal("an allow past a stuck scorer must not report itself as scored")
}
// A scorer that answers again is answering again: the state is a measurement,
// not a latch.
sem.answered(time.Now())
if v := decide(sem, context.Background(), "acme", grant); v.Refusal != RefusalBusy {
t.Fatalf("after an answer came back: %+v, want %s", v, RefusalBusy)
}
}
func TestDecide_PassesAScoredVerdictThrough(t *testing.T) {
resetScorer(t)
SetRiskScorer(func(_ context.Context, org string, q RiskQuery) (RiskVerdict, error) {
if org != "acme" {
t.Fatalf("the scorer was asked about org %q, not the one Decide was called for", org)
}
return RiskVerdict{ID: "d-1", Action: ActionBlock, Score: 0.97, Agency: edge.AgencyBot, Cause: "peers"}, nil
})
v := Decide(context.Background(), "acme", RiskQuery{Stage: StageUsage})
if !v.Scored() {
t.Fatalf("a scored verdict must carry no refusal, got %q", v.Refusal)
}
if v.Action != ActionBlock || v.ID != "d-1" || v.Agency != edge.AgencyBot {
t.Fatalf("verdict was reshaped in transit: %+v", v)
}
}
// A scored verdict must not be produced by the fail policy just because the
// caller marked the request privileged: the flag selects a BRANCH of the fail
// policy, it does not deny on its own.
func TestDecide_PrivilegedIsNotItselfADenial(t *testing.T) {
resetScorer(t)
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{ID: "d-2", Action: ActionAllow}, nil
})
v := Decide(context.Background(), "acme", RiskQuery{Stage: StageUsage, Privileged: true})
if v.Action != ActionAllow {
t.Fatalf("a scored allow on a privileged grant must be honoured, got %q", v.Action)
}
}
// The budget is the gate's, not the scorer's: a scorer that ignores its context
// must not be able to hold the request path open.
func TestDecide_ReturnsInsideTheBudget(t *testing.T) {
resetScorer(t)
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
time.Sleep(2 * time.Second) // deliberately ignores ctx
return RiskVerdict{Action: ActionAllow}, nil
})
start := time.Now()
v := Decide(context.Background(), "acme", RiskQuery{Stage: StageUsage})
if elapsed := time.Since(start); elapsed > 10*RiskBudget {
t.Fatalf("Decide took %s; the budget is %s", elapsed, RiskBudget)
}
if v.Refusal != RefusalTimeout {
t.Fatalf("refusal = %q, want %q", v.Refusal, RefusalTimeout)
}
}
func TestRiskScorerInstalled(t *testing.T) {
resetScorer(t)
SetRiskScorer(nil)
if RiskScorerInstalled() {
t.Fatal("no scorer is installed, but the seam says there is one")
}
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{Action: ActionAllow}, nil
})
if !RiskScorerInstalled() {
t.Fatal("a scorer is installed, but the seam says there is not")
}
SetRiskScorer(nil) // a degraded scorer withdraws rather than answering badly.
if RiskScorerInstalled() {
t.Fatal("uninstalling must be possible — a scorer that cannot withdraw cannot degrade safely")
}
}
func TestPrivileged(t *testing.T) {
cases := []struct {
method, path string
want bool
why string
}{
{"POST", "/v1/iam/mint-user-keys", true, "minting a credential is a grant"},
{"POST", "/v1/iam/issue-user-token", true, "issuing a token is a grant"},
{"POST", "/v1/iam/revoke-user-keys", true, "revocation changes who can act"},
{"POST", "/v1/iam/signup", true, "creating an account is a grant"},
{"POST", "/v1/iam/onboard", true, "minting a tenant is a grant"},
{"GET", "/v1/kms/orgs/acme/secrets/db", true, "reading a secret with a stolen key IS the attack"},
{"DELETE", "/v1/admin/orgs/acme", true, "an admin mutation changes who can do what"},
{"GET", "/v1/admin/orgs", false, "an admin read is audited and access-controlled, not a grant"},
{"POST", "/v1/orgs/acme/members", true, "adding a member grants standing authority"},
{"GET", "/v1/models", false, "an ordinary read is not a grant"},
{"POST", "/v1/ai/chat/completions", false, "inference is not a grant"},
{"GET", "/v1/iam/whoami", false, "reading your own identity grants nothing"},
// REGRESSION — the spelling must not decide. fiber routes case-insensitively
// and ignores a trailing slash, so each of these reaches the SAME handler the
// canonical spelling does; a prefix test over the raw path matched none of
// them, and the scorer's silence then ALLOWED what it must refuse.
{"GET", "/V1/KMS/orgs/acme/secrets/db", true, "one capital letter is not a different route"},
{"GET", "/v1/Kms/orgs/acme/secrets/db", true, "nor is one capital letter in the middle"},
{"POST", "/V1/IAM/MINT-USER-KEYS", true, "a shouted grant is still a grant"},
{"GET", "/v1/kms/", true, "the subtree root is in the subtree"},
{"GET", "/v1/kms", true, "and so is the root without its slash — one route, per StrictRouting"},
{"DELETE", "/V1/Admin/orgs/acme", true, "an admin mutation, whatever its case"},
{"POST", "/v1/orgs/", true, "the org subtree root"},
// And normalization must not WIDEN the list either: a neighbouring name that
// merely shares a prefix is a different route and must stay ordinary.
{"GET", "/v1/kmsx/keys", false, "a prefix of a name is not the subtree"},
{"GET", "/v1/iam/signup-preflight", false, "a longer name is a different route"},
{"POST", "/v1/organizations/x", false, "/v1/orgs is not /v1/organizations"},
}
for _, tc := range cases {
if got := Privileged(tc.method, tc.path); got != tc.want {
t.Errorf("Privileged(%s %s) = %v, want %v — %s", tc.method, tc.path, got, tc.want, tc.why)
}
}
}
// RoutePath is the router's own rule, and a security comparison may use no other.
func TestRoutePath(t *testing.T) {
cases := map[string]string{
"/v1/kms/x": "/v1/kms/x",
"/V1/KMS/X": "/v1/kms/x",
"/v1/kms/": "/v1/kms",
"/v1/kms///": "/v1/kms",
"/": "/",
"": "",
// Percent-encoding is NOT decoded, deliberately: fiber does not decode it
// either, so this spelling routes nowhere and normalizing it here would
// make the predicate match a route that does not exist.
"/v1/%6bms/x": "/v1/%6bms/x",
}
for in, want := range cases {
if got := RoutePath(in); got != want {
t.Errorf("RoutePath(%q) = %q, want %q", in, got, want)
}
}
}
// The scorer is bounded. Every ask costs a goroutine that lives until the scorer
// returns, so a scorer that has stopped returning must stop being asked — past
// the ceiling the fail policy answers, which is the same answer a timeout gives
// and reaches it without allocating anything.
//
// The ceiling is passed in so the property is asserted deterministically rather
// than by launching MaxScorerCalls goroutines and hoping none of them belongs to
// another test. It is the SAME function the production path runs; only the size
// of the ceiling differs.
func TestDecide_IsBoundedWhenTheScorerStops(t *testing.T) {
resetScorer(t)
sem := newCalls(2)
release := make(chan struct{})
t.Cleanup(func() { close(release) })
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
<-release // never answers within the test
return RiskVerdict{Action: ActionAllow}, nil
})
// Fill every slot. Each call returns at the budget; the goroutine behind it
// stays parked, which is precisely the cost being bounded.
var wg sync.WaitGroup
refusals := make([]string, 2)
for i := range refusals {
wg.Add(1)
go func(i int) {
defer wg.Done()
refusals[i] = decide(sem, context.Background(), "acme", RiskQuery{Stage: StageUsage}).Refusal
}(i)
}
wg.Wait()
for i, r := range refusals {
if r != RefusalTimeout {
t.Fatalf("call %d refusal = %q, want %q", i, r, RefusalTimeout)
}
}
// The next one is not asked at all — and the fail policy still holds both ways.
if v := decide(sem, context.Background(), "acme", RiskQuery{Stage: StageUsage}); v.Refusal != RefusalBusy || v.Action != ActionAllow {
t.Fatalf("past the ceiling: %+v, want allow/%s", v, RefusalBusy)
}
if v := decide(sem, context.Background(), "acme", RiskQuery{Stage: StageUsage, Privileged: true}); v.Refusal != RefusalBusy || v.Action != ActionBlock {
t.Fatalf("past the ceiling on a grant: %+v, want block/%s", v, RefusalBusy)
}
}
// The slot is released by the goroutine that HELD it, when the scorer actually
// returns — not when the budget expires. Releasing it at the timeout would let a
// stalled scorer exceed the ceiling without bound, which is the thing the ceiling
// exists to prevent.
func TestDecide_ReleasesItsSlotWhenTheScorerAnswers(t *testing.T) {
resetScorer(t)
sem := newCalls(1)
SetRiskScorer(func(context.Context, string, RiskQuery) (RiskVerdict, error) {
return RiskVerdict{Action: ActionAllow}, nil
})
for i := 0; i < 50; i++ {
if v := decide(sem, context.Background(), "acme", RiskQuery{Stage: StageUsage}); v.Refusal != "" {
t.Fatalf("call %d was refused %q — a returned scorer must give its slot back", i, v.Refusal)
}
}
}
// A SCOPE KEY MUST NAME THE ROUTE. The service axis of the per-scope rate rule
// and of the per-scope spend cap is derived from the path, and it was derived
// from the RAW path — so "/V1/AI/chat" produced the service "AI" while reaching
// exactly the "/v1/ai/chat" handler. A different scope key is a different rate
// bucket and a different cap: an authenticated caller could multiply its own
// ceiling by holding down the shift key.
func TestCanonicalService_NamesTheRouteNotTheSpelling(t *testing.T) {
for _, spelling := range []string{
"/v1/ai/chat", "/V1/AI/chat", "/v1/AI/Chat", "/v1/ai/chat/", "/V1/Ai/",
} {
if got := canonicalService(spelling); got != "ai" {
t.Errorf("canonicalService(%q) = %q, want %q", spelling, got, "ai")
}
}
// The alias table is applied to the normalized segment, not before it.
if got := canonicalService("/V1/ML/models"); got != "compute" {
t.Errorf("canonicalService(/V1/ML/models) = %q, want compute", got)
}
// And a non-/v1 path still has no service.
if got := canonicalService("/healthz"); got != "" {
t.Errorf("canonicalService(/healthz) = %q, want empty", got)
}
}
// A fact we do not have is ABSENT, not empty. An empty string is a VALUE, and a
// scorer keying velocity on one would group every request whose client address
// never arrived — behind a TCP load balancer with no PROXY protocol, that is all
// of them — into a single very busy caller, and refuse the lot.
func TestFacts_AMissingFactIsAbsent(t *testing.T) {
got := Facts(map[string]string{"ip": "", "path": "/v1/models", "credential": "", "peers": "0"})
if _, ok := got["ip"]; ok {
t.Error("an empty ip was sent as a fact; the scorer will treat it as an identity")
}
if _, ok := got["credential"]; ok {
t.Error("an empty credential class was sent as a fact")
}
if got["path"] != "/v1/models" || got["peers"] != "0" {
t.Errorf("a real fact was dropped: %v", got)
}
}
serve.go
+10
View File
@@ -382,6 +382,16 @@ func Listen(plugins []Plugin, enable []string) error {
// BOTH sources are absent.
app.Use(ScopeRateLimit(deps.Metering, deps.GatewayPolicy))
// Lifecycle defense (middleware_abuse.go). Runs AFTER ScopeRateLimit so plain
// over-rate traffic is already 429'd and never reaches the scorer, INSIDE
// AuditTrail so a refusal lands in the tamper-evident trail without a second
// write, and BEFORE the two funding gates so an abusive request cannot consume
// a balance. It keys on the CREDENTIAL, which neither limiter above can see —
// a stolen key inside its org's normal ceiling is invisible to both. SHADOW per
// org by default: it senses and reports, and enforces nothing until an operator
// arms that org at PUT /v1/gateway/config.
app.Use(AbuseGate(deps, deps.Traffic))
// Starter credit — the funding path the two gates below are sequenced behind.
// It needs the gateway-asserted principal to resolve a
// wallet; an unvalidated caller is skipped) and BEFORE both gates, so a brand-new