Compare commits
8
Commits
main
...
blue/risk-fix
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4a10e0f49a | ||
|
|
d10b2d7716 | ||
|
|
2428ef6cad | ||
|
|
0f2e340b76 | ||
|
|
b08faf767b | ||
|
|
fe28b77239 | ||
|
|
cccacd65e3 | ||
|
|
383ce0eee4 |
@@ -4067,3 +4067,214 @@ SDK and every MCP tool list that answer 404 in production: the same dark hole th
|
||||
`api-hanzo-ai-catalog` router opened under `/v1/models` and `/v1/pricing`, which
|
||||
cost 17 documented-but-uncallable operations. Re-home that surface only after the
|
||||
upstream ships it, and prove the upstream answers before declaring anything.
|
||||
## Hanzo Risk (`apps/risk`) — `/v1/risk`
|
||||
|
||||
ONE app, ONE prefix, one core: `/v1/risk` DECIDES and LEARNS for any entity
|
||||
(account, transaction, session, agent, merchant, payout). Fraud is a USE of
|
||||
`/v1/risk` — so is abuse, so are bots, account takeover, spam and pay-as-you-go
|
||||
abuse. **There is no `/v1/fraud`.**
|
||||
|
||||
**THREE FACES, NO OVERLAP.** `/v1/aml` is compliance (cases, sanctions,
|
||||
retention — LIVE at v0.3.7, served by the amld pod). `/v1/ml` is model SERVING
|
||||
(`apps/ml`: InferenceServices, `/v1/ml/models`, predict — LIVE, other customers).
|
||||
`/v1/risk` is this one. An earlier cut put the learning leaves — score, train,
|
||||
state, features, search, snapshot, restore — under `/v1/ml`, on the premise that
|
||||
the prefix was free. **It is not: `/v1/ml/health` answers 200 in production and
|
||||
`/v1/ml/models` is auth-gated at 403 today.** "Models you serve" and "models that
|
||||
learn" are two concepts, and two concepts under one name is what the manifest
|
||||
exists to refuse. The move is a declaration change (typed zip ops), the risk row
|
||||
is now `{"/v1/risk"}` alone, and the live serving paths are byte-identical in the
|
||||
woven document. There is no alias from the old spelling: it never shipped.
|
||||
|
||||
Deciding and learning stay in ONE binary because they are ONE state — one
|
||||
half-space forest per tenant, held as mass counters that every score reads and
|
||||
every train writes. Split across two rows they would be two processes holding
|
||||
different counters, with no error, no log and no 404: just two different answers
|
||||
to one question.
|
||||
|
||||
**The engine is a MODULE DEP, and only part of it is linked.** `github.com/luxfi/aml`
|
||||
enters the way `luxfi/kms` and `hanzoai/o11y` do — no source vendored. Only four
|
||||
packages are linked: `types`, `velocity`, `anomaly`, `replay`.
|
||||
|
||||
**MEASURED CORRECTION, worth not re-learning: `pkg/engine` is NOT base-free.**
|
||||
A grep for `hanzoai/base` inside its own files finds nothing, which is how the
|
||||
claim gets made. `go list -deps` finds the truth:
|
||||
|
||||
pkg/engine -> pkg/history -> pkg/store -> github.com/hanzoai/base/core
|
||||
-> github.com/hanzoai/tasks (Temporal SDK)
|
||||
|
||||
Same for `pkg/measure` and `pkg/history`. Linking them would drag an application
|
||||
framework and a workflow engine into a payment-authorization path. So the risk
|
||||
rule plane is a CLOSED ALGEBRA declared cloud-side (`rule.go`): a fixed field
|
||||
vocabulary, a fixed operator set, conjunction only. That is injection-safe by
|
||||
construction, is a TYPED wire shape (so it reaches the SDKs, the CLI and the MCP
|
||||
tools as a schema instead of an opaque expression string), and replays without an
|
||||
interpreter. The upstream fix — split `pkg/history`'s Base-backed store into its
|
||||
own package — is owed and would let `pkg/engine` in later without a rewrite here.
|
||||
|
||||
**The hot path never reads the warehouse.** A payment decide sits inside a
|
||||
processor's authorization window and the datastore is one StatefulSet pod that has
|
||||
taken api.hanzo.ai down before. `POST /v1/risk/decide` reads in-memory velocity
|
||||
rings, the tenant's own SQLite, and the in-process model. ClickHouse
|
||||
(`hanzo.risk_feature`) is the BACKFILL that warms the rings and the read behind
|
||||
the dictionary and the search sandbox. Measure p50/p99 with the warehouse DOWN; a
|
||||
decide that needs analytics up is a payment plane that fails when analytics does.
|
||||
|
||||
**Cross-org learning is AGGREGATE-ONLY and the boundary is three layers deep:**
|
||||
|
||||
type level apps/risk/feature.go is the ONLY file that reads a feature table,
|
||||
and every function there takes a minted `Tenant`. Tenant has no
|
||||
exported constructor and no json tags, so it cannot arrive off the
|
||||
wire; tenantOf(ctx) is the only source and it reads the VALIDATED
|
||||
principal cloud.Bridge parked.
|
||||
value level hanzo.risk_baseline has NO tenant column, no subject, no id and no
|
||||
pseudonym — quantiles only. There is no query that returns one
|
||||
org's rows because the rows do not exist. The populate statement
|
||||
is a package CONSTANT with no placeholder and a HAVING clause
|
||||
carrying the k-anonymity floor (25 orgs, 1000 observations).
|
||||
key level the tenant key is `<brand>/<org>` (tenant.go qualify), minted in
|
||||
ONE place, and the brand half comes from deps.Brand and NEVER a
|
||||
header. It is the store index, the engine's org column AND the
|
||||
seed of the per-tenant tree geometry — so two tenants do not
|
||||
merely hold different counters, they hold different TREES.
|
||||
|
||||
`$public` — the reserved anonymous lane the event door files credential-less
|
||||
writes under — is refused at the mint AND excluded from the baseline statement.
|
||||
|
||||
**Shadow is the default and the default is not configurable.** Two gates in
|
||||
series: the engine store is constructed `Shadow: true`, and a tenant's own record
|
||||
plane holds `mode` (shadow|live, `PUT /v1/risk/mode`). In shadow every rule runs,
|
||||
the model scores and learns, every decision is recorded, and the action is always
|
||||
allow. Statistical evidence is capped at `review` (`modelCeiling`) and that cap is
|
||||
NOT weakened for the payment stage: a model can put a transaction in front of a
|
||||
person; it cannot decline one.
|
||||
|
||||
**Durable first, analytics after.** Every decision lands in the tenant's own
|
||||
encrypted SQLite before `analytics.PublishEvents` emits the copy. `/v1/event` is
|
||||
best-effort by design — it answers `{accepted, dropped}` and the anonymous lane
|
||||
drops on purpose — so a record that rode it would be lost by a bus hiccup,
|
||||
invisibly.
|
||||
|
||||
**Shutdown is not housekeeping.** cloud deploys `strategy: Recreate` at 1 replica,
|
||||
so every rollout drops every in-memory model. `Plugin.Shutdown` snapshots every
|
||||
resident tenant and `tenantState` restores on first touch. Without it every deploy
|
||||
returns every tenant to warming — and a warming model REFUSES to score, which
|
||||
reads as "clean" to anything that does not check `Refusal`.
|
||||
|
||||
**Metering: `decide` meters AFTER, `train` and `search` gate BEFORE.**
|
||||
`cloud.DenyResource` writes the fleet's NESTED `{"error":{...}}` 402 in band and a
|
||||
typed op's error renders FLAT, so a pre-work gate on the hot path would either
|
||||
move the 402 body every balance-aware client parses or force the route untyped.
|
||||
The screen is billed on the decision that was actually produced. `train` and
|
||||
`search` are real CPU and ARE gated — they are new routes, so no client parses a
|
||||
nested body from them and a typed 402 costs nothing.
|
||||
|
||||
**Untyped by design: exactly one route.** `GET /v1/risk/health` answers 503
|
||||
carrying the degraded REPORT as its body. `apps/risk/typed_wire_test.go` holds the
|
||||
closed list, and it also pins the whole served surface as a diffable list —
|
||||
35 operations, 34 MCP tools.
|
||||
|
||||
**Using this plane is not GOVERNING it.** One predicate, one place
|
||||
(`governState`): a write that can turn a control off — `PUT /v1/risk/mode`,
|
||||
retiring a rule, a blanket suppression, an allow-list entry, appetite, snapshot,
|
||||
restore — requires `principal.IsOrgAdmin` for the caller's OWN org. It is
|
||||
org-scoped and org-scoped only; there is no cross-tenant surface here, so there
|
||||
is nothing for platform authority to reach and conflating the two scopes would be
|
||||
a privilege escalation. Scoring, deciding, reading and labelling are unchanged for
|
||||
an ordinary member. Every governance write emits with its actor.
|
||||
|
||||
**A value that outlives its request OWNS ITS BYTES.** fasthttp owns the buffers
|
||||
behind header values and the request path and REUSES them for the next request on
|
||||
the connection, so `sc.org`, `by(sc)` and every path parameter are strings
|
||||
pointing at memory the server is about to overwrite. `emit` published those from a
|
||||
goroutine, which marshalled whatever the NEXT request wrote there — another
|
||||
tenant's user id under this tenant's `DistinctID`, a data race under `-race` and a
|
||||
silent wrong analytics record without it. `detach` (store.go) takes ownership at
|
||||
the ONE publish site, and `analytics.PublishEvents` is spelled exactly once in the
|
||||
package so a second detach cannot skip it.
|
||||
|
||||
**An op cannot obtain a file handle without the error that says it has none.**
|
||||
`residency.close` retires EVERY cell the instant teardown runs — no idle
|
||||
requirement — and `(*sql.DB)(nil).QueryRow` locks a nil mutex, so on a Recreate
|
||||
one-replica rollout a request already holding a cell could nil-dereference and
|
||||
take every tenant on the pod with it. `resident.file()` answers `(handle, error)`
|
||||
and `tenantState` resolves it once for all 27 op sites, so the nil is
|
||||
unrepresentable downstream rather than a rule each caller has to remember.
|
||||
|
||||
### The operating point an operator has to know
|
||||
|
||||
**ONE BOUND, AND IT IS BYTES.** `RISK_MEMORY` (default 512 MiB) is the node's
|
||||
whole budget for resident risk state. There is no tenant COUNT: a count over
|
||||
state the tenant sizes is not a bound, and pricing every tenant at its worst case
|
||||
is what refused the 49th concurrently-active org the entire risk surface. A
|
||||
tenant is charged for what it HOLDS.
|
||||
|
||||
textMax 256 B the cap on ANY caller-supplied text — subject id,
|
||||
signal value, agent ref, list entry, rule term, path
|
||||
segment. Enforced ONCE at the wire door (door.go) so
|
||||
every count below is a byte figure. 400 past it.
|
||||
bodyMax 4 MiB per request, under the edge's fleet-wide 16 MiB.
|
||||
bytesPerKey 6,560 B 94 buckets x 48 + 1,280 overhead + 3 x textMax.
|
||||
MEASURED at 5,965 B for the longest value the door
|
||||
accepts (hold_test.go), so the published figure over-
|
||||
states, which is the direction a ceiling must err in.
|
||||
velBytes 8 MiB per-tenant aggregate budget -> maxKeys = 1,278.
|
||||
cellBytes 784 KiB an armed cell before it counts anything: forest
|
||||
336 KiB + file 64 KiB + agency memo 384 KiB.
|
||||
governMemo ≤3.5 MiB rules 1 MiB + suppressions 512 KiB + lists 2 MiB.
|
||||
Priced LIVE, never reserved and never refused —
|
||||
refusing to load a tenant's rules would disarm its
|
||||
controls to save memory.
|
||||
per tenant ≤12.2 MiB cellBytes + velBytes + governMemo.
|
||||
recordBudget 256 MiB per tenant on the DISK volume: the decision log is a
|
||||
RING (recordCap = 16,384 rows at recordMax 16 KiB),
|
||||
pruned at open and every 256 writes. Refusing a
|
||||
decision would refuse the authorization; dropping the
|
||||
oldest is only honest if the window is published, so
|
||||
`GET /v1/risk/decisions` carries `retained`+`oldest`.
|
||||
|
||||
RISK_MEMORY 512 MiB = ~660 ordinary tenants, or 42 simultaneously at their
|
||||
full ceiling. Scaling past that is a SHARDING answer:
|
||||
the shard router pins an org to one pod.
|
||||
|
||||
**EVERY DEGRADATION IS NAMED, AND READABLE.** `GET /v1/risk/health` carries
|
||||
`{tenants, bytes, bytes_max, refused, reclaimed, strained, model, warehouse,
|
||||
billing}` and goes 503 for `capacityAlarm` after a refusal.
|
||||
|
||||
strained (decision + probe) this tenant's rings refused a NEW key — its own
|
||||
cardinality bound or the node's memory. Its
|
||||
counts may under-report its OWN traffic, so a
|
||||
rule on `velocity.ip.1h.count` is reading a
|
||||
partial ring. The gate refuses EXACTLY at the
|
||||
bound and counts it; the engine's own per-shard
|
||||
LRU is put out of reach (MaxKeys x 64) because
|
||||
its eviction is silent and starts at ~72% of the
|
||||
nominal bound.
|
||||
reclaimed (probe) cells taken back under memory pressure. Only a
|
||||
cell silent past `idleFloor` (16 min) may go, so
|
||||
no tenant's arrival ever costs a tenant that is
|
||||
working its rings.
|
||||
refused (probe, 503) a newcomer turned away: the node had no memory
|
||||
AND nothing idle to reclaim. Add a writer or
|
||||
raise RISK_MEMORY.
|
||||
disarmed (decision) learned state existed and this process does not
|
||||
have it. `POST /v1/risk/restore` reinstates.
|
||||
|
||||
**`luxfi/aml` v0.3.5 pulls `luxfi/fhe` v1.8.2 -> v1.8.8 (indirect).** Not a bump
|
||||
anyone chose: `go mod graph` shows `luxfi/aml@v0.3.5 -> luxfi/fhe@v1.8.8` and MVS
|
||||
takes the max against `hanzoai/base`'s v1.8.2. It stays inside v1.x. Removing the
|
||||
aml dependency is the only way to avoid it.
|
||||
|
||||
**Build:** `-tags sqlite_math_functions` under CGO (hanzoai/base's gate).
|
||||
`modernc.org/sqlite` is NOT in the risk graph, so the image's SQLITE-GATE stays
|
||||
green — verify with
|
||||
`CGO_ENABLED=1 go list -tags 'libsqlite3 sqlite_fts5 sqlite_math_functions' -deps ./plugin/risk | grep modernc`.
|
||||
|
||||
**No chart change is needed to serve it.** `api.hanzo.ai/` already routes to
|
||||
cloud, `CLOUD_ENABLE` is unset in `charts/app/values/hanzo/cloud.yaml` (so every
|
||||
manifest app is enabled), and the image builds one binary per `plugin/<app>`.
|
||||
Landing it is an image pin. **`/v1/aml` is deliberately NOT claimed here** — the
|
||||
Ingress still points that path at `service: aml` (the amld pod holding the live
|
||||
5-year retention plane), and claiming it in cloud without deleting that rule in
|
||||
the SAME commit gives a compliance surface answering from a store that is not the
|
||||
record.
|
||||
|
||||
@@ -395,6 +395,11 @@ func Mount(app cloud.Router, deps cloud.Deps) error {
|
||||
// ends where the product starts (provenance.go).
|
||||
mountProvenance(s)
|
||||
|
||||
// The registry's yes/no, for a subsystem in ANOTHER process (declared_rpc.go).
|
||||
// ListForOrg serves the in-binary readers; this serves the risk plane, whose
|
||||
// agent-versus-bot decision would otherwise have to trust a request field.
|
||||
exposeDeclared()
|
||||
|
||||
log.Info("agents mounted", "ai", s.State.ai != nil, "billing", s.State.bill.Enabled(),
|
||||
"scheduler", s.State.sched != nil, "brand", deps.Brand)
|
||||
return nil
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
package agents
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/hanzoai/cloud"
|
||||
"github.com/hanzoai/cloud/plane"
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// The registry's yes/no, published on the internal plane.
|
||||
//
|
||||
// ListForOrg is the same fact for a subsystem sharing this binary. This is the
|
||||
// same fact for one that does not — and the risk plane is exactly that: its own
|
||||
// manifest row, its own process, and a decision that turns on whether an actor
|
||||
// is a DECLARED agent or an undeclared script. Without this op that distinction
|
||||
// could only be read off the request body, which is the caller asserting its own
|
||||
// agency.
|
||||
//
|
||||
// The projection is one boolean. See plane.AgentDeclared for why the record
|
||||
// itself is not the answer.
|
||||
|
||||
// exposeDeclared publishes the resolution. Mount calls it.
|
||||
//
|
||||
// A NAMED handler, not a closure: zipdoc lifts an op's prose off its handler's
|
||||
// doc comment and can lift nothing from an anonymous one.
|
||||
func exposeDeclared() {
|
||||
zip.Post[plane.AgentRef, plane.AgentDeclared](cloud.Plane(), "/agents/declared",
|
||||
declared,
|
||||
zip.WithOperationID(plane.AgentsDeclared),
|
||||
zip.WithSummary("Whether a reference names an agent in the caller's own org"))
|
||||
}
|
||||
|
||||
// declared answers whether ref resolves to an agent in the CALLER'S OWN org.
|
||||
//
|
||||
// The org is taken from the authenticated call and can never be an argument: it
|
||||
// is the registry's tenancy key, so a caller able to name it could probe another
|
||||
// tenant's agents by reference. A call carrying no org is refused rather than
|
||||
// answered false — "no" and "we would not say" are different facts, and only the
|
||||
// second one is a reason for the asker to treat the actor as unclassified.
|
||||
//
|
||||
// The store comes from mountedStore, the ONE door for an in-process seam whose
|
||||
// org was resolved server-side. It re-applies the same bound the HTTP path
|
||||
// applies, so this op cannot be granted an org key a request would have been
|
||||
// refused — and it FAILS CLOSED when the registry is not open, because answering
|
||||
// false there would silently reclassify every one of an org's real agents as
|
||||
// undeclared automation.
|
||||
func declared(ctx context.Context, in *plane.AgentRef) (*plane.AgentDeclared, error) {
|
||||
org := strings.TrimSpace(cloud.Who(ctx).Org)
|
||||
if org == "" {
|
||||
return nil, zip.ErrUnauthorized("agents: no org on the call")
|
||||
}
|
||||
ref := strings.TrimSpace(in.Ref)
|
||||
if ref == "" || len(ref) > maxAgentLabel {
|
||||
// An empty or oversized reference names nothing. It is a well-formed
|
||||
// question with the answer "no", not an error.
|
||||
return &plane.AgentDeclared{}, nil
|
||||
}
|
||||
sto, org, err := mountedStore(org)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("agents: registry unavailable: %w", err)
|
||||
}
|
||||
if _, err := sto.Resolve(ctx, org, ref); err != nil {
|
||||
if err == errNotFound {
|
||||
return &plane.AgentDeclared{}, nil
|
||||
}
|
||||
return nil, fmt.Errorf("agents: resolve: %w", err)
|
||||
}
|
||||
return &plane.AgentDeclared{Declared: true}, nil
|
||||
}
|
||||
+51
-23
@@ -10,14 +10,14 @@ import (
|
||||
|
||||
func init() {
|
||||
zip.Describe("DELETE /v1/agents/:ref", zip.Doc{
|
||||
Description: "DeleteAgent removes an agent and every run recorded against it. Answers 204.",
|
||||
Description: "Removes an agent and every run recorded against it. Answers 204.",
|
||||
Fields: map[string]string{
|
||||
"agentRef.ref": "Ref is the agent's public id (the agent_… handle create and list return) or\nits org-unique name, from the path. Either resolves the same agent.",
|
||||
},
|
||||
Example: json.RawMessage(`{"ref":"helper"}`),
|
||||
})
|
||||
zip.Describe("DELETE /v1/agents/targets/:id", zip.Doc{
|
||||
Description: "DeleteTarget deregisters one machine. Only its owner, or an org admin, may\nremove it; an unknown id, a cross-org id and a machine owned by someone else\nall answer the same not-found, so a probe learns nothing about what exists.",
|
||||
Description: "Deregisters one machine. Only its owner, or an org admin, may\nremove it; an unknown id, a cross-org id and a machine owned by someone else\nall answer the same not-found, so a probe learns nothing about what exists.",
|
||||
Fields: map[string]string{
|
||||
"targetDeleted.deleted": "Deleted is true when the target was removed.",
|
||||
"targetDeleted.id": "ID is the target that was removed.",
|
||||
@@ -26,20 +26,20 @@ func init() {
|
||||
Example: json.RawMessage(`{"id":"tgt_1"}`),
|
||||
})
|
||||
zip.Describe("GET /v1/agents", zip.Doc{
|
||||
Description: "ListAgents returns every agent defined in the caller's org, each with the\nnumber of runs recorded against it.",
|
||||
Description: "Returns every agent defined in the caller's org, each with the\nnumber of runs recorded against it.",
|
||||
Fields: map[string]string{
|
||||
"agentList.agents": "Agents is the org's agents, each carrying its recorded run count.",
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/agents/:ref", zip.Doc{
|
||||
Description: "GetAgent returns one agent with its system prompt and its 20 most recent runs.\nThe ref is the agent's public id or its org-unique name — a created agent is\nimmediately gettable by whatever create handed back.",
|
||||
Description: "Returns one agent with its system prompt and its 20 most recent runs.\nThe ref is the agent's public id or its org-unique name — a created agent is\nimmediately gettable by whatever create handed back.",
|
||||
Fields: map[string]string{
|
||||
"agentRef.ref": "Ref is the agent's public id (the agent_… handle create and list return) or\nits org-unique name, from the path. Either resolves the same agent.",
|
||||
},
|
||||
Example: json.RawMessage(`{"ref":"helper"}`),
|
||||
})
|
||||
zip.Describe("GET /v1/agents/:ref/runs", zip.Doc{
|
||||
Description: "ListAgentRuns returns one agent's execution history, newest first — each run's\ninput, its output or its error, and how long it took. Every row is a run that\nactually happened.",
|
||||
Description: "Returns one agent's execution history, newest first — each run's\ninput, its output or its error, and how long it took. Every row is a run that\nactually happened.",
|
||||
Fields: map[string]string{
|
||||
"runList.runs": "Runs is the agent's executions, newest first.",
|
||||
"runsQuery.limit": "Limit caps how many runs come back, newest first. Absent, zero or out of\nrange (1..200) reads as 50.",
|
||||
@@ -48,7 +48,7 @@ func init() {
|
||||
Example: json.RawMessage(`{"ref":"helper","limit":20}`),
|
||||
})
|
||||
zip.Describe("GET /v1/agents/activity", zip.Doc{
|
||||
Description: "AgentActivity serves the org-wide recent-activity feed. Events are REAL: each\nrecorded run is an invoked (ok) or failed (error) event; each agent's own\ncreate/update timestamps are created/updated events. Merged, newest first,\ncapped. Nothing is invented — an org with no agents and no runs gets [].",
|
||||
Description: "Serves the org-wide recent-activity feed. Events are REAL: each\nrecorded run is an invoked (ok) or failed (error) event; each agent's own\ncreate/update timestamps are created/updated events. Merged, newest first,\ncapped. Nothing is invented — an org with no agents and no runs gets [].",
|
||||
Fields: map[string]string{
|
||||
"activityFeed.activity": "Activity is the merged run/create/update events, newest first, capped at 50.",
|
||||
"activityView.agent": "agent name",
|
||||
@@ -57,14 +57,14 @@ func init() {
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/agents/builds", zip.Doc{
|
||||
Description: "ListBuilds returns the public index of every published build, most recently\nupdated first, so a gallery can link straight to the story behind each product.\nPUBLIC, no tenancy: publishing is the author's act, and only published root\nsessions appear here.",
|
||||
Description: "Returns the public index of every published build, most recently\nupdated first, so a gallery can link straight to the story behind each product.\nPUBLIC, no tenancy: publishing is the author's act, and only published root\nsessions appear here.",
|
||||
Fields: map[string]string{
|
||||
"buildList.builds": "Builds is every published build, most recently updated first.",
|
||||
"buildsQuery.limit": "Limit caps the page. Absent, zero or over 500 reads as 100.",
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/agents/builds/:org/:project", zip.Doc{
|
||||
Description: "ReadBuild returns the readable build of one product: the agent session that\nproduced it, turn by turn — the prompts, the reasoning, the commits each turn\nproduced — plus the exact `git log` that re-derives every commit binding from\ngit itself, so nothing here has to be taken on trust.\n\nPUBLIC, no tenancy: it answers only for a session its author explicitly\npublished, which is what makes it safe to be anonymous. An unpublished session\nis invisible here no matter who asks; its owner reads it through the org-scoped\n/v1/agents/sessions routes, which need a validated principal.",
|
||||
Description: "Returns the readable build of one product: the agent session that\nproduced it, turn by turn — the prompts, the reasoning, the commits each turn\nproduced — plus the exact `git log` that re-derives every commit binding from\ngit itself, so nothing here has to be taken on trust.\n\nPUBLIC, no tenancy: it answers only for a session its author explicitly\npublished, which is what makes it safe to be anonymous. An unpublished session\nis invisible here no matter who asks; its owner reads it through the org-scoped\n/v1/agents/sessions routes, which need a validated principal.",
|
||||
Fields: map[string]string{
|
||||
"buildRef.org": "Org is the org that published the build, from the path.",
|
||||
"buildRef.project": "Project is the product's slug, from the path.",
|
||||
@@ -73,7 +73,7 @@ func init() {
|
||||
Example: json.RawMessage(`{"org":"hanzo","project":"landing"}`),
|
||||
})
|
||||
zip.Describe("GET /v1/agents/metrics", zip.Doc{
|
||||
Description: "AgentMetrics serves the invocations-over-time histogram for the org's Agents\ndashboard. Every point is a REAL count of recorded runs in that time bucket —\none series line per agent that ran in the window. The Resource Usage rollup is\nall-null because this store meters no CPU/memory/storage/cost; the console\nrenders those as \"—\" rather than a fabricated figure. No runs => empty series\n(an honest \"not connected / no activity yet\"), never a synthesized trend.",
|
||||
Description: "Serves the invocations-over-time histogram for the org's Agents\ndashboard. Every point is a REAL count of recorded runs in that time bucket —\none series line per agent that ran in the window. The Resource Usage rollup is\nall-null because this store meters no CPU/memory/storage/cost; the console\nrenders those as \"—\" rather than a fabricated figure. No runs => empty series\n(an honest \"not connected / no activity yet\"), never a synthesized trend.",
|
||||
Fields: map[string]string{
|
||||
"metricsQuery.range": "Range is the window to bucket: 24H, 7D or 30D. Anything else reads as 30D.",
|
||||
"metricsView.range": "echoes the requested window (24H|7D|30D)",
|
||||
@@ -85,7 +85,7 @@ func init() {
|
||||
Example: json.RawMessage(`{"range":"7D"}`),
|
||||
})
|
||||
zip.Describe("GET /v1/agents/sessions", zip.Doc{
|
||||
Description: "ListSessions returns the caller org's live sessions, newest first — each with\nits event count, its direct-child count and a one-line preview of its latest\nevent. With no filter it returns ROOT sessions only, so a dashboard shows one\nrow per flow rather than one per subagent; ?root= or ?parent= descends.",
|
||||
Description: "Returns the caller org's live sessions, newest first — each with\nits event count, its direct-child count and a one-line preview of its latest\nevent. With no filter it returns ROOT sessions only, so a dashboard shows one\nrow per flow rather than one per subagent; ?root= or ?parent= descends.",
|
||||
Fields: map[string]string{
|
||||
"sessionList.sessions": "Sessions is the matching sessions, each with its event and child counts and\na one-line preview of its latest event.",
|
||||
"sessionQuery.limit": "Limit caps the page. Absent, zero or over 500 reads as 100.",
|
||||
@@ -102,7 +102,7 @@ func init() {
|
||||
Example: json.RawMessage(`{"status":"running","limit":20}`),
|
||||
})
|
||||
zip.Describe("GET /v1/agents/sessions/:id", zip.Doc{
|
||||
Description: "GetSession returns one session with its direct child sessions and its 50 most\nrecent events, oldest of those first.",
|
||||
Description: "Returns one session with its direct child sessions and its 50 most\nrecent events, oldest of those first.",
|
||||
Fields: map[string]string{
|
||||
"sessionRef.id": "ID is the session to act on, from the path.",
|
||||
"sessionView.host": "Execution context (mission-control): the machine/repo/cwd a card shows and\nthe run-target a session is dispatched to. Omitted when a surface didn't report it.",
|
||||
@@ -114,7 +114,7 @@ func init() {
|
||||
Example: json.RawMessage(`{"id":"sess_1"}`),
|
||||
})
|
||||
zip.Describe("GET /v1/agents/sessions/:id/control", zip.Doc{
|
||||
Description: "DrainSessionControl returns the steering commands (pause/resume/stop/message)\nrecorded against the caller's own session that are newer than the cursor,\noldest first, with the cursor to poll from next. It is how a locally started\n`hanzo code` session — which is not task-backed, so nothing forwards its\ncommands to an execution engine — consumes what the dashboard posted. Read-only\nand bounded at 200 per poll, so a steady poll is cheap and an applied command is\nnever redelivered.",
|
||||
Description: "Returns the steering commands (pause/resume/stop/message)\nrecorded against the caller's own session that are newer than the cursor,\noldest first, with the cursor to poll from next. It is how a locally started\n`hanzo code` session — which is not task-backed, so nothing forwards its\ncommands to an execution engine — consumes what the dashboard posted. Read-only\nand bounded at 200 per poll, so a steady poll is cheap and an applied command is\nnever redelivered.",
|
||||
Fields: map[string]string{
|
||||
"controlDrain.commands": "Commands is the session's control commands newer than the cursor, oldest first.",
|
||||
"controlDrain.cursor": "Cursor is the seq to send as `after` on the next poll — the highest seq in\nthis page, or the cursor sent in when the page is empty.",
|
||||
@@ -124,7 +124,7 @@ func init() {
|
||||
Example: json.RawMessage(`{"id":"sess_1","after":12}`),
|
||||
})
|
||||
zip.Describe("GET /v1/agents/sessions/:id/tree", zip.Doc{
|
||||
Description: "SessionTree returns the subagent-flow graph rooted at this session: the session,\nits children, their children, each node carrying its own event count. One\nindexed read pulls the whole flow (every node of a flow shares a root id), so\nthe shape is assembled in memory rather than by walking the store per node.",
|
||||
Description: "Returns the subagent-flow graph rooted at this session: the session,\nits children, their children, each node carrying its own event count. One\nindexed read pulls the whole flow (every node of a flow shares a root id), so\nthe shape is assembled in memory rather than by walking the store per node.",
|
||||
Fields: map[string]string{
|
||||
"sessionRef.id": "ID is the session to act on, from the path.",
|
||||
"sessionView.host": "Execution context (mission-control): the machine/repo/cwd a card shows and\nthe run-target a session is dispatched to. Omitted when a surface didn't report it.",
|
||||
@@ -135,8 +135,11 @@ func init() {
|
||||
},
|
||||
Example: json.RawMessage(`{"id":"sess_1"}`),
|
||||
})
|
||||
zip.Describe("GET /v1/agents/sessions/stream", zip.Doc{
|
||||
Description: "Binds a Service-scoped handler to a route: it adapts a\n`func(*Service[S], *zip.Ctx) error` to the plain `func(*zip.Ctx) error` the\nrouter takes, capturing s. One adapter, so packages write free-function\nhandlers and register them with `app.Get(\"/path\", cloud.Handle(s, myHandler))`.",
|
||||
})
|
||||
zip.Describe("GET /v1/agents/targets", zip.Doc{
|
||||
Description: "ListTargets returns every machine registered to the caller's org, newest\nfirst, each with its live session load.",
|
||||
Description: "Returns every machine registered to the caller's org, newest\nfirst, each with its live session load.",
|
||||
Fields: map[string]string{
|
||||
"GPU.memory": "VRAM bytes, 0 = unknown",
|
||||
"GPU.model": "\"GB10\", \"8060S\", \"RTX 4090\"",
|
||||
@@ -153,7 +156,7 @@ func init() {
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/agents/targets/:id", zip.Doc{
|
||||
Description: "GetTarget returns one registered machine, with its live session load.",
|
||||
Description: "Returns one registered machine, with its live session load.",
|
||||
Fields: map[string]string{
|
||||
"GPU.memory": "VRAM bytes, 0 = unknown",
|
||||
"GPU.model": "\"GB10\", \"8060S\", \"RTX 4090\"",
|
||||
@@ -171,14 +174,14 @@ func init() {
|
||||
Example: json.RawMessage(`{"id":"tgt_1"}`),
|
||||
})
|
||||
zip.Describe("PATCH /v1/agents/:ref", zip.Doc{
|
||||
Description: "UpdateAgent changes an agent in place. Every field is optional; a field the\nrequest omits keeps its stored value. The resulting mode+schedule are\nre-validated together, so a partial update can never leave a long-running\nagent without the cron the scheduler needs to fire it, and a transition INTO\nlong-running counts against the per-org cap on scheduled agents.",
|
||||
Description: "Changes an agent in place. Every field is optional; a field the\nrequest omits keeps its stored value. The resulting mode+schedule are\nre-validated together, so a partial update can never leave a long-running\nagent without the cron the scheduler needs to fire it, and a transition INTO\nlong-running counts against the per-org cap on scheduled agents.",
|
||||
Fields: map[string]string{
|
||||
"updateAgentIn.ref": "Ref is the agent to update — its public id or org-unique name, from the path.",
|
||||
},
|
||||
Example: json.RawMessage(`{"ref":"helper","instructions":"be terse and cite sources"}`),
|
||||
})
|
||||
zip.Describe("PATCH /v1/agents/sessions/:id", zip.Doc{
|
||||
Description: "PatchSession updates a session's surface-owned truth: its status, its title,\nthe run-target it is dispatched to, and the product it built plus whether that\nbuild's story is public. A FINISHED session stays finished — reopening a\ndone/error run would fabricate liveness — and publishing is refused unless the\nsession names the project it built, because the public build route is keyed on\n(org, project).",
|
||||
Description: "Updates a session's surface-owned truth: its status, its title,\nthe run-target it is dispatched to, and the product it built plus whether that\nbuild's story is public. A FINISHED session stays finished — reopening a\ndone/error run would fabricate liveness — and publishing is refused unless the\nsession names the project it built, because the public build route is keyed on\n(org, project).",
|
||||
Fields: map[string]string{
|
||||
"patchSessionIn.id": "ID is the session to update, from the path.",
|
||||
"patchSessionIn.project": "Project tags the product this session built; Published is the author's\ndecision to let anyone read the story (provenance.go). Both are pointers so\n\"absent\" and \"cleared\" are different requests.",
|
||||
@@ -193,7 +196,7 @@ func init() {
|
||||
Example: json.RawMessage(`{"id":"sess_1","status":"done"}`),
|
||||
})
|
||||
zip.Describe("PATCH /v1/agents/targets/:id", zip.Doc{
|
||||
Description: "PatchTarget updates one machine in place. Every field is optional; a field the\nrequest omits is left alone. A metrics patch IS a heartbeat — the server stamps\nits own clock, so a client can neither forge nor backdate staleness.",
|
||||
Description: "Updates one machine in place. Every field is optional; a field the\nrequest omits is left alone. A metrics patch IS a heartbeat — the server stamps\nits own clock, so a client can neither forge nor backdate staleness.",
|
||||
Fields: map[string]string{
|
||||
"GPU.memory": "VRAM bytes, 0 = unknown",
|
||||
"GPU.model": "\"GB10\", \"8060S\", \"RTX 4090\"",
|
||||
@@ -211,12 +214,22 @@ func init() {
|
||||
},
|
||||
Example: json.RawMessage(`{"id":"tgt_1","status":"draining"}`),
|
||||
})
|
||||
zip.Describe("POST /agents/declared", zip.Doc{
|
||||
Description: "Answers whether ref resolves to an agent in the CALLER'S OWN org.\n\nThe org is taken from the authenticated call and can never be an argument: it\nis the registry's tenancy key, so a caller able to name it could probe another\ntenant's agents by reference. A call carrying no org is refused rather than\nanswered false — \"no\" and \"we would not say\" are different facts, and only the\nsecond one is a reason for the asker to treat the actor as unclassified.\n\nThe store comes from mountedStore, the ONE door for an in-process seam whose\norg was resolved server-side. It re-applies the same bound the HTTP path\napplies, so this op cannot be granted an org key a request would have been\nrefused — and it FAILS CLOSED when the registry is not open, because answering\nfalse there would silently reclassify every one of an org's real agents as\nundeclared automation.",
|
||||
Fields: map[string]string{
|
||||
"AgentDeclared.declared": "true when the reference resolves in the caller's own registry",
|
||||
"AgentRef.ref": "the id or name to resolve, as the caller's own org spells it",
|
||||
},
|
||||
})
|
||||
zip.Describe("POST /v1/agents", zip.Doc{
|
||||
Description: "CreateAgent defines an agent in the caller's org: a model, a system prompt\n(instructions) and a set of tool names. The name must be unique in the org and\nmatch ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$. An omitted model takes the\ndeployment's configured default; a named one is checked against the gateway's\nserved catalog, so a model this deployment never serves is refused here rather\nthan failing at run time. A long-running agent must carry a 5-field cron\nschedule (the scheduler would otherwise never fire it) and counts against a\nper-org cap on scheduled agents.",
|
||||
Description: "Defines an agent in the caller's org: a model, a system prompt\n(instructions) and a set of tool names. The name must be unique in the org and\nmatch ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$. An omitted model takes the\ndeployment's configured default; a named one is checked against the gateway's\nserved catalog, so a model this deployment never serves is refused here rather\nthan failing at run time. A long-running agent must carry a 5-field cron\nschedule (the scheduler would otherwise never fire it) and counts against a\nper-org cap on scheduled agents.",
|
||||
Example: json.RawMessage(`{"name":"helper","model":"enso-flash","instructions":"be terse"}`),
|
||||
})
|
||||
zip.Describe("POST /v1/agents/:ref/run", zip.Doc{
|
||||
Description: "Binds a Service-scoped handler to a route: it adapts a\n`func(*Service[S], *zip.Ctx) error` to the plain `func(*zip.Ctx) error` the\nrouter takes, capturing s. One adapter, so packages write free-function\nhandlers and register them with `app.Get(\"/path\", cloud.Handle(s, myHandler))`.",
|
||||
})
|
||||
zip.Describe("POST /v1/agents/sessions", zip.Doc{
|
||||
Description: "RegisterSession opens a live agent session in the caller's org — the row every\nsurface (the CLI's outer agent, hanzo.bot, the console, chat) hangs its\nactivity off. A session with a parentSessionId becomes a subagent of that\nsession and inherits its root, so one flow is one tree; without one it is\nitself a root. Registering with a terminal status records a session that has\nalready finished.",
|
||||
Description: "Opens a live agent session in the caller's org — the row every\nsurface (the CLI's outer agent, hanzo.bot, the console, chat) hangs its\nactivity off. A session with a parentSessionId becomes a subagent of that\nsession and inherits its root, so one flow is one tree; without one it is\nitself a root. Registering with a terminal status records a session that has\nalready finished.",
|
||||
Fields: map[string]string{
|
||||
"registerReq.host": "Execution context — where this session runs (all optional).",
|
||||
"registerReq.project": "The readable build (provenance.go): which product this session builds, and\nwhether its story may be read by the world.",
|
||||
@@ -230,8 +243,23 @@ func init() {
|
||||
},
|
||||
Example: json.RawMessage(`{"agent":"hanzo-dev","title":"ship the landing page","host":"gpu-01"}`),
|
||||
})
|
||||
zip.Describe("POST /v1/agents/sessions/:id/events", zip.Doc{
|
||||
Description: "Binds a Service-scoped handler to a route: it adapts a\n`func(*Service[S], *zip.Ctx) error` to the plain `func(*zip.Ctx) error` the\nrouter takes, capturing s. One adapter, so packages write free-function\nhandlers and register them with `app.Get(\"/path\", cloud.Handle(s, myHandler))`.",
|
||||
})
|
||||
zip.Describe("POST /v1/agents/sessions/:id/message", zip.Doc{
|
||||
Description: "Binds a Service-scoped handler to a route: it adapts a\n`func(*Service[S], *zip.Ctx) error` to the plain `func(*zip.Ctx) error` the\nrouter takes, capturing s. One adapter, so packages write free-function\nhandlers and register them with `app.Get(\"/path\", cloud.Handle(s, myHandler))`.",
|
||||
})
|
||||
zip.Describe("POST /v1/agents/sessions/:id/pause", zip.Doc{
|
||||
Description: "Binds a Service-scoped handler to a route: it adapts a\n`func(*Service[S], *zip.Ctx) error` to the plain `func(*zip.Ctx) error` the\nrouter takes, capturing s. One adapter, so packages write free-function\nhandlers and register them with `app.Get(\"/path\", cloud.Handle(s, myHandler))`.",
|
||||
})
|
||||
zip.Describe("POST /v1/agents/sessions/:id/resume", zip.Doc{
|
||||
Description: "Binds a Service-scoped handler to a route: it adapts a\n`func(*Service[S], *zip.Ctx) error` to the plain `func(*zip.Ctx) error` the\nrouter takes, capturing s. One adapter, so packages write free-function\nhandlers and register them with `app.Get(\"/path\", cloud.Handle(s, myHandler))`.",
|
||||
})
|
||||
zip.Describe("POST /v1/agents/sessions/:id/stop", zip.Doc{
|
||||
Description: "Binds a Service-scoped handler to a route: it adapts a\n`func(*Service[S], *zip.Ctx) error` to the plain `func(*zip.Ctx) error` the\nrouter takes, capturing s. One adapter, so packages write free-function\nhandlers and register them with `app.Get(\"/path\", cloud.Handle(s, myHandler))`.",
|
||||
})
|
||||
zip.Describe("POST /v1/agents/targets", zip.Doc{
|
||||
Description: "RegisterTarget registers a machine as an agent target, or re-links one that is\nalready registered. Re-linking is idempotent and keyed on org+host+owner, so a\nmachine that reconnects refreshes its own row rather than piling up duplicates;\nit answers 200, while a first registration answers 201.",
|
||||
Description: "Registers a machine as an agent target, or re-links one that is\nalready registered. Re-linking is idempotent and keyed on org+host+owner, so a\nmachine that reconnects refreshes its own row rather than piling up duplicates;\nit answers 200, while a first registration answers 201.",
|
||||
Fields: map[string]string{
|
||||
"GPU.memory": "VRAM bytes, 0 = unknown",
|
||||
"GPU.model": "\"GB10\", \"8060S\", \"RTX 4090\"",
|
||||
@@ -258,7 +286,7 @@ func init() {
|
||||
Example: json.RawMessage(`{"id":"tgt_1"}`),
|
||||
})
|
||||
zip.Describe("POST /v1/agents/targets/:id/key", zip.Doc{
|
||||
Description: "MintTargetClaimKey mints (or rotates) the claim key a `hanzo code --serve`\ndaemon presents to claim work for this machine, and returns it ONCE: only its\nSHA-256 hash is stored. Rotating supersedes any prior daemon, so only the\nmachine's owner — or an org admin — may call it; every other caller gets the\nsame not-found an unknown id gets, and learns nothing about what exists.",
|
||||
Description: "Mints (or rotates) the claim key a `hanzo code --serve`\ndaemon presents to claim work for this machine, and returns it ONCE: only its\nSHA-256 hash is stored. Rotating supersedes any prior daemon, so only the\nmachine's owner — or an org admin — may call it; every other caller gets the\nsame not-found an unknown id gets, and learns nothing about what exists.",
|
||||
Fields: map[string]string{
|
||||
"claimKeyOut.claimKey": "ClaimKey is the capability itself. It is returned ONCE and never again — only\nits SHA-256 hash is stored — so a daemon that loses it mints a new one.",
|
||||
"claimKeyOut.targetId": "TargetID is the machine the key authenticates.",
|
||||
@@ -267,7 +295,7 @@ func init() {
|
||||
Example: json.RawMessage(`{"id":"tgt_1"}`),
|
||||
})
|
||||
zip.Describe("POST /v1/agents/targets/:id/runs/:runId/report", zip.Doc{
|
||||
Description: "ReportRoutedRun completes a claimed run: it delivers the terminal result to the\nrun's durable owner, which is what lets that workflow finish. Scoped to (org,\ntarget, run) and claim-key authenticated, so a machine can only ever report a\nrun it legitimately holds. Idempotent — a report for an unknown or\nalready-finished run answers delivered:false rather than failing, because the\nsession's terminal state was already set by the machine's own stream.",
|
||||
Description: "Completes a claimed run: it delivers the terminal result to the\nrun's durable owner, which is what lets that workflow finish. Scoped to (org,\ntarget, run) and claim-key authenticated, so a machine can only ever report a\nrun it legitimately holds. Idempotent — a report for an unknown or\nalready-finished run answers delivered:false rather than failing, because the\nsession's terminal state was already set by the machine's own stream.",
|
||||
Fields: map[string]string{
|
||||
"reportOut.delivered": "Delivered is true when a waiting durable owner received this result. False\nmeans there was none to deliver to — an unknown or already-finished run — which\nis a clean no-op, not an error.",
|
||||
"reportRunIn.branch": "Branch, CommitSha and Diffstat describe what the run produced; Error is the\nfailure when OK is false. Each is clamped, never rejected.",
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
# Generated by plugin/gen-app-cmds. DO NOT EDIT.
|
||||
#
|
||||
# The build contract is mk/plugin.mk — one file carrying every target an app
|
||||
# needs: build, test, vet, openapi, clean. This names the app(s) this package
|
||||
# backs and includes it. Written from the same apps.Wire() parse that writes
|
||||
# plugin/<app>/main.go, so an app cannot have a main and no Makefile.
|
||||
APPS := risk
|
||||
include ../../mk/plugin.mk
|
||||
@@ -0,0 +1,230 @@
|
||||
package risk
|
||||
|
||||
// agency.go decides what KIND of actor a decision is about, from facts the
|
||||
// caller does not get to state.
|
||||
//
|
||||
// THIS IS THE DIFFERENTIATOR, SO IT HAS TO BE REAL. Telling a declared,
|
||||
// registered, metered agent from an anonymous script is the thing nobody who does
|
||||
// not run agents can compute — and it is worth exactly nothing if the answer is a
|
||||
// string off the request body. The first cut read `actor.agent != ""` and called
|
||||
// that "declared", while its own comment claimed a registry lookup: any caller
|
||||
// could type a name and be classified `agent`, and the `bot` lane was
|
||||
// structurally unreachable besides, because reaching any op here already requires
|
||||
// a validated principal.
|
||||
//
|
||||
// WHAT IS ACTUALLY KNOWN. The registry of a tenant's agents belongs to the agents
|
||||
// app, in another process, so it is asked over the internal plane
|
||||
// (plane.AgentsDeclared) with the caller's own identity — the org rides the
|
||||
// caller and can never be an argument, so a reference can only ever resolve
|
||||
// against the ASKING org's registry.
|
||||
//
|
||||
// resolves the org registered this agent → AgencyAgent
|
||||
// does not the caller claimed one that is not → AgencyBot
|
||||
// nothing named no claim to check → AgencyHuman or AgencyUnknown
|
||||
// cannot ask the registry is unreachable → AgencyUnknown + RefusalUnverified
|
||||
//
|
||||
// A CLAIM THAT DOES NOT RESOLVE IS THE STRONGEST BOT SIGNAL AVAILABLE, which is
|
||||
// why it is not merely downgraded to unknown: an actor asserting an agency the
|
||||
// registry can disprove has done something an honest one never does. That is also
|
||||
// what makes the bot lane reachable — it is entered by a fact, not by the absence
|
||||
// of a credential the surface already requires.
|
||||
//
|
||||
// WHAT BOUNDS THE LOOKUP, EXACTLY. It happens only when the caller CLAIMS an
|
||||
// agent, so the cost lands on the claimant, and every decide that reaches it is
|
||||
// itself metered on that caller's ledger — the registry call is one per PAID
|
||||
// decision, never an amplification of one. A repeated reference costs nothing
|
||||
// after the first: answers are memoised per tenant, negatives included, under
|
||||
// that tenant's own bound. A NOVEL reference on every request is still a call on
|
||||
// every request, which is why the call carries a deadline measured against an
|
||||
// authorization window: a payment decision waiting on a registry is a payment
|
||||
// plane that fails when the registry does.
|
||||
//
|
||||
// A reference longer than the registry can name is answered HERE, without the
|
||||
// call. agents bounds an agent label at 128 bytes and answers "not declared" to
|
||||
// anything longer, so asking is a round trip whose answer is already known — and
|
||||
// taking that answer as a verdict would classify a legitimately long name as a
|
||||
// bot on a fact about our own field width rather than about the caller.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/hanzoai/cloud"
|
||||
"github.com/hanzoai/cloud/plane"
|
||||
)
|
||||
|
||||
// agentsApp is the manifest name of the process holding the registry.
|
||||
const agentsApp = "agents"
|
||||
|
||||
// verifyBudget is how long a decision will wait for the registry. An
|
||||
// authorization window is tens of milliseconds of slack, not seconds; past this
|
||||
// the honest answer is that agency could not be verified, and that answer is
|
||||
// published rather than guessed.
|
||||
const verifyBudget = 250 * time.Millisecond
|
||||
|
||||
// The cache. Positive and negative answers are both kept, because a negative is
|
||||
// the answer a flood produces and caching only the positives would leave exactly
|
||||
// the abusive case uncached.
|
||||
const (
|
||||
// agencyTTL is how long a resolution is trusted. Short: an agent retired
|
||||
// this minute must stop counting as declared this minute.
|
||||
agencyTTL = time.Minute
|
||||
// agencyCacheMax bounds the per-tenant cache. It is per tenant like
|
||||
// everything else here, so a tenant inventing references evicts its OWN
|
||||
// oldest answers and nobody else's — and every entry is at most refMax
|
||||
// bytes, so the count is a byte figure (agencyMemo in bound.go) and not a
|
||||
// number over values the caller sizes.
|
||||
agencyCacheMax = 1024
|
||||
// refMax is the longest agent reference the registry can name: agents'
|
||||
// own maxAgentLabel (apps/agents/sessions.go). Stated here because it is
|
||||
// what makes a longer claim answerable without a call.
|
||||
refMax = 128
|
||||
)
|
||||
|
||||
// registry answers whether a reference names an agent in an org's own registry.
|
||||
// An interface with one method, so a test can state the registry's answer
|
||||
// without a second process — and so the production path has exactly one
|
||||
// implementation and no flag choosing between them.
|
||||
type registry interface {
|
||||
declared(ctx context.Context, ref string) (bool, error)
|
||||
}
|
||||
|
||||
// peerRegistry is the production implementation: the agents app, over the
|
||||
// internal plane, as the caller.
|
||||
type peerRegistry struct{}
|
||||
|
||||
func (peerRegistry) declared(ctx context.Context, ref string) (bool, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, verifyBudget)
|
||||
defer cancel()
|
||||
out, err := cloud.Ask[plane.AgentRef, plane.AgentDeclared](ctx, agentsApp, plane.AgentsDeclared,
|
||||
&plane.AgentRef{Ref: ref})
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return out.Declared, nil
|
||||
}
|
||||
|
||||
// agencyCache is one tenant's memo of registry answers.
|
||||
type agencyCache struct {
|
||||
mu sync.Mutex
|
||||
at map[string]agencyAnswer
|
||||
seq uint64
|
||||
}
|
||||
|
||||
type agencyAnswer struct {
|
||||
declared bool
|
||||
until time.Time
|
||||
// used orders eviction within this tenant's own cache.
|
||||
used uint64
|
||||
}
|
||||
|
||||
func (c *agencyCache) get(ref string, now time.Time) (bool, bool) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
a, ok := c.at[ref]
|
||||
if !ok || now.After(a.until) {
|
||||
return false, false
|
||||
}
|
||||
c.seq++
|
||||
a.used = c.seq
|
||||
c.at[ref] = a
|
||||
return a.declared, true
|
||||
}
|
||||
|
||||
func (c *agencyCache) put(ref string, declared bool, now time.Time) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if c.at == nil {
|
||||
c.at = make(map[string]agencyAnswer, 16)
|
||||
}
|
||||
if len(c.at) >= agencyCacheMax {
|
||||
c.evictLocked()
|
||||
}
|
||||
c.seq++
|
||||
c.at[ref] = agencyAnswer{declared: declared, until: now.Add(agencyTTL), used: c.seq}
|
||||
}
|
||||
|
||||
// clear forgets everything this tenant memoised. Called when its cell is
|
||||
// retired, so a retired tenant costs nothing at all — a bounded map left behind
|
||||
// by every tenant that ever passed through is still an unbounded process.
|
||||
func (c *agencyCache) clear() {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.at = nil
|
||||
}
|
||||
|
||||
// evictLocked drops this tenant's least recently used answer. Caller holds mu.
|
||||
func (c *agencyCache) evictLocked() {
|
||||
var oldest string
|
||||
var at uint64
|
||||
first := true
|
||||
for ref, a := range c.at {
|
||||
if first || a.used < at {
|
||||
oldest, at, first = ref, a.used, false
|
||||
}
|
||||
}
|
||||
if oldest != "" {
|
||||
delete(c.at, oldest)
|
||||
}
|
||||
}
|
||||
|
||||
// agencyOf classifies the actor a decision is about.
|
||||
//
|
||||
// It returns the agency AND a refusal, because "we could not check" is a third
|
||||
// answer and folding it into `unknown` would make an unreachable registry look
|
||||
// like an ordinary unclassified caller. A refusal here rides the decision, so a
|
||||
// reader can tell a classification from a gap in one.
|
||||
// user is the VALIDATED user id the scope carries — empty for a machine
|
||||
// credential. It is taken as a value rather than read off the request here,
|
||||
// because tenantOf already resolved the whole principal and two readers of the
|
||||
// raw request would be two places the identity rules live.
|
||||
func agencyOf(ctx context.Context, reg registry, cache *agencyCache, user string, obs observation) (string, string) {
|
||||
ref := strings.TrimSpace(obs.agent)
|
||||
if ref == "" {
|
||||
// No claim to check. A live session bound to a validated user is a
|
||||
// person; anything else is honestly unclassified, and an honest "we
|
||||
// cannot tell" is worth more than a guess in either direction.
|
||||
if user != "" && strings.TrimSpace(obs.session) != "" {
|
||||
return AgencyHuman, ""
|
||||
}
|
||||
return AgencyUnknown, ""
|
||||
}
|
||||
if len(ref) > refMax {
|
||||
// Longer than the registry can name, so there is no lookup to make and no
|
||||
// verdict to reach. Unknown + unverified is the honest answer: the claim
|
||||
// was not checked, and saying it was disproved would be a statement about
|
||||
// a field width rather than about the caller.
|
||||
return AgencyUnknown, RefusalUnverified
|
||||
}
|
||||
if reg == nil {
|
||||
return AgencyUnknown, RefusalUnverified
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
if declared, hit := cache.get(ref, now); hit {
|
||||
return classify(declared), ""
|
||||
}
|
||||
declared, err := reg.declared(ctx, ref)
|
||||
if err != nil {
|
||||
// The registry could not answer. NOT a bot and NOT an agent: the claim
|
||||
// is unchecked, and saying so is the only answer that does not silently
|
||||
// promote or demote whoever is asking.
|
||||
return AgencyUnknown, RefusalUnverified
|
||||
}
|
||||
cache.put(ref, declared, now)
|
||||
return classify(declared), ""
|
||||
}
|
||||
|
||||
// classify turns the registry's verdict into an agency.
|
||||
//
|
||||
// Two lanes from one verified fact, which is the whole vocabulary this can
|
||||
// honestly support: the org registered this agent, or it did not and something is
|
||||
// claiming otherwise.
|
||||
func classify(declared bool) string {
|
||||
if declared {
|
||||
return AgencyAgent
|
||||
}
|
||||
return AgencyBot
|
||||
}
|
||||
@@ -0,0 +1,236 @@
|
||||
package risk
|
||||
|
||||
// agency_test.go is the regression suite for the forgeable classification.
|
||||
//
|
||||
// THE DEFECT: `declared` was `actor.agent != ""`, so any caller could type a name
|
||||
// and be classified `agent`; the published field said the reference was resolved
|
||||
// in the org's own registry, and no such lookup existed. Worse, `bot` was
|
||||
// STRUCTURALLY UNREACHABLE — every op here requires a validated principal, so the
|
||||
// credential terms the bot lane turned on could never both hold.
|
||||
//
|
||||
// Each test below fails if either half comes back.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// stubRegistry answers for a fixed set of references, counts its calls, and can
|
||||
// be made unreachable — the three behaviours the classification turns on.
|
||||
type stubRegistry struct {
|
||||
mu sync.Mutex
|
||||
known map[string]bool
|
||||
down bool
|
||||
calls int
|
||||
}
|
||||
|
||||
func (s *stubRegistry) declared(_ context.Context, ref string) (bool, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.calls++
|
||||
if s.down {
|
||||
return false, fmt.Errorf("registry unreachable")
|
||||
}
|
||||
return s.known[ref], nil
|
||||
}
|
||||
|
||||
func (s *stubRegistry) count() int {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.calls
|
||||
}
|
||||
|
||||
// TestAgencyIsResolvedNotAsserted is the headline. A caller naming an agent this
|
||||
// org never registered must NOT be classified as an agent, and the answer must be
|
||||
// the strongest thing that fact supports: it claimed an agency we can disprove.
|
||||
func TestAgencyIsResolvedNotAsserted(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
reg := &stubRegistry{known: map[string]bool{"agent_real": true}}
|
||||
s.State.reg = reg
|
||||
|
||||
for _, tc := range []struct {
|
||||
name, actor, want, refusal string
|
||||
}{
|
||||
{"a reference this org registered", `"actor":{"agent":"agent_real"},`, AgencyAgent, ""},
|
||||
{"a reference it did not", `"actor":{"agent":"agent_invented"},`, AgencyBot, ""},
|
||||
{"a session and no agent", `"actor":{"session":"sess_1"},`, AgencyHuman, ""},
|
||||
{"nothing named at all", "", AgencyUnknown, ""},
|
||||
} {
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"signup","subject":{"kind":"account","id":"a1"},`+tc.actor+`"signals":{"ip":"203.0.113.1"}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("%s: decide = %d %s", tc.name, code, body)
|
||||
}
|
||||
var out riskDecision
|
||||
if err := json.Unmarshal(body, &out); err != nil {
|
||||
t.Fatalf("%s: %v", tc.name, err)
|
||||
}
|
||||
if out.Agency != tc.want {
|
||||
t.Errorf("%s: agency = %q, want %q", tc.name, out.Agency, tc.want)
|
||||
}
|
||||
if tc.refusal != "" && out.Refusal != tc.refusal {
|
||||
t.Errorf("%s: refusal = %q, want %q", tc.name, out.Refusal, tc.refusal)
|
||||
}
|
||||
}
|
||||
if reg.count() == 0 {
|
||||
t.Fatal("the registry was never asked — the classification is still reading the request body")
|
||||
}
|
||||
}
|
||||
|
||||
// TestTheBotLaneIsReachable pins the second half of the defect. A vocabulary term
|
||||
// no input can produce is not a classification, it is decoration — and this one
|
||||
// was the product's whole differentiator.
|
||||
func TestTheBotLaneIsReachable(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
s.State.reg = &stubRegistry{known: map[string]bool{}}
|
||||
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"payment","subject":{"kind":"transaction","id":"tx1"},"actor":{"agent":"scripted"}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("decide = %d %s", code, body)
|
||||
}
|
||||
var out riskDecision
|
||||
_ = json.Unmarshal(body, &out)
|
||||
if out.Agency != AgencyBot {
|
||||
t.Fatalf("agency = %q — undeclared automation cannot be reached, so the bad-bot lane classifies nobody", out.Agency)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAnUnreachableRegistryIsAGapAndSaysSo. Believing the caller when the check
|
||||
// could not be made is the forgery back again; silently answering `unknown` is the
|
||||
// same gap with the evidence removed. The honest answer names itself.
|
||||
func TestAnUnreachableRegistryIsAGapAndSaysSo(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
s.State.reg = &stubRegistry{known: map[string]bool{"agent_real": true}, down: true}
|
||||
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"signup","subject":{"kind":"account","id":"a1"},"actor":{"agent":"agent_real"}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("decide = %d %s", code, body)
|
||||
}
|
||||
var out riskDecision
|
||||
_ = json.Unmarshal(body, &out)
|
||||
if out.Agency == AgencyAgent {
|
||||
t.Fatal("an unreachable registry classified a claimed reference as an agent — the claim was believed")
|
||||
}
|
||||
if out.Agency != AgencyUnknown {
|
||||
t.Fatalf("agency = %q, want unknown when the registry could not answer", out.Agency)
|
||||
}
|
||||
if out.Refusal != RefusalUnverified {
|
||||
t.Fatalf("refusal = %q, want %q — an unchecked classification that does not say so is indistinguishable from a checked one",
|
||||
out.Refusal, RefusalUnverified)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTheLookupIsMemoisedPerTenantAndBounded pins the amplification bound. One
|
||||
// request must not become one internal call, or a caller with a loop turns a
|
||||
// decision into a fan-out; and the memo must be this tenant's own, with this
|
||||
// tenant's own eviction.
|
||||
func TestTheLookupIsMemoisedPerTenantAndBounded(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
reg := &stubRegistry{known: map[string]bool{"agent_real": true}}
|
||||
s.State.reg = reg
|
||||
|
||||
for i := 0; i < 5; i++ {
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"signup","subject":{"kind":"account","id":"a1"},"actor":{"agent":"agent_real"}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("decide = %d %s", code, body)
|
||||
}
|
||||
}
|
||||
if n := reg.count(); n != 1 {
|
||||
t.Fatalf("five decisions made %d registry calls — one request is one internal call, which is a fan-out a caller controls", n)
|
||||
}
|
||||
|
||||
// A negative is memoised too: the flood case is exactly the uncached one if
|
||||
// only the positives are kept.
|
||||
for i := 0; i < 3; i++ {
|
||||
req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"signup","subject":{"kind":"account","id":"a1"},"actor":{"agent":"nope"}}`)
|
||||
}
|
||||
if n := reg.count(); n != 2 {
|
||||
t.Fatalf("three decisions naming one unknown reference made %d total calls, want 2 — a negative is not memoised", n)
|
||||
}
|
||||
|
||||
// And the memo is bounded, by this tenant's own eviction.
|
||||
cache := &resOf(t, s, Tenant("hanzo/acme")).agency
|
||||
now := time.Now()
|
||||
for i := 0; i < agencyCacheMax*2; i++ {
|
||||
cache.put(fmt.Sprintf("ref-%d", i), false, now)
|
||||
}
|
||||
cache.mu.Lock()
|
||||
held := len(cache.at)
|
||||
cache.mu.Unlock()
|
||||
if held > agencyCacheMax {
|
||||
t.Fatalf("the agency memo holds %d entries against a bound of %d", held, agencyCacheMax)
|
||||
}
|
||||
}
|
||||
|
||||
// TestOneTenantsRegistryAnswerIsNotAnothers. The memo is per resident, so a
|
||||
// reference that resolves for one org can never resolve for another off the back
|
||||
// of it — which would be a cross-tenant read of the registry.
|
||||
func TestOneTenantsRegistryAnswerIsNotAnothers(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
// The registry answers for whoever asks; the ISOLATION being tested is that
|
||||
// A's cached "yes" is not readable by B, so B's own lookup decides.
|
||||
reg := &perOrgRegistry{yes: map[string]bool{"agent_a": true}}
|
||||
s.State.reg = reg
|
||||
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"signup","subject":{"kind":"account","id":"a1"},"actor":{"agent":"agent_a"}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("A decide = %d %s", code, body)
|
||||
}
|
||||
var mine riskDecision
|
||||
_ = json.Unmarshal(body, &mine)
|
||||
if mine.Agency != AgencyAgent {
|
||||
t.Fatalf("A's own agent classified %q", mine.Agency)
|
||||
}
|
||||
|
||||
// B names the SAME reference. It is not B's, so B must not inherit the answer.
|
||||
reg.yes = map[string]bool{} // whoever asks now, the answer is no
|
||||
code, body = req(t, app, http.MethodPost, "/v1/risk/decide", "beta", "u_beta",
|
||||
`{"stage":"signup","subject":{"kind":"account","id":"b1"},"actor":{"agent":"agent_a"}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("B decide = %d %s", code, body)
|
||||
}
|
||||
var theirs riskDecision
|
||||
_ = json.Unmarshal(body, &theirs)
|
||||
if theirs.Agency == AgencyAgent {
|
||||
t.Fatal("B read A's cached registry answer — one tenant's agent became another's")
|
||||
}
|
||||
}
|
||||
|
||||
type perOrgRegistry struct{ yes map[string]bool }
|
||||
|
||||
func (p *perOrgRegistry) declared(_ context.Context, ref string) (bool, error) {
|
||||
return p.yes[ref], nil
|
||||
}
|
||||
|
||||
// TestTheDocumentDoesNotClaimWhatTheCodeDoesNot. The published Agency field once
|
||||
// described a registry lookup that did not exist. Prose is the contract an SDK
|
||||
// user and a model read, so a claim in it is a claim we owe.
|
||||
func TestTheDocumentDoesNotClaimWhatTheCodeDoesNot(t *testing.T) {
|
||||
body, err := os.ReadFile("typed.go")
|
||||
if err != nil {
|
||||
t.Fatalf("read: %v", err)
|
||||
}
|
||||
src := string(body)
|
||||
// Whatever the field says, it must not promise facts nothing computes. These
|
||||
// two were in the old text and neither was ever derived.
|
||||
for _, gone := range []string{"the credential class", "the account's metered shape"} {
|
||||
if strings.Contains(src, gone) {
|
||||
t.Errorf("the published Agency description still promises %q, and nothing computes it", gone)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(src, "looked up in") {
|
||||
t.Error("the published Agency description no longer says the reference is looked up — the one thing that IS true of it")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,499 @@
|
||||
package risk
|
||||
|
||||
// bound.go is where every number that costs memory is decided, ONCE, and where
|
||||
// the only constructor for a tenant's aggregates lives.
|
||||
//
|
||||
// TWO DEFECT CLASSES THIS FILE EXISTS TO MAKE UNREPRESENTABLE.
|
||||
//
|
||||
// CLASS A — A BOUND ON A COUNT OF CALLER-SIZED VALUES IS NOT A BOUND. "100,000
|
||||
// keys", "20,000 list entries", "1,024 cached answers": every one of those is a
|
||||
// count over a value the CALLER sizes, so the byte figure an operator reads is
|
||||
// whatever the caller decides to make it. A 64 KiB subject id put one tenant at
|
||||
// 100 MiB against a published 8 MiB. The shape that cannot express it: ONE cap
|
||||
// on the LENGTH of any caller-supplied text (textMax), enforced at the wire door
|
||||
// (door.go) before the value reaches anything that keeps it — so count x textMax
|
||||
// IS the byte bound, and every count below is DERIVED from a byte budget rather
|
||||
// than chosen.
|
||||
//
|
||||
// CLASS B — ONE STORE SHARED BY EVERY TENANT WITH A GLOBAL CAP IS A CROSS-TENANT
|
||||
// EVICTOR. velocity.Store evicts the least-recently-updated key in a SHARD, and
|
||||
// shards are keyed by a hash that mixes tenants together, so one org filling the
|
||||
// store deletes ANOTHER org's counters — the victim's rules then read zero,
|
||||
// decline nothing, and report success. The shape that cannot express it: state
|
||||
// is PER TENANT with a PER-TENANT bound. aggregates() and forest() are the ONLY
|
||||
// constructors in this package, they are unexported, and they always apply the
|
||||
// bound (pinned by TestOnlyOneConstructorIsBounded).
|
||||
//
|
||||
// AND NOTHING DEGRADES QUIETLY. The engine's own eviction is invisible — no
|
||||
// counter, no callback, nothing a caller can read — so this app does not use it.
|
||||
// The store is built with a cardinality this app's own gate can never reach, and
|
||||
// the GATE is what refuses: exactly at the bound, counted, and published as
|
||||
// `strained` on the decision and on the probe. See rings.record.
|
||||
//
|
||||
// THE WORST CASE IS ARITHMETIC, NOT A HOPE:
|
||||
//
|
||||
// per key bucketBytes * buckets + keyOverhead + 3*textMax = 6,560 B
|
||||
// per tenant velBytes (aggregates) = 8 MiB → 1,278 keys
|
||||
// + cellBytes (forest + file + agency memo) = 784 KiB
|
||||
// + the governance cache at its byte budgets ≤ 3.25 MiB
|
||||
// ≤ 12 MiB
|
||||
// node memBytes, and a tenant is priced at what it HOLDS, not at its
|
||||
// ceiling: an ordinary tenant costs cellBytes, so 512 MiB serves
|
||||
// ~660 of them, or 42 simultaneously at their full ceiling.
|
||||
//
|
||||
// THE NODE BOUND IS BYTES, AND THAT IS THE WHOLE OPERATING POINT. An earlier cut
|
||||
// bounded the COUNT of resident tenants at 48 and priced every one of them at its
|
||||
// worst case, so the 49th concurrently-active org was refused the entire risk
|
||||
// surface — class A again, at the process layer. Bytes is the dimension that
|
||||
// matters, a tenant is charged for the counters it actually holds, and the
|
||||
// refusal is the LAST answer rather than the first: a node with no room reclaims
|
||||
// tenants that have been silent past the retire floor before it turns anyone
|
||||
// away, and both the reclaim and the refusal are counted where an operator reads
|
||||
// them.
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/aml/pkg/anomaly"
|
||||
"github.com/luxfi/aml/pkg/velocity"
|
||||
)
|
||||
|
||||
// ── the one cap on caller-supplied text ─────────────────────────────────────
|
||||
|
||||
// textMax is the longest any single text value this app is handed may be: a
|
||||
// subject id, a signal value, an agent reference, a list entry, a rule term, a
|
||||
// path segment, a query value. ONE number for all of them, enforced once at the
|
||||
// wire door, because a rule per field is a rule the next field will not have.
|
||||
//
|
||||
// 256 bytes is chosen from what an identifier has to be able to say: an RFC 5321
|
||||
// address is at most 254 octets, a UUID is 36, a ULID 26, an IPv6 literal 45, and
|
||||
// every payment-processor object id in circulation is well under a hundred. A
|
||||
// value longer than that is not naming a subject, and this app aggregates on it —
|
||||
// so accepting one is accepting an unpriced key.
|
||||
//
|
||||
// It is what turns every count below into a byte figure. Change it and every
|
||||
// derived cap moves with it, which is the point.
|
||||
const textMax = 256
|
||||
|
||||
// bodyMax is the largest request body this app accepts. The edge's own limit is
|
||||
// 16 MiB (GATEWAY_BODY_LIMIT) for the whole fleet; a decision is a few hundred
|
||||
// bytes and the largest batch any op here takes is 1,000 observations, so a
|
||||
// quarter of a megabyte per observation is already absurd. It bounds the
|
||||
// TRANSIENT cost of decoding, which the resident ceilings above do not cover:
|
||||
// without it, N connections each hold N x 16 MiB of half-decoded JSON.
|
||||
const bodyMax = 4 << 20
|
||||
|
||||
// ── the aggregates ──────────────────────────────────────────────────────────
|
||||
|
||||
// windows is the ONE declaration of the sliding aggregates this plane keeps.
|
||||
//
|
||||
// The NAMES and SPANS are the engine's standard set, because they are the rule
|
||||
// vocabulary: `velocity.<axis>.<window>.<stat>` is what a tenant's rules are
|
||||
// written against, the dictionary publishes them, and anomaly's inventory reads
|
||||
// them by name at construction.
|
||||
//
|
||||
// The RESOLUTIONS are this plane's own, and they are lower than the AML defaults
|
||||
// on purpose. Those were chosen for statutory structuring detection over a
|
||||
// compliance feed — 444 buckets per key, a measured 22.7 KiB — which puts ONE
|
||||
// tenant's plausible entity cardinality into gigabytes on a shared pod. Here the
|
||||
// same four windows cost 94 buckets, 4.4 KiB of rings per key, and the
|
||||
// quantisation is stated rather than assumed: a window of W with B buckets
|
||||
// resolves to W/B, so the boundary of the 1h window is exact to five minutes and
|
||||
// of the 30d window to a day. Every rule in the starter set compares a COUNT or a
|
||||
// SUM over a whole window, and none of them can distinguish a boundary that
|
||||
// fuzzy — precision that outruns the meaning of the number it measures is not
|
||||
// precision, it is four times the memory.
|
||||
//
|
||||
// An operator who wants the compliance resolution raises velBytes; the two are
|
||||
// the same knob seen from either end.
|
||||
func windows() []velocity.Window {
|
||||
return []velocity.Window{
|
||||
{Name: "1h", Span: time.Hour, Buckets: 12},
|
||||
{Name: "24h", Span: 24 * time.Hour, Buckets: 24},
|
||||
{Name: "7d", Span: 7 * 24 * time.Hour, Buckets: 28},
|
||||
{Name: "30d", Span: 30 * 24 * time.Hour, Buckets: 30},
|
||||
}
|
||||
}
|
||||
|
||||
// longestWindow is the span past which every ring of an untouched key reads
|
||||
// zero. A key idle for longer holds no information, so reclaiming it costs
|
||||
// nothing — which is the one reclaim that is free of any judgement call.
|
||||
func longestWindow() time.Duration {
|
||||
var d time.Duration
|
||||
for _, w := range windows() {
|
||||
if w.Span > d {
|
||||
d = w.Span
|
||||
}
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// bucketBytes is sizeof(velocity's ring bucket) on a 64-bit build: an int64
|
||||
// index, two ints, two float64s and a uint64 day mask. It is stated here rather
|
||||
// than measured at run time because a bound computed from the thing it bounds is
|
||||
// a bound that moves.
|
||||
const bucketBytes = 48
|
||||
|
||||
// keyOverhead is everything a key costs BESIDES its buckets and its text: four
|
||||
// ring headers, the pointer slice holding them, the entry, the engine's map slot
|
||||
// and this app's ledger slot, plus allocator rounding. Deliberately generous — a
|
||||
// bound that under-states is not a bound.
|
||||
const keyOverhead = 1280
|
||||
|
||||
// bytesPerKey is what one aggregated entity costs THIS tenant. Three text values
|
||||
// fit inside it: the engine keeps the composite id (org + kind + value) and this
|
||||
// app's admission ledger keeps its own (kind + value), so 3 x textMax covers both
|
||||
// with the axis name inside the slack.
|
||||
//
|
||||
// Derived from the windows and from textMax, so a change to either moves the key
|
||||
// count and can never leave a stale constant behind.
|
||||
func bytesPerKey() int {
|
||||
n := 0
|
||||
for _, w := range windows() {
|
||||
n += w.Buckets
|
||||
}
|
||||
return n*bucketBytes + keyOverhead + 3*textMax
|
||||
}
|
||||
|
||||
// The knobs. Each is an environment override over a documented default, because
|
||||
// the right ceiling is a property of the pod's memory limit and not of this
|
||||
// source file — and because an operator who has to patch a constant to survive a
|
||||
// capacity event will instead raise the constant to infinity.
|
||||
const (
|
||||
envVelBytes = "RISK_VELOCITY_BYTES" // per-tenant aggregate budget, bytes
|
||||
envMemory = "RISK_MEMORY" // the node's whole budget for risk state, bytes
|
||||
envIdle = "RISK_RECLAIM_IDLE" // how long a tenant must be silent before it is retired
|
||||
)
|
||||
|
||||
// velBytes is the per-tenant aggregate budget. 8 MiB is ~1,278 entities at this
|
||||
// plane's resolution: enough that a tenant's ACTIVE set fits, small enough that
|
||||
// a node holding a lot of them is still a node.
|
||||
func velBytes() int { return envInt(envVelBytes, 8<<20, 1<<20, 1<<30) }
|
||||
|
||||
// memBytes is the node's whole budget for resident risk state, and it is the
|
||||
// ONLY process bound. 512 MiB sits well inside cloud's 6 GiB request / 11 GiB
|
||||
// limit and leaves the rest of the binary — 130-odd other apps in the same
|
||||
// process — the room it had.
|
||||
//
|
||||
// A pod serving more than this is a SHARDING answer, not a bigger-number answer:
|
||||
// the shard router pins an org to one pod, so capacity scales by adding writers.
|
||||
// Raising the knob past what the pod's memory limit supports trades a loud
|
||||
// refusal for an OOM kill, which takes every tenant down.
|
||||
func memBytes() int { return envInt(envMemory, 512<<20, cellBytes, 32<<30) }
|
||||
|
||||
// idleReclaim is how long a tenant must send NOTHING before the background sweep
|
||||
// retires it. Six hours: long enough that a bursty tenant keeps its 24h and 7d
|
||||
// counts across a quiet afternoon, short enough that memory comes back the same
|
||||
// day. The trigger is the tenant's own silence and nothing else.
|
||||
//
|
||||
// THE FLOOR IS LOAD-BEARING, not caution. Retiring a tenant CLOSES its file, and
|
||||
// the one thing that holds that file outside a request is a search worker
|
||||
// (searchBudget). A floor comfortably above that budget is what makes the close
|
||||
// safe without a reference count on every op: a cell can only be retired when no
|
||||
// request has resolved it for idleFloor, and nothing in this process can hold a
|
||||
// handle that long. TestARetirementCannotRaceAWorker pins it.
|
||||
//
|
||||
// The floor is ALSO what a node under memory pressure reclaims at: pressure may
|
||||
// take a cell that has been silent that long and no other, so a tenant that is
|
||||
// using the node can never lose its rings to a tenant that is arriving.
|
||||
func idleReclaim() time.Duration {
|
||||
d := time.Duration(envInt(envIdle, int(6*time.Hour/time.Second), 60, int(400*24*time.Hour/time.Second))) * time.Second
|
||||
if d < idleFloor {
|
||||
return idleFloor
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// idleFloor is the shortest idleness that may retire a tenant: eight times the
|
||||
// longest a search may hold a tenant's file.
|
||||
const idleFloor = 8 * searchBudget
|
||||
|
||||
// maxKeys is the per-tenant cardinality bound, computed from the budget rather
|
||||
// than chosen. At least one, so a misconfigured budget degrades to a tiny store
|
||||
// rather than to velocity's own 100,000-key default.
|
||||
func maxKeys() int {
|
||||
n := velBytes() / bytesPerKey()
|
||||
if n < 1 {
|
||||
return 1
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// envInt reads a bounded integer override. Out of range or unparseable is the
|
||||
// default, not a failure: a typo in a capacity knob must not stop a payment
|
||||
// plane from booting, and the value it lands on is the documented one.
|
||||
func envInt(name string, def, lo, hi int) int {
|
||||
v, err := strconv.Atoi(os.Getenv(name))
|
||||
if err != nil || v < lo || v > hi {
|
||||
return def
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// ── one tenant's aggregates, and the gate that bounds them ──────────────────
|
||||
|
||||
// rings is ONE TENANT's sliding aggregates together with the ledger that bounds
|
||||
// them. The two travel as one value because a store without its gate is exactly
|
||||
// the shape both defect classes had: bounded by a number nobody can read, and
|
||||
// evicting silently when it binds.
|
||||
//
|
||||
// THE LEDGER IS WHAT MAKES `strained` EXACT. velocity offers no membership test,
|
||||
// so without it the app cannot tell a new key from one it already holds and
|
||||
// cannot refuse at the bound — which is how the engine's per-SHARD eviction
|
||||
// (MaxKeys/64+1) came to fire at ~72% of the nominal bound while the tenant was
|
||||
// told nothing. The ledger costs one map entry per key and it is PRICED: see
|
||||
// bytesPerKey.
|
||||
type rings struct {
|
||||
mu sync.Mutex
|
||||
vel *velocity.Store
|
||||
seen map[string]struct{}
|
||||
// held and dropped are the two numbers ANYONE may read: what this tenant is
|
||||
// priced at, and whether it is strained. They are atomics rather than fields
|
||||
// under mu because the node's memory gate is asked from INSIDE admit — so a
|
||||
// reclaim triggered by one tenant's new key prices another tenant's cell, and
|
||||
// pricing it through mu would re-enter this lock from underneath itself. One
|
||||
// writer (admit, under mu) keeps them exact.
|
||||
held atomic.Int64
|
||||
dropped atomic.Int64
|
||||
}
|
||||
|
||||
// record writes one observation onto every ring it has a value for, through the
|
||||
// gate. The rings are keyed {OrgID, Kind, Value} with the tenant leading, so two
|
||||
// tenants naming the same device or the same address never share a counter.
|
||||
//
|
||||
// A key the gate refuses is COUNTED and nothing else happens to it: the tenant's
|
||||
// existing counters keep moving (refusing those too would turn a cardinality
|
||||
// bound into a total stop), and `strained` becomes true for as long as this arm
|
||||
// lasts, on the decision and on the probe.
|
||||
//
|
||||
// room is the node's own gate — nil when the caller is not accounting, which is
|
||||
// what lets a search sandbox replay a tenant's history under the same per-tenant
|
||||
// bound without charging the node twice for counters it already holds.
|
||||
func (g *rings) record(t Tenant, o observation, room func(int) bool) {
|
||||
usd := nanoUSD(o.amount)
|
||||
for axis := range velocityAxes {
|
||||
v := o.axisOf(axis)
|
||||
if v == "" {
|
||||
continue
|
||||
}
|
||||
if !g.admit(axis, v, room) {
|
||||
continue
|
||||
}
|
||||
g.vel.Record(velocity.Key{OrgID: t.String(), Kind: axis, Value: v}, o.at, usd, structuringThreshold)
|
||||
}
|
||||
}
|
||||
|
||||
// admit reports whether this key may be written, reserving its cost when it is
|
||||
// new. A key already in the ledger is always admitted: it costs nothing more.
|
||||
func (g *rings) admit(axis, value string, room func(int) bool) bool {
|
||||
id := axis + "\x00" + value
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
if _, have := g.seen[id]; have {
|
||||
return true
|
||||
}
|
||||
if len(g.seen) >= maxKeys() {
|
||||
g.dropped.Add(1)
|
||||
return false
|
||||
}
|
||||
if room != nil && !room(bytesPerKey()) {
|
||||
g.dropped.Add(1)
|
||||
return false
|
||||
}
|
||||
g.seen[id] = struct{}{}
|
||||
g.held.Store(int64(len(g.seen)))
|
||||
return true
|
||||
}
|
||||
|
||||
// observe reads one key's windows. It is the only read path, so the ledger and
|
||||
// the store can never be consulted through different doors.
|
||||
func (g *rings) observe(t Tenant, axis, value string) []velocity.Observation {
|
||||
return g.vel.Observe(velocity.Key{OrgID: t.String(), Kind: axis, Value: value})
|
||||
}
|
||||
|
||||
// keys is how many distinct entities this tenant's rings hold. Exact, because
|
||||
// nothing inside the store evicts.
|
||||
func (g *rings) keys() int { return int(g.held.Load()) }
|
||||
|
||||
// stored is how many keys the ENGINE holds, asked of the engine.
|
||||
//
|
||||
// It exists because `keys` cannot answer the question the margin above is bought
|
||||
// to answer. The ledger counts what the GATE admitted and knows nothing about
|
||||
// what the store did with it, so an assertion on `keys` is true whether or not
|
||||
// the engine evicted — which is a test that cannot fail for the thing it names.
|
||||
// The two numbers agree exactly when nothing inside the store has evicted, and
|
||||
// that agreement is the property, so it has to be readable from both sides.
|
||||
func (g *rings) stored() int { return g.vel.Keys() }
|
||||
|
||||
// strained reports that this tenant's rings stopped tracking its own traffic.
|
||||
//
|
||||
// It is published rather than logged because the consumer is the TENANT: a rule
|
||||
// that fires on `velocity.ip.1h.count >= 5` stops firing on a key that was never
|
||||
// admitted, and there is no other way for the tenant to learn that its threshold
|
||||
// is being measured against a partial ring.
|
||||
func (g *rings) strained() bool { return g.missed() > 0 }
|
||||
|
||||
// missed is how many distinct keys this tenant's rings could not take. On the
|
||||
// probe, so an operator sees the size of the gap and not only its existence.
|
||||
func (g *rings) missed() int64 { return g.dropped.Load() }
|
||||
|
||||
// aggregates builds ONE TENANT's rings.
|
||||
//
|
||||
// It is the only call to velocity.New in this package and it is unexported, so
|
||||
// `velocity.New` with a zero Config — a 100,000-key store shared by everyone —
|
||||
// is not a thing a future edit can reach for by accident.
|
||||
//
|
||||
// MaxKeys is set BEYOND ANYTHING THE GATE ADMITS, and that is deliberate. The
|
||||
// engine's cap is enforced per shard at MaxKeys/shardCount+1 and its eviction is
|
||||
// silent; multiplying by the shard count puts it out of reach even if every key
|
||||
// this tenant is allowed hashed into one shard, so the only thing that can ever
|
||||
// refuse a key is this app's own gate, which says so. MaxKeys costs nothing —
|
||||
// the store allocates per key recorded, never per key allowed.
|
||||
func aggregates() *rings {
|
||||
return &rings{
|
||||
vel: velocity.New(velocity.Config{Windows: windows(), MaxKeys: maxKeys()*velShards + velShards}),
|
||||
seen: make(map[string]struct{}, 64),
|
||||
}
|
||||
}
|
||||
|
||||
// velShards is the number of shards velocity spreads keys over (its own
|
||||
// shardCount, pkg/velocity/velocity.go). Stated here because the multiplication
|
||||
// above is what makes the engine's silent eviction unreachable, and a change
|
||||
// upstream has to be met here. TestNothingIsEvictedInsideATenantsAggregates
|
||||
// measures it rather than trusting it.
|
||||
const velShards = 64
|
||||
|
||||
// forest builds ONE TENANT's half-space forest over its own rings. It is the
|
||||
// only call to anomaly.New in this package, and it OVERRIDES two fields of
|
||||
// whatever config it is handed, which is why the live plane and the search
|
||||
// sandbox can share it.
|
||||
//
|
||||
// MaxOrgs is forced to 1 and that is the whole point. anomaly.Store holds a map
|
||||
// of tenants with a GLOBAL LRU over it, so a shared store lets one tenant's
|
||||
// arrival delete another's learned model — and an evicted model returns to
|
||||
// warming, which declines to score and reads as clean. One tenant per store makes
|
||||
// that map hold exactly one key, so the eviction path is unreachable rather than
|
||||
// merely unlikely.
|
||||
//
|
||||
// Shadow is forced on: it is the deployment-wide default, and the tenant's own
|
||||
// live switch in its record plane is the second gate. Two in series, so a flag
|
||||
// flipped by mistake still cannot make a tenant act, and a search sandbox can
|
||||
// never alert for real.
|
||||
func forest(cfg anomaly.Config, g *rings) (*anomaly.Store, error) {
|
||||
cfg.MaxOrgs, cfg.Shadow = 1, true
|
||||
m, err := anomaly.New(cfg, g.vel)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("risk: %w", err)
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
// ── what an armed cell costs before it has counted anything ─────────────────
|
||||
|
||||
// The fixed half of a resident, reserved at admission. Everything here is
|
||||
// bounded by a constant rather than by the caller, so reserving it is exact;
|
||||
// the caller-scaled half (the rings, and the governance cache below) is priced
|
||||
// live as it lands. That split is the operating point: an ordinary tenant costs
|
||||
// cellBytes and nothing else, so a node serves hundreds of them.
|
||||
const (
|
||||
// forestBytes is luxfi/aml's own measured figure for one half-space forest
|
||||
// at this app's geometry.
|
||||
forestBytes = 336 << 10
|
||||
// fileBytes is the tenant's open SQLite handle, its connection pool and the
|
||||
// cell's own structs.
|
||||
fileBytes = 64 << 10
|
||||
// agencyEntry is one memoised registry answer: the reference, the verdict,
|
||||
// its expiry and the map slot.
|
||||
agencyEntry = textMax + 128
|
||||
// agencyMemo is the whole per-tenant agency cache. Bounded by the entry
|
||||
// count TIMES the one text cap, which is what makes it a byte figure.
|
||||
agencyMemo = agencyCacheMax * agencyEntry
|
||||
// cellBytes is what one armed cell costs before it counts anything.
|
||||
cellBytes = forestBytes + fileBytes + agencyMemo
|
||||
)
|
||||
|
||||
// ── the governance bounds ───────────────────────────────────────────────────
|
||||
//
|
||||
// Every one of these is a BYTE budget, and the row count is DERIVED from it —
|
||||
// the other way round is class A. Each is enforced at WRITE time and INSIDE the
|
||||
// transaction that does the write, which is what makes it a bound rather than a
|
||||
// truncation and what stops two concurrent writes from both seeing room. The
|
||||
// matching LIMIT on the read is defence in depth for rows that predate a cap.
|
||||
|
||||
const (
|
||||
// ruleMax is the largest a rule's stored form may be. A rule is an id, a
|
||||
// name and a handful of terms; 4 KiB holds sixteen terms at textMax each,
|
||||
// which is far past anything readable.
|
||||
ruleMax = 4 << 10
|
||||
// ruleBudget is the per-tenant budget for rules. They are ALL evaluated on
|
||||
// every decision, so this is a latency bound as much as a memory one.
|
||||
ruleBudget = 1 << 20
|
||||
// supMax is the largest a suppression's stored form may be: seven text
|
||||
// fields at textMax each, with room over.
|
||||
supMax = 2 << 10
|
||||
// supBudget is the per-tenant budget for suppressions. Each one is matched
|
||||
// against every hit.
|
||||
supBudget = 512 << 10
|
||||
// entryMax is what one list entry costs in the map the authorization path
|
||||
// loads it into: the value at its cap plus the map slot.
|
||||
entryMax = textMax + 128
|
||||
// listBudget is the per-tenant budget for list membership across ALL of a
|
||||
// tenant's lists — the map is one map, so a per-list cap would be a bound on
|
||||
// nothing, reachable by creating more lists.
|
||||
listBudget = 2 << 20
|
||||
// controlMax and controlBudget bound the controls plane. Controls are not on
|
||||
// the decision path — the money plane reads them — so this is a disk figure
|
||||
// rather than a resident one, and the budget is larger.
|
||||
controlMax = 2 << 10
|
||||
controlBudget = 4 << 20
|
||||
// recordMax is the largest one decision row may be: the subject, the stage,
|
||||
// the signals map and the evidence, all at textMax.
|
||||
recordMax = 16 << 10
|
||||
// recordBudget is how much of the shared volume ONE tenant's decision log
|
||||
// may hold. THIS IS THE SAME CLASS ONE LAYER DOWN: every tenant's SQLite file
|
||||
// lives on the pod's one DataDir volume, so a log that only ever grows is
|
||||
// "one org quiets another" moved from RAM to disk — a few thousand calls
|
||||
// fill the volume and every tenant's writes start failing. 256 MiB is a
|
||||
// long history for a tenant and a small share of a pod volume.
|
||||
recordBudget = 256 << 20
|
||||
// pruneEvery is how many writes pass between prunes. Counting the log on
|
||||
// every decision would be a table scan inside an authorization window, so
|
||||
// the overshoot is bounded at this many rows instead — and the log is also
|
||||
// pruned once when the tenant's file is opened, which catches a tenant that
|
||||
// writes fewer than this between rollouts.
|
||||
pruneEvery = 256
|
||||
)
|
||||
|
||||
// recordCap is how many decisions one tenant's log retains. Derived from the
|
||||
// budget, like every other count here.
|
||||
func recordCap() int { return recordBudget / recordMax }
|
||||
|
||||
// governMemo is the whole per-tenant governance cache at its budgets. Published
|
||||
// so the per-tenant ceiling is arithmetic; priced LIVE rather than reserved,
|
||||
// because governance is loaded on the authorization path and refusing it would
|
||||
// disarm the tenant's controls — the one degradation this package will not do.
|
||||
const governMemo = ruleBudget + supBudget + listBudget
|
||||
|
||||
// The derived row counts. Each is budget / largest-row, so the count and the
|
||||
// byte figure can never disagree.
|
||||
func ruleCap() int { return ruleBudget / ruleMax }
|
||||
func supCap() int { return supBudget / supMax }
|
||||
func listCap() int { return listBudget / entryMax }
|
||||
func controlCap() int { return controlBudget / controlMax }
|
||||
|
||||
// errCap is what a write past a budget answers. It names the budget and the
|
||||
// number, so the tenant can act on it rather than guess.
|
||||
func errCap(what string, cap int) error {
|
||||
return fmt.Errorf("this tenant already holds the maximum of %d %s; retire one before adding another", cap, what)
|
||||
}
|
||||
|
||||
// errLong is what the wire door answers for a value it cannot price. Stated
|
||||
// here, beside the cap it enforces, so the number and its refusal are one edit.
|
||||
func errLong(what string) error {
|
||||
return fmt.Errorf("%s is longer than %d bytes, which is the most this plane will aggregate on or store", what, textMax)
|
||||
}
|
||||
@@ -0,0 +1,286 @@
|
||||
package risk
|
||||
|
||||
// bound_test.go is the regression suite for the two defects that held this app:
|
||||
// a process-wide velocity store whose eviction crossed tenants, and a per-key
|
||||
// cost of ~22 KB against no per-tenant ceiling at all.
|
||||
//
|
||||
// Each test was written by reintroducing the defect and checking that THIS test
|
||||
// goes red.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TestOneTenantCannotEvictAnothersAggregates is the load-bearing one.
|
||||
//
|
||||
// THE DEFECT: one velocity.Store for the whole process, MaxKeys 100,000 over 64
|
||||
// shards, keys hashed across those shards without regard to tenant. Tenant A
|
||||
// pushing distinct subjects evicted tenant B's counters, so B's velocity rules
|
||||
// read zero, fired on nothing, and reported success. No error, no log, no alert.
|
||||
//
|
||||
// THE TEST: give a tenant the smallest bound the knob allows, drive A far past
|
||||
// it — so A's own store DEMONSTRABLY evicts — and require that B's single key
|
||||
// still reads every observation B made. With a shared store B's key is gone.
|
||||
func TestOneTenantCannotEvictAnothersAggregates(t *testing.T) {
|
||||
t.Setenv(envVelBytes, "1048576") // 1 MiB: the floor, so the bound is reachable in a test
|
||||
_, s := wireApp(t)
|
||||
|
||||
a, b := Tenant("hanzo/acme"), Tenant("hanzo/beta")
|
||||
bees := resOf(t, s, b)
|
||||
bvel, _, _ := bees.arms()
|
||||
|
||||
// B records five observations on ONE subject.
|
||||
const bCount = 5
|
||||
for i := 0; i < bCount; i++ {
|
||||
bees.record(observation{
|
||||
at: time.Now(), kind: "account", subject: "b-account",
|
||||
amount: 1_000_000_000, signals: map[string]string{"ip": "198.51.100.7"},
|
||||
})
|
||||
}
|
||||
|
||||
// A floods, well past its OWN bound.
|
||||
ay := resOf(t, s, a)
|
||||
avel, _, _ := ay.arms()
|
||||
flood := maxKeys() * 3
|
||||
for i := 0; i < flood; i++ {
|
||||
ay.record(observation{
|
||||
at: time.Now(), kind: "account", subject: fmt.Sprintf("a-account-%d", i),
|
||||
amount: 1_000_000_000,
|
||||
})
|
||||
}
|
||||
|
||||
// A really did hit its own bound — otherwise this test proves nothing,
|
||||
// because nothing was ever under pressure. And it hit it AT the bound: the
|
||||
// gate refuses, the engine never evicts.
|
||||
if got := avel.keys(); got != maxKeys() {
|
||||
t.Fatalf("the flooding tenant holds %d keys against a bound of %d — the gate is not what bound it", got, maxKeys())
|
||||
}
|
||||
if !ay.strained() {
|
||||
t.Fatal("the flooding tenant is past its own bound and does not say so")
|
||||
}
|
||||
|
||||
// B is untouched. This is the property.
|
||||
obs := bvel.observe(b, "account", "b-account")
|
||||
var got int
|
||||
for _, o := range obs {
|
||||
if o.Window == "24h" {
|
||||
got = o.Count
|
||||
}
|
||||
}
|
||||
if got != bCount {
|
||||
t.Fatalf("B's 24h count is %d, want %d — another tenant's volume evicted B's counters, and B's velocity rules now fire on nothing", got, bCount)
|
||||
}
|
||||
if avel == bvel {
|
||||
t.Fatal("two tenants share one velocity store — the eviction boundary is a hash, not a tenant")
|
||||
}
|
||||
}
|
||||
|
||||
// TestTheWorstCaseIsArithmeticAndConservative pins the memory bound as something
|
||||
// computed rather than hoped for, and proves the computation is an OVER-estimate
|
||||
// by measuring the real thing.
|
||||
//
|
||||
// A ceiling that under-states is not a ceiling. The formula must be at least as
|
||||
// large as the bytes a key actually costs, or the per-tenant budget buys more
|
||||
// keys than it can hold.
|
||||
func TestTheWorstCaseIsArithmeticAndConservative(t *testing.T) {
|
||||
// The formula is the buckets actually configured, not a constant left behind
|
||||
// by an earlier window set.
|
||||
buckets := 0
|
||||
for _, w := range windows() {
|
||||
buckets += w.Buckets
|
||||
}
|
||||
if want := buckets*bucketBytes + keyOverhead + 3*textMax; bytesPerKey() != want {
|
||||
t.Fatalf("bytesPerKey() = %d, want %d — the estimate has drifted from the windows and the text cap it is computed over", bytesPerKey(), want)
|
||||
}
|
||||
// And the budget really does bound the key count.
|
||||
if maxKeys()*bytesPerKey() > velBytes() {
|
||||
t.Fatalf("%d keys x %d B = %d B exceeds the %d B budget", maxKeys(), bytesPerKey(), maxKeys()*bytesPerKey(), velBytes())
|
||||
}
|
||||
|
||||
// MEASURED. Fill one tenant's rings with distinct keys and weigh it. An
|
||||
// ORDINARY identifier here; the WORST case the door admits is measured by
|
||||
// TestThePublishedCeilingHoldsForTheLongestValueTheDoorAccepts, which is the
|
||||
// one that matters and the one this test used to be missing.
|
||||
tn := Tenant("hanzo/acme")
|
||||
n := maxKeys()
|
||||
runtime.GC()
|
||||
var before, after runtime.MemStats
|
||||
runtime.ReadMemStats(&before)
|
||||
vel := aggregates()
|
||||
at := time.Now()
|
||||
for i := 0; i < n; i++ {
|
||||
vel.record(tn, observation{at: at, kind: "account", subject: fmt.Sprintf("s-%d", i), amount: 1_000_000_000}, nil)
|
||||
}
|
||||
runtime.GC()
|
||||
runtime.ReadMemStats(&after)
|
||||
runtime.KeepAlive(vel)
|
||||
|
||||
measured := int(after.HeapAlloc-before.HeapAlloc) / n
|
||||
if measured > bytesPerKey() {
|
||||
t.Fatalf("a key measures %d B against a published ceiling of %d B — the per-tenant budget buys more keys than it can hold", measured, bytesPerKey())
|
||||
}
|
||||
t.Logf("per key: measured %d B, published ceiling %d B; per tenant %d keys in %d B; node budget %d B",
|
||||
measured, bytesPerKey(), maxKeys(), velBytes(), memBytes())
|
||||
}
|
||||
|
||||
// TestTheDefaultCeilingIsWhatIsDocumented pins the published numbers. The whole
|
||||
// point of a computable worst case is that it is written down somewhere an
|
||||
// operator reads, so a change to a default has to be a change to the document.
|
||||
func TestTheDefaultCeilingIsWhatIsDocumented(t *testing.T) {
|
||||
for _, k := range []string{envVelBytes, envMemory, envIdle} {
|
||||
t.Setenv(k, "")
|
||||
}
|
||||
if got := velBytes(); got != 8<<20 {
|
||||
t.Errorf("default per-tenant aggregate budget = %d, want 8 MiB", got)
|
||||
}
|
||||
if got := memBytes(); got != 512<<20 {
|
||||
t.Errorf("default node budget = %d, want 512 MiB", got)
|
||||
}
|
||||
if got := idleReclaim(); got != 6*time.Hour {
|
||||
t.Errorf("default reclaim idleness = %s, want 6h", got)
|
||||
}
|
||||
// The per-tenant CEILING, stated, and the node budget it is charged against.
|
||||
// If either number moves, the comment at the top of bound.go must move with
|
||||
// it. Note what this is NOT: a count of tenants. A tenant is charged for what
|
||||
// it holds, so the node serves memBytes/cellBytes ordinary tenants and
|
||||
// memBytes/perTenant simultaneously at their ceiling.
|
||||
perTenant := cellBytes + velBytes() + governMemo
|
||||
if perTenant > 16<<20 {
|
||||
t.Fatalf("one tenant may hold %d B (%d MiB) — beyond what a shared pod may promise any single org", perTenant, perTenant>>20)
|
||||
}
|
||||
if memBytes() < 8*perTenant {
|
||||
t.Fatalf("the node budget (%d B) is under eight tenants at their ceiling (%d B) — a node that cannot hold a handful of busy orgs is not an operating point", memBytes(), perTenant)
|
||||
}
|
||||
t.Logf("per tenant ceiling %d B; node %d B = %d ordinary cells or %d at their ceiling",
|
||||
perTenant, memBytes(), memBytes()/cellBytes, memBytes()/perTenant)
|
||||
}
|
||||
|
||||
// TestARetirementCannotRaceAWorker pins the invariant that lets a retire CLOSE a
|
||||
// tenant's file without a reference count on every op.
|
||||
//
|
||||
// Retirement is safe only because a cell reaches it having served no request for
|
||||
// idleReclaim, and nothing in this process can hold a tenant's handle that long:
|
||||
// the longest is a search worker, bounded by searchBudget. If someone lowers the
|
||||
// floor under that budget — or raises the budget over the floor — the close
|
||||
// starts racing a live worker, and the failure is a write error on a durable
|
||||
// report rather than anything this suite would otherwise notice.
|
||||
func TestARetirementCannotRaceAWorker(t *testing.T) {
|
||||
if idleFloor <= searchBudget {
|
||||
t.Fatalf("the retire floor (%s) is not above the longest a worker holds a tenant's file (%s) — closing it races that worker", idleFloor, searchBudget)
|
||||
}
|
||||
// And the floor really is a floor: no environment value can go under it.
|
||||
for _, v := range []string{"60", "1", "0", "-5", "600"} {
|
||||
t.Setenv(envIdle, v)
|
||||
if got := idleReclaim(); got < idleFloor {
|
||||
t.Fatalf("%s=%s yields %s, under the %s floor", envIdle, v, got, idleFloor)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestOnlyOneConstructorIsBounded is the structural half: it is not enough that
|
||||
// today's call sites are bounded, it must be hard to add an unbounded one.
|
||||
//
|
||||
// velocity.New with a zero Config is a 100,000-key store, and anomaly.New with a
|
||||
// zero Config holds 256 tenants under a global LRU. Both are exactly the shape
|
||||
// the defect had. This asserts that neither is spelled anywhere in the package
|
||||
// except inside the two constructors that force the bound.
|
||||
func TestOnlyOneConstructorIsBounded(t *testing.T) {
|
||||
for _, call := range []struct{ pkg, fn, allowedIn string }{
|
||||
{"velocity", "New", "bound.go"},
|
||||
{"anomaly", "New", "bound.go"},
|
||||
} {
|
||||
for file, n := range callsIn(t, call.pkg, call.fn) {
|
||||
if file != call.allowedIn {
|
||||
t.Errorf("%s calls %s.%s %d time(s); the only bounded constructor is in %s — an unbounded store shared across tenants is the defect this package was held for",
|
||||
file, call.pkg, call.fn, n, call.allowedIn)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// callsIn counts <pkg>.<fn>( per file across the package's non-test sources.
|
||||
func callsIn(t *testing.T, pkg, fn string) map[string]int {
|
||||
t.Helper()
|
||||
out := map[string]int{}
|
||||
fset := token.NewFileSet()
|
||||
pkgs, err := parser.ParseDir(fset, ".", func(fi os.FileInfo) bool {
|
||||
return !strings.HasSuffix(fi.Name(), "_test.go")
|
||||
}, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("parse: %v", err)
|
||||
}
|
||||
for _, p := range pkgs {
|
||||
for name, f := range p.Files {
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
call, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
sel, ok := call.Fun.(*ast.SelectorExpr)
|
||||
if !ok || sel.Sel.Name != fn {
|
||||
return true
|
||||
}
|
||||
id, ok := sel.X.(*ast.Ident)
|
||||
if !ok || id.Name != pkg {
|
||||
return true
|
||||
}
|
||||
out[filepath.Base(name)]++
|
||||
return true
|
||||
})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// TestAStrainedTenantSaysSo pins the loud half of the cardinality bound. A
|
||||
// tenant that has hit its own bound is reading partial rings, and a partial ring
|
||||
// under-counts, which is the failure mode that reads as a clean result.
|
||||
func TestAStrainedTenantSaysSo(t *testing.T) {
|
||||
t.Setenv(envVelBytes, "1048576")
|
||||
app, s := wireApp(t)
|
||||
|
||||
// Not strained to begin with, or the assertion below proves nothing.
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"signup","subject":{"kind":"account","id":"first"}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("decide = %d %s", code, body)
|
||||
}
|
||||
var first riskDecision
|
||||
_ = json.Unmarshal(body, &first)
|
||||
if first.Strained {
|
||||
t.Fatal("a fresh tenant reports its aggregates as strained")
|
||||
}
|
||||
if first.Since == "" {
|
||||
t.Fatal("a decision does not publish the period its aggregates cover — a 30-day count over ten minutes of rings reads as a 30-day fact")
|
||||
}
|
||||
|
||||
// Fill this tenant's own store past its own bound.
|
||||
acme := resOf(t, s, Tenant("hanzo/acme"))
|
||||
for i := 0; i < maxKeys()*2; i++ {
|
||||
acme.record(observation{
|
||||
at: time.Now(), kind: "account", subject: fmt.Sprintf("filler-%d", i), amount: 1,
|
||||
})
|
||||
}
|
||||
|
||||
code, body = req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"signup","subject":{"kind":"account","id":"later"}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("decide = %d %s", code, body)
|
||||
}
|
||||
var later riskDecision
|
||||
_ = json.Unmarshal(body, &later)
|
||||
if !later.Strained {
|
||||
t.Fatal("a tenant at its own cardinality bound does not say so, so an under-counted velocity reads as a clean one")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,414 @@
|
||||
package risk
|
||||
|
||||
// decide.go is the hot path: one call, one decision, an answer that says what to
|
||||
// do and why.
|
||||
//
|
||||
// ORDER, and why it is this order:
|
||||
//
|
||||
// 1. RECORD the observation into the in-memory velocity rings. Everything after
|
||||
// reads them, and the numbers quoted in the decision must be the numbers an
|
||||
// investigator sees when they look at the subject.
|
||||
// 2. CLASSIFY agency — agent, human, bot or unknown — from facts only we hold.
|
||||
// 3. SCORE the model. It learns online and its evidence is capped at review.
|
||||
// 4. EVALUATE the rules over the recorded aggregates plus the model's score.
|
||||
// 5. SUPPRESS. A suppressed hit is RECORDED with suppressed=true, never dropped:
|
||||
// silence must never read as a clean result.
|
||||
// 6. COMBINE into one action, and never let statistical evidence exceed the
|
||||
// ceiling.
|
||||
//
|
||||
// NOTHING HERE READS THE WAREHOUSE. The rings are in memory, the rules and lists
|
||||
// are the tenant's own SQLite file, the model is in process. That is deliberate:
|
||||
// the warehouse is one pod and this call sits inside a card processor's
|
||||
// authorization window.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"math"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/aml/pkg/anomaly"
|
||||
"github.com/luxfi/aml/pkg/types"
|
||||
)
|
||||
|
||||
// Agency is what kind of actor this is. It is the differentiator: we run the
|
||||
// agents, so we can ask the org's OWN registry whether the agent a decision
|
||||
// names is one it registered — a fact nobody who does not run agents holds. See
|
||||
// agency.go, which is where it is derived and where the registry is asked.
|
||||
const (
|
||||
// AgencyAgent is an agent the org registered: the reference on the
|
||||
// observation RESOLVED in this org's own agent registry.
|
||||
AgencyAgent = "agent"
|
||||
// AgencyHuman is a live session bound to a validated user, naming no agent.
|
||||
AgencyHuman = "human"
|
||||
// AgencyBot is undeclared automation: something named an agent reference
|
||||
// this org's registry does not know. A claim the registry can disprove is
|
||||
// the strongest signal available here.
|
||||
AgencyBot = "bot"
|
||||
// AgencyUnknown names nothing to check, or nothing checkable. Scored
|
||||
// normally — an honest "we cannot tell" is worth more than a guess in either
|
||||
// direction.
|
||||
AgencyUnknown = "unknown"
|
||||
)
|
||||
|
||||
// Refusals. A refusal is COUNTED and NAMED, never silent, because a control that
|
||||
// declined to answer and a control that found nothing are the same bytes on the
|
||||
// wire and opposite facts about the system.
|
||||
const (
|
||||
// RefusalWarming means the model has not learned enough of this tenant's
|
||||
// behaviour for its scores to mean anything. Rules still ran.
|
||||
RefusalWarming = "warming"
|
||||
// RefusalDisarmed means this tenant HAD learned state and this process does
|
||||
// not have it: the snapshot was unreadable, or the engine refused it. It is
|
||||
// warming's opposite, not its synonym — warming is a control coming up, and
|
||||
// this is a control that is off. Named separately because reporting a
|
||||
// disarmed model as "warming" is exactly how one stays off unnoticed.
|
||||
RefusalDisarmed = "disarmed"
|
||||
// RefusalUnverified means the actor's claimed agency could not be checked
|
||||
// against the org's registry, because the registry could not be reached. The
|
||||
// classification on this decision is a gap, not a verdict.
|
||||
RefusalUnverified = "unverified"
|
||||
// RefusalUnidentified means the observation named no subject the aggregates
|
||||
// can be keyed on.
|
||||
RefusalUnidentified = "unidentified"
|
||||
// RefusalShadow means the tenant is in shadow: everything was computed and
|
||||
// recorded, and nothing acts.
|
||||
RefusalShadow = "shadow"
|
||||
)
|
||||
|
||||
// hit is one piece of evidence.
|
||||
type hit struct {
|
||||
// Rule is the identifier of the rule or model that produced this evidence.
|
||||
Rule string `json:"rule"`
|
||||
// Name is the human-readable detection name.
|
||||
Name string `json:"name"`
|
||||
// Action is what this evidence alone asks for.
|
||||
Action string `json:"action"`
|
||||
// Weight is how much this evidence contributes, in [0,1].
|
||||
Weight float64 `json:"weight"`
|
||||
// Severity is the reviewer-facing grading.
|
||||
Severity string `json:"severity,omitempty"`
|
||||
// Suppressed marks evidence a tenant suppression muted. It still contributes
|
||||
// nothing to the action and is still recorded, because a muted control that
|
||||
// leaves no trace is indistinguishable from one that was never running.
|
||||
Suppressed bool `json:"suppressed,omitempty"`
|
||||
}
|
||||
|
||||
// observation is everything the decide path was given plus everything it derived
|
||||
// before scoring. It is the value the rules read and the value the record plane
|
||||
// stores, so what fired and what was stored cannot disagree.
|
||||
type observation struct {
|
||||
id string
|
||||
at time.Time
|
||||
stage string
|
||||
kind string
|
||||
subject string
|
||||
agency string
|
||||
amount int64
|
||||
currency string
|
||||
direction string
|
||||
signals map[string]string
|
||||
agent string
|
||||
session string
|
||||
// refusal is what was already short BEFORE scoring — today, an agency the
|
||||
// registry could not confirm. It rides the observation rather than being a
|
||||
// parameter of its own because it is derived from the observation, in the
|
||||
// same call that derives agency.
|
||||
refusal string
|
||||
}
|
||||
|
||||
// outcome is the decision, before it is rendered onto the wire.
|
||||
type outcome struct {
|
||||
id string
|
||||
action string
|
||||
score float64
|
||||
agency string
|
||||
hits []hit
|
||||
causes []types.Cause
|
||||
shadow bool
|
||||
refusal string
|
||||
}
|
||||
|
||||
// axisOf maps a velocity axis to the observation's value on that axis. It is a
|
||||
// closed map because a rule may only name an axis this returns something for,
|
||||
// and admitTerm holds it to the same set.
|
||||
func (o observation) axisOf(axis string) string {
|
||||
switch axis {
|
||||
case "account":
|
||||
return o.subject
|
||||
case "device":
|
||||
return o.signals["device"]
|
||||
case "ip":
|
||||
return o.signals["ip"]
|
||||
case "email":
|
||||
return o.signals["email"]
|
||||
case "bin":
|
||||
return o.signals["bin"]
|
||||
case "pair":
|
||||
if cp := o.signals["counterparty"]; cp != "" {
|
||||
return o.subject + "\x1f" + cp
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// structuringThreshold is the reporting threshold the "just under" counters are
|
||||
// measured against — the standard USD 10,000 figure. It is a constant because a
|
||||
// per-tenant threshold is a per-tenant policy and this is the statutory one.
|
||||
const structuringThreshold = 10_000
|
||||
|
||||
// nanoUSD converts the wire's integer nano-units to the float the aggregates and
|
||||
// the model read. Money is carried as an integer on the wire so no rounding
|
||||
// happens between the caller and the ledger; the aggregate is a statistic and a
|
||||
// float is the right shape for it.
|
||||
func nanoUSD(nano int64) float64 { return float64(nano) / 1e9 }
|
||||
|
||||
// observe builds the fact set the rules read. Every velocity number the closed
|
||||
// vocabulary can name is materialised here, once, so a rule set of any size
|
||||
// reads the rings a bounded number of times.
|
||||
func observe(vel *rings, t Tenant, o observation, modelScore float64, warming bool) factSet {
|
||||
f := factSet{
|
||||
scalar: map[string]string{
|
||||
"stage": o.stage,
|
||||
"subject.kind": o.kind,
|
||||
"subject.id": o.subject,
|
||||
"agency": o.agency,
|
||||
"amount.currency": o.currency,
|
||||
"amount.direction": o.direction,
|
||||
"actor.agent": o.agent,
|
||||
"actor.session": o.session,
|
||||
"model.warming": strconv.FormatBool(warming),
|
||||
},
|
||||
number: map[string]float64{
|
||||
"amount.nano": float64(o.amount),
|
||||
"model.score": modelScore,
|
||||
},
|
||||
}
|
||||
for k, v := range o.signals {
|
||||
f.scalar["signal."+strings.ToLower(k)] = v
|
||||
}
|
||||
for axis := range velocityAxes {
|
||||
v := o.axisOf(axis)
|
||||
if v == "" {
|
||||
continue
|
||||
}
|
||||
for _, obs := range vel.observe(t, axis, v) {
|
||||
p := "velocity." + axis + "." + obs.Window + "."
|
||||
f.number[p+"count"] = float64(obs.Count)
|
||||
f.number[p+"sum"] = obs.Sum
|
||||
f.number[p+"near"] = float64(obs.Near)
|
||||
f.number[p+"days"] = float64(obs.Days)
|
||||
}
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
// bench is everything a decision is made AGAINST: the tenant's two in-memory
|
||||
// planes, its governance, and how to grade the model's own refusal.
|
||||
//
|
||||
// It is a VALUE, so decide can be exercised without a store — which is what
|
||||
// makes the ordering above testable rather than assertable — and it is ONE
|
||||
// value, so the next thing a decision must read arrives as a named field rather
|
||||
// than as a tenth positional argument nobody reads at the call site.
|
||||
type bench struct {
|
||||
// t is the tenant. It leads every velocity key and every model entity, so it
|
||||
// is what makes two tenants' counters disjoint.
|
||||
t Tenant
|
||||
// vel and model are THIS tenant's own aggregates and forest. Nothing here is
|
||||
// shared with another tenant; see bound.go.
|
||||
vel *rings
|
||||
model *anomaly.Store
|
||||
// rules is this tenant's rule set, already loaded.
|
||||
rules []rule
|
||||
// lists answers whether a value is in one of this tenant's named lists.
|
||||
lists func(name, value string) bool
|
||||
// mute answers whether a suppression covers this hit. A muted hit is still
|
||||
// recorded; see step 5.
|
||||
mute func(h hit, o observation) bool
|
||||
// shadow is the tenant observing rather than acting.
|
||||
shadow bool
|
||||
// room is the node's memory gate, asked before a NEW counter is taken. Nil
|
||||
// when nothing is accounting — a bench built by a test, or a search sandbox.
|
||||
room func(int) bool
|
||||
// grade turns the model's own refusal into the word that is true of it —
|
||||
// `warming` for a control coming up, `disarmed` for one that is off. Nil
|
||||
// grades nothing, which is what a test without a store wants.
|
||||
grade func(reason string) string
|
||||
}
|
||||
|
||||
// gradeOf applies the bench's grader, if it has one.
|
||||
func (b bench) gradeOf(reason string) string {
|
||||
if b.grade == nil {
|
||||
return reason
|
||||
}
|
||||
return b.grade(reason)
|
||||
}
|
||||
|
||||
// decide is the whole path.
|
||||
func decide(ctx context.Context, b bench, o observation) outcome {
|
||||
_ = ctx
|
||||
|
||||
out := outcome{id: o.id, shadow: b.shadow, agency: o.agency, action: ActionAllow, refusal: o.refusal}
|
||||
if strings.TrimSpace(o.subject) == "" {
|
||||
out.refusal = RefusalUnidentified
|
||||
return out
|
||||
}
|
||||
|
||||
// 1. Record first: everything after reads these rings, and the numbers in the
|
||||
// decision must be the ones an investigator sees on the subject. It goes
|
||||
// through the tenant's own gate, so a key the bound refuses is counted and
|
||||
// published as `strained` rather than silently dropped inside the engine.
|
||||
b.vel.record(b.t, o, b.room)
|
||||
|
||||
// 2. Score the model. Assess LEARNS; the tenant's own traffic is its training
|
||||
// set and there is no separate training pass.
|
||||
tx := types.Transaction{
|
||||
ID: o.id,
|
||||
OrgID: b.t.String(),
|
||||
UserID: o.subject,
|
||||
AccountID: o.subject,
|
||||
Currency: o.currency,
|
||||
Direction: o.direction,
|
||||
IPAddress: o.signals["ip"],
|
||||
DeviceFingerprint: o.signals["device"],
|
||||
Counterparty: o.signals["counterparty"],
|
||||
Timestamp: o.at,
|
||||
USD: nanoUSD(o.amount),
|
||||
}
|
||||
entity := types.Entity{ID: o.subject, OrgID: b.t.String()}
|
||||
assessment := b.model.Inspect(tx, entity)
|
||||
var modelHit *hit
|
||||
if mh, ok := b.model.Assess(tx, entity); ok {
|
||||
action := mh.Rule.Action
|
||||
if actionRank(action) > actionRank(modelCeiling) {
|
||||
action = modelCeiling
|
||||
}
|
||||
modelHit = &hit{
|
||||
Rule: mh.Rule.ID, Name: mh.Rule.Name, Action: action,
|
||||
Weight: mh.Rule.Weight, Severity: mh.Rule.Severity,
|
||||
}
|
||||
out.causes = mh.Causes
|
||||
}
|
||||
// GRADED HERE, where the model's own reason is produced and still the only
|
||||
// thing being named. Grading the merged word instead would let any
|
||||
// higher-ranked reason hide a model that is off.
|
||||
if !assessment.Scored && assessment.Reason != "" {
|
||||
out.refusal = worse(out.refusal, b.gradeOf(assessment.Reason))
|
||||
}
|
||||
|
||||
// 3. Rules, over the recorded aggregates plus the model's score.
|
||||
f := observe(b.vel, b.t, o, assessment.Score, !assessment.Scored)
|
||||
f.lists = b.lists
|
||||
hits := evaluate(b.rules, o.stage, f)
|
||||
if modelHit != nil {
|
||||
hits = append(hits, *modelHit)
|
||||
}
|
||||
|
||||
// 4. Suppression. A suppressed hit is kept, marked and given zero weight —
|
||||
// it appears in the record and contributes nothing to the action.
|
||||
scoring := make([]hit, 0, len(hits))
|
||||
for i := range hits {
|
||||
if b.mute != nil && b.mute(hits[i], o) {
|
||||
hits[i].Suppressed = true
|
||||
continue
|
||||
}
|
||||
scoring = append(scoring, hits[i])
|
||||
}
|
||||
|
||||
// 5. Combine. Sorting again puts the model's hit in its place among the rest.
|
||||
sort.SliceStable(hits, func(i, j int) bool {
|
||||
if a, b := actionRank(hits[i].Action), actionRank(hits[j].Action); a != b {
|
||||
return a > b
|
||||
}
|
||||
return hits[i].Weight > hits[j].Weight
|
||||
})
|
||||
score, action := combine(scoring)
|
||||
out.hits, out.score = hits, round4(score)
|
||||
|
||||
// 6. Shadow. Everything above ran, everything is recorded, nothing acts.
|
||||
if b.shadow {
|
||||
out.action = ActionAllow
|
||||
out.refusal = worse(out.refusal, RefusalShadow)
|
||||
return out
|
||||
}
|
||||
out.action = action
|
||||
return out
|
||||
}
|
||||
|
||||
// worse keeps the refusal a reader most needs to see.
|
||||
//
|
||||
// A decision can be short of more than one thing at once and the wire carries ONE
|
||||
// word, so the word is chosen by a stated precedence rather than by whichever
|
||||
// line happened to run last.
|
||||
//
|
||||
// THE ORDER IS "WHAT WILL NOT FIX ITSELF", most consequential first:
|
||||
//
|
||||
// unidentified the observation named nobody, so there was nothing to key on.
|
||||
// disarmed a control that WAS on is off. Nothing this tenant does next
|
||||
// turns it back on; an operator has to.
|
||||
// unverified a check that was supposed to happen did not, because another
|
||||
// process could not answer. Also nobody's own traffic to fix.
|
||||
// unusable the engine says its own output is not usable yet.
|
||||
// warming the model is coming up. ORDINARY — it is the state of every new
|
||||
// tenant, and the tenant's own next requests resolve it.
|
||||
// shadow the tenant chose to observe. Not a shortfall at all, and also
|
||||
// published as its own boolean.
|
||||
//
|
||||
// warming ranking BELOW unverified is deliberate and was a defect the other way
|
||||
// round: warming is the common case, so letting it win meant a registry outage
|
||||
// was reported as a model that is merely young, on the majority of decisions.
|
||||
//
|
||||
// A refusal this table does not name still outranks silence: an unnamed reason
|
||||
// is one the engine added and this file has not met, and dropping it would be
|
||||
// exactly the silent gap the whole vocabulary exists to prevent.
|
||||
func worse(a, b string) string {
|
||||
switch {
|
||||
case a == "":
|
||||
return b
|
||||
case b == "":
|
||||
return a
|
||||
case rank(b) > rank(a):
|
||||
return b
|
||||
default:
|
||||
return a
|
||||
}
|
||||
}
|
||||
|
||||
func rank(refusal string) int {
|
||||
if r, named := refusalRank[refusal]; named {
|
||||
return r
|
||||
}
|
||||
return refusalRank[anomaly.ReasonUnusable]
|
||||
}
|
||||
|
||||
var refusalRank = map[string]int{
|
||||
RefusalShadow: 1,
|
||||
RefusalWarming: 2,
|
||||
anomaly.ReasonUnusable: 3,
|
||||
RefusalUnverified: 4,
|
||||
RefusalDisarmed: 5,
|
||||
RefusalUnidentified: 6,
|
||||
}
|
||||
|
||||
// round4 trims a score to four places. A score is a judgement, not a
|
||||
// measurement, and rendering seventeen digits of float noise invites a client to
|
||||
// compare two scores that differ in the fifteenth.
|
||||
func round4(f float64) float64 {
|
||||
if math.IsNaN(f) || math.IsInf(f, 0) {
|
||||
return 0
|
||||
}
|
||||
return math.Round(f*1e4) / 1e4
|
||||
}
|
||||
|
||||
// newID mints a decision identifier. Random rather than sequential: a sequential
|
||||
// id over a shared surface is a volume oracle — a tenant can count another
|
||||
// tenant's decisions by watching its own ids skip.
|
||||
func newID(prefix string) string {
|
||||
var b [12]byte
|
||||
_, _ = rand.Read(b[:])
|
||||
return prefix + "_" + hex.EncodeToString(b[:])
|
||||
}
|
||||
@@ -0,0 +1,454 @@
|
||||
package risk
|
||||
|
||||
// decide_test.go pins the decision path's ORDER and its two hard limits: the
|
||||
// model may never act alone, and shadow may never act at all.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/aml/pkg/anomaly"
|
||||
)
|
||||
|
||||
// TestRecordHappensBeforeScoring pins step 1. Everything after reads the rings,
|
||||
// so the numbers quoted in a decision must be the ones an investigator sees when
|
||||
// they look at the subject — including THIS observation.
|
||||
func TestRecordHappensBeforeScoring(t *testing.T) {
|
||||
vel := aggregates()
|
||||
model, err := forest(anomaly.Config{}, vel)
|
||||
if err != nil {
|
||||
t.Fatalf("anomaly.New: %v", err)
|
||||
}
|
||||
tn := Tenant("hanzo/acme")
|
||||
|
||||
// A rule that can only hold if THIS observation is already in the ring.
|
||||
rules := []rule{{
|
||||
ID: "r1", Name: "counted", Stage: StagePayment, Action: ActionReview,
|
||||
Weight: 0.5, Enabled: true,
|
||||
All: []term{{Field: "velocity.ip.1h.count", Op: OpGte, Number: 1}},
|
||||
}}
|
||||
o := observation{
|
||||
id: "d1", at: time.Now(), stage: StagePayment, kind: "transaction", subject: "tx1",
|
||||
amount: 1_000_000_000, currency: "USD", direction: "in",
|
||||
signals: map[string]string{"ip": "203.0.113.1"},
|
||||
}
|
||||
out := decide(context.Background(), bench{t: tn, vel: vel, model: model, rules: rules}, o)
|
||||
if len(out.hits) != 1 {
|
||||
t.Fatalf("hits = %v; the rule did not see this observation in the ring, so the record ran after the read", out.hits)
|
||||
}
|
||||
}
|
||||
|
||||
// TestModelEvidenceCannotExceedTheCeiling pins the structural backstop. A
|
||||
// statistical judgement may summon a person; it may not decline a payment,
|
||||
// because an unexplainable refusal is not a decision anybody can defend to a
|
||||
// customer or a chargeback network.
|
||||
func TestModelEvidenceCannotExceedTheCeiling(t *testing.T) {
|
||||
if actionRank(modelCeiling) != actionRank(ActionReview) {
|
||||
t.Fatalf("the model ceiling is %q, not review — statistical evidence can now act alone", modelCeiling)
|
||||
}
|
||||
// And the combine step's ordering must never promote a capped hit.
|
||||
hits := []hit{{Rule: "model", Action: modelCeiling, Weight: 1.0}}
|
||||
_, action := combine(hits)
|
||||
if actionRank(action) > actionRank(ActionReview) {
|
||||
t.Fatalf("a weight-1.0 model hit produced %q — the cap is not holding", action)
|
||||
}
|
||||
}
|
||||
|
||||
// TestShadowActsOnNothing pins the default. In shadow every rule runs, the model
|
||||
// scores and learns, every decision is recorded, and the action is always allow.
|
||||
func TestShadowActsOnNothing(t *testing.T) {
|
||||
vel := aggregates()
|
||||
model, _ := forest(anomaly.Config{}, vel)
|
||||
rules := []rule{{
|
||||
ID: "block-all", Name: "would block", Stage: StagePayment, Action: ActionBlock,
|
||||
Weight: 1, Enabled: true,
|
||||
All: []term{{Field: "subject.kind", Op: OpEq, Value: "transaction"}},
|
||||
}}
|
||||
o := observation{
|
||||
id: "d1", at: time.Now(), stage: StagePayment, kind: "transaction", subject: "tx1",
|
||||
signals: map[string]string{},
|
||||
}
|
||||
out := decide(context.Background(), bench{t: Tenant("hanzo/acme"), vel: vel, model: model, rules: rules, shadow: true}, o)
|
||||
if out.action != ActionAllow {
|
||||
t.Fatalf("shadow produced action %q — a shadow tenant acted", out.action)
|
||||
}
|
||||
// The answer must SAY it was shadow, or silence reads as a clean result. The
|
||||
// refusal keeps the MODEL's more specific reason when it has one (a warming
|
||||
// model is a different fact from a shadow tenant, and hiding the first behind
|
||||
// the second is exactly the conflation this field exists to prevent) — so
|
||||
// `shadow` is the flag that must always hold, and `refusal` must always be
|
||||
// populated with something.
|
||||
if !out.shadow {
|
||||
t.Fatal("a shadow decision does not say it was shadow")
|
||||
}
|
||||
if out.refusal == "" {
|
||||
t.Fatal("a shadow decision names no refusal at all, so it is indistinguishable from a clean live one")
|
||||
}
|
||||
if out.refusal != RefusalWarming && out.refusal != RefusalShadow {
|
||||
t.Fatalf("refusal = %q, want the model's own reason or shadow", out.refusal)
|
||||
}
|
||||
if len(out.hits) != 1 {
|
||||
t.Fatalf("shadow computed %d hits, want 1 — shadow must compute everything and act on nothing", len(out.hits))
|
||||
}
|
||||
// Live, the same input blocks. Without this the test above would pass on a
|
||||
// rule that simply never fires.
|
||||
out = decide(context.Background(), bench{t: Tenant("hanzo/acme"), vel: vel, model: model, rules: rules}, o)
|
||||
if out.action != ActionBlock {
|
||||
t.Fatalf("live produced %q, want block — the shadow assertion proves nothing if the rule cannot fire", out.action)
|
||||
}
|
||||
}
|
||||
|
||||
// TestADisarmedModelIsNamedEvenWhenSomethingElseIsAlsoShort.
|
||||
//
|
||||
// THE DEFECT: the model's refusal was graded AFTER every reason had been merged
|
||||
// into the single word the wire carries. Grading only fired on the literal word
|
||||
// `warming`, so a decision that was ALSO short of something else — an agency the
|
||||
// registry could not confirm, a tenant in shadow — went out under that other
|
||||
// reason's name and the disarmed model was never said out loud. A grader that
|
||||
// stops working as soon as anything else goes wrong is a grader that stops
|
||||
// working exactly when a reader most needs it.
|
||||
//
|
||||
// THE FIX IS ORDER: the model's own reason is graded where it is produced, before
|
||||
// the merge, so the merge chooses between TRUE words.
|
||||
func TestADisarmedModelIsNamedEvenWhenSomethingElseIsAlsoShort(t *testing.T) {
|
||||
vel := aggregates()
|
||||
model, err := forest(anomaly.Config{}, vel)
|
||||
if err != nil {
|
||||
t.Fatalf("forest: %v", err)
|
||||
}
|
||||
o := observation{
|
||||
id: "d1", at: time.Now(), stage: StageSignup, kind: "account", subject: "a1",
|
||||
signals: map[string]string{},
|
||||
// The agency claim could not be checked — a second, independently true
|
||||
// shortfall that outranks warming.
|
||||
refusal: RefusalUnverified,
|
||||
}
|
||||
// A grader standing in for a tenant whose learned state is gone.
|
||||
disarm := func(reason string) string {
|
||||
if reason == RefusalWarming {
|
||||
return RefusalDisarmed
|
||||
}
|
||||
return reason
|
||||
}
|
||||
|
||||
out := decide(context.Background(), bench{
|
||||
t: Tenant("hanzo/acme"), vel: vel, model: model, grade: disarm,
|
||||
}, o)
|
||||
if out.refusal != RefusalDisarmed {
|
||||
t.Fatalf("refusal = %q, want %q — a control that is OFF was hidden behind another reason, which is the silent disarm in one word",
|
||||
out.refusal, RefusalDisarmed)
|
||||
}
|
||||
|
||||
// The control: with no grader the same decision reports the reason the engine
|
||||
// actually gave, so the assertion above is about grading and not about rank.
|
||||
plain := decide(context.Background(), bench{
|
||||
t: Tenant("hanzo/acme"), vel: vel, model: model,
|
||||
}, o)
|
||||
if plain.refusal == RefusalDisarmed {
|
||||
t.Fatal("an ungraded decision reported disarmed, so the test above proves nothing")
|
||||
}
|
||||
if plain.refusal != RefusalUnverified {
|
||||
t.Fatalf("ungraded refusal = %q, want %q — an unchecked agency must outrank a model that is merely young", plain.refusal, RefusalUnverified)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSuppressedEvidenceIsRecordedNotDropped pins the doctrine: a muted control
|
||||
// that leaves no trace is indistinguishable from one that was never running.
|
||||
func TestSuppressedEvidenceIsRecordedNotDropped(t *testing.T) {
|
||||
vel := aggregates()
|
||||
model, _ := forest(anomaly.Config{}, vel)
|
||||
rules := []rule{{
|
||||
ID: "r1", Name: "noisy", Stage: StageSignup, Action: ActionBlock, Weight: 1, Enabled: true,
|
||||
All: []term{{Field: "subject.kind", Op: OpEq, Value: "account"}},
|
||||
}}
|
||||
o := observation{id: "d1", at: time.Now(), stage: StageSignup, kind: "account", subject: "a1", signals: map[string]string{}}
|
||||
|
||||
out := decide(context.Background(), bench{
|
||||
t: Tenant("hanzo/acme"), vel: vel, model: model, rules: rules,
|
||||
mute: func(h hit, _ observation) bool { return h.Rule == "r1" },
|
||||
}, o)
|
||||
|
||||
if len(out.hits) != 1 {
|
||||
t.Fatalf("the suppressed hit was DROPPED (%d hits) — the record no longer says the rule fired", len(out.hits))
|
||||
}
|
||||
if !out.hits[0].Suppressed {
|
||||
t.Fatal("the hit is recorded but not marked suppressed")
|
||||
}
|
||||
if out.action != ActionAllow {
|
||||
t.Fatalf("a suppressed hit still drove the action to %q", out.action)
|
||||
}
|
||||
if out.score != 0 {
|
||||
t.Fatalf("a suppressed hit contributed %v to the score", out.score)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRuleAdmissionRefusesWhatWouldReadAsWorking pins the admission gate. Each
|
||||
// of these produces a rule that looks live and detects nothing, or one that
|
||||
// detects everything.
|
||||
func TestRuleAdmissionRefusesWhatWouldReadAsWorking(t *testing.T) {
|
||||
base := rule{Name: "r", Action: ActionBlock, Weight: 0.5, Enabled: true,
|
||||
All: []term{{Field: "signal.ip", Op: OpEq, Value: "x"}}}
|
||||
ok := base
|
||||
if err := admit(ok); err != nil {
|
||||
t.Fatalf("a well-formed rule was refused: %v", err)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
mut func(*rule)
|
||||
}{
|
||||
{"no terms holds on everything", func(r *rule) { r.All = nil }},
|
||||
{"no name cannot be read back in an alert", func(r *rule) { r.Name = "" }},
|
||||
{"an unknown action", func(r *rule) { r.Action = "quarantine" }},
|
||||
{"a weight outside [0,1]", func(r *rule) { r.Weight = 2 }},
|
||||
{"an unknown stage", func(r *rule) { r.Stage = "checkout" }},
|
||||
{"a field the vocabulary does not carry", func(r *rule) { r.All = []term{{Field: "tx.amount", Op: OpEq}} }},
|
||||
{"an unknown operator", func(r *rule) { r.All = []term{{Field: "signal.ip", Op: "matches"}} }},
|
||||
{"an unknown velocity axis", func(r *rule) { r.All = []term{{Field: "velocity.wallet.1h.count", Op: OpGte}} }},
|
||||
{"an unknown velocity window", func(r *rule) { r.All = []term{{Field: "velocity.ip.90d.count", Op: OpGte}} }},
|
||||
{"an empty in-set holds on nothing", func(r *rule) { r.All = []term{{Field: "agency", Op: OpIn}} }},
|
||||
{"an inlist naming no list", func(r *rule) { r.All = []term{{Field: "signal.ip", Op: OpInList}} }},
|
||||
} {
|
||||
r := base
|
||||
r.All = append([]term(nil), base.All...)
|
||||
tc.mut(&r)
|
||||
if err := admit(r); err == nil {
|
||||
t.Errorf("admitted a rule with %s", tc.name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestWeightOfEvidenceCompounds pins the score's shape. Two independent weak
|
||||
// signals must compound and no single one may saturate — a summed score clamps
|
||||
// at 1 and then loses every further signal.
|
||||
func TestWeightOfEvidenceCompounds(t *testing.T) {
|
||||
one, _ := combine([]hit{{Weight: 0.4, Action: ActionAllow}})
|
||||
two, _ := combine([]hit{{Weight: 0.4, Action: ActionAllow}, {Weight: 0.4, Action: ActionAllow}})
|
||||
three, _ := combine([]hit{{Weight: 0.4}, {Weight: 0.4}, {Weight: 0.4}})
|
||||
four, _ := combine([]hit{{Weight: 0.4}, {Weight: 0.4}, {Weight: 0.4}, {Weight: 0.4}})
|
||||
if !(one < two && two < three && three < four) {
|
||||
t.Fatalf("scores %v %v %v %v do not compound — a fourth signal changed nothing", one, two, three, four)
|
||||
}
|
||||
if four >= 1 {
|
||||
t.Fatalf("four 0.4 signals saturated at %v; nothing after this could raise the score", four)
|
||||
}
|
||||
}
|
||||
|
||||
// TestIdempotentDecideDoesNotScoreTwice pins the wire promise. A retried decide
|
||||
// must return the SAME decision and must not move the counters again — a
|
||||
// double-counted payment is a velocity rule firing on one transaction.
|
||||
func TestIdempotentDecideDoesNotScoreTwice(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
body := `{"stage":"payment","subject":{"kind":"transaction","id":"tx-9"},
|
||||
"amount":{"nano":1000000000,"currency":"USD","direction":"in"},
|
||||
"signals":{"ip":"198.51.100.7"},"idem":"k-1"}`
|
||||
|
||||
code, first := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme", body)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("first decide = %d %s", code, first)
|
||||
}
|
||||
code, second := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme", body)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("second decide = %d %s", code, second)
|
||||
}
|
||||
var a, b riskDecision
|
||||
_ = json.Unmarshal(first, &a)
|
||||
_ = json.Unmarshal(second, &b)
|
||||
if a.ID != b.ID {
|
||||
t.Fatalf("the same idempotency key produced two decisions (%s, %s)", a.ID, b.ID)
|
||||
}
|
||||
|
||||
_, list := req(t, app, http.MethodGet, "/v1/risk/decisions", "acme", "u_acme", "")
|
||||
var page riskDecisionPage
|
||||
_ = json.Unmarshal(list, &page)
|
||||
if len(page.Items) != 1 {
|
||||
t.Fatalf("%d decisions recorded for one idempotency key", len(page.Items))
|
||||
}
|
||||
}
|
||||
|
||||
// TestConcurrentRetriesUnderOneKeyProduceOneDecision pins the race the
|
||||
// two-statement version had: insert the row, then claim the key, and two
|
||||
// simultaneous retries both find no row, both insert, and the loser's claim
|
||||
// violates the index — so a caller that retried CORRECTLY gets a 500 and the
|
||||
// tenant's counters moved twice.
|
||||
//
|
||||
// Claiming the key inside the insert makes the loser lose at the index, before a
|
||||
// second decision exists, and read the winner's answer back.
|
||||
func TestConcurrentRetriesUnderOneKeyProduceOneDecision(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
body := `{"stage":"payment","subject":{"kind":"transaction","id":"tx-race"},
|
||||
"amount":{"nano":2500000000,"currency":"USD","direction":"in"},
|
||||
"signals":{"ip":"198.51.100.99"},"idem":"race-1"}`
|
||||
|
||||
const n = 8
|
||||
type result struct {
|
||||
code int
|
||||
id string
|
||||
}
|
||||
out := make(chan result, n)
|
||||
var start sync.WaitGroup
|
||||
start.Add(1)
|
||||
for i := 0; i < n; i++ {
|
||||
go func() {
|
||||
start.Wait()
|
||||
code, b := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme", body)
|
||||
var d riskDecision
|
||||
_ = json.Unmarshal(b, &d)
|
||||
out <- result{code, d.ID}
|
||||
}()
|
||||
}
|
||||
start.Done()
|
||||
|
||||
ids := map[string]int{}
|
||||
for i := 0; i < n; i++ {
|
||||
r := <-out
|
||||
if r.code != http.StatusOK {
|
||||
t.Errorf("a concurrent retry answered %d — a correct retry must never be an error", r.code)
|
||||
continue
|
||||
}
|
||||
ids[r.id]++
|
||||
}
|
||||
if len(ids) != 1 {
|
||||
t.Fatalf("%d concurrent requests under one key produced %d distinct decisions: %v", n, len(ids), ids)
|
||||
}
|
||||
// And exactly one row was written, so the counters moved once.
|
||||
_, list := req(t, app, http.MethodGet, "/v1/risk/decisions", "acme", "u_acme", "")
|
||||
var page riskDecisionPage
|
||||
_ = json.Unmarshal(list, &page)
|
||||
if len(page.Items) != 1 {
|
||||
t.Fatalf("%d decisions recorded under one idempotency key", len(page.Items))
|
||||
}
|
||||
}
|
||||
|
||||
// TestDecideRefusesAnUnknownStageOrKind pins the two vocabularies at the door.
|
||||
func TestDecideRefusesAnUnknownStageOrKind(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
for _, body := range []string{
|
||||
`{"stage":"checkout","subject":{"kind":"account","id":"a"}}`,
|
||||
`{"stage":"signup","subject":{"kind":"wallet","id":"a"}}`,
|
||||
`{"stage":"signup","subject":{"kind":"account","id":""}}`,
|
||||
} {
|
||||
code, got := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme", body)
|
||||
if code != http.StatusBadRequest {
|
||||
t.Errorf("%s = %d %s, want 400", body, code, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSimulateRefusesAnEmptyHistory pins the sandbox's whole reason for
|
||||
// existing: "no alerts" is what a quiet rule and an unrun rule both look like.
|
||||
func TestSimulateRefusesAnEmptyHistory(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/simulate", "acme", "u_acme",
|
||||
`{"candidate":{"name":"c","action":"review","weight":0.5,
|
||||
"all":[{"field":"signal.ip","op":"eq","value":"1.1.1.1"}]}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("simulate = %d %s", code, body)
|
||||
}
|
||||
var rep riskSimulateReport
|
||||
_ = json.Unmarshal(body, &rep)
|
||||
if rep.Refusal == "" {
|
||||
t.Fatal("an empty history reported zero alerts instead of refusing")
|
||||
}
|
||||
if rep.Events != 0 {
|
||||
t.Fatalf("events = %d on an empty history", rep.Events)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSimulateMeasuresAgainstRealHistory drives a decision in, then replays a
|
||||
// candidate over it, so the report is measured rather than asserted.
|
||||
func TestSimulateMeasuresAgainstRealHistory(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
mustOK(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"signup","subject":{"kind":"account","id":"a-1"},"signals":{"ip":"192.0.2.10"}}`,
|
||||
http.StatusOK)
|
||||
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/simulate", "acme", "u_acme",
|
||||
`{"candidate":{"name":"c","stage":"signup","action":"review","weight":0.5,
|
||||
"all":[{"field":"signal.ip","op":"eq","value":"192.0.2.10"}]}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("simulate = %d %s", code, body)
|
||||
}
|
||||
var rep riskSimulateReport
|
||||
_ = json.Unmarshal(body, &rep)
|
||||
if rep.Events != 1 || rep.Alerts != 1 {
|
||||
t.Fatalf("events=%d alerts=%d, want 1/1 — the replay did not read the recorded facts", rep.Events, rep.Alerts)
|
||||
}
|
||||
if len(rep.Added) != 1 {
|
||||
t.Fatalf("added=%v, want the one decision the candidate newly catches", rep.Added)
|
||||
}
|
||||
}
|
||||
|
||||
// TestModeDefaultsToShadowAndSaysSo pins the default at the wire.
|
||||
func TestModeDefaultsToShadowAndSaysSo(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
code, body := req(t, app, http.MethodGet, "/v1/risk/mode", "acme", "u_acme", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("mode = %d %s", code, body)
|
||||
}
|
||||
var m riskModeView
|
||||
_ = json.Unmarshal(body, &m)
|
||||
if m.Mode != "shadow" {
|
||||
t.Fatalf("a fresh tenant is %q, not shadow — it would act on its very first request", m.Mode)
|
||||
}
|
||||
|
||||
// A decision in shadow must SAY it was shadow.
|
||||
_, body = req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"signup","subject":{"kind":"account","id":"a-1"}}`)
|
||||
var d riskDecision
|
||||
_ = json.Unmarshal(body, &d)
|
||||
if !d.Shadow {
|
||||
t.Fatal("a shadow tenant's decision does not say so")
|
||||
}
|
||||
}
|
||||
|
||||
// TestHealthCarriesItsReport pins the reason the probe is untyped: the body IS
|
||||
// the answer.
|
||||
func TestHealthCarriesItsReport(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
code, body := req(t, app, http.MethodGet, "/v1/risk/health", "", "", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("health = %d %s", code, body)
|
||||
}
|
||||
for _, k := range []string{"status", "model", "warehouse", "billing"} {
|
||||
if !strings.Contains(string(body), `"`+k+`"`) {
|
||||
t.Errorf("the probe body does not carry %q — a typed op would have dropped exactly this", k)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestTrainIsOnlineAndPerTenant drives training through the wire and proves the
|
||||
// counters moved for the caller's tenant and nobody else's.
|
||||
func TestTrainIsOnlineAndPerTenant(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
obs := make([]string, 0, 20)
|
||||
for i := 0; i < 20; i++ {
|
||||
obs = append(obs, `{"subject":{"kind":"account","id":"a-1"},"amount":{"nano":1000000000,"currency":"USD","direction":"in"}}`)
|
||||
}
|
||||
body := `{"observations":[` + strings.Join(obs, ",") + `]}`
|
||||
code, got := req(t, app, http.MethodPost, "/v1/risk/train", "acme", "u_acme", body)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("train = %d %s", code, got)
|
||||
}
|
||||
var out riskTrainOut
|
||||
_ = json.Unmarshal(got, &out)
|
||||
if out.Learned != 20 {
|
||||
t.Fatalf("learned %d of 20", out.Learned)
|
||||
}
|
||||
_, mine, _ := resOf(t, s, Tenant("hanzo/acme")).arms()
|
||||
if mine.State("hanzo/acme").Learned == 0 {
|
||||
t.Fatal("the caller's model learned nothing")
|
||||
}
|
||||
_, theirs, _ := resOf(t, s, Tenant("hanzo/beta")).arms()
|
||||
if theirs.State("hanzo/beta").Learned != 0 {
|
||||
t.Fatal("another tenant's model learned from this tenant's data")
|
||||
}
|
||||
// And the tenant key really is qualified, not bare.
|
||||
if mine.State("acme").Learned != 0 {
|
||||
t.Fatal("the model is indexed on the BARE org — two brands' same-named orgs would share it")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
package risk
|
||||
|
||||
// door.go is the one place a value this app cannot price is refused.
|
||||
//
|
||||
// WHY IT IS ONE MIDDLEWARE AND NOT A RULE PER FIELD. Every bound in bound.go is
|
||||
// a count multiplied by textMax — 1,278 keys, 8,192 list entries, 1,024 memoised
|
||||
// agency answers — and that multiplication is only arithmetic if NO value can be
|
||||
// longer than textMax. A per-field check would have to be written again for the
|
||||
// next field, and the field it was not written for is the one that reopens the
|
||||
// hole: a subject id, a signal value, an agent reference, a rule term and a path
|
||||
// segment all end up in the same rings, the same maps and the same rows.
|
||||
//
|
||||
// So the door reads the REQUEST, not the shape: every string in the body, every
|
||||
// path segment, every query value, and the body's own length. Whatever an op
|
||||
// adds tomorrow arrives through here.
|
||||
//
|
||||
// IT REFUSES, IT DOES NOT TRUNCATE. A truncated identifier is a different
|
||||
// subject silently sharing a counter with the one that was asked about, which is
|
||||
// the same cross-key confusion the tenant boundary exists to prevent, one layer
|
||||
// down.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"strings"
|
||||
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// door refuses any request carrying a value longer than textMax, or a body
|
||||
// larger than bodyMax, before a typed op ever sees it.
|
||||
func door() zip.Handler {
|
||||
return func(c *zip.Ctx) error {
|
||||
for _, seg := range strings.Split(c.Path(), "/") {
|
||||
if len(seg) > textMax {
|
||||
return zip.ErrBadRequest(errLong("a path segment").Error())
|
||||
}
|
||||
}
|
||||
for k, v := range c.Fiber().Queries() {
|
||||
if len(k) > textMax || len(v) > textMax {
|
||||
return zip.ErrBadRequest(errLong("query parameter " + trim(k)).Error())
|
||||
}
|
||||
}
|
||||
body := c.Body()
|
||||
if len(body) > bodyMax {
|
||||
return zip.Errorf(413, "this request is %d bytes; this plane accepts at most %d", len(body), bodyMax)
|
||||
}
|
||||
if len(body) == 0 {
|
||||
return c.Next()
|
||||
}
|
||||
if err := admitJSON(body); err != nil {
|
||||
return zip.ErrBadRequest(err.Error())
|
||||
}
|
||||
return c.Next()
|
||||
}
|
||||
}
|
||||
|
||||
// admitJSON walks the body as a token stream and refuses the first over-long
|
||||
// string, whether it is a key or a value.
|
||||
//
|
||||
// A TOKEN WALK, NOT A DECODE. Decoding into `any` would allocate the whole
|
||||
// document — including the very strings being refused — which is the cost the
|
||||
// cap exists to prevent. The stream reads one token at a time and returns on the
|
||||
// first offender.
|
||||
//
|
||||
// A body that is not JSON is not this door's business: the typed op's own bind
|
||||
// answers for that, and refusing here would turn every malformed-body 400 into a
|
||||
// message about a length nobody exceeded.
|
||||
func admitJSON(body []byte) error {
|
||||
dec := json.NewDecoder(bytes.NewReader(body))
|
||||
dec.UseNumber()
|
||||
for {
|
||||
tok, err := dec.Token()
|
||||
if err == io.EOF {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil // not JSON, or truncated: the op's bind is the authority
|
||||
}
|
||||
if s, ok := tok.(string); ok && len(s) > textMax {
|
||||
return errLong("a value in this request (" + trim(s) + "…)")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// trim renders enough of an offending name to find it without echoing the whole
|
||||
// of what was refused back onto the wire.
|
||||
func trim(s string) string {
|
||||
const shown = 32
|
||||
if len(s) <= shown {
|
||||
return s
|
||||
}
|
||||
return s[:shown]
|
||||
}
|
||||
@@ -0,0 +1,333 @@
|
||||
package risk
|
||||
|
||||
// feature.go is the ONLY file in this package that reads the warehouse. Every
|
||||
// other file reaches a tenant's numbers through a function declared here, and
|
||||
// every function declared here takes a minted Tenant — so "a feature read is
|
||||
// tenant-scoped" is a property of the type signature, not of a reviewer's
|
||||
// attention.
|
||||
//
|
||||
// TWO PLANES, AND THE SECOND IS WHY THE FIRST IS SAFE TO SHARE.
|
||||
//
|
||||
// hanzo.risk_feature per-tenant. `org` is the FIRST sort key, so a tenant read
|
||||
// is a prefix scan and not a filter, and the predicate is
|
||||
// bound rather than interpolated.
|
||||
// hanzo.risk_baseline the network baseline. It has NO tenant column, no subject,
|
||||
// no id and no pseudonym — only quantiles over a k-anonymous
|
||||
// set of contributing orgs. There is no query against it that
|
||||
// returns one org's rows, because the rows do not exist. That
|
||||
// makes cross-tenant learning UNCOMPUTABLE rather than merely
|
||||
// disallowed, which is the only form of that boundary worth
|
||||
// having.
|
||||
//
|
||||
// THE HOT PATH DOES NOT COME THROUGH HERE. A payment-stage decision must answer
|
||||
// inside the processor's authorization window, and the warehouse is a single
|
||||
// StatefulSet pod that has taken api.hanzo.ai down once already. Scoring reads
|
||||
// the in-memory velocity rings (constant time, fixed memory per key); this file
|
||||
// is the BACKFILL that warms those rings and the read behind the dictionary and
|
||||
// the search sandbox. A decide that needs analytics up is a payment plane that
|
||||
// fails when analytics does.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/hanzoai/cloud/apps/datastore"
|
||||
)
|
||||
|
||||
// featureTable is the per-tenant feature surface. It is a projection of the ONE
|
||||
// event door (POST /v1/event -> bus -> event.event / event.error) plus the LLM
|
||||
// spend ledger, reduced to the counts a risk decision reads and bucketed at five
|
||||
// minutes, so a subject's recent shape is one prefix scan.
|
||||
//
|
||||
// org LEADS the sort key deliberately, unlike hanzo.cloud_usage's
|
||||
// (timestamp, organization, ...): a per-tenant read must be a prefix scan.
|
||||
const featureTable = "hanzo.risk_feature"
|
||||
|
||||
// baselineTable is the network baseline: quantiles per (bucket, subject kind,
|
||||
// feature), over a k-anonymous set of contributing orgs. No tenant column
|
||||
// exists, so no tenant row can be selected from it.
|
||||
const baselineTable = "hanzo.risk_baseline"
|
||||
|
||||
const featureDDL = `
|
||||
CREATE TABLE IF NOT EXISTS hanzo.risk_feature (
|
||||
org String,
|
||||
subject_kind LowCardinality(String),
|
||||
subject String,
|
||||
bucket DateTime,
|
||||
events UInt32,
|
||||
sessions UInt32,
|
||||
distincts UInt32,
|
||||
errors UInt32,
|
||||
spend_nano Int64,
|
||||
tokens UInt64,
|
||||
ips UInt16,
|
||||
uas UInt16,
|
||||
countries UInt16,
|
||||
signups UInt32,
|
||||
payments UInt32,
|
||||
declines UInt32,
|
||||
disputes UInt32,
|
||||
payouts UInt32
|
||||
) ENGINE = SummingMergeTree()
|
||||
ORDER BY (org, subject_kind, subject, bucket)
|
||||
TTL bucket + INTERVAL 400 DAY`
|
||||
|
||||
const baselineDDL = `
|
||||
CREATE TABLE IF NOT EXISTS hanzo.risk_baseline (
|
||||
bucket Date,
|
||||
subject_kind LowCardinality(String),
|
||||
feature LowCardinality(String),
|
||||
q10 Float64,
|
||||
q50 Float64,
|
||||
q90 Float64,
|
||||
q99 Float64,
|
||||
orgs UInt32,
|
||||
n UInt64
|
||||
) ENGINE = ReplacingMergeTree()
|
||||
ORDER BY (bucket, subject_kind, feature)
|
||||
TTL bucket + INTERVAL 400 DAY`
|
||||
|
||||
// ensureTables creates both planes idempotently. cloud reads and writes rows on
|
||||
// the event plane but does not own its DDL (o11y does); it DOES own these two,
|
||||
// so they are created here the way apps/datastore/cloudusage.go creates its own —
|
||||
// CREATE IF NOT EXISTS, plus additive ADD COLUMN IF NOT EXISTS migrations, so a
|
||||
// fresh warehouse and a legacy one converge on the same shape.
|
||||
func ensureTables(ctx context.Context) error {
|
||||
if !datastore.Ready() {
|
||||
return errWarehouse
|
||||
}
|
||||
for _, stmt := range []string{featureDDL, baselineDDL} {
|
||||
if err := datastore.Exec(ctx, stmt); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// featureColumns is the FIXED allowlist of columns a caller may name. Nothing
|
||||
// user-derived is ever spelled into a statement: a caller names a column, this
|
||||
// map answers whether that name is one of ours, and the map's own key is what
|
||||
// reaches the SQL. A name that is not here is refused, never interpolated.
|
||||
var featureColumns = map[string]string{
|
||||
"events": "events",
|
||||
"sessions": "sessions",
|
||||
"distincts": "distincts",
|
||||
"errors": "errors",
|
||||
"spend": "spend_nano",
|
||||
"tokens": "tokens",
|
||||
"ips": "ips",
|
||||
"uas": "uas",
|
||||
"countries": "countries",
|
||||
"signups": "signups",
|
||||
"payments": "payments",
|
||||
"declines": "declines",
|
||||
"disputes": "disputes",
|
||||
"payouts": "payouts",
|
||||
}
|
||||
|
||||
// subjectKinds is the FIXED allowlist of aggregation axes. `pair` and `device`
|
||||
// are the axes that surface several nominally unrelated customers acting as one,
|
||||
// which is what multi-account abuse and account sharing look like from here.
|
||||
var subjectKinds = map[string]bool{
|
||||
"account": true, "transaction": true, "session": true, "agent": true,
|
||||
"merchant": true, "payout": true, "user": true, "device": true,
|
||||
"ip": true, "pair": true,
|
||||
}
|
||||
|
||||
// featureRow is one bucket of one subject's activity, as the reader materialises
|
||||
// it. The tenant is NOT a field: a row is only ever produced by a read that was
|
||||
// already scoped, so carrying the key back would be the one place it could be
|
||||
// compared against the wrong thing.
|
||||
type featureRow struct {
|
||||
bucket time.Time
|
||||
values map[string]float64
|
||||
}
|
||||
|
||||
// riskWhere is the tenancy predicate. org LEADS and is BOUND; the window is
|
||||
// bound; the subject kind has already passed the allowlist and is bound anyway.
|
||||
// Nothing user-derived reaches the statement as an identifier.
|
||||
//
|
||||
// The shape is the one apps/analytics/query.go and o11y's eventsql use, which is
|
||||
// the point: a third shape is a third place for the boundary to be got wrong.
|
||||
func riskWhere(t Tenant, kind, subject string, start, end time.Time) (string, []any) {
|
||||
return "org = ? AND subject_kind = ? AND subject = ? AND bucket >= ? AND bucket < ?",
|
||||
[]any{t.String(), kind, subject, tsLiteral(start), tsLiteral(end)}
|
||||
}
|
||||
|
||||
// tsLiteral renders a time the way the warehouse driver binds a DateTime. Same
|
||||
// transport apps/analytics uses; the value is still a bound parameter.
|
||||
func tsLiteral(t time.Time) string { return t.UTC().Format("2006-01-02 15:04:05") }
|
||||
|
||||
// errWarehouse is the honest gap: the warehouse is unreachable, so a read
|
||||
// returns nothing AND says why. A reader that answered zero would be reporting
|
||||
// "this subject did nothing", which is the one answer a risk surface must never
|
||||
// invent.
|
||||
var errWarehouse = fmt.Errorf("risk: the warehouse is unreachable, so this subject's history cannot be read")
|
||||
|
||||
// window reads one subject's buckets over [start,end).
|
||||
//
|
||||
// The caller supplies a Tenant, which it can only have obtained from a validated
|
||||
// principal, and a kind that must be in the allowlist. The column list is built
|
||||
// from featureColumns' own values, never from the caller's spelling.
|
||||
func window(ctx context.Context, t Tenant, kind, subject string, start, end time.Time) ([]featureRow, error) {
|
||||
if !subjectKinds[kind] {
|
||||
return nil, fmt.Errorf("risk: %q is not a subject kind", kind)
|
||||
}
|
||||
if strings.TrimSpace(subject) == "" {
|
||||
return nil, fmt.Errorf("risk: no subject, so the window would name nobody")
|
||||
}
|
||||
if !datastore.Ready() {
|
||||
return nil, errWarehouse
|
||||
}
|
||||
where, args := riskWhere(t, kind, subject, start, end)
|
||||
|
||||
// Ordered so the projection is stable across calls; the names come from our
|
||||
// own map, so this concatenation carries nothing the caller wrote.
|
||||
names := columnNames()
|
||||
sel := make([]string, 0, len(names)+1)
|
||||
sel = append(sel, "bucket")
|
||||
for _, n := range names {
|
||||
sel = append(sel, "sum("+featureColumns[n]+") AS "+n)
|
||||
}
|
||||
q := "SELECT " + strings.Join(sel, ", ") + " FROM " + featureTable +
|
||||
" WHERE " + where + " GROUP BY bucket ORDER BY bucket"
|
||||
|
||||
rows, err := datastore.Query(ctx, q, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]featureRow, 0, len(rows))
|
||||
for _, r := range rows {
|
||||
fr := featureRow{values: make(map[string]float64, len(names))}
|
||||
if b, ok := r["bucket"].(time.Time); ok {
|
||||
fr.bucket = b
|
||||
}
|
||||
for _, n := range names {
|
||||
fr.values[n] = num(r[n])
|
||||
}
|
||||
out = append(out, fr)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// columnNames returns the allowlist keys in a stable order.
|
||||
func columnNames() []string {
|
||||
out := make([]string, 0, len(featureColumns))
|
||||
for n := range featureColumns {
|
||||
out = append(out, n)
|
||||
}
|
||||
sortStrings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// ── the network baseline ────────────────────────────────────────────────────
|
||||
|
||||
// kAnonMin is how many DISTINCT orgs must contribute to a bucket before its
|
||||
// quantiles may be published. Below it a quantile is close enough to one
|
||||
// business's own numbers to be that business's numbers.
|
||||
const kAnonMin = 25
|
||||
|
||||
// nMin is how many observations a published bucket needs. k orgs each
|
||||
// contributing one row is k-anonymous and still statistically meaningless.
|
||||
const nMin = 1000
|
||||
|
||||
// baselinePopulate is the WHOLE cross-org surface. It is never composed from
|
||||
// request data, has no placeholder a caller could reach, and its projection
|
||||
// carries no org, no subject, no id and no pseudonym — only quantiles and two
|
||||
// counts. The HAVING clause is the k-anonymity gate and is part of the statement
|
||||
// rather than a filter applied to its result, so a bucket below the threshold is
|
||||
// never materialised at all.
|
||||
//
|
||||
// THE FLOOR IS SPELLED ONCE. It used to be a package constant with the numbers
|
||||
// written into the SQL as literals AND declared again as kAnonMin/nMin above, so
|
||||
// raising the constant raised only the read-side belt: the writer kept
|
||||
// publishing below the intended floor and the reader silently discarded
|
||||
// everything it wrote. Two spellings of one number is one number that will drift,
|
||||
// and the direction it drifts in here is a privacy floor.
|
||||
//
|
||||
// The reserved `$public` tenant is excluded here as well as at the mint: an
|
||||
// unauthenticated stranger writes into that lane, and a stranger who can move
|
||||
// the network baseline can move every tenant's comparison against it.
|
||||
var baselinePopulate = `
|
||||
INSERT INTO hanzo.risk_baseline (bucket, subject_kind, feature, q10, q50, q90, q99, orgs, n)
|
||||
SELECT
|
||||
toDate(bucket) AS bucket,
|
||||
subject_kind,
|
||||
'events' AS feature,
|
||||
quantileExact(0.10)(events) AS q10,
|
||||
quantileExact(0.50)(events) AS q50,
|
||||
quantileExact(0.90)(events) AS q90,
|
||||
quantileExact(0.99)(events) AS q99,
|
||||
uniqExact(org) AS orgs,
|
||||
count() AS n
|
||||
FROM hanzo.risk_feature
|
||||
WHERE org != '$public' AND org NOT LIKE '%/$public'
|
||||
AND bucket >= toDateTime(toDate(now()) - 1) AND bucket < toDateTime(toDate(now()))
|
||||
GROUP BY bucket, subject_kind
|
||||
HAVING orgs >= ` + strconv.Itoa(kAnonMin) + ` AND n >= ` + strconv.Itoa(nMin)
|
||||
|
||||
// baselineRow is one published quantile bucket. Reflection over this type is
|
||||
// part of the isolation proof: a field naming a tenant, a subject or a person
|
||||
// would be a leak, and the test that asserts none exists reads THIS type and the
|
||||
// DDL above rather than a comment.
|
||||
type baselineRow struct {
|
||||
bucket time.Time
|
||||
kind string
|
||||
feature string
|
||||
q10 float64
|
||||
q50 float64
|
||||
q90 float64
|
||||
q99 float64
|
||||
orgs uint64
|
||||
n uint64
|
||||
}
|
||||
|
||||
// publishBaseline recomputes yesterday's network quantiles. It is the ONLY
|
||||
// writer of the baseline plane and it runs on a schedule, never on a request, so
|
||||
// no caller can time it, steer it or observe its cost.
|
||||
func publishBaseline(ctx context.Context) error {
|
||||
if !datastore.Ready() {
|
||||
return errWarehouse
|
||||
}
|
||||
return datastore.Exec(ctx, baselinePopulate)
|
||||
}
|
||||
|
||||
// baseline reads the published quantiles for one subject kind. There is no
|
||||
// tenant argument BECAUSE THERE IS NO TENANT COLUMN — every caller reads the
|
||||
// same rows, which is what makes them safe to read at all.
|
||||
func baseline(ctx context.Context, kind string, day time.Time) ([]baselineRow, error) {
|
||||
if !subjectKinds[kind] {
|
||||
return nil, fmt.Errorf("risk: %q is not a subject kind", kind)
|
||||
}
|
||||
if !datastore.Ready() {
|
||||
return nil, errWarehouse
|
||||
}
|
||||
rows, err := datastore.Query(ctx,
|
||||
"SELECT bucket, subject_kind, feature, q10, q50, q90, q99, orgs, n FROM "+baselineTable+
|
||||
" WHERE bucket = ? AND subject_kind = ? ORDER BY feature",
|
||||
day.UTC().Format("2006-01-02"), kind)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]baselineRow, 0, len(rows))
|
||||
for _, r := range rows {
|
||||
br := baselineRow{
|
||||
kind: str(r["subject_kind"]), feature: str(r["feature"]),
|
||||
q10: num(r["q10"]), q50: num(r["q50"]), q90: num(r["q90"]), q99: num(r["q99"]),
|
||||
orgs: uint64(num(r["orgs"])), n: uint64(num(r["n"])),
|
||||
}
|
||||
if b, ok := r["bucket"].(time.Time); ok {
|
||||
br.bucket = b
|
||||
}
|
||||
// Belt on top of the HAVING: a row that predates the gate, or one an
|
||||
// operator inserted by hand, is dropped on read rather than trusted.
|
||||
if br.orgs < kAnonMin || br.n < nMin {
|
||||
continue
|
||||
}
|
||||
out = append(out, br)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,176 @@
|
||||
package risk
|
||||
|
||||
// govern_test.go is the regression suite for the unbounded reads on the
|
||||
// authorization path.
|
||||
//
|
||||
// THE DEFECT: every single decide ran three SELECTs with no LIMIT — the whole
|
||||
// rule table, the whole entry table, the whole suppression table — and every row
|
||||
// of all three was then evaluated. A tenant could make its own authorization path
|
||||
// arbitrarily slow, and there was nothing between "one rule" and "a million".
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestTheGovernedPlanesAreBoundedAtTheWrite pins the caps where they belong. A
|
||||
// read-time truncation would be a tenant's controls silently switching off; a
|
||||
// write-time refusal is a tenant being told.
|
||||
func TestTheGovernedPlanesAreBoundedAtTheWrite(t *testing.T) {
|
||||
_, s := wireApp(t)
|
||||
db := dbOf(t, resOf(t, s, Tenant("hanzo/acme")))
|
||||
|
||||
// Rules, to the cap and one past it.
|
||||
base := rule{Name: "r", Stage: StageSignup, Action: ActionReview, Weight: 0.2, Enabled: true,
|
||||
All: []term{{Field: "signal.ip", Op: OpEq, Value: "1.1.1.1"}}}
|
||||
var held int
|
||||
if err := db.QueryRow(`SELECT COUNT(*) FROM rule`).Scan(&held); err != nil {
|
||||
t.Fatalf("count: %v", err)
|
||||
}
|
||||
for i := held; i < ruleCap(); i++ {
|
||||
r := base
|
||||
r.ID = fmt.Sprintf("bulk-%d", i)
|
||||
if err := putRule(db, r); err != nil {
|
||||
t.Fatalf("rule %d of %d: %v", i, ruleCap(), err)
|
||||
}
|
||||
}
|
||||
over := base
|
||||
over.ID = "one-too-many"
|
||||
if err := putRule(db, over); err == nil {
|
||||
t.Fatalf("a tenant wrote rule %d past a cap of %d — the authorization path has no bound", ruleCap()+1, ruleCap())
|
||||
} else if statusOf(err) != 409 {
|
||||
t.Fatalf("the cap refusal answers %d, want 409", statusOf(err))
|
||||
}
|
||||
// A tenant AT its cap can still replace a rule it already has: the cap bounds
|
||||
// growth, and a tenant that cannot fix a bad rule is worse off than one that
|
||||
// cannot add a good one.
|
||||
fix := base
|
||||
fix.ID = "bulk-10"
|
||||
fix.Name = "fixed"
|
||||
if err := putRule(db, fix); err != nil {
|
||||
t.Fatalf("a tenant at its cap could not replace an existing rule: %v", err)
|
||||
}
|
||||
|
||||
// And the read is bounded too, for a row that arrived by any other route.
|
||||
rules, err := loadRules(db)
|
||||
if err != nil {
|
||||
t.Fatalf("loadRules: %v", err)
|
||||
}
|
||||
if len(rules) > ruleCap() {
|
||||
t.Fatalf("the authorization path read %d rules against a cap of %d", len(rules), ruleCap())
|
||||
}
|
||||
}
|
||||
|
||||
// TestListEntriesAreBoundedAcrossEveryList pins that the cap is over the tenant's
|
||||
// WHOLE entry plane. A per-list cap would be a bound on nothing — the decision
|
||||
// path loads every list into one map, and a caller can make more lists.
|
||||
func TestListEntriesAreBoundedAcrossEveryList(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
|
||||
values := make([]string, listCap()+1)
|
||||
for i := range values {
|
||||
values[i] = fmt.Sprintf("10.0.%d.%d", i/256, i%256)
|
||||
}
|
||||
body, _ := json.Marshal(struct {
|
||||
Values []string `json:"values"`
|
||||
}{values})
|
||||
code, out := reqAdmin(t, app, http.MethodPost, "/v1/risk/lists/ip-deny/entries", "acme", "u_acme", string(body))
|
||||
if code != http.StatusConflict {
|
||||
t.Fatalf("adding %d entries answered %d, want 409 — the map the authorization path builds has no bound", len(values), code)
|
||||
}
|
||||
if !strings.Contains(string(out), fmt.Sprint(listCap())) {
|
||||
t.Errorf("the refusal does not name the cap, so a tenant cannot act on it: %s", out)
|
||||
}
|
||||
|
||||
// A batch within the cap still works, or the bound is just breakage.
|
||||
body, _ = json.Marshal(struct {
|
||||
Values []string `json:"values"`
|
||||
}{[]string{"203.0.113.9"}})
|
||||
code, out = reqAdmin(t, app, http.MethodPost, "/v1/risk/lists/ip-deny/entries", "acme", "u_acme", string(body))
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("a normal add answered %d %s", code, out)
|
||||
}
|
||||
}
|
||||
|
||||
// TestGovernanceIsLoadedOncePerChange pins the cache AND its invalidation. A
|
||||
// cache with no invalidation is a worse defect than the unbounded read it
|
||||
// replaces, so both halves are asserted here rather than only the fast one.
|
||||
func TestGovernanceIsLoadedOncePerChange(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
r := resOf(t, s, Tenant("hanzo/acme"))
|
||||
|
||||
rules, _, _, _, err := r.governance()
|
||||
if err != nil {
|
||||
t.Fatalf("governance: %v", err)
|
||||
}
|
||||
before := len(rules)
|
||||
if before == 0 {
|
||||
t.Fatal("a seeded tenant has no rules, so this test cannot see a change")
|
||||
}
|
||||
|
||||
// A write BEHIND the cache — no dirty() — must not be seen. That is what
|
||||
// proves there is a cache at all rather than a re-read every time.
|
||||
if err := putRule(dbOf(t, r), rule{ID: "behind-the-cache", Name: "unseen", Stage: StageSignup,
|
||||
Action: ActionReview, Weight: 0.1, Enabled: true,
|
||||
All: []term{{Field: "signal.ip", Op: OpEq, Value: "9.9.9.9"}}}); err != nil {
|
||||
t.Fatalf("putRule: %v", err)
|
||||
}
|
||||
again, _, _, _, _ := r.governance()
|
||||
if len(again) != before {
|
||||
t.Fatalf("the authorization path re-read the rule table (%d then %d) — three unbounded SELECTs per decision is the defect", before, len(again))
|
||||
}
|
||||
|
||||
// A write THROUGH the op is seen immediately: every writer calls dirty.
|
||||
code, body := reqAdmin(t, app, http.MethodPost, "/v1/risk/rules", "acme", "u_acme",
|
||||
`{"rule":{"name":"through the op","stage":"signup","action":"review","weight":0.3,"enabled":true,
|
||||
"all":[{"field":"signal.ip","op":"eq","value":"8.8.8.8"}]}}`)
|
||||
if code != http.StatusCreated {
|
||||
t.Fatalf("createRule = %d %s", code, body)
|
||||
}
|
||||
after, _, _, _, _ := r.governance()
|
||||
if len(after) <= before {
|
||||
t.Fatalf("a rule created through the op is not visible to the decision path (%d then %d) — the cache is stale", before, len(after))
|
||||
}
|
||||
|
||||
// The live/shadow switch rides the same cache and the same invalidation.
|
||||
if _, _, _, live, _ := r.governance(); live {
|
||||
t.Fatal("a fresh tenant is live; shadow is the default and the default is not configurable")
|
||||
}
|
||||
code, body = reqAdmin(t, app, http.MethodPut, "/v1/risk/mode", "acme", "u_acme", `{"mode":"live"}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("setMode = %d %s", code, body)
|
||||
}
|
||||
if _, _, _, live, _ := r.governance(); !live {
|
||||
t.Fatal("a tenant that went live is still shadow to the decision path — the mode change did not invalidate the cache")
|
||||
}
|
||||
}
|
||||
|
||||
// TestOneTenantsGovernanceCacheIsNotAnothers. The cache lives on the resident, so
|
||||
// there is no map keyed by more than one tenant and therefore no statement that
|
||||
// could return the wrong tenant's rules.
|
||||
func TestOneTenantsGovernanceCacheIsNotAnothers(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
|
||||
code, body := reqAdmin(t, app, http.MethodPost, "/v1/risk/rules", "acme", "u_acme",
|
||||
`{"rule":{"name":"acme only","stage":"signup","action":"block","weight":0.9,"enabled":true,
|
||||
"all":[{"field":"signal.ip","op":"eq","value":"7.7.7.7"}]}}`)
|
||||
if code != http.StatusCreated {
|
||||
t.Fatalf("createRule = %d %s", code, body)
|
||||
}
|
||||
|
||||
mine, _, _, _, _ := resOf(t, s, Tenant("hanzo/acme")).governance()
|
||||
theirs, _, _, _, _ := resOf(t, s, Tenant("hanzo/beta")).governance()
|
||||
for _, r := range theirs {
|
||||
if r.Name == "acme only" {
|
||||
t.Fatal("one tenant's cached rule set is readable by another")
|
||||
}
|
||||
}
|
||||
if len(mine) == len(theirs) {
|
||||
// Both start from the same seed, so equal lengths after A added one means
|
||||
// B got it too.
|
||||
t.Fatalf("A holds %d rules and B holds %d after A added one — the caches are the same map", len(mine), len(theirs))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,717 @@
|
||||
package risk
|
||||
|
||||
// hold_test.go carries the proofs for the five findings that held this branch.
|
||||
// Each one was written FIRST, run against the code that had the defect, and
|
||||
// watched go red; the fix is what turns it green, and reintroducing the defect
|
||||
// turns it red again.
|
||||
//
|
||||
// They live together rather than beside their subject because they are one
|
||||
// story — three of the five are the SAME two defect classes reappearing one
|
||||
// layer down:
|
||||
//
|
||||
// class A a bound on the COUNT of caller-sized values is not a bound.
|
||||
// Bound the bytes, or cap the value at the wire door so that
|
||||
// count x cap IS the byte bound.
|
||||
// class B one store shared by every tenant with a global cap is a
|
||||
// cross-tenant evictor. Per-tenant state, per-tenant bounds.
|
||||
// loud a bound that binds, a model that disarms, a counter that is
|
||||
// dropped: every one of them is a NAMED state an operator and the
|
||||
// tenant can read. A control that switches off quietly is worse
|
||||
// than no control.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"runtime"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
"unsafe"
|
||||
|
||||
luxlog "github.com/luxfi/log"
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// ── 1. the loud half of the per-tenant bound ────────────────────────────────
|
||||
|
||||
// TestStrainedIsTrueTheMomentACounterIsDropped is the whole design resting on
|
||||
// one boolean.
|
||||
//
|
||||
// THE DEFECT: strained() read `vel.Keys() >= maxKeys()`, but the store evicts
|
||||
// PER SHARD at MaxKeys/shardCount+1, and shard skew starts dropping keys long
|
||||
// before the total reaches the bound. A tenant's own counters went missing
|
||||
// while its decisions reported `strained: false` — a rule written as
|
||||
// `velocity.ip.1h.count >= 5` stops firing on an evicted key and the decision
|
||||
// reads clean. That is the silently-disarmed control, reproduced inside a single
|
||||
// tenant.
|
||||
//
|
||||
// THE TEST: record distinct subjects one at a time and, at every step, require
|
||||
// that the tenant is told the moment its live cardinality stops tracking what it
|
||||
// recorded.
|
||||
func TestStrainedIsTrueTheMomentACounterIsDropped(t *testing.T) {
|
||||
t.Setenv(envVelBytes, "1048576") // the floor, so the bound is reachable in a test
|
||||
_, s := wireApp(t)
|
||||
|
||||
tn := Tenant("hanzo/acme")
|
||||
r := resOf(t, s, tn)
|
||||
at := time.Now()
|
||||
|
||||
for i := 0; i < maxKeys()+16; i++ {
|
||||
r.record(observation{at: at, kind: "account", subject: fmt.Sprintf("s-%d", i), amount: 1_000_000_000})
|
||||
vel, _, _ := r.arms()
|
||||
if vel.keys() < i+1 && !r.strained() {
|
||||
t.Fatalf("after %d distinct subjects this tenant holds %d counters — %d of its own are gone — and strained is false",
|
||||
i+1, vel.keys(), i+1-vel.keys())
|
||||
}
|
||||
}
|
||||
if !r.strained() {
|
||||
t.Fatal("the tenant is past its own cardinality bound and strained is still false")
|
||||
}
|
||||
}
|
||||
|
||||
// TestNothingIsEvictedInsideATenantsAggregates is the structural half.
|
||||
//
|
||||
// The engine's own eviction is LRU inside a shard, and it is invisible: no
|
||||
// counter, no callback, nothing a caller can read. So this app does not use it.
|
||||
// The store is built with a cardinality the app's own gate can never reach, and
|
||||
// the gate is what refuses — loudly, and exactly. This test pins that the store
|
||||
// really does hold every key the gate admitted, which is what makes
|
||||
// `Keys() == admitted` true and therefore what makes `strained` exact.
|
||||
func TestNothingIsEvictedInsideATenantsAggregates(t *testing.T) {
|
||||
t.Setenv(envVelBytes, "1048576")
|
||||
tn := Tenant("hanzo/acme")
|
||||
vel := aggregates()
|
||||
at := time.Now()
|
||||
n := maxKeys()
|
||||
for i := 0; i < n; i++ {
|
||||
vel.record(tn, observation{at: at, kind: "account", subject: fmt.Sprintf("s-%d", i), amount: 1_000_000_000}, nil)
|
||||
}
|
||||
// ASKED OF THE ENGINE, not of the ledger. `keys()` counts what the GATE
|
||||
// admitted and cannot see an eviction, so asserting on it would be true
|
||||
// whether or not the store dropped anything — the exact shape of test that
|
||||
// let the original defect through.
|
||||
if got := vel.stored(); got != n {
|
||||
t.Fatalf("the engine holds %d of the %d keys the gate admitted — its own per-shard eviction (MaxKeys/%d+1) is reachable under this app's gate, so a dropped counter is invisible again", got, n, velShards)
|
||||
}
|
||||
if got := vel.keys(); got != n {
|
||||
t.Fatalf("the ledger holds %d of the %d keys it admitted", got, n)
|
||||
}
|
||||
if got := vel.missed(); got != 0 {
|
||||
t.Fatalf("the gate refused %d keys inside its own bound", got)
|
||||
}
|
||||
// And the very next one is refused BY THE GATE, loudly, rather than
|
||||
// disappearing inside a shard.
|
||||
vel.record(tn, observation{at: at, kind: "account", subject: "one-too-many", amount: 1_000_000_000}, nil)
|
||||
if vel.keys() != n || vel.missed() != 1 || !vel.strained() {
|
||||
t.Fatalf("past the bound: keys=%d missed=%d strained=%v — want %d, 1, true",
|
||||
vel.keys(), vel.missed(), vel.strained(), n)
|
||||
}
|
||||
}
|
||||
|
||||
// ── 2. the bound is on bytes, not on a count of caller-sized values ─────────
|
||||
|
||||
// TestThePublishedCeilingHoldsForTheLongestValueTheDoorAccepts is class A.
|
||||
//
|
||||
// THE DEFECT: every bound in this design was a COUNT — 1,387 keys, 1,024 agency
|
||||
// answers, 20,000 list entries — over values the CALLER sizes. The published
|
||||
// 8 MiB per-tenant ceiling was measured against a six-byte identifier; a 64 KiB
|
||||
// subject id put ONE tenant at 100 MiB, thirteen times the number an operator
|
||||
// was reading.
|
||||
//
|
||||
// THE TEST: measure a key whose value is the LONGEST the wire door will accept,
|
||||
// and fail if the published per-key figure under-states it. `fmt.Sprintf("s-%d")`
|
||||
// is not a worst case and a ceiling proven against it proves nothing.
|
||||
func TestThePublishedCeilingHoldsForTheLongestValueTheDoorAccepts(t *testing.T) {
|
||||
tn := Tenant(strings.Repeat("o", textMax/2) + "/" + strings.Repeat("g", textMax/2-1))
|
||||
value := strings.Repeat("v", textMax)
|
||||
n := maxKeys()
|
||||
|
||||
runtime.GC()
|
||||
var before, after runtime.MemStats
|
||||
runtime.ReadMemStats(&before)
|
||||
vel := aggregates()
|
||||
at := time.Now()
|
||||
for i := 0; i < n; i++ {
|
||||
vel.record(tn, observation{at: at, kind: "account", subject: fmt.Sprintf("%d%s", i, value[:textMax-8]), amount: 1_000_000_000}, nil)
|
||||
}
|
||||
runtime.GC()
|
||||
runtime.ReadMemStats(&after)
|
||||
runtime.KeepAlive(vel)
|
||||
|
||||
if vel.keys() != n {
|
||||
t.Fatalf("the gate admitted %d of %d keys", vel.keys(), n)
|
||||
}
|
||||
measured := int(after.HeapAlloc-before.HeapAlloc) / n
|
||||
if measured > bytesPerKey() {
|
||||
t.Fatalf("a worst-case key measures %d B against a published ceiling of %d B — the per-tenant budget buys %d keys it cannot hold",
|
||||
measured, bytesPerKey(), maxKeys())
|
||||
}
|
||||
if held := n * measured; held > velBytes() {
|
||||
t.Fatalf("a full tenant measures %d B against a published per-tenant budget of %d B", held, velBytes())
|
||||
}
|
||||
t.Logf("worst-case key: measured %d B, published %d B; %d keys per tenant, %d B measured against a %d B budget",
|
||||
measured, bytesPerKey(), n, n*measured, velBytes())
|
||||
}
|
||||
|
||||
// TestTheWireDoorRefusesAValueTheBoundCannotPrice is the other half of class A:
|
||||
// the cap has to be enforced where the value ARRIVES, or the arithmetic above is
|
||||
// about a value the app never sees.
|
||||
//
|
||||
// It is one middleware over both groups rather than a rule per field, because a
|
||||
// rule per field is a rule the next field will not have.
|
||||
func TestTheWireDoorRefusesAValueTheBoundCannotPrice(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
long := strings.Repeat("x", textMax+1)
|
||||
|
||||
for _, c := range []struct {
|
||||
what string
|
||||
method string
|
||||
path string
|
||||
body string
|
||||
}{
|
||||
{"a subject id", http.MethodPost, "/v1/risk/decide", `{"subject":{"kind":"account","id":"` + long + `"},"stage":"payment"}`},
|
||||
{"a signal value", http.MethodPost, "/v1/risk/decide", `{"subject":{"kind":"account","id":"a"},"signals":{"ip":"` + long + `"}}`},
|
||||
{"an agent reference", http.MethodPost, "/v1/risk/decide", `{"subject":{"kind":"account","id":"a"},"actor":{"agent":"` + long + `"}}`},
|
||||
{"a list entry", http.MethodPost, "/v1/risk/lists/deny-ip/entries", `{"values":["` + long + `"]}`},
|
||||
{"a rule expression", http.MethodPost, "/v1/risk/rules", `{"id":"r","when":"` + long + `","action":"decline"}`},
|
||||
{"a path segment", http.MethodDelete, "/v1/risk/lists/deny-ip/entries/" + long, ""},
|
||||
{"a scored observation", http.MethodPost, "/v1/risk/score", `{"observation":{"subject":{"kind":"account","id":"` + long + `"}}}`},
|
||||
} {
|
||||
code, body := req(t, app, c.method, c.path, "acme", "u_acme", c.body)
|
||||
if code != http.StatusBadRequest {
|
||||
t.Errorf("%s of %d bytes answered %d, want 400 — a value the ceiling cannot price reached the store: %s",
|
||||
c.what, len(long), code, string(body))
|
||||
}
|
||||
}
|
||||
|
||||
// And the door does not refuse what the app is for.
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"subject":{"kind":"account","id":"a-1"},"stage":"payment","amount":{"nano":1000000000,"currency":"USD"}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("an ordinary decision answered %d: %s", code, string(body))
|
||||
}
|
||||
}
|
||||
|
||||
// ── 3. a rollout does not run the backlog it is tearing down ────────────────
|
||||
|
||||
// TestAStoppedRunnerDoesNotRunItsBacklog.
|
||||
//
|
||||
// THE DEFECT: stop() cancelled only the claims that had a cancel func — a job
|
||||
// still in the BACKLOG has none, because execute() installs it — and then closed
|
||||
// the channel. The workers drained the remainder, found each claim still there,
|
||||
// installed a FRESH budget and ran the whole thing. Worst case stop() blocks for
|
||||
// backlog/workers x searchBudget while SIGTERM's grace period is ~40s, so the
|
||||
// pod is killed before teardown snapshots anything: every tenant reverts to its
|
||||
// last snapshot or to `warming`. Ship-blocker 6, fleet-wide, armed by any
|
||||
// authenticated tenant queueing searches before a deploy.
|
||||
func TestAStoppedRunnerDoesNotRunItsBacklog(t *testing.T) {
|
||||
_, s := wireApp(t)
|
||||
db := dbOf(t, resOf(t, s, Tenant("hanzo/acme")))
|
||||
|
||||
r := newRunner(luxlog.New("risktest"))
|
||||
var ran int64
|
||||
var mu sync.Mutex
|
||||
const queued = searchQueue
|
||||
for i := 0; i < queued; i++ {
|
||||
id := fmt.Sprintf("search_%d", i)
|
||||
if err := putSearch(db, id, searchRunning, []byte(`{}`)); err != nil {
|
||||
t.Fatalf("seeding the row: %v", err)
|
||||
}
|
||||
err := r.start(job{
|
||||
t: Tenant(fmt.Sprintf("hanzo/t%d", i)), id: id, db: db,
|
||||
load: func() ([]observation, error) {
|
||||
mu.Lock()
|
||||
ran++
|
||||
mu.Unlock()
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
return []observation{{at: time.Now(), kind: "account", subject: "s"}}, nil
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("queueing %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
r.stop()
|
||||
took := time.Since(start)
|
||||
|
||||
if took > 10*time.Second {
|
||||
t.Fatalf("stop() took %s — a rollout's grace period is ~40s and teardown snapshots every model AFTER this returns", took)
|
||||
}
|
||||
for i := 0; i < queued; i++ {
|
||||
status, _, err := getSearch(db, fmt.Sprintf("search_%d", i))
|
||||
if err != nil {
|
||||
t.Fatalf("reading row %d: %v", i, err)
|
||||
}
|
||||
if status == searchDone {
|
||||
t.Fatalf("search_%d ran to completion during shutdown — the backlog is work a rollout still does", i)
|
||||
}
|
||||
if status == searchRunning {
|
||||
t.Fatalf("search_%d was left `running` by a process that is gone", i)
|
||||
}
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if ran > searchWorkers {
|
||||
t.Fatalf("%d queued searches read their inputs while the node was shutting down", ran)
|
||||
}
|
||||
}
|
||||
|
||||
// TestASearchIsBoundedBeforeItReadsHistory. The expensive part ran BEFORE the
|
||||
// bound: replayHistory materialised up to 5,000 rows into observations and only
|
||||
// then asked whether this tenant already had a run going. Thirty concurrent
|
||||
// calls each paid the full read and 29 answered 409 — none of them metered, and
|
||||
// nothing limited how many held 5,000 observations at once.
|
||||
//
|
||||
// The inputs are now the WORKER's to load, so a refusal costs a map lookup and
|
||||
// at most searchWorkers loads exist at any instant.
|
||||
func TestASearchIsBoundedBeforeItReadsHistory(t *testing.T) {
|
||||
r := idle()
|
||||
tn := Tenant("hanzo/acme")
|
||||
var loads int64
|
||||
load := func() ([]observation, error) {
|
||||
loads++
|
||||
return nil, nil
|
||||
}
|
||||
if err := r.start(job{t: tn, id: "search_1", load: load}); err != nil {
|
||||
t.Fatalf("the first search was refused: %v", err)
|
||||
}
|
||||
for i := 0; i < 30; i++ {
|
||||
if err := r.start(job{t: tn, id: fmt.Sprintf("search_r%d", i), load: load}); err == nil {
|
||||
t.Fatal("a second concurrent search for one tenant was accepted")
|
||||
}
|
||||
}
|
||||
if loads != 0 {
|
||||
t.Fatalf("%d refused searches read this tenant's history first — the expensive half runs before the bound", loads)
|
||||
}
|
||||
}
|
||||
|
||||
// ── 4. an operating point, not a ceiling of 48 ──────────────────────────────
|
||||
|
||||
// TestANewTenantIsAdmittedWhileTheNodeHasMemory.
|
||||
//
|
||||
// THE DEFECT: tenantMax defaulted to 48 and priced every tenant at its WORST
|
||||
// case — the full 8 MiB aggregate budget — whether it held two entities or two
|
||||
// thousand. The 49th concurrently-active org was refused the entire risk surface
|
||||
// with 503, and reclaim only freed a cell after six hours of that tenant's own
|
||||
// silence, so on a one-replica pod the refusal stood for most of a day. Raising
|
||||
// the knob broke the ceiling it was computed from: 4,096 x 10.4 MiB = 42 GiB.
|
||||
//
|
||||
// It is class A again, at the process layer: a bound on the COUNT of things
|
||||
// whose size the tenant decides. The bound is now BYTES, and a tenant is priced
|
||||
// at what it actually holds.
|
||||
func TestANewTenantIsAdmittedWhileTheNodeHasMemory(t *testing.T) {
|
||||
_, s := wireApp(t)
|
||||
const orgs = 200
|
||||
for i := 0; i < orgs; i++ {
|
||||
tn := Tenant(fmt.Sprintf("hanzo/org-%d", i))
|
||||
r, err := s.State.res.of(tn, s.Log)
|
||||
if err != nil {
|
||||
held, _, _, _ := s.State.res.count()
|
||||
t.Fatalf("org %d of %d was refused the whole risk surface with %d resident and %d B of %d B in use: %v",
|
||||
i+1, orgs, held, s.State.res.bytes(), memBytes(), err)
|
||||
}
|
||||
// Each one is a real tenant doing real work, not an empty cell.
|
||||
r.record(observation{at: time.Now(), kind: "account", subject: "a-1", amount: 1_000_000_000})
|
||||
}
|
||||
held, _, _, _ := s.State.res.count()
|
||||
if held != orgs {
|
||||
t.Fatalf("%d tenants resident, want %d", held, orgs)
|
||||
}
|
||||
t.Logf("%d active tenants resident in %d B of a %d B node budget", held, s.State.res.bytes(), memBytes())
|
||||
}
|
||||
|
||||
// TestAFullNodeReclaimsBeforeItRefuses. The refusal is still there — a node with
|
||||
// no memory and nothing to reclaim must say so rather than be OOM-killed with
|
||||
// every tenant on board — but it is the LAST answer, not the first. A tenant
|
||||
// that has been silent past the retire floor is reclaimed to make room, which
|
||||
// costs it its rings (published as `since`) and nothing durable.
|
||||
func TestAFullNodeReclaimsBeforeItRefuses(t *testing.T) {
|
||||
t.Setenv(envMemory, fmt.Sprint(2*cellBytes)) // room for two cells
|
||||
_, s := wireApp(t)
|
||||
|
||||
a, b := Tenant("hanzo/acme"), Tenant("hanzo/beta")
|
||||
ra := resOf(t, s, a)
|
||||
resOf(t, s, b)
|
||||
|
||||
// Both are hot: nothing may be taken from either, so the newcomer is refused.
|
||||
if _, err := s.State.res.of(Tenant("hanzo/gamma"), s.Log); err == nil {
|
||||
t.Fatal("a third tenant was admitted onto a full node whose incumbents are both active")
|
||||
}
|
||||
|
||||
// A goes quiet past the retire floor. Now the node can make room without
|
||||
// taking anything from a tenant that is using it.
|
||||
ra.mu.Lock()
|
||||
ra.touched = time.Now().Add(-2 * idleFloor)
|
||||
ra.mu.Unlock()
|
||||
|
||||
if _, err := s.State.res.of(Tenant("hanzo/gamma"), s.Log); err != nil {
|
||||
t.Fatalf("a newcomer was refused while a tenant silent for %s held a cell: %v", 2*idleFloor, err)
|
||||
}
|
||||
if _, _, reclaimed, _ := s.State.res.count(); reclaimed != 1 {
|
||||
t.Fatalf("reclaimed = %d, want 1 — a cell taken under pressure has to be counted where an operator reads it", reclaimed)
|
||||
}
|
||||
}
|
||||
|
||||
// ── 5. a cell is never closed under a request that is holding it ────────────
|
||||
|
||||
// TestACellIsNotClosedUnderAnotherRequest.
|
||||
//
|
||||
// THE DEFECT: `of` called abandon() when open() or arm() failed, and abandon
|
||||
// released the cell it found in the map — which, for a concurrent first request
|
||||
// on the same org, is the cell the OTHER request is already using. release()
|
||||
// did `_ = r.db.Close(); r.db = nil`, so the winner's next query dereferenced a
|
||||
// nil *sql.DB. There is no recover() in the request path and cloud runs ONE
|
||||
// replica, so a transient SQLITE_BUSY on a cold tenant is a total outage for
|
||||
// every tenant on the pod.
|
||||
//
|
||||
// THE SHAPE THAT CANNOT EXPRESS IT: a cell is built COMPLETE and only then
|
||||
// published. The only cell `of` can release is one it built and nobody has ever
|
||||
// seen, so there is no abandon() to get wrong.
|
||||
func TestACellIsNotClosedUnderAnotherRequest(t *testing.T) {
|
||||
_, s := wireApp(t)
|
||||
tn := Tenant("hanzo/acme")
|
||||
|
||||
var wg sync.WaitGroup
|
||||
cells := make([]*resident, 8)
|
||||
for i := range cells {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
r, err := s.State.res.of(tn, s.Log)
|
||||
if err != nil {
|
||||
t.Errorf("concurrent admission %d: %v", i, err)
|
||||
return
|
||||
}
|
||||
cells[i] = r
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
// Every racer holds a usable handle onto the SAME cell — one tenant, one
|
||||
// cell, one file.
|
||||
for i, r := range cells {
|
||||
if r == nil {
|
||||
continue
|
||||
}
|
||||
if r != cells[0] {
|
||||
t.Fatalf("racer %d holds a different cell for the same tenant — two writers on one file", i)
|
||||
}
|
||||
if _, _, err := getSearch(dbOf(t, r), "search_absent"); err == nil {
|
||||
t.Fatalf("racer %d: reading an absent row succeeded", i)
|
||||
} else if strings.Contains(err.Error(), "closed") || strings.Contains(err.Error(), "nil") {
|
||||
t.Fatalf("racer %d holds a closed or nil handle: %v", i, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestAPanicInAWorkerIsNotAnOutage. cloud runs one replica: an unrecovered panic
|
||||
// in a background goroutine takes every tenant down. The worker owns the panic,
|
||||
// writes it onto the run's own row, and keeps serving.
|
||||
func TestAPanicInAWorkerIsNotAnOutage(t *testing.T) {
|
||||
_, s := wireApp(t)
|
||||
db := dbOf(t, resOf(t, s, Tenant("hanzo/acme")))
|
||||
|
||||
r := newRunner(luxlog.New("risktest"))
|
||||
defer r.stop()
|
||||
if err := putSearch(db, "search_boom", searchRunning, []byte(`{}`)); err != nil {
|
||||
t.Fatalf("seeding: %v", err)
|
||||
}
|
||||
err := r.start(job{
|
||||
t: Tenant("hanzo/acme"), id: "search_boom", db: db,
|
||||
load: func() ([]observation, error) { panic("the input plane exploded") },
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("queueing: %v", err)
|
||||
}
|
||||
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
status, body, err := getSearch(db, "search_boom")
|
||||
if err != nil {
|
||||
t.Fatalf("reading the row: %v", err)
|
||||
}
|
||||
if status != searchRunning {
|
||||
if status != searchRefused {
|
||||
t.Fatalf("a panicking search left status %q, want %q", status, searchRefused)
|
||||
}
|
||||
var rep searchReport
|
||||
if err := json.Unmarshal(body, &rep); err != nil {
|
||||
t.Fatalf("decoding the report: %v", err)
|
||||
}
|
||||
if rep.Refusal == "" {
|
||||
t.Fatal("the row carries no reason, so nobody can tell a crash from a cancel")
|
||||
}
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("the row is still `running` — the worker died and took its claim with it")
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
|
||||
// And the pool still serves.
|
||||
if err := putSearch(db, "search_after", searchRunning, []byte(`{}`)); err != nil {
|
||||
t.Fatalf("seeding: %v", err)
|
||||
}
|
||||
if err := r.start(job{
|
||||
t: Tenant("hanzo/beta"), id: "search_after", db: db,
|
||||
load: func() ([]observation, error) { return nil, nil },
|
||||
}); err != nil {
|
||||
t.Fatalf("the pool stopped serving after a panic: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// ── 6. using this plane is not governing it ─────────────────────────────────
|
||||
|
||||
// TestGoverningThisTenantNeedsItsOwnAdmin.
|
||||
//
|
||||
// FOUND WHILE FIXING THE FIVE ABOVE, and reported: there was no role, scope or
|
||||
// admin predicate anywhere in the app, so any principal carrying the org claim
|
||||
// could turn the org's fraud plane off. PUT /v1/risk/mode {"mode":"shadow"}
|
||||
// makes every rule observe and nothing act; DELETE a rule deletes a detection;
|
||||
// a blanket suppression mutes one; an allow-list entry is a bypass; appetite
|
||||
// decides how much of the stream the model may look at. A leaked low-privilege
|
||||
// customer key reached all of them, and the customer would find out from a
|
||||
// chargeback.
|
||||
//
|
||||
// One predicate, one place (governState), and the negative and positive halves
|
||||
// are both here: a member is refused and an admin of the SAME org is not, or the
|
||||
// gate is either absent or a wall.
|
||||
func TestGoverningThisTenantNeedsItsOwnAdmin(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
|
||||
for _, c := range []struct {
|
||||
what string
|
||||
method string
|
||||
path string
|
||||
body string
|
||||
admin int
|
||||
}{
|
||||
{"taking the tenant to shadow", http.MethodPut, "/v1/risk/mode", `{"mode":"shadow"}`, http.StatusOK},
|
||||
{"retiring a rule", http.MethodDelete, "/v1/risk/rules/signup-burst-ip", "", http.StatusOK},
|
||||
{"blanket-muting a rule", http.MethodPost, "/v1/risk/suppressions", `{"rule":"payment-card-testing","reason":"x"}`, http.StatusCreated},
|
||||
{"adding an allow-list bypass", http.MethodPost, "/v1/risk/lists/ip-allow/entries", `{"values":["1.2.3.4"]}`, http.StatusOK},
|
||||
{"narrowing what the model looks at", http.MethodPut, "/v1/risk/state/appetite", `{"review":0.001,"sample":0.001}`, http.StatusOK},
|
||||
{"overwriting the learned state", http.MethodPost, "/v1/risk/snapshot", "", http.StatusCreated},
|
||||
} {
|
||||
if code, body := req(t, app, c.method, c.path, "acme", "u_member", c.body); code != http.StatusForbidden {
|
||||
t.Errorf("%s as an ordinary member of the org answered %d, want 403 — a leaked customer key turns this tenant's fraud plane off: %s",
|
||||
c.what, code, string(body))
|
||||
}
|
||||
if code, body := reqAdmin(t, app, c.method, c.path, "acme", "u_admin", c.body); code != c.admin {
|
||||
t.Errorf("%s as an admin OF THIS ORG answered %d, want %d — the gate is a wall, not a scope: %s",
|
||||
c.what, code, c.admin, string(body))
|
||||
}
|
||||
}
|
||||
|
||||
// And USING the plane is untouched: an ordinary member still scores, reads
|
||||
// and labels. A gate that also stopped the product would be a different bug.
|
||||
for _, c := range []struct {
|
||||
what string
|
||||
method string
|
||||
path string
|
||||
body string
|
||||
want int
|
||||
}{
|
||||
{"deciding", http.MethodPost, "/v1/risk/decide", `{"stage":"payment","subject":{"kind":"account","id":"a-1"}}`, http.StatusOK},
|
||||
{"scoring", http.MethodPost, "/v1/risk/score", `{"observation":{"subject":{"kind":"account","id":"a-1"}}}`, http.StatusOK},
|
||||
{"reading the rules", http.MethodGet, "/v1/risk/rules", "", http.StatusOK},
|
||||
{"reading the model state", http.MethodGet, "/v1/risk/state", "", http.StatusOK},
|
||||
} {
|
||||
if code, body := req(t, app, c.method, c.path, "acme", "u_member", c.body); code != c.want {
|
||||
t.Errorf("%s as an ordinary member answered %d, want %d: %s", c.what, code, c.want, string(body))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── 7. the same class, on the volume ────────────────────────────────────────
|
||||
|
||||
// TestTheDecisionLogIsARingAndSaysHowFarBack.
|
||||
//
|
||||
// REPORTED, NOT FIXED IN THE LAST CUT: there was no DELETE, no TTL and no prune
|
||||
// on any durable plane. Every decide wrote a row forever, and every tenant's
|
||||
// SQLite file lives on the pod's ONE volume — so one authenticated org filling
|
||||
// the disk is "one org quiets another" moved from RAM to disk, with every other
|
||||
// tenant's writes failing behind it.
|
||||
//
|
||||
// The log is bounded in BYTES like everything else, and it is a RING rather than
|
||||
// a refusal: refusing a governance write costs a rule the tenant can retry,
|
||||
// refusing a DECISION costs the authorization it asked for. Dropping the oldest
|
||||
// is only honest if the window is published, so the page carries it.
|
||||
func TestTheDecisionLogIsARingAndSaysHowFarBack(t *testing.T) {
|
||||
_, s := wireApp(t)
|
||||
tn := Tenant("hanzo/acme")
|
||||
db := dbOf(t, resOf(t, s, tn))
|
||||
|
||||
// Well past the retention, written straight at the store so the test is
|
||||
// about the ring and not about the decide path's speed.
|
||||
over := recordCap() + 2*pruneEvery
|
||||
at := time.Now().Add(-time.Duration(over) * time.Second)
|
||||
for i := 0; i < over; i++ {
|
||||
err := putDecision(db, observation{
|
||||
id: fmt.Sprintf("dec_%06d", i), at: at.Add(time.Duration(i) * time.Second),
|
||||
stage: StagePayment, kind: "account", subject: "a-1",
|
||||
}, outcome{id: fmt.Sprintf("dec_%06d", i), action: ActionAllow}, "digest", "", at, false)
|
||||
if err != nil {
|
||||
t.Fatalf("writing %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
if err := prune(db); err != nil {
|
||||
t.Fatalf("prune: %v", err)
|
||||
}
|
||||
|
||||
held, oldest, err := retention(db)
|
||||
if err != nil {
|
||||
t.Fatalf("retention: %v", err)
|
||||
}
|
||||
if held > recordCap() {
|
||||
t.Fatalf("the log holds %d decisions against a retention of %d — one tenant's write volume is every tenant's disk", held, recordCap())
|
||||
}
|
||||
if held != recordCap() {
|
||||
t.Fatalf("the log holds %d, want exactly %d — the ring dropped more than it had to", held, recordCap())
|
||||
}
|
||||
if bytes := held * recordMax; bytes > recordBudget {
|
||||
t.Fatalf("the retained log may reach %d B against a published budget of %d B", bytes, recordBudget)
|
||||
}
|
||||
if oldest == "" {
|
||||
t.Fatal("the log does not say how far back it goes, so a period that was never retained reads as a period with nothing in it")
|
||||
}
|
||||
// The NEWEST survived and the OLDEST went: a ring that dropped the wrong end
|
||||
// would pass every count assertion above.
|
||||
rows, err := decisionsPage(db, "", "", "", "", 1)
|
||||
if err != nil || len(rows) != 1 {
|
||||
t.Fatalf("reading the newest: %v %d", err, len(rows))
|
||||
}
|
||||
if rows[0].ID != fmt.Sprintf("dec_%06d", over-1) {
|
||||
t.Fatalf("the newest decision is %s, want dec_%06d — the ring dropped the wrong end", rows[0].ID, over-1)
|
||||
}
|
||||
if _, _, _, _, err := decisionDetail(db, "dec_000000"); err == nil {
|
||||
t.Fatal("the oldest decision is still there, so nothing was pruned")
|
||||
}
|
||||
}
|
||||
|
||||
// ── 6. a value that outlives the request must own its bytes ─────────────────
|
||||
|
||||
// TestADetachedEventDoesNotAliasTheRequest is a defect RED DID NOT REPORT, found
|
||||
// by running the suite under -race: two DATA RACES, both between
|
||||
// analytics.PublishEvents marshalling a risk event and fasthttp parsing the NEXT
|
||||
// request on the same connection.
|
||||
//
|
||||
// THE DEFECT: emit() built the event from strings the caller handed it and then
|
||||
// detached a goroutine to publish it. fasthttp owns the byte buffers behind
|
||||
// header values and the path, and REUSES them for the next request — so
|
||||
// `by(sc)` (X-User-Id), `sc.org` (X-Org-Id) and every path parameter (in.ID,
|
||||
// in.Name) were strings pointing into memory the server was about to overwrite.
|
||||
// The published event therefore carried whatever the NEXT request wrote there,
|
||||
// which on a shared pod is another tenant's user id under this tenant's
|
||||
// DistinctID. Not a crash — a silent, wrong analytics record, and unbounded
|
||||
// undefined behaviour under the race.
|
||||
//
|
||||
// It is the same shape as class A and class B one layer further down: a value
|
||||
// whose LIFETIME is the request, complected with a consumer whose lifetime is
|
||||
// not. The fix is that the detach point takes ownership, once, for everyone.
|
||||
//
|
||||
// THE TEST models exactly what fasthttp does — it hands emit a string aliasing a
|
||||
// buffer and then overwrites the buffer — so it is deterministic rather than a
|
||||
// race the scheduler has to be persuaded into.
|
||||
func TestADetachedEventDoesNotAliasTheRequest(t *testing.T) {
|
||||
buf := []byte("user-alpha")
|
||||
aliased := unsafe.String(unsafe.SliceData(buf), len(buf))
|
||||
|
||||
kept := detach(aliased, "risk.rule.retired", map[string]any{
|
||||
"by": aliased, "rule": aliased, "count": 3,
|
||||
})
|
||||
|
||||
// fasthttp reuses the buffer for the next request on this connection.
|
||||
copy(buf, "user-BETA!")
|
||||
|
||||
if kept.DistinctID != "user-alpha" {
|
||||
t.Errorf("the detached event's tenant is %q after the buffer was reused, want %q — the event aliases request memory", kept.DistinctID, "user-alpha")
|
||||
}
|
||||
for _, k := range []string{"by", "rule"} {
|
||||
if got := kept.Properties[k]; got != "user-alpha" {
|
||||
t.Errorf("the detached event's %q is %q after the buffer was reused, want %q — the event aliases request memory", k, got, "user-alpha")
|
||||
}
|
||||
}
|
||||
if got := kept.Properties["count"]; got != 3 {
|
||||
t.Errorf("the detached event's non-text property is %v, want 3 — taking ownership must not change the value", got)
|
||||
}
|
||||
// The caller's own map must not be the one that was detached: a caller that
|
||||
// reuses or mutates its map after emit returns would otherwise mutate an
|
||||
// event already in flight.
|
||||
props := map[string]any{"by": "u1"}
|
||||
ev := detach("acme", "risk.mode", props)
|
||||
props["by"] = "u2"
|
||||
if ev.Properties["by"] != "u1" {
|
||||
t.Error("the detached event shares the caller's map, so a caller that reuses it rewrites an event already in flight")
|
||||
}
|
||||
}
|
||||
|
||||
// TestThereIsOneDetachAndItTakesOwnership pins the structure rather than the
|
||||
// instance. Cloning inside emit only holds while emit is the ONLY place this
|
||||
// package hands request-derived text to something that outlives the request; a
|
||||
// second `go analytics.Publish...` written next year would reopen the defect
|
||||
// with the fix still sitting in the file. So: exactly one publish site, and it
|
||||
// is reached through detach().
|
||||
func TestThereIsOneDetachAndItTakesOwnership(t *testing.T) {
|
||||
sites := callsIn(t, "analytics", "PublishEvents")
|
||||
total := 0
|
||||
for file, n := range sites {
|
||||
total += n
|
||||
if file != "store.go" {
|
||||
t.Errorf("%s publishes onto the bus %d time(s); the only publish site is emit() in store.go, which detaches an owned copy first", file, n)
|
||||
}
|
||||
}
|
||||
if total != 1 {
|
||||
t.Errorf("analytics.PublishEvents is called %d time(s) in this package, want exactly 1 — a second detach is a second chance to hand a goroutine memory the server is about to overwrite", total)
|
||||
}
|
||||
}
|
||||
|
||||
// TestARolloutDoesNotHandAnInFlightRequestAClosedFile is the OTHER half of
|
||||
// ship-blocker 5, and it is the half that is reachable on every single rollout.
|
||||
//
|
||||
// abandon() is gone and a cell is built complete before it is published, so no
|
||||
// request can have its file closed by a racing admission. But `residency.close`
|
||||
// — what teardown runs on SIGTERM — retires EVERY cell immediately, with no idle
|
||||
// requirement, and cloud deploys Recreate at ONE replica. A request that has
|
||||
// already resolved its tenant and is between two queries then reads
|
||||
// `r.handle == nil`, and `(*sql.DB)(nil).QueryRow` locks a nil mutex: the same
|
||||
// nil dereference, arrived at from shutdown instead of from a lost race, on a
|
||||
// path every deploy takes.
|
||||
//
|
||||
// THE SHAPE THAT CANNOT EXPRESS IT: an op cannot obtain a file handle without
|
||||
// obtaining an error alongside it. tenantState is the ONE door every op passes,
|
||||
// so resolving the handle THERE — once, checked — is what makes a nil handle
|
||||
// unrepresentable downstream rather than a rule 27 call sites have to remember.
|
||||
func TestARolloutDoesNotHandAnInFlightRequestAClosedFile(t *testing.T) {
|
||||
_, s := wireApp(t)
|
||||
tn := Tenant("hanzo/acme")
|
||||
|
||||
// A request that has resolved its tenant and is about to read its file.
|
||||
res, err := s.State.res.of(tn, s.Log)
|
||||
if err != nil {
|
||||
t.Fatalf("resolving the tenant: %v", err)
|
||||
}
|
||||
|
||||
// SIGTERM. teardown snapshots every model and closes every file.
|
||||
s.State.res.close(s.Log)
|
||||
|
||||
// The in-flight request now goes to read. It must be REFUSED, not handed a
|
||||
// handle it will dereference.
|
||||
db, err := res.file()
|
||||
if err == nil {
|
||||
t.Fatal("a cell closed by teardown still hands out a file handle; the next query nil-dereferences and takes every tenant on the pod with it")
|
||||
}
|
||||
if db != nil {
|
||||
t.Fatalf("the refusal came with a handle anyway (%v)", db)
|
||||
}
|
||||
var he *zip.HTTPError
|
||||
if !errors.As(err, &he) || he.Status != http.StatusServiceUnavailable {
|
||||
t.Errorf("a request that lost its cell to a rollout answers %v, want a 503 it can retry against the next pod", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,335 @@
|
||||
package risk
|
||||
|
||||
// learn.go is the model plane: train, exhaustive-search, score — on the AI
|
||||
// cloud's own substrate, in process, per tenant.
|
||||
//
|
||||
// WHERE THE MODELS TRAIN, AND WHY NOT THE ALTERNATIVES.
|
||||
//
|
||||
// - hanzoai/ai is the LLM router. It has no trainer.
|
||||
// - candle is Rust and cloud does not link it.
|
||||
// - Kubeflow (/v1/train/jobs, /v1/train/experiments) is real and stays the
|
||||
// escape hatch for a customer who wants a heavy supervised sweep — but
|
||||
// apps/ml fails closed on it today because no kserve/katib/trainer chart is
|
||||
// deployed. It is not a substrate this product can stand on.
|
||||
//
|
||||
// So: luxfi/aml's half-space trees, in this process. That is not a placeholder —
|
||||
// it is the right shape for the problem, for five reasons that a neural
|
||||
// alternative cannot offer:
|
||||
//
|
||||
// 1. NO TRAINING PASS AND NO RETAINED SAMPLE. The geometry is built BEFORE any
|
||||
// data arrives; the model IS a set of mass counters. Training is one online
|
||||
// increment per observation, so there is no job, no queue and no window in
|
||||
// which the tenant is protected by a stale model.
|
||||
// 2. PER TENANT INCLUDING THE GEOMETRY. The seed is mix(cfg.Seed, orgID), so two
|
||||
// tenants do not merely hold different counters — they hold DIFFERENT TREES.
|
||||
// Probing one reveals nothing about where another's regions lie.
|
||||
// 3. BOUNDED, AND BOUNDED PER TENANT. A measured 336 KB per tenant at the
|
||||
// defaults, in a store that holds exactly ONE tenant (bound.go: forest
|
||||
// forces MaxOrgs to 1), so the engine's cross-tenant LRU is unreachable — one
|
||||
// org can never evict another's learned model. A tenant that goes idle has
|
||||
// its model written down and its memory reclaimed; one that comes back has
|
||||
// it restored, not re-warmed.
|
||||
// 4. ATTRIBUTION IS A COUNTERFACTUAL ON THE MODEL THAT RAISED THE ALERT. Move
|
||||
// one coordinate to its neutral value, rescore, and the drop IS that
|
||||
// feature's contribution. No second explainer model, and therefore no second
|
||||
// thing that can be wrong. That is what makes a decline defensible.
|
||||
// 5. IT CANNOT ACT ALONE. Evidence is capped at review; weight is non-negative;
|
||||
// NaN and Inf are refused.
|
||||
//
|
||||
// EXHAUSTIVE SEARCH WITHOUT KUBEFLOW. luxfi/aml's replay package already replays
|
||||
// a candidate over real history through the engine's own evaluator, writing
|
||||
// nothing — it reaches the evaluator through a one-method interface and history
|
||||
// through another, and imports no store, so a dry run is STRUCTURALLY dry.
|
||||
// mlSearch generalises that from one rule candidate to a grid over the model
|
||||
// topology, replays each candidate over the tenant's own recent decisions, and
|
||||
// returns the learning curve and the winning topology. Native Go, per tenant, no
|
||||
// GPU, no CRD.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"math"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/aml/pkg/anomaly"
|
||||
"github.com/luxfi/aml/pkg/types"
|
||||
)
|
||||
|
||||
// snapshotKey names the row a tenant's learned state is kept under. One row: a
|
||||
// tenant has one model, and a second row would be a second answer to one
|
||||
// question.
|
||||
//
|
||||
// Writing it is what makes a rollout survivable. cloud is strategy Recreate at
|
||||
// one replica, so every deploy drops the process — and with it every warming
|
||||
// model and the appetite threshold it had computed. Without the snapshot, every
|
||||
// deploy silently resets every tenant to warming, and a warming model REFUSES to
|
||||
// score, which reads as "clean" to anything that does not check Refusal. See
|
||||
// resident.go for the read side, which is also the ONE place the reload happens.
|
||||
const snapshotKey = "anomaly"
|
||||
|
||||
// encodeSnapshot and decodeSnapshot are the snapshot's storage form, named so
|
||||
// the two halves are one edit. A snapshot whose shape does not match the running
|
||||
// inventory is REFUSED by the engine on restore, not coerced: state the model
|
||||
// would treat as its own memory has to have come from this algorithm over this
|
||||
// feature set.
|
||||
func encodeSnapshot(s anomaly.Snapshot) ([]byte, error) { return json.Marshal(s) }
|
||||
|
||||
func decodeSnapshot(body []byte) (anomaly.Snapshot, error) {
|
||||
var s anomaly.Snapshot
|
||||
err := json.Unmarshal(body, &s)
|
||||
return s, err
|
||||
}
|
||||
|
||||
// ── exhaustive search ───────────────────────────────────────────────────────
|
||||
|
||||
// axis is one dimension of the topology grid, with the values the search may
|
||||
// try. Both are CLOSED: an unbounded grid is an unbounded amount of a shared
|
||||
// pod's CPU, reachable by anyone with a key.
|
||||
type axis struct {
|
||||
name string
|
||||
values []float64
|
||||
}
|
||||
|
||||
// grid is the search space. Trees x Depth x Window x Blend x Review is 3*3*3*3*3
|
||||
// = 243 candidates at the widest, which is seconds of one core over a few
|
||||
// thousand replayed decisions — bounded by construction rather than by a
|
||||
// timeout.
|
||||
func grid() []axis {
|
||||
return []axis{
|
||||
{"trees", []float64{15, 25, 40}},
|
||||
{"depth", []float64{6, 8, 10}},
|
||||
{"window", []float64{128, 256, 512}},
|
||||
{"blend", []float64{0.1, 0.25, 0.5}},
|
||||
{"review", []float64{0.005, 0.01, 0.02}},
|
||||
}
|
||||
}
|
||||
|
||||
// candidate is one point in the grid.
|
||||
type candidate struct {
|
||||
// Trees is how many half-space trees the model holds.
|
||||
Trees int `json:"trees"`
|
||||
// Depth is how deep each tree splits.
|
||||
Depth int `json:"depth"`
|
||||
// Window is how many observations make up one reference window.
|
||||
Window int `json:"window"`
|
||||
// Blend is how much of a closing window folds into the reference.
|
||||
Blend float64 `json:"blend"`
|
||||
// Review is the share of the stream this topology may send for examination.
|
||||
Review float64 `json:"review"`
|
||||
}
|
||||
|
||||
// trial is what one candidate did over the replayed history.
|
||||
type trial struct {
|
||||
// Candidate is the topology tried.
|
||||
Candidate candidate `json:"candidate"`
|
||||
// Scored is how many observations the model was able to score. A topology
|
||||
// that warms slowly scores fewer, which is a real cost and is reported.
|
||||
Scored int `json:"scored"`
|
||||
// Alerted is how many of those it would have alerted on.
|
||||
Alerted int `json:"alerted"`
|
||||
// Realised is Alerted/Scored — the share actually reached, against the
|
||||
// Review share intended. The gap between the two IS the governance report.
|
||||
Realised float64 `json:"realised"`
|
||||
// Separation is the mean score of the alerted set minus the mean score of
|
||||
// the rest. It is the ranking objective: a topology that separates the tail
|
||||
// from the body is doing the job whatever its absolute scores look like.
|
||||
Separation float64 `json:"separation"`
|
||||
// Warm is how many observations passed before the model would score at all.
|
||||
Warm int `json:"warm"`
|
||||
}
|
||||
|
||||
// searchReport is the whole answer.
|
||||
type searchReport struct {
|
||||
// Events is how many historical observations were replayed.
|
||||
Events int `json:"events"`
|
||||
// From and To are the period they span.
|
||||
From time.Time `json:"from,omitzero"`
|
||||
To time.Time `json:"to,omitzero"`
|
||||
// Trials is every candidate tried, best first.
|
||||
Trials []trial `json:"trials"`
|
||||
// Winner is the best-separating topology that also honoured its stated
|
||||
// appetite. Absent when no candidate did both.
|
||||
Winner *candidate `json:"winner,omitempty"`
|
||||
// Curve is the learning curve of the winner: separation as a function of how
|
||||
// much of the history it had seen, in ten steps.
|
||||
Curve []float64 `json:"curve,omitempty"`
|
||||
// Refusal names why a report is empty when it is. An empty report and a
|
||||
// report of no alerts are opposite facts.
|
||||
Refusal string `json:"refusal,omitempty"`
|
||||
}
|
||||
|
||||
// errNoHistory is what a search returns instead of a clean-looking zero. replay
|
||||
// refuses an empty history for exactly this reason: "no alerts" is what a quiet
|
||||
// rule and an unrun rule both look like, and the difference is the whole reason
|
||||
// a sandbox exists.
|
||||
var errNoHistory = fmt.Errorf("risk: no history to replay, so a result would be indistinguishable from a quiet model")
|
||||
|
||||
// searchRun replays every candidate over one tenant's own recorded observations.
|
||||
//
|
||||
// Nothing is written and nothing outside this function's own scratch models is
|
||||
// touched: each candidate gets a FRESH anomaly.Store over a FRESH velocity store
|
||||
// seeded from the tenant key, so a search can never move the live model's
|
||||
// counters, and two searches for two tenants can never see each other's.
|
||||
func searchRun(ctx context.Context, t Tenant, history []observation) (searchReport, error) {
|
||||
if len(history) == 0 {
|
||||
return searchReport{Refusal: errNoHistory.Error()}, errNoHistory
|
||||
}
|
||||
rep := searchReport{Events: len(history), From: history[0].at, To: history[len(history)-1].at}
|
||||
|
||||
for _, c := range candidates() {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return rep, ctx.Err()
|
||||
default:
|
||||
}
|
||||
tr, err := replayCandidate(t, c, history)
|
||||
if err != nil {
|
||||
continue // a topology the engine refuses is not a result, it is a non-candidate
|
||||
}
|
||||
rep.Trials = append(rep.Trials, tr)
|
||||
}
|
||||
sort.SliceStable(rep.Trials, func(i, j int) bool {
|
||||
return rep.Trials[i].Separation > rep.Trials[j].Separation
|
||||
})
|
||||
for i := range rep.Trials {
|
||||
// The winner must have SCORED something and must have honoured its stated
|
||||
// appetite within a factor of two. A topology that separates beautifully
|
||||
// while alerting on nothing is not a control.
|
||||
tr := rep.Trials[i]
|
||||
if tr.Scored == 0 || tr.Realised == 0 {
|
||||
continue
|
||||
}
|
||||
if tr.Realised > 2*tr.Candidate.Review {
|
||||
continue
|
||||
}
|
||||
w := tr.Candidate
|
||||
rep.Winner = &w
|
||||
rep.Curve = curve(ctx, t, w, history)
|
||||
break
|
||||
}
|
||||
if rep.Winner == nil {
|
||||
rep.Refusal = "no candidate both scored and honoured its stated appetite over this history"
|
||||
}
|
||||
return rep, nil
|
||||
}
|
||||
|
||||
// candidates expands the grid. Written as an explicit product so the count is
|
||||
// visible in the code rather than emergent from a recursion.
|
||||
func candidates() []candidate {
|
||||
g := grid()
|
||||
var out []candidate
|
||||
for _, trees := range g[0].values {
|
||||
for _, depth := range g[1].values {
|
||||
for _, win := range g[2].values {
|
||||
for _, blend := range g[3].values {
|
||||
for _, review := range g[4].values {
|
||||
out = append(out, candidate{
|
||||
Trees: int(trees), Depth: int(depth), Window: int(win),
|
||||
Blend: blend, Review: review,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// replayCandidate runs one topology over the history in a sandbox.
|
||||
//
|
||||
// The sandbox's aggregates carry the SAME per-tenant bound the live plane does
|
||||
// (aggregates()), so a search over five thousand distinct subjects costs what one
|
||||
// tenant is allowed to cost and not a multiple of it. A sandbox that could
|
||||
// allocate without a ceiling would be the memory bomb again, reachable by an op
|
||||
// that answers 202 and then runs for minutes.
|
||||
func replayCandidate(t Tenant, c candidate, history []observation) (trial, error) {
|
||||
vel := aggregates()
|
||||
model, err := forest(anomaly.Config{
|
||||
Trees: c.Trees, Depth: c.Depth, Window: c.Window, Blend: c.Blend,
|
||||
Appetite: anomaly.Appetite{Review: c.Review, Sample: 0.001},
|
||||
}, vel)
|
||||
if err != nil {
|
||||
return trial{}, err
|
||||
}
|
||||
|
||||
tr := trial{Candidate: c}
|
||||
var alerted, rest []float64
|
||||
for _, o := range history {
|
||||
vel.record(t, o, nil)
|
||||
tx := types.Transaction{
|
||||
ID: o.id, OrgID: t.String(), UserID: o.subject, AccountID: o.subject,
|
||||
Currency: o.currency, Direction: o.direction,
|
||||
IPAddress: o.signals["ip"], DeviceFingerprint: o.signals["device"],
|
||||
Timestamp: o.at, USD: nanoUSD(o.amount),
|
||||
}
|
||||
a := model.Inspect(tx, types.Entity{ID: o.subject, OrgID: t.String()})
|
||||
// Inspect does not learn, so a second pass through the learning path is
|
||||
// what advances the model. judge(learn=true) is Assess; in shadow it
|
||||
// returns no hit, and the counters still move — which is exactly the
|
||||
// replay we want.
|
||||
_, _ = model.Assess(tx, types.Entity{ID: o.subject, OrgID: t.String()})
|
||||
if !a.Scored {
|
||||
tr.Warm++
|
||||
continue
|
||||
}
|
||||
tr.Scored++
|
||||
if a.Score >= a.Cut {
|
||||
tr.Alerted++
|
||||
alerted = append(alerted, a.Score)
|
||||
} else {
|
||||
rest = append(rest, a.Score)
|
||||
}
|
||||
}
|
||||
if tr.Scored > 0 {
|
||||
tr.Realised = round4(float64(tr.Alerted) / float64(tr.Scored))
|
||||
}
|
||||
tr.Separation = round4(mean(alerted) - mean(rest))
|
||||
return tr, nil
|
||||
}
|
||||
|
||||
// curve is the winner's separation as a function of how much history it had
|
||||
// seen. Ten steps: enough to see whether the model is still improving, few
|
||||
// enough that the answer is a chart and not a data set.
|
||||
//
|
||||
// It polls the deadline like the grid above does. Ten more replays after the
|
||||
// grid finished is the same work again, and an op that stops honouring its
|
||||
// budget on the last step is an op with no budget.
|
||||
func curve(ctx context.Context, t Tenant, c candidate, history []observation) []float64 {
|
||||
out := make([]float64, 0, 10)
|
||||
for i := 1; i <= 10; i++ {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return out
|
||||
default:
|
||||
}
|
||||
n := len(history) * i / 10
|
||||
if n == 0 {
|
||||
out = append(out, 0)
|
||||
continue
|
||||
}
|
||||
tr, err := replayCandidate(t, c, history[:n])
|
||||
if err != nil {
|
||||
out = append(out, 0)
|
||||
continue
|
||||
}
|
||||
out = append(out, tr.Separation)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func mean(xs []float64) float64 {
|
||||
if len(xs) == 0 {
|
||||
return 0
|
||||
}
|
||||
var s float64
|
||||
for _, x := range xs {
|
||||
s += x
|
||||
}
|
||||
v := s / float64(len(xs))
|
||||
if math.IsNaN(v) || math.IsInf(v, 0) {
|
||||
return 0
|
||||
}
|
||||
return v
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
package risk
|
||||
|
||||
// devmaster keys this test binary: cek opens nothing without a master and a test
|
||||
// process has no KMS. Risk needs it because every decision is DURABLE FIRST —
|
||||
// the record lands in the tenant's own encrypted file before anything else — so
|
||||
// a suite that cannot open a store cannot exercise the decision path at all.
|
||||
import _ "github.com/hanzoai/cloud/internal/devmaster"
|
||||
@@ -0,0 +1,913 @@
|
||||
package risk
|
||||
|
||||
// resident.go is the ONE registry of live per-tenant state, and the only place a
|
||||
// tenant's aggregates, model, file and governance cache are created, reclaimed or
|
||||
// closed.
|
||||
//
|
||||
// ONE CELL PER TENANT, NOTHING SHARED. A resident holds this tenant's own
|
||||
// velocity rings, its own half-space forest, its own SQLite file and its own
|
||||
// cached rule set. No map inside any of them is indexed by more than one tenant,
|
||||
// so there is no eviction, no read and no write that could cross a tenant even by
|
||||
// mistake. That is the structural answer to the defect three planes shipped: a
|
||||
// process-wide store with a GLOBAL cap, where one org's volume silently deletes
|
||||
// another org's state.
|
||||
//
|
||||
// A CELL IS BUILT COMPLETE AND ONLY THEN PUBLISHED, and that is what makes the
|
||||
// lifetime safe. An earlier cut published an empty cell, opened its file, and on
|
||||
// any failure called abandon() — which released whatever cell it found in the
|
||||
// map. For a concurrent first request on the same org that is the cell the OTHER
|
||||
// request is already holding: release() closed the handle and nil'd it, and the
|
||||
// winner's next query dereferenced a nil *sql.DB. cloud runs ONE replica, so a
|
||||
// transient SQLITE_BUSY on a cold tenant was a total outage for every tenant on
|
||||
// the pod. Here the only cell `of` can discard is one it built and nobody has
|
||||
// ever seen, so there is no abandon to get wrong, and the handle is read through
|
||||
// db() under the cell's own lock rather than off the struct.
|
||||
//
|
||||
// EVERY DEGRADATION IS SELF-INFLICTED OR LOUD.
|
||||
//
|
||||
// own cardinality bound the tenant's own gate refuses a NEW key, counts it,
|
||||
// and reports `strained` on the decision and the probe.
|
||||
// Its existing counters keep moving.
|
||||
// own idleness after idleReclaim of SILENCE the tenant is RETIRED:
|
||||
// its model is snapshotted onto its own file first, then
|
||||
// its aggregates, caches and file handle are released.
|
||||
// Nothing durable is lost, so its next request comes
|
||||
// back with what it learned.
|
||||
// the node is full cells silent past idleFloor are reclaimed FIRST —
|
||||
// counted, and readable on the probe. Only if there is
|
||||
// nothing to reclaim is a newcomer refused, loudly
|
||||
// (503 + an error log + a degraded probe). No tenant
|
||||
// that is USING the node is ever taken from.
|
||||
//
|
||||
// RETIREMENT IS THE WHOLE LIFETIME, not a half of one. An earlier cut dropped a
|
||||
// tenant's aggregates but KEPT its cell forever, so the map only ever grew: once
|
||||
// the ceiling of distinct tenants had passed through, the next one was refused
|
||||
// for the life of the process even with the pod idle. A high-water mark is not a
|
||||
// bound — it is the same "one tenant's presence denies another" defect wearing an
|
||||
// admission badge.
|
||||
//
|
||||
// AND NOTHING IS EVER SILENTLY UNDER-COUNTED. `since` is the instant a tenant's
|
||||
// rings started, and it rides every decision. A 30-day count computed from ten
|
||||
// minutes of rings is a true number about the wrong period, and the only way that
|
||||
// is not a lie is to publish the period. It matters on far more than reclaim:
|
||||
// cloud deploys strategy Recreate at ONE replica, so every rollout starts every
|
||||
// tenant's rings from zero.
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/hanzoai/cloud"
|
||||
"github.com/luxfi/aml/pkg/anomaly"
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// resident is one tenant's live state.
|
||||
type resident struct {
|
||||
t Tenant
|
||||
// res is the node budget this cell draws on. Nil for a cell nothing is
|
||||
// accounting — a search sandbox — which is then bounded by its own maxKeys
|
||||
// alone.
|
||||
res *residency
|
||||
|
||||
mu sync.Mutex
|
||||
// handle is the tenant's own encrypted file. Read through db() under this
|
||||
// lock and never off the struct: release() writes it, and an unsynchronised
|
||||
// read of a field another goroutine may be closing is how a nil dereference
|
||||
// takes a one-replica pod down.
|
||||
handle *sql.DB
|
||||
// The ARM: the two in-memory planes and when they started. Nil when this
|
||||
// tenant has been reclaimed; re-armed on its next request, which is also
|
||||
// the path that reloads its model.
|
||||
vel *rings
|
||||
model *anomaly.Store
|
||||
since time.Time
|
||||
// touched is the last time this tenant asked for anything. The reclaim
|
||||
// sweep reads it and nothing else.
|
||||
touched time.Time
|
||||
// kept is what the tenant's DURABLE snapshot said it had learned when this
|
||||
// cell last armed. Compared against what the live model holds, it is how
|
||||
// "warming, because new" is told apart from "warming, because the learned
|
||||
// state is gone" — the difference between a control that is coming up and
|
||||
// one that is off.
|
||||
kept int64
|
||||
// disarmed says the learned state existed and this process does not have
|
||||
// it: the snapshot was unreadable, or refused by the engine. A decision
|
||||
// made in that condition carries RefusalDisarmed, never RefusalWarming.
|
||||
disarmed bool
|
||||
|
||||
// retired latches the ONE release of this cell. Two sweeps racing the same
|
||||
// cell would otherwise each price it and each hand its bytes back, and a
|
||||
// budget that can be credited twice is not a budget.
|
||||
retired bool
|
||||
// replaying is this tenant's ONE slot for a synchronous replay of its own
|
||||
// history. Atomic rather than under mu because it is held across the replay
|
||||
// itself, and holding the cell's lock for the length of a 5,000-row read
|
||||
// would stall every other request for this tenant.
|
||||
replaying atomic.Bool
|
||||
// writes counts decisions recorded since this cell armed. It is what paces
|
||||
// the decision log's prune without a table scan on the authorization path.
|
||||
writes int
|
||||
|
||||
// The governance cache. Loaded once per change, not once per decision: three
|
||||
// unbounded SELECTs on every authorization was the second defect. Every
|
||||
// writer calls dirty, and both live under this same mutex, so a read can
|
||||
// never see a half-applied change.
|
||||
loaded bool
|
||||
rules []rule
|
||||
lists map[string]map[string]bool
|
||||
sups []suppression
|
||||
live bool
|
||||
// govern is what the loaded governance measures, in bytes. Priced live
|
||||
// rather than reserved because a tenant with ten rules must not be charged
|
||||
// for a budget it is not using — that pricing is the whole operating point.
|
||||
govern int
|
||||
|
||||
// agency memoises this tenant's agent-registry answers, under this tenant's
|
||||
// own bound. It has its own lock because a registry call must not be made
|
||||
// while holding the lock a decision needs. Its ceiling is reserved in
|
||||
// cellBytes.
|
||||
agency agencyCache
|
||||
}
|
||||
|
||||
// residency is the set of residents, bounded in BYTES.
|
||||
type residency struct {
|
||||
dataDir string
|
||||
|
||||
mu sync.Mutex
|
||||
cells map[Tenant]*resident
|
||||
// held is the live sum of what the cells cost. Kept as a running total
|
||||
// rather than recomputed, because a new key on the hot path must not walk
|
||||
// every tenant on the node to find out whether it fits.
|
||||
held int
|
||||
// building serialises the first touch of a tenant, so two concurrent
|
||||
// newcomers open one file rather than two. A second builder waits on the
|
||||
// first's channel and then finds the published cell.
|
||||
building map[Tenant]chan struct{}
|
||||
// The saturation view. All three are on the probe: a node that is refusing
|
||||
// tenants, or reclaiming them under pressure, must page an operator rather
|
||||
// than be discovered in a support ticket.
|
||||
refused int64
|
||||
reclaimed int64
|
||||
refusedAt time.Time
|
||||
}
|
||||
|
||||
func newResidency(dataDir string) *residency {
|
||||
return &residency{dataDir: dataDir, cells: map[Tenant]*resident{}, building: map[Tenant]chan struct{}{}}
|
||||
}
|
||||
|
||||
// errFull is the capacity refusal. 503 and not 403: nothing about the caller is
|
||||
// wrong, this node is out of memory and has nothing idle left to reclaim, and a
|
||||
// retry against a node with room succeeds.
|
||||
var errFull = zip.Errorf(503, "this node has no memory left for another tenant and will not take a live tenant's state to make room")
|
||||
|
||||
// of resolves the caller's resident, admitting and arming it if this process has
|
||||
// not seen it. It is the ONE entry: every typed op reaches its tenant through
|
||||
// here, so admission, the bound, the reclaim and the model reload each have
|
||||
// exactly one site.
|
||||
func (rs *residency) of(t Tenant, log logger) (*resident, error) {
|
||||
for {
|
||||
r, wait := rs.lookup(t)
|
||||
if r != nil {
|
||||
r.touch()
|
||||
return r, nil
|
||||
}
|
||||
if wait != nil {
|
||||
<-wait // another request is opening this tenant's file; take its cell
|
||||
continue
|
||||
}
|
||||
return rs.build(t, log)
|
||||
}
|
||||
}
|
||||
|
||||
// lookup answers with the published cell, or with the channel to wait on when
|
||||
// another request is building it, or with neither — in which case the caller
|
||||
// holds the build claim and must call build.
|
||||
func (rs *residency) lookup(t Tenant) (*resident, chan struct{}) {
|
||||
rs.mu.Lock()
|
||||
defer rs.mu.Unlock()
|
||||
if r, held := rs.cells[t]; held {
|
||||
return r, nil
|
||||
}
|
||||
if ch, going := rs.building[t]; going {
|
||||
return nil, ch
|
||||
}
|
||||
rs.building[t] = make(chan struct{})
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// build opens and arms a cell OFF the registry and publishes it whole. Nothing
|
||||
// can observe a half-built cell, so nothing has to be undone if it fails: the
|
||||
// handle this call opened is the only one that could be closed, and closing it
|
||||
// is safe precisely because no other request has ever been handed it.
|
||||
func (rs *residency) build(t Tenant, log logger) (*resident, error) {
|
||||
defer rs.finish(t)
|
||||
|
||||
if err := rs.reserve(cellBytes, log); err != nil {
|
||||
log.Error("risk: refusing a new tenant — this node has no memory left and nothing idle to reclaim",
|
||||
"tenant", t.String(), "bytes", rs.bytes(), "bytes_max", memBytes())
|
||||
return nil, err
|
||||
}
|
||||
r := &resident{t: t, res: rs, touched: time.Now()}
|
||||
if err := rs.open(r); err != nil {
|
||||
rs.release(cellBytes)
|
||||
return nil, err
|
||||
}
|
||||
if err := rs.arm(r, log); err != nil {
|
||||
r.close()
|
||||
rs.release(cellBytes)
|
||||
return nil, err
|
||||
}
|
||||
|
||||
rs.mu.Lock()
|
||||
rs.cells[t] = r
|
||||
rs.mu.Unlock()
|
||||
return r, nil
|
||||
}
|
||||
|
||||
// finish publishes the build claim's completion to anyone waiting on it.
|
||||
func (rs *residency) finish(t Tenant) {
|
||||
rs.mu.Lock()
|
||||
ch := rs.building[t]
|
||||
delete(rs.building, t)
|
||||
rs.mu.Unlock()
|
||||
if ch != nil {
|
||||
close(ch)
|
||||
}
|
||||
}
|
||||
|
||||
// reserve takes n bytes of the node's budget, RECLAIMING tenants that have been
|
||||
// silent past the retire floor before it refuses.
|
||||
//
|
||||
// The order is the whole design. Reclaim first, refuse last: a node that turned
|
||||
// a newcomer away while holding cells nobody has touched for hours is denying
|
||||
// service to keep memory it is not using. But only cells past idleFloor may go,
|
||||
// and that threshold is the tenant's OWN silence — so no tenant's arrival can
|
||||
// ever cost a tenant that is working its rings, which is the difference between
|
||||
// reclaim and eviction.
|
||||
func (rs *residency) reserve(n int, log logger) error {
|
||||
rs.mu.Lock()
|
||||
if rs.held+n <= memBytes() {
|
||||
rs.held += n
|
||||
rs.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
rs.mu.Unlock()
|
||||
|
||||
if freed := rs.pressure(n, log); freed {
|
||||
rs.mu.Lock()
|
||||
if rs.held+n <= memBytes() {
|
||||
rs.held += n
|
||||
rs.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
rs.mu.Unlock()
|
||||
}
|
||||
|
||||
rs.mu.Lock()
|
||||
rs.refused, rs.refusedAt = rs.refused+1, time.Now()
|
||||
rs.mu.Unlock()
|
||||
return errFull
|
||||
}
|
||||
|
||||
// release gives n bytes back to the node's budget.
|
||||
func (rs *residency) release(n int) {
|
||||
rs.mu.Lock()
|
||||
rs.held -= n
|
||||
if rs.held < 0 {
|
||||
rs.held = 0
|
||||
}
|
||||
rs.mu.Unlock()
|
||||
}
|
||||
|
||||
// room is the node gate a tenant's rings ask before taking a new key. Same
|
||||
// budget, same reclaim, no refusal log: a key that does not fit is reported to
|
||||
// the tenant as `strained`, which is the honest word for it.
|
||||
func (rs *residency) room(n int) bool {
|
||||
rs.mu.Lock()
|
||||
if rs.held+n <= memBytes() {
|
||||
rs.held += n
|
||||
rs.mu.Unlock()
|
||||
return true
|
||||
}
|
||||
rs.mu.Unlock()
|
||||
if !rs.pressure(n, discard{}) {
|
||||
return false
|
||||
}
|
||||
rs.mu.Lock()
|
||||
defer rs.mu.Unlock()
|
||||
if rs.held+n <= memBytes() {
|
||||
rs.held += n
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// pressure retires cells silent past idleFloor until n bytes are free, and
|
||||
// reports whether it freed anything. Counted, so `reclaimed` on the probe tells
|
||||
// an operator the node is running on reclamation rather than on headroom.
|
||||
func (rs *residency) pressure(n int, log logger) bool {
|
||||
freed := false
|
||||
for _, r := range rs.idle(idleFloor) {
|
||||
if rs.retire(r, log) {
|
||||
freed = true
|
||||
rs.mu.Lock()
|
||||
rs.reclaimed++
|
||||
room := rs.held+n <= memBytes()
|
||||
rs.mu.Unlock()
|
||||
if room {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return freed
|
||||
}
|
||||
|
||||
// idle snapshots the cells silent for at least d, OLDEST FIRST.
|
||||
//
|
||||
// A SNAPSHOT, taken under rs.mu and walked outside it. Holding the registry lock
|
||||
// while taking each cell's lock is a cross-tenant latency coupling on a path the
|
||||
// package itself describes as sitting inside a card processor's authorization
|
||||
// window: one cold tenant's file I/O would stall every other tenant's decide.
|
||||
func (rs *residency) idle(d time.Duration) []*resident {
|
||||
now := time.Now()
|
||||
rs.mu.Lock()
|
||||
out := make([]*resident, 0, len(rs.cells))
|
||||
for _, r := range rs.cells {
|
||||
if now.Sub(r.idleFor()) >= d {
|
||||
out = append(out, r)
|
||||
}
|
||||
}
|
||||
rs.mu.Unlock()
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].idleFor().Before(out[j].idleFor()) })
|
||||
return out
|
||||
}
|
||||
|
||||
// idleFor is when this cell was last touched.
|
||||
func (r *resident) idleFor() time.Time {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return r.touched
|
||||
}
|
||||
|
||||
// touch records that a request resolved this tenant. The retire threshold is
|
||||
// measured from here, which is the fact the safety of closing its file rests on.
|
||||
func (r *resident) touch() {
|
||||
r.mu.Lock()
|
||||
r.touched = time.Now()
|
||||
r.mu.Unlock()
|
||||
}
|
||||
|
||||
// open resolves the tenant's file once. The ORG half is what cloud.OrgNamespace
|
||||
// takes — the deployment does its own brand scoping through DataDir, so handing
|
||||
// it the qualified key would put the brand in the name twice.
|
||||
//
|
||||
// It runs on an UNPUBLISHED cell, so no lock is taken across the file I/O:
|
||||
// creating and seeding an encrypted SQLite file under a lock a request needs is
|
||||
// how one cold tenant's first touch became every other tenant's latency.
|
||||
func (rs *residency) open(r *resident) error {
|
||||
ns, err := cloud.OrgNamespace(r.t.org(), "")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
db, err := cloud.OrgDB(rs.dataDir, ns, "risk")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := db.Exec(schema); err != nil {
|
||||
_ = db.Close()
|
||||
return err
|
||||
}
|
||||
if err := seed(db); err != nil {
|
||||
_ = db.Close()
|
||||
return err
|
||||
}
|
||||
// A search row still marked running was running in a process that no longer
|
||||
// exists — cloud deploys Recreate, so there is no other way for one to
|
||||
// survive. Left alone it would be read forever as "in progress", which is
|
||||
// the same bytes as a search that is about to answer and the opposite fact.
|
||||
if err := abandonSearches(db); err != nil {
|
||||
_ = db.Close()
|
||||
return err
|
||||
}
|
||||
// The decision log is pruned to its retention here as well as every
|
||||
// pruneEvery writes, so a tenant that writes fewer than that between
|
||||
// rollouts still comes back inside its budget.
|
||||
if err := prune(db); err != nil {
|
||||
_ = db.Close()
|
||||
return err
|
||||
}
|
||||
r.handle = db
|
||||
return nil
|
||||
}
|
||||
|
||||
// arm gives an unpublished resident its aggregates and its model, and restores
|
||||
// what the tenant had learned.
|
||||
//
|
||||
// THE RELOAD IS HERE AND ONLY HERE, which is what closes the silent-disarm
|
||||
// defect. The process used to latch "restored" in a map that outlived the model,
|
||||
// so a model dropped by the shared store's LRU was never reloaded — the tenant
|
||||
// scored nothing for the rest of the process's life while reporting only
|
||||
// "warming". A cell that has no model has no latch either, because the latch IS
|
||||
// the model.
|
||||
func (rs *residency) arm(r *resident, log logger) error {
|
||||
vel := aggregates()
|
||||
model, err := forest(anomaly.Config{}, vel)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
kept, disarmed := int64(0), false
|
||||
if snap, ok, err := readSnapshot(r.handle, r.t); err != nil {
|
||||
disarmed = true
|
||||
log.Error("risk: a tenant's learned state is on file and could not be read; the model is DISARMED, not warming",
|
||||
"tenant", r.t.String(), "err", err)
|
||||
} else if ok {
|
||||
kept = snap.Learned
|
||||
if err := model.Restore(snap); err != nil {
|
||||
disarmed = true
|
||||
log.Error("risk: a tenant's learned state was refused by the engine; the model is DISARMED, not warming",
|
||||
"tenant", r.t.String(), "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
r.vel, r.model, r.since = vel, model, time.Now().UTC()
|
||||
r.kept, r.disarmed = kept, disarmed
|
||||
return nil
|
||||
}
|
||||
|
||||
// retire releases ONE tenant that has been silent: its model is written to its
|
||||
// own file, then its aggregates, its caches and its file handle go and its cell
|
||||
// is dropped. Answers whether it did.
|
||||
//
|
||||
// IT IS NOT EVICTION AND THE DIFFERENCE IS THE WHOLE POINT: the trigger is the
|
||||
// retired tenant's own silence, so no tenant's traffic can ever cost another
|
||||
// tenant a ring. Nothing durable is lost — the model is snapshotted first, and
|
||||
// the rules, lists and decisions were always on the tenant's own file — so the
|
||||
// tenant's next request comes back with what it learned. The aggregates are the
|
||||
// one thing that does not survive, which is why `since` rides every decision
|
||||
// instead of the loss being left for someone to notice.
|
||||
//
|
||||
// CLOSING THE FILE IS SAFE WITHOUT A REFERENCE COUNT because the caller only
|
||||
// ever hands it cells idle past idleFloor, which is floored above the longest
|
||||
// any worker may hold one (see idleReclaim).
|
||||
func (rs *residency) retire(r *resident, log logger) bool {
|
||||
r.mu.Lock()
|
||||
if r.retired {
|
||||
r.mu.Unlock()
|
||||
return false
|
||||
}
|
||||
if err := keep(r.handle, r.t, r.model); err != nil {
|
||||
// Never drop a model we could not write down: the cell is HELD, and the
|
||||
// next sweep tries again. Losing learned state to a housekeeping pass is
|
||||
// exactly the silent disarm this file exists to prevent.
|
||||
r.mu.Unlock()
|
||||
log.Error("risk: a retiring tenant's learned state was not kept, so its cell is held", "tenant", r.t.String(), "err", err)
|
||||
return false
|
||||
}
|
||||
idleFor := time.Since(r.touched)
|
||||
cost := r.costLocked()
|
||||
r.retired = true
|
||||
r.releaseLocked()
|
||||
r.mu.Unlock()
|
||||
|
||||
rs.mu.Lock()
|
||||
if rs.cells[r.t] == r {
|
||||
delete(rs.cells, r.t)
|
||||
}
|
||||
rs.mu.Unlock()
|
||||
rs.release(cost)
|
||||
|
||||
log.Info("risk: a tenant was retired after its own idleness; its learned state is on its own file",
|
||||
"tenant", r.t.String(), "idle", idleFor.String())
|
||||
return true
|
||||
}
|
||||
|
||||
// costLocked is what this cell is charged to the node's budget. Caller holds
|
||||
// r.mu.
|
||||
func (r *resident) costLocked() int {
|
||||
n := cellBytes + r.govern
|
||||
if r.vel != nil {
|
||||
n += r.vel.keys() * bytesPerKey()
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// bytes is what this cell costs the node right now.
|
||||
func (r *resident) bytes() int {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return r.costLocked()
|
||||
}
|
||||
|
||||
// releaseLocked drops everything this cell holds. Caller holds r.mu.
|
||||
func (r *resident) releaseLocked() {
|
||||
r.vel, r.model = nil, nil
|
||||
r.loaded, r.rules, r.lists, r.sups, r.govern = false, nil, nil, nil, 0
|
||||
r.agency.clear()
|
||||
if r.handle != nil {
|
||||
_ = r.handle.Close()
|
||||
r.handle = nil
|
||||
}
|
||||
}
|
||||
|
||||
// close drops an UNPUBLISHED cell. The only caller is the build that made it,
|
||||
// which is what makes closing the handle safe: nobody else has ever held it.
|
||||
func (r *resident) close() {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.releaseLocked()
|
||||
}
|
||||
|
||||
// sweep is the background retire. It runs on a timer so memory comes back from a
|
||||
// silent tenant without waiting for a busy one to need it.
|
||||
func (rs *residency) sweep(log logger) {
|
||||
for _, r := range rs.idle(idleReclaim()) {
|
||||
rs.retire(r, log)
|
||||
}
|
||||
}
|
||||
|
||||
// tenants lists the tenants this process holds, in a stable order.
|
||||
func (rs *residency) tenants() []Tenant {
|
||||
rs.mu.Lock()
|
||||
defer rs.mu.Unlock()
|
||||
out := make([]Tenant, 0, len(rs.cells))
|
||||
for t := range rs.cells {
|
||||
out = append(out, t)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i] < out[j] })
|
||||
return out
|
||||
}
|
||||
|
||||
// bytes is what every resident tenant costs this node right now.
|
||||
func (rs *residency) bytes() int {
|
||||
rs.mu.Lock()
|
||||
defer rs.mu.Unlock()
|
||||
return rs.held
|
||||
}
|
||||
|
||||
// count reports the saturation view: how many tenants this process holds, how
|
||||
// many admissions it has refused, how many cells it has reclaimed under memory
|
||||
// pressure, and when the last refusal was. All four are on the probe — a
|
||||
// control that runs out of room quietly is a control nobody knows is off.
|
||||
func (rs *residency) count() (tenants int, refused, reclaimed int64, at time.Time) {
|
||||
rs.mu.Lock()
|
||||
defer rs.mu.Unlock()
|
||||
return len(rs.cells), rs.refused, rs.reclaimed, rs.refusedAt
|
||||
}
|
||||
|
||||
// strained is how many resident tenants are reading partial rings.
|
||||
func (rs *residency) strained() int {
|
||||
rs.mu.Lock()
|
||||
cells := make([]*resident, 0, len(rs.cells))
|
||||
for _, r := range rs.cells {
|
||||
cells = append(cells, r)
|
||||
}
|
||||
rs.mu.Unlock()
|
||||
n := 0
|
||||
for _, r := range cells {
|
||||
if r.strained() {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// close snapshots every armed tenant and closes every file. It is what makes a
|
||||
// rollout survivable: cloud deploys Recreate at one replica, so every deploy
|
||||
// drops the process, and a model that comes back with nothing learned declines to
|
||||
// score for its whole warm period.
|
||||
func (rs *residency) close(log logger) (kept, failed int) {
|
||||
rs.mu.Lock()
|
||||
cells := make(map[Tenant]*resident, len(rs.cells))
|
||||
for t, r := range rs.cells {
|
||||
cells[t] = r
|
||||
}
|
||||
rs.cells = map[Tenant]*resident{}
|
||||
rs.held = 0
|
||||
rs.mu.Unlock()
|
||||
|
||||
for t, r := range cells {
|
||||
r.mu.Lock()
|
||||
if r.model != nil {
|
||||
if err := keep(r.handle, r.t, r.model); err != nil {
|
||||
failed++
|
||||
log.Error("risk: a tenant's learned state was not kept", "tenant", t.String(), "err", err)
|
||||
} else {
|
||||
kept++
|
||||
}
|
||||
}
|
||||
r.retired = true
|
||||
r.releaseLocked()
|
||||
r.mu.Unlock()
|
||||
}
|
||||
return kept, failed
|
||||
}
|
||||
|
||||
// ── what an op reads off a resident ─────────────────────────────────────────
|
||||
|
||||
// file is the tenant's own file handle, read under this cell's own lock.
|
||||
//
|
||||
// IT ANSWERS WITH AN ERROR RATHER THAN WITH A HANDLE NOBODY MAY USE, and that is
|
||||
// the whole of it. `(*sql.DB)(nil).QueryRow` locks a nil mutex, so a cell whose
|
||||
// file has been closed must never hand one out: on a one-replica pod that panic
|
||||
// is every tenant's outage. Two things close a cell — retire, after the tenant's
|
||||
// own idleFloor of silence, and teardown, which closes EVERY cell the instant
|
||||
// SIGTERM lands. The second has no idle requirement and cloud deploys Recreate,
|
||||
// so a request holding a cell it resolved a millisecond ago can find the handle
|
||||
// gone on every single rollout.
|
||||
//
|
||||
// Returning (handle, error) is what makes the nil unrepresentable downstream: an
|
||||
// op cannot obtain the file without also obtaining the reason it has none, and
|
||||
// the check lives at the ONE door (tenantState) instead of at 27 call sites that
|
||||
// each have to remember it.
|
||||
func (r *resident) file() (*sql.DB, error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if r.handle == nil {
|
||||
return nil, errRetired
|
||||
}
|
||||
return r.handle, nil
|
||||
}
|
||||
|
||||
// arms returns the tenant's two in-memory planes and the instant they started.
|
||||
// A disarmed resident cannot be reached this way: `of` arms before publishing.
|
||||
func (r *resident) arms() (*rings, *anomaly.Store, time.Time) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return r.vel, r.model, r.since
|
||||
}
|
||||
|
||||
// recorded says one decision was written, and answers whether the log is due
|
||||
// for its prune. Counted per cell, so nothing on the hot path counts rows.
|
||||
func (r *resident) recorded() bool {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.writes++
|
||||
return r.writes%pruneEvery == 0
|
||||
}
|
||||
|
||||
// alone runs f while this tenant holds the ONE slot for a synchronous replay of
|
||||
// its own history, and refuses rather than queues when it is taken.
|
||||
//
|
||||
// THE BOUND IS CONCURRENCY, NOT PRICE. /v1/risk/simulate reads up to 5,000 of
|
||||
// this tenant's decisions and materialises them, and it is deliberately UNPRICED
|
||||
// because it is the rehearsal a tenant runs before activating a rule — charging
|
||||
// for the safe path discourages the safe path. But nothing bounded how many ran
|
||||
// at once, so N connections held N x 5,000 rows, which appears in no ceiling.
|
||||
// One at a time per tenant is the honest bound: the rehearsal stays free, and
|
||||
// the memory it can reach is a fixed multiple of what one replay costs.
|
||||
func (r *resident) alone(f func() error) error {
|
||||
if !r.replaying.CompareAndSwap(false, true) {
|
||||
return zip.Errorf(409, "this tenant already has a replay of its own history running; it reads up to %d decisions and runs one at a time", searchHistoryMax)
|
||||
}
|
||||
defer r.replaying.Store(false)
|
||||
return f()
|
||||
}
|
||||
|
||||
// record writes an observation onto this tenant's rings, through this tenant's
|
||||
// own cardinality gate and the node's byte gate. It is the ONLY write path into
|
||||
// the aggregates, so a key that was never priced cannot exist.
|
||||
func (r *resident) record(o observation) {
|
||||
vel, _, _ := r.arms()
|
||||
if vel == nil {
|
||||
return
|
||||
}
|
||||
vel.record(r.t, o, r.room)
|
||||
}
|
||||
|
||||
// room is this cell's view of the node gate. A cell with no residency behind it
|
||||
// — a search sandbox replaying history — is bounded by its own maxKeys and
|
||||
// charges the node nothing, because the node already paid for the live rings the
|
||||
// sandbox is a copy of.
|
||||
func (r *resident) room(n int) bool {
|
||||
if r.res == nil {
|
||||
return true
|
||||
}
|
||||
return r.res.room(n)
|
||||
}
|
||||
|
||||
// strained reports that this tenant's rings stopped tracking its own traffic:
|
||||
// a key it needed was refused, by its own cardinality bound or by the node's
|
||||
// memory. Either way a count it reads may be an under-count of its own traffic,
|
||||
// and a rule written on that count is measuring a partial ring.
|
||||
func (r *resident) strained() bool {
|
||||
vel, _, _ := r.arms()
|
||||
return vel != nil && vel.strained()
|
||||
}
|
||||
|
||||
// grade turns THE MODEL'S OWN refusal into the word that is true of it, against
|
||||
// what this tenant is known to have learned. Warming and disarmed are the same
|
||||
// bytes on the wire and opposite facts about the system, so they are never the
|
||||
// same word.
|
||||
//
|
||||
// It grades the model's reason and nothing else. An earlier cut ran it over the
|
||||
// FINISHED refusal, after every reason had been merged into one word — so a
|
||||
// decision that was also short of something ranked higher never reached the
|
||||
// warming branch, and a disarmed model went out under the other reason's name.
|
||||
// A grader that only fires when nothing else went wrong is not a grader.
|
||||
func (r *resident) grade(reason string) string {
|
||||
if reason != RefusalWarming {
|
||||
return reason
|
||||
}
|
||||
r.mu.Lock()
|
||||
disarmed, kept, model := r.disarmed, r.kept, r.model
|
||||
r.mu.Unlock()
|
||||
if disarmed {
|
||||
return RefusalDisarmed
|
||||
}
|
||||
if model != nil && kept > 0 && model.State(r.t.String()).Learned < kept {
|
||||
return RefusalDisarmed
|
||||
}
|
||||
return reason
|
||||
}
|
||||
|
||||
// governance is the tenant's rule set, list membership, suppressions and live
|
||||
// switch, loaded ONCE per change.
|
||||
//
|
||||
// Three unbounded SELECTs on every authorization was the defect; a cache with no
|
||||
// invalidation would be a worse one, so every writer on this tenant's file calls
|
||||
// dirty and both live under this mutex. cloud runs one replica of this app and
|
||||
// the shard router pins an org to one pod, so this process is the only writer of
|
||||
// this file — the cache cannot be stale with respect to a writer it cannot see.
|
||||
//
|
||||
// IT IS PRICED, NOT GATED. What it loads is bounded per tenant by the write-time
|
||||
// byte budgets (governMemo), and it is measured onto the node's total the moment
|
||||
// it lands — but it is never refused. Refusing to load a tenant's rules would
|
||||
// disarm that tenant's controls to save memory, which is the one degradation
|
||||
// this package will not do; the node answers instead by admitting no new cells
|
||||
// and no new counters until the total comes down.
|
||||
func (r *resident) governance() ([]rule, map[string]map[string]bool, []suppression, bool, error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if r.loaded {
|
||||
return r.rules, r.lists, r.sups, r.live, nil
|
||||
}
|
||||
if r.handle == nil {
|
||||
// Retired out from under a request. idleFloor is what makes this
|
||||
// unreachable — nothing in this process holds a cell for sixteen minutes
|
||||
// — so say so loudly rather than dereference a nil handle and take the
|
||||
// pod down with every tenant on it.
|
||||
return nil, nil, nil, false, errRetired
|
||||
}
|
||||
rules, err := loadRules(r.handle)
|
||||
if err != nil {
|
||||
return nil, nil, nil, false, err
|
||||
}
|
||||
lists, err := loadLists(r.handle)
|
||||
if err != nil {
|
||||
return nil, nil, nil, false, err
|
||||
}
|
||||
sups, err := loadSuppressions(r.handle)
|
||||
if err != nil {
|
||||
return nil, nil, nil, false, err
|
||||
}
|
||||
r.rules, r.lists, r.sups = rules, lists, sups
|
||||
r.live = mode(r.handle) == "live"
|
||||
r.loaded = true
|
||||
was := r.govern
|
||||
r.govern = governBytesOf(rules, lists, sups)
|
||||
if r.res != nil {
|
||||
r.res.charge(r.govern - was)
|
||||
}
|
||||
return r.rules, r.lists, r.sups, r.live, nil
|
||||
}
|
||||
|
||||
// governBytesOf measures a loaded governance set. Each row is priced at what it
|
||||
// costs in the maps the authorization path holds it in, at the caps bound.go
|
||||
// publishes — so the node's total is the sum of real rows and never of budgets
|
||||
// nobody is using.
|
||||
func governBytesOf(rules []rule, lists map[string]map[string]bool, sups []suppression) int {
|
||||
n := len(rules)*ruleMax + len(sups)*supMax
|
||||
for _, l := range lists {
|
||||
n += len(l) * entryMax
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// charge moves the node's running total by delta, which may be negative.
|
||||
func (rs *residency) charge(delta int) {
|
||||
if delta == 0 {
|
||||
return
|
||||
}
|
||||
rs.mu.Lock()
|
||||
rs.held += delta
|
||||
if rs.held < 0 {
|
||||
rs.held = 0
|
||||
}
|
||||
rs.mu.Unlock()
|
||||
}
|
||||
|
||||
// reload reinstates this tenant's learned state from its own pinned snapshot,
|
||||
// over whatever the live model holds.
|
||||
//
|
||||
// It is what POST /v1/risk/restore does, and it is also the honest recovery from
|
||||
// a disarmed model: a tenant told its control is off can put it back on with the
|
||||
// state it kept. A snapshot belonging to another tenant is refused twice — here,
|
||||
// by the tenant that asked, and again inside the engine.
|
||||
func (r *resident) reload() error {
|
||||
r.mu.Lock()
|
||||
db, model := r.handle, r.model
|
||||
r.mu.Unlock()
|
||||
if model == nil {
|
||||
return fmt.Errorf("risk: this tenant holds no model to restore into")
|
||||
}
|
||||
snap, ok, err := readSnapshot(db, r.t)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !ok {
|
||||
return fmt.Errorf("risk: this tenant has no pinned snapshot")
|
||||
}
|
||||
if err := model.Restore(snap); err != nil {
|
||||
return err
|
||||
}
|
||||
r.mu.Lock()
|
||||
r.kept, r.disarmed = snap.Learned, false
|
||||
r.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
|
||||
// dirty drops the governance cache. Called by every write to a governed plane,
|
||||
// beside the write, so a new plane cannot be added without the author meeting
|
||||
// this line.
|
||||
func (r *resident) dirty() {
|
||||
r.mu.Lock()
|
||||
was := r.govern
|
||||
r.loaded, r.rules, r.lists, r.sups, r.govern = false, nil, nil, nil, 0
|
||||
res := r.res
|
||||
r.mu.Unlock()
|
||||
if res != nil {
|
||||
res.charge(-was)
|
||||
}
|
||||
}
|
||||
|
||||
// ── the durable half of the model ───────────────────────────────────────────
|
||||
|
||||
// keep writes a tenant's learned state into its own file. Nothing learned is not
|
||||
// a failure — a tenant that has scored nothing has nothing to keep.
|
||||
func keep(db *sql.DB, t Tenant, model *anomaly.Store) error {
|
||||
if db == nil || model == nil {
|
||||
return nil
|
||||
}
|
||||
snap, ok := model.Snapshot(t.String())
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
body, err := encodeSnapshot(snap)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return putModel(db, snapshotKey, body)
|
||||
}
|
||||
|
||||
// readSnapshot reads a tenant's kept state back.
|
||||
//
|
||||
// The snapshot's tenant must be the tenant asking for it. The engine checks this
|
||||
// too; checking here as well means a restore of A's file into B's request is
|
||||
// refused at the boundary that knows who asked.
|
||||
func readSnapshot(db *sql.DB, t Tenant) (anomaly.Snapshot, bool, error) {
|
||||
body, err := getModel(db, snapshotKey)
|
||||
if err != nil {
|
||||
var he *zip.HTTPError
|
||||
if errors.As(err, &he) && he.Status == 404 {
|
||||
return anomaly.Snapshot{}, false, nil // no snapshot is the normal first run
|
||||
}
|
||||
return anomaly.Snapshot{}, false, err
|
||||
}
|
||||
snap, err := decodeSnapshot(body)
|
||||
if err != nil {
|
||||
return anomaly.Snapshot{}, false, err
|
||||
}
|
||||
if snap.OrgID != t.String() {
|
||||
return anomaly.Snapshot{}, false, fmt.Errorf("risk: snapshot belongs to another tenant")
|
||||
}
|
||||
return snap, true, nil
|
||||
}
|
||||
|
||||
// errRetired is what a cell answers when its file has already been closed. A
|
||||
// retry lands on a fresh cell, so it is a 503 and not a 500.
|
||||
var errRetired = zip.Errorf(503, "this tenant's state was reclaimed while the request was in flight; retry")
|
||||
|
||||
// logger is the slice of the service's logger this file uses. Narrow on purpose:
|
||||
// residency is reached from teardown and from a background sweep as well as from
|
||||
// a request, and a package that took the whole service would be reaching for
|
||||
// state it has no business touching.
|
||||
type logger interface {
|
||||
Info(msg string, args ...any)
|
||||
Error(msg string, args ...any)
|
||||
}
|
||||
|
||||
// discard is the logger the memory gate uses. A key that does not fit is
|
||||
// reported to the tenant as `strained` and counted on the probe; logging a line
|
||||
// per refused counter would be a log entry per request on exactly the node that
|
||||
// has no room to spare.
|
||||
type discard struct{}
|
||||
|
||||
func (discard) Info(string, ...any) {}
|
||||
func (discard) Error(string, ...any) {}
|
||||
@@ -0,0 +1,421 @@
|
||||
package risk
|
||||
|
||||
// resident_test.go covers the residency: admission under a bound, reclaim driven
|
||||
// only by a tenant's OWN idleness, and the reload that makes a disarmed model
|
||||
// impossible to mistake for a warming one.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/hanzoai/cloud"
|
||||
luxlog "github.com/luxfi/log"
|
||||
)
|
||||
|
||||
// node is the RISK_MEMORY value that holds n cells with a little counting room
|
||||
// each. The bound is BYTES, so a test that wants "room for n tenants" has to say
|
||||
// what those tenants hold — how many a node serves is an OUTCOME of that, which
|
||||
// is the whole point of the fix.
|
||||
func node(n int) string { return fmt.Sprint(n * (cellBytes + 64*bytesPerKey())) }
|
||||
|
||||
// TestAFullNodeRefusesRatherThanEvicts pins the admission rule.
|
||||
//
|
||||
// When a pod is out of room there are two things it can do: turn the newcomer
|
||||
// away, or take an incumbent's state. The second is the defect — silent, and
|
||||
// aimed at whoever happens to be quietest — so this asserts the first, and
|
||||
// asserts the incumbent is untouched afterwards.
|
||||
func TestAFullNodeRefusesRatherThanEvicts(t *testing.T) {
|
||||
t.Setenv(envMemory, node(1))
|
||||
_, s := wireApp(t)
|
||||
|
||||
a := Tenant("hanzo/acme")
|
||||
first, err := s.State.res.of(a, s.Log)
|
||||
if err != nil {
|
||||
t.Fatalf("the first tenant was refused: %v", err)
|
||||
}
|
||||
vel, model, _ := first.arms()
|
||||
first.record(observation{at: time.Now(), kind: "account", subject: "keep-me", amount: 1})
|
||||
|
||||
if _, err := s.State.res.of(Tenant("hanzo/beta"), s.Log); err == nil {
|
||||
t.Fatal("a second tenant was admitted onto a full node — something was taken to make room")
|
||||
}
|
||||
|
||||
// The incumbent still has everything.
|
||||
vel2, model2, _ := first.arms()
|
||||
if vel2 != vel || model2 != model {
|
||||
t.Fatal("the incumbent's planes were replaced by the refused admission")
|
||||
}
|
||||
if vel2.keys() == 0 {
|
||||
t.Fatal("the incumbent's aggregates were dropped to make room for a tenant that was refused anyway")
|
||||
}
|
||||
|
||||
// And the refusal is LOUD: the probe goes degraded and names it.
|
||||
if _, refused, _, _ := s.State.res.count(); refused == 0 {
|
||||
t.Fatal("the refusal was not counted, so nothing pages an operator about a node that is out of room")
|
||||
}
|
||||
}
|
||||
|
||||
// TestTheCeilingIsAWorkingSetAndNotAHighWaterMark.
|
||||
//
|
||||
// THE DEFECT: reclaim dropped a tenant's aggregates but KEPT its cell, and the
|
||||
// admission bound counted cells. So the total only ever went up: once the node's
|
||||
// distinct tenants had passed through, the next one was refused for the life of
|
||||
// the process — with the pod completely idle and nothing to reclaim. A ceiling
|
||||
// that a tenant can raise by leaving is not a ceiling, it is a fuse.
|
||||
func TestTheCeilingIsAWorkingSetAndNotAHighWaterMark(t *testing.T) {
|
||||
t.Setenv(envMemory, node(2))
|
||||
_, s := wireApp(t)
|
||||
|
||||
for _, name := range []Tenant{"hanzo/one", "hanzo/two"} {
|
||||
if _, err := s.State.res.of(name, s.Log); err != nil {
|
||||
t.Fatalf("%s was refused: %v", name, err)
|
||||
}
|
||||
}
|
||||
if n, _, _, _ := s.State.res.count(); n != 2 {
|
||||
t.Fatalf("the node holds %d tenants, want 2", n)
|
||||
}
|
||||
// Both go silent. Retirement is driven by their OWN idleness.
|
||||
for _, name := range []Tenant{"hanzo/one", "hanzo/two"} {
|
||||
r := resOf(t, s, name)
|
||||
r.mu.Lock()
|
||||
r.touched = time.Now().Add(-24 * time.Hour)
|
||||
r.mu.Unlock()
|
||||
}
|
||||
s.State.res.sweep(s.Log)
|
||||
if n, _, _, _ := s.State.res.count(); n != 0 {
|
||||
t.Fatalf("after both tenants went silent the node still holds %d cells — the map only grows, so the ceiling is a fuse", n)
|
||||
}
|
||||
|
||||
// And the room really is usable: two NEW tenants are admitted.
|
||||
for _, name := range []Tenant{"hanzo/three", "hanzo/four"} {
|
||||
if _, err := s.State.res.of(name, s.Log); err != nil {
|
||||
t.Fatalf("%s was refused after the node emptied: %v", name, err)
|
||||
}
|
||||
}
|
||||
if _, refused, _, _ := s.State.res.count(); refused != 0 {
|
||||
t.Fatalf("%d admissions were refused by a node with room", refused)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRetirementLosesNothingDurable. Retiring closes a tenant's file, so the
|
||||
// test that matters is not that memory came back but that the tenant did: its
|
||||
// rules, its decisions and its learned state must all still be there on the
|
||||
// request that brings it home.
|
||||
func TestRetirementLosesNothingDurable(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
tn := Tenant("hanzo/acme")
|
||||
|
||||
code, body := reqAdmin(t, app, http.MethodPost, "/v1/risk/rules", "acme", "u_acme",
|
||||
`{"rule":{"name":"keep me","stage":"signup","action":"review","weight":0.5,"enabled":true,`+
|
||||
`"all":[{"field":"subject.kind","op":"eq","value":"account"}]}}`)
|
||||
if code != http.StatusCreated {
|
||||
t.Fatalf("rule = %d %s", code, body)
|
||||
}
|
||||
feed(t, s, tn, 60)
|
||||
before := resOf(t, s, tn)
|
||||
_, model, _ := before.arms()
|
||||
learned := model.State(tn.String()).Learned
|
||||
if learned == 0 {
|
||||
t.Fatal("the tenant learned nothing, so a reload proves nothing")
|
||||
}
|
||||
|
||||
before.mu.Lock()
|
||||
before.touched = time.Now().Add(-24 * time.Hour)
|
||||
before.mu.Unlock()
|
||||
s.State.res.sweep(s.Log)
|
||||
if n, _, _, _ := s.State.res.count(); n != 0 {
|
||||
t.Fatalf("the silent tenant was not retired (%d cells)", n)
|
||||
}
|
||||
|
||||
// It comes home on its next request, through the ordinary door.
|
||||
code, body = req(t, app, http.MethodGet, "/v1/risk/rules", "acme", "u_acme", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("a retired tenant's rules read %d %s — retiring closed a file it could not reopen", code, body)
|
||||
}
|
||||
if !strings.Contains(string(body), "keep me") {
|
||||
t.Fatalf("the rule did not survive the retirement: %s", body)
|
||||
}
|
||||
_, back, _ := resOf(t, s, tn).arms()
|
||||
if got := back.State(tn.String()).Learned; got != learned {
|
||||
t.Fatalf("the retired tenant came back having learned %d of %d — retirement is losing state, which is eviction with a nicer name", got, learned)
|
||||
}
|
||||
}
|
||||
|
||||
// TestConcurrentAdmissionDoesNotStall is the regression for a lock order that
|
||||
// deadlocked.
|
||||
//
|
||||
// THE DEFECT: arming took the residency lock and then a cell's lock, and called
|
||||
// the sweep — which locks EVERY cell — from inside both. The first tenant to arm
|
||||
// while the node was at its ceiling took its own cell's lock twice and the whole
|
||||
// process stopped deciding, for everyone, with no error and no log.
|
||||
//
|
||||
// Concurrency is the only way to catch it and `go test -race -timeout` is the
|
||||
// assertion: a deadlock here does not fail, it hangs.
|
||||
func TestConcurrentAdmissionDoesNotStall(t *testing.T) {
|
||||
t.Setenv(envMemory, node(8))
|
||||
_, s := wireApp(t)
|
||||
|
||||
const workers, each = 16, 12
|
||||
var wg sync.WaitGroup
|
||||
var admitted, refused atomic.Int64
|
||||
for w := 0; w < workers; w++ {
|
||||
wg.Add(1)
|
||||
go func(w int) {
|
||||
defer wg.Done()
|
||||
for i := 0; i < each; i++ {
|
||||
// Deliberately more distinct tenants than the ceiling, so the
|
||||
// admission path runs its sweep-then-refuse arm under contention.
|
||||
tn := Tenant(fmt.Sprintf("hanzo/t%d", (w*each+i)%12))
|
||||
if _, err := s.State.res.of(tn, s.Log); err != nil {
|
||||
refused.Add(1)
|
||||
continue
|
||||
}
|
||||
admitted.Add(1)
|
||||
}
|
||||
}(w)
|
||||
}
|
||||
done := make(chan struct{})
|
||||
go func() { wg.Wait(); close(done) }()
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(60 * time.Second):
|
||||
t.Fatal("concurrent admission did not finish — the residency deadlocked, which stops every tenant's decisions at once")
|
||||
}
|
||||
if admitted.Load() == 0 {
|
||||
t.Fatal("nothing was admitted at all")
|
||||
}
|
||||
// Whatever the split, the node never holds more than it promised.
|
||||
if held, max := s.State.res.bytes(), memBytes(); held > max {
|
||||
t.Fatalf("the node holds %d B against a budget of %d B", held, max)
|
||||
}
|
||||
n, _, reclaimed, _ := s.State.res.count()
|
||||
t.Logf("admitted %d, refused %d, reclaimed %d, resident %d in %d B of %d B",
|
||||
admitted.Load(), refused.Load(), reclaimed, n, s.State.res.bytes(), memBytes())
|
||||
}
|
||||
|
||||
// TestReclaimIsDrivenOnlyByATenantsOwnIdleness pins the difference between
|
||||
// reclaim and eviction. The trigger must be the reclaimed tenant's own silence —
|
||||
// never another tenant's arrival, and never memory pressure, because both make
|
||||
// one tenant's traffic the reason another tenant's control weakened.
|
||||
func TestReclaimIsDrivenOnlyByATenantsOwnIdleness(t *testing.T) {
|
||||
_, s := wireApp(t)
|
||||
a, b := Tenant("hanzo/acme"), Tenant("hanzo/beta")
|
||||
|
||||
ra, rb := resOf(t, s, a), resOf(t, s, b)
|
||||
ra.record(observation{at: time.Now(), kind: "account", subject: "a1", amount: 1})
|
||||
rb.record(observation{at: time.Now(), kind: "account", subject: "b1", amount: 1})
|
||||
|
||||
// A is made to look silent; B was touched just now.
|
||||
ra.mu.Lock()
|
||||
ra.touched = time.Now().Add(-24 * time.Hour)
|
||||
ra.mu.Unlock()
|
||||
|
||||
for _, r := range s.State.res.idle(time.Hour) {
|
||||
s.State.res.retire(r, s.Log)
|
||||
}
|
||||
|
||||
if v, m, _ := ra.arms(); v != nil || m != nil {
|
||||
t.Fatal("the silent tenant was not reclaimed")
|
||||
}
|
||||
if v, _, _ := rb.arms(); v == nil {
|
||||
t.Fatal("a tenant that was active a moment ago had its aggregates reclaimed — the trigger is not its own idleness")
|
||||
}
|
||||
if v, _, _ := rb.arms(); v.keys() == 0 {
|
||||
t.Fatal("the active tenant's counters are gone")
|
||||
}
|
||||
}
|
||||
|
||||
// TestAReclaimedTenantComesBackWithWhatItLearned is the regression for the
|
||||
// silent-disarm defect.
|
||||
//
|
||||
// THE DEFECT: the process latched "this tenant has been restored" in a map that
|
||||
// OUTLIVED the model. When the shared store's LRU dropped a tenant's model, the
|
||||
// latch still said restored, so it was never reloaded — the tenant scored nothing
|
||||
// for the rest of the process's life while reporting only "warming".
|
||||
//
|
||||
// THE FIX IS STRUCTURAL: the latch IS the model. A cell with no model has no
|
||||
// latch, so the only way to come back is through the path that reloads.
|
||||
func TestAReclaimedTenantComesBackWithWhatItLearned(t *testing.T) {
|
||||
_, s := wireApp(t)
|
||||
tn := Tenant("hanzo/acme")
|
||||
|
||||
feed(t, s, tn, 60)
|
||||
r := resOf(t, s, tn)
|
||||
_, model, _ := r.arms()
|
||||
learned := model.State(tn.String()).Learned
|
||||
if learned == 0 {
|
||||
t.Fatal("the tenant learned nothing, so this test cannot tell a reload from a fresh start")
|
||||
}
|
||||
if err := keep(dbOf(t, r), tn, model); err != nil {
|
||||
t.Fatalf("keep: %v", err)
|
||||
}
|
||||
|
||||
// Reclaim it, exactly as the idle sweep would.
|
||||
r.mu.Lock()
|
||||
r.touched = time.Now().Add(-24 * time.Hour)
|
||||
r.mu.Unlock()
|
||||
for _, cell := range s.State.res.idle(time.Hour) {
|
||||
s.State.res.retire(cell, s.Log)
|
||||
}
|
||||
if v, _, _ := r.arms(); v != nil {
|
||||
t.Fatal("reclaim did not disarm")
|
||||
}
|
||||
|
||||
// It comes back with what it knew.
|
||||
back := resOf(t, s, tn)
|
||||
_, model2, _ := back.arms()
|
||||
if model2 == nil {
|
||||
t.Fatal("the tenant did not re-arm")
|
||||
}
|
||||
if got := model2.State(tn.String()).Learned; got != learned {
|
||||
t.Fatalf("the model came back having learned %d of %d — a reclaimed tenant is silently back to warming", got, learned)
|
||||
}
|
||||
if back.grade(RefusalWarming) == RefusalDisarmed {
|
||||
t.Fatal("a correctly reloaded tenant is being reported as disarmed")
|
||||
}
|
||||
}
|
||||
|
||||
// TestLostLearnedStateIsCalledDisarmedAndNotWarming pins the loud half.
|
||||
//
|
||||
// Warming is a control coming up. Disarmed is a control that is OFF. They are the
|
||||
// same bytes on the wire unless they are different words, and reporting the second
|
||||
// as the first is exactly how a control stays off with nobody looking.
|
||||
func TestLostLearnedStateIsCalledDisarmedAndNotWarming(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
deps := cloud.Deps{Logger: luxlog.New("risktest"), DataDir: dir, Brand: "hanzo"}
|
||||
|
||||
// First process: learn something and pin it.
|
||||
one, err := build(deps)
|
||||
if err != nil {
|
||||
t.Fatalf("build: %v", err)
|
||||
}
|
||||
tn := Tenant("hanzo/acme")
|
||||
feed(t, one, tn, 60)
|
||||
r := resOf(t, one, tn)
|
||||
_, model, _ := r.arms()
|
||||
snap, ok := model.Snapshot(tn.String())
|
||||
if !ok {
|
||||
t.Fatal("the tenant learned nothing, so there is no state to lose")
|
||||
}
|
||||
_, _ = one.State.res.close(one.Log)
|
||||
one.State.runs.stop()
|
||||
|
||||
// Damage the pinned state the way a shape change or a bad write would: the
|
||||
// row is there and says a great deal was learned, and the engine will refuse
|
||||
// it. The tenant HAD a model and the next process cannot have it.
|
||||
//
|
||||
// AFTER the shutdown, deliberately: shutdown's whole job is to snapshot every
|
||||
// resident tenant, so damaging the file first would simply be overwritten by
|
||||
// the good state on the way down — and the test would then be asserting that
|
||||
// a healthy model is disarmed, which it is not.
|
||||
snap.Digest = "not-this-shape"
|
||||
body, err := encodeSnapshot(snap)
|
||||
if err != nil {
|
||||
t.Fatalf("encode: %v", err)
|
||||
}
|
||||
db, err := cloud.OrgDB(dir, cloud.MustOrgNamespace(tn.org(), ""), "risk")
|
||||
if err != nil {
|
||||
t.Fatalf("reopen: %v", err)
|
||||
}
|
||||
if err := putModel(db, snapshotKey, body); err != nil {
|
||||
t.Fatalf("putModel: %v", err)
|
||||
}
|
||||
if err := db.Close(); err != nil {
|
||||
t.Fatalf("close: %v", err)
|
||||
}
|
||||
|
||||
// Second process over the same directory — the rollout.
|
||||
two, err := build(deps)
|
||||
if err != nil {
|
||||
t.Fatalf("build: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _, _ = two.State.res.close(two.Log); two.State.runs.stop() })
|
||||
back := resOf(t, two, tn)
|
||||
if got := back.grade(RefusalWarming); got != RefusalDisarmed {
|
||||
t.Fatalf("a model whose learned state was refused reports %q — a control that is OFF is reporting itself as one that is coming up", got)
|
||||
}
|
||||
if got := back.grade(RefusalUnidentified); got != RefusalUnidentified {
|
||||
t.Fatalf("grading rewrote an unrelated refusal to %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestARolloutDoesNotSilentlyResetEveryTenant pins the durability contract. cloud
|
||||
// deploys strategy Recreate at one replica, so every deploy drops the process; a
|
||||
// model that comes back with nothing learned declines to score for its whole warm
|
||||
// period, and reads as clean to anything that does not check the refusal.
|
||||
func TestARolloutDoesNotSilentlyResetEveryTenant(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
deps := cloud.Deps{Logger: luxlog.New("risktest"), DataDir: dir, Brand: "hanzo"}
|
||||
tn := Tenant("hanzo/acme")
|
||||
|
||||
one, err := build(deps)
|
||||
if err != nil {
|
||||
t.Fatalf("build: %v", err)
|
||||
}
|
||||
feed(t, one, tn, 60)
|
||||
_, model, _ := resOf(t, one, tn).arms()
|
||||
learned := model.State(tn.String()).Learned
|
||||
if learned == 0 {
|
||||
t.Fatal("nothing was learned, so the restart proves nothing")
|
||||
}
|
||||
// Shutdown is the snapshot: nothing else runs on a rollout.
|
||||
kept, failed := one.State.res.close(one.Log)
|
||||
one.State.runs.stop()
|
||||
if kept != 1 || failed != 0 {
|
||||
t.Fatalf("shutdown kept %d and lost %d models", kept, failed)
|
||||
}
|
||||
|
||||
two, err := build(deps)
|
||||
if err != nil {
|
||||
t.Fatalf("build: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _, _ = two.State.res.close(two.Log); two.State.runs.stop() })
|
||||
_, back, since := resOf(t, two, tn).arms()
|
||||
if got := back.State(tn.String()).Learned; got != learned {
|
||||
t.Fatalf("after a restart the model has learned %d of %d", got, learned)
|
||||
}
|
||||
// The AGGREGATES do not survive — they are not durable — and the decision
|
||||
// says so rather than presenting a ten-minute ring as a thirty-day count.
|
||||
if since.Before(time.Now().Add(-time.Minute)) {
|
||||
t.Fatalf("the aggregates claim to have been running since %s across a restart", since)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTheProbeGoesDegradedWhenTheNodeIsFull pins that capacity is an ALARM. A
|
||||
// counter nobody reads is how a pod quietly stops protecting new tenants.
|
||||
func TestTheProbeGoesDegradedWhenTheNodeIsFull(t *testing.T) {
|
||||
t.Setenv(envMemory, node(1))
|
||||
app, s := wireApp(t)
|
||||
|
||||
code, _ := req(t, app, http.MethodGet, "/v1/risk/health", "", "", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("a healthy node probes %d, want 200", code)
|
||||
}
|
||||
if _, err := s.State.res.of(Tenant("hanzo/acme"), s.Log); err != nil {
|
||||
t.Fatalf("first tenant: %v", err)
|
||||
}
|
||||
if _, err := s.State.res.of(Tenant("hanzo/beta"), s.Log); err == nil {
|
||||
t.Fatal("the second tenant was admitted")
|
||||
}
|
||||
|
||||
code, body := req(t, app, http.MethodGet, "/v1/risk/health", "", "", "")
|
||||
if code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("a node that has refused a tenant probes %d, want 503 — nothing pages an operator", code)
|
||||
}
|
||||
var report map[string]any
|
||||
_ = json.Unmarshal(body, &report)
|
||||
if report["status"] != "degraded" {
|
||||
t.Fatalf("probe status = %v, want degraded", report["status"])
|
||||
}
|
||||
for _, k := range []string{"refused", "reclaimed", "strained", "bytes", "bytes_max"} {
|
||||
if report[k] == nil {
|
||||
t.Fatalf("the probe does not carry %q, so the saturation view is readable by nobody: %s", k, body)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,428 @@
|
||||
// Package risk is Hanzo Risk: the scoring and decision plane for any entity a
|
||||
// tenant has — an account, a transaction, a session, an agent, a merchant, a
|
||||
// payout — and the shared model plane both it and the compliance face read.
|
||||
//
|
||||
// THREE VERBS, ONE CORE: decide, record, learn.
|
||||
//
|
||||
// ONE FACE, AND IT IS /v1/risk. POST /v1/risk/decide answers allow / challenge /
|
||||
// review / restrict / block WITH A REASON; the planes around it (decisions,
|
||||
// rules, lists, suppressions, controls, dictionary, activity) are how a tenant
|
||||
// governs it; and score, train, search, state, features, snapshot and restore
|
||||
// are how it learns. Deciding and learning are the same state, so they are the
|
||||
// same face.
|
||||
//
|
||||
// THE THREE FACES DO NOT OVERLAP. /v1/aml is compliance — cases, sanctions,
|
||||
// retention. /v1/ml is SERVING — apps/ml deploys InferenceServices there and
|
||||
// /v1/ml/models means "models you serve", which is a different concept from
|
||||
// "models that learn"; this app claims nothing under it. /v1/risk is this one.
|
||||
//
|
||||
// Fraud is a USE of /v1/risk, not a sibling of it, and neither is abuse, bots,
|
||||
// account takeover, spam or pay-as-you-go abuse. There is no /v1/fraud.
|
||||
//
|
||||
// WHY IT IS IN CLOUD AND NOT A SERVICE OF ITS OWN. Being in cloud is what earns
|
||||
// the five things a standalone engine would each have to grow: IAM-validated
|
||||
// identity, the tenant gate, usage metering, billing and structured logs. None
|
||||
// of them is free — every one is wired explicitly below, and the table in
|
||||
// Mount's comment says where.
|
||||
//
|
||||
// WHAT IS LINKED FROM THE ENGINE, AND WHAT IS DELIBERATELY NOT. github.com/luxfi/aml
|
||||
// is a MODULE DEPENDENCY, exactly as luxfi/kms and hanzoai/o11y are; no source is
|
||||
// vendored. Only its transitively base-free packages are linked — types,
|
||||
// velocity, anomaly, replay — because pkg/engine, pkg/measure and pkg/history all
|
||||
// reach github.com/hanzoai/base/core and github.com/hanzoai/tasks through
|
||||
// pkg/history's Base-backed store, and dragging an application framework and a
|
||||
// workflow engine into a payment-authorization path is not a trade worth making.
|
||||
// See rule.go for the measurement and the upstream fix.
|
||||
package risk
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/hanzoai/cloud"
|
||||
"github.com/hanzoai/cloud/apps/datastore"
|
||||
"github.com/hanzoai/cloud/openapi"
|
||||
"github.com/luxfi/aml/pkg/anomaly"
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// state is everything the surface reads. Every per-tenant plane it can reach —
|
||||
// the model's counters, the velocity rings, the SQLite file and the governance
|
||||
// cache — lives inside ONE resident per tenant (resident.go) and is SINGLE-WRITER
|
||||
// by nature, which is why this app must run under the shard router that pins an
|
||||
// org to one pod, and why Shutdown must snapshot.
|
||||
//
|
||||
// NOTHING PER-TENANT IS HELD HERE. A field on this struct is a field every tenant
|
||||
// shares, and a shared store with a global cap is how one org silently evicts
|
||||
// another's fraud controls. The residency is the only tenant-indexed thing in the
|
||||
// app and it holds cells, never rows.
|
||||
type state struct {
|
||||
// brand is the deployment's brand, the half of the tenant key that never
|
||||
// comes from a header.
|
||||
brand string
|
||||
// dataDir is where the per-tenant files live.
|
||||
dataDir string
|
||||
|
||||
// res is the bounded set of live tenants: each one's own aggregates, own
|
||||
// model, own file, own cached rules.
|
||||
res *residency
|
||||
// runs is the process's bounded search queue.
|
||||
runs *runner
|
||||
// reg answers whether an agent reference is one the asking org registered.
|
||||
// An interface so the wire is one implementation and a test is another, with
|
||||
// no flag in production choosing between them.
|
||||
reg registry
|
||||
|
||||
// digest names the model SHAPE in force. It is a pure function of the
|
||||
// configuration, identical for every tenant, so it is computed once here
|
||||
// rather than read off whichever tenant's store came to hand.
|
||||
digest string
|
||||
|
||||
// bill is the shared per-org gate and meter, on the "risk" product.
|
||||
bill *cloud.ResourceMeter
|
||||
|
||||
// warehouse records whether the feature tables were created. A false value
|
||||
// is an honest gap on the dictionary and the backfill, never a zero.
|
||||
mu sync.Mutex
|
||||
warehouse bool
|
||||
}
|
||||
|
||||
// Mount wires /v1/risk onto app.
|
||||
//
|
||||
// EVERY INHERITED CAPABILITY IS WIRED HERE, EXPLICITLY. Being embedded in cloud
|
||||
// makes each one AVAILABLE; none of them is automatic:
|
||||
//
|
||||
// IAM auth SanitizeIdentity mints X-Org-Id from the verified bearer.
|
||||
// Global (serve.go) — nothing to do here, and that is the point:
|
||||
// this app never validates a token and never can.
|
||||
// tenant gate cloud.Bridge() on EACH group, FIRST, before any leaf. A typed
|
||||
// op receives only a context; Bridge is what parks the validated
|
||||
// org and validated-ness in it. Installed after the leaves it
|
||||
// would never run — fiber orders middleware by registration.
|
||||
// org sub-scope principal.ValidatedProject, read in tenantOf.
|
||||
// meter + gate cloud.NewResourceMeter(deps, "risk"); rm.Gate before priced
|
||||
// work, rm.Meter after it. Wired per op, in typed.go.
|
||||
// logs cloud.NewBase(deps, "risk") gives the scoped luxlog.
|
||||
// traces global and already ZAP-native (OTLZ). This package imports no
|
||||
// otlp transport, deliberately.
|
||||
// health OwnsHealth on the plugin plus the real probe below.
|
||||
//
|
||||
// The route registration follows apps/ml's own form, including the one shape
|
||||
// cmd/zipdoc can read: ONE `g := <router>.Group("/prefix")` per line.
|
||||
func Mount(app cloud.Router, deps cloud.Deps) error {
|
||||
if app == nil {
|
||||
return fmt.Errorf("risk.Mount: nil app")
|
||||
}
|
||||
s, err := build(deps)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// The warehouse is not on any request path this app serves, so a warehouse
|
||||
// that is down at boot must not stop the decision plane from mounting. The
|
||||
// tables are ensured once, in the background, and the honest gap is recorded.
|
||||
go warehouse(s)
|
||||
// Memory comes back from a SILENT tenant on a timer, never under another
|
||||
// tenant's pressure — that ordering is what makes it reclaim and not
|
||||
// eviction. See residency.reclaimLocked.
|
||||
go reclaim(s)
|
||||
|
||||
mount(s, app)
|
||||
s.Log.Info("risk surface mounted",
|
||||
"brand", deps.Brand, "env", deps.Env,
|
||||
"billing", s.State.bill.Enabled(), "model", s.State.digest,
|
||||
"text_max", textMax, "tenant_bytes", velBytes(), "tenant_keys", maxKeys(),
|
||||
"cell_bytes", cellBytes, "node_bytes", memBytes(), "reclaim_idle", idleReclaim().String())
|
||||
|
||||
// Shutdown is registered by the plugin (plugin/risk/main.go) and calls back
|
||||
// here; holding the service in a package var would be a second owner of the
|
||||
// state, so the closure captures it instead.
|
||||
shutdown = func(context.Context) error { return teardown(s) }
|
||||
return nil
|
||||
}
|
||||
|
||||
// warehouse creates the two feature planes and then keeps the NETWORK BASELINE
|
||||
// current. It is the only background work this app does and it never touches a
|
||||
// request path.
|
||||
//
|
||||
// The baseline recompute is here, on a timer, rather than on any route — which
|
||||
// is what makes the cross-org surface unreachable by a caller. Nobody can time
|
||||
// it, steer it, or observe its cost, and the statement it runs is a package
|
||||
// constant with no placeholder, so nothing a caller sends reaches it. What it
|
||||
// writes carries no tenant, no subject and no pseudonym: quantiles over a
|
||||
// k-anonymous set of contributing orgs, and nothing else.
|
||||
func warehouse(s *stateService) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
if err := ensureTables(ctx); err != nil {
|
||||
cancel()
|
||||
s.Log.Warn("risk: feature tables unavailable; the dictionary, the peer comparison and the search sandbox will report an honest gap", "err", err)
|
||||
return
|
||||
}
|
||||
cancel()
|
||||
s.State.mu.Lock()
|
||||
s.State.warehouse = true
|
||||
s.State.mu.Unlock()
|
||||
|
||||
for {
|
||||
bg, stop := context.WithTimeout(context.Background(), 10*time.Minute)
|
||||
if err := publishBaseline(bg); err != nil {
|
||||
s.Log.Warn("risk: the network baseline was not recomputed; peer comparison holds its last published day", "err", err)
|
||||
}
|
||||
stop()
|
||||
time.Sleep(baselineEvery)
|
||||
}
|
||||
}
|
||||
|
||||
// baselineEvery is how often the network quantiles are recomputed. Six hours:
|
||||
// the statement covers whole days, so anything faster republishes the same
|
||||
// numbers, and anything slower lets a day go unpublished after a restart.
|
||||
const baselineEvery = 6 * time.Hour
|
||||
|
||||
// reclaim returns memory from tenants that have gone SILENT.
|
||||
//
|
||||
// It runs on a timer rather than on admission pressure, and that is the whole
|
||||
// design: a reclaim triggered by a NEW tenant's arrival would make one tenant's
|
||||
// traffic the reason another tenant's aggregates went away, which is the
|
||||
// cross-tenant eviction this app exists without. Here the only input is the
|
||||
// reclaimed tenant's own silence.
|
||||
func reclaim(s *stateService) {
|
||||
for {
|
||||
time.Sleep(sweepEvery)
|
||||
s.State.res.sweep(s.Log)
|
||||
}
|
||||
}
|
||||
|
||||
// sweepEvery is how often idleness is checked. A fraction of idleReclaim, so a
|
||||
// tenant is reclaimed near its own threshold rather than up to a threshold later.
|
||||
const sweepEvery = 5 * time.Minute
|
||||
|
||||
// capacityAlarm is how long a refusal keeps the probe degraded. Longer than the
|
||||
// sweep, so an operator sees the alarm across at least one chance for the node to
|
||||
// free itself; short enough that a resolved event stops paging.
|
||||
const capacityAlarm = 3 * sweepEvery
|
||||
|
||||
// build constructs the state. Separate from Mount because Mount also starts the
|
||||
// background work and installs the routes, and a test wants the state without
|
||||
// either — the same split apps/ml makes for the same reason.
|
||||
func build(deps cloud.Deps) (*stateService, error) {
|
||||
if deps.Logger == nil {
|
||||
return nil, fmt.Errorf("risk.Mount: nil deps.Logger")
|
||||
}
|
||||
if deps.Brand == "" {
|
||||
// The brand is half the tenant key. A deployment that did not state one
|
||||
// cannot mint a key, and every request would 403 — better to say so at
|
||||
// boot than once per request.
|
||||
return nil, fmt.Errorf("risk.Mount: no brand, so no tenant key can be minted")
|
||||
}
|
||||
|
||||
// The model SHAPE, from a store that will never hold a tenant. The digest is
|
||||
// a pure function of the configuration — the inventory in order and the
|
||||
// geometry parameters — so it can be settled once at boot, and reading it
|
||||
// off a throwaway is better than keeping a spare store around that something
|
||||
// could later be tempted to score with.
|
||||
shape, err := forest(anomaly.Config{}, aggregates())
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("risk.Mount: %w", err)
|
||||
}
|
||||
|
||||
base := cloud.NewBase(deps, "risk")
|
||||
return &cloud.Service[state]{
|
||||
Base: base,
|
||||
State: state{
|
||||
brand: deps.Brand,
|
||||
dataDir: deps.DataDir,
|
||||
res: newResidency(deps.DataDir),
|
||||
runs: newRunner(base.Log),
|
||||
reg: peerRegistry{},
|
||||
digest: shape.Digest(),
|
||||
bill: cloud.NewResourceMeter(deps, "risk"),
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
// stateService is the concrete service this package builds. An alias, so the
|
||||
// helpers below read as functions of the service and no second type exists.
|
||||
type stateService = cloud.Service[state]
|
||||
|
||||
// shutdown is set by Mount and read by the plugin. A nil value means the app
|
||||
// never mounted, and tearing down what never came up is a no-op rather than a
|
||||
// panic.
|
||||
var shutdown func(context.Context) error
|
||||
|
||||
// Shutdown snapshots every resident tenant's model and closes the record planes.
|
||||
//
|
||||
// THIS IS NOT HOUSEKEEPING. cloud deploys with strategy Recreate at one replica:
|
||||
// every rollout is a hard all-endpoint window AND drops every in-memory model.
|
||||
// A model that comes back with nothing learned declines to score for the whole
|
||||
// warm period, and a control that is off for however long that takes is a
|
||||
// control that was off — reported, if anyone reads Refusal, and silent if not.
|
||||
func Shutdown(ctx context.Context) error {
|
||||
if shutdown == nil {
|
||||
return nil
|
||||
}
|
||||
return shutdown(ctx)
|
||||
}
|
||||
|
||||
func teardown(s *stateService) error {
|
||||
// Stop the search queue FIRST: a worker holds a tenant's file handle, and
|
||||
// closing under it would turn a rollout into a write error on a durable
|
||||
// report. Every in-flight run is cancelled and its row says so.
|
||||
s.State.runs.stop()
|
||||
kept, failed := s.State.res.close(s.Log)
|
||||
s.Log.Info("risk surface down", "models_kept", kept, "models_lost", failed)
|
||||
if failed > 0 {
|
||||
return fmt.Errorf("risk: %d tenant model(s) could not be snapshotted", failed)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// tenantState resolves the caller's scope and its resident — the tenant's own
|
||||
// aggregates, model, file and cached governance, armed and restored if this
|
||||
// process does not already hold them.
|
||||
//
|
||||
// Every typed op starts here, so there is ONE place the tenant is established,
|
||||
// ONE place the bound is applied, and ONE place the learned state is reloaded.
|
||||
// IT ALSO RESOLVES THE TENANT'S FILE, and that is what keeps a closed handle
|
||||
// off the query path. residency.close retires EVERY cell the moment teardown
|
||||
// runs — no idle requirement, and cloud deploys Recreate at ONE replica — so a
|
||||
// request already holding a cell can find its handle gone, and a nil *sql.DB
|
||||
// locks a nil mutex on its first use. Handing the handle out HERE, resolved and
|
||||
// checked once, is what makes that unrepresentable downstream: an op cannot
|
||||
// obtain a file without also obtaining the error that says it has none.
|
||||
func tenantState(ctx context.Context, s *stateService) (scope, *resident, *sql.DB, error) {
|
||||
sc, err := tenantOf(ctx, s.State.brand)
|
||||
if err != nil {
|
||||
return scope{}, nil, nil, err
|
||||
}
|
||||
// The mint's own shape check, asserted at the boundary rather than assumed.
|
||||
// A regression here is the whole product: the key is the store index, the
|
||||
// history org column and the model's tree seed at once.
|
||||
if !qualified(s.State.brand, sc.tenant) {
|
||||
return scope{}, nil, nil, zip.ErrForbidden("the tenant key is not qualified")
|
||||
}
|
||||
r, err := s.State.res.of(sc.tenant, s.Log)
|
||||
if err != nil {
|
||||
return scope{}, nil, nil, err
|
||||
}
|
||||
db, err := r.file()
|
||||
if err != nil {
|
||||
return scope{}, nil, nil, err
|
||||
}
|
||||
return sc, r, db, nil
|
||||
}
|
||||
|
||||
// governState is tenantState plus the ONE predicate that separates USING this
|
||||
// plane from GOVERNING it: the caller must be an admin of its own org.
|
||||
//
|
||||
// EVERY WRITE BEHIND IT CAN TURN A CONTROL OFF. Shadow mode makes every rule
|
||||
// observe and nothing act; retiring a rule deletes a detection; a blanket
|
||||
// suppression mutes one; an allow-list entry is a bypass; appetite decides how
|
||||
// much of the stream the model may even look at. A leaked low-privilege customer
|
||||
// key that can reach any of those turns the customer's fraud plane off, and the
|
||||
// customer finds out from a chargeback.
|
||||
//
|
||||
// It is org-scoped and org-scoped only. There is no cross-tenant surface in this
|
||||
// app, so there is nothing for platform authority to reach and no reason to ask
|
||||
// for it — conflating the two scopes is a privilege escalation, not a
|
||||
// convenience.
|
||||
//
|
||||
// A governance write is also EMITTED with its actor, so the change is readable
|
||||
// after the fact by whoever has to explain why a control was off.
|
||||
func governState(ctx context.Context, s *stateService) (scope, *resident, *sql.DB, error) {
|
||||
sc, res, db, err := tenantState(ctx, s)
|
||||
if err != nil {
|
||||
return scope{}, nil, nil, err
|
||||
}
|
||||
if !sc.admin {
|
||||
return scope{}, nil, nil, zip.ErrForbidden("governing this tenant's risk controls requires an admin of this org; scoring and reading do not")
|
||||
}
|
||||
return sc, res, db, nil
|
||||
}
|
||||
|
||||
// health is the app's real, fail-closed probe.
|
||||
//
|
||||
// UNTYPED BY DESIGN. A real probe answers 503 CARRYING THE DEGRADED REPORT AS
|
||||
// ITS BODY, and a typed op reaches a non-2xx only by returning an error, which
|
||||
// zip renders as its own envelope — dropping exactly the report the probe
|
||||
// exists to deliver. Same reason apps/ml keeps its two health routes raw.
|
||||
func health(s *stateService) func(*zip.Ctx) error {
|
||||
return func(c *zip.Ctx) error {
|
||||
s.State.mu.Lock()
|
||||
warehouse := s.State.warehouse
|
||||
s.State.mu.Unlock()
|
||||
tenants, refused, reclaimed, refusedAt := s.State.res.count()
|
||||
|
||||
// THE SATURATION VIEW, and every number on it is one an operator acts
|
||||
// on. A bound that binds silently is the defect this app was held for,
|
||||
// so each way this node can run short says so here: `bytes` against
|
||||
// `bytes_max` is the headroom, `reclaimed` is how many cells the node
|
||||
// has taken back under pressure, `refused` is how many tenants it turned
|
||||
// away, and `strained` is how many resident tenants are reading partial
|
||||
// rings right now. A control that switches off quietly is worse than no
|
||||
// control.
|
||||
report := map[string]any{
|
||||
"status": "ok",
|
||||
"model": s.State.digest,
|
||||
"tenants": tenants,
|
||||
"bytes": s.State.res.bytes(),
|
||||
"bytes_max": memBytes(),
|
||||
"refused": refused,
|
||||
"reclaimed": reclaimed,
|
||||
"strained": s.State.res.strained(),
|
||||
"warehouse": warehouse || datastore.Ready(),
|
||||
"billing": s.State.bill.Enabled(),
|
||||
}
|
||||
// The DECISION plane is what this app promises, and it does not need the
|
||||
// warehouse: rings are in memory, rules and lists are the tenant's own
|
||||
// file, the model is in process. The probe is degraded only when
|
||||
// something the hot path actually needs is missing.
|
||||
if s.State.res == nil || s.State.runs == nil || s.State.dataDir == "" {
|
||||
report["status"] = "degraded"
|
||||
report["error"] = "the decision plane is not constructed"
|
||||
return c.JSON(http.StatusServiceUnavailable, report)
|
||||
}
|
||||
// A node that has REFUSED a tenant is out of room, and the tenants it
|
||||
// turned away are unprotected. That is a capacity event an operator must
|
||||
// be paged for, not a counter someone finds later — so the probe goes
|
||||
// degraded while it stands, and the reason names itself.
|
||||
//
|
||||
// WHILE IT STANDS, not forever. The counter is monotone, so latching on
|
||||
// it would leave the pod unready for the rest of its life over one
|
||||
// refusal — turning a capacity event for ONE tenant into an outage for
|
||||
// every tenant the node is serving perfectly well, which is a worse
|
||||
// version of the same defect. The alarm clears once the node has gone
|
||||
// capacityAlarm without turning anyone away; the count stays on the
|
||||
// report either way, so nothing is hidden.
|
||||
if !refusedAt.IsZero() && time.Since(refusedAt) < capacityAlarm {
|
||||
report["status"] = "degraded"
|
||||
report["error"] = "this node has no memory left for another tenant and has refused admissions; it will not take a live tenant's state to make room"
|
||||
return c.JSON(http.StatusServiceUnavailable, report)
|
||||
}
|
||||
return c.JSON(http.StatusOK, report)
|
||||
}
|
||||
}
|
||||
|
||||
// The prose for the ONE route above that is untyped by design. zipdoc lifts
|
||||
// prose from a typed handler's doc comment, and this is not one — so without a
|
||||
// Describe the probe would publish an operationId and nothing else. Declared
|
||||
// beside the wire fact it belongs to.
|
||||
func init() {
|
||||
openapi.Describe("/v1/risk/health", http.MethodGet,
|
||||
"Report whether the decision plane can decide",
|
||||
"Answers 200 with a report of what is up, or 503 CARRYING THE SAME REPORT as its "+
|
||||
"body — which is the whole point of a real probe, and the reason this one route is "+
|
||||
"not a typed op: a typed op reaches a non-2xx only by returning an error, and that "+
|
||||
"renders as an envelope with the report dropped.\n\n"+
|
||||
"The report names the model digest in force, how many tenants this process holds "+
|
||||
"resident, whether the feature warehouse is reachable, and whether metering is "+
|
||||
"configured. Only the first is load-bearing for a decision: the rings are in "+
|
||||
"memory, the rules and lists are the tenant's own file, and the model is in "+
|
||||
"process, so a decision does NOT need the warehouse and the probe stays green "+
|
||||
"without it. A warehouse that is down costs the field dictionary and the search "+
|
||||
"sandbox, and is reported as exactly that rather than as a failure of the "+
|
||||
"authorization path.")
|
||||
}
|
||||
@@ -0,0 +1,474 @@
|
||||
package risk
|
||||
|
||||
// rule.go is the risk rule algebra: a CLOSED set of fields and a CLOSED set of
|
||||
// comparisons, evaluated against one decision's facts.
|
||||
//
|
||||
// WHY IT IS NOT luxfi/aml's expr-lang evaluator, stated with the measurement.
|
||||
// The design this app was built from claimed pkg/engine is base-free. It is not.
|
||||
// Measured with `go list -deps`:
|
||||
//
|
||||
// pkg/engine -> pkg/history -> pkg/store -> github.com/hanzoai/base/core
|
||||
// -> github.com/hanzoai/tasks (Temporal SDK)
|
||||
// -> github.com/hanzoai/csqlite, hanzoai/sqlite
|
||||
//
|
||||
// A grep for `hanzoai/base` in pkg/engine's own files finds nothing, which is
|
||||
// how the claim was arrived at; the import is TRANSITIVE, through the measures
|
||||
// in scope.go. Linking it would pull the whole Base application framework and a
|
||||
// workflow engine into a binary whose job is to answer a payment-authorization
|
||||
// call in single-digit milliseconds — and cloud's image carries a SQLITE-GATE
|
||||
// that fails the BUILD, not the tests, when a per-app binary drags a second
|
||||
// sqlite driver in under CGO.
|
||||
//
|
||||
// So the risk plane brings in only the transitively base-free packages
|
||||
// (types, velocity, anomaly, replay, standard) and states its rules as data.
|
||||
// That is the better shape here anyway: a risk rule is a conjunction of
|
||||
// comparisons over a fixed vocabulary, which is a closed algebra, not a
|
||||
// language. A closed algebra is injection-safe by construction, is a TYPED
|
||||
// wire shape (so it reaches the SDKs, the CLI and the MCP tools as a schema
|
||||
// instead of as an opaque string), and can be replayed without an interpreter.
|
||||
//
|
||||
// The right long-term fix is upstream and is named in the handoff: split
|
||||
// pkg/history's Base-backed store into its own package so pkg/engine is
|
||||
// base-free in fact as well as in intent.
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Actions, strongest last. `challenge` and `restrict` are risk's own: the
|
||||
// compliance vocabulary (allow/flag/review/block) has no word for "ask the human
|
||||
// to prove they are one" or "let the money in but not out", and both are the
|
||||
// normal answer at signup and at payout.
|
||||
const (
|
||||
ActionAllow = "allow"
|
||||
ActionChallenge = "challenge"
|
||||
ActionReview = "review"
|
||||
ActionRestrict = "restrict"
|
||||
ActionBlock = "block"
|
||||
)
|
||||
|
||||
// actionRank orders actions by how much they demand, so "the strongest of these"
|
||||
// and "no stronger than this" are one comparison rather than two tables. An
|
||||
// action nobody defined ranks lowest: it must never outrank one that was.
|
||||
func actionRank(a string) int {
|
||||
switch a {
|
||||
case ActionAllow:
|
||||
return 1
|
||||
case ActionChallenge:
|
||||
return 2
|
||||
case ActionReview:
|
||||
return 3
|
||||
case ActionRestrict:
|
||||
return 4
|
||||
case ActionBlock:
|
||||
return 5
|
||||
default:
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
// modelCeiling is the strongest action STATISTICAL evidence may reach on its
|
||||
// own. It is luxfi/aml's types.ActionCeiling and it is not weakened for the
|
||||
// payment stage: a model can put a transaction in front of a person; it cannot
|
||||
// decline one, because an unexplainable refusal is not a decision anybody can
|
||||
// defend to the customer or to a chargeback network.
|
||||
const modelCeiling = ActionReview
|
||||
|
||||
// Stages are the lifecycle moments a decision can be asked about. The stage
|
||||
// selects the feature window and the rule set; it never selects a different
|
||||
// tenant gate.
|
||||
const (
|
||||
StageSignup = "signup"
|
||||
StagePayment = "payment"
|
||||
StageSession = "session"
|
||||
StageUsage = "usage"
|
||||
StagePayout = "payout"
|
||||
StageDispute = "dispute"
|
||||
)
|
||||
|
||||
var stages = map[string]bool{
|
||||
StageSignup: true, StagePayment: true, StageSession: true,
|
||||
StageUsage: true, StagePayout: true, StageDispute: true,
|
||||
}
|
||||
|
||||
// Operators, closed.
|
||||
const (
|
||||
OpEq = "eq"
|
||||
OpNe = "ne"
|
||||
OpGt = "gt"
|
||||
OpGte = "gte"
|
||||
OpLt = "lt"
|
||||
OpLte = "lte"
|
||||
OpIn = "in"
|
||||
OpNotIn = "notin"
|
||||
OpInList = "inlist"
|
||||
OpNotList = "notinlist"
|
||||
OpExists = "exists"
|
||||
OpAbsent = "absent"
|
||||
OpContains = "contains"
|
||||
OpPrefix = "prefix"
|
||||
OpSuffix = "suffix"
|
||||
)
|
||||
|
||||
var operators = map[string]bool{
|
||||
OpEq: true, OpNe: true, OpGt: true, OpGte: true, OpLt: true, OpLte: true,
|
||||
OpIn: true, OpNotIn: true, OpInList: true, OpNotList: true,
|
||||
OpExists: true, OpAbsent: true, OpContains: true, OpPrefix: true, OpSuffix: true,
|
||||
}
|
||||
|
||||
// facts is the CLOSED vocabulary of scalar fields a term may name. Two families
|
||||
// are open by shape and closed by source: `signal.<name>` reads the caller's own
|
||||
// signal map (the caller's data, compared in memory, never spelled into SQL),
|
||||
// and `velocity.<axis>.<window>.<stat>` reads the in-memory rings through an
|
||||
// allowlisted axis and window.
|
||||
var facts = map[string]bool{
|
||||
"stage": true,
|
||||
"subject.kind": true,
|
||||
"subject.id": true,
|
||||
"agency": true,
|
||||
"amount.nano": true,
|
||||
"amount.currency": true,
|
||||
"amount.direction": true,
|
||||
"model.score": true,
|
||||
"model.warming": true,
|
||||
"actor.agent": true,
|
||||
"actor.session": true,
|
||||
}
|
||||
|
||||
// velocityStats is the closed set of numbers a rule may read off a ring.
|
||||
var velocityStats = map[string]bool{"count": true, "sum": true, "near": true, "days": true}
|
||||
|
||||
// term is one comparison. Value carries strings, Number carries numerics and
|
||||
// Values carries a set — three fields rather than one `any`, because the wire
|
||||
// shape is a published schema and `any` publishes as "anything".
|
||||
type term struct {
|
||||
// Field is the fact to read. One of the closed vocabulary, or
|
||||
// `signal.<name>`, or `velocity.<axis>.<window>.<stat>`.
|
||||
Field string `json:"field"`
|
||||
// Op is the comparison. One of the closed operator set.
|
||||
Op string `json:"op"`
|
||||
// Value is the string operand, for the textual comparisons.
|
||||
Value string `json:"value,omitempty"`
|
||||
// Number is the numeric operand, for the ordered comparisons.
|
||||
Number float64 `json:"number,omitempty"`
|
||||
// Values is the set operand, for `in` and `notin`.
|
||||
Values []string `json:"values,omitempty"`
|
||||
}
|
||||
|
||||
// rule is one detection. All terms must hold — a conjunction, deliberately: a
|
||||
// disjunction is two rules, and two rules are two things a reviewer can judge,
|
||||
// retire and measure separately. Nothing is lost and the report gets sharper.
|
||||
type rule struct {
|
||||
// ID is the rule's stable identifier within the tenant.
|
||||
ID string `json:"id"`
|
||||
// Name is what a reviewer reads in an alert.
|
||||
Name string `json:"name"`
|
||||
// Stage narrows the rule to one lifecycle moment; empty means every stage.
|
||||
Stage string `json:"stage,omitempty"`
|
||||
// Action is what the rule asks for when it holds.
|
||||
Action string `json:"action"`
|
||||
// Weight is how much evidence a hit contributes, in [0,1].
|
||||
Weight float64 `json:"weight"`
|
||||
// Severity is the reviewer-facing grading: low, medium, high or critical.
|
||||
Severity string `json:"severity"`
|
||||
// Enabled governs the live path. A disabled rule still replays, because the
|
||||
// question a simulation asks is what happens ON activation.
|
||||
Enabled bool `json:"enabled"`
|
||||
// All is the conjunction. An empty conjunction is refused at admission: a
|
||||
// rule that holds on everything is not a detection.
|
||||
All []term `json:"all"`
|
||||
}
|
||||
|
||||
// admit validates a rule before anything depends on it. Every refusal here is
|
||||
// one that would otherwise become a rule that fires on everything, on nothing,
|
||||
// or on a field that does not exist — all three read as a working control.
|
||||
func admit(r rule) error {
|
||||
switch {
|
||||
case strings.TrimSpace(r.Name) == "":
|
||||
return fmt.Errorf("risk: a rule with no name cannot be read back in an alert")
|
||||
case r.Stage != "" && !stages[r.Stage]:
|
||||
return fmt.Errorf("risk: %q is not a lifecycle stage", r.Stage)
|
||||
case actionRank(r.Action) == 0:
|
||||
return fmt.Errorf("risk: %q is not an action", r.Action)
|
||||
case r.Weight < 0 || r.Weight > 1:
|
||||
return fmt.Errorf("risk: weight %v is outside [0,1]", r.Weight)
|
||||
case len(r.All) == 0:
|
||||
return fmt.Errorf("risk: a rule with no terms holds on everything, which is not a detection")
|
||||
}
|
||||
if r.Severity != "" && !severities[r.Severity] {
|
||||
return fmt.Errorf("risk: %q is not a severity", r.Severity)
|
||||
}
|
||||
for i, t := range r.All {
|
||||
if err := admitTerm(t); err != nil {
|
||||
return fmt.Errorf("risk: term %d: %w", i, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
var severities = map[string]bool{"low": true, "medium": true, "high": true, "critical": true}
|
||||
|
||||
// admitTerm holds the field and operator vocabularies. It is the ONE gate: a
|
||||
// field is either in the closed set or matches one of the two structured
|
||||
// families, and an unknown field is an error rather than a silent miss.
|
||||
func admitTerm(t term) error {
|
||||
if !operators[t.Op] {
|
||||
return fmt.Errorf("%q is not an operator", t.Op)
|
||||
}
|
||||
switch {
|
||||
case facts[t.Field]:
|
||||
case strings.HasPrefix(t.Field, "signal."):
|
||||
if strings.TrimPrefix(t.Field, "signal.") == "" {
|
||||
return fmt.Errorf("signal. names no signal")
|
||||
}
|
||||
case strings.HasPrefix(t.Field, "velocity."):
|
||||
parts := strings.Split(t.Field, ".")
|
||||
if len(parts) != 4 {
|
||||
return fmt.Errorf("%q is not velocity.<axis>.<window>.<stat>", t.Field)
|
||||
}
|
||||
if !velocityAxes[parts[1]] {
|
||||
return fmt.Errorf("%q is not a velocity axis", parts[1])
|
||||
}
|
||||
if !velocityWindows[parts[2]] {
|
||||
return fmt.Errorf("%q is not a velocity window", parts[2])
|
||||
}
|
||||
if !velocityStats[parts[3]] {
|
||||
return fmt.Errorf("%q is not a velocity statistic", parts[3])
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("%q is not a fact this vocabulary carries", t.Field)
|
||||
}
|
||||
if (t.Op == OpIn || t.Op == OpNotIn) && len(t.Values) == 0 {
|
||||
return fmt.Errorf("%s with an empty set holds on nothing", t.Op)
|
||||
}
|
||||
if (t.Op == OpInList || t.Op == OpNotList) && strings.TrimSpace(t.Value) == "" {
|
||||
return fmt.Errorf("%s names no list", t.Op)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// velocityAxes and velocityWindows mirror the rings the decide path keeps. They
|
||||
// are declared here because this is where a caller's spelling is checked against
|
||||
// them; decide.go records on exactly these.
|
||||
var velocityAxes = map[string]bool{"account": true, "device": true, "ip": true, "pair": true, "email": true, "bin": true}
|
||||
|
||||
var velocityWindows = map[string]bool{"1h": true, "24h": true, "7d": true, "30d": true}
|
||||
|
||||
// facts is what a term reads: the decision's own inputs plus everything the
|
||||
// scoring path has computed by the time rules run.
|
||||
type factSet struct {
|
||||
scalar map[string]string
|
||||
number map[string]float64
|
||||
lists func(name, value string) bool
|
||||
}
|
||||
|
||||
func (f factSet) str(field string) (string, bool) {
|
||||
v, ok := f.scalar[field]
|
||||
return v, ok
|
||||
}
|
||||
|
||||
func (f factSet) num(field string) (float64, bool) {
|
||||
v, ok := f.number[field]
|
||||
if ok {
|
||||
return v, true
|
||||
}
|
||||
// A scalar that parses as a number is comparable as one: `signal.bin` is a
|
||||
// string on the wire and an ordered value in a rule.
|
||||
if s, ok := f.scalar[field]; ok {
|
||||
if n, err := strconv.ParseFloat(s, 64); err == nil {
|
||||
return n, true
|
||||
}
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// holds evaluates one term. An unreadable field is FALSE, never an error that
|
||||
// aborts the decision: a rule over a signal this caller did not send has not
|
||||
// matched, and the alternative — failing the whole decision — turns one
|
||||
// mis-specified rule into an outage on the authorization path.
|
||||
func (f factSet) holds(t term) bool {
|
||||
switch t.Op {
|
||||
case OpExists:
|
||||
if _, ok := f.str(t.Field); ok {
|
||||
return true
|
||||
}
|
||||
_, ok := f.num(t.Field)
|
||||
return ok
|
||||
case OpAbsent:
|
||||
if _, ok := f.str(t.Field); ok {
|
||||
return false
|
||||
}
|
||||
_, ok := f.num(t.Field)
|
||||
return !ok
|
||||
case OpGt, OpGte, OpLt, OpLte:
|
||||
v, ok := f.num(t.Field)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
switch t.Op {
|
||||
case OpGt:
|
||||
return v > t.Number
|
||||
case OpGte:
|
||||
return v >= t.Number
|
||||
case OpLt:
|
||||
return v < t.Number
|
||||
default:
|
||||
return v <= t.Number
|
||||
}
|
||||
case OpInList, OpNotList:
|
||||
v, ok := f.str(t.Field)
|
||||
if !ok || f.lists == nil {
|
||||
return t.Op == OpNotList
|
||||
}
|
||||
in := f.lists(t.Value, v)
|
||||
return in == (t.Op == OpInList)
|
||||
}
|
||||
|
||||
v, ok := f.str(t.Field)
|
||||
if !ok {
|
||||
if n, isNum := f.num(t.Field); isNum {
|
||||
v, ok = strconv.FormatFloat(n, 'f', -1, 64), true
|
||||
}
|
||||
}
|
||||
if !ok {
|
||||
return t.Op == OpNe || t.Op == OpNotIn
|
||||
}
|
||||
switch t.Op {
|
||||
case OpEq:
|
||||
return v == t.Value
|
||||
case OpNe:
|
||||
return v != t.Value
|
||||
case OpContains:
|
||||
return strings.Contains(v, t.Value)
|
||||
case OpPrefix:
|
||||
return strings.HasPrefix(v, t.Value)
|
||||
case OpSuffix:
|
||||
return strings.HasSuffix(v, t.Value)
|
||||
case OpIn:
|
||||
return contains(t.Values, v)
|
||||
case OpNotIn:
|
||||
return !contains(t.Values, v)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func contains(set []string, v string) bool {
|
||||
for _, s := range set {
|
||||
if s == v {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// evaluate runs the enabled rules for a stage and returns the hits in a stable
|
||||
// order (strongest action first, then by weight, then by id) so two identical
|
||||
// decisions render identically.
|
||||
func evaluate(rules []rule, stage string, f factSet) []hit {
|
||||
var hits []hit
|
||||
for _, r := range rules {
|
||||
if !r.Enabled || (r.Stage != "" && r.Stage != stage) {
|
||||
continue
|
||||
}
|
||||
matched := true
|
||||
for _, t := range r.All {
|
||||
if !f.holds(t) {
|
||||
matched = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if matched {
|
||||
hits = append(hits, hit{Rule: r.ID, Name: r.Name, Action: r.Action, Weight: r.Weight, Severity: r.Severity})
|
||||
}
|
||||
}
|
||||
sort.SliceStable(hits, func(i, j int) bool {
|
||||
if a, b := actionRank(hits[i].Action), actionRank(hits[j].Action); a != b {
|
||||
return a > b
|
||||
}
|
||||
if hits[i].Weight != hits[j].Weight {
|
||||
return hits[i].Weight > hits[j].Weight
|
||||
}
|
||||
return hits[i].Rule < hits[j].Rule
|
||||
})
|
||||
return hits
|
||||
}
|
||||
|
||||
// combine turns hits into a score and an action.
|
||||
//
|
||||
// The score is weight-of-evidence: 1 - prod(1-w). Two independent weak signals
|
||||
// compound and no single one saturates, which is the property a summed score
|
||||
// does not have (three 0.4 rules sum past 1 and clamp, losing the fourth).
|
||||
//
|
||||
// The action is the strongest any hit asked for, and the model's own hit is
|
||||
// capped at the ceiling BEFORE it gets here, so no arrangement of weights lets
|
||||
// statistical evidence decline anything on its own.
|
||||
func combine(hits []hit) (float64, string) {
|
||||
remain, action := 1.0, ActionAllow
|
||||
for _, h := range hits {
|
||||
w := h.Weight
|
||||
if w < 0 {
|
||||
w = 0
|
||||
}
|
||||
if w > 1 {
|
||||
w = 1
|
||||
}
|
||||
remain *= 1 - w
|
||||
if actionRank(h.Action) > actionRank(action) {
|
||||
action = h.Action
|
||||
}
|
||||
}
|
||||
return 1 - remain, action
|
||||
}
|
||||
|
||||
// starter is the rule set a tenant gets on its first decision. Every one of
|
||||
// these is a lifecycle-stage detection the product spec names, expressed in the
|
||||
// vocabulary above so a tenant can read, copy and retire them. They are seeded
|
||||
// ENABLED but the tenant starts in SHADOW, so nothing acts until an operator
|
||||
// says so and can see what would have happened.
|
||||
func starter() []rule {
|
||||
return []rule{{
|
||||
ID: "signup-burst-ip", Name: "Many signups from one address", Stage: StageSignup,
|
||||
Action: ActionChallenge, Weight: 0.4, Severity: "medium", Enabled: true,
|
||||
All: []term{{Field: "velocity.ip.1h.count", Op: OpGte, Number: 5}},
|
||||
}, {
|
||||
ID: "signup-shared-device", Name: "One device onboarding several accounts", Stage: StageSignup,
|
||||
Action: ActionReview, Weight: 0.5, Severity: "high", Enabled: true,
|
||||
All: []term{{Field: "velocity.device.7d.count", Op: OpGte, Number: 4}},
|
||||
}, {
|
||||
ID: "signup-disposable-email", Name: "Disposable email domain", Stage: StageSignup,
|
||||
Action: ActionChallenge, Weight: 0.35, Severity: "medium", Enabled: true,
|
||||
All: []term{{Field: "signal.emaildomain", Op: OpInList, Value: "email-deny"}},
|
||||
}, {
|
||||
ID: "payment-card-testing", Name: "Card testing: a burst of small charges", Stage: StagePayment,
|
||||
Action: ActionBlock, Weight: 0.7, Severity: "critical", Enabled: true,
|
||||
All: []term{
|
||||
{Field: "velocity.bin.1h.count", Op: OpGte, Number: 10},
|
||||
{Field: "amount.nano", Op: OpLte, Number: 5_000_000_000},
|
||||
},
|
||||
}, {
|
||||
ID: "payment-denied-ip", Name: "Payment from a denied address", Stage: StagePayment,
|
||||
Action: ActionBlock, Weight: 0.9, Severity: "critical", Enabled: true,
|
||||
All: []term{{Field: "signal.ip", Op: OpInList, Value: "ip-deny"}},
|
||||
}, {
|
||||
ID: "usage-spend-spike", Name: "Pay-as-you-go spend far above this account's own shape", Stage: StageUsage,
|
||||
Action: ActionRestrict, Weight: 0.5, Severity: "high", Enabled: true,
|
||||
All: []term{{Field: "velocity.account.24h.sum", Op: OpGte, Number: 100_000_000_000_000}},
|
||||
}, {
|
||||
ID: "payout-velocity", Name: "Payouts leaving faster than they arrived", Stage: StagePayout,
|
||||
Action: ActionRestrict, Weight: 0.6, Severity: "high", Enabled: true,
|
||||
All: []term{
|
||||
{Field: "amount.direction", Op: OpEq, Value: "out"},
|
||||
{Field: "velocity.account.24h.count", Op: OpGte, Number: 5},
|
||||
},
|
||||
}, {
|
||||
ID: "bot-anonymous-burst", Name: "Undeclared automation at machine cadence", Stage: StageSession,
|
||||
Action: ActionChallenge, Weight: 0.45, Severity: "medium", Enabled: true,
|
||||
All: []term{
|
||||
{Field: "agency", Op: OpEq, Value: AgencyBot},
|
||||
{Field: "velocity.ip.1h.count", Op: OpGte, Number: 60},
|
||||
},
|
||||
}}
|
||||
}
|
||||
@@ -0,0 +1,304 @@
|
||||
package risk
|
||||
|
||||
// search.go bounds the one expensive thing this app will do on a caller's word.
|
||||
//
|
||||
// WHAT IT WAS. POST /v1/risk/search started a bare goroutine per call with a
|
||||
// ten-minute budget, replaying 243 topologies over up to five thousand recorded
|
||||
// decisions. Nothing bounded how many ran at once, nothing could stop one, and a
|
||||
// rollout — Recreate at one replica — left the row saying `running` forever, which
|
||||
// is the same bytes as a search about to answer and the opposite fact. One
|
||||
// authenticated caller with a loop could hold every core on a shared pod.
|
||||
//
|
||||
// WHAT IT IS. A queue with a fixed number of workers, ONE run per tenant, a
|
||||
// bounded backlog, a deadline, a cancel op, and a durable status that a restart
|
||||
// resolves rather than abandons:
|
||||
//
|
||||
// one per tenant a second start answers 409 with the run already going, so a
|
||||
// loop costs one slot and not one per iteration.
|
||||
// workers searchWorkers goroutines for the whole process. The grid is
|
||||
// CPU, and CPU is the pod's, not the caller's.
|
||||
// backlog searchQueue deep. Full answers 429 — the honest word for
|
||||
// "come back", and one a client can act on.
|
||||
// deadline searchBudget, enforced by the context every candidate polls.
|
||||
// cancel DELETE /v1/risk/search/{id} stops it and says so on the row.
|
||||
// shutdown a stopped runner does not run its backlog; every queued and
|
||||
// in-flight run leaves a `cancelled` row.
|
||||
// priced gated before and metered after, on the caller's own ledger.
|
||||
//
|
||||
// THE INPUTS ARE THE WORKER'S TO LOAD, and that is a bound and not a style. The
|
||||
// handler used to replay up to 5,000 rows into observations and only THEN ask
|
||||
// whether this tenant already had a run going: thirty concurrent calls each paid
|
||||
// the full SQLite read and materialised 5,000 observations, twenty-nine of them
|
||||
// to answer 409. Now a refusal costs a map lookup, and at most searchWorkers
|
||||
// loads exist at any instant.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
const (
|
||||
// searchWorkers is how many searches run at once across the WHOLE process.
|
||||
// Two: the grid is seconds to minutes of one core, and a decision path
|
||||
// sharing the pod must keep its cores.
|
||||
searchWorkers = 2
|
||||
// searchQueue is how deep the backlog goes before a start is refused. Short
|
||||
// on purpose — a long queue turns a capacity refusal into a latency mystery.
|
||||
searchQueue = 8
|
||||
// searchBudget bounds one run. The grid is closed (243 candidates over at
|
||||
// most 5,000 events), so this is a ceiling for a pathological host and not
|
||||
// the thing that makes the work finite.
|
||||
searchBudget = 2 * time.Minute
|
||||
)
|
||||
|
||||
// The durable statuses a search row carries. `running` is the only one a reader
|
||||
// must be able to trust, which is why nothing may leave one behind.
|
||||
const (
|
||||
searchRunning = "running"
|
||||
searchDone = "done"
|
||||
searchRefused = "refused"
|
||||
searchCancelled = "cancelled"
|
||||
)
|
||||
|
||||
// runner is the process's search queue. One for the app, built by build() and
|
||||
// stopped by teardown, so nothing here is a package global that a second mount
|
||||
// would share with the first.
|
||||
type runner struct {
|
||||
jobs chan job
|
||||
log logger
|
||||
|
||||
mu sync.Mutex
|
||||
running map[Tenant]*run
|
||||
stopped bool
|
||||
wg sync.WaitGroup
|
||||
}
|
||||
|
||||
// job is one queued search. It carries how to LOAD its inputs rather than the
|
||||
// inputs themselves, so the expensive read happens on a worker that already has
|
||||
// a slot — never on a caller that is about to be refused one.
|
||||
type job struct {
|
||||
t Tenant
|
||||
id string
|
||||
db *sql.DB
|
||||
load func() ([]observation, error)
|
||||
}
|
||||
|
||||
// run is a search in flight, and the handle that stops it.
|
||||
type run struct {
|
||||
id string
|
||||
cancel context.CancelFunc
|
||||
}
|
||||
|
||||
func newRunner(log logger) *runner {
|
||||
r := &runner{jobs: make(chan job, searchQueue), log: log, running: map[Tenant]*run{}}
|
||||
for i := 0; i < searchWorkers; i++ {
|
||||
r.wg.Add(1)
|
||||
go r.work()
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// errBusy and errBacklog are the two refusals, and they are DIFFERENT statuses
|
||||
// because they are different facts: one says this tenant already has a search
|
||||
// going, the other says the node does. A client retries the second and reads the
|
||||
// first.
|
||||
var (
|
||||
errBusy = zip.Errorf(409, "this tenant already has a search running; read it or cancel it before starting another")
|
||||
errBacklog = zip.Errorf(429, "this node's search queue is full; retry shortly")
|
||||
)
|
||||
|
||||
// start queues one search. It refuses rather than queues when the tenant already
|
||||
// has one going, so a caller in a loop occupies one slot forever instead of
|
||||
// filling the backlog and then the pod.
|
||||
func (r *runner) start(j job) error {
|
||||
r.mu.Lock()
|
||||
if r.stopped {
|
||||
r.mu.Unlock()
|
||||
return zip.Errorf(503, "this node is shutting down and is not starting searches")
|
||||
}
|
||||
if _, going := r.running[j.t]; going {
|
||||
r.mu.Unlock()
|
||||
return errBusy
|
||||
}
|
||||
// Claimed BEFORE the send, so two concurrent starts cannot both queue.
|
||||
r.running[j.t] = &run{id: j.id}
|
||||
r.mu.Unlock()
|
||||
|
||||
select {
|
||||
case r.jobs <- j:
|
||||
return nil
|
||||
default:
|
||||
r.mu.Lock()
|
||||
delete(r.running, j.t)
|
||||
r.mu.Unlock()
|
||||
return errBacklog
|
||||
}
|
||||
}
|
||||
|
||||
// cancel stops a tenant's run if the named one is the one going. Answers whether
|
||||
// it did: cancelling a search that already finished is not an error, and telling
|
||||
// the caller which happened is the whole content of the reply.
|
||||
//
|
||||
// A run still in the QUEUE has no cancel function yet — no worker has reached
|
||||
// it. Dropping the claim is how that one is cancelled: the worker that picks the
|
||||
// job up finds no claim and writes the cancelled row instead of doing the work.
|
||||
// Handling only the started case would leave a queued search uncancellable
|
||||
// precisely while the queue is full, which is when it matters.
|
||||
func (r *runner) cancel(t Tenant, id string) bool {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
going, ok := r.running[t]
|
||||
if !ok || going.id != id {
|
||||
return false
|
||||
}
|
||||
if going.cancel != nil {
|
||||
going.cancel()
|
||||
return true
|
||||
}
|
||||
delete(r.running, t)
|
||||
return true
|
||||
}
|
||||
|
||||
// work is one worker. It runs until the queue closes.
|
||||
func (r *runner) work() {
|
||||
defer r.wg.Done()
|
||||
for j := range r.jobs {
|
||||
r.execute(j)
|
||||
}
|
||||
}
|
||||
|
||||
// execute runs one job to its durable row.
|
||||
//
|
||||
// A PANIC IS THIS WORKER'S TO OWN. cloud runs ONE replica: an unrecovered panic
|
||||
// in a background goroutine takes the process down and with it every tenant on
|
||||
// the node, over one tenant's malformed history. The crash lands on the run's
|
||||
// own row, where the caller who started it can read it, and the pool keeps
|
||||
// serving.
|
||||
func (r *runner) execute(j job) {
|
||||
defer func() {
|
||||
if p := recover(); p != nil {
|
||||
r.log.Error("risk: a search worker panicked; the run is refused and the pool keeps serving",
|
||||
"tenant", j.t.String(), "search", j.id, "panic", fmt.Sprint(p))
|
||||
r.write(j, searchRefused, searchReport{Refusal: fmt.Sprintf("this search stopped on an internal error: %v", p)})
|
||||
r.done(j)
|
||||
}
|
||||
}()
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), searchBudget)
|
||||
defer cancel()
|
||||
|
||||
// THE STOP CHECK IS HERE, AT THE TOP, and it is what makes a rollout safe.
|
||||
// stop() cancels the claims that HAVE a cancel func, but a job still in the
|
||||
// backlog has none — execute installs it. Without this line the workers
|
||||
// drained the remaining backlog, found each claim still present, installed a
|
||||
// FRESH budget and ran the whole thing: worst case stop() blocked for
|
||||
// backlog/workers x searchBudget while SIGTERM's grace period is ~40s, so the
|
||||
// pod was killed before teardown snapshotted anything and EVERY tenant
|
||||
// reverted to its last snapshot or to `warming`. Any authenticated tenant
|
||||
// could arm that by queueing searches before a deploy.
|
||||
r.mu.Lock()
|
||||
stopped := r.stopped
|
||||
going, claimed := r.running[j.t]
|
||||
switch {
|
||||
case stopped || !claimed || going.id != j.id:
|
||||
r.mu.Unlock()
|
||||
reason := "cancelled before it started"
|
||||
if stopped {
|
||||
reason = "this node stopped before the search started; start it again"
|
||||
}
|
||||
r.write(j, searchCancelled, searchReport{Refusal: reason})
|
||||
r.done(j)
|
||||
return
|
||||
default:
|
||||
going.cancel = cancel
|
||||
r.mu.Unlock()
|
||||
}
|
||||
|
||||
obs, err := j.load()
|
||||
if err != nil {
|
||||
r.write(j, searchRefused, searchReport{Refusal: err.Error()})
|
||||
r.done(j)
|
||||
return
|
||||
}
|
||||
if len(obs) == 0 {
|
||||
r.write(j, searchRefused, searchReport{Refusal: errNoHistory.Error()})
|
||||
r.done(j)
|
||||
return
|
||||
}
|
||||
|
||||
rep, err := searchRun(ctx, j.t, obs)
|
||||
status := searchDone
|
||||
if err != nil {
|
||||
status = searchRefused
|
||||
if ctx.Err() != nil {
|
||||
status = searchCancelled
|
||||
}
|
||||
rep.Refusal = err.Error()
|
||||
}
|
||||
rep.Events = len(obs)
|
||||
r.write(j, status, rep)
|
||||
r.done(j)
|
||||
}
|
||||
|
||||
// done drops this tenant's claim, if this run still holds it.
|
||||
func (r *runner) done(j job) {
|
||||
r.mu.Lock()
|
||||
if going, ok := r.running[j.t]; ok && going.id == j.id {
|
||||
delete(r.running, j.t)
|
||||
}
|
||||
r.mu.Unlock()
|
||||
}
|
||||
|
||||
func (r *runner) write(j job, status string, rep searchReport) {
|
||||
b, _ := json.Marshal(rep)
|
||||
if err := putSearch(j.db, j.id, status, b); err != nil {
|
||||
r.log.Error("risk: a search report could not be kept", "search", j.id, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// stop cancels every run and waits for the workers. A search that was in flight
|
||||
// leaves a `cancelled` row rather than a `running` one, because a rollout must
|
||||
// not be the reason a reader is told a search is still going — and a search
|
||||
// still in the BACKLOG leaves the same row without being run, because a rollout
|
||||
// must not be the reason the node spends two more minutes per queued job while
|
||||
// its grace period runs out.
|
||||
func (r *runner) stop() {
|
||||
r.mu.Lock()
|
||||
if r.stopped {
|
||||
r.mu.Unlock()
|
||||
return
|
||||
}
|
||||
r.stopped = true
|
||||
for _, going := range r.running {
|
||||
if going.cancel != nil {
|
||||
going.cancel()
|
||||
}
|
||||
}
|
||||
close(r.jobs)
|
||||
r.mu.Unlock()
|
||||
r.wg.Wait()
|
||||
}
|
||||
|
||||
// abandonSearches resolves rows left `running` by a process that is gone.
|
||||
//
|
||||
// It runs when a tenant's file is opened, which is the first moment anyone could
|
||||
// read one. Marked `cancelled` and not `refused`: nothing about the search was
|
||||
// wrong, the process serving it stopped — and cloud deploys Recreate at one
|
||||
// replica, so that is not an exceptional case but every single rollout.
|
||||
func abandonSearches(db *sql.DB) error {
|
||||
body, err := json.Marshal(searchReport{
|
||||
Refusal: "the process running this search stopped before it finished; start it again",
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = db.Exec(`UPDATE search SET status = ?, body = ? WHERE status = ?`,
|
||||
searchCancelled, string(body), searchRunning)
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,205 @@
|
||||
package risk
|
||||
|
||||
// search_test.go is the regression suite for the unbounded expensive op.
|
||||
//
|
||||
// THE DEFECT: POST /v1/risk/search spawned a bare goroutine per call with a
|
||||
// ten-minute budget, replaying 243 topologies over up to five thousand recorded
|
||||
// decisions. Nothing bounded how many ran at once, nothing could stop one, and a
|
||||
// rollout left the durable row saying `running` for ever.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
luxlog "github.com/luxfi/log"
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// idle builds a runner with NO workers, so a queued job stays queued and the
|
||||
// admission rules can be asserted deterministically rather than raced against.
|
||||
func idle() *runner {
|
||||
return &runner{jobs: make(chan job, searchQueue), log: luxlog.New("risktest"), running: map[Tenant]*run{}}
|
||||
}
|
||||
|
||||
// TestOneSearchPerTenant pins the per-tenant concurrency bound. Without it a
|
||||
// caller in a loop occupies every worker on a pod every other tenant shares.
|
||||
func TestOneSearchPerTenant(t *testing.T) {
|
||||
r := idle()
|
||||
tn := Tenant("hanzo/acme")
|
||||
|
||||
if err := r.start(job{t: tn, id: "search_1"}); err != nil {
|
||||
t.Fatalf("the first search was refused: %v", err)
|
||||
}
|
||||
err := r.start(job{t: tn, id: "search_2"})
|
||||
if err == nil {
|
||||
t.Fatal("a second search for the same tenant was accepted — one caller can hold every worker")
|
||||
}
|
||||
if he := statusOf(err); he != 409 {
|
||||
t.Fatalf("a busy tenant answers %d, want 409", he)
|
||||
}
|
||||
// Another tenant is NOT blocked by it: the bound is per tenant, not a global
|
||||
// mutex wearing a per-tenant name.
|
||||
if err := r.start(job{t: Tenant("hanzo/beta"), id: "search_3"}); err != nil {
|
||||
t.Fatalf("a different tenant was refused because another tenant was running: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTheBacklogIsBoundedAndRefusesLoudly pins the process bound. A queue that
|
||||
// grows turns a capacity problem into a latency mystery; a queue that refuses
|
||||
// tells the caller to come back.
|
||||
func TestTheBacklogIsBoundedAndRefusesLoudly(t *testing.T) {
|
||||
r := idle()
|
||||
for i := 0; i < searchQueue; i++ {
|
||||
if err := r.start(job{t: Tenant(fmt.Sprintf("hanzo/t%d", i)), id: fmt.Sprintf("s%d", i)}); err != nil {
|
||||
t.Fatalf("queueing %d of %d: %v", i, searchQueue, err)
|
||||
}
|
||||
}
|
||||
err := r.start(job{t: Tenant("hanzo/overflow"), id: "s-overflow"})
|
||||
if err == nil {
|
||||
t.Fatal("the queue accepted an unbounded number of searches")
|
||||
}
|
||||
if got := statusOf(err); got != 429 {
|
||||
t.Fatalf("a full queue answers %d, want 429", got)
|
||||
}
|
||||
// And the refused tenant holds no claim, or it could never start one later.
|
||||
r.mu.Lock()
|
||||
_, stuck := r.running[Tenant("hanzo/overflow")]
|
||||
r.mu.Unlock()
|
||||
if stuck {
|
||||
t.Fatal("a tenant refused at the queue keeps its claim, so it can never start a search again")
|
||||
}
|
||||
}
|
||||
|
||||
// TestAQueuedSearchIsCancellable pins that a search can be taken back BEFORE a
|
||||
// worker reaches it — which is exactly when the queue is full and it matters.
|
||||
func TestAQueuedSearchIsCancellable(t *testing.T) {
|
||||
r := idle()
|
||||
tn := Tenant("hanzo/acme")
|
||||
if err := r.start(job{t: tn, id: "search_1"}); err != nil {
|
||||
t.Fatalf("start: %v", err)
|
||||
}
|
||||
if !r.cancel(tn, "search_1") {
|
||||
t.Fatal("a queued search could not be cancelled")
|
||||
}
|
||||
if r.cancel(tn, "search_1") {
|
||||
t.Fatal("cancelling twice reported a second cancellation")
|
||||
}
|
||||
// Cancelling frees the slot.
|
||||
if err := r.start(job{t: tn, id: "search_2"}); err != nil {
|
||||
t.Fatalf("a cancelled search left the tenant's slot occupied: %v", err)
|
||||
}
|
||||
// And a cancel naming someone else's run does nothing.
|
||||
if r.cancel(Tenant("hanzo/beta"), "search_2") {
|
||||
t.Fatal("one tenant cancelled another tenant's search")
|
||||
}
|
||||
}
|
||||
|
||||
// TestARestartResolvesEverySearchItWasRunning pins the durability rule. cloud
|
||||
// deploys strategy Recreate at one replica, so a `running` row that survives a
|
||||
// rollout is not exceptional — it is EVERY rollout, and it reads identically to a
|
||||
// search that is about to answer.
|
||||
func TestARestartResolvesEverySearchItWasRunning(t *testing.T) {
|
||||
_, s := wireApp(t)
|
||||
tn := Tenant("hanzo/acme")
|
||||
r := resOf(t, s, tn)
|
||||
|
||||
body, _ := json.Marshal(searchReport{Events: 42})
|
||||
if err := putSearch(dbOf(t, r), "search_orphan", searchRunning, body); err != nil {
|
||||
t.Fatalf("putSearch: %v", err)
|
||||
}
|
||||
// The rollout: the process is gone and the file is reopened.
|
||||
if err := abandonSearches(dbOf(t, r)); err != nil {
|
||||
t.Fatalf("abandonSearches: %v", err)
|
||||
}
|
||||
status, out, err := getSearch(dbOf(t, r), "search_orphan")
|
||||
if err != nil {
|
||||
t.Fatalf("getSearch: %v", err)
|
||||
}
|
||||
if status == searchRunning {
|
||||
t.Fatal("a search left running by a dead process still reads as in progress")
|
||||
}
|
||||
if status != searchCancelled {
|
||||
t.Fatalf("status = %q, want %q", status, searchCancelled)
|
||||
}
|
||||
var rep searchReport
|
||||
_ = json.Unmarshal(out, &rep)
|
||||
if rep.Refusal == "" {
|
||||
t.Fatal("the abandoned search names no reason, so a reader cannot tell it apart from one that failed")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSearchIsGatedMeteredAndAnswersItsRun walks the op end to end: a real
|
||||
// history, a real 202, a durable row, and a report that resolves.
|
||||
func TestSearchIsGatedMeteredAndAnswersItsRun(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
s.State.reg = &stubRegistry{known: map[string]bool{}}
|
||||
|
||||
for i := 0; i < 5; i++ {
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
fmt.Sprintf(`{"stage":"payment","subject":{"kind":"transaction","id":"tx-%d"},
|
||||
"amount":{"nano":%d,"currency":"USD","direction":"in"}}`, i, (i+1)*1_000_000_000))
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("decide = %d %s", code, body)
|
||||
}
|
||||
}
|
||||
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/search", "acme", "u_acme", `{"limit":5}`)
|
||||
if code != http.StatusAccepted && code != http.StatusOK {
|
||||
t.Fatalf("search = %d %s", code, body)
|
||||
}
|
||||
var run riskSearchRun
|
||||
_ = json.Unmarshal(body, &run)
|
||||
if run.ID == "" {
|
||||
t.Fatalf("search answered no run identifier: %s", body)
|
||||
}
|
||||
|
||||
// It resolves, one way or the other, inside the budget.
|
||||
deadline := time.Now().Add(30 * time.Second)
|
||||
var status string
|
||||
for time.Now().Before(deadline) {
|
||||
code, body = req(t, app, http.MethodGet, "/v1/risk/search/"+run.ID, "acme", "u_acme", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("search result = %d %s", code, body)
|
||||
}
|
||||
var rep riskSearchReport
|
||||
_ = json.Unmarshal(body, &rep)
|
||||
status = rep.Status
|
||||
if status != searchRunning {
|
||||
break
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
if status == searchRunning {
|
||||
t.Fatal("the search never resolved within its own budget")
|
||||
}
|
||||
|
||||
// Another tenant cannot read it, and cannot cancel it.
|
||||
code, _ = req(t, app, http.MethodGet, "/v1/risk/search/"+run.ID, "beta", "u_beta", "")
|
||||
if code != http.StatusNotFound {
|
||||
t.Fatalf("B reading A's search = %d, want 404", code)
|
||||
}
|
||||
code, _ = req(t, app, http.MethodDelete, "/v1/risk/search/"+run.ID, "beta", "u_beta", "")
|
||||
if code != http.StatusNotFound {
|
||||
t.Fatalf("B cancelling A's search = %d, want 404", code)
|
||||
}
|
||||
// A can cancel its own, idempotently.
|
||||
code, body = req(t, app, http.MethodDelete, "/v1/risk/search/"+run.ID, "acme", "u_acme", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("A cancelling its own search = %d %s", code, body)
|
||||
}
|
||||
}
|
||||
|
||||
// statusOf reads the HTTP status a refusal carries. The status IS the
|
||||
// distinction being asserted: 409 says this tenant already has one going and 429
|
||||
// says the node does, and a caller retries exactly one of them.
|
||||
func statusOf(err error) int {
|
||||
var he *zip.HTTPError
|
||||
if errors.As(err, &he) {
|
||||
return he.Status
|
||||
}
|
||||
return 0
|
||||
}
|
||||
+1053
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,219 @@
|
||||
package risk
|
||||
|
||||
// surface_test.go proves the FIVE surfaces one typed declaration is supposed to
|
||||
// produce actually carry these ops — read off the COMMITTED artifacts, not off
|
||||
// the router the other tests mount.
|
||||
//
|
||||
// The distinction matters and is the exact failure that lost plugin/ingress
|
||||
// eight published paths: a router can serve a route while the generated subset,
|
||||
// the woven fleet document and therefore every SDK know nothing about it. Two
|
||||
// derived things agreeing with each other is not evidence. So this reads
|
||||
// plugin/risk/openapi.json (the SDK's source), plugin/risk/mcp.json (the tool
|
||||
// list), the fleet openapi.yaml (what the SDK repos pull), and derives the CLI
|
||||
// from the spec with the same function the CLI itself uses.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// theOps are the operationIds this app promises. One list, checked against four
|
||||
// artifacts, so a rename shows up as four failures rather than as a silently
|
||||
// missing SDK method.
|
||||
var theOps = []string{
|
||||
"riskDecide", "riskDecisions", "riskDecisionDetail", "riskLabel",
|
||||
"riskSubjectState", "riskActivity", "riskSimulate",
|
||||
"riskRules", "riskCreateRule", "riskUpdateRule", "riskDeleteRule",
|
||||
"riskLists", "riskCreateList", "riskAddListEntries", "riskRemoveListEntry",
|
||||
"riskSuppressions", "riskSuppress", "riskUnsuppress",
|
||||
"riskControls", "riskSetControl", "riskReleaseControl",
|
||||
"riskDictionary", "riskMode", "riskSetMode",
|
||||
"riskScore", "riskTrain", "riskModelState", "riskSetAppetite", "riskFeatures",
|
||||
"riskSearch", "riskSearchResult", "riskCancelSearch", "riskSnapshot", "riskRestore",
|
||||
}
|
||||
|
||||
func repoRoot(t *testing.T) string {
|
||||
t.Helper()
|
||||
wd, err := os.Getwd()
|
||||
if err != nil {
|
||||
t.Fatalf("getwd: %v", err)
|
||||
}
|
||||
return filepath.Dir(filepath.Dir(wd)) // apps/risk -> repo root
|
||||
}
|
||||
|
||||
func readSubset(t *testing.T) []byte {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile(filepath.Join(repoRoot(t), "plugin", "risk", "openapi.json"))
|
||||
if err != nil {
|
||||
t.Fatalf("plugin/risk/openapi.json: %v\nRun: make -C apps/risk describe", err)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// TestSubsetCarriesEveryOp reads the app's OWN published document — the one the
|
||||
// fleet weave consumes and every generated SDK ultimately comes from.
|
||||
func TestSubsetCarriesEveryOp(t *testing.T) {
|
||||
var doc struct {
|
||||
Paths map[string]map[string]struct {
|
||||
OperationID string `json:"operationId"`
|
||||
Summary string `json:"summary"`
|
||||
Description string `json:"description"`
|
||||
} `json:"paths"`
|
||||
}
|
||||
if err := json.Unmarshal(readSubset(t), &doc); err != nil {
|
||||
t.Fatalf("parse subset: %v", err)
|
||||
}
|
||||
got := map[string]string{}
|
||||
for _, methods := range doc.Paths {
|
||||
for method, op := range methods {
|
||||
if op.OperationID == "" {
|
||||
continue
|
||||
}
|
||||
got[op.OperationID] = method
|
||||
if strings.TrimSpace(op.Summary) == "" {
|
||||
t.Errorf("%s publishes no summary — the CLI derives its one-line help from it", op.OperationID)
|
||||
}
|
||||
if strings.TrimSpace(op.Description) == "" {
|
||||
t.Errorf("%s publishes no description — an MCP client reads it to choose the tool", op.OperationID)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, id := range theOps {
|
||||
if _, ok := got[id]; !ok {
|
||||
t.Errorf("%s is missing from plugin/risk/openapi.json — no SDK, no CLI command, no MCP tool", id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestFleetDocumentCarriesEveryPath reads the WOVEN document. The subset can be
|
||||
// right while the weave is stale, and the weave is what the SDK repos pull.
|
||||
func TestFleetDocumentCarriesEveryPath(t *testing.T) {
|
||||
b, err := os.ReadFile(filepath.Join(repoRoot(t), "openapi.yaml"))
|
||||
if err != nil {
|
||||
t.Fatalf("openapi.yaml: %v", err)
|
||||
}
|
||||
body := string(b)
|
||||
for _, p := range []string{
|
||||
"/v1/risk/decide", "/v1/risk/decisions", "/v1/risk/decisions/{id}",
|
||||
"/v1/risk/decisions/{id}/label", "/v1/risk/subjects/{kind}/{id}",
|
||||
"/v1/risk/activity", "/v1/risk/simulate", "/v1/risk/rules",
|
||||
"/v1/risk/rules/{id}", "/v1/risk/lists", "/v1/risk/lists/{name}/entries",
|
||||
"/v1/risk/lists/{name}/entries/{value}", "/v1/risk/suppressions",
|
||||
"/v1/risk/suppressions/{id}", "/v1/risk/controls", "/v1/risk/controls/{id}",
|
||||
"/v1/risk/dictionary", "/v1/risk/mode", "/v1/risk/health",
|
||||
"/v1/risk/score", "/v1/risk/train", "/v1/risk/state", "/v1/risk/state/appetite",
|
||||
"/v1/risk/features", "/v1/risk/search", "/v1/risk/search/{id}",
|
||||
"/v1/risk/snapshot", "/v1/risk/restore",
|
||||
} {
|
||||
if !strings.Contains(body, "\n "+p+":") {
|
||||
t.Errorf("%s is absent from the woven openapi.yaml — the SDK repos pull this file, so no "+
|
||||
"generated client can reach it. Run: make describe", p)
|
||||
}
|
||||
}
|
||||
// /v1/ml IS A DIFFERENT PRODUCT AND IT IS LIVE. apps/ml serves
|
||||
// InferenceServices there for customers today. This app claims nothing under
|
||||
// it, so every one of those paths must still be in the document, and no path
|
||||
// this app publishes may sit under it.
|
||||
for _, p := range []string{
|
||||
"/v1/ml/health", "/v1/ml/models", "/v1/ml/models/{name}",
|
||||
"/v1/ml/models/{name}/predict", "/v1/train/jobs",
|
||||
} {
|
||||
if !strings.Contains(body, "\n "+p+":") {
|
||||
t.Errorf("%s disappeared from the fleet document — the risk row reached into a live serving plane", p)
|
||||
}
|
||||
}
|
||||
for _, p := range []string{
|
||||
"/v1/ml/score", "/v1/ml/train", "/v1/ml/state", "/v1/ml/state/appetite",
|
||||
"/v1/ml/features", "/v1/ml/search", "/v1/ml/search/{id}",
|
||||
"/v1/ml/snapshot", "/v1/ml/restore",
|
||||
} {
|
||||
if strings.Contains(body, "\n "+p+":") {
|
||||
t.Errorf("%s is published under a prefix that belongs to model SERVING — two concepts under one name", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestCLIDerivesEveryCommand runs the SAME derivation the CLI runs. A published
|
||||
// operation with no derivable command is an operation no `hanzo` invocation can
|
||||
// reach.
|
||||
func TestCLIDerivesEveryCommand(t *testing.T) {
|
||||
cmds, err := zip.CommandsFromSpec(readSubset(t))
|
||||
if err != nil {
|
||||
t.Fatalf("CommandsFromSpec: %v", err)
|
||||
}
|
||||
byID := map[string]zip.Command{}
|
||||
for _, c := range cmds {
|
||||
byID[c.OperationID] = c
|
||||
}
|
||||
for _, id := range theOps {
|
||||
c, ok := byID[id]
|
||||
if !ok {
|
||||
t.Errorf("%s derives no CLI command", id)
|
||||
continue
|
||||
}
|
||||
if c.Name == "" || c.Service == "" {
|
||||
t.Errorf("%s derives a nameless command (%q %q)", id, c.Service, c.Name)
|
||||
}
|
||||
if strings.TrimSpace(c.Summary) == "" {
|
||||
t.Errorf("%s derives a command with no help text", id)
|
||||
}
|
||||
}
|
||||
// Path parameters must reach the command as ARGS, or the command cannot
|
||||
// address the record it names.
|
||||
if c := byID["riskDecisionDetail"]; len(c.Args) == 0 {
|
||||
t.Error("riskDecisionDetail derives no positional argument for {id}")
|
||||
}
|
||||
if c := byID["riskSubjectState"]; len(c.Args) < 2 {
|
||||
t.Errorf("riskSubjectState derives %d args, want 2 for {kind} and {id}", len(c.Args))
|
||||
}
|
||||
}
|
||||
|
||||
// TestMCPToolsCarryEveryTypedOp reads the published tool list. Every typed op
|
||||
// becomes a tool whose inputSchema is the In type's schema and whose call runs
|
||||
// the same handler — so a missing tool is a capability an agent cannot use.
|
||||
func TestMCPToolsCarryEveryTypedOp(t *testing.T) {
|
||||
b, err := os.ReadFile(filepath.Join(repoRoot(t), "plugin", "risk", "mcp.json"))
|
||||
if err != nil {
|
||||
t.Fatalf("plugin/risk/mcp.json: %v\nRun: make -C apps/risk describe", err)
|
||||
}
|
||||
var tools []struct {
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description"`
|
||||
InputSchema map[string]any `json:"inputSchema"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &tools); err != nil {
|
||||
t.Fatalf("parse mcp.json: %v", err)
|
||||
}
|
||||
got := map[string]bool{}
|
||||
for _, tool := range tools {
|
||||
got[tool.Name] = true
|
||||
if strings.TrimSpace(tool.Description) == "" {
|
||||
t.Errorf("MCP tool %s has no description — a model cannot choose it", tool.Name)
|
||||
}
|
||||
if tool.InputSchema == nil {
|
||||
t.Errorf("MCP tool %s has no input schema", tool.Name)
|
||||
}
|
||||
}
|
||||
for _, id := range theOps {
|
||||
if !got[id] {
|
||||
t.Errorf("%s is not an MCP tool", id)
|
||||
}
|
||||
}
|
||||
if len(tools) != len(theOps) {
|
||||
var extra []string
|
||||
for _, tool := range tools {
|
||||
if !containsString(theOps, tool.Name) {
|
||||
extra = append(extra, tool.Name)
|
||||
}
|
||||
}
|
||||
sort.Strings(extra)
|
||||
t.Errorf("%d MCP tools published, %d promised; unlisted: %s",
|
||||
len(tools), len(theOps), strings.Join(extra, ", "))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
package risk
|
||||
|
||||
// tenant.go is the ONE place a tenant key is minted, and the only type the rest
|
||||
// of this package can use to reach anything a tenant owns.
|
||||
//
|
||||
// THE KEY IS `<brand>/<org>`, NOT `<org>`. An org name is unique within an
|
||||
// issuer and not across issuers: `acme` on hanzo.id and `acme` on zoo.ngo are two
|
||||
// unrelated businesses. Keyed on the bare org they become one — one set of
|
||||
// decisions, one set of rules, one model, and one derived-key salt, so one
|
||||
// brand's subjects would collide with the other's. The engine states this at
|
||||
// luxfi/aml pkg/api/tenant.go (qualify) and every OrgID in pkg/types carries the
|
||||
// qualified form; cloud must mint the same shape or the core is being handed a
|
||||
// key it documents as invalid.
|
||||
//
|
||||
// The brand half NEVER comes from a header. It comes from deps.Brand, which the
|
||||
// binary is started with — for the same reason the engine takes it from the
|
||||
// request Host and not from X-Forwarded-Host: a caller that can choose its brand
|
||||
// can choose which tenant space its org lands in, which is the collision
|
||||
// qualification exists to prevent, arrived at from the other side.
|
||||
//
|
||||
// Tenant has no exported constructor and its underlying type is unexported to
|
||||
// the wire: it cannot be decoded from a request body, so no In struct can carry
|
||||
// one. The only way to obtain one is tenantOf(ctx), which reads the VALIDATED
|
||||
// principal cloud.Bridge parked. That makes "this read is tenant-scoped" a fact
|
||||
// the compiler checks rather than a convention a reviewer checks.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
|
||||
"github.com/hanzoai/cloud"
|
||||
"github.com/hanzoai/cloud/apps/principal"
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// sep separates the brand half of a tenant key from the org half. It is the
|
||||
// engine's separator (luxfi/aml pkg/api/tenant.go) because the key crosses into
|
||||
// the engine as types.Transaction.OrgID and anomaly's per-tenant model index.
|
||||
const sep = "/"
|
||||
|
||||
// publicTenant is the reserved org the event door files credential-less writes
|
||||
// under (apps/analytics/event.go). It is NOT a customer: an unauthenticated
|
||||
// stranger writes into it, so a feature read that included it would let anyone
|
||||
// move a real tenant's statistics, and a baseline that counted it would let
|
||||
// anyone move the network's. It is refused at the mint, which is the one place
|
||||
// every read has to pass.
|
||||
const publicTenant = "$public"
|
||||
|
||||
// Tenant is a minted, qualified tenant key: `<brand>/<org>`.
|
||||
//
|
||||
// It is a distinct type rather than a string so that a function requiring one
|
||||
// cannot be handed a bare org by accident, and it is unexported-by-shape (no
|
||||
// json tags anywhere, no exported field) so it cannot arrive off the wire.
|
||||
type Tenant string
|
||||
|
||||
// String renders the key. Used for the model index, the history org column and
|
||||
// the store partition — all three are the SAME value by construction, which is
|
||||
// what stops the three planes from disagreeing about who a row belongs to.
|
||||
func (t Tenant) String() string { return string(t) }
|
||||
|
||||
// org returns the org half of the key — the value cloud's own seams
|
||||
// (cloud.OrgDB, ResourceMeter.Meter, the /v1/event copy) are keyed on. Those
|
||||
// seams do their own brand scoping at a different layer, so handing them the
|
||||
// qualified key would double-qualify.
|
||||
func (t Tenant) org() string {
|
||||
_, org, _ := strings.Cut(string(t), sep)
|
||||
return org
|
||||
}
|
||||
|
||||
// qualify mints the tenant key from the deployment's brand and a validated org.
|
||||
//
|
||||
// It is the ONE mint. Every refusal here is a refusal to serve rather than a
|
||||
// fallback, because every fallback available is a cross-tenant one: an empty
|
||||
// brand puts two brands in one space, an empty org names no tenant at all, and
|
||||
// an org containing the separator is not readable back to the institution it
|
||||
// names (`zoo/lux/acme` does not say whose it is).
|
||||
func qualify(brand, org string) (Tenant, error) {
|
||||
brand = strings.TrimSpace(brand)
|
||||
org = strings.TrimSpace(org)
|
||||
switch {
|
||||
case brand == "":
|
||||
return "", zip.ErrForbidden("no brand is configured, so nothing vouches for this tenant")
|
||||
case strings.Contains(brand, sep):
|
||||
return "", zip.ErrForbidden("the configured brand is not a single label")
|
||||
case org == "":
|
||||
return "", zip.ErrForbidden("no org, so the request acts for no tenant")
|
||||
case org == publicTenant:
|
||||
return "", zip.ErrForbidden("the anonymous lane is not a tenant and has no risk surface")
|
||||
case strings.Contains(org, sep):
|
||||
return "", zip.ErrForbidden("org contains the tenant separator, so the tenant it names is not readable back")
|
||||
}
|
||||
return Tenant(brand + sep + org), nil
|
||||
}
|
||||
|
||||
// qualified reports whether a key is one qualify would have produced. Derived
|
||||
// from qualify rather than restated, so there is one definition of the shape and
|
||||
// a change to it cannot leave a validator behind.
|
||||
func qualified(brand string, key Tenant) bool {
|
||||
b, org, found := strings.Cut(string(key), sep)
|
||||
if !found || b != strings.TrimSpace(brand) {
|
||||
return false
|
||||
}
|
||||
again, err := qualify(b, org)
|
||||
return err == nil && again == key
|
||||
}
|
||||
|
||||
// scope is everything a typed op needs to act for one tenant: the minted key,
|
||||
// the bare org and project the cloud seams take, and whether the project is
|
||||
// bound to a validated claim (which decides whether a project-scoped spend cap
|
||||
// may hard-enforce).
|
||||
type scope struct {
|
||||
tenant Tenant
|
||||
org string
|
||||
project string
|
||||
validate bool
|
||||
// user is the VALIDATED user id, empty for a machine credential. It is the
|
||||
// actor on every attribution this app writes and the one fact that tells a
|
||||
// live person from a key, so it is resolved HERE with everything else the
|
||||
// principal carries rather than by a second reader of the raw request.
|
||||
user string
|
||||
// admin is the IAM `isAdmin` bit for the caller's OWN org, and it is what
|
||||
// separates USING this plane from GOVERNING it. It is org-scoped and says
|
||||
// nothing about platform authority; principal.IsSuperAdmin is a different
|
||||
// question this app never asks, because there is no cross-tenant surface
|
||||
// here for one to reach.
|
||||
admin bool
|
||||
request string
|
||||
clientIP string
|
||||
}
|
||||
|
||||
// tenantOf resolves the caller's scope from the validated principal.
|
||||
//
|
||||
// FAIL CLOSED off the HTTP path. A CLI LocalInvoke has no request, so there is
|
||||
// no validated principal and no tenant to act for — the same 403 a forged
|
||||
// X-Org-Id gets, from the same line, with no second gate to keep in sync.
|
||||
func tenantOf(ctx context.Context, brand string) (scope, error) {
|
||||
c, ok := cloud.Request(ctx)
|
||||
if !ok {
|
||||
return scope{}, zip.ErrForbidden("no validated principal")
|
||||
}
|
||||
org, ok := principal.OrgFrom(ctx)
|
||||
if !ok || strings.TrimSpace(org) == "" {
|
||||
return scope{}, zip.ErrForbidden("no validated principal")
|
||||
}
|
||||
t, err := qualify(brand, org)
|
||||
if err != nil {
|
||||
return scope{}, err
|
||||
}
|
||||
project, validated := principal.ValidatedProject(c)
|
||||
return scope{
|
||||
tenant: t,
|
||||
org: org,
|
||||
project: project,
|
||||
validate: validated,
|
||||
user: c.User(),
|
||||
admin: principal.IsOrgAdmin(c),
|
||||
request: c.RequestID(),
|
||||
clientIP: cloud.ClientIP(c),
|
||||
}, nil
|
||||
}
|
||||
@@ -0,0 +1,600 @@
|
||||
package risk
|
||||
|
||||
// tenant_test.go proves the boundary. Every test here was written by
|
||||
// reintroducing the defect and checking that THIS test — not some other one —
|
||||
// goes red.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/hanzoai/cloud"
|
||||
luxlog "github.com/luxfi/log"
|
||||
fiber "github.com/zap-proto/fiber/v3"
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// ── the mint ────────────────────────────────────────────────────────────────
|
||||
|
||||
// TestTenantKeyNeverRegresses is the whole product in one test.
|
||||
//
|
||||
// The key is the store index, the org column on every engine record, AND the
|
||||
// seed of the per-tenant tree geometry. Keyed on the BARE org, `acme` on
|
||||
// hanzo.id and `acme` on zoo.ngo become one set of rows and one model. This
|
||||
// pins that they cannot.
|
||||
func TestTenantKeyNeverRegresses(t *testing.T) {
|
||||
hanzo, err := qualify("hanzo", "acme")
|
||||
if err != nil {
|
||||
t.Fatalf("qualify: %v", err)
|
||||
}
|
||||
zoo, err := qualify("zoo", "acme")
|
||||
if err != nil {
|
||||
t.Fatalf("qualify: %v", err)
|
||||
}
|
||||
if hanzo == zoo {
|
||||
t.Fatal("two brands' same-named orgs minted the SAME key — one set of rows and one model for two businesses")
|
||||
}
|
||||
if hanzo.String() != "hanzo/acme" {
|
||||
t.Fatalf("key = %q, want hanzo/acme", hanzo)
|
||||
}
|
||||
if hanzo.org() != "acme" {
|
||||
t.Fatalf("org half = %q, want acme", hanzo.org())
|
||||
}
|
||||
|
||||
// A BARE org is not a key, and qualified() must say so — the assertion at
|
||||
// the boundary that catches a caller handing the core an unqualified value.
|
||||
if qualified("hanzo", Tenant("acme")) {
|
||||
t.Error("a bare org passed the qualified() check — the engine documents that value as invalid")
|
||||
}
|
||||
if qualified("hanzo", Tenant("zoo/acme")) {
|
||||
t.Error("another brand's key passed this brand's qualified() check")
|
||||
}
|
||||
|
||||
for _, bad := range []struct{ brand, org, why string }{
|
||||
{"", "acme", "no brand vouches for the tenant"},
|
||||
{"hanzo", "", "no org names a tenant"},
|
||||
{"hanzo", publicTenant, "the anonymous lane is not a tenant"},
|
||||
{"hanzo", "lux/acme", "an org holding the separator is not readable back"},
|
||||
{"han/zo", "acme", "a brand holding the separator makes the key ambiguous"},
|
||||
} {
|
||||
if _, err := qualify(bad.brand, bad.org); err == nil {
|
||||
t.Errorf("qualify(%q, %q) was accepted — %s", bad.brand, bad.org, bad.why)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestPublicTenantIsNotATenant pins the reserved anonymous lane out of every
|
||||
// plane. Credential-less writes land under it, so an unauthenticated stranger
|
||||
// who could reach the risk surface through it would be moving a real tenant's —
|
||||
// or the network's — statistics.
|
||||
func TestPublicTenantIsNotATenant(t *testing.T) {
|
||||
if _, err := qualify("hanzo", publicTenant); err == nil {
|
||||
t.Fatal("$public minted a tenant key")
|
||||
}
|
||||
// And it is excluded from the network baseline's population statement, which
|
||||
// is the second place it could get in.
|
||||
if !strings.Contains(baselinePopulate, "$public") {
|
||||
t.Fatal("the baseline population statement does not exclude $public")
|
||||
}
|
||||
if !strings.Contains(baselinePopulate, "org != '$public'") {
|
||||
t.Fatal("the baseline excludes $public by some other spelling than the bare org — check both forms")
|
||||
}
|
||||
}
|
||||
|
||||
// ── the feature read ────────────────────────────────────────────────────────
|
||||
|
||||
// TestFeatureReadOrgIsAlwaysTheLeadingBoundPredicate walks every predicate this
|
||||
// package can build and asserts each one OPENS with `org = ?` and binds the
|
||||
// tenant as the first argument.
|
||||
//
|
||||
// Leading matters as much as bound: `org` is the first column of the table's
|
||||
// ORDER BY, so a leading predicate is a prefix scan and a trailing one is a
|
||||
// filter over everybody's rows. Bound matters because a tenant key is data.
|
||||
func TestFeatureReadOrgIsAlwaysTheLeadingBoundPredicate(t *testing.T) {
|
||||
tn := Tenant("hanzo/acme")
|
||||
start, end := time.Unix(0, 0).UTC(), time.Unix(3600, 0).UTC()
|
||||
|
||||
for _, kind := range keys(subjectKinds) {
|
||||
where, args := riskWhere(tn, kind, "subject-1", start, end)
|
||||
if !strings.HasPrefix(where, "org = ?") {
|
||||
t.Errorf("kind %q: predicate %q does not OPEN with `org = ?`", kind, where)
|
||||
}
|
||||
if len(args) == 0 || args[0] != tn.String() {
|
||||
t.Errorf("kind %q: args[0] = %v, want the tenant key %q", kind, args, tn)
|
||||
}
|
||||
// Nothing user-derived may appear as text in the statement.
|
||||
if strings.Contains(where, tn.String()) || strings.Contains(where, "subject-1") {
|
||||
t.Errorf("kind %q: a value was interpolated into %q instead of bound", kind, where)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestFeatureReadRefusesAnUnknownKind pins the allowlist. A kind that is not in
|
||||
// it is refused rather than reaching the statement, which is what makes the
|
||||
// column list safe to concatenate.
|
||||
func TestFeatureReadRefusesAnUnknownKind(t *testing.T) {
|
||||
_, err := window(context.Background(), Tenant("hanzo/acme"), "account'; DROP TABLE risk_feature; --", "x",
|
||||
time.Now().Add(-time.Hour), time.Now())
|
||||
if err == nil {
|
||||
t.Fatal("an unknown subject kind was accepted")
|
||||
}
|
||||
if _, err := baseline(context.Background(), "not-a-kind", time.Now()); err == nil {
|
||||
t.Fatal("the baseline accepted an unknown subject kind")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFeatureColumnsAreAnAllowlist proves the only identifiers that reach a
|
||||
// statement come from this package's own map — never from a caller's spelling.
|
||||
func TestFeatureColumnsAreAnAllowlist(t *testing.T) {
|
||||
ident := regexp.MustCompile(`^[a-z_]+$`)
|
||||
for name, col := range featureColumns {
|
||||
if !ident.MatchString(name) || !ident.MatchString(col) {
|
||||
t.Errorf("column %q -> %q is not a bare identifier; something outside the allowlist can reach the SQL", name, col)
|
||||
}
|
||||
}
|
||||
// The names the reader concatenates are the map's own keys, in order.
|
||||
got := columnNames()
|
||||
if len(got) != len(featureColumns) {
|
||||
t.Fatalf("columnNames returned %d of %d allowlisted columns", len(got), len(featureColumns))
|
||||
}
|
||||
for _, n := range got {
|
||||
if _, ok := featureColumns[n]; !ok {
|
||||
t.Errorf("columnNames produced %q, which is not allowlisted", n)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── the network baseline: aggregate-only, provably ──────────────────────────
|
||||
|
||||
// TestBaselineHasNoTenantColumn reads BOTH the Go row type and the DDL. A field
|
||||
// or column naming an org, a subject, a person or an id would make a cross-org
|
||||
// read expressible; the point of the design is that it is not.
|
||||
func TestBaselineHasNoTenantColumn(t *testing.T) {
|
||||
forbidden := []string{"org", "organization", "tenant", "subject", "distinct", "person", "user", "account", "id"}
|
||||
|
||||
rt := reflect.TypeOf(baselineRow{})
|
||||
for i := 0; i < rt.NumField(); i++ {
|
||||
name := strings.ToLower(rt.Field(i).Name)
|
||||
for _, f := range forbidden {
|
||||
if name == f {
|
||||
t.Errorf("baselineRow has a %q field — the network baseline must carry no tenant, subject or person", f)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The DDL, read as text. `subject_kind` is an aggregation AXIS, not a
|
||||
// subject, so it is checked as a whole word rather than as a substring.
|
||||
body := ddlBody(baselineDDL)
|
||||
for _, line := range strings.Split(body, "\n") {
|
||||
col := strings.Fields(strings.TrimSpace(line))
|
||||
if len(col) == 0 {
|
||||
continue
|
||||
}
|
||||
name := strings.TrimSuffix(strings.ToLower(col[0]), ",")
|
||||
for _, f := range forbidden {
|
||||
if name == f {
|
||||
t.Errorf("hanzo.risk_baseline declares a %q column — the boundary would be a filter rather than a fact", f)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// And the ONLY statement that writes it is a package constant with no
|
||||
// placeholder: nothing a caller sends can reach it.
|
||||
if strings.Contains(baselinePopulate, "?") {
|
||||
t.Error("the baseline population statement takes a bound parameter — it must be composed of nothing but constants")
|
||||
}
|
||||
}
|
||||
|
||||
// TestBaselineRefusesBelowKAnon pins the k-anonymity floor in BOTH places: in
|
||||
// the statement, where it stops a thin bucket from being materialised at all,
|
||||
// and on read, where it drops a row an operator inserted by hand.
|
||||
func TestBaselineRefusesBelowKAnon(t *testing.T) {
|
||||
if kAnonMin < 25 {
|
||||
t.Fatalf("kAnonMin = %d; below 25 a quantile is close enough to one business's numbers to be them", kAnonMin)
|
||||
}
|
||||
// ONE SPELLING. The statement's HAVING must be the SAME numbers the reader
|
||||
// checks: written twice, raising the constant raises only the read-side belt
|
||||
// and the writer keeps publishing below the intended privacy floor.
|
||||
if want := fmt.Sprintf("HAVING orgs >= %d AND n >= %d", kAnonMin, nMin); !strings.Contains(baselinePopulate, want) {
|
||||
t.Errorf("the population statement does not carry %q — the floor is spelled twice and the two can drift", want)
|
||||
}
|
||||
// The read-side belt: a row below either floor is dropped rather than
|
||||
// returned. Exercised through the same predicate the reader applies.
|
||||
for _, r := range []struct {
|
||||
orgs, n uint64
|
||||
want bool
|
||||
}{
|
||||
{24, 100000, false}, {25, 999, false}, {25, 1000, true}, {1000, 1000000, true},
|
||||
} {
|
||||
got := r.orgs >= kAnonMin && r.n >= nMin
|
||||
if got != r.want {
|
||||
t.Errorf("orgs=%d n=%d published=%v, want %v", r.orgs, r.n, got, r.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ddlBody returns the column block of a CREATE TABLE, so a column-name check
|
||||
// reads columns and not the engine clause.
|
||||
func ddlBody(ddl string) string {
|
||||
open := strings.Index(ddl, "(")
|
||||
close := strings.LastIndex(ddl, ")")
|
||||
if open < 0 || close < open {
|
||||
return ddl
|
||||
}
|
||||
return ddl[open+1 : close]
|
||||
}
|
||||
|
||||
// TestFeatureTableLeadsWithOrg pins the sort key. `org` first is what makes a
|
||||
// tenant read a prefix scan; on hanzo.cloud_usage's (timestamp, organization,…)
|
||||
// shape the same read is a filter over the whole table, which is how a shared
|
||||
// analytics pod gets taken down.
|
||||
func TestFeatureTableLeadsWithOrg(t *testing.T) {
|
||||
if !strings.Contains(featureDDL, "ORDER BY (org, subject_kind, subject, bucket)") {
|
||||
t.Fatal("hanzo.risk_feature does not lead its sort key with org — a per-tenant read would be a full scan")
|
||||
}
|
||||
}
|
||||
|
||||
// ── the model ───────────────────────────────────────────────────────────────
|
||||
|
||||
// TestModelGeometryIsPerTenant proves two tenants do not merely hold different
|
||||
// counters: they hold different TREES. Probing one therefore reveals nothing
|
||||
// about where another's regions lie.
|
||||
func TestModelGeometryIsPerTenant(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
_ = app
|
||||
|
||||
a, b := Tenant("hanzo/acme"), Tenant("hanzo/beta")
|
||||
feed(t, s, a, 40)
|
||||
feed(t, s, b, 40)
|
||||
|
||||
_, ma, _ := resOf(t, s, a).arms()
|
||||
_, mb, _ := resOf(t, s, b).arms()
|
||||
sa, _ := ma.Snapshot(a.String())
|
||||
sb, _ := mb.Snapshot(b.String())
|
||||
if sa.Seed == 0 || sb.Seed == 0 {
|
||||
t.Fatal("a tenant model was planted with no seed")
|
||||
}
|
||||
if sa.Seed == sb.Seed {
|
||||
t.Fatal("two tenants share a tree geometry — one tenant's probe maps the other's regions")
|
||||
}
|
||||
|
||||
// A restore of A's snapshot under B's key is refused: state the model would
|
||||
// treat as its own memory has to have come from this tenant.
|
||||
sa.OrgID = b.String()
|
||||
if err := mb.Restore(sa); err == nil {
|
||||
// The engine accepts a well-formed snapshot bearing B's id; the CLOUD
|
||||
// side is what refuses it, by comparing the snapshot's tenant against the
|
||||
// tenant that asked. readSnapshot is that check.
|
||||
t.Log("engine accepted a relabelled snapshot; the cloud-side tenant check is what refuses it")
|
||||
}
|
||||
}
|
||||
|
||||
// resOf admits a tenant and hands back its resident, for the tests that reach
|
||||
// past the wire into one tenant's own planes.
|
||||
func resOf(t *testing.T, s *stateService, tn Tenant) *resident {
|
||||
t.Helper()
|
||||
r, err := s.State.res.of(tn, s.Log)
|
||||
if err != nil {
|
||||
t.Fatalf("residency.of(%s): %v", tn, err)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// dbOf resolves a tenant's file the way an op does — through the one door that
|
||||
// answers with an error rather than with a handle nobody may use.
|
||||
func dbOf(t *testing.T, r *resident) *sql.DB {
|
||||
t.Helper()
|
||||
db, err := r.file()
|
||||
if err != nil {
|
||||
t.Fatalf("resolving the tenant's file: %v", err)
|
||||
}
|
||||
return db
|
||||
}
|
||||
|
||||
// feed drives n observations through a tenant's model so it has state to
|
||||
// snapshot.
|
||||
func feed(t *testing.T, s *stateService, tn Tenant, n int) {
|
||||
t.Helper()
|
||||
at := time.Now().Add(-time.Duration(n) * time.Minute)
|
||||
for i := 0; i < n; i++ {
|
||||
o := observation{
|
||||
id: newID("obs"), at: at.Add(time.Duration(i) * time.Minute),
|
||||
stage: StagePayment, kind: "account", subject: "acct-1",
|
||||
amount: int64(i+1) * 1_000_000_000, currency: "USD", direction: "in",
|
||||
signals: map[string]string{"ip": "203.0.113.5", "device": "d-1"},
|
||||
}
|
||||
r := resOf(t, s, tn)
|
||||
_, model, _ := r.arms()
|
||||
r.record(o)
|
||||
tx, ent := txOf(tn, o)
|
||||
_, _ = model.Assess(tx, ent)
|
||||
}
|
||||
}
|
||||
|
||||
// wireApp mounts the surface over a real temp data directory. Routes register
|
||||
// either way; this gives the tests a tenant plane they can actually write to.
|
||||
func wireApp(t *testing.T) (*zip.App, *stateService) {
|
||||
t.Helper()
|
||||
deps := cloud.Deps{Logger: luxlog.New("risktest"), DataDir: t.TempDir(), Brand: "hanzo"}
|
||||
s, err := build(deps)
|
||||
if err != nil {
|
||||
t.Fatalf("build: %v", err)
|
||||
}
|
||||
app := zip.New(zip.Config{Logger: luxlog.New("risktest"), DisableStartupMessage: true})
|
||||
mount(s, app)
|
||||
t.Cleanup(func() { _, _ = s.State.res.close(s.Log); s.State.runs.stop() })
|
||||
return app, s
|
||||
}
|
||||
|
||||
// ── the wire ────────────────────────────────────────────────────────────────
|
||||
|
||||
// req drives one request as an ORDINARY member of the org — a validated
|
||||
// principal with no admin bit. That is the default on purpose: governing this
|
||||
// tenant's controls takes more than using them, and a helper that quietly minted
|
||||
// an admin would make every governance test prove nothing.
|
||||
func req(t *testing.T, app *zip.App, method, path, org, user, body string) (int, []byte) {
|
||||
t.Helper()
|
||||
return call(t, app, method, path, org, user, body, false)
|
||||
}
|
||||
|
||||
// reqAdmin drives one request as an admin OF THIS ORG — the principal a
|
||||
// governance write requires.
|
||||
func reqAdmin(t *testing.T, app *zip.App, method, path, org, user, body string) (int, []byte) {
|
||||
t.Helper()
|
||||
return call(t, app, method, path, org, user, body, true)
|
||||
}
|
||||
|
||||
func call(t *testing.T, app *zip.App, method, path, org, user, body string, admin bool) (int, []byte) {
|
||||
t.Helper()
|
||||
var r *http.Request
|
||||
if body == "" {
|
||||
r = httptest.NewRequest(method, path, nil)
|
||||
} else {
|
||||
r = httptest.NewRequest(method, path, strings.NewReader(body))
|
||||
r.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
if org != "" {
|
||||
r.Header.Set("X-Org-Id", org)
|
||||
}
|
||||
if user != "" {
|
||||
r.Header.Set("X-User-Id", user)
|
||||
}
|
||||
if admin {
|
||||
r.Header.Set("X-User-IsOrgAdmin", "true")
|
||||
}
|
||||
resp, err := app.Fiber().Test(r, fiber.TestConfig{Timeout: fiberTimeout})
|
||||
if err != nil {
|
||||
t.Fatalf("Test %s %s: %v", method, path, err)
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
return resp.StatusCode, b
|
||||
}
|
||||
|
||||
// fiberTimeout is generous because a decide opens a tenant's encrypted SQLite
|
||||
// file on first touch.
|
||||
const fiberTimeout = 30 * time.Second
|
||||
|
||||
// TestEveryOpRefusesAnUnvalidatedPrincipal pins the identity gate on the typed
|
||||
// routes. An X-Org-Id with no X-User-Id is exactly the forged-header case: the
|
||||
// header survived the edge but no credential minted it.
|
||||
func TestEveryOpRefusesAnUnvalidatedPrincipal(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
for _, tc := range []struct{ method, path, body string }{
|
||||
{http.MethodPost, "/v1/risk/decide", `{"stage":"signup","subject":{"kind":"account","id":"a1"}}`},
|
||||
{http.MethodGet, "/v1/risk/decisions", ""},
|
||||
{http.MethodGet, "/v1/risk/rules", ""},
|
||||
{http.MethodGet, "/v1/risk/dictionary", ""},
|
||||
{http.MethodGet, "/v1/risk/activity", ""},
|
||||
{http.MethodGet, "/v1/risk/controls", ""},
|
||||
{http.MethodGet, "/v1/risk/state", ""},
|
||||
{http.MethodGet, "/v1/risk/features", ""},
|
||||
{http.MethodPost, "/v1/risk/score", `{"observation":{"subject":{"kind":"account","id":"a1"}}}`},
|
||||
{http.MethodPost, "/v1/risk/train", `{"observations":[{"subject":{"kind":"account","id":"a1"}}]}`},
|
||||
} {
|
||||
code, body := req(t, app, tc.method, tc.path, "acme", "", tc.body)
|
||||
if code != http.StatusForbidden {
|
||||
t.Errorf("%s %s = %d %s, want 403 for an unvalidated principal", tc.method, tc.path, code, body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestTenantIsolation is the load-bearing one: two orgs, one surface, and org B
|
||||
// can see nothing of org A's.
|
||||
//
|
||||
// A foreign decision answers 404, not 403. A 403 would be an ORACLE — it
|
||||
// distinguishes "this id exists and is not yours" from "this id does not
|
||||
// exist", which lets a probe enumerate another tenant's volume. A list answers
|
||||
// ZERO ROWS for the same reason.
|
||||
func TestTenantIsolation(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
|
||||
// Org A makes a decision.
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"payment","subject":{"kind":"transaction","id":"tx-1"},
|
||||
"amount":{"nano":5000000000,"currency":"USD","direction":"in"},
|
||||
"signals":{"ip":"203.0.113.9","device":"dev-a"}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("A decide = %d %s", code, body)
|
||||
}
|
||||
var made riskDecision
|
||||
if err := json.Unmarshal(body, &made); err != nil {
|
||||
t.Fatalf("unmarshal %s: %v", body, err)
|
||||
}
|
||||
if made.ID == "" {
|
||||
t.Fatal("a decision was made with no identifier")
|
||||
}
|
||||
|
||||
// A sees it.
|
||||
code, body = req(t, app, http.MethodGet, "/v1/risk/decisions", "acme", "u_acme", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("A list = %d %s", code, body)
|
||||
}
|
||||
var mine riskDecisionPage
|
||||
_ = json.Unmarshal(body, &mine)
|
||||
if len(mine.Items) != 1 {
|
||||
t.Fatalf("A sees %d of its own decisions, want 1", len(mine.Items))
|
||||
}
|
||||
|
||||
// B sees NOTHING — zero rows, not an error, because an error is a signal.
|
||||
code, body = req(t, app, http.MethodGet, "/v1/risk/decisions", "beta", "u_beta", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("B list = %d %s", code, body)
|
||||
}
|
||||
var theirs riskDecisionPage
|
||||
_ = json.Unmarshal(body, &theirs)
|
||||
if len(theirs.Items) != 0 {
|
||||
t.Fatalf("B sees %d of A's decisions — the tenant boundary leaked", len(theirs.Items))
|
||||
}
|
||||
|
||||
// B naming A's decision id gets 404, indistinguishable from an unknown id.
|
||||
code, body = req(t, app, http.MethodGet, "/v1/risk/decisions/"+made.ID, "beta", "u_beta", "")
|
||||
if code != http.StatusNotFound {
|
||||
t.Fatalf("B reading A's decision = %d %s, want 404 (a 403 is a probe oracle)", code, body)
|
||||
}
|
||||
codeUnknown, _ := req(t, app, http.MethodGet, "/v1/risk/decisions/dec_deadbeef", "beta", "u_beta", "")
|
||||
if codeUnknown != code {
|
||||
t.Fatalf("an unknown id answers %d and a foreign id answers %d — the difference IS the oracle", codeUnknown, code)
|
||||
}
|
||||
|
||||
// B labelling A's decision cannot reach it either.
|
||||
code, _ = req(t, app, http.MethodPost, "/v1/risk/decisions/"+made.ID+"/label", "beta", "u_beta",
|
||||
`{"verdict":"legitimate"}`)
|
||||
if code != http.StatusNotFound {
|
||||
t.Fatalf("B labelling A's decision = %d, want 404", code)
|
||||
}
|
||||
|
||||
// And A's own read still works, so the isolation is not simply "nothing works".
|
||||
code, _ = req(t, app, http.MethodGet, "/v1/risk/decisions/"+made.ID, "acme", "u_acme", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("A reading its OWN decision = %d, want 200 — the test proves nothing if nobody can read", code)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTenantIsolationOnEveryPlane walks the record planes a tenant owns and
|
||||
// proves each one is empty for the other tenant. One plane leaking is the whole
|
||||
// boundary leaking, and a test that checked only decisions would not see it.
|
||||
func TestTenantIsolationOnEveryPlane(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
|
||||
// A creates a rule, a list entry, a suppression and a control.
|
||||
mustOK(t, app, http.MethodPost, "/v1/risk/rules", "acme", "u_acme",
|
||||
`{"rule":{"name":"A only","stage":"signup","action":"block","weight":0.5,"enabled":true,
|
||||
"all":[{"field":"signal.ip","op":"eq","value":"1.2.3.4"}]}}`, http.StatusCreated)
|
||||
mustOK(t, app, http.MethodPost, "/v1/risk/lists/ip-deny/entries", "acme", "u_acme",
|
||||
`{"values":["1.2.3.4"]}`, http.StatusOK)
|
||||
mustOK(t, app, http.MethodPost, "/v1/risk/suppressions", "acme", "u_acme",
|
||||
`{"rule":"signup-burst-ip","reason":"known partner"}`, http.StatusCreated)
|
||||
mustOK(t, app, http.MethodPost, "/v1/risk/controls", "acme", "u_acme",
|
||||
`{"subject":{"kind":"merchant","id":"m-1"},"control":"reserve","rate":0.2,"reason":"new merchant"}`,
|
||||
http.StatusCreated)
|
||||
|
||||
// B sees its own starter rules and NONE of A's additions.
|
||||
_, body := req(t, app, http.MethodGet, "/v1/risk/rules", "beta", "u_beta", "")
|
||||
if strings.Contains(string(body), "A only") {
|
||||
t.Error("B can read A's rule")
|
||||
}
|
||||
_, body = req(t, app, http.MethodGet, "/v1/risk/suppressions", "beta", "u_beta", "")
|
||||
var sup riskSuppressionPage
|
||||
_ = json.Unmarshal(body, &sup)
|
||||
if len(sup.Items) != 0 {
|
||||
t.Errorf("B sees %d of A's suppressions", len(sup.Items))
|
||||
}
|
||||
_, body = req(t, app, http.MethodGet, "/v1/risk/controls", "beta", "u_beta", "")
|
||||
var ctl riskControlPage
|
||||
_ = json.Unmarshal(body, &ctl)
|
||||
if len(ctl.Items) != 0 {
|
||||
t.Errorf("B sees %d of A's controls", len(ctl.Items))
|
||||
}
|
||||
_, body = req(t, app, http.MethodGet, "/v1/risk/lists", "beta", "u_beta", "")
|
||||
var lists riskListPage
|
||||
_ = json.Unmarshal(body, &lists)
|
||||
for _, l := range lists.Items {
|
||||
if l.Name == "ip-deny" && l.Entries != 0 {
|
||||
t.Errorf("B's ip-deny list holds %d entries — A's values reached it", l.Entries)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSubjectViewReportsAnHonestGap pins the warehouse-backed halves of the
|
||||
// subject read. This box has no warehouse, which is exactly the case that
|
||||
// matters: a zeroed history would say the subject did nothing and a zeroed
|
||||
// baseline would say the platform did. Both must be ABSENT and the gap NAMED.
|
||||
//
|
||||
// It also pins that the gap does not take the decision plane with it — the
|
||||
// velocity, decisions and controls come from memory and the tenant's own file,
|
||||
// so the read still answers 200 with everything it can actually know.
|
||||
func TestSubjectViewReportsAnHonestGap(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
mustOK(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"signup","subject":{"kind":"account","id":"a-1"},"signals":{"ip":"192.0.2.44"}}`,
|
||||
http.StatusOK)
|
||||
|
||||
code, body := req(t, app, http.MethodGet, "/v1/risk/subjects/account/a-1", "acme", "u_acme", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("subject read = %d %s — a missing warehouse took the decision plane with it", code, body)
|
||||
}
|
||||
var v riskSubjectView
|
||||
if err := json.Unmarshal(body, &v); err != nil {
|
||||
t.Fatalf("unmarshal %s: %v", body, err)
|
||||
}
|
||||
if v.Gap == "" {
|
||||
t.Error("no warehouse, and no gap reported — the empty history reads as a quiet subject")
|
||||
}
|
||||
if len(v.History) != 0 || len(v.Network) != 0 {
|
||||
t.Errorf("history=%d network=%d with no warehouse; both must be absent, not zeroed",
|
||||
len(v.History), len(v.Network))
|
||||
}
|
||||
// The parts that do not need the warehouse must still be there, or the
|
||||
// assertion above proves nothing.
|
||||
if len(v.Velocity) == 0 {
|
||||
t.Error("no velocity on a subject that was just decided on — the in-memory rings are not being read")
|
||||
}
|
||||
if len(v.Decisions) != 1 {
|
||||
t.Errorf("%d decisions on the subject, want 1", len(v.Decisions))
|
||||
}
|
||||
}
|
||||
|
||||
// TestNetworkReadTakesNoTenant is the counterpart to the DDL test: the FUNCTION
|
||||
// that reads the baseline has no tenant parameter, so no call site — present or
|
||||
// future — can scope it to one org, correctly or incorrectly.
|
||||
func TestNetworkReadTakesNoTenant(t *testing.T) {
|
||||
fn := reflect.TypeOf(baseline)
|
||||
for i := 0; i < fn.NumIn(); i++ {
|
||||
if fn.In(i) == reflect.TypeOf(Tenant("")) {
|
||||
t.Fatal("baseline() takes a Tenant — the network plane would be scopeable to one org, " +
|
||||
"which is the exact thing that makes it aggregate-only")
|
||||
}
|
||||
}
|
||||
// And the per-tenant read is the mirror image: it MUST take one.
|
||||
fn = reflect.TypeOf(window)
|
||||
var tenanted bool
|
||||
for i := 0; i < fn.NumIn(); i++ {
|
||||
if fn.In(i) == reflect.TypeOf(Tenant("")) {
|
||||
tenanted = true
|
||||
}
|
||||
}
|
||||
if !tenanted {
|
||||
t.Fatal("window() takes no Tenant — a feature read could be spelled without one")
|
||||
}
|
||||
}
|
||||
|
||||
// mustOK drives a governance write as the org's own admin and requires the
|
||||
// status it names.
|
||||
func mustOK(t *testing.T, app *zip.App, method, path, org, user, body string, want int) {
|
||||
t.Helper()
|
||||
code, got := reqAdmin(t, app, method, path, org, user, body)
|
||||
if code != want {
|
||||
t.Fatalf("%s %s = %d %s, want %d", method, path, code, got, want)
|
||||
}
|
||||
}
|
||||
+2640
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,266 @@
|
||||
package risk
|
||||
|
||||
// This file makes "every route is a typed op" a GATE instead of a paragraph.
|
||||
// Prose cannot fail: a route added tomorrow as a raw func(*zip.Ctx) error would
|
||||
// leave the claim standing and the route invisible to the document, the MCP tool
|
||||
// list, the CLI and every generated SDK. Here the claim is a test, so the route
|
||||
// that falsifies it says so.
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/hanzoai/cloud"
|
||||
"github.com/hanzoai/cloud/openapi"
|
||||
luxlog "github.com/luxfi/log"
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// untypedByDesign is the CLOSED list of risk operations that are NOT typed ops,
|
||||
// each with the WIRE fact that keeps it raw. A typed op is a route PLUS a
|
||||
// registry entry — the one value the OpenAPI operation, the MCP tool, the CLI
|
||||
// command and the SDK method all come from — so an operation missing from that
|
||||
// registry is invisible to all four. This one is missing on purpose. The address
|
||||
// is written the way the DOCUMENT writes it, which is the identity every
|
||||
// projection keys on.
|
||||
var untypedByDesign = map[string]string{
|
||||
// The real probe. It answers 503 CARRYING THE DEGRADED REPORT as its body
|
||||
// (status/model/tenants/warehouse/billing), which is the whole point of it: a
|
||||
// probe that says only "unhealthy" tells an operator nothing about which of
|
||||
// the four planes is down. A typed op reaches a non-2xx only by returning an
|
||||
// error, and zip renders that as its own envelope, dropping the report.
|
||||
"GET /v1/risk/health": healthWire,
|
||||
}
|
||||
|
||||
const healthWire = "a REAL probe: 503 carries the degraded REPORT as its body " +
|
||||
"(status/model/tenants/warehouse/billing), which is the whole point of it. A typed op reaches a " +
|
||||
"non-2xx only by returning an error, and zip renders that as its own envelope, dropping the report."
|
||||
|
||||
// The two ops that COULD have been forced untyped, and were not — recorded here
|
||||
// so a reviewer can check the reasoning and not only the list.
|
||||
//
|
||||
// POST /v1/risk/decide is TYPED AND IS NEVER BALANCE-GATED. cloud.DenyResource
|
||||
// renders a pre-work refusal as the fleet's NESTED {"error":{"code","message"}}
|
||||
// 402 with c.JSON, and a typed op's returned error renders as the FLAT
|
||||
// {"status","code","error"} envelope — so a gated decide would either change the
|
||||
// 402 body every balance-aware client parses, or have to be untyped. It meters
|
||||
// AFTER instead: the screen is billed on the decision that was actually
|
||||
// produced, the hot path stays typed, and no existing client's parse moves.
|
||||
//
|
||||
// POST /v1/risk/train and POST /v1/risk/search ARE gated before the work, because
|
||||
// both are real CPU on a shared pod. They stay typed and return a typed 402:
|
||||
// they are NEW routes, so no client parses a nested body from them, and a new
|
||||
// route may as well carry the contract we want rather than the one we inherited.
|
||||
|
||||
// mountApp mounts risk the way plugin/risk does — the whole Mount, so the
|
||||
// projection ledgers below read the surface a deployed binary serves and not a
|
||||
// test-only subset.
|
||||
func mountApp(t *testing.T) *zip.App {
|
||||
t.Helper()
|
||||
app := zip.New(zip.Config{Logger: luxlog.New("risktest"), DisableStartupMessage: true})
|
||||
if err := Mount(app, cloud.Deps{
|
||||
Logger: luxlog.New("risktest"), DataDir: t.TempDir(), Brand: "hanzo",
|
||||
}); err != nil {
|
||||
t.Fatalf("Mount: %v", err)
|
||||
}
|
||||
return app
|
||||
}
|
||||
|
||||
// riskOps reads BOTH projections of the live router at their one shared address
|
||||
// form: what the document says is served, and which of those carry a typed
|
||||
// registry entry. EVERY served operation counts, so a route mounted at an
|
||||
// address nobody expected is caught rather than filtered out.
|
||||
func riskOps(t *testing.T) (served map[string]bool, typed map[string]string, schemas map[string]any) {
|
||||
t.Helper()
|
||||
app := mountApp(t)
|
||||
doc, err := openapi.Spec(app, openapi.Info{Title: "risk", Version: "v1"})
|
||||
if err != nil {
|
||||
t.Fatalf("spec: %v", err)
|
||||
}
|
||||
reg, err := openapi.Typed(app)
|
||||
if err != nil {
|
||||
t.Fatalf("typed registry: %v", err)
|
||||
}
|
||||
served, typed = map[string]bool{}, map[string]string{}
|
||||
for path, item := range doc.Paths {
|
||||
for method := range item {
|
||||
served[strings.ToUpper(method)+" "+path] = true
|
||||
}
|
||||
}
|
||||
for key, op := range reg.Ops {
|
||||
typed[key] = op.Description
|
||||
}
|
||||
return served, typed, reg.Schemas
|
||||
}
|
||||
|
||||
// TestEveryRouteIsTypedOrNamed fails when a risk operation is neither a typed op
|
||||
// nor named above — so the next route added here is typed by default, and
|
||||
// dropping one out of the registry takes a deliberate edit carrying a reason.
|
||||
func TestEveryRouteIsTypedOrNamed(t *testing.T) {
|
||||
served, typed, _ := riskOps(t)
|
||||
|
||||
var untyped []string
|
||||
for key := range served {
|
||||
if _, ok := typed[key]; ok {
|
||||
continue
|
||||
}
|
||||
if _, named := untypedByDesign[key]; named {
|
||||
continue
|
||||
}
|
||||
untyped = append(untyped, key)
|
||||
}
|
||||
if len(untyped) > 0 {
|
||||
sort.Strings(untyped)
|
||||
t.Errorf("operation(s) with no registry entry and no reason: %s\n"+
|
||||
"A route that is not a typed op has no schema, no prose, no MCP tool, no CLI command and no "+
|
||||
"SDK method. Convert it (zip.Get/Post/... on the /v1/risk group), or add it to "+
|
||||
"untypedByDesign with the reason typing it would move the wire.", strings.Join(untyped, ", "))
|
||||
}
|
||||
for key := range untypedByDesign {
|
||||
if !served[key] {
|
||||
t.Errorf("untypedByDesign names %q, which risk no longer serves", key)
|
||||
}
|
||||
if _, ok := typed[key]; ok {
|
||||
t.Errorf("untypedByDesign names %q, which IS a typed op — delete the entry", key)
|
||||
}
|
||||
}
|
||||
if got := len(typed) + len(untypedByDesign); got != len(served) {
|
||||
t.Errorf("%d typed + %d named = %d, but risk serves %d operations",
|
||||
len(typed), len(untypedByDesign), got, len(served))
|
||||
}
|
||||
}
|
||||
|
||||
// TestTheSurfaceIsWhatWasPromised pins the operations this app exists to serve.
|
||||
// A rename or a silent drop is a broken SDK for every caller, and a diff of a
|
||||
// list is the only thing that catches it before they do.
|
||||
func TestTheSurfaceIsWhatWasPromised(t *testing.T) {
|
||||
served, _, _ := riskOps(t)
|
||||
want := []string{
|
||||
// decide
|
||||
"POST /v1/risk/decide",
|
||||
// record
|
||||
"GET /v1/risk/decisions",
|
||||
"GET /v1/risk/decisions/{id}",
|
||||
"POST /v1/risk/decisions/{id}/label",
|
||||
"GET /v1/risk/subjects/{kind}/{id}",
|
||||
"GET /v1/risk/activity",
|
||||
// govern
|
||||
"POST /v1/risk/simulate",
|
||||
"GET /v1/risk/rules", "POST /v1/risk/rules",
|
||||
"PATCH /v1/risk/rules/{id}", "DELETE /v1/risk/rules/{id}",
|
||||
"GET /v1/risk/lists", "POST /v1/risk/lists",
|
||||
"POST /v1/risk/lists/{name}/entries",
|
||||
"DELETE /v1/risk/lists/{name}/entries/{value}",
|
||||
"GET /v1/risk/suppressions", "POST /v1/risk/suppressions",
|
||||
"DELETE /v1/risk/suppressions/{id}",
|
||||
"GET /v1/risk/controls", "POST /v1/risk/controls",
|
||||
"DELETE /v1/risk/controls/{id}",
|
||||
"GET /v1/risk/dictionary",
|
||||
"GET /v1/risk/mode", "PUT /v1/risk/mode",
|
||||
"GET /v1/risk/health",
|
||||
// learn
|
||||
"POST /v1/risk/score",
|
||||
"POST /v1/risk/train",
|
||||
"GET /v1/risk/state", "PUT /v1/risk/state/appetite",
|
||||
"GET /v1/risk/features",
|
||||
"POST /v1/risk/search", "GET /v1/risk/search/{id}", "DELETE /v1/risk/search/{id}",
|
||||
"POST /v1/risk/snapshot", "POST /v1/risk/restore",
|
||||
}
|
||||
for _, w := range want {
|
||||
if !served[w] {
|
||||
t.Errorf("%s is not served — it is in the contract and not on the router", w)
|
||||
}
|
||||
}
|
||||
if len(served) != len(want) {
|
||||
var extra []string
|
||||
for k := range served {
|
||||
if !containsString(want, k) {
|
||||
extra = append(extra, k)
|
||||
}
|
||||
}
|
||||
sort.Strings(extra)
|
||||
t.Errorf("risk serves %d operations, the contract names %d; unlisted: %s",
|
||||
len(served), len(want), strings.Join(extra, ", "))
|
||||
}
|
||||
}
|
||||
|
||||
func containsString(set []string, v string) bool {
|
||||
for _, s := range set {
|
||||
if s == v {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// TestEveryTypedOpIsDescribed proves the lifted prose reached the binary. That
|
||||
// prose IS the product surface: it becomes the OpenAPI description AND the MCP
|
||||
// tool description a model reads to pick the tool. zipdoc_gen.go is what carries
|
||||
// it in, so an op added without regenerating shows up here as a nameless tool.
|
||||
func TestEveryTypedOpIsDescribed(t *testing.T) {
|
||||
_, typed, _ := riskOps(t)
|
||||
if len(typed) == 0 {
|
||||
t.Fatal("no typed risk ops in the registry at all")
|
||||
}
|
||||
for key, desc := range typed {
|
||||
if strings.TrimSpace(desc) == "" {
|
||||
t.Errorf("%s has no description — run: go generate -run zipdoc ./apps/risk/...", key)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestEveryPublishedFieldIsDescribed closes the half of the surface the op-level
|
||||
// gate cannot see. Typing a route documents its ADDRESS and its SHAPE; it does
|
||||
// not document the shape's FIELDS, and those come from doc comments on the In/Out
|
||||
// struct fields, which zipdoc lifts per field.
|
||||
func TestEveryPublishedFieldIsDescribed(t *testing.T) {
|
||||
_, _, schemas := riskOps(t)
|
||||
if len(schemas) == 0 {
|
||||
t.Fatal("no risk schemas in the typed registry at all")
|
||||
}
|
||||
var bare []string
|
||||
for name, raw := range schemas {
|
||||
sch, ok := raw.(map[string]any)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
props, ok := sch["properties"].(map[string]any)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
for field, praw := range props {
|
||||
p, ok := praw.(map[string]any)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if desc, _ := p["description"].(string); strings.TrimSpace(desc) == "" {
|
||||
bare = append(bare, name+"."+field)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(bare) > 0 {
|
||||
sort.Strings(bare)
|
||||
t.Errorf("published propert(ies) with no description: %s\n"+
|
||||
"Every field of a published schema is read by SDK users and by a model choosing a tool. "+
|
||||
"Write a doc comment on the struct field and run: go generate -run zipdoc ./apps/risk/...",
|
||||
strings.Join(bare, ", "))
|
||||
}
|
||||
}
|
||||
|
||||
// TestEverySchemaNameCarriesItsFace guards the ONE failure mode the fleet weave
|
||||
// cannot recover from. openapi.Weave refuses one schema name with two shapes
|
||||
// across apps because a generated SDK binds whichever it read last — and this
|
||||
// app introduces forty types into a namespace already flat across the whole
|
||||
// fleet. `ref`, `list`, `page` and `state` are exactly the names the next app
|
||||
// reaches for.
|
||||
func TestEverySchemaNameCarriesItsFace(t *testing.T) {
|
||||
_, _, schemas := riskOps(t)
|
||||
for name := range schemas {
|
||||
if strings.HasPrefix(name, "risk") || strings.HasPrefix(name, "ml") {
|
||||
continue
|
||||
}
|
||||
t.Errorf("schema %q carries no face — prefix it risk* or ml*, or the fleet weave will "+
|
||||
"eventually refuse it against another app's type of the same name", name)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,689 @@
|
||||
// Code generated by zipdoc; DO NOT EDIT.
|
||||
|
||||
package risk
|
||||
|
||||
import (
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
func init() {
|
||||
zip.Describe("DELETE /v1/risk/controls/:id", zip.Doc{
|
||||
Description: "Lifts a control. Answers 204, or 404 for an identifier this\ntenant does not own.",
|
||||
Fields: map[string]string{
|
||||
"riskRef.id": "ID is the record to act on, taken from the path.",
|
||||
},
|
||||
})
|
||||
zip.Describe("DELETE /v1/risk/lists/:name/entries/:value", zip.Doc{
|
||||
Description: "Removes one value from a list. Answers 204, or 404 when the\nlist does not hold it.",
|
||||
Fields: map[string]string{
|
||||
"riskListEntryRef.name": "Name is the list, from the path.",
|
||||
"riskListEntryRef.value": "Value is the value to remove, from the path.",
|
||||
},
|
||||
})
|
||||
zip.Describe("DELETE /v1/risk/rules/:id", zip.Doc{
|
||||
Description: "Retires a detection. Answers 204, or 404 for an identifier this\ntenant does not own.",
|
||||
Fields: map[string]string{
|
||||
"riskRef.id": "ID is the record to act on, taken from the path.",
|
||||
},
|
||||
})
|
||||
zip.Describe("DELETE /v1/risk/search/:id", zip.Doc{
|
||||
Description: "Stops a search this tenant started.\n\nAn expensive op that cannot be stopped is an expensive op a caller cannot take\nback — and the run holds a worker every other tenant is queued behind. It is\nidempotent: cancelling a search that has already answered is not an error, and\nthe reply says which happened.",
|
||||
Fields: map[string]string{
|
||||
"riskRef.id": "ID is the record to act on, taken from the path.",
|
||||
"riskSearchRun.candidates": "Candidates is how many topologies will be tried.",
|
||||
"riskSearchRun.id": "ID identifies the run; read it back at GET /v1/risk/search/{id}.",
|
||||
"riskSearchRun.status": "Status is \"running\" or \"done\".",
|
||||
},
|
||||
})
|
||||
zip.Describe("DELETE /v1/risk/suppressions/:id", zip.Doc{
|
||||
Description: "Unsuppress lifts a mute. Answers 204, or 404 for an identifier this tenant\ndoes not own.",
|
||||
Fields: map[string]string{
|
||||
"riskRef.id": "ID is the record to act on, taken from the path.",
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/risk/activity", zip.Doc{
|
||||
Description: "Activity is the watcher: what is firing right now, what is being muted, and\nwhich way the decisions are going.\n\nIt reads the DECISION LOG rather than a second bus. The log is already the\nrecord and the analytics copies already reach the platform stream; a dedicated\nactivation bus would be a third representation of one fact and a third thing\nthat can be behind.",
|
||||
Fields: map[string]string{
|
||||
"riskActivityIn.limit": "Limit bounds how many recent decisions are summarised, 1..500.",
|
||||
"riskActivityRule.activations": "Activations is how many of the sampled decisions it fired on.",
|
||||
"riskActivityRule.name": "Name is its detection name.",
|
||||
"riskActivityRule.rule": "Rule is the rule that fired.",
|
||||
"riskActivityRule.suppressed": "Suppressed is how many of those were muted.",
|
||||
"riskActivityView.actions": "Actions is how many decisions reached each action.",
|
||||
"riskActivityView.agency": "Agency is how many decisions were of each actor class.",
|
||||
"riskActivityView.refusals": "Refusals is how many decisions the model declined to score, by reason.",
|
||||
"riskActivityView.rules": "Rules is per-rule activation, most active first.",
|
||||
"riskActivityView.sampled": "Sampled is how many recent decisions this summarises.",
|
||||
"riskActivityView.shadow": "Shadow is whether this tenant is observing rather than acting.",
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/risk/controls", zip.Doc{
|
||||
Description: "Controls lists the platform controls declared on this tenant's subjects:\nreserves, payout holds and blocks.\n\nThey are DECLARATIONS the money plane reads. Risk never moves money —\nhanzoai/commerce owns payouts, disputes and balances — so a marketplace reads\nthis before a payout and calls decide at authorization, and risk touches no\nprocessor. That is what \"processor-agnostic\" is: a structural property, not a\nfeature.",
|
||||
Fields: map[string]string{
|
||||
"riskControl.at": "At is when, RFC 3339 in UTC.",
|
||||
"riskControl.by": "By is who declared it, taken from the validated principal.",
|
||||
"riskControl.control": "Control is which one.",
|
||||
"riskControl.id": "ID identifies the control.",
|
||||
"riskControl.rate": "Rate is the reserve fraction, for a reserve.",
|
||||
"riskControl.reason": "Reason is why it was declared.",
|
||||
"riskControl.subject": "Subject is what it applies to.",
|
||||
"riskControl.until": "Until is when it lapses, RFC 3339. Absent means it does not.",
|
||||
"riskControlPage.items": "Items is every control, newest first.",
|
||||
"riskControlsIn.kind": "Kind narrows to one subject kind.",
|
||||
"riskControlsIn.subject": "Subject narrows to one subject.",
|
||||
"riskSubject.id": "ID is the subject's identifier within the caller's own org. It is never\ninterpreted, only aggregated on, so it may be any stable string.",
|
||||
"riskSubject.kind": "Kind is the sort of thing this is: account, transaction, session, agent,\nmerchant or payout. It selects the aggregation axes, not the tenant gate.",
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/risk/decisions", zip.Doc{
|
||||
Description: "Decisions lists this tenant's recent decisions, newest first. Every filter is\nan equality on a column the tenant's own file owns, so a filter can narrow the\ncaller's own log and can never widen it.",
|
||||
Fields: map[string]string{
|
||||
"riskDecisionBrief.action": "Action is what was decided.",
|
||||
"riskDecisionBrief.agency": "Agency is the actor class.",
|
||||
"riskDecisionBrief.at": "At is when it was made, RFC 3339 in UTC.",
|
||||
"riskDecisionBrief.id": "ID identifies the decision.",
|
||||
"riskDecisionBrief.kind": "Kind is the subject kind.",
|
||||
"riskDecisionBrief.label": "Label is what a human later concluded, when anyone has.",
|
||||
"riskDecisionBrief.refusal": "Refusal names what the decision was short of, when it was short of\nanything.",
|
||||
"riskDecisionBrief.score": "Score is the weight-of-evidence score.",
|
||||
"riskDecisionBrief.shadow": "Shadow is whether the decision acted.",
|
||||
"riskDecisionBrief.since": "Since is when the aggregates this decision read had started, RFC 3339 in\nUTC — the period its velocity numbers actually cover.",
|
||||
"riskDecisionBrief.stage": "Stage is the lifecycle moment.",
|
||||
"riskDecisionBrief.strained": "Strained is true when the aggregates were at this tenant's own cardinality\nbound when the decision was made.",
|
||||
"riskDecisionBrief.subject": "Subject is the subject identifier.",
|
||||
"riskDecisionPage.items": "Items is the page, newest first.",
|
||||
"riskDecisionPage.oldest": "Oldest is the instant of the oldest decision still retained, RFC 3339 in\nUTC — the far edge of the window every read of this log looks through. It\nis published rather than left to be inferred: a query that finds nothing\nbefore a date and a period that was never retained are the same empty\nanswer and opposite facts.",
|
||||
"riskDecisionPage.retained": "Retained is how many decisions this tenant's log holds. The log is a ring:\npast its retention the OLDEST decisions are dropped so a new one can\nalways be recorded, because refusing a decision would be refusing the\nauthorization the caller asked for.",
|
||||
"riskDecisionsIn.action": "Action narrows to one outcome.",
|
||||
"riskDecisionsIn.kind": "Kind narrows to one subject kind.",
|
||||
"riskDecisionsIn.limit": "Limit bounds the page, 1..500, default 100.",
|
||||
"riskDecisionsIn.stage": "Stage narrows to one lifecycle moment.",
|
||||
"riskDecisionsIn.subject": "Subject narrows to one subject.",
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/risk/decisions/:id", zip.Doc{
|
||||
Description: "Reads one decision with everything behind it: the evidence, the\nmodel's per-feature attribution and the digest of the model that produced it.\nThat set IS the dispute packet — what was decided, on what, by which model.\n\nA decision another tenant owns answers 404, exactly as an unknown identifier\ndoes, because the tenant's file is the only place looked in: there is no query\nthat could reach another tenant's row, so a probe learns nothing.",
|
||||
Fields: map[string]string{
|
||||
"riskCause.baseline": "Baseline is what is unremarkable for this subject.",
|
||||
"riskCause.citation": "Citation is where those words come from.",
|
||||
"riskCause.feature": "Feature is the model dimension.",
|
||||
"riskCause.indicator": "Indicator is the supervisor's own words for what is being looked for.",
|
||||
"riskCause.observed": "Observed is this subject's value.",
|
||||
"riskCause.share": "Share is this feature's part of the score, in [0,1].",
|
||||
"riskCause.typology": "Typology is the pattern the feature expresses.",
|
||||
"riskCause.unit": "Unit is how to read Observed and Baseline.",
|
||||
"riskCause.without": "Without is the score the model would have produced with this feature\nneutral, holding everything else.",
|
||||
"riskDecisionBrief.action": "Action is what was decided.",
|
||||
"riskDecisionBrief.agency": "Agency is the actor class.",
|
||||
"riskDecisionBrief.at": "At is when it was made, RFC 3339 in UTC.",
|
||||
"riskDecisionBrief.id": "ID identifies the decision.",
|
||||
"riskDecisionBrief.kind": "Kind is the subject kind.",
|
||||
"riskDecisionBrief.label": "Label is what a human later concluded, when anyone has.",
|
||||
"riskDecisionBrief.refusal": "Refusal names what the decision was short of, when it was short of\nanything.",
|
||||
"riskDecisionBrief.score": "Score is the weight-of-evidence score.",
|
||||
"riskDecisionBrief.shadow": "Shadow is whether the decision acted.",
|
||||
"riskDecisionBrief.since": "Since is when the aggregates this decision read had started, RFC 3339 in\nUTC — the period its velocity numbers actually cover.",
|
||||
"riskDecisionBrief.stage": "Stage is the lifecycle moment.",
|
||||
"riskDecisionBrief.strained": "Strained is true when the aggregates were at this tenant's own cardinality\nbound when the decision was made.",
|
||||
"riskDecisionBrief.subject": "Subject is the subject identifier.",
|
||||
"riskDecisionView.causes": "Causes is the model's attribution.",
|
||||
"riskDecisionView.decision": "Decision is the row.",
|
||||
"riskDecisionView.hits": "Hits is the evidence.",
|
||||
"riskDecisionView.model": "Model is the digest of the model that produced it.",
|
||||
"riskHit.action": "Action is what this evidence alone asked for.",
|
||||
"riskHit.name": "Name is the detection's human-readable name.",
|
||||
"riskHit.rule": "Rule is the identifier of the rule or model that produced this evidence.",
|
||||
"riskHit.severity": "Severity is the reviewer-facing grading.",
|
||||
"riskHit.suppressed": "Suppressed marks evidence a suppression muted. It is reported and\ncontributes nothing — a muted control that left no trace would be\nindistinguishable from one that was never running.",
|
||||
"riskHit.weight": "Weight is how much it contributed, in [0,1].",
|
||||
"riskRef.id": "ID is the record to act on, taken from the path.",
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/risk/dictionary", zip.Doc{
|
||||
Description: "Dictionary is the vocabulary a rule may be written in: every fact, every\noperator, every velocity axis and window, this tenant's own lists, and the\nsignal keys this tenant has actually sent.\n\nTwo lenses on one catalogue. The product half is closed and is what admission\nchecks against. The tenant half — the signals — is the tenant's own, read back\noff its recent decisions, which is what makes a rule builder able to offer the\nfields that exist here rather than the fields that exist in general.",
|
||||
Fields: map[string]string{
|
||||
"riskDictionaryView.actions": "Actions are the outcomes, weakest first.",
|
||||
"riskDictionaryView.axes": "Axes are the velocity axes.",
|
||||
"riskDictionaryView.fields": "Fields is every scalar fact.",
|
||||
"riskDictionaryView.lists": "Lists are this tenant's own lists, which an inlist term may name.",
|
||||
"riskDictionaryView.signals": "Signals are the signal keys this tenant has actually sent, which is the\nhalf of the catalogue that is the tenant's own rather than the product's.",
|
||||
"riskDictionaryView.stages": "Stages are the lifecycle moments.",
|
||||
"riskDictionaryView.windows": "Windows are the velocity spans.",
|
||||
"riskField.field": "Field is how a rule names it.",
|
||||
"riskField.kind": "Kind is what sort of value it holds: \"string\" or \"number\".",
|
||||
"riskField.note": "Note says what it means.",
|
||||
"riskField.ops": "Ops are the comparisons that make sense on it.",
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/risk/features", zip.Doc{
|
||||
Description: "Features is the typology-to-feature inventory the model is built on: what each\ndimension measures, the risk indicator it serves, and the published source\nthose words come from.\n\nIt is code rather than a document because a mapping kept beside the model\ncannot drift away from what the model actually reads — and the attributability\nit makes possible is why this is a tree and not a net.",
|
||||
Fields: map[string]string{
|
||||
"riskFeature.citation": "Citation is where those words come from, so the claim is checkable.",
|
||||
"riskFeature.indicator": "Indicator is the supervisor's own words for what is being looked for.",
|
||||
"riskFeature.name": "Name is the dimension.",
|
||||
"riskFeature.neutral": "Neutral is the value at which this feature is unremarkable — the\ncoordinate the counterfactual moves it to.",
|
||||
"riskFeature.severity": "Severity is the grading a hit on this feature carries.",
|
||||
"riskFeature.typology": "Typology is the pattern it expresses.",
|
||||
"riskFeature.unit": "Unit is how to read the raw number.",
|
||||
"riskFeature.window": "Window is the span it is measured over, when it has one.",
|
||||
"riskFeatureInventory.digest": "Digest is the identity of this shape.",
|
||||
"riskFeatureInventory.items": "Items is every dimension, in coordinate order. The ORDER is part of the\nmodel's identity: adding, removing or reordering one invalidates learned\nstate, which the digest enforces.",
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/risk/health", zip.Doc{
|
||||
Description: "Is the app's real, fail-closed probe.\n\nUNTYPED BY DESIGN. A real probe answers 503 CARRYING THE DEGRADED REPORT AS\nITS BODY, and a typed op reaches a non-2xx only by returning an error, which\nzip renders as its own envelope — dropping exactly the report the probe\nexists to deliver. Same reason apps/ml keeps its two health routes raw.",
|
||||
})
|
||||
zip.Describe("GET /v1/risk/lists", zip.Doc{
|
||||
Description: "Lists shows this tenant's allow and deny lists and how many values each holds.\n\nThese are the TENANT's operational lists. Sanctions and PEP designations are\nnot here and are never merged into them: a tenant may add to its own deny\nlist, and a tenant may not edit OFAC.",
|
||||
Fields: map[string]string{
|
||||
"riskListPage.items": "Items is every list.",
|
||||
"riskListView.createdAt": "CreatedAt is when it was created, RFC 3339 in UTC.",
|
||||
"riskListView.entries": "Entries is how many values it holds.",
|
||||
"riskListView.kind": "Kind is whether membership allows or denies: \"allow\" or \"deny\".",
|
||||
"riskListView.name": "Name is the list's identifier, which a rule names in an inlist term.",
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/risk/mode", zip.Doc{
|
||||
Description: "Mode reports whether this tenant's decisions act or only observe.",
|
||||
Fields: map[string]string{
|
||||
"riskModeView.mode": "Mode is \"shadow\" or \"live\".",
|
||||
"riskModeView.since": "Since is when it was last changed, RFC 3339 in UTC. Empty means never.",
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/risk/rules", zip.Doc{
|
||||
Description: "Rules lists this tenant's detections. A tenant starts with a starter set\ncovering the lifecycle the product names — signup burst, shared device,\ndisposable email, card testing, denied address, spend spike, payout velocity,\nundeclared automation — all of which it may read, copy, edit and retire.",
|
||||
Fields: map[string]string{
|
||||
"riskRule.action": "Action is what the rule asks for when it holds.",
|
||||
"riskRule.all": "All is the conjunction. An empty conjunction is refused: a rule that holds\non everything is not a detection.",
|
||||
"riskRule.enabled": "Enabled governs the live path. A disabled rule still replays, because the\nquestion a simulation asks is what happens ON activation.",
|
||||
"riskRule.id": "ID is the rule's identifier within this tenant.",
|
||||
"riskRule.name": "Name is what a reviewer reads in an alert.",
|
||||
"riskRule.severity": "Severity is the reviewer-facing grading: low, medium, high or critical.",
|
||||
"riskRule.stage": "Stage narrows the rule to one lifecycle moment; empty means every stage.",
|
||||
"riskRule.weight": "Weight is how much a hit contributes, in [0,1].",
|
||||
"riskRuleList.items": "Items is every rule, by identifier.",
|
||||
"riskTerm.field": "Field is the fact to read: one of the closed vocabulary, or signal.<name>,\nor velocity.<axis>.<window>.<stat>. GET /v1/risk/dictionary lists them.",
|
||||
"riskTerm.number": "Number is the numeric operand.",
|
||||
"riskTerm.op": "Op is the comparison: eq, ne, gt, gte, lt, lte, in, notin, inlist,\nnotinlist, exists, absent, contains, prefix or suffix.",
|
||||
"riskTerm.value": "Value is the string operand.",
|
||||
"riskTerm.values": "Values is the set operand, for in and notin.",
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/risk/search/:id", zip.Doc{
|
||||
Description: "Reads an exhaustive search's report: every candidate tried, the\nwinner, and the winner's learning curve.",
|
||||
Fields: map[string]string{
|
||||
"riskCandidate.blend": "Blend is how much of a closing window folds into the reference.",
|
||||
"riskCandidate.depth": "Depth is how deep each tree splits.",
|
||||
"riskCandidate.review": "Review is the share of the stream this topology may examine.",
|
||||
"riskCandidate.trees": "Trees is how many half-space trees the model holds.",
|
||||
"riskCandidate.window": "Window is how many observations make up one reference window.",
|
||||
"riskRef.id": "ID is the record to act on, taken from the path.",
|
||||
"riskSearchReport.curve": "Curve is the winner's separation as a function of how much history it had\nseen, in ten steps — the learning curve.",
|
||||
"riskSearchReport.events": "Events is how many historical observations were replayed.",
|
||||
"riskSearchReport.id": "ID identifies the run.",
|
||||
"riskSearchReport.refusal": "Refusal names why the report is empty when it is. An empty history is\nrefused rather than reported as zero alerts, because a quiet model and an\nunrun model produce the same number.",
|
||||
"riskSearchReport.status": "Status is \"running\", \"done\" or \"refused\".",
|
||||
"riskSearchReport.trials": "Trials is every candidate tried, best-separating first.",
|
||||
"riskSearchReport.winner": "Winner is the best-separating topology that also honoured its stated\nappetite. Absent when no candidate did both.",
|
||||
"riskTrial.alerted": "Alerted is how many of those it would have alerted on.",
|
||||
"riskTrial.candidate": "Candidate is the topology tried.",
|
||||
"riskTrial.realised": "Realised is Alerted over Scored, against the Review share intended.",
|
||||
"riskTrial.scored": "Scored is how many observations it was able to score.",
|
||||
"riskTrial.separation": "Separation is the mean score of the alerted set minus the mean of the\nrest. It is the ranking objective: a topology that separates the tail from\nthe body is doing the job whatever its absolute scores look like.",
|
||||
"riskTrial.warm": "Warm is how many observations passed before it would score at all.",
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/risk/state", zip.Doc{
|
||||
Description: "ModelState is what a review of the model reads: the appetite it was given, the\nthreshold that appetite produced, and the share it ACTUALLY reached.\n\nStated against realised is the whole governance report. An appetite is a\nmeasured commitment or it is nothing, and a fixed threshold on a drifting\ndistribution silently becomes either silence or a flood.",
|
||||
Fields: map[string]string{
|
||||
"riskAppetite.review": "Review is the share of the stream that may be sent for examination. The\nalert threshold is derived from it as a quantile of the scores actually\nobserved, rather than fixed at a number someone liked — so an alert level\nis governed rather than tuned, and a drifting distribution does not turn\nit into silence or a flood.",
|
||||
"riskAppetite.sample": "Sample is the share of NON-alerting traffic retained for review below the\nline. It is the only instrument that can measure what the model missed,\nbecause nothing in the stream is labelled.",
|
||||
"riskAppetite.warm": "Warm is how many observations the model must learn before it may score.",
|
||||
"riskFeature.citation": "Citation is where those words come from, so the claim is checkable.",
|
||||
"riskFeature.indicator": "Indicator is the supervisor's own words for what is being looked for.",
|
||||
"riskFeature.name": "Name is the dimension.",
|
||||
"riskFeature.neutral": "Neutral is the value at which this feature is unremarkable — the\ncoordinate the counterfactual moves it to.",
|
||||
"riskFeature.severity": "Severity is the grading a hit on this feature carries.",
|
||||
"riskFeature.typology": "Typology is the pattern it expresses.",
|
||||
"riskFeature.unit": "Unit is how to read the raw number.",
|
||||
"riskFeature.window": "Window is the span it is measured over, when it has one.",
|
||||
"riskModelState.alerted": "Alerted is how many of those it alerted on — the numerator of Realised.",
|
||||
"riskModelState.appetite": "Appetite is the stated share of the stream that may be examined.",
|
||||
"riskModelState.blind": "Blind counts, per feature, how often it took its neutral value for want of\ndata.",
|
||||
"riskModelState.digest": "Digest is the model identity an auditor pins.",
|
||||
"riskModelState.distribution": "Distribution is the score distribution the threshold is cut from, in 32\nbands.",
|
||||
"riskModelState.inventory": "Inventory is the feature set, so the state and the shape are read\ntogether.",
|
||||
"riskModelState.learned": "Learned is how many observations this tenant's model has taken in.",
|
||||
"riskModelState.realised": "Realised is the share that ACTUALLY alerted. Stated against realised is\nthe governance report — an appetite is a measured commitment or it is\nnothing.",
|
||||
"riskModelState.refusals": "Refusals counts what the model declined to score, by reason. None of these\nwas examined by the model; all of them were examined by the rules.",
|
||||
"riskModelState.saturated": "Saturated means the appetite cannot be honoured by any threshold because\ntoo much of the stream scores in the top band. It is the one state that\nmust never be mistaken for quiet.",
|
||||
"riskModelState.scored": "Scored is how many observations the model was able to score — the\ndenominator of Realised.",
|
||||
"riskModelState.threshold": "Threshold is the score cut currently in force, recomputed each window as\nthe quantile that admits the stated share.",
|
||||
"riskModelState.warming": "Warming is true while the model has not learned enough to score. A warming\nmodel REFUSES rather than scoring low, and the refusal is counted below.",
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/risk/subjects/:kind/:id", zip.Doc{
|
||||
Description: "Reads one subject's current risk state: every live velocity\naggregate it has, its recent decisions, whatever controls are declared on it,\nits thirty-day history, and where this KIND of subject sits across the\nplatform. This is the continuous-monitoring read — a merchant, an account or\nan agent, on one page.\n\nThe last two come from the warehouse and are BEST EFFORT: an unreachable\nwarehouse omits them and names the gap, because a zeroed history would say the\nsubject did nothing and a zeroed baseline would say the platform did.\n\nThe network comparison is the only cross-org value this API returns and it is\naggregate BY CONSTRUCTION: the table has no tenant column, so there is no\nquery — here or anywhere — that could return another tenant's rows.",
|
||||
Fields: map[string]string{
|
||||
"riskBaseline.day": "Day is the period the band covers, YYYY-MM-DD.",
|
||||
"riskBaseline.feature": "Feature is what is being compared.",
|
||||
"riskBaseline.observations": "Observations is how many measurements the band was cut from.",
|
||||
"riskBaseline.orgs": "Orgs is how many distinct organisations contributed to this band. It is\nreported so a reader can see the k-anonymity floor was met rather than\ntake it on trust.",
|
||||
"riskBaseline.q10": "Q10 is the tenth percentile across the platform for this subject kind.",
|
||||
"riskBaseline.q50": "Q50 is the median.",
|
||||
"riskBaseline.q90": "Q90 is the ninetieth percentile.",
|
||||
"riskBaseline.q99": "Q99 is the ninety-ninth — where the tail this product is looking for\nbegins.",
|
||||
"riskControl.at": "At is when, RFC 3339 in UTC.",
|
||||
"riskControl.by": "By is who declared it, taken from the validated principal.",
|
||||
"riskControl.control": "Control is which one.",
|
||||
"riskControl.id": "ID identifies the control.",
|
||||
"riskControl.rate": "Rate is the reserve fraction, for a reserve.",
|
||||
"riskControl.reason": "Reason is why it was declared.",
|
||||
"riskControl.subject": "Subject is what it applies to.",
|
||||
"riskControl.until": "Until is when it lapses, RFC 3339. Absent means it does not.",
|
||||
"riskDecisionBrief.action": "Action is what was decided.",
|
||||
"riskDecisionBrief.agency": "Agency is the actor class.",
|
||||
"riskDecisionBrief.at": "At is when it was made, RFC 3339 in UTC.",
|
||||
"riskDecisionBrief.id": "ID identifies the decision.",
|
||||
"riskDecisionBrief.kind": "Kind is the subject kind.",
|
||||
"riskDecisionBrief.label": "Label is what a human later concluded, when anyone has.",
|
||||
"riskDecisionBrief.refusal": "Refusal names what the decision was short of, when it was short of\nanything.",
|
||||
"riskDecisionBrief.score": "Score is the weight-of-evidence score.",
|
||||
"riskDecisionBrief.shadow": "Shadow is whether the decision acted.",
|
||||
"riskDecisionBrief.since": "Since is when the aggregates this decision read had started, RFC 3339 in\nUTC — the period its velocity numbers actually cover.",
|
||||
"riskDecisionBrief.stage": "Stage is the lifecycle moment.",
|
||||
"riskDecisionBrief.strained": "Strained is true when the aggregates were at this tenant's own cardinality\nbound when the decision was made.",
|
||||
"riskDecisionBrief.subject": "Subject is the subject identifier.",
|
||||
"riskHistory.bucket": "Bucket is the five-minute period, RFC 3339 in UTC.",
|
||||
"riskHistory.values": "Values is each counted feature over that period, keyed by the names\nGET /v1/risk/dictionary publishes.",
|
||||
"riskSubject.id": "ID is the subject's identifier within the caller's own org. It is never\ninterpreted, only aggregated on, so it may be any stable string.",
|
||||
"riskSubject.kind": "Kind is the sort of thing this is: account, transaction, session, agent,\nmerchant or payout. It selects the aggregation axes, not the tenant gate.",
|
||||
"riskSubjectRef.id": "ID is the subject identifier, from the path.",
|
||||
"riskSubjectRef.kind": "Kind is the subject kind, from the path.",
|
||||
"riskSubjectView.controls": "Controls is what is currently declared on the subject.",
|
||||
"riskSubjectView.decisions": "Decisions is the subject's recent decision history, newest first.",
|
||||
"riskSubjectView.gap": "Gap names what could not be read, when something could not be. It is\npresent precisely so that a missing History or Network is legible as a\ngap rather than as an answer.",
|
||||
"riskSubjectView.history": "History is the subject's activity over the last thirty days from the\nwarehouse. Absent when the warehouse is unreachable — an absent history is\nan honest gap, where a zeroed one would say the subject did nothing.",
|
||||
"riskSubjectView.network": "Network is where this subject kind sits across the platform, as quantiles\nonly. Absent when no band has met the k-anonymity floor.",
|
||||
"riskSubjectView.since": "Since is when those aggregates started, RFC 3339 in UTC. A window wider\nthan the interval since this instant is only partly covered, and saying so\nis the difference between a small count and a wrong one.",
|
||||
"riskSubjectView.strained": "Strained is true when this tenant's aggregates are at their own cardinality\nbound, so a count may be short of this tenant's own traffic.",
|
||||
"riskSubjectView.subject": "Subject is what this describes.",
|
||||
"riskSubjectView.velocity": "Velocity is every live aggregate the subject has.",
|
||||
"riskVelocity.axis": "Axis is what is being aggregated over: account, device, ip, pair, email or\nbin.",
|
||||
"riskVelocity.count": "Count is how many observations fell in the window.",
|
||||
"riskVelocity.days": "Days is how many distinct calendar days contributed.",
|
||||
"riskVelocity.near": "Near is how many fell just below the reporting threshold — the structuring\nsignal.",
|
||||
"riskVelocity.sum": "Sum is their total value, in units of currency.",
|
||||
"riskVelocity.window": "Window is the span: 1h, 24h, 7d or 30d.",
|
||||
},
|
||||
})
|
||||
zip.Describe("GET /v1/risk/suppressions", zip.Doc{
|
||||
Description: "Suppressions lists the active mutes. A suppression that has expired is not\nlisted and mutes nothing — a forgotten mute stops muting instead of quietly\nstaying on forever.",
|
||||
Fields: map[string]string{
|
||||
"riskSuppression.at": "At is when it was applied, RFC 3339 in UTC.",
|
||||
"riskSuppression.by": "By is who applied it, taken from the validated principal and never from\nthe request body.",
|
||||
"riskSuppression.id": "ID identifies the suppression.",
|
||||
"riskSuppression.kind": "Kind is the subject kind muted, when it names one.",
|
||||
"riskSuppression.reason": "Reason is why it was applied.",
|
||||
"riskSuppression.rule": "Rule is the rule muted, when it names one.",
|
||||
"riskSuppression.subject": "Subject is the subject muted, when it names one.",
|
||||
"riskSuppression.until": "Until is when it expires, RFC 3339. Absent means it does not.",
|
||||
"riskSuppressionPage.items": "Items is every suppression, newest first.",
|
||||
},
|
||||
})
|
||||
zip.Describe("PATCH /v1/risk/rules/:id", zip.Doc{
|
||||
Description: "Replaces a detection whole. There is no partial update: a rule is a\nconjunction of terms, and merging a partial term list into an existing one is\na change nobody can read off the request.",
|
||||
Fields: map[string]string{
|
||||
"riskRule.action": "Action is what the rule asks for when it holds.",
|
||||
"riskRule.all": "All is the conjunction. An empty conjunction is refused: a rule that holds\non everything is not a detection.",
|
||||
"riskRule.enabled": "Enabled governs the live path. A disabled rule still replays, because the\nquestion a simulation asks is what happens ON activation.",
|
||||
"riskRule.id": "ID is the rule's identifier within this tenant.",
|
||||
"riskRule.name": "Name is what a reviewer reads in an alert.",
|
||||
"riskRule.severity": "Severity is the reviewer-facing grading: low, medium, high or critical.",
|
||||
"riskRule.stage": "Stage narrows the rule to one lifecycle moment; empty means every stage.",
|
||||
"riskRule.weight": "Weight is how much a hit contributes, in [0,1].",
|
||||
"riskRuleIn.id": "ID is the rule to replace, from the path on a PATCH. Ignored on create,\nwhere the server mints one.",
|
||||
"riskRuleIn.rule": "Rule is the detection.",
|
||||
"riskTerm.field": "Field is the fact to read: one of the closed vocabulary, or signal.<name>,\nor velocity.<axis>.<window>.<stat>. GET /v1/risk/dictionary lists them.",
|
||||
"riskTerm.number": "Number is the numeric operand.",
|
||||
"riskTerm.op": "Op is the comparison: eq, ne, gt, gte, lt, lte, in, notin, inlist,\nnotinlist, exists, absent, contains, prefix or suffix.",
|
||||
"riskTerm.value": "Value is the string operand.",
|
||||
"riskTerm.values": "Values is the set operand, for in and notin.",
|
||||
},
|
||||
})
|
||||
zip.Describe("POST /v1/risk/controls", zip.Doc{
|
||||
Description: "Declares a reserve, a payout hold or a block on a subject.",
|
||||
Fields: map[string]string{
|
||||
"riskControl.at": "At is when, RFC 3339 in UTC.",
|
||||
"riskControl.by": "By is who declared it, taken from the validated principal.",
|
||||
"riskControl.control": "Control is which one.",
|
||||
"riskControl.id": "ID identifies the control.",
|
||||
"riskControl.rate": "Rate is the reserve fraction, for a reserve.",
|
||||
"riskControl.reason": "Reason is why it was declared.",
|
||||
"riskControl.subject": "Subject is what it applies to.",
|
||||
"riskControl.until": "Until is when it lapses, RFC 3339. Absent means it does not.",
|
||||
"riskControlIn.control": "Control is which one: reserve, payout-hold or block.",
|
||||
"riskControlIn.rate": "Rate is the reserve fraction, in [0,1], for a reserve.",
|
||||
"riskControlIn.reason": "Reason is why.",
|
||||
"riskControlIn.subject": "Subject is what the control applies to.",
|
||||
"riskControlIn.until": "Until is when it lapses, RFC 3339. Absent means it does not.",
|
||||
"riskSubject.id": "ID is the subject's identifier within the caller's own org. It is never\ninterpreted, only aggregated on, so it may be any stable string.",
|
||||
"riskSubject.kind": "Kind is the sort of thing this is: account, transaction, session, agent,\nmerchant or payout. It selects the aggregation axes, not the tenant gate.",
|
||||
},
|
||||
})
|
||||
zip.Describe("POST /v1/risk/decide", zip.Doc{
|
||||
Description: "Decide scores one entity or event and answers what to do about it: allow,\nchallenge, review, restrict or block, with the evidence behind it.\n\nOne op, not three. \"Score this thing\" is one verb, and a stage-specific op per\nlifecycle moment would be three places for the tenant gate to drift. The stage\nselects the feature window and the rule set and nothing else.\n\nIt is METERED AFTER the work, never gated before it. A pre-work balance gate\nwould have to render its refusal in the fleet's nested error contract, which a\ntyped op cannot do — so the hot path stays typed, the screen is billed on the\ndecision that was actually produced, and no funded client's 402 parsing is\naffected. Overage lands on the caller's OWN ledger: Meter sends the org as\nboth the commerce identity and the org header, which is the anti-cross-org\nproperty.",
|
||||
Fields: map[string]string{
|
||||
"riskActor.agent": "Agent is the agent reference the caller claims to be acting as. It is\nresolved against THIS org's agent registry; an unresolvable reference is\nsimply not a declaration.",
|
||||
"riskActor.session": "Session is the session reference this action belongs to.",
|
||||
"riskAmount.currency": "Currency is the ISO 4217 code.",
|
||||
"riskAmount.direction": "Direction is whether value came in or went out: \"in\" or \"out\". Without it\nthe model cannot see funds passing through, which is the layering shape.",
|
||||
"riskAmount.nano": "Nano is the amount in billionths of one unit of Currency.",
|
||||
"riskCause.baseline": "Baseline is what is unremarkable for this subject.",
|
||||
"riskCause.citation": "Citation is where those words come from.",
|
||||
"riskCause.feature": "Feature is the model dimension.",
|
||||
"riskCause.indicator": "Indicator is the supervisor's own words for what is being looked for.",
|
||||
"riskCause.observed": "Observed is this subject's value.",
|
||||
"riskCause.share": "Share is this feature's part of the score, in [0,1].",
|
||||
"riskCause.typology": "Typology is the pattern the feature expresses.",
|
||||
"riskCause.unit": "Unit is how to read Observed and Baseline.",
|
||||
"riskCause.without": "Without is the score the model would have produced with this feature\nneutral, holding everything else.",
|
||||
"riskDecideIn.actor": "Actor is who is acting, if the caller knows.",
|
||||
"riskDecideIn.amount": "Amount is the money involved, for the stages where there is any.",
|
||||
"riskDecideIn.idem": "Idem makes a re-decide idempotent: the same key returns the same decision\nrather than scoring twice and moving the counters twice.",
|
||||
"riskDecideIn.signals": "Signals are the observations the caller can supply: ip, email,\nemaildomain, device, bin, asn, country, counterparty. Unknown keys are\ncarried and are readable by a rule as signal.<key>, so a tenant can score\non facts only it has.",
|
||||
"riskDecideIn.stage": "Stage is the lifecycle moment being judged: signup, payment, session,\nusage, payout or dispute. It selects the feature window and the rule set;\nit does NOT select a different tenant gate.",
|
||||
"riskDecideIn.subject": "Subject is what is being judged.",
|
||||
"riskDecision.action": "Action is what to do: allow, challenge, review, restrict or block.",
|
||||
"riskDecision.agency": "Agency is what kind of actor this was: agent, human, bot or unknown.\nRESOLVED server-side: an agent reference on the observation is looked up in\nYOUR OWN agent registry, and it is `agent` only when that lookup succeeds.\nA reference the registry does not know is `bot` — a claim we can disprove\nis the strongest signal there is — and a lookup that could not be made\nleaves this `unknown` with Refusal set to `unverified`. It is never read\noff a user-agent string, which is a claim rather than a fact.",
|
||||
"riskDecision.causes": "Causes is the model's per-feature attribution, when the model contributed.",
|
||||
"riskDecision.hits": "Hits is the evidence, strongest first.",
|
||||
"riskDecision.id": "ID identifies this decision for the whole of its life: the label, the\nevidence read and the dispute packet all key on it.",
|
||||
"riskDecision.model": "Model is the digest of the model that produced this, so an auditor can pin\nthe exact geometry that raised an alert.",
|
||||
"riskDecision.refusal": "Refusal names what this decision was short of, when it was short of\nanything. It is present precisely so that silence never reads as a clean\nresult. ONE word, and when more than one applies it is the one that will\nnot fix itself, in this order:\n\n\tunidentified nothing to key the aggregates on.\n\tdisarmed learned state existed and this process does not have it.\n\t A control that is OFF, and never reported as `warming`.\n\tunverified a claimed agent reference could not be checked against your\n\t registry, so the Agency on this decision is a gap.\n\twarming the model is still building this tenant's baseline. Ordinary,\n\t and resolved by your own traffic.\n\tshadow this tenant is observing; nothing acted.",
|
||||
"riskDecision.score": "Score is the weight-of-evidence score in [0,1].",
|
||||
"riskDecision.shadow": "Shadow is true when this tenant is observing rather than acting. In shadow\nAction is always allow and everything else is computed and recorded.",
|
||||
"riskDecision.since": "Since is when the sliding aggregates behind this decision started, RFC 3339\nin UTC. A velocity count over a 30-day window computed from ten minutes of\naggregates is a true number about the wrong period, so the period is\npublished rather than assumed: this app is deployed at one replica with a\nrecreate rollout, and a reclaim of an idle tenant restarts them too.",
|
||||
"riskDecision.strained": "Strained is true when this tenant's aggregates were at their own\ncardinality bound, so a count may be short of this tenant's own traffic.\nOnly this tenant's activity can cause it, and only this tenant's counts are\naffected.",
|
||||
"riskHit.action": "Action is what this evidence alone asked for.",
|
||||
"riskHit.name": "Name is the detection's human-readable name.",
|
||||
"riskHit.rule": "Rule is the identifier of the rule or model that produced this evidence.",
|
||||
"riskHit.severity": "Severity is the reviewer-facing grading.",
|
||||
"riskHit.suppressed": "Suppressed marks evidence a suppression muted. It is reported and\ncontributes nothing — a muted control that left no trace would be\nindistinguishable from one that was never running.",
|
||||
"riskHit.weight": "Weight is how much it contributed, in [0,1].",
|
||||
"riskSubject.id": "ID is the subject's identifier within the caller's own org. It is never\ninterpreted, only aggregated on, so it may be any stable string.",
|
||||
"riskSubject.kind": "Kind is the sort of thing this is: account, transaction, session, agent,\nmerchant or payout. It selects the aggregation axes, not the tenant gate.",
|
||||
},
|
||||
})
|
||||
zip.Describe("POST /v1/risk/decisions/:id/label", zip.Doc{
|
||||
Description: "Label records what a human concluded about a decision — fraud, legitimate,\nchargeback, abuse or unknown.\n\nTHE DECIDER IS SERVER-SET. It comes from the validated principal and never\nfrom the request body: the engine's own resolutions carry an unauthenticated\n`by`, and importing that gap into a product that bills on the decision would\nmake the attribution worthless the moment anyone asked who cleared what.\n\nThis is also the only supervision this product has. Everything the model knows\nabout its own miss rate comes from these labels against the below-the-line\nsample.",
|
||||
Fields: map[string]string{
|
||||
"riskCause.baseline": "Baseline is what is unremarkable for this subject.",
|
||||
"riskCause.citation": "Citation is where those words come from.",
|
||||
"riskCause.feature": "Feature is the model dimension.",
|
||||
"riskCause.indicator": "Indicator is the supervisor's own words for what is being looked for.",
|
||||
"riskCause.observed": "Observed is this subject's value.",
|
||||
"riskCause.share": "Share is this feature's part of the score, in [0,1].",
|
||||
"riskCause.typology": "Typology is the pattern the feature expresses.",
|
||||
"riskCause.unit": "Unit is how to read Observed and Baseline.",
|
||||
"riskCause.without": "Without is the score the model would have produced with this feature\nneutral, holding everything else.",
|
||||
"riskDecisionBrief.action": "Action is what was decided.",
|
||||
"riskDecisionBrief.agency": "Agency is the actor class.",
|
||||
"riskDecisionBrief.at": "At is when it was made, RFC 3339 in UTC.",
|
||||
"riskDecisionBrief.id": "ID identifies the decision.",
|
||||
"riskDecisionBrief.kind": "Kind is the subject kind.",
|
||||
"riskDecisionBrief.label": "Label is what a human later concluded, when anyone has.",
|
||||
"riskDecisionBrief.refusal": "Refusal names what the decision was short of, when it was short of\nanything.",
|
||||
"riskDecisionBrief.score": "Score is the weight-of-evidence score.",
|
||||
"riskDecisionBrief.shadow": "Shadow is whether the decision acted.",
|
||||
"riskDecisionBrief.since": "Since is when the aggregates this decision read had started, RFC 3339 in\nUTC — the period its velocity numbers actually cover.",
|
||||
"riskDecisionBrief.stage": "Stage is the lifecycle moment.",
|
||||
"riskDecisionBrief.strained": "Strained is true when the aggregates were at this tenant's own cardinality\nbound when the decision was made.",
|
||||
"riskDecisionBrief.subject": "Subject is the subject identifier.",
|
||||
"riskDecisionView.causes": "Causes is the model's attribution.",
|
||||
"riskDecisionView.decision": "Decision is the row.",
|
||||
"riskDecisionView.hits": "Hits is the evidence.",
|
||||
"riskDecisionView.model": "Model is the digest of the model that produced it.",
|
||||
"riskHit.action": "Action is what this evidence alone asked for.",
|
||||
"riskHit.name": "Name is the detection's human-readable name.",
|
||||
"riskHit.rule": "Rule is the identifier of the rule or model that produced this evidence.",
|
||||
"riskHit.severity": "Severity is the reviewer-facing grading.",
|
||||
"riskHit.suppressed": "Suppressed marks evidence a suppression muted. It is reported and\ncontributes nothing — a muted control that left no trace would be\nindistinguishable from one that was never running.",
|
||||
"riskHit.weight": "Weight is how much it contributed, in [0,1].",
|
||||
"riskLabelIn.id": "ID is the decision, from the path.",
|
||||
"riskLabelIn.verdict": "Verdict is what it turned out to be: fraud, legitimate, chargeback, abuse\nor unknown. It is the only supervision this product has, and it is what\nturns a false-positive rate from a guess into a measurement.",
|
||||
},
|
||||
})
|
||||
zip.Describe("POST /v1/risk/lists", zip.Doc{
|
||||
Description: "Creates an allow or deny list a rule can name in an inlist term.",
|
||||
Fields: map[string]string{
|
||||
"riskListIn.kind": "Kind is whether membership allows or denies.",
|
||||
"riskListIn.name": "Name is the list's identifier.",
|
||||
"riskListView.createdAt": "CreatedAt is when it was created, RFC 3339 in UTC.",
|
||||
"riskListView.entries": "Entries is how many values it holds.",
|
||||
"riskListView.kind": "Kind is whether membership allows or denies: \"allow\" or \"deny\".",
|
||||
"riskListView.name": "Name is the list's identifier, which a rule names in an inlist term.",
|
||||
},
|
||||
})
|
||||
zip.Describe("POST /v1/risk/lists/:name/entries", zip.Doc{
|
||||
Description: "Adds values to a list. Values are folded to lower case so a\nvalue added in one case matches a signal sent in another.",
|
||||
Fields: map[string]string{
|
||||
"riskListEntriesIn.name": "Name is the list, from the path.",
|
||||
"riskListEntriesIn.values": "Values are the values to add. They are folded to lower case, so a value\nadded in one case matches a signal in another.",
|
||||
"riskListView.createdAt": "CreatedAt is when it was created, RFC 3339 in UTC.",
|
||||
"riskListView.entries": "Entries is how many values it holds.",
|
||||
"riskListView.kind": "Kind is whether membership allows or denies: \"allow\" or \"deny\".",
|
||||
"riskListView.name": "Name is the list's identifier, which a rule names in an inlist term.",
|
||||
},
|
||||
})
|
||||
zip.Describe("POST /v1/risk/restore", zip.Doc{
|
||||
Description: "Restore reinstates this tenant's learned state from its pinned snapshot.\n\nA snapshot whose shape does not match the running inventory is REFUSED, not\ncoerced, and a snapshot belonging to another tenant is refused twice — here,\nby the tenant that asked, and again inside the engine. State the model would\ntreat as its own memory has to have come from this algorithm over this feature\nset for this tenant.",
|
||||
Fields: map[string]string{
|
||||
"riskAppetite.review": "Review is the share of the stream that may be sent for examination. The\nalert threshold is derived from it as a quantile of the scores actually\nobserved, rather than fixed at a number someone liked — so an alert level\nis governed rather than tuned, and a drifting distribution does not turn\nit into silence or a flood.",
|
||||
"riskAppetite.sample": "Sample is the share of NON-alerting traffic retained for review below the\nline. It is the only instrument that can measure what the model missed,\nbecause nothing in the stream is labelled.",
|
||||
"riskAppetite.warm": "Warm is how many observations the model must learn before it may score.",
|
||||
"riskFeature.citation": "Citation is where those words come from, so the claim is checkable.",
|
||||
"riskFeature.indicator": "Indicator is the supervisor's own words for what is being looked for.",
|
||||
"riskFeature.name": "Name is the dimension.",
|
||||
"riskFeature.neutral": "Neutral is the value at which this feature is unremarkable — the\ncoordinate the counterfactual moves it to.",
|
||||
"riskFeature.severity": "Severity is the grading a hit on this feature carries.",
|
||||
"riskFeature.typology": "Typology is the pattern it expresses.",
|
||||
"riskFeature.unit": "Unit is how to read the raw number.",
|
||||
"riskFeature.window": "Window is the span it is measured over, when it has one.",
|
||||
"riskModelState.alerted": "Alerted is how many of those it alerted on — the numerator of Realised.",
|
||||
"riskModelState.appetite": "Appetite is the stated share of the stream that may be examined.",
|
||||
"riskModelState.blind": "Blind counts, per feature, how often it took its neutral value for want of\ndata.",
|
||||
"riskModelState.digest": "Digest is the model identity an auditor pins.",
|
||||
"riskModelState.distribution": "Distribution is the score distribution the threshold is cut from, in 32\nbands.",
|
||||
"riskModelState.inventory": "Inventory is the feature set, so the state and the shape are read\ntogether.",
|
||||
"riskModelState.learned": "Learned is how many observations this tenant's model has taken in.",
|
||||
"riskModelState.realised": "Realised is the share that ACTUALLY alerted. Stated against realised is\nthe governance report — an appetite is a measured commitment or it is\nnothing.",
|
||||
"riskModelState.refusals": "Refusals counts what the model declined to score, by reason. None of these\nwas examined by the model; all of them were examined by the rules.",
|
||||
"riskModelState.saturated": "Saturated means the appetite cannot be honoured by any threshold because\ntoo much of the stream scores in the top band. It is the one state that\nmust never be mistaken for quiet.",
|
||||
"riskModelState.scored": "Scored is how many observations the model was able to score — the\ndenominator of Realised.",
|
||||
"riskModelState.threshold": "Threshold is the score cut currently in force, recomputed each window as\nthe quantile that admits the stated share.",
|
||||
"riskModelState.warming": "Warming is true while the model has not learned enough to score. A warming\nmodel REFUSES rather than scoring low, and the refusal is counted below.",
|
||||
},
|
||||
})
|
||||
zip.Describe("POST /v1/risk/rules", zip.Doc{
|
||||
Description: "Admits a new detection. Admission is strict on purpose: a rule with\nno terms holds on everything, a rule naming a field that does not exist holds\non nothing, and both read as a working control from the outside.",
|
||||
Fields: map[string]string{
|
||||
"riskRule.action": "Action is what the rule asks for when it holds.",
|
||||
"riskRule.all": "All is the conjunction. An empty conjunction is refused: a rule that holds\non everything is not a detection.",
|
||||
"riskRule.enabled": "Enabled governs the live path. A disabled rule still replays, because the\nquestion a simulation asks is what happens ON activation.",
|
||||
"riskRule.id": "ID is the rule's identifier within this tenant.",
|
||||
"riskRule.name": "Name is what a reviewer reads in an alert.",
|
||||
"riskRule.severity": "Severity is the reviewer-facing grading: low, medium, high or critical.",
|
||||
"riskRule.stage": "Stage narrows the rule to one lifecycle moment; empty means every stage.",
|
||||
"riskRule.weight": "Weight is how much a hit contributes, in [0,1].",
|
||||
"riskRuleIn.id": "ID is the rule to replace, from the path on a PATCH. Ignored on create,\nwhere the server mints one.",
|
||||
"riskRuleIn.rule": "Rule is the detection.",
|
||||
"riskTerm.field": "Field is the fact to read: one of the closed vocabulary, or signal.<name>,\nor velocity.<axis>.<window>.<stat>. GET /v1/risk/dictionary lists them.",
|
||||
"riskTerm.number": "Number is the numeric operand.",
|
||||
"riskTerm.op": "Op is the comparison: eq, ne, gt, gte, lt, lte, in, notin, inlist,\nnotinlist, exists, absent, contains, prefix or suffix.",
|
||||
"riskTerm.value": "Value is the string operand.",
|
||||
"riskTerm.values": "Values is the set operand, for in and notin.",
|
||||
},
|
||||
})
|
||||
zip.Describe("POST /v1/risk/score", zip.Doc{
|
||||
Description: "Score scores one observation and LEARNS NOTHING from it. It is how a candidate\nis tried against a tenant's real behaviour before anything depends on the\nanswer, and it is the model's analogue of testing a rule: because it records\nnothing, the aggregates it reads do not include the candidate.",
|
||||
Fields: map[string]string{
|
||||
"riskAmount.currency": "Currency is the ISO 4217 code.",
|
||||
"riskAmount.direction": "Direction is whether value came in or went out: \"in\" or \"out\". Without it\nthe model cannot see funds passing through, which is the layering shape.",
|
||||
"riskAmount.nano": "Nano is the amount in billionths of one unit of Currency.",
|
||||
"riskCause.baseline": "Baseline is what is unremarkable for this subject.",
|
||||
"riskCause.citation": "Citation is where those words come from.",
|
||||
"riskCause.feature": "Feature is the model dimension.",
|
||||
"riskCause.indicator": "Indicator is the supervisor's own words for what is being looked for.",
|
||||
"riskCause.observed": "Observed is this subject's value.",
|
||||
"riskCause.share": "Share is this feature's part of the score, in [0,1].",
|
||||
"riskCause.typology": "Typology is the pattern the feature expresses.",
|
||||
"riskCause.unit": "Unit is how to read Observed and Baseline.",
|
||||
"riskCause.without": "Without is the score the model would have produced with this feature\nneutral, holding everything else.",
|
||||
"riskObservation.amount": "Amount is the money involved, if any.",
|
||||
"riskObservation.at": "At is when it happened, RFC 3339. Absent means now.",
|
||||
"riskObservation.signals": "Signals are the observations: ip, device, counterparty and the rest.",
|
||||
"riskObservation.subject": "Subject is whose behaviour this is.",
|
||||
"riskScoreIn.observation": "Observation is what to score.",
|
||||
"riskScoreOut.alert": "Alert is whether this would become evidence.",
|
||||
"riskScoreOut.causes": "Causes is the per-feature attribution, ordered by contribution.",
|
||||
"riskScoreOut.cut": "Cut is the threshold in force, derived from the appetite.",
|
||||
"riskScoreOut.model": "Model is the digest of the model that produced this.",
|
||||
"riskScoreOut.refusal": "Refusal names the decline: warming or unidentified.",
|
||||
"riskScoreOut.score": "Score is the anomaly score in [0,1]: 0 where this tenant's recent\nbehaviour is densest, 1 where there is none of it.",
|
||||
"riskScoreOut.scored": "Scored is false when the model declined; Refusal says which refusal.",
|
||||
"riskScoreOut.shadow": "Shadow is whether the model is observing rather than contributing.",
|
||||
"riskScoreOut.values": "Values is every coordinate, including the blind ones.",
|
||||
"riskSubject.id": "ID is the subject's identifier within the caller's own org. It is never\ninterpreted, only aggregated on, so it may be any stable string.",
|
||||
"riskSubject.kind": "Kind is the sort of thing this is: account, transaction, session, agent,\nmerchant or payout. It selects the aggregation axes, not the tenant gate.",
|
||||
"riskValue.baseline": "Baseline is what is unremarkable here.",
|
||||
"riskValue.blind": "Blind is true when this feature had no usable coordinate, so it took its\nneutral value. A feature blind on most traffic is not contributing\nwhatever the inventory claims for it.",
|
||||
"riskValue.feature": "Feature is the dimension.",
|
||||
"riskValue.observed": "Observed is the raw number behind it.",
|
||||
"riskValue.unit": "Unit is how to read Observed and Baseline.",
|
||||
"riskValue.x": "X is the coordinate the model used, in [0,1].",
|
||||
},
|
||||
})
|
||||
zip.Describe("POST /v1/risk/search", zip.Doc{
|
||||
Description: "Search exhaustively tries the model topology over this tenant's own history\nand returns the learning curve and the winning shape.\n\nThe grid is CLOSED — trees x depth x window x blend x appetite — because an\nunbounded grid is an unbounded amount of a shared pod's CPU reachable by\nanyone with a key. Each candidate runs in a sandbox over fresh counters\ncarrying the SAME per-tenant memory bound the live plane does, so a search can\nnever move the live model, can never cost more than a tenant is allowed to\ncost, and can never see another tenant's history because the replay reads this\ntenant's own file.\n\nIT IS QUEUED, AND THE QUEUE IS THE BOUND. One run per tenant (a second answers\n409 naming the one already going), a fixed number of workers for the whole\nprocess, a short backlog beyond which a start answers 429, and a deadline every\ncandidate polls. Priced before the work and metered after it, on the caller's\nown ledger. See search.go.\n\nIt answers 202 with a run identifier; read the report back at\nGET /v1/risk/search/{id} and stop it at DELETE /v1/risk/search/{id}.",
|
||||
Fields: map[string]string{
|
||||
"riskSearchIn.limit": "Limit bounds how many recorded observations are replayed, 1..5000. The\ngrid itself is closed and needs no bound from the caller.",
|
||||
"riskSearchRun.candidates": "Candidates is how many topologies will be tried.",
|
||||
"riskSearchRun.id": "ID identifies the run; read it back at GET /v1/risk/search/{id}.",
|
||||
"riskSearchRun.status": "Status is \"running\" or \"done\".",
|
||||
},
|
||||
})
|
||||
zip.Describe("POST /v1/risk/simulate", zip.Doc{
|
||||
Description: "Simulate replays a candidate rule over this tenant's OWN recorded decisions\nand reports what it would have caught, what it would have given up, and how\nmuch of that a human has already judged.\n\nIt is a gate and not a nicety: a new typology has to be tested before live\nactivation, and a retirement has to be justified against the outgoing rule's\nperformance. Nothing is written — the replay reads the decision log and\nevaluates in memory.\n\nAn EMPTY history is REFUSED rather than reported as zero alerts. \"No alerts\"\nis exactly what a quiet rule looks like, and being unable to tell the two\napart is the failure a sandbox exists to prevent.\n\nBOUNDED AND SYNCHRONOUS, AND DELIBERATELY NOT PRICED. The work is ONE rule\nover at most 5,000 of the tenant's own recorded decisions, evaluated in memory\nwith no model and no writes, and the caller holds its own request open for all\nof it — so the cost is bounded per call and self-throttling across calls, which\nis what the search plane needed a queue to achieve. Not priced because this is\nthe op a tenant runs BEFORE activating a rule, and charging for the rehearsal\nis how you teach people to skip it.",
|
||||
Fields: map[string]string{
|
||||
"riskRule.action": "Action is what the rule asks for when it holds.",
|
||||
"riskRule.all": "All is the conjunction. An empty conjunction is refused: a rule that holds\non everything is not a detection.",
|
||||
"riskRule.enabled": "Enabled governs the live path. A disabled rule still replays, because the\nquestion a simulation asks is what happens ON activation.",
|
||||
"riskRule.id": "ID is the rule's identifier within this tenant.",
|
||||
"riskRule.name": "Name is what a reviewer reads in an alert.",
|
||||
"riskRule.severity": "Severity is the reviewer-facing grading: low, medium, high or critical.",
|
||||
"riskRule.stage": "Stage narrows the rule to one lifecycle moment; empty means every stage.",
|
||||
"riskRule.weight": "Weight is how much a hit contributes, in [0,1].",
|
||||
"riskSimulateIn.candidate": "Candidate is the rule to try. It is evaluated as activated whatever its\nenabled flag says: the question is what happens on activation.",
|
||||
"riskSimulateIn.incumbent": "Incumbent is the rule the candidate would replace, when it replaces one.\nNaming it is what turns a report into a justification for a retirement.",
|
||||
"riskSimulateIn.limit": "Limit bounds how many recorded decisions are replayed, 1..5000.",
|
||||
"riskSimulateReport.added": "Added is what the candidate catches and the incumbent does not — the new\ncoverage, and the new volume.",
|
||||
"riskSimulateReport.alerts": "Alerts is how many the candidate would have fired on.",
|
||||
"riskSimulateReport.dropped": "Dropped is what the incumbent catches and the candidate does not — the\ncoverage being given up, which is what a retirement has to justify.",
|
||||
"riskSimulateReport.events": "Events is how many recorded decisions were replayed.",
|
||||
"riskSimulateReport.falsePositive": "FalsePositive is the share of judged alerts a human called legitimate. It\nis ABSENT rather than zero when nothing was judged: an unmeasured\nproportion reported as 0.0 reads as a perfect rule.",
|
||||
"riskSimulateReport.judged": "Judged is how many of the candidate's alerts fall on decisions a human has\nlabelled.",
|
||||
"riskSimulateReport.kept": "Kept is what both catch.",
|
||||
"riskSimulateReport.refusal": "Refusal names why a report is empty when it is. An empty history is\nrefused rather than reported as zero alerts, because a quiet rule and an\nunrun rule produce the same number.",
|
||||
"riskTerm.field": "Field is the fact to read: one of the closed vocabulary, or signal.<name>,\nor velocity.<axis>.<window>.<stat>. GET /v1/risk/dictionary lists them.",
|
||||
"riskTerm.number": "Number is the numeric operand.",
|
||||
"riskTerm.op": "Op is the comparison: eq, ne, gt, gte, lt, lte, in, notin, inlist,\nnotinlist, exists, absent, contains, prefix or suffix.",
|
||||
"riskTerm.value": "Value is the string operand.",
|
||||
"riskTerm.values": "Values is the set operand, for in and notin.",
|
||||
},
|
||||
})
|
||||
zip.Describe("POST /v1/risk/snapshot", zip.Doc{
|
||||
Description: "Snapshot pins this tenant's learned state into its own encrypted file.\n\nIt is what an auditor asks for: the exact model that raised an alert, kept so\nthat the same input can be scored again a year later. It is also what makes a\nrollout survivable — the shutdown path snapshots every resident tenant,\nbecause cloud deploys Recreate at one replica and a model that comes back with\nnothing learned refuses to score for its whole warm period.",
|
||||
Fields: map[string]string{
|
||||
"riskSnapshotOut.at": "At is when it was pinned, RFC 3339 in UTC.",
|
||||
"riskSnapshotOut.digest": "Digest is the model identity that was pinned.",
|
||||
"riskSnapshotOut.learned": "Learned is how many observations that state had taken in.",
|
||||
},
|
||||
})
|
||||
zip.Describe("POST /v1/risk/suppressions", zip.Doc{
|
||||
Description: "Suppress mutes a rule's activations for a subject, a subject kind, or\neverywhere that rule fires.\n\nA suppressed hit is RECORDED with suppressed set and contributes nothing to\nthe action. It is never dropped, for the same reason the model counts its\nrefusals: silence must never read as a clean result, and a compliance record\nthat can be silently muted by an operational knob is not a record.",
|
||||
Fields: map[string]string{
|
||||
"riskSuppressIn.kind": "Kind narrows the suppression to one subject kind.",
|
||||
"riskSuppressIn.reason": "Reason is why. It is required, because a mute nobody can explain later is\na control that quietly stopped.",
|
||||
"riskSuppressIn.rule": "Rule is the rule to mute. At least one of Rule, Kind or Subject must be\ngiven — a suppression that names nothing would mute everything.",
|
||||
"riskSuppressIn.subject": "Subject narrows it to one subject.",
|
||||
"riskSuppressIn.until": "Until is when it expires, RFC 3339. Absent means it does not, which is a\nchoice somebody should be able to be asked about.",
|
||||
"riskSuppression.at": "At is when it was applied, RFC 3339 in UTC.",
|
||||
"riskSuppression.by": "By is who applied it, taken from the validated principal and never from\nthe request body.",
|
||||
"riskSuppression.id": "ID identifies the suppression.",
|
||||
"riskSuppression.kind": "Kind is the subject kind muted, when it names one.",
|
||||
"riskSuppression.reason": "Reason is why it was applied.",
|
||||
"riskSuppression.rule": "Rule is the rule muted, when it names one.",
|
||||
"riskSuppression.subject": "Subject is the subject muted, when it names one.",
|
||||
"riskSuppression.until": "Until is when it expires, RFC 3339. Absent means it does not.",
|
||||
},
|
||||
})
|
||||
zip.Describe("POST /v1/risk/train", zip.Doc{
|
||||
Description: "Train teaches this tenant's model from this tenant's own observations.\n\nIt is online: there is no job, no queue and no window in which the tenant is\nprotected by a stale model, because the half-space geometry is built before\nany data arrives and the model IS a set of mass counters. One observation is\none increment.\n\nTHERE IS NO CROSS-TENANT TRAINING INPUT AND NO WAY TO EXPRESS ONE. The model\nis indexed by the tenant key and its tree GEOMETRY is seeded from it, so two\ntenants do not merely hold different counters — they hold different trees.\nAny cross-org learning in this product is aggregate-only, lives in a table\nwith no tenant column, and is published only above a k-anonymity floor.",
|
||||
Fields: map[string]string{
|
||||
"riskAmount.currency": "Currency is the ISO 4217 code.",
|
||||
"riskAmount.direction": "Direction is whether value came in or went out: \"in\" or \"out\". Without it\nthe model cannot see funds passing through, which is the layering shape.",
|
||||
"riskAmount.nano": "Nano is the amount in billionths of one unit of Currency.",
|
||||
"riskObservation.amount": "Amount is the money involved, if any.",
|
||||
"riskObservation.at": "At is when it happened, RFC 3339. Absent means now.",
|
||||
"riskObservation.signals": "Signals are the observations: ip, device, counterparty and the rest.",
|
||||
"riskObservation.subject": "Subject is whose behaviour this is.",
|
||||
"riskSubject.id": "ID is the subject's identifier within the caller's own org. It is never\ninterpreted, only aggregated on, so it may be any stable string.",
|
||||
"riskSubject.kind": "Kind is the sort of thing this is: account, transaction, session, agent,\nmerchant or payout. It selects the aggregation axes, not the tenant gate.",
|
||||
"riskTrainIn.observations": "Observations are what to learn from. They are this tenant's own data and\nnothing else: there is no cross-tenant training input and no way to\nexpress one, because the model is indexed by the tenant key and its tree\nGEOMETRY is seeded from it.",
|
||||
"riskTrainOut.learned": "Learned is how many observations were incorporated.",
|
||||
"riskTrainOut.model": "Model is the digest of the model that learned them.",
|
||||
"riskTrainOut.refused": "Refused is how many were not, by reason.",
|
||||
"riskTrainOut.warm": "Warm is true once the model has learned enough of this tenant's behaviour\nto score at all.",
|
||||
},
|
||||
})
|
||||
zip.Describe("PUT /v1/risk/mode", zip.Doc{
|
||||
Description: "Takes this tenant live, or returns it to shadow.\n\nSHADOW IS THE DEFAULT AND THE DEFAULT IS NOT CONFIGURABLE. In shadow every\nrule runs, the model scores and learns, every decision is recorded, and the\naction is always allow — so what would have happened is readable over real\ntraffic before anyone depends on it. A model that quietly went live and\nstarted declining payments is the worst failure available here.",
|
||||
Fields: map[string]string{
|
||||
"riskModeIn.mode": "Mode is \"shadow\" or \"live\". Shadow is the default and the default is not\nconfigurable: a model that quietly went live and started declining\npayments is the worst failure available here, so going live is an act\nsomebody performs and can be asked about.",
|
||||
"riskModeView.mode": "Mode is \"shadow\" or \"live\".",
|
||||
"riskModeView.since": "Since is when it was last changed, RFC 3339 in UTC. Empty means never.",
|
||||
},
|
||||
})
|
||||
zip.Describe("PUT /v1/risk/state/appetite", zip.Doc{
|
||||
Description: "Sets how much of the stream the model may examine.\n\nReview is the lever, and the threshold is derived from it as a quantile of the\nscores actually observed rather than fixed at a number someone liked. That is\nwhat makes an alert level governed rather than tuned, and it is why the level\ncannot be set to fit the size of the review team.",
|
||||
Fields: map[string]string{
|
||||
"riskAppetite.review": "Review is the share of the stream that may be sent for examination. The\nalert threshold is derived from it as a quantile of the scores actually\nobserved, rather than fixed at a number someone liked — so an alert level\nis governed rather than tuned, and a drifting distribution does not turn\nit into silence or a flood.",
|
||||
"riskAppetite.sample": "Sample is the share of NON-alerting traffic retained for review below the\nline. It is the only instrument that can measure what the model missed,\nbecause nothing in the stream is labelled.",
|
||||
"riskAppetite.warm": "Warm is how many observations the model must learn before it may score.",
|
||||
"riskAppetiteIn.review": "Review is the share of the stream that may be examined, in (0, 0.5].",
|
||||
"riskAppetiteIn.sample": "Sample is the share of non-alerting traffic retained, in [0, 1].",
|
||||
"riskFeature.citation": "Citation is where those words come from, so the claim is checkable.",
|
||||
"riskFeature.indicator": "Indicator is the supervisor's own words for what is being looked for.",
|
||||
"riskFeature.name": "Name is the dimension.",
|
||||
"riskFeature.neutral": "Neutral is the value at which this feature is unremarkable — the\ncoordinate the counterfactual moves it to.",
|
||||
"riskFeature.severity": "Severity is the grading a hit on this feature carries.",
|
||||
"riskFeature.typology": "Typology is the pattern it expresses.",
|
||||
"riskFeature.unit": "Unit is how to read the raw number.",
|
||||
"riskFeature.window": "Window is the span it is measured over, when it has one.",
|
||||
"riskModelState.alerted": "Alerted is how many of those it alerted on — the numerator of Realised.",
|
||||
"riskModelState.appetite": "Appetite is the stated share of the stream that may be examined.",
|
||||
"riskModelState.blind": "Blind counts, per feature, how often it took its neutral value for want of\ndata.",
|
||||
"riskModelState.digest": "Digest is the model identity an auditor pins.",
|
||||
"riskModelState.distribution": "Distribution is the score distribution the threshold is cut from, in 32\nbands.",
|
||||
"riskModelState.inventory": "Inventory is the feature set, so the state and the shape are read\ntogether.",
|
||||
"riskModelState.learned": "Learned is how many observations this tenant's model has taken in.",
|
||||
"riskModelState.realised": "Realised is the share that ACTUALLY alerted. Stated against realised is\nthe governance report — an appetite is a measured commitment or it is\nnothing.",
|
||||
"riskModelState.refusals": "Refusals counts what the model declined to score, by reason. None of these\nwas examined by the model; all of them were examined by the rules.",
|
||||
"riskModelState.saturated": "Saturated means the appetite cannot be honoured by any threshold because\ntoo much of the stream scores in the top band. It is the one state that\nmust never be mistaken for quiet.",
|
||||
"riskModelState.scored": "Scored is how many observations the model was able to score — the\ndenominator of Realised.",
|
||||
"riskModelState.threshold": "Threshold is the score cut currently in force, recomputed each window as\nthe quantile that admits the stated share.",
|
||||
"riskModelState.warming": "Warming is true while the model has not learned enough to score. A warming\nmodel REFUSES rather than scoring low, and the refusal is counted below.",
|
||||
},
|
||||
})
|
||||
}
|
||||
@@ -33,6 +33,7 @@ require (
|
||||
github.com/hanzoai/types v0.1.0
|
||||
github.com/hanzokv/go/v9 v9.22.0
|
||||
github.com/lib/pq v1.12.3
|
||||
github.com/luxfi/aml v0.3.5
|
||||
github.com/luxfi/log v1.6.0
|
||||
github.com/luxfi/node v1.36.15
|
||||
github.com/luxfi/trace v1.4.0
|
||||
@@ -489,7 +490,7 @@ require (
|
||||
github.com/luxfi/corona v0.10.4 // indirect
|
||||
github.com/luxfi/crypto v1.20.2
|
||||
github.com/luxfi/crypto/ipa v1.2.4 // indirect
|
||||
github.com/luxfi/fhe v1.8.2 // indirect
|
||||
github.com/luxfi/fhe v1.8.8 // indirect
|
||||
github.com/luxfi/geth v1.20.1
|
||||
github.com/luxfi/ids v1.3.2
|
||||
github.com/luxfi/kms v1.12.9
|
||||
|
||||
@@ -787,6 +787,8 @@ github.com/goccy/go-yaml v1.9.5/go.mod h1:U/jl18uSupI5rdI2jmuCswEA2htH9eXfferR3K
|
||||
github.com/goccy/go-yaml v1.19.2 h1:PmFC1S6h8ljIz6gMRBopkjP1TVT7xuwrButHID66PoM=
|
||||
github.com/goccy/go-yaml v1.19.2/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
|
||||
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
|
||||
github.com/gofiber/fiber/v3 v3.2.0 h1:g9+09D320foINPpCnR3ibQ5oBEFHjAWRRfDG1te54u8=
|
||||
github.com/gofiber/fiber/v3 v3.2.0/go.mod h1:FHOsc2Db7HhHpsE62QAaJlXVV1pNkbZEptZ4jtti7m4=
|
||||
github.com/gofiber/schema v1.7.1 h1:oSJBKdgP8JeIME4TQSAqlNKTU2iBB+2RNmKi8Nsc+TI=
|
||||
github.com/gofiber/schema v1.7.1/go.mod h1:A/X5Ffyru4p9eBdp99qu+nzviHzQiZ7odLT+TwxWhbk=
|
||||
github.com/gofiber/utils/v2 v2.0.4 h1:WwAxUA7L4MW2DjdEHF234lfqvBqd2vYYuBtA9TJq2ec=
|
||||
@@ -1379,6 +1381,8 @@ github.com/luxfi/accel v1.2.4 h1:5VbIHyEvvfobn2zBiTFODxDw1CeqxCepZOLlvkuf9yQ=
|
||||
github.com/luxfi/accel v1.2.4/go.mod h1:ISIwAX+ZfsL/S5nsP2JvfldXN6Nc+QzoWf6Jtaq+xsQ=
|
||||
github.com/luxfi/age v1.6.0 h1:KMD8gSOP4NVCb7NWSlRcgBZNV2xm2a+qQWPyPmiX6f4=
|
||||
github.com/luxfi/age v1.6.0/go.mod h1:7cu9CIyikgyAvr5MlXFapEDQ15yBaHOSdKkK5lG04WE=
|
||||
github.com/luxfi/aml v0.3.5 h1:teV4i5Jbnrirkb+7qTc3Uuxvy12KUihdX4cMRbYyefo=
|
||||
github.com/luxfi/aml v0.3.5/go.mod h1:70tagqrq4OOO3BPk+vblWq8iVuGMeeeka/xwCYN2o24=
|
||||
github.com/luxfi/atomic v1.0.0 h1:xUV60MuzRvXngaQ1sM0yVC2v4TRoLlUGkkH7M9PS4yw=
|
||||
github.com/luxfi/atomic v1.0.0/go.mod h1:0G2mTlQ6TXWHICUHrUUPu1/qAiIyR4gSZ2tva9ci/bI=
|
||||
github.com/luxfi/cache v1.3.1 h1:grQhi/B5GKypG7avDMeY143QTgFbfEvQICKNIh1Cw6U=
|
||||
@@ -1405,6 +1409,8 @@ github.com/luxfi/dkg v0.3.5 h1:s2L2mMQaz+n9m0b0ghvoV5VZNxiwb2z4WrGugvK0udY=
|
||||
github.com/luxfi/dkg v0.3.5/go.mod h1:M+WH7GFRN+YUD851Rlnumdp0Md98kplNN8pVx65U8I8=
|
||||
github.com/luxfi/fhe v1.8.2 h1:QllnObNFbi6D4mvFI6uQkepW8HgLtdy4RMR1TKYAInA=
|
||||
github.com/luxfi/fhe v1.8.2/go.mod h1:16yxwhcnCez/rNcd/C9JjH9IjbEz73X+0tvlsONyLeA=
|
||||
github.com/luxfi/fhe v1.8.8 h1:3DuWY1OvnhbMPhjQT5RPmNXUGnu8x6KaJAohVQcwUXw=
|
||||
github.com/luxfi/fhe v1.8.8/go.mod h1:3EsrY0COtGmQGpab3Nb/bA67ZUeWMS6/nwepUeacjpY=
|
||||
github.com/luxfi/filesystem v0.0.1 h1:VZ6xMFKaAPBW/ddlMsDnI2G0VU1lV5rYaVcW5d+KwEY=
|
||||
github.com/luxfi/filesystem v0.0.1/go.mod h1:OQVSU6XNwqrr1AI+MqkID2taHUclx7NYmmr3svgttec=
|
||||
github.com/luxfi/geth v1.20.1 h1:QUGQr4AKvADjwMi7t8a0OfoyxShgEcI9pwie1jFYfm0=
|
||||
|
||||
@@ -106,6 +106,18 @@ var Apps = []App{
|
||||
{Name: "catalogsync", Prefixes: []string{"/v1/catalogsync"}, Eager: true},
|
||||
{Name: "webhooks", Prefixes: []string{"/v1/webhooks"}},
|
||||
{Name: "ml", Prefixes: []string{"/v1/ml/health", "/v1/ml/models", "/v1/train/experiments", "/v1/train/health", "/v1/train/jobs"}},
|
||||
// risk owns ONE prefix, and that is the point: deciding and learning are the
|
||||
// same per-tenant state — one half-space forest held as mass counters that
|
||||
// every score reads and every train writes — so they are one binary and one
|
||||
// face. Splitting them across two rows would put the counters in two
|
||||
// processes, and there would be no error, no log and no 404, just two
|
||||
// different answers to one question.
|
||||
//
|
||||
// It claims nothing under /v1/ml. That prefix is ml's row above: SERVING —
|
||||
// InferenceServices, /v1/ml/models, predict. "Models you serve" and "models
|
||||
// that learn" are two concepts, and two concepts under one name is what this
|
||||
// table exists to refuse.
|
||||
{Name: "risk", Prefixes: []string{"/v1/risk"}},
|
||||
{Name: "usage", Prefixes: []string{"/v1/usage"}},
|
||||
{Name: "leaderboard", Prefixes: []string{"/v1/usage/activity", "/v1/usage/leaderboard", "/v1/usage/rollup/backfill"}},
|
||||
{Name: "crm", Prefixes: []string{"/v1/crm"}},
|
||||
|
||||
@@ -23,7 +23,7 @@ var frozen = []string{
|
||||
"dns", "domain", "prompts", "agents", "link", "wallets",
|
||||
"x402", "deploy", "functions", "tracker", "templates", "blueprint",
|
||||
"framework", "knowledge", "help", "content", "catalogsync", "webhooks",
|
||||
"ml", "usage", "leaderboard", "crm", "marketing", "ads",
|
||||
"ml", "risk", "usage", "leaderboard", "crm", "marketing", "ads",
|
||||
"campaign", "validators", "social", "analytics", "git", "sync",
|
||||
"visor", "venue", "captable", "code", "zt", "share",
|
||||
"dataroom", "graph", "security", "integrations", "destinations", "cloudflare",
|
||||
|
||||
+2164
File diff suppressed because it is too large
Load Diff
+3
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"paths": 1399,
|
||||
"operations": 1968,
|
||||
"paths": 1427,
|
||||
"operations": 2003,
|
||||
"products": {
|
||||
"admin": 87,
|
||||
"ads": 7,
|
||||
@@ -141,6 +141,7 @@
|
||||
"rerank": 1,
|
||||
"research": 8,
|
||||
"responses": 1,
|
||||
"risk": 35,
|
||||
"router": 40,
|
||||
"run": 1,
|
||||
"runner": 3,
|
||||
|
||||
@@ -61,6 +61,12 @@ const (
|
||||
|
||||
IAMMailable = "iam_mailable"
|
||||
|
||||
// AgentsDeclared answers whether a reference names an agent in the CALLER's
|
||||
// own org. It is on the plane because the registry is the agents PROCESS's
|
||||
// own store: a peer cannot see it, and a peer that trusted the caller's word
|
||||
// instead would be treating a request field as an attestation.
|
||||
AgentsDeclared = "agents_declared"
|
||||
|
||||
GitFiles = "git_files"
|
||||
GitImport = "git_import"
|
||||
GitInbound = "git_inbound"
|
||||
@@ -310,6 +316,28 @@ type Roster struct {
|
||||
Recipients []Recipient `json:"recipients"` // everyone in the org who may be mailed; empty is a real answer, not an error
|
||||
}
|
||||
|
||||
// ---- agents.declared -------------------------------------------------------
|
||||
|
||||
// AgentRef names an agent by the handle its own org addresses it by — the id or
|
||||
// the name, which is the one lookup every path-addressed agents route uses.
|
||||
//
|
||||
// There is no org field, as everywhere on this plane: the org rides the caller,
|
||||
// so a reference can only ever be resolved against the asking org's registry.
|
||||
type AgentRef struct {
|
||||
Ref string `json:"ref"` // the id or name to resolve, as the caller's own org spells it
|
||||
}
|
||||
|
||||
// AgentDeclared is what the registry answers about a reference: whether it names
|
||||
// an agent this org actually registered.
|
||||
//
|
||||
// It is deliberately a BOOLEAN and not the agent record. The question is "did
|
||||
// this org declare this agent", and answering it with the model, instructions
|
||||
// and tool list would put an org's whole agent design on the wire to settle a
|
||||
// yes/no — and would make the answer's size a side channel on the record.
|
||||
type AgentDeclared struct {
|
||||
Declared bool `json:"declared"` // true when the reference resolves in the caller's own registry
|
||||
}
|
||||
|
||||
// ---- git -------------------------------------------------------------------
|
||||
|
||||
// ImportIn asks git to create a repo and mirror an upstream into it. It exists
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/hanzoai/cloud"
|
||||
"github.com/hanzoai/cloud/apps/risk"
|
||||
)
|
||||
|
||||
// Standalone entry for the risk app.
|
||||
//
|
||||
// This is the app's OWN composition root: it links only its own subsystem and
|
||||
// the cloud request tier, never the whole fleet, so the build is this one app
|
||||
// and not the union the fused binary was. The light host loads it as a plugin;
|
||||
// run directly it serves standalone. Its OpenAPI subset comes from
|
||||
// `risk openapi`.
|
||||
//
|
||||
// Price is cloud.Metered and not a flat edge price. Every priced act here bills
|
||||
// from the ResourceMeter on the unit it actually produced — one screen per
|
||||
// decision, one run per search — and a flat per-request edge price would charge
|
||||
// a second time for the same work.
|
||||
//
|
||||
// Shutdown is not optional for this app. The model is in-memory mutable state
|
||||
// and cloud deploys strategy Recreate at one replica, so a rollout that did not
|
||||
// snapshot would return every tenant to warming — and a warming model refuses to
|
||||
// score, which reads as "clean" to anything that does not check the refusal.
|
||||
func main() {
|
||||
if err := cloud.Listen([]cloud.Plugin{{
|
||||
Name: "risk",
|
||||
Price: cloud.Metered,
|
||||
Mount: risk.Mount,
|
||||
Shutdown: risk.Shutdown,
|
||||
OwnsHealth: true,
|
||||
}}, []string{"risk"}); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,902 @@
|
||||
[
|
||||
{
|
||||
"description": "Activity is the watcher: what is firing right now, what is being muted, and\nwhich way the decisions are going.\n\nIt reads the DECISION LOG rather than a second bus. The log is already the\nrecord and the analytics copies already reach the platform stream; a dedicated\nactivation bus would be a third representation of one fact and a third thing\nthat can be behind.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"limit": {
|
||||
"description": "Limit bounds how many recent decisions are summarised, 1..500.",
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskActivity"
|
||||
},
|
||||
{
|
||||
"description": "Adds values to a list. Values are folded to lower case so a\nvalue added in one case matches a signal sent in another.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"name": {
|
||||
"description": "Name is the list, from the path.",
|
||||
"type": "string"
|
||||
},
|
||||
"values": {
|
||||
"description": "Values are the values to add. They are folded to lower case, so a value\nadded in one case matches a signal in another.",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskAddListEntries"
|
||||
},
|
||||
{
|
||||
"description": "Stops a search this tenant started.\n\nAn expensive op that cannot be stopped is an expensive op a caller cannot take\nback — and the run holds a worker every other tenant is queued behind. It is\nidempotent: cancelling a search that has already answered is not an error, and\nthe reply says which happened.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"description": "ID is the record to act on, taken from the path.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskCancelSearch"
|
||||
},
|
||||
{
|
||||
"description": "Controls lists the platform controls declared on this tenant's subjects:\nreserves, payout holds and blocks.\n\nThey are DECLARATIONS the money plane reads. Risk never moves money —\nhanzoai/commerce owns payouts, disputes and balances — so a marketplace reads\nthis before a payout and calls decide at authorization, and risk touches no\nprocessor. That is what \"processor-agnostic\" is: a structural property, not a\nfeature.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"kind": {
|
||||
"description": "Kind narrows to one subject kind.",
|
||||
"type": "string"
|
||||
},
|
||||
"subject": {
|
||||
"description": "Subject narrows to one subject.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskControls"
|
||||
},
|
||||
{
|
||||
"description": "Creates an allow or deny list a rule can name in an inlist term.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"kind": {
|
||||
"description": "Kind is whether membership allows or denies.",
|
||||
"type": "string"
|
||||
},
|
||||
"name": {
|
||||
"description": "Name is the list's identifier.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskCreateList"
|
||||
},
|
||||
{
|
||||
"description": "Admits a new detection. Admission is strict on purpose: a rule with\nno terms holds on everything, a rule naming a field that does not exist holds\non nothing, and both read as a working control from the outside.",
|
||||
"inputSchema": {
|
||||
"$defs": {
|
||||
"riskRule": {
|
||||
"properties": {
|
||||
"action": {
|
||||
"description": "Action is what the rule asks for when it holds.",
|
||||
"type": "string"
|
||||
},
|
||||
"all": {
|
||||
"description": "All is the conjunction. An empty conjunction is refused: a rule that holds\non everything is not a detection.",
|
||||
"items": {
|
||||
"$ref": "#/$defs/riskTerm"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
"enabled": {
|
||||
"description": "Enabled governs the live path. A disabled rule still replays, because the\nquestion a simulation asks is what happens ON activation.",
|
||||
"type": "boolean"
|
||||
},
|
||||
"id": {
|
||||
"description": "ID is the rule's identifier within this tenant.",
|
||||
"type": "string"
|
||||
},
|
||||
"name": {
|
||||
"description": "Name is what a reviewer reads in an alert.",
|
||||
"type": "string"
|
||||
},
|
||||
"severity": {
|
||||
"description": "Severity is the reviewer-facing grading: low, medium, high or critical.",
|
||||
"type": "string"
|
||||
},
|
||||
"stage": {
|
||||
"description": "Stage narrows the rule to one lifecycle moment; empty means every stage.",
|
||||
"type": "string"
|
||||
},
|
||||
"weight": {
|
||||
"description": "Weight is how much a hit contributes, in [0,1].",
|
||||
"type": "number"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"riskTerm": {
|
||||
"properties": {
|
||||
"field": {
|
||||
"description": "Field is the fact to read: one of the closed vocabulary, or signal.\u003cname\u003e,\nor velocity.\u003caxis\u003e.\u003cwindow\u003e.\u003cstat\u003e. GET /v1/risk/dictionary lists them.",
|
||||
"type": "string"
|
||||
},
|
||||
"number": {
|
||||
"description": "Number is the numeric operand.",
|
||||
"type": "number"
|
||||
},
|
||||
"op": {
|
||||
"description": "Op is the comparison: eq, ne, gt, gte, lt, lte, in, notin, inlist,\nnotinlist, exists, absent, contains, prefix or suffix.",
|
||||
"type": "string"
|
||||
},
|
||||
"value": {
|
||||
"description": "Value is the string operand.",
|
||||
"type": "string"
|
||||
},
|
||||
"values": {
|
||||
"description": "Values is the set operand, for in and notin.",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"properties": {
|
||||
"id": {
|
||||
"description": "ID is the rule to replace, from the path on a PATCH. Ignored on create,\nwhere the server mints one.",
|
||||
"type": "string"
|
||||
},
|
||||
"rule": {
|
||||
"$ref": "#/$defs/riskRule",
|
||||
"description": "Rule is the detection."
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskCreateRule"
|
||||
},
|
||||
{
|
||||
"description": "Decide scores one entity or event and answers what to do about it: allow,\nchallenge, review, restrict or block, with the evidence behind it.\n\nOne op, not three. \"Score this thing\" is one verb, and a stage-specific op per\nlifecycle moment would be three places for the tenant gate to drift. The stage\nselects the feature window and the rule set and nothing else.\n\nIt is METERED AFTER the work, never gated before it. A pre-work balance gate\nwould have to render its refusal in the fleet's nested error contract, which a\ntyped op cannot do — so the hot path stays typed, the screen is billed on the\ndecision that was actually produced, and no funded client's 402 parsing is\naffected. Overage lands on the caller's OWN ledger: Meter sends the org as\nboth the commerce identity and the org header, which is the anti-cross-org\nproperty.",
|
||||
"inputSchema": {
|
||||
"$defs": {
|
||||
"riskActor": {
|
||||
"properties": {
|
||||
"agent": {
|
||||
"description": "Agent is the agent reference the caller claims to be acting as. It is\nresolved against THIS org's agent registry; an unresolvable reference is\nsimply not a declaration.",
|
||||
"type": "string"
|
||||
},
|
||||
"session": {
|
||||
"description": "Session is the session reference this action belongs to.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"riskAmount": {
|
||||
"properties": {
|
||||
"currency": {
|
||||
"description": "Currency is the ISO 4217 code.",
|
||||
"type": "string"
|
||||
},
|
||||
"direction": {
|
||||
"description": "Direction is whether value came in or went out: \"in\" or \"out\". Without it\nthe model cannot see funds passing through, which is the layering shape.",
|
||||
"type": "string"
|
||||
},
|
||||
"nano": {
|
||||
"description": "Nano is the amount in billionths of one unit of Currency.",
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"riskSubject": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"description": "ID is the subject's identifier within the caller's own org. It is never\ninterpreted, only aggregated on, so it may be any stable string.",
|
||||
"type": "string"
|
||||
},
|
||||
"kind": {
|
||||
"description": "Kind is the sort of thing this is: account, transaction, session, agent,\nmerchant or payout. It selects the aggregation axes, not the tenant gate.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"properties": {
|
||||
"actor": {
|
||||
"$ref": "#/$defs/riskActor",
|
||||
"description": "Actor is who is acting, if the caller knows."
|
||||
},
|
||||
"amount": {
|
||||
"$ref": "#/$defs/riskAmount",
|
||||
"description": "Amount is the money involved, for the stages where there is any."
|
||||
},
|
||||
"idem": {
|
||||
"description": "Idem makes a re-decide idempotent: the same key returns the same decision\nrather than scoring twice and moving the counters twice.",
|
||||
"type": "string"
|
||||
},
|
||||
"signals": {
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Signals are the observations the caller can supply: ip, email,\nemaildomain, device, bin, asn, country, counterparty. Unknown keys are\ncarried and are readable by a rule as signal.\u003ckey\u003e, so a tenant can score\non facts only it has.",
|
||||
"type": "object"
|
||||
},
|
||||
"stage": {
|
||||
"description": "Stage is the lifecycle moment being judged: signup, payment, session,\nusage, payout or dispute. It selects the feature window and the rule set;\nit does NOT select a different tenant gate.",
|
||||
"type": "string"
|
||||
},
|
||||
"subject": {
|
||||
"$ref": "#/$defs/riskSubject",
|
||||
"description": "Subject is what is being judged."
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskDecide"
|
||||
},
|
||||
{
|
||||
"description": "Reads one decision with everything behind it: the evidence, the\nmodel's per-feature attribution and the digest of the model that produced it.\nThat set IS the dispute packet — what was decided, on what, by which model.\n\nA decision another tenant owns answers 404, exactly as an unknown identifier\ndoes, because the tenant's file is the only place looked in: there is no query\nthat could reach another tenant's row, so a probe learns nothing.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"description": "ID is the record to act on, taken from the path.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskDecisionDetail"
|
||||
},
|
||||
{
|
||||
"description": "Decisions lists this tenant's recent decisions, newest first. Every filter is\nan equality on a column the tenant's own file owns, so a filter can narrow the\ncaller's own log and can never widen it.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"action": {
|
||||
"description": "Action narrows to one outcome.",
|
||||
"type": "string"
|
||||
},
|
||||
"kind": {
|
||||
"description": "Kind narrows to one subject kind.",
|
||||
"type": "string"
|
||||
},
|
||||
"limit": {
|
||||
"description": "Limit bounds the page, 1..500, default 100.",
|
||||
"type": "integer"
|
||||
},
|
||||
"stage": {
|
||||
"description": "Stage narrows to one lifecycle moment.",
|
||||
"type": "string"
|
||||
},
|
||||
"subject": {
|
||||
"description": "Subject narrows to one subject.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskDecisions"
|
||||
},
|
||||
{
|
||||
"description": "Retires a detection. Answers 204, or 404 for an identifier this\ntenant does not own.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"description": "ID is the record to act on, taken from the path.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskDeleteRule"
|
||||
},
|
||||
{
|
||||
"description": "Dictionary is the vocabulary a rule may be written in: every fact, every\noperator, every velocity axis and window, this tenant's own lists, and the\nsignal keys this tenant has actually sent.\n\nTwo lenses on one catalogue. The product half is closed and is what admission\nchecks against. The tenant half — the signals — is the tenant's own, read back\noff its recent decisions, which is what makes a rule builder able to offer the\nfields that exist here rather than the fields that exist in general.",
|
||||
"inputSchema": {
|
||||
"properties": {},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskDictionary"
|
||||
},
|
||||
{
|
||||
"description": "Features is the typology-to-feature inventory the model is built on: what each\ndimension measures, the risk indicator it serves, and the published source\nthose words come from.\n\nIt is code rather than a document because a mapping kept beside the model\ncannot drift away from what the model actually reads — and the attributability\nit makes possible is why this is a tree and not a net.",
|
||||
"inputSchema": {
|
||||
"properties": {},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskFeatures"
|
||||
},
|
||||
{
|
||||
"description": "Label records what a human concluded about a decision — fraud, legitimate,\nchargeback, abuse or unknown.\n\nTHE DECIDER IS SERVER-SET. It comes from the validated principal and never\nfrom the request body: the engine's own resolutions carry an unauthenticated\n`by`, and importing that gap into a product that bills on the decision would\nmake the attribution worthless the moment anyone asked who cleared what.\n\nThis is also the only supervision this product has. Everything the model knows\nabout its own miss rate comes from these labels against the below-the-line\nsample.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"description": "ID is the decision, from the path.",
|
||||
"type": "string"
|
||||
},
|
||||
"verdict": {
|
||||
"description": "Verdict is what it turned out to be: fraud, legitimate, chargeback, abuse\nor unknown. It is the only supervision this product has, and it is what\nturns a false-positive rate from a guess into a measurement.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskLabel"
|
||||
},
|
||||
{
|
||||
"description": "Lists shows this tenant's allow and deny lists and how many values each holds.\n\nThese are the TENANT's operational lists. Sanctions and PEP designations are\nnot here and are never merged into them: a tenant may add to its own deny\nlist, and a tenant may not edit OFAC.",
|
||||
"inputSchema": {
|
||||
"properties": {},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskLists"
|
||||
},
|
||||
{
|
||||
"description": "Mode reports whether this tenant's decisions act or only observe.",
|
||||
"inputSchema": {
|
||||
"properties": {},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskMode"
|
||||
},
|
||||
{
|
||||
"description": "ModelState is what a review of the model reads: the appetite it was given, the\nthreshold that appetite produced, and the share it ACTUALLY reached.\n\nStated against realised is the whole governance report. An appetite is a\nmeasured commitment or it is nothing, and a fixed threshold on a drifting\ndistribution silently becomes either silence or a flood.",
|
||||
"inputSchema": {
|
||||
"properties": {},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskModelState"
|
||||
},
|
||||
{
|
||||
"description": "Lifts a control. Answers 204, or 404 for an identifier this\ntenant does not own.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"description": "ID is the record to act on, taken from the path.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskReleaseControl"
|
||||
},
|
||||
{
|
||||
"description": "Removes one value from a list. Answers 204, or 404 when the\nlist does not hold it.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"name": {
|
||||
"description": "Name is the list, from the path.",
|
||||
"type": "string"
|
||||
},
|
||||
"value": {
|
||||
"description": "Value is the value to remove, from the path.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskRemoveListEntry"
|
||||
},
|
||||
{
|
||||
"description": "Restore reinstates this tenant's learned state from its pinned snapshot.\n\nA snapshot whose shape does not match the running inventory is REFUSED, not\ncoerced, and a snapshot belonging to another tenant is refused twice — here,\nby the tenant that asked, and again inside the engine. State the model would\ntreat as its own memory has to have come from this algorithm over this feature\nset for this tenant.",
|
||||
"inputSchema": {
|
||||
"properties": {},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskRestore"
|
||||
},
|
||||
{
|
||||
"description": "Rules lists this tenant's detections. A tenant starts with a starter set\ncovering the lifecycle the product names — signup burst, shared device,\ndisposable email, card testing, denied address, spend spike, payout velocity,\nundeclared automation — all of which it may read, copy, edit and retire.",
|
||||
"inputSchema": {
|
||||
"properties": {},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskRules"
|
||||
},
|
||||
{
|
||||
"description": "Score scores one observation and LEARNS NOTHING from it. It is how a candidate\nis tried against a tenant's real behaviour before anything depends on the\nanswer, and it is the model's analogue of testing a rule: because it records\nnothing, the aggregates it reads do not include the candidate.",
|
||||
"inputSchema": {
|
||||
"$defs": {
|
||||
"riskAmount": {
|
||||
"properties": {
|
||||
"currency": {
|
||||
"description": "Currency is the ISO 4217 code.",
|
||||
"type": "string"
|
||||
},
|
||||
"direction": {
|
||||
"description": "Direction is whether value came in or went out: \"in\" or \"out\". Without it\nthe model cannot see funds passing through, which is the layering shape.",
|
||||
"type": "string"
|
||||
},
|
||||
"nano": {
|
||||
"description": "Nano is the amount in billionths of one unit of Currency.",
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"riskObservation": {
|
||||
"properties": {
|
||||
"amount": {
|
||||
"$ref": "#/$defs/riskAmount",
|
||||
"description": "Amount is the money involved, if any."
|
||||
},
|
||||
"at": {
|
||||
"description": "At is when it happened, RFC 3339. Absent means now.",
|
||||
"type": "string"
|
||||
},
|
||||
"signals": {
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Signals are the observations: ip, device, counterparty and the rest.",
|
||||
"type": "object"
|
||||
},
|
||||
"subject": {
|
||||
"$ref": "#/$defs/riskSubject",
|
||||
"description": "Subject is whose behaviour this is."
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"riskSubject": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"description": "ID is the subject's identifier within the caller's own org. It is never\ninterpreted, only aggregated on, so it may be any stable string.",
|
||||
"type": "string"
|
||||
},
|
||||
"kind": {
|
||||
"description": "Kind is the sort of thing this is: account, transaction, session, agent,\nmerchant or payout. It selects the aggregation axes, not the tenant gate.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"properties": {
|
||||
"observation": {
|
||||
"$ref": "#/$defs/riskObservation",
|
||||
"description": "Observation is what to score."
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskScore"
|
||||
},
|
||||
{
|
||||
"description": "Search exhaustively tries the model topology over this tenant's own history\nand returns the learning curve and the winning shape.\n\nThe grid is CLOSED — trees x depth x window x blend x appetite — because an\nunbounded grid is an unbounded amount of a shared pod's CPU reachable by\nanyone with a key. Each candidate runs in a sandbox over fresh counters\ncarrying the SAME per-tenant memory bound the live plane does, so a search can\nnever move the live model, can never cost more than a tenant is allowed to\ncost, and can never see another tenant's history because the replay reads this\ntenant's own file.\n\nIT IS QUEUED, AND THE QUEUE IS THE BOUND. One run per tenant (a second answers\n409 naming the one already going), a fixed number of workers for the whole\nprocess, a short backlog beyond which a start answers 429, and a deadline every\ncandidate polls. Priced before the work and metered after it, on the caller's\nown ledger. See search.go.\n\nIt answers 202 with a run identifier; read the report back at\nGET /v1/risk/search/{id} and stop it at DELETE /v1/risk/search/{id}.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"limit": {
|
||||
"description": "Limit bounds how many recorded observations are replayed, 1..5000. The\ngrid itself is closed and needs no bound from the caller.",
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskSearch"
|
||||
},
|
||||
{
|
||||
"description": "Reads an exhaustive search's report: every candidate tried, the\nwinner, and the winner's learning curve.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"description": "ID is the record to act on, taken from the path.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskSearchResult"
|
||||
},
|
||||
{
|
||||
"description": "Sets how much of the stream the model may examine.\n\nReview is the lever, and the threshold is derived from it as a quantile of the\nscores actually observed rather than fixed at a number someone liked. That is\nwhat makes an alert level governed rather than tuned, and it is why the level\ncannot be set to fit the size of the review team.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"review": {
|
||||
"description": "Review is the share of the stream that may be examined, in (0, 0.5].",
|
||||
"type": "number"
|
||||
},
|
||||
"sample": {
|
||||
"description": "Sample is the share of non-alerting traffic retained, in [0, 1].",
|
||||
"type": "number"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskSetAppetite"
|
||||
},
|
||||
{
|
||||
"description": "Declares a reserve, a payout hold or a block on a subject.",
|
||||
"inputSchema": {
|
||||
"$defs": {
|
||||
"riskSubject": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"description": "ID is the subject's identifier within the caller's own org. It is never\ninterpreted, only aggregated on, so it may be any stable string.",
|
||||
"type": "string"
|
||||
},
|
||||
"kind": {
|
||||
"description": "Kind is the sort of thing this is: account, transaction, session, agent,\nmerchant or payout. It selects the aggregation axes, not the tenant gate.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"properties": {
|
||||
"control": {
|
||||
"description": "Control is which one: reserve, payout-hold or block.",
|
||||
"type": "string"
|
||||
},
|
||||
"rate": {
|
||||
"description": "Rate is the reserve fraction, in [0,1], for a reserve.",
|
||||
"type": "number"
|
||||
},
|
||||
"reason": {
|
||||
"description": "Reason is why.",
|
||||
"type": "string"
|
||||
},
|
||||
"subject": {
|
||||
"$ref": "#/$defs/riskSubject",
|
||||
"description": "Subject is what the control applies to."
|
||||
},
|
||||
"until": {
|
||||
"description": "Until is when it lapses, RFC 3339. Absent means it does not.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskSetControl"
|
||||
},
|
||||
{
|
||||
"description": "Takes this tenant live, or returns it to shadow.\n\nSHADOW IS THE DEFAULT AND THE DEFAULT IS NOT CONFIGURABLE. In shadow every\nrule runs, the model scores and learns, every decision is recorded, and the\naction is always allow — so what would have happened is readable over real\ntraffic before anyone depends on it. A model that quietly went live and\nstarted declining payments is the worst failure available here.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"mode": {
|
||||
"description": "Mode is \"shadow\" or \"live\". Shadow is the default and the default is not\nconfigurable: a model that quietly went live and started declining\npayments is the worst failure available here, so going live is an act\nsomebody performs and can be asked about.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskSetMode"
|
||||
},
|
||||
{
|
||||
"description": "Simulate replays a candidate rule over this tenant's OWN recorded decisions\nand reports what it would have caught, what it would have given up, and how\nmuch of that a human has already judged.\n\nIt is a gate and not a nicety: a new typology has to be tested before live\nactivation, and a retirement has to be justified against the outgoing rule's\nperformance. Nothing is written — the replay reads the decision log and\nevaluates in memory.\n\nAn EMPTY history is REFUSED rather than reported as zero alerts. \"No alerts\"\nis exactly what a quiet rule looks like, and being unable to tell the two\napart is the failure a sandbox exists to prevent.\n\nBOUNDED AND SYNCHRONOUS, AND DELIBERATELY NOT PRICED. The work is ONE rule\nover at most 5,000 of the tenant's own recorded decisions, evaluated in memory\nwith no model and no writes, and the caller holds its own request open for all\nof it — so the cost is bounded per call and self-throttling across calls, which\nis what the search plane needed a queue to achieve. Not priced because this is\nthe op a tenant runs BEFORE activating a rule, and charging for the rehearsal\nis how you teach people to skip it.",
|
||||
"inputSchema": {
|
||||
"$defs": {
|
||||
"riskRule": {
|
||||
"properties": {
|
||||
"action": {
|
||||
"description": "Action is what the rule asks for when it holds.",
|
||||
"type": "string"
|
||||
},
|
||||
"all": {
|
||||
"description": "All is the conjunction. An empty conjunction is refused: a rule that holds\non everything is not a detection.",
|
||||
"items": {
|
||||
"$ref": "#/$defs/riskTerm"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
"enabled": {
|
||||
"description": "Enabled governs the live path. A disabled rule still replays, because the\nquestion a simulation asks is what happens ON activation.",
|
||||
"type": "boolean"
|
||||
},
|
||||
"id": {
|
||||
"description": "ID is the rule's identifier within this tenant.",
|
||||
"type": "string"
|
||||
},
|
||||
"name": {
|
||||
"description": "Name is what a reviewer reads in an alert.",
|
||||
"type": "string"
|
||||
},
|
||||
"severity": {
|
||||
"description": "Severity is the reviewer-facing grading: low, medium, high or critical.",
|
||||
"type": "string"
|
||||
},
|
||||
"stage": {
|
||||
"description": "Stage narrows the rule to one lifecycle moment; empty means every stage.",
|
||||
"type": "string"
|
||||
},
|
||||
"weight": {
|
||||
"description": "Weight is how much a hit contributes, in [0,1].",
|
||||
"type": "number"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"riskTerm": {
|
||||
"properties": {
|
||||
"field": {
|
||||
"description": "Field is the fact to read: one of the closed vocabulary, or signal.\u003cname\u003e,\nor velocity.\u003caxis\u003e.\u003cwindow\u003e.\u003cstat\u003e. GET /v1/risk/dictionary lists them.",
|
||||
"type": "string"
|
||||
},
|
||||
"number": {
|
||||
"description": "Number is the numeric operand.",
|
||||
"type": "number"
|
||||
},
|
||||
"op": {
|
||||
"description": "Op is the comparison: eq, ne, gt, gte, lt, lte, in, notin, inlist,\nnotinlist, exists, absent, contains, prefix or suffix.",
|
||||
"type": "string"
|
||||
},
|
||||
"value": {
|
||||
"description": "Value is the string operand.",
|
||||
"type": "string"
|
||||
},
|
||||
"values": {
|
||||
"description": "Values is the set operand, for in and notin.",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"properties": {
|
||||
"candidate": {
|
||||
"$ref": "#/$defs/riskRule",
|
||||
"description": "Candidate is the rule to try. It is evaluated as activated whatever its\nenabled flag says: the question is what happens on activation."
|
||||
},
|
||||
"incumbent": {
|
||||
"description": "Incumbent is the rule the candidate would replace, when it replaces one.\nNaming it is what turns a report into a justification for a retirement.",
|
||||
"type": "string"
|
||||
},
|
||||
"limit": {
|
||||
"description": "Limit bounds how many recorded decisions are replayed, 1..5000.",
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskSimulate"
|
||||
},
|
||||
{
|
||||
"description": "Snapshot pins this tenant's learned state into its own encrypted file.\n\nIt is what an auditor asks for: the exact model that raised an alert, kept so\nthat the same input can be scored again a year later. It is also what makes a\nrollout survivable — the shutdown path snapshots every resident tenant,\nbecause cloud deploys Recreate at one replica and a model that comes back with\nnothing learned refuses to score for its whole warm period.",
|
||||
"inputSchema": {
|
||||
"properties": {},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskSnapshot"
|
||||
},
|
||||
{
|
||||
"description": "Reads one subject's current risk state: every live velocity\naggregate it has, its recent decisions, whatever controls are declared on it,\nits thirty-day history, and where this KIND of subject sits across the\nplatform. This is the continuous-monitoring read — a merchant, an account or\nan agent, on one page.\n\nThe last two come from the warehouse and are BEST EFFORT: an unreachable\nwarehouse omits them and names the gap, because a zeroed history would say the\nsubject did nothing and a zeroed baseline would say the platform did.\n\nThe network comparison is the only cross-org value this API returns and it is\naggregate BY CONSTRUCTION: the table has no tenant column, so there is no\nquery — here or anywhere — that could return another tenant's rows.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"description": "ID is the subject identifier, from the path.",
|
||||
"type": "string"
|
||||
},
|
||||
"kind": {
|
||||
"description": "Kind is the subject kind, from the path.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskSubjectState"
|
||||
},
|
||||
{
|
||||
"description": "Suppress mutes a rule's activations for a subject, a subject kind, or\neverywhere that rule fires.\n\nA suppressed hit is RECORDED with suppressed set and contributes nothing to\nthe action. It is never dropped, for the same reason the model counts its\nrefusals: silence must never read as a clean result, and a compliance record\nthat can be silently muted by an operational knob is not a record.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"kind": {
|
||||
"description": "Kind narrows the suppression to one subject kind.",
|
||||
"type": "string"
|
||||
},
|
||||
"reason": {
|
||||
"description": "Reason is why. It is required, because a mute nobody can explain later is\na control that quietly stopped.",
|
||||
"type": "string"
|
||||
},
|
||||
"rule": {
|
||||
"description": "Rule is the rule to mute. At least one of Rule, Kind or Subject must be\ngiven — a suppression that names nothing would mute everything.",
|
||||
"type": "string"
|
||||
},
|
||||
"subject": {
|
||||
"description": "Subject narrows it to one subject.",
|
||||
"type": "string"
|
||||
},
|
||||
"until": {
|
||||
"description": "Until is when it expires, RFC 3339. Absent means it does not, which is a\nchoice somebody should be able to be asked about.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskSuppress"
|
||||
},
|
||||
{
|
||||
"description": "Suppressions lists the active mutes. A suppression that has expired is not\nlisted and mutes nothing — a forgotten mute stops muting instead of quietly\nstaying on forever.",
|
||||
"inputSchema": {
|
||||
"properties": {},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskSuppressions"
|
||||
},
|
||||
{
|
||||
"description": "Train teaches this tenant's model from this tenant's own observations.\n\nIt is online: there is no job, no queue and no window in which the tenant is\nprotected by a stale model, because the half-space geometry is built before\nany data arrives and the model IS a set of mass counters. One observation is\none increment.\n\nTHERE IS NO CROSS-TENANT TRAINING INPUT AND NO WAY TO EXPRESS ONE. The model\nis indexed by the tenant key and its tree GEOMETRY is seeded from it, so two\ntenants do not merely hold different counters — they hold different trees.\nAny cross-org learning in this product is aggregate-only, lives in a table\nwith no tenant column, and is published only above a k-anonymity floor.",
|
||||
"inputSchema": {
|
||||
"$defs": {
|
||||
"riskAmount": {
|
||||
"properties": {
|
||||
"currency": {
|
||||
"description": "Currency is the ISO 4217 code.",
|
||||
"type": "string"
|
||||
},
|
||||
"direction": {
|
||||
"description": "Direction is whether value came in or went out: \"in\" or \"out\". Without it\nthe model cannot see funds passing through, which is the layering shape.",
|
||||
"type": "string"
|
||||
},
|
||||
"nano": {
|
||||
"description": "Nano is the amount in billionths of one unit of Currency.",
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"riskObservation": {
|
||||
"properties": {
|
||||
"amount": {
|
||||
"$ref": "#/$defs/riskAmount",
|
||||
"description": "Amount is the money involved, if any."
|
||||
},
|
||||
"at": {
|
||||
"description": "At is when it happened, RFC 3339. Absent means now.",
|
||||
"type": "string"
|
||||
},
|
||||
"signals": {
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Signals are the observations: ip, device, counterparty and the rest.",
|
||||
"type": "object"
|
||||
},
|
||||
"subject": {
|
||||
"$ref": "#/$defs/riskSubject",
|
||||
"description": "Subject is whose behaviour this is."
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"riskSubject": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"description": "ID is the subject's identifier within the caller's own org. It is never\ninterpreted, only aggregated on, so it may be any stable string.",
|
||||
"type": "string"
|
||||
},
|
||||
"kind": {
|
||||
"description": "Kind is the sort of thing this is: account, transaction, session, agent,\nmerchant or payout. It selects the aggregation axes, not the tenant gate.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"properties": {
|
||||
"observations": {
|
||||
"description": "Observations are what to learn from. They are this tenant's own data and\nnothing else: there is no cross-tenant training input and no way to\nexpress one, because the model is indexed by the tenant key and its tree\nGEOMETRY is seeded from it.",
|
||||
"items": {
|
||||
"$ref": "#/$defs/riskObservation"
|
||||
},
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskTrain"
|
||||
},
|
||||
{
|
||||
"description": "Unsuppress lifts a mute. Answers 204, or 404 for an identifier this tenant\ndoes not own.",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"description": "ID is the record to act on, taken from the path.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskUnsuppress"
|
||||
},
|
||||
{
|
||||
"description": "Replaces a detection whole. There is no partial update: a rule is a\nconjunction of terms, and merging a partial term list into an existing one is\na change nobody can read off the request.",
|
||||
"inputSchema": {
|
||||
"$defs": {
|
||||
"riskRule": {
|
||||
"properties": {
|
||||
"action": {
|
||||
"description": "Action is what the rule asks for when it holds.",
|
||||
"type": "string"
|
||||
},
|
||||
"all": {
|
||||
"description": "All is the conjunction. An empty conjunction is refused: a rule that holds\non everything is not a detection.",
|
||||
"items": {
|
||||
"$ref": "#/$defs/riskTerm"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
"enabled": {
|
||||
"description": "Enabled governs the live path. A disabled rule still replays, because the\nquestion a simulation asks is what happens ON activation.",
|
||||
"type": "boolean"
|
||||
},
|
||||
"id": {
|
||||
"description": "ID is the rule's identifier within this tenant.",
|
||||
"type": "string"
|
||||
},
|
||||
"name": {
|
||||
"description": "Name is what a reviewer reads in an alert.",
|
||||
"type": "string"
|
||||
},
|
||||
"severity": {
|
||||
"description": "Severity is the reviewer-facing grading: low, medium, high or critical.",
|
||||
"type": "string"
|
||||
},
|
||||
"stage": {
|
||||
"description": "Stage narrows the rule to one lifecycle moment; empty means every stage.",
|
||||
"type": "string"
|
||||
},
|
||||
"weight": {
|
||||
"description": "Weight is how much a hit contributes, in [0,1].",
|
||||
"type": "number"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"riskTerm": {
|
||||
"properties": {
|
||||
"field": {
|
||||
"description": "Field is the fact to read: one of the closed vocabulary, or signal.\u003cname\u003e,\nor velocity.\u003caxis\u003e.\u003cwindow\u003e.\u003cstat\u003e. GET /v1/risk/dictionary lists them.",
|
||||
"type": "string"
|
||||
},
|
||||
"number": {
|
||||
"description": "Number is the numeric operand.",
|
||||
"type": "number"
|
||||
},
|
||||
"op": {
|
||||
"description": "Op is the comparison: eq, ne, gt, gte, lt, lte, in, notin, inlist,\nnotinlist, exists, absent, contains, prefix or suffix.",
|
||||
"type": "string"
|
||||
},
|
||||
"value": {
|
||||
"description": "Value is the string operand.",
|
||||
"type": "string"
|
||||
},
|
||||
"values": {
|
||||
"description": "Values is the set operand, for in and notin.",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"properties": {
|
||||
"id": {
|
||||
"description": "ID is the rule to replace, from the path on a PATCH. Ignored on create,\nwhere the server mints one.",
|
||||
"type": "string"
|
||||
},
|
||||
"rule": {
|
||||
"$ref": "#/$defs/riskRule",
|
||||
"description": "Rule is the detection."
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"name": "riskUpdateRule"
|
||||
}
|
||||
]
|
||||
File diff suppressed because it is too large
Load Diff
@@ -17,7 +17,7 @@ import (
|
||||
// as a plugin; run directly it serves standalone. Its OpenAPI subset comes from
|
||||
// `tools openapi`. Hand-owned — edit the spec below directly.
|
||||
func main() {
|
||||
if err := cloud.Serve([]cloud.Plugin{{
|
||||
if err := cloud.Listen([]cloud.Plugin{{
|
||||
Name: "tools",
|
||||
Price: cloud.Metered,
|
||||
// The registry views (/v1/skills, /v1/mcp/servers, /v1/plugins) are this
|
||||
|
||||
@@ -347,6 +347,18 @@ var allowedRequestUses = map[string]string{
|
||||
"principal.OrgFrom and never through the request. Off the HTTP path it answers " +
|
||||
"principal.DefaultProject — the whole-org view, which is the honest answer where there is no " +
|
||||
"request rather than a refusal.",
|
||||
"apps/risk/tenant.go": "tenantOf — ONE seam, and the only reader of a request in this app. A risk " +
|
||||
"decision is BILLED and ATTRIBUTED, so beyond the tenant it needs four facts the principal " +
|
||||
"carries and principal.OrgFrom does not: the project SUB-SCOPE and whether it is a validated " +
|
||||
"claim (a project-scoped spend cap may hard-enforce only when it is), the validated user id " +
|
||||
"(the actor on every label, suppression and control this app writes, and the fact that tells a " +
|
||||
"live person from a machine credential when the decision classifies agency), the request id " +
|
||||
"and the client IP (both are meter arguments). Every one of them is resolved HERE, into the " +
|
||||
"scope value every op is handed, so nothing downstream reads a request: agency classification " +
|
||||
"and the actor attribution each take the user as a VALUE. The TENANT itself is " +
|
||||
"principal.OrgFrom, folded through the brand-qualified mint. It fails closed off the HTTP path " +
|
||||
"— a CLI LocalInvoke has no validated principal, and a fraud plane must not act for a tenant " +
|
||||
"nobody attested.",
|
||||
"apps/usage/account.go": "caller / noStore — the usage plane is scoped to (org, SUBJECT): the " +
|
||||
"account board carries the caller's OWN linked provider accounts, so it needs the validated user id " +
|
||||
"(c.User()) as well as the tenant, and principal.OrgFrom carries only the tenant. noStore is the " +
|
||||
|
||||
Reference in New Issue
Block a user