Compare commits
11
Commits
main
...
blue/scoring
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
db7688f4df | ||
|
|
c8bd42d514 | ||
|
|
ec6847f8fa | ||
|
|
69371ec43c | ||
|
|
9adf3601d8 | ||
|
|
2692cbd327 | ||
|
|
68edcebf04 | ||
|
|
dea22a725e | ||
|
|
2009b075ba | ||
|
|
6354816157 | ||
|
|
04aaee19a3 |
@@ -4067,3 +4067,184 @@ SDK and every MCP tool list that answer 404 in production: the same dark hole th
|
||||
`api-hanzo-ai-catalog` router opened under `/v1/models` and `/v1/pricing`, which
|
||||
cost 17 documented-but-uncallable operations. Re-home that surface only after the
|
||||
upstream ships it, and prove the upstream answers before declaring anything.
|
||||
## Hanzo Risk (`apps/risk`) — `/v1/risk` + the native `/v1/ml` leaves
|
||||
|
||||
One app, two prefix families, one core. `/v1/risk` is the DECISION plane for any
|
||||
entity (account, transaction, session, agent, merchant, payout); `/v1/ml` is the
|
||||
MODEL plane it is built on. Fraud is a USE of `/v1/risk` — so is abuse, so are
|
||||
bots, account takeover, spam and pay-as-you-go abuse. **There is no `/v1/fraud`.**
|
||||
|
||||
**The ml leaves are on THIS row, not on `ml`'s, and that is the load-bearing
|
||||
decision.** A manifest row is a BINARY, and the model is in-process MUTABLE state
|
||||
(one half-space forest per tenant, held as mass counters). If `plugin/ml` trained
|
||||
and `plugin/risk` scored, the two processes would hold different counters and
|
||||
there would be NO error — just two different answers to one question. `ml`'s row
|
||||
is unchanged and nothing moves: it never claimed bare `/v1/ml`, so longest-prefix
|
||||
match separates `/v1/ml/models` (ml) from `/v1/ml/score` (risk), exactly as it
|
||||
already separates `storage`'s `/v1/s3/buckets` from `provisioning`'s `/v1/s3`.
|
||||
|
||||
**The engine is a MODULE DEP, and only part of it is linked.** `github.com/luxfi/aml`
|
||||
enters the way `luxfi/kms` and `hanzoai/o11y` do — no source vendored. Only four
|
||||
packages are linked: `types`, `velocity`, `anomaly`, `replay`.
|
||||
|
||||
**MEASURED CORRECTION, worth not re-learning: `pkg/engine` is NOT base-free.**
|
||||
A grep for `hanzoai/base` inside its own files finds nothing, which is how the
|
||||
claim gets made. `go list -deps` finds the truth:
|
||||
|
||||
pkg/engine -> pkg/history -> pkg/store -> github.com/hanzoai/base/core
|
||||
-> github.com/hanzoai/tasks (Temporal SDK)
|
||||
|
||||
Same for `pkg/measure` and `pkg/history`. Linking them would drag an application
|
||||
framework and a workflow engine into a payment-authorization path. So the risk
|
||||
rule plane is a CLOSED ALGEBRA declared cloud-side (`rule.go`): a fixed field
|
||||
vocabulary, a fixed operator set, conjunction only. That is injection-safe by
|
||||
construction, is a TYPED wire shape (so it reaches the SDKs, the CLI and the MCP
|
||||
tools as a schema instead of an opaque expression string), and replays without an
|
||||
interpreter. The upstream fix — split `pkg/history`'s Base-backed store into its
|
||||
own package — is owed and would let `pkg/engine` in later without a rewrite here.
|
||||
|
||||
**The hot path never reads the warehouse.** A payment decide sits inside a
|
||||
processor's authorization window and the datastore is one StatefulSet pod that has
|
||||
taken api.hanzo.ai down before. `POST /v1/risk/decide` reads in-memory velocity
|
||||
rings, the tenant's own SQLite, and the in-process model. ClickHouse
|
||||
(`hanzo.risk_feature`) is the BACKFILL that warms the rings and the read behind
|
||||
the dictionary and the search sandbox. Measure p50/p99 with the warehouse DOWN; a
|
||||
decide that needs analytics up is a payment plane that fails when analytics does.
|
||||
|
||||
**Cross-org learning is AGGREGATE-ONLY and the boundary is three layers deep:**
|
||||
|
||||
type level apps/risk/feature.go is the ONLY file that reads a feature table,
|
||||
and every function there takes a minted `Tenant`. Tenant has no
|
||||
exported constructor and no json tags, so it cannot arrive off the
|
||||
wire; tenantOf(ctx) is the only source and it reads the VALIDATED
|
||||
principal cloud.Bridge parked.
|
||||
value level hanzo.risk_baseline has NO tenant column, no subject, no id and no
|
||||
pseudonym — quantiles only. There is no query that returns one
|
||||
org's rows because the rows do not exist. The populate statement
|
||||
is a package CONSTANT with no placeholder and a HAVING clause
|
||||
carrying the k-anonymity floor (25 orgs, 1000 observations).
|
||||
key level the tenant key is `<brand>/<org>` (tenant.go qualify), minted in
|
||||
ONE place, and the brand half comes from deps.Brand and NEVER a
|
||||
header. It is the store index, the engine's org column AND the
|
||||
seed of the per-tenant tree geometry — so two tenants do not
|
||||
merely hold different counters, they hold different TREES.
|
||||
|
||||
`$public` — the reserved anonymous lane the event door files credential-less
|
||||
writes under — is refused at the mint AND excluded from the baseline statement.
|
||||
|
||||
**Shadow is the default and the default is not configurable.** Two gates in
|
||||
series: the engine store is constructed `Shadow: true`, and a tenant's own record
|
||||
plane holds `mode` (shadow|live, `PUT /v1/risk/mode`). In shadow every rule runs,
|
||||
the model scores and learns, every decision is recorded, and the action is always
|
||||
allow. Statistical evidence is capped at `review` (`modelCeiling`) and that cap is
|
||||
NOT weakened for the payment stage: a model can put a transaction in front of a
|
||||
person; it cannot decline one.
|
||||
|
||||
**Durable first, analytics after.** Every decision lands in the tenant's own
|
||||
encrypted SQLite before `analytics.PublishEvents` emits the copy. `/v1/event` is
|
||||
best-effort by design — it answers `{accepted, dropped}` and the anonymous lane
|
||||
drops on purpose — so a record that rode it would be lost by a bus hiccup,
|
||||
invisibly.
|
||||
|
||||
**Shutdown is not housekeeping.** cloud deploys `strategy: Recreate` at 1 replica,
|
||||
so every rollout drops every in-memory model. `Plugin.Shutdown` snapshots every
|
||||
resident tenant and `tenantState` restores on first touch. Without it every deploy
|
||||
returns every tenant to warming — and a warming model REFUSES to score, which
|
||||
reads as "clean" to anything that does not check `Refusal`.
|
||||
|
||||
**Metering: `decide` meters AFTER, `train` and `search` gate BEFORE.**
|
||||
`cloud.DenyResource` writes the fleet's NESTED `{"error":{...}}` 402 in band and a
|
||||
typed op's error renders FLAT, so a pre-work gate on the hot path would either
|
||||
move the 402 body every balance-aware client parses or force the route untyped.
|
||||
The screen is billed on the decision that was actually produced. `train` and
|
||||
`search` are real CPU and ARE gated — they are new routes, so no client parses a
|
||||
nested body from them and a typed 402 costs nothing.
|
||||
|
||||
**Untyped by design: exactly one route.** `GET /v1/risk/health` answers 503
|
||||
carrying the degraded REPORT as its body. `apps/risk/typed_wire_test.go` holds the
|
||||
closed list, and it also pins the whole served surface as a diffable list —
|
||||
34 operations, 33 MCP tools.
|
||||
|
||||
**Build:** `-tags sqlite_math_functions` under CGO (hanzoai/base's gate).
|
||||
`modernc.org/sqlite` is NOT in the risk graph, so the image's SQLITE-GATE stays
|
||||
green — verify with
|
||||
`CGO_ENABLED=1 go list -tags 'libsqlite3 sqlite_fts5 sqlite_math_functions' -deps ./plugin/risk | grep modernc`.
|
||||
|
||||
**No chart change is needed to serve it.** `api.hanzo.ai/` already routes to
|
||||
cloud, `CLOUD_ENABLE` is unset in `charts/app/values/hanzo/cloud.yaml` (so every
|
||||
manifest app is enabled), and the image builds one binary per `plugin/<app>`.
|
||||
Landing it is an image pin. **`/v1/aml` is deliberately NOT claimed here** — the
|
||||
Ingress still points that path at `service: aml` (the amld pod holding the live
|
||||
5-year retention plane), and claiming it in cloud without deleting that rule in
|
||||
the SAME commit gives a compliance surface answering from a store that is not the
|
||||
record.
|
||||
|
||||
### What a score MEANS, and what a record IS (`apps/risk` + `luxfi/aml` scoring)
|
||||
|
||||
**A calibration cannot be read without naming the coordinates.** `calibrate.Map`
|
||||
has no read method. `Under(shape)` is the only way to obtain a `Reader`, and a
|
||||
`Reader` has no exported constructor — so a function taking one is a function
|
||||
whose caller went through the training–serving skew gate, and the compiler checks
|
||||
it. The gate used to be `P(score, shape)` and every internal call site passed
|
||||
`m.Shape`: the value being compared came from the value being checked, so the
|
||||
control was inert everywhere except the one place a live shape happened to be
|
||||
threaded in. `/v1/ml/evaluate`, `/v1/ml/replay` and the reliability chart all
|
||||
answered under a map the decide path refuses. **A gate a caller can satisfy is
|
||||
not a gate — bind at the boundary, not at the read.**
|
||||
|
||||
**Ties are the only case, not an edge case.** A decision score is
|
||||
`1 - prod(1-weight)` over FIXED per-rule weights, `round4`'d — so the alphabet is
|
||||
a handful of atoms and the modal one is exactly 0. Isotonic regression that pools
|
||||
per SAMPLE rather than per DISTINCT SCORE reports every plateau holding a
|
||||
positive as certainty. `pava` builds one block per distinct score, and a rounding
|
||||
collision between two centroids POOLS rather than picking one: either endpoint
|
||||
alone states a probability the pooled evidence does not support.
|
||||
|
||||
**A decision records the shape it was scored under** (`decision.shape`), a fit
|
||||
reads ONE coordinate system, and the refusal says how much evidence the boundary
|
||||
put out of reach (`superseded`). Without the column a refit re-blesses stale
|
||||
rows — the gate's own documented remedy walking through the gate. The bounded
|
||||
history read takes the NEWEST rows (`DESC` + `LIMIT n+1`) and reports truncation;
|
||||
ascending plus a limit freezes every measurement on the tenant's first 50k
|
||||
decisions, forever, silently.
|
||||
|
||||
**A rule-wide mute is IN the scoring shape.** `combine()` sums only unsuppressed
|
||||
hits, so muting a rule for the whole population moves every score it touched
|
||||
exactly as retiring the rule would. A mute naming a subject is operational data
|
||||
and stays out.
|
||||
|
||||
**The policy ladder cannot decline on the model alone.** The calibrated
|
||||
probability is a pure function of the same score `modelCeiling` already capped,
|
||||
so an uncapped escalation would void the cap by arithmetic. `escalate` caps the
|
||||
policy at what the evidence behind it can justify; a decline needs a rule the
|
||||
organisation wrote.
|
||||
|
||||
**One cell per tenant; nothing is shared.** `bound.go` holds every number that
|
||||
costs memory and the ONLY two constructors — `aggregates()` and `forest()`, both
|
||||
unexported, both always bounded, `forest` pinning `MaxOrgs: 1` so the
|
||||
cross-tenant eviction path is unreachable rather than unlikely. Worst case is
|
||||
arithmetic: 6,048 B/key × an 8 MiB per-tenant cap (~1,386 keys) × 48 tenants
|
||||
≈ 400 MiB. At the ceiling a NEW tenant is REFUSED (503 + error log + degraded
|
||||
probe); admitting it by evicting an incumbent is the same defect wearing an
|
||||
admission badge. Memory returns from a tenant's OWN silence, floored at the
|
||||
longest window so every retired ring held zeros, and at most `retireBatch` (4)
|
||||
snapshots are written per pass because the registry lock is held while they are.
|
||||
|
||||
**Every record ships before it is acknowledged.** `shelf.commit` writes to the
|
||||
tenant's own file and `Sync`s it, fenced at the lease round, as ONE step — so
|
||||
"written" and "durable" cannot come apart at a call site. An UNACKED ship is an
|
||||
error, never a warning: unacked means this pod is not the org's elected writer.
|
||||
The writes easiest to forget are the RETIREMENTS (rule deleted, mute lifted, hold
|
||||
released, mode returned to shadow) — an unshipped delete brings the old state
|
||||
back at the next rollout, firing or blocking, after a 204 said it was gone.
|
||||
`TestEveryMutatingRouteIsCoveredByTheDurabilityTable` reads
|
||||
`plugin/risk/openapi.json` and fails on any mutating route the table does not
|
||||
exercise, so this stops being a rule somebody has to remember.
|
||||
|
||||
**`GET /v1/<app>/health` is unauthenticated by design across this fleet** — the
|
||||
billing gate, the tracing filter and the identity middleware each exempt it by
|
||||
suffix. So the capacity report carries a strained COUNT and never a tenant key: a
|
||||
roster there is a customer list served to an anonymous GET, along with which pod
|
||||
holds each. The organisation whose own rings are partial is told on its own
|
||||
scoped state (`GET /v1/ml/state` → `strained`), and no other organisation is told
|
||||
anything.
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
# Generated by plugin/gen-app-cmds. DO NOT EDIT.
|
||||
#
|
||||
# The build contract is mk/plugin.mk — one file carrying every target an app
|
||||
# needs: build, test, vet, openapi, clean. This names the app(s) this package
|
||||
# backs and includes it. Written from the same apps.Wire() parse that writes
|
||||
# plugin/<app>/main.go, so an app cannot have a main and no Makefile.
|
||||
APPS := risk
|
||||
include ../../mk/plugin.mk
|
||||
@@ -0,0 +1,219 @@
|
||||
package risk
|
||||
|
||||
// bound.go is where every number that costs memory is decided, ONCE, and where
|
||||
// the only constructors for the two in-memory planes live.
|
||||
//
|
||||
// THE DEFECT THIS FILE EXISTS TO MAKE UNREPRESENTABLE. One process-wide
|
||||
// velocity.Store with a GLOBAL key cap, shared by every tenant, is not a
|
||||
// performance choice. velocity.Store evicts the least-recently-updated key in a
|
||||
// SHARD, and a shard is chosen by hashing the whole key — so tenants are mixed
|
||||
// together in it and one org filling the store DELETES ANOTHER ORG'S counters.
|
||||
// The victim's rules then read zero, fire nothing, and report success. A
|
||||
// silently disarmed control is worse than an absent one, because nobody goes
|
||||
// looking for it. anomaly.Store has the same shape one level up: a map of
|
||||
// tenants with a global LRU, where a new tenant's arrival drops another's
|
||||
// learned model and returns it to warming — which declines to score, and reads
|
||||
// on a screen exactly like a clean customer.
|
||||
//
|
||||
// The shape that cannot express it: state is PER TENANT with a PER-TENANT bound.
|
||||
// aggregates() and forest() are the ONLY constructors in this package, they are
|
||||
// unexported, and they always apply the bound (pinned by
|
||||
// TestOnlyTheBoundedConstructorsBuildThePlanes). A tenant that reaches its own
|
||||
// bound evicts its OWN oldest key and nobody else's, and the probe says so.
|
||||
//
|
||||
// THE WORST CASE IS ARITHMETIC, NOT A HOPE. Every quantity below is a constant
|
||||
// or an environment override, and the ceiling is their product:
|
||||
//
|
||||
// per key bucketBytes * buckets(windows()) + keyOverhead = 6,048 B
|
||||
// per tenant velBytes (a CAP, allocated per key on demand) = 8 MiB → 1,386 keys
|
||||
// + one half-space forest, allocated on arming ≈ 336 KiB
|
||||
// process tenantMax tenants holding all of it = 48
|
||||
// ≈ 400 MiB
|
||||
//
|
||||
// A tenant beyond tenantMax is REFUSED ADMISSION, loudly (503 + an error log +
|
||||
// a degraded probe) — never admitted by evicting an incumbent, which is the
|
||||
// defect wearing a different hat. Reclaim is driven by a tenant's OWN idleness
|
||||
// and by nothing else, so no tenant's activity can ever cost another tenant a
|
||||
// ring.
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/aml/pkg/anomaly"
|
||||
"github.com/luxfi/aml/pkg/velocity"
|
||||
)
|
||||
|
||||
// windows is the ONE declaration of the sliding aggregates this plane keeps.
|
||||
//
|
||||
// The NAMES and SPANS are the engine's standard set, because they are the rule
|
||||
// vocabulary: `velocity.<axis>.<window>.<stat>` is what a tenant's rules are
|
||||
// written against, the dictionary publishes them, and anomaly's inventory reads
|
||||
// them by name at construction.
|
||||
//
|
||||
// The RESOLUTIONS are this plane's own and they are lower than the engine's
|
||||
// defaults on purpose. Those were chosen for statutory structuring detection
|
||||
// over a compliance feed — 444 buckets per key, 22.7 KiB — which puts one
|
||||
// tenant's plausible entity cardinality into gigabytes. Here the same four
|
||||
// windows cost 94 buckets and 6 KiB per key, and the quantisation is stated
|
||||
// rather than assumed: a window of W with B buckets resolves to W/B, so the 1h
|
||||
// window is exact to five minutes and the 30d window to a day. Every rule in the
|
||||
// starter set compares a COUNT or a SUM over a whole window and none of them can
|
||||
// distinguish a boundary that fuzzy — precision that outruns the meaning of the
|
||||
// number it measures is not precision, it is four times the memory.
|
||||
func windows() []velocity.Window {
|
||||
return []velocity.Window{
|
||||
{Name: "1h", Span: time.Hour, Buckets: 12},
|
||||
{Name: "24h", Span: 24 * time.Hour, Buckets: 24},
|
||||
{Name: "7d", Span: 7 * 24 * time.Hour, Buckets: 28},
|
||||
{Name: "30d", Span: 30 * 24 * time.Hour, Buckets: 30},
|
||||
}
|
||||
}
|
||||
|
||||
// longestWindow is the span past which every ring of an untouched key reads
|
||||
// zero. A tenant silent for longer holds no information in memory, which is what
|
||||
// makes retiring it free rather than a judgement call.
|
||||
func longestWindow() time.Duration {
|
||||
var d time.Duration
|
||||
for _, w := range windows() {
|
||||
if w.Span > d {
|
||||
d = w.Span
|
||||
}
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// bucketBytes is sizeof(velocity's ring bucket) on a 64-bit build: an int64
|
||||
// index, two ints, two float64s and a uint64 day mask. Stated here rather than
|
||||
// measured at run time, because a bound computed from the thing it bounds is a
|
||||
// bound that moves.
|
||||
const bucketBytes = 48
|
||||
|
||||
// keyOverhead is everything a key costs BESIDES its buckets: four ring headers,
|
||||
// the slice holding them, the entry, the map slot and the composite id string,
|
||||
// plus allocator rounding. Deliberately generous — a bound that under-states is
|
||||
// not a bound.
|
||||
const keyOverhead = 1536
|
||||
|
||||
// bytesPerKey is what one aggregated entity costs, derived from the windows
|
||||
// actually configured so a change to windows() moves the key count and can never
|
||||
// leave a stale constant behind.
|
||||
func bytesPerKey() int {
|
||||
n := 0
|
||||
for _, w := range windows() {
|
||||
n += w.Buckets
|
||||
}
|
||||
return n*bucketBytes + keyOverhead
|
||||
}
|
||||
|
||||
// The knobs. Each is an environment override over a documented default, because
|
||||
// the right ceiling is a property of the pod's memory limit and not of this
|
||||
// source file — and because an operator who has to patch a constant to survive a
|
||||
// capacity event will instead patch it to infinity.
|
||||
const (
|
||||
envVelBytes = "RISK_VELOCITY_BYTES" // per-tenant aggregate budget, bytes
|
||||
envTenantMax = "RISK_TENANTS" // tenants this process arms at once
|
||||
envIdle = "RISK_RECLAIM_IDLE" // how long a tenant must be silent before it is retired, seconds
|
||||
)
|
||||
|
||||
// velBytes is the per-tenant aggregate budget: the CAP on what one tenant's
|
||||
// rings may grow to, not an allocation. 8 MiB is ~1,386 entities at this plane's
|
||||
// resolution — enough that a tenant's active set fits, small enough that
|
||||
// tenantMax of them is well inside a pod.
|
||||
func velBytes() int { return envInt(envVelBytes, 8<<20, 1<<20, 1<<30) }
|
||||
|
||||
// tenantMax is how many tenants this process ARMS at once — each with its own
|
||||
// aggregates and its own forest. It is the process's memory bound and the only
|
||||
// one.
|
||||
//
|
||||
// A pod serving more tenants than this is a SHARDING answer, not a bigger-number
|
||||
// answer: the shard router already pins an org to one pod, so capacity scales by
|
||||
// adding writers. Raising the knob past what the pod's memory limit supports
|
||||
// trades a loud refusal for an OOM kill, which takes every tenant down.
|
||||
func tenantMax() int { return envInt(envTenantMax, 48, 1, 4096) }
|
||||
|
||||
// idleReclaim is how long a tenant must send NOTHING before its arms are
|
||||
// retired. The trigger is that tenant's own silence and nothing else — no other
|
||||
// tenant's activity can ever cause it, which is the property that makes this
|
||||
// reclaim and not eviction.
|
||||
//
|
||||
// THE FLOOR IS THE WHOLE ARGUMENT. It is the longest window, so a retired
|
||||
// tenant's rings held nothing but zeros: every bucket of every ring of every key
|
||||
// it owns has rotated past. Retirement is therefore information-free by
|
||||
// construction rather than by hope, and a small tenant that decides once a day
|
||||
// can never be quietly reset to zero counts — which is the failure a shorter
|
||||
// reclaim would introduce while claiming to fix a memory bug.
|
||||
func idleReclaim() time.Duration {
|
||||
floor := longestWindow()
|
||||
d := time.Duration(envInt(envIdle, int((floor+6*time.Hour)/time.Second), 60, int(400*24*time.Hour/time.Second))) * time.Second
|
||||
if d < floor {
|
||||
return floor
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// searchBudget is the longest one exhaustive search may run. The grid is closed
|
||||
// at 243 candidates, so this is a backstop rather than the bound — but a
|
||||
// detached goroutine with no deadline is a goroutine that outlives the reason it
|
||||
// was started.
|
||||
const searchBudget = 10 * time.Minute
|
||||
|
||||
// maxKeys is the per-tenant cardinality bound, computed from the budget rather
|
||||
// than chosen. At least one, so a misconfigured budget degrades to a tiny store
|
||||
// rather than to velocity's own 100,000-key default.
|
||||
func maxKeys() int {
|
||||
n := velBytes() / bytesPerKey()
|
||||
if n < 1 {
|
||||
return 1
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// envInt reads a bounded integer override. Out of range or unparseable is the
|
||||
// default, not a failure: a typo in a capacity knob must not stop a payment
|
||||
// plane from booting, and the value it lands on is the documented one.
|
||||
func envInt(name string, def, lo, hi int) int {
|
||||
v, err := strconv.Atoi(os.Getenv(name))
|
||||
if err != nil || v < lo || v > hi {
|
||||
return def
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// aggregates builds ONE TENANT's sliding aggregates.
|
||||
//
|
||||
// It is the only call to velocity.New in this package and it is unexported, so
|
||||
// `velocity.New` with a zero Config — a 100,000-key store shared by everyone —
|
||||
// is not something a later edit can reach for by accident. The bound it carries
|
||||
// is per tenant BECAUSE THE STORE IS per tenant: an eviction inside it can only
|
||||
// ever drop a key this same tenant put there.
|
||||
func aggregates() *velocity.Store {
|
||||
return velocity.New(velocity.Config{Windows: windows(), MaxKeys: maxKeys()})
|
||||
}
|
||||
|
||||
// forest builds ONE TENANT's half-space forest over that tenant's own
|
||||
// aggregates. It is the only call to anomaly.New in this package and it
|
||||
// OVERRIDES two fields of whatever config it is handed, which is why the live
|
||||
// plane and the search sandbox can share it.
|
||||
//
|
||||
// MaxOrgs is forced to 1 and that is the point. anomaly.Store holds a map of
|
||||
// tenants under a GLOBAL LRU, so a shared store lets one tenant's arrival delete
|
||||
// another's learned model — and an evicted model returns to warming, which
|
||||
// declines to score and reads as clean. One tenant per store makes that map hold
|
||||
// exactly one key, so the eviction path is unreachable rather than merely
|
||||
// unlikely.
|
||||
//
|
||||
// Shadow is forced on: it is the deployment-wide default, and the tenant's own
|
||||
// live switch in its record plane is the second gate. Two in series, so a flag
|
||||
// flipped by mistake still cannot make an organisation act, and a sandbox can
|
||||
// never alert for real.
|
||||
func forest(cfg anomaly.Config, vel *velocity.Store) (*anomaly.Store, error) {
|
||||
cfg.MaxOrgs, cfg.Shadow = 1, true
|
||||
m, err := anomaly.New(cfg, vel)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("risk: %w", err)
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
@@ -0,0 +1,398 @@
|
||||
package risk
|
||||
|
||||
// bound_test.go pins the property bound.go exists for: a tenant may only ever
|
||||
// degrade ITSELF.
|
||||
//
|
||||
// Each test here was written by reintroducing the defect and checking that this
|
||||
// test goes red. The defect is one line — `velocity.New(velocity.Config{})` on a
|
||||
// process-wide field — so the guard that survives a revert is the SOURCE test at
|
||||
// the bottom: it reads this package's own files and fails the moment a second
|
||||
// constructor appears, whichever file it appears in.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/aml/pkg/velocity"
|
||||
)
|
||||
|
||||
// TestASharedStoreEvictsAcrossTenants is the DEFECT, demonstrated on the library
|
||||
// value this package used to hold one of.
|
||||
//
|
||||
// It is not a test of this package — it is the reason the rest of this file
|
||||
// exists, and it is written first so the property below cannot be read as
|
||||
// defending against something hypothetical. velocity.Store shards by hashing the
|
||||
// WHOLE key, so two tenants land in the same shard, and the overflow eviction
|
||||
// takes the least-recently-updated key IN THAT SHARD — whoever owns it.
|
||||
func TestASharedStoreEvictsAcrossTenants(t *testing.T) {
|
||||
shared := velocity.New(velocity.Config{Windows: windows(), MaxKeys: 64})
|
||||
now := time.Now()
|
||||
victim := velocity.Key{OrgID: "hanzo/victim", Kind: "ip", Value: "203.0.113.5"}
|
||||
shared.Record(victim, now, 10, 10_000)
|
||||
if count(shared, victim) != 1 {
|
||||
t.Fatalf("the victim's own record did not land")
|
||||
}
|
||||
for i := range 4_000 {
|
||||
shared.Record(velocity.Key{OrgID: "hanzo/flooder", Kind: "ip", Value: fmt.Sprintf("198.51.100.%d", i)}, now, 1, 10_000)
|
||||
}
|
||||
if got := count(shared, victim); got != 0 {
|
||||
t.Skipf("the shared store still holds the victim's key (%v) — the eviction is probabilistic in the shard, "+
|
||||
"but the shape that allows it is what this file rules out", got)
|
||||
}
|
||||
t.Log("a shared store with a global cap silently deleted another tenant's counter: this is the shape bound.go forbids")
|
||||
}
|
||||
|
||||
// TestOneTenantsFloodCannotQuietAnother is the property.
|
||||
//
|
||||
// The flooder blows through its OWN cardinality bound many times over. The
|
||||
// victim's ring — one key, recorded before the flood and never touched again —
|
||||
// must still read exactly what the victim put in it. Under the shared store
|
||||
// above it reads zero, the victim's `velocity.ip.1h.count >= 5` rule stops
|
||||
// firing, and nothing anywhere says so.
|
||||
func TestOneTenantsFloodCannotQuietAnother(t *testing.T) {
|
||||
t.Setenv(envVelBytes, strconv.Itoa(1<<20)) // the smallest budget the knob allows
|
||||
_, s := wireApp(t)
|
||||
|
||||
victim, flooder := Tenant("hanzo/victim"), Tenant("hanzo/flooder")
|
||||
vvel, _, _ := armsOf(t, s, victim)
|
||||
fvel, _, _ := armsOf(t, s, flooder)
|
||||
if vvel == fvel {
|
||||
t.Fatal("two tenants were handed the SAME aggregate store — one org's volume can evict another's counters")
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
key := velocity.Key{OrgID: victim.String(), Kind: "ip", Value: "203.0.113.5"}
|
||||
for range 5 {
|
||||
vvel.Record(key, now, 100, 10_000)
|
||||
}
|
||||
|
||||
for i := range maxKeys() * 4 {
|
||||
fvel.Record(velocity.Key{OrgID: flooder.String(), Kind: "ip", Value: fmt.Sprintf("198.51.100.%d", i)}, now, 1, 10_000)
|
||||
}
|
||||
|
||||
if got := count(vvel, key); got != 5 {
|
||||
t.Fatalf("the victim's 1h count is %d, want 5 — another tenant's traffic quieted this tenant's control", got)
|
||||
}
|
||||
// And the flooder degraded ITSELF, which is the other half of the property:
|
||||
// its own bound held, and it is reported rather than silent.
|
||||
// velocity rounds its cap UP to a whole number of keys per shard, so the true
|
||||
// ceiling is a little above the budget's key count and never a multiple of it.
|
||||
// What has to hold is that the flood is bounded at all: it recorded four times
|
||||
// the budget and kept about one.
|
||||
if got := fvel.Keys(); got >= 2*maxKeys() {
|
||||
t.Fatalf("the flooder holds %d keys against a budget of %d — its own bound did not hold", got, maxKeys())
|
||||
}
|
||||
c, err := s.State.shelf.of(flooder)
|
||||
if err != nil {
|
||||
t.Fatalf("resolve flooder: %v", err)
|
||||
}
|
||||
if !c.strained() {
|
||||
t.Fatal("a tenant at its own cardinality bound does not report itself strained, so a partial ring reads as a complete one")
|
||||
}
|
||||
if vc, err := s.State.shelf.of(victim); err != nil || vc.strained() {
|
||||
t.Fatalf("the victim reports strained (%v) because of another tenant's traffic", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAFullNodeRefusesANewTenantAndKeepsEveryIncumbent pins the admission rule.
|
||||
//
|
||||
// At the ceiling the answer is 503 and an incumbent keeps everything: its rings,
|
||||
// its model and its file. Admitting by eviction would be the same defect wearing
|
||||
// an admission badge — the newcomer's arrival would be the reason a tenant that
|
||||
// did nothing lost its counters.
|
||||
func TestAFullNodeRefusesANewTenantAndKeepsEveryIncumbent(t *testing.T) {
|
||||
t.Setenv(envTenantMax, "2")
|
||||
_, s := wireApp(t)
|
||||
|
||||
a, b := Tenant("hanzo/one"), Tenant("hanzo/two")
|
||||
avel, amodel, _ := armsOf(t, s, a)
|
||||
bvel, _, _ := armsOf(t, s, b)
|
||||
|
||||
key := velocity.Key{OrgID: a.String(), Kind: "ip", Value: "203.0.113.9"}
|
||||
avel.Record(key, time.Now(), 10, 10_000)
|
||||
|
||||
if _, err := s.State.shelf.of(Tenant("hanzo/three")); err == nil {
|
||||
t.Fatal("a third tenant was admitted past the ceiling of two")
|
||||
} else if !strings.Contains(err.Error(), "tenant capacity") {
|
||||
t.Fatalf("refusal says %q, want the capacity refusal", err)
|
||||
}
|
||||
|
||||
av2, am2, _ := armsOf(t, s, a)
|
||||
bv2, _, _ := armsOf(t, s, b)
|
||||
if av2 != avel || am2 != amodel || bv2 != bvel {
|
||||
t.Fatal("an incumbent's state was replaced to make room — that is eviction with a different name")
|
||||
}
|
||||
if got := count(avel, key); got != 1 {
|
||||
t.Fatalf("an incumbent's count is %d after another tenant was refused, want 1", got)
|
||||
}
|
||||
if _, refused, _ := s.State.shelf.count(); refused != 1 {
|
||||
t.Fatalf("the refusal was not counted (%d), so the probe cannot report a pod at capacity", refused)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTheProbeReportsAPodAtCapacity: a refusal is an operator's problem and it
|
||||
// has to reach one. A pod that quietly turns tenants away while answering 200 is
|
||||
// a pod nobody investigates.
|
||||
func TestTheProbeReportsAPodAtCapacity(t *testing.T) {
|
||||
t.Setenv(envTenantMax, "1")
|
||||
app, s := wireApp(t)
|
||||
if _, err := s.State.shelf.of(Tenant("hanzo/one")); err != nil {
|
||||
t.Fatalf("first tenant: %v", err)
|
||||
}
|
||||
code, _ := req(t, app, http.MethodGet, "/v1/risk/health", "", "", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("probe = %d before any refusal, want 200", code)
|
||||
}
|
||||
if _, err := s.State.shelf.of(Tenant("hanzo/two")); err == nil {
|
||||
t.Fatal("a second tenant was admitted past the ceiling of one")
|
||||
}
|
||||
code, body := req(t, app, http.MethodGet, "/v1/risk/health", "", "", "")
|
||||
if code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("probe = %d %s after a refusal, want 503 carrying the report", code, body)
|
||||
}
|
||||
if !strings.Contains(string(body), "tenant ceiling") {
|
||||
t.Fatalf("the degraded probe does not name the capacity event: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTheProbeNamesNoTenant.
|
||||
//
|
||||
// GET /v1/<app>/health is unauthenticated BY DESIGN across this fleet — the
|
||||
// billing gate, the tracing filter and the identity middleware each exempt it by
|
||||
// suffix — so everything the probe says, it says to the internet. The capacity
|
||||
// report needs a strained COUNT; it must never carry the KEY of a strained
|
||||
// organisation, because that publishes who is a customer and which pod holds
|
||||
// them, from an anonymous GET.
|
||||
//
|
||||
// The tenant that needs the fact gets it on its own scoped surface.
|
||||
func TestTheProbeNamesNoTenant(t *testing.T) {
|
||||
t.Setenv(envVelBytes, strconv.Itoa(1<<20)) // the smallest budget the knob allows
|
||||
app, s := wireApp(t)
|
||||
tn := Tenant("hanzo/acme")
|
||||
vel, _, _ := armsOf(t, s, tn)
|
||||
now := time.Now()
|
||||
for i := range maxKeys() * 2 {
|
||||
vel.Record(velocity.Key{OrgID: tn.String(), Kind: "ip", Value: fmt.Sprintf("198.51.100.%d", i)}, now, 1, 1_000)
|
||||
}
|
||||
c, err := s.State.shelf.of(tn)
|
||||
if err != nil {
|
||||
t.Fatalf("resolve: %v", err)
|
||||
}
|
||||
if !c.strained() {
|
||||
t.Fatal("the tenant is not at its own bound, so this test proves nothing")
|
||||
}
|
||||
|
||||
code, body := req(t, app, http.MethodGet, "/v1/risk/health", "", "", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("probe = %d %s", code, body)
|
||||
}
|
||||
var report map[string]any
|
||||
if err := json.Unmarshal(body, &report); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n, ok := report["strained"].(float64); !ok || n != 1 {
|
||||
t.Fatalf("the probe does not report the strained COUNT an operator needs: %s", body)
|
||||
}
|
||||
for _, leak := range []string{tn.String(), tn.org(), "acme"} {
|
||||
if strings.Contains(string(body), leak) {
|
||||
t.Errorf("the unauthenticated probe names a tenant (%q): an anonymous GET reads this "+
|
||||
"organisation's key off the customer list — %s", leak, body)
|
||||
}
|
||||
}
|
||||
|
||||
// And the organisation itself IS told, on its own authenticated state.
|
||||
code, body = req(t, app, http.MethodGet, "/v1/ml/state", "acme", "u_acme", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("state = %d %s", code, body)
|
||||
}
|
||||
var st mlModelState
|
||||
if err := json.Unmarshal(body, &st); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !st.Strained {
|
||||
t.Error("the organisation whose own rings are partial is not told so on its own state, " +
|
||||
"so a velocity rule that stopped firing reads as a clean stream")
|
||||
}
|
||||
|
||||
// A DIFFERENT organisation is told nothing about this one.
|
||||
code, body = req(t, app, http.MethodGet, "/v1/ml/state", "beta", "u_beta", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("beta state = %d %s", code, body)
|
||||
}
|
||||
var other mlModelState
|
||||
if err := json.Unmarshal(body, &other); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if other.Strained {
|
||||
t.Error("one organisation's strain is reported on another organisation's state")
|
||||
}
|
||||
}
|
||||
|
||||
// TestARetiredTenantHeldNothingButZeros pins the reclaim's safety argument.
|
||||
//
|
||||
// Reclaim is floored at the longest window, so every ring of a retired tenant
|
||||
// has already rotated to zero and retiring it cannot lose a count. The tenant
|
||||
// comes back with its LEARNED state, because the model is snapshotted to its own
|
||||
// file before the cell is dropped.
|
||||
func TestARetiredTenantHeldNothingButZeros(t *testing.T) {
|
||||
if idleReclaim() < longestWindow() {
|
||||
t.Fatalf("the reclaim threshold %s is under the longest window %s: retiring a tenant would delete live counts",
|
||||
idleReclaim(), longestWindow())
|
||||
}
|
||||
_, s := wireApp(t)
|
||||
tn := Tenant("hanzo/quiet")
|
||||
vel, _, _ := armsOf(t, s, tn)
|
||||
key := velocity.Key{OrgID: tn.String(), Kind: "ip", Value: "203.0.113.7"}
|
||||
vel.Record(key, time.Now(), 10, 10_000)
|
||||
|
||||
// A sweep at the real threshold must NOT touch a tenant that just spoke.
|
||||
s.State.shelf.sweep()
|
||||
if n, _, _ := s.State.shelf.count(); n != 1 {
|
||||
t.Fatalf("a tenant that just decided was retired (%d resident)", n)
|
||||
}
|
||||
// Silent for longer than the longest window: now it goes, and its rings held
|
||||
// nothing.
|
||||
c, err := s.State.shelf.of(tn)
|
||||
if err != nil {
|
||||
t.Fatalf("resolve: %v", err)
|
||||
}
|
||||
c.mu.Lock()
|
||||
c.touched = time.Now().Add(-2 * idleReclaim())
|
||||
c.mu.Unlock()
|
||||
s.State.shelf.sweep()
|
||||
if n, _, _ := s.State.shelf.count(); n != 0 {
|
||||
t.Fatalf("a tenant silent for twice the reclaim threshold is still resident (%d)", n)
|
||||
}
|
||||
// It comes back armed, on its own file, with no trace of another tenant.
|
||||
back, _, _ := armsOf(t, s, tn)
|
||||
if back == vel {
|
||||
t.Fatal("the retired tenant came back with the SAME store, so nothing was released")
|
||||
}
|
||||
}
|
||||
|
||||
// TestOneSweepWritesABoundedNumberOfSnapshots.
|
||||
//
|
||||
// The retire holds the REGISTRY lock while it writes a tenant's model to that
|
||||
// tenant's encrypted file, and every other organisation's cell lookup queues
|
||||
// behind that lock. Unbounded, a sweep that finds the whole shelf idle would
|
||||
// serialise a snapshot per resident tenant under it — a fleet-wide stall
|
||||
// arriving through the housekeeping door, which is the exact shape the rest of
|
||||
// this file rules out on the request path.
|
||||
func TestOneSweepWritesABoundedNumberOfSnapshots(t *testing.T) {
|
||||
_, s := wireApp(t)
|
||||
// More idle tenants than one pass may retire.
|
||||
n := retireBatch + 3
|
||||
for i := range n {
|
||||
tn := Tenant(fmt.Sprintf("hanzo/idle-%d", i))
|
||||
vel, model, _ := armsOf(t, s, tn)
|
||||
vel.Record(velocity.Key{OrgID: tn.String(), Kind: "ip", Value: "203.0.113.1"}, time.Now(), 1, 1_000)
|
||||
// Something learned, so the retire has a snapshot to write rather than a
|
||||
// nil model it skips for free.
|
||||
tx, ent := txOf(tn, observation{
|
||||
id: "o", at: time.Now(), stage: StagePayment, kind: "transaction", subject: "tx",
|
||||
amount: 1_000_000_000, currency: "USD", direction: "in", signals: map[string]string{},
|
||||
})
|
||||
_, _ = model.Assess(tx, ent)
|
||||
c, err := s.State.shelf.of(tn)
|
||||
if err != nil {
|
||||
t.Fatalf("resolve: %v", err)
|
||||
}
|
||||
c.mu.Lock()
|
||||
c.touched = time.Now().Add(-2 * idleReclaim())
|
||||
c.mu.Unlock()
|
||||
}
|
||||
if got, _, _ := s.State.shelf.count(); got != n {
|
||||
t.Fatalf("%d tenants resident, want %d", got, n)
|
||||
}
|
||||
s.State.shelf.sweep()
|
||||
after, _, _ := s.State.shelf.count()
|
||||
if n-after > retireBatch {
|
||||
t.Fatalf("one sweep retired %d tenants, writing that many snapshots under the registry lock; "+
|
||||
"the bound is %d per pass", n-after, retireBatch)
|
||||
}
|
||||
if n-after == 0 {
|
||||
t.Fatal("the sweep retired nothing, so this test proves nothing about the bound")
|
||||
}
|
||||
}
|
||||
|
||||
// TestOnlyTheBoundedConstructorsBuildThePlanes is the guard that survives a
|
||||
// revert.
|
||||
//
|
||||
// It reads this package's own source. velocity.New and anomaly.New may appear in
|
||||
// bound.go and nowhere else — including tests, except the one above that
|
||||
// demonstrates the defect. Reintroduce the process-wide store anywhere and this
|
||||
// fails, whatever it is called and whoever calls it.
|
||||
func TestOnlyTheBoundedConstructorsBuildThePlanes(t *testing.T) {
|
||||
allowed := map[string]bool{"bound.go": true, "bound_test.go": true}
|
||||
files, err := filepath.Glob("*.go")
|
||||
if err != nil {
|
||||
t.Fatalf("glob: %v", err)
|
||||
}
|
||||
if len(files) == 0 {
|
||||
t.Fatal("no source read, so this guard proved nothing")
|
||||
}
|
||||
for _, f := range files {
|
||||
if allowed[filepath.Base(f)] {
|
||||
continue
|
||||
}
|
||||
body, err := os.ReadFile(f)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", f, err)
|
||||
}
|
||||
for _, ctor := range []string{"velocity.New(", "anomaly.New("} {
|
||||
if strings.Contains(string(body), ctor) {
|
||||
t.Errorf("%s calls %s — the two in-memory planes are built ONLY by aggregates() and forest() "+
|
||||
"in bound.go, which is what makes a process-wide store with a global cap unrepresentable", f, ctor)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// count reads a key's 1h count out of a store.
|
||||
func count(s *velocity.Store, k velocity.Key) int {
|
||||
for _, o := range s.Observe(k) {
|
||||
if o.Window == "1h" {
|
||||
return o.Count
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// TestOneSearchPerTenantAndOnlyPerTenant pins the bound on the most expensive op
|
||||
// this plane has.
|
||||
//
|
||||
// An exhaustive search is up to searchBudget of CPU across the whole grid, in a
|
||||
// goroutine detached from the request. Unbounded, one key loops the route and
|
||||
// takes down every product on the pod. Bounded per tenant, a caller that loops it
|
||||
// spends its own slot and nobody else's — which is the same shape the measurement
|
||||
// plane uses, for the same reason.
|
||||
func TestOneSearchPerTenantAndOnlyPerTenant(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
|
||||
release, err := s.State.running.claim(Tenant("hanzo/acme"))
|
||||
if err != nil {
|
||||
t.Fatalf("claim: %v", err)
|
||||
}
|
||||
code, body := req(t, app, http.MethodPost, "/v1/ml/search", "acme", "u_acme", `{"limit":10}`)
|
||||
if code != http.StatusTooManyRequests {
|
||||
t.Fatalf("a second concurrent search = %d %s, want 429 — the run is unbounded", code, body)
|
||||
}
|
||||
// Another tenant is untouched: the bound is per tenant, not a fleet-wide slot
|
||||
// that one org can hold against everybody.
|
||||
code, body = req(t, app, http.MethodPost, "/v1/ml/search", "beta", "u_beta", `{"limit":10}`)
|
||||
if code == http.StatusTooManyRequests {
|
||||
t.Fatalf("another tenant's search = %d %s — one org's run is blocking the fleet", code, body)
|
||||
}
|
||||
release()
|
||||
code, body = req(t, app, http.MethodPost, "/v1/ml/search", "acme", "u_acme", `{"limit":10}`)
|
||||
if code == http.StatusTooManyRequests {
|
||||
t.Fatalf("the slot was not released: %d %s", code, body)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,365 @@
|
||||
package risk
|
||||
|
||||
// decide.go is the hot path: one call, one decision, an answer that says what to
|
||||
// do and why.
|
||||
//
|
||||
// ORDER, and why it is this order:
|
||||
//
|
||||
// 1. RECORD the observation into the in-memory velocity rings. Everything after
|
||||
// reads them, and the numbers quoted in the decision must be the numbers an
|
||||
// investigator sees when they look at the subject.
|
||||
// 2. CLASSIFY agency — agent, human, bot or unknown — from facts only we hold.
|
||||
// 3. SCORE the model. It learns online and its evidence is capped at review.
|
||||
// 4. EVALUATE the rules over the recorded aggregates plus the model's score.
|
||||
// 5. SUPPRESS. A suppressed hit is RECORDED with suppressed=true, never dropped:
|
||||
// silence must never read as a clean result.
|
||||
// 6. COMBINE into one action, and never let statistical evidence exceed the
|
||||
// ceiling.
|
||||
//
|
||||
// NOTHING HERE READS THE WAREHOUSE. The rings are in memory, the rules and lists
|
||||
// are the tenant's own SQLite file, the model is in process. That is deliberate:
|
||||
// the warehouse is one pod and this call sits inside a card processor's
|
||||
// authorization window.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"math"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/aml/pkg/anomaly"
|
||||
"github.com/luxfi/aml/pkg/types"
|
||||
"github.com/luxfi/aml/pkg/velocity"
|
||||
)
|
||||
|
||||
// Agency is what kind of actor this is. It is the differentiator: we run the
|
||||
// agents, so we can tell a declared, credentialed, metered agent from an
|
||||
// anonymous script, and nobody who does not run agents can compute it.
|
||||
const (
|
||||
// AgencyAgent is a declared agent: it named an agent reference, it
|
||||
// authenticated with a principal-bearing credential, and its traffic is
|
||||
// metered on this org's own ledger.
|
||||
AgencyAgent = "agent"
|
||||
// AgencyHuman is a browser session bound to a validated user.
|
||||
AgencyHuman = "human"
|
||||
// AgencyBot is undeclared automation: no agent reference, and either no
|
||||
// credential or a publishable key, which by construction identifies a tenant
|
||||
// and authenticates nobody.
|
||||
AgencyBot = "bot"
|
||||
// AgencyUnknown is credentialed but undeclared. Scored normally — an honest
|
||||
// "we cannot tell" is worth more than a guess in either direction.
|
||||
AgencyUnknown = "unknown"
|
||||
)
|
||||
|
||||
// Refusals. A refusal is COUNTED and NAMED, never silent, because a control that
|
||||
// declined to answer and a control that found nothing are the same bytes on the
|
||||
// wire and opposite facts about the system.
|
||||
const (
|
||||
// RefusalWarming means the model has not learned enough of this tenant's
|
||||
// behaviour for its scores to mean anything. Rules still ran.
|
||||
RefusalWarming = "warming"
|
||||
// RefusalUnidentified means the observation named no subject the aggregates
|
||||
// can be keyed on.
|
||||
RefusalUnidentified = "unidentified"
|
||||
// RefusalShadow means the tenant is in shadow: everything was computed and
|
||||
// recorded, and nothing acts.
|
||||
RefusalShadow = "shadow"
|
||||
)
|
||||
|
||||
// hit is one piece of evidence.
|
||||
type hit struct {
|
||||
// Rule is the identifier of the rule or model that produced this evidence.
|
||||
Rule string `json:"rule"`
|
||||
// Name is the human-readable detection name.
|
||||
Name string `json:"name"`
|
||||
// Action is what this evidence alone asks for.
|
||||
Action string `json:"action"`
|
||||
// Weight is how much this evidence contributes, in [0,1].
|
||||
Weight float64 `json:"weight"`
|
||||
// Severity is the reviewer-facing grading.
|
||||
Severity string `json:"severity,omitempty"`
|
||||
// Suppressed marks evidence a tenant suppression muted. It still contributes
|
||||
// nothing to the action and is still recorded, because a muted control that
|
||||
// leaves no trace is indistinguishable from one that was never running.
|
||||
Suppressed bool `json:"suppressed,omitempty"`
|
||||
}
|
||||
|
||||
// observation is everything the decide path was given plus everything it derived
|
||||
// before scoring. It is the value the rules read and the value the record plane
|
||||
// stores, so what fired and what was stored cannot disagree.
|
||||
type observation struct {
|
||||
id string
|
||||
at time.Time
|
||||
stage string
|
||||
kind string
|
||||
subject string
|
||||
agency string
|
||||
amount int64
|
||||
currency string
|
||||
direction string
|
||||
signals map[string]string
|
||||
agent string
|
||||
session string
|
||||
}
|
||||
|
||||
// outcome is the decision, before it is rendered onto the wire.
|
||||
type outcome struct {
|
||||
id string
|
||||
action string
|
||||
score float64
|
||||
agency string
|
||||
hits []hit
|
||||
causes []types.Cause
|
||||
shadow bool
|
||||
refusal string
|
||||
}
|
||||
|
||||
// axisOf maps a velocity axis to the observation's value on that axis. It is a
|
||||
// closed map because a rule may only name an axis this returns something for,
|
||||
// and admitTerm holds it to the same set.
|
||||
func (o observation) axisOf(axis string) string {
|
||||
switch axis {
|
||||
case "account":
|
||||
return o.subject
|
||||
case "device":
|
||||
return o.signals["device"]
|
||||
case "ip":
|
||||
return o.signals["ip"]
|
||||
case "email":
|
||||
return o.signals["email"]
|
||||
case "bin":
|
||||
return o.signals["bin"]
|
||||
case "pair":
|
||||
if cp := o.signals["counterparty"]; cp != "" {
|
||||
return o.subject + "\x1f" + cp
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// record writes the observation onto every ring it has a value for. The rings
|
||||
// are keyed {OrgID, Kind, Value} with the tenant leading, so two tenants naming
|
||||
// the same device or the same address never share a counter.
|
||||
func record(vel *velocity.Store, t Tenant, o observation) {
|
||||
usd := nanoUSD(o.amount)
|
||||
for axis := range velocityAxes {
|
||||
v := o.axisOf(axis)
|
||||
if v == "" {
|
||||
continue
|
||||
}
|
||||
vel.Record(velocity.Key{OrgID: t.String(), Kind: axis, Value: v}, o.at, usd, structuringThreshold)
|
||||
}
|
||||
}
|
||||
|
||||
// structuringThreshold is the reporting threshold the "just under" counters are
|
||||
// measured against — the standard USD 10,000 figure. It is a constant because a
|
||||
// per-tenant threshold is a per-tenant policy and this is the statutory one.
|
||||
const structuringThreshold = 10_000
|
||||
|
||||
// nanoUSD converts the wire's integer nano-units to the float the aggregates and
|
||||
// the model read. Money is carried as an integer on the wire so no rounding
|
||||
// happens between the caller and the ledger; the aggregate is a statistic and a
|
||||
// float is the right shape for it.
|
||||
func nanoUSD(nano int64) float64 { return float64(nano) / 1e9 }
|
||||
|
||||
// observe builds the fact set the rules read. Every velocity number the closed
|
||||
// vocabulary can name is materialised here, once, so a rule set of any size
|
||||
// reads the rings a bounded number of times.
|
||||
func observe(vel *velocity.Store, t Tenant, o observation, modelScore float64, warming bool) factSet {
|
||||
f := factSet{
|
||||
scalar: map[string]string{
|
||||
"stage": o.stage,
|
||||
"subject.kind": o.kind,
|
||||
"subject.id": o.subject,
|
||||
"agency": o.agency,
|
||||
"amount.currency": o.currency,
|
||||
"amount.direction": o.direction,
|
||||
"actor.agent": o.agent,
|
||||
"actor.session": o.session,
|
||||
"model.warming": strconv.FormatBool(warming),
|
||||
},
|
||||
number: map[string]float64{
|
||||
"amount.nano": float64(o.amount),
|
||||
"model.score": modelScore,
|
||||
},
|
||||
}
|
||||
for k, v := range o.signals {
|
||||
f.scalar["signal."+strings.ToLower(k)] = v
|
||||
}
|
||||
for axis := range velocityAxes {
|
||||
v := o.axisOf(axis)
|
||||
if v == "" {
|
||||
continue
|
||||
}
|
||||
for _, obs := range vel.Observe(velocity.Key{OrgID: t.String(), Kind: axis, Value: v}) {
|
||||
p := "velocity." + axis + "." + obs.Window + "."
|
||||
f.number[p+"count"] = float64(obs.Count)
|
||||
f.number[p+"sum"] = obs.Sum
|
||||
f.number[p+"near"] = float64(obs.Near)
|
||||
f.number[p+"days"] = float64(obs.Days)
|
||||
}
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
// classify derives agency from four facts cloud already holds, and from NO
|
||||
// user-agent string. A user agent is a caller-supplied claim, so sniffing it
|
||||
// classifies whoever is honest and misses whoever is not.
|
||||
//
|
||||
// credential a publishable key resolves an ORG and no principal, so it cannot
|
||||
// be an agent; a secret key or a bearer resolves a principal.
|
||||
// declared does actor.agentRef resolve in THIS org's agent registry?
|
||||
// session is there a live agent session for it?
|
||||
// metered does this account have priced rows on the org's own ledger?
|
||||
//
|
||||
// declared AND credentialed => agent, and the org's agent policy applies.
|
||||
// undeclared AND anonymous => bot, and the anonymous lane's bounds apply.
|
||||
// Anything else is unknown and is scored normally.
|
||||
func classify(credentialed, publishable, declared, humanSession bool) string {
|
||||
switch {
|
||||
case declared && credentialed && !publishable:
|
||||
return AgencyAgent
|
||||
case !declared && (!credentialed || publishable):
|
||||
return AgencyBot
|
||||
case humanSession && !declared:
|
||||
return AgencyHuman
|
||||
default:
|
||||
return AgencyUnknown
|
||||
}
|
||||
}
|
||||
|
||||
// decide is the whole path. It takes the tenant's rules, lists and suppressions
|
||||
// as values so it can be exercised without a store, which is what makes the
|
||||
// ordering above testable rather than assertable.
|
||||
func decide(
|
||||
ctx context.Context,
|
||||
vel *velocity.Store,
|
||||
model *anomaly.Store,
|
||||
t Tenant,
|
||||
o observation,
|
||||
rules []rule,
|
||||
lists func(name, value string) bool,
|
||||
suppressed func(h hit, o observation) bool,
|
||||
shadow bool,
|
||||
) outcome {
|
||||
_ = ctx
|
||||
|
||||
out := outcome{id: o.id, shadow: shadow, agency: o.agency, action: ActionAllow}
|
||||
if strings.TrimSpace(o.subject) == "" {
|
||||
out.refusal = RefusalUnidentified
|
||||
return out
|
||||
}
|
||||
|
||||
// 1. Record first: everything after reads these rings, and the numbers in the
|
||||
// decision must be the ones an investigator sees on the subject.
|
||||
record(vel, t, o)
|
||||
|
||||
// 2. Score the model. Assess LEARNS; the tenant's own traffic is its training
|
||||
// set and there is no separate training pass.
|
||||
tx := types.Transaction{
|
||||
ID: o.id,
|
||||
OrgID: t.String(),
|
||||
UserID: o.subject,
|
||||
AccountID: o.subject,
|
||||
Currency: o.currency,
|
||||
Direction: o.direction,
|
||||
IPAddress: o.signals["ip"],
|
||||
DeviceFingerprint: o.signals["device"],
|
||||
Counterparty: o.signals["counterparty"],
|
||||
Timestamp: o.at,
|
||||
USD: nanoUSD(o.amount),
|
||||
}
|
||||
assessment := model.Inspect(tx, types.Entity{ID: o.subject, OrgID: t.String()})
|
||||
// The attribution comes from the assessment that produced the SCORE THIS
|
||||
// DECISION RECORDS, not from the alert. Two reasons, and both are the whole
|
||||
// point of having an explanation at all.
|
||||
//
|
||||
// Inspect does not learn and Assess does; they run back to back, so the
|
||||
// masses move between them and the alert's causes explain a marginally
|
||||
// different score than the one written down. Reading the recorded
|
||||
// assessment's causes means the reasons and the number they explain are the
|
||||
// same arithmetic.
|
||||
//
|
||||
// And Assess yields a hit only when the model ALERTS, which it never does
|
||||
// while the engine-level shadow is set — so a decision below the line, or any
|
||||
// decision at all in shadow, would carry no explanation whatever. A score with
|
||||
// no reason is exactly what an adverse-action regime does not allow.
|
||||
out.causes = assessment.Causes
|
||||
var modelHit *hit
|
||||
if mh, ok := model.Assess(tx, types.Entity{ID: o.subject, OrgID: t.String()}); ok {
|
||||
action := mh.Rule.Action
|
||||
if actionRank(action) > actionRank(modelCeiling) {
|
||||
action = modelCeiling
|
||||
}
|
||||
modelHit = &hit{
|
||||
Rule: mh.Rule.ID, Name: mh.Rule.Name, Action: action,
|
||||
Weight: mh.Rule.Weight, Severity: mh.Rule.Severity,
|
||||
}
|
||||
}
|
||||
if !assessment.Scored && assessment.Reason != "" {
|
||||
out.refusal = assessment.Reason
|
||||
}
|
||||
|
||||
// 3. Rules, over the recorded aggregates plus the model's score.
|
||||
f := observe(vel, t, o, assessment.Score, !assessment.Scored)
|
||||
f.lists = lists
|
||||
hits := evaluate(rules, o.stage, f)
|
||||
if modelHit != nil {
|
||||
hits = append(hits, *modelHit)
|
||||
}
|
||||
|
||||
// 4. Suppression. A suppressed hit is kept, marked and given zero weight —
|
||||
// it appears in the record and contributes nothing to the action.
|
||||
scoring := make([]hit, 0, len(hits))
|
||||
for i := range hits {
|
||||
if suppressed != nil && suppressed(hits[i], o) {
|
||||
hits[i].Suppressed = true
|
||||
continue
|
||||
}
|
||||
scoring = append(scoring, hits[i])
|
||||
}
|
||||
|
||||
// 5. Combine. Sorting again puts the model's hit in its place among the rest.
|
||||
sort.SliceStable(hits, func(i, j int) bool {
|
||||
if a, b := actionRank(hits[i].Action), actionRank(hits[j].Action); a != b {
|
||||
return a > b
|
||||
}
|
||||
return hits[i].Weight > hits[j].Weight
|
||||
})
|
||||
score, action := combine(scoring)
|
||||
out.hits, out.score = hits, round4(score)
|
||||
|
||||
// 6. Shadow. Everything above ran, everything is recorded, nothing acts.
|
||||
if shadow {
|
||||
out.action = ActionAllow
|
||||
if out.refusal == "" {
|
||||
out.refusal = RefusalShadow
|
||||
}
|
||||
return out
|
||||
}
|
||||
out.action = action
|
||||
return out
|
||||
}
|
||||
|
||||
// round4 trims a score to four places. A score is a judgement, not a
|
||||
// measurement, and rendering seventeen digits of float noise invites a client to
|
||||
// compare two scores that differ in the fifteenth.
|
||||
func round4(f float64) float64 {
|
||||
if math.IsNaN(f) || math.IsInf(f, 0) {
|
||||
return 0
|
||||
}
|
||||
return math.Round(f*1e4) / 1e4
|
||||
}
|
||||
|
||||
// newID mints a decision identifier. Random rather than sequential: a sequential
|
||||
// id over a shared surface is a volume oracle — a tenant can count another
|
||||
// tenant's decisions by watching its own ids skip.
|
||||
func newID(prefix string) string {
|
||||
var b [12]byte
|
||||
_, _ = rand.Read(b[:])
|
||||
return prefix + "_" + hex.EncodeToString(b[:])
|
||||
}
|
||||
@@ -0,0 +1,421 @@
|
||||
package risk
|
||||
|
||||
// decide_test.go pins the decision path's ORDER and its two hard limits: the
|
||||
// model may never act alone, and shadow may never act at all.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/aml/pkg/anomaly"
|
||||
)
|
||||
|
||||
// TestRecordHappensBeforeScoring pins step 1. Everything after reads the rings,
|
||||
// so the numbers quoted in a decision must be the ones an investigator sees when
|
||||
// they look at the subject — including THIS observation.
|
||||
func TestRecordHappensBeforeScoring(t *testing.T) {
|
||||
vel := aggregates()
|
||||
model, err := forest(anomaly.Config{}, vel)
|
||||
if err != nil {
|
||||
t.Fatalf("forest: %v", err)
|
||||
}
|
||||
tn := Tenant("hanzo/acme")
|
||||
|
||||
// A rule that can only hold if THIS observation is already in the ring.
|
||||
rules := []rule{{
|
||||
ID: "r1", Name: "counted", Stage: StagePayment, Action: ActionReview,
|
||||
Weight: 0.5, Enabled: true,
|
||||
All: []term{{Field: "velocity.ip.1h.count", Op: OpGte, Number: 1}},
|
||||
}}
|
||||
o := observation{
|
||||
id: "d1", at: time.Now(), stage: StagePayment, kind: "transaction", subject: "tx1",
|
||||
amount: 1_000_000_000, currency: "USD", direction: "in",
|
||||
signals: map[string]string{"ip": "203.0.113.1"},
|
||||
}
|
||||
out := decide(context.Background(), vel, model, tn, o, rules, nil, nil, false)
|
||||
if len(out.hits) != 1 {
|
||||
t.Fatalf("hits = %v; the rule did not see this observation in the ring, so the record ran after the read", out.hits)
|
||||
}
|
||||
}
|
||||
|
||||
// TestModelEvidenceCannotExceedTheCeiling pins the structural backstop. A
|
||||
// statistical judgement may summon a person; it may not decline a payment,
|
||||
// because an unexplainable refusal is not a decision anybody can defend to a
|
||||
// customer or a chargeback network.
|
||||
func TestModelEvidenceCannotExceedTheCeiling(t *testing.T) {
|
||||
if actionRank(modelCeiling) != actionRank(ActionReview) {
|
||||
t.Fatalf("the model ceiling is %q, not review — statistical evidence can now act alone", modelCeiling)
|
||||
}
|
||||
// And the combine step's ordering must never promote a capped hit.
|
||||
hits := []hit{{Rule: "model", Action: modelCeiling, Weight: 1.0}}
|
||||
_, action := combine(hits)
|
||||
if actionRank(action) > actionRank(ActionReview) {
|
||||
t.Fatalf("a weight-1.0 model hit produced %q — the cap is not holding", action)
|
||||
}
|
||||
}
|
||||
|
||||
// TestShadowActsOnNothing pins the default. In shadow every rule runs, the model
|
||||
// scores and learns, every decision is recorded, and the action is always allow.
|
||||
func TestShadowActsOnNothing(t *testing.T) {
|
||||
vel := aggregates()
|
||||
model, _ := forest(anomaly.Config{}, vel)
|
||||
rules := []rule{{
|
||||
ID: "block-all", Name: "would block", Stage: StagePayment, Action: ActionBlock,
|
||||
Weight: 1, Enabled: true,
|
||||
All: []term{{Field: "subject.kind", Op: OpEq, Value: "transaction"}},
|
||||
}}
|
||||
o := observation{
|
||||
id: "d1", at: time.Now(), stage: StagePayment, kind: "transaction", subject: "tx1",
|
||||
signals: map[string]string{},
|
||||
}
|
||||
out := decide(context.Background(), vel, model, Tenant("hanzo/acme"), o, rules, nil, nil, true)
|
||||
if out.action != ActionAllow {
|
||||
t.Fatalf("shadow produced action %q — a shadow tenant acted", out.action)
|
||||
}
|
||||
// The answer must SAY it was shadow, or silence reads as a clean result. The
|
||||
// refusal keeps the MODEL's more specific reason when it has one (a warming
|
||||
// model is a different fact from a shadow tenant, and hiding the first behind
|
||||
// the second is exactly the conflation this field exists to prevent) — so
|
||||
// `shadow` is the flag that must always hold, and `refusal` must always be
|
||||
// populated with something.
|
||||
if !out.shadow {
|
||||
t.Fatal("a shadow decision does not say it was shadow")
|
||||
}
|
||||
if out.refusal == "" {
|
||||
t.Fatal("a shadow decision names no refusal at all, so it is indistinguishable from a clean live one")
|
||||
}
|
||||
if out.refusal != RefusalWarming && out.refusal != RefusalShadow {
|
||||
t.Fatalf("refusal = %q, want the model's own reason or shadow", out.refusal)
|
||||
}
|
||||
if len(out.hits) != 1 {
|
||||
t.Fatalf("shadow computed %d hits, want 1 — shadow must compute everything and act on nothing", len(out.hits))
|
||||
}
|
||||
// Live, the same input blocks. Without this the test above would pass on a
|
||||
// rule that simply never fires.
|
||||
out = decide(context.Background(), vel, model, Tenant("hanzo/acme"), o, rules, nil, nil, false)
|
||||
if out.action != ActionBlock {
|
||||
t.Fatalf("live produced %q, want block — the shadow assertion proves nothing if the rule cannot fire", out.action)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSuppressedEvidenceIsRecordedNotDropped pins the doctrine: a muted control
|
||||
// that leaves no trace is indistinguishable from one that was never running.
|
||||
func TestSuppressedEvidenceIsRecordedNotDropped(t *testing.T) {
|
||||
vel := aggregates()
|
||||
model, _ := forest(anomaly.Config{}, vel)
|
||||
rules := []rule{{
|
||||
ID: "r1", Name: "noisy", Stage: StageSignup, Action: ActionBlock, Weight: 1, Enabled: true,
|
||||
All: []term{{Field: "subject.kind", Op: OpEq, Value: "account"}},
|
||||
}}
|
||||
o := observation{id: "d1", at: time.Now(), stage: StageSignup, kind: "account", subject: "a1", signals: map[string]string{}}
|
||||
|
||||
out := decide(context.Background(), vel, model, Tenant("hanzo/acme"), o, rules, nil,
|
||||
func(h hit, _ observation) bool { return h.Rule == "r1" }, false)
|
||||
|
||||
if len(out.hits) != 1 {
|
||||
t.Fatalf("the suppressed hit was DROPPED (%d hits) — the record no longer says the rule fired", len(out.hits))
|
||||
}
|
||||
if !out.hits[0].Suppressed {
|
||||
t.Fatal("the hit is recorded but not marked suppressed")
|
||||
}
|
||||
if out.action != ActionAllow {
|
||||
t.Fatalf("a suppressed hit still drove the action to %q", out.action)
|
||||
}
|
||||
if out.score != 0 {
|
||||
t.Fatalf("a suppressed hit contributed %v to the score", out.score)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAgencyIsDerivedNotDeclared pins the differentiator. The classification
|
||||
// turns on facts we hold — the credential class and this org's own agent
|
||||
// registry — and never on a user-agent string, which is a claim.
|
||||
func TestAgencyIsDerivedNotDeclared(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
credentialed, publishable, declared, humanSes bool
|
||||
want string
|
||||
}{
|
||||
{"declared agent on a secret key", true, false, true, false, AgencyAgent},
|
||||
{"declared agent on a publishable key", true, true, true, false, AgencyUnknown},
|
||||
{"anonymous script", false, false, false, false, AgencyBot},
|
||||
{"publishable-key script", true, true, false, false, AgencyBot},
|
||||
{"browser session", true, false, false, true, AgencyHuman},
|
||||
{"credentialed but undeclared", true, false, false, false, AgencyUnknown},
|
||||
} {
|
||||
if got := classify(tc.credentialed, tc.publishable, tc.declared, tc.humanSes); got != tc.want {
|
||||
t.Errorf("%s: agency = %q, want %q", tc.name, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestRuleAdmissionRefusesWhatWouldReadAsWorking pins the admission gate. Each
|
||||
// of these produces a rule that looks live and detects nothing, or one that
|
||||
// detects everything.
|
||||
func TestRuleAdmissionRefusesWhatWouldReadAsWorking(t *testing.T) {
|
||||
base := rule{Name: "r", Action: ActionBlock, Weight: 0.5, Enabled: true,
|
||||
All: []term{{Field: "signal.ip", Op: OpEq, Value: "x"}}}
|
||||
ok := base
|
||||
if err := admit(ok); err != nil {
|
||||
t.Fatalf("a well-formed rule was refused: %v", err)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
mut func(*rule)
|
||||
}{
|
||||
{"no terms holds on everything", func(r *rule) { r.All = nil }},
|
||||
{"no name cannot be read back in an alert", func(r *rule) { r.Name = "" }},
|
||||
{"an unknown action", func(r *rule) { r.Action = "quarantine" }},
|
||||
{"a weight outside [0,1]", func(r *rule) { r.Weight = 2 }},
|
||||
{"an unknown stage", func(r *rule) { r.Stage = "checkout" }},
|
||||
{"a field the vocabulary does not carry", func(r *rule) { r.All = []term{{Field: "tx.amount", Op: OpEq}} }},
|
||||
{"an unknown operator", func(r *rule) { r.All = []term{{Field: "signal.ip", Op: "matches"}} }},
|
||||
{"an unknown velocity axis", func(r *rule) { r.All = []term{{Field: "velocity.wallet.1h.count", Op: OpGte}} }},
|
||||
{"an unknown velocity window", func(r *rule) { r.All = []term{{Field: "velocity.ip.90d.count", Op: OpGte}} }},
|
||||
{"an empty in-set holds on nothing", func(r *rule) { r.All = []term{{Field: "agency", Op: OpIn}} }},
|
||||
{"an inlist naming no list", func(r *rule) { r.All = []term{{Field: "signal.ip", Op: OpInList}} }},
|
||||
} {
|
||||
r := base
|
||||
r.All = append([]term(nil), base.All...)
|
||||
tc.mut(&r)
|
||||
if err := admit(r); err == nil {
|
||||
t.Errorf("admitted a rule with %s", tc.name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestWeightOfEvidenceCompounds pins the score's shape. Two independent weak
|
||||
// signals must compound and no single one may saturate — a summed score clamps
|
||||
// at 1 and then loses every further signal.
|
||||
func TestWeightOfEvidenceCompounds(t *testing.T) {
|
||||
one, _ := combine([]hit{{Weight: 0.4, Action: ActionAllow}})
|
||||
two, _ := combine([]hit{{Weight: 0.4, Action: ActionAllow}, {Weight: 0.4, Action: ActionAllow}})
|
||||
three, _ := combine([]hit{{Weight: 0.4}, {Weight: 0.4}, {Weight: 0.4}})
|
||||
four, _ := combine([]hit{{Weight: 0.4}, {Weight: 0.4}, {Weight: 0.4}, {Weight: 0.4}})
|
||||
if !(one < two && two < three && three < four) {
|
||||
t.Fatalf("scores %v %v %v %v do not compound — a fourth signal changed nothing", one, two, three, four)
|
||||
}
|
||||
if four >= 1 {
|
||||
t.Fatalf("four 0.4 signals saturated at %v; nothing after this could raise the score", four)
|
||||
}
|
||||
}
|
||||
|
||||
// TestIdempotentDecideDoesNotScoreTwice pins the wire promise. A retried decide
|
||||
// must return the SAME decision and must not move the counters again — a
|
||||
// double-counted payment is a velocity rule firing on one transaction.
|
||||
func TestIdempotentDecideDoesNotScoreTwice(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
body := `{"stage":"payment","subject":{"kind":"transaction","id":"tx-9"},
|
||||
"amount":{"nano":1000000000,"currency":"USD","direction":"in"},
|
||||
"signals":{"ip":"198.51.100.7"},"idem":"k-1"}`
|
||||
|
||||
code, first := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme", body)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("first decide = %d %s", code, first)
|
||||
}
|
||||
code, second := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme", body)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("second decide = %d %s", code, second)
|
||||
}
|
||||
var a, b riskDecision
|
||||
_ = json.Unmarshal(first, &a)
|
||||
_ = json.Unmarshal(second, &b)
|
||||
if a.ID != b.ID {
|
||||
t.Fatalf("the same idempotency key produced two decisions (%s, %s)", a.ID, b.ID)
|
||||
}
|
||||
|
||||
_, list := req(t, app, http.MethodGet, "/v1/risk/decisions", "acme", "u_acme", "")
|
||||
var page riskDecisionPage
|
||||
_ = json.Unmarshal(list, &page)
|
||||
if len(page.Items) != 1 {
|
||||
t.Fatalf("%d decisions recorded for one idempotency key", len(page.Items))
|
||||
}
|
||||
}
|
||||
|
||||
// TestConcurrentRetriesUnderOneKeyProduceOneDecision pins the race the
|
||||
// two-statement version had: insert the row, then claim the key, and two
|
||||
// simultaneous retries both find no row, both insert, and the loser's claim
|
||||
// violates the index — so a caller that retried CORRECTLY gets a 500 and the
|
||||
// tenant's counters moved twice.
|
||||
//
|
||||
// Claiming the key inside the insert makes the loser lose at the index, before a
|
||||
// second decision exists, and read the winner's answer back.
|
||||
func TestConcurrentRetriesUnderOneKeyProduceOneDecision(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
body := `{"stage":"payment","subject":{"kind":"transaction","id":"tx-race"},
|
||||
"amount":{"nano":2500000000,"currency":"USD","direction":"in"},
|
||||
"signals":{"ip":"198.51.100.99"},"idem":"race-1"}`
|
||||
|
||||
const n = 8
|
||||
type result struct {
|
||||
code int
|
||||
id string
|
||||
}
|
||||
out := make(chan result, n)
|
||||
var start sync.WaitGroup
|
||||
start.Add(1)
|
||||
for i := 0; i < n; i++ {
|
||||
go func() {
|
||||
start.Wait()
|
||||
code, b := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme", body)
|
||||
var d riskDecision
|
||||
_ = json.Unmarshal(b, &d)
|
||||
out <- result{code, d.ID}
|
||||
}()
|
||||
}
|
||||
start.Done()
|
||||
|
||||
ids := map[string]int{}
|
||||
for i := 0; i < n; i++ {
|
||||
r := <-out
|
||||
if r.code != http.StatusOK {
|
||||
t.Errorf("a concurrent retry answered %d — a correct retry must never be an error", r.code)
|
||||
continue
|
||||
}
|
||||
ids[r.id]++
|
||||
}
|
||||
if len(ids) != 1 {
|
||||
t.Fatalf("%d concurrent requests under one key produced %d distinct decisions: %v", n, len(ids), ids)
|
||||
}
|
||||
// And exactly one row was written, so the counters moved once.
|
||||
_, list := req(t, app, http.MethodGet, "/v1/risk/decisions", "acme", "u_acme", "")
|
||||
var page riskDecisionPage
|
||||
_ = json.Unmarshal(list, &page)
|
||||
if len(page.Items) != 1 {
|
||||
t.Fatalf("%d decisions recorded under one idempotency key", len(page.Items))
|
||||
}
|
||||
}
|
||||
|
||||
// TestDecideRefusesAnUnknownStageOrKind pins the two vocabularies at the door.
|
||||
func TestDecideRefusesAnUnknownStageOrKind(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
for _, body := range []string{
|
||||
`{"stage":"checkout","subject":{"kind":"account","id":"a"}}`,
|
||||
`{"stage":"signup","subject":{"kind":"wallet","id":"a"}}`,
|
||||
`{"stage":"signup","subject":{"kind":"account","id":""}}`,
|
||||
} {
|
||||
code, got := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme", body)
|
||||
if code != http.StatusBadRequest {
|
||||
t.Errorf("%s = %d %s, want 400", body, code, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSimulateRefusesAnEmptyHistory pins the sandbox's whole reason for
|
||||
// existing: "no alerts" is what a quiet rule and an unrun rule both look like.
|
||||
func TestSimulateRefusesAnEmptyHistory(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/simulate", "acme", "u_acme",
|
||||
`{"candidate":{"name":"c","action":"review","weight":0.5,
|
||||
"all":[{"field":"signal.ip","op":"eq","value":"1.1.1.1"}]}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("simulate = %d %s", code, body)
|
||||
}
|
||||
var rep riskSimulateReport
|
||||
_ = json.Unmarshal(body, &rep)
|
||||
if rep.Refusal == "" {
|
||||
t.Fatal("an empty history reported zero alerts instead of refusing")
|
||||
}
|
||||
if rep.Events != 0 {
|
||||
t.Fatalf("events = %d on an empty history", rep.Events)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSimulateMeasuresAgainstRealHistory drives a decision in, then replays a
|
||||
// candidate over it, so the report is measured rather than asserted.
|
||||
func TestSimulateMeasuresAgainstRealHistory(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
mustOK(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"signup","subject":{"kind":"account","id":"a-1"},"signals":{"ip":"192.0.2.10"}}`,
|
||||
http.StatusOK)
|
||||
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/simulate", "acme", "u_acme",
|
||||
`{"candidate":{"name":"c","stage":"signup","action":"review","weight":0.5,
|
||||
"all":[{"field":"signal.ip","op":"eq","value":"192.0.2.10"}]}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("simulate = %d %s", code, body)
|
||||
}
|
||||
var rep riskSimulateReport
|
||||
_ = json.Unmarshal(body, &rep)
|
||||
if rep.Events != 1 || rep.Alerts != 1 {
|
||||
t.Fatalf("events=%d alerts=%d, want 1/1 — the replay did not read the recorded facts", rep.Events, rep.Alerts)
|
||||
}
|
||||
if len(rep.Added) != 1 {
|
||||
t.Fatalf("added=%v, want the one decision the candidate newly catches", rep.Added)
|
||||
}
|
||||
}
|
||||
|
||||
// TestModeDefaultsToShadowAndSaysSo pins the default at the wire.
|
||||
func TestModeDefaultsToShadowAndSaysSo(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
code, body := req(t, app, http.MethodGet, "/v1/risk/mode", "acme", "u_acme", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("mode = %d %s", code, body)
|
||||
}
|
||||
var m riskModeView
|
||||
_ = json.Unmarshal(body, &m)
|
||||
if m.Mode != "shadow" {
|
||||
t.Fatalf("a fresh tenant is %q, not shadow — it would act on its very first request", m.Mode)
|
||||
}
|
||||
|
||||
// A decision in shadow must SAY it was shadow.
|
||||
_, body = req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"signup","subject":{"kind":"account","id":"a-1"}}`)
|
||||
var d riskDecision
|
||||
_ = json.Unmarshal(body, &d)
|
||||
if !d.Shadow {
|
||||
t.Fatal("a shadow tenant's decision does not say so")
|
||||
}
|
||||
}
|
||||
|
||||
// TestHealthCarriesItsReport pins the reason the probe is untyped: the body IS
|
||||
// the answer.
|
||||
func TestHealthCarriesItsReport(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
code, body := req(t, app, http.MethodGet, "/v1/risk/health", "", "", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("health = %d %s", code, body)
|
||||
}
|
||||
for _, k := range []string{"status", "model", "warehouse", "billing"} {
|
||||
if !strings.Contains(string(body), `"`+k+`"`) {
|
||||
t.Errorf("the probe body does not carry %q — a typed op would have dropped exactly this", k)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestTrainIsOnlineAndPerTenant drives training through the wire and proves the
|
||||
// counters moved for the caller's tenant and nobody else's.
|
||||
func TestTrainIsOnlineAndPerTenant(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
obs := make([]string, 0, 20)
|
||||
for i := 0; i < 20; i++ {
|
||||
obs = append(obs, `{"subject":{"kind":"account","id":"a-1"},"amount":{"nano":1000000000,"currency":"USD","direction":"in"}}`)
|
||||
}
|
||||
body := `{"observations":[` + strings.Join(obs, ",") + `]}`
|
||||
code, got := req(t, app, http.MethodPost, "/v1/ml/train", "acme", "u_acme", body)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("train = %d %s", code, got)
|
||||
}
|
||||
var out mlTrainOut
|
||||
_ = json.Unmarshal(got, &out)
|
||||
if out.Learned != 20 {
|
||||
t.Fatalf("learned %d of 20", out.Learned)
|
||||
}
|
||||
_, acme, _ := armsOf(t, s, Tenant("hanzo/acme"))
|
||||
_, beta, _ := armsOf(t, s, Tenant("hanzo/beta"))
|
||||
if acme.State("hanzo/acme").Learned == 0 {
|
||||
t.Fatal("the caller's model learned nothing")
|
||||
}
|
||||
if beta.State("hanzo/beta").Learned != 0 {
|
||||
t.Fatal("another tenant's model learned from this tenant's data")
|
||||
}
|
||||
// And the tenant key really is qualified, not bare — asked of the caller's
|
||||
// OWN forest, which is the only one that could have been indexed wrongly.
|
||||
if acme.State("acme").Learned != 0 {
|
||||
t.Fatal("the model is indexed on the BARE org — two brands' same-named orgs would share it")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
package risk
|
||||
|
||||
// devmaster keys this test binary: cek opens nothing without a master, and a test
|
||||
// process has no KMS to resolve one from.
|
||||
import _ "github.com/hanzoai/cloud/internal/devmaster"
|
||||
@@ -0,0 +1,494 @@
|
||||
package risk
|
||||
|
||||
// durable_test.go pins the property store.go's header claims: a record is not
|
||||
// acknowledged until it is durable.
|
||||
//
|
||||
// The test drives the REAL durable plane — cloud.OrgStore over org.Durability
|
||||
// over a conditional object store — with the object store standing in as an
|
||||
// in-memory CAS register. Nothing here is a mock of our own code: the ship, the
|
||||
// fence and the lease are the production ones, and the only substitution is the
|
||||
// bucket.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/hanzoai/cloud"
|
||||
"github.com/hanzoai/cloud/internal/org"
|
||||
sqlitedrv "github.com/hanzoai/sqlite"
|
||||
"github.com/hanzoai/vfs/replica"
|
||||
"github.com/luxfi/aml/pkg/anomaly"
|
||||
luxlog "github.com/luxfi/log"
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// TestADecisionIsShippedBeforeItIsAcknowledged.
|
||||
//
|
||||
// A decision is the record an adverse action is defended with. cloud deploys
|
||||
// Recreate at one replica, so a decision that lives only on the pod's volume is
|
||||
// one a rollout can lose AFTER the caller was told it was taken. The ship
|
||||
// happens before the 200, or the 200 is a lie.
|
||||
func TestADecisionIsShippedBeforeItIsAcknowledged(t *testing.T) {
|
||||
app, _, store := wireDurable(t)
|
||||
|
||||
before := store.puts()
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"payment","subject":{"kind":"transaction","id":"tx-1"},
|
||||
"amount":{"nano":12000000000,"currency":"USD","direction":"in"}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("decide = %d %s", code, body)
|
||||
}
|
||||
if got := store.puts() - before; got == 0 {
|
||||
t.Fatal("the decision was acknowledged without a single write to the durable object: " +
|
||||
"a rollout loses it and the caller was told it was taken")
|
||||
}
|
||||
}
|
||||
|
||||
// TestAWriteThatCannotBeShippedIsNotAcknowledged.
|
||||
//
|
||||
// The object store goes away mid-flight. EVERY record write must then FAIL —
|
||||
// loudly, with a status a caller retries on — rather than answer 200 over a row
|
||||
// that exists on one pod's disk and nowhere else. The retry is exact because the
|
||||
// caller's idempotency key is claimed inside the same transaction.
|
||||
//
|
||||
// EVERY MUTATING ROUTE IS IN THE TABLE, and that is what makes this the guard
|
||||
// rather than three examples. Durability applied one call site at a time is a
|
||||
// rule somebody has to remember, and the writes easiest to forget are the
|
||||
// RETIREMENTS — a rule deleted, a mute lifted, a hold released — because an
|
||||
// unshipped delete brings the old state back at the next rollout, firing or
|
||||
// blocking, after a 204 said it was gone. A mutating op absent from the table
|
||||
// and from the records-nothing list fails the companion test below, which reads
|
||||
// the published subset.
|
||||
//
|
||||
// ONE app and ONE store for all of it. Every subject a retirement retires is
|
||||
// created first, while the store is up; then the store fails ONCE and every
|
||||
// write runs against it. A fresh durable harness per case would re-encrypt and
|
||||
// fsync the whole file per setup write, which is minutes of the suite's wall
|
||||
// clock to prove nothing the shared setup does not.
|
||||
func TestAWriteThatCannotBeShippedIsNotAcknowledged(t *testing.T) {
|
||||
app, _, store := wireDurable(t)
|
||||
ids := seedForRetirement(t, app)
|
||||
|
||||
store.fail(errors.New("the object store is unreachable"))
|
||||
for _, w := range durableWrites(ids) {
|
||||
code, body := req(t, app, w.method, w.path, "acme", "u_acme", w.body)
|
||||
if code < 500 {
|
||||
t.Errorf("%s (%s %s) answered %d %s while the durable object store was down — "+
|
||||
"the record exists on this pod only and the caller was told otherwise",
|
||||
w.what, w.method, w.path, code, body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// subjects are the server-minted identifiers a retirement needs. Every id here
|
||||
// is READ BACK from the create, never guessed: this plane mints its own, so a
|
||||
// literal in a test would be testing a 404 path.
|
||||
type subjects struct{ rule, spare, mute, control, decision string }
|
||||
|
||||
func seedForRetirement(t *testing.T, app *zip.App) subjects {
|
||||
t.Helper()
|
||||
var s subjects
|
||||
s.rule = mintedID(t, app, http.MethodPost, "/v1/risk/rules", ruleBody("durable"))
|
||||
s.spare = mintedID(t, app, http.MethodPost, "/v1/risk/rules", ruleBody("spare"))
|
||||
s.mute = mintedID(t, app, http.MethodPost, "/v1/risk/suppressions",
|
||||
`{"rule":"`+s.rule+`","kind":"transaction","reason":"noisy under test"}`)
|
||||
s.control = mintedID(t, app, http.MethodPost, "/v1/risk/controls",
|
||||
`{"subject":{"kind":"account","id":"acct-1"},"control":"payout-hold","reason":"under review"}`)
|
||||
s.decision = mintedID(t, app, http.MethodPost, "/v1/risk/decide",
|
||||
`{"stage":"payment","subject":{"kind":"transaction","id":"tx-seed"}}`)
|
||||
for _, c := range []call{
|
||||
{http.MethodPost, "/v1/risk/lists", `{"name":"blocked-ips","kind":"deny"}`},
|
||||
{http.MethodPost, "/v1/risk/lists/blocked-ips/entries", `{"values":["203.0.113.9"]}`},
|
||||
{http.MethodPost, "/v1/ml/train", trainBody},
|
||||
} {
|
||||
if code, body := req(t, app, c.method, c.path, "acme", "u_acme", c.body); code >= 300 {
|
||||
t.Fatalf("setup %s %s = %d %s", c.method, c.path, code, body)
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// mintedID performs one create and returns the id the SERVER chose.
|
||||
func mintedID(t *testing.T, app *zip.App, method, path, body string) string {
|
||||
t.Helper()
|
||||
code, out := req(t, app, method, path, "acme", "u_acme", body)
|
||||
if code >= 300 {
|
||||
t.Fatalf("setup %s %s = %d %s", method, path, code, out)
|
||||
}
|
||||
var v struct {
|
||||
ID string `json:"id"`
|
||||
}
|
||||
if err := json.Unmarshal(out, &v); err != nil || v.ID == "" {
|
||||
t.Fatalf("setup %s %s did not answer with an id: %s", method, path, out)
|
||||
}
|
||||
return v.ID
|
||||
}
|
||||
|
||||
// call is one request.
|
||||
type call struct{ method, path, body string }
|
||||
|
||||
// durableWrite is a mutating route and the name it fails under.
|
||||
type durableWrite struct{ what, method, path, body string }
|
||||
|
||||
func ruleBody(name string) string {
|
||||
return `{"rule":{"name":"` + name + `","stage":"payment","action":"review","weight":0.5,"enabled":true,
|
||||
"all":[{"field":"subject.kind","op":"eq","value":"transaction"}]}}`
|
||||
}
|
||||
|
||||
// durableWrites is the table. The RETIREMENTS come last and each takes a
|
||||
// DIFFERENT subject from the one an earlier case needs, because a write's local
|
||||
// half lands even when its ship does not — that is the defect under test.
|
||||
func durableWrites(s subjects) []durableWrite {
|
||||
return []durableWrite{
|
||||
{"a decision", http.MethodPost, "/v1/risk/decide",
|
||||
`{"stage":"payment","subject":{"kind":"transaction","id":"tx-2"}}`},
|
||||
{"a label on a decision", http.MethodPost, "/v1/risk/decisions/" + s.decision + "/label",
|
||||
`{"verdict":"fraud"}`},
|
||||
{"a policy version", http.MethodPut, "/v1/risk/policy",
|
||||
`{"stage":"payment","floor":"allow","reason":"the dispute rate doubled","bands":[{"at":0.9,"action":"block"}]}`},
|
||||
{"a rule", http.MethodPost, "/v1/risk/rules", ruleBody("another")},
|
||||
{"a rule change", http.MethodPatch, "/v1/risk/rules/" + s.rule, ruleBody("changed")},
|
||||
{"a list", http.MethodPost, "/v1/risk/lists", `{"name":"watched-ips","kind":"allow"}`},
|
||||
{"a deny-list entry", http.MethodPost, "/v1/risk/lists/blocked-ips/entries", `{"values":["198.51.100.4"]}`},
|
||||
{"a deny-list REMOVAL", http.MethodDelete, "/v1/risk/lists/blocked-ips/entries/203.0.113.9", ""},
|
||||
{"a mute", http.MethodPost, "/v1/risk/suppressions",
|
||||
`{"rule":"` + s.spare + `","kind":"account","reason":"noisy too"}`},
|
||||
{"a mute LIFTED", http.MethodDelete, "/v1/risk/suppressions/" + s.mute, ""},
|
||||
{"a control", http.MethodPost, "/v1/risk/controls",
|
||||
`{"subject":{"kind":"account","id":"acct-2"},"control":"block","reason":"card testing"}`},
|
||||
{"a control RELEASED", http.MethodDelete, "/v1/risk/controls/" + s.control, ""},
|
||||
{"the live/shadow switch", http.MethodPut, "/v1/risk/mode", `{"mode":"shadow"}`},
|
||||
{"the appetite", http.MethodPut, "/v1/ml/state/appetite", `{"review":0.01,"sample":0.001}`},
|
||||
{"a model snapshot", http.MethodPost, "/v1/ml/snapshot", `{}`},
|
||||
{"a search run", http.MethodPost, "/v1/ml/search", `{"limit":10}`},
|
||||
{"a rule RETIREMENT", http.MethodDelete, "/v1/risk/rules/" + s.spare, ""},
|
||||
}
|
||||
}
|
||||
|
||||
// trainBody is enough observations that the model has something to snapshot.
|
||||
const trainBody = `{"observations":[
|
||||
{"stage":"payment","subject":{"kind":"transaction","id":"t1"},"amount":{"nano":1000000000,"currency":"USD","direction":"in"}},
|
||||
{"stage":"payment","subject":{"kind":"transaction","id":"t2"},"amount":{"nano":2000000000,"currency":"USD","direction":"in"}},
|
||||
{"stage":"payment","subject":{"kind":"transaction","id":"t3"},"amount":{"nano":3000000000,"currency":"USD","direction":"in"}}]}`
|
||||
|
||||
// TestEveryMutatingRouteIsCoveredByTheDurabilityTable is the guard that survives
|
||||
// a NEW op rather than a revert.
|
||||
//
|
||||
// The table above is only a guard while it is complete, and the way it stops
|
||||
// being complete is somebody adding a route. This reads the app's OWN published
|
||||
// subset — the artifact the SDKs are generated from — and insists every mutating
|
||||
// path is either exercised by the table or named below as recording nothing.
|
||||
func TestEveryMutatingRouteIsCoveredByTheDurabilityTable(t *testing.T) {
|
||||
// Routes that record NOTHING: they answer from memory or from a read, so
|
||||
// there is no row whose durability could be in question. Each is named, so
|
||||
// admitting one is a decision somebody wrote down.
|
||||
//
|
||||
// /v1/ml/calibrate and /v1/ml/replay DO record, and both ship — they are
|
||||
// exercised by their own tests in skew_test.go, which need 200 judged rows
|
||||
// apiece and would cost this table two more fits to say the same thing.
|
||||
silent := map[string]bool{
|
||||
"POST /v1/ml/score": true, // scores one observation and learns nothing
|
||||
"POST /v1/ml/train": true, // moves in-memory counters; /v1/ml/snapshot is the record
|
||||
"POST /v1/ml/restore": true, // reads a snapshot back into memory
|
||||
"POST /v1/ml/evaluate": true, // measures over rows already written
|
||||
"POST /v1/risk/simulate": true, // tries a rule against history and writes nothing
|
||||
"POST /v1/ml/calibrate": true, // records, and ships — covered in skew_test.go
|
||||
"POST /v1/ml/replay": true, // records, and ships — covered in skew_test.go
|
||||
}
|
||||
covered := map[string]bool{}
|
||||
for _, w := range durableWrites(subjects{rule: "R", spare: "S", mute: "M", control: "C", decision: "D"}) {
|
||||
covered[w.method+" "+routePattern(w.path)] = true
|
||||
}
|
||||
|
||||
var doc struct {
|
||||
Paths map[string]map[string]any `json:"paths"`
|
||||
}
|
||||
if err := json.Unmarshal(readSubset(t), &doc); err != nil {
|
||||
t.Fatalf("plugin/risk/openapi.json: %v", err)
|
||||
}
|
||||
if len(doc.Paths) == 0 {
|
||||
t.Fatal("no paths read, so this guard proved nothing")
|
||||
}
|
||||
for path, methods := range doc.Paths {
|
||||
for m := range methods {
|
||||
method := strings.ToUpper(m)
|
||||
switch method {
|
||||
case http.MethodPost, http.MethodPut, http.MethodPatch, http.MethodDelete:
|
||||
default:
|
||||
continue
|
||||
}
|
||||
key := method + " " + path
|
||||
if silent[key] || covered[key] {
|
||||
continue
|
||||
}
|
||||
t.Errorf("%s is a mutating route the durability table does not exercise: either it "+
|
||||
"writes a record and must ship before it acknowledges, or it records nothing and "+
|
||||
"must say so in the list above", key)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// routePattern turns a concrete request path back into the pattern the published
|
||||
// subset names it by, so the table can be written with real, server-minted ids.
|
||||
func routePattern(path string) string {
|
||||
switch {
|
||||
case strings.HasPrefix(path, "/v1/risk/rules/"):
|
||||
return "/v1/risk/rules/{id}"
|
||||
case strings.HasPrefix(path, "/v1/risk/suppressions/"):
|
||||
return "/v1/risk/suppressions/{id}"
|
||||
case strings.HasPrefix(path, "/v1/risk/controls/"):
|
||||
return "/v1/risk/controls/{id}"
|
||||
case strings.HasPrefix(path, "/v1/risk/lists/") && strings.Contains(path, "/entries/"):
|
||||
return "/v1/risk/lists/{name}/entries/{value}"
|
||||
case strings.HasPrefix(path, "/v1/risk/lists/") && strings.HasSuffix(path, "/entries"):
|
||||
return "/v1/risk/lists/{name}/entries"
|
||||
case strings.HasPrefix(path, "/v1/risk/decisions/") && strings.HasSuffix(path, "/label"):
|
||||
return "/v1/risk/decisions/{id}/label"
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
// TestADeposedWriterDoesNotAcknowledgeItsOwnWrite is the OTHER half of the ship,
|
||||
// and the half a warning would hide.
|
||||
//
|
||||
// A ship has three outcomes, not two. It can fail (an error — covered above); it
|
||||
// can be acknowledged; or it can come back UNACKNOWLEDGED WITH NO ERROR, which
|
||||
// is what the fence answers when a successor pod has advanced the round: this
|
||||
// process is no longer this org's elected writer, so the row it just wrote is on
|
||||
// a deposed pod's disk and the org's record is elsewhere. Nothing failed. If the
|
||||
// plane treats that as success — or logs it and answers 200 — the caller is told
|
||||
// a decision is on file that the surviving writer will never produce.
|
||||
func TestADeposedWriterDoesNotAcknowledgeItsOwnWrite(t *testing.T) {
|
||||
app, _, store := wireDurable(t)
|
||||
|
||||
// One good decision first, so the file, the lease and the object all exist:
|
||||
// what follows is a writer that was deposed, not one that never held the org.
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"payment","subject":{"kind":"transaction","id":"tx-1"}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("first decide = %d %s", code, body)
|
||||
}
|
||||
|
||||
store.depose()
|
||||
code, body = req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"payment","subject":{"kind":"transaction","id":"tx-2"}}`)
|
||||
if code < 500 {
|
||||
t.Fatalf("a deposed writer answered %d %s — the decision is on this pod's disk and "+
|
||||
"the org's elected writer will never produce it", code, body)
|
||||
}
|
||||
if !strings.Contains(string(body), "elected writer") {
|
||||
t.Errorf("the refusal does not say why, so an operator reads it as a generic 5xx: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTheRecordPlaneIsOpenedThroughTheDurableStore is the guard that survives a
|
||||
// revert. The bare encrypted open is the open and nothing more, so a package
|
||||
// that calls it directly has no hydrate, no fence and no ship whatever its
|
||||
// comments say. This package reaches a file through the durable org store and no
|
||||
// other way.
|
||||
func TestTheRecordPlaneIsOpenedThroughTheDurableStore(t *testing.T) {
|
||||
// Spelled in two halves so this file does not match its own guard.
|
||||
bareOpen := "cloud.Org" + "DB("
|
||||
|
||||
files, err := filepath.Glob("*.go")
|
||||
if err != nil {
|
||||
t.Fatalf("glob: %v", err)
|
||||
}
|
||||
if len(files) == 0 {
|
||||
t.Fatal("no source read, so this guard proved nothing")
|
||||
}
|
||||
found := false
|
||||
for _, f := range files {
|
||||
body, err := os.ReadFile(f)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", f, err)
|
||||
}
|
||||
if strings.Contains(string(body), "cloud.NewOrgStore") {
|
||||
found = true
|
||||
}
|
||||
if strings.Contains(string(body), bareOpen) {
|
||||
t.Errorf("%s opens a tenant file with cloud.OrgDB — that open is local-only, so every record "+
|
||||
"in it is as durable as one pod's volume", f)
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Error("nothing in this package opens its files through cloud.NewOrgStore, so nothing is durable")
|
||||
}
|
||||
}
|
||||
|
||||
// TestCommitRefusesToShipAWriteThatFailed pins the order inside commit: the ship
|
||||
// is the acknowledgement of a write that HAPPENED, so a failed write must not
|
||||
// produce one.
|
||||
func TestCommitRefusesToShipAWriteThatFailed(t *testing.T) {
|
||||
_, s := wireApp(t)
|
||||
boom := errors.New("the write failed")
|
||||
err := s.State.shelf.commit(Tenant("hanzo/acme"), func(*sql.DB) error { return boom })
|
||||
if !errors.Is(err, boom) {
|
||||
t.Fatalf("commit returned %v, want the write's own error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// ── the durable harness ─────────────────────────────────────────────────────
|
||||
|
||||
// wireDurable mounts the surface over a REAL durable org store whose object
|
||||
// store is an in-memory CAS register.
|
||||
func wireDurable(t *testing.T) (*zip.App, *stateService, *bucket) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
log := luxlog.New("risktest")
|
||||
b := cloud.NewBase(cloud.Deps{Logger: log, DataDir: dir, Brand: "hanzo"}, "risk")
|
||||
store := newBucket()
|
||||
// The SAME checkpoint the composition root wires (build.go durableCheckpoint):
|
||||
// the ship reads the real on-disk path, and on an encrypting backend that path
|
||||
// is only fresh after the WAL is folded and re-encrypted. Without it this
|
||||
// harness would be testing a ship of a file that does not exist.
|
||||
b.Durable = org.NewDurability(store, soleWriter{}, nil, org.WithCheckpoint(checkpoint))
|
||||
|
||||
shape, err := forest(anomaly.Config{}, aggregates())
|
||||
if err != nil {
|
||||
t.Fatalf("forest: %v", err)
|
||||
}
|
||||
s := &cloud.Service[state]{Base: b, State: state{
|
||||
brand: "hanzo",
|
||||
dataDir: dir,
|
||||
shelf: newShelf(b),
|
||||
inflight: newInflight("a measurement"),
|
||||
running: newInflight("an exhaustive search"),
|
||||
digest: shape.Digest(),
|
||||
bill: cloud.NewResourceMeter(cloud.Deps{Logger: log}, "risk"),
|
||||
}}
|
||||
app := zip.New(zip.Config{Logger: log, DisableStartupMessage: true})
|
||||
mount(s, app)
|
||||
t.Cleanup(func() { s.State.shelf.close() })
|
||||
return app, s, store
|
||||
}
|
||||
|
||||
// checkpoint folds the WAL into the real path and re-encrypts it, which is what
|
||||
// makes the bytes a ship reads the bytes a caller was told were written.
|
||||
func checkpoint(ctx context.Context, db *sql.DB) error {
|
||||
// The fold runs on its OWN connection and RELEASES it before the re-encrypt:
|
||||
// the store holds one connection, and an envelope re-encrypt that asks for a
|
||||
// second while the first is still open waits for itself. (Production splits
|
||||
// these two for the same reason — build.go walCheckpointTruncate.)
|
||||
if err := fold(ctx, db); err != nil {
|
||||
return err
|
||||
}
|
||||
return sqlitedrv.Checkpoint(db)
|
||||
}
|
||||
|
||||
func fold(ctx context.Context, db *sql.DB) error {
|
||||
conn, err := db.Conn(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = conn.Close() }()
|
||||
var busy, frames, done int
|
||||
if err := conn.QueryRowContext(ctx, "PRAGMA wal_checkpoint(TRUNCATE)").Scan(&busy, &frames, &done); err != nil {
|
||||
return err
|
||||
}
|
||||
if busy != 0 {
|
||||
return fmt.Errorf("checkpoint busy=%d", busy)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// soleWriter is the membership: one pod, which is therefore every org's elected
|
||||
// writer. The election itself is tested in internal/org; here it only has to
|
||||
// resolve to "this process owns the lease" so a ship can be acknowledged.
|
||||
type soleWriter struct{}
|
||||
|
||||
func (soleWriter) Self() string { return "pod-a" }
|
||||
func (soleWriter) Members() []org.Member { return []org.Member{{ID: "pod-a"}} }
|
||||
|
||||
// bucket is an in-memory conditional store: the CAS register the fence and the
|
||||
// ship both run over. It counts writes and can be made to fail.
|
||||
type bucket struct {
|
||||
mu sync.Mutex
|
||||
slots map[string]*slot
|
||||
writes int
|
||||
err error
|
||||
fenced bool
|
||||
}
|
||||
|
||||
type slot struct {
|
||||
data []byte
|
||||
ver int
|
||||
}
|
||||
|
||||
func newBucket() *bucket { return &bucket{slots: map[string]*slot{}} }
|
||||
|
||||
func (b *bucket) puts() int {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
return b.writes
|
||||
}
|
||||
|
||||
func (b *bucket) fail(err error) {
|
||||
b.mu.Lock()
|
||||
b.err = err
|
||||
b.mu.Unlock()
|
||||
}
|
||||
|
||||
// depose models a successor pod that has advanced this org's recorded round: the
|
||||
// store admits the lease renewal and refuses every RECORD ship as fenced. That
|
||||
// is not a failure — the fence is working — so Sync answers (false, nil), which
|
||||
// is the outcome an error check alone never sees.
|
||||
func (b *bucket) depose() {
|
||||
b.mu.Lock()
|
||||
b.fenced = true
|
||||
b.mu.Unlock()
|
||||
}
|
||||
|
||||
func (b *bucket) Get(_ context.Context, key string) ([]byte, string, error) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
if b.err != nil {
|
||||
return nil, "", b.err
|
||||
}
|
||||
s, ok := b.slots[key]
|
||||
if !ok {
|
||||
return nil, "", replica.ErrNotFound
|
||||
}
|
||||
return append([]byte(nil), s.data...), strconv.Itoa(s.ver), nil
|
||||
}
|
||||
|
||||
func (b *bucket) PutIfVersion(_ context.Context, key string, data []byte, expect string) (string, error) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
if b.err != nil {
|
||||
return "", b.err
|
||||
}
|
||||
// A lease renewal is admitted — this process still runs; it is the RECORD
|
||||
// ship that a successor's higher round fences.
|
||||
if b.fenced && !strings.Contains(key, "lease") {
|
||||
return "", fmt.Errorf("bucket: %w", replica.ErrStaleRound)
|
||||
}
|
||||
s, ok := b.slots[key]
|
||||
cur := ""
|
||||
if ok {
|
||||
cur = strconv.Itoa(s.ver)
|
||||
}
|
||||
if cur != expect {
|
||||
return "", fmt.Errorf("bucket: %w: have %q want %q", replica.ErrConflict, cur, expect)
|
||||
}
|
||||
if !ok {
|
||||
s = &slot{}
|
||||
b.slots[key] = s
|
||||
}
|
||||
s.ver++
|
||||
s.data = append([]byte(nil), data...)
|
||||
if !strings.Contains(key, "lease") { // a lease renewal is not a record ship
|
||||
b.writes++
|
||||
}
|
||||
return strconv.Itoa(s.ver), nil
|
||||
}
|
||||
@@ -0,0 +1,325 @@
|
||||
package risk
|
||||
|
||||
// feature.go is the ONLY file in this package that reads the warehouse. Every
|
||||
// other file reaches a tenant's numbers through a function declared here, and
|
||||
// every function declared here takes a minted Tenant — so "a feature read is
|
||||
// tenant-scoped" is a property of the type signature, not of a reviewer's
|
||||
// attention.
|
||||
//
|
||||
// TWO PLANES, AND THE SECOND IS WHY THE FIRST IS SAFE TO SHARE.
|
||||
//
|
||||
// hanzo.risk_feature per-tenant. `org` is the FIRST sort key, so a tenant read
|
||||
// is a prefix scan and not a filter, and the predicate is
|
||||
// bound rather than interpolated.
|
||||
// hanzo.risk_baseline the network baseline. It has NO tenant column, no subject,
|
||||
// no id and no pseudonym — only quantiles over a k-anonymous
|
||||
// set of contributing orgs. There is no query against it that
|
||||
// returns one org's rows, because the rows do not exist. That
|
||||
// makes cross-tenant learning UNCOMPUTABLE rather than merely
|
||||
// disallowed, which is the only form of that boundary worth
|
||||
// having.
|
||||
//
|
||||
// THE HOT PATH DOES NOT COME THROUGH HERE. A payment-stage decision must answer
|
||||
// inside the processor's authorization window, and the warehouse is a single
|
||||
// StatefulSet pod that has taken api.hanzo.ai down once already. Scoring reads
|
||||
// the in-memory velocity rings (constant time, fixed memory per key); this file
|
||||
// is the BACKFILL that warms those rings and the read behind the dictionary and
|
||||
// the search sandbox. A decide that needs analytics up is a payment plane that
|
||||
// fails when analytics does.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/hanzoai/cloud/apps/datastore"
|
||||
)
|
||||
|
||||
// featureTable is the per-tenant feature surface. It is a projection of the ONE
|
||||
// event door (POST /v1/event -> bus -> event.event / event.error) plus the LLM
|
||||
// spend ledger, reduced to the counts a risk decision reads and bucketed at five
|
||||
// minutes, so a subject's recent shape is one prefix scan.
|
||||
//
|
||||
// org LEADS the sort key deliberately, unlike hanzo.cloud_usage's
|
||||
// (timestamp, organization, ...): a per-tenant read must be a prefix scan.
|
||||
const featureTable = "hanzo.risk_feature"
|
||||
|
||||
// baselineTable is the network baseline: quantiles per (bucket, subject kind,
|
||||
// feature), over a k-anonymous set of contributing orgs. No tenant column
|
||||
// exists, so no tenant row can be selected from it.
|
||||
const baselineTable = "hanzo.risk_baseline"
|
||||
|
||||
const featureDDL = `
|
||||
CREATE TABLE IF NOT EXISTS hanzo.risk_feature (
|
||||
org String,
|
||||
subject_kind LowCardinality(String),
|
||||
subject String,
|
||||
bucket DateTime,
|
||||
events UInt32,
|
||||
sessions UInt32,
|
||||
distincts UInt32,
|
||||
errors UInt32,
|
||||
spend_nano Int64,
|
||||
tokens UInt64,
|
||||
ips UInt16,
|
||||
uas UInt16,
|
||||
countries UInt16,
|
||||
signups UInt32,
|
||||
payments UInt32,
|
||||
declines UInt32,
|
||||
disputes UInt32,
|
||||
payouts UInt32
|
||||
) ENGINE = SummingMergeTree()
|
||||
ORDER BY (org, subject_kind, subject, bucket)
|
||||
TTL bucket + INTERVAL 400 DAY`
|
||||
|
||||
const baselineDDL = `
|
||||
CREATE TABLE IF NOT EXISTS hanzo.risk_baseline (
|
||||
bucket Date,
|
||||
subject_kind LowCardinality(String),
|
||||
feature LowCardinality(String),
|
||||
q10 Float64,
|
||||
q50 Float64,
|
||||
q90 Float64,
|
||||
q99 Float64,
|
||||
orgs UInt32,
|
||||
n UInt64
|
||||
) ENGINE = ReplacingMergeTree()
|
||||
ORDER BY (bucket, subject_kind, feature)
|
||||
TTL bucket + INTERVAL 400 DAY`
|
||||
|
||||
// ensureTables creates both planes idempotently. cloud reads and writes rows on
|
||||
// the event plane but does not own its DDL (o11y does); it DOES own these two,
|
||||
// so they are created here the way apps/datastore/cloudusage.go creates its own —
|
||||
// CREATE IF NOT EXISTS, plus additive ADD COLUMN IF NOT EXISTS migrations, so a
|
||||
// fresh warehouse and a legacy one converge on the same shape.
|
||||
func ensureTables(ctx context.Context) error {
|
||||
if !datastore.Ready() {
|
||||
return errWarehouse
|
||||
}
|
||||
for _, stmt := range []string{featureDDL, baselineDDL} {
|
||||
if err := datastore.Exec(ctx, stmt); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// featureColumns is the FIXED allowlist of columns a caller may name. Nothing
|
||||
// user-derived is ever spelled into a statement: a caller names a column, this
|
||||
// map answers whether that name is one of ours, and the map's own key is what
|
||||
// reaches the SQL. A name that is not here is refused, never interpolated.
|
||||
var featureColumns = map[string]string{
|
||||
"events": "events",
|
||||
"sessions": "sessions",
|
||||
"distincts": "distincts",
|
||||
"errors": "errors",
|
||||
"spend": "spend_nano",
|
||||
"tokens": "tokens",
|
||||
"ips": "ips",
|
||||
"uas": "uas",
|
||||
"countries": "countries",
|
||||
"signups": "signups",
|
||||
"payments": "payments",
|
||||
"declines": "declines",
|
||||
"disputes": "disputes",
|
||||
"payouts": "payouts",
|
||||
}
|
||||
|
||||
// subjectKinds is the FIXED allowlist of aggregation axes. `pair` and `device`
|
||||
// are the axes that surface several nominally unrelated customers acting as one,
|
||||
// which is what multi-account abuse and account sharing look like from here.
|
||||
var subjectKinds = map[string]bool{
|
||||
"account": true, "transaction": true, "session": true, "agent": true,
|
||||
"merchant": true, "payout": true, "user": true, "device": true,
|
||||
"ip": true, "pair": true,
|
||||
}
|
||||
|
||||
// featureRow is one bucket of one subject's activity, as the reader materialises
|
||||
// it. The tenant is NOT a field: a row is only ever produced by a read that was
|
||||
// already scoped, so carrying the key back would be the one place it could be
|
||||
// compared against the wrong thing.
|
||||
type featureRow struct {
|
||||
bucket time.Time
|
||||
values map[string]float64
|
||||
}
|
||||
|
||||
// riskWhere is the tenancy predicate. org LEADS and is BOUND; the window is
|
||||
// bound; the subject kind has already passed the allowlist and is bound anyway.
|
||||
// Nothing user-derived reaches the statement as an identifier.
|
||||
//
|
||||
// The shape is the one apps/analytics/query.go and o11y's eventsql use, which is
|
||||
// the point: a third shape is a third place for the boundary to be got wrong.
|
||||
func riskWhere(t Tenant, kind, subject string, start, end time.Time) (string, []any) {
|
||||
return "org = ? AND subject_kind = ? AND subject = ? AND bucket >= ? AND bucket < ?",
|
||||
[]any{t.String(), kind, subject, tsLiteral(start), tsLiteral(end)}
|
||||
}
|
||||
|
||||
// tsLiteral renders a time the way the warehouse driver binds a DateTime. Same
|
||||
// transport apps/analytics uses; the value is still a bound parameter.
|
||||
func tsLiteral(t time.Time) string { return t.UTC().Format("2006-01-02 15:04:05") }
|
||||
|
||||
// errWarehouse is the honest gap: the warehouse is unreachable, so a read
|
||||
// returns nothing AND says why. A reader that answered zero would be reporting
|
||||
// "this subject did nothing", which is the one answer a risk surface must never
|
||||
// invent.
|
||||
var errWarehouse = fmt.Errorf("risk: the warehouse is unreachable, so this subject's history cannot be read")
|
||||
|
||||
// window reads one subject's buckets over [start,end).
|
||||
//
|
||||
// The caller supplies a Tenant, which it can only have obtained from a validated
|
||||
// principal, and a kind that must be in the allowlist. The column list is built
|
||||
// from featureColumns' own values, never from the caller's spelling.
|
||||
func window(ctx context.Context, t Tenant, kind, subject string, start, end time.Time) ([]featureRow, error) {
|
||||
if !subjectKinds[kind] {
|
||||
return nil, fmt.Errorf("risk: %q is not a subject kind", kind)
|
||||
}
|
||||
if strings.TrimSpace(subject) == "" {
|
||||
return nil, fmt.Errorf("risk: no subject, so the window would name nobody")
|
||||
}
|
||||
if !datastore.Ready() {
|
||||
return nil, errWarehouse
|
||||
}
|
||||
where, args := riskWhere(t, kind, subject, start, end)
|
||||
|
||||
// Ordered so the projection is stable across calls; the names come from our
|
||||
// own map, so this concatenation carries nothing the caller wrote.
|
||||
names := columnNames()
|
||||
sel := make([]string, 0, len(names)+1)
|
||||
sel = append(sel, "bucket")
|
||||
for _, n := range names {
|
||||
sel = append(sel, "sum("+featureColumns[n]+") AS "+n)
|
||||
}
|
||||
q := "SELECT " + strings.Join(sel, ", ") + " FROM " + featureTable +
|
||||
" WHERE " + where + " GROUP BY bucket ORDER BY bucket"
|
||||
|
||||
rows, err := datastore.Query(ctx, q, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]featureRow, 0, len(rows))
|
||||
for _, r := range rows {
|
||||
fr := featureRow{values: make(map[string]float64, len(names))}
|
||||
if b, ok := r["bucket"].(time.Time); ok {
|
||||
fr.bucket = b
|
||||
}
|
||||
for _, n := range names {
|
||||
fr.values[n] = num(r[n])
|
||||
}
|
||||
out = append(out, fr)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// columnNames returns the allowlist keys in a stable order.
|
||||
func columnNames() []string {
|
||||
out := make([]string, 0, len(featureColumns))
|
||||
for n := range featureColumns {
|
||||
out = append(out, n)
|
||||
}
|
||||
sortStrings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// ── the network baseline ────────────────────────────────────────────────────
|
||||
|
||||
// kAnonMin is how many DISTINCT orgs must contribute to a bucket before its
|
||||
// quantiles may be published. Below it a quantile is close enough to one
|
||||
// business's own numbers to be that business's numbers.
|
||||
const kAnonMin = 25
|
||||
|
||||
// nMin is how many observations a published bucket needs. k orgs each
|
||||
// contributing one row is k-anonymous and still statistically meaningless.
|
||||
const nMin = 1000
|
||||
|
||||
// baselinePopulate is the WHOLE cross-org surface, and it is a package CONSTANT.
|
||||
// It is never composed from request data, has no placeholder a caller could
|
||||
// reach, and its projection carries no org, no subject, no id and no pseudonym —
|
||||
// only quantiles and two counts. The HAVING clause is the k-anonymity gate and
|
||||
// is part of the statement rather than a filter applied to its result, so a
|
||||
// bucket below the threshold is never materialised at all.
|
||||
//
|
||||
// The reserved `$public` tenant is excluded here as well as at the mint: an
|
||||
// unauthenticated stranger writes into that lane, and a stranger who can move
|
||||
// the network baseline can move every tenant's comparison against it.
|
||||
const baselinePopulate = `
|
||||
INSERT INTO hanzo.risk_baseline (bucket, subject_kind, feature, q10, q50, q90, q99, orgs, n)
|
||||
SELECT
|
||||
toDate(bucket) AS bucket,
|
||||
subject_kind,
|
||||
'events' AS feature,
|
||||
quantileExact(0.10)(events) AS q10,
|
||||
quantileExact(0.50)(events) AS q50,
|
||||
quantileExact(0.90)(events) AS q90,
|
||||
quantileExact(0.99)(events) AS q99,
|
||||
uniqExact(org) AS orgs,
|
||||
count() AS n
|
||||
FROM hanzo.risk_feature
|
||||
WHERE org != '$public' AND org NOT LIKE '%/$public'
|
||||
AND bucket >= toDateTime(toDate(now()) - 1) AND bucket < toDateTime(toDate(now()))
|
||||
GROUP BY bucket, subject_kind
|
||||
HAVING orgs >= 25 AND n >= 1000`
|
||||
|
||||
// baselineRow is one published quantile bucket. Reflection over this type is
|
||||
// part of the isolation proof: a field naming a tenant, a subject or a person
|
||||
// would be a leak, and the test that asserts none exists reads THIS type and the
|
||||
// DDL above rather than a comment.
|
||||
type baselineRow struct {
|
||||
bucket time.Time
|
||||
kind string
|
||||
feature string
|
||||
q10 float64
|
||||
q50 float64
|
||||
q90 float64
|
||||
q99 float64
|
||||
orgs uint64
|
||||
n uint64
|
||||
}
|
||||
|
||||
// publishBaseline recomputes yesterday's network quantiles. It is the ONLY
|
||||
// writer of the baseline plane and it runs on a schedule, never on a request, so
|
||||
// no caller can time it, steer it or observe its cost.
|
||||
func publishBaseline(ctx context.Context) error {
|
||||
if !datastore.Ready() {
|
||||
return errWarehouse
|
||||
}
|
||||
return datastore.Exec(ctx, baselinePopulate)
|
||||
}
|
||||
|
||||
// baseline reads the published quantiles for one subject kind. There is no
|
||||
// tenant argument BECAUSE THERE IS NO TENANT COLUMN — every caller reads the
|
||||
// same rows, which is what makes them safe to read at all.
|
||||
func baseline(ctx context.Context, kind string, day time.Time) ([]baselineRow, error) {
|
||||
if !subjectKinds[kind] {
|
||||
return nil, fmt.Errorf("risk: %q is not a subject kind", kind)
|
||||
}
|
||||
if !datastore.Ready() {
|
||||
return nil, errWarehouse
|
||||
}
|
||||
rows, err := datastore.Query(ctx,
|
||||
"SELECT bucket, subject_kind, feature, q10, q50, q90, q99, orgs, n FROM "+baselineTable+
|
||||
" WHERE bucket = ? AND subject_kind = ? ORDER BY feature",
|
||||
day.UTC().Format("2006-01-02"), kind)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]baselineRow, 0, len(rows))
|
||||
for _, r := range rows {
|
||||
br := baselineRow{
|
||||
kind: str(r["subject_kind"]), feature: str(r["feature"]),
|
||||
q10: num(r["q10"]), q50: num(r["q50"]), q90: num(r["q90"]), q99: num(r["q99"]),
|
||||
orgs: uint64(num(r["orgs"])), n: uint64(num(r["n"])),
|
||||
}
|
||||
if b, ok := r["bucket"].(time.Time); ok {
|
||||
br.bucket = b
|
||||
}
|
||||
// Belt on top of the HAVING: a row that predates the gate, or one an
|
||||
// operator inserted by hand, is dropped on read rather than trusted.
|
||||
if br.orgs < kAnonMin || br.n < nMin {
|
||||
continue
|
||||
}
|
||||
out = append(out, br)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,770 @@
|
||||
package risk
|
||||
|
||||
// grade.go is the record plane and the arithmetic behind SCORING QUALITY: what a
|
||||
// score means, why a decision went the way it did, and what the organisation
|
||||
// asked for at that probability.
|
||||
//
|
||||
// FOUR RECORDS, ALL IN THE TENANT'S OWN FILE. A policy version, a calibration,
|
||||
// the verdict attached to one decision and a replay report are each something an
|
||||
// auditor, a regulator or a declined customer can ask about, so none of them
|
||||
// lives in memory. They ride the same cek-encrypted per-org SQLite file the
|
||||
// decision log already uses, which is what makes them survive the
|
||||
// Recreate-at-one-replica rollout that drops every in-memory model.
|
||||
//
|
||||
// THAT FILE IS OPENED WITH cloud.OrgDB AND IS THEREFORE LOCAL. OrgDB is the
|
||||
// encrypted open and nothing more; the ship-before-ack path is cloud.OrgStore
|
||||
// configured WithDurable, which this package does not use — so a write here is
|
||||
// as durable as the pod's volume and no more. Every record in this file has that
|
||||
// property, the decision log included, and it predates this plane. Named here
|
||||
// rather than claimed away: the two sibling durable planes (apps/research
|
||||
// compose.go, apps/books books.go) ship after every commit and treat an unacked
|
||||
// ship as an error, and moving the shelf onto OrgStore is the change that would
|
||||
// give these records the same guarantee.
|
||||
//
|
||||
// NOTHING IS CACHED, DELIBERATELY. The decide path already reads this tenant's
|
||||
// rules, lists and suppressions from its own file on every decision; reading the
|
||||
// policy and the calibration the same way adds one idiom rather than a second.
|
||||
// It also removes an entire class of bug — a cache with no invalidation path
|
||||
// that keeps deciding under a policy an operator has already replaced — and it
|
||||
// is why a rollout needs no rehydration step here: the record IS the state.
|
||||
//
|
||||
// THE ESCALATION RULE, ONCE. The evidence (rules and the model) proposes an
|
||||
// action, and the policy proposes another from the calibrated probability. The
|
||||
// decision takes the STRONGER of the two, capped at what the evidence can
|
||||
// justify. A deny-list rule must not be overruled by a low probability — it is
|
||||
// the tenant's explicit instruction — and a high probability must not be talked
|
||||
// down by quiet evidence. But the probability is a pure function of the same
|
||||
// score the model's own ceiling already capped, so an uncapped ladder would void
|
||||
// that ceiling by arithmetic: see ceiling() below. Two authorities for one
|
||||
// decision would otherwise be a question with two answers.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/aml/pkg/calibrate"
|
||||
// The package is aml's `evaluate`; it is bound to `quality` here because
|
||||
// rule.go already declares a function called evaluate — the rule evaluator —
|
||||
// and one name cannot mean two things in one package.
|
||||
quality "github.com/luxfi/aml/pkg/evaluate"
|
||||
"github.com/luxfi/aml/pkg/policy"
|
||||
"github.com/luxfi/aml/pkg/reason"
|
||||
"github.com/luxfi/aml/pkg/replay"
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// qualitySchema is applied beside the decision plane's own schema, on the same
|
||||
// open, in the same file. Separate constants rather than one, because these are
|
||||
// a different concern with a different owner; the same file, because a verdict
|
||||
// that could outlive or precede the decision it grades is not a record.
|
||||
//
|
||||
// NO TTL and no expiry anywhere here. These are records: a policy version says
|
||||
// what was in force on the day someone was declined, and a verdict says why.
|
||||
// Only the retention plane decides when a tenant's records go, per tenant.
|
||||
const qualitySchema = `
|
||||
CREATE TABLE IF NOT EXISTS policy (
|
||||
stage TEXT NOT NULL,
|
||||
version INTEGER NOT NULL,
|
||||
at TEXT NOT NULL,
|
||||
by TEXT NOT NULL DEFAULT '',
|
||||
reason TEXT NOT NULL DEFAULT '',
|
||||
digest TEXT NOT NULL,
|
||||
body TEXT NOT NULL,
|
||||
PRIMARY KEY (stage, version)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS calibration (
|
||||
version INTEGER PRIMARY KEY,
|
||||
at TEXT NOT NULL,
|
||||
by TEXT NOT NULL DEFAULT '',
|
||||
shape TEXT NOT NULL,
|
||||
horizon INTEGER NOT NULL DEFAULT 0,
|
||||
digest TEXT NOT NULL,
|
||||
body TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS verdict (
|
||||
decision TEXT PRIMARY KEY,
|
||||
at TEXT NOT NULL,
|
||||
probability REAL,
|
||||
calibration TEXT NOT NULL DEFAULT '',
|
||||
policy TEXT NOT NULL DEFAULT '',
|
||||
reasons TEXT NOT NULL DEFAULT '[]',
|
||||
refusal TEXT NOT NULL DEFAULT ''
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS replay (
|
||||
id TEXT PRIMARY KEY,
|
||||
at TEXT NOT NULL,
|
||||
by TEXT NOT NULL DEFAULT '',
|
||||
digest TEXT NOT NULL,
|
||||
body TEXT NOT NULL
|
||||
);
|
||||
`
|
||||
|
||||
// topReasons is how many principal reasons a decision cites.
|
||||
//
|
||||
// Four, from Reg B: a creditor taking adverse action need not describe more than
|
||||
// four reasons, and a list of every contributing feature is not an explanation
|
||||
// anybody can act on. The model has nine coordinates and almost always moves on
|
||||
// two or three of them.
|
||||
const topReasons = 4
|
||||
|
||||
// verdict is the defensibility layer over one outcome: what the score meant,
|
||||
// what moved it, and which ladder read it.
|
||||
type verdict struct {
|
||||
// probability is the calibrated likelihood. Absent — a nil pointer, not a
|
||||
// zero — when no calibration is in force or the one on file was fitted under
|
||||
// a different scoring shape. A density reported as a probability is a number
|
||||
// that lies.
|
||||
probability *float64
|
||||
// reasons are the principal reasons, strongest first.
|
||||
reasons []reason.Reason
|
||||
// calibration and policy are the digests of what was applied, so the
|
||||
// decision can be replayed against exactly them.
|
||||
calibration string
|
||||
policy string
|
||||
// action is what the policy asked for, before the escalation rule. Empty
|
||||
// when no ladder was reachable.
|
||||
action string
|
||||
// refusal names why there is no probability, when there is none. Silence and
|
||||
// a low probability render the same on a screen and are opposite facts.
|
||||
refusal string
|
||||
}
|
||||
|
||||
// dispositions maps the decision log's verdict vocabulary onto the engine's.
|
||||
//
|
||||
// It is a CLOSED map with no default arm: a verdict this table does not name is
|
||||
// unjudged, which is the honest reading of a word nobody has defined the meaning
|
||||
// of. fraud, chargeback and abuse are all "the decision was right to be
|
||||
// worried"; only legitimate is the other way; unknown says a human looked and
|
||||
// could not tell, which is not evidence for either side.
|
||||
var dispositions = map[string]replay.Disposition{
|
||||
"fraud": replay.Productive,
|
||||
"chargeback": replay.Productive,
|
||||
"abuse": replay.Productive,
|
||||
"legitimate": replay.Unproductive,
|
||||
}
|
||||
|
||||
// scoringShape is the fingerprint everything a calibration depends on must agree
|
||||
// on, and it is the training-serving skew control.
|
||||
//
|
||||
// A calibration maps THE DECISION SCORE to a probability, and that score is the
|
||||
// combination of the model's own score with whatever this tenant's rules
|
||||
// contributed. So the shape covers BOTH: the detector's geometry and feature
|
||||
// inventory (anomaly.Store.Digest, which already folds the inventory in order
|
||||
// with its neutral values), and this tenant's enabled rule set with the weights
|
||||
// and actions each rule carries. Change either and the score distribution moves;
|
||||
// the map fitted before the change then refuses to answer rather than quietly
|
||||
// reporting probabilities for coordinates that no longer mean what they meant.
|
||||
//
|
||||
// A RULE-WIDE SUPPRESSION IS IN IT. Muting a rule is the day-to-day tuning knob
|
||||
// and retiring one is the rare act, but combine() sums only the hits that were
|
||||
// not suppressed — so a mute moves every score that rule touched EXACTLY as
|
||||
// retiring it would. A control that noticed the rare change and not the common
|
||||
// one would be a control nobody could rely on.
|
||||
//
|
||||
// LISTS AND SUBJECT-SCOPED SUPPRESSIONS ARE DELIBERATELY NOT IN IT, by the same
|
||||
// argument in both directions. A list entry is data a rule reads, and a
|
||||
// suppression naming one subject is a statement about that subject — neither
|
||||
// changes what the scorer computes for the population, and both change
|
||||
// constantly in normal operation. A plane whose calibration expired every time a
|
||||
// stolen card was added to a deny list, or every time one merchant was excused,
|
||||
// would never have a calibration at all. The boundary is stated rather than
|
||||
// assumed: what moves the DISTRIBUTION invalidates, operational data about one
|
||||
// row does not.
|
||||
func scoringShape(db *sql.DB, model string) (string, error) {
|
||||
rules, err := loadRules(db)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
sups, err := loadSuppressions(db)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return shapeOf(model, rules, sups, time.Now()), nil
|
||||
}
|
||||
|
||||
// shapeOf folds the coordinate system from the values that produce a score.
|
||||
//
|
||||
// It takes the values rather than the store so the DECIDE path can fold the
|
||||
// exact rules and suppressions it just scored under, instead of re-reading them
|
||||
// and recording a shape a concurrent write may already have moved. scoringShape
|
||||
// is the same fold over a fresh read, for the callers that hold neither.
|
||||
func shapeOf(model string, rules []rule, sups []suppression, now time.Time) string {
|
||||
// Sorted by id so the fingerprint is a property of the SET and not of the
|
||||
// order rows happened to come back in.
|
||||
rules = append([]rule(nil), rules...)
|
||||
sort.Slice(rules, func(i, j int) bool { return rules[i].ID < rules[j].ID })
|
||||
muted := make([]string, 0, len(sups))
|
||||
for _, s := range sups {
|
||||
// Expired mutes nothing — `until` in the past is the store's own expiry and
|
||||
// the decide path already reads it that way, so a shape that still counted
|
||||
// one would describe a scorer that no longer exists.
|
||||
if !s.Until.IsZero() && now.After(s.Until) {
|
||||
continue
|
||||
}
|
||||
// A mute that names a subject is about that subject. A mute that names a
|
||||
// rule and no subject silences it for the whole population, which is a move
|
||||
// of the distribution.
|
||||
if s.Rule == "" || s.Subject != "" {
|
||||
continue
|
||||
}
|
||||
muted = append(muted, s.Rule+":"+s.Kind)
|
||||
}
|
||||
sort.Strings(muted)
|
||||
|
||||
h := sha256.New()
|
||||
fmt.Fprintf(h, "shape/v2|%s|", model)
|
||||
for _, r := range rules {
|
||||
if !r.Enabled {
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(h, "%s:%s:%g:%s|", r.ID, r.Stage, r.Weight, r.Action)
|
||||
}
|
||||
fmt.Fprint(h, "muted|")
|
||||
for _, m := range muted {
|
||||
fmt.Fprintf(h, "%s|", m)
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil))
|
||||
}
|
||||
|
||||
// grade turns an outcome into a defensible one.
|
||||
//
|
||||
// It reads the tenant's own calibration and its ladder for this stage, maps the
|
||||
// recorded score to a probability, ranks the reasons out of the model's exact
|
||||
// counterfactual attribution and the rules that actually fired, and returns the
|
||||
// action the policy asks for. It writes nothing: the caller records the verdict
|
||||
// in the same place and at the same moment it records the decision, because a
|
||||
// decision whose reasons landed separately could exist without them.
|
||||
func grade(db *sql.DB, shape string, stage string, out outcome) verdict {
|
||||
v := verdict{reasons: reasonsOf(out)}
|
||||
|
||||
cal, _, _, ok, err := currentCalibration(db)
|
||||
switch {
|
||||
case err != nil:
|
||||
v.refusal = "the calibration could not be read, so this score has no probability"
|
||||
return v
|
||||
case !ok:
|
||||
v.refusal = "no calibration is fitted for this organisation, so a score is a rank and not a probability"
|
||||
return v
|
||||
}
|
||||
read, err := cal.Under(shape)
|
||||
if err != nil {
|
||||
// The commonest cause by far, and the one worth naming: the model, the rule
|
||||
// set or a rule-wide mute moved since the fit.
|
||||
v.refusal = "the calibration on file was fitted under a different scoring shape: " + err.Error()
|
||||
return v
|
||||
}
|
||||
p := read.P(out.score)
|
||||
v.probability, v.calibration = &p, read.Digest()
|
||||
|
||||
pol, ok, err := currentPolicy(db, stage)
|
||||
switch {
|
||||
case err != nil:
|
||||
v.refusal = "the policy could not be read, so no threshold was applied"
|
||||
return v
|
||||
case !ok:
|
||||
v.refusal = "no policy is set for this stage, so the probability was computed and nothing was decided from it"
|
||||
return v
|
||||
}
|
||||
v.policy, v.action = pol.Digest, pol.Action(p)
|
||||
return v
|
||||
}
|
||||
|
||||
// escalate is the ONE place two authorities become one decision: the stronger of
|
||||
// what the evidence asked for and what the policy asked for, with the policy
|
||||
// held to what the evidence behind it can justify.
|
||||
func escalate(evidence, policy string, out outcome) string {
|
||||
policy = atMost(policy, ceiling(out))
|
||||
if actionRank(policy) > actionRank(evidence) {
|
||||
return policy
|
||||
}
|
||||
return evidence
|
||||
}
|
||||
|
||||
// ceiling is the strongest action the evidence behind one outcome can carry, or
|
||||
// empty when it carries no ceiling at all.
|
||||
//
|
||||
// A decision the MODEL alone moved can put a transaction in front of a person
|
||||
// and cannot decline one: an unexplainable refusal is not a decision anybody can
|
||||
// defend to the customer or to a chargeback network, which is why decide caps
|
||||
// the model's own hit at modelCeiling (decide.go, rule.go). The policy ladder is
|
||||
// not a second, independent authority for that decline — it reads the calibrated
|
||||
// probability, and the probability is a pure function of the SAME score — so
|
||||
// applying the ladder over the cap would void the cap by arithmetic and the
|
||||
// documented invariant would hold only until somebody set a band.
|
||||
//
|
||||
// A RULE the organisation wrote is different in kind. It is an explicit
|
||||
// instruction with a name, a weight and a sentence a person can read, so a
|
||||
// decline it reaches is explainable and no ceiling applies. A suppressed hit is
|
||||
// not evidence: it contributed zero weight and it is not cited as a reason, so
|
||||
// citing it here would let a mute both silence a rule and license a decline.
|
||||
func ceiling(out outcome) string {
|
||||
for _, h := range out.hits {
|
||||
if h.Suppressed || h.Rule == modelRuleID {
|
||||
continue
|
||||
}
|
||||
return ""
|
||||
}
|
||||
return modelCeiling
|
||||
}
|
||||
|
||||
// atMost caps an action at a ceiling. An empty ceiling caps nothing.
|
||||
func atMost(action, ceiling string) string {
|
||||
if ceiling == "" || actionRank(action) <= actionRank(ceiling) {
|
||||
return action
|
||||
}
|
||||
return ceiling
|
||||
}
|
||||
|
||||
// reasonsOf ranks the principal reasons behind one outcome.
|
||||
//
|
||||
// The model's contribution comes from its own per-feature counterfactual — move
|
||||
// one coordinate to neutral, rescore on the same trees — so it is a measurement
|
||||
// on the model that decided rather than a surrogate fitted afterwards. Each
|
||||
// rule that fired contributes one reason naming itself.
|
||||
//
|
||||
// A SUPPRESSED HIT IS NOT A REASON. It is recorded, it is visible, and it
|
||||
// contributed zero weight to the action — so citing it as a reason for the
|
||||
// action would be false. That is the same argument the suppression path already
|
||||
// makes for keeping the hit in the record.
|
||||
func reasonsOf(out outcome) []reason.Reason {
|
||||
rs := reason.Rank(out.causes, 0)
|
||||
for _, h := range out.hits {
|
||||
if h.Suppressed || h.Rule == modelRuleID {
|
||||
continue
|
||||
}
|
||||
rs = append(rs, reason.OfRule(h.Rule, h.Name, h.Severity, h.Weight))
|
||||
}
|
||||
sort.SliceStable(rs, func(i, j int) bool { return rs[i].Weight > rs[j].Weight })
|
||||
if len(rs) > topReasons {
|
||||
rs = rs[:topReasons]
|
||||
}
|
||||
return rs
|
||||
}
|
||||
|
||||
// modelRuleID is the identifier the engine's own detector files its hit under.
|
||||
// Its contribution is already expressed feature by feature in the causes, so
|
||||
// citing the hit as well would count one piece of evidence twice.
|
||||
const modelRuleID = "anomaly"
|
||||
|
||||
// ── the policy record ───────────────────────────────────────────────────────
|
||||
|
||||
// currentPolicy reads the ladder in force for one stage: the highest version.
|
||||
func currentPolicy(db *sql.DB, stage string) (policy.Policy, bool, error) {
|
||||
var body string
|
||||
err := db.QueryRow(`SELECT body FROM policy WHERE stage = ? ORDER BY version DESC LIMIT 1`, stage).Scan(&body)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return policy.Policy{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return policy.Policy{}, false, err
|
||||
}
|
||||
var p policy.Policy
|
||||
if err := json.Unmarshal([]byte(body), &p); err != nil {
|
||||
return policy.Policy{}, false, err
|
||||
}
|
||||
return p, true, nil
|
||||
}
|
||||
|
||||
// putPolicy mints the next version of a stage's ladder.
|
||||
//
|
||||
// Nothing is ever updated in place. A policy change is a governance decision and
|
||||
// the question "what was in force when this customer was declined" has to be a
|
||||
// lookup rather than a reconstruction — so the row is an insert, the version is
|
||||
// the successor of whatever is there, and the previous version stays exactly as
|
||||
// it was.
|
||||
func putPolicy(db *sql.DB, p policy.Policy) (policy.Policy, error) {
|
||||
sealed, err := policy.Seal(p)
|
||||
if err != nil {
|
||||
return policy.Policy{}, zip.ErrBadRequest(err.Error())
|
||||
}
|
||||
tx, err := db.Begin()
|
||||
if err != nil {
|
||||
return policy.Policy{}, err
|
||||
}
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
var last sql.NullInt64
|
||||
if err := tx.QueryRow(`SELECT MAX(version) FROM policy WHERE stage = ?`, sealed.Stage).Scan(&last); err != nil {
|
||||
return policy.Policy{}, err
|
||||
}
|
||||
sealed.Version = int(last.Int64) + 1
|
||||
sealed.At = time.Now().UTC()
|
||||
// Digest is over what the ladder DECIDES, so it does not move with the
|
||||
// version or the timestamp. Re-sealing after stamping those would be a no-op
|
||||
// and is not done, which is what keeps two identical ladders comparable.
|
||||
|
||||
body, err := json.Marshal(sealed)
|
||||
if err != nil {
|
||||
return policy.Policy{}, err
|
||||
}
|
||||
if _, err := tx.Exec(
|
||||
`INSERT INTO policy (stage, version, at, by, reason, digest, body) VALUES (?,?,?,?,?,?,?)`,
|
||||
sealed.Stage, sealed.Version, stamp(sealed.At), sealed.By, sealed.Reason, sealed.Digest, string(body),
|
||||
); err != nil {
|
||||
return policy.Policy{}, err
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return policy.Policy{}, err
|
||||
}
|
||||
return sealed, nil
|
||||
}
|
||||
|
||||
// policyVersions is the audit trail for one stage, newest first.
|
||||
func policyVersions(db *sql.DB, stage string, limit int) ([]policy.Policy, error) {
|
||||
rows, err := db.Query(
|
||||
`SELECT body FROM policy WHERE stage = ? ORDER BY version DESC LIMIT ?`, stage, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer func() { _ = rows.Close() }()
|
||||
var out []policy.Policy
|
||||
for rows.Next() {
|
||||
var body string
|
||||
if err := rows.Scan(&body); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var p policy.Policy
|
||||
if err := json.Unmarshal([]byte(body), &p); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, p)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// policyStages lists the stages this tenant has ever set a ladder for.
|
||||
func policyStages(db *sql.DB) ([]string, error) {
|
||||
rows, err := db.Query(`SELECT DISTINCT stage FROM policy ORDER BY stage`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer func() { _ = rows.Close() }()
|
||||
var out []string
|
||||
for rows.Next() {
|
||||
var s string
|
||||
if err := rows.Scan(&s); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, s)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// ── the calibration record ──────────────────────────────────────────────────
|
||||
|
||||
// currentCalibration reads the map in force: the highest version. One
|
||||
// calibration per tenant rather than one per stage — the map turns THIS model's
|
||||
// score into a probability and the model is per tenant, so a per-stage split
|
||||
// would divide the same evidence into thinner samples of the same relationship.
|
||||
func currentCalibration(db *sql.DB) (calibrate.Map, int, time.Time, bool, error) {
|
||||
var version, horizon int
|
||||
var at, body string
|
||||
err := db.QueryRow(
|
||||
`SELECT version, at, horizon, body FROM calibration ORDER BY version DESC LIMIT 1`,
|
||||
).Scan(&version, &at, &horizon, &body)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return calibrate.Map{}, 0, time.Time{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return calibrate.Map{}, 0, time.Time{}, false, err
|
||||
}
|
||||
var m calibrate.Map
|
||||
if err := json.Unmarshal([]byte(body), &m); err != nil {
|
||||
return calibrate.Map{}, 0, time.Time{}, false, err
|
||||
}
|
||||
return m, version, unstamp(at), true, nil
|
||||
}
|
||||
|
||||
// putCalibration records a fit as the next version.
|
||||
func putCalibration(db *sql.DB, m calibrate.Map, by string, horizon int) (int, time.Time, error) {
|
||||
body, err := json.Marshal(m)
|
||||
if err != nil {
|
||||
return 0, time.Time{}, err
|
||||
}
|
||||
tx, err := db.Begin()
|
||||
if err != nil {
|
||||
return 0, time.Time{}, err
|
||||
}
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
var last sql.NullInt64
|
||||
if err := tx.QueryRow(`SELECT MAX(version) FROM calibration`).Scan(&last); err != nil {
|
||||
return 0, time.Time{}, err
|
||||
}
|
||||
version := int(last.Int64) + 1
|
||||
at := time.Now().UTC()
|
||||
if _, err := tx.Exec(
|
||||
`INSERT INTO calibration (version, at, by, shape, horizon, digest, body) VALUES (?,?,?,?,?,?,?)`,
|
||||
version, stamp(at), by, m.Shape, horizon, m.Digest, string(body),
|
||||
); err != nil {
|
||||
return 0, time.Time{}, err
|
||||
}
|
||||
return version, at, tx.Commit()
|
||||
}
|
||||
|
||||
// ── the verdict record ──────────────────────────────────────────────────────
|
||||
|
||||
// getVerdict reads one decision's grading back, so a repeat of an idempotent
|
||||
// decide returns the same reasons rather than a decision with none.
|
||||
func getVerdict(db *sql.DB, id string) (verdict, bool, error) {
|
||||
var p sql.NullFloat64
|
||||
var cal, pol, reasons, refusal string
|
||||
err := db.QueryRow(
|
||||
`SELECT probability, calibration, policy, reasons, refusal FROM verdict WHERE decision = ?`, id,
|
||||
).Scan(&p, &cal, &pol, &reasons, &refusal)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return verdict{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return verdict{}, false, err
|
||||
}
|
||||
v := verdict{calibration: cal, policy: pol, refusal: refusal}
|
||||
if p.Valid {
|
||||
val := p.Float64
|
||||
v.probability = &val
|
||||
}
|
||||
if err := json.Unmarshal([]byte(reasons), &v.reasons); err != nil {
|
||||
return verdict{}, false, err
|
||||
}
|
||||
return v, true, nil
|
||||
}
|
||||
|
||||
// graded is how every READ path asks for a decision's grading: the verdict, or a
|
||||
// verdict that SAYS there is none.
|
||||
//
|
||||
// The not-found used to be discarded with `_`, which made an ungraded decision
|
||||
// indistinguishable from a graded one with nothing to say — so a block could be
|
||||
// served on the dispute-packet surface with no probability, no principal reasons
|
||||
// and no refusal. The rows are written in one transaction now, so this should be
|
||||
// unreachable; a record plane that answered "nothing to say" for a state it
|
||||
// believes impossible would be hiding exactly the corruption worth knowing about.
|
||||
func graded(db *sql.DB, id string) (verdict, error) {
|
||||
v, ok, err := getVerdict(db, id)
|
||||
if err != nil {
|
||||
return verdict{}, err
|
||||
}
|
||||
if !ok {
|
||||
return verdict{refusal: "this decision has no recorded grading, so there is no probability, " +
|
||||
"no principal reason and nothing this plane can defend it with"}, nil
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
// ── the replay record ───────────────────────────────────────────────────────
|
||||
|
||||
func putReplay(db *sql.DB, id string, at time.Time, by string, rep quality.Report) error {
|
||||
body, err := json.Marshal(rep)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = db.Exec(`INSERT INTO replay (id, at, by, digest, body) VALUES (?,?,?,?,?)`,
|
||||
id, stamp(at), by, rep.Digest, string(body))
|
||||
return err
|
||||
}
|
||||
|
||||
func getReplay(db *sql.DB, id string) (quality.Report, time.Time, bool, error) {
|
||||
var at, body string
|
||||
err := db.QueryRow(`SELECT at, body FROM replay WHERE id = ?`, id).Scan(&at, &body)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return quality.Report{}, time.Time{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return quality.Report{}, time.Time{}, false, err
|
||||
}
|
||||
var rep quality.Report
|
||||
if err := json.Unmarshal([]byte(body), &rep); err != nil {
|
||||
return quality.Report{}, time.Time{}, false, err
|
||||
}
|
||||
return rep, unstamp(at), true, nil
|
||||
}
|
||||
|
||||
// ── reading the history back ────────────────────────────────────────────────
|
||||
|
||||
// maxHistory bounds every read of the decision log for measurement.
|
||||
//
|
||||
// The same bound the search sandbox uses, and for the same reason: this runs
|
||||
// in-request against a single-writer file, so an unbounded scan is a tenant
|
||||
// holding the only connection its own decisions need.
|
||||
const maxHistory = 50_000
|
||||
|
||||
// evidence is one bounded read of the decision log: the rows a measurement was
|
||||
// computed from, and the two facts that say what the bounds left out. Both are
|
||||
// reported on every answer, because a measurement is only as good as the sample
|
||||
// it saw and a sample silently cut reads as a complete one.
|
||||
type evidence struct {
|
||||
// history is the rows, OLDEST FIRST, ready to measure over.
|
||||
history []quality.Recorded
|
||||
// truncated says the bound cut the read: there are older mature decisions
|
||||
// under this shape that no number here was computed from.
|
||||
truncated bool
|
||||
// superseded counts the mature judged decisions excluded because they were
|
||||
// scored under a DIFFERENT coordinate system. It is the evidence a refit
|
||||
// cannot honestly use, and the number that says why a fit went thin after a
|
||||
// governed change.
|
||||
superseded int
|
||||
}
|
||||
|
||||
// recorded reads this tenant's decision log as measurable observations, under
|
||||
// ONE scoring shape, restricted to decisions old enough for their outcome to
|
||||
// have arrived and bounded to the most recent of them.
|
||||
//
|
||||
// THE MATURITY HORIZON IS THE WHOLE OF WHY THIS TAKES A PARAMETER. Analysts
|
||||
// judge within hours; a card network dispute lands 30 to 120 days after the
|
||||
// transaction it disputes. Measure over everything and the recent tail is
|
||||
// enriched with analyst clearances and stripped of the chargebacks that have not
|
||||
// arrived yet, so the prevalence is understated and every threshold derived from
|
||||
// it sits too high. Excluding decisions younger than the horizon is what makes
|
||||
// the judged set representative rather than merely recent.
|
||||
//
|
||||
// THE SHAPE IS THE OTHER FILTER AND IT IS NOT OPTIONAL. A score is a coordinate.
|
||||
// Rows scored before a rule was written, retired, reweighted or muted are
|
||||
// coordinates in a system that no longer exists, and a fit taken over them and
|
||||
// stamped with today's shape is precisely the training–serving skew the shape
|
||||
// gate exists to refuse — arrived at from the inside, by the remedy the gate's
|
||||
// own message recommends. Filtering here means a refit after a governed change
|
||||
// finds thin evidence and SAYS SO, instead of re-blessing history it cannot read.
|
||||
//
|
||||
// THE BOUND TAKES THE MOST RECENT ROWS. Ascending plus LIMIT takes the oldest,
|
||||
// which past the bound freezes every fit, every measurement and every replay on
|
||||
// the dawn of the tenant's log forever. The read is descending and the slice is
|
||||
// reversed once, so what comes back is the LATEST maxHistory decisions in time
|
||||
// order — and truncated says when there were more.
|
||||
//
|
||||
// The ordering is on the second-truncated timestamp and then the id, which is a
|
||||
// TOTAL order: the stored form is RFC 3339 with a variable-length fraction, so a
|
||||
// plain string comparison sorts "…:00.5Z" before "…:00Z". At the day scale a
|
||||
// horizon works on that is immaterial, but a learning curve that split its
|
||||
// history on it would not be reproducible, and reproducibility is the property
|
||||
// being sold.
|
||||
func recorded(ctx context.Context, db *sql.DB, horizon int, shape string, limit int) (evidence, error) {
|
||||
if limit <= 0 || limit > maxHistory {
|
||||
limit = maxHistory
|
||||
}
|
||||
cutoff := time.Now().UTC().AddDate(0, 0, -horizon).Format("2006-01-02T15:04:05")
|
||||
|
||||
// The scan is its own scope, and that is load-bearing rather than tidy: the
|
||||
// org file is opened with ONE connection, so an open *sql.Rows holds the only
|
||||
// one there is. Reading limit+1 and stopping early leaves it open, and the
|
||||
// count below would then wait for a connection this function is itself
|
||||
// holding — a deadlock, not an error, bounded by nothing.
|
||||
scan := func() ([]quality.Recorded, bool, error) {
|
||||
// limit+1 answers "was there more" exactly, and for free — a COUNT over the
|
||||
// same predicate is a second scan of the same rows to learn one bit.
|
||||
rows, err := db.QueryContext(ctx,
|
||||
`SELECT id, at, action, score, label FROM decision
|
||||
WHERE substr(at,1,19) <= ? AND shape = ?
|
||||
ORDER BY substr(at,1,19) DESC, id DESC LIMIT ?`, cutoff, shape, limit+1)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
out := make([]quality.Recorded, 0, limit)
|
||||
truncated := false
|
||||
for rows.Next() {
|
||||
if len(out) == limit {
|
||||
truncated = true
|
||||
break
|
||||
}
|
||||
var id, at, action, label string
|
||||
var score float64
|
||||
if err := rows.Scan(&id, &at, &action, &score, &label); err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
out = append(out, quality.Recorded{
|
||||
Observation: quality.Observation{
|
||||
ID: id, At: unstamp(at), Score: score, Action: action,
|
||||
Disposition: dispositions[strings.ToLower(label)],
|
||||
},
|
||||
// Every judgement this plane holds today came from a person reviewing
|
||||
// a decision. The below-the-line sample arm the engine already selects
|
||||
// is not yet wired to a label source, so the honest exploration share
|
||||
// is zero and Replay reports it as such rather than assuming it away.
|
||||
Source: "review",
|
||||
})
|
||||
}
|
||||
return out, truncated, rows.Err()
|
||||
}
|
||||
|
||||
out, truncated, err := scan()
|
||||
if err != nil {
|
||||
return evidence{}, err
|
||||
}
|
||||
// Back into time order: every consumer of a history is temporal — the
|
||||
// learning curve splits it, the replay walks it, the report bounds it.
|
||||
for i, j := 0, len(out)-1; i < j; i, j = i+1, j-1 {
|
||||
out[i], out[j] = out[j], out[i]
|
||||
}
|
||||
w := evidence{history: out, truncated: truncated}
|
||||
|
||||
if err := db.QueryRowContext(ctx,
|
||||
`SELECT COUNT(*) FROM decision WHERE label != '' AND substr(at,1,19) <= ? AND shape != ?`,
|
||||
cutoff, shape).Scan(&w.superseded); err != nil {
|
||||
return evidence{}, err
|
||||
}
|
||||
return w, nil
|
||||
}
|
||||
|
||||
// immature counts the labelled decisions the horizon excluded. It is reported
|
||||
// beside every fit and every measurement, because a horizon that quietly drops
|
||||
// most of the evidence is the difference between a thin answer and a wrong one.
|
||||
func immature(ctx context.Context, db *sql.DB, horizon int) (int, error) {
|
||||
cutoff := time.Now().UTC().AddDate(0, 0, -horizon).Format("2006-01-02T15:04:05")
|
||||
var n int
|
||||
err := db.QueryRowContext(ctx,
|
||||
`SELECT COUNT(*) FROM decision WHERE label != '' AND substr(at,1,19) > ?`, cutoff).Scan(&n)
|
||||
return n, err
|
||||
}
|
||||
|
||||
// samples reduces recorded observations to what a calibration fit reads.
|
||||
// Unjudged rows are dropped rather than counted as unproductive: a fit over
|
||||
// "unknown means fine" is a fit against the incumbent policy, not against the
|
||||
// world.
|
||||
func samples(history []quality.Recorded) []calibrate.Sample {
|
||||
out := make([]calibrate.Sample, 0, len(history))
|
||||
for _, h := range history {
|
||||
switch h.Disposition {
|
||||
case replay.Productive:
|
||||
out = append(out, calibrate.Sample{Score: h.Score, Productive: true})
|
||||
case replay.Unproductive:
|
||||
out = append(out, calibrate.Sample{Score: h.Score})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// unacted is the share of the judged evidence that came from decisions the
|
||||
// policy LET THROUGH.
|
||||
//
|
||||
// It is the honest limit on every number measured from this log. A blocked
|
||||
// decision has no outcome — the counterfactual is unobserved and no arithmetic
|
||||
// recovers it — so a judged set made entirely of decisions the incumbent chose
|
||||
// to act on measures agreement with the incumbent rather than accuracy. Near
|
||||
// zero here means exactly that, and it is reported rather than assumed away.
|
||||
func unacted(history []quality.Recorded) (float64, int) {
|
||||
var judged, allowed int
|
||||
for _, h := range history {
|
||||
if h.Disposition != replay.Productive && h.Disposition != replay.Unproductive {
|
||||
continue
|
||||
}
|
||||
judged++
|
||||
if h.Action == ActionAllow {
|
||||
allowed++
|
||||
}
|
||||
}
|
||||
if judged == 0 {
|
||||
return 0, 0
|
||||
}
|
||||
return round4(float64(allowed) / float64(judged)), judged
|
||||
}
|
||||
@@ -0,0 +1,352 @@
|
||||
package risk
|
||||
|
||||
// learn.go is the model plane: train, exhaustive-search, score — on the AI
|
||||
// cloud's own substrate, in process, per tenant.
|
||||
//
|
||||
// WHERE THE MODELS TRAIN, AND WHY NOT THE ALTERNATIVES.
|
||||
//
|
||||
// - hanzoai/ai is the LLM router. It has no trainer.
|
||||
// - candle is Rust and cloud does not link it.
|
||||
// - Kubeflow (/v1/train/jobs, /v1/train/experiments) is real and stays the
|
||||
// escape hatch for a customer who wants a heavy supervised sweep — but
|
||||
// apps/ml fails closed on it today because no kserve/katib/trainer chart is
|
||||
// deployed. It is not a substrate this product can stand on.
|
||||
//
|
||||
// So: luxfi/aml's half-space trees, in this process. That is not a placeholder —
|
||||
// it is the right shape for the problem, for five reasons that a neural
|
||||
// alternative cannot offer:
|
||||
//
|
||||
// 1. NO TRAINING PASS AND NO RETAINED SAMPLE. The geometry is built BEFORE any
|
||||
// data arrives; the model IS a set of mass counters. Training is one online
|
||||
// increment per observation, so there is no job, no queue and no window in
|
||||
// which the tenant is protected by a stale model.
|
||||
// 2. PER TENANT INCLUDING THE GEOMETRY. The seed is mix(cfg.Seed, orgID), so two
|
||||
// tenants do not merely hold different counters — they hold DIFFERENT TREES.
|
||||
// Probing one reveals nothing about where another's regions lie.
|
||||
// 3. BOUNDED, PER TENANT. 336 KB of forest per tenant at the defaults, and a
|
||||
// tenant's aggregates are capped by its OWN budget (bound.go) — one store per
|
||||
// tenant, so an eviction can only ever drop a key that tenant put there. A
|
||||
// tenant that goes idle costs nothing and one that comes back re-warms.
|
||||
// 4. ATTRIBUTION IS A COUNTERFACTUAL ON THE MODEL THAT RAISED THE ALERT. Move
|
||||
// one coordinate to its neutral value, rescore, and the drop IS that
|
||||
// feature's contribution. No second explainer model, and therefore no second
|
||||
// thing that can be wrong. That is what makes a decline defensible.
|
||||
// 5. IT CANNOT ACT ALONE. Evidence is capped at review; weight is non-negative;
|
||||
// NaN and Inf are refused.
|
||||
//
|
||||
// EXHAUSTIVE SEARCH WITHOUT KUBEFLOW. luxfi/aml's replay package already replays
|
||||
// a candidate over real history through the engine's own evaluator, writing
|
||||
// nothing — it reaches the evaluator through a one-method interface and history
|
||||
// through another, and imports no store, so a dry run is STRUCTURALLY dry.
|
||||
// mlSearch generalises that from one rule candidate to a grid over the model
|
||||
// topology, replays each candidate over the tenant's own recent decisions, and
|
||||
// returns the learning curve and the winning topology. Native Go, per tenant, no
|
||||
// GPU, no CRD.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"math"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/aml/pkg/anomaly"
|
||||
"github.com/luxfi/aml/pkg/types"
|
||||
)
|
||||
|
||||
// snapshotKey names the row a tenant's learned state is kept under. One row: a
|
||||
// tenant has one model, and a second row would be a second answer to one
|
||||
// question.
|
||||
const snapshotKey = "anomaly"
|
||||
|
||||
// keep writes a tenant's learned state into its own encrypted file.
|
||||
//
|
||||
// This is what makes a rollout survivable. cloud is strategy Recreate at one
|
||||
// replica, so every deploy drops the process — and with it every warming model
|
||||
// and the appetite threshold it had computed. Without this, every deploy
|
||||
// silently resets every tenant to warming, and a warming model REFUSES to score,
|
||||
// which reads as "clean" to anything that does not check Refusal.
|
||||
//
|
||||
// It takes the tenant's own db and the tenant's own model, because both come off
|
||||
// one cell: a keep that resolved either of them for itself could pair one
|
||||
// tenant's file with another tenant's state.
|
||||
func keep(db *sql.DB, t Tenant, model *anomaly.Store) error {
|
||||
if db == nil || model == nil {
|
||||
return nil
|
||||
}
|
||||
snap, ok := model.Snapshot(t.String())
|
||||
if !ok {
|
||||
return nil // nothing learned for this tenant; nothing to keep
|
||||
}
|
||||
body, err := json.Marshal(snap)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return putModel(db, snapshotKey, body)
|
||||
}
|
||||
|
||||
// restore reinstates a tenant's learned state into that tenant's own model.
|
||||
// A snapshot whose shape does not match the running inventory is REFUSED by the
|
||||
// engine, not coerced: state the model would treat as its own memory has to have
|
||||
// come from this algorithm over this feature set.
|
||||
func restore(db *sql.DB, model *anomaly.Store, t Tenant) error {
|
||||
body, err := getModel(db, snapshotKey)
|
||||
if err != nil {
|
||||
return nil // no snapshot is the normal first-run state, not a failure
|
||||
}
|
||||
var snap anomaly.Snapshot
|
||||
if err := json.Unmarshal(body, &snap); err != nil {
|
||||
return err
|
||||
}
|
||||
// The snapshot's tenant must be the tenant asking for it. The engine checks
|
||||
// this too; checking here as well means a restore of A's file into B's
|
||||
// request is refused at the boundary that knows who asked.
|
||||
if snap.OrgID != t.String() {
|
||||
return fmt.Errorf("risk: snapshot belongs to another tenant")
|
||||
}
|
||||
return model.Restore(snap)
|
||||
}
|
||||
|
||||
// ── exhaustive search ───────────────────────────────────────────────────────
|
||||
|
||||
// axis is one dimension of the topology grid, with the values the search may
|
||||
// try. Both are CLOSED: an unbounded grid is an unbounded amount of a shared
|
||||
// pod's CPU, reachable by anyone with a key.
|
||||
type axis struct {
|
||||
name string
|
||||
values []float64
|
||||
}
|
||||
|
||||
// grid is the search space. Trees x Depth x Window x Blend x Review is 3*3*3*3*3
|
||||
// = 243 candidates at the widest, which is seconds of one core over a few
|
||||
// thousand replayed decisions — bounded by construction rather than by a
|
||||
// timeout.
|
||||
func grid() []axis {
|
||||
return []axis{
|
||||
{"trees", []float64{15, 25, 40}},
|
||||
{"depth", []float64{6, 8, 10}},
|
||||
{"window", []float64{128, 256, 512}},
|
||||
{"blend", []float64{0.1, 0.25, 0.5}},
|
||||
{"review", []float64{0.005, 0.01, 0.02}},
|
||||
}
|
||||
}
|
||||
|
||||
// candidate is one point in the grid.
|
||||
type candidate struct {
|
||||
// Trees is how many half-space trees the model holds.
|
||||
Trees int `json:"trees"`
|
||||
// Depth is how deep each tree splits.
|
||||
Depth int `json:"depth"`
|
||||
// Window is how many observations make up one reference window.
|
||||
Window int `json:"window"`
|
||||
// Blend is how much of a closing window folds into the reference.
|
||||
Blend float64 `json:"blend"`
|
||||
// Review is the share of the stream this topology may send for examination.
|
||||
Review float64 `json:"review"`
|
||||
}
|
||||
|
||||
// trial is what one candidate did over the replayed history.
|
||||
type trial struct {
|
||||
// Candidate is the topology tried.
|
||||
Candidate candidate `json:"candidate"`
|
||||
// Scored is how many observations the model was able to score. A topology
|
||||
// that warms slowly scores fewer, which is a real cost and is reported.
|
||||
Scored int `json:"scored"`
|
||||
// Alerted is how many of those it would have alerted on.
|
||||
Alerted int `json:"alerted"`
|
||||
// Realised is Alerted/Scored — the share actually reached, against the
|
||||
// Review share intended. The gap between the two IS the governance report.
|
||||
Realised float64 `json:"realised"`
|
||||
// Separation is the mean score of the alerted set minus the mean score of
|
||||
// the rest. It is the ranking objective: a topology that separates the tail
|
||||
// from the body is doing the job whatever its absolute scores look like.
|
||||
Separation float64 `json:"separation"`
|
||||
// Warm is how many observations passed before the model would score at all.
|
||||
Warm int `json:"warm"`
|
||||
}
|
||||
|
||||
// searchReport is the whole answer.
|
||||
type searchReport struct {
|
||||
// Events is how many historical observations were replayed.
|
||||
Events int `json:"events"`
|
||||
// From and To are the period they span.
|
||||
From time.Time `json:"from,omitzero"`
|
||||
To time.Time `json:"to,omitzero"`
|
||||
// Trials is every candidate tried, best first.
|
||||
Trials []trial `json:"trials"`
|
||||
// Winner is the best-separating topology that also honoured its stated
|
||||
// appetite. Absent when no candidate did both.
|
||||
Winner *candidate `json:"winner,omitempty"`
|
||||
// Curve is the learning curve of the winner: separation as a function of how
|
||||
// much of the history it had seen, in ten steps.
|
||||
Curve []float64 `json:"curve,omitempty"`
|
||||
// Refusal names why a report is empty when it is. An empty report and a
|
||||
// report of no alerts are opposite facts.
|
||||
Refusal string `json:"refusal,omitempty"`
|
||||
}
|
||||
|
||||
// errNoHistory is what a search returns instead of a clean-looking zero. replay
|
||||
// refuses an empty history for exactly this reason: "no alerts" is what a quiet
|
||||
// rule and an unrun rule both look like, and the difference is the whole reason
|
||||
// a sandbox exists.
|
||||
var errNoHistory = fmt.Errorf("risk: no history to replay, so a result would be indistinguishable from a quiet model")
|
||||
|
||||
// searchRun replays every candidate over one tenant's own recorded observations.
|
||||
//
|
||||
// Nothing is written and nothing outside this function's own scratch models is
|
||||
// touched: each candidate gets a FRESH anomaly.Store over a FRESH velocity store
|
||||
// seeded from the tenant key, so a search can never move the live model's
|
||||
// counters, and two searches for two tenants can never see each other's.
|
||||
func searchRun(ctx context.Context, t Tenant, history []observation) (searchReport, error) {
|
||||
if len(history) == 0 {
|
||||
return searchReport{Refusal: errNoHistory.Error()}, errNoHistory
|
||||
}
|
||||
rep := searchReport{Events: len(history), From: history[0].at, To: history[len(history)-1].at}
|
||||
|
||||
for _, c := range candidates() {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return rep, ctx.Err()
|
||||
default:
|
||||
}
|
||||
tr, err := replayCandidate(t, c, history)
|
||||
if err != nil {
|
||||
continue // a topology the engine refuses is not a result, it is a non-candidate
|
||||
}
|
||||
rep.Trials = append(rep.Trials, tr)
|
||||
}
|
||||
sort.SliceStable(rep.Trials, func(i, j int) bool {
|
||||
return rep.Trials[i].Separation > rep.Trials[j].Separation
|
||||
})
|
||||
for i := range rep.Trials {
|
||||
// The winner must have SCORED something and must have honoured its stated
|
||||
// appetite within a factor of two. A topology that separates beautifully
|
||||
// while alerting on nothing is not a control.
|
||||
tr := rep.Trials[i]
|
||||
if tr.Scored == 0 || tr.Realised == 0 {
|
||||
continue
|
||||
}
|
||||
if tr.Realised > 2*tr.Candidate.Review {
|
||||
continue
|
||||
}
|
||||
w := tr.Candidate
|
||||
rep.Winner = &w
|
||||
rep.Curve = curve(t, w, history)
|
||||
break
|
||||
}
|
||||
if rep.Winner == nil {
|
||||
rep.Refusal = "no candidate both scored and honoured its stated appetite over this history"
|
||||
}
|
||||
return rep, nil
|
||||
}
|
||||
|
||||
// candidates expands the grid. Written as an explicit product so the count is
|
||||
// visible in the code rather than emergent from a recursion.
|
||||
func candidates() []candidate {
|
||||
g := grid()
|
||||
var out []candidate
|
||||
for _, trees := range g[0].values {
|
||||
for _, depth := range g[1].values {
|
||||
for _, win := range g[2].values {
|
||||
for _, blend := range g[3].values {
|
||||
for _, review := range g[4].values {
|
||||
out = append(out, candidate{
|
||||
Trees: int(trees), Depth: int(depth), Window: int(win),
|
||||
Blend: blend, Review: review,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// replayCandidate runs one topology over the history in a sandbox.
|
||||
//
|
||||
// The sandbox is built by the SAME two bounded constructors the live plane uses
|
||||
// (bound.go), so a search cannot be the one place a 100,000-key store shared by
|
||||
// everyone comes back — and the sandbox holds one tenant's replay, under that
|
||||
// tenant's own bound, exactly like the plane it is a model of.
|
||||
func replayCandidate(t Tenant, c candidate, history []observation) (trial, error) {
|
||||
vel := aggregates()
|
||||
model, err := forest(anomaly.Config{
|
||||
Trees: c.Trees, Depth: c.Depth, Window: c.Window, Blend: c.Blend,
|
||||
Appetite: anomaly.Appetite{Review: c.Review, Sample: 0.001},
|
||||
}, vel)
|
||||
if err != nil {
|
||||
return trial{}, err
|
||||
}
|
||||
|
||||
tr := trial{Candidate: c}
|
||||
var alerted, rest []float64
|
||||
for _, o := range history {
|
||||
record(vel, t, o)
|
||||
tx := types.Transaction{
|
||||
ID: o.id, OrgID: t.String(), UserID: o.subject, AccountID: o.subject,
|
||||
Currency: o.currency, Direction: o.direction,
|
||||
IPAddress: o.signals["ip"], DeviceFingerprint: o.signals["device"],
|
||||
Timestamp: o.at, USD: nanoUSD(o.amount),
|
||||
}
|
||||
a := model.Inspect(tx, types.Entity{ID: o.subject, OrgID: t.String()})
|
||||
// Inspect does not learn, so a second pass through the learning path is
|
||||
// what advances the model. judge(learn=true) is Assess; in shadow it
|
||||
// returns no hit, and the counters still move — which is exactly the
|
||||
// replay we want.
|
||||
_, _ = model.Assess(tx, types.Entity{ID: o.subject, OrgID: t.String()})
|
||||
if !a.Scored {
|
||||
tr.Warm++
|
||||
continue
|
||||
}
|
||||
tr.Scored++
|
||||
if a.Score >= a.Cut {
|
||||
tr.Alerted++
|
||||
alerted = append(alerted, a.Score)
|
||||
} else {
|
||||
rest = append(rest, a.Score)
|
||||
}
|
||||
}
|
||||
if tr.Scored > 0 {
|
||||
tr.Realised = round4(float64(tr.Alerted) / float64(tr.Scored))
|
||||
}
|
||||
tr.Separation = round4(mean(alerted) - mean(rest))
|
||||
return tr, nil
|
||||
}
|
||||
|
||||
// curve is the winner's separation as a function of how much history it had
|
||||
// seen. Ten steps: enough to see whether the model is still improving, few
|
||||
// enough that the answer is a chart and not a data set.
|
||||
func curve(t Tenant, c candidate, history []observation) []float64 {
|
||||
out := make([]float64, 0, 10)
|
||||
for i := 1; i <= 10; i++ {
|
||||
n := len(history) * i / 10
|
||||
if n == 0 {
|
||||
out = append(out, 0)
|
||||
continue
|
||||
}
|
||||
tr, err := replayCandidate(t, c, history[:n])
|
||||
if err != nil {
|
||||
out = append(out, 0)
|
||||
continue
|
||||
}
|
||||
out = append(out, tr.Separation)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func mean(xs []float64) float64 {
|
||||
if len(xs) == 0 {
|
||||
return 0
|
||||
}
|
||||
var s float64
|
||||
for _, x := range xs {
|
||||
s += x
|
||||
}
|
||||
v := s / float64(len(xs))
|
||||
if math.IsNaN(v) || math.IsInf(v, 0) {
|
||||
return 0
|
||||
}
|
||||
return v
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,381 @@
|
||||
// Package risk is Hanzo Risk: the scoring and decision plane for any entity a
|
||||
// tenant has — an account, a transaction, a session, an agent, a merchant, a
|
||||
// payout — and the shared model plane both it and the compliance face read.
|
||||
//
|
||||
// THREE VERBS, ONE CORE: decide, record, learn.
|
||||
//
|
||||
// /v1/risk decide + record. One hot op (POST /v1/risk/decide) answers
|
||||
// allow / challenge / review / restrict / block WITH A REASON, and
|
||||
// the planes around it (decisions, rules, lists, suppressions,
|
||||
// controls, dictionary, activity) are how a tenant governs it.
|
||||
// /v1/ml learn. train, exhaustive-search, score, plus the state a reviewer
|
||||
// reads and the snapshot an auditor pins.
|
||||
//
|
||||
// Fraud is a USE of /v1/risk, not a sibling of it, and neither is abuse, bots,
|
||||
// account takeover, spam or pay-as-you-go abuse. There is no /v1/fraud.
|
||||
//
|
||||
// WHY IT IS IN CLOUD AND NOT A SERVICE OF ITS OWN. Being in cloud is what earns
|
||||
// the five things a standalone engine would each have to grow: IAM-validated
|
||||
// identity, the tenant gate, usage metering, billing and structured logs. None
|
||||
// of them is free — every one is wired explicitly below, and the table in
|
||||
// Mount's comment says where.
|
||||
//
|
||||
// WHAT IS LINKED FROM THE ENGINE, AND WHAT IS DELIBERATELY NOT. github.com/luxfi/aml
|
||||
// is a MODULE DEPENDENCY, exactly as luxfi/kms and hanzoai/o11y are; no source is
|
||||
// vendored. Only its transitively base-free packages are linked — types,
|
||||
// velocity, anomaly, replay — because pkg/engine, pkg/measure and pkg/history all
|
||||
// reach github.com/hanzoai/base/core and github.com/hanzoai/tasks through
|
||||
// pkg/history's Base-backed store, and dragging an application framework and a
|
||||
// workflow engine into a payment-authorization path is not a trade worth making.
|
||||
// See rule.go for the measurement and the upstream fix.
|
||||
package risk
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/hanzoai/cloud"
|
||||
"github.com/hanzoai/cloud/apps/datastore"
|
||||
"github.com/hanzoai/cloud/openapi"
|
||||
"github.com/luxfi/aml/pkg/anomaly"
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// state is everything the surface reads. The per-tenant halves — the model's
|
||||
// counters, the velocity rings and the SQLite file — are SINGLE-WRITER by
|
||||
// nature, which is why this app must run under the shard router that pins an org
|
||||
// to one pod, and why Shutdown must snapshot.
|
||||
//
|
||||
// THE IN-MEMORY PLANES ARE NOT HERE, and that is the point. One velocity store
|
||||
// and one forest on this struct meant every tenant shared them under a GLOBAL
|
||||
// cap, so one org's volume evicted another org's counters and another org's
|
||||
// learned model. They live on the tenant's own cell now (store.go), each under
|
||||
// that tenant's own bound (bound.go).
|
||||
type state struct {
|
||||
// brand is the deployment's brand, the half of the tenant key that never
|
||||
// comes from a header.
|
||||
brand string
|
||||
// dataDir is where the per-tenant files live.
|
||||
dataDir string
|
||||
|
||||
// shelf is the bounded registry of live tenants: one cell each, holding that
|
||||
// tenant's file, its aggregates and its model.
|
||||
shelf *shelf
|
||||
// inflight is the per-tenant bound on measurement: one at a time, per tenant,
|
||||
// so a caller that loops the measurement surface degrades only itself.
|
||||
inflight *inflight
|
||||
// running is the per-tenant bound on the exhaustive search: one at a time,
|
||||
// per tenant, for the same reason and with the same shape.
|
||||
running *inflight
|
||||
|
||||
// digest is the model SHAPE — the feature inventory in order and the
|
||||
// detector's geometry parameters. It is a pure function of the configuration,
|
||||
// identical for every tenant, so it is settled once at boot rather than read
|
||||
// off whichever tenant's forest is at hand.
|
||||
digest string
|
||||
|
||||
// bill is the shared per-org gate and meter, on the "risk" product.
|
||||
bill *cloud.ResourceMeter
|
||||
|
||||
// warehouse records whether the feature tables were created. A false value
|
||||
// is an honest gap on the dictionary and the backfill, never a zero.
|
||||
mu sync.Mutex
|
||||
warehouse bool
|
||||
}
|
||||
|
||||
// Mount wires /v1/risk and the native /v1/ml leaves onto app.
|
||||
//
|
||||
// EVERY INHERITED CAPABILITY IS WIRED HERE, EXPLICITLY. Being embedded in cloud
|
||||
// makes each one AVAILABLE; none of them is automatic:
|
||||
//
|
||||
// IAM auth SanitizeIdentity mints X-Org-Id from the verified bearer.
|
||||
// Global (serve.go) — nothing to do here, and that is the point:
|
||||
// this app never validates a token and never can.
|
||||
// tenant gate cloud.Bridge() on EACH group, FIRST, before any leaf. A typed
|
||||
// op receives only a context; Bridge is what parks the validated
|
||||
// org and validated-ness in it. Installed after the leaves it
|
||||
// would never run — fiber orders middleware by registration.
|
||||
// org sub-scope principal.ValidatedProject, read in tenantOf.
|
||||
// meter + gate cloud.NewResourceMeter(deps, "risk"); rm.Gate before priced
|
||||
// work, rm.Meter after it. Wired per op, in typed.go.
|
||||
// logs cloud.NewBase(deps, "risk") gives the scoped luxlog.
|
||||
// traces global and already ZAP-native (OTLZ). This package imports no
|
||||
// otlp transport, deliberately.
|
||||
// health OwnsHealth on the plugin plus the real probe below.
|
||||
//
|
||||
// The route registration follows apps/ml exactly, including the one form
|
||||
// cmd/zipdoc can read: ONE `g := <router>.Group("/prefix")` per line.
|
||||
func Mount(app cloud.Router, deps cloud.Deps) error {
|
||||
if app == nil {
|
||||
return fmt.Errorf("risk.Mount: nil app")
|
||||
}
|
||||
s, err := build(deps)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// The warehouse is not on any request path this app serves, so a warehouse
|
||||
// that is down at boot must not stop the decision plane from mounting. The
|
||||
// tables are ensured once, in the background, and the honest gap is recorded.
|
||||
go warehouse(s)
|
||||
// Memory comes back from a tenant's OWN silence on a timer, so a quiet tenant
|
||||
// releases its rings without a busy one having to need them first.
|
||||
go reclaim(s)
|
||||
|
||||
mount(s, app)
|
||||
s.Log.Info("risk surface mounted",
|
||||
"brand", deps.Brand, "env", deps.Env,
|
||||
"billing", s.State.bill.Enabled(), "model", s.State.digest,
|
||||
"tenants", tenantMax(), "keys_per_tenant", maxKeys())
|
||||
|
||||
// Shutdown is registered by the plugin (plugin/risk/main.go) and calls back
|
||||
// here; holding the service in a package var would be a second owner of the
|
||||
// state, so the closure captures it instead.
|
||||
shutdown = func(context.Context) error { return teardown(s) }
|
||||
return nil
|
||||
}
|
||||
|
||||
// warehouse creates the two feature planes and then keeps the NETWORK BASELINE
|
||||
// current. It is the only background work this app does and it never touches a
|
||||
// request path.
|
||||
//
|
||||
// The baseline recompute is here, on a timer, rather than on any route — which
|
||||
// is what makes the cross-org surface unreachable by a caller. Nobody can time
|
||||
// it, steer it, or observe its cost, and the statement it runs is a package
|
||||
// constant with no placeholder, so nothing a caller sends reaches it. What it
|
||||
// writes carries no tenant, no subject and no pseudonym: quantiles over a
|
||||
// k-anonymous set of contributing orgs, and nothing else.
|
||||
func warehouse(s *stateService) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
if err := ensureTables(ctx); err != nil {
|
||||
cancel()
|
||||
s.Log.Warn("risk: feature tables unavailable; the dictionary, the peer comparison and the search sandbox will report an honest gap", "err", err)
|
||||
return
|
||||
}
|
||||
cancel()
|
||||
s.State.mu.Lock()
|
||||
s.State.warehouse = true
|
||||
s.State.mu.Unlock()
|
||||
|
||||
for {
|
||||
bg, stop := context.WithTimeout(context.Background(), 10*time.Minute)
|
||||
if err := publishBaseline(bg); err != nil {
|
||||
s.Log.Warn("risk: the network baseline was not recomputed; peer comparison holds its last published day", "err", err)
|
||||
}
|
||||
stop()
|
||||
time.Sleep(baselineEvery)
|
||||
}
|
||||
}
|
||||
|
||||
// baselineEvery is how often the network quantiles are recomputed. Six hours:
|
||||
// the statement covers whole days, so anything faster republishes the same
|
||||
// numbers, and anything slower lets a day go unpublished after a restart.
|
||||
const baselineEvery = 6 * time.Hour
|
||||
|
||||
// reclaim retires tenants that have gone silent for longer than their rings hold
|
||||
// anything. It is the only thing in this process that releases a tenant's
|
||||
// memory, and it is driven by that tenant's own idleness — never by another
|
||||
// tenant's arrival, which is what makes it reclaim rather than eviction.
|
||||
func reclaim(s *stateService) {
|
||||
for {
|
||||
time.Sleep(sweepEvery)
|
||||
s.State.shelf.sweep()
|
||||
}
|
||||
}
|
||||
|
||||
// sweepEvery is how often the reclaim runs. It is far shorter than the idleness
|
||||
// it looks for, so the granularity of "when memory comes back" is minutes rather
|
||||
// than a multiple of the threshold.
|
||||
const sweepEvery = 10 * time.Minute
|
||||
|
||||
// build constructs the state. Separate from Mount because Mount also starts the
|
||||
// background work and installs the routes, and a test wants the state without
|
||||
// either — the same split apps/ml makes for the same reason.
|
||||
func build(deps cloud.Deps) (*stateService, error) {
|
||||
if deps.Logger == nil {
|
||||
return nil, fmt.Errorf("risk.Mount: nil deps.Logger")
|
||||
}
|
||||
if deps.Brand == "" {
|
||||
// The brand is half the tenant key. A deployment that did not state one
|
||||
// cannot mint a key, and every request would 403 — better to say so at
|
||||
// boot than once per request.
|
||||
return nil, fmt.Errorf("risk.Mount: no brand, so no tenant key can be minted")
|
||||
}
|
||||
|
||||
// The model SHAPE, off a forest that will never hold a tenant. The digest is
|
||||
// a pure function of the configuration — the inventory in order and the
|
||||
// geometry parameters — so it can be settled once at boot, and reading it off
|
||||
// a throwaway is better than keeping a spare store around that something
|
||||
// could later be tempted to score with.
|
||||
//
|
||||
// SHADOW IS THE DEFAULT AT THE ENGINE TOO (forest forces it), and the
|
||||
// per-tenant switch in the record plane is what turns a tenant live. Two gates
|
||||
// in series rather than one: a deployment-wide flag flipped by mistake still
|
||||
// cannot make a tenant act.
|
||||
shape, err := forest(anomaly.Config{}, aggregates())
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("risk.Mount: %w", err)
|
||||
}
|
||||
|
||||
base := cloud.NewBase(deps, "risk")
|
||||
return &cloud.Service[state]{
|
||||
Base: base,
|
||||
State: state{
|
||||
brand: deps.Brand,
|
||||
dataDir: deps.DataDir,
|
||||
shelf: newShelf(base),
|
||||
inflight: newInflight("a measurement"),
|
||||
running: newInflight("an exhaustive search"),
|
||||
digest: shape.Digest(),
|
||||
bill: cloud.NewResourceMeter(deps, "risk"),
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
// stateService is the concrete service this package builds. An alias, so the
|
||||
// helpers below read as functions of the service and no second type exists.
|
||||
type stateService = cloud.Service[state]
|
||||
|
||||
// shutdown is set by Mount and read by the plugin. A nil value means the app
|
||||
// never mounted, and tearing down what never came up is a no-op rather than a
|
||||
// panic.
|
||||
var shutdown func(context.Context) error
|
||||
|
||||
// Shutdown snapshots every resident tenant's model and closes the record planes.
|
||||
//
|
||||
// THIS IS NOT HOUSEKEEPING. cloud deploys with strategy Recreate at one replica:
|
||||
// every rollout is a hard all-endpoint window AND drops every in-memory model.
|
||||
// A model that comes back with nothing learned declines to score for the whole
|
||||
// warm period, and a control that is off for however long that takes is a
|
||||
// control that was off — reported, if anyone reads Refusal, and silent if not.
|
||||
func Shutdown(ctx context.Context) error {
|
||||
if shutdown == nil {
|
||||
return nil
|
||||
}
|
||||
return shutdown(ctx)
|
||||
}
|
||||
|
||||
func teardown(s *stateService) error {
|
||||
kept, failed := s.State.shelf.close()
|
||||
s.Log.Info("risk surface down", "models_kept", kept, "models_lost", failed)
|
||||
if failed > 0 {
|
||||
return fmt.Errorf("risk: %d tenant model(s) could not be snapshotted", failed)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// tenantCell resolves the caller's scope and its live state: its own file, its
|
||||
// own aggregates and its own model, restored from its own snapshot the first
|
||||
// time this process arms it. Every typed op starts here, so the tenant is
|
||||
// established in one place, admitted in one place and armed in one place.
|
||||
func tenantCell(ctx context.Context, s *stateService) (scope, *cell, error) {
|
||||
sc, err := tenantOf(ctx, s.State.brand)
|
||||
if err != nil {
|
||||
return scope{}, nil, err
|
||||
}
|
||||
// The mint's own shape check, asserted at the boundary rather than assumed.
|
||||
// A regression here is the whole product: the key is the store index, the
|
||||
// history org column and the model's tree seed at once.
|
||||
if !qualified(s.State.brand, sc.tenant) {
|
||||
return scope{}, nil, zip.ErrForbidden("the tenant key is not qualified")
|
||||
}
|
||||
c, err := s.State.shelf.of(sc.tenant)
|
||||
if err != nil {
|
||||
return scope{}, nil, err
|
||||
}
|
||||
return sc, c, nil
|
||||
}
|
||||
|
||||
// tenantState is the file-only door for the ops that touch the record plane and
|
||||
// not the two in-memory planes.
|
||||
func tenantState(ctx context.Context, s *stateService) (scope, *sql.DB, error) {
|
||||
sc, c, err := tenantCell(ctx, s)
|
||||
if err != nil {
|
||||
return scope{}, nil, err
|
||||
}
|
||||
return sc, c.db, nil
|
||||
}
|
||||
|
||||
// health is the app's real, fail-closed probe.
|
||||
//
|
||||
// UNTYPED BY DESIGN. A real probe answers 503 CARRYING THE DEGRADED REPORT AS
|
||||
// ITS BODY, and a typed op reaches a non-2xx only by returning an error, which
|
||||
// zip renders as its own envelope — dropping exactly the report the probe
|
||||
// exists to deliver. Same reason apps/ml keeps its two health routes raw.
|
||||
func health(s *stateService) func(*zip.Ctx) error {
|
||||
return func(c *zip.Ctx) error {
|
||||
s.State.mu.Lock()
|
||||
warehouse := s.State.warehouse
|
||||
s.State.mu.Unlock()
|
||||
tenants, refused, refusedAt := s.State.shelf.count()
|
||||
|
||||
report := map[string]any{
|
||||
"status": "ok",
|
||||
"model": s.State.digest,
|
||||
"tenants": tenants,
|
||||
"capacity": tenantMax(),
|
||||
// strained COUNTS the tenants whose own aggregates are at their own
|
||||
// cardinality bound, so a count they read may under-state their own
|
||||
// traffic. It is on the probe because a partial ring reads exactly like
|
||||
// a quiet one, and nobody goes looking for a control that went quiet —
|
||||
// and it is a NUMBER, because this route is unauthenticated by design
|
||||
// and a tenant roster served to an anonymous GET is a customer list.
|
||||
// The tenant learns about its OWN ring on its own scoped surface,
|
||||
// GET /v1/ml/state.
|
||||
"strained": s.State.shelf.strained(),
|
||||
"warehouse": warehouse || datastore.Ready(),
|
||||
"billing": s.State.bill.Enabled(),
|
||||
}
|
||||
// The DECISION plane is what this app promises, and it does not need the
|
||||
// warehouse: rings are in memory, rules and lists are the tenant's own
|
||||
// file, the model is in process. The probe is degraded only when
|
||||
// something the hot path actually needs is missing.
|
||||
if s.State.shelf == nil || s.State.dataDir == "" {
|
||||
report["status"] = "degraded"
|
||||
report["error"] = "the decision plane is not constructed"
|
||||
return c.JSON(http.StatusServiceUnavailable, report)
|
||||
}
|
||||
// A REFUSED ADMISSION IS A PAGE, not a log line. This pod turned a tenant
|
||||
// away rather than evict an incumbent, which is the right answer and also
|
||||
// an outage for whoever was turned away — so it degrades the probe until
|
||||
// an operator shards or raises the ceiling.
|
||||
if refused > 0 && time.Since(refusedAt) < capacityAlarm {
|
||||
report["status"] = "degraded"
|
||||
report["refused"] = refused
|
||||
report["refused_at"] = refusedAt.UTC().Format(time.RFC3339)
|
||||
report["error"] = "this node is at its tenant ceiling and is refusing new tenants; shard, or raise RISK_TENANTS to what the pod's memory allows"
|
||||
return c.JSON(http.StatusServiceUnavailable, report)
|
||||
}
|
||||
return c.JSON(http.StatusOK, report)
|
||||
}
|
||||
}
|
||||
|
||||
// capacityAlarm is how long a refusal keeps the probe degraded. Three sweeps:
|
||||
// long enough that a refusal cannot be missed between two scrapes, short enough
|
||||
// that a pod which has since reclaimed room reports itself healthy again.
|
||||
const capacityAlarm = 3 * sweepEvery
|
||||
|
||||
// The prose for the ONE route above that is untyped by design. zipdoc lifts
|
||||
// prose from a typed handler's doc comment, and this is not one — so without a
|
||||
// Describe the probe would publish an operationId and nothing else. Declared
|
||||
// beside the wire fact it belongs to.
|
||||
func init() {
|
||||
openapi.Describe("/v1/risk/health", http.MethodGet,
|
||||
"Report whether the decision plane can decide",
|
||||
"Answers 200 with a report of what is up, or 503 CARRYING THE SAME REPORT as its "+
|
||||
"body — which is the whole point of a real probe, and the reason this one route is "+
|
||||
"not a typed op: a typed op reaches a non-2xx only by returning an error, and that "+
|
||||
"renders as an envelope with the report dropped.\n\n"+
|
||||
"The report names the model digest in force, how many tenants this process holds "+
|
||||
"resident, whether the feature warehouse is reachable, and whether metering is "+
|
||||
"configured. Only the first is load-bearing for a decision: the rings are in "+
|
||||
"memory, the rules and lists are the tenant's own file, and the model is in "+
|
||||
"process, so a decision does NOT need the warehouse and the probe stays green "+
|
||||
"without it. A warehouse that is down costs the field dictionary and the search "+
|
||||
"sandbox, and is reported as exactly that rather than as a failure of the "+
|
||||
"authorization path.")
|
||||
}
|
||||
@@ -0,0 +1,474 @@
|
||||
package risk
|
||||
|
||||
// rule.go is the risk rule algebra: a CLOSED set of fields and a CLOSED set of
|
||||
// comparisons, evaluated against one decision's facts.
|
||||
//
|
||||
// WHY IT IS NOT luxfi/aml's expr-lang evaluator, stated with the measurement.
|
||||
// The design this app was built from claimed pkg/engine is base-free. It is not.
|
||||
// Measured with `go list -deps`:
|
||||
//
|
||||
// pkg/engine -> pkg/history -> pkg/store -> github.com/hanzoai/base/core
|
||||
// -> github.com/hanzoai/tasks (Temporal SDK)
|
||||
// -> github.com/hanzoai/csqlite, hanzoai/sqlite
|
||||
//
|
||||
// A grep for `hanzoai/base` in pkg/engine's own files finds nothing, which is
|
||||
// how the claim was arrived at; the import is TRANSITIVE, through the measures
|
||||
// in scope.go. Linking it would pull the whole Base application framework and a
|
||||
// workflow engine into a binary whose job is to answer a payment-authorization
|
||||
// call in single-digit milliseconds — and cloud's image carries a SQLITE-GATE
|
||||
// that fails the BUILD, not the tests, when a per-app binary drags a second
|
||||
// sqlite driver in under CGO.
|
||||
//
|
||||
// So the risk plane brings in only the transitively base-free packages
|
||||
// (types, velocity, anomaly, replay, standard) and states its rules as data.
|
||||
// That is the better shape here anyway: a risk rule is a conjunction of
|
||||
// comparisons over a fixed vocabulary, which is a closed algebra, not a
|
||||
// language. A closed algebra is injection-safe by construction, is a TYPED
|
||||
// wire shape (so it reaches the SDKs, the CLI and the MCP tools as a schema
|
||||
// instead of as an opaque string), and can be replayed without an interpreter.
|
||||
//
|
||||
// The right long-term fix is upstream and is named in the handoff: split
|
||||
// pkg/history's Base-backed store into its own package so pkg/engine is
|
||||
// base-free in fact as well as in intent.
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Actions, strongest last. `challenge` and `restrict` are risk's own: the
|
||||
// compliance vocabulary (allow/flag/review/block) has no word for "ask the human
|
||||
// to prove they are one" or "let the money in but not out", and both are the
|
||||
// normal answer at signup and at payout.
|
||||
const (
|
||||
ActionAllow = "allow"
|
||||
ActionChallenge = "challenge"
|
||||
ActionReview = "review"
|
||||
ActionRestrict = "restrict"
|
||||
ActionBlock = "block"
|
||||
)
|
||||
|
||||
// actionRank orders actions by how much they demand, so "the strongest of these"
|
||||
// and "no stronger than this" are one comparison rather than two tables. An
|
||||
// action nobody defined ranks lowest: it must never outrank one that was.
|
||||
func actionRank(a string) int {
|
||||
switch a {
|
||||
case ActionAllow:
|
||||
return 1
|
||||
case ActionChallenge:
|
||||
return 2
|
||||
case ActionReview:
|
||||
return 3
|
||||
case ActionRestrict:
|
||||
return 4
|
||||
case ActionBlock:
|
||||
return 5
|
||||
default:
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
// modelCeiling is the strongest action STATISTICAL evidence may reach on its
|
||||
// own. It is luxfi/aml's types.ActionCeiling and it is not weakened for the
|
||||
// payment stage: a model can put a transaction in front of a person; it cannot
|
||||
// decline one, because an unexplainable refusal is not a decision anybody can
|
||||
// defend to the customer or to a chargeback network.
|
||||
const modelCeiling = ActionReview
|
||||
|
||||
// Stages are the lifecycle moments a decision can be asked about. The stage
|
||||
// selects the feature window and the rule set; it never selects a different
|
||||
// tenant gate.
|
||||
const (
|
||||
StageSignup = "signup"
|
||||
StagePayment = "payment"
|
||||
StageSession = "session"
|
||||
StageUsage = "usage"
|
||||
StagePayout = "payout"
|
||||
StageDispute = "dispute"
|
||||
)
|
||||
|
||||
var stages = map[string]bool{
|
||||
StageSignup: true, StagePayment: true, StageSession: true,
|
||||
StageUsage: true, StagePayout: true, StageDispute: true,
|
||||
}
|
||||
|
||||
// Operators, closed.
|
||||
const (
|
||||
OpEq = "eq"
|
||||
OpNe = "ne"
|
||||
OpGt = "gt"
|
||||
OpGte = "gte"
|
||||
OpLt = "lt"
|
||||
OpLte = "lte"
|
||||
OpIn = "in"
|
||||
OpNotIn = "notin"
|
||||
OpInList = "inlist"
|
||||
OpNotList = "notinlist"
|
||||
OpExists = "exists"
|
||||
OpAbsent = "absent"
|
||||
OpContains = "contains"
|
||||
OpPrefix = "prefix"
|
||||
OpSuffix = "suffix"
|
||||
)
|
||||
|
||||
var operators = map[string]bool{
|
||||
OpEq: true, OpNe: true, OpGt: true, OpGte: true, OpLt: true, OpLte: true,
|
||||
OpIn: true, OpNotIn: true, OpInList: true, OpNotList: true,
|
||||
OpExists: true, OpAbsent: true, OpContains: true, OpPrefix: true, OpSuffix: true,
|
||||
}
|
||||
|
||||
// facts is the CLOSED vocabulary of scalar fields a term may name. Two families
|
||||
// are open by shape and closed by source: `signal.<name>` reads the caller's own
|
||||
// signal map (the caller's data, compared in memory, never spelled into SQL),
|
||||
// and `velocity.<axis>.<window>.<stat>` reads the in-memory rings through an
|
||||
// allowlisted axis and window.
|
||||
var facts = map[string]bool{
|
||||
"stage": true,
|
||||
"subject.kind": true,
|
||||
"subject.id": true,
|
||||
"agency": true,
|
||||
"amount.nano": true,
|
||||
"amount.currency": true,
|
||||
"amount.direction": true,
|
||||
"model.score": true,
|
||||
"model.warming": true,
|
||||
"actor.agent": true,
|
||||
"actor.session": true,
|
||||
}
|
||||
|
||||
// velocityStats is the closed set of numbers a rule may read off a ring.
|
||||
var velocityStats = map[string]bool{"count": true, "sum": true, "near": true, "days": true}
|
||||
|
||||
// term is one comparison. Value carries strings, Number carries numerics and
|
||||
// Values carries a set — three fields rather than one `any`, because the wire
|
||||
// shape is a published schema and `any` publishes as "anything".
|
||||
type term struct {
|
||||
// Field is the fact to read. One of the closed vocabulary, or
|
||||
// `signal.<name>`, or `velocity.<axis>.<window>.<stat>`.
|
||||
Field string `json:"field"`
|
||||
// Op is the comparison. One of the closed operator set.
|
||||
Op string `json:"op"`
|
||||
// Value is the string operand, for the textual comparisons.
|
||||
Value string `json:"value,omitempty"`
|
||||
// Number is the numeric operand, for the ordered comparisons.
|
||||
Number float64 `json:"number,omitempty"`
|
||||
// Values is the set operand, for `in` and `notin`.
|
||||
Values []string `json:"values,omitempty"`
|
||||
}
|
||||
|
||||
// rule is one detection. All terms must hold — a conjunction, deliberately: a
|
||||
// disjunction is two rules, and two rules are two things a reviewer can judge,
|
||||
// retire and measure separately. Nothing is lost and the report gets sharper.
|
||||
type rule struct {
|
||||
// ID is the rule's stable identifier within the tenant.
|
||||
ID string `json:"id"`
|
||||
// Name is what a reviewer reads in an alert.
|
||||
Name string `json:"name"`
|
||||
// Stage narrows the rule to one lifecycle moment; empty means every stage.
|
||||
Stage string `json:"stage,omitempty"`
|
||||
// Action is what the rule asks for when it holds.
|
||||
Action string `json:"action"`
|
||||
// Weight is how much evidence a hit contributes, in [0,1].
|
||||
Weight float64 `json:"weight"`
|
||||
// Severity is the reviewer-facing grading: low, medium, high or critical.
|
||||
Severity string `json:"severity"`
|
||||
// Enabled governs the live path. A disabled rule still replays, because the
|
||||
// question a simulation asks is what happens ON activation.
|
||||
Enabled bool `json:"enabled"`
|
||||
// All is the conjunction. An empty conjunction is refused at admission: a
|
||||
// rule that holds on everything is not a detection.
|
||||
All []term `json:"all"`
|
||||
}
|
||||
|
||||
// admit validates a rule before anything depends on it. Every refusal here is
|
||||
// one that would otherwise become a rule that fires on everything, on nothing,
|
||||
// or on a field that does not exist — all three read as a working control.
|
||||
func admit(r rule) error {
|
||||
switch {
|
||||
case strings.TrimSpace(r.Name) == "":
|
||||
return fmt.Errorf("risk: a rule with no name cannot be read back in an alert")
|
||||
case r.Stage != "" && !stages[r.Stage]:
|
||||
return fmt.Errorf("risk: %q is not a lifecycle stage", r.Stage)
|
||||
case actionRank(r.Action) == 0:
|
||||
return fmt.Errorf("risk: %q is not an action", r.Action)
|
||||
case r.Weight < 0 || r.Weight > 1:
|
||||
return fmt.Errorf("risk: weight %v is outside [0,1]", r.Weight)
|
||||
case len(r.All) == 0:
|
||||
return fmt.Errorf("risk: a rule with no terms holds on everything, which is not a detection")
|
||||
}
|
||||
if r.Severity != "" && !severities[r.Severity] {
|
||||
return fmt.Errorf("risk: %q is not a severity", r.Severity)
|
||||
}
|
||||
for i, t := range r.All {
|
||||
if err := admitTerm(t); err != nil {
|
||||
return fmt.Errorf("risk: term %d: %w", i, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
var severities = map[string]bool{"low": true, "medium": true, "high": true, "critical": true}
|
||||
|
||||
// admitTerm holds the field and operator vocabularies. It is the ONE gate: a
|
||||
// field is either in the closed set or matches one of the two structured
|
||||
// families, and an unknown field is an error rather than a silent miss.
|
||||
func admitTerm(t term) error {
|
||||
if !operators[t.Op] {
|
||||
return fmt.Errorf("%q is not an operator", t.Op)
|
||||
}
|
||||
switch {
|
||||
case facts[t.Field]:
|
||||
case strings.HasPrefix(t.Field, "signal."):
|
||||
if strings.TrimPrefix(t.Field, "signal.") == "" {
|
||||
return fmt.Errorf("signal. names no signal")
|
||||
}
|
||||
case strings.HasPrefix(t.Field, "velocity."):
|
||||
parts := strings.Split(t.Field, ".")
|
||||
if len(parts) != 4 {
|
||||
return fmt.Errorf("%q is not velocity.<axis>.<window>.<stat>", t.Field)
|
||||
}
|
||||
if !velocityAxes[parts[1]] {
|
||||
return fmt.Errorf("%q is not a velocity axis", parts[1])
|
||||
}
|
||||
if !velocityWindows[parts[2]] {
|
||||
return fmt.Errorf("%q is not a velocity window", parts[2])
|
||||
}
|
||||
if !velocityStats[parts[3]] {
|
||||
return fmt.Errorf("%q is not a velocity statistic", parts[3])
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("%q is not a fact this vocabulary carries", t.Field)
|
||||
}
|
||||
if (t.Op == OpIn || t.Op == OpNotIn) && len(t.Values) == 0 {
|
||||
return fmt.Errorf("%s with an empty set holds on nothing", t.Op)
|
||||
}
|
||||
if (t.Op == OpInList || t.Op == OpNotList) && strings.TrimSpace(t.Value) == "" {
|
||||
return fmt.Errorf("%s names no list", t.Op)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// velocityAxes and velocityWindows mirror the rings the decide path keeps. They
|
||||
// are declared here because this is where a caller's spelling is checked against
|
||||
// them; decide.go records on exactly these.
|
||||
var velocityAxes = map[string]bool{"account": true, "device": true, "ip": true, "pair": true, "email": true, "bin": true}
|
||||
|
||||
var velocityWindows = map[string]bool{"1h": true, "24h": true, "7d": true, "30d": true}
|
||||
|
||||
// facts is what a term reads: the decision's own inputs plus everything the
|
||||
// scoring path has computed by the time rules run.
|
||||
type factSet struct {
|
||||
scalar map[string]string
|
||||
number map[string]float64
|
||||
lists func(name, value string) bool
|
||||
}
|
||||
|
||||
func (f factSet) str(field string) (string, bool) {
|
||||
v, ok := f.scalar[field]
|
||||
return v, ok
|
||||
}
|
||||
|
||||
func (f factSet) num(field string) (float64, bool) {
|
||||
v, ok := f.number[field]
|
||||
if ok {
|
||||
return v, true
|
||||
}
|
||||
// A scalar that parses as a number is comparable as one: `signal.bin` is a
|
||||
// string on the wire and an ordered value in a rule.
|
||||
if s, ok := f.scalar[field]; ok {
|
||||
if n, err := strconv.ParseFloat(s, 64); err == nil {
|
||||
return n, true
|
||||
}
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// holds evaluates one term. An unreadable field is FALSE, never an error that
|
||||
// aborts the decision: a rule over a signal this caller did not send has not
|
||||
// matched, and the alternative — failing the whole decision — turns one
|
||||
// mis-specified rule into an outage on the authorization path.
|
||||
func (f factSet) holds(t term) bool {
|
||||
switch t.Op {
|
||||
case OpExists:
|
||||
if _, ok := f.str(t.Field); ok {
|
||||
return true
|
||||
}
|
||||
_, ok := f.num(t.Field)
|
||||
return ok
|
||||
case OpAbsent:
|
||||
if _, ok := f.str(t.Field); ok {
|
||||
return false
|
||||
}
|
||||
_, ok := f.num(t.Field)
|
||||
return !ok
|
||||
case OpGt, OpGte, OpLt, OpLte:
|
||||
v, ok := f.num(t.Field)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
switch t.Op {
|
||||
case OpGt:
|
||||
return v > t.Number
|
||||
case OpGte:
|
||||
return v >= t.Number
|
||||
case OpLt:
|
||||
return v < t.Number
|
||||
default:
|
||||
return v <= t.Number
|
||||
}
|
||||
case OpInList, OpNotList:
|
||||
v, ok := f.str(t.Field)
|
||||
if !ok || f.lists == nil {
|
||||
return t.Op == OpNotList
|
||||
}
|
||||
in := f.lists(t.Value, v)
|
||||
return in == (t.Op == OpInList)
|
||||
}
|
||||
|
||||
v, ok := f.str(t.Field)
|
||||
if !ok {
|
||||
if n, isNum := f.num(t.Field); isNum {
|
||||
v, ok = strconv.FormatFloat(n, 'f', -1, 64), true
|
||||
}
|
||||
}
|
||||
if !ok {
|
||||
return t.Op == OpNe || t.Op == OpNotIn
|
||||
}
|
||||
switch t.Op {
|
||||
case OpEq:
|
||||
return v == t.Value
|
||||
case OpNe:
|
||||
return v != t.Value
|
||||
case OpContains:
|
||||
return strings.Contains(v, t.Value)
|
||||
case OpPrefix:
|
||||
return strings.HasPrefix(v, t.Value)
|
||||
case OpSuffix:
|
||||
return strings.HasSuffix(v, t.Value)
|
||||
case OpIn:
|
||||
return contains(t.Values, v)
|
||||
case OpNotIn:
|
||||
return !contains(t.Values, v)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func contains(set []string, v string) bool {
|
||||
for _, s := range set {
|
||||
if s == v {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// evaluate runs the enabled rules for a stage and returns the hits in a stable
|
||||
// order (strongest action first, then by weight, then by id) so two identical
|
||||
// decisions render identically.
|
||||
func evaluate(rules []rule, stage string, f factSet) []hit {
|
||||
var hits []hit
|
||||
for _, r := range rules {
|
||||
if !r.Enabled || (r.Stage != "" && r.Stage != stage) {
|
||||
continue
|
||||
}
|
||||
matched := true
|
||||
for _, t := range r.All {
|
||||
if !f.holds(t) {
|
||||
matched = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if matched {
|
||||
hits = append(hits, hit{Rule: r.ID, Name: r.Name, Action: r.Action, Weight: r.Weight, Severity: r.Severity})
|
||||
}
|
||||
}
|
||||
sort.SliceStable(hits, func(i, j int) bool {
|
||||
if a, b := actionRank(hits[i].Action), actionRank(hits[j].Action); a != b {
|
||||
return a > b
|
||||
}
|
||||
if hits[i].Weight != hits[j].Weight {
|
||||
return hits[i].Weight > hits[j].Weight
|
||||
}
|
||||
return hits[i].Rule < hits[j].Rule
|
||||
})
|
||||
return hits
|
||||
}
|
||||
|
||||
// combine turns hits into a score and an action.
|
||||
//
|
||||
// The score is weight-of-evidence: 1 - prod(1-w). Two independent weak signals
|
||||
// compound and no single one saturates, which is the property a summed score
|
||||
// does not have (three 0.4 rules sum past 1 and clamp, losing the fourth).
|
||||
//
|
||||
// The action is the strongest any hit asked for, and the model's own hit is
|
||||
// capped at the ceiling BEFORE it gets here, so no arrangement of weights lets
|
||||
// statistical evidence decline anything on its own.
|
||||
func combine(hits []hit) (float64, string) {
|
||||
remain, action := 1.0, ActionAllow
|
||||
for _, h := range hits {
|
||||
w := h.Weight
|
||||
if w < 0 {
|
||||
w = 0
|
||||
}
|
||||
if w > 1 {
|
||||
w = 1
|
||||
}
|
||||
remain *= 1 - w
|
||||
if actionRank(h.Action) > actionRank(action) {
|
||||
action = h.Action
|
||||
}
|
||||
}
|
||||
return 1 - remain, action
|
||||
}
|
||||
|
||||
// starter is the rule set a tenant gets on its first decision. Every one of
|
||||
// these is a lifecycle-stage detection the product spec names, expressed in the
|
||||
// vocabulary above so a tenant can read, copy and retire them. They are seeded
|
||||
// ENABLED but the tenant starts in SHADOW, so nothing acts until an operator
|
||||
// says so and can see what would have happened.
|
||||
func starter() []rule {
|
||||
return []rule{{
|
||||
ID: "signup-burst-ip", Name: "Many signups from one address", Stage: StageSignup,
|
||||
Action: ActionChallenge, Weight: 0.4, Severity: "medium", Enabled: true,
|
||||
All: []term{{Field: "velocity.ip.1h.count", Op: OpGte, Number: 5}},
|
||||
}, {
|
||||
ID: "signup-shared-device", Name: "One device onboarding several accounts", Stage: StageSignup,
|
||||
Action: ActionReview, Weight: 0.5, Severity: "high", Enabled: true,
|
||||
All: []term{{Field: "velocity.device.7d.count", Op: OpGte, Number: 4}},
|
||||
}, {
|
||||
ID: "signup-disposable-email", Name: "Disposable email domain", Stage: StageSignup,
|
||||
Action: ActionChallenge, Weight: 0.35, Severity: "medium", Enabled: true,
|
||||
All: []term{{Field: "signal.emaildomain", Op: OpInList, Value: "email-deny"}},
|
||||
}, {
|
||||
ID: "payment-card-testing", Name: "Card testing: a burst of small charges", Stage: StagePayment,
|
||||
Action: ActionBlock, Weight: 0.7, Severity: "critical", Enabled: true,
|
||||
All: []term{
|
||||
{Field: "velocity.bin.1h.count", Op: OpGte, Number: 10},
|
||||
{Field: "amount.nano", Op: OpLte, Number: 5_000_000_000},
|
||||
},
|
||||
}, {
|
||||
ID: "payment-denied-ip", Name: "Payment from a denied address", Stage: StagePayment,
|
||||
Action: ActionBlock, Weight: 0.9, Severity: "critical", Enabled: true,
|
||||
All: []term{{Field: "signal.ip", Op: OpInList, Value: "ip-deny"}},
|
||||
}, {
|
||||
ID: "usage-spend-spike", Name: "Pay-as-you-go spend far above this account's own shape", Stage: StageUsage,
|
||||
Action: ActionRestrict, Weight: 0.5, Severity: "high", Enabled: true,
|
||||
All: []term{{Field: "velocity.account.24h.sum", Op: OpGte, Number: 100_000_000_000_000}},
|
||||
}, {
|
||||
ID: "payout-velocity", Name: "Payouts leaving faster than they arrived", Stage: StagePayout,
|
||||
Action: ActionRestrict, Weight: 0.6, Severity: "high", Enabled: true,
|
||||
All: []term{
|
||||
{Field: "amount.direction", Op: OpEq, Value: "out"},
|
||||
{Field: "velocity.account.24h.count", Op: OpGte, Number: 5},
|
||||
},
|
||||
}, {
|
||||
ID: "bot-anonymous-burst", Name: "Undeclared automation at machine cadence", Stage: StageSession,
|
||||
Action: ActionChallenge, Weight: 0.45, Severity: "medium", Enabled: true,
|
||||
All: []term{
|
||||
{Field: "agency", Op: OpEq, Value: AgencyBot},
|
||||
{Field: "velocity.ip.1h.count", Op: OpGte, Number: 60},
|
||||
},
|
||||
}}
|
||||
}
|
||||
@@ -0,0 +1,586 @@
|
||||
package risk
|
||||
|
||||
// skew_test.go proves the training–serving skew control actually controls
|
||||
// something, and that the record behind a decision is one record.
|
||||
//
|
||||
// Every test here was written against the defect it names: the fix was reverted,
|
||||
// the test was run, and it went red. A control whose test passes with the control
|
||||
// removed is a comment.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A REFIT CANNOT RE-BLESS STALE COORDINATES.
|
||||
//
|
||||
// The shape gate refuses a map fitted under coordinates that have moved, and the
|
||||
// refusal's own remedy is "fit again". If a refit reads the same rows — every one
|
||||
// of them scored under the OLD shape — and stamps the NEW shape on them, then the
|
||||
// gate is a door that opens from the inside: one POST clears the control with no
|
||||
// new evidence at all, and the plane resumes reporting probabilities derived from
|
||||
// a coordinate system that no longer exists.
|
||||
//
|
||||
// A fit reads ONE coordinate system. After a governed change the evidence under
|
||||
// the new shape is empty, so the fit is refused and says how much evidence it had
|
||||
// to leave behind and why.
|
||||
func TestARefitCannotReblessCoordinatesThatMoved(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
tn := Tenant("hanzo/acme")
|
||||
seedJudged(t, s, tn, 200)
|
||||
|
||||
code, body := req(t, app, http.MethodPost, "/v1/ml/calibrate", "acme", "u_acme", `{"horizon":0}`)
|
||||
if code != http.StatusCreated {
|
||||
t.Fatalf("calibrate = %d %s", code, body)
|
||||
}
|
||||
var first mlCalibrationView
|
||||
if err := json.Unmarshal(body, &first); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !first.Current || first.Rows != 200 {
|
||||
t.Fatalf("the first fit is not the premise this test needs: current=%v rows=%d", first.Current, first.Rows)
|
||||
}
|
||||
|
||||
// A governed change. Every one of the 200 rows was scored under the old shape.
|
||||
code, body = req(t, app, http.MethodPost, "/v1/risk/rules", "acme", "u_acme",
|
||||
`{"rule":{"id":"skew-new","name":"new","stage":"payment","action":"review",
|
||||
"weight":0.5,"severity":"high","enabled":true,
|
||||
"all":[{"field":"amount.nano","op":"gte","number":1}]}}`)
|
||||
if code != http.StatusOK && code != http.StatusCreated {
|
||||
t.Fatalf("rule = %d %s", code, body)
|
||||
}
|
||||
|
||||
// The documented remedy, run against evidence that predates the change.
|
||||
code, body = req(t, app, http.MethodPost, "/v1/ml/calibrate", "acme", "u_acme", `{"horizon":0}`)
|
||||
if code != http.StatusCreated {
|
||||
t.Fatalf("refit = %d %s", code, body)
|
||||
}
|
||||
var second mlCalibrationView
|
||||
if err := json.Unmarshal(body, &second); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if second.Fitted {
|
||||
t.Errorf("the refit produced a map (shape %.12s, %d rows) out of history scored entirely under "+
|
||||
"shape %.12s. The skew control was cleared by one POST with no new observation.",
|
||||
second.Shape, second.Rows, first.Shape)
|
||||
}
|
||||
if second.Refusal == "" {
|
||||
t.Error("the refit refused and did not say why")
|
||||
}
|
||||
if second.Superseded != 200 {
|
||||
t.Errorf("superseded = %d, want 200 — the answer does not say how much evidence the shape "+
|
||||
"boundary put out of reach, so an operator cannot tell a thin tenant from a moved one",
|
||||
second.Superseded)
|
||||
}
|
||||
|
||||
// And the map in force still refuses, because nothing about the world changed.
|
||||
code, body = req(t, app, http.MethodGet, "/v1/ml/calibration", "acme", "u_acme", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("calibration = %d %s", code, body)
|
||||
}
|
||||
var now mlCalibrationView
|
||||
if err := json.Unmarshal(body, &now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if now.Current {
|
||||
t.Error("the map in force reports current after a shape change it was not fitted under")
|
||||
}
|
||||
}
|
||||
|
||||
// EVERY OP READS THE MAP THROUGH THE SHAPE IN FORCE.
|
||||
//
|
||||
// The gate used to be Map.P(score, shape), and every internal caller passed the
|
||||
// map's OWN shape — so the comparison was between a value and itself and the
|
||||
// control was inert everywhere except the decide path. The consequence was not
|
||||
// subtle: with the live plane refusing to state any probability, /v1/ml/evaluate
|
||||
// still reported a Brier, /v1/ml/replay still moved rows onto rungs nothing could
|
||||
// reach, and the reliability chart drew the map the same response refused to use.
|
||||
//
|
||||
// Binding at the boundary makes all three answer the same way the decide path
|
||||
// does. This test moves the shape and then asks all three.
|
||||
func TestMeasurementRefusesTheMapTheDecidePathRefuses(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
tn := Tenant("hanzo/acme")
|
||||
seedJudged(t, s, tn, 200)
|
||||
|
||||
if code, body := req(t, app, http.MethodPost, "/v1/ml/calibrate", "acme", "u_acme", `{"horizon":0}`); code != http.StatusCreated {
|
||||
t.Fatalf("calibrate = %d %s", code, body)
|
||||
}
|
||||
if code, body := req(t, app, http.MethodPut, "/v1/risk/policy", "acme", "u_acme",
|
||||
`{"stage":"payment","floor":"allow","reason":"under test","bands":[{"at":0.2,"action":"review"},{"at":0.6,"action":"block"}]}`); code != http.StatusOK {
|
||||
t.Fatalf("policy = %d %s", code, body)
|
||||
}
|
||||
if code, body := req(t, app, http.MethodPost, "/v1/risk/rules", "acme", "u_acme",
|
||||
`{"rule":{"id":"skew-two","name":"new","stage":"payment","action":"review",
|
||||
"weight":0.5,"severity":"high","enabled":true,
|
||||
"all":[{"field":"amount.nano","op":"gte","number":1}]}}`); code != http.StatusOK && code != http.StatusCreated {
|
||||
t.Fatalf("rule = %d %s", code, body)
|
||||
}
|
||||
|
||||
code, body := req(t, app, http.MethodGet, "/v1/ml/calibration", "acme", "u_acme", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("calibration = %d %s", code, body)
|
||||
}
|
||||
var view mlCalibrationView
|
||||
if err := json.Unmarshal(body, &view); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if view.Current {
|
||||
t.Fatal("the shape did not move; the test proves nothing")
|
||||
}
|
||||
if len(view.Reliability) > 0 {
|
||||
t.Errorf("the response that says the map refuses to answer shipped %d reliability bins, "+
|
||||
"computed by applying that very map", len(view.Reliability))
|
||||
}
|
||||
|
||||
code, body = req(t, app, http.MethodPost, "/v1/ml/evaluate", "acme", "u_acme", `{"stage":"payment","horizon":0}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("evaluate = %d %s", code, body)
|
||||
}
|
||||
var meas mlMeasurement
|
||||
if err := json.Unmarshal(body, &meas); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if meas.Metrics.Brier != nil {
|
||||
t.Errorf("evaluate reported Brier=%.6f under a calibration the decide path refuses", *meas.Metrics.Brier)
|
||||
}
|
||||
if meas.Calibrated {
|
||||
t.Error("evaluate says it was calibrated while the map in force refuses to answer")
|
||||
}
|
||||
if meas.Refusal == "" {
|
||||
t.Error("evaluate measured without a probability and did not say so")
|
||||
}
|
||||
|
||||
code, body = req(t, app, http.MethodPost, "/v1/ml/replay", "acme", "u_acme",
|
||||
`{"stage":"payment","horizon":0,"bands":[{"at":0.2,"action":"review"},{"at":0.6,"action":"block"}]}`)
|
||||
if code != http.StatusCreated {
|
||||
t.Fatalf("replay = %d %s", code, body)
|
||||
}
|
||||
var rep mlReplayReport
|
||||
if err := json.Unmarshal(body, &rep); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
acted := 0
|
||||
for _, w := range rep.Would {
|
||||
if w.Action != ActionAllow {
|
||||
acted += w.Count
|
||||
}
|
||||
}
|
||||
if acted > 0 {
|
||||
t.Errorf("the replay says the candidate would act on %d decisions. Under the shape in force "+
|
||||
"there is no probability, so no rung is reachable and the candidate acts on none — and "+
|
||||
"this report was written down as the evidence for a threshold change.", acted)
|
||||
}
|
||||
if rep.Refusal == "" {
|
||||
t.Error("the replay reached no rung and did not say why")
|
||||
}
|
||||
}
|
||||
|
||||
// THE BOUNDED READ TAKES THE MOST RECENT DECISIONS.
|
||||
//
|
||||
// Ascending order plus LIMIT takes the OLDEST rows. Past the bound that freezes
|
||||
// the calibration, the evaluation, the learning curve and every replay on the
|
||||
// tenant's first maxHistory decisions forever, with nothing on any response
|
||||
// saying the read was cut.
|
||||
func TestTheBoundedReadTakesTheNewestDecisionsAndSaysWhenItCut(t *testing.T) {
|
||||
_, s := wireApp(t)
|
||||
tn := Tenant("hanzo/acme")
|
||||
db, err := s.State.shelf.open(tn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
shape, err := scoringShape(db, s.State.digest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
base := time.Now().UTC().Add(-1000 * time.Hour)
|
||||
const n = 300
|
||||
for i := range n {
|
||||
id := fmt.Sprintf("dec_w_%05d", i)
|
||||
o := observation{
|
||||
id: id, at: base.Add(time.Duration(i) * time.Minute),
|
||||
stage: StagePayment, kind: "transaction", subject: "tx", agency: AgencyUnknown,
|
||||
amount: 1_000_000_000, currency: "USD", direction: "in", signals: map[string]string{},
|
||||
}
|
||||
out := outcome{id: id, action: ActionAllow, score: float64(i) / n, agency: o.agency}
|
||||
if err := putDecision(db, o, out, "d", shape, "", verdict{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := label(db, id, "legitimate", "u"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
// The bound is maxHistory in production; the statement is identical and only
|
||||
// the number differs, so a smaller one exercises the same path in a test.
|
||||
w, err := recorded(context.Background(), db, 0, shape, 100)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(w.history) != 100 {
|
||||
t.Fatalf("read %d rows, want 100", len(w.history))
|
||||
}
|
||||
first, last := w.history[0].ID, w.history[len(w.history)-1].ID
|
||||
if last != fmt.Sprintf("dec_w_%05d", n-1) {
|
||||
t.Errorf("the bounded read returned %s..%s — the OLDEST rows, not the most recent. Past "+
|
||||
"maxHistory=%d decisions every measurement is pinned to the dawn of the log.", first, last, maxHistory)
|
||||
}
|
||||
if first != fmt.Sprintf("dec_w_%05d", n-100) {
|
||||
t.Errorf("the window starts at %s, want %s: the rows must be the last 100 IN TIME ORDER",
|
||||
first, fmt.Sprintf("dec_w_%05d", n-100))
|
||||
}
|
||||
if !w.truncated {
|
||||
t.Error("the read was cut and no field says so, so a report over a truncated window reads as complete")
|
||||
}
|
||||
// And an unbounded read reports no truncation.
|
||||
all, err := recorded(context.Background(), db, 0, shape, n)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if all.truncated {
|
||||
t.Error("a read that saw everything reported itself truncated")
|
||||
}
|
||||
}
|
||||
|
||||
// A MUTE MOVES THE SCORE DISTRIBUTION, SO IT MOVES THE SHAPE.
|
||||
//
|
||||
// combine() sums only the hits that were not suppressed, so muting a rule for
|
||||
// every subject moves every score that rule touched exactly as retiring the rule
|
||||
// would. Muting is the day-to-day tuning knob and retiring is the rare act, so a
|
||||
// control blind to the mute is blind to the common change.
|
||||
//
|
||||
// A mute that names a SUBJECT is a statement about that subject — operational
|
||||
// data, like a list entry — and must not invalidate the tenant's calibration.
|
||||
func TestARuleWideMuteMovesTheShapeAndASubjectMuteDoesNot(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
db, err := s.State.shelf.open(Tenant("hanzo/acme"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
before, err := scoringShape(db, s.State.digest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/suppressions", "acme", "u_acme",
|
||||
`{"rule":"payment-card-testing","reason":"too noisy this week"}`)
|
||||
if code != http.StatusOK && code != http.StatusCreated {
|
||||
t.Fatalf("suppress = %d %s", code, body)
|
||||
}
|
||||
muted, err := scoringShape(db, s.State.digest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if muted == before {
|
||||
t.Errorf("muting a weight-bearing rule for every subject left the shape at %.12s. Every score "+
|
||||
"that rule touched has moved and the calibration still reports itself current.", muted)
|
||||
}
|
||||
|
||||
code, body = req(t, app, http.MethodPost, "/v1/risk/suppressions", "acme", "u_acme",
|
||||
`{"rule":"payment-velocity-burst","subject":"tx-one-merchant","reason":"known good"}`)
|
||||
if code != http.StatusOK && code != http.StatusCreated {
|
||||
t.Fatalf("suppress subject = %d %s", code, body)
|
||||
}
|
||||
after, err := scoringShape(db, s.State.digest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if after != muted {
|
||||
t.Errorf("excusing ONE subject moved the shape from %.12s to %.12s. Operational data about "+
|
||||
"one row would expire the tenant's calibration, and it would never have one.", muted, after)
|
||||
}
|
||||
}
|
||||
|
||||
// THE LADDER CANNOT DECLINE ON THE MODEL'S EVIDENCE ALONE.
|
||||
//
|
||||
// decide caps the model's own hit at modelCeiling because an unexplainable
|
||||
// refusal is not a decision anybody can defend to the customer or to a chargeback
|
||||
// network. The policy ladder reads the calibrated probability, and that
|
||||
// probability is a pure function of the SAME score — so an uncapped escalation
|
||||
// voids the cap by arithmetic, and the invariant holds only until somebody sets a
|
||||
// band.
|
||||
func TestTheLadderCannotDeclineOnTheModelAlone(t *testing.T) {
|
||||
modelOnly := outcome{hits: []hit{{Rule: modelRuleID, Action: ActionReview, Weight: 0.9}}}
|
||||
nothing := outcome{}
|
||||
muted := outcome{hits: []hit{{Rule: "a-rule", Action: ActionBlock, Weight: 0.7, Suppressed: true}}}
|
||||
explained := outcome{hits: []hit{{Rule: "a-rule", Action: ActionBlock, Weight: 0.7}}}
|
||||
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
out outcome
|
||||
evidence, policy string
|
||||
want string
|
||||
}{
|
||||
{"model alone cannot block", modelOnly, ActionReview, ActionBlock, ActionReview},
|
||||
{"model alone cannot restrict", modelOnly, ActionAllow, ActionRestrict, ActionReview},
|
||||
{"model alone can still review", modelOnly, ActionAllow, ActionReview, ActionReview},
|
||||
{"no evidence at all cannot block", nothing, ActionAllow, ActionBlock, ActionReview},
|
||||
{"a muted hit is not evidence", muted, ActionAllow, ActionBlock, ActionReview},
|
||||
{"a rule the org wrote can block", explained, ActionAllow, ActionBlock, ActionBlock},
|
||||
} {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
if got := escalate(c.evidence, c.policy, c.out); got != c.want {
|
||||
t.Errorf("escalate(%q, %q) = %q, want %q — %s",
|
||||
c.evidence, c.policy, got, c.want,
|
||||
"a decline the model alone is behind has no reason a person can read")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// THE DECISION AND ITS GRADING ARE ONE RECORD.
|
||||
//
|
||||
// Two independent statements can land one and not the other, and the half that
|
||||
// lands is the half that acts: a BLOCK on the dispute-packet surface with no
|
||||
// probability, no principal reason and no refusal saying why. This proves both
|
||||
// halves — that a failed grading takes the decision down with it, and that a
|
||||
// decision found without one says so instead of reading as ungraded.
|
||||
func TestTheDecisionAndItsGradingAreOneRecord(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
db, err := s.State.shelf.open(Tenant("hanzo/acme"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
o := observation{
|
||||
id: "dec_atomic", at: time.Now().UTC(), stage: StagePayment, kind: "transaction",
|
||||
subject: "tx", agency: AgencyUnknown, amount: 1_000_000_000, currency: "USD",
|
||||
direction: "in", signals: map[string]string{},
|
||||
}
|
||||
out := outcome{id: o.id, action: ActionBlock, score: 0.97, agency: o.agency}
|
||||
|
||||
// Make the SECOND write fail. Whatever the cause in production — a disk, a
|
||||
// lock, a crash between two Execs — the property under test is that the first
|
||||
// write does not survive it.
|
||||
if _, err := db.Exec(`ALTER TABLE verdict RENAME TO verdict_away`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := putDecision(db, o, out, "d", "shape", "", verdict{}); err == nil {
|
||||
t.Fatal("the grading could not be written and the write reported success")
|
||||
}
|
||||
if _, err := db.Exec(`ALTER TABLE verdict_away RENAME TO verdict`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var n int
|
||||
if err := db.QueryRow(`SELECT COUNT(*) FROM decision WHERE id = ?`, o.id).Scan(&n); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n != 0 {
|
||||
t.Errorf("the decision survived a grading that did not: a BLOCK exists with no probability, "+
|
||||
"no principal reason and nothing this plane can defend it with (%d row(s))", n)
|
||||
}
|
||||
|
||||
// The read path must never present an ungraded decision as a graded one with
|
||||
// nothing to say.
|
||||
orphan := o
|
||||
orphan.id = "dec_orphan"
|
||||
if _, err := db.Exec(`INSERT INTO decision (id, at, stage, kind, subject, action, score, agency, shadow)
|
||||
VALUES (?,?,?,?,?,?,?,?,0)`,
|
||||
orphan.id, stamp(orphan.at), orphan.stage, orphan.kind, orphan.subject,
|
||||
ActionBlock, 0.97, AgencyUnknown); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
code, body := req(t, app, http.MethodGet, "/v1/risk/decisions/dec_orphan", "acme", "u_acme", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("detail = %d %s", code, body)
|
||||
}
|
||||
var packet riskDecisionView
|
||||
if err := json.Unmarshal(body, &packet); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(packet.Reasons) == 0 && packet.Refusal == "" {
|
||||
t.Error("a BLOCK is served on the dispute-packet surface with no reasons, no probability and " +
|
||||
"no refusal, so an ungraded decision is indistinguishable from one with nothing to say")
|
||||
}
|
||||
}
|
||||
|
||||
// A GOVERNANCE RECORD NAMES THE PERSON.
|
||||
//
|
||||
// "The org changed the thresholds" is not an answer to who changed them, and the
|
||||
// helper that resolves the validated principal already exists and is already used
|
||||
// by the suppression and control records in this same package.
|
||||
func TestGovernanceRecordsNameThePersonAndNotTheOrg(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
tn := Tenant("hanzo/acme")
|
||||
seedJudged(t, s, tn, 200)
|
||||
db, err := s.State.shelf.open(tn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if code, body := req(t, app, http.MethodPut, "/v1/risk/policy", "acme", "u_acme",
|
||||
`{"stage":"payment","floor":"allow","reason":"under test","bands":[{"at":0.6,"action":"review"}]}`); code != http.StatusOK {
|
||||
t.Fatalf("policy = %d %s", code, body)
|
||||
}
|
||||
if code, body := req(t, app, http.MethodPost, "/v1/ml/calibrate", "acme", "u_acme", `{"horizon":0}`); code != http.StatusCreated {
|
||||
t.Fatalf("calibrate = %d %s", code, body)
|
||||
}
|
||||
if code, body := req(t, app, http.MethodPost, "/v1/ml/replay", "acme", "u_acme",
|
||||
`{"stage":"payment","horizon":0}`); code != http.StatusCreated {
|
||||
t.Fatalf("replay = %d %s", code, body)
|
||||
}
|
||||
|
||||
for _, q := range []struct{ what, stmt string }{
|
||||
{"policy", `SELECT by FROM policy ORDER BY version DESC LIMIT 1`},
|
||||
{"calibration", `SELECT by FROM calibration ORDER BY version DESC LIMIT 1`},
|
||||
{"replay", `SELECT by FROM replay ORDER BY at DESC LIMIT 1`},
|
||||
} {
|
||||
var who string
|
||||
if err := db.QueryRow(q.stmt).Scan(&who); err != nil {
|
||||
t.Fatalf("%s: %v", q.what, err)
|
||||
}
|
||||
if who != "u_acme" {
|
||||
t.Errorf("the %s record names %q as its author; the validated principal is %q",
|
||||
q.what, who, "u_acme")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// MEASUREMENT IS BOUNDED PER TENANT, AND ONLY PER TENANT.
|
||||
//
|
||||
// These ops read the tenant's single-writer file, so a second concurrent
|
||||
// measurement for one tenant queues behind the first holding the connection that
|
||||
// tenant's own decisions need. The bound is per tenant by construction: there is
|
||||
// no fleet-wide number, so a caller that loops this surface degrades itself and
|
||||
// nobody else.
|
||||
func TestMeasurementIsBoundedPerTenantAndOnlyPerTenant(t *testing.T) {
|
||||
f := newInflight("a measurement")
|
||||
release, err := f.claim(Tenant("hanzo/acme"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := f.claim(Tenant("hanzo/acme")); err == nil {
|
||||
t.Error("a tenant took two measurement slots at once, so it can hold every connection its " +
|
||||
"own decide path needs")
|
||||
}
|
||||
// A NEIGHBOUR IS UNAFFECTED. This is the property a fleet-wide cap does not
|
||||
// have, and the whole reason the bound is keyed on the tenant.
|
||||
other, err := f.claim(Tenant("hanzo/other"))
|
||||
if err != nil {
|
||||
t.Fatalf("one tenant's measurement refused another tenant's: %v", err)
|
||||
}
|
||||
other()
|
||||
release()
|
||||
again, err := f.claim(Tenant("hanzo/acme"))
|
||||
if err != nil {
|
||||
t.Fatalf("the slot was not released: %v", err)
|
||||
}
|
||||
again()
|
||||
// Release is idempotent: an op that returns through two paths must not free a
|
||||
// slot a later request already took.
|
||||
release()
|
||||
if _, err := f.claim(Tenant("hanzo/acme")); err != nil {
|
||||
t.Fatalf("a double release corrupted the bound: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A MEASUREMENT IS PRICED BY THE WORK IT DOES.
|
||||
//
|
||||
// A flat price over a scan whose cost is linear in the rows it reads is a lie
|
||||
// about the cheap call: a fit over forty judged decisions and a twenty-step
|
||||
// learning curve over fifty thousand cost the same, and the second is the one a
|
||||
// caller loops.
|
||||
func TestAMeasurementIsPricedByTheRowsItReads(t *testing.T) {
|
||||
small, large := measureCents(40), measureCents(maxHistory)
|
||||
if small != qualityCents {
|
||||
t.Errorf("a read of 40 rows costs %d, want the floor %d", small, qualityCents)
|
||||
}
|
||||
if large <= small {
|
||||
t.Errorf("a read of %d rows costs %d and a read of 40 costs %d — the price does not move "+
|
||||
"with the work", maxHistory, large, small)
|
||||
}
|
||||
if measureCents(-1) != qualityCents {
|
||||
t.Error("a negative row count priced below the floor")
|
||||
}
|
||||
// And the gate is taken on the CEILING a request could reach, so the ledger is
|
||||
// debited before the work rather than after it.
|
||||
if bounded(0) != maxHistory || bounded(999999) != maxHistory || bounded(10) != 10 {
|
||||
t.Errorf("bounded(0)=%d bounded(999999)=%d bounded(10)=%d", bounded(0), bounded(999999), bounded(10))
|
||||
}
|
||||
}
|
||||
|
||||
// A SCORE MEANS A PROBABILITY ON THE SCORES THIS PLANE ACTUALLY PRODUCES.
|
||||
//
|
||||
// The whole track is sold on that sentence, and the score distribution it has to
|
||||
// hold for is not a continuum. combine() returns 1 - prod(1-weight) over a FIXED
|
||||
// set of rule weights and decide rounds it to four places, so thousands of
|
||||
// decisions land on a handful of atoms and the modal atom is exactly zero. An
|
||||
// isotonic fit that does not pool ties collapses on exactly that input: every
|
||||
// plateau holding a positive is reported as CERTAINTY, at the top of the range,
|
||||
// where declines happen and where the number lands on an adverse-action record.
|
||||
//
|
||||
// End to end through the op, because the unit test in the engine cannot see the
|
||||
// alphabet cloud feeds it.
|
||||
func TestAScoreMeansAProbabilityOnThePlateausThisPlaneProduces(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
tn := Tenant("hanzo/acme")
|
||||
db, err := s.State.shelf.open(tn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
shape, err := scoringShape(db, s.State.digest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
atoms := []struct {
|
||||
score float64
|
||||
clean, fraud int
|
||||
}{
|
||||
{0.00, 900, 2},
|
||||
{0.35, 190, 10},
|
||||
{0.50, 85, 15},
|
||||
{0.70, 45, 55},
|
||||
{0.90, 10, 40},
|
||||
}
|
||||
base := time.Now().UTC().Add(-500 * time.Hour)
|
||||
n := 0
|
||||
for _, a := range atoms {
|
||||
for i := range a.clean + a.fraud {
|
||||
judgement := "legitimate"
|
||||
if i >= a.clean {
|
||||
judgement = "fraud"
|
||||
}
|
||||
id := fmt.Sprintf("dec_atom_%05d", n)
|
||||
o := observation{
|
||||
id: id, at: base.Add(time.Duration(n) * time.Minute),
|
||||
stage: StagePayment, kind: "transaction", subject: fmt.Sprintf("tx-%d", n),
|
||||
agency: AgencyUnknown, amount: 1_000_000_000, currency: "USD", direction: "in",
|
||||
signals: map[string]string{},
|
||||
}
|
||||
out := outcome{id: id, action: ActionAllow, score: a.score, agency: o.agency}
|
||||
if err := putDecision(db, o, out, "seed-digest", shape, "", verdict{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := label(db, id, judgement, "u_seed"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
n++
|
||||
}
|
||||
}
|
||||
|
||||
code, body := req(t, app, http.MethodPost, "/v1/ml/calibrate", "acme", "u_acme", `{"horizon":0}`)
|
||||
if code != http.StatusCreated {
|
||||
t.Fatalf("calibrate = %d %s", code, body)
|
||||
}
|
||||
cal, _, _, ok, err := currentCalibration(db)
|
||||
if err != nil || !ok {
|
||||
t.Fatalf("read back: %v ok=%v", err, ok)
|
||||
}
|
||||
read, err := cal.Under(shape)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, a := range atoms {
|
||||
want := float64(a.fraud) / float64(a.clean+a.fraud)
|
||||
got := read.P(a.score)
|
||||
if d := got - want; d > 0.05 || d < -0.05 {
|
||||
t.Errorf("score %.2f reports probability %.6f; %d of %d at that score turned out productive, "+
|
||||
"so the truth is %.4f (off by %+.4f)", a.score, got, a.fraud, a.clean+a.fraud, want, d)
|
||||
}
|
||||
}
|
||||
if p := read.P(0.90); p >= 1 {
|
||||
t.Errorf("the top plateau reports certainty (%.6f). That number goes on an adverse-action record.", p)
|
||||
}
|
||||
}
|
||||
+1336
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,208 @@
|
||||
package risk
|
||||
|
||||
// surface_test.go proves the FIVE surfaces one typed declaration is supposed to
|
||||
// produce actually carry these ops — read off the COMMITTED artifacts, not off
|
||||
// the router the other tests mount.
|
||||
//
|
||||
// The distinction matters and is the exact failure that lost plugin/ingress
|
||||
// eight published paths: a router can serve a route while the generated subset,
|
||||
// the woven fleet document and therefore every SDK know nothing about it. Two
|
||||
// derived things agreeing with each other is not evidence. So this reads
|
||||
// plugin/risk/openapi.json (the SDK's source), plugin/risk/mcp.json (the tool
|
||||
// list), the fleet openapi.yaml (what the SDK repos pull), and derives the CLI
|
||||
// from the spec with the same function the CLI itself uses.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// theOps are the operationIds this app promises. One list, checked against four
|
||||
// artifacts, so a rename shows up as four failures rather than as a silently
|
||||
// missing SDK method.
|
||||
var theOps = []string{
|
||||
"riskDecide", "riskDecisions", "riskDecisionDetail", "riskLabel",
|
||||
"riskSubjectState", "riskActivity", "riskSimulate",
|
||||
"riskRules", "riskCreateRule", "riskUpdateRule", "riskDeleteRule",
|
||||
"riskLists", "riskCreateList", "riskAddListEntries", "riskRemoveListEntry",
|
||||
"riskSuppressions", "riskSuppress", "riskUnsuppress",
|
||||
"riskControls", "riskSetControl", "riskReleaseControl",
|
||||
"riskDictionary", "riskMode", "riskSetMode",
|
||||
"mlScore", "mlTrain", "mlState", "mlSetAppetite", "mlFeatures",
|
||||
"mlSearch", "mlSearchResult", "mlSnapshot", "mlRestore",
|
||||
"riskReasons", "riskPolicy", "riskSetPolicy", "riskPolicyVersions",
|
||||
"mlCalibrate", "mlCalibration", "mlEvaluate", "mlLearning",
|
||||
"mlReplay", "mlReplayReport",
|
||||
}
|
||||
|
||||
func repoRoot(t *testing.T) string {
|
||||
t.Helper()
|
||||
wd, err := os.Getwd()
|
||||
if err != nil {
|
||||
t.Fatalf("getwd: %v", err)
|
||||
}
|
||||
return filepath.Dir(filepath.Dir(wd)) // apps/risk -> repo root
|
||||
}
|
||||
|
||||
func readSubset(t *testing.T) []byte {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile(filepath.Join(repoRoot(t), "plugin", "risk", "openapi.json"))
|
||||
if err != nil {
|
||||
t.Fatalf("plugin/risk/openapi.json: %v\nRun: make -C apps/risk describe", err)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// TestSubsetCarriesEveryOp reads the app's OWN published document — the one the
|
||||
// fleet weave consumes and every generated SDK ultimately comes from.
|
||||
func TestSubsetCarriesEveryOp(t *testing.T) {
|
||||
var doc struct {
|
||||
Paths map[string]map[string]struct {
|
||||
OperationID string `json:"operationId"`
|
||||
Summary string `json:"summary"`
|
||||
Description string `json:"description"`
|
||||
} `json:"paths"`
|
||||
}
|
||||
if err := json.Unmarshal(readSubset(t), &doc); err != nil {
|
||||
t.Fatalf("parse subset: %v", err)
|
||||
}
|
||||
got := map[string]string{}
|
||||
for _, methods := range doc.Paths {
|
||||
for method, op := range methods {
|
||||
if op.OperationID == "" {
|
||||
continue
|
||||
}
|
||||
got[op.OperationID] = method
|
||||
if strings.TrimSpace(op.Summary) == "" {
|
||||
t.Errorf("%s publishes no summary — the CLI derives its one-line help from it", op.OperationID)
|
||||
}
|
||||
if strings.TrimSpace(op.Description) == "" {
|
||||
t.Errorf("%s publishes no description — an MCP client reads it to choose the tool", op.OperationID)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, id := range theOps {
|
||||
if _, ok := got[id]; !ok {
|
||||
t.Errorf("%s is missing from plugin/risk/openapi.json — no SDK, no CLI command, no MCP tool", id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestFleetDocumentCarriesEveryPath reads the WOVEN document. The subset can be
|
||||
// right while the weave is stale, and the weave is what the SDK repos pull.
|
||||
func TestFleetDocumentCarriesEveryPath(t *testing.T) {
|
||||
b, err := os.ReadFile(filepath.Join(repoRoot(t), "openapi.yaml"))
|
||||
if err != nil {
|
||||
t.Fatalf("openapi.yaml: %v", err)
|
||||
}
|
||||
body := string(b)
|
||||
for _, p := range []string{
|
||||
"/v1/risk/decide", "/v1/risk/decisions", "/v1/risk/decisions/{id}",
|
||||
"/v1/risk/decisions/{id}/label", "/v1/risk/subjects/{kind}/{id}",
|
||||
"/v1/risk/activity", "/v1/risk/simulate", "/v1/risk/rules",
|
||||
"/v1/risk/rules/{id}", "/v1/risk/lists", "/v1/risk/lists/{name}/entries",
|
||||
"/v1/risk/lists/{name}/entries/{value}", "/v1/risk/suppressions",
|
||||
"/v1/risk/suppressions/{id}", "/v1/risk/controls", "/v1/risk/controls/{id}",
|
||||
"/v1/risk/dictionary", "/v1/risk/mode", "/v1/risk/health",
|
||||
"/v1/ml/score", "/v1/ml/train", "/v1/ml/state", "/v1/ml/state/appetite",
|
||||
"/v1/ml/features", "/v1/ml/search", "/v1/ml/search/{id}",
|
||||
"/v1/ml/snapshot", "/v1/ml/restore",
|
||||
} {
|
||||
if !strings.Contains(body, "\n "+p+":") {
|
||||
t.Errorf("%s is absent from the woven openapi.yaml — the SDK repos pull this file, so no "+
|
||||
"generated client can reach it. Run: make describe", p)
|
||||
}
|
||||
}
|
||||
// The ml app's own paths must still be there: risk claims LEAVES under
|
||||
// /v1/ml, it does not take the stem.
|
||||
for _, p := range []string{"/v1/ml/models", "/v1/ml/health", "/v1/train/jobs"} {
|
||||
if !strings.Contains(body, "\n "+p+":") {
|
||||
t.Errorf("%s disappeared from the fleet document — the risk row swallowed an ml route", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestCLIDerivesEveryCommand runs the SAME derivation the CLI runs. A published
|
||||
// operation with no derivable command is an operation no `hanzo` invocation can
|
||||
// reach.
|
||||
func TestCLIDerivesEveryCommand(t *testing.T) {
|
||||
cmds, err := zip.CommandsFromSpec(readSubset(t))
|
||||
if err != nil {
|
||||
t.Fatalf("CommandsFromSpec: %v", err)
|
||||
}
|
||||
byID := map[string]zip.Command{}
|
||||
for _, c := range cmds {
|
||||
byID[c.OperationID] = c
|
||||
}
|
||||
for _, id := range theOps {
|
||||
c, ok := byID[id]
|
||||
if !ok {
|
||||
t.Errorf("%s derives no CLI command", id)
|
||||
continue
|
||||
}
|
||||
if c.Name == "" || c.Service == "" {
|
||||
t.Errorf("%s derives a nameless command (%q %q)", id, c.Service, c.Name)
|
||||
}
|
||||
if strings.TrimSpace(c.Summary) == "" {
|
||||
t.Errorf("%s derives a command with no help text", id)
|
||||
}
|
||||
}
|
||||
// Path parameters must reach the command as ARGS, or the command cannot
|
||||
// address the record it names.
|
||||
if c := byID["riskDecisionDetail"]; len(c.Args) == 0 {
|
||||
t.Error("riskDecisionDetail derives no positional argument for {id}")
|
||||
}
|
||||
if c := byID["riskSubjectState"]; len(c.Args) < 2 {
|
||||
t.Errorf("riskSubjectState derives %d args, want 2 for {kind} and {id}", len(c.Args))
|
||||
}
|
||||
}
|
||||
|
||||
// TestMCPToolsCarryEveryTypedOp reads the published tool list. Every typed op
|
||||
// becomes a tool whose inputSchema is the In type's schema and whose call runs
|
||||
// the same handler — so a missing tool is a capability an agent cannot use.
|
||||
func TestMCPToolsCarryEveryTypedOp(t *testing.T) {
|
||||
b, err := os.ReadFile(filepath.Join(repoRoot(t), "plugin", "risk", "mcp.json"))
|
||||
if err != nil {
|
||||
t.Fatalf("plugin/risk/mcp.json: %v\nRun: make -C apps/risk describe", err)
|
||||
}
|
||||
var tools []struct {
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description"`
|
||||
InputSchema map[string]any `json:"inputSchema"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &tools); err != nil {
|
||||
t.Fatalf("parse mcp.json: %v", err)
|
||||
}
|
||||
got := map[string]bool{}
|
||||
for _, tool := range tools {
|
||||
got[tool.Name] = true
|
||||
if strings.TrimSpace(tool.Description) == "" {
|
||||
t.Errorf("MCP tool %s has no description — a model cannot choose it", tool.Name)
|
||||
}
|
||||
if tool.InputSchema == nil {
|
||||
t.Errorf("MCP tool %s has no input schema", tool.Name)
|
||||
}
|
||||
}
|
||||
for _, id := range theOps {
|
||||
if !got[id] {
|
||||
t.Errorf("%s is not an MCP tool", id)
|
||||
}
|
||||
}
|
||||
if len(tools) != len(theOps) {
|
||||
var extra []string
|
||||
for _, tool := range tools {
|
||||
if !containsString(theOps, tool.Name) {
|
||||
extra = append(extra, tool.Name)
|
||||
}
|
||||
}
|
||||
sort.Strings(extra)
|
||||
t.Errorf("%d MCP tools published, %d promised; unlisted: %s",
|
||||
len(tools), len(theOps), strings.Join(extra, ", "))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
package risk
|
||||
|
||||
// tenant.go is the ONE place a tenant key is minted, and the only type the rest
|
||||
// of this package can use to reach anything a tenant owns.
|
||||
//
|
||||
// THE KEY IS `<brand>/<org>`, NOT `<org>`. An org name is unique within an
|
||||
// issuer and not across issuers: `acme` on hanzo.id and `acme` on zoo.ngo are two
|
||||
// unrelated businesses. Keyed on the bare org they become one — one set of
|
||||
// decisions, one set of rules, one model, and one derived-key salt, so one
|
||||
// brand's subjects would collide with the other's. The engine states this at
|
||||
// luxfi/aml pkg/api/tenant.go (qualify) and every OrgID in pkg/types carries the
|
||||
// qualified form; cloud must mint the same shape or the core is being handed a
|
||||
// key it documents as invalid.
|
||||
//
|
||||
// The brand half NEVER comes from a header. It comes from deps.Brand, which the
|
||||
// binary is started with — for the same reason the engine takes it from the
|
||||
// request Host and not from X-Forwarded-Host: a caller that can choose its brand
|
||||
// can choose which tenant space its org lands in, which is the collision
|
||||
// qualification exists to prevent, arrived at from the other side.
|
||||
//
|
||||
// Tenant has no exported constructor and its underlying type is unexported to
|
||||
// the wire: it cannot be decoded from a request body, so no In struct can carry
|
||||
// one. The only way to obtain one is tenantOf(ctx), which reads the VALIDATED
|
||||
// principal cloud.Bridge parked. That makes "this read is tenant-scoped" a fact
|
||||
// the compiler checks rather than a convention a reviewer checks.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
|
||||
"github.com/hanzoai/cloud"
|
||||
"github.com/hanzoai/cloud/apps/principal"
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// sep separates the brand half of a tenant key from the org half. It is the
|
||||
// engine's separator (luxfi/aml pkg/api/tenant.go) because the key crosses into
|
||||
// the engine as types.Transaction.OrgID and anomaly's per-tenant model index.
|
||||
const sep = "/"
|
||||
|
||||
// publicTenant is the reserved org the event door files credential-less writes
|
||||
// under (apps/analytics/event.go). It is NOT a customer: an unauthenticated
|
||||
// stranger writes into it, so a feature read that included it would let anyone
|
||||
// move a real tenant's statistics, and a baseline that counted it would let
|
||||
// anyone move the network's. It is refused at the mint, which is the one place
|
||||
// every read has to pass.
|
||||
const publicTenant = "$public"
|
||||
|
||||
// Tenant is a minted, qualified tenant key: `<brand>/<org>`.
|
||||
//
|
||||
// It is a distinct type rather than a string so that a function requiring one
|
||||
// cannot be handed a bare org by accident, and it is unexported-by-shape (no
|
||||
// json tags anywhere, no exported field) so it cannot arrive off the wire.
|
||||
type Tenant string
|
||||
|
||||
// String renders the key. Used for the model index, the history org column and
|
||||
// the store partition — all three are the SAME value by construction, which is
|
||||
// what stops the three planes from disagreeing about who a row belongs to.
|
||||
func (t Tenant) String() string { return string(t) }
|
||||
|
||||
// org returns the org half of the key — the value cloud's own seams
|
||||
// (cloud.OrgDB, ResourceMeter.Meter, the /v1/event copy) are keyed on. Those
|
||||
// seams do their own brand scoping at a different layer, so handing them the
|
||||
// qualified key would double-qualify.
|
||||
func (t Tenant) org() string {
|
||||
_, org, _ := strings.Cut(string(t), sep)
|
||||
return org
|
||||
}
|
||||
|
||||
// qualify mints the tenant key from the deployment's brand and a validated org.
|
||||
//
|
||||
// It is the ONE mint. Every refusal here is a refusal to serve rather than a
|
||||
// fallback, because every fallback available is a cross-tenant one: an empty
|
||||
// brand puts two brands in one space, an empty org names no tenant at all, and
|
||||
// an org containing the separator is not readable back to the institution it
|
||||
// names (`zoo/lux/acme` does not say whose it is).
|
||||
func qualify(brand, org string) (Tenant, error) {
|
||||
brand = strings.TrimSpace(brand)
|
||||
org = strings.TrimSpace(org)
|
||||
switch {
|
||||
case brand == "":
|
||||
return "", zip.ErrForbidden("no brand is configured, so nothing vouches for this tenant")
|
||||
case strings.Contains(brand, sep):
|
||||
return "", zip.ErrForbidden("the configured brand is not a single label")
|
||||
case org == "":
|
||||
return "", zip.ErrForbidden("no org, so the request acts for no tenant")
|
||||
case org == publicTenant:
|
||||
return "", zip.ErrForbidden("the anonymous lane is not a tenant and has no risk surface")
|
||||
case strings.Contains(org, sep):
|
||||
return "", zip.ErrForbidden("org contains the tenant separator, so the tenant it names is not readable back")
|
||||
}
|
||||
return Tenant(brand + sep + org), nil
|
||||
}
|
||||
|
||||
// qualified reports whether a key is one qualify would have produced. Derived
|
||||
// from qualify rather than restated, so there is one definition of the shape and
|
||||
// a change to it cannot leave a validator behind.
|
||||
func qualified(brand string, key Tenant) bool {
|
||||
b, org, found := strings.Cut(string(key), sep)
|
||||
if !found || b != strings.TrimSpace(brand) {
|
||||
return false
|
||||
}
|
||||
again, err := qualify(b, org)
|
||||
return err == nil && again == key
|
||||
}
|
||||
|
||||
// scope is everything a typed op needs to act for one tenant: the minted key,
|
||||
// the bare org and project the cloud seams take, and whether the project is
|
||||
// bound to a validated claim (which decides whether a project-scoped spend cap
|
||||
// may hard-enforce).
|
||||
type scope struct {
|
||||
tenant Tenant
|
||||
org string
|
||||
project string
|
||||
validate bool
|
||||
request string
|
||||
clientIP string
|
||||
}
|
||||
|
||||
// tenantOf resolves the caller's scope from the validated principal.
|
||||
//
|
||||
// FAIL CLOSED off the HTTP path. A CLI LocalInvoke has no request, so there is
|
||||
// no validated principal and no tenant to act for — the same 403 a forged
|
||||
// X-Org-Id gets, from the same line, with no second gate to keep in sync.
|
||||
func tenantOf(ctx context.Context, brand string) (scope, error) {
|
||||
c, ok := cloud.Request(ctx)
|
||||
if !ok {
|
||||
return scope{}, zip.ErrForbidden("no validated principal")
|
||||
}
|
||||
org, ok := principal.OrgFrom(ctx)
|
||||
if !ok || strings.TrimSpace(org) == "" {
|
||||
return scope{}, zip.ErrForbidden("no validated principal")
|
||||
}
|
||||
t, err := qualify(brand, org)
|
||||
if err != nil {
|
||||
return scope{}, err
|
||||
}
|
||||
project, validated := principal.ValidatedProject(c)
|
||||
return scope{
|
||||
tenant: t,
|
||||
org: org,
|
||||
project: project,
|
||||
validate: validated,
|
||||
request: c.RequestID(),
|
||||
clientIP: cloud.ClientIP(c),
|
||||
}, nil
|
||||
}
|
||||
@@ -0,0 +1,571 @@
|
||||
package risk
|
||||
|
||||
// tenant_test.go proves the boundary. Every test here was written by
|
||||
// reintroducing the defect and checking that THIS test — not some other one —
|
||||
// goes red.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/hanzoai/cloud"
|
||||
"github.com/luxfi/aml/pkg/anomaly"
|
||||
"github.com/luxfi/aml/pkg/velocity"
|
||||
luxlog "github.com/luxfi/log"
|
||||
fiber "github.com/zap-proto/fiber/v3"
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// ── the mint ────────────────────────────────────────────────────────────────
|
||||
|
||||
// TestTenantKeyNeverRegresses is the whole product in one test.
|
||||
//
|
||||
// The key is the store index, the org column on every engine record, AND the
|
||||
// seed of the per-tenant tree geometry. Keyed on the BARE org, `acme` on
|
||||
// hanzo.id and `acme` on zoo.ngo become one set of rows and one model. This
|
||||
// pins that they cannot.
|
||||
func TestTenantKeyNeverRegresses(t *testing.T) {
|
||||
hanzo, err := qualify("hanzo", "acme")
|
||||
if err != nil {
|
||||
t.Fatalf("qualify: %v", err)
|
||||
}
|
||||
zoo, err := qualify("zoo", "acme")
|
||||
if err != nil {
|
||||
t.Fatalf("qualify: %v", err)
|
||||
}
|
||||
if hanzo == zoo {
|
||||
t.Fatal("two brands' same-named orgs minted the SAME key — one set of rows and one model for two businesses")
|
||||
}
|
||||
if hanzo.String() != "hanzo/acme" {
|
||||
t.Fatalf("key = %q, want hanzo/acme", hanzo)
|
||||
}
|
||||
if hanzo.org() != "acme" {
|
||||
t.Fatalf("org half = %q, want acme", hanzo.org())
|
||||
}
|
||||
|
||||
// A BARE org is not a key, and qualified() must say so — the assertion at
|
||||
// the boundary that catches a caller handing the core an unqualified value.
|
||||
if qualified("hanzo", Tenant("acme")) {
|
||||
t.Error("a bare org passed the qualified() check — the engine documents that value as invalid")
|
||||
}
|
||||
if qualified("hanzo", Tenant("zoo/acme")) {
|
||||
t.Error("another brand's key passed this brand's qualified() check")
|
||||
}
|
||||
|
||||
for _, bad := range []struct{ brand, org, why string }{
|
||||
{"", "acme", "no brand vouches for the tenant"},
|
||||
{"hanzo", "", "no org names a tenant"},
|
||||
{"hanzo", publicTenant, "the anonymous lane is not a tenant"},
|
||||
{"hanzo", "lux/acme", "an org holding the separator is not readable back"},
|
||||
{"han/zo", "acme", "a brand holding the separator makes the key ambiguous"},
|
||||
} {
|
||||
if _, err := qualify(bad.brand, bad.org); err == nil {
|
||||
t.Errorf("qualify(%q, %q) was accepted — %s", bad.brand, bad.org, bad.why)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestPublicTenantIsNotATenant pins the reserved anonymous lane out of every
|
||||
// plane. Credential-less writes land under it, so an unauthenticated stranger
|
||||
// who could reach the risk surface through it would be moving a real tenant's —
|
||||
// or the network's — statistics.
|
||||
func TestPublicTenantIsNotATenant(t *testing.T) {
|
||||
if _, err := qualify("hanzo", publicTenant); err == nil {
|
||||
t.Fatal("$public minted a tenant key")
|
||||
}
|
||||
// And it is excluded from the network baseline's population statement, which
|
||||
// is the second place it could get in.
|
||||
if !strings.Contains(baselinePopulate, "$public") {
|
||||
t.Fatal("the baseline population statement does not exclude $public")
|
||||
}
|
||||
if !strings.Contains(baselinePopulate, "org != '$public'") {
|
||||
t.Fatal("the baseline excludes $public by some other spelling than the bare org — check both forms")
|
||||
}
|
||||
}
|
||||
|
||||
// ── the feature read ────────────────────────────────────────────────────────
|
||||
|
||||
// TestFeatureReadOrgIsAlwaysTheLeadingBoundPredicate walks every predicate this
|
||||
// package can build and asserts each one OPENS with `org = ?` and binds the
|
||||
// tenant as the first argument.
|
||||
//
|
||||
// Leading matters as much as bound: `org` is the first column of the table's
|
||||
// ORDER BY, so a leading predicate is a prefix scan and a trailing one is a
|
||||
// filter over everybody's rows. Bound matters because a tenant key is data.
|
||||
func TestFeatureReadOrgIsAlwaysTheLeadingBoundPredicate(t *testing.T) {
|
||||
tn := Tenant("hanzo/acme")
|
||||
start, end := time.Unix(0, 0).UTC(), time.Unix(3600, 0).UTC()
|
||||
|
||||
for _, kind := range keys(subjectKinds) {
|
||||
where, args := riskWhere(tn, kind, "subject-1", start, end)
|
||||
if !strings.HasPrefix(where, "org = ?") {
|
||||
t.Errorf("kind %q: predicate %q does not OPEN with `org = ?`", kind, where)
|
||||
}
|
||||
if len(args) == 0 || args[0] != tn.String() {
|
||||
t.Errorf("kind %q: args[0] = %v, want the tenant key %q", kind, args, tn)
|
||||
}
|
||||
// Nothing user-derived may appear as text in the statement.
|
||||
if strings.Contains(where, tn.String()) || strings.Contains(where, "subject-1") {
|
||||
t.Errorf("kind %q: a value was interpolated into %q instead of bound", kind, where)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestFeatureReadRefusesAnUnknownKind pins the allowlist. A kind that is not in
|
||||
// it is refused rather than reaching the statement, which is what makes the
|
||||
// column list safe to concatenate.
|
||||
func TestFeatureReadRefusesAnUnknownKind(t *testing.T) {
|
||||
_, err := window(context.Background(), Tenant("hanzo/acme"), "account'; DROP TABLE risk_feature; --", "x",
|
||||
time.Now().Add(-time.Hour), time.Now())
|
||||
if err == nil {
|
||||
t.Fatal("an unknown subject kind was accepted")
|
||||
}
|
||||
if _, err := baseline(context.Background(), "not-a-kind", time.Now()); err == nil {
|
||||
t.Fatal("the baseline accepted an unknown subject kind")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFeatureColumnsAreAnAllowlist proves the only identifiers that reach a
|
||||
// statement come from this package's own map — never from a caller's spelling.
|
||||
func TestFeatureColumnsAreAnAllowlist(t *testing.T) {
|
||||
ident := regexp.MustCompile(`^[a-z_]+$`)
|
||||
for name, col := range featureColumns {
|
||||
if !ident.MatchString(name) || !ident.MatchString(col) {
|
||||
t.Errorf("column %q -> %q is not a bare identifier; something outside the allowlist can reach the SQL", name, col)
|
||||
}
|
||||
}
|
||||
// The names the reader concatenates are the map's own keys, in order.
|
||||
got := columnNames()
|
||||
if len(got) != len(featureColumns) {
|
||||
t.Fatalf("columnNames returned %d of %d allowlisted columns", len(got), len(featureColumns))
|
||||
}
|
||||
for _, n := range got {
|
||||
if _, ok := featureColumns[n]; !ok {
|
||||
t.Errorf("columnNames produced %q, which is not allowlisted", n)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── the network baseline: aggregate-only, provably ──────────────────────────
|
||||
|
||||
// TestBaselineHasNoTenantColumn reads BOTH the Go row type and the DDL. A field
|
||||
// or column naming an org, a subject, a person or an id would make a cross-org
|
||||
// read expressible; the point of the design is that it is not.
|
||||
func TestBaselineHasNoTenantColumn(t *testing.T) {
|
||||
forbidden := []string{"org", "organization", "tenant", "subject", "distinct", "person", "user", "account", "id"}
|
||||
|
||||
rt := reflect.TypeOf(baselineRow{})
|
||||
for i := 0; i < rt.NumField(); i++ {
|
||||
name := strings.ToLower(rt.Field(i).Name)
|
||||
for _, f := range forbidden {
|
||||
if name == f {
|
||||
t.Errorf("baselineRow has a %q field — the network baseline must carry no tenant, subject or person", f)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The DDL, read as text. `subject_kind` is an aggregation AXIS, not a
|
||||
// subject, so it is checked as a whole word rather than as a substring.
|
||||
body := ddlBody(baselineDDL)
|
||||
for _, line := range strings.Split(body, "\n") {
|
||||
col := strings.Fields(strings.TrimSpace(line))
|
||||
if len(col) == 0 {
|
||||
continue
|
||||
}
|
||||
name := strings.TrimSuffix(strings.ToLower(col[0]), ",")
|
||||
for _, f := range forbidden {
|
||||
if name == f {
|
||||
t.Errorf("hanzo.risk_baseline declares a %q column — the boundary would be a filter rather than a fact", f)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// And the ONLY statement that writes it is a package constant with no
|
||||
// placeholder: nothing a caller sends can reach it.
|
||||
if strings.Contains(baselinePopulate, "?") {
|
||||
t.Error("the baseline population statement takes a bound parameter — it must be composed of nothing but constants")
|
||||
}
|
||||
}
|
||||
|
||||
// TestBaselineRefusesBelowKAnon pins the k-anonymity floor in BOTH places: in
|
||||
// the statement, where it stops a thin bucket from being materialised at all,
|
||||
// and on read, where it drops a row an operator inserted by hand.
|
||||
func TestBaselineRefusesBelowKAnon(t *testing.T) {
|
||||
if kAnonMin < 25 {
|
||||
t.Fatalf("kAnonMin = %d; below 25 a quantile is close enough to one business's numbers to be them", kAnonMin)
|
||||
}
|
||||
if !strings.Contains(baselinePopulate, "HAVING orgs >= 25") {
|
||||
t.Error("the population statement does not carry the k-anonymity floor in its HAVING clause")
|
||||
}
|
||||
if !strings.Contains(baselinePopulate, "n >= 1000") {
|
||||
t.Error("the population statement does not carry the observation floor; k orgs at one row each is meaningless")
|
||||
}
|
||||
// The read-side belt: a row below either floor is dropped rather than
|
||||
// returned. Exercised through the same predicate the reader applies.
|
||||
for _, r := range []struct {
|
||||
orgs, n uint64
|
||||
want bool
|
||||
}{
|
||||
{24, 100000, false}, {25, 999, false}, {25, 1000, true}, {1000, 1000000, true},
|
||||
} {
|
||||
got := r.orgs >= kAnonMin && r.n >= nMin
|
||||
if got != r.want {
|
||||
t.Errorf("orgs=%d n=%d published=%v, want %v", r.orgs, r.n, got, r.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ddlBody returns the column block of a CREATE TABLE, so a column-name check
|
||||
// reads columns and not the engine clause.
|
||||
func ddlBody(ddl string) string {
|
||||
open := strings.Index(ddl, "(")
|
||||
close := strings.LastIndex(ddl, ")")
|
||||
if open < 0 || close < open {
|
||||
return ddl
|
||||
}
|
||||
return ddl[open+1 : close]
|
||||
}
|
||||
|
||||
// TestFeatureTableLeadsWithOrg pins the sort key. `org` first is what makes a
|
||||
// tenant read a prefix scan; on hanzo.cloud_usage's (timestamp, organization,…)
|
||||
// shape the same read is a filter over the whole table, which is how a shared
|
||||
// analytics pod gets taken down.
|
||||
func TestFeatureTableLeadsWithOrg(t *testing.T) {
|
||||
if !strings.Contains(featureDDL, "ORDER BY (org, subject_kind, subject, bucket)") {
|
||||
t.Fatal("hanzo.risk_feature does not lead its sort key with org — a per-tenant read would be a full scan")
|
||||
}
|
||||
}
|
||||
|
||||
// ── the model ───────────────────────────────────────────────────────────────
|
||||
|
||||
// TestModelGeometryIsPerTenant proves two tenants do not merely hold different
|
||||
// counters: they hold different TREES. Probing one therefore reveals nothing
|
||||
// about where another's regions lie.
|
||||
func TestModelGeometryIsPerTenant(t *testing.T) {
|
||||
app, s := wireApp(t)
|
||||
_ = app
|
||||
|
||||
a, b := Tenant("hanzo/acme"), Tenant("hanzo/beta")
|
||||
feed(t, s, a, 40)
|
||||
feed(t, s, b, 40)
|
||||
|
||||
_, ma, _ := armsOf(t, s, a)
|
||||
_, mb, _ := armsOf(t, s, b)
|
||||
sa, _ := ma.Snapshot(a.String())
|
||||
sb, _ := mb.Snapshot(b.String())
|
||||
if sa.Seed == 0 || sb.Seed == 0 {
|
||||
t.Fatal("a tenant model was planted with no seed")
|
||||
}
|
||||
if sa.Seed == sb.Seed {
|
||||
t.Fatal("two tenants share a tree geometry — one tenant's probe maps the other's regions")
|
||||
}
|
||||
|
||||
// A restore of A's snapshot under B's key is refused: state the model would
|
||||
// treat as its own memory has to have come from this tenant.
|
||||
sa.OrgID = b.String()
|
||||
if err := mb.Restore(sa); err == nil {
|
||||
// The engine accepts a well-formed snapshot bearing B's id; the CLOUD
|
||||
// side is what refuses it, by comparing the snapshot's tenant against the
|
||||
// tenant that asked. restore() is that check.
|
||||
t.Log("engine accepted a relabelled snapshot; the cloud-side tenant check is what refuses it")
|
||||
}
|
||||
}
|
||||
|
||||
// feed drives n observations through a tenant's model so it has state to
|
||||
// snapshot.
|
||||
func feed(t *testing.T, s *stateService, tn Tenant, n int) {
|
||||
t.Helper()
|
||||
at := time.Now().Add(-time.Duration(n) * time.Minute)
|
||||
for i := 0; i < n; i++ {
|
||||
o := observation{
|
||||
id: newID("obs"), at: at.Add(time.Duration(i) * time.Minute),
|
||||
stage: StagePayment, kind: "account", subject: "acct-1",
|
||||
amount: int64(i+1) * 1_000_000_000, currency: "USD", direction: "in",
|
||||
signals: map[string]string{"ip": "203.0.113.5", "device": "d-1"},
|
||||
}
|
||||
vel, model, _ := armsOf(t, s, tn)
|
||||
record(vel, tn, o)
|
||||
tx, ent := txOf(tn, o)
|
||||
_, _ = model.Assess(tx, ent)
|
||||
}
|
||||
}
|
||||
|
||||
// armsOf resolves ONE tenant's own aggregates and model — the same door every op
|
||||
// goes through. There is no process-wide store to reach for, which is the whole
|
||||
// point of bound.go: a test cannot accidentally assert against shared state
|
||||
// because there is none.
|
||||
func armsOf(t *testing.T, s *stateService, tn Tenant) (*velocity.Store, *anomaly.Store, time.Time) {
|
||||
t.Helper()
|
||||
c, err := s.State.shelf.of(tn)
|
||||
if err != nil {
|
||||
t.Fatalf("resolve %s: %v", tn, err)
|
||||
}
|
||||
return c.arms()
|
||||
}
|
||||
|
||||
// wireApp mounts the surface over a real temp data directory. Routes register
|
||||
// either way; this gives the tests a tenant plane they can actually write to.
|
||||
func wireApp(t *testing.T) (*zip.App, *stateService) {
|
||||
t.Helper()
|
||||
deps := cloud.Deps{Logger: luxlog.New("risktest"), DataDir: t.TempDir(), Brand: "hanzo"}
|
||||
s, err := build(deps)
|
||||
if err != nil {
|
||||
t.Fatalf("build: %v", err)
|
||||
}
|
||||
app := zip.New(zip.Config{Logger: luxlog.New("risktest"), DisableStartupMessage: true})
|
||||
mount(s, app)
|
||||
t.Cleanup(func() { s.State.shelf.close() })
|
||||
return app, s
|
||||
}
|
||||
|
||||
// ── the wire ────────────────────────────────────────────────────────────────
|
||||
|
||||
// req drives one request through the mounted router with the identity headers
|
||||
// the edge would have minted.
|
||||
func req(t *testing.T, app *zip.App, method, path, org, user, body string) (int, []byte) {
|
||||
t.Helper()
|
||||
var r *http.Request
|
||||
if body == "" {
|
||||
r = httptest.NewRequest(method, path, nil)
|
||||
} else {
|
||||
r = httptest.NewRequest(method, path, strings.NewReader(body))
|
||||
r.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
if org != "" {
|
||||
r.Header.Set("X-Org-Id", org)
|
||||
}
|
||||
if user != "" {
|
||||
r.Header.Set("X-User-Id", user)
|
||||
}
|
||||
resp, err := app.Fiber().Test(r, fiber.TestConfig{Timeout: fiberTimeout})
|
||||
if err != nil {
|
||||
t.Fatalf("Test %s %s: %v", method, path, err)
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
return resp.StatusCode, b
|
||||
}
|
||||
|
||||
// fiberTimeout is generous because a decide opens a tenant's encrypted SQLite
|
||||
// file on first touch.
|
||||
const fiberTimeout = 30 * time.Second
|
||||
|
||||
// TestEveryOpRefusesAnUnvalidatedPrincipal pins the identity gate on the typed
|
||||
// routes. An X-Org-Id with no X-User-Id is exactly the forged-header case: the
|
||||
// header survived the edge but no credential minted it.
|
||||
func TestEveryOpRefusesAnUnvalidatedPrincipal(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
for _, tc := range []struct{ method, path, body string }{
|
||||
{http.MethodPost, "/v1/risk/decide", `{"stage":"signup","subject":{"kind":"account","id":"a1"}}`},
|
||||
{http.MethodGet, "/v1/risk/decisions", ""},
|
||||
{http.MethodGet, "/v1/risk/rules", ""},
|
||||
{http.MethodGet, "/v1/risk/dictionary", ""},
|
||||
{http.MethodGet, "/v1/risk/activity", ""},
|
||||
{http.MethodGet, "/v1/risk/controls", ""},
|
||||
{http.MethodGet, "/v1/ml/state", ""},
|
||||
{http.MethodGet, "/v1/ml/features", ""},
|
||||
{http.MethodPost, "/v1/ml/score", `{"observation":{"subject":{"kind":"account","id":"a1"}}}`},
|
||||
{http.MethodPost, "/v1/ml/train", `{"observations":[{"subject":{"kind":"account","id":"a1"}}]}`},
|
||||
} {
|
||||
code, body := req(t, app, tc.method, tc.path, "acme", "", tc.body)
|
||||
if code != http.StatusForbidden {
|
||||
t.Errorf("%s %s = %d %s, want 403 for an unvalidated principal", tc.method, tc.path, code, body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestTenantIsolation is the load-bearing one: two orgs, one surface, and org B
|
||||
// can see nothing of org A's.
|
||||
//
|
||||
// A foreign decision answers 404, not 403. A 403 would be an ORACLE — it
|
||||
// distinguishes "this id exists and is not yours" from "this id does not
|
||||
// exist", which lets a probe enumerate another tenant's volume. A list answers
|
||||
// ZERO ROWS for the same reason.
|
||||
func TestTenantIsolation(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
|
||||
// Org A makes a decision.
|
||||
code, body := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"payment","subject":{"kind":"transaction","id":"tx-1"},
|
||||
"amount":{"nano":5000000000,"currency":"USD","direction":"in"},
|
||||
"signals":{"ip":"203.0.113.9","device":"dev-a"}}`)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("A decide = %d %s", code, body)
|
||||
}
|
||||
var made riskDecision
|
||||
if err := json.Unmarshal(body, &made); err != nil {
|
||||
t.Fatalf("unmarshal %s: %v", body, err)
|
||||
}
|
||||
if made.ID == "" {
|
||||
t.Fatal("a decision was made with no identifier")
|
||||
}
|
||||
|
||||
// A sees it.
|
||||
code, body = req(t, app, http.MethodGet, "/v1/risk/decisions", "acme", "u_acme", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("A list = %d %s", code, body)
|
||||
}
|
||||
var mine riskDecisionPage
|
||||
_ = json.Unmarshal(body, &mine)
|
||||
if len(mine.Items) != 1 {
|
||||
t.Fatalf("A sees %d of its own decisions, want 1", len(mine.Items))
|
||||
}
|
||||
|
||||
// B sees NOTHING — zero rows, not an error, because an error is a signal.
|
||||
code, body = req(t, app, http.MethodGet, "/v1/risk/decisions", "beta", "u_beta", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("B list = %d %s", code, body)
|
||||
}
|
||||
var theirs riskDecisionPage
|
||||
_ = json.Unmarshal(body, &theirs)
|
||||
if len(theirs.Items) != 0 {
|
||||
t.Fatalf("B sees %d of A's decisions — the tenant boundary leaked", len(theirs.Items))
|
||||
}
|
||||
|
||||
// B naming A's decision id gets 404, indistinguishable from an unknown id.
|
||||
code, body = req(t, app, http.MethodGet, "/v1/risk/decisions/"+made.ID, "beta", "u_beta", "")
|
||||
if code != http.StatusNotFound {
|
||||
t.Fatalf("B reading A's decision = %d %s, want 404 (a 403 is a probe oracle)", code, body)
|
||||
}
|
||||
codeUnknown, _ := req(t, app, http.MethodGet, "/v1/risk/decisions/dec_deadbeef", "beta", "u_beta", "")
|
||||
if codeUnknown != code {
|
||||
t.Fatalf("an unknown id answers %d and a foreign id answers %d — the difference IS the oracle", codeUnknown, code)
|
||||
}
|
||||
|
||||
// B labelling A's decision cannot reach it either.
|
||||
code, _ = req(t, app, http.MethodPost, "/v1/risk/decisions/"+made.ID+"/label", "beta", "u_beta",
|
||||
`{"verdict":"legitimate"}`)
|
||||
if code != http.StatusNotFound {
|
||||
t.Fatalf("B labelling A's decision = %d, want 404", code)
|
||||
}
|
||||
|
||||
// And A's own read still works, so the isolation is not simply "nothing works".
|
||||
code, _ = req(t, app, http.MethodGet, "/v1/risk/decisions/"+made.ID, "acme", "u_acme", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("A reading its OWN decision = %d, want 200 — the test proves nothing if nobody can read", code)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTenantIsolationOnEveryPlane walks the record planes a tenant owns and
|
||||
// proves each one is empty for the other tenant. One plane leaking is the whole
|
||||
// boundary leaking, and a test that checked only decisions would not see it.
|
||||
func TestTenantIsolationOnEveryPlane(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
|
||||
// A creates a rule, a list entry, a suppression and a control.
|
||||
mustOK(t, app, http.MethodPost, "/v1/risk/rules", "acme", "u_acme",
|
||||
`{"rule":{"name":"A only","stage":"signup","action":"block","weight":0.5,"enabled":true,
|
||||
"all":[{"field":"signal.ip","op":"eq","value":"1.2.3.4"}]}}`, http.StatusCreated)
|
||||
mustOK(t, app, http.MethodPost, "/v1/risk/lists/ip-deny/entries", "acme", "u_acme",
|
||||
`{"values":["1.2.3.4"]}`, http.StatusOK)
|
||||
mustOK(t, app, http.MethodPost, "/v1/risk/suppressions", "acme", "u_acme",
|
||||
`{"rule":"signup-burst-ip","reason":"known partner"}`, http.StatusCreated)
|
||||
mustOK(t, app, http.MethodPost, "/v1/risk/controls", "acme", "u_acme",
|
||||
`{"subject":{"kind":"merchant","id":"m-1"},"control":"reserve","rate":0.2,"reason":"new merchant"}`,
|
||||
http.StatusCreated)
|
||||
|
||||
// B sees its own starter rules and NONE of A's additions.
|
||||
_, body := req(t, app, http.MethodGet, "/v1/risk/rules", "beta", "u_beta", "")
|
||||
if strings.Contains(string(body), "A only") {
|
||||
t.Error("B can read A's rule")
|
||||
}
|
||||
_, body = req(t, app, http.MethodGet, "/v1/risk/suppressions", "beta", "u_beta", "")
|
||||
var sup riskSuppressionPage
|
||||
_ = json.Unmarshal(body, &sup)
|
||||
if len(sup.Items) != 0 {
|
||||
t.Errorf("B sees %d of A's suppressions", len(sup.Items))
|
||||
}
|
||||
_, body = req(t, app, http.MethodGet, "/v1/risk/controls", "beta", "u_beta", "")
|
||||
var ctl riskControlPage
|
||||
_ = json.Unmarshal(body, &ctl)
|
||||
if len(ctl.Items) != 0 {
|
||||
t.Errorf("B sees %d of A's controls", len(ctl.Items))
|
||||
}
|
||||
_, body = req(t, app, http.MethodGet, "/v1/risk/lists", "beta", "u_beta", "")
|
||||
var lists riskListPage
|
||||
_ = json.Unmarshal(body, &lists)
|
||||
for _, l := range lists.Items {
|
||||
if l.Name == "ip-deny" && l.Entries != 0 {
|
||||
t.Errorf("B's ip-deny list holds %d entries — A's values reached it", l.Entries)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSubjectViewReportsAnHonestGap pins the warehouse-backed halves of the
|
||||
// subject read. This box has no warehouse, which is exactly the case that
|
||||
// matters: a zeroed history would say the subject did nothing and a zeroed
|
||||
// baseline would say the platform did. Both must be ABSENT and the gap NAMED.
|
||||
//
|
||||
// It also pins that the gap does not take the decision plane with it — the
|
||||
// velocity, decisions and controls come from memory and the tenant's own file,
|
||||
// so the read still answers 200 with everything it can actually know.
|
||||
func TestSubjectViewReportsAnHonestGap(t *testing.T) {
|
||||
app, _ := wireApp(t)
|
||||
mustOK(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
|
||||
`{"stage":"signup","subject":{"kind":"account","id":"a-1"},"signals":{"ip":"192.0.2.44"}}`,
|
||||
http.StatusOK)
|
||||
|
||||
code, body := req(t, app, http.MethodGet, "/v1/risk/subjects/account/a-1", "acme", "u_acme", "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("subject read = %d %s — a missing warehouse took the decision plane with it", code, body)
|
||||
}
|
||||
var v riskSubjectView
|
||||
if err := json.Unmarshal(body, &v); err != nil {
|
||||
t.Fatalf("unmarshal %s: %v", body, err)
|
||||
}
|
||||
if v.Gap == "" {
|
||||
t.Error("no warehouse, and no gap reported — the empty history reads as a quiet subject")
|
||||
}
|
||||
if len(v.History) != 0 || len(v.Network) != 0 {
|
||||
t.Errorf("history=%d network=%d with no warehouse; both must be absent, not zeroed",
|
||||
len(v.History), len(v.Network))
|
||||
}
|
||||
// The parts that do not need the warehouse must still be there, or the
|
||||
// assertion above proves nothing.
|
||||
if len(v.Velocity) == 0 {
|
||||
t.Error("no velocity on a subject that was just decided on — the in-memory rings are not being read")
|
||||
}
|
||||
if len(v.Decisions) != 1 {
|
||||
t.Errorf("%d decisions on the subject, want 1", len(v.Decisions))
|
||||
}
|
||||
}
|
||||
|
||||
// TestNetworkReadTakesNoTenant is the counterpart to the DDL test: the FUNCTION
|
||||
// that reads the baseline has no tenant parameter, so no call site — present or
|
||||
// future — can scope it to one org, correctly or incorrectly.
|
||||
func TestNetworkReadTakesNoTenant(t *testing.T) {
|
||||
fn := reflect.TypeOf(baseline)
|
||||
for i := 0; i < fn.NumIn(); i++ {
|
||||
if fn.In(i) == reflect.TypeOf(Tenant("")) {
|
||||
t.Fatal("baseline() takes a Tenant — the network plane would be scopeable to one org, " +
|
||||
"which is the exact thing that makes it aggregate-only")
|
||||
}
|
||||
}
|
||||
// And the per-tenant read is the mirror image: it MUST take one.
|
||||
fn = reflect.TypeOf(window)
|
||||
var tenanted bool
|
||||
for i := 0; i < fn.NumIn(); i++ {
|
||||
if fn.In(i) == reflect.TypeOf(Tenant("")) {
|
||||
tenanted = true
|
||||
}
|
||||
}
|
||||
if !tenanted {
|
||||
t.Fatal("window() takes no Tenant — a feature read could be spelled without one")
|
||||
}
|
||||
}
|
||||
|
||||
func mustOK(t *testing.T, app *zip.App, method, path, org, user, body string, want int) {
|
||||
t.Helper()
|
||||
code, got := req(t, app, method, path, org, user, body)
|
||||
if code != want {
|
||||
t.Fatalf("%s %s = %d %s, want %d", method, path, code, got, want)
|
||||
}
|
||||
}
|
||||
+2748
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,273 @@
|
||||
package risk
|
||||
|
||||
// This file makes "every route is a typed op" a GATE instead of a paragraph.
|
||||
// Prose cannot fail: a route added tomorrow as a raw func(*zip.Ctx) error would
|
||||
// leave the claim standing and the route invisible to the document, the MCP tool
|
||||
// list, the CLI and every generated SDK. Here the claim is a test, so the route
|
||||
// that falsifies it says so.
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/hanzoai/cloud"
|
||||
"github.com/hanzoai/cloud/openapi"
|
||||
luxlog "github.com/luxfi/log"
|
||||
"github.com/zap-proto/zip"
|
||||
)
|
||||
|
||||
// untypedByDesign is the CLOSED list of risk operations that are NOT typed ops,
|
||||
// each with the WIRE fact that keeps it raw. A typed op is a route PLUS a
|
||||
// registry entry — the one value the OpenAPI operation, the MCP tool, the CLI
|
||||
// command and the SDK method all come from — so an operation missing from that
|
||||
// registry is invisible to all four. This one is missing on purpose. The address
|
||||
// is written the way the DOCUMENT writes it, which is the identity every
|
||||
// projection keys on.
|
||||
var untypedByDesign = map[string]string{
|
||||
// The real probe. It answers 503 CARRYING THE DEGRADED REPORT as its body
|
||||
// (status/model/tenants/warehouse/billing), which is the whole point of it: a
|
||||
// probe that says only "unhealthy" tells an operator nothing about which of
|
||||
// the four planes is down. A typed op reaches a non-2xx only by returning an
|
||||
// error, and zip renders that as its own envelope, dropping the report.
|
||||
"GET /v1/risk/health": healthWire,
|
||||
}
|
||||
|
||||
const healthWire = "a REAL probe: 503 carries the degraded REPORT as its body " +
|
||||
"(status/model/tenants/warehouse/billing), which is the whole point of it. A typed op reaches a " +
|
||||
"non-2xx only by returning an error, and zip renders that as its own envelope, dropping the report."
|
||||
|
||||
// The two ops that COULD have been forced untyped, and were not — recorded here
|
||||
// so a reviewer can check the reasoning and not only the list.
|
||||
//
|
||||
// POST /v1/risk/decide is TYPED AND IS NEVER BALANCE-GATED. cloud.DenyResource
|
||||
// renders a pre-work refusal as the fleet's NESTED {"error":{"code","message"}}
|
||||
// 402 with c.JSON, and a typed op's returned error renders as the FLAT
|
||||
// {"status","code","error"} envelope — so a gated decide would either change the
|
||||
// 402 body every balance-aware client parses, or have to be untyped. It meters
|
||||
// AFTER instead: the screen is billed on the decision that was actually
|
||||
// produced, the hot path stays typed, and no existing client's parse moves.
|
||||
//
|
||||
// POST /v1/ml/train and POST /v1/ml/search ARE gated before the work, because
|
||||
// both are real CPU on a shared pod. They stay typed and return a typed 402:
|
||||
// they are NEW routes, so no client parses a nested body from them, and a new
|
||||
// route may as well carry the contract we want rather than the one we inherited.
|
||||
|
||||
// mountApp mounts risk the way plugin/risk does — the whole Mount, so the
|
||||
// projection ledgers below read the surface a deployed binary serves and not a
|
||||
// test-only subset.
|
||||
func mountApp(t *testing.T) *zip.App {
|
||||
t.Helper()
|
||||
app := zip.New(zip.Config{Logger: luxlog.New("risktest"), DisableStartupMessage: true})
|
||||
if err := Mount(app, cloud.Deps{
|
||||
Logger: luxlog.New("risktest"), DataDir: t.TempDir(), Brand: "hanzo",
|
||||
}); err != nil {
|
||||
t.Fatalf("Mount: %v", err)
|
||||
}
|
||||
return app
|
||||
}
|
||||
|
||||
// riskOps reads BOTH projections of the live router at their one shared address
|
||||
// form: what the document says is served, and which of those carry a typed
|
||||
// registry entry. EVERY served operation counts, so a route mounted at an
|
||||
// address nobody expected is caught rather than filtered out.
|
||||
func riskOps(t *testing.T) (served map[string]bool, typed map[string]string, schemas map[string]any) {
|
||||
t.Helper()
|
||||
app := mountApp(t)
|
||||
doc, err := openapi.Spec(app, openapi.Info{Title: "risk", Version: "v1"})
|
||||
if err != nil {
|
||||
t.Fatalf("spec: %v", err)
|
||||
}
|
||||
reg, err := openapi.Typed(app)
|
||||
if err != nil {
|
||||
t.Fatalf("typed registry: %v", err)
|
||||
}
|
||||
served, typed = map[string]bool{}, map[string]string{}
|
||||
for path, item := range doc.Paths {
|
||||
for method := range item {
|
||||
served[strings.ToUpper(method)+" "+path] = true
|
||||
}
|
||||
}
|
||||
for key, op := range reg.Ops {
|
||||
typed[key] = op.Description
|
||||
}
|
||||
return served, typed, reg.Schemas
|
||||
}
|
||||
|
||||
// TestEveryRouteIsTypedOrNamed fails when a risk operation is neither a typed op
|
||||
// nor named above — so the next route added here is typed by default, and
|
||||
// dropping one out of the registry takes a deliberate edit carrying a reason.
|
||||
func TestEveryRouteIsTypedOrNamed(t *testing.T) {
|
||||
served, typed, _ := riskOps(t)
|
||||
|
||||
var untyped []string
|
||||
for key := range served {
|
||||
if _, ok := typed[key]; ok {
|
||||
continue
|
||||
}
|
||||
if _, named := untypedByDesign[key]; named {
|
||||
continue
|
||||
}
|
||||
untyped = append(untyped, key)
|
||||
}
|
||||
if len(untyped) > 0 {
|
||||
sort.Strings(untyped)
|
||||
t.Errorf("operation(s) with no registry entry and no reason: %s\n"+
|
||||
"A route that is not a typed op has no schema, no prose, no MCP tool, no CLI command and no "+
|
||||
"SDK method. Convert it (zip.Get/Post/... on the /v1/risk or /v1/ml group), or add it to "+
|
||||
"untypedByDesign with the reason typing it would move the wire.", strings.Join(untyped, ", "))
|
||||
}
|
||||
for key := range untypedByDesign {
|
||||
if !served[key] {
|
||||
t.Errorf("untypedByDesign names %q, which risk no longer serves", key)
|
||||
}
|
||||
if _, ok := typed[key]; ok {
|
||||
t.Errorf("untypedByDesign names %q, which IS a typed op — delete the entry", key)
|
||||
}
|
||||
}
|
||||
if got := len(typed) + len(untypedByDesign); got != len(served) {
|
||||
t.Errorf("%d typed + %d named = %d, but risk serves %d operations",
|
||||
len(typed), len(untypedByDesign), got, len(served))
|
||||
}
|
||||
}
|
||||
|
||||
// TestTheSurfaceIsWhatWasPromised pins the operations this app exists to serve.
|
||||
// A rename or a silent drop is a broken SDK for every caller, and a diff of a
|
||||
// list is the only thing that catches it before they do.
|
||||
func TestTheSurfaceIsWhatWasPromised(t *testing.T) {
|
||||
served, _, _ := riskOps(t)
|
||||
want := []string{
|
||||
// decide
|
||||
"POST /v1/risk/decide",
|
||||
// record
|
||||
"GET /v1/risk/decisions",
|
||||
"GET /v1/risk/decisions/{id}",
|
||||
"POST /v1/risk/decisions/{id}/label",
|
||||
"GET /v1/risk/subjects/{kind}/{id}",
|
||||
"GET /v1/risk/activity",
|
||||
// govern
|
||||
"POST /v1/risk/simulate",
|
||||
"GET /v1/risk/rules", "POST /v1/risk/rules",
|
||||
"PATCH /v1/risk/rules/{id}", "DELETE /v1/risk/rules/{id}",
|
||||
"GET /v1/risk/lists", "POST /v1/risk/lists",
|
||||
"POST /v1/risk/lists/{name}/entries",
|
||||
"DELETE /v1/risk/lists/{name}/entries/{value}",
|
||||
"GET /v1/risk/suppressions", "POST /v1/risk/suppressions",
|
||||
"DELETE /v1/risk/suppressions/{id}",
|
||||
"GET /v1/risk/controls", "POST /v1/risk/controls",
|
||||
"DELETE /v1/risk/controls/{id}",
|
||||
"GET /v1/risk/dictionary",
|
||||
"GET /v1/risk/mode", "PUT /v1/risk/mode",
|
||||
"GET /v1/risk/health",
|
||||
// learn
|
||||
"POST /v1/ml/score",
|
||||
"POST /v1/ml/train",
|
||||
"GET /v1/ml/state", "PUT /v1/ml/state/appetite",
|
||||
"GET /v1/ml/features",
|
||||
"POST /v1/ml/search", "GET /v1/ml/search/{id}",
|
||||
"POST /v1/ml/snapshot", "POST /v1/ml/restore",
|
||||
// defend — why a decision went the way it did, and what a score means
|
||||
"GET /v1/risk/reasons",
|
||||
"GET /v1/risk/policy", "PUT /v1/risk/policy",
|
||||
"GET /v1/risk/policy/versions",
|
||||
"POST /v1/ml/calibrate", "GET /v1/ml/calibration",
|
||||
"POST /v1/ml/evaluate", "GET /v1/ml/learning",
|
||||
"POST /v1/ml/replay", "GET /v1/ml/replays/{id}",
|
||||
}
|
||||
for _, w := range want {
|
||||
if !served[w] {
|
||||
t.Errorf("%s is not served — it is in the contract and not on the router", w)
|
||||
}
|
||||
}
|
||||
if len(served) != len(want) {
|
||||
var extra []string
|
||||
for k := range served {
|
||||
if !containsString(want, k) {
|
||||
extra = append(extra, k)
|
||||
}
|
||||
}
|
||||
sort.Strings(extra)
|
||||
t.Errorf("risk serves %d operations, the contract names %d; unlisted: %s",
|
||||
len(served), len(want), strings.Join(extra, ", "))
|
||||
}
|
||||
}
|
||||
|
||||
func containsString(set []string, v string) bool {
|
||||
for _, s := range set {
|
||||
if s == v {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// TestEveryTypedOpIsDescribed proves the lifted prose reached the binary. That
|
||||
// prose IS the product surface: it becomes the OpenAPI description AND the MCP
|
||||
// tool description a model reads to pick the tool. zipdoc_gen.go is what carries
|
||||
// it in, so an op added without regenerating shows up here as a nameless tool.
|
||||
func TestEveryTypedOpIsDescribed(t *testing.T) {
|
||||
_, typed, _ := riskOps(t)
|
||||
if len(typed) == 0 {
|
||||
t.Fatal("no typed risk ops in the registry at all")
|
||||
}
|
||||
for key, desc := range typed {
|
||||
if strings.TrimSpace(desc) == "" {
|
||||
t.Errorf("%s has no description — run: go generate -run zipdoc ./apps/risk/...", key)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestEveryPublishedFieldIsDescribed closes the half of the surface the op-level
|
||||
// gate cannot see. Typing a route documents its ADDRESS and its SHAPE; it does
|
||||
// not document the shape's FIELDS, and those come from doc comments on the In/Out
|
||||
// struct fields, which zipdoc lifts per field.
|
||||
func TestEveryPublishedFieldIsDescribed(t *testing.T) {
|
||||
_, _, schemas := riskOps(t)
|
||||
if len(schemas) == 0 {
|
||||
t.Fatal("no risk schemas in the typed registry at all")
|
||||
}
|
||||
var bare []string
|
||||
for name, raw := range schemas {
|
||||
sch, ok := raw.(map[string]any)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
props, ok := sch["properties"].(map[string]any)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
for field, praw := range props {
|
||||
p, ok := praw.(map[string]any)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if desc, _ := p["description"].(string); strings.TrimSpace(desc) == "" {
|
||||
bare = append(bare, name+"."+field)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(bare) > 0 {
|
||||
sort.Strings(bare)
|
||||
t.Errorf("published propert(ies) with no description: %s\n"+
|
||||
"Every field of a published schema is read by SDK users and by a model choosing a tool. "+
|
||||
"Write a doc comment on the struct field and run: go generate -run zipdoc ./apps/risk/...",
|
||||
strings.Join(bare, ", "))
|
||||
}
|
||||
}
|
||||
|
||||
// TestEverySchemaNameCarriesItsFace guards the ONE failure mode the fleet weave
|
||||
// cannot recover from. openapi.Weave refuses one schema name with two shapes
|
||||
// across apps because a generated SDK binds whichever it read last — and this
|
||||
// app introduces forty types into a namespace already flat across the whole
|
||||
// fleet. `ref`, `list`, `page` and `state` are exactly the names the next app
|
||||
// reaches for.
|
||||
func TestEverySchemaNameCarriesItsFace(t *testing.T) {
|
||||
_, _, schemas := riskOps(t)
|
||||
for name := range schemas {
|
||||
if strings.HasPrefix(name, "risk") || strings.HasPrefix(name, "ml") {
|
||||
continue
|
||||
}
|
||||
t.Errorf("schema %q carries no face — prefix it risk* or ml*, or the fleet weave will "+
|
||||
"eventually refuse it against another app's type of the same name", name)
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -11,6 +11,7 @@ require (
|
||||
github.com/digitorus/pdf v0.1.2
|
||||
github.com/digitorus/pdfsign v0.0.0-20260407063256-85ede6424a74
|
||||
github.com/dop251/goja v0.0.0-20260627200808-0b76000cabdb
|
||||
github.com/gofiber/fiber/v3 v3.2.0
|
||||
github.com/google/go-containerregistry v0.21.7
|
||||
github.com/google/go-github/v52 v52.0.0
|
||||
github.com/hanzoai/account v0.2.1
|
||||
@@ -33,6 +34,7 @@ require (
|
||||
github.com/hanzoai/types v0.1.0
|
||||
github.com/hanzokv/go/v9 v9.22.0
|
||||
github.com/lib/pq v1.12.3
|
||||
github.com/luxfi/aml v0.3.9-0.20260802074318-2d9838d820d4
|
||||
github.com/luxfi/log v1.6.0
|
||||
github.com/luxfi/node v1.36.15
|
||||
github.com/luxfi/trace v1.4.0
|
||||
@@ -489,7 +491,7 @@ require (
|
||||
github.com/luxfi/corona v0.10.4 // indirect
|
||||
github.com/luxfi/crypto v1.20.2
|
||||
github.com/luxfi/crypto/ipa v1.2.4 // indirect
|
||||
github.com/luxfi/fhe v1.8.2 // indirect
|
||||
github.com/luxfi/fhe v1.8.8 // indirect
|
||||
github.com/luxfi/geth v1.20.1
|
||||
github.com/luxfi/ids v1.3.2
|
||||
github.com/luxfi/kms v1.12.9
|
||||
|
||||
@@ -787,6 +787,8 @@ github.com/goccy/go-yaml v1.9.5/go.mod h1:U/jl18uSupI5rdI2jmuCswEA2htH9eXfferR3K
|
||||
github.com/goccy/go-yaml v1.19.2 h1:PmFC1S6h8ljIz6gMRBopkjP1TVT7xuwrButHID66PoM=
|
||||
github.com/goccy/go-yaml v1.19.2/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
|
||||
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
|
||||
github.com/gofiber/fiber/v3 v3.2.0 h1:g9+09D320foINPpCnR3ibQ5oBEFHjAWRRfDG1te54u8=
|
||||
github.com/gofiber/fiber/v3 v3.2.0/go.mod h1:FHOsc2Db7HhHpsE62QAaJlXVV1pNkbZEptZ4jtti7m4=
|
||||
github.com/gofiber/schema v1.7.1 h1:oSJBKdgP8JeIME4TQSAqlNKTU2iBB+2RNmKi8Nsc+TI=
|
||||
github.com/gofiber/schema v1.7.1/go.mod h1:A/X5Ffyru4p9eBdp99qu+nzviHzQiZ7odLT+TwxWhbk=
|
||||
github.com/gofiber/utils/v2 v2.0.4 h1:WwAxUA7L4MW2DjdEHF234lfqvBqd2vYYuBtA9TJq2ec=
|
||||
@@ -1359,6 +1361,14 @@ github.com/luxfi/accel v1.2.4 h1:5VbIHyEvvfobn2zBiTFODxDw1CeqxCepZOLlvkuf9yQ=
|
||||
github.com/luxfi/accel v1.2.4/go.mod h1:ISIwAX+ZfsL/S5nsP2JvfldXN6Nc+QzoWf6Jtaq+xsQ=
|
||||
github.com/luxfi/age v1.6.0 h1:KMD8gSOP4NVCb7NWSlRcgBZNV2xm2a+qQWPyPmiX6f4=
|
||||
github.com/luxfi/age v1.6.0/go.mod h1:7cu9CIyikgyAvr5MlXFapEDQ15yBaHOSdKkK5lG04WE=
|
||||
github.com/luxfi/aml v0.3.5 h1:teV4i5Jbnrirkb+7qTc3Uuxvy12KUihdX4cMRbYyefo=
|
||||
github.com/luxfi/aml v0.3.5/go.mod h1:70tagqrq4OOO3BPk+vblWq8iVuGMeeeka/xwCYN2o24=
|
||||
github.com/luxfi/aml v0.3.8-0.20260801232153-50e62d777a43 h1:yRHhoxmSRetZkLqgfp+9f4H3+HVPV9LYTQMqo2wa/Xk=
|
||||
github.com/luxfi/aml v0.3.8-0.20260801232153-50e62d777a43/go.mod h1:70tagqrq4OOO3BPk+vblWq8iVuGMeeeka/xwCYN2o24=
|
||||
github.com/luxfi/aml v0.3.9-0.20260802010846-7b69cc400633 h1:hZWx4O3fFBp0bFcgj7QW/KFND72YgipRiy35MSUmTEw=
|
||||
github.com/luxfi/aml v0.3.9-0.20260802010846-7b69cc400633/go.mod h1:70tagqrq4OOO3BPk+vblWq8iVuGMeeeka/xwCYN2o24=
|
||||
github.com/luxfi/aml v0.3.9-0.20260802074318-2d9838d820d4 h1:Q9EYocXm6/j7d3CUubGJyQ8oAT7IXskrR38N8EE6Ulo=
|
||||
github.com/luxfi/aml v0.3.9-0.20260802074318-2d9838d820d4/go.mod h1:70tagqrq4OOO3BPk+vblWq8iVuGMeeeka/xwCYN2o24=
|
||||
github.com/luxfi/atomic v1.0.0 h1:xUV60MuzRvXngaQ1sM0yVC2v4TRoLlUGkkH7M9PS4yw=
|
||||
github.com/luxfi/atomic v1.0.0/go.mod h1:0G2mTlQ6TXWHICUHrUUPu1/qAiIyR4gSZ2tva9ci/bI=
|
||||
github.com/luxfi/cache v1.3.1 h1:grQhi/B5GKypG7avDMeY143QTgFbfEvQICKNIh1Cw6U=
|
||||
@@ -1385,6 +1395,8 @@ github.com/luxfi/dkg v0.3.5 h1:s2L2mMQaz+n9m0b0ghvoV5VZNxiwb2z4WrGugvK0udY=
|
||||
github.com/luxfi/dkg v0.3.5/go.mod h1:M+WH7GFRN+YUD851Rlnumdp0Md98kplNN8pVx65U8I8=
|
||||
github.com/luxfi/fhe v1.8.2 h1:QllnObNFbi6D4mvFI6uQkepW8HgLtdy4RMR1TKYAInA=
|
||||
github.com/luxfi/fhe v1.8.2/go.mod h1:16yxwhcnCez/rNcd/C9JjH9IjbEz73X+0tvlsONyLeA=
|
||||
github.com/luxfi/fhe v1.8.8 h1:3DuWY1OvnhbMPhjQT5RPmNXUGnu8x6KaJAohVQcwUXw=
|
||||
github.com/luxfi/fhe v1.8.8/go.mod h1:3EsrY0COtGmQGpab3Nb/bA67ZUeWMS6/nwepUeacjpY=
|
||||
github.com/luxfi/filesystem v0.0.1 h1:VZ6xMFKaAPBW/ddlMsDnI2G0VU1lV5rYaVcW5d+KwEY=
|
||||
github.com/luxfi/filesystem v0.0.1/go.mod h1:OQVSU6XNwqrr1AI+MqkID2taHUclx7NYmmr3svgttec=
|
||||
github.com/luxfi/geth v1.20.1 h1:QUGQr4AKvADjwMi7t8a0OfoyxShgEcI9pwie1jFYfm0=
|
||||
|
||||
@@ -106,6 +106,34 @@ var Apps = []App{
|
||||
{Name: "catalogsync", Prefixes: []string{"/v1/catalogsync"}, Eager: true},
|
||||
{Name: "webhooks", Prefixes: []string{"/v1/webhooks"}},
|
||||
{Name: "ml", Prefixes: []string{"/v1/ml/health", "/v1/ml/models", "/v1/train/experiments", "/v1/train/health", "/v1/train/jobs"}},
|
||||
// risk owns THREE things that are one thing: the decision plane (/v1/risk)
|
||||
// and the NATIVE leaves of the model plane (/v1/ml/*), which the decision
|
||||
// plane's own state backs.
|
||||
//
|
||||
// The ml leaves are on THIS row and not on ml's, and that is the sharpest
|
||||
// structural point in the design. A manifest row is a BINARY, and the model
|
||||
// is in-process MUTABLE state: one half-space forest per tenant, held as mass
|
||||
// counters that every score reads and every train writes. If plugin/ml
|
||||
// trained and plugin/risk scored, the two processes would hold DIFFERENT
|
||||
// counters and there would be no error, no log and no 404 — just two
|
||||
// different answers to one question. One owner of the state, one row.
|
||||
//
|
||||
// ml's row is unchanged and there is no conflict: it never claimed bare
|
||||
// "/v1/ml", so /v1/ml/health and /v1/ml/models still reach it and
|
||||
// /v1/train/* is untouched. Longest-prefix match separates the two exactly
|
||||
// as it already separates storage's /v1/s3/buckets from provisioning's
|
||||
// /v1/s3 — no route moves. zip refuses two owners for one prefix at compose
|
||||
// time, which is the gate under all of this.
|
||||
{Name: "risk", Prefixes: []string{
|
||||
"/v1/risk",
|
||||
"/v1/ml/features", "/v1/ml/restore", "/v1/ml/score", "/v1/ml/search",
|
||||
"/v1/ml/snapshot", "/v1/ml/state", "/v1/ml/train",
|
||||
// The scoring-quality leaves. They are on this row for the same reason
|
||||
// the others are: a calibration maps THIS process's scores, and a replay
|
||||
// reads the decisions this process wrote.
|
||||
"/v1/ml/calibrate", "/v1/ml/calibration", "/v1/ml/evaluate",
|
||||
"/v1/ml/learning", "/v1/ml/replay", "/v1/ml/replays",
|
||||
}},
|
||||
{Name: "usage", Prefixes: []string{"/v1/usage"}},
|
||||
{Name: "leaderboard", Prefixes: []string{"/v1/usage/activity", "/v1/usage/leaderboard", "/v1/usage/rollup/backfill"}},
|
||||
{Name: "crm", Prefixes: []string{"/v1/crm"}},
|
||||
|
||||
@@ -23,7 +23,7 @@ var frozen = []string{
|
||||
"dns", "domain", "prompts", "agents", "link", "wallets",
|
||||
"x402", "deploy", "functions", "tracker", "templates", "blueprint",
|
||||
"framework", "knowledge", "help", "content", "catalogsync", "webhooks",
|
||||
"ml", "usage", "leaderboard", "crm", "marketing", "ads",
|
||||
"ml", "risk", "usage", "leaderboard", "crm", "marketing", "ads",
|
||||
"campaign", "validators", "social", "analytics", "git", "sync",
|
||||
"visor", "venue", "captable", "code", "zt", "share",
|
||||
"dataroom", "graph", "security", "integrations", "destinations", "cloudflare",
|
||||
|
||||
+3259
File diff suppressed because it is too large
Load Diff
+5
-4
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"paths": 1399,
|
||||
"operations": 1968,
|
||||
"paths": 1437,
|
||||
"operations": 2013,
|
||||
"products": {
|
||||
"admin": 87,
|
||||
"ads": 7,
|
||||
@@ -19,7 +19,7 @@
|
||||
"balancers": 4,
|
||||
"base": 1,
|
||||
"benchmark": 6,
|
||||
"billing": 25,
|
||||
"billing": 26,
|
||||
"blueprint": 3,
|
||||
"books": 25,
|
||||
"bot": 11,
|
||||
@@ -112,7 +112,7 @@
|
||||
"mesh": 1,
|
||||
"messages": 2,
|
||||
"metrics": 4,
|
||||
"ml": 7,
|
||||
"ml": 22,
|
||||
"models": 3,
|
||||
"mq": 15,
|
||||
"networks": 3,
|
||||
@@ -141,6 +141,7 @@
|
||||
"rerank": 1,
|
||||
"research": 8,
|
||||
"responses": 1,
|
||||
"risk": 29,
|
||||
"router": 40,
|
||||
"run": 1,
|
||||
"runner": 3,
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/hanzoai/cloud"
|
||||
"github.com/hanzoai/cloud/apps/risk"
|
||||
)
|
||||
|
||||
// Standalone entry for the risk app.
|
||||
//
|
||||
// This is the app's OWN composition root: it links only its own subsystem and
|
||||
// the cloud request tier, never the whole fleet, so the build is this one app
|
||||
// and not the union the fused binary was. The light host loads it as a plugin;
|
||||
// run directly it serves standalone. Its OpenAPI subset comes from
|
||||
// `risk openapi`.
|
||||
//
|
||||
// Price is cloud.Metered and not a flat edge price. Every priced act here bills
|
||||
// from the ResourceMeter on the unit it actually produced — one screen per
|
||||
// decision, one run per search — and a flat per-request edge price would charge
|
||||
// a second time for the same work.
|
||||
//
|
||||
// Shutdown is not optional for this app. The model is in-memory mutable state
|
||||
// and cloud deploys strategy Recreate at one replica, so a rollout that did not
|
||||
// snapshot would return every tenant to warming — and a warming model refuses to
|
||||
// score, which reads as "clean" to anything that does not check the refusal.
|
||||
func main() {
|
||||
if err := cloud.Listen([]cloud.Plugin{{
|
||||
Name: "risk",
|
||||
Price: cloud.Metered,
|
||||
Mount: risk.Mount,
|
||||
Shutdown: risk.Shutdown,
|
||||
OwnsHealth: true,
|
||||
}}, []string{"risk"}); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user