Compare commits

...
11 Commits
Author SHA1 Message Date
hanzo-dev db7688f4df openapi: raise the floor for the route main added without it
/v1/billing/credits landed on main (commerce's row and its subset both name it)
and openapi/floor.json was not raised in the same commit, so the weave lowers
nothing and simply reports a count one higher than the recorded floor. Weaving
on a PRISTINE origin/main produces exactly this one-line bump, so it is main's
and not this branch's — carried here because a branch whose own weave is not
idempotent cannot prove anything about its own surface.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-02 12:20:37 -07:00
hanzo-dev c8bd42d514 risk: call Listen, the verb the composition root now has — and write the plane down
main renamed cloud.Serve to Listen (f403c603) while this branch was out; every
other plugin main moved with it and this one did not, so the app that carries
the decision plane was the one binary in the fleet that no longer compiled.

LLM.md gains the scoring contract, because these are the facts that cost the
most to re-derive: a Map cannot be read without naming the coordinates (Under
returns a Reader and a Reader has no constructor, so the skew gate is a thing
the compiler checks rather than a thing a call site remembers); ties are the
only case on the score alphabet this plane produces; a decision records the
shape it was scored under so a refit cannot re-bless stale rows; one cell per
tenant under that tenant's own bound; every record ships before it is
acknowledged, retirements included; and the health probe is unauthenticated,
so it carries a count and never a tenant key.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-02 12:20:37 -07:00
hanzo-dev ec6847f8fa risk: the durability table is one harness, and the sweep is bounded like everything else
THE TABLE COULD NOT HAVE CAUGHT ANYTHING. Five of its cases addressed
identifiers this plane never mints — rules, mutes and controls all get a
server-chosen id, so the retirements were asking a 404 whether it shipped, and
a control's subject is an object, not a string. Each id is READ BACK from its
create now, and a fixture that stops matching the wire is a setup failure
rather than a green test proving nothing. The label route was missing outright
and the companion guard said so, which is the guard doing its job on its first
run.

It is also ONE harness instead of sixteen. Every subject a retirement retires
is created while the store is up, the store fails once, and every write runs
against it — 75 seconds rather than the minutes sixteen encrypted re-opens
cost to prove the same thing.

THE SWEEP HELD THE REGISTRY LOCK ACROSS N ENCRYPTED WRITES. A retire writes
the tenant's model to its own file, and every other organisation's cell lookup
queues behind that lock. Unbounded, one pass over an idle shelf serialises
tenantMax snapshots under it and stalls the decision path for every
organisation on the pod — the same fleet-wide degradation the request path is
built to rule out, arriving through the housekeeping door. Four per pass: one
is all an admission at the ceiling needs, and the rest are idle by definition
so nothing waits on them.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-02 12:20:37 -07:00
hanzo-dev 69371ec43c deps: aml at the branch tip the cloud plane is tested against
The pin named 2d9838d's parent — valid, since it is an ancestor of the branch —
but a citation that is not the tip is one somebody has to resolve by hand. The
commit it moves to adds a test and no behaviour: a rounding collision between
two isotonic centroids pools rather than picks.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-02 12:20:37 -07:00
hanzo-dev 9adf3601d8 risk: every record ships, including the ones that take a control away
THREE MORE WRITES WERE ACKNOWLEDGED WITHOUT SHIPPING, and they are the ones
whose loss is worst.

A DELETE THAT DOES NOT SHIP RESURRECTS. An unshipped create loses a control
nobody depended on yet. An unshipped RETIREMENT — a rule deleted, a mute
lifted, a payout hold released, a deny-list entry removed — brings the old
state back at the next rollout, firing or blocking, after a 204 said it was
gone. The live/shadow switch is the sharpest of them: an organisation that
answered 200 to "stop acting" and then went back to declining payments.

So the durability table is now built from the ROUTES. Every mutating path in
the published subset is either exercised against a dead object store, or named
in a short list of ops that record nothing. A new op that writes and does not
ship fails a test that reads plugin/risk/openapi.json, which is the same
artifact the SDKs come from — this stops being a rule somebody has to remember
at each call site.

THE PROBE NAMED ITS TENANTS. GET /v1/<app>/health is unauthenticated by design
across this fleet — the billing gate, the tracing filter and the identity
middleware each exempt it by suffix — so the capacity report's list of
strained organisations was a customer list served to an anonymous GET, along
with which pod holds each. It is a COUNT now. The organisation that needs the
fact — its own rings are partial, so a `velocity.ip.1h.count >= 5` rule is
being measured against a ring that dropped keys — is told on its own scoped
state, GET /v1/ml/state, and no other organisation is told anything.

Also: the appetite is a governed statement of how much of the stream may be
examined, so it ships too.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-02 12:20:37 -07:00
hanzo-dev 2692cbd327 risk: a tenant's memory is its own, and a record is not a record until it ships
Three defects, each an instance of a class red has now found in five planes.

ONE STORE, ONE GLOBAL CAP, EVERY TENANT IN IT. The velocity store was
process-wide with velocity's default 100,000-key LRU, and the anomaly store
was process-wide with a global MaxOrgs LRU over tenants. Both evict. A
velocity shard is chosen by hashing the whole key, so tenants are mixed
together in one and the busiest org DELETES another org's counters; the
victim's rules then read zero, fire nothing and report success. The anomaly
LRU is worse: a new tenant's arrival drops an incumbent's learned model, and a
model returned to warming REFUSES to score, which reads on a screen exactly
like a clean customer. Neither failure raises anything.

State is per tenant with a per-tenant bound now. A cell holds one tenant's
file, one tenant's aggregates and one tenant's forest; aggregates() and
forest() are the only constructors in the package, both unexported, both
always bounded, and forest pins MaxOrgs to 1 so the cross-tenant eviction path
is unreachable rather than unlikely. The worst case is arithmetic and it is
written down: 6,048 B per key, an 8 MiB per-tenant cap, 48 tenants, ~400 MiB.
At the ceiling a NEW tenant is refused — 503, an error log and a degraded
probe — because admitting it by evicting an incumbent is the same defect
wearing a different hat. Memory comes back from a tenant's OWN silence, on a
timer, floored at the longest window so every ring it held had already rotated
to zero.

THE RECORD PLANE WAS NOT DURABLE. It opened bare OrgDBs, so the pod's volume
was the only copy of every decision — and cloud deploys Recreate at one
replica. The two sibling durable planes (apps/research, apps/books) both go
through OrgStore and Sync after every commit; this one now does the same,
through ONE commit() that writes and ships as a single step, so "written" and
"durable" cannot come apart at a call site. An unacked ship is an ERROR, never
a warning: unacked means this pod is not the org's elected writer, so the
local row is not the org's record and answering 200 over it would promise a
decision that a takeover will not find.

THE EXHAUSTIVE SEARCH HAD NO BOUND. Gated and metered, but nothing stopped one
tenant from holding N detached goroutines each grinding a 243-candidate grid
for ten minutes on the single-replica pod that serves every product on
api.hanzo.ai. One search per tenant at a time, on the same per-tenant
primitive the measurement plane already uses — a caller that loops the surface
spends its own slot and nobody else's — and the detached worker runs under a
named budget instead of a bare literal.

Also: the model digest is a pure function of the configuration, so it is
settled once at boot rather than read off whichever tenant's forest is at
hand; the reload latch IS the model now, so a dropped model can never be
stranded un-restorable for the life of the process; and the search sandbox is
built by the same two bounded constructors, so it cannot be the one place the
shared store comes back.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-02 12:20:37 -07:00
hanzo-dev 68edcebf04 risk: the suite is keyed the way every other test binary in the fleet is
cek opens nothing without a master, and a test process has no KMS to resolve
one from. This package answered that with its own TestMain setting an env var
nothing reads — a private copy of a decision that is stated once, in
internal/devmaster, and imported for its side effect by seventy other suites.
The copy was merely redundant until main moved the at-rest layer onto cek:
then every store this plane opens refused, and the whole decision path went
untestable.

One import, no TestMain, and the fact lives where the fleet keeps it.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-02 12:20:37 -07:00
blueandhanzo-dev dea22a725e risk: a probability is read under the shape in force, and a decision is one record
Six defects, all in the properties this plane is sold on.

TIES. The engine's isotonic fit did not pool tied scores, and the score
alphabet this plane produces is a handful of atoms — 1 - prod(1-weight)
over fixed rule weights, round4'd. Every plateau holding a positive was
reported as certainty. Fixed in luxfi/aml calibrate; the end-to-end proof
over the alphabet cloud actually writes is here.

THE SHAPE GATE WAS SELF-SATISFIED. Every internal read was
cal.P(score, cal.Shape) — the value being compared came from the value
being checked — so /v1/ml/evaluate, /v1/ml/replay and the reliability
chart all answered under a calibration the decide path refuses. A Map can
no longer be read at all: Under(shape) returns a Reader, bound once at the
boundary that knows the live shape, and every consumer holds a value with
no shape to pass.

A REFIT RE-BLESSED STALE COORDINATES. The gate's own remedy walked through
it: a fit read history scored under the old shape and stamped the new one
on it, clearing the control with no new evidence. Decisions record the
shape they were scored under, a fit reads one coordinate system, and the
refusal says how much evidence the boundary put out of reach.

THE BOUNDED READ TOOK THE OLDEST ROWS. Ascending plus LIMIT: past
maxHistory every measurement was frozen on the tenant's first 50k
decisions, with nothing saying the read was cut. It takes the most recent
now and reports truncation.

A MUTE MOVED THE DISTRIBUTION AND NOT THE SHAPE. combine() sums only
unsuppressed hits, so muting a rule for every subject moves every score it
touched exactly as retiring it would. Rule-wide mutes are in the shape; a
mute naming one subject is operational data and stays out, with the
argument stated.

THE LADDER VOIDED modelCeiling. The policy reads a probability that is a
pure function of the score the model's ceiling already capped, so an
uncapped escalation voided the cap by arithmetic. A decline now needs a
rule the organisation wrote behind it.

Also: the decision and its grading are ONE transaction and the read path
states a refusal instead of swallowing a missing one; governance records
name the person, not the org; measurement is bounded one-at-a-time PER
TENANT, runs under a deadline on cancellable reads, and is priced by the
rows it reads.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-02 12:20:37 -07:00
blueandhanzo-dev 2009b075ba risk: a score means a probability, a decision says why, and a change is tried before it is made
Ten typed zip ops on /v1/risk and /v1/ml, four durable records in the tenant's
own file, and three fields on the decision. One declaration each, so the route,
the OpenAPI operation, the MCP tool, the CLI command and every SDK come from the
same place.

REASON CODES. Every decision now carries the principal reasons it went the way
it did, strongest first, capped at four — the Reg B number, and the point past
which a list stops being an explanation. The codes are a closed vocabulary
DERIVED from the detector's own feature inventory, so it cannot name a feature
the model does not read; the model's weight is its exact counterfactual (one
coordinate to neutral, rescored on the same trees), never a surrogate. The
attribution is read off the assessment that produced the recorded score rather
than off the alert, so the reasons explain the number that was written down and
a decision below the line — or any decision in shadow — still has an
explanation. A suppressed hit contributed nothing and is not cited; the model's
own hit is not cited beside its features, or one piece of evidence counts twice.

CALIBRATION. A half-space tree's score is a density: it ranks, and it means
nothing. /v1/ml/calibrate fits isotonic or Platt on this organisation's own
judged decisions and the decision plane reports a probability — absent, never
zero, when no map is fitted. Every map records the SCORING SHAPE it was fitted
under: the detector's geometry and inventory folded with this tenant's enabled
rule set. Write a rule and the map refuses rather than answering for coordinates
that have moved. That is the training-serving skew control, and it is one
function used by the fit and by the decide path, so the two cannot disagree.

BANDS AND POLICY. Thresholds are a per-organisation versioned record with an
author, a stated reason and a digest over what the ladder DECIDES. Nothing is
edited: what was in force on the day someone was declined is a lookup. Evidence
and policy are two authorities and escalate() is the one place they become one
answer — the stronger wins, so a deny-list rule is not talked down by a low
probability and a high probability is not talked down by quiet evidence.

EVALUATION. Accuracy is not computed; on a one-in-a-thousand stream it is
misleading rather than weak. ROC-AUC with the prevalence beside it, average
precision by the step definition, Brier, lift, the confusion, and a
cost-weighted number in exact int64 nano with the cost-minimising threshold
reported next to the one in force. Every rate is a pointer: absent means absent.
The learning curve has two arms over a FIXED later window, so two steps are
comparable and no step can see the future.

REPLAY. A candidate ladder over recorded history, deterministic — the recorded
score is replayed and the model is never re-run, because it has since learned
from these very events. Two runs agree on the digest or the inputs moved, and
the report is written down, because it is the justification attached to a
threshold change.

All four planes are records: durable in the tenant's own cek-encrypted file
beside the decision log, no TTL, expiry belongs to retention alone. Nothing is
cached, so the Recreate rollout that drops every in-memory model needs no
rehydration here — the record IS the state.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-02 12:20:37 -07:00
hanzo-dev 6354816157 risk: the network baseline is published, the peer comparison reads it, and a retry cannot double-decide
Three things this closes, each of which was scaffolding or a defect.

The baseline plane was declared and never written, so the aggregate-only
cross-org boundary existed as a table and a test and not as a capability.
Mount now recomputes it on a timer — never on a route, which is what keeps the
cross-org surface unreachable by a caller: nobody can time it, steer it or
observe its cost, and the statement it runs is a package constant with no
placeholder. GET /v1/risk/subjects/{kind}/{id} reads it back as the peer
comparison, alongside the subject's thirty-day warehouse history. Both are
best effort and both name their gap when the warehouse is down: a zeroed
history would say the subject did nothing and a zeroed baseline would say the
platform did.

The analytics copy was published inline. PublishEvents dials the bus and
publishes in the caller's goroutine, and analytics itself only ever calls it
detached for that reason — on this path it would have been a bus dial inside a
card processor's authorization window. The record is already durable when it
runs, so there is nothing to wait for.

The idempotency key was claimed in a second statement after the insert, which
left a window: two simultaneous retries both found no row, both inserted, and
the loser's claim then violated the index, so a caller that retried correctly
got a 500 and the counters moved twice. The key is part of the insert now, so
the loser loses at the index before a second decision exists and reads the
winner's answer back. Eight concurrent retries under one key, one decision,
one row.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-02 12:20:37 -07:00
hanzo-dev 04aaee19a3 risk: the decision plane answers with a reason, and the model it reads is the tenant's own
apps/risk mounts /v1/risk and the native leaves of /v1/ml as 34 typed zip
operations, so one declaration produces the route, the OpenAPI operation, the
MCP tool, the CLI command and every generated SDK method. POST /v1/risk/decide
answers allow / challenge / review / restrict / block with the evidence behind
it, for any subject a tenant has, at any point in its lifecycle.

The ml leaves are on THIS manifest row rather than apps/ml's because the model
is in-process mutable state: two binaries holding two sets of mass counters
would give two answers to one question and raise no error. ml's row is
unchanged and longest-prefix match separates them, so no route moves.

github.com/luxfi/aml enters as a module dependency and only its transitively
base-free packages are linked. pkg/engine is NOT base-free — go list -deps has
it reaching hanzoai/base/core and hanzoai/tasks through pkg/history's
Base-backed store — so the rule plane is a closed algebra declared here: a
fixed field vocabulary, a fixed operator set, conjunction only. Injection-safe
by construction, and a typed wire shape rather than an opaque expression.

Every capability the embed earns is wired rather than assumed: cloud.Bridge on
each group before its leaves, the tenant read from the validated principal,
ResourceMeter metering the screen after the decision and gating train and
search before them, the scoped logger, and a real fail-closed probe.

Cross-org learning is aggregate-only at three layers. The only file that reads
a feature table takes a minted Tenant that cannot be decoded off the wire; the
network baseline is a table with no tenant column, written by one constant
statement behind a k-anonymity floor; and the tenant key is <brand>/<org>,
minted once, which is the store index, the engine's org column and the seed of
the per-tenant tree geometry at the same time.

Shadow is the default and the model's evidence is capped at review, so a
statistical judgement summons a person and never declines a payment. Every
decision is durable in the tenant's own encrypted file before the analytics
copy is emitted, because the event door drops by design. Shutdown snapshots
every resident model, because a Recreate rollout otherwise returns every tenant
to warming and a warming model refuses to score.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-02 12:20:37 -07:00
32 changed files with 21963 additions and 6 deletions
+181
View File
@@ -4067,3 +4067,184 @@ SDK and every MCP tool list that answer 404 in production: the same dark hole th
`api-hanzo-ai-catalog` router opened under `/v1/models` and `/v1/pricing`, which
cost 17 documented-but-uncallable operations. Re-home that surface only after the
upstream ships it, and prove the upstream answers before declaring anything.
## Hanzo Risk (`apps/risk`) — `/v1/risk` + the native `/v1/ml` leaves
One app, two prefix families, one core. `/v1/risk` is the DECISION plane for any
entity (account, transaction, session, agent, merchant, payout); `/v1/ml` is the
MODEL plane it is built on. Fraud is a USE of `/v1/risk` — so is abuse, so are
bots, account takeover, spam and pay-as-you-go abuse. **There is no `/v1/fraud`.**
**The ml leaves are on THIS row, not on `ml`'s, and that is the load-bearing
decision.** A manifest row is a BINARY, and the model is in-process MUTABLE state
(one half-space forest per tenant, held as mass counters). If `plugin/ml` trained
and `plugin/risk` scored, the two processes would hold different counters and
there would be NO error — just two different answers to one question. `ml`'s row
is unchanged and nothing moves: it never claimed bare `/v1/ml`, so longest-prefix
match separates `/v1/ml/models` (ml) from `/v1/ml/score` (risk), exactly as it
already separates `storage`'s `/v1/s3/buckets` from `provisioning`'s `/v1/s3`.
**The engine is a MODULE DEP, and only part of it is linked.** `github.com/luxfi/aml`
enters the way `luxfi/kms` and `hanzoai/o11y` do — no source vendored. Only four
packages are linked: `types`, `velocity`, `anomaly`, `replay`.
**MEASURED CORRECTION, worth not re-learning: `pkg/engine` is NOT base-free.**
A grep for `hanzoai/base` inside its own files finds nothing, which is how the
claim gets made. `go list -deps` finds the truth:
pkg/engine -> pkg/history -> pkg/store -> github.com/hanzoai/base/core
-> github.com/hanzoai/tasks (Temporal SDK)
Same for `pkg/measure` and `pkg/history`. Linking them would drag an application
framework and a workflow engine into a payment-authorization path. So the risk
rule plane is a CLOSED ALGEBRA declared cloud-side (`rule.go`): a fixed field
vocabulary, a fixed operator set, conjunction only. That is injection-safe by
construction, is a TYPED wire shape (so it reaches the SDKs, the CLI and the MCP
tools as a schema instead of an opaque expression string), and replays without an
interpreter. The upstream fix — split `pkg/history`'s Base-backed store into its
own package — is owed and would let `pkg/engine` in later without a rewrite here.
**The hot path never reads the warehouse.** A payment decide sits inside a
processor's authorization window and the datastore is one StatefulSet pod that has
taken api.hanzo.ai down before. `POST /v1/risk/decide` reads in-memory velocity
rings, the tenant's own SQLite, and the in-process model. ClickHouse
(`hanzo.risk_feature`) is the BACKFILL that warms the rings and the read behind
the dictionary and the search sandbox. Measure p50/p99 with the warehouse DOWN; a
decide that needs analytics up is a payment plane that fails when analytics does.
**Cross-org learning is AGGREGATE-ONLY and the boundary is three layers deep:**
type level apps/risk/feature.go is the ONLY file that reads a feature table,
and every function there takes a minted `Tenant`. Tenant has no
exported constructor and no json tags, so it cannot arrive off the
wire; tenantOf(ctx) is the only source and it reads the VALIDATED
principal cloud.Bridge parked.
value level hanzo.risk_baseline has NO tenant column, no subject, no id and no
pseudonym — quantiles only. There is no query that returns one
org's rows because the rows do not exist. The populate statement
is a package CONSTANT with no placeholder and a HAVING clause
carrying the k-anonymity floor (25 orgs, 1000 observations).
key level the tenant key is `<brand>/<org>` (tenant.go qualify), minted in
ONE place, and the brand half comes from deps.Brand and NEVER a
header. It is the store index, the engine's org column AND the
seed of the per-tenant tree geometry — so two tenants do not
merely hold different counters, they hold different TREES.
`$public` — the reserved anonymous lane the event door files credential-less
writes under — is refused at the mint AND excluded from the baseline statement.
**Shadow is the default and the default is not configurable.** Two gates in
series: the engine store is constructed `Shadow: true`, and a tenant's own record
plane holds `mode` (shadow|live, `PUT /v1/risk/mode`). In shadow every rule runs,
the model scores and learns, every decision is recorded, and the action is always
allow. Statistical evidence is capped at `review` (`modelCeiling`) and that cap is
NOT weakened for the payment stage: a model can put a transaction in front of a
person; it cannot decline one.
**Durable first, analytics after.** Every decision lands in the tenant's own
encrypted SQLite before `analytics.PublishEvents` emits the copy. `/v1/event` is
best-effort by design — it answers `{accepted, dropped}` and the anonymous lane
drops on purpose — so a record that rode it would be lost by a bus hiccup,
invisibly.
**Shutdown is not housekeeping.** cloud deploys `strategy: Recreate` at 1 replica,
so every rollout drops every in-memory model. `Plugin.Shutdown` snapshots every
resident tenant and `tenantState` restores on first touch. Without it every deploy
returns every tenant to warming — and a warming model REFUSES to score, which
reads as "clean" to anything that does not check `Refusal`.
**Metering: `decide` meters AFTER, `train` and `search` gate BEFORE.**
`cloud.DenyResource` writes the fleet's NESTED `{"error":{...}}` 402 in band and a
typed op's error renders FLAT, so a pre-work gate on the hot path would either
move the 402 body every balance-aware client parses or force the route untyped.
The screen is billed on the decision that was actually produced. `train` and
`search` are real CPU and ARE gated — they are new routes, so no client parses a
nested body from them and a typed 402 costs nothing.
**Untyped by design: exactly one route.** `GET /v1/risk/health` answers 503
carrying the degraded REPORT as its body. `apps/risk/typed_wire_test.go` holds the
closed list, and it also pins the whole served surface as a diffable list —
34 operations, 33 MCP tools.
**Build:** `-tags sqlite_math_functions` under CGO (hanzoai/base's gate).
`modernc.org/sqlite` is NOT in the risk graph, so the image's SQLITE-GATE stays
green — verify with
`CGO_ENABLED=1 go list -tags 'libsqlite3 sqlite_fts5 sqlite_math_functions' -deps ./plugin/risk | grep modernc`.
**No chart change is needed to serve it.** `api.hanzo.ai/` already routes to
cloud, `CLOUD_ENABLE` is unset in `charts/app/values/hanzo/cloud.yaml` (so every
manifest app is enabled), and the image builds one binary per `plugin/<app>`.
Landing it is an image pin. **`/v1/aml` is deliberately NOT claimed here** — the
Ingress still points that path at `service: aml` (the amld pod holding the live
5-year retention plane), and claiming it in cloud without deleting that rule in
the SAME commit gives a compliance surface answering from a store that is not the
record.
### What a score MEANS, and what a record IS (`apps/risk` + `luxfi/aml` scoring)
**A calibration cannot be read without naming the coordinates.** `calibrate.Map`
has no read method. `Under(shape)` is the only way to obtain a `Reader`, and a
`Reader` has no exported constructor — so a function taking one is a function
whose caller went through the trainingserving skew gate, and the compiler checks
it. The gate used to be `P(score, shape)` and every internal call site passed
`m.Shape`: the value being compared came from the value being checked, so the
control was inert everywhere except the one place a live shape happened to be
threaded in. `/v1/ml/evaluate`, `/v1/ml/replay` and the reliability chart all
answered under a map the decide path refuses. **A gate a caller can satisfy is
not a gate — bind at the boundary, not at the read.**
**Ties are the only case, not an edge case.** A decision score is
`1 - prod(1-weight)` over FIXED per-rule weights, `round4`'d — so the alphabet is
a handful of atoms and the modal one is exactly 0. Isotonic regression that pools
per SAMPLE rather than per DISTINCT SCORE reports every plateau holding a
positive as certainty. `pava` builds one block per distinct score, and a rounding
collision between two centroids POOLS rather than picking one: either endpoint
alone states a probability the pooled evidence does not support.
**A decision records the shape it was scored under** (`decision.shape`), a fit
reads ONE coordinate system, and the refusal says how much evidence the boundary
put out of reach (`superseded`). Without the column a refit re-blesses stale
rows — the gate's own documented remedy walking through the gate. The bounded
history read takes the NEWEST rows (`DESC` + `LIMIT n+1`) and reports truncation;
ascending plus a limit freezes every measurement on the tenant's first 50k
decisions, forever, silently.
**A rule-wide mute is IN the scoring shape.** `combine()` sums only unsuppressed
hits, so muting a rule for the whole population moves every score it touched
exactly as retiring the rule would. A mute naming a subject is operational data
and stays out.
**The policy ladder cannot decline on the model alone.** The calibrated
probability is a pure function of the same score `modelCeiling` already capped,
so an uncapped escalation would void the cap by arithmetic. `escalate` caps the
policy at what the evidence behind it can justify; a decline needs a rule the
organisation wrote.
**One cell per tenant; nothing is shared.** `bound.go` holds every number that
costs memory and the ONLY two constructors — `aggregates()` and `forest()`, both
unexported, both always bounded, `forest` pinning `MaxOrgs: 1` so the
cross-tenant eviction path is unreachable rather than unlikely. Worst case is
arithmetic: 6,048 B/key × an 8 MiB per-tenant cap (~1,386 keys) × 48 tenants
≈ 400 MiB. At the ceiling a NEW tenant is REFUSED (503 + error log + degraded
probe); admitting it by evicting an incumbent is the same defect wearing an
admission badge. Memory returns from a tenant's OWN silence, floored at the
longest window so every retired ring held zeros, and at most `retireBatch` (4)
snapshots are written per pass because the registry lock is held while they are.
**Every record ships before it is acknowledged.** `shelf.commit` writes to the
tenant's own file and `Sync`s it, fenced at the lease round, as ONE step — so
"written" and "durable" cannot come apart at a call site. An UNACKED ship is an
error, never a warning: unacked means this pod is not the org's elected writer.
The writes easiest to forget are the RETIREMENTS (rule deleted, mute lifted, hold
released, mode returned to shadow) — an unshipped delete brings the old state
back at the next rollout, firing or blocking, after a 204 said it was gone.
`TestEveryMutatingRouteIsCoveredByTheDurabilityTable` reads
`plugin/risk/openapi.json` and fails on any mutating route the table does not
exercise, so this stops being a rule somebody has to remember.
**`GET /v1/<app>/health` is unauthenticated by design across this fleet** — the
billing gate, the tracing filter and the identity middleware each exempt it by
suffix. So the capacity report carries a strained COUNT and never a tenant key: a
roster there is a customer list served to an anonymous GET, along with which pod
holds each. The organisation whose own rings are partial is told on its own
scoped state (`GET /v1/ml/state``strained`), and no other organisation is told
anything.
+8
View File
@@ -0,0 +1,8 @@
# Generated by plugin/gen-app-cmds. DO NOT EDIT.
#
# The build contract is mk/plugin.mk — one file carrying every target an app
# needs: build, test, vet, openapi, clean. This names the app(s) this package
# backs and includes it. Written from the same apps.Wire() parse that writes
# plugin/<app>/main.go, so an app cannot have a main and no Makefile.
APPS := risk
include ../../mk/plugin.mk
+219
View File
@@ -0,0 +1,219 @@
package risk
// bound.go is where every number that costs memory is decided, ONCE, and where
// the only constructors for the two in-memory planes live.
//
// THE DEFECT THIS FILE EXISTS TO MAKE UNREPRESENTABLE. One process-wide
// velocity.Store with a GLOBAL key cap, shared by every tenant, is not a
// performance choice. velocity.Store evicts the least-recently-updated key in a
// SHARD, and a shard is chosen by hashing the whole key — so tenants are mixed
// together in it and one org filling the store DELETES ANOTHER ORG'S counters.
// The victim's rules then read zero, fire nothing, and report success. A
// silently disarmed control is worse than an absent one, because nobody goes
// looking for it. anomaly.Store has the same shape one level up: a map of
// tenants with a global LRU, where a new tenant's arrival drops another's
// learned model and returns it to warming — which declines to score, and reads
// on a screen exactly like a clean customer.
//
// The shape that cannot express it: state is PER TENANT with a PER-TENANT bound.
// aggregates() and forest() are the ONLY constructors in this package, they are
// unexported, and they always apply the bound (pinned by
// TestOnlyTheBoundedConstructorsBuildThePlanes). A tenant that reaches its own
// bound evicts its OWN oldest key and nobody else's, and the probe says so.
//
// THE WORST CASE IS ARITHMETIC, NOT A HOPE. Every quantity below is a constant
// or an environment override, and the ceiling is their product:
//
// per key bucketBytes * buckets(windows()) + keyOverhead = 6,048 B
// per tenant velBytes (a CAP, allocated per key on demand) = 8 MiB → 1,386 keys
// + one half-space forest, allocated on arming ≈ 336 KiB
// process tenantMax tenants holding all of it = 48
// ≈ 400 MiB
//
// A tenant beyond tenantMax is REFUSED ADMISSION, loudly (503 + an error log +
// a degraded probe) — never admitted by evicting an incumbent, which is the
// defect wearing a different hat. Reclaim is driven by a tenant's OWN idleness
// and by nothing else, so no tenant's activity can ever cost another tenant a
// ring.
import (
"fmt"
"os"
"strconv"
"time"
"github.com/luxfi/aml/pkg/anomaly"
"github.com/luxfi/aml/pkg/velocity"
)
// windows is the ONE declaration of the sliding aggregates this plane keeps.
//
// The NAMES and SPANS are the engine's standard set, because they are the rule
// vocabulary: `velocity.<axis>.<window>.<stat>` is what a tenant's rules are
// written against, the dictionary publishes them, and anomaly's inventory reads
// them by name at construction.
//
// The RESOLUTIONS are this plane's own and they are lower than the engine's
// defaults on purpose. Those were chosen for statutory structuring detection
// over a compliance feed — 444 buckets per key, 22.7 KiB — which puts one
// tenant's plausible entity cardinality into gigabytes. Here the same four
// windows cost 94 buckets and 6 KiB per key, and the quantisation is stated
// rather than assumed: a window of W with B buckets resolves to W/B, so the 1h
// window is exact to five minutes and the 30d window to a day. Every rule in the
// starter set compares a COUNT or a SUM over a whole window and none of them can
// distinguish a boundary that fuzzy — precision that outruns the meaning of the
// number it measures is not precision, it is four times the memory.
func windows() []velocity.Window {
return []velocity.Window{
{Name: "1h", Span: time.Hour, Buckets: 12},
{Name: "24h", Span: 24 * time.Hour, Buckets: 24},
{Name: "7d", Span: 7 * 24 * time.Hour, Buckets: 28},
{Name: "30d", Span: 30 * 24 * time.Hour, Buckets: 30},
}
}
// longestWindow is the span past which every ring of an untouched key reads
// zero. A tenant silent for longer holds no information in memory, which is what
// makes retiring it free rather than a judgement call.
func longestWindow() time.Duration {
var d time.Duration
for _, w := range windows() {
if w.Span > d {
d = w.Span
}
}
return d
}
// bucketBytes is sizeof(velocity's ring bucket) on a 64-bit build: an int64
// index, two ints, two float64s and a uint64 day mask. Stated here rather than
// measured at run time, because a bound computed from the thing it bounds is a
// bound that moves.
const bucketBytes = 48
// keyOverhead is everything a key costs BESIDES its buckets: four ring headers,
// the slice holding them, the entry, the map slot and the composite id string,
// plus allocator rounding. Deliberately generous — a bound that under-states is
// not a bound.
const keyOverhead = 1536
// bytesPerKey is what one aggregated entity costs, derived from the windows
// actually configured so a change to windows() moves the key count and can never
// leave a stale constant behind.
func bytesPerKey() int {
n := 0
for _, w := range windows() {
n += w.Buckets
}
return n*bucketBytes + keyOverhead
}
// The knobs. Each is an environment override over a documented default, because
// the right ceiling is a property of the pod's memory limit and not of this
// source file — and because an operator who has to patch a constant to survive a
// capacity event will instead patch it to infinity.
const (
envVelBytes = "RISK_VELOCITY_BYTES" // per-tenant aggregate budget, bytes
envTenantMax = "RISK_TENANTS" // tenants this process arms at once
envIdle = "RISK_RECLAIM_IDLE" // how long a tenant must be silent before it is retired, seconds
)
// velBytes is the per-tenant aggregate budget: the CAP on what one tenant's
// rings may grow to, not an allocation. 8 MiB is ~1,386 entities at this plane's
// resolution — enough that a tenant's active set fits, small enough that
// tenantMax of them is well inside a pod.
func velBytes() int { return envInt(envVelBytes, 8<<20, 1<<20, 1<<30) }
// tenantMax is how many tenants this process ARMS at once — each with its own
// aggregates and its own forest. It is the process's memory bound and the only
// one.
//
// A pod serving more tenants than this is a SHARDING answer, not a bigger-number
// answer: the shard router already pins an org to one pod, so capacity scales by
// adding writers. Raising the knob past what the pod's memory limit supports
// trades a loud refusal for an OOM kill, which takes every tenant down.
func tenantMax() int { return envInt(envTenantMax, 48, 1, 4096) }
// idleReclaim is how long a tenant must send NOTHING before its arms are
// retired. The trigger is that tenant's own silence and nothing else — no other
// tenant's activity can ever cause it, which is the property that makes this
// reclaim and not eviction.
//
// THE FLOOR IS THE WHOLE ARGUMENT. It is the longest window, so a retired
// tenant's rings held nothing but zeros: every bucket of every ring of every key
// it owns has rotated past. Retirement is therefore information-free by
// construction rather than by hope, and a small tenant that decides once a day
// can never be quietly reset to zero counts — which is the failure a shorter
// reclaim would introduce while claiming to fix a memory bug.
func idleReclaim() time.Duration {
floor := longestWindow()
d := time.Duration(envInt(envIdle, int((floor+6*time.Hour)/time.Second), 60, int(400*24*time.Hour/time.Second))) * time.Second
if d < floor {
return floor
}
return d
}
// searchBudget is the longest one exhaustive search may run. The grid is closed
// at 243 candidates, so this is a backstop rather than the bound — but a
// detached goroutine with no deadline is a goroutine that outlives the reason it
// was started.
const searchBudget = 10 * time.Minute
// maxKeys is the per-tenant cardinality bound, computed from the budget rather
// than chosen. At least one, so a misconfigured budget degrades to a tiny store
// rather than to velocity's own 100,000-key default.
func maxKeys() int {
n := velBytes() / bytesPerKey()
if n < 1 {
return 1
}
return n
}
// envInt reads a bounded integer override. Out of range or unparseable is the
// default, not a failure: a typo in a capacity knob must not stop a payment
// plane from booting, and the value it lands on is the documented one.
func envInt(name string, def, lo, hi int) int {
v, err := strconv.Atoi(os.Getenv(name))
if err != nil || v < lo || v > hi {
return def
}
return v
}
// aggregates builds ONE TENANT's sliding aggregates.
//
// It is the only call to velocity.New in this package and it is unexported, so
// `velocity.New` with a zero Config — a 100,000-key store shared by everyone —
// is not something a later edit can reach for by accident. The bound it carries
// is per tenant BECAUSE THE STORE IS per tenant: an eviction inside it can only
// ever drop a key this same tenant put there.
func aggregates() *velocity.Store {
return velocity.New(velocity.Config{Windows: windows(), MaxKeys: maxKeys()})
}
// forest builds ONE TENANT's half-space forest over that tenant's own
// aggregates. It is the only call to anomaly.New in this package and it
// OVERRIDES two fields of whatever config it is handed, which is why the live
// plane and the search sandbox can share it.
//
// MaxOrgs is forced to 1 and that is the point. anomaly.Store holds a map of
// tenants under a GLOBAL LRU, so a shared store lets one tenant's arrival delete
// another's learned model — and an evicted model returns to warming, which
// declines to score and reads as clean. One tenant per store makes that map hold
// exactly one key, so the eviction path is unreachable rather than merely
// unlikely.
//
// Shadow is forced on: it is the deployment-wide default, and the tenant's own
// live switch in its record plane is the second gate. Two in series, so a flag
// flipped by mistake still cannot make an organisation act, and a sandbox can
// never alert for real.
func forest(cfg anomaly.Config, vel *velocity.Store) (*anomaly.Store, error) {
cfg.MaxOrgs, cfg.Shadow = 1, true
m, err := anomaly.New(cfg, vel)
if err != nil {
return nil, fmt.Errorf("risk: %w", err)
}
return m, nil
}
+398
View File
@@ -0,0 +1,398 @@
package risk
// bound_test.go pins the property bound.go exists for: a tenant may only ever
// degrade ITSELF.
//
// Each test here was written by reintroducing the defect and checking that this
// test goes red. The defect is one line — `velocity.New(velocity.Config{})` on a
// process-wide field — so the guard that survives a revert is the SOURCE test at
// the bottom: it reads this package's own files and fails the moment a second
// constructor appears, whichever file it appears in.
import (
"encoding/json"
"fmt"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
"github.com/luxfi/aml/pkg/velocity"
)
// TestASharedStoreEvictsAcrossTenants is the DEFECT, demonstrated on the library
// value this package used to hold one of.
//
// It is not a test of this package — it is the reason the rest of this file
// exists, and it is written first so the property below cannot be read as
// defending against something hypothetical. velocity.Store shards by hashing the
// WHOLE key, so two tenants land in the same shard, and the overflow eviction
// takes the least-recently-updated key IN THAT SHARD — whoever owns it.
func TestASharedStoreEvictsAcrossTenants(t *testing.T) {
shared := velocity.New(velocity.Config{Windows: windows(), MaxKeys: 64})
now := time.Now()
victim := velocity.Key{OrgID: "hanzo/victim", Kind: "ip", Value: "203.0.113.5"}
shared.Record(victim, now, 10, 10_000)
if count(shared, victim) != 1 {
t.Fatalf("the victim's own record did not land")
}
for i := range 4_000 {
shared.Record(velocity.Key{OrgID: "hanzo/flooder", Kind: "ip", Value: fmt.Sprintf("198.51.100.%d", i)}, now, 1, 10_000)
}
if got := count(shared, victim); got != 0 {
t.Skipf("the shared store still holds the victim's key (%v) — the eviction is probabilistic in the shard, "+
"but the shape that allows it is what this file rules out", got)
}
t.Log("a shared store with a global cap silently deleted another tenant's counter: this is the shape bound.go forbids")
}
// TestOneTenantsFloodCannotQuietAnother is the property.
//
// The flooder blows through its OWN cardinality bound many times over. The
// victim's ring — one key, recorded before the flood and never touched again —
// must still read exactly what the victim put in it. Under the shared store
// above it reads zero, the victim's `velocity.ip.1h.count >= 5` rule stops
// firing, and nothing anywhere says so.
func TestOneTenantsFloodCannotQuietAnother(t *testing.T) {
t.Setenv(envVelBytes, strconv.Itoa(1<<20)) // the smallest budget the knob allows
_, s := wireApp(t)
victim, flooder := Tenant("hanzo/victim"), Tenant("hanzo/flooder")
vvel, _, _ := armsOf(t, s, victim)
fvel, _, _ := armsOf(t, s, flooder)
if vvel == fvel {
t.Fatal("two tenants were handed the SAME aggregate store — one org's volume can evict another's counters")
}
now := time.Now()
key := velocity.Key{OrgID: victim.String(), Kind: "ip", Value: "203.0.113.5"}
for range 5 {
vvel.Record(key, now, 100, 10_000)
}
for i := range maxKeys() * 4 {
fvel.Record(velocity.Key{OrgID: flooder.String(), Kind: "ip", Value: fmt.Sprintf("198.51.100.%d", i)}, now, 1, 10_000)
}
if got := count(vvel, key); got != 5 {
t.Fatalf("the victim's 1h count is %d, want 5 — another tenant's traffic quieted this tenant's control", got)
}
// And the flooder degraded ITSELF, which is the other half of the property:
// its own bound held, and it is reported rather than silent.
// velocity rounds its cap UP to a whole number of keys per shard, so the true
// ceiling is a little above the budget's key count and never a multiple of it.
// What has to hold is that the flood is bounded at all: it recorded four times
// the budget and kept about one.
if got := fvel.Keys(); got >= 2*maxKeys() {
t.Fatalf("the flooder holds %d keys against a budget of %d — its own bound did not hold", got, maxKeys())
}
c, err := s.State.shelf.of(flooder)
if err != nil {
t.Fatalf("resolve flooder: %v", err)
}
if !c.strained() {
t.Fatal("a tenant at its own cardinality bound does not report itself strained, so a partial ring reads as a complete one")
}
if vc, err := s.State.shelf.of(victim); err != nil || vc.strained() {
t.Fatalf("the victim reports strained (%v) because of another tenant's traffic", err)
}
}
// TestAFullNodeRefusesANewTenantAndKeepsEveryIncumbent pins the admission rule.
//
// At the ceiling the answer is 503 and an incumbent keeps everything: its rings,
// its model and its file. Admitting by eviction would be the same defect wearing
// an admission badge — the newcomer's arrival would be the reason a tenant that
// did nothing lost its counters.
func TestAFullNodeRefusesANewTenantAndKeepsEveryIncumbent(t *testing.T) {
t.Setenv(envTenantMax, "2")
_, s := wireApp(t)
a, b := Tenant("hanzo/one"), Tenant("hanzo/two")
avel, amodel, _ := armsOf(t, s, a)
bvel, _, _ := armsOf(t, s, b)
key := velocity.Key{OrgID: a.String(), Kind: "ip", Value: "203.0.113.9"}
avel.Record(key, time.Now(), 10, 10_000)
if _, err := s.State.shelf.of(Tenant("hanzo/three")); err == nil {
t.Fatal("a third tenant was admitted past the ceiling of two")
} else if !strings.Contains(err.Error(), "tenant capacity") {
t.Fatalf("refusal says %q, want the capacity refusal", err)
}
av2, am2, _ := armsOf(t, s, a)
bv2, _, _ := armsOf(t, s, b)
if av2 != avel || am2 != amodel || bv2 != bvel {
t.Fatal("an incumbent's state was replaced to make room — that is eviction with a different name")
}
if got := count(avel, key); got != 1 {
t.Fatalf("an incumbent's count is %d after another tenant was refused, want 1", got)
}
if _, refused, _ := s.State.shelf.count(); refused != 1 {
t.Fatalf("the refusal was not counted (%d), so the probe cannot report a pod at capacity", refused)
}
}
// TestTheProbeReportsAPodAtCapacity: a refusal is an operator's problem and it
// has to reach one. A pod that quietly turns tenants away while answering 200 is
// a pod nobody investigates.
func TestTheProbeReportsAPodAtCapacity(t *testing.T) {
t.Setenv(envTenantMax, "1")
app, s := wireApp(t)
if _, err := s.State.shelf.of(Tenant("hanzo/one")); err != nil {
t.Fatalf("first tenant: %v", err)
}
code, _ := req(t, app, http.MethodGet, "/v1/risk/health", "", "", "")
if code != http.StatusOK {
t.Fatalf("probe = %d before any refusal, want 200", code)
}
if _, err := s.State.shelf.of(Tenant("hanzo/two")); err == nil {
t.Fatal("a second tenant was admitted past the ceiling of one")
}
code, body := req(t, app, http.MethodGet, "/v1/risk/health", "", "", "")
if code != http.StatusServiceUnavailable {
t.Fatalf("probe = %d %s after a refusal, want 503 carrying the report", code, body)
}
if !strings.Contains(string(body), "tenant ceiling") {
t.Fatalf("the degraded probe does not name the capacity event: %s", body)
}
}
// TestTheProbeNamesNoTenant.
//
// GET /v1/<app>/health is unauthenticated BY DESIGN across this fleet — the
// billing gate, the tracing filter and the identity middleware each exempt it by
// suffix — so everything the probe says, it says to the internet. The capacity
// report needs a strained COUNT; it must never carry the KEY of a strained
// organisation, because that publishes who is a customer and which pod holds
// them, from an anonymous GET.
//
// The tenant that needs the fact gets it on its own scoped surface.
func TestTheProbeNamesNoTenant(t *testing.T) {
t.Setenv(envVelBytes, strconv.Itoa(1<<20)) // the smallest budget the knob allows
app, s := wireApp(t)
tn := Tenant("hanzo/acme")
vel, _, _ := armsOf(t, s, tn)
now := time.Now()
for i := range maxKeys() * 2 {
vel.Record(velocity.Key{OrgID: tn.String(), Kind: "ip", Value: fmt.Sprintf("198.51.100.%d", i)}, now, 1, 1_000)
}
c, err := s.State.shelf.of(tn)
if err != nil {
t.Fatalf("resolve: %v", err)
}
if !c.strained() {
t.Fatal("the tenant is not at its own bound, so this test proves nothing")
}
code, body := req(t, app, http.MethodGet, "/v1/risk/health", "", "", "")
if code != http.StatusOK {
t.Fatalf("probe = %d %s", code, body)
}
var report map[string]any
if err := json.Unmarshal(body, &report); err != nil {
t.Fatal(err)
}
if n, ok := report["strained"].(float64); !ok || n != 1 {
t.Fatalf("the probe does not report the strained COUNT an operator needs: %s", body)
}
for _, leak := range []string{tn.String(), tn.org(), "acme"} {
if strings.Contains(string(body), leak) {
t.Errorf("the unauthenticated probe names a tenant (%q): an anonymous GET reads this "+
"organisation's key off the customer list — %s", leak, body)
}
}
// And the organisation itself IS told, on its own authenticated state.
code, body = req(t, app, http.MethodGet, "/v1/ml/state", "acme", "u_acme", "")
if code != http.StatusOK {
t.Fatalf("state = %d %s", code, body)
}
var st mlModelState
if err := json.Unmarshal(body, &st); err != nil {
t.Fatal(err)
}
if !st.Strained {
t.Error("the organisation whose own rings are partial is not told so on its own state, " +
"so a velocity rule that stopped firing reads as a clean stream")
}
// A DIFFERENT organisation is told nothing about this one.
code, body = req(t, app, http.MethodGet, "/v1/ml/state", "beta", "u_beta", "")
if code != http.StatusOK {
t.Fatalf("beta state = %d %s", code, body)
}
var other mlModelState
if err := json.Unmarshal(body, &other); err != nil {
t.Fatal(err)
}
if other.Strained {
t.Error("one organisation's strain is reported on another organisation's state")
}
}
// TestARetiredTenantHeldNothingButZeros pins the reclaim's safety argument.
//
// Reclaim is floored at the longest window, so every ring of a retired tenant
// has already rotated to zero and retiring it cannot lose a count. The tenant
// comes back with its LEARNED state, because the model is snapshotted to its own
// file before the cell is dropped.
func TestARetiredTenantHeldNothingButZeros(t *testing.T) {
if idleReclaim() < longestWindow() {
t.Fatalf("the reclaim threshold %s is under the longest window %s: retiring a tenant would delete live counts",
idleReclaim(), longestWindow())
}
_, s := wireApp(t)
tn := Tenant("hanzo/quiet")
vel, _, _ := armsOf(t, s, tn)
key := velocity.Key{OrgID: tn.String(), Kind: "ip", Value: "203.0.113.7"}
vel.Record(key, time.Now(), 10, 10_000)
// A sweep at the real threshold must NOT touch a tenant that just spoke.
s.State.shelf.sweep()
if n, _, _ := s.State.shelf.count(); n != 1 {
t.Fatalf("a tenant that just decided was retired (%d resident)", n)
}
// Silent for longer than the longest window: now it goes, and its rings held
// nothing.
c, err := s.State.shelf.of(tn)
if err != nil {
t.Fatalf("resolve: %v", err)
}
c.mu.Lock()
c.touched = time.Now().Add(-2 * idleReclaim())
c.mu.Unlock()
s.State.shelf.sweep()
if n, _, _ := s.State.shelf.count(); n != 0 {
t.Fatalf("a tenant silent for twice the reclaim threshold is still resident (%d)", n)
}
// It comes back armed, on its own file, with no trace of another tenant.
back, _, _ := armsOf(t, s, tn)
if back == vel {
t.Fatal("the retired tenant came back with the SAME store, so nothing was released")
}
}
// TestOneSweepWritesABoundedNumberOfSnapshots.
//
// The retire holds the REGISTRY lock while it writes a tenant's model to that
// tenant's encrypted file, and every other organisation's cell lookup queues
// behind that lock. Unbounded, a sweep that finds the whole shelf idle would
// serialise a snapshot per resident tenant under it — a fleet-wide stall
// arriving through the housekeeping door, which is the exact shape the rest of
// this file rules out on the request path.
func TestOneSweepWritesABoundedNumberOfSnapshots(t *testing.T) {
_, s := wireApp(t)
// More idle tenants than one pass may retire.
n := retireBatch + 3
for i := range n {
tn := Tenant(fmt.Sprintf("hanzo/idle-%d", i))
vel, model, _ := armsOf(t, s, tn)
vel.Record(velocity.Key{OrgID: tn.String(), Kind: "ip", Value: "203.0.113.1"}, time.Now(), 1, 1_000)
// Something learned, so the retire has a snapshot to write rather than a
// nil model it skips for free.
tx, ent := txOf(tn, observation{
id: "o", at: time.Now(), stage: StagePayment, kind: "transaction", subject: "tx",
amount: 1_000_000_000, currency: "USD", direction: "in", signals: map[string]string{},
})
_, _ = model.Assess(tx, ent)
c, err := s.State.shelf.of(tn)
if err != nil {
t.Fatalf("resolve: %v", err)
}
c.mu.Lock()
c.touched = time.Now().Add(-2 * idleReclaim())
c.mu.Unlock()
}
if got, _, _ := s.State.shelf.count(); got != n {
t.Fatalf("%d tenants resident, want %d", got, n)
}
s.State.shelf.sweep()
after, _, _ := s.State.shelf.count()
if n-after > retireBatch {
t.Fatalf("one sweep retired %d tenants, writing that many snapshots under the registry lock; "+
"the bound is %d per pass", n-after, retireBatch)
}
if n-after == 0 {
t.Fatal("the sweep retired nothing, so this test proves nothing about the bound")
}
}
// TestOnlyTheBoundedConstructorsBuildThePlanes is the guard that survives a
// revert.
//
// It reads this package's own source. velocity.New and anomaly.New may appear in
// bound.go and nowhere else — including tests, except the one above that
// demonstrates the defect. Reintroduce the process-wide store anywhere and this
// fails, whatever it is called and whoever calls it.
func TestOnlyTheBoundedConstructorsBuildThePlanes(t *testing.T) {
allowed := map[string]bool{"bound.go": true, "bound_test.go": true}
files, err := filepath.Glob("*.go")
if err != nil {
t.Fatalf("glob: %v", err)
}
if len(files) == 0 {
t.Fatal("no source read, so this guard proved nothing")
}
for _, f := range files {
if allowed[filepath.Base(f)] {
continue
}
body, err := os.ReadFile(f)
if err != nil {
t.Fatalf("read %s: %v", f, err)
}
for _, ctor := range []string{"velocity.New(", "anomaly.New("} {
if strings.Contains(string(body), ctor) {
t.Errorf("%s calls %s — the two in-memory planes are built ONLY by aggregates() and forest() "+
"in bound.go, which is what makes a process-wide store with a global cap unrepresentable", f, ctor)
}
}
}
}
// count reads a key's 1h count out of a store.
func count(s *velocity.Store, k velocity.Key) int {
for _, o := range s.Observe(k) {
if o.Window == "1h" {
return o.Count
}
}
return 0
}
// TestOneSearchPerTenantAndOnlyPerTenant pins the bound on the most expensive op
// this plane has.
//
// An exhaustive search is up to searchBudget of CPU across the whole grid, in a
// goroutine detached from the request. Unbounded, one key loops the route and
// takes down every product on the pod. Bounded per tenant, a caller that loops it
// spends its own slot and nobody else's — which is the same shape the measurement
// plane uses, for the same reason.
func TestOneSearchPerTenantAndOnlyPerTenant(t *testing.T) {
app, s := wireApp(t)
release, err := s.State.running.claim(Tenant("hanzo/acme"))
if err != nil {
t.Fatalf("claim: %v", err)
}
code, body := req(t, app, http.MethodPost, "/v1/ml/search", "acme", "u_acme", `{"limit":10}`)
if code != http.StatusTooManyRequests {
t.Fatalf("a second concurrent search = %d %s, want 429 — the run is unbounded", code, body)
}
// Another tenant is untouched: the bound is per tenant, not a fleet-wide slot
// that one org can hold against everybody.
code, body = req(t, app, http.MethodPost, "/v1/ml/search", "beta", "u_beta", `{"limit":10}`)
if code == http.StatusTooManyRequests {
t.Fatalf("another tenant's search = %d %s — one org's run is blocking the fleet", code, body)
}
release()
code, body = req(t, app, http.MethodPost, "/v1/ml/search", "acme", "u_acme", `{"limit":10}`)
if code == http.StatusTooManyRequests {
t.Fatalf("the slot was not released: %d %s", code, body)
}
}
+365
View File
@@ -0,0 +1,365 @@
package risk
// decide.go is the hot path: one call, one decision, an answer that says what to
// do and why.
//
// ORDER, and why it is this order:
//
// 1. RECORD the observation into the in-memory velocity rings. Everything after
// reads them, and the numbers quoted in the decision must be the numbers an
// investigator sees when they look at the subject.
// 2. CLASSIFY agency — agent, human, bot or unknown — from facts only we hold.
// 3. SCORE the model. It learns online and its evidence is capped at review.
// 4. EVALUATE the rules over the recorded aggregates plus the model's score.
// 5. SUPPRESS. A suppressed hit is RECORDED with suppressed=true, never dropped:
// silence must never read as a clean result.
// 6. COMBINE into one action, and never let statistical evidence exceed the
// ceiling.
//
// NOTHING HERE READS THE WAREHOUSE. The rings are in memory, the rules and lists
// are the tenant's own SQLite file, the model is in process. That is deliberate:
// the warehouse is one pod and this call sits inside a card processor's
// authorization window.
import (
"context"
"crypto/rand"
"encoding/hex"
"math"
"sort"
"strconv"
"strings"
"time"
"github.com/luxfi/aml/pkg/anomaly"
"github.com/luxfi/aml/pkg/types"
"github.com/luxfi/aml/pkg/velocity"
)
// Agency is what kind of actor this is. It is the differentiator: we run the
// agents, so we can tell a declared, credentialed, metered agent from an
// anonymous script, and nobody who does not run agents can compute it.
const (
// AgencyAgent is a declared agent: it named an agent reference, it
// authenticated with a principal-bearing credential, and its traffic is
// metered on this org's own ledger.
AgencyAgent = "agent"
// AgencyHuman is a browser session bound to a validated user.
AgencyHuman = "human"
// AgencyBot is undeclared automation: no agent reference, and either no
// credential or a publishable key, which by construction identifies a tenant
// and authenticates nobody.
AgencyBot = "bot"
// AgencyUnknown is credentialed but undeclared. Scored normally — an honest
// "we cannot tell" is worth more than a guess in either direction.
AgencyUnknown = "unknown"
)
// Refusals. A refusal is COUNTED and NAMED, never silent, because a control that
// declined to answer and a control that found nothing are the same bytes on the
// wire and opposite facts about the system.
const (
// RefusalWarming means the model has not learned enough of this tenant's
// behaviour for its scores to mean anything. Rules still ran.
RefusalWarming = "warming"
// RefusalUnidentified means the observation named no subject the aggregates
// can be keyed on.
RefusalUnidentified = "unidentified"
// RefusalShadow means the tenant is in shadow: everything was computed and
// recorded, and nothing acts.
RefusalShadow = "shadow"
)
// hit is one piece of evidence.
type hit struct {
// Rule is the identifier of the rule or model that produced this evidence.
Rule string `json:"rule"`
// Name is the human-readable detection name.
Name string `json:"name"`
// Action is what this evidence alone asks for.
Action string `json:"action"`
// Weight is how much this evidence contributes, in [0,1].
Weight float64 `json:"weight"`
// Severity is the reviewer-facing grading.
Severity string `json:"severity,omitempty"`
// Suppressed marks evidence a tenant suppression muted. It still contributes
// nothing to the action and is still recorded, because a muted control that
// leaves no trace is indistinguishable from one that was never running.
Suppressed bool `json:"suppressed,omitempty"`
}
// observation is everything the decide path was given plus everything it derived
// before scoring. It is the value the rules read and the value the record plane
// stores, so what fired and what was stored cannot disagree.
type observation struct {
id string
at time.Time
stage string
kind string
subject string
agency string
amount int64
currency string
direction string
signals map[string]string
agent string
session string
}
// outcome is the decision, before it is rendered onto the wire.
type outcome struct {
id string
action string
score float64
agency string
hits []hit
causes []types.Cause
shadow bool
refusal string
}
// axisOf maps a velocity axis to the observation's value on that axis. It is a
// closed map because a rule may only name an axis this returns something for,
// and admitTerm holds it to the same set.
func (o observation) axisOf(axis string) string {
switch axis {
case "account":
return o.subject
case "device":
return o.signals["device"]
case "ip":
return o.signals["ip"]
case "email":
return o.signals["email"]
case "bin":
return o.signals["bin"]
case "pair":
if cp := o.signals["counterparty"]; cp != "" {
return o.subject + "\x1f" + cp
}
}
return ""
}
// record writes the observation onto every ring it has a value for. The rings
// are keyed {OrgID, Kind, Value} with the tenant leading, so two tenants naming
// the same device or the same address never share a counter.
func record(vel *velocity.Store, t Tenant, o observation) {
usd := nanoUSD(o.amount)
for axis := range velocityAxes {
v := o.axisOf(axis)
if v == "" {
continue
}
vel.Record(velocity.Key{OrgID: t.String(), Kind: axis, Value: v}, o.at, usd, structuringThreshold)
}
}
// structuringThreshold is the reporting threshold the "just under" counters are
// measured against — the standard USD 10,000 figure. It is a constant because a
// per-tenant threshold is a per-tenant policy and this is the statutory one.
const structuringThreshold = 10_000
// nanoUSD converts the wire's integer nano-units to the float the aggregates and
// the model read. Money is carried as an integer on the wire so no rounding
// happens between the caller and the ledger; the aggregate is a statistic and a
// float is the right shape for it.
func nanoUSD(nano int64) float64 { return float64(nano) / 1e9 }
// observe builds the fact set the rules read. Every velocity number the closed
// vocabulary can name is materialised here, once, so a rule set of any size
// reads the rings a bounded number of times.
func observe(vel *velocity.Store, t Tenant, o observation, modelScore float64, warming bool) factSet {
f := factSet{
scalar: map[string]string{
"stage": o.stage,
"subject.kind": o.kind,
"subject.id": o.subject,
"agency": o.agency,
"amount.currency": o.currency,
"amount.direction": o.direction,
"actor.agent": o.agent,
"actor.session": o.session,
"model.warming": strconv.FormatBool(warming),
},
number: map[string]float64{
"amount.nano": float64(o.amount),
"model.score": modelScore,
},
}
for k, v := range o.signals {
f.scalar["signal."+strings.ToLower(k)] = v
}
for axis := range velocityAxes {
v := o.axisOf(axis)
if v == "" {
continue
}
for _, obs := range vel.Observe(velocity.Key{OrgID: t.String(), Kind: axis, Value: v}) {
p := "velocity." + axis + "." + obs.Window + "."
f.number[p+"count"] = float64(obs.Count)
f.number[p+"sum"] = obs.Sum
f.number[p+"near"] = float64(obs.Near)
f.number[p+"days"] = float64(obs.Days)
}
}
return f
}
// classify derives agency from four facts cloud already holds, and from NO
// user-agent string. A user agent is a caller-supplied claim, so sniffing it
// classifies whoever is honest and misses whoever is not.
//
// credential a publishable key resolves an ORG and no principal, so it cannot
// be an agent; a secret key or a bearer resolves a principal.
// declared does actor.agentRef resolve in THIS org's agent registry?
// session is there a live agent session for it?
// metered does this account have priced rows on the org's own ledger?
//
// declared AND credentialed => agent, and the org's agent policy applies.
// undeclared AND anonymous => bot, and the anonymous lane's bounds apply.
// Anything else is unknown and is scored normally.
func classify(credentialed, publishable, declared, humanSession bool) string {
switch {
case declared && credentialed && !publishable:
return AgencyAgent
case !declared && (!credentialed || publishable):
return AgencyBot
case humanSession && !declared:
return AgencyHuman
default:
return AgencyUnknown
}
}
// decide is the whole path. It takes the tenant's rules, lists and suppressions
// as values so it can be exercised without a store, which is what makes the
// ordering above testable rather than assertable.
func decide(
ctx context.Context,
vel *velocity.Store,
model *anomaly.Store,
t Tenant,
o observation,
rules []rule,
lists func(name, value string) bool,
suppressed func(h hit, o observation) bool,
shadow bool,
) outcome {
_ = ctx
out := outcome{id: o.id, shadow: shadow, agency: o.agency, action: ActionAllow}
if strings.TrimSpace(o.subject) == "" {
out.refusal = RefusalUnidentified
return out
}
// 1. Record first: everything after reads these rings, and the numbers in the
// decision must be the ones an investigator sees on the subject.
record(vel, t, o)
// 2. Score the model. Assess LEARNS; the tenant's own traffic is its training
// set and there is no separate training pass.
tx := types.Transaction{
ID: o.id,
OrgID: t.String(),
UserID: o.subject,
AccountID: o.subject,
Currency: o.currency,
Direction: o.direction,
IPAddress: o.signals["ip"],
DeviceFingerprint: o.signals["device"],
Counterparty: o.signals["counterparty"],
Timestamp: o.at,
USD: nanoUSD(o.amount),
}
assessment := model.Inspect(tx, types.Entity{ID: o.subject, OrgID: t.String()})
// The attribution comes from the assessment that produced the SCORE THIS
// DECISION RECORDS, not from the alert. Two reasons, and both are the whole
// point of having an explanation at all.
//
// Inspect does not learn and Assess does; they run back to back, so the
// masses move between them and the alert's causes explain a marginally
// different score than the one written down. Reading the recorded
// assessment's causes means the reasons and the number they explain are the
// same arithmetic.
//
// And Assess yields a hit only when the model ALERTS, which it never does
// while the engine-level shadow is set — so a decision below the line, or any
// decision at all in shadow, would carry no explanation whatever. A score with
// no reason is exactly what an adverse-action regime does not allow.
out.causes = assessment.Causes
var modelHit *hit
if mh, ok := model.Assess(tx, types.Entity{ID: o.subject, OrgID: t.String()}); ok {
action := mh.Rule.Action
if actionRank(action) > actionRank(modelCeiling) {
action = modelCeiling
}
modelHit = &hit{
Rule: mh.Rule.ID, Name: mh.Rule.Name, Action: action,
Weight: mh.Rule.Weight, Severity: mh.Rule.Severity,
}
}
if !assessment.Scored && assessment.Reason != "" {
out.refusal = assessment.Reason
}
// 3. Rules, over the recorded aggregates plus the model's score.
f := observe(vel, t, o, assessment.Score, !assessment.Scored)
f.lists = lists
hits := evaluate(rules, o.stage, f)
if modelHit != nil {
hits = append(hits, *modelHit)
}
// 4. Suppression. A suppressed hit is kept, marked and given zero weight —
// it appears in the record and contributes nothing to the action.
scoring := make([]hit, 0, len(hits))
for i := range hits {
if suppressed != nil && suppressed(hits[i], o) {
hits[i].Suppressed = true
continue
}
scoring = append(scoring, hits[i])
}
// 5. Combine. Sorting again puts the model's hit in its place among the rest.
sort.SliceStable(hits, func(i, j int) bool {
if a, b := actionRank(hits[i].Action), actionRank(hits[j].Action); a != b {
return a > b
}
return hits[i].Weight > hits[j].Weight
})
score, action := combine(scoring)
out.hits, out.score = hits, round4(score)
// 6. Shadow. Everything above ran, everything is recorded, nothing acts.
if shadow {
out.action = ActionAllow
if out.refusal == "" {
out.refusal = RefusalShadow
}
return out
}
out.action = action
return out
}
// round4 trims a score to four places. A score is a judgement, not a
// measurement, and rendering seventeen digits of float noise invites a client to
// compare two scores that differ in the fifteenth.
func round4(f float64) float64 {
if math.IsNaN(f) || math.IsInf(f, 0) {
return 0
}
return math.Round(f*1e4) / 1e4
}
// newID mints a decision identifier. Random rather than sequential: a sequential
// id over a shared surface is a volume oracle — a tenant can count another
// tenant's decisions by watching its own ids skip.
func newID(prefix string) string {
var b [12]byte
_, _ = rand.Read(b[:])
return prefix + "_" + hex.EncodeToString(b[:])
}
+421
View File
@@ -0,0 +1,421 @@
package risk
// decide_test.go pins the decision path's ORDER and its two hard limits: the
// model may never act alone, and shadow may never act at all.
import (
"context"
"encoding/json"
"net/http"
"strings"
"sync"
"testing"
"time"
"github.com/luxfi/aml/pkg/anomaly"
)
// TestRecordHappensBeforeScoring pins step 1. Everything after reads the rings,
// so the numbers quoted in a decision must be the ones an investigator sees when
// they look at the subject — including THIS observation.
func TestRecordHappensBeforeScoring(t *testing.T) {
vel := aggregates()
model, err := forest(anomaly.Config{}, vel)
if err != nil {
t.Fatalf("forest: %v", err)
}
tn := Tenant("hanzo/acme")
// A rule that can only hold if THIS observation is already in the ring.
rules := []rule{{
ID: "r1", Name: "counted", Stage: StagePayment, Action: ActionReview,
Weight: 0.5, Enabled: true,
All: []term{{Field: "velocity.ip.1h.count", Op: OpGte, Number: 1}},
}}
o := observation{
id: "d1", at: time.Now(), stage: StagePayment, kind: "transaction", subject: "tx1",
amount: 1_000_000_000, currency: "USD", direction: "in",
signals: map[string]string{"ip": "203.0.113.1"},
}
out := decide(context.Background(), vel, model, tn, o, rules, nil, nil, false)
if len(out.hits) != 1 {
t.Fatalf("hits = %v; the rule did not see this observation in the ring, so the record ran after the read", out.hits)
}
}
// TestModelEvidenceCannotExceedTheCeiling pins the structural backstop. A
// statistical judgement may summon a person; it may not decline a payment,
// because an unexplainable refusal is not a decision anybody can defend to a
// customer or a chargeback network.
func TestModelEvidenceCannotExceedTheCeiling(t *testing.T) {
if actionRank(modelCeiling) != actionRank(ActionReview) {
t.Fatalf("the model ceiling is %q, not review — statistical evidence can now act alone", modelCeiling)
}
// And the combine step's ordering must never promote a capped hit.
hits := []hit{{Rule: "model", Action: modelCeiling, Weight: 1.0}}
_, action := combine(hits)
if actionRank(action) > actionRank(ActionReview) {
t.Fatalf("a weight-1.0 model hit produced %q — the cap is not holding", action)
}
}
// TestShadowActsOnNothing pins the default. In shadow every rule runs, the model
// scores and learns, every decision is recorded, and the action is always allow.
func TestShadowActsOnNothing(t *testing.T) {
vel := aggregates()
model, _ := forest(anomaly.Config{}, vel)
rules := []rule{{
ID: "block-all", Name: "would block", Stage: StagePayment, Action: ActionBlock,
Weight: 1, Enabled: true,
All: []term{{Field: "subject.kind", Op: OpEq, Value: "transaction"}},
}}
o := observation{
id: "d1", at: time.Now(), stage: StagePayment, kind: "transaction", subject: "tx1",
signals: map[string]string{},
}
out := decide(context.Background(), vel, model, Tenant("hanzo/acme"), o, rules, nil, nil, true)
if out.action != ActionAllow {
t.Fatalf("shadow produced action %q — a shadow tenant acted", out.action)
}
// The answer must SAY it was shadow, or silence reads as a clean result. The
// refusal keeps the MODEL's more specific reason when it has one (a warming
// model is a different fact from a shadow tenant, and hiding the first behind
// the second is exactly the conflation this field exists to prevent) — so
// `shadow` is the flag that must always hold, and `refusal` must always be
// populated with something.
if !out.shadow {
t.Fatal("a shadow decision does not say it was shadow")
}
if out.refusal == "" {
t.Fatal("a shadow decision names no refusal at all, so it is indistinguishable from a clean live one")
}
if out.refusal != RefusalWarming && out.refusal != RefusalShadow {
t.Fatalf("refusal = %q, want the model's own reason or shadow", out.refusal)
}
if len(out.hits) != 1 {
t.Fatalf("shadow computed %d hits, want 1 — shadow must compute everything and act on nothing", len(out.hits))
}
// Live, the same input blocks. Without this the test above would pass on a
// rule that simply never fires.
out = decide(context.Background(), vel, model, Tenant("hanzo/acme"), o, rules, nil, nil, false)
if out.action != ActionBlock {
t.Fatalf("live produced %q, want block — the shadow assertion proves nothing if the rule cannot fire", out.action)
}
}
// TestSuppressedEvidenceIsRecordedNotDropped pins the doctrine: a muted control
// that leaves no trace is indistinguishable from one that was never running.
func TestSuppressedEvidenceIsRecordedNotDropped(t *testing.T) {
vel := aggregates()
model, _ := forest(anomaly.Config{}, vel)
rules := []rule{{
ID: "r1", Name: "noisy", Stage: StageSignup, Action: ActionBlock, Weight: 1, Enabled: true,
All: []term{{Field: "subject.kind", Op: OpEq, Value: "account"}},
}}
o := observation{id: "d1", at: time.Now(), stage: StageSignup, kind: "account", subject: "a1", signals: map[string]string{}}
out := decide(context.Background(), vel, model, Tenant("hanzo/acme"), o, rules, nil,
func(h hit, _ observation) bool { return h.Rule == "r1" }, false)
if len(out.hits) != 1 {
t.Fatalf("the suppressed hit was DROPPED (%d hits) — the record no longer says the rule fired", len(out.hits))
}
if !out.hits[0].Suppressed {
t.Fatal("the hit is recorded but not marked suppressed")
}
if out.action != ActionAllow {
t.Fatalf("a suppressed hit still drove the action to %q", out.action)
}
if out.score != 0 {
t.Fatalf("a suppressed hit contributed %v to the score", out.score)
}
}
// TestAgencyIsDerivedNotDeclared pins the differentiator. The classification
// turns on facts we hold — the credential class and this org's own agent
// registry — and never on a user-agent string, which is a claim.
func TestAgencyIsDerivedNotDeclared(t *testing.T) {
for _, tc := range []struct {
name string
credentialed, publishable, declared, humanSes bool
want string
}{
{"declared agent on a secret key", true, false, true, false, AgencyAgent},
{"declared agent on a publishable key", true, true, true, false, AgencyUnknown},
{"anonymous script", false, false, false, false, AgencyBot},
{"publishable-key script", true, true, false, false, AgencyBot},
{"browser session", true, false, false, true, AgencyHuman},
{"credentialed but undeclared", true, false, false, false, AgencyUnknown},
} {
if got := classify(tc.credentialed, tc.publishable, tc.declared, tc.humanSes); got != tc.want {
t.Errorf("%s: agency = %q, want %q", tc.name, got, tc.want)
}
}
}
// TestRuleAdmissionRefusesWhatWouldReadAsWorking pins the admission gate. Each
// of these produces a rule that looks live and detects nothing, or one that
// detects everything.
func TestRuleAdmissionRefusesWhatWouldReadAsWorking(t *testing.T) {
base := rule{Name: "r", Action: ActionBlock, Weight: 0.5, Enabled: true,
All: []term{{Field: "signal.ip", Op: OpEq, Value: "x"}}}
ok := base
if err := admit(ok); err != nil {
t.Fatalf("a well-formed rule was refused: %v", err)
}
for _, tc := range []struct {
name string
mut func(*rule)
}{
{"no terms holds on everything", func(r *rule) { r.All = nil }},
{"no name cannot be read back in an alert", func(r *rule) { r.Name = "" }},
{"an unknown action", func(r *rule) { r.Action = "quarantine" }},
{"a weight outside [0,1]", func(r *rule) { r.Weight = 2 }},
{"an unknown stage", func(r *rule) { r.Stage = "checkout" }},
{"a field the vocabulary does not carry", func(r *rule) { r.All = []term{{Field: "tx.amount", Op: OpEq}} }},
{"an unknown operator", func(r *rule) { r.All = []term{{Field: "signal.ip", Op: "matches"}} }},
{"an unknown velocity axis", func(r *rule) { r.All = []term{{Field: "velocity.wallet.1h.count", Op: OpGte}} }},
{"an unknown velocity window", func(r *rule) { r.All = []term{{Field: "velocity.ip.90d.count", Op: OpGte}} }},
{"an empty in-set holds on nothing", func(r *rule) { r.All = []term{{Field: "agency", Op: OpIn}} }},
{"an inlist naming no list", func(r *rule) { r.All = []term{{Field: "signal.ip", Op: OpInList}} }},
} {
r := base
r.All = append([]term(nil), base.All...)
tc.mut(&r)
if err := admit(r); err == nil {
t.Errorf("admitted a rule with %s", tc.name)
}
}
}
// TestWeightOfEvidenceCompounds pins the score's shape. Two independent weak
// signals must compound and no single one may saturate — a summed score clamps
// at 1 and then loses every further signal.
func TestWeightOfEvidenceCompounds(t *testing.T) {
one, _ := combine([]hit{{Weight: 0.4, Action: ActionAllow}})
two, _ := combine([]hit{{Weight: 0.4, Action: ActionAllow}, {Weight: 0.4, Action: ActionAllow}})
three, _ := combine([]hit{{Weight: 0.4}, {Weight: 0.4}, {Weight: 0.4}})
four, _ := combine([]hit{{Weight: 0.4}, {Weight: 0.4}, {Weight: 0.4}, {Weight: 0.4}})
if !(one < two && two < three && three < four) {
t.Fatalf("scores %v %v %v %v do not compound — a fourth signal changed nothing", one, two, three, four)
}
if four >= 1 {
t.Fatalf("four 0.4 signals saturated at %v; nothing after this could raise the score", four)
}
}
// TestIdempotentDecideDoesNotScoreTwice pins the wire promise. A retried decide
// must return the SAME decision and must not move the counters again — a
// double-counted payment is a velocity rule firing on one transaction.
func TestIdempotentDecideDoesNotScoreTwice(t *testing.T) {
app, _ := wireApp(t)
body := `{"stage":"payment","subject":{"kind":"transaction","id":"tx-9"},
"amount":{"nano":1000000000,"currency":"USD","direction":"in"},
"signals":{"ip":"198.51.100.7"},"idem":"k-1"}`
code, first := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme", body)
if code != http.StatusOK {
t.Fatalf("first decide = %d %s", code, first)
}
code, second := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme", body)
if code != http.StatusOK {
t.Fatalf("second decide = %d %s", code, second)
}
var a, b riskDecision
_ = json.Unmarshal(first, &a)
_ = json.Unmarshal(second, &b)
if a.ID != b.ID {
t.Fatalf("the same idempotency key produced two decisions (%s, %s)", a.ID, b.ID)
}
_, list := req(t, app, http.MethodGet, "/v1/risk/decisions", "acme", "u_acme", "")
var page riskDecisionPage
_ = json.Unmarshal(list, &page)
if len(page.Items) != 1 {
t.Fatalf("%d decisions recorded for one idempotency key", len(page.Items))
}
}
// TestConcurrentRetriesUnderOneKeyProduceOneDecision pins the race the
// two-statement version had: insert the row, then claim the key, and two
// simultaneous retries both find no row, both insert, and the loser's claim
// violates the index — so a caller that retried CORRECTLY gets a 500 and the
// tenant's counters moved twice.
//
// Claiming the key inside the insert makes the loser lose at the index, before a
// second decision exists, and read the winner's answer back.
func TestConcurrentRetriesUnderOneKeyProduceOneDecision(t *testing.T) {
app, _ := wireApp(t)
body := `{"stage":"payment","subject":{"kind":"transaction","id":"tx-race"},
"amount":{"nano":2500000000,"currency":"USD","direction":"in"},
"signals":{"ip":"198.51.100.99"},"idem":"race-1"}`
const n = 8
type result struct {
code int
id string
}
out := make(chan result, n)
var start sync.WaitGroup
start.Add(1)
for i := 0; i < n; i++ {
go func() {
start.Wait()
code, b := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme", body)
var d riskDecision
_ = json.Unmarshal(b, &d)
out <- result{code, d.ID}
}()
}
start.Done()
ids := map[string]int{}
for i := 0; i < n; i++ {
r := <-out
if r.code != http.StatusOK {
t.Errorf("a concurrent retry answered %d — a correct retry must never be an error", r.code)
continue
}
ids[r.id]++
}
if len(ids) != 1 {
t.Fatalf("%d concurrent requests under one key produced %d distinct decisions: %v", n, len(ids), ids)
}
// And exactly one row was written, so the counters moved once.
_, list := req(t, app, http.MethodGet, "/v1/risk/decisions", "acme", "u_acme", "")
var page riskDecisionPage
_ = json.Unmarshal(list, &page)
if len(page.Items) != 1 {
t.Fatalf("%d decisions recorded under one idempotency key", len(page.Items))
}
}
// TestDecideRefusesAnUnknownStageOrKind pins the two vocabularies at the door.
func TestDecideRefusesAnUnknownStageOrKind(t *testing.T) {
app, _ := wireApp(t)
for _, body := range []string{
`{"stage":"checkout","subject":{"kind":"account","id":"a"}}`,
`{"stage":"signup","subject":{"kind":"wallet","id":"a"}}`,
`{"stage":"signup","subject":{"kind":"account","id":""}}`,
} {
code, got := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme", body)
if code != http.StatusBadRequest {
t.Errorf("%s = %d %s, want 400", body, code, got)
}
}
}
// TestSimulateRefusesAnEmptyHistory pins the sandbox's whole reason for
// existing: "no alerts" is what a quiet rule and an unrun rule both look like.
func TestSimulateRefusesAnEmptyHistory(t *testing.T) {
app, _ := wireApp(t)
code, body := req(t, app, http.MethodPost, "/v1/risk/simulate", "acme", "u_acme",
`{"candidate":{"name":"c","action":"review","weight":0.5,
"all":[{"field":"signal.ip","op":"eq","value":"1.1.1.1"}]}}`)
if code != http.StatusOK {
t.Fatalf("simulate = %d %s", code, body)
}
var rep riskSimulateReport
_ = json.Unmarshal(body, &rep)
if rep.Refusal == "" {
t.Fatal("an empty history reported zero alerts instead of refusing")
}
if rep.Events != 0 {
t.Fatalf("events = %d on an empty history", rep.Events)
}
}
// TestSimulateMeasuresAgainstRealHistory drives a decision in, then replays a
// candidate over it, so the report is measured rather than asserted.
func TestSimulateMeasuresAgainstRealHistory(t *testing.T) {
app, _ := wireApp(t)
mustOK(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
`{"stage":"signup","subject":{"kind":"account","id":"a-1"},"signals":{"ip":"192.0.2.10"}}`,
http.StatusOK)
code, body := req(t, app, http.MethodPost, "/v1/risk/simulate", "acme", "u_acme",
`{"candidate":{"name":"c","stage":"signup","action":"review","weight":0.5,
"all":[{"field":"signal.ip","op":"eq","value":"192.0.2.10"}]}}`)
if code != http.StatusOK {
t.Fatalf("simulate = %d %s", code, body)
}
var rep riskSimulateReport
_ = json.Unmarshal(body, &rep)
if rep.Events != 1 || rep.Alerts != 1 {
t.Fatalf("events=%d alerts=%d, want 1/1 — the replay did not read the recorded facts", rep.Events, rep.Alerts)
}
if len(rep.Added) != 1 {
t.Fatalf("added=%v, want the one decision the candidate newly catches", rep.Added)
}
}
// TestModeDefaultsToShadowAndSaysSo pins the default at the wire.
func TestModeDefaultsToShadowAndSaysSo(t *testing.T) {
app, _ := wireApp(t)
code, body := req(t, app, http.MethodGet, "/v1/risk/mode", "acme", "u_acme", "")
if code != http.StatusOK {
t.Fatalf("mode = %d %s", code, body)
}
var m riskModeView
_ = json.Unmarshal(body, &m)
if m.Mode != "shadow" {
t.Fatalf("a fresh tenant is %q, not shadow — it would act on its very first request", m.Mode)
}
// A decision in shadow must SAY it was shadow.
_, body = req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
`{"stage":"signup","subject":{"kind":"account","id":"a-1"}}`)
var d riskDecision
_ = json.Unmarshal(body, &d)
if !d.Shadow {
t.Fatal("a shadow tenant's decision does not say so")
}
}
// TestHealthCarriesItsReport pins the reason the probe is untyped: the body IS
// the answer.
func TestHealthCarriesItsReport(t *testing.T) {
app, _ := wireApp(t)
code, body := req(t, app, http.MethodGet, "/v1/risk/health", "", "", "")
if code != http.StatusOK {
t.Fatalf("health = %d %s", code, body)
}
for _, k := range []string{"status", "model", "warehouse", "billing"} {
if !strings.Contains(string(body), `"`+k+`"`) {
t.Errorf("the probe body does not carry %q — a typed op would have dropped exactly this", k)
}
}
}
// TestTrainIsOnlineAndPerTenant drives training through the wire and proves the
// counters moved for the caller's tenant and nobody else's.
func TestTrainIsOnlineAndPerTenant(t *testing.T) {
app, s := wireApp(t)
obs := make([]string, 0, 20)
for i := 0; i < 20; i++ {
obs = append(obs, `{"subject":{"kind":"account","id":"a-1"},"amount":{"nano":1000000000,"currency":"USD","direction":"in"}}`)
}
body := `{"observations":[` + strings.Join(obs, ",") + `]}`
code, got := req(t, app, http.MethodPost, "/v1/ml/train", "acme", "u_acme", body)
if code != http.StatusOK {
t.Fatalf("train = %d %s", code, got)
}
var out mlTrainOut
_ = json.Unmarshal(got, &out)
if out.Learned != 20 {
t.Fatalf("learned %d of 20", out.Learned)
}
_, acme, _ := armsOf(t, s, Tenant("hanzo/acme"))
_, beta, _ := armsOf(t, s, Tenant("hanzo/beta"))
if acme.State("hanzo/acme").Learned == 0 {
t.Fatal("the caller's model learned nothing")
}
if beta.State("hanzo/beta").Learned != 0 {
t.Fatal("another tenant's model learned from this tenant's data")
}
// And the tenant key really is qualified, not bare — asked of the caller's
// OWN forest, which is the only one that could have been indexed wrongly.
if acme.State("acme").Learned != 0 {
t.Fatal("the model is indexed on the BARE org — two brands' same-named orgs would share it")
}
}
+5
View File
@@ -0,0 +1,5 @@
package risk
// devmaster keys this test binary: cek opens nothing without a master, and a test
// process has no KMS to resolve one from.
import _ "github.com/hanzoai/cloud/internal/devmaster"
+494
View File
@@ -0,0 +1,494 @@
package risk
// durable_test.go pins the property store.go's header claims: a record is not
// acknowledged until it is durable.
//
// The test drives the REAL durable plane — cloud.OrgStore over org.Durability
// over a conditional object store — with the object store standing in as an
// in-memory CAS register. Nothing here is a mock of our own code: the ship, the
// fence and the lease are the production ones, and the only substitution is the
// bucket.
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"testing"
"github.com/hanzoai/cloud"
"github.com/hanzoai/cloud/internal/org"
sqlitedrv "github.com/hanzoai/sqlite"
"github.com/hanzoai/vfs/replica"
"github.com/luxfi/aml/pkg/anomaly"
luxlog "github.com/luxfi/log"
"github.com/zap-proto/zip"
)
// TestADecisionIsShippedBeforeItIsAcknowledged.
//
// A decision is the record an adverse action is defended with. cloud deploys
// Recreate at one replica, so a decision that lives only on the pod's volume is
// one a rollout can lose AFTER the caller was told it was taken. The ship
// happens before the 200, or the 200 is a lie.
func TestADecisionIsShippedBeforeItIsAcknowledged(t *testing.T) {
app, _, store := wireDurable(t)
before := store.puts()
code, body := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
`{"stage":"payment","subject":{"kind":"transaction","id":"tx-1"},
"amount":{"nano":12000000000,"currency":"USD","direction":"in"}}`)
if code != http.StatusOK {
t.Fatalf("decide = %d %s", code, body)
}
if got := store.puts() - before; got == 0 {
t.Fatal("the decision was acknowledged without a single write to the durable object: " +
"a rollout loses it and the caller was told it was taken")
}
}
// TestAWriteThatCannotBeShippedIsNotAcknowledged.
//
// The object store goes away mid-flight. EVERY record write must then FAIL —
// loudly, with a status a caller retries on — rather than answer 200 over a row
// that exists on one pod's disk and nowhere else. The retry is exact because the
// caller's idempotency key is claimed inside the same transaction.
//
// EVERY MUTATING ROUTE IS IN THE TABLE, and that is what makes this the guard
// rather than three examples. Durability applied one call site at a time is a
// rule somebody has to remember, and the writes easiest to forget are the
// RETIREMENTS — a rule deleted, a mute lifted, a hold released — because an
// unshipped delete brings the old state back at the next rollout, firing or
// blocking, after a 204 said it was gone. A mutating op absent from the table
// and from the records-nothing list fails the companion test below, which reads
// the published subset.
//
// ONE app and ONE store for all of it. Every subject a retirement retires is
// created first, while the store is up; then the store fails ONCE and every
// write runs against it. A fresh durable harness per case would re-encrypt and
// fsync the whole file per setup write, which is minutes of the suite's wall
// clock to prove nothing the shared setup does not.
func TestAWriteThatCannotBeShippedIsNotAcknowledged(t *testing.T) {
app, _, store := wireDurable(t)
ids := seedForRetirement(t, app)
store.fail(errors.New("the object store is unreachable"))
for _, w := range durableWrites(ids) {
code, body := req(t, app, w.method, w.path, "acme", "u_acme", w.body)
if code < 500 {
t.Errorf("%s (%s %s) answered %d %s while the durable object store was down — "+
"the record exists on this pod only and the caller was told otherwise",
w.what, w.method, w.path, code, body)
}
}
}
// subjects are the server-minted identifiers a retirement needs. Every id here
// is READ BACK from the create, never guessed: this plane mints its own, so a
// literal in a test would be testing a 404 path.
type subjects struct{ rule, spare, mute, control, decision string }
func seedForRetirement(t *testing.T, app *zip.App) subjects {
t.Helper()
var s subjects
s.rule = mintedID(t, app, http.MethodPost, "/v1/risk/rules", ruleBody("durable"))
s.spare = mintedID(t, app, http.MethodPost, "/v1/risk/rules", ruleBody("spare"))
s.mute = mintedID(t, app, http.MethodPost, "/v1/risk/suppressions",
`{"rule":"`+s.rule+`","kind":"transaction","reason":"noisy under test"}`)
s.control = mintedID(t, app, http.MethodPost, "/v1/risk/controls",
`{"subject":{"kind":"account","id":"acct-1"},"control":"payout-hold","reason":"under review"}`)
s.decision = mintedID(t, app, http.MethodPost, "/v1/risk/decide",
`{"stage":"payment","subject":{"kind":"transaction","id":"tx-seed"}}`)
for _, c := range []call{
{http.MethodPost, "/v1/risk/lists", `{"name":"blocked-ips","kind":"deny"}`},
{http.MethodPost, "/v1/risk/lists/blocked-ips/entries", `{"values":["203.0.113.9"]}`},
{http.MethodPost, "/v1/ml/train", trainBody},
} {
if code, body := req(t, app, c.method, c.path, "acme", "u_acme", c.body); code >= 300 {
t.Fatalf("setup %s %s = %d %s", c.method, c.path, code, body)
}
}
return s
}
// mintedID performs one create and returns the id the SERVER chose.
func mintedID(t *testing.T, app *zip.App, method, path, body string) string {
t.Helper()
code, out := req(t, app, method, path, "acme", "u_acme", body)
if code >= 300 {
t.Fatalf("setup %s %s = %d %s", method, path, code, out)
}
var v struct {
ID string `json:"id"`
}
if err := json.Unmarshal(out, &v); err != nil || v.ID == "" {
t.Fatalf("setup %s %s did not answer with an id: %s", method, path, out)
}
return v.ID
}
// call is one request.
type call struct{ method, path, body string }
// durableWrite is a mutating route and the name it fails under.
type durableWrite struct{ what, method, path, body string }
func ruleBody(name string) string {
return `{"rule":{"name":"` + name + `","stage":"payment","action":"review","weight":0.5,"enabled":true,
"all":[{"field":"subject.kind","op":"eq","value":"transaction"}]}}`
}
// durableWrites is the table. The RETIREMENTS come last and each takes a
// DIFFERENT subject from the one an earlier case needs, because a write's local
// half lands even when its ship does not — that is the defect under test.
func durableWrites(s subjects) []durableWrite {
return []durableWrite{
{"a decision", http.MethodPost, "/v1/risk/decide",
`{"stage":"payment","subject":{"kind":"transaction","id":"tx-2"}}`},
{"a label on a decision", http.MethodPost, "/v1/risk/decisions/" + s.decision + "/label",
`{"verdict":"fraud"}`},
{"a policy version", http.MethodPut, "/v1/risk/policy",
`{"stage":"payment","floor":"allow","reason":"the dispute rate doubled","bands":[{"at":0.9,"action":"block"}]}`},
{"a rule", http.MethodPost, "/v1/risk/rules", ruleBody("another")},
{"a rule change", http.MethodPatch, "/v1/risk/rules/" + s.rule, ruleBody("changed")},
{"a list", http.MethodPost, "/v1/risk/lists", `{"name":"watched-ips","kind":"allow"}`},
{"a deny-list entry", http.MethodPost, "/v1/risk/lists/blocked-ips/entries", `{"values":["198.51.100.4"]}`},
{"a deny-list REMOVAL", http.MethodDelete, "/v1/risk/lists/blocked-ips/entries/203.0.113.9", ""},
{"a mute", http.MethodPost, "/v1/risk/suppressions",
`{"rule":"` + s.spare + `","kind":"account","reason":"noisy too"}`},
{"a mute LIFTED", http.MethodDelete, "/v1/risk/suppressions/" + s.mute, ""},
{"a control", http.MethodPost, "/v1/risk/controls",
`{"subject":{"kind":"account","id":"acct-2"},"control":"block","reason":"card testing"}`},
{"a control RELEASED", http.MethodDelete, "/v1/risk/controls/" + s.control, ""},
{"the live/shadow switch", http.MethodPut, "/v1/risk/mode", `{"mode":"shadow"}`},
{"the appetite", http.MethodPut, "/v1/ml/state/appetite", `{"review":0.01,"sample":0.001}`},
{"a model snapshot", http.MethodPost, "/v1/ml/snapshot", `{}`},
{"a search run", http.MethodPost, "/v1/ml/search", `{"limit":10}`},
{"a rule RETIREMENT", http.MethodDelete, "/v1/risk/rules/" + s.spare, ""},
}
}
// trainBody is enough observations that the model has something to snapshot.
const trainBody = `{"observations":[
{"stage":"payment","subject":{"kind":"transaction","id":"t1"},"amount":{"nano":1000000000,"currency":"USD","direction":"in"}},
{"stage":"payment","subject":{"kind":"transaction","id":"t2"},"amount":{"nano":2000000000,"currency":"USD","direction":"in"}},
{"stage":"payment","subject":{"kind":"transaction","id":"t3"},"amount":{"nano":3000000000,"currency":"USD","direction":"in"}}]}`
// TestEveryMutatingRouteIsCoveredByTheDurabilityTable is the guard that survives
// a NEW op rather than a revert.
//
// The table above is only a guard while it is complete, and the way it stops
// being complete is somebody adding a route. This reads the app's OWN published
// subset — the artifact the SDKs are generated from — and insists every mutating
// path is either exercised by the table or named below as recording nothing.
func TestEveryMutatingRouteIsCoveredByTheDurabilityTable(t *testing.T) {
// Routes that record NOTHING: they answer from memory or from a read, so
// there is no row whose durability could be in question. Each is named, so
// admitting one is a decision somebody wrote down.
//
// /v1/ml/calibrate and /v1/ml/replay DO record, and both ship — they are
// exercised by their own tests in skew_test.go, which need 200 judged rows
// apiece and would cost this table two more fits to say the same thing.
silent := map[string]bool{
"POST /v1/ml/score": true, // scores one observation and learns nothing
"POST /v1/ml/train": true, // moves in-memory counters; /v1/ml/snapshot is the record
"POST /v1/ml/restore": true, // reads a snapshot back into memory
"POST /v1/ml/evaluate": true, // measures over rows already written
"POST /v1/risk/simulate": true, // tries a rule against history and writes nothing
"POST /v1/ml/calibrate": true, // records, and ships — covered in skew_test.go
"POST /v1/ml/replay": true, // records, and ships — covered in skew_test.go
}
covered := map[string]bool{}
for _, w := range durableWrites(subjects{rule: "R", spare: "S", mute: "M", control: "C", decision: "D"}) {
covered[w.method+" "+routePattern(w.path)] = true
}
var doc struct {
Paths map[string]map[string]any `json:"paths"`
}
if err := json.Unmarshal(readSubset(t), &doc); err != nil {
t.Fatalf("plugin/risk/openapi.json: %v", err)
}
if len(doc.Paths) == 0 {
t.Fatal("no paths read, so this guard proved nothing")
}
for path, methods := range doc.Paths {
for m := range methods {
method := strings.ToUpper(m)
switch method {
case http.MethodPost, http.MethodPut, http.MethodPatch, http.MethodDelete:
default:
continue
}
key := method + " " + path
if silent[key] || covered[key] {
continue
}
t.Errorf("%s is a mutating route the durability table does not exercise: either it "+
"writes a record and must ship before it acknowledges, or it records nothing and "+
"must say so in the list above", key)
}
}
}
// routePattern turns a concrete request path back into the pattern the published
// subset names it by, so the table can be written with real, server-minted ids.
func routePattern(path string) string {
switch {
case strings.HasPrefix(path, "/v1/risk/rules/"):
return "/v1/risk/rules/{id}"
case strings.HasPrefix(path, "/v1/risk/suppressions/"):
return "/v1/risk/suppressions/{id}"
case strings.HasPrefix(path, "/v1/risk/controls/"):
return "/v1/risk/controls/{id}"
case strings.HasPrefix(path, "/v1/risk/lists/") && strings.Contains(path, "/entries/"):
return "/v1/risk/lists/{name}/entries/{value}"
case strings.HasPrefix(path, "/v1/risk/lists/") && strings.HasSuffix(path, "/entries"):
return "/v1/risk/lists/{name}/entries"
case strings.HasPrefix(path, "/v1/risk/decisions/") && strings.HasSuffix(path, "/label"):
return "/v1/risk/decisions/{id}/label"
}
return path
}
// TestADeposedWriterDoesNotAcknowledgeItsOwnWrite is the OTHER half of the ship,
// and the half a warning would hide.
//
// A ship has three outcomes, not two. It can fail (an error — covered above); it
// can be acknowledged; or it can come back UNACKNOWLEDGED WITH NO ERROR, which
// is what the fence answers when a successor pod has advanced the round: this
// process is no longer this org's elected writer, so the row it just wrote is on
// a deposed pod's disk and the org's record is elsewhere. Nothing failed. If the
// plane treats that as success — or logs it and answers 200 — the caller is told
// a decision is on file that the surviving writer will never produce.
func TestADeposedWriterDoesNotAcknowledgeItsOwnWrite(t *testing.T) {
app, _, store := wireDurable(t)
// One good decision first, so the file, the lease and the object all exist:
// what follows is a writer that was deposed, not one that never held the org.
code, body := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
`{"stage":"payment","subject":{"kind":"transaction","id":"tx-1"}}`)
if code != http.StatusOK {
t.Fatalf("first decide = %d %s", code, body)
}
store.depose()
code, body = req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
`{"stage":"payment","subject":{"kind":"transaction","id":"tx-2"}}`)
if code < 500 {
t.Fatalf("a deposed writer answered %d %s — the decision is on this pod's disk and "+
"the org's elected writer will never produce it", code, body)
}
if !strings.Contains(string(body), "elected writer") {
t.Errorf("the refusal does not say why, so an operator reads it as a generic 5xx: %s", body)
}
}
// TestTheRecordPlaneIsOpenedThroughTheDurableStore is the guard that survives a
// revert. The bare encrypted open is the open and nothing more, so a package
// that calls it directly has no hydrate, no fence and no ship whatever its
// comments say. This package reaches a file through the durable org store and no
// other way.
func TestTheRecordPlaneIsOpenedThroughTheDurableStore(t *testing.T) {
// Spelled in two halves so this file does not match its own guard.
bareOpen := "cloud.Org" + "DB("
files, err := filepath.Glob("*.go")
if err != nil {
t.Fatalf("glob: %v", err)
}
if len(files) == 0 {
t.Fatal("no source read, so this guard proved nothing")
}
found := false
for _, f := range files {
body, err := os.ReadFile(f)
if err != nil {
t.Fatalf("read %s: %v", f, err)
}
if strings.Contains(string(body), "cloud.NewOrgStore") {
found = true
}
if strings.Contains(string(body), bareOpen) {
t.Errorf("%s opens a tenant file with cloud.OrgDB — that open is local-only, so every record "+
"in it is as durable as one pod's volume", f)
}
}
if !found {
t.Error("nothing in this package opens its files through cloud.NewOrgStore, so nothing is durable")
}
}
// TestCommitRefusesToShipAWriteThatFailed pins the order inside commit: the ship
// is the acknowledgement of a write that HAPPENED, so a failed write must not
// produce one.
func TestCommitRefusesToShipAWriteThatFailed(t *testing.T) {
_, s := wireApp(t)
boom := errors.New("the write failed")
err := s.State.shelf.commit(Tenant("hanzo/acme"), func(*sql.DB) error { return boom })
if !errors.Is(err, boom) {
t.Fatalf("commit returned %v, want the write's own error", err)
}
}
// ── the durable harness ─────────────────────────────────────────────────────
// wireDurable mounts the surface over a REAL durable org store whose object
// store is an in-memory CAS register.
func wireDurable(t *testing.T) (*zip.App, *stateService, *bucket) {
t.Helper()
dir := t.TempDir()
log := luxlog.New("risktest")
b := cloud.NewBase(cloud.Deps{Logger: log, DataDir: dir, Brand: "hanzo"}, "risk")
store := newBucket()
// The SAME checkpoint the composition root wires (build.go durableCheckpoint):
// the ship reads the real on-disk path, and on an encrypting backend that path
// is only fresh after the WAL is folded and re-encrypted. Without it this
// harness would be testing a ship of a file that does not exist.
b.Durable = org.NewDurability(store, soleWriter{}, nil, org.WithCheckpoint(checkpoint))
shape, err := forest(anomaly.Config{}, aggregates())
if err != nil {
t.Fatalf("forest: %v", err)
}
s := &cloud.Service[state]{Base: b, State: state{
brand: "hanzo",
dataDir: dir,
shelf: newShelf(b),
inflight: newInflight("a measurement"),
running: newInflight("an exhaustive search"),
digest: shape.Digest(),
bill: cloud.NewResourceMeter(cloud.Deps{Logger: log}, "risk"),
}}
app := zip.New(zip.Config{Logger: log, DisableStartupMessage: true})
mount(s, app)
t.Cleanup(func() { s.State.shelf.close() })
return app, s, store
}
// checkpoint folds the WAL into the real path and re-encrypts it, which is what
// makes the bytes a ship reads the bytes a caller was told were written.
func checkpoint(ctx context.Context, db *sql.DB) error {
// The fold runs on its OWN connection and RELEASES it before the re-encrypt:
// the store holds one connection, and an envelope re-encrypt that asks for a
// second while the first is still open waits for itself. (Production splits
// these two for the same reason — build.go walCheckpointTruncate.)
if err := fold(ctx, db); err != nil {
return err
}
return sqlitedrv.Checkpoint(db)
}
func fold(ctx context.Context, db *sql.DB) error {
conn, err := db.Conn(ctx)
if err != nil {
return err
}
defer func() { _ = conn.Close() }()
var busy, frames, done int
if err := conn.QueryRowContext(ctx, "PRAGMA wal_checkpoint(TRUNCATE)").Scan(&busy, &frames, &done); err != nil {
return err
}
if busy != 0 {
return fmt.Errorf("checkpoint busy=%d", busy)
}
return nil
}
// soleWriter is the membership: one pod, which is therefore every org's elected
// writer. The election itself is tested in internal/org; here it only has to
// resolve to "this process owns the lease" so a ship can be acknowledged.
type soleWriter struct{}
func (soleWriter) Self() string { return "pod-a" }
func (soleWriter) Members() []org.Member { return []org.Member{{ID: "pod-a"}} }
// bucket is an in-memory conditional store: the CAS register the fence and the
// ship both run over. It counts writes and can be made to fail.
type bucket struct {
mu sync.Mutex
slots map[string]*slot
writes int
err error
fenced bool
}
type slot struct {
data []byte
ver int
}
func newBucket() *bucket { return &bucket{slots: map[string]*slot{}} }
func (b *bucket) puts() int {
b.mu.Lock()
defer b.mu.Unlock()
return b.writes
}
func (b *bucket) fail(err error) {
b.mu.Lock()
b.err = err
b.mu.Unlock()
}
// depose models a successor pod that has advanced this org's recorded round: the
// store admits the lease renewal and refuses every RECORD ship as fenced. That
// is not a failure — the fence is working — so Sync answers (false, nil), which
// is the outcome an error check alone never sees.
func (b *bucket) depose() {
b.mu.Lock()
b.fenced = true
b.mu.Unlock()
}
func (b *bucket) Get(_ context.Context, key string) ([]byte, string, error) {
b.mu.Lock()
defer b.mu.Unlock()
if b.err != nil {
return nil, "", b.err
}
s, ok := b.slots[key]
if !ok {
return nil, "", replica.ErrNotFound
}
return append([]byte(nil), s.data...), strconv.Itoa(s.ver), nil
}
func (b *bucket) PutIfVersion(_ context.Context, key string, data []byte, expect string) (string, error) {
b.mu.Lock()
defer b.mu.Unlock()
if b.err != nil {
return "", b.err
}
// A lease renewal is admitted — this process still runs; it is the RECORD
// ship that a successor's higher round fences.
if b.fenced && !strings.Contains(key, "lease") {
return "", fmt.Errorf("bucket: %w", replica.ErrStaleRound)
}
s, ok := b.slots[key]
cur := ""
if ok {
cur = strconv.Itoa(s.ver)
}
if cur != expect {
return "", fmt.Errorf("bucket: %w: have %q want %q", replica.ErrConflict, cur, expect)
}
if !ok {
s = &slot{}
b.slots[key] = s
}
s.ver++
s.data = append([]byte(nil), data...)
if !strings.Contains(key, "lease") { // a lease renewal is not a record ship
b.writes++
}
return strconv.Itoa(s.ver), nil
}
+325
View File
@@ -0,0 +1,325 @@
package risk
// feature.go is the ONLY file in this package that reads the warehouse. Every
// other file reaches a tenant's numbers through a function declared here, and
// every function declared here takes a minted Tenant — so "a feature read is
// tenant-scoped" is a property of the type signature, not of a reviewer's
// attention.
//
// TWO PLANES, AND THE SECOND IS WHY THE FIRST IS SAFE TO SHARE.
//
// hanzo.risk_feature per-tenant. `org` is the FIRST sort key, so a tenant read
// is a prefix scan and not a filter, and the predicate is
// bound rather than interpolated.
// hanzo.risk_baseline the network baseline. It has NO tenant column, no subject,
// no id and no pseudonym — only quantiles over a k-anonymous
// set of contributing orgs. There is no query against it that
// returns one org's rows, because the rows do not exist. That
// makes cross-tenant learning UNCOMPUTABLE rather than merely
// disallowed, which is the only form of that boundary worth
// having.
//
// THE HOT PATH DOES NOT COME THROUGH HERE. A payment-stage decision must answer
// inside the processor's authorization window, and the warehouse is a single
// StatefulSet pod that has taken api.hanzo.ai down once already. Scoring reads
// the in-memory velocity rings (constant time, fixed memory per key); this file
// is the BACKFILL that warms those rings and the read behind the dictionary and
// the search sandbox. A decide that needs analytics up is a payment plane that
// fails when analytics does.
import (
"context"
"fmt"
"strings"
"time"
"github.com/hanzoai/cloud/apps/datastore"
)
// featureTable is the per-tenant feature surface. It is a projection of the ONE
// event door (POST /v1/event -> bus -> event.event / event.error) plus the LLM
// spend ledger, reduced to the counts a risk decision reads and bucketed at five
// minutes, so a subject's recent shape is one prefix scan.
//
// org LEADS the sort key deliberately, unlike hanzo.cloud_usage's
// (timestamp, organization, ...): a per-tenant read must be a prefix scan.
const featureTable = "hanzo.risk_feature"
// baselineTable is the network baseline: quantiles per (bucket, subject kind,
// feature), over a k-anonymous set of contributing orgs. No tenant column
// exists, so no tenant row can be selected from it.
const baselineTable = "hanzo.risk_baseline"
const featureDDL = `
CREATE TABLE IF NOT EXISTS hanzo.risk_feature (
org String,
subject_kind LowCardinality(String),
subject String,
bucket DateTime,
events UInt32,
sessions UInt32,
distincts UInt32,
errors UInt32,
spend_nano Int64,
tokens UInt64,
ips UInt16,
uas UInt16,
countries UInt16,
signups UInt32,
payments UInt32,
declines UInt32,
disputes UInt32,
payouts UInt32
) ENGINE = SummingMergeTree()
ORDER BY (org, subject_kind, subject, bucket)
TTL bucket + INTERVAL 400 DAY`
const baselineDDL = `
CREATE TABLE IF NOT EXISTS hanzo.risk_baseline (
bucket Date,
subject_kind LowCardinality(String),
feature LowCardinality(String),
q10 Float64,
q50 Float64,
q90 Float64,
q99 Float64,
orgs UInt32,
n UInt64
) ENGINE = ReplacingMergeTree()
ORDER BY (bucket, subject_kind, feature)
TTL bucket + INTERVAL 400 DAY`
// ensureTables creates both planes idempotently. cloud reads and writes rows on
// the event plane but does not own its DDL (o11y does); it DOES own these two,
// so they are created here the way apps/datastore/cloudusage.go creates its own —
// CREATE IF NOT EXISTS, plus additive ADD COLUMN IF NOT EXISTS migrations, so a
// fresh warehouse and a legacy one converge on the same shape.
func ensureTables(ctx context.Context) error {
if !datastore.Ready() {
return errWarehouse
}
for _, stmt := range []string{featureDDL, baselineDDL} {
if err := datastore.Exec(ctx, stmt); err != nil {
return err
}
}
return nil
}
// featureColumns is the FIXED allowlist of columns a caller may name. Nothing
// user-derived is ever spelled into a statement: a caller names a column, this
// map answers whether that name is one of ours, and the map's own key is what
// reaches the SQL. A name that is not here is refused, never interpolated.
var featureColumns = map[string]string{
"events": "events",
"sessions": "sessions",
"distincts": "distincts",
"errors": "errors",
"spend": "spend_nano",
"tokens": "tokens",
"ips": "ips",
"uas": "uas",
"countries": "countries",
"signups": "signups",
"payments": "payments",
"declines": "declines",
"disputes": "disputes",
"payouts": "payouts",
}
// subjectKinds is the FIXED allowlist of aggregation axes. `pair` and `device`
// are the axes that surface several nominally unrelated customers acting as one,
// which is what multi-account abuse and account sharing look like from here.
var subjectKinds = map[string]bool{
"account": true, "transaction": true, "session": true, "agent": true,
"merchant": true, "payout": true, "user": true, "device": true,
"ip": true, "pair": true,
}
// featureRow is one bucket of one subject's activity, as the reader materialises
// it. The tenant is NOT a field: a row is only ever produced by a read that was
// already scoped, so carrying the key back would be the one place it could be
// compared against the wrong thing.
type featureRow struct {
bucket time.Time
values map[string]float64
}
// riskWhere is the tenancy predicate. org LEADS and is BOUND; the window is
// bound; the subject kind has already passed the allowlist and is bound anyway.
// Nothing user-derived reaches the statement as an identifier.
//
// The shape is the one apps/analytics/query.go and o11y's eventsql use, which is
// the point: a third shape is a third place for the boundary to be got wrong.
func riskWhere(t Tenant, kind, subject string, start, end time.Time) (string, []any) {
return "org = ? AND subject_kind = ? AND subject = ? AND bucket >= ? AND bucket < ?",
[]any{t.String(), kind, subject, tsLiteral(start), tsLiteral(end)}
}
// tsLiteral renders a time the way the warehouse driver binds a DateTime. Same
// transport apps/analytics uses; the value is still a bound parameter.
func tsLiteral(t time.Time) string { return t.UTC().Format("2006-01-02 15:04:05") }
// errWarehouse is the honest gap: the warehouse is unreachable, so a read
// returns nothing AND says why. A reader that answered zero would be reporting
// "this subject did nothing", which is the one answer a risk surface must never
// invent.
var errWarehouse = fmt.Errorf("risk: the warehouse is unreachable, so this subject's history cannot be read")
// window reads one subject's buckets over [start,end).
//
// The caller supplies a Tenant, which it can only have obtained from a validated
// principal, and a kind that must be in the allowlist. The column list is built
// from featureColumns' own values, never from the caller's spelling.
func window(ctx context.Context, t Tenant, kind, subject string, start, end time.Time) ([]featureRow, error) {
if !subjectKinds[kind] {
return nil, fmt.Errorf("risk: %q is not a subject kind", kind)
}
if strings.TrimSpace(subject) == "" {
return nil, fmt.Errorf("risk: no subject, so the window would name nobody")
}
if !datastore.Ready() {
return nil, errWarehouse
}
where, args := riskWhere(t, kind, subject, start, end)
// Ordered so the projection is stable across calls; the names come from our
// own map, so this concatenation carries nothing the caller wrote.
names := columnNames()
sel := make([]string, 0, len(names)+1)
sel = append(sel, "bucket")
for _, n := range names {
sel = append(sel, "sum("+featureColumns[n]+") AS "+n)
}
q := "SELECT " + strings.Join(sel, ", ") + " FROM " + featureTable +
" WHERE " + where + " GROUP BY bucket ORDER BY bucket"
rows, err := datastore.Query(ctx, q, args...)
if err != nil {
return nil, err
}
out := make([]featureRow, 0, len(rows))
for _, r := range rows {
fr := featureRow{values: make(map[string]float64, len(names))}
if b, ok := r["bucket"].(time.Time); ok {
fr.bucket = b
}
for _, n := range names {
fr.values[n] = num(r[n])
}
out = append(out, fr)
}
return out, nil
}
// columnNames returns the allowlist keys in a stable order.
func columnNames() []string {
out := make([]string, 0, len(featureColumns))
for n := range featureColumns {
out = append(out, n)
}
sortStrings(out)
return out
}
// ── the network baseline ────────────────────────────────────────────────────
// kAnonMin is how many DISTINCT orgs must contribute to a bucket before its
// quantiles may be published. Below it a quantile is close enough to one
// business's own numbers to be that business's numbers.
const kAnonMin = 25
// nMin is how many observations a published bucket needs. k orgs each
// contributing one row is k-anonymous and still statistically meaningless.
const nMin = 1000
// baselinePopulate is the WHOLE cross-org surface, and it is a package CONSTANT.
// It is never composed from request data, has no placeholder a caller could
// reach, and its projection carries no org, no subject, no id and no pseudonym —
// only quantiles and two counts. The HAVING clause is the k-anonymity gate and
// is part of the statement rather than a filter applied to its result, so a
// bucket below the threshold is never materialised at all.
//
// The reserved `$public` tenant is excluded here as well as at the mint: an
// unauthenticated stranger writes into that lane, and a stranger who can move
// the network baseline can move every tenant's comparison against it.
const baselinePopulate = `
INSERT INTO hanzo.risk_baseline (bucket, subject_kind, feature, q10, q50, q90, q99, orgs, n)
SELECT
toDate(bucket) AS bucket,
subject_kind,
'events' AS feature,
quantileExact(0.10)(events) AS q10,
quantileExact(0.50)(events) AS q50,
quantileExact(0.90)(events) AS q90,
quantileExact(0.99)(events) AS q99,
uniqExact(org) AS orgs,
count() AS n
FROM hanzo.risk_feature
WHERE org != '$public' AND org NOT LIKE '%/$public'
AND bucket >= toDateTime(toDate(now()) - 1) AND bucket < toDateTime(toDate(now()))
GROUP BY bucket, subject_kind
HAVING orgs >= 25 AND n >= 1000`
// baselineRow is one published quantile bucket. Reflection over this type is
// part of the isolation proof: a field naming a tenant, a subject or a person
// would be a leak, and the test that asserts none exists reads THIS type and the
// DDL above rather than a comment.
type baselineRow struct {
bucket time.Time
kind string
feature string
q10 float64
q50 float64
q90 float64
q99 float64
orgs uint64
n uint64
}
// publishBaseline recomputes yesterday's network quantiles. It is the ONLY
// writer of the baseline plane and it runs on a schedule, never on a request, so
// no caller can time it, steer it or observe its cost.
func publishBaseline(ctx context.Context) error {
if !datastore.Ready() {
return errWarehouse
}
return datastore.Exec(ctx, baselinePopulate)
}
// baseline reads the published quantiles for one subject kind. There is no
// tenant argument BECAUSE THERE IS NO TENANT COLUMN — every caller reads the
// same rows, which is what makes them safe to read at all.
func baseline(ctx context.Context, kind string, day time.Time) ([]baselineRow, error) {
if !subjectKinds[kind] {
return nil, fmt.Errorf("risk: %q is not a subject kind", kind)
}
if !datastore.Ready() {
return nil, errWarehouse
}
rows, err := datastore.Query(ctx,
"SELECT bucket, subject_kind, feature, q10, q50, q90, q99, orgs, n FROM "+baselineTable+
" WHERE bucket = ? AND subject_kind = ? ORDER BY feature",
day.UTC().Format("2006-01-02"), kind)
if err != nil {
return nil, err
}
out := make([]baselineRow, 0, len(rows))
for _, r := range rows {
br := baselineRow{
kind: str(r["subject_kind"]), feature: str(r["feature"]),
q10: num(r["q10"]), q50: num(r["q50"]), q90: num(r["q90"]), q99: num(r["q99"]),
orgs: uint64(num(r["orgs"])), n: uint64(num(r["n"])),
}
if b, ok := r["bucket"].(time.Time); ok {
br.bucket = b
}
// Belt on top of the HAVING: a row that predates the gate, or one an
// operator inserted by hand, is dropped on read rather than trusted.
if br.orgs < kAnonMin || br.n < nMin {
continue
}
out = append(out, br)
}
return out, nil
}
+770
View File
@@ -0,0 +1,770 @@
package risk
// grade.go is the record plane and the arithmetic behind SCORING QUALITY: what a
// score means, why a decision went the way it did, and what the organisation
// asked for at that probability.
//
// FOUR RECORDS, ALL IN THE TENANT'S OWN FILE. A policy version, a calibration,
// the verdict attached to one decision and a replay report are each something an
// auditor, a regulator or a declined customer can ask about, so none of them
// lives in memory. They ride the same cek-encrypted per-org SQLite file the
// decision log already uses, which is what makes them survive the
// Recreate-at-one-replica rollout that drops every in-memory model.
//
// THAT FILE IS OPENED WITH cloud.OrgDB AND IS THEREFORE LOCAL. OrgDB is the
// encrypted open and nothing more; the ship-before-ack path is cloud.OrgStore
// configured WithDurable, which this package does not use — so a write here is
// as durable as the pod's volume and no more. Every record in this file has that
// property, the decision log included, and it predates this plane. Named here
// rather than claimed away: the two sibling durable planes (apps/research
// compose.go, apps/books books.go) ship after every commit and treat an unacked
// ship as an error, and moving the shelf onto OrgStore is the change that would
// give these records the same guarantee.
//
// NOTHING IS CACHED, DELIBERATELY. The decide path already reads this tenant's
// rules, lists and suppressions from its own file on every decision; reading the
// policy and the calibration the same way adds one idiom rather than a second.
// It also removes an entire class of bug — a cache with no invalidation path
// that keeps deciding under a policy an operator has already replaced — and it
// is why a rollout needs no rehydration step here: the record IS the state.
//
// THE ESCALATION RULE, ONCE. The evidence (rules and the model) proposes an
// action, and the policy proposes another from the calibrated probability. The
// decision takes the STRONGER of the two, capped at what the evidence can
// justify. A deny-list rule must not be overruled by a low probability — it is
// the tenant's explicit instruction — and a high probability must not be talked
// down by quiet evidence. But the probability is a pure function of the same
// score the model's own ceiling already capped, so an uncapped ladder would void
// that ceiling by arithmetic: see ceiling() below. Two authorities for one
// decision would otherwise be a question with two answers.
import (
"context"
"crypto/sha256"
"database/sql"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
"github.com/luxfi/aml/pkg/calibrate"
// The package is aml's `evaluate`; it is bound to `quality` here because
// rule.go already declares a function called evaluate — the rule evaluator —
// and one name cannot mean two things in one package.
quality "github.com/luxfi/aml/pkg/evaluate"
"github.com/luxfi/aml/pkg/policy"
"github.com/luxfi/aml/pkg/reason"
"github.com/luxfi/aml/pkg/replay"
"github.com/zap-proto/zip"
)
// qualitySchema is applied beside the decision plane's own schema, on the same
// open, in the same file. Separate constants rather than one, because these are
// a different concern with a different owner; the same file, because a verdict
// that could outlive or precede the decision it grades is not a record.
//
// NO TTL and no expiry anywhere here. These are records: a policy version says
// what was in force on the day someone was declined, and a verdict says why.
// Only the retention plane decides when a tenant's records go, per tenant.
const qualitySchema = `
CREATE TABLE IF NOT EXISTS policy (
stage TEXT NOT NULL,
version INTEGER NOT NULL,
at TEXT NOT NULL,
by TEXT NOT NULL DEFAULT '',
reason TEXT NOT NULL DEFAULT '',
digest TEXT NOT NULL,
body TEXT NOT NULL,
PRIMARY KEY (stage, version)
);
CREATE TABLE IF NOT EXISTS calibration (
version INTEGER PRIMARY KEY,
at TEXT NOT NULL,
by TEXT NOT NULL DEFAULT '',
shape TEXT NOT NULL,
horizon INTEGER NOT NULL DEFAULT 0,
digest TEXT NOT NULL,
body TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS verdict (
decision TEXT PRIMARY KEY,
at TEXT NOT NULL,
probability REAL,
calibration TEXT NOT NULL DEFAULT '',
policy TEXT NOT NULL DEFAULT '',
reasons TEXT NOT NULL DEFAULT '[]',
refusal TEXT NOT NULL DEFAULT ''
);
CREATE TABLE IF NOT EXISTS replay (
id TEXT PRIMARY KEY,
at TEXT NOT NULL,
by TEXT NOT NULL DEFAULT '',
digest TEXT NOT NULL,
body TEXT NOT NULL
);
`
// topReasons is how many principal reasons a decision cites.
//
// Four, from Reg B: a creditor taking adverse action need not describe more than
// four reasons, and a list of every contributing feature is not an explanation
// anybody can act on. The model has nine coordinates and almost always moves on
// two or three of them.
const topReasons = 4
// verdict is the defensibility layer over one outcome: what the score meant,
// what moved it, and which ladder read it.
type verdict struct {
// probability is the calibrated likelihood. Absent — a nil pointer, not a
// zero — when no calibration is in force or the one on file was fitted under
// a different scoring shape. A density reported as a probability is a number
// that lies.
probability *float64
// reasons are the principal reasons, strongest first.
reasons []reason.Reason
// calibration and policy are the digests of what was applied, so the
// decision can be replayed against exactly them.
calibration string
policy string
// action is what the policy asked for, before the escalation rule. Empty
// when no ladder was reachable.
action string
// refusal names why there is no probability, when there is none. Silence and
// a low probability render the same on a screen and are opposite facts.
refusal string
}
// dispositions maps the decision log's verdict vocabulary onto the engine's.
//
// It is a CLOSED map with no default arm: a verdict this table does not name is
// unjudged, which is the honest reading of a word nobody has defined the meaning
// of. fraud, chargeback and abuse are all "the decision was right to be
// worried"; only legitimate is the other way; unknown says a human looked and
// could not tell, which is not evidence for either side.
var dispositions = map[string]replay.Disposition{
"fraud": replay.Productive,
"chargeback": replay.Productive,
"abuse": replay.Productive,
"legitimate": replay.Unproductive,
}
// scoringShape is the fingerprint everything a calibration depends on must agree
// on, and it is the training-serving skew control.
//
// A calibration maps THE DECISION SCORE to a probability, and that score is the
// combination of the model's own score with whatever this tenant's rules
// contributed. So the shape covers BOTH: the detector's geometry and feature
// inventory (anomaly.Store.Digest, which already folds the inventory in order
// with its neutral values), and this tenant's enabled rule set with the weights
// and actions each rule carries. Change either and the score distribution moves;
// the map fitted before the change then refuses to answer rather than quietly
// reporting probabilities for coordinates that no longer mean what they meant.
//
// A RULE-WIDE SUPPRESSION IS IN IT. Muting a rule is the day-to-day tuning knob
// and retiring one is the rare act, but combine() sums only the hits that were
// not suppressed — so a mute moves every score that rule touched EXACTLY as
// retiring it would. A control that noticed the rare change and not the common
// one would be a control nobody could rely on.
//
// LISTS AND SUBJECT-SCOPED SUPPRESSIONS ARE DELIBERATELY NOT IN IT, by the same
// argument in both directions. A list entry is data a rule reads, and a
// suppression naming one subject is a statement about that subject — neither
// changes what the scorer computes for the population, and both change
// constantly in normal operation. A plane whose calibration expired every time a
// stolen card was added to a deny list, or every time one merchant was excused,
// would never have a calibration at all. The boundary is stated rather than
// assumed: what moves the DISTRIBUTION invalidates, operational data about one
// row does not.
func scoringShape(db *sql.DB, model string) (string, error) {
rules, err := loadRules(db)
if err != nil {
return "", err
}
sups, err := loadSuppressions(db)
if err != nil {
return "", err
}
return shapeOf(model, rules, sups, time.Now()), nil
}
// shapeOf folds the coordinate system from the values that produce a score.
//
// It takes the values rather than the store so the DECIDE path can fold the
// exact rules and suppressions it just scored under, instead of re-reading them
// and recording a shape a concurrent write may already have moved. scoringShape
// is the same fold over a fresh read, for the callers that hold neither.
func shapeOf(model string, rules []rule, sups []suppression, now time.Time) string {
// Sorted by id so the fingerprint is a property of the SET and not of the
// order rows happened to come back in.
rules = append([]rule(nil), rules...)
sort.Slice(rules, func(i, j int) bool { return rules[i].ID < rules[j].ID })
muted := make([]string, 0, len(sups))
for _, s := range sups {
// Expired mutes nothing — `until` in the past is the store's own expiry and
// the decide path already reads it that way, so a shape that still counted
// one would describe a scorer that no longer exists.
if !s.Until.IsZero() && now.After(s.Until) {
continue
}
// A mute that names a subject is about that subject. A mute that names a
// rule and no subject silences it for the whole population, which is a move
// of the distribution.
if s.Rule == "" || s.Subject != "" {
continue
}
muted = append(muted, s.Rule+":"+s.Kind)
}
sort.Strings(muted)
h := sha256.New()
fmt.Fprintf(h, "shape/v2|%s|", model)
for _, r := range rules {
if !r.Enabled {
continue
}
fmt.Fprintf(h, "%s:%s:%g:%s|", r.ID, r.Stage, r.Weight, r.Action)
}
fmt.Fprint(h, "muted|")
for _, m := range muted {
fmt.Fprintf(h, "%s|", m)
}
return hex.EncodeToString(h.Sum(nil))
}
// grade turns an outcome into a defensible one.
//
// It reads the tenant's own calibration and its ladder for this stage, maps the
// recorded score to a probability, ranks the reasons out of the model's exact
// counterfactual attribution and the rules that actually fired, and returns the
// action the policy asks for. It writes nothing: the caller records the verdict
// in the same place and at the same moment it records the decision, because a
// decision whose reasons landed separately could exist without them.
func grade(db *sql.DB, shape string, stage string, out outcome) verdict {
v := verdict{reasons: reasonsOf(out)}
cal, _, _, ok, err := currentCalibration(db)
switch {
case err != nil:
v.refusal = "the calibration could not be read, so this score has no probability"
return v
case !ok:
v.refusal = "no calibration is fitted for this organisation, so a score is a rank and not a probability"
return v
}
read, err := cal.Under(shape)
if err != nil {
// The commonest cause by far, and the one worth naming: the model, the rule
// set or a rule-wide mute moved since the fit.
v.refusal = "the calibration on file was fitted under a different scoring shape: " + err.Error()
return v
}
p := read.P(out.score)
v.probability, v.calibration = &p, read.Digest()
pol, ok, err := currentPolicy(db, stage)
switch {
case err != nil:
v.refusal = "the policy could not be read, so no threshold was applied"
return v
case !ok:
v.refusal = "no policy is set for this stage, so the probability was computed and nothing was decided from it"
return v
}
v.policy, v.action = pol.Digest, pol.Action(p)
return v
}
// escalate is the ONE place two authorities become one decision: the stronger of
// what the evidence asked for and what the policy asked for, with the policy
// held to what the evidence behind it can justify.
func escalate(evidence, policy string, out outcome) string {
policy = atMost(policy, ceiling(out))
if actionRank(policy) > actionRank(evidence) {
return policy
}
return evidence
}
// ceiling is the strongest action the evidence behind one outcome can carry, or
// empty when it carries no ceiling at all.
//
// A decision the MODEL alone moved can put a transaction in front of a person
// and cannot decline one: an unexplainable refusal is not a decision anybody can
// defend to the customer or to a chargeback network, which is why decide caps
// the model's own hit at modelCeiling (decide.go, rule.go). The policy ladder is
// not a second, independent authority for that decline — it reads the calibrated
// probability, and the probability is a pure function of the SAME score — so
// applying the ladder over the cap would void the cap by arithmetic and the
// documented invariant would hold only until somebody set a band.
//
// A RULE the organisation wrote is different in kind. It is an explicit
// instruction with a name, a weight and a sentence a person can read, so a
// decline it reaches is explainable and no ceiling applies. A suppressed hit is
// not evidence: it contributed zero weight and it is not cited as a reason, so
// citing it here would let a mute both silence a rule and license a decline.
func ceiling(out outcome) string {
for _, h := range out.hits {
if h.Suppressed || h.Rule == modelRuleID {
continue
}
return ""
}
return modelCeiling
}
// atMost caps an action at a ceiling. An empty ceiling caps nothing.
func atMost(action, ceiling string) string {
if ceiling == "" || actionRank(action) <= actionRank(ceiling) {
return action
}
return ceiling
}
// reasonsOf ranks the principal reasons behind one outcome.
//
// The model's contribution comes from its own per-feature counterfactual — move
// one coordinate to neutral, rescore on the same trees — so it is a measurement
// on the model that decided rather than a surrogate fitted afterwards. Each
// rule that fired contributes one reason naming itself.
//
// A SUPPRESSED HIT IS NOT A REASON. It is recorded, it is visible, and it
// contributed zero weight to the action — so citing it as a reason for the
// action would be false. That is the same argument the suppression path already
// makes for keeping the hit in the record.
func reasonsOf(out outcome) []reason.Reason {
rs := reason.Rank(out.causes, 0)
for _, h := range out.hits {
if h.Suppressed || h.Rule == modelRuleID {
continue
}
rs = append(rs, reason.OfRule(h.Rule, h.Name, h.Severity, h.Weight))
}
sort.SliceStable(rs, func(i, j int) bool { return rs[i].Weight > rs[j].Weight })
if len(rs) > topReasons {
rs = rs[:topReasons]
}
return rs
}
// modelRuleID is the identifier the engine's own detector files its hit under.
// Its contribution is already expressed feature by feature in the causes, so
// citing the hit as well would count one piece of evidence twice.
const modelRuleID = "anomaly"
// ── the policy record ───────────────────────────────────────────────────────
// currentPolicy reads the ladder in force for one stage: the highest version.
func currentPolicy(db *sql.DB, stage string) (policy.Policy, bool, error) {
var body string
err := db.QueryRow(`SELECT body FROM policy WHERE stage = ? ORDER BY version DESC LIMIT 1`, stage).Scan(&body)
if errors.Is(err, sql.ErrNoRows) {
return policy.Policy{}, false, nil
}
if err != nil {
return policy.Policy{}, false, err
}
var p policy.Policy
if err := json.Unmarshal([]byte(body), &p); err != nil {
return policy.Policy{}, false, err
}
return p, true, nil
}
// putPolicy mints the next version of a stage's ladder.
//
// Nothing is ever updated in place. A policy change is a governance decision and
// the question "what was in force when this customer was declined" has to be a
// lookup rather than a reconstruction — so the row is an insert, the version is
// the successor of whatever is there, and the previous version stays exactly as
// it was.
func putPolicy(db *sql.DB, p policy.Policy) (policy.Policy, error) {
sealed, err := policy.Seal(p)
if err != nil {
return policy.Policy{}, zip.ErrBadRequest(err.Error())
}
tx, err := db.Begin()
if err != nil {
return policy.Policy{}, err
}
defer func() { _ = tx.Rollback() }()
var last sql.NullInt64
if err := tx.QueryRow(`SELECT MAX(version) FROM policy WHERE stage = ?`, sealed.Stage).Scan(&last); err != nil {
return policy.Policy{}, err
}
sealed.Version = int(last.Int64) + 1
sealed.At = time.Now().UTC()
// Digest is over what the ladder DECIDES, so it does not move with the
// version or the timestamp. Re-sealing after stamping those would be a no-op
// and is not done, which is what keeps two identical ladders comparable.
body, err := json.Marshal(sealed)
if err != nil {
return policy.Policy{}, err
}
if _, err := tx.Exec(
`INSERT INTO policy (stage, version, at, by, reason, digest, body) VALUES (?,?,?,?,?,?,?)`,
sealed.Stage, sealed.Version, stamp(sealed.At), sealed.By, sealed.Reason, sealed.Digest, string(body),
); err != nil {
return policy.Policy{}, err
}
if err := tx.Commit(); err != nil {
return policy.Policy{}, err
}
return sealed, nil
}
// policyVersions is the audit trail for one stage, newest first.
func policyVersions(db *sql.DB, stage string, limit int) ([]policy.Policy, error) {
rows, err := db.Query(
`SELECT body FROM policy WHERE stage = ? ORDER BY version DESC LIMIT ?`, stage, limit)
if err != nil {
return nil, err
}
defer func() { _ = rows.Close() }()
var out []policy.Policy
for rows.Next() {
var body string
if err := rows.Scan(&body); err != nil {
return nil, err
}
var p policy.Policy
if err := json.Unmarshal([]byte(body), &p); err != nil {
return nil, err
}
out = append(out, p)
}
return out, rows.Err()
}
// policyStages lists the stages this tenant has ever set a ladder for.
func policyStages(db *sql.DB) ([]string, error) {
rows, err := db.Query(`SELECT DISTINCT stage FROM policy ORDER BY stage`)
if err != nil {
return nil, err
}
defer func() { _ = rows.Close() }()
var out []string
for rows.Next() {
var s string
if err := rows.Scan(&s); err != nil {
return nil, err
}
out = append(out, s)
}
return out, rows.Err()
}
// ── the calibration record ──────────────────────────────────────────────────
// currentCalibration reads the map in force: the highest version. One
// calibration per tenant rather than one per stage — the map turns THIS model's
// score into a probability and the model is per tenant, so a per-stage split
// would divide the same evidence into thinner samples of the same relationship.
func currentCalibration(db *sql.DB) (calibrate.Map, int, time.Time, bool, error) {
var version, horizon int
var at, body string
err := db.QueryRow(
`SELECT version, at, horizon, body FROM calibration ORDER BY version DESC LIMIT 1`,
).Scan(&version, &at, &horizon, &body)
if errors.Is(err, sql.ErrNoRows) {
return calibrate.Map{}, 0, time.Time{}, false, nil
}
if err != nil {
return calibrate.Map{}, 0, time.Time{}, false, err
}
var m calibrate.Map
if err := json.Unmarshal([]byte(body), &m); err != nil {
return calibrate.Map{}, 0, time.Time{}, false, err
}
return m, version, unstamp(at), true, nil
}
// putCalibration records a fit as the next version.
func putCalibration(db *sql.DB, m calibrate.Map, by string, horizon int) (int, time.Time, error) {
body, err := json.Marshal(m)
if err != nil {
return 0, time.Time{}, err
}
tx, err := db.Begin()
if err != nil {
return 0, time.Time{}, err
}
defer func() { _ = tx.Rollback() }()
var last sql.NullInt64
if err := tx.QueryRow(`SELECT MAX(version) FROM calibration`).Scan(&last); err != nil {
return 0, time.Time{}, err
}
version := int(last.Int64) + 1
at := time.Now().UTC()
if _, err := tx.Exec(
`INSERT INTO calibration (version, at, by, shape, horizon, digest, body) VALUES (?,?,?,?,?,?,?)`,
version, stamp(at), by, m.Shape, horizon, m.Digest, string(body),
); err != nil {
return 0, time.Time{}, err
}
return version, at, tx.Commit()
}
// ── the verdict record ──────────────────────────────────────────────────────
// getVerdict reads one decision's grading back, so a repeat of an idempotent
// decide returns the same reasons rather than a decision with none.
func getVerdict(db *sql.DB, id string) (verdict, bool, error) {
var p sql.NullFloat64
var cal, pol, reasons, refusal string
err := db.QueryRow(
`SELECT probability, calibration, policy, reasons, refusal FROM verdict WHERE decision = ?`, id,
).Scan(&p, &cal, &pol, &reasons, &refusal)
if errors.Is(err, sql.ErrNoRows) {
return verdict{}, false, nil
}
if err != nil {
return verdict{}, false, err
}
v := verdict{calibration: cal, policy: pol, refusal: refusal}
if p.Valid {
val := p.Float64
v.probability = &val
}
if err := json.Unmarshal([]byte(reasons), &v.reasons); err != nil {
return verdict{}, false, err
}
return v, true, nil
}
// graded is how every READ path asks for a decision's grading: the verdict, or a
// verdict that SAYS there is none.
//
// The not-found used to be discarded with `_`, which made an ungraded decision
// indistinguishable from a graded one with nothing to say — so a block could be
// served on the dispute-packet surface with no probability, no principal reasons
// and no refusal. The rows are written in one transaction now, so this should be
// unreachable; a record plane that answered "nothing to say" for a state it
// believes impossible would be hiding exactly the corruption worth knowing about.
func graded(db *sql.DB, id string) (verdict, error) {
v, ok, err := getVerdict(db, id)
if err != nil {
return verdict{}, err
}
if !ok {
return verdict{refusal: "this decision has no recorded grading, so there is no probability, " +
"no principal reason and nothing this plane can defend it with"}, nil
}
return v, nil
}
// ── the replay record ───────────────────────────────────────────────────────
func putReplay(db *sql.DB, id string, at time.Time, by string, rep quality.Report) error {
body, err := json.Marshal(rep)
if err != nil {
return err
}
_, err = db.Exec(`INSERT INTO replay (id, at, by, digest, body) VALUES (?,?,?,?,?)`,
id, stamp(at), by, rep.Digest, string(body))
return err
}
func getReplay(db *sql.DB, id string) (quality.Report, time.Time, bool, error) {
var at, body string
err := db.QueryRow(`SELECT at, body FROM replay WHERE id = ?`, id).Scan(&at, &body)
if errors.Is(err, sql.ErrNoRows) {
return quality.Report{}, time.Time{}, false, nil
}
if err != nil {
return quality.Report{}, time.Time{}, false, err
}
var rep quality.Report
if err := json.Unmarshal([]byte(body), &rep); err != nil {
return quality.Report{}, time.Time{}, false, err
}
return rep, unstamp(at), true, nil
}
// ── reading the history back ────────────────────────────────────────────────
// maxHistory bounds every read of the decision log for measurement.
//
// The same bound the search sandbox uses, and for the same reason: this runs
// in-request against a single-writer file, so an unbounded scan is a tenant
// holding the only connection its own decisions need.
const maxHistory = 50_000
// evidence is one bounded read of the decision log: the rows a measurement was
// computed from, and the two facts that say what the bounds left out. Both are
// reported on every answer, because a measurement is only as good as the sample
// it saw and a sample silently cut reads as a complete one.
type evidence struct {
// history is the rows, OLDEST FIRST, ready to measure over.
history []quality.Recorded
// truncated says the bound cut the read: there are older mature decisions
// under this shape that no number here was computed from.
truncated bool
// superseded counts the mature judged decisions excluded because they were
// scored under a DIFFERENT coordinate system. It is the evidence a refit
// cannot honestly use, and the number that says why a fit went thin after a
// governed change.
superseded int
}
// recorded reads this tenant's decision log as measurable observations, under
// ONE scoring shape, restricted to decisions old enough for their outcome to
// have arrived and bounded to the most recent of them.
//
// THE MATURITY HORIZON IS THE WHOLE OF WHY THIS TAKES A PARAMETER. Analysts
// judge within hours; a card network dispute lands 30 to 120 days after the
// transaction it disputes. Measure over everything and the recent tail is
// enriched with analyst clearances and stripped of the chargebacks that have not
// arrived yet, so the prevalence is understated and every threshold derived from
// it sits too high. Excluding decisions younger than the horizon is what makes
// the judged set representative rather than merely recent.
//
// THE SHAPE IS THE OTHER FILTER AND IT IS NOT OPTIONAL. A score is a coordinate.
// Rows scored before a rule was written, retired, reweighted or muted are
// coordinates in a system that no longer exists, and a fit taken over them and
// stamped with today's shape is precisely the trainingserving skew the shape
// gate exists to refuse — arrived at from the inside, by the remedy the gate's
// own message recommends. Filtering here means a refit after a governed change
// finds thin evidence and SAYS SO, instead of re-blessing history it cannot read.
//
// THE BOUND TAKES THE MOST RECENT ROWS. Ascending plus LIMIT takes the oldest,
// which past the bound freezes every fit, every measurement and every replay on
// the dawn of the tenant's log forever. The read is descending and the slice is
// reversed once, so what comes back is the LATEST maxHistory decisions in time
// order — and truncated says when there were more.
//
// The ordering is on the second-truncated timestamp and then the id, which is a
// TOTAL order: the stored form is RFC 3339 with a variable-length fraction, so a
// plain string comparison sorts "…:00.5Z" before "…:00Z". At the day scale a
// horizon works on that is immaterial, but a learning curve that split its
// history on it would not be reproducible, and reproducibility is the property
// being sold.
func recorded(ctx context.Context, db *sql.DB, horizon int, shape string, limit int) (evidence, error) {
if limit <= 0 || limit > maxHistory {
limit = maxHistory
}
cutoff := time.Now().UTC().AddDate(0, 0, -horizon).Format("2006-01-02T15:04:05")
// The scan is its own scope, and that is load-bearing rather than tidy: the
// org file is opened with ONE connection, so an open *sql.Rows holds the only
// one there is. Reading limit+1 and stopping early leaves it open, and the
// count below would then wait for a connection this function is itself
// holding — a deadlock, not an error, bounded by nothing.
scan := func() ([]quality.Recorded, bool, error) {
// limit+1 answers "was there more" exactly, and for free — a COUNT over the
// same predicate is a second scan of the same rows to learn one bit.
rows, err := db.QueryContext(ctx,
`SELECT id, at, action, score, label FROM decision
WHERE substr(at,1,19) <= ? AND shape = ?
ORDER BY substr(at,1,19) DESC, id DESC LIMIT ?`, cutoff, shape, limit+1)
if err != nil {
return nil, false, err
}
defer func() { _ = rows.Close() }()
out := make([]quality.Recorded, 0, limit)
truncated := false
for rows.Next() {
if len(out) == limit {
truncated = true
break
}
var id, at, action, label string
var score float64
if err := rows.Scan(&id, &at, &action, &score, &label); err != nil {
return nil, false, err
}
out = append(out, quality.Recorded{
Observation: quality.Observation{
ID: id, At: unstamp(at), Score: score, Action: action,
Disposition: dispositions[strings.ToLower(label)],
},
// Every judgement this plane holds today came from a person reviewing
// a decision. The below-the-line sample arm the engine already selects
// is not yet wired to a label source, so the honest exploration share
// is zero and Replay reports it as such rather than assuming it away.
Source: "review",
})
}
return out, truncated, rows.Err()
}
out, truncated, err := scan()
if err != nil {
return evidence{}, err
}
// Back into time order: every consumer of a history is temporal — the
// learning curve splits it, the replay walks it, the report bounds it.
for i, j := 0, len(out)-1; i < j; i, j = i+1, j-1 {
out[i], out[j] = out[j], out[i]
}
w := evidence{history: out, truncated: truncated}
if err := db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM decision WHERE label != '' AND substr(at,1,19) <= ? AND shape != ?`,
cutoff, shape).Scan(&w.superseded); err != nil {
return evidence{}, err
}
return w, nil
}
// immature counts the labelled decisions the horizon excluded. It is reported
// beside every fit and every measurement, because a horizon that quietly drops
// most of the evidence is the difference between a thin answer and a wrong one.
func immature(ctx context.Context, db *sql.DB, horizon int) (int, error) {
cutoff := time.Now().UTC().AddDate(0, 0, -horizon).Format("2006-01-02T15:04:05")
var n int
err := db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM decision WHERE label != '' AND substr(at,1,19) > ?`, cutoff).Scan(&n)
return n, err
}
// samples reduces recorded observations to what a calibration fit reads.
// Unjudged rows are dropped rather than counted as unproductive: a fit over
// "unknown means fine" is a fit against the incumbent policy, not against the
// world.
func samples(history []quality.Recorded) []calibrate.Sample {
out := make([]calibrate.Sample, 0, len(history))
for _, h := range history {
switch h.Disposition {
case replay.Productive:
out = append(out, calibrate.Sample{Score: h.Score, Productive: true})
case replay.Unproductive:
out = append(out, calibrate.Sample{Score: h.Score})
}
}
return out
}
// unacted is the share of the judged evidence that came from decisions the
// policy LET THROUGH.
//
// It is the honest limit on every number measured from this log. A blocked
// decision has no outcome — the counterfactual is unobserved and no arithmetic
// recovers it — so a judged set made entirely of decisions the incumbent chose
// to act on measures agreement with the incumbent rather than accuracy. Near
// zero here means exactly that, and it is reported rather than assumed away.
func unacted(history []quality.Recorded) (float64, int) {
var judged, allowed int
for _, h := range history {
if h.Disposition != replay.Productive && h.Disposition != replay.Unproductive {
continue
}
judged++
if h.Action == ActionAllow {
allowed++
}
}
if judged == 0 {
return 0, 0
}
return round4(float64(allowed) / float64(judged)), judged
}
+352
View File
@@ -0,0 +1,352 @@
package risk
// learn.go is the model plane: train, exhaustive-search, score — on the AI
// cloud's own substrate, in process, per tenant.
//
// WHERE THE MODELS TRAIN, AND WHY NOT THE ALTERNATIVES.
//
// - hanzoai/ai is the LLM router. It has no trainer.
// - candle is Rust and cloud does not link it.
// - Kubeflow (/v1/train/jobs, /v1/train/experiments) is real and stays the
// escape hatch for a customer who wants a heavy supervised sweep — but
// apps/ml fails closed on it today because no kserve/katib/trainer chart is
// deployed. It is not a substrate this product can stand on.
//
// So: luxfi/aml's half-space trees, in this process. That is not a placeholder —
// it is the right shape for the problem, for five reasons that a neural
// alternative cannot offer:
//
// 1. NO TRAINING PASS AND NO RETAINED SAMPLE. The geometry is built BEFORE any
// data arrives; the model IS a set of mass counters. Training is one online
// increment per observation, so there is no job, no queue and no window in
// which the tenant is protected by a stale model.
// 2. PER TENANT INCLUDING THE GEOMETRY. The seed is mix(cfg.Seed, orgID), so two
// tenants do not merely hold different counters — they hold DIFFERENT TREES.
// Probing one reveals nothing about where another's regions lie.
// 3. BOUNDED, PER TENANT. 336 KB of forest per tenant at the defaults, and a
// tenant's aggregates are capped by its OWN budget (bound.go) — one store per
// tenant, so an eviction can only ever drop a key that tenant put there. A
// tenant that goes idle costs nothing and one that comes back re-warms.
// 4. ATTRIBUTION IS A COUNTERFACTUAL ON THE MODEL THAT RAISED THE ALERT. Move
// one coordinate to its neutral value, rescore, and the drop IS that
// feature's contribution. No second explainer model, and therefore no second
// thing that can be wrong. That is what makes a decline defensible.
// 5. IT CANNOT ACT ALONE. Evidence is capped at review; weight is non-negative;
// NaN and Inf are refused.
//
// EXHAUSTIVE SEARCH WITHOUT KUBEFLOW. luxfi/aml's replay package already replays
// a candidate over real history through the engine's own evaluator, writing
// nothing — it reaches the evaluator through a one-method interface and history
// through another, and imports no store, so a dry run is STRUCTURALLY dry.
// mlSearch generalises that from one rule candidate to a grid over the model
// topology, replays each candidate over the tenant's own recent decisions, and
// returns the learning curve and the winning topology. Native Go, per tenant, no
// GPU, no CRD.
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"math"
"sort"
"time"
"github.com/luxfi/aml/pkg/anomaly"
"github.com/luxfi/aml/pkg/types"
)
// snapshotKey names the row a tenant's learned state is kept under. One row: a
// tenant has one model, and a second row would be a second answer to one
// question.
const snapshotKey = "anomaly"
// keep writes a tenant's learned state into its own encrypted file.
//
// This is what makes a rollout survivable. cloud is strategy Recreate at one
// replica, so every deploy drops the process — and with it every warming model
// and the appetite threshold it had computed. Without this, every deploy
// silently resets every tenant to warming, and a warming model REFUSES to score,
// which reads as "clean" to anything that does not check Refusal.
//
// It takes the tenant's own db and the tenant's own model, because both come off
// one cell: a keep that resolved either of them for itself could pair one
// tenant's file with another tenant's state.
func keep(db *sql.DB, t Tenant, model *anomaly.Store) error {
if db == nil || model == nil {
return nil
}
snap, ok := model.Snapshot(t.String())
if !ok {
return nil // nothing learned for this tenant; nothing to keep
}
body, err := json.Marshal(snap)
if err != nil {
return err
}
return putModel(db, snapshotKey, body)
}
// restore reinstates a tenant's learned state into that tenant's own model.
// A snapshot whose shape does not match the running inventory is REFUSED by the
// engine, not coerced: state the model would treat as its own memory has to have
// come from this algorithm over this feature set.
func restore(db *sql.DB, model *anomaly.Store, t Tenant) error {
body, err := getModel(db, snapshotKey)
if err != nil {
return nil // no snapshot is the normal first-run state, not a failure
}
var snap anomaly.Snapshot
if err := json.Unmarshal(body, &snap); err != nil {
return err
}
// The snapshot's tenant must be the tenant asking for it. The engine checks
// this too; checking here as well means a restore of A's file into B's
// request is refused at the boundary that knows who asked.
if snap.OrgID != t.String() {
return fmt.Errorf("risk: snapshot belongs to another tenant")
}
return model.Restore(snap)
}
// ── exhaustive search ───────────────────────────────────────────────────────
// axis is one dimension of the topology grid, with the values the search may
// try. Both are CLOSED: an unbounded grid is an unbounded amount of a shared
// pod's CPU, reachable by anyone with a key.
type axis struct {
name string
values []float64
}
// grid is the search space. Trees x Depth x Window x Blend x Review is 3*3*3*3*3
// = 243 candidates at the widest, which is seconds of one core over a few
// thousand replayed decisions — bounded by construction rather than by a
// timeout.
func grid() []axis {
return []axis{
{"trees", []float64{15, 25, 40}},
{"depth", []float64{6, 8, 10}},
{"window", []float64{128, 256, 512}},
{"blend", []float64{0.1, 0.25, 0.5}},
{"review", []float64{0.005, 0.01, 0.02}},
}
}
// candidate is one point in the grid.
type candidate struct {
// Trees is how many half-space trees the model holds.
Trees int `json:"trees"`
// Depth is how deep each tree splits.
Depth int `json:"depth"`
// Window is how many observations make up one reference window.
Window int `json:"window"`
// Blend is how much of a closing window folds into the reference.
Blend float64 `json:"blend"`
// Review is the share of the stream this topology may send for examination.
Review float64 `json:"review"`
}
// trial is what one candidate did over the replayed history.
type trial struct {
// Candidate is the topology tried.
Candidate candidate `json:"candidate"`
// Scored is how many observations the model was able to score. A topology
// that warms slowly scores fewer, which is a real cost and is reported.
Scored int `json:"scored"`
// Alerted is how many of those it would have alerted on.
Alerted int `json:"alerted"`
// Realised is Alerted/Scored — the share actually reached, against the
// Review share intended. The gap between the two IS the governance report.
Realised float64 `json:"realised"`
// Separation is the mean score of the alerted set minus the mean score of
// the rest. It is the ranking objective: a topology that separates the tail
// from the body is doing the job whatever its absolute scores look like.
Separation float64 `json:"separation"`
// Warm is how many observations passed before the model would score at all.
Warm int `json:"warm"`
}
// searchReport is the whole answer.
type searchReport struct {
// Events is how many historical observations were replayed.
Events int `json:"events"`
// From and To are the period they span.
From time.Time `json:"from,omitzero"`
To time.Time `json:"to,omitzero"`
// Trials is every candidate tried, best first.
Trials []trial `json:"trials"`
// Winner is the best-separating topology that also honoured its stated
// appetite. Absent when no candidate did both.
Winner *candidate `json:"winner,omitempty"`
// Curve is the learning curve of the winner: separation as a function of how
// much of the history it had seen, in ten steps.
Curve []float64 `json:"curve,omitempty"`
// Refusal names why a report is empty when it is. An empty report and a
// report of no alerts are opposite facts.
Refusal string `json:"refusal,omitempty"`
}
// errNoHistory is what a search returns instead of a clean-looking zero. replay
// refuses an empty history for exactly this reason: "no alerts" is what a quiet
// rule and an unrun rule both look like, and the difference is the whole reason
// a sandbox exists.
var errNoHistory = fmt.Errorf("risk: no history to replay, so a result would be indistinguishable from a quiet model")
// searchRun replays every candidate over one tenant's own recorded observations.
//
// Nothing is written and nothing outside this function's own scratch models is
// touched: each candidate gets a FRESH anomaly.Store over a FRESH velocity store
// seeded from the tenant key, so a search can never move the live model's
// counters, and two searches for two tenants can never see each other's.
func searchRun(ctx context.Context, t Tenant, history []observation) (searchReport, error) {
if len(history) == 0 {
return searchReport{Refusal: errNoHistory.Error()}, errNoHistory
}
rep := searchReport{Events: len(history), From: history[0].at, To: history[len(history)-1].at}
for _, c := range candidates() {
select {
case <-ctx.Done():
return rep, ctx.Err()
default:
}
tr, err := replayCandidate(t, c, history)
if err != nil {
continue // a topology the engine refuses is not a result, it is a non-candidate
}
rep.Trials = append(rep.Trials, tr)
}
sort.SliceStable(rep.Trials, func(i, j int) bool {
return rep.Trials[i].Separation > rep.Trials[j].Separation
})
for i := range rep.Trials {
// The winner must have SCORED something and must have honoured its stated
// appetite within a factor of two. A topology that separates beautifully
// while alerting on nothing is not a control.
tr := rep.Trials[i]
if tr.Scored == 0 || tr.Realised == 0 {
continue
}
if tr.Realised > 2*tr.Candidate.Review {
continue
}
w := tr.Candidate
rep.Winner = &w
rep.Curve = curve(t, w, history)
break
}
if rep.Winner == nil {
rep.Refusal = "no candidate both scored and honoured its stated appetite over this history"
}
return rep, nil
}
// candidates expands the grid. Written as an explicit product so the count is
// visible in the code rather than emergent from a recursion.
func candidates() []candidate {
g := grid()
var out []candidate
for _, trees := range g[0].values {
for _, depth := range g[1].values {
for _, win := range g[2].values {
for _, blend := range g[3].values {
for _, review := range g[4].values {
out = append(out, candidate{
Trees: int(trees), Depth: int(depth), Window: int(win),
Blend: blend, Review: review,
})
}
}
}
}
}
return out
}
// replayCandidate runs one topology over the history in a sandbox.
//
// The sandbox is built by the SAME two bounded constructors the live plane uses
// (bound.go), so a search cannot be the one place a 100,000-key store shared by
// everyone comes back — and the sandbox holds one tenant's replay, under that
// tenant's own bound, exactly like the plane it is a model of.
func replayCandidate(t Tenant, c candidate, history []observation) (trial, error) {
vel := aggregates()
model, err := forest(anomaly.Config{
Trees: c.Trees, Depth: c.Depth, Window: c.Window, Blend: c.Blend,
Appetite: anomaly.Appetite{Review: c.Review, Sample: 0.001},
}, vel)
if err != nil {
return trial{}, err
}
tr := trial{Candidate: c}
var alerted, rest []float64
for _, o := range history {
record(vel, t, o)
tx := types.Transaction{
ID: o.id, OrgID: t.String(), UserID: o.subject, AccountID: o.subject,
Currency: o.currency, Direction: o.direction,
IPAddress: o.signals["ip"], DeviceFingerprint: o.signals["device"],
Timestamp: o.at, USD: nanoUSD(o.amount),
}
a := model.Inspect(tx, types.Entity{ID: o.subject, OrgID: t.String()})
// Inspect does not learn, so a second pass through the learning path is
// what advances the model. judge(learn=true) is Assess; in shadow it
// returns no hit, and the counters still move — which is exactly the
// replay we want.
_, _ = model.Assess(tx, types.Entity{ID: o.subject, OrgID: t.String()})
if !a.Scored {
tr.Warm++
continue
}
tr.Scored++
if a.Score >= a.Cut {
tr.Alerted++
alerted = append(alerted, a.Score)
} else {
rest = append(rest, a.Score)
}
}
if tr.Scored > 0 {
tr.Realised = round4(float64(tr.Alerted) / float64(tr.Scored))
}
tr.Separation = round4(mean(alerted) - mean(rest))
return tr, nil
}
// curve is the winner's separation as a function of how much history it had
// seen. Ten steps: enough to see whether the model is still improving, few
// enough that the answer is a chart and not a data set.
func curve(t Tenant, c candidate, history []observation) []float64 {
out := make([]float64, 0, 10)
for i := 1; i <= 10; i++ {
n := len(history) * i / 10
if n == 0 {
out = append(out, 0)
continue
}
tr, err := replayCandidate(t, c, history[:n])
if err != nil {
out = append(out, 0)
continue
}
out = append(out, tr.Separation)
}
return out
}
func mean(xs []float64) float64 {
if len(xs) == 0 {
return 0
}
var s float64
for _, x := range xs {
s += x
}
v := s / float64(len(xs))
if math.IsNaN(v) || math.IsInf(v, 0) {
return 0
}
return v
}
+1489
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+381
View File
@@ -0,0 +1,381 @@
// Package risk is Hanzo Risk: the scoring and decision plane for any entity a
// tenant has — an account, a transaction, a session, an agent, a merchant, a
// payout — and the shared model plane both it and the compliance face read.
//
// THREE VERBS, ONE CORE: decide, record, learn.
//
// /v1/risk decide + record. One hot op (POST /v1/risk/decide) answers
// allow / challenge / review / restrict / block WITH A REASON, and
// the planes around it (decisions, rules, lists, suppressions,
// controls, dictionary, activity) are how a tenant governs it.
// /v1/ml learn. train, exhaustive-search, score, plus the state a reviewer
// reads and the snapshot an auditor pins.
//
// Fraud is a USE of /v1/risk, not a sibling of it, and neither is abuse, bots,
// account takeover, spam or pay-as-you-go abuse. There is no /v1/fraud.
//
// WHY IT IS IN CLOUD AND NOT A SERVICE OF ITS OWN. Being in cloud is what earns
// the five things a standalone engine would each have to grow: IAM-validated
// identity, the tenant gate, usage metering, billing and structured logs. None
// of them is free — every one is wired explicitly below, and the table in
// Mount's comment says where.
//
// WHAT IS LINKED FROM THE ENGINE, AND WHAT IS DELIBERATELY NOT. github.com/luxfi/aml
// is a MODULE DEPENDENCY, exactly as luxfi/kms and hanzoai/o11y are; no source is
// vendored. Only its transitively base-free packages are linked — types,
// velocity, anomaly, replay — because pkg/engine, pkg/measure and pkg/history all
// reach github.com/hanzoai/base/core and github.com/hanzoai/tasks through
// pkg/history's Base-backed store, and dragging an application framework and a
// workflow engine into a payment-authorization path is not a trade worth making.
// See rule.go for the measurement and the upstream fix.
package risk
import (
"context"
"database/sql"
"fmt"
"net/http"
"sync"
"time"
"github.com/hanzoai/cloud"
"github.com/hanzoai/cloud/apps/datastore"
"github.com/hanzoai/cloud/openapi"
"github.com/luxfi/aml/pkg/anomaly"
"github.com/zap-proto/zip"
)
// state is everything the surface reads. The per-tenant halves — the model's
// counters, the velocity rings and the SQLite file — are SINGLE-WRITER by
// nature, which is why this app must run under the shard router that pins an org
// to one pod, and why Shutdown must snapshot.
//
// THE IN-MEMORY PLANES ARE NOT HERE, and that is the point. One velocity store
// and one forest on this struct meant every tenant shared them under a GLOBAL
// cap, so one org's volume evicted another org's counters and another org's
// learned model. They live on the tenant's own cell now (store.go), each under
// that tenant's own bound (bound.go).
type state struct {
// brand is the deployment's brand, the half of the tenant key that never
// comes from a header.
brand string
// dataDir is where the per-tenant files live.
dataDir string
// shelf is the bounded registry of live tenants: one cell each, holding that
// tenant's file, its aggregates and its model.
shelf *shelf
// inflight is the per-tenant bound on measurement: one at a time, per tenant,
// so a caller that loops the measurement surface degrades only itself.
inflight *inflight
// running is the per-tenant bound on the exhaustive search: one at a time,
// per tenant, for the same reason and with the same shape.
running *inflight
// digest is the model SHAPE — the feature inventory in order and the
// detector's geometry parameters. It is a pure function of the configuration,
// identical for every tenant, so it is settled once at boot rather than read
// off whichever tenant's forest is at hand.
digest string
// bill is the shared per-org gate and meter, on the "risk" product.
bill *cloud.ResourceMeter
// warehouse records whether the feature tables were created. A false value
// is an honest gap on the dictionary and the backfill, never a zero.
mu sync.Mutex
warehouse bool
}
// Mount wires /v1/risk and the native /v1/ml leaves onto app.
//
// EVERY INHERITED CAPABILITY IS WIRED HERE, EXPLICITLY. Being embedded in cloud
// makes each one AVAILABLE; none of them is automatic:
//
// IAM auth SanitizeIdentity mints X-Org-Id from the verified bearer.
// Global (serve.go) — nothing to do here, and that is the point:
// this app never validates a token and never can.
// tenant gate cloud.Bridge() on EACH group, FIRST, before any leaf. A typed
// op receives only a context; Bridge is what parks the validated
// org and validated-ness in it. Installed after the leaves it
// would never run — fiber orders middleware by registration.
// org sub-scope principal.ValidatedProject, read in tenantOf.
// meter + gate cloud.NewResourceMeter(deps, "risk"); rm.Gate before priced
// work, rm.Meter after it. Wired per op, in typed.go.
// logs cloud.NewBase(deps, "risk") gives the scoped luxlog.
// traces global and already ZAP-native (OTLZ). This package imports no
// otlp transport, deliberately.
// health OwnsHealth on the plugin plus the real probe below.
//
// The route registration follows apps/ml exactly, including the one form
// cmd/zipdoc can read: ONE `g := <router>.Group("/prefix")` per line.
func Mount(app cloud.Router, deps cloud.Deps) error {
if app == nil {
return fmt.Errorf("risk.Mount: nil app")
}
s, err := build(deps)
if err != nil {
return err
}
// The warehouse is not on any request path this app serves, so a warehouse
// that is down at boot must not stop the decision plane from mounting. The
// tables are ensured once, in the background, and the honest gap is recorded.
go warehouse(s)
// Memory comes back from a tenant's OWN silence on a timer, so a quiet tenant
// releases its rings without a busy one having to need them first.
go reclaim(s)
mount(s, app)
s.Log.Info("risk surface mounted",
"brand", deps.Brand, "env", deps.Env,
"billing", s.State.bill.Enabled(), "model", s.State.digest,
"tenants", tenantMax(), "keys_per_tenant", maxKeys())
// Shutdown is registered by the plugin (plugin/risk/main.go) and calls back
// here; holding the service in a package var would be a second owner of the
// state, so the closure captures it instead.
shutdown = func(context.Context) error { return teardown(s) }
return nil
}
// warehouse creates the two feature planes and then keeps the NETWORK BASELINE
// current. It is the only background work this app does and it never touches a
// request path.
//
// The baseline recompute is here, on a timer, rather than on any route — which
// is what makes the cross-org surface unreachable by a caller. Nobody can time
// it, steer it, or observe its cost, and the statement it runs is a package
// constant with no placeholder, so nothing a caller sends reaches it. What it
// writes carries no tenant, no subject and no pseudonym: quantiles over a
// k-anonymous set of contributing orgs, and nothing else.
func warehouse(s *stateService) {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
if err := ensureTables(ctx); err != nil {
cancel()
s.Log.Warn("risk: feature tables unavailable; the dictionary, the peer comparison and the search sandbox will report an honest gap", "err", err)
return
}
cancel()
s.State.mu.Lock()
s.State.warehouse = true
s.State.mu.Unlock()
for {
bg, stop := context.WithTimeout(context.Background(), 10*time.Minute)
if err := publishBaseline(bg); err != nil {
s.Log.Warn("risk: the network baseline was not recomputed; peer comparison holds its last published day", "err", err)
}
stop()
time.Sleep(baselineEvery)
}
}
// baselineEvery is how often the network quantiles are recomputed. Six hours:
// the statement covers whole days, so anything faster republishes the same
// numbers, and anything slower lets a day go unpublished after a restart.
const baselineEvery = 6 * time.Hour
// reclaim retires tenants that have gone silent for longer than their rings hold
// anything. It is the only thing in this process that releases a tenant's
// memory, and it is driven by that tenant's own idleness — never by another
// tenant's arrival, which is what makes it reclaim rather than eviction.
func reclaim(s *stateService) {
for {
time.Sleep(sweepEvery)
s.State.shelf.sweep()
}
}
// sweepEvery is how often the reclaim runs. It is far shorter than the idleness
// it looks for, so the granularity of "when memory comes back" is minutes rather
// than a multiple of the threshold.
const sweepEvery = 10 * time.Minute
// build constructs the state. Separate from Mount because Mount also starts the
// background work and installs the routes, and a test wants the state without
// either — the same split apps/ml makes for the same reason.
func build(deps cloud.Deps) (*stateService, error) {
if deps.Logger == nil {
return nil, fmt.Errorf("risk.Mount: nil deps.Logger")
}
if deps.Brand == "" {
// The brand is half the tenant key. A deployment that did not state one
// cannot mint a key, and every request would 403 — better to say so at
// boot than once per request.
return nil, fmt.Errorf("risk.Mount: no brand, so no tenant key can be minted")
}
// The model SHAPE, off a forest that will never hold a tenant. The digest is
// a pure function of the configuration — the inventory in order and the
// geometry parameters — so it can be settled once at boot, and reading it off
// a throwaway is better than keeping a spare store around that something
// could later be tempted to score with.
//
// SHADOW IS THE DEFAULT AT THE ENGINE TOO (forest forces it), and the
// per-tenant switch in the record plane is what turns a tenant live. Two gates
// in series rather than one: a deployment-wide flag flipped by mistake still
// cannot make a tenant act.
shape, err := forest(anomaly.Config{}, aggregates())
if err != nil {
return nil, fmt.Errorf("risk.Mount: %w", err)
}
base := cloud.NewBase(deps, "risk")
return &cloud.Service[state]{
Base: base,
State: state{
brand: deps.Brand,
dataDir: deps.DataDir,
shelf: newShelf(base),
inflight: newInflight("a measurement"),
running: newInflight("an exhaustive search"),
digest: shape.Digest(),
bill: cloud.NewResourceMeter(deps, "risk"),
},
}, nil
}
// stateService is the concrete service this package builds. An alias, so the
// helpers below read as functions of the service and no second type exists.
type stateService = cloud.Service[state]
// shutdown is set by Mount and read by the plugin. A nil value means the app
// never mounted, and tearing down what never came up is a no-op rather than a
// panic.
var shutdown func(context.Context) error
// Shutdown snapshots every resident tenant's model and closes the record planes.
//
// THIS IS NOT HOUSEKEEPING. cloud deploys with strategy Recreate at one replica:
// every rollout is a hard all-endpoint window AND drops every in-memory model.
// A model that comes back with nothing learned declines to score for the whole
// warm period, and a control that is off for however long that takes is a
// control that was off — reported, if anyone reads Refusal, and silent if not.
func Shutdown(ctx context.Context) error {
if shutdown == nil {
return nil
}
return shutdown(ctx)
}
func teardown(s *stateService) error {
kept, failed := s.State.shelf.close()
s.Log.Info("risk surface down", "models_kept", kept, "models_lost", failed)
if failed > 0 {
return fmt.Errorf("risk: %d tenant model(s) could not be snapshotted", failed)
}
return nil
}
// tenantCell resolves the caller's scope and its live state: its own file, its
// own aggregates and its own model, restored from its own snapshot the first
// time this process arms it. Every typed op starts here, so the tenant is
// established in one place, admitted in one place and armed in one place.
func tenantCell(ctx context.Context, s *stateService) (scope, *cell, error) {
sc, err := tenantOf(ctx, s.State.brand)
if err != nil {
return scope{}, nil, err
}
// The mint's own shape check, asserted at the boundary rather than assumed.
// A regression here is the whole product: the key is the store index, the
// history org column and the model's tree seed at once.
if !qualified(s.State.brand, sc.tenant) {
return scope{}, nil, zip.ErrForbidden("the tenant key is not qualified")
}
c, err := s.State.shelf.of(sc.tenant)
if err != nil {
return scope{}, nil, err
}
return sc, c, nil
}
// tenantState is the file-only door for the ops that touch the record plane and
// not the two in-memory planes.
func tenantState(ctx context.Context, s *stateService) (scope, *sql.DB, error) {
sc, c, err := tenantCell(ctx, s)
if err != nil {
return scope{}, nil, err
}
return sc, c.db, nil
}
// health is the app's real, fail-closed probe.
//
// UNTYPED BY DESIGN. A real probe answers 503 CARRYING THE DEGRADED REPORT AS
// ITS BODY, and a typed op reaches a non-2xx only by returning an error, which
// zip renders as its own envelope — dropping exactly the report the probe
// exists to deliver. Same reason apps/ml keeps its two health routes raw.
func health(s *stateService) func(*zip.Ctx) error {
return func(c *zip.Ctx) error {
s.State.mu.Lock()
warehouse := s.State.warehouse
s.State.mu.Unlock()
tenants, refused, refusedAt := s.State.shelf.count()
report := map[string]any{
"status": "ok",
"model": s.State.digest,
"tenants": tenants,
"capacity": tenantMax(),
// strained COUNTS the tenants whose own aggregates are at their own
// cardinality bound, so a count they read may under-state their own
// traffic. It is on the probe because a partial ring reads exactly like
// a quiet one, and nobody goes looking for a control that went quiet —
// and it is a NUMBER, because this route is unauthenticated by design
// and a tenant roster served to an anonymous GET is a customer list.
// The tenant learns about its OWN ring on its own scoped surface,
// GET /v1/ml/state.
"strained": s.State.shelf.strained(),
"warehouse": warehouse || datastore.Ready(),
"billing": s.State.bill.Enabled(),
}
// The DECISION plane is what this app promises, and it does not need the
// warehouse: rings are in memory, rules and lists are the tenant's own
// file, the model is in process. The probe is degraded only when
// something the hot path actually needs is missing.
if s.State.shelf == nil || s.State.dataDir == "" {
report["status"] = "degraded"
report["error"] = "the decision plane is not constructed"
return c.JSON(http.StatusServiceUnavailable, report)
}
// A REFUSED ADMISSION IS A PAGE, not a log line. This pod turned a tenant
// away rather than evict an incumbent, which is the right answer and also
// an outage for whoever was turned away — so it degrades the probe until
// an operator shards or raises the ceiling.
if refused > 0 && time.Since(refusedAt) < capacityAlarm {
report["status"] = "degraded"
report["refused"] = refused
report["refused_at"] = refusedAt.UTC().Format(time.RFC3339)
report["error"] = "this node is at its tenant ceiling and is refusing new tenants; shard, or raise RISK_TENANTS to what the pod's memory allows"
return c.JSON(http.StatusServiceUnavailable, report)
}
return c.JSON(http.StatusOK, report)
}
}
// capacityAlarm is how long a refusal keeps the probe degraded. Three sweeps:
// long enough that a refusal cannot be missed between two scrapes, short enough
// that a pod which has since reclaimed room reports itself healthy again.
const capacityAlarm = 3 * sweepEvery
// The prose for the ONE route above that is untyped by design. zipdoc lifts
// prose from a typed handler's doc comment, and this is not one — so without a
// Describe the probe would publish an operationId and nothing else. Declared
// beside the wire fact it belongs to.
func init() {
openapi.Describe("/v1/risk/health", http.MethodGet,
"Report whether the decision plane can decide",
"Answers 200 with a report of what is up, or 503 CARRYING THE SAME REPORT as its "+
"body — which is the whole point of a real probe, and the reason this one route is "+
"not a typed op: a typed op reaches a non-2xx only by returning an error, and that "+
"renders as an envelope with the report dropped.\n\n"+
"The report names the model digest in force, how many tenants this process holds "+
"resident, whether the feature warehouse is reachable, and whether metering is "+
"configured. Only the first is load-bearing for a decision: the rings are in "+
"memory, the rules and lists are the tenant's own file, and the model is in "+
"process, so a decision does NOT need the warehouse and the probe stays green "+
"without it. A warehouse that is down costs the field dictionary and the search "+
"sandbox, and is reported as exactly that rather than as a failure of the "+
"authorization path.")
}
+474
View File
@@ -0,0 +1,474 @@
package risk
// rule.go is the risk rule algebra: a CLOSED set of fields and a CLOSED set of
// comparisons, evaluated against one decision's facts.
//
// WHY IT IS NOT luxfi/aml's expr-lang evaluator, stated with the measurement.
// The design this app was built from claimed pkg/engine is base-free. It is not.
// Measured with `go list -deps`:
//
// pkg/engine -> pkg/history -> pkg/store -> github.com/hanzoai/base/core
// -> github.com/hanzoai/tasks (Temporal SDK)
// -> github.com/hanzoai/csqlite, hanzoai/sqlite
//
// A grep for `hanzoai/base` in pkg/engine's own files finds nothing, which is
// how the claim was arrived at; the import is TRANSITIVE, through the measures
// in scope.go. Linking it would pull the whole Base application framework and a
// workflow engine into a binary whose job is to answer a payment-authorization
// call in single-digit milliseconds — and cloud's image carries a SQLITE-GATE
// that fails the BUILD, not the tests, when a per-app binary drags a second
// sqlite driver in under CGO.
//
// So the risk plane brings in only the transitively base-free packages
// (types, velocity, anomaly, replay, standard) and states its rules as data.
// That is the better shape here anyway: a risk rule is a conjunction of
// comparisons over a fixed vocabulary, which is a closed algebra, not a
// language. A closed algebra is injection-safe by construction, is a TYPED
// wire shape (so it reaches the SDKs, the CLI and the MCP tools as a schema
// instead of as an opaque string), and can be replayed without an interpreter.
//
// The right long-term fix is upstream and is named in the handoff: split
// pkg/history's Base-backed store into its own package so pkg/engine is
// base-free in fact as well as in intent.
import (
"fmt"
"sort"
"strconv"
"strings"
)
// Actions, strongest last. `challenge` and `restrict` are risk's own: the
// compliance vocabulary (allow/flag/review/block) has no word for "ask the human
// to prove they are one" or "let the money in but not out", and both are the
// normal answer at signup and at payout.
const (
ActionAllow = "allow"
ActionChallenge = "challenge"
ActionReview = "review"
ActionRestrict = "restrict"
ActionBlock = "block"
)
// actionRank orders actions by how much they demand, so "the strongest of these"
// and "no stronger than this" are one comparison rather than two tables. An
// action nobody defined ranks lowest: it must never outrank one that was.
func actionRank(a string) int {
switch a {
case ActionAllow:
return 1
case ActionChallenge:
return 2
case ActionReview:
return 3
case ActionRestrict:
return 4
case ActionBlock:
return 5
default:
return 0
}
}
// modelCeiling is the strongest action STATISTICAL evidence may reach on its
// own. It is luxfi/aml's types.ActionCeiling and it is not weakened for the
// payment stage: a model can put a transaction in front of a person; it cannot
// decline one, because an unexplainable refusal is not a decision anybody can
// defend to the customer or to a chargeback network.
const modelCeiling = ActionReview
// Stages are the lifecycle moments a decision can be asked about. The stage
// selects the feature window and the rule set; it never selects a different
// tenant gate.
const (
StageSignup = "signup"
StagePayment = "payment"
StageSession = "session"
StageUsage = "usage"
StagePayout = "payout"
StageDispute = "dispute"
)
var stages = map[string]bool{
StageSignup: true, StagePayment: true, StageSession: true,
StageUsage: true, StagePayout: true, StageDispute: true,
}
// Operators, closed.
const (
OpEq = "eq"
OpNe = "ne"
OpGt = "gt"
OpGte = "gte"
OpLt = "lt"
OpLte = "lte"
OpIn = "in"
OpNotIn = "notin"
OpInList = "inlist"
OpNotList = "notinlist"
OpExists = "exists"
OpAbsent = "absent"
OpContains = "contains"
OpPrefix = "prefix"
OpSuffix = "suffix"
)
var operators = map[string]bool{
OpEq: true, OpNe: true, OpGt: true, OpGte: true, OpLt: true, OpLte: true,
OpIn: true, OpNotIn: true, OpInList: true, OpNotList: true,
OpExists: true, OpAbsent: true, OpContains: true, OpPrefix: true, OpSuffix: true,
}
// facts is the CLOSED vocabulary of scalar fields a term may name. Two families
// are open by shape and closed by source: `signal.<name>` reads the caller's own
// signal map (the caller's data, compared in memory, never spelled into SQL),
// and `velocity.<axis>.<window>.<stat>` reads the in-memory rings through an
// allowlisted axis and window.
var facts = map[string]bool{
"stage": true,
"subject.kind": true,
"subject.id": true,
"agency": true,
"amount.nano": true,
"amount.currency": true,
"amount.direction": true,
"model.score": true,
"model.warming": true,
"actor.agent": true,
"actor.session": true,
}
// velocityStats is the closed set of numbers a rule may read off a ring.
var velocityStats = map[string]bool{"count": true, "sum": true, "near": true, "days": true}
// term is one comparison. Value carries strings, Number carries numerics and
// Values carries a set — three fields rather than one `any`, because the wire
// shape is a published schema and `any` publishes as "anything".
type term struct {
// Field is the fact to read. One of the closed vocabulary, or
// `signal.<name>`, or `velocity.<axis>.<window>.<stat>`.
Field string `json:"field"`
// Op is the comparison. One of the closed operator set.
Op string `json:"op"`
// Value is the string operand, for the textual comparisons.
Value string `json:"value,omitempty"`
// Number is the numeric operand, for the ordered comparisons.
Number float64 `json:"number,omitempty"`
// Values is the set operand, for `in` and `notin`.
Values []string `json:"values,omitempty"`
}
// rule is one detection. All terms must hold — a conjunction, deliberately: a
// disjunction is two rules, and two rules are two things a reviewer can judge,
// retire and measure separately. Nothing is lost and the report gets sharper.
type rule struct {
// ID is the rule's stable identifier within the tenant.
ID string `json:"id"`
// Name is what a reviewer reads in an alert.
Name string `json:"name"`
// Stage narrows the rule to one lifecycle moment; empty means every stage.
Stage string `json:"stage,omitempty"`
// Action is what the rule asks for when it holds.
Action string `json:"action"`
// Weight is how much evidence a hit contributes, in [0,1].
Weight float64 `json:"weight"`
// Severity is the reviewer-facing grading: low, medium, high or critical.
Severity string `json:"severity"`
// Enabled governs the live path. A disabled rule still replays, because the
// question a simulation asks is what happens ON activation.
Enabled bool `json:"enabled"`
// All is the conjunction. An empty conjunction is refused at admission: a
// rule that holds on everything is not a detection.
All []term `json:"all"`
}
// admit validates a rule before anything depends on it. Every refusal here is
// one that would otherwise become a rule that fires on everything, on nothing,
// or on a field that does not exist — all three read as a working control.
func admit(r rule) error {
switch {
case strings.TrimSpace(r.Name) == "":
return fmt.Errorf("risk: a rule with no name cannot be read back in an alert")
case r.Stage != "" && !stages[r.Stage]:
return fmt.Errorf("risk: %q is not a lifecycle stage", r.Stage)
case actionRank(r.Action) == 0:
return fmt.Errorf("risk: %q is not an action", r.Action)
case r.Weight < 0 || r.Weight > 1:
return fmt.Errorf("risk: weight %v is outside [0,1]", r.Weight)
case len(r.All) == 0:
return fmt.Errorf("risk: a rule with no terms holds on everything, which is not a detection")
}
if r.Severity != "" && !severities[r.Severity] {
return fmt.Errorf("risk: %q is not a severity", r.Severity)
}
for i, t := range r.All {
if err := admitTerm(t); err != nil {
return fmt.Errorf("risk: term %d: %w", i, err)
}
}
return nil
}
var severities = map[string]bool{"low": true, "medium": true, "high": true, "critical": true}
// admitTerm holds the field and operator vocabularies. It is the ONE gate: a
// field is either in the closed set or matches one of the two structured
// families, and an unknown field is an error rather than a silent miss.
func admitTerm(t term) error {
if !operators[t.Op] {
return fmt.Errorf("%q is not an operator", t.Op)
}
switch {
case facts[t.Field]:
case strings.HasPrefix(t.Field, "signal."):
if strings.TrimPrefix(t.Field, "signal.") == "" {
return fmt.Errorf("signal. names no signal")
}
case strings.HasPrefix(t.Field, "velocity."):
parts := strings.Split(t.Field, ".")
if len(parts) != 4 {
return fmt.Errorf("%q is not velocity.<axis>.<window>.<stat>", t.Field)
}
if !velocityAxes[parts[1]] {
return fmt.Errorf("%q is not a velocity axis", parts[1])
}
if !velocityWindows[parts[2]] {
return fmt.Errorf("%q is not a velocity window", parts[2])
}
if !velocityStats[parts[3]] {
return fmt.Errorf("%q is not a velocity statistic", parts[3])
}
default:
return fmt.Errorf("%q is not a fact this vocabulary carries", t.Field)
}
if (t.Op == OpIn || t.Op == OpNotIn) && len(t.Values) == 0 {
return fmt.Errorf("%s with an empty set holds on nothing", t.Op)
}
if (t.Op == OpInList || t.Op == OpNotList) && strings.TrimSpace(t.Value) == "" {
return fmt.Errorf("%s names no list", t.Op)
}
return nil
}
// velocityAxes and velocityWindows mirror the rings the decide path keeps. They
// are declared here because this is where a caller's spelling is checked against
// them; decide.go records on exactly these.
var velocityAxes = map[string]bool{"account": true, "device": true, "ip": true, "pair": true, "email": true, "bin": true}
var velocityWindows = map[string]bool{"1h": true, "24h": true, "7d": true, "30d": true}
// facts is what a term reads: the decision's own inputs plus everything the
// scoring path has computed by the time rules run.
type factSet struct {
scalar map[string]string
number map[string]float64
lists func(name, value string) bool
}
func (f factSet) str(field string) (string, bool) {
v, ok := f.scalar[field]
return v, ok
}
func (f factSet) num(field string) (float64, bool) {
v, ok := f.number[field]
if ok {
return v, true
}
// A scalar that parses as a number is comparable as one: `signal.bin` is a
// string on the wire and an ordered value in a rule.
if s, ok := f.scalar[field]; ok {
if n, err := strconv.ParseFloat(s, 64); err == nil {
return n, true
}
}
return 0, false
}
// holds evaluates one term. An unreadable field is FALSE, never an error that
// aborts the decision: a rule over a signal this caller did not send has not
// matched, and the alternative — failing the whole decision — turns one
// mis-specified rule into an outage on the authorization path.
func (f factSet) holds(t term) bool {
switch t.Op {
case OpExists:
if _, ok := f.str(t.Field); ok {
return true
}
_, ok := f.num(t.Field)
return ok
case OpAbsent:
if _, ok := f.str(t.Field); ok {
return false
}
_, ok := f.num(t.Field)
return !ok
case OpGt, OpGte, OpLt, OpLte:
v, ok := f.num(t.Field)
if !ok {
return false
}
switch t.Op {
case OpGt:
return v > t.Number
case OpGte:
return v >= t.Number
case OpLt:
return v < t.Number
default:
return v <= t.Number
}
case OpInList, OpNotList:
v, ok := f.str(t.Field)
if !ok || f.lists == nil {
return t.Op == OpNotList
}
in := f.lists(t.Value, v)
return in == (t.Op == OpInList)
}
v, ok := f.str(t.Field)
if !ok {
if n, isNum := f.num(t.Field); isNum {
v, ok = strconv.FormatFloat(n, 'f', -1, 64), true
}
}
if !ok {
return t.Op == OpNe || t.Op == OpNotIn
}
switch t.Op {
case OpEq:
return v == t.Value
case OpNe:
return v != t.Value
case OpContains:
return strings.Contains(v, t.Value)
case OpPrefix:
return strings.HasPrefix(v, t.Value)
case OpSuffix:
return strings.HasSuffix(v, t.Value)
case OpIn:
return contains(t.Values, v)
case OpNotIn:
return !contains(t.Values, v)
}
return false
}
func contains(set []string, v string) bool {
for _, s := range set {
if s == v {
return true
}
}
return false
}
// evaluate runs the enabled rules for a stage and returns the hits in a stable
// order (strongest action first, then by weight, then by id) so two identical
// decisions render identically.
func evaluate(rules []rule, stage string, f factSet) []hit {
var hits []hit
for _, r := range rules {
if !r.Enabled || (r.Stage != "" && r.Stage != stage) {
continue
}
matched := true
for _, t := range r.All {
if !f.holds(t) {
matched = false
break
}
}
if matched {
hits = append(hits, hit{Rule: r.ID, Name: r.Name, Action: r.Action, Weight: r.Weight, Severity: r.Severity})
}
}
sort.SliceStable(hits, func(i, j int) bool {
if a, b := actionRank(hits[i].Action), actionRank(hits[j].Action); a != b {
return a > b
}
if hits[i].Weight != hits[j].Weight {
return hits[i].Weight > hits[j].Weight
}
return hits[i].Rule < hits[j].Rule
})
return hits
}
// combine turns hits into a score and an action.
//
// The score is weight-of-evidence: 1 - prod(1-w). Two independent weak signals
// compound and no single one saturates, which is the property a summed score
// does not have (three 0.4 rules sum past 1 and clamp, losing the fourth).
//
// The action is the strongest any hit asked for, and the model's own hit is
// capped at the ceiling BEFORE it gets here, so no arrangement of weights lets
// statistical evidence decline anything on its own.
func combine(hits []hit) (float64, string) {
remain, action := 1.0, ActionAllow
for _, h := range hits {
w := h.Weight
if w < 0 {
w = 0
}
if w > 1 {
w = 1
}
remain *= 1 - w
if actionRank(h.Action) > actionRank(action) {
action = h.Action
}
}
return 1 - remain, action
}
// starter is the rule set a tenant gets on its first decision. Every one of
// these is a lifecycle-stage detection the product spec names, expressed in the
// vocabulary above so a tenant can read, copy and retire them. They are seeded
// ENABLED but the tenant starts in SHADOW, so nothing acts until an operator
// says so and can see what would have happened.
func starter() []rule {
return []rule{{
ID: "signup-burst-ip", Name: "Many signups from one address", Stage: StageSignup,
Action: ActionChallenge, Weight: 0.4, Severity: "medium", Enabled: true,
All: []term{{Field: "velocity.ip.1h.count", Op: OpGte, Number: 5}},
}, {
ID: "signup-shared-device", Name: "One device onboarding several accounts", Stage: StageSignup,
Action: ActionReview, Weight: 0.5, Severity: "high", Enabled: true,
All: []term{{Field: "velocity.device.7d.count", Op: OpGte, Number: 4}},
}, {
ID: "signup-disposable-email", Name: "Disposable email domain", Stage: StageSignup,
Action: ActionChallenge, Weight: 0.35, Severity: "medium", Enabled: true,
All: []term{{Field: "signal.emaildomain", Op: OpInList, Value: "email-deny"}},
}, {
ID: "payment-card-testing", Name: "Card testing: a burst of small charges", Stage: StagePayment,
Action: ActionBlock, Weight: 0.7, Severity: "critical", Enabled: true,
All: []term{
{Field: "velocity.bin.1h.count", Op: OpGte, Number: 10},
{Field: "amount.nano", Op: OpLte, Number: 5_000_000_000},
},
}, {
ID: "payment-denied-ip", Name: "Payment from a denied address", Stage: StagePayment,
Action: ActionBlock, Weight: 0.9, Severity: "critical", Enabled: true,
All: []term{{Field: "signal.ip", Op: OpInList, Value: "ip-deny"}},
}, {
ID: "usage-spend-spike", Name: "Pay-as-you-go spend far above this account's own shape", Stage: StageUsage,
Action: ActionRestrict, Weight: 0.5, Severity: "high", Enabled: true,
All: []term{{Field: "velocity.account.24h.sum", Op: OpGte, Number: 100_000_000_000_000}},
}, {
ID: "payout-velocity", Name: "Payouts leaving faster than they arrived", Stage: StagePayout,
Action: ActionRestrict, Weight: 0.6, Severity: "high", Enabled: true,
All: []term{
{Field: "amount.direction", Op: OpEq, Value: "out"},
{Field: "velocity.account.24h.count", Op: OpGte, Number: 5},
},
}, {
ID: "bot-anonymous-burst", Name: "Undeclared automation at machine cadence", Stage: StageSession,
Action: ActionChallenge, Weight: 0.45, Severity: "medium", Enabled: true,
All: []term{
{Field: "agency", Op: OpEq, Value: AgencyBot},
{Field: "velocity.ip.1h.count", Op: OpGte, Number: 60},
},
}}
}
+586
View File
@@ -0,0 +1,586 @@
package risk
// skew_test.go proves the trainingserving skew control actually controls
// something, and that the record behind a decision is one record.
//
// Every test here was written against the defect it names: the fix was reverted,
// the test was run, and it went red. A control whose test passes with the control
// removed is a comment.
import (
"context"
"encoding/json"
"fmt"
"net/http"
"testing"
"time"
)
// A REFIT CANNOT RE-BLESS STALE COORDINATES.
//
// The shape gate refuses a map fitted under coordinates that have moved, and the
// refusal's own remedy is "fit again". If a refit reads the same rows — every one
// of them scored under the OLD shape — and stamps the NEW shape on them, then the
// gate is a door that opens from the inside: one POST clears the control with no
// new evidence at all, and the plane resumes reporting probabilities derived from
// a coordinate system that no longer exists.
//
// A fit reads ONE coordinate system. After a governed change the evidence under
// the new shape is empty, so the fit is refused and says how much evidence it had
// to leave behind and why.
func TestARefitCannotReblessCoordinatesThatMoved(t *testing.T) {
app, s := wireApp(t)
tn := Tenant("hanzo/acme")
seedJudged(t, s, tn, 200)
code, body := req(t, app, http.MethodPost, "/v1/ml/calibrate", "acme", "u_acme", `{"horizon":0}`)
if code != http.StatusCreated {
t.Fatalf("calibrate = %d %s", code, body)
}
var first mlCalibrationView
if err := json.Unmarshal(body, &first); err != nil {
t.Fatal(err)
}
if !first.Current || first.Rows != 200 {
t.Fatalf("the first fit is not the premise this test needs: current=%v rows=%d", first.Current, first.Rows)
}
// A governed change. Every one of the 200 rows was scored under the old shape.
code, body = req(t, app, http.MethodPost, "/v1/risk/rules", "acme", "u_acme",
`{"rule":{"id":"skew-new","name":"new","stage":"payment","action":"review",
"weight":0.5,"severity":"high","enabled":true,
"all":[{"field":"amount.nano","op":"gte","number":1}]}}`)
if code != http.StatusOK && code != http.StatusCreated {
t.Fatalf("rule = %d %s", code, body)
}
// The documented remedy, run against evidence that predates the change.
code, body = req(t, app, http.MethodPost, "/v1/ml/calibrate", "acme", "u_acme", `{"horizon":0}`)
if code != http.StatusCreated {
t.Fatalf("refit = %d %s", code, body)
}
var second mlCalibrationView
if err := json.Unmarshal(body, &second); err != nil {
t.Fatal(err)
}
if second.Fitted {
t.Errorf("the refit produced a map (shape %.12s, %d rows) out of history scored entirely under "+
"shape %.12s. The skew control was cleared by one POST with no new observation.",
second.Shape, second.Rows, first.Shape)
}
if second.Refusal == "" {
t.Error("the refit refused and did not say why")
}
if second.Superseded != 200 {
t.Errorf("superseded = %d, want 200 — the answer does not say how much evidence the shape "+
"boundary put out of reach, so an operator cannot tell a thin tenant from a moved one",
second.Superseded)
}
// And the map in force still refuses, because nothing about the world changed.
code, body = req(t, app, http.MethodGet, "/v1/ml/calibration", "acme", "u_acme", "")
if code != http.StatusOK {
t.Fatalf("calibration = %d %s", code, body)
}
var now mlCalibrationView
if err := json.Unmarshal(body, &now); err != nil {
t.Fatal(err)
}
if now.Current {
t.Error("the map in force reports current after a shape change it was not fitted under")
}
}
// EVERY OP READS THE MAP THROUGH THE SHAPE IN FORCE.
//
// The gate used to be Map.P(score, shape), and every internal caller passed the
// map's OWN shape — so the comparison was between a value and itself and the
// control was inert everywhere except the decide path. The consequence was not
// subtle: with the live plane refusing to state any probability, /v1/ml/evaluate
// still reported a Brier, /v1/ml/replay still moved rows onto rungs nothing could
// reach, and the reliability chart drew the map the same response refused to use.
//
// Binding at the boundary makes all three answer the same way the decide path
// does. This test moves the shape and then asks all three.
func TestMeasurementRefusesTheMapTheDecidePathRefuses(t *testing.T) {
app, s := wireApp(t)
tn := Tenant("hanzo/acme")
seedJudged(t, s, tn, 200)
if code, body := req(t, app, http.MethodPost, "/v1/ml/calibrate", "acme", "u_acme", `{"horizon":0}`); code != http.StatusCreated {
t.Fatalf("calibrate = %d %s", code, body)
}
if code, body := req(t, app, http.MethodPut, "/v1/risk/policy", "acme", "u_acme",
`{"stage":"payment","floor":"allow","reason":"under test","bands":[{"at":0.2,"action":"review"},{"at":0.6,"action":"block"}]}`); code != http.StatusOK {
t.Fatalf("policy = %d %s", code, body)
}
if code, body := req(t, app, http.MethodPost, "/v1/risk/rules", "acme", "u_acme",
`{"rule":{"id":"skew-two","name":"new","stage":"payment","action":"review",
"weight":0.5,"severity":"high","enabled":true,
"all":[{"field":"amount.nano","op":"gte","number":1}]}}`); code != http.StatusOK && code != http.StatusCreated {
t.Fatalf("rule = %d %s", code, body)
}
code, body := req(t, app, http.MethodGet, "/v1/ml/calibration", "acme", "u_acme", "")
if code != http.StatusOK {
t.Fatalf("calibration = %d %s", code, body)
}
var view mlCalibrationView
if err := json.Unmarshal(body, &view); err != nil {
t.Fatal(err)
}
if view.Current {
t.Fatal("the shape did not move; the test proves nothing")
}
if len(view.Reliability) > 0 {
t.Errorf("the response that says the map refuses to answer shipped %d reliability bins, "+
"computed by applying that very map", len(view.Reliability))
}
code, body = req(t, app, http.MethodPost, "/v1/ml/evaluate", "acme", "u_acme", `{"stage":"payment","horizon":0}`)
if code != http.StatusOK {
t.Fatalf("evaluate = %d %s", code, body)
}
var meas mlMeasurement
if err := json.Unmarshal(body, &meas); err != nil {
t.Fatal(err)
}
if meas.Metrics.Brier != nil {
t.Errorf("evaluate reported Brier=%.6f under a calibration the decide path refuses", *meas.Metrics.Brier)
}
if meas.Calibrated {
t.Error("evaluate says it was calibrated while the map in force refuses to answer")
}
if meas.Refusal == "" {
t.Error("evaluate measured without a probability and did not say so")
}
code, body = req(t, app, http.MethodPost, "/v1/ml/replay", "acme", "u_acme",
`{"stage":"payment","horizon":0,"bands":[{"at":0.2,"action":"review"},{"at":0.6,"action":"block"}]}`)
if code != http.StatusCreated {
t.Fatalf("replay = %d %s", code, body)
}
var rep mlReplayReport
if err := json.Unmarshal(body, &rep); err != nil {
t.Fatal(err)
}
acted := 0
for _, w := range rep.Would {
if w.Action != ActionAllow {
acted += w.Count
}
}
if acted > 0 {
t.Errorf("the replay says the candidate would act on %d decisions. Under the shape in force "+
"there is no probability, so no rung is reachable and the candidate acts on none — and "+
"this report was written down as the evidence for a threshold change.", acted)
}
if rep.Refusal == "" {
t.Error("the replay reached no rung and did not say why")
}
}
// THE BOUNDED READ TAKES THE MOST RECENT DECISIONS.
//
// Ascending order plus LIMIT takes the OLDEST rows. Past the bound that freezes
// the calibration, the evaluation, the learning curve and every replay on the
// tenant's first maxHistory decisions forever, with nothing on any response
// saying the read was cut.
func TestTheBoundedReadTakesTheNewestDecisionsAndSaysWhenItCut(t *testing.T) {
_, s := wireApp(t)
tn := Tenant("hanzo/acme")
db, err := s.State.shelf.open(tn)
if err != nil {
t.Fatal(err)
}
shape, err := scoringShape(db, s.State.digest)
if err != nil {
t.Fatal(err)
}
base := time.Now().UTC().Add(-1000 * time.Hour)
const n = 300
for i := range n {
id := fmt.Sprintf("dec_w_%05d", i)
o := observation{
id: id, at: base.Add(time.Duration(i) * time.Minute),
stage: StagePayment, kind: "transaction", subject: "tx", agency: AgencyUnknown,
amount: 1_000_000_000, currency: "USD", direction: "in", signals: map[string]string{},
}
out := outcome{id: id, action: ActionAllow, score: float64(i) / n, agency: o.agency}
if err := putDecision(db, o, out, "d", shape, "", verdict{}); err != nil {
t.Fatal(err)
}
if err := label(db, id, "legitimate", "u"); err != nil {
t.Fatal(err)
}
}
// The bound is maxHistory in production; the statement is identical and only
// the number differs, so a smaller one exercises the same path in a test.
w, err := recorded(context.Background(), db, 0, shape, 100)
if err != nil {
t.Fatal(err)
}
if len(w.history) != 100 {
t.Fatalf("read %d rows, want 100", len(w.history))
}
first, last := w.history[0].ID, w.history[len(w.history)-1].ID
if last != fmt.Sprintf("dec_w_%05d", n-1) {
t.Errorf("the bounded read returned %s..%s — the OLDEST rows, not the most recent. Past "+
"maxHistory=%d decisions every measurement is pinned to the dawn of the log.", first, last, maxHistory)
}
if first != fmt.Sprintf("dec_w_%05d", n-100) {
t.Errorf("the window starts at %s, want %s: the rows must be the last 100 IN TIME ORDER",
first, fmt.Sprintf("dec_w_%05d", n-100))
}
if !w.truncated {
t.Error("the read was cut and no field says so, so a report over a truncated window reads as complete")
}
// And an unbounded read reports no truncation.
all, err := recorded(context.Background(), db, 0, shape, n)
if err != nil {
t.Fatal(err)
}
if all.truncated {
t.Error("a read that saw everything reported itself truncated")
}
}
// A MUTE MOVES THE SCORE DISTRIBUTION, SO IT MOVES THE SHAPE.
//
// combine() sums only the hits that were not suppressed, so muting a rule for
// every subject moves every score that rule touched exactly as retiring the rule
// would. Muting is the day-to-day tuning knob and retiring is the rare act, so a
// control blind to the mute is blind to the common change.
//
// A mute that names a SUBJECT is a statement about that subject — operational
// data, like a list entry — and must not invalidate the tenant's calibration.
func TestARuleWideMuteMovesTheShapeAndASubjectMuteDoesNot(t *testing.T) {
app, s := wireApp(t)
db, err := s.State.shelf.open(Tenant("hanzo/acme"))
if err != nil {
t.Fatal(err)
}
before, err := scoringShape(db, s.State.digest)
if err != nil {
t.Fatal(err)
}
code, body := req(t, app, http.MethodPost, "/v1/risk/suppressions", "acme", "u_acme",
`{"rule":"payment-card-testing","reason":"too noisy this week"}`)
if code != http.StatusOK && code != http.StatusCreated {
t.Fatalf("suppress = %d %s", code, body)
}
muted, err := scoringShape(db, s.State.digest)
if err != nil {
t.Fatal(err)
}
if muted == before {
t.Errorf("muting a weight-bearing rule for every subject left the shape at %.12s. Every score "+
"that rule touched has moved and the calibration still reports itself current.", muted)
}
code, body = req(t, app, http.MethodPost, "/v1/risk/suppressions", "acme", "u_acme",
`{"rule":"payment-velocity-burst","subject":"tx-one-merchant","reason":"known good"}`)
if code != http.StatusOK && code != http.StatusCreated {
t.Fatalf("suppress subject = %d %s", code, body)
}
after, err := scoringShape(db, s.State.digest)
if err != nil {
t.Fatal(err)
}
if after != muted {
t.Errorf("excusing ONE subject moved the shape from %.12s to %.12s. Operational data about "+
"one row would expire the tenant's calibration, and it would never have one.", muted, after)
}
}
// THE LADDER CANNOT DECLINE ON THE MODEL'S EVIDENCE ALONE.
//
// decide caps the model's own hit at modelCeiling because an unexplainable
// refusal is not a decision anybody can defend to the customer or to a chargeback
// network. The policy ladder reads the calibrated probability, and that
// probability is a pure function of the SAME score — so an uncapped escalation
// voids the cap by arithmetic, and the invariant holds only until somebody sets a
// band.
func TestTheLadderCannotDeclineOnTheModelAlone(t *testing.T) {
modelOnly := outcome{hits: []hit{{Rule: modelRuleID, Action: ActionReview, Weight: 0.9}}}
nothing := outcome{}
muted := outcome{hits: []hit{{Rule: "a-rule", Action: ActionBlock, Weight: 0.7, Suppressed: true}}}
explained := outcome{hits: []hit{{Rule: "a-rule", Action: ActionBlock, Weight: 0.7}}}
for _, c := range []struct {
name string
out outcome
evidence, policy string
want string
}{
{"model alone cannot block", modelOnly, ActionReview, ActionBlock, ActionReview},
{"model alone cannot restrict", modelOnly, ActionAllow, ActionRestrict, ActionReview},
{"model alone can still review", modelOnly, ActionAllow, ActionReview, ActionReview},
{"no evidence at all cannot block", nothing, ActionAllow, ActionBlock, ActionReview},
{"a muted hit is not evidence", muted, ActionAllow, ActionBlock, ActionReview},
{"a rule the org wrote can block", explained, ActionAllow, ActionBlock, ActionBlock},
} {
t.Run(c.name, func(t *testing.T) {
if got := escalate(c.evidence, c.policy, c.out); got != c.want {
t.Errorf("escalate(%q, %q) = %q, want %q — %s",
c.evidence, c.policy, got, c.want,
"a decline the model alone is behind has no reason a person can read")
}
})
}
}
// THE DECISION AND ITS GRADING ARE ONE RECORD.
//
// Two independent statements can land one and not the other, and the half that
// lands is the half that acts: a BLOCK on the dispute-packet surface with no
// probability, no principal reason and no refusal saying why. This proves both
// halves — that a failed grading takes the decision down with it, and that a
// decision found without one says so instead of reading as ungraded.
func TestTheDecisionAndItsGradingAreOneRecord(t *testing.T) {
app, s := wireApp(t)
db, err := s.State.shelf.open(Tenant("hanzo/acme"))
if err != nil {
t.Fatal(err)
}
o := observation{
id: "dec_atomic", at: time.Now().UTC(), stage: StagePayment, kind: "transaction",
subject: "tx", agency: AgencyUnknown, amount: 1_000_000_000, currency: "USD",
direction: "in", signals: map[string]string{},
}
out := outcome{id: o.id, action: ActionBlock, score: 0.97, agency: o.agency}
// Make the SECOND write fail. Whatever the cause in production — a disk, a
// lock, a crash between two Execs — the property under test is that the first
// write does not survive it.
if _, err := db.Exec(`ALTER TABLE verdict RENAME TO verdict_away`); err != nil {
t.Fatal(err)
}
if err := putDecision(db, o, out, "d", "shape", "", verdict{}); err == nil {
t.Fatal("the grading could not be written and the write reported success")
}
if _, err := db.Exec(`ALTER TABLE verdict_away RENAME TO verdict`); err != nil {
t.Fatal(err)
}
var n int
if err := db.QueryRow(`SELECT COUNT(*) FROM decision WHERE id = ?`, o.id).Scan(&n); err != nil {
t.Fatal(err)
}
if n != 0 {
t.Errorf("the decision survived a grading that did not: a BLOCK exists with no probability, "+
"no principal reason and nothing this plane can defend it with (%d row(s))", n)
}
// The read path must never present an ungraded decision as a graded one with
// nothing to say.
orphan := o
orphan.id = "dec_orphan"
if _, err := db.Exec(`INSERT INTO decision (id, at, stage, kind, subject, action, score, agency, shadow)
VALUES (?,?,?,?,?,?,?,?,0)`,
orphan.id, stamp(orphan.at), orphan.stage, orphan.kind, orphan.subject,
ActionBlock, 0.97, AgencyUnknown); err != nil {
t.Fatal(err)
}
code, body := req(t, app, http.MethodGet, "/v1/risk/decisions/dec_orphan", "acme", "u_acme", "")
if code != http.StatusOK {
t.Fatalf("detail = %d %s", code, body)
}
var packet riskDecisionView
if err := json.Unmarshal(body, &packet); err != nil {
t.Fatal(err)
}
if len(packet.Reasons) == 0 && packet.Refusal == "" {
t.Error("a BLOCK is served on the dispute-packet surface with no reasons, no probability and " +
"no refusal, so an ungraded decision is indistinguishable from one with nothing to say")
}
}
// A GOVERNANCE RECORD NAMES THE PERSON.
//
// "The org changed the thresholds" is not an answer to who changed them, and the
// helper that resolves the validated principal already exists and is already used
// by the suppression and control records in this same package.
func TestGovernanceRecordsNameThePersonAndNotTheOrg(t *testing.T) {
app, s := wireApp(t)
tn := Tenant("hanzo/acme")
seedJudged(t, s, tn, 200)
db, err := s.State.shelf.open(tn)
if err != nil {
t.Fatal(err)
}
if code, body := req(t, app, http.MethodPut, "/v1/risk/policy", "acme", "u_acme",
`{"stage":"payment","floor":"allow","reason":"under test","bands":[{"at":0.6,"action":"review"}]}`); code != http.StatusOK {
t.Fatalf("policy = %d %s", code, body)
}
if code, body := req(t, app, http.MethodPost, "/v1/ml/calibrate", "acme", "u_acme", `{"horizon":0}`); code != http.StatusCreated {
t.Fatalf("calibrate = %d %s", code, body)
}
if code, body := req(t, app, http.MethodPost, "/v1/ml/replay", "acme", "u_acme",
`{"stage":"payment","horizon":0}`); code != http.StatusCreated {
t.Fatalf("replay = %d %s", code, body)
}
for _, q := range []struct{ what, stmt string }{
{"policy", `SELECT by FROM policy ORDER BY version DESC LIMIT 1`},
{"calibration", `SELECT by FROM calibration ORDER BY version DESC LIMIT 1`},
{"replay", `SELECT by FROM replay ORDER BY at DESC LIMIT 1`},
} {
var who string
if err := db.QueryRow(q.stmt).Scan(&who); err != nil {
t.Fatalf("%s: %v", q.what, err)
}
if who != "u_acme" {
t.Errorf("the %s record names %q as its author; the validated principal is %q",
q.what, who, "u_acme")
}
}
}
// MEASUREMENT IS BOUNDED PER TENANT, AND ONLY PER TENANT.
//
// These ops read the tenant's single-writer file, so a second concurrent
// measurement for one tenant queues behind the first holding the connection that
// tenant's own decisions need. The bound is per tenant by construction: there is
// no fleet-wide number, so a caller that loops this surface degrades itself and
// nobody else.
func TestMeasurementIsBoundedPerTenantAndOnlyPerTenant(t *testing.T) {
f := newInflight("a measurement")
release, err := f.claim(Tenant("hanzo/acme"))
if err != nil {
t.Fatal(err)
}
if _, err := f.claim(Tenant("hanzo/acme")); err == nil {
t.Error("a tenant took two measurement slots at once, so it can hold every connection its " +
"own decide path needs")
}
// A NEIGHBOUR IS UNAFFECTED. This is the property a fleet-wide cap does not
// have, and the whole reason the bound is keyed on the tenant.
other, err := f.claim(Tenant("hanzo/other"))
if err != nil {
t.Fatalf("one tenant's measurement refused another tenant's: %v", err)
}
other()
release()
again, err := f.claim(Tenant("hanzo/acme"))
if err != nil {
t.Fatalf("the slot was not released: %v", err)
}
again()
// Release is idempotent: an op that returns through two paths must not free a
// slot a later request already took.
release()
if _, err := f.claim(Tenant("hanzo/acme")); err != nil {
t.Fatalf("a double release corrupted the bound: %v", err)
}
}
// A MEASUREMENT IS PRICED BY THE WORK IT DOES.
//
// A flat price over a scan whose cost is linear in the rows it reads is a lie
// about the cheap call: a fit over forty judged decisions and a twenty-step
// learning curve over fifty thousand cost the same, and the second is the one a
// caller loops.
func TestAMeasurementIsPricedByTheRowsItReads(t *testing.T) {
small, large := measureCents(40), measureCents(maxHistory)
if small != qualityCents {
t.Errorf("a read of 40 rows costs %d, want the floor %d", small, qualityCents)
}
if large <= small {
t.Errorf("a read of %d rows costs %d and a read of 40 costs %d — the price does not move "+
"with the work", maxHistory, large, small)
}
if measureCents(-1) != qualityCents {
t.Error("a negative row count priced below the floor")
}
// And the gate is taken on the CEILING a request could reach, so the ledger is
// debited before the work rather than after it.
if bounded(0) != maxHistory || bounded(999999) != maxHistory || bounded(10) != 10 {
t.Errorf("bounded(0)=%d bounded(999999)=%d bounded(10)=%d", bounded(0), bounded(999999), bounded(10))
}
}
// A SCORE MEANS A PROBABILITY ON THE SCORES THIS PLANE ACTUALLY PRODUCES.
//
// The whole track is sold on that sentence, and the score distribution it has to
// hold for is not a continuum. combine() returns 1 - prod(1-weight) over a FIXED
// set of rule weights and decide rounds it to four places, so thousands of
// decisions land on a handful of atoms and the modal atom is exactly zero. An
// isotonic fit that does not pool ties collapses on exactly that input: every
// plateau holding a positive is reported as CERTAINTY, at the top of the range,
// where declines happen and where the number lands on an adverse-action record.
//
// End to end through the op, because the unit test in the engine cannot see the
// alphabet cloud feeds it.
func TestAScoreMeansAProbabilityOnThePlateausThisPlaneProduces(t *testing.T) {
app, s := wireApp(t)
tn := Tenant("hanzo/acme")
db, err := s.State.shelf.open(tn)
if err != nil {
t.Fatal(err)
}
shape, err := scoringShape(db, s.State.digest)
if err != nil {
t.Fatal(err)
}
atoms := []struct {
score float64
clean, fraud int
}{
{0.00, 900, 2},
{0.35, 190, 10},
{0.50, 85, 15},
{0.70, 45, 55},
{0.90, 10, 40},
}
base := time.Now().UTC().Add(-500 * time.Hour)
n := 0
for _, a := range atoms {
for i := range a.clean + a.fraud {
judgement := "legitimate"
if i >= a.clean {
judgement = "fraud"
}
id := fmt.Sprintf("dec_atom_%05d", n)
o := observation{
id: id, at: base.Add(time.Duration(n) * time.Minute),
stage: StagePayment, kind: "transaction", subject: fmt.Sprintf("tx-%d", n),
agency: AgencyUnknown, amount: 1_000_000_000, currency: "USD", direction: "in",
signals: map[string]string{},
}
out := outcome{id: id, action: ActionAllow, score: a.score, agency: o.agency}
if err := putDecision(db, o, out, "seed-digest", shape, "", verdict{}); err != nil {
t.Fatal(err)
}
if err := label(db, id, judgement, "u_seed"); err != nil {
t.Fatal(err)
}
n++
}
}
code, body := req(t, app, http.MethodPost, "/v1/ml/calibrate", "acme", "u_acme", `{"horizon":0}`)
if code != http.StatusCreated {
t.Fatalf("calibrate = %d %s", code, body)
}
cal, _, _, ok, err := currentCalibration(db)
if err != nil || !ok {
t.Fatalf("read back: %v ok=%v", err, ok)
}
read, err := cal.Under(shape)
if err != nil {
t.Fatal(err)
}
for _, a := range atoms {
want := float64(a.fraud) / float64(a.clean+a.fraud)
got := read.P(a.score)
if d := got - want; d > 0.05 || d < -0.05 {
t.Errorf("score %.2f reports probability %.6f; %d of %d at that score turned out productive, "+
"so the truth is %.4f (off by %+.4f)", a.score, got, a.fraud, a.clean+a.fraud, want, d)
}
}
if p := read.P(0.90); p >= 1 {
t.Errorf("the top plateau reports certainty (%.6f). That number goes on an adverse-action record.", p)
}
}
+1336
View File
File diff suppressed because it is too large Load Diff
+208
View File
@@ -0,0 +1,208 @@
package risk
// surface_test.go proves the FIVE surfaces one typed declaration is supposed to
// produce actually carry these ops — read off the COMMITTED artifacts, not off
// the router the other tests mount.
//
// The distinction matters and is the exact failure that lost plugin/ingress
// eight published paths: a router can serve a route while the generated subset,
// the woven fleet document and therefore every SDK know nothing about it. Two
// derived things agreeing with each other is not evidence. So this reads
// plugin/risk/openapi.json (the SDK's source), plugin/risk/mcp.json (the tool
// list), the fleet openapi.yaml (what the SDK repos pull), and derives the CLI
// from the spec with the same function the CLI itself uses.
import (
"encoding/json"
"os"
"path/filepath"
"sort"
"strings"
"testing"
"github.com/zap-proto/zip"
)
// theOps are the operationIds this app promises. One list, checked against four
// artifacts, so a rename shows up as four failures rather than as a silently
// missing SDK method.
var theOps = []string{
"riskDecide", "riskDecisions", "riskDecisionDetail", "riskLabel",
"riskSubjectState", "riskActivity", "riskSimulate",
"riskRules", "riskCreateRule", "riskUpdateRule", "riskDeleteRule",
"riskLists", "riskCreateList", "riskAddListEntries", "riskRemoveListEntry",
"riskSuppressions", "riskSuppress", "riskUnsuppress",
"riskControls", "riskSetControl", "riskReleaseControl",
"riskDictionary", "riskMode", "riskSetMode",
"mlScore", "mlTrain", "mlState", "mlSetAppetite", "mlFeatures",
"mlSearch", "mlSearchResult", "mlSnapshot", "mlRestore",
"riskReasons", "riskPolicy", "riskSetPolicy", "riskPolicyVersions",
"mlCalibrate", "mlCalibration", "mlEvaluate", "mlLearning",
"mlReplay", "mlReplayReport",
}
func repoRoot(t *testing.T) string {
t.Helper()
wd, err := os.Getwd()
if err != nil {
t.Fatalf("getwd: %v", err)
}
return filepath.Dir(filepath.Dir(wd)) // apps/risk -> repo root
}
func readSubset(t *testing.T) []byte {
t.Helper()
b, err := os.ReadFile(filepath.Join(repoRoot(t), "plugin", "risk", "openapi.json"))
if err != nil {
t.Fatalf("plugin/risk/openapi.json: %v\nRun: make -C apps/risk describe", err)
}
return b
}
// TestSubsetCarriesEveryOp reads the app's OWN published document — the one the
// fleet weave consumes and every generated SDK ultimately comes from.
func TestSubsetCarriesEveryOp(t *testing.T) {
var doc struct {
Paths map[string]map[string]struct {
OperationID string `json:"operationId"`
Summary string `json:"summary"`
Description string `json:"description"`
} `json:"paths"`
}
if err := json.Unmarshal(readSubset(t), &doc); err != nil {
t.Fatalf("parse subset: %v", err)
}
got := map[string]string{}
for _, methods := range doc.Paths {
for method, op := range methods {
if op.OperationID == "" {
continue
}
got[op.OperationID] = method
if strings.TrimSpace(op.Summary) == "" {
t.Errorf("%s publishes no summary — the CLI derives its one-line help from it", op.OperationID)
}
if strings.TrimSpace(op.Description) == "" {
t.Errorf("%s publishes no description — an MCP client reads it to choose the tool", op.OperationID)
}
}
}
for _, id := range theOps {
if _, ok := got[id]; !ok {
t.Errorf("%s is missing from plugin/risk/openapi.json — no SDK, no CLI command, no MCP tool", id)
}
}
}
// TestFleetDocumentCarriesEveryPath reads the WOVEN document. The subset can be
// right while the weave is stale, and the weave is what the SDK repos pull.
func TestFleetDocumentCarriesEveryPath(t *testing.T) {
b, err := os.ReadFile(filepath.Join(repoRoot(t), "openapi.yaml"))
if err != nil {
t.Fatalf("openapi.yaml: %v", err)
}
body := string(b)
for _, p := range []string{
"/v1/risk/decide", "/v1/risk/decisions", "/v1/risk/decisions/{id}",
"/v1/risk/decisions/{id}/label", "/v1/risk/subjects/{kind}/{id}",
"/v1/risk/activity", "/v1/risk/simulate", "/v1/risk/rules",
"/v1/risk/rules/{id}", "/v1/risk/lists", "/v1/risk/lists/{name}/entries",
"/v1/risk/lists/{name}/entries/{value}", "/v1/risk/suppressions",
"/v1/risk/suppressions/{id}", "/v1/risk/controls", "/v1/risk/controls/{id}",
"/v1/risk/dictionary", "/v1/risk/mode", "/v1/risk/health",
"/v1/ml/score", "/v1/ml/train", "/v1/ml/state", "/v1/ml/state/appetite",
"/v1/ml/features", "/v1/ml/search", "/v1/ml/search/{id}",
"/v1/ml/snapshot", "/v1/ml/restore",
} {
if !strings.Contains(body, "\n "+p+":") {
t.Errorf("%s is absent from the woven openapi.yaml — the SDK repos pull this file, so no "+
"generated client can reach it. Run: make describe", p)
}
}
// The ml app's own paths must still be there: risk claims LEAVES under
// /v1/ml, it does not take the stem.
for _, p := range []string{"/v1/ml/models", "/v1/ml/health", "/v1/train/jobs"} {
if !strings.Contains(body, "\n "+p+":") {
t.Errorf("%s disappeared from the fleet document — the risk row swallowed an ml route", p)
}
}
}
// TestCLIDerivesEveryCommand runs the SAME derivation the CLI runs. A published
// operation with no derivable command is an operation no `hanzo` invocation can
// reach.
func TestCLIDerivesEveryCommand(t *testing.T) {
cmds, err := zip.CommandsFromSpec(readSubset(t))
if err != nil {
t.Fatalf("CommandsFromSpec: %v", err)
}
byID := map[string]zip.Command{}
for _, c := range cmds {
byID[c.OperationID] = c
}
for _, id := range theOps {
c, ok := byID[id]
if !ok {
t.Errorf("%s derives no CLI command", id)
continue
}
if c.Name == "" || c.Service == "" {
t.Errorf("%s derives a nameless command (%q %q)", id, c.Service, c.Name)
}
if strings.TrimSpace(c.Summary) == "" {
t.Errorf("%s derives a command with no help text", id)
}
}
// Path parameters must reach the command as ARGS, or the command cannot
// address the record it names.
if c := byID["riskDecisionDetail"]; len(c.Args) == 0 {
t.Error("riskDecisionDetail derives no positional argument for {id}")
}
if c := byID["riskSubjectState"]; len(c.Args) < 2 {
t.Errorf("riskSubjectState derives %d args, want 2 for {kind} and {id}", len(c.Args))
}
}
// TestMCPToolsCarryEveryTypedOp reads the published tool list. Every typed op
// becomes a tool whose inputSchema is the In type's schema and whose call runs
// the same handler — so a missing tool is a capability an agent cannot use.
func TestMCPToolsCarryEveryTypedOp(t *testing.T) {
b, err := os.ReadFile(filepath.Join(repoRoot(t), "plugin", "risk", "mcp.json"))
if err != nil {
t.Fatalf("plugin/risk/mcp.json: %v\nRun: make -C apps/risk describe", err)
}
var tools []struct {
Name string `json:"name"`
Description string `json:"description"`
InputSchema map[string]any `json:"inputSchema"`
}
if err := json.Unmarshal(b, &tools); err != nil {
t.Fatalf("parse mcp.json: %v", err)
}
got := map[string]bool{}
for _, tool := range tools {
got[tool.Name] = true
if strings.TrimSpace(tool.Description) == "" {
t.Errorf("MCP tool %s has no description — a model cannot choose it", tool.Name)
}
if tool.InputSchema == nil {
t.Errorf("MCP tool %s has no input schema", tool.Name)
}
}
for _, id := range theOps {
if !got[id] {
t.Errorf("%s is not an MCP tool", id)
}
}
if len(tools) != len(theOps) {
var extra []string
for _, tool := range tools {
if !containsString(theOps, tool.Name) {
extra = append(extra, tool.Name)
}
}
sort.Strings(extra)
t.Errorf("%d MCP tools published, %d promised; unlisted: %s",
len(tools), len(theOps), strings.Join(extra, ", "))
}
}
+147
View File
@@ -0,0 +1,147 @@
package risk
// tenant.go is the ONE place a tenant key is minted, and the only type the rest
// of this package can use to reach anything a tenant owns.
//
// THE KEY IS `<brand>/<org>`, NOT `<org>`. An org name is unique within an
// issuer and not across issuers: `acme` on hanzo.id and `acme` on zoo.ngo are two
// unrelated businesses. Keyed on the bare org they become one — one set of
// decisions, one set of rules, one model, and one derived-key salt, so one
// brand's subjects would collide with the other's. The engine states this at
// luxfi/aml pkg/api/tenant.go (qualify) and every OrgID in pkg/types carries the
// qualified form; cloud must mint the same shape or the core is being handed a
// key it documents as invalid.
//
// The brand half NEVER comes from a header. It comes from deps.Brand, which the
// binary is started with — for the same reason the engine takes it from the
// request Host and not from X-Forwarded-Host: a caller that can choose its brand
// can choose which tenant space its org lands in, which is the collision
// qualification exists to prevent, arrived at from the other side.
//
// Tenant has no exported constructor and its underlying type is unexported to
// the wire: it cannot be decoded from a request body, so no In struct can carry
// one. The only way to obtain one is tenantOf(ctx), which reads the VALIDATED
// principal cloud.Bridge parked. That makes "this read is tenant-scoped" a fact
// the compiler checks rather than a convention a reviewer checks.
import (
"context"
"strings"
"github.com/hanzoai/cloud"
"github.com/hanzoai/cloud/apps/principal"
"github.com/zap-proto/zip"
)
// sep separates the brand half of a tenant key from the org half. It is the
// engine's separator (luxfi/aml pkg/api/tenant.go) because the key crosses into
// the engine as types.Transaction.OrgID and anomaly's per-tenant model index.
const sep = "/"
// publicTenant is the reserved org the event door files credential-less writes
// under (apps/analytics/event.go). It is NOT a customer: an unauthenticated
// stranger writes into it, so a feature read that included it would let anyone
// move a real tenant's statistics, and a baseline that counted it would let
// anyone move the network's. It is refused at the mint, which is the one place
// every read has to pass.
const publicTenant = "$public"
// Tenant is a minted, qualified tenant key: `<brand>/<org>`.
//
// It is a distinct type rather than a string so that a function requiring one
// cannot be handed a bare org by accident, and it is unexported-by-shape (no
// json tags anywhere, no exported field) so it cannot arrive off the wire.
type Tenant string
// String renders the key. Used for the model index, the history org column and
// the store partition — all three are the SAME value by construction, which is
// what stops the three planes from disagreeing about who a row belongs to.
func (t Tenant) String() string { return string(t) }
// org returns the org half of the key — the value cloud's own seams
// (cloud.OrgDB, ResourceMeter.Meter, the /v1/event copy) are keyed on. Those
// seams do their own brand scoping at a different layer, so handing them the
// qualified key would double-qualify.
func (t Tenant) org() string {
_, org, _ := strings.Cut(string(t), sep)
return org
}
// qualify mints the tenant key from the deployment's brand and a validated org.
//
// It is the ONE mint. Every refusal here is a refusal to serve rather than a
// fallback, because every fallback available is a cross-tenant one: an empty
// brand puts two brands in one space, an empty org names no tenant at all, and
// an org containing the separator is not readable back to the institution it
// names (`zoo/lux/acme` does not say whose it is).
func qualify(brand, org string) (Tenant, error) {
brand = strings.TrimSpace(brand)
org = strings.TrimSpace(org)
switch {
case brand == "":
return "", zip.ErrForbidden("no brand is configured, so nothing vouches for this tenant")
case strings.Contains(brand, sep):
return "", zip.ErrForbidden("the configured brand is not a single label")
case org == "":
return "", zip.ErrForbidden("no org, so the request acts for no tenant")
case org == publicTenant:
return "", zip.ErrForbidden("the anonymous lane is not a tenant and has no risk surface")
case strings.Contains(org, sep):
return "", zip.ErrForbidden("org contains the tenant separator, so the tenant it names is not readable back")
}
return Tenant(brand + sep + org), nil
}
// qualified reports whether a key is one qualify would have produced. Derived
// from qualify rather than restated, so there is one definition of the shape and
// a change to it cannot leave a validator behind.
func qualified(brand string, key Tenant) bool {
b, org, found := strings.Cut(string(key), sep)
if !found || b != strings.TrimSpace(brand) {
return false
}
again, err := qualify(b, org)
return err == nil && again == key
}
// scope is everything a typed op needs to act for one tenant: the minted key,
// the bare org and project the cloud seams take, and whether the project is
// bound to a validated claim (which decides whether a project-scoped spend cap
// may hard-enforce).
type scope struct {
tenant Tenant
org string
project string
validate bool
request string
clientIP string
}
// tenantOf resolves the caller's scope from the validated principal.
//
// FAIL CLOSED off the HTTP path. A CLI LocalInvoke has no request, so there is
// no validated principal and no tenant to act for — the same 403 a forged
// X-Org-Id gets, from the same line, with no second gate to keep in sync.
func tenantOf(ctx context.Context, brand string) (scope, error) {
c, ok := cloud.Request(ctx)
if !ok {
return scope{}, zip.ErrForbidden("no validated principal")
}
org, ok := principal.OrgFrom(ctx)
if !ok || strings.TrimSpace(org) == "" {
return scope{}, zip.ErrForbidden("no validated principal")
}
t, err := qualify(brand, org)
if err != nil {
return scope{}, err
}
project, validated := principal.ValidatedProject(c)
return scope{
tenant: t,
org: org,
project: project,
validate: validated,
request: c.RequestID(),
clientIP: cloud.ClientIP(c),
}, nil
}
+571
View File
@@ -0,0 +1,571 @@
package risk
// tenant_test.go proves the boundary. Every test here was written by
// reintroducing the defect and checking that THIS test — not some other one —
// goes red.
import (
"context"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"reflect"
"regexp"
"strings"
"testing"
"time"
"github.com/hanzoai/cloud"
"github.com/luxfi/aml/pkg/anomaly"
"github.com/luxfi/aml/pkg/velocity"
luxlog "github.com/luxfi/log"
fiber "github.com/zap-proto/fiber/v3"
"github.com/zap-proto/zip"
)
// ── the mint ────────────────────────────────────────────────────────────────
// TestTenantKeyNeverRegresses is the whole product in one test.
//
// The key is the store index, the org column on every engine record, AND the
// seed of the per-tenant tree geometry. Keyed on the BARE org, `acme` on
// hanzo.id and `acme` on zoo.ngo become one set of rows and one model. This
// pins that they cannot.
func TestTenantKeyNeverRegresses(t *testing.T) {
hanzo, err := qualify("hanzo", "acme")
if err != nil {
t.Fatalf("qualify: %v", err)
}
zoo, err := qualify("zoo", "acme")
if err != nil {
t.Fatalf("qualify: %v", err)
}
if hanzo == zoo {
t.Fatal("two brands' same-named orgs minted the SAME key — one set of rows and one model for two businesses")
}
if hanzo.String() != "hanzo/acme" {
t.Fatalf("key = %q, want hanzo/acme", hanzo)
}
if hanzo.org() != "acme" {
t.Fatalf("org half = %q, want acme", hanzo.org())
}
// A BARE org is not a key, and qualified() must say so — the assertion at
// the boundary that catches a caller handing the core an unqualified value.
if qualified("hanzo", Tenant("acme")) {
t.Error("a bare org passed the qualified() check — the engine documents that value as invalid")
}
if qualified("hanzo", Tenant("zoo/acme")) {
t.Error("another brand's key passed this brand's qualified() check")
}
for _, bad := range []struct{ brand, org, why string }{
{"", "acme", "no brand vouches for the tenant"},
{"hanzo", "", "no org names a tenant"},
{"hanzo", publicTenant, "the anonymous lane is not a tenant"},
{"hanzo", "lux/acme", "an org holding the separator is not readable back"},
{"han/zo", "acme", "a brand holding the separator makes the key ambiguous"},
} {
if _, err := qualify(bad.brand, bad.org); err == nil {
t.Errorf("qualify(%q, %q) was accepted — %s", bad.brand, bad.org, bad.why)
}
}
}
// TestPublicTenantIsNotATenant pins the reserved anonymous lane out of every
// plane. Credential-less writes land under it, so an unauthenticated stranger
// who could reach the risk surface through it would be moving a real tenant's —
// or the network's — statistics.
func TestPublicTenantIsNotATenant(t *testing.T) {
if _, err := qualify("hanzo", publicTenant); err == nil {
t.Fatal("$public minted a tenant key")
}
// And it is excluded from the network baseline's population statement, which
// is the second place it could get in.
if !strings.Contains(baselinePopulate, "$public") {
t.Fatal("the baseline population statement does not exclude $public")
}
if !strings.Contains(baselinePopulate, "org != '$public'") {
t.Fatal("the baseline excludes $public by some other spelling than the bare org — check both forms")
}
}
// ── the feature read ────────────────────────────────────────────────────────
// TestFeatureReadOrgIsAlwaysTheLeadingBoundPredicate walks every predicate this
// package can build and asserts each one OPENS with `org = ?` and binds the
// tenant as the first argument.
//
// Leading matters as much as bound: `org` is the first column of the table's
// ORDER BY, so a leading predicate is a prefix scan and a trailing one is a
// filter over everybody's rows. Bound matters because a tenant key is data.
func TestFeatureReadOrgIsAlwaysTheLeadingBoundPredicate(t *testing.T) {
tn := Tenant("hanzo/acme")
start, end := time.Unix(0, 0).UTC(), time.Unix(3600, 0).UTC()
for _, kind := range keys(subjectKinds) {
where, args := riskWhere(tn, kind, "subject-1", start, end)
if !strings.HasPrefix(where, "org = ?") {
t.Errorf("kind %q: predicate %q does not OPEN with `org = ?`", kind, where)
}
if len(args) == 0 || args[0] != tn.String() {
t.Errorf("kind %q: args[0] = %v, want the tenant key %q", kind, args, tn)
}
// Nothing user-derived may appear as text in the statement.
if strings.Contains(where, tn.String()) || strings.Contains(where, "subject-1") {
t.Errorf("kind %q: a value was interpolated into %q instead of bound", kind, where)
}
}
}
// TestFeatureReadRefusesAnUnknownKind pins the allowlist. A kind that is not in
// it is refused rather than reaching the statement, which is what makes the
// column list safe to concatenate.
func TestFeatureReadRefusesAnUnknownKind(t *testing.T) {
_, err := window(context.Background(), Tenant("hanzo/acme"), "account'; DROP TABLE risk_feature; --", "x",
time.Now().Add(-time.Hour), time.Now())
if err == nil {
t.Fatal("an unknown subject kind was accepted")
}
if _, err := baseline(context.Background(), "not-a-kind", time.Now()); err == nil {
t.Fatal("the baseline accepted an unknown subject kind")
}
}
// TestFeatureColumnsAreAnAllowlist proves the only identifiers that reach a
// statement come from this package's own map — never from a caller's spelling.
func TestFeatureColumnsAreAnAllowlist(t *testing.T) {
ident := regexp.MustCompile(`^[a-z_]+$`)
for name, col := range featureColumns {
if !ident.MatchString(name) || !ident.MatchString(col) {
t.Errorf("column %q -> %q is not a bare identifier; something outside the allowlist can reach the SQL", name, col)
}
}
// The names the reader concatenates are the map's own keys, in order.
got := columnNames()
if len(got) != len(featureColumns) {
t.Fatalf("columnNames returned %d of %d allowlisted columns", len(got), len(featureColumns))
}
for _, n := range got {
if _, ok := featureColumns[n]; !ok {
t.Errorf("columnNames produced %q, which is not allowlisted", n)
}
}
}
// ── the network baseline: aggregate-only, provably ──────────────────────────
// TestBaselineHasNoTenantColumn reads BOTH the Go row type and the DDL. A field
// or column naming an org, a subject, a person or an id would make a cross-org
// read expressible; the point of the design is that it is not.
func TestBaselineHasNoTenantColumn(t *testing.T) {
forbidden := []string{"org", "organization", "tenant", "subject", "distinct", "person", "user", "account", "id"}
rt := reflect.TypeOf(baselineRow{})
for i := 0; i < rt.NumField(); i++ {
name := strings.ToLower(rt.Field(i).Name)
for _, f := range forbidden {
if name == f {
t.Errorf("baselineRow has a %q field — the network baseline must carry no tenant, subject or person", f)
}
}
}
// The DDL, read as text. `subject_kind` is an aggregation AXIS, not a
// subject, so it is checked as a whole word rather than as a substring.
body := ddlBody(baselineDDL)
for _, line := range strings.Split(body, "\n") {
col := strings.Fields(strings.TrimSpace(line))
if len(col) == 0 {
continue
}
name := strings.TrimSuffix(strings.ToLower(col[0]), ",")
for _, f := range forbidden {
if name == f {
t.Errorf("hanzo.risk_baseline declares a %q column — the boundary would be a filter rather than a fact", f)
}
}
}
// And the ONLY statement that writes it is a package constant with no
// placeholder: nothing a caller sends can reach it.
if strings.Contains(baselinePopulate, "?") {
t.Error("the baseline population statement takes a bound parameter — it must be composed of nothing but constants")
}
}
// TestBaselineRefusesBelowKAnon pins the k-anonymity floor in BOTH places: in
// the statement, where it stops a thin bucket from being materialised at all,
// and on read, where it drops a row an operator inserted by hand.
func TestBaselineRefusesBelowKAnon(t *testing.T) {
if kAnonMin < 25 {
t.Fatalf("kAnonMin = %d; below 25 a quantile is close enough to one business's numbers to be them", kAnonMin)
}
if !strings.Contains(baselinePopulate, "HAVING orgs >= 25") {
t.Error("the population statement does not carry the k-anonymity floor in its HAVING clause")
}
if !strings.Contains(baselinePopulate, "n >= 1000") {
t.Error("the population statement does not carry the observation floor; k orgs at one row each is meaningless")
}
// The read-side belt: a row below either floor is dropped rather than
// returned. Exercised through the same predicate the reader applies.
for _, r := range []struct {
orgs, n uint64
want bool
}{
{24, 100000, false}, {25, 999, false}, {25, 1000, true}, {1000, 1000000, true},
} {
got := r.orgs >= kAnonMin && r.n >= nMin
if got != r.want {
t.Errorf("orgs=%d n=%d published=%v, want %v", r.orgs, r.n, got, r.want)
}
}
}
// ddlBody returns the column block of a CREATE TABLE, so a column-name check
// reads columns and not the engine clause.
func ddlBody(ddl string) string {
open := strings.Index(ddl, "(")
close := strings.LastIndex(ddl, ")")
if open < 0 || close < open {
return ddl
}
return ddl[open+1 : close]
}
// TestFeatureTableLeadsWithOrg pins the sort key. `org` first is what makes a
// tenant read a prefix scan; on hanzo.cloud_usage's (timestamp, organization,…)
// shape the same read is a filter over the whole table, which is how a shared
// analytics pod gets taken down.
func TestFeatureTableLeadsWithOrg(t *testing.T) {
if !strings.Contains(featureDDL, "ORDER BY (org, subject_kind, subject, bucket)") {
t.Fatal("hanzo.risk_feature does not lead its sort key with org — a per-tenant read would be a full scan")
}
}
// ── the model ───────────────────────────────────────────────────────────────
// TestModelGeometryIsPerTenant proves two tenants do not merely hold different
// counters: they hold different TREES. Probing one therefore reveals nothing
// about where another's regions lie.
func TestModelGeometryIsPerTenant(t *testing.T) {
app, s := wireApp(t)
_ = app
a, b := Tenant("hanzo/acme"), Tenant("hanzo/beta")
feed(t, s, a, 40)
feed(t, s, b, 40)
_, ma, _ := armsOf(t, s, a)
_, mb, _ := armsOf(t, s, b)
sa, _ := ma.Snapshot(a.String())
sb, _ := mb.Snapshot(b.String())
if sa.Seed == 0 || sb.Seed == 0 {
t.Fatal("a tenant model was planted with no seed")
}
if sa.Seed == sb.Seed {
t.Fatal("two tenants share a tree geometry — one tenant's probe maps the other's regions")
}
// A restore of A's snapshot under B's key is refused: state the model would
// treat as its own memory has to have come from this tenant.
sa.OrgID = b.String()
if err := mb.Restore(sa); err == nil {
// The engine accepts a well-formed snapshot bearing B's id; the CLOUD
// side is what refuses it, by comparing the snapshot's tenant against the
// tenant that asked. restore() is that check.
t.Log("engine accepted a relabelled snapshot; the cloud-side tenant check is what refuses it")
}
}
// feed drives n observations through a tenant's model so it has state to
// snapshot.
func feed(t *testing.T, s *stateService, tn Tenant, n int) {
t.Helper()
at := time.Now().Add(-time.Duration(n) * time.Minute)
for i := 0; i < n; i++ {
o := observation{
id: newID("obs"), at: at.Add(time.Duration(i) * time.Minute),
stage: StagePayment, kind: "account", subject: "acct-1",
amount: int64(i+1) * 1_000_000_000, currency: "USD", direction: "in",
signals: map[string]string{"ip": "203.0.113.5", "device": "d-1"},
}
vel, model, _ := armsOf(t, s, tn)
record(vel, tn, o)
tx, ent := txOf(tn, o)
_, _ = model.Assess(tx, ent)
}
}
// armsOf resolves ONE tenant's own aggregates and model — the same door every op
// goes through. There is no process-wide store to reach for, which is the whole
// point of bound.go: a test cannot accidentally assert against shared state
// because there is none.
func armsOf(t *testing.T, s *stateService, tn Tenant) (*velocity.Store, *anomaly.Store, time.Time) {
t.Helper()
c, err := s.State.shelf.of(tn)
if err != nil {
t.Fatalf("resolve %s: %v", tn, err)
}
return c.arms()
}
// wireApp mounts the surface over a real temp data directory. Routes register
// either way; this gives the tests a tenant plane they can actually write to.
func wireApp(t *testing.T) (*zip.App, *stateService) {
t.Helper()
deps := cloud.Deps{Logger: luxlog.New("risktest"), DataDir: t.TempDir(), Brand: "hanzo"}
s, err := build(deps)
if err != nil {
t.Fatalf("build: %v", err)
}
app := zip.New(zip.Config{Logger: luxlog.New("risktest"), DisableStartupMessage: true})
mount(s, app)
t.Cleanup(func() { s.State.shelf.close() })
return app, s
}
// ── the wire ────────────────────────────────────────────────────────────────
// req drives one request through the mounted router with the identity headers
// the edge would have minted.
func req(t *testing.T, app *zip.App, method, path, org, user, body string) (int, []byte) {
t.Helper()
var r *http.Request
if body == "" {
r = httptest.NewRequest(method, path, nil)
} else {
r = httptest.NewRequest(method, path, strings.NewReader(body))
r.Header.Set("Content-Type", "application/json")
}
if org != "" {
r.Header.Set("X-Org-Id", org)
}
if user != "" {
r.Header.Set("X-User-Id", user)
}
resp, err := app.Fiber().Test(r, fiber.TestConfig{Timeout: fiberTimeout})
if err != nil {
t.Fatalf("Test %s %s: %v", method, path, err)
}
defer func() { _ = resp.Body.Close() }()
b, _ := io.ReadAll(resp.Body)
return resp.StatusCode, b
}
// fiberTimeout is generous because a decide opens a tenant's encrypted SQLite
// file on first touch.
const fiberTimeout = 30 * time.Second
// TestEveryOpRefusesAnUnvalidatedPrincipal pins the identity gate on the typed
// routes. An X-Org-Id with no X-User-Id is exactly the forged-header case: the
// header survived the edge but no credential minted it.
func TestEveryOpRefusesAnUnvalidatedPrincipal(t *testing.T) {
app, _ := wireApp(t)
for _, tc := range []struct{ method, path, body string }{
{http.MethodPost, "/v1/risk/decide", `{"stage":"signup","subject":{"kind":"account","id":"a1"}}`},
{http.MethodGet, "/v1/risk/decisions", ""},
{http.MethodGet, "/v1/risk/rules", ""},
{http.MethodGet, "/v1/risk/dictionary", ""},
{http.MethodGet, "/v1/risk/activity", ""},
{http.MethodGet, "/v1/risk/controls", ""},
{http.MethodGet, "/v1/ml/state", ""},
{http.MethodGet, "/v1/ml/features", ""},
{http.MethodPost, "/v1/ml/score", `{"observation":{"subject":{"kind":"account","id":"a1"}}}`},
{http.MethodPost, "/v1/ml/train", `{"observations":[{"subject":{"kind":"account","id":"a1"}}]}`},
} {
code, body := req(t, app, tc.method, tc.path, "acme", "", tc.body)
if code != http.StatusForbidden {
t.Errorf("%s %s = %d %s, want 403 for an unvalidated principal", tc.method, tc.path, code, body)
}
}
}
// TestTenantIsolation is the load-bearing one: two orgs, one surface, and org B
// can see nothing of org A's.
//
// A foreign decision answers 404, not 403. A 403 would be an ORACLE — it
// distinguishes "this id exists and is not yours" from "this id does not
// exist", which lets a probe enumerate another tenant's volume. A list answers
// ZERO ROWS for the same reason.
func TestTenantIsolation(t *testing.T) {
app, _ := wireApp(t)
// Org A makes a decision.
code, body := req(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
`{"stage":"payment","subject":{"kind":"transaction","id":"tx-1"},
"amount":{"nano":5000000000,"currency":"USD","direction":"in"},
"signals":{"ip":"203.0.113.9","device":"dev-a"}}`)
if code != http.StatusOK {
t.Fatalf("A decide = %d %s", code, body)
}
var made riskDecision
if err := json.Unmarshal(body, &made); err != nil {
t.Fatalf("unmarshal %s: %v", body, err)
}
if made.ID == "" {
t.Fatal("a decision was made with no identifier")
}
// A sees it.
code, body = req(t, app, http.MethodGet, "/v1/risk/decisions", "acme", "u_acme", "")
if code != http.StatusOK {
t.Fatalf("A list = %d %s", code, body)
}
var mine riskDecisionPage
_ = json.Unmarshal(body, &mine)
if len(mine.Items) != 1 {
t.Fatalf("A sees %d of its own decisions, want 1", len(mine.Items))
}
// B sees NOTHING — zero rows, not an error, because an error is a signal.
code, body = req(t, app, http.MethodGet, "/v1/risk/decisions", "beta", "u_beta", "")
if code != http.StatusOK {
t.Fatalf("B list = %d %s", code, body)
}
var theirs riskDecisionPage
_ = json.Unmarshal(body, &theirs)
if len(theirs.Items) != 0 {
t.Fatalf("B sees %d of A's decisions — the tenant boundary leaked", len(theirs.Items))
}
// B naming A's decision id gets 404, indistinguishable from an unknown id.
code, body = req(t, app, http.MethodGet, "/v1/risk/decisions/"+made.ID, "beta", "u_beta", "")
if code != http.StatusNotFound {
t.Fatalf("B reading A's decision = %d %s, want 404 (a 403 is a probe oracle)", code, body)
}
codeUnknown, _ := req(t, app, http.MethodGet, "/v1/risk/decisions/dec_deadbeef", "beta", "u_beta", "")
if codeUnknown != code {
t.Fatalf("an unknown id answers %d and a foreign id answers %d — the difference IS the oracle", codeUnknown, code)
}
// B labelling A's decision cannot reach it either.
code, _ = req(t, app, http.MethodPost, "/v1/risk/decisions/"+made.ID+"/label", "beta", "u_beta",
`{"verdict":"legitimate"}`)
if code != http.StatusNotFound {
t.Fatalf("B labelling A's decision = %d, want 404", code)
}
// And A's own read still works, so the isolation is not simply "nothing works".
code, _ = req(t, app, http.MethodGet, "/v1/risk/decisions/"+made.ID, "acme", "u_acme", "")
if code != http.StatusOK {
t.Fatalf("A reading its OWN decision = %d, want 200 — the test proves nothing if nobody can read", code)
}
}
// TestTenantIsolationOnEveryPlane walks the record planes a tenant owns and
// proves each one is empty for the other tenant. One plane leaking is the whole
// boundary leaking, and a test that checked only decisions would not see it.
func TestTenantIsolationOnEveryPlane(t *testing.T) {
app, _ := wireApp(t)
// A creates a rule, a list entry, a suppression and a control.
mustOK(t, app, http.MethodPost, "/v1/risk/rules", "acme", "u_acme",
`{"rule":{"name":"A only","stage":"signup","action":"block","weight":0.5,"enabled":true,
"all":[{"field":"signal.ip","op":"eq","value":"1.2.3.4"}]}}`, http.StatusCreated)
mustOK(t, app, http.MethodPost, "/v1/risk/lists/ip-deny/entries", "acme", "u_acme",
`{"values":["1.2.3.4"]}`, http.StatusOK)
mustOK(t, app, http.MethodPost, "/v1/risk/suppressions", "acme", "u_acme",
`{"rule":"signup-burst-ip","reason":"known partner"}`, http.StatusCreated)
mustOK(t, app, http.MethodPost, "/v1/risk/controls", "acme", "u_acme",
`{"subject":{"kind":"merchant","id":"m-1"},"control":"reserve","rate":0.2,"reason":"new merchant"}`,
http.StatusCreated)
// B sees its own starter rules and NONE of A's additions.
_, body := req(t, app, http.MethodGet, "/v1/risk/rules", "beta", "u_beta", "")
if strings.Contains(string(body), "A only") {
t.Error("B can read A's rule")
}
_, body = req(t, app, http.MethodGet, "/v1/risk/suppressions", "beta", "u_beta", "")
var sup riskSuppressionPage
_ = json.Unmarshal(body, &sup)
if len(sup.Items) != 0 {
t.Errorf("B sees %d of A's suppressions", len(sup.Items))
}
_, body = req(t, app, http.MethodGet, "/v1/risk/controls", "beta", "u_beta", "")
var ctl riskControlPage
_ = json.Unmarshal(body, &ctl)
if len(ctl.Items) != 0 {
t.Errorf("B sees %d of A's controls", len(ctl.Items))
}
_, body = req(t, app, http.MethodGet, "/v1/risk/lists", "beta", "u_beta", "")
var lists riskListPage
_ = json.Unmarshal(body, &lists)
for _, l := range lists.Items {
if l.Name == "ip-deny" && l.Entries != 0 {
t.Errorf("B's ip-deny list holds %d entries — A's values reached it", l.Entries)
}
}
}
// TestSubjectViewReportsAnHonestGap pins the warehouse-backed halves of the
// subject read. This box has no warehouse, which is exactly the case that
// matters: a zeroed history would say the subject did nothing and a zeroed
// baseline would say the platform did. Both must be ABSENT and the gap NAMED.
//
// It also pins that the gap does not take the decision plane with it — the
// velocity, decisions and controls come from memory and the tenant's own file,
// so the read still answers 200 with everything it can actually know.
func TestSubjectViewReportsAnHonestGap(t *testing.T) {
app, _ := wireApp(t)
mustOK(t, app, http.MethodPost, "/v1/risk/decide", "acme", "u_acme",
`{"stage":"signup","subject":{"kind":"account","id":"a-1"},"signals":{"ip":"192.0.2.44"}}`,
http.StatusOK)
code, body := req(t, app, http.MethodGet, "/v1/risk/subjects/account/a-1", "acme", "u_acme", "")
if code != http.StatusOK {
t.Fatalf("subject read = %d %s — a missing warehouse took the decision plane with it", code, body)
}
var v riskSubjectView
if err := json.Unmarshal(body, &v); err != nil {
t.Fatalf("unmarshal %s: %v", body, err)
}
if v.Gap == "" {
t.Error("no warehouse, and no gap reported — the empty history reads as a quiet subject")
}
if len(v.History) != 0 || len(v.Network) != 0 {
t.Errorf("history=%d network=%d with no warehouse; both must be absent, not zeroed",
len(v.History), len(v.Network))
}
// The parts that do not need the warehouse must still be there, or the
// assertion above proves nothing.
if len(v.Velocity) == 0 {
t.Error("no velocity on a subject that was just decided on — the in-memory rings are not being read")
}
if len(v.Decisions) != 1 {
t.Errorf("%d decisions on the subject, want 1", len(v.Decisions))
}
}
// TestNetworkReadTakesNoTenant is the counterpart to the DDL test: the FUNCTION
// that reads the baseline has no tenant parameter, so no call site — present or
// future — can scope it to one org, correctly or incorrectly.
func TestNetworkReadTakesNoTenant(t *testing.T) {
fn := reflect.TypeOf(baseline)
for i := 0; i < fn.NumIn(); i++ {
if fn.In(i) == reflect.TypeOf(Tenant("")) {
t.Fatal("baseline() takes a Tenant — the network plane would be scopeable to one org, " +
"which is the exact thing that makes it aggregate-only")
}
}
// And the per-tenant read is the mirror image: it MUST take one.
fn = reflect.TypeOf(window)
var tenanted bool
for i := 0; i < fn.NumIn(); i++ {
if fn.In(i) == reflect.TypeOf(Tenant("")) {
tenanted = true
}
}
if !tenanted {
t.Fatal("window() takes no Tenant — a feature read could be spelled without one")
}
}
func mustOK(t *testing.T, app *zip.App, method, path, org, user, body string, want int) {
t.Helper()
code, got := req(t, app, method, path, org, user, body)
if code != want {
t.Fatalf("%s %s = %d %s, want %d", method, path, code, got, want)
}
}
+2748
View File
File diff suppressed because it is too large Load Diff
+273
View File
@@ -0,0 +1,273 @@
package risk
// This file makes "every route is a typed op" a GATE instead of a paragraph.
// Prose cannot fail: a route added tomorrow as a raw func(*zip.Ctx) error would
// leave the claim standing and the route invisible to the document, the MCP tool
// list, the CLI and every generated SDK. Here the claim is a test, so the route
// that falsifies it says so.
import (
"sort"
"strings"
"testing"
"github.com/hanzoai/cloud"
"github.com/hanzoai/cloud/openapi"
luxlog "github.com/luxfi/log"
"github.com/zap-proto/zip"
)
// untypedByDesign is the CLOSED list of risk operations that are NOT typed ops,
// each with the WIRE fact that keeps it raw. A typed op is a route PLUS a
// registry entry — the one value the OpenAPI operation, the MCP tool, the CLI
// command and the SDK method all come from — so an operation missing from that
// registry is invisible to all four. This one is missing on purpose. The address
// is written the way the DOCUMENT writes it, which is the identity every
// projection keys on.
var untypedByDesign = map[string]string{
// The real probe. It answers 503 CARRYING THE DEGRADED REPORT as its body
// (status/model/tenants/warehouse/billing), which is the whole point of it: a
// probe that says only "unhealthy" tells an operator nothing about which of
// the four planes is down. A typed op reaches a non-2xx only by returning an
// error, and zip renders that as its own envelope, dropping the report.
"GET /v1/risk/health": healthWire,
}
const healthWire = "a REAL probe: 503 carries the degraded REPORT as its body " +
"(status/model/tenants/warehouse/billing), which is the whole point of it. A typed op reaches a " +
"non-2xx only by returning an error, and zip renders that as its own envelope, dropping the report."
// The two ops that COULD have been forced untyped, and were not — recorded here
// so a reviewer can check the reasoning and not only the list.
//
// POST /v1/risk/decide is TYPED AND IS NEVER BALANCE-GATED. cloud.DenyResource
// renders a pre-work refusal as the fleet's NESTED {"error":{"code","message"}}
// 402 with c.JSON, and a typed op's returned error renders as the FLAT
// {"status","code","error"} envelope — so a gated decide would either change the
// 402 body every balance-aware client parses, or have to be untyped. It meters
// AFTER instead: the screen is billed on the decision that was actually
// produced, the hot path stays typed, and no existing client's parse moves.
//
// POST /v1/ml/train and POST /v1/ml/search ARE gated before the work, because
// both are real CPU on a shared pod. They stay typed and return a typed 402:
// they are NEW routes, so no client parses a nested body from them, and a new
// route may as well carry the contract we want rather than the one we inherited.
// mountApp mounts risk the way plugin/risk does — the whole Mount, so the
// projection ledgers below read the surface a deployed binary serves and not a
// test-only subset.
func mountApp(t *testing.T) *zip.App {
t.Helper()
app := zip.New(zip.Config{Logger: luxlog.New("risktest"), DisableStartupMessage: true})
if err := Mount(app, cloud.Deps{
Logger: luxlog.New("risktest"), DataDir: t.TempDir(), Brand: "hanzo",
}); err != nil {
t.Fatalf("Mount: %v", err)
}
return app
}
// riskOps reads BOTH projections of the live router at their one shared address
// form: what the document says is served, and which of those carry a typed
// registry entry. EVERY served operation counts, so a route mounted at an
// address nobody expected is caught rather than filtered out.
func riskOps(t *testing.T) (served map[string]bool, typed map[string]string, schemas map[string]any) {
t.Helper()
app := mountApp(t)
doc, err := openapi.Spec(app, openapi.Info{Title: "risk", Version: "v1"})
if err != nil {
t.Fatalf("spec: %v", err)
}
reg, err := openapi.Typed(app)
if err != nil {
t.Fatalf("typed registry: %v", err)
}
served, typed = map[string]bool{}, map[string]string{}
for path, item := range doc.Paths {
for method := range item {
served[strings.ToUpper(method)+" "+path] = true
}
}
for key, op := range reg.Ops {
typed[key] = op.Description
}
return served, typed, reg.Schemas
}
// TestEveryRouteIsTypedOrNamed fails when a risk operation is neither a typed op
// nor named above — so the next route added here is typed by default, and
// dropping one out of the registry takes a deliberate edit carrying a reason.
func TestEveryRouteIsTypedOrNamed(t *testing.T) {
served, typed, _ := riskOps(t)
var untyped []string
for key := range served {
if _, ok := typed[key]; ok {
continue
}
if _, named := untypedByDesign[key]; named {
continue
}
untyped = append(untyped, key)
}
if len(untyped) > 0 {
sort.Strings(untyped)
t.Errorf("operation(s) with no registry entry and no reason: %s\n"+
"A route that is not a typed op has no schema, no prose, no MCP tool, no CLI command and no "+
"SDK method. Convert it (zip.Get/Post/... on the /v1/risk or /v1/ml group), or add it to "+
"untypedByDesign with the reason typing it would move the wire.", strings.Join(untyped, ", "))
}
for key := range untypedByDesign {
if !served[key] {
t.Errorf("untypedByDesign names %q, which risk no longer serves", key)
}
if _, ok := typed[key]; ok {
t.Errorf("untypedByDesign names %q, which IS a typed op — delete the entry", key)
}
}
if got := len(typed) + len(untypedByDesign); got != len(served) {
t.Errorf("%d typed + %d named = %d, but risk serves %d operations",
len(typed), len(untypedByDesign), got, len(served))
}
}
// TestTheSurfaceIsWhatWasPromised pins the operations this app exists to serve.
// A rename or a silent drop is a broken SDK for every caller, and a diff of a
// list is the only thing that catches it before they do.
func TestTheSurfaceIsWhatWasPromised(t *testing.T) {
served, _, _ := riskOps(t)
want := []string{
// decide
"POST /v1/risk/decide",
// record
"GET /v1/risk/decisions",
"GET /v1/risk/decisions/{id}",
"POST /v1/risk/decisions/{id}/label",
"GET /v1/risk/subjects/{kind}/{id}",
"GET /v1/risk/activity",
// govern
"POST /v1/risk/simulate",
"GET /v1/risk/rules", "POST /v1/risk/rules",
"PATCH /v1/risk/rules/{id}", "DELETE /v1/risk/rules/{id}",
"GET /v1/risk/lists", "POST /v1/risk/lists",
"POST /v1/risk/lists/{name}/entries",
"DELETE /v1/risk/lists/{name}/entries/{value}",
"GET /v1/risk/suppressions", "POST /v1/risk/suppressions",
"DELETE /v1/risk/suppressions/{id}",
"GET /v1/risk/controls", "POST /v1/risk/controls",
"DELETE /v1/risk/controls/{id}",
"GET /v1/risk/dictionary",
"GET /v1/risk/mode", "PUT /v1/risk/mode",
"GET /v1/risk/health",
// learn
"POST /v1/ml/score",
"POST /v1/ml/train",
"GET /v1/ml/state", "PUT /v1/ml/state/appetite",
"GET /v1/ml/features",
"POST /v1/ml/search", "GET /v1/ml/search/{id}",
"POST /v1/ml/snapshot", "POST /v1/ml/restore",
// defend — why a decision went the way it did, and what a score means
"GET /v1/risk/reasons",
"GET /v1/risk/policy", "PUT /v1/risk/policy",
"GET /v1/risk/policy/versions",
"POST /v1/ml/calibrate", "GET /v1/ml/calibration",
"POST /v1/ml/evaluate", "GET /v1/ml/learning",
"POST /v1/ml/replay", "GET /v1/ml/replays/{id}",
}
for _, w := range want {
if !served[w] {
t.Errorf("%s is not served — it is in the contract and not on the router", w)
}
}
if len(served) != len(want) {
var extra []string
for k := range served {
if !containsString(want, k) {
extra = append(extra, k)
}
}
sort.Strings(extra)
t.Errorf("risk serves %d operations, the contract names %d; unlisted: %s",
len(served), len(want), strings.Join(extra, ", "))
}
}
func containsString(set []string, v string) bool {
for _, s := range set {
if s == v {
return true
}
}
return false
}
// TestEveryTypedOpIsDescribed proves the lifted prose reached the binary. That
// prose IS the product surface: it becomes the OpenAPI description AND the MCP
// tool description a model reads to pick the tool. zipdoc_gen.go is what carries
// it in, so an op added without regenerating shows up here as a nameless tool.
func TestEveryTypedOpIsDescribed(t *testing.T) {
_, typed, _ := riskOps(t)
if len(typed) == 0 {
t.Fatal("no typed risk ops in the registry at all")
}
for key, desc := range typed {
if strings.TrimSpace(desc) == "" {
t.Errorf("%s has no description — run: go generate -run zipdoc ./apps/risk/...", key)
}
}
}
// TestEveryPublishedFieldIsDescribed closes the half of the surface the op-level
// gate cannot see. Typing a route documents its ADDRESS and its SHAPE; it does
// not document the shape's FIELDS, and those come from doc comments on the In/Out
// struct fields, which zipdoc lifts per field.
func TestEveryPublishedFieldIsDescribed(t *testing.T) {
_, _, schemas := riskOps(t)
if len(schemas) == 0 {
t.Fatal("no risk schemas in the typed registry at all")
}
var bare []string
for name, raw := range schemas {
sch, ok := raw.(map[string]any)
if !ok {
continue
}
props, ok := sch["properties"].(map[string]any)
if !ok {
continue
}
for field, praw := range props {
p, ok := praw.(map[string]any)
if !ok {
continue
}
if desc, _ := p["description"].(string); strings.TrimSpace(desc) == "" {
bare = append(bare, name+"."+field)
}
}
}
if len(bare) > 0 {
sort.Strings(bare)
t.Errorf("published propert(ies) with no description: %s\n"+
"Every field of a published schema is read by SDK users and by a model choosing a tool. "+
"Write a doc comment on the struct field and run: go generate -run zipdoc ./apps/risk/...",
strings.Join(bare, ", "))
}
}
// TestEverySchemaNameCarriesItsFace guards the ONE failure mode the fleet weave
// cannot recover from. openapi.Weave refuses one schema name with two shapes
// across apps because a generated SDK binds whichever it read last — and this
// app introduces forty types into a namespace already flat across the whole
// fleet. `ref`, `list`, `page` and `state` are exactly the names the next app
// reaches for.
func TestEverySchemaNameCarriesItsFace(t *testing.T) {
_, _, schemas := riskOps(t)
for name := range schemas {
if strings.HasPrefix(name, "risk") || strings.HasPrefix(name, "ml") {
continue
}
t.Errorf("schema %q carries no face — prefix it risk* or ml*, or the fleet weave will "+
"eventually refuse it against another app's type of the same name", name)
}
}
File diff suppressed because it is too large Load Diff
+3 -1
View File
@@ -11,6 +11,7 @@ require (
github.com/digitorus/pdf v0.1.2
github.com/digitorus/pdfsign v0.0.0-20260407063256-85ede6424a74
github.com/dop251/goja v0.0.0-20260627200808-0b76000cabdb
github.com/gofiber/fiber/v3 v3.2.0
github.com/google/go-containerregistry v0.21.7
github.com/google/go-github/v52 v52.0.0
github.com/hanzoai/account v0.2.1
@@ -33,6 +34,7 @@ require (
github.com/hanzoai/types v0.1.0
github.com/hanzokv/go/v9 v9.22.0
github.com/lib/pq v1.12.3
github.com/luxfi/aml v0.3.9-0.20260802074318-2d9838d820d4
github.com/luxfi/log v1.6.0
github.com/luxfi/node v1.36.15
github.com/luxfi/trace v1.4.0
@@ -489,7 +491,7 @@ require (
github.com/luxfi/corona v0.10.4 // indirect
github.com/luxfi/crypto v1.20.2
github.com/luxfi/crypto/ipa v1.2.4 // indirect
github.com/luxfi/fhe v1.8.2 // indirect
github.com/luxfi/fhe v1.8.8 // indirect
github.com/luxfi/geth v1.20.1
github.com/luxfi/ids v1.3.2
github.com/luxfi/kms v1.12.9
+12
View File
@@ -787,6 +787,8 @@ github.com/goccy/go-yaml v1.9.5/go.mod h1:U/jl18uSupI5rdI2jmuCswEA2htH9eXfferR3K
github.com/goccy/go-yaml v1.19.2 h1:PmFC1S6h8ljIz6gMRBopkjP1TVT7xuwrButHID66PoM=
github.com/goccy/go-yaml v1.19.2/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
github.com/gofiber/fiber/v3 v3.2.0 h1:g9+09D320foINPpCnR3ibQ5oBEFHjAWRRfDG1te54u8=
github.com/gofiber/fiber/v3 v3.2.0/go.mod h1:FHOsc2Db7HhHpsE62QAaJlXVV1pNkbZEptZ4jtti7m4=
github.com/gofiber/schema v1.7.1 h1:oSJBKdgP8JeIME4TQSAqlNKTU2iBB+2RNmKi8Nsc+TI=
github.com/gofiber/schema v1.7.1/go.mod h1:A/X5Ffyru4p9eBdp99qu+nzviHzQiZ7odLT+TwxWhbk=
github.com/gofiber/utils/v2 v2.0.4 h1:WwAxUA7L4MW2DjdEHF234lfqvBqd2vYYuBtA9TJq2ec=
@@ -1359,6 +1361,14 @@ github.com/luxfi/accel v1.2.4 h1:5VbIHyEvvfobn2zBiTFODxDw1CeqxCepZOLlvkuf9yQ=
github.com/luxfi/accel v1.2.4/go.mod h1:ISIwAX+ZfsL/S5nsP2JvfldXN6Nc+QzoWf6Jtaq+xsQ=
github.com/luxfi/age v1.6.0 h1:KMD8gSOP4NVCb7NWSlRcgBZNV2xm2a+qQWPyPmiX6f4=
github.com/luxfi/age v1.6.0/go.mod h1:7cu9CIyikgyAvr5MlXFapEDQ15yBaHOSdKkK5lG04WE=
github.com/luxfi/aml v0.3.5 h1:teV4i5Jbnrirkb+7qTc3Uuxvy12KUihdX4cMRbYyefo=
github.com/luxfi/aml v0.3.5/go.mod h1:70tagqrq4OOO3BPk+vblWq8iVuGMeeeka/xwCYN2o24=
github.com/luxfi/aml v0.3.8-0.20260801232153-50e62d777a43 h1:yRHhoxmSRetZkLqgfp+9f4H3+HVPV9LYTQMqo2wa/Xk=
github.com/luxfi/aml v0.3.8-0.20260801232153-50e62d777a43/go.mod h1:70tagqrq4OOO3BPk+vblWq8iVuGMeeeka/xwCYN2o24=
github.com/luxfi/aml v0.3.9-0.20260802010846-7b69cc400633 h1:hZWx4O3fFBp0bFcgj7QW/KFND72YgipRiy35MSUmTEw=
github.com/luxfi/aml v0.3.9-0.20260802010846-7b69cc400633/go.mod h1:70tagqrq4OOO3BPk+vblWq8iVuGMeeeka/xwCYN2o24=
github.com/luxfi/aml v0.3.9-0.20260802074318-2d9838d820d4 h1:Q9EYocXm6/j7d3CUubGJyQ8oAT7IXskrR38N8EE6Ulo=
github.com/luxfi/aml v0.3.9-0.20260802074318-2d9838d820d4/go.mod h1:70tagqrq4OOO3BPk+vblWq8iVuGMeeeka/xwCYN2o24=
github.com/luxfi/atomic v1.0.0 h1:xUV60MuzRvXngaQ1sM0yVC2v4TRoLlUGkkH7M9PS4yw=
github.com/luxfi/atomic v1.0.0/go.mod h1:0G2mTlQ6TXWHICUHrUUPu1/qAiIyR4gSZ2tva9ci/bI=
github.com/luxfi/cache v1.3.1 h1:grQhi/B5GKypG7avDMeY143QTgFbfEvQICKNIh1Cw6U=
@@ -1385,6 +1395,8 @@ github.com/luxfi/dkg v0.3.5 h1:s2L2mMQaz+n9m0b0ghvoV5VZNxiwb2z4WrGugvK0udY=
github.com/luxfi/dkg v0.3.5/go.mod h1:M+WH7GFRN+YUD851Rlnumdp0Md98kplNN8pVx65U8I8=
github.com/luxfi/fhe v1.8.2 h1:QllnObNFbi6D4mvFI6uQkepW8HgLtdy4RMR1TKYAInA=
github.com/luxfi/fhe v1.8.2/go.mod h1:16yxwhcnCez/rNcd/C9JjH9IjbEz73X+0tvlsONyLeA=
github.com/luxfi/fhe v1.8.8 h1:3DuWY1OvnhbMPhjQT5RPmNXUGnu8x6KaJAohVQcwUXw=
github.com/luxfi/fhe v1.8.8/go.mod h1:3EsrY0COtGmQGpab3Nb/bA67ZUeWMS6/nwepUeacjpY=
github.com/luxfi/filesystem v0.0.1 h1:VZ6xMFKaAPBW/ddlMsDnI2G0VU1lV5rYaVcW5d+KwEY=
github.com/luxfi/filesystem v0.0.1/go.mod h1:OQVSU6XNwqrr1AI+MqkID2taHUclx7NYmmr3svgttec=
github.com/luxfi/geth v1.20.1 h1:QUGQr4AKvADjwMi7t8a0OfoyxShgEcI9pwie1jFYfm0=
+28
View File
@@ -106,6 +106,34 @@ var Apps = []App{
{Name: "catalogsync", Prefixes: []string{"/v1/catalogsync"}, Eager: true},
{Name: "webhooks", Prefixes: []string{"/v1/webhooks"}},
{Name: "ml", Prefixes: []string{"/v1/ml/health", "/v1/ml/models", "/v1/train/experiments", "/v1/train/health", "/v1/train/jobs"}},
// risk owns THREE things that are one thing: the decision plane (/v1/risk)
// and the NATIVE leaves of the model plane (/v1/ml/*), which the decision
// plane's own state backs.
//
// The ml leaves are on THIS row and not on ml's, and that is the sharpest
// structural point in the design. A manifest row is a BINARY, and the model
// is in-process MUTABLE state: one half-space forest per tenant, held as mass
// counters that every score reads and every train writes. If plugin/ml
// trained and plugin/risk scored, the two processes would hold DIFFERENT
// counters and there would be no error, no log and no 404 — just two
// different answers to one question. One owner of the state, one row.
//
// ml's row is unchanged and there is no conflict: it never claimed bare
// "/v1/ml", so /v1/ml/health and /v1/ml/models still reach it and
// /v1/train/* is untouched. Longest-prefix match separates the two exactly
// as it already separates storage's /v1/s3/buckets from provisioning's
// /v1/s3 — no route moves. zip refuses two owners for one prefix at compose
// time, which is the gate under all of this.
{Name: "risk", Prefixes: []string{
"/v1/risk",
"/v1/ml/features", "/v1/ml/restore", "/v1/ml/score", "/v1/ml/search",
"/v1/ml/snapshot", "/v1/ml/state", "/v1/ml/train",
// The scoring-quality leaves. They are on this row for the same reason
// the others are: a calibration maps THIS process's scores, and a replay
// reads the decisions this process wrote.
"/v1/ml/calibrate", "/v1/ml/calibration", "/v1/ml/evaluate",
"/v1/ml/learning", "/v1/ml/replay", "/v1/ml/replays",
}},
{Name: "usage", Prefixes: []string{"/v1/usage"}},
{Name: "leaderboard", Prefixes: []string{"/v1/usage/activity", "/v1/usage/leaderboard", "/v1/usage/rollup/backfill"}},
{Name: "crm", Prefixes: []string{"/v1/crm"}},
+1 -1
View File
@@ -23,7 +23,7 @@ var frozen = []string{
"dns", "domain", "prompts", "agents", "link", "wallets",
"x402", "deploy", "functions", "tracker", "templates", "blueprint",
"framework", "knowledge", "help", "content", "catalogsync", "webhooks",
"ml", "usage", "leaderboard", "crm", "marketing", "ads",
"ml", "risk", "usage", "leaderboard", "crm", "marketing", "ads",
"campaign", "validators", "social", "analytics", "git", "sync",
"visor", "venue", "captable", "code", "zt", "share",
"dataroom", "graph", "security", "integrations", "destinations", "cloudflare",
+3259
View File
File diff suppressed because it is too large Load Diff
+5 -4
View File
@@ -1,6 +1,6 @@
{
"paths": 1399,
"operations": 1968,
"paths": 1437,
"operations": 2013,
"products": {
"admin": 87,
"ads": 7,
@@ -19,7 +19,7 @@
"balancers": 4,
"base": 1,
"benchmark": 6,
"billing": 25,
"billing": 26,
"blueprint": 3,
"books": 25,
"bot": 11,
@@ -112,7 +112,7 @@
"mesh": 1,
"messages": 2,
"metrics": 4,
"ml": 7,
"ml": 22,
"models": 3,
"mq": 15,
"networks": 3,
@@ -141,6 +141,7 @@
"rerank": 1,
"research": 8,
"responses": 1,
"risk": 29,
"router": 40,
"run": 1,
"runner": 3,
+39
View File
@@ -0,0 +1,39 @@
package main
import (
"fmt"
"os"
"github.com/hanzoai/cloud"
"github.com/hanzoai/cloud/apps/risk"
)
// Standalone entry for the risk app.
//
// This is the app's OWN composition root: it links only its own subsystem and
// the cloud request tier, never the whole fleet, so the build is this one app
// and not the union the fused binary was. The light host loads it as a plugin;
// run directly it serves standalone. Its OpenAPI subset comes from
// `risk openapi`.
//
// Price is cloud.Metered and not a flat edge price. Every priced act here bills
// from the ResourceMeter on the unit it actually produced — one screen per
// decision, one run per search — and a flat per-request edge price would charge
// a second time for the same work.
//
// Shutdown is not optional for this app. The model is in-memory mutable state
// and cloud deploys strategy Recreate at one replica, so a rollout that did not
// snapshot would return every tenant to warming — and a warming model refuses to
// score, which reads as "clean" to anything that does not check the refusal.
func main() {
if err := cloud.Listen([]cloud.Plugin{{
Name: "risk",
Price: cloud.Metered,
Mount: risk.Mount,
Shutdown: risk.Shutdown,
OwnsHealth: true,
}}, []string{"risk"}); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
+1180
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff