lux/aml
2
0
Fork 0
mirror of https://github.com/luxfi/aml.git synced 2026-09-24 00:25:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

planes: the five surfaces a monitoring programme is operated through

Browse Source 
Lists, suppressions, activations, the field catalog and the model plane. Each is
its own package, each is Base-backed, and none of them can destroy anything —
disposal is pkg/retention's decision alone, per tenant, and every one of them has
a test that reads its own source for a call to Delete.

lists holds the institution's own allow and deny lists, by class, normalised once
so an address written two ways is one address. It reaches a rule as one term,
Listed(name, value), and it refuses both a list nobody declared and an empty
value: either would be a rule that is catalogued, reported as coverage, and
incapable of firing. Removing a value stops it matching and keeps the row, because
a list entry is the reason a payment was refused.

suppress holds the decisions to keep a detection out of a queue. Reason and
decider are required and neither is defaulted; a suppression naming neither a rule
nor a subject is refused, because that is a kill switch. Lifting keeps the row.
The narrowest cover wins, so a specific decision about one account is the one named
on the activation rather than the rule-wide one behind it.

watch writes every activation before anything happens to it, and marks the ones a
suppression covered or a fold matched rather than dropping them: a system that
discards what it decided not to show cannot say how much it is not showing. Action
is what the rule asked for, Response is what the plane concluded, and exactly two
declared mechanisms move it. A rung may only raise — lowering is a suppression by
another name — and elevation beats folding, because an activation that reached an
escalation rung is the escalation. The live feed is a convenience over the durable
rows; it drops rather than blocking ingest, and the drops are counted per tenant
and published on every poll.

dictionary reports what a tenant's payloads carry: declared fields discovered by
reflection over the transaction rather than from a second list, the tenant's own
Raw keys, fill, blindness, and a distinct count kept as a bitmap sketch — hashed
with the tenant, so two tenants' bitmaps cannot be compared, and saturating into
"no longer a count" rather than drifting down. No value is ever stored. It is
write-behind and says so: Pending is exactly what a restart would lose.

topology searches the space of model shapes over a tenant's own history, through
the detector's own learning path rather than a copy of it, and reports the learning
curve, the realised alert share against the stated appetite, where each candidate
went blind, and which features carried the signal. It has no store and imports
nothing that has one. It names NO winner where nothing has been judged: ranking
needs an outcome, and a ranking invented from unlabelled data would look like a
recommendation and be a preference. Ties go to the smaller model.

models keeps every search and every fitted state, because both are governed acts,
and adopts a fit into the live model with its decider recorded. The learned state
never leaves the store — mass counters describe where a tenant's activity is dense.
A fit installs only into a model of the same shape, which is what stops a tenant's
model being swapped underneath it by restoring a file.

Around them: anomaly.Learn exposes the assessment Assess was already computing and
throwing away, so a study of the model reads the same scores production does;
engine gains one provider and one vocabulary term; and the tenant key moves to
pkg/brand, where the brand registry is, so the five new planes hold the same shape
at their own write boundary instead of each growing an opinion about what a tenant
is. api/tenant.go's qualify and qualified are now aliases of it, not a second copy.

Every operation is func(ctx, org, *In) (*Out, error): the tenant is an argument and
never something the operation resolves for itself. api/typed.go is the whole of
what turns one into a route — query into the input for a GET, body for a POST, the
path bound last and winning — and api/planes.go is a routing table with no handler
bodies in it, which a test reads as source. The contract is the Go types and
nothing else, so a zip face over the same functions restates none of it.

Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
zeekay
2026-08-01 16:37:20 -07:00
co-authored by Hanzo Dev
parent 2d940e06bb
commit 7c314d0b30
36 changed files with 8633 additions and 62 deletions
Download Patch File Download Diff File Expand all files Collapse all files
LLM.md
+218 -1
View File
@@ -98,6 +98,23 @@ pkg/
replay/replay.go -- Rule sandbox: replays a candidate over real history
through the engine's own evaluator, writes nothing
webhook/webhook.go -- Signed delivery (HMAC-SHA256) with retry + dead-letter
lists/ -- The institution's own allow and deny lists, and the
one rule term over them: Listed(name, value)
suppress/ -- The decisions to keep a detection out of a queue:
reason, decider, window, and the narrowest-wins cover
watch/ -- Every activation as it fires, the rungs that raise a
response, the fold that marks a repeat, the live feed
dictionary/ -- What a tenant's payloads carry: declared fields by
reflection, the tenant's own Raw keys, fill, blindness
and a distinct count kept as a bitmap and never values
topology/ -- The space of model shapes, searched over a tenant's
own history. Pure: no store, nothing it could write to
models/ -- The record of every search and every fitted state, and
the one path that adopts one into the live model
api/typed.go -- One operation shape, two adapters: the tenant is an
argument, the input is one struct, the output another
api/planes.go -- The five planes on the router, and the one place
ingest meets them
api/routes.go -- /v1/aml/* HTTP routes + SanctionsStore on Base
api/records.go -- The one place retention, token and replay are joined
api/anomaly.go -- Model state for governance + candidate scoring
@@ -112,6 +129,8 @@ pkg/
api/brand.go -- GET /v1/aml/config: the brand identity of this Host
brand/brand.go -- Brand id and request Host -> issuer + domains
(HIP-0111; the same registry as hanzoai/cloud brand)
brand/tenant.go -- The tenant key <brand>/<org>: Qualify, Qualified and
the boundary check every record plane calls
```
## Behavioural detection
@@ -195,6 +214,24 @@ scores, learns and reports what it *would* have alerted on, changing nothing.
| GET | /v1/aml/catalog | Coverage: installed rules, their citations, and the stated gaps |
| GET | /v1/aml/health | Health check, 503 when records cannot be kept |
| GET | /v1/aml/config | Brand identity of the request's Host: brand, display name, issuer, domain |
| GET/POST | /v1/aml/lists | The tenant's declared lists / declare one |
| GET/POST | /v1/aml/lists/{name}/entries | Read a list / put values on it |
| POST | /v1/aml/lists/{name}/entries/remove | Take a value off; the row stays |
| GET | /v1/aml/lists/{name}/lookup | Is this value listed, and on what entry |
| GET/POST | /v1/aml/suppressions | The suppression ledger / declare one |
| POST | /v1/aml/suppressions/{id}/lift | End one, with its own reason and decider |
| GET/POST | /v1/aml/activations | The activation feed / record one |
| GET | /v1/aml/activations/rates | Per rule: fired, live, silenced, folded, elevated |
| GET/POST | /v1/aml/rungs | The declared ladder / declare a rung |
| POST | /v1/aml/rungs/{id}/retire | Retire one; the row stays |
| GET | /v1/aml/dictionary | What this tenant's payloads carry, and what is blind |
| GET/POST | /v1/aml/models/runs | Past searches / run one over this tenant's history |
| GET | /v1/aml/models/runs/{id} | One whole search: every trial, every curve |
| GET/POST | /v1/aml/models/fits | Fitted states / fit one under a shape |
| POST | /v1/aml/models/fits/{id}/adopt | Install a fit into the live model |
Every route above is `get(h, <plane>.<Op>)` or `post(h, <plane>.<Op>)` and nothing
else — see "The five planes" below.
Every route resolves its tenant through `Handler.Identity`. Exactly two do not:
`/v1/aml/config`, which names the issuer a caller needs in order to obtain a token
@@ -248,6 +285,171 @@ The isolation boundary is the deployment: each brand runs its own store, issuer
ingress host. Brand-qualifying the key is what makes that boundary something other
than the only thing standing between two institutions.
## The five planes
Five capabilities an institution operates its monitoring programme through, each
its own package, each a durable record plane, and none of them able to destroy
anything: disposal is `pkg/retention`'s decision alone (AMLR Art. 77), per tenant,
and every one of these packages has a test that reads its own source for a call to
`Delete`.
| Plane | Package | What it is | What it refuses |
|---|---|---|---|
| Lists | `pkg/lists` | The institution's own allow and deny lists, by class, reachable from a rule as `Listed(name, value)` | A list nobody declared, and an empty value — both would be a rule that is catalogued, reported as coverage, and incapable of firing |
| Suppression | `pkg/suppress` | The decisions to keep a detection out of a queue: reason, decider, window | A suppression naming neither a rule nor a subject: that is a kill switch, not a suppression |
| Watcher | `pkg/watch` | Every activation as it fires, what became of it, and a live feed over the durable rows | An activation with no subject — it would pool every anonymous detection under one imaginary customer |
| Dictionary | `pkg/dictionary` | What a tenant's payloads carry: fill, blindness, distinct-count, and the model's own inventory | Nothing. It is a diagnostic and must never be able to refuse a payment |
| Models | `pkg/topology` + `pkg/models` | The space of model shapes searched over a tenant's own history, and the record of every search and fit | A winner nobody can justify, and a fit installed into a model of another shape |
### Suppression is a marking, never a drop
A detection a suppression covers, or a repeat inside a fold window, is WRITTEN and
marked with the reason it did not reach a queue. `Action` is what the rule asked
for; `Response` is what the plane concluded. A monitoring system that discards what
it decided not to show cannot answer how much it is not showing, and "no alerts"
then means either a quiet institution or a silent control with no way to tell
which. Because the suppression's id is on every activation it covered, "how much
has this silenced" is a query — which is why nothing keeps a counter of it.
### Elevation and folding are one mechanism
A RUNG is a tenant's declared policy: when this rule has fired this many times on
one subject within this window, do this. `To: watch.Fold` marks the repeat as a
duplicate; `To: <action>` raises the response. A rung can only RAISE — lowering is
a suppression by another name, and a suppression is asked for a reason and a
decider that a rung is not asked for on every activation it touches. Elevation
beats folding, because an activation that reached an escalation rung IS the
escalation and not a repeat of one. With no rung declared nothing folds: silence is
a decision, never a default.
### The dictionary stores no value, ever
A distinct-value count is a bitmap sketch (linear counting, 4096 bits per field per
tenant), hashed with the tenant in the key so two tenants' bitmaps cannot be
compared to learn whether they share a customer. Identifiers belong in the retained
record plane, sealed and purpose-gated; a statistics table does not get a second
copy of them under a weaker regime. The estimate SATURATES — "at least this many,
and no longer a count" — rather than drifting down, because a cardinality that
silently stops rising reads as a field that stopped varying.
The catalog is write-behind: observations accumulate and are written on a cadence
and at shutdown. That is deliberate — a field statistic is not a compliance record,
and paying an indexed write per field per transaction would spend the ingest path's
latency on a diagnostic. What is not acceptable is silence about it, so `Pending`
is published: a restart loses exactly that many observations and the answer says
so.
### A search names no winner it cannot justify
Ranking model shapes needs an outcome, and the only honest one is whether a shape
separates the events a human judged productive from the events a human dismissed
(the area under the ROC curve of its scores against those dispositions). Where
nothing has been judged, `topology.Search` reports every trial and NO winner, with
the reason. A ranking invented from unlabelled data — most alerts, fewest alerts,
best-behaved threshold — would look like a recommendation and would be a
preference. Ties go to the smaller model: a shape that ties on evidence and costs
less is the better answer, and preferring the bigger one drifts the fleet upward
every time a search runs.
`pkg/topology` has no store and imports nothing that has one, exactly as
`pkg/replay` does not — and a test reads its imports to keep it that way. A trial
builds its own detector and its own aggregate store, uses them, and drops them.
`models.Fit` produces learned state, and `anomaly.Restore` will only install it
into a model of the SAME shape (the digest). That is a governance property, not a
limitation: a tenant's model shape cannot be swapped underneath it by restoring a
file. The learned state itself never leaves the store — mass counters describe
where a tenant's activity is dense, and handing them out over an API would publish
the shape of an institution's customer behaviour to whoever holds a token.
### One operation shape, two faces
Every operation on every plane is
```go
func(ctx context.Context, org string, in *In) (*Out, error)
```
— the tenant is an argument and never something the operation resolves for itself,
the input is one typed struct, the output is another. `pkg/api/typed.go` is the
whole of what turns one into an HTTP route: `get` binds the input from the query
string, `post` from the body, both driven by the same json tags, and the route's
own path parameters are bound last and win. `pkg/api/planes.go` is then a routing
table with no handler bodies in it, and a test reads it as source to keep it that
way.
The point is that the CONTRACT is the pair of Go types and nothing else. The cloud
mount wraps the same functions as zip typed operations — where the same In and Out
become the OpenAPI schema, the CLI, the SDKs and the MCP tools — with no line of
the contract restated. See "A native zip conversion" below for what that costs.
### The tenant key, and where it is checked
`brand.Qualify(brandID, org)` mints `<brand>/<org>` and `brand.Qualified` is
derived from it, so there is one definition of the shape. `pkg/api/tenant.go`'s
`qualify`/`qualified` are aliases of those, not a second implementation.
Every one of the five planes calls `brand.Tenant(org)` at its own boundary before
it writes. The check is in the planes and not only where the key is minted because
the identity comes from a seam the DEPLOYMENT supplies: this is where the value
crosses into a store index, and an unqualified org reaching one puts two brands'
institutions of the same name into one tenant. Each plane has a
`TestBareOrgIsRefused` over every operation and a `TestTenantIsolation` that uses
the same org name under a second brand — which is the case a bare-org regression
collides.
## A native zip conversion
The typed operations above were written so that the cloud mount is a wrap. What a
full native-zip conversion of this engine would take, measured against the code as
it stands:
**Already done, and it is most of the work.** The 19 plane routes are typed
operations over 24 In/Out struct pairs with json tags on every field. A zip face
over them is one line each:
```go
zip.Post(g, "/lists", ops.declare, zip.WithOperationID("amlDeclareList"), zip.WithTags("aml"))
```
where `ops.declare` reads the tenant from the validated principal and calls
`lists.Shelf.Declare` — the same function this repo's Base router calls. Nothing in
the contract is restated.
**What is left, and it is the older half.** The 19 pre-existing routes in
`api/routes.go` are `func(*core.RequestEvent) error` closures that decode an
anonymous struct inline, so each one's request shape exists only inside its
handler. Converting them is: lift each anonymous struct to a named In type, lift
each `e.JSON(...)` literal to a named Out type, and move the body to
`func(ctx, org, *In) (*Out, error)`. `POST /v1/aml/transactions` is the largest —
its In is the embedded `types.Transaction` plus `relationship` and `entity`, and
its Out is the anonymous `EvalResult`+`record` struct at routes.go. Roughly 19
types to name and 19 handler bodies to move; no logic changes and no behaviour
changes.
**Three things a conversion must decide rather than discover.**
1. **Schema names are flat across the fleet.** Every type this repo would publish
needs a prefix — `amlDeclareListIn`, `amlActivation`, `mlSearchRun` — because a
weave refuses one name with two shapes and binds an SDK to whichever it read
last. The plane types here are currently named for their package
(`lists.DeclareIn`, `watch.DeclareIn`) and are distinct only by import path,
which a flat namespace does not have.
2. **`GET /v1/aml/health` cannot be a typed operation.** A real probe answers 503
carrying the degraded report AS ITS BODY (`readiness.go`), and a typed operation
reaches a non-2xx only by returning an error, which renders as the framework's
envelope and drops the report. It stays untyped, deliberately, and belongs on
whatever closed list the mount keeps of routes that are untyped by design.
3. **The tenant must come from the principal, not from an In field.** Every
operation here already takes `org` as its first argument for exactly this
reason. A conversion that put the org on the In struct would make it
caller-supplied, which is a cross-tenant read the caller asserted for itself.
**What does not convert.** `watch.Shelf.Subscribe` is a channel, not a request. A
live feed is a transport concern (server-sent events, or a ZAP stream) over the
same durable rows that `GET /v1/aml/activations` pages; it is not a typed operation
and should not be made to look like one.
## Records, tokenisation, and the sandbox
Three obligations, three packages, one join in `pkg/api/records.go`.
@@ -310,6 +512,11 @@ processed.
| `AML_BUSINESS_ZONE` | Zone business-day and business-hour rules are answered in. Default UTC |
| `AML_ANOMALY` | `live` leaves shadow mode. Anything else scores without contributing |
The five planes take no configuration. A list, a suppression and a rung are all
tenant DECLARATIONS carrying a reason and a decider, so none of them is a switch an
operator flips in a manifest — which is the point: a control that can be turned off
by an environment variable has no record of who turned it off.
## Console (`ui/`)
A compiled SPA, its own image, its own host. `ghcr.io/luxfi/aml-ui` is
@@ -623,6 +830,11 @@ record-keeping rules make an exception for.
| Retained records | `aml_retention`, `aml_retention_parties` | `retention.Ensure` + `retention.NewBase` | `TestRetainedRecordsSurviveARestart` |
| Cases + timelines | `aml_cases`, `aml_case_events` | `cases.Ensure` + `cases.NewBase` | `TestCasesSurviveARestart` |
| Alerts | `aml_alerts` | `api.EnsureAlerts` + `api.NewAlertStoreBase` | `TestAlertsSurviveARestart` |
| Lists | `aml_lists`, `aml_list_entries` | `lists.Ensure` + `lists.NewBase` | `lists.TestRestart` |
| Suppressions | `aml_suppressions` | `suppress.Ensure` + `suppress.NewBase` | `suppress.TestRestart` |
| Activations + rungs | `aml_activations`, `aml_rungs` | `watch.Ensure` + `watch.NewBase` | `watch.TestRestart` |
| Field catalog | `aml_fields`, `aml_payloads` | `dictionary.Ensure` + `dictionary.NewBase` | `dictionary.TestRestart` |
| Model runs + fits | `aml_model_runs`, `aml_model_fits` | `models.Ensure` + `models.NewBase` | `models.TestRestart` |
Each test writes, copies the data directory as it stands, drops the instance,
opens a second one over those bytes, and asks again — including that the tenant
@@ -662,11 +874,16 @@ v0.3.x tag before v0.3.6 is a tag with nothing behind it for these reasons.
## Test Coverage
366 Go tests across 18 packages, all green under `-race`, plus 12 browser tests.
Go tests across 24 packages, all green under `-race`, plus 12 browser tests.
- api: 52 · retention: 43 · cases: 40 · sanctions: 39 · engine: 38 · anomaly: 27
- measure: 26 · screen: 15 · token: 14 · replay: 14 · rules: 13 · velocity: 11
- brand: 8 · chain: 7 · workflows: 7 · webhook: 5 · store: 4 · history: 3
- lists · suppress · watch · dictionary · topology · models — each carries a
`TestTenantIsolation` (the same org name under a second brand), a
`TestBareOrgIsRefused` over every operation, a `TestNothingDeletes` that reads
the package's own source, and a `TestRestart` that opens a second instance over
the first one's bytes (`internal/instance`)
The browser suite is `ui/e2e`, run by `npm --prefix ui run e2e` and by CI. It
builds the bundle, serves it under `ui/csp.txt`, signs in before the first
cmd/amld/main.go
+59 -7
View File
@@ -28,14 +28,19 @@ import (
"github.com/luxfi/aml/pkg/anomaly"
"github.com/luxfi/aml/pkg/api"
"github.com/luxfi/aml/pkg/cases"
"github.com/luxfi/aml/pkg/dictionary"
"github.com/luxfi/aml/pkg/engine"
"github.com/luxfi/aml/pkg/history"
"github.com/luxfi/aml/pkg/lists"
"github.com/luxfi/aml/pkg/models"
"github.com/luxfi/aml/pkg/reference"
"github.com/luxfi/aml/pkg/retention"
"github.com/luxfi/aml/pkg/rules"
"github.com/luxfi/aml/pkg/screen"
"github.com/luxfi/aml/pkg/suppress"
"github.com/luxfi/aml/pkg/token"
"github.com/luxfi/aml/pkg/velocity"
"github.com/luxfi/aml/pkg/watch"
)
var version = "(dev)"
@@ -97,7 +102,7 @@ func main() {
// endpoints read it; there is no second store for either to talk to instead —
// two stores is how sanctions search came to answer "no match" for every name
// while the refresh was loading designations somewhere else.
lists := screen.New(listStale, nil)
screening := screen.New(listStale, nil)
// Per-source readiness. The store answers whether screening can be relied on at
// all; this answers which publisher is behind it, with a count and a date,
@@ -168,11 +173,34 @@ func main() {
return fmt.Errorf("refusing to start, alerts cannot be recorded: %w", err)
}
// The five planes the monitoring programme is operated through:
// the institution's own lists, the suppressions it has decided,
// every activation as it fires, the catalog of what its payloads
// carry, and the studies of its model. Every one is Base-backed
// for the same reason the three above are — a control that empties
// on a rollout is a control that was off, and from the outside a
// silent one is indistinguishable from a quiet institution.
for _, ensure := range []func(core.App) error{
lists.Ensure, suppress.Ensure, watch.Ensure, dictionary.Ensure, models.Ensure,
} {
if err := ensure(app); err != nil {
return fmt.Errorf("refusing to start, a record plane cannot be created: %w", err)
}
}
deny := lists.NewBase(app)
silence := suppress.NewBase(app)
monitor := watch.NewBase(app)
monitor.Cover = silence
catalog := dictionary.NewBase(app)
study := models.NewBase(app)
study.Model = model
rates := reference.RatesFromEnv()
eng := engine.New(engine.Providers{
History: events,
Screen: lists,
Lists: deny,
Screen: screening,
Reference: reference.JurisdictionsFromEnv(),
Rate: rates,
Zone: zone,
@@ -206,7 +234,7 @@ func main() {
ClientID: client,
Cases: cases.NewBase(app),
Alerts: api.NewAlertStoreBase(app),
Screen: lists,
Screen: screening,
Readiness: readiness,
History: events,
Rate: rates,
@@ -214,10 +242,34 @@ func main() {
Keys: keys,
Velocity: windows,
Anomaly: model,
Planes: api.Planes{
Lists: deny, Suppress: silence, Watch: monitor,
Dictionary: catalog, Models: study,
},
}
// The model plane reads a tenant's history through the same join a
// rule replay uses, so a study of the model and a study of a rule
// see the same events.
study.History = api.Replayed{H: handler}
handler.Register(se)
refresh(app, lists, readiness)
// The field catalog accumulates in memory and is written on a
// cadence and at shutdown. Both are wired here rather than left to
// a caller: an accumulator nobody flushes is a statistic that only
// ever reads as pending.
app.Cron().Add("dictionary-flush", "*/5 * * * *", func() {
if err := catalog.Flush(context.Background()); err != nil {
app.Logger().Error("field catalog flush failed", "error", err)
}
})
app.OnTerminate().BindFunc(func(te *core.TerminateEvent) error {
if err := catalog.Flush(context.Background()); err != nil {
app.Logger().Error("field catalog flush failed at shutdown", "error", err)
}
return te.Next()
})
refresh(app, screening, readiness)
// Destroy records whose retention period has run out, daily.
retention.Cron(app, records)
@@ -240,7 +292,7 @@ func main() {
// refresh registers the daily sanctions load and runs one immediately, so the
// process does not serve a whole day with nothing loaded.
func refresh(app core.App, lists *screen.Store, readiness *screen.Readiness) {
func refresh(app core.App, screening *screen.Store, readiness *screen.Readiness) {
load := func() {
results := screen.Fetch(context.Background())
for i := range results {
@@ -249,7 +301,7 @@ func refresh(app core.App, lists *screen.Store, readiness *screen.Readiness) {
app.Logger().Error("sanctions list load failed", "source", r.Source, "error", r.Err)
continue
}
if err := lists.Load(r.Source, r.Entries); err != nil {
if err := screening.Load(r.Source, r.Entries); err != nil {
// Recorded on the result too, so readiness reports a rejected list as
// unfit rather than as a successful load of nothing.
r.Err = err
@@ -259,7 +311,7 @@ func refresh(app core.App, lists *screen.Store, readiness *screen.Readiness) {
app.Logger().Info("sanctions list loaded", "source", r.Source, "designations", len(r.Entries))
}
readiness.Record(results, time.Now().UTC())
if err := lists.Ready(); err != nil {
if err := screening.Ready(); err != nil {
app.Logger().Error("screening is not ready after a refresh", "error", err)
}
}
internal/instance/instance.go
+84
View File
@@ -0,0 +1,84 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// Package instance opens a second Base app over a copy of a first one's data
// directory — which is what a restarted pod reads, and the only honest way to
// test that a record plane is durable.
//
// It is one helper because there is one question: is the row still there when the
// process that wrote it is gone. A test that reopens the SAME app answers a
// different and much weaker question, and a per-package copy of this walk is five
// chances for one of them to quietly reopen the same handle.
package instance
import (
"io"
"os"
"path/filepath"
"testing"
"github.com/hanzoai/base/tests"
)
// New opens a fresh instance over an empty data directory.
func New(t testing.TB) *tests.TestApp {
t.Helper()
app, err := tests.NewTestApp()
if err != nil {
t.Fatalf("instance: %v", err)
}
return app
}
// Restart takes the first instance's data directory as it stands, drops the
// instance, and opens a second one over those bytes.
//
// The first app is cleaned up here rather than left to the caller, because a test
// that keeps both open is testing two handles on one file and not a restart.
func Restart(t testing.TB, first *tests.TestApp) *tests.TestApp {
t.Helper()
dir := copyTree(t, first.DataDir())
first.Cleanup()
second, err := tests.NewTestApp(dir)
if err != nil {
t.Fatalf("instance: second instance: %v", err)
}
t.Cleanup(second.Cleanup)
return second
}
func copyTree(t testing.TB, src string) string {
t.Helper()
dst := t.TempDir()
err := filepath.WalkDir(src, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
rel, err := filepath.Rel(src, path)
if err != nil {
return err
}
target := filepath.Join(dst, rel)
if d.IsDir() {
return os.MkdirAll(target, 0o755)
}
in, err := os.Open(path)
if err != nil {
return err
}
defer in.Close()
out, err := os.Create(target)
if err != nil {
return err
}
defer out.Close()
_, err = io.Copy(out, in)
return err
})
if err != nil {
t.Fatalf("instance: copy data dir: %v", err)
}
return dst
}
pkg/anomaly/anomaly.go
+15 -1
View File
@@ -307,7 +307,7 @@ func (s *Store) Config() Config {
// every baseline in the feature set has the transaction removed from it
// arithmetically so that nothing is measured against itself.
func (s *Store) Assess(tx types.Transaction, entity types.Entity) (types.RuleHit, bool) {
a := s.judge(tx, true)
a := s.Learn(tx, entity)
if !a.Alert {
return types.RuleHit{}, false
}
@@ -330,6 +330,20 @@ func (s *Store) Assess(tx types.Transaction, entity types.Entity) (types.RuleHit
}, true
}
// Learn scores one transaction, lets the model learn from it, and returns the
// whole assessment.
//
// It is what Assess does, with the answer not yet reduced to the engine's Scorer
// shape. The two cannot disagree about what the model did, because Assess is
// defined in terms of this and there is no second scoring path — which is the
// property that lets a study of the model (pkg/topology) replay a tenant's own
// history through the code that scores production, and read the score of every
// event as the model learned it, rather than through a copy of the algorithm
// that would be free to be wrong.
func (s *Store) Learn(tx types.Transaction, entity types.Entity) Assessment {
return s.judge(tx, true)
}
// Inspect scores one transaction without learning from it, moving any counter, or
// touching the aggregate store. It is how a candidate is tried against a
// tenant's real behaviour before anything depends on the answer, and it is the
pkg/api/monitor_test.go
+233
View File
@@ -0,0 +1,233 @@
package api
import (
"context"
"encoding/json"
"net/http"
"testing"
"time"
"github.com/hanzoai/base/core"
"github.com/luxfi/aml/internal/instance"
"github.com/luxfi/aml/pkg/dictionary"
"github.com/luxfi/aml/pkg/engine"
"github.com/luxfi/aml/pkg/lists"
"github.com/luxfi/aml/pkg/reference"
"github.com/luxfi/aml/pkg/suppress"
"github.com/luxfi/aml/pkg/types"
"github.com/luxfi/aml/pkg/watch"
)
// The ingest path meets the five planes here, and this is where that meeting is
// checked end to end: a rule fires, an activation is written, a declared
// suppression changes the answer the caller gets, and the field catalog sees the
// payload.
//
// The planes' own tests establish that each one is correct. This one establishes
// that ingest is WIRED to them — the failure it exists to catch is five correct
// planes that nothing calls.
func monitored(t *testing.T) *Handler {
t.Helper()
app := instance.New(t)
t.Cleanup(app.Cleanup)
for _, ensure := range []func(core.App) error{lists.Ensure, suppress.Ensure, watch.Ensure, dictionary.Ensure} {
if err := ensure(app); err != nil {
t.Fatalf("ensure: %v", err)
}
}
silence := suppress.NewBase(app)
monitor := watch.NewBase(app)
monitor.Cover = silence
h := plane(t)
h.Planes = Planes{
Lists: lists.NewBase(app),
Suppress: silence,
Watch: monitor,
Dictionary: dictionary.NewBase(app),
}
return h
}
// payment is the wire shape the ingest route takes: the transaction at the top
// level, with the customer beside it.
func payment(id string, usd float64) map[string]any {
return map[string]any{
"id": id, "user_id": "u1", "account_id": "acct-1",
"currency": "USD", "notional": usd, "timestamp": time.Now().UTC(),
"entity": map[string]any{"id": "u1", "name": "Acme Trading"},
}
}
// TestIngestRecordsWhatTheRulesDid.
func TestIngestRecordsWhatTheRulesDid(t *testing.T) {
h := monitored(t)
ctx := context.Background()
rec := ingest(t, h, payment("tx-1", 25_000))
if rec.Code != http.StatusOK {
t.Fatalf("ingest = %d: %s", rec.Code, rec.Body.String())
}
var out types.EvalResult
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatal(err)
}
if out.Action != types.ActionReport {
t.Fatalf("the rule asked for report, got %q", out.Action)
}
feed, err := h.Planes.Watch.Feed(ctx, acme, &watch.FeedIn{})
if err != nil {
t.Fatal(err)
}
if len(feed.Activations) != 1 {
t.Fatalf("the rule fired and nothing recorded it: %+v", feed.Activations)
}
a := feed.Activations[0]
if a.Rule != "ctr" || a.Tx != "tx-1" || a.Subject.Value != "acct-1" || a.Subject.Kind != "account" {
t.Fatalf("the activation does not describe what happened: %+v", a)
}
if a.Action != types.ActionReport || a.Response != types.ActionReport {
t.Fatalf("with nothing declared, the response is what the rule asked for: %+v", a)
}
// And the payload reached the catalog.
cat, err := h.Planes.Dictionary.Catalog(ctx, acme, &dictionary.CatalogIn{})
if err != nil {
t.Fatal(err)
}
if cat.Payloads != 1 {
t.Fatalf("the field catalog did not see the payload: %d", cat.Payloads)
}
}
// TestADeclaredSuppressionChangesTheAnswer is the whole of what suppression does,
// exercised where a customer would feel it.
func TestADeclaredSuppressionChangesTheAnswer(t *testing.T) {
h := monitored(t)
ctx := context.Background()
if _, err := h.Planes.Suppress.Suppress(ctx, acme, &suppress.SuppressIn{
Rule: "ctr", Kind: "account", Value: "acct-1",
Reason: "treasury sweep, agreed with the MLRO 2026-02", By: "a.mensah",
}); err != nil {
t.Fatal(err)
}
rec := ingest(t, h, payment("tx-2", 25_000))
if rec.Code != http.StatusOK {
t.Fatalf("ingest = %d: %s", rec.Code, rec.Body.String())
}
var out types.EvalResult
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatal(err)
}
if out.Action != types.ActionAllow {
t.Fatalf("a covered detection must not reach the caller as a report: %q", out.Action)
}
// The alert itself is still evidence and is still there.
if len(out.AlertIDs) != 1 {
t.Fatalf("suppression must not destroy the alert: %+v", out.AlertIDs)
}
feed, err := h.Planes.Watch.Feed(ctx, acme, &watch.FeedIn{})
if err != nil {
t.Fatal(err)
}
if len(feed.Activations) != 1 {
t.Fatalf("a suppressed detection is recorded, not dropped: %+v", feed.Activations)
}
a := feed.Activations[0]
if !a.Suppressed || a.Cause != watch.CauseSuppressed || a.By == "" {
t.Fatalf("the activation must name the suppression that covered it: %+v", a)
}
if a.Action != types.ActionReport {
t.Fatalf("what the rule asked for is still on the row: %+v", a)
}
}
// TestADeclaredRungRaisesTheAnswer.
func TestADeclaredRungRaisesTheAnswer(t *testing.T) {
h := monitored(t)
ctx := context.Background()
if _, err := h.Planes.Watch.Declare(ctx, acme, &watch.DeclareIn{
Rule: "ctr", Kind: "account", Count: 2, Within: watch.Span(time.Hour), To: types.ActionBlock,
Reason: "two reportable transactions in an hour on one account", By: "a.mensah",
}); err != nil {
t.Fatal(err)
}
if rec := ingest(t, h, payment("tx-3", 25_000)); rec.Code != http.StatusOK {
t.Fatalf("ingest = %d: %s", rec.Code, rec.Body.String())
}
rec := ingest(t, h, payment("tx-4", 25_000))
if rec.Code != http.StatusOK {
t.Fatalf("ingest = %d: %s", rec.Code, rec.Body.String())
}
var out types.EvalResult
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatal(err)
}
if out.Action != types.ActionBlock {
t.Fatalf("the declared escalation did not reach the caller: %q", out.Action)
}
}
// TestARuleReadsTheTenantsOwnDenyList — the list plane reaching the rule
// vocabulary, through the engine's own evaluator.
func TestARuleReadsTheTenantsOwnDenyList(t *testing.T) {
h := monitored(t)
ctx := context.Background()
if _, err := h.Planes.Lists.Declare(ctx, acme, &lists.DeclareIn{
Name: "ip-deny", Kind: lists.Deny, Class: lists.IP, By: "a.mensah",
}); err != nil {
t.Fatal(err)
}
if _, err := h.Planes.Lists.Add(ctx, acme, &lists.AddIn{
Name: "ip-deny", By: "a.mensah",
Values: []lists.Value{{Value: "198.51.100.7", Reason: "credential stuffing source"}},
}); err != nil {
t.Fatal(err)
}
// A rule over the list, admitted through the engine's own vocabulary check —
// which is what refuses the rule outright if the provider is not wired.
h.Engine = engine.New(engine.Providers{
Rate: reference.Rates{USDPer: map[string]float64{}},
Lists: h.Planes.Lists,
Zone: time.UTC,
})
if err := h.Engine.SetRules([]types.Rule{{
ID: "denied-address", Name: "Address on the deny list",
DSL: `Tx.IPAddress != "" && Listed("ip-deny", Tx.IPAddress)`,
Severity: types.SeverityHigh, Weight: 0.4, Action: types.ActionBlock, Enabled: true,
}}); err != nil {
t.Fatalf("the deny-list rule must install where the list plane is wired: %v", err)
}
body := payment("tx-5", 100)
body["ip_address"] = "198.51.100.7"
rec := ingest(t, h, body)
if rec.Code != http.StatusOK {
t.Fatalf("ingest = %d: %s", rec.Code, rec.Body.String())
}
var out types.EvalResult
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatal(err)
}
if out.Action != types.ActionBlock {
t.Fatalf("a payment from a listed address must be blocked, got %q", out.Action)
}
// And an address nobody listed passes.
clean := payment("tx-6", 100)
clean["ip_address"] = "203.0.113.9"
rec = ingest(t, h, clean)
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatal(err)
}
if out.Action != types.ActionAllow {
t.Fatalf("an address nobody listed must pass, got %q", out.Action)
}
}
pkg/api/planes.go
+164
View File
@@ -0,0 +1,164 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package api
// The five record planes this file puts on the router, and the one place ingest
// meets them.
//
// Every route here is `post(h, <plane>.<Op>)` or `get(h, <plane>.<Op>)` and
// nothing else. There is no handler with a body: the operation is the contract,
// the adapter is in typed.go, and this file is a routing table. That is what makes
// the cloud mount a wrap of the same functions rather than a second
// implementation of them — a route that unpacked fields here would have to be
// unpacked again there, differently, and the difference is what a customer finds.
import (
"context"
"log"
"github.com/hanzoai/base/core"
"github.com/luxfi/aml/pkg/dictionary"
"github.com/luxfi/aml/pkg/lists"
"github.com/luxfi/aml/pkg/models"
"github.com/luxfi/aml/pkg/replay"
"github.com/luxfi/aml/pkg/suppress"
"github.com/luxfi/aml/pkg/types"
"github.com/luxfi/aml/pkg/watch"
)
// Planes are the five record planes the Handler serves alongside the engine.
//
// Each is optional and each is nil-checked at registration, so a deployment that
// has not wired one serves no route for it — which is the honest state. A route
// that answered an empty list for a plane nobody wired would report an
// institution with no deny lists and no suppressions, and that is exactly the
// reading a reviewer must never be handed by accident.
type Planes struct {
Lists *lists.Shelf
Suppress *suppress.Shelf
Watch *watch.Shelf
Dictionary *dictionary.Shelf
Models *models.Shelf
}
// registerPlanes puts the planes on the router. Called from Register.
func (h *Handler) registerPlanes(se *core.ServeEvent) {
if l := h.Planes.Lists; l != nil {
se.Router.GET("/v1/aml/lists", get(h, l.Catalog))
se.Router.POST("/v1/aml/lists", post(h, l.Declare, true))
se.Router.GET("/v1/aml/lists/{name}/entries", get(h, l.Entries))
se.Router.POST("/v1/aml/lists/{name}/entries", post(h, l.Add, false))
se.Router.POST("/v1/aml/lists/{name}/entries/remove", post(h, l.Remove, false))
se.Router.GET("/v1/aml/lists/{name}/lookup", get(h, l.Lookup))
}
if s := h.Planes.Suppress; s != nil {
se.Router.GET("/v1/aml/suppressions", get(h, s.Ledger))
se.Router.POST("/v1/aml/suppressions", post(h, s.Suppress, true))
se.Router.POST("/v1/aml/suppressions/{id}/lift", post(h, s.Lift, false))
}
if w := h.Planes.Watch; w != nil {
se.Router.GET("/v1/aml/activations", get(h, w.Feed))
se.Router.POST("/v1/aml/activations", post(h, w.Record, true))
se.Router.GET("/v1/aml/activations/rates", get(h, w.Rates))
se.Router.GET("/v1/aml/rungs", get(h, w.Ladder))
se.Router.POST("/v1/aml/rungs", post(h, w.Declare, true))
se.Router.POST("/v1/aml/rungs/{id}/retire", post(h, w.Retire, false))
}
if d := h.Planes.Dictionary; d != nil {
se.Router.GET("/v1/aml/dictionary", get(h, d.Catalog))
}
if m := h.Planes.Models; m != nil {
se.Router.GET("/v1/aml/models/runs", get(h, m.Runs))
se.Router.POST("/v1/aml/models/runs", post(h, m.Search, true))
se.Router.GET("/v1/aml/models/runs/{id}", get(h, m.Run))
se.Router.GET("/v1/aml/models/fits", get(h, m.Fits))
se.Router.POST("/v1/aml/models/fits", post(h, m.Fit, true))
se.Router.POST("/v1/aml/models/fits/{id}/adopt", post(h, m.Adopt, false))
}
}
// monitor records what the rules did to a transaction and returns the response
// the monitoring plane concluded.
//
// This is the one place suppression and elevation reach an outcome. Both are
// declared decisions of the institution — a suppression carries a reason and a
// decider, a rung carries both as well — so the transaction's action is the
// strongest RESPONSE across the activations, not the strongest action the rules
// asked for. That difference is the whole of what suppression does, and it is
// visible on every row: Action is what the rule wanted, Response is what happened.
//
// A failure to record is a refusal to answer, on the same terms as a failure to
// retain: a control whose activations are not being recorded cannot be shown to
// have run, and processing a transaction it cannot account for is worse than
// declining it.
func (h *Handler) monitor(ctx context.Context, org string, tx types.Transaction, alerts []types.Alert, action string) (string, error) {
w := h.Planes.Watch
if w == nil || len(alerts) == 0 {
return action, nil
}
// The subject the activation is about: the account the aggregates are kept
// under, falling back to the user, which is the same choice the model makes.
subject := watch.Subject{Kind: "account", Value: tx.AccountID}
if subject.Value == "" {
subject = watch.Subject{Kind: "user", Value: tx.UserID}
}
if subject.Value == "" {
// Nothing to attribute the activations to. The rules' own verdict stands;
// recording an activation against an empty subject would pool every
// anonymous transaction in the tenant under one imaginary customer, which
// is the failure anomaly refuses by name.
return action, nil
}
best := types.ActionAllow
for _, a := range alerts {
rec, err := w.Record(ctx, org, &watch.RecordIn{
Rule: a.RuleID, RuleName: a.RuleName, Typology: a.Typology,
Severity: a.Severity, Action: a.ActionTaken, Score: a.Score,
Tx: tx.ID, Subject: subject, At: tx.Timestamp,
})
if err != nil {
return "", err
}
if types.ActionRank(rec.Response) > types.ActionRank(best) {
best = rec.Response
}
}
return best, nil
}
// observe offers a transaction to the field catalog.
//
// A failure is logged and nothing else. The catalog is a diagnostic — nothing is
// decided on it — so a statistic that could not be taken must not be able to
// refuse a payment. That is the opposite of the rule for every record plane
// above, and the difference is exactly whether an obligation rests on it.
func (h *Handler) observe(org string, tx types.Transaction) {
if d := h.Planes.Dictionary; d != nil {
if err := d.Observe(org, tx); err != nil {
log.Printf("[aml] dictionary: %v", err)
}
}
}
// Replayed hands a tenant's retained transactions to a model search.
//
// It is the models plane's Source, satisfied by the SAME join that answers a rule
// replay — one history seam, so a study of the model and a study of a rule see
// the same events and their answers can be held against each other.
type Replayed struct{ H *Handler }
// History returns this tenant's replayable events, opened under the monitoring
// purpose. Testing a shape before activation is monitoring work and not a
// commercial read (AMLD4 Art. 41(2)), which is what entitles this read to the
// sealed bodies at all.
func (r Replayed) History(ctx context.Context, org string) (replay.History, error) {
vault, err := r.H.vault(org)
if err != nil {
return nil, err
}
return r.H.history(org, vault)
}
pkg/api/routes.go
+18
View File
@@ -66,6 +66,12 @@ type Handler struct {
// Anomaly scores whether a transaction is unusual for the entity, alongside
// the rules rather than instead of them. Nil runs on rules alone.
Anomaly *anomaly.Store
// Planes are the five record planes served alongside the engine: lists,
// suppressions, activations, the field catalog and the model plane. Each is
// optional and an absent one serves no route, because an empty answer from a
// plane nobody wired reads as an institution with no controls.
Planes Planes
// ClientID is this deployment's IAM application — the audience every token
// is pinned to. It is served by /v1/aml/config so the console reads its own
// identity from the engine that enforces it.
@@ -216,6 +222,7 @@ func (h *Handler) Register(se *core.ServeEvent) {
se.Router.GET("/v1/aml/catalog", h.catalog())
se.Router.GET("/v1/aml/config", h.brandConfig())
se.Router.GET("/v1/aml/health", h.health())
h.registerPlanes(se)
}
// refuse answers an unauthenticated request. The reason is logged for the
@@ -410,6 +417,17 @@ func (h *Handler) ingestTransaction() func(e *core.RequestEvent) error {
}
h.Alerts.Add(tx.ID, alerts)
// The monitoring plane records what fired and decides what the response
// is: a suppression the institution declared lowers it, a rung the
// institution declared raises it, and every activation is written either
// way. It runs after the record is retained and after the alerts are
// stored, so the activation cannot outlive the evidence it names.
action, err = h.monitor(e.Request.Context(), orgID, tx, alerts, action)
if err != nil {
return unavailable(e, "record activations", err)
}
h.observe(orgID, tx)
result := struct {
types.EvalResult
Record string `json:"record"`
pkg/api/tenant.go
+13 -53
View File
@@ -31,60 +31,20 @@ type Identity func(*http.Request) (tenant string, err error)
var ErrNoIdentity = errors.New("no identity configured, so no request can be attributed to a tenant")
// sep divides a tenant key's two halves.
const sep = "/"
const sep = brand.Sep
// qualify builds a tenant key from the brand whose issuer vouched for the caller
// and the org that caller acts for.
//
// It is one string because it is one key. The same value is the store index for
// alerts, cases and retained records, the org column on a history row, and the
// HKDF salt every tokenisation key is derived from (pkg/token New: salt =
// "aml/token/org/" + org). An org name is unique within an issuer and not across
// issuers, so the bare org is not an identity: `acme` on lux.id and `acme` on
// zoolabs.id are two unrelated financial institutions. Keyed on the bare org they
// are one — one set of rows, and worse, one vault: the same salt derives the same
// keys, so one brand's customer names tokenise to the other's pseudonyms and its
// sealed records open under the other's tenant. That is a cross-brand plaintext
// disclosure of retained personal data, against the purpose limitation retention
// is held to (Directive (EU) 2015/849 Art. 41(2)) and against HIP-0302, which is
// the requirement that the determinism stops at the tenant boundary.
//
// Qualifying at the key, rather than filtering on the brand afterwards, is what
// makes that leak uncomputable instead of merely disallowed: there is no query
// that returns the other brand's rows and no key that opens its vault. It is done
// here, before any record exists, because doing it later re-keys every vault —
// pseudonyms and sealed bodies both.
func qualify(brandID, org string) (string, error) {
b, ok := brand.For(brandID)
if !ok {
return "", fmt.Errorf("no brand is registered as %q, so nothing vouches for this tenant", brandID)
}
org = strings.TrimSpace(org)
if org == "" {
return "", errors.New("no org, so the request acts for no tenant")
}
// Brand ids are a closed set and none contains the separator, so the first one
// always ends the brand and the key is unambiguous either way. The org half is
// held to it so the key stays readable back to the org it names: an operator
// reading a retention row or a vault refusal has to be able to say which
// institution it belongs to, and `zoo/lux/acme` does not say.
if strings.Contains(org, sep) {
return "", fmt.Errorf("org %q contains %q, so the tenant it names is not readable back", org, sep)
}
return b.ID + sep + org, nil
}
// qualified reports whether a key is one qualify would have produced. Derived from
// qualify rather than stated again, so there is one definition of the shape and a
// change to it cannot leave a validator behind.
func qualified(key string) bool {
brandID, org, found := strings.Cut(key, sep)
if !found {
return false
}
again, err := qualify(brandID, org)
return err == nil && again == key
}
// qualify and qualified are the tenant key, and they live in pkg/brand because
// the key is a brand fact: an org is only an identity once the brand whose issuer
// vouched for it is named. They are aliases rather than a second implementation
// so that the record planes below this package — lists, suppressions,
// activations, field statistics, model runs — hold the same shape at their own
// write boundary without any of them getting a private opinion about what a
// tenant is. See brand.Qualify for why the key is one string and what a bare org
// collides.
var (
qualify = brand.Qualify
qualified = brand.Qualified
)
// TrustedProxyHeader resolves the tenant's org from a header written by an
// authenticating proxy, and its brand from the request's own Host.
pkg/api/typed.go
+268
View File
@@ -0,0 +1,268 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package api
// One shape for every operation, and two adapters onto it.
//
// An operation is `func(ctx, org, *In) (*Out, error)`: the tenant is an argument
// and never something the operation resolves for itself, the input is one typed
// struct, and the output is another. Every record plane below this package
// exposes exactly that, and this file is the whole of what turns one into an HTTP
// route.
//
// The point is that the CONTRACT is the pair of Go types and nothing else. A
// second face over the same planes — the zip typed operations in the cloud mount,
// where the same In and Out become the OpenAPI schema, the CLI, the SDKs and the
// MCP tools — wraps the same functions with no line of the contract restated. A
// handler that unpacked fields by hand would be that second copy, and the copy is
// what drifts.
//
// GET binds its input from the query string and POST from the body, both driven
// by the same json tags on the same struct. There is one binder for each and no
// third way.
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
"net/url"
"reflect"
"strconv"
"strings"
"time"
"github.com/hanzoai/base/core"
"github.com/luxfi/aml/pkg/brand"
"github.com/luxfi/aml/pkg/dictionary"
"github.com/luxfi/aml/pkg/lists"
"github.com/luxfi/aml/pkg/models"
"github.com/luxfi/aml/pkg/suppress"
"github.com/luxfi/aml/pkg/topology"
"github.com/luxfi/aml/pkg/watch"
)
// op is an operation over one tenant's records.
type op[In, Out any] func(ctx context.Context, org string, in *In) (*Out, error)
// maxBody bounds a request body. A list import is the largest legitimate one and
// it is bounded at a thousand values, so a megabyte is generous for every
// operation here and small enough that a body cannot be the attack.
const maxBody = 1 << 20
// post adapts an operation to a route that reads its input from the body.
func post[In, Out any](h *Handler, fn op[In, Out], created bool) func(*core.RequestEvent) error {
return func(e *core.RequestEvent) error {
orgID, err := h.tenant(e)
if err != nil {
return refuse(e, err)
}
var in In
body := http.MaxBytesReader(e.Response, e.Request.Body, maxBody)
if err := json.NewDecoder(body).Decode(&in); err != nil && !errors.Is(err, io.EOF) {
return fail(e, http.StatusBadRequest, "invalid request body")
}
// The path is bound last and wins. A body naming a different list than
// the URL is a request whose meaning depends on which half the reader
// believes, and the URL is the one the router matched on.
if err := path(e.Request, &in); err != nil {
return fail(e, http.StatusBadRequest, err.Error())
}
out, err := fn(e.Request.Context(), orgID, &in)
if err != nil {
return answer(e, err)
}
code := http.StatusOK
if created {
code = http.StatusCreated
}
return e.JSON(code, out)
}
}
// get adapts an operation to a route that reads its input from the query string.
func get[In, Out any](h *Handler, fn op[In, Out]) func(*core.RequestEvent) error {
return func(e *core.RequestEvent) error {
orgID, err := h.tenant(e)
if err != nil {
return refuse(e, err)
}
var in In
if err := bind(e.Request.URL.Query(), &in); err != nil {
return fail(e, http.StatusBadRequest, err.Error())
}
if err := path(e.Request, &in); err != nil {
return fail(e, http.StatusBadRequest, err.Error())
}
out, err := fn(e.Request.Context(), orgID, &in)
if err != nil {
return answer(e, err)
}
return e.JSON(http.StatusOK, out)
}
}
// bind fills a typed input from a query string, by the same json tags the body
// decoder reads.
//
// A parameter that cannot be read as its field's type is an ERROR and never a
// zero value. `?limit=all` silently becoming limit 0 becoming the default is how
// a caller asks for everything and is handed a page, and `?live=yes` silently
// becoming false is how a filter turns itself off.
func bind(q url.Values, target any) error {
v := reflect.ValueOf(target).Elem()
t := v.Type()
for i := 0; i < t.NumField(); i++ {
name, _, _ := strings.Cut(t.Field(i).Tag.Get("json"), ",")
if name == "" || name == "-" {
continue
}
raw := strings.TrimSpace(q.Get(name))
if raw == "" {
continue
}
if err := set(v.Field(i), name, raw); err != nil {
return err
}
}
return nil
}
// path overlays the route's own parameters onto a typed input, by the same json
// tags. A field whose name the route does not carry as a parameter is untouched,
// so one binder serves every route and no route needs a list of which parameters
// it has.
func path(r *http.Request, target any) error {
v := reflect.ValueOf(target).Elem()
t := v.Type()
for i := 0; i < t.NumField(); i++ {
name, _, _ := strings.Cut(t.Field(i).Tag.Get("json"), ",")
if name == "" || name == "-" {
continue
}
raw := strings.TrimSpace(r.PathValue(name))
if raw == "" {
continue
}
if err := set(v.Field(i), name, raw); err != nil {
return err
}
}
return nil
}
func set(f reflect.Value, name, raw string) error {
if f.Type() == reflect.TypeOf(time.Time{}) {
at, err := time.Parse(time.RFC3339, raw)
if err != nil {
return fmt.Errorf("%s: %q is not an RFC 3339 instant", name, raw)
}
f.Set(reflect.ValueOf(at.UTC()))
return nil
}
switch f.Kind() {
case reflect.String:
f.SetString(raw)
case reflect.Bool:
b, err := strconv.ParseBool(raw)
if err != nil {
return fmt.Errorf("%s: %q is not true or false", name, raw)
}
f.SetBool(b)
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
n, err := strconv.ParseInt(raw, 10, 64)
if err != nil {
return fmt.Errorf("%s: %q is not a whole number", name, raw)
}
f.SetInt(n)
case reflect.Float32, reflect.Float64:
x, err := strconv.ParseFloat(raw, 64)
if err != nil {
return fmt.Errorf("%s: %q is not a number", name, raw)
}
f.SetFloat(x)
default:
return fmt.Errorf("%s cannot be given in a query string", name)
}
return nil
}
// answer turns a refusal into a status.
//
// The table is here, at the one place the domain meets the transport, rather than
// as a status code carried on each error — an error that knows its HTTP status is
// an error that has learned about HTTP, and the planes below this package have
// no business knowing there is any.
//
// A refusal the table does not name is a fault and not a bad request: it is
// logged with its detail and answered generically, because an unrecognised error
// is the one whose text nobody has read for what it might disclose.
func answer(e *core.RequestEvent, err error) error {
for _, known := range gone {
if errors.Is(err, known) {
return fail(e, http.StatusNotFound, err.Error())
}
}
for _, known := range taken {
if errors.Is(err, known) {
return fail(e, http.StatusConflict, err.Error())
}
}
for _, known := range refused {
if errors.Is(err, known) {
return fail(e, http.StatusBadRequest, err.Error())
}
}
for _, known := range broken {
if errors.Is(err, known) {
return unavailable(e, "record plane", err)
}
}
if errors.Is(err, brand.ErrTenant) {
// The identity resolved something that is not a tenant key. That is this
// deployment's configuration and not the caller's request, so it is a
// fault here and an unauthenticated request there.
return refuse(e, err)
}
log.Printf("[aml] %s: %v", e.Request.URL.Path, err)
return fail(e, http.StatusInternalServerError, "the request could not be completed")
}
// The refusal table. Every sentinel each plane publishes appears exactly once, so
// a plane that adds one and forgets it here answers 500 — loudly — rather than
// being quietly classified as a bad request.
var (
gone = []error{
lists.ErrNoList, lists.ErrNoEntry,
suppress.ErrNotHere,
watch.ErrNotHere,
models.ErrNotHere, models.ErrNoFit,
}
taken = []error{
lists.ErrExists, lists.ErrRetired,
suppress.ErrLifted,
watch.ErrRetired,
models.ErrAdopted,
}
refused = []error{
lists.ErrName, lists.ErrKind, lists.ErrClass, lists.ErrValue, lists.ErrEmpty,
lists.ErrDecider, lists.ErrReason, lists.ErrCrowded, lists.ErrMaxValues,
suppress.ErrReason, suppress.ErrDecider, suppress.ErrBroad, suppress.ErrKind,
suppress.ErrSubject, suppress.ErrWindow,
watch.ErrRule, watch.ErrSubject, watch.ErrKind, watch.ErrAction, watch.ErrTo,
watch.ErrCount, watch.ErrWithin, watch.ErrReason, watch.ErrDecider,
models.ErrDecider, models.ErrShape, models.ErrNoModel, models.ErrNoHistory,
topology.ErrEmptySpace, topology.ErrHuge, topology.ErrEmpty, topology.ErrShape,
topology.ErrOrg, topology.ErrNoHistory,
}
broken = []error{
lists.ErrStore, suppress.ErrStore, watch.ErrStore,
models.ErrStore, dictionary.ErrStore,
}
)
pkg/api/typed_test.go
+192
View File
@@ -0,0 +1,192 @@
package api
import (
"net/http"
"net/http/httptest"
"net/url"
"os"
"reflect"
"strings"
"testing"
"time"
"github.com/luxfi/aml/pkg/lists"
"github.com/luxfi/aml/pkg/models"
"github.com/luxfi/aml/pkg/suppress"
"github.com/luxfi/aml/pkg/topology"
"github.com/luxfi/aml/pkg/watch"
)
// TestQueryBindingRefusesRatherThanZeroing.
//
// `?limit=all` silently becoming 0 becoming the default is how a caller asks for
// everything and is handed a page; `?live=yes` silently becoming false is how a
// filter turns itself off. Both read as a successful request for something else.
func TestQueryBindingRefusesRatherThanZeroing(t *testing.T) {
var in watch.FeedIn
q := url.Values{"since": {"2026-03-01T12:00:00Z"}, "rule": {"r1"}, "live": {"true"}, "limit": {"25"}}
if err := bind(q, &in); err != nil {
t.Fatalf("bind: %v", err)
}
if in.Rule != "r1" || !in.Live || in.Limit != 25 {
t.Fatalf("bound %+v", in)
}
if !in.Since.Equal(time.Date(2026, 3, 1, 12, 0, 0, 0, time.UTC)) {
t.Fatalf("since = %v", in.Since)
}
for _, bad := range []url.Values{
{"limit": {"all"}},
{"live": {"yes"}},
{"since": {"yesterday"}},
} {
var out watch.FeedIn
if err := bind(bad, &out); err == nil {
t.Errorf("%v must be refused rather than read as a zero", bad)
}
}
}
// TestPathWinsOverTheBody. A body naming a different list than the URL is a
// request whose meaning depends on which half the reader believes, and the URL is
// the one the router matched on.
func TestPathWinsOverTheBody(t *testing.T) {
in := lists.AddIn{Name: "from-the-body"}
req := httptest.NewRequest(http.MethodPost, "/v1/aml/lists/from-the-path/entries", nil)
req.SetPathValue("name", "from-the-path")
if err := path(req, &in); err != nil {
t.Fatal(err)
}
if in.Name != "from-the-path" {
t.Fatalf("name = %q, want the path's", in.Name)
}
// A route with no such parameter leaves the field alone.
in = lists.AddIn{Name: "from-the-body"}
bare := httptest.NewRequest(http.MethodPost, "/v1/aml/lists", nil)
if err := path(bare, &in); err != nil {
t.Fatal(err)
}
if in.Name != "from-the-body" {
t.Fatalf("name = %q, want the body's", in.Name)
}
}
// TestEveryPlaneSentinelIsClassified.
//
// A refusal the table does not name answers 500, which is the right failure —
// loud — but it must not happen for an error a plane publishes. This walks every
// exported sentinel of every plane and checks the transport has an opinion about
// it, so adding one and forgetting the table is a red test rather than a 500 in
// production.
func TestEveryPlaneSentinelIsClassified(t *testing.T) {
classified := map[error]bool{}
for _, set := range [][]error{gone, taken, refused, broken} {
for _, err := range set {
if classified[err] {
t.Errorf("%v appears twice in the refusal table", err)
}
classified[err] = true
}
}
for name, sentinels := range map[string][]error{
"lists": {lists.ErrNoList, lists.ErrExists, lists.ErrName, lists.ErrKind, lists.ErrClass, lists.ErrValue, lists.ErrEmpty, lists.ErrDecider, lists.ErrReason, lists.ErrCrowded, lists.ErrNoEntry, lists.ErrStore, lists.ErrRetired, lists.ErrMaxValues},
"suppress": {suppress.ErrReason, suppress.ErrDecider, suppress.ErrBroad, suppress.ErrKind, suppress.ErrSubject, suppress.ErrWindow, suppress.ErrNotHere, suppress.ErrLifted, suppress.ErrStore},
"watch": {watch.ErrRule, watch.ErrSubject, watch.ErrKind, watch.ErrAction, watch.ErrTo, watch.ErrCount, watch.ErrWithin, watch.ErrReason, watch.ErrDecider, watch.ErrNotHere, watch.ErrRetired, watch.ErrStore},
"models": {models.ErrNoHistory, models.ErrNoModel, models.ErrNotHere, models.ErrNoFit, models.ErrDecider, models.ErrShape, models.ErrAdopted, models.ErrStore},
"topology": {topology.ErrEmptySpace, topology.ErrHuge, topology.ErrNoHistory, topology.ErrEmpty, topology.ErrShape, topology.ErrOrg},
} {
for _, err := range sentinels {
if !classified[err] {
t.Errorf("%s publishes %v and the transport has no status for it", name, err)
}
}
}
}
// TestEveryPlaneRouteIsATypedOperation.
//
// The routing table is read as source. A handler with a body in planes.go is the
// second copy of a contract that the cloud mount would then have to re-implement
// differently, and the difference is what a customer finds.
func TestEveryPlaneRouteIsATypedOperation(t *testing.T) {
src := readSource(t, "planes.go")
for _, line := range strings.Split(src, "\n") {
line = strings.TrimSpace(line)
if !strings.HasPrefix(line, "se.Router.") {
continue
}
if !strings.Contains(line, "get(h,") && !strings.Contains(line, "post(h,") {
t.Errorf("route is not a typed operation: %s", line)
}
}
if strings.Count(src, "se.Router.") < 18 {
t.Fatalf("the routing table looks truncated: %d routes", strings.Count(src, "se.Router."))
}
}
// TestPathParametersMatchTheirFields.
//
// A route naming {name} binds onto the In struct's json tag `name`. A mismatch
// does not error: the field simply stays empty and the operation refuses for a
// reason that names the wrong thing. Checked here against the actual structs.
func TestPathParametersMatchTheirFields(t *testing.T) {
for param, in := range map[string]any{
"name": lists.AddIn{},
"id": suppress.LiftIn{},
} {
if !hasTag(reflect.TypeOf(in), param) {
t.Errorf("%T has no field tagged %q, so the route parameter binds to nothing", in, param)
}
}
for _, in := range []any{lists.EntriesIn{}, lists.RemoveIn{}, lists.LookupIn{}} {
if !hasTag(reflect.TypeOf(in), "name") {
t.Errorf("%T has no field tagged \"name\"", in)
}
}
for _, in := range []any{watch.RetireIn{}, models.RefIn{}, models.AdoptIn{}} {
if !hasTag(reflect.TypeOf(in), "id") {
t.Errorf("%T has no field tagged \"id\"", in)
}
}
}
func hasTag(t reflect.Type, want string) bool {
for i := 0; i < t.NumField(); i++ {
name, _, _ := strings.Cut(t.Field(i).Tag.Get("json"), ",")
if name == want {
return true
}
}
return false
}
// TestTenantKeyIsTheBrandRegistrys. The api package must not hold a second
// definition of the tenant key's shape.
func TestTenantKeyIsTheBrandRegistrys(t *testing.T) {
key, err := qualify("hanzo", "acme")
if err != nil {
t.Fatal(err)
}
if key != "hanzo/acme" || !qualified(key) {
t.Fatalf("tenant key = %q", key)
}
if qualified("acme") {
t.Fatal("a bare org is not a tenant key")
}
if _, err := qualify("hanzo", ""); err == nil {
t.Fatal("an empty org acts for no tenant")
}
}
// readSource reads one of this package's own files, which is how the routing
// table is held to being a routing table.
func readSource(t *testing.T, name string) string {
t.Helper()
raw, err := os.ReadFile(name)
if err != nil {
t.Fatalf("read %s: %v", name, err)
}
return string(raw)
}
pkg/brand/tenant.go
+95
View File
@@ -0,0 +1,95 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package brand
// The tenant key lives here because it is a brand fact: a customer organisation
// is only identified once the brand whose issuer vouched for it is named. Every
// plane that indexes, salts or scopes on the tenant reads the shape from this one
// definition, so a new record plane cannot arrive with a second opinion about
// what a tenant is.
import (
"errors"
"fmt"
"strings"
)
// Sep divides a tenant key's two halves.
const Sep = "/"
// Qualify builds a tenant key from the brand whose issuer vouched for the caller
// and the org that caller acts for.
//
// It is one string because it is one key. The same value is the store index for
// alerts, cases, retained records, lists, suppressions, activations, field stats
// and model runs; it is the org column on a history row; and it is the HKDF salt
// every tokenisation key is derived from (pkg/token New: salt =
// "aml/token/org/" + org). An org name is unique within an issuer and not across
// issuers, so the bare org is not an identity: `acme` on lux.id and `acme` on
// zoolabs.id are two unrelated financial institutions. Keyed on the bare org they
// are one — one set of rows, and worse, one vault: the same salt derives the same
// keys, so one brand's customer names tokenise to the other's pseudonyms and its
// sealed records open under the other's tenant. That is a cross-brand plaintext
// disclosure of retained personal data, against the purpose limitation retention
// is held to (Directive (EU) 2015/849 Art. 41(2)) and against HIP-0302, which is
// the requirement that the determinism stops at the tenant boundary.
//
// Qualifying at the key, rather than filtering on the brand afterwards, is what
// makes that leak uncomputable instead of merely disallowed: there is no query
// that returns the other brand's rows and no key that opens its vault. It is done
// before any record exists, because doing it later re-keys every vault —
// pseudonyms and sealed bodies both.
func Qualify(brandID, org string) (string, error) {
b, ok := For(brandID)
if !ok {
return "", fmt.Errorf("no brand is registered as %q, so nothing vouches for this tenant", brandID)
}
org = strings.TrimSpace(org)
if org == "" {
return "", errors.New("no org, so the request acts for no tenant")
}
// Brand ids are a closed set and none contains the separator, so the first one
// always ends the brand and the key is unambiguous either way. The org half is
// held to it so the key stays readable back to the org it names: an operator
// reading a retention row or a vault refusal has to be able to say which
// institution it belongs to, and `zoo/lux/acme` does not say.
if strings.Contains(org, Sep) {
return "", fmt.Errorf("org %q contains %q, so the tenant it names is not readable back", org, Sep)
}
return b.ID + Sep + org, nil
}
// Qualified reports whether a key is one Qualify would have produced. Derived
// from Qualify rather than stated again, so there is one definition of the shape
// and a change to it cannot leave a validator behind.
func Qualified(key string) bool {
brandID, org, found := strings.Cut(key, Sep)
if !found {
return false
}
again, err := Qualify(brandID, org)
return err == nil && again == key
}
// ErrTenant is what a record plane returns for a key that is not brand-qualified.
//
// It is an error and never a filter, because the two failures are opposite in
// consequence: a refused write loses nothing, while a write admitted under a bare
// org lands in a tenant space shared with every other brand's institution of the
// same name — and the row is then indistinguishable from a legitimate one.
var ErrTenant = errors.New("tenant key is not <brand>/<org>")
// Tenant checks a key at the boundary of a record plane and names the failure.
//
// Every new plane calls this before it writes. The check exists in the planes and
// not only where the key is minted because the key arrives from a seam the
// deployment supplies: this is where the value crosses into a store index, and an
// unqualified org reaching one is the cross-brand collision.
func Tenant(key string) error {
if !Qualified(key) {
return fmt.Errorf("%w: %q", ErrTenant, key)
}
return nil
}
pkg/brand/tenant_test.go
+75
View File
@@ -0,0 +1,75 @@
package brand
import (
"errors"
"strings"
"testing"
)
// TestTenantKeyNeverRegresses holds the one property every record plane rests on:
// a tenant is `<brand>/<org>` and a bare org is not a tenant.
//
// It is here rather than in each plane because the shape is defined once. A plane
// that grew its own copy of the check would be free to disagree with this one, and
// the disagreement is a set of rows written under a key that opens another brand's
// vault.
func TestTenantKeyNeverRegresses(t *testing.T) {
key, err := Qualify("hanzo", "acme")
if err != nil {
t.Fatalf("qualify: %v", err)
}
if key != "hanzo/acme" {
t.Fatalf("tenant key = %q, want hanzo/acme", key)
}
if !Qualified(key) {
t.Fatal("a key Qualify produced must be one Qualified accepts, or the two have drifted apart")
}
// A bare org is refused everywhere a key is taken.
for _, bare := range []string{"acme", "", " ", "/acme", "acme/", "unknownbrand/acme"} {
if Qualified(bare) {
t.Errorf("%q is not a tenant key and must not be accepted as one", bare)
}
if err := Tenant(bare); !errors.Is(err, ErrTenant) {
t.Errorf("Tenant(%q) = %v, want ErrTenant", bare, err)
}
}
}
// TestBrandQualifiesTheOrg is the sharp end: two brands' institutions of the same
// name must be two tenants.
//
// Keyed on the bare org they are one — one set of rows, and one tokenisation
// vault, because the salt is this key. That is a cross-brand plaintext disclosure
// of retained personal data, and it is what qualifying at the key makes
// uncomputable rather than merely disallowed.
func TestBrandQualifiesTheOrg(t *testing.T) {
hanzo, err := Qualify("hanzo", "acme")
if err != nil {
t.Fatal(err)
}
zoo, err := Qualify("zoo", "acme")
if err != nil {
t.Fatal(err)
}
if hanzo == zoo {
t.Fatalf("two brands' institutions named acme resolved to one tenant key %q", hanzo)
}
if !strings.HasPrefix(hanzo, "hanzo"+Sep) || !strings.HasPrefix(zoo, "zoo"+Sep) {
t.Fatalf("a tenant key must lead with the brand that vouched for it: %q, %q", hanzo, zoo)
}
}
// TestUnreadableOrgRefused keeps the key readable back to the institution it
// names. `zoo/lux/acme` does not say which one that is.
func TestUnreadableOrgRefused(t *testing.T) {
if _, err := Qualify("zoo", "lux/acme"); err == nil {
t.Fatal("an org containing the separator makes the key ambiguous and must be refused")
}
if _, err := Qualify("nobody", "acme"); err == nil {
t.Fatal("a brand no registry claims vouches for nothing and must be refused")
}
if _, err := Qualify("hanzo", " "); err == nil {
t.Fatal("an empty org acts for no tenant and must be refused")
}
}
pkg/dictionary/dictionary.go
+367
View File
@@ -0,0 +1,367 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// Package dictionary is the catalog of what a tenant's payloads actually carry:
// which fields exist, what shape they are, how often they are filled, and how
// much they vary.
//
// It answers the question a monitoring programme cannot answer from its rule
// catalog alone. A rule set can look complete and be reading a field nobody
// populates; a model can list nine features and be blind on four of them. Both
// read as "no findings", and the only way to tell that apart from a clean
// institution is to measure what arrived.
//
// # Three lenses on one catalog
//
// DECLARED fields are the transaction's own, discovered by reflection over
// types.Transaction rather than by a hand-kept list, so the catalog cannot
// describe a field the engine does not read or miss one it does.
//
// CUSTOM fields are the top-level keys of the tenant's own Raw payload. They are
// how an institution's own vocabulary appears in the catalog without anybody
// declaring it here.
//
// FEATURES are the model's inventory (pkg/anomaly), carried verbatim with their
// typology, indicator and citation. They are not measured here — the model's own
// State reports what it is blind on — and they are in the catalog because a
// reviewer reading "what does this system look at" needs one answer, not two.
//
// # No value is ever stored
//
// A distinct-value count is kept as a bitmap sketch and never as values or as
// hashes of values (see sketch.go). Identifiers belong in the retained record
// plane, sealed and purpose-gated; a statistics table does not get a second copy
// of them under a weaker regime.
//
// # Write-behind, and it says so
//
// Observations accumulate in memory and are written on Flush and on Close.
// A field statistic is not a compliance record — no decision is taken on it — so
// paying an indexed write per field per transaction to make it one would be
// spending the ingest path's latency on a diagnostic. What is NOT acceptable is
// silence about it, so the catalog reports Pending: how many observations are
// accumulated and not yet durable. A restart loses exactly that number and the
// answer says so, rather than reading as a quieter institution.
package dictionary
import (
"encoding/json"
"errors"
"reflect"
"strings"
"time"
"github.com/luxfi/aml/pkg/anomaly"
"github.com/luxfi/aml/pkg/types"
)
// Origins of a field in the catalog.
const (
// Declared is a field of the transaction the engine itself reads.
Declared = "declared"
// Custom is a top-level key of the tenant's own Raw payload.
Custom = "custom"
)
// Shapes a value can have. A closed set, because the shape decides which
// statistics mean anything: a mean over text is not a number anybody should read.
const (
Text = "text"
Number = "number"
Time = "time"
Bool = "bool"
Object = "object"
List = "list"
)
// Prefix under which a tenant's own payload keys appear, so a custom field named
// `amount` is never confused with the declared one.
const Prefix = "raw."
// ErrStore is what a catalog read or write returns when the store refuses.
var ErrStore = errors.New("dictionary: store")
// MaxKeys bounds how many top-level keys of one Raw payload are catalogued.
//
// A payload with thousands of distinct keys is not a payload with a vocabulary,
// it is a map used as a bag, and cataloguing it would grow one row per key per
// tenant without bound. The excess is counted and reported rather than absorbed.
const MaxKeys = 256
// Field is one entry in the catalog.
type Field struct {
Name string `json:"name"`
Origin string `json:"origin"`
Shape string `json:"shape"`
// Seen is how many payloads carried this field with a value. Fill is that
// against the payloads examined, which is what says whether a rule reading it
// can ever fire.
Seen int64 `json:"seen"`
Fill float64 `json:"fill"`
// Blind is a declared field no payload has ever filled. It is stated rather
// than left as a zero, because a coverage claim resting on a blind field is
// the failure this catalog exists to surface.
Blind bool `json:"blind,omitempty"`
// Distinct is the estimated number of values the field has taken, and
// Saturated says the estimate has stopped being a count — the field varies
// more than the sketch can measure, which is itself the finding.
Distinct int64 `json:"distinct"`
Saturated bool `json:"saturated,omitempty"`
// Numbers only. Absent — not zero — for a field that is not a number, because
// a mean of 0.0 over an identifier reads as a fact.
Min *float64 `json:"min,omitempty"`
Max *float64 `json:"max,omitempty"`
Mean *float64 `json:"mean,omitempty"`
Deviation *float64 `json:"deviation,omitempty"`
First time.Time `json:"first,omitzero"`
Last time.Time `json:"last,omitzero"`
}
// The typed operations.
type (
// CatalogIn takes no argument beyond the tenant: a catalog is of the tenant's
// own payloads and there is no other one to ask for.
CatalogIn struct{}
// Catalog is what a tenant's payloads carry.
Catalog struct {
// Payloads is how many transactions the statistics were computed from.
Payloads int64 `json:"payloads"`
First time.Time `json:"first,omitzero"`
Last time.Time `json:"last,omitzero"`
// Pending is how many observations are accumulated in memory and not yet
// written. It is published because a restart loses exactly this many, and
// a statistic that quietly shrank is worse than one that says it might.
Pending int64 `json:"pending"`
// Skipped counts payload keys past MaxKeys that were not catalogued.
Skipped int64 `json:"skipped"`
Fields []Field `json:"fields"`
// Features is the model's own inventory, carried whole. Its statistics are
// the model's to report (GET /v1/aml/anomaly), not this catalog's, and
// inventing them here would be a second account of the same thing.
Features []anomaly.Feature `json:"features"`
}
)
// declared is the transaction's own field list, derived from the struct once.
//
// Reflection over the json tags rather than a list written out here, because a
// list is a second declaration of the payload and the two drift the first time a
// field is added — and the drift is invisible: the catalog simply never mentions
// the new field, which reads as a field nobody sends.
var declared = fields(reflect.TypeOf(types.Transaction{}))
type spec struct {
name string
shape string
index int
}
func fields(t reflect.Type) []spec {
out := make([]spec, 0, t.NumField())
for i := 0; i < t.NumField(); i++ {
f := t.Field(i)
tag := f.Tag.Get("json")
if tag == "-" || tag == "" {
continue
}
name, _, _ := strings.Cut(tag, ",")
if name == "" {
continue
}
out = append(out, spec{name: name, shape: shapeOf(f.Type), index: i})
}
return out
}
func shapeOf(t reflect.Type) string {
switch {
case t == reflect.TypeOf(time.Time{}):
return Time
case t == reflect.TypeOf(json.RawMessage{}):
return Object
}
switch t.Kind() {
case reflect.String:
return Text
case reflect.Bool:
return Bool
case reflect.Float32, reflect.Float64,
reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64,
reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
return Number
case reflect.Slice, reflect.Array:
return List
default:
return Object
}
}
// Fields is the transaction fields the engine reads, in declaration order. It is
// exported so a console can render the vocabulary a rule may name without keeping
// its own copy of it.
func Fields() []string {
out := make([]string, 0, len(declared))
for _, s := range declared {
out = append(out, s.name)
}
return out
}
// reading is one field's value in one payload, reduced to what the statistics
// need: whether it was filled, its text form for the distinct count, and its
// numeric value where it has one.
type reading struct {
filled bool
text string
num float64
isNum bool
}
// read pulls one declared field out of a transaction.
//
// Emptiness is by value and not by presence, because JSON's zero and JSON's
// absence are the same thing on a Go struct and a fill rate that counted zeros as
// filled would report every field on every payload as populated.
func read(v reflect.Value, s spec) reading {
f := v.Field(s.index)
switch s.shape {
case Text:
str := f.String()
return reading{filled: str != "", text: str}
case Bool:
// A bool is filled whichever way it points: false is a value, not an
// absence, and a field that is false on every payload is a fact worth
// seeing in the catalog.
return reading{filled: true, text: boolText(f.Bool())}
case Number:
n := number(f)
return reading{filled: n != 0, text: "", num: n, isNum: true}
case Time:
t, _ := f.Interface().(time.Time)
return reading{filled: !t.IsZero(), text: t.UTC().Format(time.RFC3339)}
default:
return reading{filled: f.Len() > 0}
}
}
func number(v reflect.Value) float64 {
switch v.Kind() {
case reflect.Float32, reflect.Float64:
return v.Float()
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
return float64(v.Int())
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
return float64(v.Uint())
default:
return 0
}
}
func boolText(b bool) string {
if b {
return "true"
}
return "false"
}
// custom reads the top-level keys of a Raw payload.
//
// The shape is read from the JSON itself rather than assumed, so a key sent as a
// number on one payload and as text on the next is catalogued as it arrived. The
// catalog reports the LAST shape seen; a field whose shape moves is a data
// quality finding the fill and distinct counts will already be showing.
func custom(raw json.RawMessage) (map[string]reading, int) {
if len(raw) == 0 {
return nil, 0
}
var top map[string]json.RawMessage
if err := json.Unmarshal(raw, &top); err != nil {
// A payload whose extension is not an object carries no keys to catalog.
// It is not an error here: refusing the observation would make the
// dictionary decide whether a transaction is acceptable, which is the
// engine's job and not a statistic's.
return nil, 0
}
out := make(map[string]reading, len(top))
skipped := 0
names := make([]string, 0, len(top))
for k := range top {
names = append(names, k)
}
// Sorted, so which keys are dropped past the bound is deterministic rather
// than a property of map iteration order.
sortStrings(names)
for _, k := range names {
if len(out) >= MaxKeys {
skipped++
continue
}
out[Prefix+k] = value(top[k])
}
return out, skipped
}
func value(raw json.RawMessage) reading {
s := strings.TrimSpace(string(raw))
if s == "" || s == "null" {
return reading{}
}
switch s[0] {
case '"':
var str string
if err := json.Unmarshal(raw, &str); err != nil {
return reading{}
}
return reading{filled: str != "", text: str}
case '{':
return reading{filled: true, text: ""}
case '[':
return reading{filled: true, text: ""}
case 't', 'f':
return reading{filled: true, text: s}
default:
var n float64
if err := json.Unmarshal(raw, &n); err != nil {
return reading{}
}
return reading{filled: true, num: n, isNum: true}
}
}
// shapeText names the shape of a custom reading, from the value rather than from
// a declaration.
func shapeText(raw json.RawMessage) string {
s := strings.TrimSpace(string(raw))
if s == "" {
return Text
}
switch s[0] {
case '"':
return Text
case '{':
return Object
case '[':
return List
case 't', 'f':
return Bool
case 'n':
return Text
default:
return Number
}
}
func sortStrings(s []string) {
for i := 1; i < len(s); i++ {
for j := i; j > 0 && s[j] < s[j-1]; j-- {
s[j], s[j-1] = s[j-1], s[j]
}
}
}
pkg/dictionary/dictionary_test.go
+475
View File
@@ -0,0 +1,475 @@
package dictionary
import (
"context"
"encoding/json"
"fmt"
"os"
"slices"
"strings"
"testing"
"time"
"github.com/luxfi/aml/internal/instance"
"github.com/luxfi/aml/pkg/types"
)
const (
acme = "hanzo/acme"
rival = "hanzo/rival"
// other is the SAME org name under a different brand.
other = "zoo/acme"
)
var noon = time.Date(2026, 3, 1, 12, 0, 0, 0, time.UTC)
func shelf(t *testing.T) *Shelf {
t.Helper()
app := instance.New(t)
t.Cleanup(app.Cleanup)
if err := Ensure(app); err != nil {
t.Fatalf("ensure: %v", err)
}
s := NewBase(app)
s.Now = func() time.Time { return noon }
return s
}
func payment(user string, usd float64, raw string) types.Transaction {
tx := types.Transaction{
ID: fmt.Sprintf("tx-%s-%v", user, usd), UserID: user, AccountID: "acct-" + user,
Currency: "USD", Notional: usd, USD: usd, Timestamp: noon, Direction: "in",
}
if raw != "" {
tx.Raw = json.RawMessage(raw)
}
return tx
}
func field(t *testing.T, cat *Catalog, name string) Field {
t.Helper()
for _, f := range cat.Fields {
if f.Name == name {
return f
}
}
t.Fatalf("the catalog has no field %q", name)
return Field{}
}
// TestDeclaredFieldsComeFromTheStruct. A hand-kept list is a second declaration
// of the payload, and the drift is invisible: the catalog simply never mentions
// the new field, which reads as a field nobody sends.
func TestDeclaredFieldsComeFromTheStruct(t *testing.T) {
names := Fields()
for _, want := range []string{"id", "user_id", "account_id", "notional", "currency", "timestamp", "usd", "ip_address"} {
if !slices.Contains(names, want) {
t.Errorf("the catalog does not know about %q, which the engine reads", want)
}
}
if slices.Contains(names, "OrgID") {
t.Error("the catalog is built from json tags, not Go field names")
}
}
// TestBlindFieldIsStated. A coverage claim resting on a field nobody fills is the
// failure this catalog exists to surface, so the field appears with Blind set
// rather than being absent.
func TestBlindFieldIsStated(t *testing.T) {
s := shelf(t)
ctx := context.Background()
for i := 0; i < 4; i++ {
if err := s.Observe(acme, payment("u1", float64(100*(i+1)), "")); err != nil {
t.Fatal(err)
}
}
cat, err := s.Catalog(ctx, acme, &CatalogIn{})
if err != nil {
t.Fatal(err)
}
if cat.Payloads != 4 {
t.Fatalf("payloads = %d, want 4", cat.Payloads)
}
ip := field(t, cat, "ip_address")
if !ip.Blind || ip.Seen != 0 || ip.Fill != 0 {
t.Fatalf("no payload carried an address, so the field is blind: %+v", ip)
}
user := field(t, cat, "user_id")
if user.Blind || user.Seen != 4 || user.Fill != 1 {
t.Fatalf("every payload carried a user: %+v", user)
}
if user.Distinct != 1 {
t.Fatalf("one user in four payloads is one distinct value, got %d", user.Distinct)
}
}
// TestNumericStatistics are present for numbers and absent for everything else. A
// mean of 0.0 over an identifier reads as a fact.
func TestNumericStatistics(t *testing.T) {
s := shelf(t)
for _, v := range []float64{100, 200, 300} {
if err := s.Observe(acme, payment("u1", v, "")); err != nil {
t.Fatal(err)
}
}
cat, err := s.Catalog(context.Background(), acme, &CatalogIn{})
if err != nil {
t.Fatal(err)
}
usd := field(t, cat, "usd")
if usd.Mean == nil || usd.Min == nil || usd.Max == nil || usd.Deviation == nil {
t.Fatalf("a number carries its statistics: %+v", usd)
}
if *usd.Mean != 200 || *usd.Min != 100 || *usd.Max != 300 {
t.Fatalf("mean=%v min=%v max=%v, want 200/100/300", *usd.Mean, *usd.Min, *usd.Max)
}
if user := field(t, cat, "user_id"); user.Mean != nil {
t.Fatalf("text has no mean: %+v", user)
}
}
// TestCustomFieldsAreTheTenantsOwnVocabulary.
func TestCustomFieldsAreTheTenantsOwnVocabulary(t *testing.T) {
s := shelf(t)
if err := s.Observe(acme, payment("u1", 100, `{"channel":"pos","terminal":42,"reversed":false}`)); err != nil {
t.Fatal(err)
}
cat, err := s.Catalog(context.Background(), acme, &CatalogIn{})
if err != nil {
t.Fatal(err)
}
channel := field(t, cat, Prefix+"channel")
if channel.Origin != Custom || channel.Shape != Text || channel.Seen != 1 {
t.Fatalf("a payload key is catalogued as the tenant's own: %+v", channel)
}
terminal := field(t, cat, Prefix+"terminal")
if terminal.Shape != Number || terminal.Mean == nil || *terminal.Mean != 42 {
t.Fatalf("a numeric payload key carries its statistics: %+v", terminal)
}
reversed := field(t, cat, Prefix+"reversed")
if reversed.Shape != Bool || reversed.Seen != 1 {
t.Fatalf("false is a value and not an absence: %+v", reversed)
}
// The declared `raw` field is an object and is not confused with its keys.
if raw := field(t, cat, "raw"); raw.Origin != Declared {
t.Fatalf("the raw payload itself stays a declared field: %+v", raw)
}
}
// TestFeaturesTravelWholeAndUnmeasured. Their statistics are the model's to
// report; inventing them here would be a second account of the same thing.
func TestFeaturesTravelWholeAndUnmeasured(t *testing.T) {
s := shelf(t)
cat, err := s.Catalog(context.Background(), acme, &CatalogIn{})
if err != nil {
t.Fatal(err)
}
if len(cat.Features) != 9 {
t.Fatalf("the model's inventory is nine dimensions, got %d", len(cat.Features))
}
for _, f := range cat.Features {
if f.Typology == "" || f.Citation == "" || f.Unit == "" {
t.Fatalf("a feature travels with what makes it checkable: %+v", f)
}
}
}
// TestNoValueIsEverStored. The distinct count is a bitmap, and a bitmap is not a
// pseudonym: a statistics table does not get a second copy of an institution's
// identifiers under a weaker regime than the retained record plane's.
func TestNoValueIsEverStored(t *testing.T) {
s := shelf(t)
ctx := context.Background()
secret := "[email protected]"
tx := payment("u1", 100, `{"contact":"`+secret+`"}`)
if err := s.Observe(acme, tx); err != nil {
t.Fatal(err)
}
if err := s.Flush(ctx); err != nil {
t.Fatal(err)
}
cat, err := s.Catalog(ctx, acme, &CatalogIn{})
if err != nil {
t.Fatal(err)
}
rendered, err := json.Marshal(cat)
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(rendered), secret) {
t.Fatal("the catalog handed back a payload value")
}
if strings.Contains(string(rendered), "u1") || strings.Contains(string(rendered), "acct-u1") {
t.Fatal("the catalog handed back an identifier")
}
// And the row itself holds no value either.
if strings.Contains(dump(t, s, acme), secret) {
t.Fatal("a payload value reached the store")
}
}
// dump renders every stored field row, which is what the no-value invariant is
// checked against.
func dump(t *testing.T, s *Shelf, org string) string {
t.Helper()
rows, err := fieldKind.Find(s.app, org, "", "", 0, nil)
if err != nil {
t.Fatal(err)
}
var b strings.Builder
for _, row := range rows {
raw, err := json.Marshal(row.PublicExport())
if err != nil {
t.Fatal(err)
}
b.Write(raw)
}
return b.String()
}
// TestDistinctSaturatesRatherThanDrifting. A cardinality that silently stops
// rising reads as a field that stopped varying.
func TestDistinctSaturatesRatherThanDrifting(t *testing.T) {
var s sketch
for i := 0; i < 500; i++ {
s.add(acme, "user_id", fmt.Sprintf("u%d", i))
}
n, saturated := s.estimate()
if saturated {
t.Fatalf("500 values must not saturate a %d-bit bitmap", buckets)
}
if n < 450 || n > 550 {
t.Fatalf("estimate %d is not within 10%% of 500", n)
}
var full sketch
for i := 0; i < 100_000; i++ {
full.add(acme, "user_id", fmt.Sprintf("u%d", i))
}
if _, saturated := full.estimate(); !saturated {
t.Fatal("a bitmap this full is no longer counting, and must say so")
}
}
// TestSketchIsMergeableAndPerTenant.
//
// Mergeable is what makes a write-behind census correct across a restart rather
// than approximately correct. Per tenant is what stops two tenants' bitmaps being
// compared to learn whether they share a customer — an inference computed from a
// diagnostics table.
func TestSketchIsMergeableAndPerTenant(t *testing.T) {
var a, b, both sketch
for i := 0; i < 100; i++ {
v := fmt.Sprintf("u%d", i)
a.add(acme, "user_id", v)
both.add(acme, "user_id", v)
}
for i := 50; i < 150; i++ {
v := fmt.Sprintf("u%d", i)
b.add(acme, "user_id", v)
both.add(acme, "user_id", v)
}
a.merge(b)
if a != both {
t.Fatal("merging two halves must equal observing both, or a restart double-counts the overlap")
}
var mine, theirs sketch
for i := 0; i < 200; i++ {
v := fmt.Sprintf("u%d", i)
mine.add(acme, "user_id", v)
theirs.add(other, "user_id", v)
}
if mine == theirs {
t.Fatal("two tenants observing the same values produced the same bitmap: they are comparable")
}
// And the encoding round-trips, which is what the durable row depends on.
if decode(mine.encode()) != mine {
t.Fatal("a bitmap must survive the row it is written to")
}
if decode("not base64") != (sketch{}) {
t.Fatal("an unreadable bitmap restarts the count rather than corrupting it")
}
}
// TestPendingIsPublished. A restart loses exactly the un-flushed observations, and
// the answer says how many that is rather than reading as a quieter institution.
func TestPendingIsPublished(t *testing.T) {
s := shelf(t)
ctx := context.Background()
for i := 0; i < 3; i++ {
if err := s.Observe(acme, payment("u1", float64(i+1), "")); err != nil {
t.Fatal(err)
}
}
cat, err := s.Catalog(ctx, acme, &CatalogIn{})
if err != nil {
t.Fatal(err)
}
if cat.Pending != 3 || cat.Payloads != 3 {
t.Fatalf("un-flushed observations are counted and published: pending=%d payloads=%d", cat.Pending, cat.Payloads)
}
if err := s.Flush(ctx); err != nil {
t.Fatal(err)
}
cat, err = s.Catalog(ctx, acme, &CatalogIn{})
if err != nil {
t.Fatal(err)
}
if cat.Pending != 0 || cat.Payloads != 3 {
t.Fatalf("after a flush nothing is pending and nothing is lost: pending=%d payloads=%d", cat.Pending, cat.Payloads)
}
}
// TestTenantIsolation, including the same org name under a second brand.
func TestTenantIsolation(t *testing.T) {
s := shelf(t)
ctx := context.Background()
if err := s.Observe(acme, payment("u1", 100, `{"channel":"pos"}`)); err != nil {
t.Fatal(err)
}
if err := s.Observe(other, payment("z9", 900, `{"lane":"web"}`)); err != nil {
t.Fatal(err)
}
if err := s.Flush(ctx); err != nil {
t.Fatal(err)
}
mine, err := s.Catalog(ctx, acme, &CatalogIn{})
if err != nil {
t.Fatal(err)
}
if mine.Payloads != 1 {
t.Fatalf("payloads = %d, want 1 — another tenant's payloads were counted", mine.Payloads)
}
for _, f := range mine.Fields {
if f.Name == Prefix+"lane" {
t.Fatal("another tenant's payload key is in this catalog")
}
}
theirs, err := s.Catalog(ctx, other, &CatalogIn{})
if err != nil {
t.Fatal(err)
}
if theirs.Payloads != 1 {
t.Fatalf("the second brand's catalog counted %d payloads, want 1", theirs.Payloads)
}
empty, err := s.Catalog(ctx, rival, &CatalogIn{})
if err != nil {
t.Fatal(err)
}
if empty.Payloads != 0 {
t.Fatalf("a tenant that sent nothing has a catalog of nothing, got %d payloads", empty.Payloads)
}
// Its declared fields are all blind, which is the honest reading.
for _, f := range empty.Fields {
if f.Origin == Declared && !f.Blind {
t.Fatalf("a tenant that sent nothing has no filled field: %+v", f)
}
}
}
// TestBareOrgIsRefused.
func TestBareOrgIsRefused(t *testing.T) {
s := shelf(t)
for _, bare := range []string{"acme", "", "unknown/acme"} {
if err := s.Observe(bare, payment("u1", 100, "")); err == nil {
t.Errorf("Observe accepted %q as a tenant", bare)
}
if _, err := s.Catalog(context.Background(), bare, &CatalogIn{}); err == nil {
t.Errorf("Catalog accepted %q as a tenant", bare)
}
}
}
// TestNothingDeletes.
func TestNothingDeletes(t *testing.T) {
for _, name := range []string{"dictionary.go", "shelf.go", "sketch.go"} {
raw, err := os.ReadFile(name)
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(raw), ".Delete(") {
t.Errorf("%s calls Delete: disposal is pkg/retention's decision and nobody else's", name)
}
}
}
// TestRestart: what a restarted pod reads, and that the counts continue rather
// than starting again.
func TestRestart(t *testing.T) {
first := instance.New(t)
if err := Ensure(first); err != nil {
t.Fatal(err)
}
before := NewBase(first)
before.Now = func() time.Time { return noon }
for i := 0; i < 10; i++ {
if err := before.Observe(acme, payment(fmt.Sprintf("u%d", i), float64(100*(i+1)), `{"channel":"pos"}`)); err != nil {
t.Fatal(err)
}
}
if err := before.Observe(other, payment("z9", 900, "")); err != nil {
t.Fatal(err)
}
if err := before.Flush(context.Background()); err != nil {
t.Fatal(err)
}
second := instance.Restart(t, first)
if err := Ensure(second); err != nil {
t.Fatal(err)
}
after := NewBase(second)
after.Now = func() time.Time { return noon }
cat, err := after.Catalog(context.Background(), acme, &CatalogIn{})
if err != nil {
t.Fatal(err)
}
if cat.Payloads != 10 {
t.Fatalf("the census did not survive the restart: %d payloads, want 10", cat.Payloads)
}
user := field(t, cat, "user_id")
if user.Seen != 10 || user.Distinct < 9 || user.Distinct > 11 {
t.Fatalf("the distinct count did not survive: %+v", user)
}
if field(t, cat, Prefix+"channel").Seen != 10 {
t.Fatal("the tenant's own payload key did not survive the restart")
}
// The counts CONTINUE. A census that restarted at zero would read as an
// institution that just started sending.
for i := 10; i < 15; i++ {
if err := after.Observe(acme, payment(fmt.Sprintf("u%d", i), 100, `{"channel":"pos"}`)); err != nil {
t.Fatal(err)
}
}
if err := after.Flush(context.Background()); err != nil {
t.Fatal(err)
}
cat, err = after.Catalog(context.Background(), acme, &CatalogIn{})
if err != nil {
t.Fatal(err)
}
if cat.Payloads != 15 {
t.Fatalf("the census restarted instead of continuing: %d payloads, want 15", cat.Payloads)
}
user = field(t, cat, "user_id")
if user.Distinct < 13 || user.Distinct > 17 {
t.Fatalf("the distinct count restarted instead of merging: %+v", user)
}
// Tenancy still holds.
theirs, err := after.Catalog(context.Background(), other, &CatalogIn{})
if err != nil {
t.Fatal(err)
}
if theirs.Payloads != 1 {
t.Fatalf("the tenant boundary did not survive the restart: %d", theirs.Payloads)
}
}
pkg/dictionary/shelf.go
+528
View File
@@ -0,0 +1,528 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package dictionary
import (
"context"
"encoding/json"
"fmt"
"math"
"reflect"
"sort"
"sync"
"time"
"github.com/hanzoai/base/core"
"github.com/hanzoai/dbx"
"github.com/luxfi/aml/pkg/anomaly"
"github.com/luxfi/aml/pkg/brand"
"github.com/luxfi/aml/pkg/store"
"github.com/luxfi/aml/pkg/types"
)
const (
fieldName = "name"
fieldOrigin = "origin"
fieldShape = "shape"
fieldSeen = "seen"
fieldBits = "bits"
fieldCount = "num_count"
fieldSum = "num_sum"
fieldSquare = "num_square"
fieldMin = "num_min"
fieldMax = "num_max"
fieldFirst = "first"
fieldLast = "last"
fieldPayloads = "payloads"
fieldSkipped = "skipped"
)
var fieldKind = store.Kind{
Name: "aml_fields",
Fields: []core.Field{
&core.TextField{Name: fieldName, Required: true},
&core.TextField{Name: fieldOrigin, Required: true},
&core.TextField{Name: fieldShape, Required: true},
&core.NumberField{Name: fieldSeen},
&core.TextField{Name: fieldBits},
&core.NumberField{Name: fieldCount},
&core.NumberField{Name: fieldSum},
&core.NumberField{Name: fieldSquare},
&core.NumberField{Name: fieldMin},
&core.NumberField{Name: fieldMax},
&core.DateField{Name: fieldFirst},
&core.DateField{Name: fieldLast},
},
Indexes: []store.Index{
{Name: "name", Fields: []string{store.Org, fieldName}, Unique: true},
},
}
var censusKind = store.Kind{
Name: "aml_payloads",
Fields: []core.Field{
&core.NumberField{Name: fieldPayloads},
&core.NumberField{Name: fieldSkipped},
&core.DateField{Name: fieldFirst},
&core.DateField{Name: fieldLast},
},
Indexes: []store.Index{
// One row per tenant. Unique, so a concurrent first flush cannot leave two
// counts that each look like the whole.
{Name: "org", Fields: []string{store.Org}, Unique: true},
},
}
// Ensure creates what the catalog is kept in. Idempotent, so it runs on every
// start.
func Ensure(app core.App) error {
if err := fieldKind.Ensure(app); err != nil {
return err
}
return censusKind.Ensure(app)
}
// stat is one field's accumulated observations, in memory between flushes.
type stat struct {
origin string
shape string
seen int64
bits sketch
count int64
sum float64
square float64
min float64
max float64
first time.Time
last time.Time
}
func (s *stat) observe(org, name string, r reading, at time.Time) {
if !r.filled {
return
}
s.seen++
if r.text != "" {
s.bits.add(org, name, r.text)
}
if r.isNum {
s.bits.add(org, name, fmt.Sprintf("%v", r.num))
s.count++
s.sum += r.num
s.square += r.num * r.num
if s.count == 1 || r.num < s.min {
s.min = r.num
}
if s.count == 1 || r.num > s.max {
s.max = r.num
}
}
if s.first.IsZero() || at.Before(s.first) {
s.first = at
}
if at.After(s.last) {
s.last = at
}
}
type census struct {
fields map[string]*stat
payloads int64
skipped int64
first time.Time
last time.Time
}
// Shelf is the durable catalog, with the accumulator in front of it.
type Shelf struct {
app core.App
// Now supplies the instant an observation is stamped with. Tests set it.
Now func() time.Time
mu sync.Mutex
pending map[string]*census
// count is how many observations are accumulated and not yet written, across
// every tenant. It is published on every catalog so a reader can see how much
// of the answer a restart would lose.
count int64
}
// NewBase returns the durable catalog. Ensure has to have run first.
func NewBase(app core.App) *Shelf {
return &Shelf{app: app, pending: map[string]*census{}}
}
func (s *Shelf) now() time.Time {
if s.Now != nil {
return s.Now().UTC()
}
return time.Now().UTC()
}
// Observe accumulates one payload.
//
// It writes nothing: the observation joins this tenant's accumulator and reaches
// the store on the next Flush. That keeps a per-field indexed write off the
// ingest path, and the cost of it — what a restart would lose — is published as
// Pending rather than absorbed.
func (s *Shelf) Observe(org string, tx types.Transaction) error {
if err := brand.Tenant(org); err != nil {
return err
}
at := tx.Timestamp.UTC()
if at.IsZero() {
at = s.now()
}
v := reflect.ValueOf(tx)
extra, skipped := custom(tx.Raw)
var top map[string]json.RawMessage
if len(tx.Raw) > 0 {
_ = json.Unmarshal(tx.Raw, &top)
}
s.mu.Lock()
defer s.mu.Unlock()
c := s.pending[org]
if c == nil {
c = &census{fields: map[string]*stat{}}
s.pending[org] = c
}
c.payloads++
c.skipped += int64(skipped)
if c.first.IsZero() || at.Before(c.first) {
c.first = at
}
if at.After(c.last) {
c.last = at
}
for _, spec := range declared {
st := c.field(spec.name, Declared, spec.shape)
st.observe(org, spec.name, read(v, spec), at)
}
for name, r := range extra {
st := c.field(name, Custom, shapeText(top[name[len(Prefix):]]))
st.shape = shapeText(top[name[len(Prefix):]])
st.observe(org, name, r, at)
}
s.count++
return nil
}
func (c *census) field(name, origin, shape string) *stat {
st := c.fields[name]
if st == nil {
st = &stat{origin: origin, shape: shape}
c.fields[name] = st
}
return st
}
// Pending is how many observations are accumulated and not yet durable.
func (s *Shelf) Pending() int64 {
s.mu.Lock()
defer s.mu.Unlock()
return s.count
}
// Flush writes every tenant's accumulated observations.
//
// One transaction per tenant: a half-written census is a fill rate computed over
// a payload count that does not match it, and a fill rate over the wrong
// denominator is a number that reads as a finding.
func (s *Shelf) Flush(ctx context.Context) error {
s.mu.Lock()
batch := s.pending
s.pending = map[string]*census{}
s.count = 0
s.mu.Unlock()
var failed error
for org, c := range batch {
if err := s.write(org, c); err != nil {
// Put the tenant's observations back, so a store that was briefly
// unavailable costs latency rather than the census.
s.restore(org, c)
failed = err
}
}
return failed
}
// restore folds an unwritten census back into the accumulator.
func (s *Shelf) restore(org string, c *census) {
s.mu.Lock()
defer s.mu.Unlock()
held := s.pending[org]
if held == nil {
s.pending[org] = c
s.count += c.payloads
return
}
merge(held, c)
s.count += c.payloads
}
func merge(into, from *census) {
into.payloads += from.payloads
into.skipped += from.skipped
if into.first.IsZero() || (!from.first.IsZero() && from.first.Before(into.first)) {
into.first = from.first
}
if from.last.After(into.last) {
into.last = from.last
}
for name, st := range from.fields {
held := into.fields[name]
if held == nil {
into.fields[name] = st
continue
}
held.seen += st.seen
held.bits.merge(st.bits)
if st.count > 0 {
if held.count == 0 || st.min < held.min {
held.min = st.min
}
if held.count == 0 || st.max > held.max {
held.max = st.max
}
held.count += st.count
held.sum += st.sum
held.square += st.square
}
if held.first.IsZero() || (!st.first.IsZero() && st.first.Before(held.first)) {
held.first = st.first
}
if st.last.After(held.last) {
held.last = st.last
}
}
}
func (s *Shelf) write(org string, c *census) error {
if err := brand.Tenant(org); err != nil {
return err
}
return s.app.RunInTransaction(func(tx core.App) error {
for name, st := range c.fields {
rows, err := fieldKind.Find(tx, org, fieldName+" = {:name}", "", 1, dbx.Params{"name": name})
if err != nil {
return fmt.Errorf("%w: %w", ErrStore, err)
}
var row *core.Record
held := &stat{}
if len(rows) > 0 {
row = rows[0]
held = readStat(row)
} else {
row, err = fieldKind.New(tx, org)
if err != nil {
return fmt.Errorf("%w: %w", ErrStore, err)
}
}
fold(held, st)
held.origin, held.shape = st.origin, st.shape
row.Set(fieldName, name)
writeStat(row, held)
if err := tx.Save(row); err != nil {
return fmt.Errorf("%w: %s: %w", ErrStore, name, err)
}
}
rows, err := censusKind.Find(tx, org, "", "", 1, nil)
if err != nil {
return fmt.Errorf("%w: %w", ErrStore, err)
}
var row *core.Record
var payloads, skipped int64
var first, last time.Time
if len(rows) > 0 {
row = rows[0]
payloads = int64(row.GetInt(fieldPayloads))
skipped = int64(row.GetInt(fieldSkipped))
first = row.GetDateTime(fieldFirst).Time()
last = row.GetDateTime(fieldLast).Time()
} else {
row, err = censusKind.New(tx, org)
if err != nil {
return fmt.Errorf("%w: %w", ErrStore, err)
}
}
payloads += c.payloads
skipped += c.skipped
if first.IsZero() || (!c.first.IsZero() && c.first.Before(first)) {
first = c.first
}
if c.last.After(last) {
last = c.last
}
row.Set(fieldPayloads, payloads)
row.Set(fieldSkipped, skipped)
row.Set(fieldFirst, first.UTC())
row.Set(fieldLast, last.UTC())
if err := tx.Save(row); err != nil {
return fmt.Errorf("%w: census: %w", ErrStore, err)
}
return nil
})
}
// fold merges an accumulated stat into the stored one. Every accumulator here is
// a monoid — a sum, a max, a bitwise union — which is what makes a write-behind
// census correct across a restart rather than approximately correct.
func fold(into, from *stat) {
into.seen += from.seen
into.bits.merge(from.bits)
if from.count > 0 {
if into.count == 0 || from.min < into.min {
into.min = from.min
}
if into.count == 0 || from.max > into.max {
into.max = from.max
}
into.count += from.count
into.sum += from.sum
into.square += from.square
}
if into.first.IsZero() || (!from.first.IsZero() && from.first.Before(into.first)) {
into.first = from.first
}
if from.last.After(into.last) {
into.last = from.last
}
}
// Catalog reports what a tenant's payloads carry.
//
// It reads the durable rows and folds this tenant's un-flushed accumulator on
// top, so the answer is current rather than as-of-the-last-flush — and Pending
// says how much of it a restart would take back.
func (s *Shelf) Catalog(ctx context.Context, org string, _ *CatalogIn) (*Catalog, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
rows, err := fieldKind.Find(s.app, org, "", fieldName, 0, nil)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
held := &census{fields: map[string]*stat{}}
for _, row := range rows {
st := readStat(row)
st.origin = row.GetString(fieldOrigin)
st.shape = row.GetString(fieldShape)
held.fields[row.GetString(fieldName)] = st
}
crows, err := censusKind.Find(s.app, org, "", "", 1, nil)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
if len(crows) > 0 {
held.payloads = int64(crows[0].GetInt(fieldPayloads))
held.skipped = int64(crows[0].GetInt(fieldSkipped))
held.first = crows[0].GetDateTime(fieldFirst).Time()
held.last = crows[0].GetDateTime(fieldLast).Time()
}
s.mu.Lock()
pending := int64(0)
if c := s.pending[org]; c != nil {
pending = c.payloads
merge(held, copyCensus(c))
}
s.mu.Unlock()
out := &Catalog{
Payloads: held.payloads, First: held.first, Last: held.last,
Pending: pending, Skipped: held.skipped,
Fields: make([]Field, 0, len(held.fields)+len(declared)),
Features: inventory(),
}
// Every declared field appears whether or not it was ever filled: a field
// missing from the catalog reads as a field nobody sends, and a field present
// with Blind set reads as one nobody fills. Only the second is true.
for _, spec := range declared {
if held.fields[spec.name] == nil {
held.fields[spec.name] = &stat{origin: Declared, shape: spec.shape}
}
}
for name, st := range held.fields {
out.Fields = append(out.Fields, project(name, st, held.payloads))
}
sort.Slice(out.Fields, func(i, j int) bool { return out.Fields[i].Name < out.Fields[j].Name })
return out, nil
}
// copyCensus takes a snapshot of an accumulator under the lock, so folding it
// into an answer cannot race the next Observe.
func copyCensus(c *census) *census {
out := &census{fields: make(map[string]*stat, len(c.fields)),
payloads: c.payloads, skipped: c.skipped, first: c.first, last: c.last}
for name, st := range c.fields {
copied := *st
out.fields[name] = &copied
}
return out
}
func project(name string, st *stat, payloads int64) Field {
distinct, saturated := st.bits.estimate()
f := Field{
Name: name, Origin: st.origin, Shape: st.shape,
Seen: st.seen, Distinct: distinct, Saturated: saturated,
First: st.first, Last: st.last,
Blind: st.seen == 0,
}
if payloads > 0 {
f.Fill = float64(st.seen) / float64(payloads)
}
if st.count > 0 {
mean := st.sum / float64(st.count)
variance := st.square/float64(st.count) - mean*mean
if variance < 0 {
variance = 0
}
dev := math.Sqrt(variance)
min, max := st.min, st.max
f.Min, f.Max, f.Mean, f.Deviation = &min, &max, &mean, &dev
}
return f
}
// inventory is the model's feature list, carried whole.
func inventory() []anomaly.Feature {
inv := anomaly.Inventory()
return append([]anomaly.Feature(nil), inv[:]...)
}
func writeStat(row *core.Record, s *stat) {
row.Set(fieldOrigin, s.origin)
row.Set(fieldShape, s.shape)
row.Set(fieldSeen, s.seen)
row.Set(fieldBits, s.bits.encode())
row.Set(fieldCount, s.count)
row.Set(fieldSum, s.sum)
row.Set(fieldSquare, s.square)
row.Set(fieldMin, s.min)
row.Set(fieldMax, s.max)
row.Set(fieldFirst, s.first.UTC())
row.Set(fieldLast, s.last.UTC())
}
func readStat(row *core.Record) *stat {
return &stat{
seen: int64(row.GetInt(fieldSeen)),
bits: decode(row.GetString(fieldBits)),
count: int64(row.GetInt(fieldCount)),
sum: row.GetFloat(fieldSum),
square: row.GetFloat(fieldSquare),
min: row.GetFloat(fieldMin),
max: row.GetFloat(fieldMax),
first: row.GetDateTime(fieldFirst).Time(),
last: row.GetDateTime(fieldLast).Time(),
}
}
pkg/dictionary/sketch.go
+135
View File
@@ -0,0 +1,135 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package dictionary
import (
"encoding/base64"
"math"
)
// A sketch counts how many DISTINCT values a field has taken, without keeping any
// of them.
//
// That constraint is the whole design. A field catalog is a diagnostic plane, and
// the ordinary way to count distinct values — keep the values, or keep their
// hashes — puts a pseudonymised set of customer identifiers, addresses and email
// addresses into a table whose purpose is to describe shapes. The retained record
// plane is where identifiers belong, sealed and purpose-gated (pkg/token,
// pkg/retention); a statistics table is not entitled to a second copy of them
// under a weaker regime.
//
// So: linear counting over a fixed bitmap. Each value sets one bit chosen by a
// hash; the estimate is derived from how many bits are still clear. Two
// properties follow, and both matter here. It is MERGEABLE — a bitwise OR — so an
// accumulator flushed to a row and picked up again after a restart continues the
// same count rather than starting one. And it SATURATES rather than drifting: as
// the bitmap fills the estimate loses accuracy in a way that is computable from
// the bitmap itself, so the answer is "at least this many, and the count is no
// longer reliable" instead of a smaller number that reads as fact.
//
// Whitang, Vitter & others' linear counting: n̂ = -m·ln(z/m), for m buckets of
// which z are empty. At the bucket count below it is within about 1% up to a few
// thousand distinct values, which is the range where the number is interesting —
// a field with more distinct values than that is an identifier, and knowing it is
// an identifier is the whole of what the catalog needs to say.
// buckets is the bitmap's width in bits. 4096 bits is 512 bytes per field per
// tenant, and roughly 1% accurate to a few thousand distinct values.
const buckets = 4096
// full is the share of set buckets past which the estimate is no longer reported
// as a count. At 95% occupancy linear counting is extrapolating from 5% of the
// bitmap and the error grows without bound.
const full = 0.95
type sketch [buckets / 8]byte
// add records one value against a tenant-and-field-specific hash.
//
// The org is part of the hash so the same value in two tenants sets different
// buckets. Without that, two tenants' bitmaps could be compared to learn whether
// they share a customer — a cross-tenant inference computed from a diagnostics
// table, which is exactly the shape of leak the tenant key exists to make
// impossible.
func (s *sketch) add(org, field, value string) {
h := hash(org, field, value)
i := h % buckets
s[i/8] |= 1 << (i % 8)
}
// merge folds another bitmap in. Union, because a distinct count is a set union
// and the same value seen in both halves must not count twice.
func (s *sketch) merge(other sketch) {
for i := range s {
s[i] |= other[i]
}
}
// set counts the bits that are on.
func (s sketch) set() int {
n := 0
for _, b := range s {
for ; b != 0; b &= b - 1 {
n++
}
}
return n
}
// estimate is how many distinct values the bitmap has seen, and whether the
// answer is still a count.
//
// saturated true means the bitmap is too full for the estimate to mean anything
// beyond "at least this many". It is returned rather than folded into the number,
// because a saturated bitmap reports FEWER distinct values than there are, and a
// cardinality that silently stops rising reads as a field that stopped varying.
func (s sketch) estimate() (n int64, saturated bool) {
on := s.set()
if on == 0 {
return 0, false
}
z := buckets - on
if z == 0 {
return int64(math.Round(-float64(buckets) * math.Log(1/float64(buckets)))), true
}
est := -float64(buckets) * math.Log(float64(z)/float64(buckets))
return int64(math.Round(est)), float64(on)/float64(buckets) >= full
}
// encode renders a bitmap for a row. Base64 of the raw bits: fixed width, no
// interpretation, and nothing in it derived from a value that could be read back.
func (s sketch) encode() string { return base64.StdEncoding.EncodeToString(s[:]) }
// decode reads a bitmap back. An unreadable or wrong-width one yields an empty
// bitmap rather than an error: the count then restarts, which understates, and
// understating a diagnostic is preferable to refusing to serve the catalog. The
// caller reports the width mismatch through the saturation flag never being set
// on a bitmap that was thrown away, and the row is rewritten on the next flush.
func decode(s string) sketch {
var out sketch
raw, err := base64.StdEncoding.DecodeString(s)
if err != nil || len(raw) != len(out) {
return out
}
copy(out[:], raw)
return out
}
// hash is FNV-1a over the tenant, the field and the value.
func hash(org, field, value string) uint64 {
var h uint64 = 14695981039346656037
mix := func(s string) {
for i := 0; i < len(s); i++ {
h ^= uint64(s[i])
h *= 1099511628211
}
h ^= 0
h *= 1099511628211
}
mix(org)
mix(field)
mix(value)
return h
}
pkg/engine/provider.go
+21
View File
@@ -49,6 +49,22 @@ type Reference interface {
Jurisdiction(code string) (string, error)
}
// Lists answers whether a value is on one of the institution's own allow or deny
// lists.
//
// The org is the tenant KEY the transaction under evaluation carries, so a rule
// can only ever read its own institution's lists — the same value that indexes
// every other record plane, passed through rather than resolved again here.
//
// It returns an error for a list nobody declared, and for an empty value, rather
// than reporting "not listed". Both would be a rule that is present in the
// catalog, visible in the interface, counted as coverage, and incapable of
// firing — which is the failure ErrNoProvider exists to prevent, arrived at from
// the data side instead of the configuration side.
type Lists interface {
Listed(ctx context.Context, org, name, value string) (bool, error)
}
// Rate converts an amount into USD.
//
// It returns an error for a currency it does not know rather than passing the
@@ -71,6 +87,7 @@ type Providers struct {
Screen Screen
Reference Reference
Rate Rate
Lists Lists
// Now supplies the evaluation instant. Tests set it; production leaves it nil
// and gets the wall clock.
@@ -89,6 +106,7 @@ const (
capScreen capability = "screen"
capReference capability = "reference"
capRate capability = "rate"
capLists capability = "lists"
)
// vocabulary maps each term a rule expression may call to the provider that
@@ -114,6 +132,7 @@ var vocabulary = map[string]capability{
"Tier": capReference,
"USD": capRate,
"Near": capRate,
"Listed": capLists,
}
func (p Providers) has(c capability) bool {
@@ -126,6 +145,8 @@ func (p Providers) has(c capability) bool {
return p.Reference != nil
case capRate:
return p.Rate != nil
case capLists:
return p.Lists != nil
}
return false
}
pkg/engine/scope.go
+20
View File
@@ -310,6 +310,26 @@ func (s *scope) Screened(name, class string) (bool, error) {
return hit.Matched, nil
}
// Listed reports whether a value is on one of this institution's own lists.
//
// The tenant is the transaction's, never the caller's and never an argument: a
// rule that could name the org it reads lists from would be a cross-tenant read
// the rule author asserted for itself.
//
// An empty value is an error and not a miss, for the same reason Seen refuses
// one. A transaction carrying no address must not quietly satisfy — or quietly
// fail — a rule that checks addresses against a deny list: that reports "nothing
// to see" for precisely the transactions that withheld the evidence.
func (s *scope) Listed(name, value string) (bool, error) {
if name == "" {
return false, fmt.Errorf("Listed: no list named")
}
if value == "" {
return false, fmt.Errorf("Listed: value for list %q is empty", name)
}
return s.p.Lists.Listed(s.ctx, s.Tx.OrgID, name, value)
}
// Tier is the published risk tier of a jurisdiction: empty, monitoring, or
// action.
func (s *scope) Tier(code string) (string, error) {
pkg/lists/lists.go
+323
View File
@@ -0,0 +1,323 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// Package lists holds a tenant's own allow and deny lists, and answers whether a
// value is on one.
//
// A list is a control an institution states in its own terms — these addresses
// are ours, these are not to be served, this card range is under review — and it
// is referenced from a rule by name: `Listed("ip-deny", Tx.IPAddress)`. That is
// the whole of the language it adds, because a list either has a value or it does
// not, and every question that is more than that is a rule.
//
// Three properties are held up by the code rather than by discipline.
//
// A list that was never declared is an ERROR and not a miss. A rule naming a list
// nobody created would otherwise evaluate to "not listed" for every value on
// earth: the rule appears in the catalog, appears in the interface, is reported as
// coverage, and can never fire. That is the same failure the evaluator refuses at
// admission time for a missing provider, arrived at from the data side.
//
// Nothing here deletes. Removing a value stops it matching and records who
// removed it and why; the row stays, because a list entry is the reason a payment
// was refused and the disposal of evidence is the retention plane's decision and
// nobody else's (AMLR Art. 77). There is no eviction, no bound and no sweep in
// this package — grep it for Delete and the result is empty.
//
// A value is normalised by the list's CLASS before it is stored or matched, once,
// here. An address written 10.0.0.1 and ::ffff:10.0.0.1 are one address and a
// deny list that holds only one of them is a control with a documented bypass.
package lists
import (
"context"
"errors"
"fmt"
"net/netip"
"strings"
"time"
)
// Kinds. A list decides one of two things, and which it is has to be declared
// rather than inferred from the name: `partners` allows on one deployment and
// denies on another, and a rule reading the wrong sense inverts a control.
const (
Allow = "allow"
Deny = "deny"
)
// Classes are the kinds of value a list holds. It is a closed set because the
// class decides normalisation, and a list of "anything" cannot be normalised —
// which means it matches by luck.
const (
IP = "ip"
Email = "email"
Account = "account"
Device = "device"
Country = "country"
BIN = "bin"
ASN = "asn"
)
// Kinds and Classes are the closed sets, exported so a console renders the
// choices from the engine rather than from a copy of them.
var (
Kinds = []string{Allow, Deny}
Classes = []string{IP, Email, Account, Device, Country, BIN, ASN}
)
// Errors.
var (
ErrNoList = errors.New("lists: no such list")
ErrExists = errors.New("lists: a list of that name already exists")
ErrName = errors.New("lists: a list needs a name")
ErrKind = errors.New("lists: a list is allow or deny")
ErrClass = errors.New("lists: unknown class")
ErrValue = errors.New("lists: value cannot be read as this class")
ErrEmpty = errors.New("lists: no value to look up")
ErrDecider = errors.New("lists: no decider, so the entry records nobody's decision")
ErrReason = errors.New("lists: no reason, so the entry records no decision")
ErrCrowded = errors.New("lists: too many ranges on one list")
ErrNoEntry = errors.New("lists: no such entry")
ErrStore = errors.New("lists: store")
ErrRetired = errors.New("lists: entry is already removed")
ErrMaxValues = errors.New("lists: too many values in one request")
)
// MaxRanges bounds how many address RANGES one list may hold.
//
// A range cannot be looked up by equality, so every one of them is tested on
// every match. The bound is what keeps a match constant-ish rather than growing
// with the list, and it is refused at write time rather than degrading silently
// at read time. Exact values are unbounded: they are one indexed lookup however
// many there are.
const MaxRanges = 4096
// MaxValues bounds one Add. A bulk import arrives in pages; a single request that
// writes a million rows inside one transaction is a lock nobody can wait out.
const MaxValues = 1000
// List is a declared list: what it decides, what class of value it holds, and who
// declared it.
type List struct {
Org string `json:"org"`
Name string `json:"name"`
Kind string `json:"kind"`
Class string `json:"class"`
Note string `json:"note,omitempty"`
By string `json:"by"`
Added int64 `json:"added"`
Removed int64 `json:"removed"`
Ranges int64 `json:"ranges"`
Created time.Time `json:"created"`
Updated time.Time `json:"updated"`
}
// Entry is one value on a list, and the decision that put it there.
//
// Until is an operational validity window and NOT an expiry: past it the entry
// stops matching, and the row stays exactly where it is. Removed is the same
// thing reached by an operator instead of by the clock. Neither destroys
// anything, because the reason a transaction was refused outlives the control
// that refused it.
type Entry struct {
Org string `json:"org"`
List string `json:"list"`
Value string `json:"value"`
Range bool `json:"range,omitempty"`
Reason string `json:"reason"`
By string `json:"by"`
At time.Time `json:"at"`
Until time.Time `json:"until,omitzero"`
Removed time.Time `json:"removed,omitzero"`
RemoveBy string `json:"remove_by,omitempty"`
RemoveWhy string `json:"remove_why,omitempty"`
}
// Live reports whether an entry matches at the given instant.
func (e Entry) Live(at time.Time) bool {
if !e.Removed.IsZero() {
return false
}
return e.Until.IsZero() || e.Until.After(at)
}
// Match is what a lookup found: whether the value is listed, and on which entry.
//
// The entry travels with the answer because a refusal has to be able to say what
// refused it. A rule that blocks a payment on a deny list and cannot produce the
// reason and the decider behind that list entry has made a decision nobody can
// explain to the customer.
type Match struct {
Listed bool `json:"listed"`
Kind string `json:"kind,omitempty"`
Entry *Entry `json:"entry,omitempty"`
}
// Value is one value being put on a list.
type Value struct {
Value string `json:"value"`
Reason string `json:"reason"`
Until time.Time `json:"until,omitzero"`
}
// The typed operations. Each op is one In struct and one Out struct, so an HTTP
// face — this repo's Base router, or a zip typed op in the cloud mount — is a
// decode, a call, and an encode, with no second copy of the contract to drift.
type (
// DeclareIn declares a list. It is not an upsert: redeclaring an existing
// name with a different class would silently reinterpret every value already
// on it.
DeclareIn struct {
Name string `json:"name"`
Kind string `json:"kind"`
Class string `json:"class"`
Note string `json:"note,omitempty"`
By string `json:"by"`
}
// AddIn puts values on a list. Adding a value already there restates it: the
// row keeps its identity and takes the new reason, decider and window, which
// is also how a removed value is put back.
AddIn struct {
Name string `json:"name"`
Values []Value `json:"values"`
By string `json:"by"`
}
// RemoveIn takes a value off a list. The row stays; it stops matching.
RemoveIn struct {
Name string `json:"name"`
Value string `json:"value"`
Reason string `json:"reason"`
By string `json:"by"`
}
// EntriesIn reads a list. Live restricts the answer to entries that match
// right now; the default is everything, because the removed ones are the
// audit.
EntriesIn struct {
Name string `json:"name"`
Live bool `json:"live,omitempty"`
Limit int `json:"limit,omitempty"`
}
// LookupIn asks whether one value is listed.
LookupIn struct {
Name string `json:"name"`
Value string `json:"value"`
}
// CatalogIn takes no argument: a tenant's catalog is its own.
CatalogIn struct{}
// Catalog is every list this tenant has declared.
Catalog struct {
Kinds []string `json:"kinds"`
Classes []string `json:"classes"`
Lists []List `json:"lists"`
}
// Page is a list and the entries read from it.
Page struct {
List List `json:"list"`
Entries []Entry `json:"entries"`
// Cut is true when Limit stopped the read before the list ended, so an
// empty tail is never mistaken for the end of the list.
Cut bool `json:"cut,omitempty"`
}
)
// DefaultLimit is how many entries a read returns when the caller names no bound.
const DefaultLimit = 500
// normalise puts a value into the one form its class is stored and matched in,
// and reports whether it is a range.
//
// Every class that has a canonical form gets one here, once. The alternative is a
// deny list holding "[email protected] " that never matches "[email protected]",
// which is a control that reports itself installed and is not.
func normalise(class, value string) (norm string, isRange bool, err error) {
v := strings.TrimSpace(value)
if v == "" {
return "", false, ErrEmpty
}
switch class {
case IP:
if strings.Contains(v, "/") {
p, perr := netip.ParsePrefix(v)
if perr != nil {
return "", false, fmt.Errorf("%w: %q is not an address range: %v", ErrValue, value, perr)
}
// Masked, so 10.0.0.7/8 and 10.0.0.0/8 are one range rather than two
// rows that test identically.
return p.Masked().String(), true, nil
}
a, aerr := netip.ParseAddr(v)
if aerr != nil {
return "", false, fmt.Errorf("%w: %q is not an address: %v", ErrValue, value, aerr)
}
return a.Unmap().String(), false, nil
case Email:
v = strings.ToLower(v)
if strings.Count(v, "@") != 1 || strings.HasPrefix(v, "@") || strings.HasSuffix(v, "@") {
return "", false, fmt.Errorf("%w: %q is not an address", ErrValue, value)
}
return v, false, nil
case Country:
v = strings.ToUpper(v)
if len(v) != 2 {
return "", false, fmt.Errorf("%w: %q is not an ISO 3166-1 alpha-2 code", ErrValue, value)
}
return v, false, nil
case BIN, ASN:
// Digits only, so a BIN written with spaces or an ASN written AS15169 is
// one value rather than three.
v = strings.TrimPrefix(strings.ToUpper(v), "AS")
v = strings.Map(func(r rune) rune {
if r >= '0' && r <= '9' {
return r
}
return -1
}, v)
if v == "" {
return "", false, fmt.Errorf("%w: %q holds no digits", ErrValue, value)
}
return v, false, nil
case Account, Device:
// Identifiers an institution issues are case-sensitive and opaque, so
// trimming is the only safe normalisation.
return v, false, nil
default:
return "", false, fmt.Errorf("%w: %q", ErrClass, class)
}
}
// contains reports whether a normalised address falls inside a normalised range.
func contains(rangeCIDR, addr string) bool {
p, err := netip.ParsePrefix(rangeCIDR)
if err != nil {
return false
}
a, err := netip.ParseAddr(addr)
if err != nil {
return false
}
return p.Contains(a.Unmap())
}
// Listed answers the rule vocabulary's question: is this value on this list.
//
// It satisfies engine.Lists, and its refusals are the point. An empty value is an
// error, because a transaction that carries no address must not quietly pass a
// rule that checks addresses against a deny list — that is silence reading as a
// clean result for precisely the transactions that withheld the evidence. A list
// that does not exist is an error for the same reason.
func (s *Shelf) Listed(ctx context.Context, org, name, value string) (bool, error) {
m, err := s.Lookup(ctx, org, &LookupIn{Name: name, Value: value})
if err != nil {
return false, err
}
return m.Listed, nil
}
pkg/lists/lists_test.go
+380
View File
@@ -0,0 +1,380 @@
package lists
import (
"context"
"errors"
"strings"
"testing"
"time"
"github.com/hanzoai/base/core"
"github.com/luxfi/aml/internal/instance"
)
const (
acme = "hanzo/acme"
rival = "hanzo/rival"
// other is the SAME org name under a different brand. Every isolation test
// uses it as well as a different org, because the brand half of the key is the
// half a bare-org regression would drop.
other = "zoo/acme"
)
func shelf(t *testing.T) (*Shelf, core.App) {
t.Helper()
app := instance.New(t)
t.Cleanup(app.Cleanup)
if err := Ensure(app); err != nil {
t.Fatalf("ensure: %v", err)
}
return NewBase(app), app
}
func declare(t *testing.T, s *Shelf, org, name, kind, class string) {
t.Helper()
if _, err := s.Declare(context.Background(), org, &DeclareIn{
Name: name, Kind: kind, Class: class, By: "a.mensah",
}); err != nil {
t.Fatalf("declare %s: %v", name, err)
}
}
func add(t *testing.T, s *Shelf, org, name string, values ...Value) {
t.Helper()
if _, err := s.Add(context.Background(), org, &AddIn{Name: name, Values: values, By: "a.mensah"}); err != nil {
t.Fatalf("add to %s: %v", name, err)
}
}
func listed(t *testing.T, s *Shelf, org, name, value string) bool {
t.Helper()
hit, err := s.Listed(context.Background(), org, name, value)
if err != nil {
t.Fatalf("listed %s/%s: %v", name, value, err)
}
return hit
}
// TestListNobodyDeclaredIsAnError is the reason this package refuses rather than
// reporting a miss. A rule naming a list that does not exist would otherwise be
// installed, catalogued, reported as coverage, and incapable of firing.
func TestListNobodyDeclaredIsAnError(t *testing.T) {
s, _ := shelf(t)
_, err := s.Listed(context.Background(), acme, "ip-deny", "10.0.0.1")
if !errors.Is(err, ErrNoList) {
t.Fatalf("a rule naming an undeclared list must error, got %v", err)
}
}
// TestEmptyValueIsAnError: a transaction carrying no address must not quietly
// pass an address deny list.
func TestEmptyValueIsAnError(t *testing.T) {
s, _ := shelf(t)
declare(t, s, acme, "ip-deny", Deny, IP)
if _, err := s.Listed(context.Background(), acme, "ip-deny", ""); !errors.Is(err, ErrEmpty) {
t.Fatalf("an empty value must error, got %v", err)
}
}
// TestValuesAreNormalisedByClass. One address written two ways is one address; a
// deny list holding only one spelling is a control with a documented bypass.
func TestValuesAreNormalisedByClass(t *testing.T) {
s, _ := shelf(t)
declare(t, s, acme, "ip-deny", Deny, IP)
declare(t, s, acme, "mail-deny", Deny, Email)
declare(t, s, acme, "bin-watch", Deny, BIN)
add(t, s, acme, "ip-deny", Value{Value: "10.0.0.1", Reason: "known proxy"})
add(t, s, acme, "mail-deny", Value{Value: " [email protected] ", Reason: "chargebacks"})
add(t, s, acme, "bin-watch", Value{Value: "4111 11", Reason: "test range"})
for _, c := range []struct{ list, value string }{
{"ip-deny", "::ffff:10.0.0.1"},
{"ip-deny", "10.0.0.1"},
{"mail-deny", "[email protected]"},
{"mail-deny", "[email protected]"},
{"bin-watch", "411111"},
{"bin-watch", "4111-11"},
} {
if !listed(t, s, acme, c.list, c.value) {
t.Errorf("%s should hold %q after normalisation", c.list, c.value)
}
}
if listed(t, s, acme, "ip-deny", "10.0.0.2") {
t.Error("an address that was never listed must not match")
}
}
// TestRangesMatchAddresses. A deny list of single addresses is not a usable
// control against a hosting range.
func TestRangesMatchAddresses(t *testing.T) {
s, _ := shelf(t)
declare(t, s, acme, "ip-deny", Deny, IP)
add(t, s, acme, "ip-deny", Value{Value: "10.0.0.7/8", Reason: "hosting range"})
if !listed(t, s, acme, "ip-deny", "10.9.9.9") {
t.Fatal("an address inside a listed range must match")
}
if listed(t, s, acme, "ip-deny", "11.0.0.1") {
t.Fatal("an address outside every listed range must not match")
}
// A range is stored masked, so 10.0.0.7/8 and 10.0.0.0/8 are one row.
page, err := s.Entries(context.Background(), acme, &EntriesIn{Name: "ip-deny"})
if err != nil {
t.Fatal(err)
}
if len(page.Entries) != 1 || page.Entries[0].Value != "10.0.0.0/8" {
t.Fatalf("a range must be stored masked, got %+v", page.Entries)
}
}
// TestRemovalKeepsTheRecord. A list entry is the reason a payment was refused; it
// stops matching and it does not go.
func TestRemovalKeepsTheRecord(t *testing.T) {
s, _ := shelf(t)
declare(t, s, acme, "ip-deny", Deny, IP)
add(t, s, acme, "ip-deny", Value{Value: "10.0.0.1", Reason: "known proxy"})
if _, err := s.Remove(context.Background(), acme, &RemoveIn{
Name: "ip-deny", Value: "10.0.0.1", Reason: "customer appealed", By: "r.okafor",
}); err != nil {
t.Fatalf("remove: %v", err)
}
if listed(t, s, acme, "ip-deny", "10.0.0.1") {
t.Fatal("a removed value must stop matching")
}
page, err := s.Entries(context.Background(), acme, &EntriesIn{Name: "ip-deny"})
if err != nil {
t.Fatal(err)
}
if len(page.Entries) != 1 {
t.Fatalf("removal destroyed the record: %d entries, want 1", len(page.Entries))
}
e := page.Entries[0]
if e.Removed.IsZero() || e.RemoveBy != "r.okafor" || e.RemoveWhy != "customer appealed" {
t.Fatalf("the removal must name its decider and its reason: %+v", e)
}
if e.Reason != "known proxy" || e.By != "a.mensah" {
t.Fatalf("the original decision must survive the removal: %+v", e)
}
live, err := s.Entries(context.Background(), acme, &EntriesIn{Name: "ip-deny", Live: true})
if err != nil {
t.Fatal(err)
}
if len(live.Entries) != 0 {
t.Fatalf("a removed entry is not live: %+v", live.Entries)
}
}
// TestValidityWindowStopsMatchingWithoutDeleting.
func TestValidityWindowStopsMatchingWithoutDeleting(t *testing.T) {
now := time.Date(2026, 3, 1, 12, 0, 0, 0, time.UTC)
s, _ := shelf(t)
s.Now = func() time.Time { return now }
declare(t, s, acme, "ip-deny", Deny, IP)
add(t, s, acme, "ip-deny", Value{Value: "10.0.0.1", Reason: "temporary", Until: now.Add(time.Hour)})
if !listed(t, s, acme, "ip-deny", "10.0.0.1") {
t.Fatal("an entry inside its window must match")
}
s.Now = func() time.Time { return now.Add(2 * time.Hour) }
if listed(t, s, acme, "ip-deny", "10.0.0.1") {
t.Fatal("an entry past its window must not match")
}
page, err := s.Entries(context.Background(), acme, &EntriesIn{Name: "ip-deny"})
if err != nil {
t.Fatal(err)
}
if len(page.Entries) != 1 {
t.Fatal("a window closing must not destroy the row — only retention decides that")
}
}
// TestRestatementPutsAValueBack, and it names who put it back.
func TestRestatementPutsAValueBack(t *testing.T) {
s, _ := shelf(t)
declare(t, s, acme, "ip-deny", Deny, IP)
add(t, s, acme, "ip-deny", Value{Value: "10.0.0.1", Reason: "known proxy"})
if _, err := s.Remove(context.Background(), acme, &RemoveIn{
Name: "ip-deny", Value: "10.0.0.1", Reason: "appealed", By: "r.okafor",
}); err != nil {
t.Fatal(err)
}
if _, err := s.Add(context.Background(), acme, &AddIn{
Name: "ip-deny", By: "s.tan", Values: []Value{{Value: "10.0.0.1", Reason: "reoffended"}},
}); err != nil {
t.Fatal(err)
}
if !listed(t, s, acme, "ip-deny", "10.0.0.1") {
t.Fatal("a restated value must match again")
}
page, _ := s.Entries(context.Background(), acme, &EntriesIn{Name: "ip-deny"})
if len(page.Entries) != 1 {
t.Fatalf("a restatement must not double the row: %d", len(page.Entries))
}
if page.Entries[0].By != "s.tan" || page.Entries[0].Reason != "reoffended" {
t.Fatalf("the row must name the current decision: %+v", page.Entries[0])
}
}
// TestEveryDecisionNamesADeciderAndAReason.
func TestEveryDecisionNamesADeciderAndAReason(t *testing.T) {
s, _ := shelf(t)
ctx := context.Background()
if _, err := s.Declare(ctx, acme, &DeclareIn{Name: "x", Kind: Deny, Class: IP}); !errors.Is(err, ErrDecider) {
t.Errorf("declaring without a decider must be refused, got %v", err)
}
declare(t, s, acme, "ip-deny", Deny, IP)
if _, err := s.Add(ctx, acme, &AddIn{Name: "ip-deny", Values: []Value{{Value: "1.1.1.1"}}, By: "a"}); !errors.Is(err, ErrReason) {
t.Errorf("adding without a reason must be refused, got %v", err)
}
if _, err := s.Add(ctx, acme, &AddIn{Name: "ip-deny", Values: []Value{{Value: "1.1.1.1", Reason: "r"}}}); !errors.Is(err, ErrDecider) {
t.Errorf("adding without a decider must be refused, got %v", err)
}
if _, err := s.Remove(ctx, acme, &RemoveIn{Name: "ip-deny", Value: "1.1.1.1", By: "a"}); !errors.Is(err, ErrReason) {
t.Errorf("removing without a reason must be refused, got %v", err)
}
}
// TestPartialImportIsRefused: one unreadable value refuses the whole request
// rather than landing half of it, because a list whose contents nobody can state
// is not a control.
func TestPartialImportIsRefused(t *testing.T) {
s, _ := shelf(t)
declare(t, s, acme, "ip-deny", Deny, IP)
_, err := s.Add(context.Background(), acme, &AddIn{
Name: "ip-deny", By: "a.mensah",
Values: []Value{{Value: "10.0.0.1", Reason: "ok"}, {Value: "not-an-address", Reason: "ok"}},
})
if !errors.Is(err, ErrValue) {
t.Fatalf("an unreadable value must refuse the request, got %v", err)
}
page, _ := s.Entries(context.Background(), acme, &EntriesIn{Name: "ip-deny"})
if len(page.Entries) != 0 {
t.Fatalf("a refused import must have written nothing: %+v", page.Entries)
}
}
// TestTenantIsolation is the property the whole product rests on, exercised over
// every operation this package has.
//
// The second tenant is the SAME org name under a different brand, which is the
// case a bare-org regression collides. Reads answer as though the other tenant's
// list does not exist, because a refusal that distinguished "not yours" from "not
// there" is a probe for which institutions exist.
func TestTenantIsolation(t *testing.T) {
s, _ := shelf(t)
ctx := context.Background()
declare(t, s, acme, "ip-deny", Deny, IP)
add(t, s, acme, "ip-deny", Value{Value: "10.0.0.1", Reason: "known proxy"})
for _, stranger := range []string{other, rival} {
if _, err := s.Listed(ctx, stranger, "ip-deny", "10.0.0.1"); !errors.Is(err, ErrNoList) {
t.Errorf("%s can see %s's list: %v", stranger, acme, err)
}
if _, err := s.Entries(ctx, stranger, &EntriesIn{Name: "ip-deny"}); !errors.Is(err, ErrNoList) {
t.Errorf("%s can read %s's entries: %v", stranger, acme, err)
}
if _, err := s.Remove(ctx, stranger, &RemoveIn{Name: "ip-deny", Value: "10.0.0.1", Reason: "r", By: "b"}); !errors.Is(err, ErrNoList) {
t.Errorf("%s can remove from %s's list: %v", stranger, acme, err)
}
if _, err := s.Add(ctx, stranger, &AddIn{Name: "ip-deny", By: "b", Values: []Value{{Value: "9.9.9.9", Reason: "r"}}}); !errors.Is(err, ErrNoList) {
t.Errorf("%s can write to %s's list: %v", stranger, acme, err)
}
cat, err := s.Catalog(ctx, stranger, &CatalogIn{})
if err != nil {
t.Fatal(err)
}
if len(cat.Lists) != 0 {
t.Errorf("%s's catalog shows %d of another tenant's lists", stranger, len(cat.Lists))
}
}
// And the two brands may each hold a list of the same name without either
// seeing the other's values.
declare(t, s, other, "ip-deny", Allow, IP)
add(t, s, other, "ip-deny", Value{Value: "203.0.113.5", Reason: "our own egress"})
if listed(t, s, acme, "ip-deny", "203.0.113.5") {
t.Fatal("one brand's entry matched under another brand's tenant of the same org name")
}
if listed(t, s, other, "ip-deny", "10.0.0.1") {
t.Fatal("one brand's entry matched under another brand's tenant of the same org name")
}
}
// TestBareOrgIsRefused at the write boundary of every operation. The check is in
// the plane and not only where the key is minted, because the identity comes from
// a seam the deployment supplies.
func TestBareOrgIsRefused(t *testing.T) {
s, _ := shelf(t)
ctx := context.Background()
for _, bare := range []string{"acme", "", "unknown/acme"} {
if _, err := s.Declare(ctx, bare, &DeclareIn{Name: "x", Kind: Deny, Class: IP, By: "a"}); err == nil {
t.Errorf("Declare accepted %q as a tenant", bare)
}
if _, err := s.Catalog(ctx, bare, &CatalogIn{}); err == nil {
t.Errorf("Catalog accepted %q as a tenant", bare)
}
if _, err := s.Lookup(ctx, bare, &LookupIn{Name: "x", Value: "1.1.1.1"}); err == nil {
t.Errorf("Lookup accepted %q as a tenant", bare)
}
}
}
// TestNothingDeletes reads the package's own source for a destructive call.
//
// Stated as a test rather than as a comment, because the property is what the
// retention plane's exclusivity rests on: if any operational knob can destroy a
// list entry, the five-year record is whatever survived the operator.
func TestNothingDeletes(t *testing.T) {
for _, name := range []string{"lists.go", "shelf.go"} {
src := read(t, name)
if strings.Contains(src, ".Delete(") {
t.Errorf("%s calls Delete: disposal is pkg/retention's decision and nobody else's", name)
}
}
}
// TestRestart is the durability half: what a restarted pod reads.
func TestRestart(t *testing.T) {
first := instance.New(t)
if err := Ensure(first); err != nil {
t.Fatal(err)
}
before := NewBase(first)
declare(t, before, acme, "ip-deny", Deny, IP)
add(t, before, acme, "ip-deny", Value{Value: "10.0.0.1", Reason: "known proxy"})
declare(t, before, other, "ip-deny", Allow, IP)
add(t, before, other, "ip-deny", Value{Value: "203.0.113.5", Reason: "our own egress"})
second := instance.Restart(t, first)
if err := Ensure(second); err != nil {
t.Fatal(err)
}
after := NewBase(second)
if !listed(t, after, acme, "ip-deny", "10.0.0.1") {
t.Fatal("the deny list did not survive the restart — the control was off after the rollout and nothing would have said so")
}
page, err := after.Entries(context.Background(), acme, &EntriesIn{Name: "ip-deny"})
if err != nil {
t.Fatal(err)
}
if len(page.Entries) != 1 || page.Entries[0].Reason != "known proxy" || page.Entries[0].By != "a.mensah" {
t.Fatalf("the decision behind the entry did not survive: %+v", page.Entries)
}
if page.List.Added != 1 {
t.Fatalf("the list's own counters did not survive: %+v", page.List)
}
// Tenancy still holds on the other side of the restart.
if listed(t, after, other, "ip-deny", "10.0.0.1") {
t.Fatal("the tenant boundary did not survive the restart")
}
if _, err := after.Listed(context.Background(), rival, "ip-deny", "10.0.0.1"); !errors.Is(err, ErrNoList) {
t.Fatal("a third tenant can see a list after the restart")
}
}
pkg/lists/shelf.go
+495
View File
@@ -0,0 +1,495 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package lists
import (
"context"
"errors"
"fmt"
"strings"
"time"
"github.com/hanzoai/base/core"
"github.com/hanzoai/dbx"
"github.com/luxfi/aml/pkg/brand"
"github.com/luxfi/aml/pkg/store"
)
// Field names. Constants because the writer and the reader must name the same
// column: a silent disagreement between them returns nothing, and nothing is
// indistinguishable from a list with no entries.
const (
fieldName = "name"
fieldKind = "kind"
fieldClass = "class"
fieldNote = "note"
fieldBy = "by"
fieldAdded = "added"
fieldRemoved = "removed"
fieldRanges = "ranges"
fieldCreated = "created"
fieldUpdated = "updated"
fieldList = "list"
fieldValue = "value"
fieldRange = "range"
fieldReason = "reason"
fieldAt = "at"
fieldUntil = "until"
fieldOff = "off"
fieldOffBy = "off_by"
fieldOffWhy = "off_why"
)
var listKind = store.Kind{
Name: "aml_lists",
Fields: []core.Field{
&core.TextField{Name: fieldName, Required: true},
&core.TextField{Name: fieldKind, Required: true},
&core.TextField{Name: fieldClass, Required: true},
&core.TextField{Name: fieldNote},
&core.TextField{Name: fieldBy, Required: true},
&core.NumberField{Name: fieldAdded},
&core.NumberField{Name: fieldRemoved},
&core.NumberField{Name: fieldRanges},
&core.DateField{Name: fieldCreated, Required: true},
&core.DateField{Name: fieldUpdated, Required: true},
},
Indexes: []store.Index{
// One list of a name per tenant. Unique on the tenant AND the name, never
// on the name alone: two institutions naming a list `deny` is ordinary.
{Name: "name", Fields: []string{store.Org, fieldName}, Unique: true},
},
}
var entryKind = store.Kind{
Name: "aml_list_entries",
Fields: []core.Field{
&core.TextField{Name: fieldList, Required: true},
&core.TextField{Name: fieldValue, Required: true},
&core.BoolField{Name: fieldRange},
&core.TextField{Name: fieldReason, Required: true},
&core.TextField{Name: fieldBy, Required: true},
&core.DateField{Name: fieldAt, Required: true},
&core.DateField{Name: fieldUntil},
&core.DateField{Name: fieldOff},
&core.TextField{Name: fieldOffBy},
&core.TextField{Name: fieldOffWhy},
},
Indexes: []store.Index{
// The match: this tenant, this list, this exact value. Unique, so one
// value is one row and a restatement cannot double it.
{Name: "value", Fields: []string{store.Org, fieldList, fieldValue}, Unique: true},
// The range scan: this tenant's ranges on this list, which is the only
// read that cannot be an equality.
{Name: "range", Fields: []string{store.Org, fieldList, fieldRange}},
},
}
// Ensure creates what lists are kept in. Idempotent, so it runs on every start.
func Ensure(app core.App) error {
if err := listKind.Ensure(app); err != nil {
return err
}
return entryKind.Ensure(app)
}
// Shelf is the durable list plane. There is no memory implementation: a deny list
// that empties on a rollout is a control that was off, and the difference is
// invisible from the outside because "not listed" is the ordinary answer.
type Shelf struct {
app core.App
// Now supplies the instant a validity window is judged against. Tests set it;
// production leaves it nil and gets the wall clock.
Now func() time.Time
}
// NewBase returns the durable shelf. Ensure has to have run first.
func NewBase(app core.App) *Shelf { return &Shelf{app: app} }
func (s *Shelf) now() time.Time {
if s.Now != nil {
return s.Now().UTC()
}
return time.Now().UTC()
}
// Declare creates a list.
func (s *Shelf) Declare(ctx context.Context, org string, in *DeclareIn) (*List, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
name := trim(in.Name)
switch {
case name == "":
return nil, ErrName
case in.Kind != Allow && in.Kind != Deny:
return nil, fmt.Errorf("%w: %q", ErrKind, in.Kind)
case !known(Classes, in.Class):
return nil, fmt.Errorf("%w: %q", ErrClass, in.Class)
case trim(in.By) == "":
return nil, ErrDecider
}
if _, err := s.list(org, name); err == nil {
return nil, fmt.Errorf("%w: %s", ErrExists, name)
} else if !isNoList(err) {
return nil, err
}
at := s.now()
row, err := listKind.New(s.app, org)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
l := List{Org: org, Name: name, Kind: in.Kind, Class: in.Class, Note: trim(in.Note), By: trim(in.By), Created: at, Updated: at}
writeList(row, l)
if err := s.app.Save(row); err != nil {
return nil, fmt.Errorf("%w: declaring %s: %w", ErrStore, name, err)
}
return &l, nil
}
// Catalog is every list this tenant has declared.
func (s *Shelf) Catalog(ctx context.Context, org string, _ *CatalogIn) (*Catalog, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
rows, err := listKind.Find(s.app, org, "", fieldName, 0, nil)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
out := &Catalog{Kinds: Kinds, Classes: Classes, Lists: make([]List, 0, len(rows))}
for _, row := range rows {
out.Lists = append(out.Lists, readList(row))
}
return out, nil
}
// Add puts values on a list, in one transaction: a half-written import is a
// control whose contents nobody can state.
func (s *Shelf) Add(ctx context.Context, org string, in *AddIn) (*List, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
if trim(in.By) == "" {
return nil, ErrDecider
}
if len(in.Values) == 0 {
return nil, ErrEmpty
}
if len(in.Values) > MaxValues {
return nil, fmt.Errorf("%w: %d values, at most %d", ErrMaxValues, len(in.Values), MaxValues)
}
l, err := s.list(org, trim(in.Name))
if err != nil {
return nil, err
}
// Normalise and check everything before writing anything, so one unreadable
// value refuses the request instead of landing half of it.
type prepared struct {
value string
isRange bool
reason string
until time.Time
}
ready := make([]prepared, 0, len(in.Values))
for _, v := range in.Values {
norm, isRange, err := normalise(l.Class, v.Value)
if err != nil {
return nil, err
}
if trim(v.Reason) == "" {
return nil, fmt.Errorf("%w: %s", ErrReason, v.Value)
}
ready = append(ready, prepared{value: norm, isRange: isRange, reason: trim(v.Reason), until: v.Until.UTC()})
}
at := s.now()
by := trim(in.By)
updated := *l
err = s.app.RunInTransaction(func(tx core.App) error {
for _, p := range ready {
found, ferr := entryKind.Find(tx, org, fieldList+" = {:list} && "+fieldValue+" = {:value}", "", 1,
dbx.Params{"list": l.Name, "value": p.value})
if ferr != nil {
return fmt.Errorf("%w: %w", ErrStore, ferr)
}
var row *core.Record
if len(found) > 0 {
row = found[0]
// A restatement clears the removal: the value is on the list again,
// and the row names who put it back.
row.Set(fieldOff, "")
row.Set(fieldOffBy, "")
row.Set(fieldOffWhy, "")
} else {
if p.isRange {
if updated.Ranges >= MaxRanges {
return fmt.Errorf("%w: %s holds %d, at most %d", ErrCrowded, l.Name, updated.Ranges, MaxRanges)
}
updated.Ranges++
}
updated.Added++
var nerr error
row, nerr = entryKind.New(tx, org)
if nerr != nil {
return fmt.Errorf("%w: %w", ErrStore, nerr)
}
row.Set(fieldList, l.Name)
row.Set(fieldValue, p.value)
row.Set(fieldRange, p.isRange)
}
row.Set(fieldReason, p.reason)
row.Set(fieldBy, by)
row.Set(fieldAt, at)
row.Set(fieldUntil, moment(p.until))
if serr := tx.Save(row); serr != nil {
return fmt.Errorf("%w: adding %s to %s: %w", ErrStore, p.value, l.Name, serr)
}
}
updated.Updated = at
return s.saveList(tx, updated)
})
if err != nil {
return nil, err
}
return &updated, nil
}
// Remove takes a value off a list without destroying the record of it having been
// there. The row keeps its reason and its original decider and gains the removal's
// own.
func (s *Shelf) Remove(ctx context.Context, org string, in *RemoveIn) (*List, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
if trim(in.By) == "" {
return nil, ErrDecider
}
if trim(in.Reason) == "" {
return nil, ErrReason
}
l, err := s.list(org, trim(in.Name))
if err != nil {
return nil, err
}
norm, _, err := normalise(l.Class, in.Value)
if err != nil {
return nil, err
}
at := s.now()
updated := *l
err = s.app.RunInTransaction(func(tx core.App) error {
found, ferr := entryKind.Find(tx, org, fieldList+" = {:list} && "+fieldValue+" = {:value}", "", 1,
dbx.Params{"list": l.Name, "value": norm})
if ferr != nil {
return fmt.Errorf("%w: %w", ErrStore, ferr)
}
if len(found) == 0 {
return fmt.Errorf("%w: %s on %s", ErrNoEntry, norm, l.Name)
}
row := found[0]
if row.GetString(fieldOff) != "" {
return fmt.Errorf("%w: %s on %s", ErrRetired, norm, l.Name)
}
row.Set(fieldOff, at)
row.Set(fieldOffBy, trim(in.By))
row.Set(fieldOffWhy, trim(in.Reason))
if serr := tx.Save(row); serr != nil {
return fmt.Errorf("%w: removing %s from %s: %w", ErrStore, norm, l.Name, serr)
}
updated.Removed++
updated.Updated = at
return s.saveList(tx, updated)
})
if err != nil {
return nil, err
}
return &updated, nil
}
// Entries reads a list.
func (s *Shelf) Entries(ctx context.Context, org string, in *EntriesIn) (*Page, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
l, err := s.list(org, trim(in.Name))
if err != nil {
return nil, err
}
limit := in.Limit
if limit <= 0 {
limit = DefaultLimit
}
// One more than asked, so a full page can be told from the end of the list.
rows, err := entryKind.Find(s.app, org, fieldList+" = {:list}", fieldValue, limit+1, dbx.Params{"list": l.Name})
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
page := &Page{List: *l, Entries: make([]Entry, 0, len(rows))}
if len(rows) > limit {
rows, page.Cut = rows[:limit], true
}
at := s.now()
for _, row := range rows {
e := readEntry(row)
if in.Live && !e.Live(at) {
continue
}
page.Entries = append(page.Entries, e)
}
return page, nil
}
// Lookup answers whether one value is on one list, and on what entry.
func (s *Shelf) Lookup(ctx context.Context, org string, in *LookupIn) (*Match, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
l, err := s.list(org, trim(in.Name))
if err != nil {
return nil, err
}
norm, isRange, err := normalise(l.Class, in.Value)
if err != nil {
return nil, err
}
if isRange {
return nil, fmt.Errorf("%w: %q is a range, and a range is not a value to look up", ErrValue, in.Value)
}
at := s.now()
rows, err := entryKind.Find(s.app, org, fieldList+" = {:list} && "+fieldValue+" = {:value}", "", 1,
dbx.Params{"list": l.Name, "value": norm})
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
if len(rows) > 0 {
if e := readEntry(rows[0]); e.Live(at) {
return &Match{Listed: true, Kind: l.Kind, Entry: &e}, nil
}
}
if l.Class != IP || l.Ranges == 0 {
return &Match{Kind: l.Kind}, nil
}
// The ranges. Bounded by MaxRanges at write time, so this is a bounded read
// and not a scan that grows with the list.
ranges, err := entryKind.Find(s.app, org, fieldList+" = {:list} && "+fieldRange+" = true", fieldValue, MaxRanges,
dbx.Params{"list": l.Name})
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
for _, row := range ranges {
e := readEntry(row)
if e.Live(at) && contains(e.Value, norm) {
return &Match{Listed: true, Kind: l.Kind, Entry: &e}, nil
}
}
return &Match{Kind: l.Kind}, nil
}
// list reads one declaration, or reports that the caller named a list nobody
// declared.
func (s *Shelf) list(org, name string) (*List, error) {
if name == "" {
return nil, ErrName
}
rows, err := listKind.Find(s.app, org, fieldName+" = {:name}", "", 1, dbx.Params{"name": name})
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
if len(rows) == 0 {
return nil, fmt.Errorf("%w: %s", ErrNoList, name)
}
l := readList(rows[0])
return &l, nil
}
func (s *Shelf) saveList(app core.App, l List) error {
rows, err := listKind.Find(app, l.Org, fieldName+" = {:name}", "", 1, dbx.Params{"name": l.Name})
if err != nil {
return fmt.Errorf("%w: %w", ErrStore, err)
}
if len(rows) == 0 {
return fmt.Errorf("%w: %s", ErrNoList, l.Name)
}
writeList(rows[0], l)
if err := app.Save(rows[0]); err != nil {
return fmt.Errorf("%w: %w", ErrStore, err)
}
return nil
}
func writeList(row *core.Record, l List) {
row.Set(fieldName, l.Name)
row.Set(fieldKind, l.Kind)
row.Set(fieldClass, l.Class)
row.Set(fieldNote, l.Note)
row.Set(fieldBy, l.By)
row.Set(fieldAdded, l.Added)
row.Set(fieldRemoved, l.Removed)
row.Set(fieldRanges, l.Ranges)
row.Set(fieldCreated, l.Created.UTC())
row.Set(fieldUpdated, l.Updated.UTC())
}
func readList(row *core.Record) List {
return List{
Org: row.GetString(store.Org),
Name: row.GetString(fieldName),
Kind: row.GetString(fieldKind),
Class: row.GetString(fieldClass),
Note: row.GetString(fieldNote),
By: row.GetString(fieldBy),
Added: int64(row.GetInt(fieldAdded)),
Removed: int64(row.GetInt(fieldRemoved)),
Ranges: int64(row.GetInt(fieldRanges)),
Created: row.GetDateTime(fieldCreated).Time(),
Updated: row.GetDateTime(fieldUpdated).Time(),
}
}
func readEntry(row *core.Record) Entry {
return Entry{
Org: row.GetString(store.Org),
List: row.GetString(fieldList),
Value: row.GetString(fieldValue),
Range: row.GetBool(fieldRange),
Reason: row.GetString(fieldReason),
By: row.GetString(fieldBy),
At: row.GetDateTime(fieldAt).Time(),
Until: row.GetDateTime(fieldUntil).Time(),
Removed: row.GetDateTime(fieldOff).Time(),
RemoveBy: row.GetString(fieldOffBy),
RemoveWhy: row.GetString(fieldOffWhy),
}
}
// moment renders an optional instant the way a date column holds "unset": the
// empty string, which is what the filters compare against. A zero time written as
// a date is year one, which sorts and compares as a real moment in the past.
func moment(t time.Time) any {
if t.IsZero() {
return ""
}
return t.UTC()
}
func trim(s string) string { return strings.TrimSpace(s) }
func known(set []string, v string) bool {
for _, s := range set {
if s == v {
return true
}
}
return false
}
func isNoList(err error) bool { return errors.Is(err, ErrNoList) }
pkg/lists/source_test.go
+16
View File
@@ -0,0 +1,16 @@
package lists
import (
"os"
"testing"
)
// read is how the source-level invariants read this package's own files.
func read(t *testing.T, name string) string {
t.Helper()
raw, err := os.ReadFile(name)
if err != nil {
t.Fatalf("read %s: %v", name, err)
}
return string(raw)
}
pkg/models/models.go
+162
View File
@@ -0,0 +1,162 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// Package models keeps the record of what a tenant's model is, and of every
// study that produced it.
//
// Two things are kept, and both are kept because both are governed acts.
//
// A RUN is a search over the space of model shapes (pkg/topology) against this
// tenant's own history. JMLSG 5.7.18 requires new detection to be implemented and
// tested before live activation and FCG 3.2.5A requires a retirement to be
// justified against the outgoing configuration's performance; a search is how
// both questions get an answer, and an answer nobody can retrieve six months
// later did not answer them. The run holds the space asked for, the options,
// every trial, and the seed — so the whole thing can be re-run and checked rather
// than believed.
//
// A FIT is learned state built from this tenant's history under one shape. It is
// what makes a recommendation actionable: adopt it and the model starts warm
// rather than blind. Adoption is recorded with its decider, because a model that
// changed and cannot say who changed it is a control nobody owns.
//
// # The learned state never leaves the tenant's store
//
// A fit's snapshot is written and read here and is not on any answer this package
// returns. Mass counters describe where a tenant's activity is dense; handing
// them out over an API would publish the shape of an institution's customer
// behaviour to whoever holds a token, which is a disclosure the alerts themselves
// are careful not to make. Callers get the digest, the trial and whether the fit
// is adoptable — everything needed to decide, and nothing that describes the
// customers.
//
// Nothing here deletes and nothing evicts, on the same terms as every other
// record plane: disposal is pkg/retention's decision alone.
package models
import (
"errors"
"time"
"github.com/luxfi/aml/pkg/topology"
)
// Errors.
var (
ErrNoHistory = errors.New("models: no history source is wired, so a search has nothing to replay")
ErrNoModel = errors.New("models: no live model is wired, so nothing can be adopted into one")
ErrNotHere = errors.New("models: no such run")
ErrNoFit = errors.New("models: no such fit")
ErrDecider = errors.New("models: no decider, so the record names nobody's decision")
ErrShape = errors.New("models: the fitted shape is not the shape the model is running")
ErrAdopted = errors.New("models: already adopted")
ErrStore = errors.New("models: store")
)
// Summary is a run as a list reads it: enough to choose one to open, and none of
// the trials.
//
// A run over a full grid carries a curve per candidate, which is the right size
// for one answer and the wrong size for a page of twenty. The two are separate
// shapes rather than one shape with a flag, so a list can never accidentally be
// the heavy read.
type Summary struct {
ID string `json:"id"`
At time.Time `json:"at"`
By string `json:"by"`
Trials int `json:"trials"`
Events int `json:"events"`
Judged int `json:"judged"`
Elapsed time.Duration `json:"elapsed"`
// Winner is the shape the run recommended, and Refusal is why it recommended
// none. Exactly one of them is set, on the summary as on the report.
Winner *topology.Topology `json:"winner,omitempty"`
Refusal string `json:"refusal,omitempty"`
}
// Run is a whole search, kept.
type Run struct {
ID string `json:"id"`
Org string `json:"org"`
At time.Time `json:"at"`
By string `json:"by"`
Space topology.Space `json:"space"`
Options topology.Options `json:"options"`
Report topology.Report `json:"report"`
}
// Fit is learned state built under one shape, as a caller sees it.
//
// The snapshot itself is deliberately absent — see the package doc. Digest is
// what identifies the shape, and Adoptable says whether the live model is running
// that shape today.
type Fit struct {
ID string `json:"id"`
Org string `json:"org"`
At time.Time `json:"at"`
By string `json:"by"`
Topology topology.Topology `json:"topology"`
Digest string `json:"digest"`
Trial topology.Trial `json:"trial"`
// Adoptable is whether this fit's shape matches the model this deployment is
// running. A fit of a different shape is a recommendation to change the
// deployment, not something that can be installed by restoring a file.
Adoptable bool `json:"adoptable"`
Adopted time.Time `json:"adopted,omitzero"`
AdoptedBy string `json:"adopted_by,omitempty"`
}
// The typed operations.
type (
// SearchIn runs a search over this tenant's own history.
SearchIn struct {
Space topology.Space `json:"space"`
Options topology.Options `json:"options,omitempty"`
By string `json:"by"`
}
// RefIn names one run or one fit.
RefIn struct {
ID string `json:"id"`
}
// RunsIn lists this tenant's runs, most recent first.
RunsIn struct {
Limit int `json:"limit,omitempty"`
}
// Runs is the answer to RunsIn.
Runs struct {
Runs []Summary `json:"runs"`
Cut bool `json:"cut,omitempty"`
}
// FitIn builds learned state under one shape.
FitIn struct {
Topology topology.Topology `json:"topology"`
Options topology.Options `json:"options,omitempty"`
By string `json:"by"`
}
// FitsIn lists this tenant's fits, most recent first.
FitsIn struct {
Limit int `json:"limit,omitempty"`
}
// Fits is the answer to FitsIn.
Fits struct {
Fits []Fit `json:"fits"`
Cut bool `json:"cut,omitempty"`
}
// AdoptIn installs a fit into the live model.
AdoptIn struct {
ID string `json:"id"`
Reason string `json:"reason"`
By string `json:"by"`
}
)
// DefaultLimit is how many rows a list returns when the caller names no bound.
const DefaultLimit = 50
pkg/models/models_test.go
+377
View File
@@ -0,0 +1,377 @@
package models
import (
"context"
"encoding/json"
"errors"
"os"
"strings"
"testing"
"time"
"github.com/luxfi/aml/internal/instance"
"github.com/luxfi/aml/pkg/anomaly"
"github.com/luxfi/aml/pkg/replay"
"github.com/luxfi/aml/pkg/topology"
"github.com/luxfi/aml/pkg/types"
"github.com/luxfi/aml/pkg/velocity"
)
const (
acme = "hanzo/acme"
rival = "hanzo/rival"
// other is the SAME org name under a different brand.
other = "zoo/acme"
)
var (
noon = time.Date(2026, 3, 1, 12, 0, 0, 0, time.UTC)
shape = topology.Topology{Trees: 8, Depth: 6, Window: 16, Blend: 0.25, Review: 0.02}
)
// history is a tenant's own traffic, with the outliers a human judged productive.
type history struct{ n int }
func (h history) History(ctx context.Context, org string) (replay.History, error) {
out := make(replay.Slice, 0, h.n)
for i := 0; i < h.n; i++ {
usd := 100 + float64(i%17)*3
odd := i%53 == 52
if odd {
usd = 90_000
}
e := replay.Event{
Tx: types.Transaction{
ID: "tx-" + time.Duration(i).String(), OrgID: org, UserID: "u1", AccountID: "acct-1",
Currency: "USD", Notional: usd, USD: usd, Direction: "in",
Timestamp: noon.Add(time.Duration(i) * time.Minute),
},
Entity: types.Entity{ID: "u1", OrgID: org},
}
switch {
case odd:
e.Disposition = replay.Productive
case i%23 == 0:
e.Disposition = replay.Unproductive
}
out = append(out, e)
}
return out, nil
}
func live(t *testing.T, s topology.Topology, seed uint64) *anomaly.Store {
t.Helper()
model, err := anomaly.New(s.Config(seed), velocity.New(velocity.Config{}))
if err != nil {
t.Fatal(err)
}
return model
}
func shelf(t *testing.T) *Shelf {
t.Helper()
app := instance.New(t)
t.Cleanup(app.Cleanup)
if err := Ensure(app); err != nil {
t.Fatalf("ensure: %v", err)
}
s := NewBase(app)
s.Now = func() time.Time { return noon }
s.History = history{n: 600}
s.Model = live(t, shape, 7)
return s
}
// TestASearchIsKept. An answer that reached a screen and not the store is one
// nobody can produce when asked why the model looks the way it does.
func TestASearchIsKept(t *testing.T) {
s := shelf(t)
ctx := context.Background()
run, err := s.Search(ctx, acme, &SearchIn{
Space: topology.Space{Trees: []int{8}, Depth: []int{6}, Window: []int{16}, Blend: []float64{0.25}, Review: []float64{0.02}},
Options: topology.Options{Seed: 7, Curve: 8},
By: "a.mensah",
})
if err != nil {
t.Fatal(err)
}
if len(run.Report.Trials) != 1 || run.Report.Seed != 7 {
t.Fatalf("the run must carry the whole report: %+v", run.Report)
}
back, err := s.Run(ctx, acme, &RefIn{ID: run.ID})
if err != nil {
t.Fatal(err)
}
if len(back.Report.Trials) != 1 || back.By != "a.mensah" || back.Report.Seed != 7 {
t.Fatalf("the run did not come back whole: %+v", back)
}
if len(back.Report.Trials[0].Curve) < 8 {
t.Fatalf("the learning curve did not survive the store: %d points", len(back.Report.Trials[0].Curve))
}
list, err := s.Runs(ctx, acme, &RunsIn{})
if err != nil {
t.Fatal(err)
}
if len(list.Runs) != 1 || list.Runs[0].ID != run.ID || list.Runs[0].Trials != 1 {
t.Fatalf("the list must summarise the run: %+v", list.Runs)
}
// A list is a summary and never the heavy read.
raw, err := json.Marshal(list)
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(raw), "\"curve\"") {
t.Fatal("a list of runs must not carry a curve per candidate")
}
}
// TestASearchNeedsAHistoryAndADecider.
func TestASearchNeedsAHistoryAndADecider(t *testing.T) {
s := shelf(t)
ctx := context.Background()
if _, err := s.Search(ctx, acme, &SearchIn{Space: topology.Space{}}); !errors.Is(err, ErrDecider) {
t.Errorf("a search must name who asked for it, got %v", err)
}
s.History = nil
if _, err := s.Search(ctx, acme, &SearchIn{Space: topology.Space{}, By: "a"}); !errors.Is(err, ErrNoHistory) {
t.Errorf("a search over no history would report that every shape alerts on nothing, got %v", err)
}
}
// TestFitIsAdoptableOnlyIntoItsOwnShape. That is the governance property: a
// tenant's model shape cannot be swapped underneath it by restoring a file.
func TestFitIsAdoptableOnlyIntoItsOwnShape(t *testing.T) {
s := shelf(t)
ctx := context.Background()
mine, err := s.Fit(ctx, acme, &FitIn{Topology: shape, Options: topology.Options{Seed: 7}, By: "a.mensah"})
if err != nil {
t.Fatal(err)
}
if !mine.Adoptable {
t.Fatalf("a fit of the running shape must be adoptable: %+v", mine)
}
bigger := shape
bigger.Trees = 16
elsewhere, err := s.Fit(ctx, acme, &FitIn{Topology: bigger, Options: topology.Options{Seed: 7}, By: "a.mensah"})
if err != nil {
t.Fatal(err)
}
if elsewhere.Adoptable {
t.Fatalf("a fit of a shape the deployment is not running must not be adoptable: %+v", elsewhere)
}
if _, err := s.Adopt(ctx, acme, &AdoptIn{ID: elsewhere.ID, Reason: "the search recommended it", By: "a.mensah"}); err == nil {
t.Fatal("adopting a fit of another shape must be refused by the model itself")
}
adopted, err := s.Adopt(ctx, acme, &AdoptIn{ID: mine.ID, Reason: "the search recommended it", By: "r.okafor"})
if err != nil {
t.Fatalf("adopt: %v", err)
}
if adopted.Adopted.IsZero() || adopted.AdoptedBy != "r.okafor" {
t.Fatalf("an adoption must name who took it and when: %+v", adopted)
}
if st := s.Model.State(acme); st.Learned == 0 {
t.Fatal("the live model did not take the fitted state")
}
if _, err := s.Adopt(ctx, acme, &AdoptIn{ID: mine.ID, Reason: "again", By: "r.okafor"}); !errors.Is(err, ErrAdopted) {
t.Fatalf("adopting twice must be refused, got %v", err)
}
}
// TestLearnedStateNeverLeavesTheStore.
//
// Mass counters describe where a tenant's activity is dense; handing them out
// over an API would publish the shape of an institution's customer behaviour to
// whoever holds a token.
func TestLearnedStateNeverLeavesTheStore(t *testing.T) {
s := shelf(t)
ctx := context.Background()
fit, err := s.Fit(ctx, acme, &FitIn{Topology: shape, Options: topology.Options{Seed: 7}, By: "a.mensah"})
if err != nil {
t.Fatal(err)
}
raw, err := json.Marshal(fit)
if err != nil {
t.Fatal(err)
}
var out map[string]any
if err := json.Unmarshal(raw, &out); err != nil {
t.Fatal(err)
}
for _, leaked := range []string{"ref", "cur", "hist", "state", "snapshot", "seed"} {
if _, there := out[leaked]; there {
t.Errorf("a fit handed back %q — the learned state must stay in the tenant's store", leaked)
}
}
if fit.Digest == "" {
t.Fatal("the digest is what a caller compares, and must be on the answer")
}
}
// TestASnapshotCannotBeInstalledIntoAnotherTenant. The tenant comes from the row's
// own scope; a snapshot claiming otherwise is refused.
func TestASnapshotCannotBeInstalledIntoAnotherTenant(t *testing.T) {
s := shelf(t)
ctx := context.Background()
mine, err := s.Fit(ctx, acme, &FitIn{Topology: shape, Options: topology.Options{Seed: 7}, By: "a.mensah"})
if err != nil {
t.Fatal(err)
}
if _, err := s.Adopt(ctx, other, &AdoptIn{ID: mine.ID, Reason: "r", By: "b"}); !errors.Is(err, ErrNoFit) {
t.Fatalf("another tenant naming this fit's id must see no such fit, got %v", err)
}
if st := s.Model.State(other); st.Learned != 0 {
t.Fatal("another tenant's model took this tenant's state")
}
}
// TestTenantIsolation over every operation.
func TestTenantIsolation(t *testing.T) {
s := shelf(t)
ctx := context.Background()
run, err := s.Search(ctx, acme, &SearchIn{
Space: topology.Space{Trees: []int{8}, Depth: []int{6}, Window: []int{16}},
Options: topology.Options{Seed: 7, Curve: 4}, By: "a.mensah",
})
if err != nil {
t.Fatal(err)
}
fit, err := s.Fit(ctx, acme, &FitIn{Topology: shape, Options: topology.Options{Seed: 7}, By: "a.mensah"})
if err != nil {
t.Fatal(err)
}
for _, stranger := range []string{other, rival} {
if _, err := s.Run(ctx, stranger, &RefIn{ID: run.ID}); !errors.Is(err, ErrNotHere) {
t.Errorf("%s can read %s's run: %v", stranger, acme, err)
}
list, err := s.Runs(ctx, stranger, &RunsIn{})
if err != nil {
t.Fatal(err)
}
if len(list.Runs) != 0 {
t.Errorf("%s can list %s's runs", stranger, acme)
}
fits, err := s.Fits(ctx, stranger, &FitsIn{})
if err != nil {
t.Fatal(err)
}
if len(fits.Fits) != 0 {
t.Errorf("%s can list %s's fits", stranger, acme)
}
if _, err := s.Adopt(ctx, stranger, &AdoptIn{ID: fit.ID, Reason: "r", By: "b"}); !errors.Is(err, ErrNoFit) {
t.Errorf("%s can adopt %s's fit: %v", stranger, acme, err)
}
}
}
// TestBareOrgIsRefused.
func TestBareOrgIsRefused(t *testing.T) {
s := shelf(t)
ctx := context.Background()
for _, bare := range []string{"acme", "", "unknown/acme"} {
if _, err := s.Search(ctx, bare, &SearchIn{By: "a"}); err == nil {
t.Errorf("Search accepted %q as a tenant", bare)
}
if _, err := s.Fit(ctx, bare, &FitIn{Topology: shape, By: "a"}); err == nil {
t.Errorf("Fit accepted %q as a tenant", bare)
}
if _, err := s.Runs(ctx, bare, &RunsIn{}); err == nil {
t.Errorf("Runs accepted %q as a tenant", bare)
}
}
}
// TestNothingDeletes.
func TestNothingDeletes(t *testing.T) {
for _, name := range []string{"models.go", "shelf.go"} {
raw, err := os.ReadFile(name)
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(raw), ".Delete(") {
t.Errorf("%s calls Delete: disposal is pkg/retention's decision and nobody else's", name)
}
}
}
// TestRestart: the study that justified a model, and the state it produced, are
// both still there after the pod that made them is gone.
func TestRestart(t *testing.T) {
first := instance.New(t)
if err := Ensure(first); err != nil {
t.Fatal(err)
}
before := NewBase(first)
before.Now = func() time.Time { return noon }
before.History = history{n: 600}
before.Model = live(t, shape, 7)
ctx := context.Background()
run, err := before.Search(ctx, acme, &SearchIn{
Space: topology.Space{Trees: []int{8}, Depth: []int{6}, Window: []int{16}, Review: []float64{0.02}},
Options: topology.Options{Seed: 7, Curve: 8}, By: "a.mensah",
})
if err != nil {
t.Fatal(err)
}
fit, err := before.Fit(ctx, acme, &FitIn{Topology: shape, Options: topology.Options{Seed: 7}, By: "a.mensah"})
if err != nil {
t.Fatal(err)
}
if _, err := before.Fit(ctx, other, &FitIn{Topology: shape, Options: topology.Options{Seed: 7}, By: "z.other"}); err != nil {
t.Fatal(err)
}
second := instance.Restart(t, first)
if err := Ensure(second); err != nil {
t.Fatal(err)
}
after := NewBase(second)
after.Now = func() time.Time { return noon }
after.History = history{n: 600}
// A FRESH model: warming, nothing learned. This is what a rollout produces.
after.Model = live(t, shape, 7)
back, err := after.Run(ctx, acme, &RefIn{ID: run.ID})
if err != nil {
t.Fatalf("the study did not survive the restart: %v", err)
}
if back.By != "a.mensah" || len(back.Report.Trials) == 0 {
t.Fatalf("the study came back changed: %+v", back)
}
if st := after.Model.State(acme); st.Learned != 0 {
t.Fatal("the fresh model should be warming — the test is not testing a restart")
}
adopted, err := after.Adopt(ctx, acme, &AdoptIn{ID: fit.ID, Reason: "restoring after a rollout", By: "r.okafor"})
if err != nil {
t.Fatalf("the fitted state did not survive the restart: %v", err)
}
if adopted.Digest != fit.Digest {
t.Fatalf("the fit came back under a different shape: %s vs %s", adopted.Digest, fit.Digest)
}
if st := after.Model.State(acme); st.Learned == 0 {
t.Fatal("a restart plus an adoption must leave the model warm, or every rollout is a blind window")
}
// Tenancy still holds.
fits, err := after.Fits(ctx, rival, &FitsIn{})
if err != nil {
t.Fatal(err)
}
if len(fits.Fits) != 0 {
t.Fatal("a third tenant can list fits after the restart")
}
mine, err := after.Fits(ctx, acme, &FitsIn{})
if err != nil {
t.Fatal(err)
}
if len(mine.Fits) != 1 {
t.Fatalf("the tenant boundary did not survive the restart: %d fits", len(mine.Fits))
}
}
pkg/models/shelf.go
+395
View File
@@ -0,0 +1,395 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package models
import (
"context"
"encoding/json"
"fmt"
"strings"
"time"
"github.com/google/uuid"
"github.com/hanzoai/base/core"
"github.com/hanzoai/dbx"
"github.com/luxfi/aml/pkg/anomaly"
"github.com/luxfi/aml/pkg/brand"
"github.com/luxfi/aml/pkg/replay"
"github.com/luxfi/aml/pkg/store"
"github.com/luxfi/aml/pkg/topology"
)
const (
fieldAt = "at"
fieldBy = "by"
fieldSpace = "space"
fieldOptions = "options"
fieldReport = "report"
fieldTrials = "trials"
fieldEvents = "events"
fieldJudged = "judged"
fieldWinner = "winner"
fieldRefusal = "refusal"
fieldElapsed = "elapsed"
fieldTopology = "topology"
fieldDigest = "digest"
fieldTrial = "trial"
fieldState = "state"
fieldAdopted = "adopted"
fieldAdoptBy = "adopted_by"
fieldAdoptWhy = "adopt_why"
)
var runKind = store.Kind{
Name: "aml_model_runs",
Fields: []core.Field{
&core.DateField{Name: fieldAt, Required: true},
&core.TextField{Name: fieldBy, Required: true},
&core.JSONField{Name: fieldSpace, Required: true},
&core.JSONField{Name: fieldOptions},
&core.JSONField{Name: fieldReport, Required: true},
// The summary columns. Denormalised deliberately: a list of runs must not
// read a report per row, and a report is the one thing here that is large.
&core.NumberField{Name: fieldTrials},
&core.NumberField{Name: fieldEvents},
&core.NumberField{Name: fieldJudged},
&core.NumberField{Name: fieldElapsed},
&core.JSONField{Name: fieldWinner},
&core.TextField{Name: fieldRefusal},
},
Indexes: []store.Index{
{Name: "at", Fields: []string{store.Org, fieldAt}},
},
}
var fitKind = store.Kind{
Name: "aml_model_fits",
Fields: []core.Field{
&core.DateField{Name: fieldAt, Required: true},
&core.TextField{Name: fieldBy, Required: true},
&core.JSONField{Name: fieldTopology, Required: true},
&core.TextField{Name: fieldDigest, Required: true},
&core.JSONField{Name: fieldTrial, Required: true},
// The learned state. It is written and read here and never leaves on an
// answer — see the package doc.
&core.JSONField{Name: fieldState, Required: true},
&core.DateField{Name: fieldAdopted},
&core.TextField{Name: fieldAdoptBy},
&core.TextField{Name: fieldAdoptWhy},
},
Indexes: []store.Index{
{Name: "at", Fields: []string{store.Org, fieldAt}},
{Name: "digest", Fields: []string{store.Org, fieldDigest}},
},
}
// Ensure creates what runs and fits are kept in. Idempotent, so it runs on every
// start.
func Ensure(app core.App) error {
if err := runKind.Ensure(app); err != nil {
return err
}
return fitKind.Ensure(app)
}
// Source hands out a tenant's replayable history.
//
// One method, and it takes the tenant, so this package cannot reach history any
// other way — the same seam pkg/replay uses, for the same reason.
type Source interface {
History(ctx context.Context, org string) (replay.History, error)
}
// Shelf is the durable model plane.
type Shelf struct {
app core.App
// History is where a search and a fit read a tenant's events from. Without it
// both refuse: a search over no history would report that every shape alerts
// on nothing, which is what a good shape and a broken store look like alike.
History Source
// Model is the live detector. Only Adopt touches it, and only to restore.
Model *anomaly.Store
// Now supplies the instant. Tests set it.
Now func() time.Time
}
// NewBase returns the durable shelf. Ensure has to have run first.
func NewBase(app core.App) *Shelf { return &Shelf{app: app} }
func (s *Shelf) now() time.Time {
if s.Now != nil {
return s.Now().UTC()
}
return time.Now().UTC()
}
// Search runs a search over this tenant's history and keeps it.
//
// The run is written before it is returned. A study whose conclusion reached a
// screen and not the store is one nobody can produce when asked why the model
// looks the way it does.
func (s *Shelf) Search(ctx context.Context, org string, in *SearchIn) (*Run, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
if strings.TrimSpace(in.By) == "" {
return nil, ErrDecider
}
if s.History == nil {
return nil, ErrNoHistory
}
h, err := s.History.History(ctx, org)
if err != nil {
return nil, err
}
report, err := topology.Search(ctx, org, h, in.Space, in.Options)
if err != nil {
return nil, err
}
run := &Run{
ID: uuid.NewString(), Org: org, At: s.now(), By: strings.TrimSpace(in.By),
Space: in.Space, Options: in.Options, Report: report,
}
row, err := runKind.New(s.app, org)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
row.Id = run.ID
row.Set(fieldAt, run.At)
row.Set(fieldBy, run.By)
if err := setJSON(row, fieldSpace, run.Space); err != nil {
return nil, err
}
if err := setJSON(row, fieldOptions, run.Options); err != nil {
return nil, err
}
if err := setJSON(row, fieldReport, run.Report); err != nil {
return nil, err
}
row.Set(fieldTrials, len(report.Trials))
row.Set(fieldEvents, report.Events)
row.Set(fieldJudged, report.Judged)
row.Set(fieldElapsed, int64(report.Elapsed))
row.Set(fieldRefusal, report.Refusal)
if report.Winner != nil {
if err := setJSON(row, fieldWinner, report.Winner.Topology); err != nil {
return nil, err
}
} else {
row.Set(fieldWinner, "")
}
if err := s.app.Save(row); err != nil {
return nil, fmt.Errorf("%w: keeping the run: %w", ErrStore, err)
}
return run, nil
}
// Run reads one whole search back.
func (s *Shelf) Run(ctx context.Context, org string, in *RefIn) (*Run, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
rows, err := runKind.Find(s.app, org, "id = {:id}", "", 1, dbx.Params{"id": strings.TrimSpace(in.ID)})
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
if len(rows) == 0 {
return nil, fmt.Errorf("%w: %s", ErrNotHere, in.ID)
}
run := &Run{ID: rows[0].Id, Org: org, At: rows[0].GetDateTime(fieldAt).Time(), By: rows[0].GetString(fieldBy)}
if err := rows[0].UnmarshalJSONField(fieldSpace, &run.Space); err != nil {
return nil, fmt.Errorf("%w: %s space: %w", ErrStore, rows[0].Id, err)
}
if err := rows[0].UnmarshalJSONField(fieldOptions, &run.Options); err != nil {
return nil, fmt.Errorf("%w: %s options: %w", ErrStore, rows[0].Id, err)
}
if err := rows[0].UnmarshalJSONField(fieldReport, &run.Report); err != nil {
return nil, fmt.Errorf("%w: %s report: %w", ErrStore, rows[0].Id, err)
}
return run, nil
}
// Runs lists this tenant's searches, most recent first, as summaries.
func (s *Shelf) Runs(ctx context.Context, org string, in *RunsIn) (*Runs, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
limit := in.Limit
if limit <= 0 {
limit = DefaultLimit
}
rows, err := runKind.Find(s.app, org, "", "-"+fieldAt, limit+1, nil)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
out := &Runs{Runs: make([]Summary, 0, len(rows))}
if len(rows) > limit {
rows, out.Cut = rows[:limit], true
}
for _, row := range rows {
sum := Summary{
ID: row.Id, At: row.GetDateTime(fieldAt).Time(), By: row.GetString(fieldBy),
Trials: row.GetInt(fieldTrials), Events: row.GetInt(fieldEvents), Judged: row.GetInt(fieldJudged),
Elapsed: time.Duration(row.GetInt(fieldElapsed)), Refusal: row.GetString(fieldRefusal),
}
if raw := row.GetString(fieldWinner); raw != "" && raw != "null" {
var t topology.Topology
if err := json.Unmarshal([]byte(raw), &t); err == nil {
sum.Winner = &t
}
}
out.Runs = append(out.Runs, sum)
}
return out, nil
}
// Fit builds learned state under one shape and keeps it.
func (s *Shelf) Fit(ctx context.Context, org string, in *FitIn) (*Fit, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
if strings.TrimSpace(in.By) == "" {
return nil, ErrDecider
}
if s.History == nil {
return nil, ErrNoHistory
}
h, err := s.History.History(ctx, org)
if err != nil {
return nil, err
}
snap, trial, err := topology.Fit(ctx, org, h, in.Topology, in.Options)
if err != nil {
return nil, err
}
fit := &Fit{
ID: uuid.NewString(), Org: org, At: s.now(), By: strings.TrimSpace(in.By),
Topology: in.Topology, Digest: trial.Digest, Trial: trial,
Adoptable: s.Model != nil && s.Model.Digest() == trial.Digest,
}
row, err := fitKind.New(s.app, org)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
row.Id = fit.ID
row.Set(fieldAt, fit.At)
row.Set(fieldBy, fit.By)
row.Set(fieldDigest, fit.Digest)
for field, v := range map[string]any{fieldTopology: fit.Topology, fieldTrial: fit.Trial, fieldState: snap} {
if err := setJSON(row, field, v); err != nil {
return nil, err
}
}
if err := s.app.Save(row); err != nil {
return nil, fmt.Errorf("%w: keeping the fit: %w", ErrStore, err)
}
return fit, nil
}
// Fits lists this tenant's fitted states, most recent first.
func (s *Shelf) Fits(ctx context.Context, org string, in *FitsIn) (*Fits, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
limit := in.Limit
if limit <= 0 {
limit = DefaultLimit
}
rows, err := fitKind.Find(s.app, org, "", "-"+fieldAt, limit+1, nil)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
out := &Fits{Fits: make([]Fit, 0, len(rows))}
if len(rows) > limit {
rows, out.Cut = rows[:limit], true
}
for _, row := range rows {
f, err := s.readFit(org, row)
if err != nil {
return nil, err
}
out.Fits = append(out.Fits, *f)
}
return out, nil
}
// Adopt installs a fit into the live model.
//
// Three refusals, and each one is the model's own rather than this package's
// opinion: no live model, a shape the running model is not, or state that fails
// the mass invariant. anomaly.Restore performs the last two, so a fit adopted
// here has passed exactly the checks a snapshot restored anywhere else does.
func (s *Shelf) Adopt(ctx context.Context, org string, in *AdoptIn) (*Fit, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
if strings.TrimSpace(in.By) == "" {
return nil, ErrDecider
}
if s.Model == nil {
return nil, ErrNoModel
}
rows, err := fitKind.Find(s.app, org, "id = {:id}", "", 1, dbx.Params{"id": strings.TrimSpace(in.ID)})
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
if len(rows) == 0 {
return nil, fmt.Errorf("%w: %s", ErrNoFit, in.ID)
}
row := rows[0]
if !row.GetDateTime(fieldAdopted).Time().IsZero() {
return nil, fmt.Errorf("%w: %s", ErrAdopted, row.Id)
}
var snap anomaly.Snapshot
if err := row.UnmarshalJSONField(fieldState, &snap); err != nil {
return nil, fmt.Errorf("%w: %s state: %w", ErrStore, row.Id, err)
}
// The tenant is taken from the row's own scope and not from the snapshot, so
// a snapshot whose OrgID says otherwise cannot install another tenant's model.
if snap.OrgID != org {
return nil, fmt.Errorf("%w: the fit was built for %q and is being adopted into %q", ErrShape, snap.OrgID, org)
}
if err := s.Model.Restore(snap); err != nil {
return nil, err
}
row.Set(fieldAdopted, s.now())
row.Set(fieldAdoptBy, strings.TrimSpace(in.By))
row.Set(fieldAdoptWhy, strings.TrimSpace(in.Reason))
if err := s.app.Save(row); err != nil {
return nil, fmt.Errorf("%w: recording the adoption: %w", ErrStore, err)
}
return s.readFit(org, row)
}
func (s *Shelf) readFit(org string, row *core.Record) (*Fit, error) {
f := &Fit{
ID: row.Id, Org: org, At: row.GetDateTime(fieldAt).Time(), By: row.GetString(fieldBy),
Digest: row.GetString(fieldDigest),
Adopted: row.GetDateTime(fieldAdopted).Time(), AdoptedBy: row.GetString(fieldAdoptBy),
}
if err := row.UnmarshalJSONField(fieldTopology, &f.Topology); err != nil {
return nil, fmt.Errorf("%w: %s topology: %w", ErrStore, row.Id, err)
}
if err := row.UnmarshalJSONField(fieldTrial, &f.Trial); err != nil {
return nil, fmt.Errorf("%w: %s trial: %w", ErrStore, row.Id, err)
}
f.Adoptable = s.Model != nil && s.Model.Digest() == f.Digest
return f, nil
}
func setJSON(row *core.Record, field string, v any) error {
raw, err := json.Marshal(v)
if err != nil {
return fmt.Errorf("%w: %s: %w", ErrStore, field, err)
}
row.Set(field, string(raw))
return nil
}
pkg/suppress/shelf.go
+310
View File
@@ -0,0 +1,310 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package suppress
import (
"context"
"fmt"
"slices"
"sort"
"strings"
"time"
"github.com/google/uuid"
"github.com/hanzoai/base/core"
"github.com/hanzoai/dbx"
"github.com/luxfi/aml/pkg/brand"
"github.com/luxfi/aml/pkg/history"
"github.com/luxfi/aml/pkg/store"
)
const (
fieldRule = "rule"
fieldKind = "kind"
fieldValue = "value"
fieldReason = "reason"
fieldBy = "by"
fieldFrom = "from_at"
fieldUntil = "until"
fieldLifted = "lifted"
fieldLiftBy = "lift_by"
fieldLiftWhy = "lift_why"
)
var kind = store.Kind{
Name: "aml_suppressions",
Fields: []core.Field{
&core.TextField{Name: fieldRule},
&core.TextField{Name: fieldKind},
&core.TextField{Name: fieldValue},
&core.TextField{Name: fieldReason, Required: true},
&core.TextField{Name: fieldBy, Required: true},
&core.DateField{Name: fieldFrom, Required: true},
&core.DateField{Name: fieldUntil},
&core.DateField{Name: fieldLifted},
&core.TextField{Name: fieldLiftBy},
&core.TextField{Name: fieldLiftWhy},
},
Indexes: []store.Index{
// The hot read: this tenant's suppressions that could cover an activation
// of this rule. The blanket ones carry an empty rule and are read by the
// same index.
{Name: "rule", Fields: []string{store.Org, fieldRule}},
// The subject read, for the ledger view of one account or address.
{Name: "subject", Fields: []string{store.Org, fieldKind, fieldValue}},
},
}
// Ensure creates what suppressions are kept in. Idempotent, so it runs on every
// start.
func Ensure(app core.App) error { return kind.Ensure(app) }
// MaxCandidates bounds the rows one Cover reads.
//
// The read is already narrowed to this tenant and to the rule named plus the
// blanket ones, so reaching this bound means an institution has declared more
// suppressions against one rule than anybody could review. It is reported rather
// than silently truncated, because a truncated cover check answers "not
// suppressed" for a detection that was.
const MaxCandidates = 2000
// Shelf is the durable suppression plane. There is no memory implementation: a
// suppression that vanishes on a rollout turns a governed silence into a
// surprise, and the reverse — a suppression that outlives its record — is worse.
type Shelf struct {
app core.App
// Now supplies the instant a window is judged against. Tests set it.
Now func() time.Time
}
// NewBase returns the durable shelf. Ensure has to have run first.
func NewBase(app core.App) *Shelf { return &Shelf{app: app} }
func (s *Shelf) now() time.Time {
if s.Now != nil {
return s.Now().UTC()
}
return time.Now().UTC()
}
// Suppress records a decision to keep a detection out of a queue.
func (s *Shelf) Suppress(ctx context.Context, org string, in *SuppressIn) (*Suppression, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
rule, kindName, value := strings.TrimSpace(in.Rule), strings.TrimSpace(in.Kind), strings.TrimSpace(in.Value)
switch {
case strings.TrimSpace(in.Reason) == "":
return nil, ErrReason
case strings.TrimSpace(in.By) == "":
return nil, ErrDecider
case rule == "" && value == "":
return nil, ErrBroad
case kindName != "" && !slices.Contains(history.Kinds, kindName):
return nil, fmt.Errorf("%w: %q, want one of %s", ErrKind, kindName, strings.Join(history.Kinds, ", "))
case value != "" && kindName == "":
return nil, fmt.Errorf("%w: %q", ErrSubject, value)
}
at := s.now()
sup := Suppression{
ID: uuid.NewString(), Org: org,
Rule: rule, Kind: kindName, Value: value,
Reason: strings.TrimSpace(in.Reason), By: strings.TrimSpace(in.By),
From: in.From.UTC(), Until: in.Until.UTC(),
}
if sup.From.IsZero() {
sup.From = at
}
if !sup.Until.IsZero() && !sup.Until.After(sup.From) {
return nil, fmt.Errorf("%w: until %s is not after from %s", ErrWindow, sup.Until, sup.From)
}
if !sup.Until.IsZero() && !sup.Until.After(at) {
return nil, fmt.Errorf("%w: until %s has passed", ErrWindow, sup.Until)
}
row, err := kind.New(s.app, org)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
row.Id = sup.ID
write(row, sup)
if err := s.app.Save(row); err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
return &sup, nil
}
// Lift ends a suppression. The row stays and records who ended it and why.
func (s *Shelf) Lift(ctx context.Context, org string, in *LiftIn) (*Suppression, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
if strings.TrimSpace(in.Reason) == "" {
return nil, ErrReason
}
if strings.TrimSpace(in.By) == "" {
return nil, ErrDecider
}
// Scoped by the tenant, so naming another institution's suppression id reads
// as "no such suppression" rather than as a refusal that confirms it exists.
rows, err := kind.Find(s.app, org, "id = {:id}", "", 1, dbx.Params{"id": strings.TrimSpace(in.ID)})
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
if len(rows) == 0 {
return nil, fmt.Errorf("%w: %s", ErrNotHere, in.ID)
}
sup := read(rows[0])
if !sup.Lifted.IsZero() {
return nil, fmt.Errorf("%w: %s", ErrLifted, sup.ID)
}
sup.Lifted = s.now()
sup.LiftedBy = strings.TrimSpace(in.By)
sup.LiftWhy = strings.TrimSpace(in.Reason)
write(rows[0], sup)
if err := s.app.Save(rows[0]); err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
return &sup, nil
}
// Ledger reads a tenant's suppressions.
func (s *Shelf) Ledger(ctx context.Context, org string, in *LedgerIn) (*Ledger, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
limit := in.Limit
if limit <= 0 {
limit = DefaultLimit
}
filter, params := "", dbx.Params{}
if r := strings.TrimSpace(in.Rule); r != "" {
filter, params[fieldRule] = fieldRule+" = {:"+fieldRule+"}", r
}
rows, err := kind.Find(s.app, org, filter, "-"+fieldFrom, limit+1, params)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
out := &Ledger{Suppressions: make([]Suppression, 0, len(rows))}
if len(rows) > limit {
rows, out.Cut = rows[:limit], true
}
at := s.now()
for _, row := range rows {
sup := read(row)
if in.InForce && !sup.InForce(at) {
continue
}
out.Suppressions = append(out.Suppressions, sup)
}
return out, nil
}
// Cover answers whether a suppression in force covers this activation, and which
// one.
//
// The narrowest wins. A tenant that has suppressed a whole rule and then made a
// specific decision about one account should see the specific one named on the
// activation, because that is the decision a reviewer will be asked about.
func (s *Shelf) Cover(ctx context.Context, org string, in *CoverIn) (*Cover, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
at := in.At.UTC()
if at.IsZero() {
at = s.now()
}
rule := strings.TrimSpace(in.Rule)
// The candidates are this tenant's suppressions naming this rule, plus the
// blanket ones. Both halves are bound parameters; neither is interpolated.
rows, err := kind.Find(s.app, org, fieldRule+" = {:rule} || "+fieldRule+" = ''", "", MaxCandidates+1,
dbx.Params{"rule": rule})
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
if len(rows) > MaxCandidates {
return nil, fmt.Errorf("%w: more than %d suppressions bear on rule %q, so a cover check cannot be answered completely",
ErrStore, MaxCandidates, rule)
}
var best *Suppression
for _, row := range rows {
sup := read(row)
if !sup.InForce(at) || !sup.Covers(rule, in.Kind, in.Value) {
continue
}
if best == nil || better(sup, *best) {
s := sup
best = &s
}
}
if best == nil {
return &Cover{}, nil
}
return &Cover{Covered: true, Suppression: best}, nil
}
// better breaks the tie between two covering suppressions: the narrower one, then
// the more recent decision, then the id. Deterministic, because an activation
// marked with a different suppression on a replay is an activation nobody can
// reconcile.
func better(a, b Suppression) bool {
if a.Reach() != b.Reach() {
return a.Reach() > b.Reach()
}
if !a.From.Equal(b.From) {
return a.From.After(b.From)
}
return a.ID < b.ID
}
// Sort orders suppressions the way a ledger reads: most recent decision first.
func Sort(in []Suppression) {
sort.SliceStable(in, func(i, j int) bool { return in[i].From.After(in[j].From) })
}
func write(row *core.Record, s Suppression) {
row.Set(fieldRule, s.Rule)
row.Set(fieldKind, s.Kind)
row.Set(fieldValue, s.Value)
row.Set(fieldReason, s.Reason)
row.Set(fieldBy, s.By)
row.Set(fieldFrom, s.From.UTC())
row.Set(fieldUntil, moment(s.Until))
row.Set(fieldLifted, moment(s.Lifted))
row.Set(fieldLiftBy, s.LiftedBy)
row.Set(fieldLiftWhy, s.LiftWhy)
}
func read(row *core.Record) Suppression {
return Suppression{
ID: row.Id,
Org: row.GetString(store.Org),
Rule: row.GetString(fieldRule),
Kind: row.GetString(fieldKind),
Value: row.GetString(fieldValue),
Reason: row.GetString(fieldReason),
By: row.GetString(fieldBy),
From: row.GetDateTime(fieldFrom).Time(),
Until: row.GetDateTime(fieldUntil).Time(),
Lifted: row.GetDateTime(fieldLifted).Time(),
LiftedBy: row.GetString(fieldLiftBy),
LiftWhy: row.GetString(fieldLiftWhy),
}
}
// moment renders an optional instant the way a date column holds "unset": the
// empty string. A zero time written as a date is year one, which compares as a
// real moment in the past — so an open-ended suppression would read as one that
// closed two thousand years ago.
func moment(t time.Time) any {
if t.IsZero() {
return ""
}
return t.UTC()
}
pkg/suppress/suppress.go
+175
View File
@@ -0,0 +1,175 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// Package suppress holds the decisions an institution has taken to stop a
// detection reaching a queue, and answers whether one covers an activation.
//
// Suppression is the most dangerous control in a monitoring system, because its
// effect is silence and silence is what an unmonitored institution also looks
// like. Everything here is shaped by that.
//
// A suppression is a DECISION and carries one: a reason and a decider, both
// required, neither defaulted. Without them the row records that alerts stopped
// and nothing about why or on whose authority, which is the state a supervisor
// asking "who turned this off" cannot be answered from.
//
// A suppression that names neither a rule nor a subject is refused. That is not
// a suppression, it is a kill switch for the whole monitoring programme, and it
// would be indistinguishable in the store from an ordinary narrow one.
//
// Lifting does not delete. The row stays and gains the lift's own decider and
// reason, so the period a detection was silent is still readable after it is
// noisy again. Nothing in this package deletes anything: disposal is the
// retention plane's decision (AMLR Art. 77) and no operational knob gets to make
// it.
//
// And this package does not suppress anything. It answers whether a suppression
// covers an activation; the activation is still RECORDED, marked, by pkg/watch.
// A detection that is dropped rather than marked is one nobody can count, and a
// suppression whose volume nobody can count is not governed.
package suppress
import (
"errors"
"time"
)
// Errors.
var (
ErrReason = errors.New("suppress: no reason, so the row records that alerts stopped and not why")
ErrDecider = errors.New("suppress: no decider, so the row records nobody's decision")
ErrBroad = errors.New("suppress: a suppression naming neither a rule nor a subject is a kill switch, not a suppression")
ErrKind = errors.New("suppress: unknown subject kind")
ErrSubject = errors.New("suppress: a subject value needs the kind it is a value of")
ErrWindow = errors.New("suppress: the window has already closed, so the suppression would never apply")
ErrNotHere = errors.New("suppress: no such suppression")
ErrLifted = errors.New("suppress: already lifted")
ErrStore = errors.New("suppress: store")
)
// Suppression is one decision to keep a detection out of a queue.
//
// Rule, Kind and Value narrow what it covers, and an empty one means "any". At
// least one of Rule and Value must be named — see ErrBroad.
type Suppression struct {
ID string `json:"id"`
Org string `json:"org"`
Rule string `json:"rule,omitempty"`
Kind string `json:"kind,omitempty"`
Value string `json:"value,omitempty"`
Reason string `json:"reason"`
By string `json:"by"`
From time.Time `json:"from"`
Until time.Time `json:"until,omitzero"`
Lifted time.Time `json:"lifted,omitzero"`
LiftedBy string `json:"lifted_by,omitempty"`
LiftWhy string `json:"lift_why,omitempty"`
}
// InForce reports whether a suppression applies at an instant. It is a pure
// function of the row and the clock, so the same question asked of a stored row
// and of a proposed one gets the same answer.
func (s Suppression) InForce(at time.Time) bool {
if !s.Lifted.IsZero() && !at.Before(s.Lifted) {
return false
}
if at.Before(s.From) {
return false
}
return s.Until.IsZero() || s.Until.After(at)
}
// Covers reports whether this suppression covers an activation of a rule on a
// subject. It does not consider the clock; InForce does that, separately, so the
// two questions stay separable in a test.
func (s Suppression) Covers(rule, kind, value string) bool {
if s.Rule != "" && s.Rule != rule {
return false
}
if s.Kind != "" && s.Kind != kind {
return false
}
if s.Value != "" && s.Value != value {
return false
}
return true
}
// Reach is how narrow a suppression is: how many of the three coordinates it
// names. The narrowest match wins, so a blanket suppression on a rule never
// shadows a specific decision about one subject.
func (s Suppression) Reach() int {
n := 0
for _, v := range []string{s.Rule, s.Kind, s.Value} {
if v != "" {
n++
}
}
return n
}
// The typed operations.
type (
// SuppressIn declares a suppression.
SuppressIn struct {
Rule string `json:"rule,omitempty"`
Kind string `json:"kind,omitempty"`
Value string `json:"value,omitempty"`
Reason string `json:"reason"`
By string `json:"by"`
From time.Time `json:"from,omitzero"`
Until time.Time `json:"until,omitzero"`
}
// LiftIn ends a suppression. It carries its own reason and decider: ending a
// suppression is as much a decision as starting one, and an alert volume that
// jumps needs an entry saying why.
LiftIn struct {
ID string `json:"id"`
Reason string `json:"reason"`
By string `json:"by"`
}
// LedgerIn reads a tenant's suppressions. InForce restricts the answer to the
// ones applying right now; the default is all of them, because the lifted ones
// are the record of what was silent and when.
LedgerIn struct {
Rule string `json:"rule,omitempty"`
InForce bool `json:"in_force,omitempty"`
Limit int `json:"limit,omitempty"`
}
// Ledger is the answer to LedgerIn.
Ledger struct {
Suppressions []Suppression `json:"suppressions"`
Cut bool `json:"cut,omitempty"`
}
// CoverIn asks whether an activation is covered. At is the instant to judge,
// zero meaning now.
CoverIn struct {
Rule string `json:"rule"`
Kind string `json:"kind"`
Value string `json:"value"`
At time.Time `json:"at,omitzero"`
}
// Cover is the answer: covered or not, and by what.
//
// The suppression travels with the answer because pkg/watch writes its id onto
// the activation it marks. That is what makes "how many detections has this
// suppression silenced" a query rather than a guess, and it is why nothing
// here keeps a counter of its own.
Cover struct {
Covered bool `json:"covered"`
Suppression *Suppression `json:"suppression,omitempty"`
}
)
// DefaultLimit is how many suppressions a read returns when the caller names no
// bound.
const DefaultLimit = 500
pkg/suppress/suppress_test.go
+285
View File
@@ -0,0 +1,285 @@
package suppress
import (
"context"
"errors"
"os"
"strings"
"testing"
"time"
"github.com/luxfi/aml/internal/instance"
)
const (
acme = "hanzo/acme"
rival = "hanzo/rival"
// other is the SAME org name under a different brand — the case a bare-org
// regression collides into one tenant.
other = "zoo/acme"
)
var noon = time.Date(2026, 3, 1, 12, 0, 0, 0, time.UTC)
func shelf(t *testing.T) *Shelf {
t.Helper()
app := instance.New(t)
t.Cleanup(app.Cleanup)
if err := Ensure(app); err != nil {
t.Fatalf("ensure: %v", err)
}
s := NewBase(app)
s.Now = func() time.Time { return noon }
return s
}
func suppressed(t *testing.T, s *Shelf, org string, in *SuppressIn) *Suppression {
t.Helper()
if in.Reason == "" {
in.Reason = "known merchant settlement pattern, reviewed 2026-02"
}
if in.By == "" {
in.By = "a.mensah"
}
out, err := s.Suppress(context.Background(), org, in)
if err != nil {
t.Fatalf("suppress: %v", err)
}
return out
}
// TestAKillSwitchIsNotASuppression. A row naming neither a rule nor a subject
// silences the whole monitoring programme and is indistinguishable in the store
// from an ordinary narrow one.
func TestAKillSwitchIsNotASuppression(t *testing.T) {
s := shelf(t)
_, err := s.Suppress(context.Background(), acme, &SuppressIn{Reason: "too noisy", By: "a.mensah"})
if !errors.Is(err, ErrBroad) {
t.Fatalf("a suppression covering everything must be refused, got %v", err)
}
}
// TestEverySuppressionCarriesADecision.
func TestEverySuppressionCarriesADecision(t *testing.T) {
s := shelf(t)
ctx := context.Background()
if _, err := s.Suppress(ctx, acme, &SuppressIn{Rule: "r1", By: "a"}); !errors.Is(err, ErrReason) {
t.Errorf("no reason must be refused, got %v", err)
}
if _, err := s.Suppress(ctx, acme, &SuppressIn{Rule: "r1", Reason: "r"}); !errors.Is(err, ErrDecider) {
t.Errorf("no decider must be refused, got %v", err)
}
sup := suppressed(t, s, acme, &SuppressIn{Rule: "r1"})
if _, err := s.Lift(ctx, acme, &LiftIn{ID: sup.ID, By: "a"}); !errors.Is(err, ErrReason) {
t.Errorf("lifting without a reason must be refused, got %v", err)
}
if _, err := s.Lift(ctx, acme, &LiftIn{ID: sup.ID, Reason: "r"}); !errors.Is(err, ErrDecider) {
t.Errorf("lifting without a decider must be refused, got %v", err)
}
}
// TestASubjectNeedsItsAxis: a value with no kind covers "any subject whose value
// happens to be this string", which spans accounts and addresses at once.
func TestASubjectNeedsItsAxis(t *testing.T) {
s := shelf(t)
_, err := s.Suppress(context.Background(), acme, &SuppressIn{Value: "acct-1", Reason: "r", By: "a"})
if !errors.Is(err, ErrSubject) {
t.Fatalf("a subject value with no kind must be refused, got %v", err)
}
}
// TestNarrowestCoverWins. A tenant that suppressed a whole rule and then took a
// specific decision about one account must see the specific one named, because
// that is the decision a reviewer will be asked about.
func TestNarrowestCoverWins(t *testing.T) {
s := shelf(t)
ctx := context.Background()
broad := suppressed(t, s, acme, &SuppressIn{Rule: "r1"})
narrow := suppressed(t, s, acme, &SuppressIn{Rule: "r1", Kind: "account", Value: "acct-1"})
cover, err := s.Cover(ctx, acme, &CoverIn{Rule: "r1", Kind: "account", Value: "acct-1"})
if err != nil {
t.Fatal(err)
}
if !cover.Covered || cover.Suppression.ID != narrow.ID {
t.Fatalf("the specific decision must be the one named, got %+v", cover.Suppression)
}
cover, err = s.Cover(ctx, acme, &CoverIn{Rule: "r1", Kind: "account", Value: "acct-2"})
if err != nil {
t.Fatal(err)
}
if !cover.Covered || cover.Suppression.ID != broad.ID {
t.Fatalf("another account falls to the rule-wide decision, got %+v", cover.Suppression)
}
cover, err = s.Cover(ctx, acme, &CoverIn{Rule: "r2", Kind: "account", Value: "acct-1"})
if err != nil {
t.Fatal(err)
}
if cover.Covered {
t.Fatalf("a suppression on r1 must not cover r2: %+v", cover.Suppression)
}
}
// TestWindowsAndLifts. Both stop a suppression applying; neither destroys it.
func TestWindowsAndLifts(t *testing.T) {
s := shelf(t)
ctx := context.Background()
windowed := suppressed(t, s, acme, &SuppressIn{Rule: "r1", Until: noon.Add(time.Hour)})
if c, _ := s.Cover(ctx, acme, &CoverIn{Rule: "r1"}); !c.Covered {
t.Fatal("a suppression inside its window must cover")
}
s.Now = func() time.Time { return noon.Add(2 * time.Hour) }
if c, _ := s.Cover(ctx, acme, &CoverIn{Rule: "r1"}); c.Covered {
t.Fatal("a suppression past its window must not cover")
}
s.Now = func() time.Time { return noon }
lifted, err := s.Lift(ctx, acme, &LiftIn{ID: windowed.ID, Reason: "volume reviewed", By: "r.okafor"})
if err != nil {
t.Fatal(err)
}
if lifted.LiftedBy != "r.okafor" || lifted.LiftWhy != "volume reviewed" || lifted.Lifted.IsZero() {
t.Fatalf("a lift must name its decider and its reason: %+v", lifted)
}
if _, err := s.Lift(ctx, acme, &LiftIn{ID: windowed.ID, Reason: "again", By: "r.okafor"}); !errors.Is(err, ErrLifted) {
t.Fatalf("lifting twice must be refused, got %v", err)
}
ledger, err := s.Ledger(ctx, acme, &LedgerIn{})
if err != nil {
t.Fatal(err)
}
if len(ledger.Suppressions) != 1 {
t.Fatalf("a lift destroyed the record: %+v", ledger.Suppressions)
}
if ledger.Suppressions[0].By != "a.mensah" {
t.Fatalf("the original decider did not survive the lift: %+v", ledger.Suppressions[0])
}
inForce, err := s.Ledger(ctx, acme, &LedgerIn{InForce: true})
if err != nil {
t.Fatal(err)
}
if len(inForce.Suppressions) != 0 {
t.Fatalf("a lifted suppression is not in force: %+v", inForce.Suppressions)
}
}
// TestClosedWindowRefused: a suppression that could never apply is an operator
// error, and accepting it hides the mistake until somebody wonders why the alerts
// never stopped.
func TestClosedWindowRefused(t *testing.T) {
s := shelf(t)
_, err := s.Suppress(context.Background(), acme, &SuppressIn{
Rule: "r1", Reason: "r", By: "a", Until: noon.Add(-time.Hour),
})
if !errors.Is(err, ErrWindow) {
t.Fatalf("a window that has already closed must be refused, got %v", err)
}
}
// TestTenantIsolation over every operation, including the same org name under a
// second brand.
func TestTenantIsolation(t *testing.T) {
s := shelf(t)
ctx := context.Background()
mine := suppressed(t, s, acme, &SuppressIn{Rule: "r1", Kind: "account", Value: "acct-1"})
for _, stranger := range []string{other, rival} {
c, err := s.Cover(ctx, stranger, &CoverIn{Rule: "r1", Kind: "account", Value: "acct-1"})
if err != nil {
t.Fatal(err)
}
if c.Covered {
t.Errorf("%s is covered by %s's suppression", stranger, acme)
}
ledger, err := s.Ledger(ctx, stranger, &LedgerIn{})
if err != nil {
t.Fatal(err)
}
if len(ledger.Suppressions) != 0 {
t.Errorf("%s can read %s's suppressions", stranger, acme)
}
if _, err := s.Lift(ctx, stranger, &LiftIn{ID: mine.ID, Reason: "r", By: "b"}); !errors.Is(err, ErrNotHere) {
t.Errorf("%s can lift %s's suppression: %v", stranger, acme, err)
}
}
if c, _ := s.Cover(ctx, acme, &CoverIn{Rule: "r1", Kind: "account", Value: "acct-1"}); !c.Covered {
t.Fatal("another tenant lifted this suppression")
}
}
// TestBareOrgIsRefused at the boundary of every operation.
func TestBareOrgIsRefused(t *testing.T) {
s := shelf(t)
ctx := context.Background()
for _, bare := range []string{"acme", "", "unknown/acme"} {
if _, err := s.Suppress(ctx, bare, &SuppressIn{Rule: "r1", Reason: "r", By: "a"}); err == nil {
t.Errorf("Suppress accepted %q as a tenant", bare)
}
if _, err := s.Cover(ctx, bare, &CoverIn{Rule: "r1"}); err == nil {
t.Errorf("Cover accepted %q as a tenant", bare)
}
if _, err := s.Ledger(ctx, bare, &LedgerIn{}); err == nil {
t.Errorf("Ledger accepted %q as a tenant", bare)
}
}
}
// TestNothingDeletes: silence is governed, so the record of it is not disposable
// by the same knob that created it.
func TestNothingDeletes(t *testing.T) {
for _, name := range []string{"suppress.go", "shelf.go"} {
raw, err := os.ReadFile(name)
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(raw), ".Delete(") {
t.Errorf("%s calls Delete: disposal is pkg/retention's decision and nobody else's", name)
}
}
}
// TestRestart: a suppression that vanished on a rollout turns a governed silence
// into a surprise, and one that outlived its record is worse.
func TestRestart(t *testing.T) {
first := instance.New(t)
if err := Ensure(first); err != nil {
t.Fatal(err)
}
before := NewBase(first)
before.Now = func() time.Time { return noon }
mine := suppressed(t, before, acme, &SuppressIn{Rule: "r1", Kind: "account", Value: "acct-1"})
suppressed(t, before, other, &SuppressIn{Rule: "r1", Kind: "account", Value: "acct-1"})
second := instance.Restart(t, first)
if err := Ensure(second); err != nil {
t.Fatal(err)
}
after := NewBase(second)
after.Now = func() time.Time { return noon }
c, err := after.Cover(context.Background(), acme, &CoverIn{Rule: "r1", Kind: "account", Value: "acct-1"})
if err != nil {
t.Fatal(err)
}
if !c.Covered || c.Suppression.ID != mine.ID {
t.Fatalf("the suppression did not survive the restart: %+v", c)
}
if c.Suppression.Reason == "" || c.Suppression.By == "" {
t.Fatalf("the decision behind it did not survive: %+v", c.Suppression)
}
ledger, err := after.Ledger(context.Background(), acme, &LedgerIn{})
if err != nil {
t.Fatal(err)
}
if len(ledger.Suppressions) != 1 {
t.Fatalf("the tenant boundary did not survive the restart: %+v", ledger.Suppressions)
}
if c, _ := after.Cover(context.Background(), rival, &CoverIn{Rule: "r1", Kind: "account", Value: "acct-1"}); c.Covered {
t.Fatal("a third tenant is covered after the restart")
}
}
pkg/topology/search.go
+384
View File
@@ -0,0 +1,384 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package topology
import (
"context"
crypto "crypto/rand"
"encoding/binary"
"fmt"
"sort"
"sync"
"time"
"github.com/luxfi/aml/pkg/anomaly"
"github.com/luxfi/aml/pkg/replay"
"github.com/luxfi/aml/pkg/velocity"
)
// reportLimit is the fallback reporting limit the aggregates are kept against.
// The same number the live path falls back to, for the same reason: a payment
// sitting just under a reporting limit is only visible as structuring if the
// aggregates know where the limit is.
const reportLimit = 10_000.0
// Search replays a tenant's history through every candidate shape and reports
// what each one would have done.
//
// The org is the tenant KEY, and it is not decoration: the detector's geometry is
// seeded from it (anomaly mix), so a search run under a different key studies a
// different set of trees than the one the tenant would get. It is required for
// the same reason a history window refuses an empty tenant.
func Search(ctx context.Context, org string, h replay.History, space Space, opt Options) (Report, error) {
if org == "" {
return Report{}, ErrOrg
}
if h == nil {
return Report{}, ErrNoHistory
}
grid, err := space.Grid()
if err != nil {
return Report{}, err
}
events, cut, from, to, judged, err := collect(h, opt.Events)
if err != nil {
return Report{}, err
}
seed, err := seedOf(opt.Seed)
if err != nil {
return Report{}, err
}
started := time.Now()
report := Report{
Events: len(events), From: from, To: to, Judged: judged, Cut: cut, Seed: seed,
Trials: make([]Trial, len(grid)),
}
workers := opt.Workers
if workers <= 0 {
workers = DefaultWorkers
}
if workers > len(grid) {
workers = len(grid)
}
var (
wg sync.WaitGroup
mu sync.Mutex
first error
next = make(chan int)
)
go func() {
defer close(next)
for i := range grid {
select {
case next <- i:
case <-ctx.Done():
return
}
}
}()
for w := 0; w < workers; w++ {
wg.Add(1)
go func() {
defer wg.Done()
for i := range next {
t, err := run(ctx, org, grid[i], events, seed, opt)
mu.Lock()
if err != nil && first == nil {
first = err
}
report.Trials[i] = t
mu.Unlock()
}
}()
}
wg.Wait()
if first != nil {
return Report{}, first
}
if err := ctx.Err(); err != nil {
return Report{}, err
}
report.Elapsed = time.Since(started)
rank(&report)
return report, nil
}
// Fit builds one shape's learned state from a tenant's own history and hands back
// a snapshot of it.
//
// The snapshot is restorable into a live model ONLY where that model already has
// this shape — anomaly.Restore checks the digest, and Trial.Digest is what a
// caller compares against the running one. That is the property that stops a
// model's shape being changed underneath a tenant by restoring a file, and it is
// why a search's winner is a recommendation to a deployment rather than something
// this package can install.
func Fit(ctx context.Context, org string, h replay.History, t Topology, opt Options) (anomaly.Snapshot, Trial, error) {
if org == "" {
return anomaly.Snapshot{}, Trial{}, ErrOrg
}
if h == nil {
return anomaly.Snapshot{}, Trial{}, ErrNoHistory
}
if err := t.Valid(); err != nil {
return anomaly.Snapshot{}, Trial{}, err
}
events, _, _, _, _, err := collect(h, opt.Events)
if err != nil {
return anomaly.Snapshot{}, Trial{}, err
}
seed, err := seedOf(opt.Seed)
if err != nil {
return anomaly.Snapshot{}, Trial{}, err
}
trial, snap, err := fit(ctx, org, t, events, seed, opt)
if err != nil {
return anomaly.Snapshot{}, Trial{}, err
}
return snap, trial, nil
}
// collect drains the history once, so N candidates cost one pass over the store
// rather than N.
func collect(h replay.History, bound int) (events []replay.Event, cut bool, from, to time.Time, judged int, err error) {
max := bound
if max <= 0 || max > MaxEvents {
max = MaxEvents
}
err = h.Each(func(e replay.Event) error {
if len(events) >= max {
cut = true
return nil
}
if ts := e.Tx.Timestamp; !ts.IsZero() {
if from.IsZero() || ts.Before(from) {
from = ts.UTC()
}
if ts.After(to) {
to = ts.UTC()
}
}
if e.Disposition != replay.Unjudged {
judged++
}
events = append(events, e)
return nil
})
if err != nil {
return nil, false, time.Time{}, time.Time{}, 0, err
}
if len(events) == 0 {
// The same refusal pkg/replay makes, for the same reason: "no alerts" is
// exactly what a quiet shape looks like, and choosing a model on the
// strength of an empty replay is the failure this package exists to
// prevent.
return nil, false, time.Time{}, time.Time{}, 0, ErrEmpty
}
return events, cut, from, to, judged, nil
}
func seedOf(given uint64) (uint64, error) {
if given != 0 {
return given, nil
}
var b [8]byte
if _, err := crypto.Read(b[:]); err != nil {
return 0, fmt.Errorf("topology: seed: %w", err)
}
s := binary.LittleEndian.Uint64(b[:])
if s == 0 {
s = 1
}
return s, nil
}
// run is one candidate, replayed.
func run(ctx context.Context, org string, t Topology, events []replay.Event, seed uint64, opt Options) (Trial, error) {
trial, _, err := fit(ctx, org, t, events, seed, opt)
return trial, err
}
// fit is the one place a candidate meets the data: it builds a detector and an
// aggregate store of its own, replays the events through the detector's own
// learning path, and drops both. Nothing it touches outlives it.
func fit(ctx context.Context, org string, t Topology, events []replay.Event, seed uint64, opt Options) (Trial, anomaly.Snapshot, error) {
vel := velocity.New(velocity.Config{})
model, err := anomaly.New(t.Config(seed), vel)
if err != nil {
return Trial{}, anomaly.Snapshot{}, err
}
limit := opt.Limit
if limit <= 0 {
limit = reportLimit
}
points := opt.Curve
if points <= 0 {
points = DefaultCurve
}
every := len(events) / points
if every < 1 {
every = 1
}
trial := Trial{
Topology: t, Digest: model.Digest(), Seed: seed,
Events: len(events), Stated: t.Review,
Refused: map[string]int64{}, Blind: map[string]int64{},
}
var (
scores []float64
positives []bool
shares = map[string]*Contribution{}
windowSum float64
windowN int64
)
for i, e := range events {
if i%512 == 0 {
if err := ctx.Err(); err != nil {
return Trial{}, anomaly.Snapshot{}, err
}
}
tx := e.Tx
tx.OrgID = org
// The aggregates are what the features are computed from, and the live
// path records the transaction BEFORE it scores it, so the numbers an
// alert quotes are the ones an investigator sees. A replay that scored
// first would measure a different thing than production does.
for _, k := range anomaly.Keys(tx) {
vel.Record(k, tx.Timestamp, tx.USD, limit)
}
a := model.Learn(tx, e.Entity)
if a.Reason != "" {
trial.Refused[a.Reason]++
}
if a.Scored {
trial.Scored++
windowSum += a.Score
windowN++
if a.Alert {
trial.Alerted++
for _, c := range a.Causes {
held := shares[c.Feature]
if held == nil {
held = &Contribution{Feature: c.Feature}
shares[c.Feature] = held
}
held.Alerts++
held.Share += c.Share
}
}
for _, v := range a.Values {
if v.Blind {
trial.Blind[v.Feature]++
}
}
if e.Disposition != replay.Unjudged {
scores = append(scores, a.Score)
positives = append(positives, e.Disposition == replay.Productive)
}
}
// The curve is sampled over the REPLAY and not over the scored events.
// Sampled over the scored ones it would have no points at all through the
// warm-up, which is the part of the curve that says whether the history is
// long enough for the shape.
if (i+1)%every == 0 || i == len(events)-1 {
st := model.State(org)
p := Point{Learned: st.Learned, Cut: st.Cut, Scored: trial.Scored, Alerted: trial.Alerted}
if trial.Scored > 0 {
p.Realised = float64(trial.Alerted) / float64(trial.Scored)
}
if windowN > 0 {
p.Mean = windowSum / float64(windowN)
}
windowSum, windowN = 0, 0
trial.Curve = append(trial.Curve, p)
}
}
st := model.State(org)
trial.Warm, trial.Saturated = st.Warm, st.Saturated
if trial.Scored > 0 {
trial.Realised = float64(trial.Alerted) / float64(trial.Scored)
}
trial.Drift = trial.Realised - trial.Stated
if trial.Drift < 0 {
trial.Drift = -trial.Drift
}
trial.Judged = len(scores)
if v, ok := auc(scores, positives); ok && finite(v) {
trial.Separation = &v
}
trial.Features = make([]Contribution, 0, len(shares))
for _, c := range shares {
if c.Alerts > 0 {
c.Share /= float64(c.Alerts)
}
trial.Features = append(trial.Features, *c)
}
sort.Slice(trial.Features, func(i, j int) bool {
if trial.Features[i].Alerts != trial.Features[j].Alerts {
return trial.Features[i].Alerts > trial.Features[j].Alerts
}
return trial.Features[i].Feature < trial.Features[j].Feature
})
snap, _ := model.Snapshot(org)
return trial, snap, nil
}
// rank picks the winner, or states why there is none.
//
// Eligibility first: a candidate that never warmed scored nothing, and a
// saturated one cannot honour any appetite. Then separation, because that is the
// only outcome measured against something a human concluded. Ties go to the
// candidate whose realised share sits closest to the one asked for, and then to
// the SMALLER model — a shape that ties on evidence and costs less is the better
// answer, and preferring the bigger one would drift the fleet upward every time a
// search ran.
func rank(r *Report) {
if r.Judged == 0 {
r.Refusal = RefusalUnjudged
return
}
var eligible []int
warm := false
for i := range r.Trials {
if r.Trials[i].Warm {
warm = true
}
if r.Trials[i].Warm && !r.Trials[i].Saturated && r.Trials[i].Separation != nil {
eligible = append(eligible, i)
}
}
if len(eligible) == 0 {
if !warm {
r.Refusal = RefusalWarming
} else {
r.Refusal = RefusalSaturated
}
return
}
sort.SliceStable(eligible, func(a, b int) bool {
x, y := r.Trials[eligible[a]], r.Trials[eligible[b]]
if *x.Separation != *y.Separation {
return *x.Separation > *y.Separation
}
if x.Drift != y.Drift {
return x.Drift < y.Drift
}
return x.Topology.Nodes() < y.Topology.Nodes()
})
winner := r.Trials[eligible[0]]
r.Winner = &winner
}
pkg/topology/topology.go
+385
View File
@@ -0,0 +1,385 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// Package topology searches the space of model shapes over a tenant's own
// history, and reports what each one would have done.
//
// A detector has a shape — how many trees, how deep, how long a reference window
// is, how fast the reference folds, and what share of the stream the appetite
// admits — and the shape that suits one institution's traffic does not suit
// another's. Picking it by hand is picking it by taste. This package picks it by
// replaying the institution's own events through the SAME detector production
// runs, one candidate at a time, and reporting the learning curve, the realised
// alert share against the stated one, and where each candidate went blind.
//
// # Dry run, structurally
//
// This package has no store, writes nothing, and imports nothing that has one.
// It reaches history through replay.History, whose one method iterates, and it
// scores through the anomaly package's own model rather than a copy of the
// algorithm — the same argument pkg/replay makes for rules, made again for the
// model. A trial builds its own detector and its own aggregate store, uses them,
// and drops them; a candidate cannot touch what a tenant is being scored by.
//
// # It refuses to name a winner it cannot justify
//
// Ranking topologies needs an outcome to rank against, and the only honest one is
// whether a candidate separates the events a human judged suspicious from the
// events a human dismissed. Where nothing has been judged, this package reports
// every trial and NO winner, with the reason. A ranking invented from unlabelled
// data — most alerts, fewest alerts, best-behaved threshold — would look like a
// recommendation and would be a preference, and an institution that changed its
// monitoring on the strength of one could not say why it did.
//
// # What a winning shape can and cannot be adopted into
//
// Learned state is only meaningful against the shape that produced it, and
// anomaly.Digest is what enforces that: a snapshot fitted under one shape cannot
// be restored into a store running another. So Fit's snapshot is adoptable only
// where the running detector already has that shape, and Trial.Digest is what a
// caller compares against the live model's. That is a governance property rather
// than a limitation — a tenant's model shape cannot be swapped underneath it by
// restoring a file.
package topology
import (
"errors"
"fmt"
"math"
"sort"
"time"
"github.com/luxfi/aml/pkg/anomaly"
)
// Errors.
var (
ErrEmptySpace = errors.New("topology: the space is empty, so there is nothing to try")
ErrHuge = errors.New("topology: the space holds more candidates than one search may run")
ErrNoHistory = errors.New("topology: no history")
ErrEmpty = errors.New("topology: history is empty, so a search proves nothing")
ErrShape = errors.New("topology: candidate shape refused")
ErrOrg = errors.New("topology: no tenant, so the geometry would not be the tenant's")
)
// MaxTrials bounds one search.
//
// A grid is a product, so four values on each of five axes is already a thousand
// candidates and each one replays the whole history. The bound is refused at the
// door rather than absorbed, because a search that silently ran a subset of the
// grid reports a winner chosen from candidates the caller cannot name.
const MaxTrials = 256
// MaxEvents bounds how much history one trial replays, and MaxNodes bounds how
// large a candidate's detector may be — Trees × nodes, which is what the fitted
// state costs to carry.
const (
MaxEvents = 50_000
MaxNodes = 262_144
)
// Refusals a search reports instead of a winner.
const (
// RefusalUnjudged means no event carried a disposition, so no candidate can
// be shown to separate anything.
RefusalUnjudged = "no event has been judged, so no shape can be shown to separate suspicious activity from ordinary activity"
// RefusalWarming means every candidate was still warming at the end of the
// history: the history is too short for the shapes asked for.
RefusalWarming = "every candidate was still warming when the history ran out, so none of them scored anything"
// RefusalSaturated means every warm candidate was saturated — too much of the
// stream scored in the top bucket for any threshold to honour the appetite.
RefusalSaturated = "every warm candidate was saturated, so none of them could honour the appetite asked for"
)
// Topology is one candidate shape of the detector.
type Topology struct {
Trees int `json:"trees"`
Depth int `json:"depth"`
Window int `json:"window"`
Blend float64 `json:"blend"`
Review float64 `json:"review"`
}
// MaxDepth is the deepest tree a candidate may ask for.
//
// It exists because the node count is exponential in the depth and a shift wide
// enough to overflow does not error — it WRAPS, and a wrapped node count is
// negative, which passes a "too large" bound and then allocates until the process
// dies. The ceiling is checked before the arithmetic, and Nodes saturates rather
// than wrapping, so neither of them can be the one that is wrong.
const MaxDepth = 24
// Nodes is how many mass counters one candidate carries per reference, which is
// what bounds both its memory and the size of a snapshot fitted from it.
//
// It saturates instead of overflowing: an unrepresentable count is reported as
// the largest one, which is above every bound and is therefore refused.
func (t Topology) Nodes() int {
if t.Depth < 0 || t.Depth > MaxDepth || t.Trees <= 0 {
return math.MaxInt32
}
per := 1<<(t.Depth+1) - 1
if t.Trees > math.MaxInt32/per {
return math.MaxInt32
}
return t.Trees * per
}
// Valid reports why a candidate cannot be built, if it cannot. The bounds are
// the detector's own — anomaly.Config would silently substitute a default for a
// value outside them, and a trial reported under a shape it did not run is worse
// than a refused one.
func (t Topology) Valid() error {
switch {
case t.Trees <= 0 || t.Depth <= 0 || t.Window <= 0:
return fmt.Errorf("%w: trees %d, depth %d, window %d must all be positive", ErrShape, t.Trees, t.Depth, t.Window)
case t.Blend <= 0 || t.Blend > 1:
return fmt.Errorf("%w: blend %v is outside (0,1]", ErrShape, t.Blend)
case t.Review <= 0 || t.Review > 0.5:
return fmt.Errorf("%w: review %v is outside (0,0.5]", ErrShape, t.Review)
case t.Depth > MaxDepth:
return fmt.Errorf("%w: depth %d, at most %d", ErrShape, t.Depth, MaxDepth)
case t.Nodes() > MaxNodes:
return fmt.Errorf("%w: %d nodes, at most %d", ErrShape, t.Nodes(), MaxNodes)
}
return nil
}
// Config is the candidate as the detector takes it. One conversion, here, so a
// trial cannot run a shape the report does not name.
func (t Topology) Config(seed uint64) anomaly.Config {
return anomaly.Config{
Trees: t.Trees, Depth: t.Depth, Window: t.Window, Blend: t.Blend,
Appetite: anomaly.Appetite{Review: t.Review},
// Not shadow: a trial exists to find out what the candidate WOULD alert
// on, and a shadow model reports no alerts by construction.
Shadow: false,
Seed: seed,
}
}
// Space is the grid to search. An empty axis takes the detector's own default for
// that axis, so a caller may search one dimension without restating the others.
type Space struct {
Trees []int `json:"trees,omitempty"`
Depth []int `json:"depth,omitempty"`
Window []int `json:"window,omitempty"`
Blend []float64 `json:"blend,omitempty"`
Review []float64 `json:"review,omitempty"`
}
// Grid enumerates the space, in a fixed order so two runs of the same space
// produce the same report in the same sequence.
func (s Space) Grid() ([]Topology, error) {
trees := ints(s.Trees, 25)
depth := ints(s.Depth, 8)
window := ints(s.Window, 256)
blend := floats(s.Blend, 0.25)
review := floats(s.Review, 0.01)
n := len(trees) * len(depth) * len(window) * len(blend) * len(review)
if n == 0 {
return nil, ErrEmptySpace
}
if n > MaxTrials {
return nil, fmt.Errorf("%w: %d candidates, at most %d", ErrHuge, n, MaxTrials)
}
out := make([]Topology, 0, n)
for _, tr := range trees {
for _, d := range depth {
for _, w := range window {
for _, b := range blend {
for _, r := range review {
t := Topology{Trees: tr, Depth: d, Window: w, Blend: b, Review: r}
if err := t.Valid(); err != nil {
return nil, err
}
out = append(out, t)
}
}
}
}
}
return out, nil
}
func ints(v []int, fallback int) []int {
if len(v) == 0 {
return []int{fallback}
}
out := append([]int(nil), v...)
sort.Ints(out)
return out
}
func floats(v []float64, fallback float64) []float64 {
if len(v) == 0 {
return []float64{fallback}
}
out := append([]float64(nil), v...)
sort.Float64s(out)
return out
}
// Point is one place on the learning curve: what the model looked like after this
// many events.
//
// The curve is what makes "the history is too short" visible. A candidate whose
// realised share is still climbing at the last point has not settled, and its
// final number is a snapshot of a moving thing rather than a property of the
// shape.
type Point struct {
Learned int64 `json:"learned"`
Cut float64 `json:"cut"`
Scored int64 `json:"scored"`
Alerted int64 `json:"alerted"`
// Realised is the share of scored events that alerted, up to this point.
Realised float64 `json:"realised"`
// Mean is the mean score of the events scored since the previous point, which
// is what shows the reference settling.
Mean float64 `json:"mean"`
}
// Contribution is one feature's part of what a candidate alerted on, summed over
// the run. It is the feature-side view of the topology: which dimensions carried
// the signal for this institution.
type Contribution struct {
Feature string `json:"feature"`
// Alerts is how many alerts this feature appeared in as a cause, and Share is
// the mean of its share across those alerts.
Alerts int64 `json:"alerts"`
Share float64 `json:"share"`
}
// Trial is one candidate's whole result.
type Trial struct {
Topology Topology `json:"topology"`
Digest string `json:"digest"`
Seed uint64 `json:"seed"`
Events int `json:"events"`
Scored int64 `json:"scored"`
Alerted int64 `json:"alerted"`
// Stated is the appetite asked for and Realised is what the candidate
// actually did. Drift is the distance between them, which is the governance
// number: an appetite nobody honours is not a control.
Stated float64 `json:"stated"`
Realised float64 `json:"realised"`
Drift float64 `json:"drift"`
Warm bool `json:"warm"`
Saturated bool `json:"saturated"`
Refused map[string]int64 `json:"refused"`
Blind map[string]int64 `json:"blind"`
Curve []Point `json:"curve"`
Features []Contribution `json:"features"`
// Separation is the area under the ROC curve of this candidate's scores
// against the dispositions a human recorded: 1 is perfect separation, 0.5 is
// none. It is ABSENT and not zero when nothing was judged, because a zero
// reads as a candidate that got everything backwards.
Separation *float64 `json:"separation,omitempty"`
// Judged is how many of the replayed events carried a disposition, which is
// what Separation was computed from.
Judged int `json:"judged"`
}
// Report is a whole search.
type Report struct {
Events int `json:"events"`
From time.Time `json:"from,omitzero"`
To time.Time `json:"to,omitzero"`
Judged int `json:"judged"`
// Cut is true when MaxEvents or the caller's own bound stopped the replay
// before the history ended.
Cut bool `json:"cut,omitempty"`
// Seed the search ran with. Reported so the run can be reproduced exactly by
// supplying it back, which is what makes a recommendation checkable.
Seed uint64 `json:"seed"`
Trials []Trial `json:"trials"`
// Winner is the shape this search recommends, and Refusal is why there is
// none. Exactly one of them is set.
Winner *Trial `json:"winner,omitempty"`
Refusal string `json:"refusal,omitempty"`
Elapsed time.Duration `json:"elapsed"`
}
// Options parameterise a search.
type Options struct {
// Curve is how many points the learning curve carries. Zero takes the
// default; the curve is sampled evenly over the replay.
Curve int `json:"curve,omitempty"`
// Events bounds the replay. Zero takes MaxEvents; a larger value is clamped
// to it and the report says the replay was cut.
Events int `json:"events,omitempty"`
// Workers is how many candidates run at once. Zero takes the default.
Workers int `json:"workers,omitempty"`
// Seed fixes every candidate's geometry, so two searches over one space and
// one history produce the same report. Zero draws one and reports it.
Seed uint64 `json:"seed,omitempty"`
// Limit is the reporting limit the aggregates are kept against, which is what
// makes a payment sitting just under it visible as structuring. Zero takes
// the same fallback the live path uses.
Limit float64 `json:"limit,omitempty"`
}
// DefaultCurve is how many points a learning curve carries when the caller names
// no number, and DefaultWorkers how many candidates run at once.
const (
DefaultCurve = 32
DefaultWorkers = 4
)
// auc is the area under the ROC curve of scores against a binary label, by the
// rank-sum identity: AUC = (R₁ − n₁(n₁+1)/2) / (n₁n₀), where R₁ is the sum of the
// positives' ranks.
//
// Ties take the mean rank, which is what keeps a candidate that gives every event
// the same score at exactly 0.5 rather than at 1 or 0 depending on sort order — a
// detector that says nothing must not be able to win a search.
func auc(scores []float64, positive []bool) (float64, bool) {
n := len(scores)
if n != len(positive) || n == 0 {
return 0, false
}
idx := make([]int, n)
for i := range idx {
idx[i] = i
}
sort.SliceStable(idx, func(a, b int) bool { return scores[idx[a]] < scores[idx[b]] })
ranks := make([]float64, n)
for i := 0; i < n; {
j := i
for j+1 < n && scores[idx[j+1]] == scores[idx[i]] {
j++
}
mean := float64(i+j)/2 + 1
for k := i; k <= j; k++ {
ranks[idx[k]] = mean
}
i = j + 1
}
var n1, n0 int
var r1 float64
for i, p := range positive {
if p {
n1++
r1 += ranks[i]
} else {
n0++
}
}
if n1 == 0 || n0 == 0 {
return 0, false
}
return (r1 - float64(n1)*float64(n1+1)/2) / (float64(n1) * float64(n0)), true
}
func finite(v float64) bool { return !math.IsNaN(v) && !math.IsInf(v, 0) }
pkg/topology/topology_test.go
+337
View File
@@ -0,0 +1,337 @@
package topology
import (
"context"
"errors"
"os"
"strings"
"testing"
"time"
"github.com/luxfi/aml/pkg/anomaly"
"github.com/luxfi/aml/pkg/replay"
"github.com/luxfi/aml/pkg/types"
"github.com/luxfi/aml/pkg/velocity"
)
const (
acme = "hanzo/acme"
other = "zoo/acme"
)
var start = time.Date(2026, 1, 1, 9, 0, 0, 0, time.UTC)
// stream is an institution's traffic: ordinary payments, with an occasional one
// far outside the account's own pattern. The outliers are what a shape either
// separates or does not.
func stream(n int, judged bool) replay.Slice {
out := make(replay.Slice, 0, n)
for i := 0; i < n; i++ {
usd := 100 + float64(i%17)*3
odd := i%53 == 52
if odd {
usd = 90_000
}
tx := types.Transaction{
ID: id(i),
OrgID: acme,
UserID: "u1",
AccountID: "acct-1",
Currency: "USD",
Notional: usd,
USD: usd,
Direction: "in",
Timestamp: start.Add(time.Duration(i) * time.Minute),
}
e := replay.Event{Tx: tx, Entity: types.Entity{ID: "u1", OrgID: acme}}
if judged {
// What a human concluded. The outliers were productive; a sample of the
// ordinary ones was looked at and dismissed.
switch {
case odd:
e.Disposition = replay.Productive
case i%23 == 0:
e.Disposition = replay.Unproductive
}
}
out = append(out, e)
}
return out
}
func id(i int) string { return "tx-" + time.Duration(i).String() }
func small() Space {
return Space{Trees: []int{8}, Depth: []int{6}, Window: []int{16}, Blend: []float64{0.25}, Review: []float64{0.02}}
}
// TestGridIsTheProductAndIsBounded. A grid is a product, so four values on five
// axes is a thousand replays; the bound is refused at the door rather than
// absorbed, because a search that ran a subset reports a winner chosen from
// candidates the caller cannot name.
func TestGridIsTheProductAndIsBounded(t *testing.T) {
grid, err := Space{Trees: []int{8, 16}, Depth: []int{4, 6}}.Grid()
if err != nil {
t.Fatal(err)
}
if len(grid) != 4 {
t.Fatalf("2x2 is four candidates, got %d", len(grid))
}
// An unnamed axis takes the detector's own default rather than being empty.
if grid[0].Window == 0 || grid[0].Blend == 0 || grid[0].Review == 0 {
t.Fatalf("an unnamed axis must take the detector's default: %+v", grid[0])
}
huge := Space{
Trees: seq(4), Depth: seq(4), Window: seq(4),
Blend: fracs(4), Review: fracs(4),
}
if _, err := huge.Grid(); !errors.Is(err, ErrHuge) {
t.Fatalf("a grid past the bound must be refused, got %v", err)
}
if _, err := (Space{Depth: []int{99}}).Grid(); !errors.Is(err, ErrShape) {
t.Fatalf("a candidate past the node bound must be refused, got %v", err)
}
}
func seq(n int) []int {
out := make([]int, n)
for i := range out {
out[i] = 4 + i
}
return out
}
func fracs(n int) []float64 {
out := make([]float64, n)
for i := range out {
out[i] = 0.05 * float64(i+1)
}
return out
}
// TestEmptyHistoryIsRefused, on the same terms as pkg/replay: "no alerts" is
// exactly what a quiet shape looks like, and choosing a model on the strength of
// an empty replay is the failure this package exists to prevent.
func TestEmptyHistoryIsRefused(t *testing.T) {
_, err := Search(context.Background(), acme, replay.Slice{}, small(), Options{Seed: 7})
if !errors.Is(err, ErrEmpty) {
t.Fatalf("an empty history must be refused, got %v", err)
}
if _, err := Search(context.Background(), "", stream(10, false), small(), Options{Seed: 7}); !errors.Is(err, ErrOrg) {
t.Fatalf("a search with no tenant studies nobody's geometry, got %v", err)
}
if _, err := Search(context.Background(), acme, nil, small(), Options{Seed: 7}); !errors.Is(err, ErrNoHistory) {
t.Fatalf("no history must be refused, got %v", err)
}
}
// TestNoWinnerWithoutJudgement is the refusal that keeps a recommendation from
// being a preference. Ranking needs an outcome, and the only honest one is whether
// a shape separates what a human judged suspicious from what a human dismissed.
func TestNoWinnerWithoutJudgement(t *testing.T) {
report, err := Search(context.Background(), acme, stream(400, false), small(), Options{Seed: 7})
if err != nil {
t.Fatal(err)
}
if report.Winner != nil {
t.Fatalf("a winner was named from unlabelled data: %+v", report.Winner)
}
if report.Refusal != RefusalUnjudged {
t.Fatalf("the refusal must say why: %q", report.Refusal)
}
if len(report.Trials) != 1 || report.Trials[0].Separation != nil {
t.Fatalf("separation is absent and not zero when nothing was judged: %+v", report.Trials[0])
}
if report.Trials[0].Judged != 0 {
t.Fatalf("nothing was judged, got %d", report.Trials[0].Judged)
}
}
// TestSearchReportsTheCurveAndTheAppetite — what a console renders and what a
// reviewer reads.
func TestSearchReportsTheCurveAndTheAppetite(t *testing.T) {
report, err := Search(context.Background(), acme, stream(600, true), small(), Options{Seed: 7, Curve: 8})
if err != nil {
t.Fatal(err)
}
trial := report.Trials[0]
if len(trial.Curve) < 8 {
t.Fatalf("the learning curve must carry the points asked for, got %d", len(trial.Curve))
}
if trial.Curve[len(trial.Curve)-1].Learned <= trial.Curve[0].Learned {
t.Fatal("the curve must show the model learning")
}
if !trial.Warm {
t.Fatalf("600 events over a 16-event window must warm a model: %+v", trial)
}
if trial.Stated != 0.02 {
t.Fatalf("the appetite asked for must be on the trial: %v", trial.Stated)
}
if trial.Realised > trial.Stated+0.05 {
t.Fatalf("realised %v is far above the stated %v — the threshold is not honouring the appetite", trial.Realised, trial.Stated)
}
if trial.Drift < 0 {
t.Fatal("drift is a distance and is never negative")
}
if trial.Digest == "" || trial.Seed == 0 {
t.Fatalf("a trial must name the shape it ran and the seed it ran with: %+v", trial)
}
if report.Seed != 7 {
t.Fatalf("the report must carry the seed so the run can be reproduced, got %d", report.Seed)
}
if trial.Separation == nil {
t.Fatal("with judged events, separation must be computed")
}
if len(trial.Features) == 0 {
t.Fatal("a shape that alerted must say which features carried it")
}
}
// TestSearchIsReproducible. A recommendation nobody can re-run is one nobody can
// check.
func TestSearchIsReproducible(t *testing.T) {
a, err := Search(context.Background(), acme, stream(400, true), small(), Options{Seed: 11})
if err != nil {
t.Fatal(err)
}
b, err := Search(context.Background(), acme, stream(400, true), small(), Options{Seed: 11})
if err != nil {
t.Fatal(err)
}
if a.Trials[0].Alerted != b.Trials[0].Alerted || a.Trials[0].Digest != b.Trials[0].Digest {
t.Fatalf("the same seed over the same history must give the same answer: %d vs %d", a.Trials[0].Alerted, b.Trials[0].Alerted)
}
}
// TestGeometryIsTheTenants. The detector is seeded from the tenant key, so a
// search run under a different key studies a different set of trees.
func TestGeometryIsTheTenants(t *testing.T) {
mine, err := Search(context.Background(), acme, stream(400, true), small(), Options{Seed: 11})
if err != nil {
t.Fatal(err)
}
theirs, err := Search(context.Background(), other, stream(400, true), small(), Options{Seed: 11})
if err != nil {
t.Fatal(err)
}
// The same brand-qualified org name under two brands must not be one study.
if mine.Trials[0].Alerted == theirs.Trials[0].Alerted && mine.Trials[0].Realised == theirs.Trials[0].Realised {
t.Log("two tenants happened to alert identically; the geometry check below is the real one")
}
sameShape := mine.Trials[0].Digest == theirs.Trials[0].Digest
if !sameShape {
t.Fatal("the digest identifies the SHAPE and must not vary with the tenant")
}
}
// TestWinnerPrefersSeparationThenTheSmallerModel. A shape that ties on evidence
// and costs less is the better answer; preferring the bigger one drifts the fleet
// upward every time a search runs.
func TestWinnerPrefersSeparationThenTheSmallerModel(t *testing.T) {
space := Space{Trees: []int{8, 24}, Depth: []int{6}, Window: []int{16}, Blend: []float64{0.25}, Review: []float64{0.02}}
report, err := Search(context.Background(), acme, stream(600, true), space, Options{Seed: 7})
if err != nil {
t.Fatal(err)
}
if report.Winner == nil {
t.Fatalf("with judged events a winner must be named or the refusal must say why: %q", report.Refusal)
}
if report.Winner.Separation == nil {
t.Fatal("a winner is chosen on separation and must carry it")
}
best := report.Trials[0]
for _, tr := range report.Trials {
if tr.Separation != nil && (best.Separation == nil || *tr.Separation > *best.Separation) {
best = tr
}
}
if *report.Winner.Separation < *best.Separation {
t.Fatalf("the winner is not the best separator: %v vs %v", *report.Winner.Separation, *best.Separation)
}
}
// TestAUCIsHalfWhenNothingIsSeparated. A detector that says nothing must not be
// able to win a search by sort order.
func TestAUCIsHalfWhenNothingIsSeparated(t *testing.T) {
flat := []float64{0.5, 0.5, 0.5, 0.5}
label := []bool{true, false, true, false}
got, ok := auc(flat, label)
if !ok || got != 0.5 {
t.Fatalf("all-tied scores are no separation: %v (ok=%v)", got, ok)
}
perfect, ok := auc([]float64{0.9, 0.8, 0.2, 0.1}, []bool{true, true, false, false})
if !ok || perfect != 1 {
t.Fatalf("perfect ordering is 1, got %v", perfect)
}
backwards, ok := auc([]float64{0.1, 0.2, 0.8, 0.9}, []bool{true, true, false, false})
if !ok || backwards != 0 {
t.Fatalf("reversed ordering is 0, got %v", backwards)
}
if _, ok := auc([]float64{0.5}, []bool{true}); ok {
t.Fatal("one class is not separation and must be absent")
}
}
// TestFitProducesRestorableState, and only into a model of the same shape.
//
// That is the governance property: a tenant's model shape cannot be swapped
// underneath it by restoring a file.
func TestFitProducesRestorableState(t *testing.T) {
shape := Topology{Trees: 8, Depth: 6, Window: 16, Blend: 0.25, Review: 0.02}
snap, trial, err := Fit(context.Background(), acme, stream(600, true), shape, Options{Seed: 7})
if err != nil {
t.Fatal(err)
}
if snap.OrgID != acme || snap.Learned == 0 {
t.Fatalf("a fit must carry the tenant's learned state: %+v", snap.OrgID)
}
live, err := anomaly.New(shape.Config(7), velocityStore())
if err != nil {
t.Fatal(err)
}
if live.Digest() != trial.Digest {
t.Fatalf("a fit of the running shape must match its digest: %s vs %s", trial.Digest, live.Digest())
}
if err := live.Restore(snap); err != nil {
t.Fatalf("a fit of the running shape must restore: %v", err)
}
if st := live.State(acme); st.Learned != snap.Learned {
t.Fatalf("the restored model did not take the learned state: %d vs %d", st.Learned, snap.Learned)
}
// A model of a DIFFERENT shape refuses it.
otherShape := Topology{Trees: 16, Depth: 6, Window: 16, Blend: 0.25, Review: 0.02}
elsewhere, err := anomaly.New(otherShape.Config(7), velocityStore())
if err != nil {
t.Fatal(err)
}
if err := elsewhere.Restore(snap); err == nil {
t.Fatal("a fit of one shape must not install into a model of another")
}
}
// TestTheSandboxIsStructurallyDry. A study that can write is not a sandbox, and
// the property is held up by the import list rather than by care.
func TestTheSandboxIsStructurallyDry(t *testing.T) {
for _, name := range []string{"topology.go", "search.go"} {
raw, err := os.ReadFile(name)
if err != nil {
t.Fatal(err)
}
src := string(raw)
for _, forbidden := range []string{
"luxfi/aml/pkg/store", "luxfi/aml/pkg/retention", "luxfi/aml/pkg/cases",
"hanzoai/base", "hanzoai/dbx",
} {
if strings.Contains(src, forbidden) {
t.Errorf("%s imports %s: a study of the model must have nothing it could write to", name, forbidden)
}
}
}
}
// velocityStore is the aggregate store a live model reads. It is here rather than
// imported into the package under test's own file because only a test needs one.
func velocityStore() *velocity.Store { return velocity.New(velocity.Config{}) }
pkg/watch/feed.go
+107
View File
@@ -0,0 +1,107 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package watch
import "sync/atomic"
// The live feed.
//
// It is a convenience over the durable plane and never a substitute for it.
// Everything it carries is already written, every question is answered from the
// rows, and a subscriber that misses one can read it back with Feed. What the
// channel adds is latency, and that is all it adds.
//
// A subscriber names its tenant and receives that tenant's activations and
// nothing else. There is no feed over every tenant, and there is no subscription
// whose tenant is decided after the fact — which is the same reason every read in
// this repo takes the tenant as an argument rather than as a filter.
// Buffer is how many activations a subscriber may fall behind by before
// deliveries to it are abandoned.
//
// A blocking send would make one slow reader stop the ingest path, which is the
// wrong failure: the monitoring plane exists to watch transactions, not to be
// paced by whoever is watching it. So the send is non-blocking, the loss is
// counted per tenant, and the count is published on every Feed — because a
// monitor that skipped a detection and does not say so reports calm.
const Buffer = 256
type feed struct {
org string
out chan Activation
}
// Subscribe opens a live feed of one tenant's activations. The returned function
// closes it, and closing twice is safe.
//
// The channel is closed when the subscription ends, so a range over it
// terminates. It is never closed by the publisher: a publisher that closed a
// subscriber's channel would panic the next send racing it.
func (s *Shelf) Subscribe(org string) (<-chan Activation, func()) {
f := &feed{org: org, out: make(chan Activation, Buffer)}
s.mu.Lock()
s.feeds[org] = append(s.feeds[org], f)
if s.dropped[org] == nil {
s.dropped[org] = &atomic.Int64{}
}
s.mu.Unlock()
var once bool
return f.out, func() {
s.mu.Lock()
defer s.mu.Unlock()
if once {
return
}
once = true
kept := s.feeds[org][:0]
for _, other := range s.feeds[org] {
if other != f {
kept = append(kept, other)
}
}
if len(kept) == 0 {
delete(s.feeds, org)
} else {
s.feeds[org] = kept
}
close(f.out)
}
}
// publish offers a written activation to that tenant's subscribers.
//
// It holds the read lock across the sends so that a subscription being closed
// cannot close a channel between the lookup and the send. The sends themselves
// never block, so holding the lock costs one non-blocking send per subscriber.
func (s *Shelf) publish(a Activation) {
s.mu.RLock()
defer s.mu.RUnlock()
for _, f := range s.feeds[a.Org] {
select {
case f.out <- a:
default:
if c := s.dropped[a.Org]; c != nil {
c.Add(1)
}
}
}
}
// Dropped is how many live deliveries this instance has abandoned for a tenant.
//
// It is per tenant and not global: a busy institution's slow console must not
// make another institution's feed look lossy, and the number travels on that
// tenant's own Feed answer.
func (s *Shelf) Dropped(org string) int64 {
s.mu.RLock()
c := s.dropped[org]
s.mu.RUnlock()
if c == nil {
return 0
}
return c.Load()
}
pkg/watch/shelf.go
+617
View File
@@ -0,0 +1,617 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package watch
import (
"context"
"fmt"
"slices"
"sort"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/google/uuid"
"github.com/hanzoai/base/core"
"github.com/hanzoai/dbx"
"github.com/luxfi/aml/pkg/brand"
"github.com/luxfi/aml/pkg/history"
"github.com/luxfi/aml/pkg/store"
"github.com/luxfi/aml/pkg/suppress"
"github.com/luxfi/aml/pkg/types"
)
const (
fieldAt = "at"
fieldRule = "rule"
fieldRuleName = "rule_name"
fieldTypology = "typology"
fieldSeverity = "severity"
fieldScore = "score"
fieldTx = "tx"
fieldKind = "kind"
fieldValue = "value"
fieldAction = "action"
fieldResponse = "response"
fieldSuppressed = "suppressed"
fieldCause = "cause"
fieldBy = "by"
fieldRung = "rung"
fieldStreak = "streak"
fieldCount = "count"
fieldWithin = "within"
fieldTo = "to"
fieldReason = "reason"
fieldRetired = "retired"
fieldOffBy = "retired_by"
fieldOffWhy = "retire_why"
)
var activationKind = store.Kind{
Name: "aml_activations",
Fields: []core.Field{
&core.DateField{Name: fieldAt, Required: true},
&core.TextField{Name: fieldRule, Required: true},
&core.TextField{Name: fieldRuleName},
&core.TextField{Name: fieldTypology},
&core.TextField{Name: fieldSeverity},
&core.NumberField{Name: fieldScore},
&core.TextField{Name: fieldTx},
&core.TextField{Name: fieldKind, Required: true},
&core.TextField{Name: fieldValue, Required: true},
&core.TextField{Name: fieldAction, Required: true},
&core.TextField{Name: fieldResponse, Required: true},
&core.BoolField{Name: fieldSuppressed},
&core.TextField{Name: fieldCause},
&core.TextField{Name: fieldBy},
&core.TextField{Name: fieldRung},
&core.NumberField{Name: fieldStreak},
},
Indexes: []store.Index{
// The feed and the rates: this tenant's activations in time order.
{Name: "at", Fields: []string{store.Org, fieldAt}},
// The streak and the fold: this rule on this subject, most recent first.
{Name: "streak", Fields: []string{store.Org, fieldRule, fieldKind, fieldValue, fieldAt}},
},
}
var rungKind = store.Kind{
Name: "aml_rungs",
Fields: []core.Field{
&core.TextField{Name: fieldRule, Required: true},
&core.TextField{Name: fieldKind, Required: true},
&core.NumberField{Name: fieldCount},
&core.NumberField{Name: fieldWithin},
&core.TextField{Name: fieldTo, Required: true},
&core.TextField{Name: fieldReason, Required: true},
&core.TextField{Name: fieldBy, Required: true},
&core.DateField{Name: fieldAt, Required: true},
&core.DateField{Name: fieldRetired},
&core.TextField{Name: fieldOffBy},
&core.TextField{Name: fieldOffWhy},
},
Indexes: []store.Index{
{Name: "rule", Fields: []string{store.Org, fieldRule, fieldKind}},
},
}
// Ensure creates what activations and rungs are kept in. Idempotent, so it runs
// on every start.
func Ensure(app core.App) error {
if err := activationKind.Ensure(app); err != nil {
return err
}
return rungKind.Ensure(app)
}
// Shelf is the durable activation plane, plus the live feed over it.
type Shelf struct {
app core.App
// Cover is consulted before an activation is marked live. Nil means no
// suppression plane is wired and nothing is ever covered — which is the safe
// default, because the failure it produces is noise rather than silence.
Cover Suppressor
// Now supplies the instant. Tests set it.
Now func() time.Time
mu sync.RWMutex
feeds map[string][]*feed
dropped map[string]*atomic.Int64
}
// Suppressor is the one method this plane needs from pkg/suppress. It is an
// interface so the activation plane can be tested without a suppression plane,
// and so the dependency runs one way only.
type Suppressor interface {
Cover(ctx context.Context, org string, in *suppress.CoverIn) (*suppress.Cover, error)
}
// NewBase returns the durable shelf. Ensure has to have run first.
func NewBase(app core.App) *Shelf {
return &Shelf{app: app, feeds: map[string][]*feed{}, dropped: map[string]*atomic.Int64{}}
}
func (s *Shelf) now() time.Time {
if s.Now != nil {
return s.Now().UTC()
}
return time.Now().UTC()
}
// Record writes one activation and returns it as it was recorded.
//
// The order is the whole design: the row is written first, and only a written row
// is offered to the live feed. Wired the other way a monitor could show a
// detection that the store never took, which is worse than showing none.
func (s *Shelf) Record(ctx context.Context, org string, in *RecordIn) (*Activation, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
rule := strings.TrimSpace(in.Rule)
subj := Subject{Kind: strings.TrimSpace(in.Subject.Kind), Value: strings.TrimSpace(in.Subject.Value)}
switch {
case rule == "":
return nil, ErrRule
case subj.Value == "":
return nil, ErrSubject
case !slices.Contains(history.Kinds, subj.Kind):
return nil, fmt.Errorf("%w: %q, want one of %s", ErrKind, subj.Kind, strings.Join(history.Kinds, ", "))
case !slices.Contains(actions, in.Action):
return nil, fmt.Errorf("%w: %q", ErrAction, in.Action)
}
at := in.At.UTC()
if at.IsZero() {
at = s.now()
}
a := Activation{
ID: uuid.NewString(), Org: org, At: at,
Rule: rule, RuleName: in.RuleName, Typology: in.Typology, Severity: in.Severity,
Score: in.Score, Tx: strings.TrimSpace(in.Tx), Subject: subj,
Action: in.Action, Response: in.Action,
}
// A declared suppression is the first question, because it is a decision
// somebody took and it outranks anything computed from repetition.
if s.Cover != nil {
cover, err := s.Cover.Cover(ctx, org, &suppress.CoverIn{Rule: rule, Kind: subj.Kind, Value: subj.Value, At: at})
if err != nil {
return nil, err
}
if cover.Covered {
a.Suppressed, a.Cause, a.By = true, CauseSuppressed, cover.Suppression.ID
a.Response = types.ActionAllow
}
}
if !a.Suppressed {
if err := s.apply(org, &a); err != nil {
return nil, err
}
}
row, err := activationKind.New(s.app, org)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
row.Id = a.ID
writeActivation(row, a)
if err := s.app.Save(row); err != nil {
return nil, fmt.Errorf("%w: recording %s on %s: %w", ErrStore, rule, subj.Value, err)
}
s.publish(a)
return &a, nil
}
// apply reads the tenant's rungs for this rule and subject kind, counts the
// streak once if any of them could bear on it, and moves the response.
//
// The streak query is skipped entirely when no rung is declared. That matters:
// the ordinary deployment has none, and a monitoring plane that costs an extra
// indexed read per detection for a policy nobody declared is a cost paid for
// nothing.
func (s *Shelf) apply(org string, a *Activation) error {
rungs, err := s.rungs(org, a.Rule, a.Subject.Kind, a.At)
if err != nil {
return err
}
if len(rungs) == 0 {
return nil
}
widest := time.Duration(0)
for _, r := range rungs {
if r.Within.Duration() > widest {
widest = r.Within.Duration()
}
}
prior, err := s.priorInWindow(org, a, widest)
if err != nil {
return err
}
a.Streak = len(prior) + 1
// Elevation first, and the furthest rung reached wins. An activation that
// reached an escalation is the escalation, not a repeat of one.
best := Rung{}
for _, r := range rungs {
if r.To == Fold || r.Count > a.Streak {
continue
}
if within(prior, a.At, r.Within.Duration())+1 < r.Count {
continue
}
if best.ID == "" || r.Count > best.Count {
best = r
}
}
if best.ID != "" && types.ActionRank(best.To) > types.ActionRank(a.Response) {
a.Response, a.Rung = best.To, best.ID
return nil
}
// Folding, only where nothing raised it.
for _, r := range rungs {
if r.To != Fold || r.Count > a.Streak {
continue
}
if within(prior, a.At, r.Within.Duration())+1 < r.Count {
continue
}
a.Suppressed, a.Cause, a.Rung = true, CauseDuplicate, r.ID
a.Response = types.ActionAllow
// By names the activation this one repeats, which is the most recent live
// one inside the window: that is the alert a reviewer already has.
for _, p := range prior {
if !p.Suppressed && a.At.Sub(p.At) <= r.Within.Duration() {
a.By = p.ID
break
}
}
return nil
}
return nil
}
// within counts how many of the prior activations fall inside a window ending at
// the given instant. The rows were read once over the widest window any rung
// declared, so each narrower rung is answered from that one read.
func within(prior []Activation, at time.Time, w time.Duration) int {
n := 0
for _, p := range prior {
if at.Sub(p.At) <= w {
n++
}
}
return n
}
// priorInWindow reads this rule's earlier activations on this subject, most
// recent first. Suppressed ones are included in the read and counted, because a
// streak that ignores what a suppression hid would restart every time somebody
// suppressed one.
func (s *Shelf) priorInWindow(org string, a *Activation, w time.Duration) ([]Activation, error) {
filter := fieldRule + " = {:rule} && " + fieldKind + " = {:kind} && " + fieldValue + " = {:value} && " +
fieldAt + " >= {:from} && " + fieldAt + " < {:at}"
rows, err := activationKind.Find(s.app, org, filter, "-"+fieldAt, MaxLimit,
dbx.Params{"rule": a.Rule, "kind": a.Subject.Kind, "value": a.Subject.Value,
"from": a.At.Add(-w), "at": a.At})
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
out := make([]Activation, 0, len(rows))
for _, row := range rows {
out = append(out, readActivation(row))
}
return out, nil
}
// Feed reads the activation plane forward from an instant.
func (s *Shelf) Feed(ctx context.Context, org string, in *FeedIn) (*Feed, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
limit := bound(in.Limit)
filter := fieldAt + " > {:since}"
params := dbx.Params{"since": in.Since.UTC()}
if r := strings.TrimSpace(in.Rule); r != "" {
filter += " && " + fieldRule + " = {:rule}"
params["rule"] = r
}
if in.Live {
filter += " && " + fieldSuppressed + " = false"
}
rows, err := activationKind.Find(s.app, org, filter, fieldAt+",id", limit+1, params)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
out := &Feed{Activations: make([]Activation, 0, len(rows)), Through: in.Since.UTC(), Dropped: s.Dropped(org)}
if len(rows) > limit {
rows, out.Cut = rows[:limit], true
}
for _, row := range rows {
a := readActivation(row)
out.Activations = append(out.Activations, a)
out.Through = a.At
}
return out, nil
}
// Rates folds the window into one row per rule.
func (s *Shelf) Rates(ctx context.Context, org string, in *RatesIn) (*Rates, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
to := in.Until.UTC()
if to.IsZero() {
to = s.now()
}
from := in.Since.UTC()
if from.IsZero() {
from = to.Add(-24 * time.Hour)
}
rows, err := activationKind.Find(s.app, org, fieldAt+" >= {:from} && "+fieldAt+" <= {:to}", fieldAt, MaxExamined+1,
dbx.Params{"from": from, "to": to})
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
out := &Rates{From: from, To: to}
if len(rows) > MaxExamined {
rows, out.Cut = rows[:MaxExamined], true
}
out.Examined = len(rows)
byRule := map[string]*Rate{}
subjects := map[string]map[string]struct{}{}
for _, row := range rows {
a := readActivation(row)
r, ok := byRule[a.Rule]
if !ok {
r = &Rate{Rule: a.Rule, RuleName: a.RuleName}
byRule[a.Rule] = r
subjects[a.Rule] = map[string]struct{}{}
}
r.Fired++
switch {
case a.Cause == CauseSuppressed:
r.Silenced++
case a.Cause == CauseDuplicate:
r.Folded++
default:
r.Live++
}
if a.Rung != "" && a.Cause == "" {
r.Elevated++
}
if a.Response == types.ActionBlock {
r.Blocked++
}
if a.At.After(r.Last) {
r.Last = a.At
}
subjects[a.Rule][a.Subject.Kind+"\x1f"+a.Subject.Value] = struct{}{}
}
out.Rules = make([]Rate, 0, len(byRule))
for id, r := range byRule {
r.Subjects = len(subjects[id])
out.Rules = append(out.Rules, *r)
}
sort.Slice(out.Rules, func(i, j int) bool {
if out.Rules[i].Fired != out.Rules[j].Fired {
return out.Rules[i].Fired > out.Rules[j].Fired
}
return out.Rules[i].Rule < out.Rules[j].Rule
})
return out, nil
}
// Declare records a rung.
func (s *Shelf) Declare(ctx context.Context, org string, in *DeclareIn) (*Rung, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
rule, kindName := strings.TrimSpace(in.Rule), strings.TrimSpace(in.Kind)
switch {
case rule == "":
return nil, ErrRule
case !slices.Contains(history.Kinds, kindName):
return nil, fmt.Errorf("%w: %q, want one of %s", ErrKind, kindName, strings.Join(history.Kinds, ", "))
case in.To != Fold && !slices.Contains(actions, in.To):
return nil, fmt.Errorf("%w: %q", ErrTo, in.To)
case in.To == types.ActionAllow:
return nil, fmt.Errorf("%w: a rung may only raise a response, and lowering one is a suppression", ErrTo)
case in.Count < 2:
return nil, fmt.Errorf("%w: %d", ErrCount, in.Count)
case in.Within <= 0:
return nil, ErrWithin
case strings.TrimSpace(in.Reason) == "":
return nil, ErrReason
case strings.TrimSpace(in.By) == "":
return nil, ErrDecider
}
r := Rung{
ID: uuid.NewString(), Org: org, Rule: rule, Kind: kindName,
Count: in.Count, Within: in.Within, To: in.To,
Reason: strings.TrimSpace(in.Reason), By: strings.TrimSpace(in.By), At: s.now(),
}
row, err := rungKind.New(s.app, org)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
row.Id = r.ID
writeRung(row, r)
if err := s.app.Save(row); err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
return &r, nil
}
// Retire ends a rung. The row stays and records who ended it and why.
func (s *Shelf) Retire(ctx context.Context, org string, in *RetireIn) (*Rung, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
if strings.TrimSpace(in.Reason) == "" {
return nil, ErrReason
}
if strings.TrimSpace(in.By) == "" {
return nil, ErrDecider
}
rows, err := rungKind.Find(s.app, org, "id = {:id}", "", 1, dbx.Params{"id": strings.TrimSpace(in.ID)})
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
if len(rows) == 0 {
return nil, fmt.Errorf("%w: %s", ErrNotHere, in.ID)
}
r := readRung(rows[0])
if !r.Retired.IsZero() {
return nil, fmt.Errorf("%w: %s", ErrRetired, r.ID)
}
r.Retired, r.RetiredBy, r.RetireWhy = s.now(), strings.TrimSpace(in.By), strings.TrimSpace(in.Reason)
writeRung(rows[0], r)
if err := s.app.Save(rows[0]); err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
return &r, nil
}
// Ladder reads a tenant's rungs, ordered by rule then by how far along the ladder
// each one sits.
func (s *Shelf) Ladder(ctx context.Context, org string, in *LadderIn) (*Ladder, error) {
if err := brand.Tenant(org); err != nil {
return nil, err
}
filter, params := "", dbx.Params{}
if r := strings.TrimSpace(in.Rule); r != "" {
filter, params["rule"] = fieldRule+" = {:rule}", r
}
rows, err := rungKind.Find(s.app, org, filter, fieldRule+","+fieldCount, 0, params)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
at := s.now()
out := &Ladder{Rungs: make([]Rung, 0, len(rows))}
for _, row := range rows {
r := readRung(row)
if in.InForce && !r.InForce(at) {
continue
}
out.Rungs = append(out.Rungs, r)
}
return out, nil
}
// rungs reads the rungs in force for one rule and subject kind.
func (s *Shelf) rungs(org, rule, kindName string, at time.Time) ([]Rung, error) {
rows, err := rungKind.Find(s.app, org, fieldRule+" = {:rule} && "+fieldKind+" = {:kind}", fieldCount, 0,
dbx.Params{"rule": rule, "kind": kindName})
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrStore, err)
}
out := make([]Rung, 0, len(rows))
for _, row := range rows {
if r := readRung(row); r.InForce(at) {
out = append(out, r)
}
}
return out, nil
}
func bound(n int) int {
switch {
case n <= 0:
return DefaultLimit
case n > MaxLimit:
return MaxLimit
default:
return n
}
}
func writeActivation(row *core.Record, a Activation) {
row.Set(fieldAt, a.At.UTC())
row.Set(fieldRule, a.Rule)
row.Set(fieldRuleName, a.RuleName)
row.Set(fieldTypology, a.Typology)
row.Set(fieldSeverity, a.Severity)
row.Set(fieldScore, a.Score)
row.Set(fieldTx, a.Tx)
row.Set(fieldKind, a.Subject.Kind)
row.Set(fieldValue, a.Subject.Value)
row.Set(fieldAction, a.Action)
row.Set(fieldResponse, a.Response)
row.Set(fieldSuppressed, a.Suppressed)
row.Set(fieldCause, a.Cause)
row.Set(fieldBy, a.By)
row.Set(fieldRung, a.Rung)
row.Set(fieldStreak, a.Streak)
}
func readActivation(row *core.Record) Activation {
return Activation{
ID: row.Id,
Org: row.GetString(store.Org),
At: row.GetDateTime(fieldAt).Time(),
Rule: row.GetString(fieldRule),
RuleName: row.GetString(fieldRuleName),
Typology: row.GetString(fieldTypology),
Severity: row.GetString(fieldSeverity),
Score: row.GetFloat(fieldScore),
Tx: row.GetString(fieldTx),
Subject: Subject{Kind: row.GetString(fieldKind), Value: row.GetString(fieldValue)},
Action: row.GetString(fieldAction),
Response: row.GetString(fieldResponse),
Suppressed: row.GetBool(fieldSuppressed),
Cause: row.GetString(fieldCause),
By: row.GetString(fieldBy),
Rung: row.GetString(fieldRung),
Streak: row.GetInt(fieldStreak),
}
}
func writeRung(row *core.Record, r Rung) {
row.Set(fieldRule, r.Rule)
row.Set(fieldKind, r.Kind)
row.Set(fieldCount, r.Count)
row.Set(fieldWithin, int64(r.Within.Duration()/time.Second))
row.Set(fieldTo, r.To)
row.Set(fieldReason, r.Reason)
row.Set(fieldBy, r.By)
row.Set(fieldAt, r.At.UTC())
row.Set(fieldRetired, moment(r.Retired))
row.Set(fieldOffBy, r.RetiredBy)
row.Set(fieldOffWhy, r.RetireWhy)
}
func readRung(row *core.Record) Rung {
return Rung{
ID: row.Id,
Org: row.GetString(store.Org),
Rule: row.GetString(fieldRule),
Kind: row.GetString(fieldKind),
Count: row.GetInt(fieldCount),
Within: Span(time.Duration(row.GetInt(fieldWithin)) * time.Second),
To: row.GetString(fieldTo),
Reason: row.GetString(fieldReason),
By: row.GetString(fieldBy),
At: row.GetDateTime(fieldAt).Time(),
Retired: row.GetDateTime(fieldRetired).Time(),
RetiredBy: row.GetString(fieldOffBy),
RetireWhy: row.GetString(fieldOffWhy),
}
}
func moment(t time.Time) any {
if t.IsZero() {
return ""
}
return t.UTC()
}
pkg/watch/span.go
+46
View File
@@ -0,0 +1,46 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package watch
import (
"encoding/json"
"fmt"
"time"
)
// Span is a length of time that reads and writes as one.
//
// A time.Duration marshals to a count of nanoseconds, so a rung counting over an
// hour arrives on the wire as 3600000000000 and a caller writing 3600 has asked
// for three and a half microseconds. Nothing errors; the rung simply never
// matches. That is the same failure the rule vocabulary refuses for a lookback
// written without a unit (engine parseWindow), and it is refused here the same
// way: the value carries its unit, or it is not a value.
type Span time.Duration
// Duration is the span as the standard library's own type.
func (s Span) Duration() time.Duration { return time.Duration(s) }
// String is the Go duration form: 90m reads back as 1h30m0s.
func (s Span) String() string { return time.Duration(s).String() }
// MarshalJSON writes the duration form.
func (s Span) MarshalJSON() ([]byte, error) { return json.Marshal(time.Duration(s).String()) }
// UnmarshalJSON reads the duration form and refuses anything else, including a
// bare number: a number has no unit, and a window off by three orders of
// magnitude is a control that does not work while appearing to.
func (s *Span) UnmarshalJSON(b []byte) error {
var text string
if err := json.Unmarshal(b, &text); err != nil {
return fmt.Errorf("watch: a window is written with its unit, as in 1h or 15m, not as %s", b)
}
d, err := time.ParseDuration(text)
if err != nil {
return fmt.Errorf("watch: window %q: %w", text, err)
}
*s = Span(d)
return nil
}
pkg/watch/watch.go
+281
View File
@@ -0,0 +1,281 @@
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// Package watch records every detection as it fires and reports what the
// monitoring programme is doing right now.
//
// An ACTIVATION is one rule firing on one subject at one instant. It is written
// before anything else happens to it, and it is never dropped: a detection that a
// suppression covers, or that repeats inside a fold window, is recorded WITH the
// reason it did not reach a queue. A monitoring system that discards what it
// decided not to show cannot answer how much it is not showing, and "no alerts"
// then means either a quiet institution or a silent control with no way to tell
// which.
//
// # The durable row is the record; the feed is a convenience
//
// Record writes the row first and only then offers it to live subscribers. A
// subscriber that cannot keep up is dropped from, and the drops are COUNTED and
// published, because a feed that silently skips is a monitor that reports calm.
// Nothing is ever served from the feed: every question this package answers is
// answered from the durable rows.
//
// # Response, and the two things that change it
//
// Action is what the rule asked for. Response is what this plane concluded, and
// exactly two declared mechanisms move it:
//
// - A SUPPRESSION (pkg/suppress) covers the activation. The response drops to
// allow, the row is marked, and the suppression's id is on it — so "how many
// detections has this suppression silenced" is a query.
// - A RUNG raises it. A rung is a tenant's declared policy: when this rule has
// fired this many times on one subject within this window, the response is
// this. A rung can only RAISE — lowering a response is a suppression by
// another name, and a suppression needs a reason and a decider, which a rung
// is not asked for on every activation it touches.
//
// Folding is the same mechanism with a different outcome: a rung whose To is Fold
// marks the repeat as a duplicate rather than raising it. Elevation beats
// folding, because an activation that reached an escalation rung is the
// escalation and not a repeat of one.
//
// Nothing here deletes and nothing evicts. The activation plane is the evidence
// that a control ran, and disposal is the retention plane's decision alone
// (AMLR Art. 77).
package watch
import (
"errors"
"time"
"github.com/luxfi/aml/pkg/types"
)
// Causes an activation did not reach a queue. Both are ordinary and neither is a
// verdict that nothing happened.
const (
// CauseSuppressed means a declared suppression covered it.
CauseSuppressed = "suppression"
// CauseDuplicate means a fold rung matched: this rule already fired on this
// subject inside the window.
CauseDuplicate = "duplicate"
)
// Fold is the rung outcome that marks a repeat as a duplicate rather than raising
// the response. It is deliberately not an action: folding does not change what
// the institution would do about the first one.
const Fold = "fold"
// Errors.
var (
ErrRule = errors.New("watch: an activation names no rule")
ErrSubject = errors.New("watch: an activation names no subject, so there is nothing for it to be about")
ErrKind = errors.New("watch: unknown subject kind")
ErrAction = errors.New("watch: unknown action")
ErrTo = errors.New("watch: a rung's outcome is fold or an action")
ErrCount = errors.New("watch: a rung counts at least two activations, or it is not about repetition")
ErrWithin = errors.New("watch: a rung needs a window to count within")
ErrReason = errors.New("watch: no reason, so the rung records no decision")
ErrDecider = errors.New("watch: no decider, so the rung records nobody's decision")
ErrNotHere = errors.New("watch: no such rung")
ErrRetired = errors.New("watch: already retired")
ErrStore = errors.New("watch: store")
)
// Subject is what an activation is about: one axis, one identifier.
//
// One subject and not a set, deliberately. A fold window and an escalation streak
// are both counts over "the same thing firing again", and a set makes "the same
// thing" ambiguous — two activations sharing an account but not a device would be
// both a repeat and not one. The caller names the axis the detection is about;
// the transaction id on the row is what reaches everything else the transaction
// touched.
type Subject struct {
Kind string `json:"kind"`
Value string `json:"value"`
}
// Activation is one rule firing on one subject, and what became of it.
type Activation struct {
ID string `json:"id"`
Org string `json:"org"`
At time.Time `json:"at"`
Rule string `json:"rule"`
RuleName string `json:"rule_name,omitempty"`
Typology string `json:"typology,omitempty"`
Severity string `json:"severity,omitempty"`
Score float64 `json:"score,omitempty"`
Tx string `json:"tx,omitempty"`
Subject Subject `json:"subject"`
// Action is what the rule asked for. Response is what this plane concluded.
// Both are kept, because the difference is the whole governance question.
Action string `json:"action"`
Response string `json:"response"`
// Suppressed marks an activation that did not reach a queue, Cause says which
// mechanism, and By names the suppression or the prior activation behind it.
Suppressed bool `json:"suppressed,omitempty"`
Cause string `json:"cause,omitempty"`
By string `json:"by,omitempty"`
// Rung names the declared policy that raised the response, when one did.
Rung string `json:"rung,omitempty"`
// Streak is how many activations of this rule on this subject fall inside the
// widest window any rung declared for it, this one included. Zero when the
// tenant has declared no rung, because the count is not computed then.
Streak int `json:"streak,omitempty"`
}
// Rung is a tenant's declared response to repetition: when this rule has fired
// Count times on one subject of this kind within Within, do To.
type Rung struct {
ID string `json:"id"`
Org string `json:"org"`
Rule string `json:"rule"`
Kind string `json:"kind"`
Count int `json:"count"`
Within Span `json:"within"`
To string `json:"to"`
Reason string `json:"reason"`
By string `json:"by"`
At time.Time `json:"at"`
Retired time.Time `json:"retired,omitzero"`
RetiredBy string `json:"retired_by,omitempty"`
RetireWhy string `json:"retire_why,omitempty"`
}
// InForce reports whether a rung applies at an instant.
func (r Rung) InForce(at time.Time) bool {
return r.Retired.IsZero() || at.Before(r.Retired)
}
// Rate is one rule's activity over a window: what it did and what became of it.
type Rate struct {
Rule string `json:"rule"`
RuleName string `json:"rule_name,omitempty"`
Fired int64 `json:"fired"`
Live int64 `json:"live"`
Silenced int64 `json:"silenced"`
Folded int64 `json:"folded"`
Elevated int64 `json:"elevated"`
Blocked int64 `json:"blocked"`
Last time.Time `json:"last"`
// Subjects is how many distinct subjects this rule fired on. A rule firing a
// thousand times on one account and a rule firing once on a thousand accounts
// are different situations and the count alone does not distinguish them.
Subjects int `json:"subjects"`
}
// The typed operations.
type (
// RecordIn is one detection as it fires.
RecordIn struct {
Rule string `json:"rule"`
RuleName string `json:"rule_name,omitempty"`
Typology string `json:"typology,omitempty"`
Severity string `json:"severity,omitempty"`
Action string `json:"action"`
Score float64 `json:"score,omitempty"`
Tx string `json:"tx,omitempty"`
Subject Subject `json:"subject"`
At time.Time `json:"at,omitzero"`
}
// FeedIn reads the activation plane forward from an instant. It is the poll a
// monitor makes; the live channel is Subscribe.
FeedIn struct {
Since time.Time `json:"since,omitzero"`
Rule string `json:"rule,omitempty"`
// Live restricts the answer to activations that reached a queue.
Live bool `json:"live,omitempty"`
Limit int `json:"limit,omitempty"`
}
// Feed is a page of activations, oldest first, so a poller can carry the last
// instant forward without missing one.
Feed struct {
Activations []Activation `json:"activations"`
// Through is the instant a caller should ask from next. It is the last
// activation's, or the request's Since when the page is empty.
Through time.Time `json:"through"`
Cut bool `json:"cut,omitempty"`
// Dropped is how many live-feed deliveries this instance has abandoned
// because a subscriber could not keep up. It travels on the poll because a
// monitor reading the feed is exactly who needs to know the feed is lossy.
Dropped int64 `json:"dropped"`
}
// RatesIn asks what the rules have been doing.
RatesIn struct {
Since time.Time `json:"since,omitzero"`
Until time.Time `json:"until,omitzero"`
}
// Rates is the answer, one row per rule, busiest first.
Rates struct {
From time.Time `json:"from"`
To time.Time `json:"to"`
Rules []Rate `json:"rules"`
// Examined is how many activations the answer was computed from, and Cut
// says the window held more than could be read. A rate computed from a
// truncated window under-reports, so the truncation is published rather
// than absorbed.
Examined int `json:"examined"`
Cut bool `json:"cut,omitempty"`
}
// DeclareIn declares a rung.
DeclareIn struct {
Rule string `json:"rule"`
Kind string `json:"kind"`
Count int `json:"count"`
Within Span `json:"within"`
To string `json:"to"`
Reason string `json:"reason"`
By string `json:"by"`
}
// RetireIn ends a rung. The row stays.
RetireIn struct {
ID string `json:"id"`
Reason string `json:"reason"`
By string `json:"by"`
}
// LadderIn reads a tenant's rungs.
LadderIn struct {
Rule string `json:"rule,omitempty"`
InForce bool `json:"in_force,omitempty"`
}
// Ladder is every rung a tenant has declared, ordered by rule then by count so
// it reads as the ladder it is.
Ladder struct {
Rungs []Rung `json:"rungs"`
}
)
// DefaultLimit is how many activations a read returns when the caller names no
// bound, and MaxLimit is the most it will return however large a bound is asked
// for.
const (
DefaultLimit = 200
MaxLimit = 5000
)
// MaxExamined bounds the rows one Rates call folds over.
//
// It is a real bound and not a page: a rate is a fold over a window and there is
// no correct partial answer, so reaching it is reported on the answer rather than
// hidden inside it.
const MaxExamined = 50_000
// actions are the responses a rung may name, alongside Fold.
var actions = []string{types.ActionAllow, types.ActionFlag, types.ActionReview, types.ActionReport, types.ActionBlock}
pkg/watch/watch_test.go
+578
View File
@@ -0,0 +1,578 @@
package watch
import (
"context"
"errors"
"os"
"strings"
"testing"
"time"
"github.com/luxfi/aml/internal/instance"
"github.com/luxfi/aml/pkg/suppress"
"github.com/luxfi/aml/pkg/types"
)
const (
acme = "hanzo/acme"
rival = "hanzo/rival"
// other is the SAME org name under a different brand.
other = "zoo/acme"
)
var noon = time.Date(2026, 3, 1, 12, 0, 0, 0, time.UTC)
func shelf(t *testing.T) (*Shelf, *suppress.Shelf) {
t.Helper()
app := instance.New(t)
t.Cleanup(app.Cleanup)
if err := Ensure(app); err != nil {
t.Fatalf("ensure: %v", err)
}
if err := suppress.Ensure(app); err != nil {
t.Fatalf("ensure suppress: %v", err)
}
silence := suppress.NewBase(app)
silence.Now = func() time.Time { return noon }
s := NewBase(app)
s.Cover = silence
s.Now = func() time.Time { return noon }
return s, silence
}
func fired(t *testing.T, s *Shelf, org, rule, account string, at time.Time) *Activation {
t.Helper()
a, err := s.Record(context.Background(), org, &RecordIn{
Rule: rule, RuleName: "structuring", Severity: types.SeverityHigh,
Action: types.ActionReview, Tx: "tx-" + at.Format(time.RFC3339Nano),
Subject: Subject{Kind: "account", Value: account}, At: at,
})
if err != nil {
t.Fatalf("record: %v", err)
}
return a
}
// TestActivationRefusalsAreRefusals: an activation with no rule or no subject has
// nothing it could be about, and recording it against an empty subject would pool
// every anonymous detection in the tenant under one imaginary customer.
func TestActivationRefusalsAreRefusals(t *testing.T) {
s, _ := shelf(t)
ctx := context.Background()
base := RecordIn{Rule: "r1", Action: types.ActionReview, Subject: Subject{Kind: "account", Value: "acct-1"}}
no := base
no.Rule = ""
if _, err := s.Record(ctx, acme, &no); !errors.Is(err, ErrRule) {
t.Errorf("an activation with no rule must be refused, got %v", err)
}
no = base
no.Subject.Value = ""
if _, err := s.Record(ctx, acme, &no); !errors.Is(err, ErrSubject) {
t.Errorf("an activation with no subject must be refused, got %v", err)
}
no = base
no.Subject.Kind = "wallet"
if _, err := s.Record(ctx, acme, &no); !errors.Is(err, ErrKind) {
t.Errorf("an unknown subject kind must be refused, got %v", err)
}
no = base
no.Action = "quarantine"
if _, err := s.Record(ctx, acme, &no); !errors.Is(err, ErrAction) {
t.Errorf("an unknown action must be refused, got %v", err)
}
}
// TestSuppressedIsRecordedNotDropped is the whole doctrine of this package.
//
// A detection that a suppression covers is written, marked, and named with the
// suppression that covered it. Dropped instead, the institution could not say how
// much it is not showing, and "no alerts" would mean either a quiet institution or
// a silent control with no way to tell which.
func TestSuppressedIsRecordedNotDropped(t *testing.T) {
s, silence := shelf(t)
ctx := context.Background()
sup, err := silence.Suppress(ctx, acme, suppressFor("r1", "acct-1"))
if err != nil {
t.Fatal(err)
}
a := fired(t, s, acme, "r1", "acct-1", noon)
if !a.Suppressed || a.Cause != CauseSuppressed || a.By != sup.ID {
t.Fatalf("a covered activation must be marked and must name the suppression: %+v", a)
}
if a.Action != types.ActionReview || a.Response != types.ActionAllow {
t.Fatalf("Action is what the rule asked for and Response what happened: %+v", a)
}
feed, err := s.Feed(ctx, acme, &FeedIn{})
if err != nil {
t.Fatal(err)
}
if len(feed.Activations) != 1 {
t.Fatalf("a suppressed activation must still be in the record: %+v", feed.Activations)
}
live, err := s.Feed(ctx, acme, &FeedIn{Live: true})
if err != nil {
t.Fatal(err)
}
if len(live.Activations) != 0 {
t.Fatalf("a suppressed activation did not reach a queue: %+v", live.Activations)
}
// The volume a suppression silences is a query, which is why nothing keeps a
// counter of it.
rates, err := s.Rates(ctx, acme, &RatesIn{Since: noon.Add(-time.Hour), Until: noon.Add(time.Hour)})
if err != nil {
t.Fatal(err)
}
if len(rates.Rules) != 1 || rates.Rules[0].Silenced != 1 || rates.Rules[0].Live != 0 {
t.Fatalf("the rates must show the silence: %+v", rates.Rules)
}
}
// suppressFor is a narrow suppression carrying the decision every one of them
// carries.
func suppressFor(rule, account string) *suppress.SuppressIn {
return &suppress.SuppressIn{
Rule: rule, Kind: "account", Value: account,
Reason: "settlement pattern reviewed 2026-02", By: "a.mensah",
}
}
// TestFoldMarksTheRepeat. Folding is a declared policy, not a default: with no
// rung, every activation is live.
func TestFoldMarksTheRepeat(t *testing.T) {
s, _ := shelf(t)
ctx := context.Background()
first := fired(t, s, acme, "r1", "acct-1", noon)
second := fired(t, s, acme, "r1", "acct-1", noon.Add(time.Minute))
if first.Suppressed || second.Suppressed {
t.Fatal("with no rung declared, nothing folds — silence must be a decision")
}
if _, err := s.Declare(ctx, acme, &DeclareIn{
Rule: "r1", Kind: "account", Count: 2, Within: Span(time.Hour), To: Fold,
Reason: "one alert per account per hour is enough to act on", By: "a.mensah",
}); err != nil {
t.Fatal(err)
}
third := fired(t, s, acme, "r1", "acct-1", noon.Add(2*time.Minute))
if !third.Suppressed || third.Cause != CauseDuplicate {
t.Fatalf("a repeat inside the window must be marked a duplicate: %+v", third)
}
if third.By != second.ID {
t.Fatalf("a duplicate must name the alert a reviewer already has: by=%s want %s", third.By, second.ID)
}
if third.Response != types.ActionAllow {
t.Fatalf("a folded activation reaches no queue: %+v", third)
}
// Past the window it is not a repeat.
later := fired(t, s, acme, "r1", "acct-1", noon.Add(2*time.Hour))
if later.Suppressed {
t.Fatalf("an activation past the fold window is not a duplicate: %+v", later)
}
// And another account was never part of the streak.
elsewhere := fired(t, s, acme, "r1", "acct-2", noon.Add(3*time.Minute))
if elsewhere.Suppressed {
t.Fatalf("a different subject is not a repeat: %+v", elsewhere)
}
}
// TestElevationRaisesAndBeatsFolding.
func TestElevationRaisesAndBeatsFolding(t *testing.T) {
s, _ := shelf(t)
ctx := context.Background()
if _, err := s.Declare(ctx, acme, &DeclareIn{
Rule: "r1", Kind: "account", Count: 2, Within: Span(time.Hour), To: Fold,
Reason: "one per hour", By: "a.mensah",
}); err != nil {
t.Fatal(err)
}
rung, err := s.Declare(ctx, acme, &DeclareIn{
Rule: "r1", Kind: "account", Count: 3, Within: Span(time.Hour), To: types.ActionBlock,
Reason: "three in an hour on one account is a decision the board has taken", By: "a.mensah",
})
if err != nil {
t.Fatal(err)
}
fired(t, s, acme, "r1", "acct-1", noon)
folded := fired(t, s, acme, "r1", "acct-1", noon.Add(time.Minute))
if !folded.Suppressed || folded.Cause != CauseDuplicate {
t.Fatalf("the second must fold: %+v", folded)
}
third := fired(t, s, acme, "r1", "acct-1", noon.Add(2*time.Minute))
if third.Suppressed {
t.Fatalf("an activation that reached an escalation rung is the escalation, not a repeat: %+v", third)
}
if third.Response != types.ActionBlock || third.Rung != rung.ID {
t.Fatalf("the third must be raised and must name the rung: %+v", third)
}
if third.Streak != 3 {
t.Fatalf("streak = %d, want 3", third.Streak)
}
}
// TestARungMayNotLowerAResponse. Lowering is a suppression by another name, and a
// suppression is asked for a reason and a decider on the decision itself.
func TestARungMayNotLowerAResponse(t *testing.T) {
s, _ := shelf(t)
ctx := context.Background()
if _, err := s.Declare(ctx, acme, &DeclareIn{
Rule: "r1", Kind: "account", Count: 2, Within: Span(time.Hour), To: types.ActionAllow,
Reason: "r", By: "a",
}); !errors.Is(err, ErrTo) {
t.Fatalf("a rung lowering a response must be refused, got %v", err)
}
// And a rung naming a weaker action than the rule asked for leaves it alone.
if _, err := s.Declare(ctx, acme, &DeclareIn{
Rule: "r1", Kind: "account", Count: 2, Within: Span(time.Hour), To: types.ActionFlag,
Reason: "r", By: "a",
}); err != nil {
t.Fatal(err)
}
fired(t, s, acme, "r1", "acct-1", noon)
second := fired(t, s, acme, "r1", "acct-1", noon.Add(time.Minute))
if second.Response != types.ActionReview {
t.Fatalf("a rung must never weaken what the rule asked for: %+v", second)
}
}
// TestRungRefusals.
func TestRungRefusals(t *testing.T) {
s, _ := shelf(t)
ctx := context.Background()
good := DeclareIn{Rule: "r1", Kind: "account", Count: 2, Within: Span(time.Hour), To: Fold, Reason: "r", By: "a"}
bad := good
bad.Count = 1
if _, err := s.Declare(ctx, acme, &bad); !errors.Is(err, ErrCount) {
t.Errorf("a rung counting one is not about repetition, got %v", err)
}
bad = good
bad.Within = 0
if _, err := s.Declare(ctx, acme, &bad); !errors.Is(err, ErrWithin) {
t.Errorf("a rung needs a window, got %v", err)
}
bad = good
bad.Reason = ""
if _, err := s.Declare(ctx, acme, &bad); !errors.Is(err, ErrReason) {
t.Errorf("a rung needs a reason, got %v", err)
}
bad = good
bad.By = ""
if _, err := s.Declare(ctx, acme, &bad); !errors.Is(err, ErrDecider) {
t.Errorf("a rung needs a decider, got %v", err)
}
}
// TestRetiringARungKeepsIt.
func TestRetiringARungKeepsIt(t *testing.T) {
s, _ := shelf(t)
ctx := context.Background()
rung, err := s.Declare(ctx, acme, &DeclareIn{
Rule: "r1", Kind: "account", Count: 2, Within: Span(time.Hour), To: Fold, Reason: "r", By: "a.mensah",
})
if err != nil {
t.Fatal(err)
}
if _, err := s.Retire(ctx, acme, &RetireIn{ID: rung.ID, Reason: "volumes fell", By: "r.okafor"}); err != nil {
t.Fatal(err)
}
if _, err := s.Retire(ctx, acme, &RetireIn{ID: rung.ID, Reason: "again", By: "r.okafor"}); !errors.Is(err, ErrRetired) {
t.Fatalf("retiring twice must be refused, got %v", err)
}
all, err := s.Ladder(ctx, acme, &LadderIn{})
if err != nil {
t.Fatal(err)
}
if len(all.Rungs) != 1 || all.Rungs[0].RetiredBy != "r.okafor" {
t.Fatalf("a retired rung stays, and names who retired it: %+v", all.Rungs)
}
inForce, err := s.Ladder(ctx, acme, &LadderIn{InForce: true})
if err != nil {
t.Fatal(err)
}
if len(inForce.Rungs) != 0 {
t.Fatalf("a retired rung is not in force: %+v", inForce.Rungs)
}
fired(t, s, acme, "r1", "acct-1", noon)
second := fired(t, s, acme, "r1", "acct-1", noon.Add(time.Minute))
if second.Suppressed {
t.Fatalf("a retired rung must not still fold: %+v", second)
}
}
// TestFeedPagesForwardWithoutSkipping. A poller carries Through forward, so a page
// boundary must not lose the activation that sat on it.
func TestFeedPagesForwardWithoutSkipping(t *testing.T) {
s, _ := shelf(t)
ctx := context.Background()
for i := 0; i < 5; i++ {
fired(t, s, acme, "r1", "acct-1", noon.Add(time.Duration(i)*time.Minute))
}
seen := map[string]bool{}
since := time.Time{}
for range 5 {
page, err := s.Feed(ctx, acme, &FeedIn{Since: since, Limit: 2})
if err != nil {
t.Fatal(err)
}
if len(page.Activations) == 0 {
break
}
for _, a := range page.Activations {
if seen[a.ID] {
t.Fatalf("the feed handed out %s twice", a.ID)
}
seen[a.ID] = true
}
since = page.Through
}
if len(seen) != 5 {
t.Fatalf("the feed skipped activations: saw %d of 5", len(seen))
}
}
// TestLiveFeedIsLossyAndSaysSo. A blocking send would let one slow console pace
// the ingest path; a silent drop would let it report calm.
func TestLiveFeedIsLossyAndSaysSo(t *testing.T) {
s, _ := shelf(t)
ctx := context.Background()
ch, close := s.Subscribe(acme)
defer close()
for i := 0; i < Buffer+5; i++ {
fired(t, s, acme, "r1", "acct-1", noon.Add(time.Duration(i)*time.Second))
}
if got := s.Dropped(acme); got == 0 {
t.Fatal("a subscriber that never read must have been dropped from, and the drops counted")
}
feed, err := s.Feed(ctx, acme, &FeedIn{Limit: 1})
if err != nil {
t.Fatal(err)
}
if feed.Dropped == 0 {
t.Fatal("the drop count must travel on the poll, because the monitor reading it is who needs to know")
}
// The durable plane lost nothing.
all, err := s.Feed(ctx, acme, &FeedIn{Limit: MaxLimit})
if err != nil {
t.Fatal(err)
}
if len(all.Activations) != Buffer+5 {
t.Fatalf("the record is the rows, not the channel: %d of %d", len(all.Activations), Buffer+5)
}
if len(ch) == 0 {
t.Fatal("the subscriber should hold a full buffer")
}
}
// TestFeedIsPerTenant: a subscription names its tenant and receives that tenant's
// activations and nothing else.
func TestFeedIsPerTenant(t *testing.T) {
s, _ := shelf(t)
ch, close := s.Subscribe(acme)
defer close()
fired(t, s, other, "r1", "acct-1", noon)
fired(t, s, rival, "r1", "acct-1", noon)
mine := fired(t, s, acme, "r1", "acct-1", noon)
select {
case got := <-ch:
if got.ID != mine.ID || got.Org != acme {
t.Fatalf("a subscriber received another tenant's activation: %+v", got)
}
case <-time.After(time.Second):
t.Fatal("the subscriber received nothing")
}
select {
case got := <-ch:
t.Fatalf("the subscriber received a second activation it should not have: %+v", got)
default:
}
}
// TestTenantIsolation over the reads and the policy writes.
func TestTenantIsolation(t *testing.T) {
s, _ := shelf(t)
ctx := context.Background()
fired(t, s, acme, "r1", "acct-1", noon)
rung, err := s.Declare(ctx, acme, &DeclareIn{
Rule: "r1", Kind: "account", Count: 2, Within: Span(time.Hour), To: Fold, Reason: "r", By: "a",
})
if err != nil {
t.Fatal(err)
}
for _, stranger := range []string{other, rival} {
feed, err := s.Feed(ctx, stranger, &FeedIn{})
if err != nil {
t.Fatal(err)
}
if len(feed.Activations) != 0 {
t.Errorf("%s can read %s's activations", stranger, acme)
}
rates, err := s.Rates(ctx, stranger, &RatesIn{Since: noon.Add(-time.Hour), Until: noon.Add(time.Hour)})
if err != nil {
t.Fatal(err)
}
if len(rates.Rules) != 0 {
t.Errorf("%s can read %s's rates", stranger, acme)
}
ladder, err := s.Ladder(ctx, stranger, &LadderIn{})
if err != nil {
t.Fatal(err)
}
if len(ladder.Rungs) != 0 {
t.Errorf("%s can read %s's rungs", stranger, acme)
}
if _, err := s.Retire(ctx, stranger, &RetireIn{ID: rung.ID, Reason: "r", By: "b"}); !errors.Is(err, ErrNotHere) {
t.Errorf("%s can retire %s's rung: %v", stranger, acme, err)
}
}
// A streak is per tenant: another brand's activations on the same subject do
// not count towards this one's fold.
fired(t, s, other, "r1", "acct-1", noon.Add(time.Minute))
fired(t, s, other, "r1", "acct-1", noon.Add(2*time.Minute))
mine := fired(t, s, acme, "r1", "acct-1", noon.Add(3*time.Minute))
if mine.Streak != 2 {
t.Fatalf("a streak counted another tenant's activations: streak=%d, want 2", mine.Streak)
}
}
// TestBareOrgIsRefused at the boundary of every operation.
func TestBareOrgIsRefused(t *testing.T) {
s, _ := shelf(t)
ctx := context.Background()
for _, bare := range []string{"acme", "", "unknown/acme"} {
in := RecordIn{Rule: "r1", Action: types.ActionReview, Subject: Subject{Kind: "account", Value: "a"}}
if _, err := s.Record(ctx, bare, &in); err == nil {
t.Errorf("Record accepted %q as a tenant", bare)
}
if _, err := s.Feed(ctx, bare, &FeedIn{}); err == nil {
t.Errorf("Feed accepted %q as a tenant", bare)
}
if _, err := s.Rates(ctx, bare, &RatesIn{}); err == nil {
t.Errorf("Rates accepted %q as a tenant", bare)
}
}
}
// TestSpanCarriesItsUnit. A window written as a bare number is off by three orders
// of magnitude and nothing errors — the rung simply never matches.
func TestSpanCarriesItsUnit(t *testing.T) {
var s Span
if err := s.UnmarshalJSON([]byte("3600")); err == nil {
t.Fatal("a window with no unit must be refused")
}
if err := s.UnmarshalJSON([]byte(`"1h30m"`)); err != nil {
t.Fatalf("a window with its unit must be accepted: %v", err)
}
if s.Duration() != 90*time.Minute {
t.Fatalf("span = %v, want 1h30m", s.Duration())
}
raw, err := s.MarshalJSON()
if err != nil {
t.Fatal(err)
}
if string(raw) != `"1h30m0s"` {
t.Fatalf("a span writes as a duration, got %s", raw)
}
}
// TestNothingDeletes.
func TestNothingDeletes(t *testing.T) {
for _, name := range []string{"watch.go", "shelf.go", "feed.go", "span.go"} {
raw, err := os.ReadFile(name)
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(raw), ".Delete(") {
t.Errorf("%s calls Delete: disposal is pkg/retention's decision and nobody else's", name)
}
}
}
// TestRestart: the activation plane is the evidence that a control ran.
func TestRestart(t *testing.T) {
first := instance.New(t)
if err := Ensure(first); err != nil {
t.Fatal(err)
}
if err := suppress.Ensure(first); err != nil {
t.Fatal(err)
}
silence := suppress.NewBase(first)
silence.Now = func() time.Time { return noon }
before := NewBase(first)
before.Cover = silence
before.Now = func() time.Time { return noon }
if _, err := silence.Suppress(context.Background(), acme, suppressFor("r2", "acct-9")); err != nil {
t.Fatal(err)
}
if _, err := before.Declare(context.Background(), acme, &DeclareIn{
Rule: "r1", Kind: "account", Count: 2, Within: Span(time.Hour), To: types.ActionBlock,
Reason: "board decision", By: "a.mensah",
}); err != nil {
t.Fatal(err)
}
live := fired(t, before, acme, "r1", "acct-1", noon)
fired(t, before, other, "r1", "acct-1", noon)
second := instance.Restart(t, first)
if err := Ensure(second); err != nil {
t.Fatal(err)
}
if err := suppress.Ensure(second); err != nil {
t.Fatal(err)
}
after := NewBase(second)
afterSilence := suppress.NewBase(second)
afterSilence.Now = func() time.Time { return noon }
after.Cover = afterSilence
after.Now = func() time.Time { return noon }
feed, err := after.Feed(context.Background(), acme, &FeedIn{})
if err != nil {
t.Fatal(err)
}
if len(feed.Activations) != 1 || feed.Activations[0].ID != live.ID {
t.Fatalf("the activation did not survive the restart: %+v", feed.Activations)
}
if feed.Activations[0].Action != types.ActionReview || feed.Activations[0].RuleName != "structuring" {
t.Fatalf("the activation came back changed: %+v", feed.Activations[0])
}
// The rung is still in force: the streak continues rather than restarting, so
// the second activation after a rollout is the second and not the first.
next := fired(t, after, acme, "r1", "acct-1", noon.Add(time.Minute))
if next.Streak != 2 || next.Response != types.ActionBlock {
t.Fatalf("the declared escalation did not survive the restart: %+v", next)
}
// And a suppression declared before the restart still covers after it.
covered, err := after.Record(context.Background(), acme, &RecordIn{
Rule: "r2", Action: types.ActionReview, Subject: Subject{Kind: "account", Value: "acct-9"}, At: noon,
})
if err != nil {
t.Fatal(err)
}
if !covered.Suppressed || covered.Cause != CauseSuppressed {
t.Fatalf("the suppression did not survive the restart: %+v", covered)
}
// Tenancy still holds.
if f, _ := after.Feed(context.Background(), rival, &FeedIn{}); len(f.Activations) != 0 {
t.Fatal("a third tenant can read activations after the restart")
}
}