mirror of
https://github.com/luxfi/aml.git
synced 2026-09-24 00:25:20 +00:00
aml: anomaly scoring, retention, tokenisation, replay, reference data
Five gaps that a monitoring product cannot be sold without. anomaly scores whether a transaction is unusual for the entity that made it, which is the statutory question — consistency with the customer's own profile — and one no fixed threshold can express. It complements the rule library rather than replacing it: rules encode typologies someone has already named, and are blind to anything nobody has written down. The design is deliberately attributable: a score can name the features that drove it, which is what a supervisor asks for and what a neural scorer cannot answer. retention keeps records for their statutory period and destroys them when it runs out, including the retained assessment whether or not it produced a report. token protects direct identifiers at rest while keeping the engine able to correlate: one key schedule, a deterministic org-scoped pseudonym to index on and a sealed record for reads that are entitled to it, so a record can be found by party without holding the party's name in the clear. replay runs a rule over real history and reports what it would have done before anyone activates it — the same question as "how many alerts would this raise", which is what makes a rule change reviewable rather than a live experiment. reference holds the published data a rule needs but cannot derive: currency conversion and jurisdiction risk listings, each carrying the date it was current at, each refusing to answer rather than guess. Passing an unknown currency through unchanged moves every threshold by the exchange rate — the structuring rule compared raw notional, so 9,500 units of a minor currency read as a near-threshold cash amount. A jurisdiction list that answers "not listed" because it was never loaded is indistinguishable from a clean world. Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
Hanzo Dev
@@ -0,0 +1,985 @@
|
||||
// Package anomaly scores whether a transaction is unusual for the entity that
|
||||
// made it, as a complement to the rule library rather than a replacement for it.
|
||||
//
|
||||
// WHY THIS EXISTS. Rules encode typologies someone has already named. They are
|
||||
// precise, defensible and blind to anything nobody has written down yet, and
|
||||
// because each one is a fixed predicate, the only way to widen coverage is to
|
||||
// widen the predicate — which is how a monitoring system arrives at a volume of
|
||||
// alerts that does not repay the cost of reading them. A density model answers a
|
||||
// different question: not "does this match a known pattern" but "is this where
|
||||
// this customer's behaviour normally lives". That is the statutory question. What
|
||||
// must be monitored is consistency with the customer's own profile, and no fixed
|
||||
// threshold can express it: a limit that fits a large customer misses a small one
|
||||
// behaving strangely, and one that fits the small customer buries the large one.
|
||||
//
|
||||
// WHY ATTRIBUTABILITY DECIDED THE ALGORITHM. Machine detection is admissible
|
||||
// here only behind three things: a mapping from typologies to the data features
|
||||
// that express them, a stated appetite for how much the model may miss, and, for
|
||||
// every alert that reaches an investigator, an explanation illustrated by the
|
||||
// features that influenced it and the risk indicators behind them. The third
|
||||
// requirement eliminates most of the field. A neural network's contribution to a
|
||||
// score is not naturally attributable to an input, so anything built on one needs
|
||||
// a second model fitted afterwards to guess at its own reasons — and a supervisor
|
||||
// is then being shown an approximation of an explanation, which is a weaker thing
|
||||
// than it appears. A tree over half-spaces can be interrogated directly: the
|
||||
// score is a pure function of the coordinates, so moving one coordinate to its
|
||||
// neutral value and rescoring gives that feature's contribution exactly, on the
|
||||
// model that raised the alert. See Store.explain. Accuracy was not the deciding
|
||||
// criterion; the ability to answer the question a regulator actually asks was.
|
||||
//
|
||||
// WHAT IT CANNOT DO, BY CONSTRUCTION. The model may summon a human and may not
|
||||
// act: its evidence is capped below any enforcement action, it can only add to a
|
||||
// transaction's score and never subtract, and where it cannot score, the
|
||||
// transaction is evaluated on rules alone and the refusal is counted rather than
|
||||
// passed off as a clean result. Silence is never evidence of innocence here, and
|
||||
// nothing downstream is allowed to read it as such.
|
||||
//
|
||||
// Detector: Tan, Ting & Liu, "Fast Anomaly Detection for Streaming Data",
|
||||
// IJCAI 2011. Requirements: HIP-0518 section 6.
|
||||
package anomaly
|
||||
|
||||
import (
|
||||
crypto "crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/binary"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"math/rand/v2"
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/aml/pkg/types"
|
||||
"github.com/luxfi/aml/pkg/velocity"
|
||||
)
|
||||
|
||||
// RuleID and RuleName are what the model's evidence carries into an alert.
|
||||
//
|
||||
// The model does not get its own alert type, its own store or its own review
|
||||
// queue. It produces the same alert a rule produces, so case management,
|
||||
// triage, webhooks and the reviewer's screen need to know nothing about how it
|
||||
// was reached — which is also what keeps the promise that a model alert is
|
||||
// reviewed by a person on the same terms as any other.
|
||||
const (
|
||||
RuleID = "anomaly"
|
||||
RuleName = "Unusual for this customer"
|
||||
)
|
||||
|
||||
// Reasons a transaction was not scored. Both are ordinary and neither is a
|
||||
// clean bill of health.
|
||||
const (
|
||||
// ReasonWarming means the model has not yet learned enough to have an
|
||||
// opinion. It is the state after every start and every restore.
|
||||
ReasonWarming = "warming"
|
||||
// ReasonUnusable means the features could not be computed to numbers the
|
||||
// model can accept, so the point was neither scored nor learned from.
|
||||
ReasonUnusable = "unusable"
|
||||
// ReasonUnidentified means the transaction names no account and no user, so
|
||||
// there is no subject whose behaviour this could be unusual for. Scoring it
|
||||
// against an aggregate keyed on nothing would pool every anonymous
|
||||
// transaction in the tenant into one imaginary customer.
|
||||
ReasonUnidentified = "unidentified"
|
||||
)
|
||||
|
||||
// histBuckets is the resolution of the score distribution the alert threshold is
|
||||
// derived from. 256 puts the threshold within 0.4% of the intended quantile,
|
||||
// which is finer than the appetite it serves is ever set.
|
||||
const histBuckets = 256
|
||||
|
||||
// snapshotVersion is the layout of persisted state. State from another version is
|
||||
// rejected rather than reinterpreted.
|
||||
const snapshotVersion = 1
|
||||
|
||||
// Appetite is the risk-appetite decision the model is not permitted to make for
|
||||
// itself.
|
||||
//
|
||||
// A model's output is a probability, so how likely it is to MISS suspicious
|
||||
// activity is a matter of appetite that has to be explained and agreed rather
|
||||
// than absorbed into a constant. This is that appetite, stated where it can be
|
||||
// read, changed and reviewed.
|
||||
//
|
||||
// Review is the lever. It is the share of the stream the model may send for
|
||||
// examination, and the alert threshold is derived from it as a quantile of the
|
||||
// scores actually observed rather than fixed at a number someone liked. That
|
||||
// matters for two reasons. An alert level must be governed rather than tuned, and
|
||||
// must not be set to fit the size of the team or to generate volume nobody reads;
|
||||
// a stated share with a measured realised share is exactly the governed form. And
|
||||
// a fixed threshold on a drifting distribution silently becomes either silence or
|
||||
// a flood, whereas a quantile holds its meaning.
|
||||
//
|
||||
// The miss rate is the consequence, and it cannot be computed from the stream:
|
||||
// there are no labels, so nothing in this package knows what it missed. Sample is
|
||||
// the instrument that measures it. It retains a share of the transactions the
|
||||
// model did NOT alert on for review below the line, which is the standard way the
|
||||
// question is answered and is separately required of any monitoring system —
|
||||
// ex-post review of a sample of all processed transactions, to test and improve
|
||||
// the system. Selection is by hash of the transaction ID, so the sample is
|
||||
// reproducible by anyone holding the same appetite and cannot be steered.
|
||||
//
|
||||
// Warm is how many transactions the model must learn before it may alert at all.
|
||||
// Below it the model has no reference to compare against and its scores mean
|
||||
// nothing, so it says nothing.
|
||||
type Appetite struct {
|
||||
Review float64 `json:"review"`
|
||||
Sample float64 `json:"sample"`
|
||||
Warm int `json:"warm"`
|
||||
}
|
||||
|
||||
// Config parameterises a Store. The zero value is valid and yields the
|
||||
// documented defaults.
|
||||
//
|
||||
// Trees, Depth and Window are the detector's shape. Window is how many
|
||||
// transactions make up one reference window; Blend is how much of a closing
|
||||
// window folds into the reference, where 1 is the published replacement and the
|
||||
// default is lower to make the reference expensive to move. Weight is how much
|
||||
// evidence one alert contributes to a transaction's score, and Action is what the
|
||||
// alert asks for — capped, because a statistical judgement may ask for a person
|
||||
// and may not stand in for one.
|
||||
//
|
||||
// Shadow is how the model is tested before it is trusted. A monitoring system has
|
||||
// to be able to implement and test new detection before live activation; in
|
||||
// shadow the model scores, learns, and records everything it would have alerted
|
||||
// on, and contributes nothing to any transaction's outcome. What it would have
|
||||
// done is then readable in State, over real traffic, before anyone depends on it.
|
||||
type Config struct {
|
||||
Trees int `json:"trees"`
|
||||
Depth int `json:"depth"`
|
||||
Window int `json:"window"`
|
||||
Blend float64 `json:"blend"`
|
||||
Appetite Appetite `json:"appetite"`
|
||||
Weight float64 `json:"weight"`
|
||||
Severity string `json:"severity"`
|
||||
Action string `json:"action"`
|
||||
Shadow bool `json:"shadow"`
|
||||
// MaxOrgs bounds how many tenants' models are held at once. Each costs
|
||||
// Trees * (2^(Depth+1)-1) * 25 bytes — about 320 KB at the defaults — and
|
||||
// models are created on a tenant's first transaction, so idle tenants cost
|
||||
// nothing. On overflow the least recently used is dropped, which returns
|
||||
// that tenant to warming: the model then declines to score rather than
|
||||
// scoring from state it does not have.
|
||||
MaxOrgs int `json:"max_orgs"`
|
||||
// Seed fixes the tree geometry. Left at zero it is drawn from the system
|
||||
// CSPRNG at construction, which is the right default: geometry that cannot
|
||||
// be predicted cannot be probed for a region to hide in. Setting it makes
|
||||
// scores reproducible across restarts at the cost of making the geometry
|
||||
// guessable from the seed, so a fixed seed belongs in KMS and not in a
|
||||
// manifest. Reproducing a past score does not need it in any case — a
|
||||
// Snapshot carries the seed it was built with.
|
||||
Seed uint64 `json:"-"`
|
||||
}
|
||||
|
||||
func (c Config) withDefaults() Config {
|
||||
if c.Trees <= 0 {
|
||||
c.Trees = 25
|
||||
}
|
||||
if c.Depth <= 0 {
|
||||
c.Depth = 8
|
||||
}
|
||||
if c.Window <= 0 {
|
||||
c.Window = 256
|
||||
}
|
||||
if c.Blend <= 0 || c.Blend > 1 {
|
||||
c.Blend = 0.25
|
||||
}
|
||||
if c.Appetite.Review <= 0 || c.Appetite.Review > 0.5 {
|
||||
c.Appetite.Review = 0.01
|
||||
}
|
||||
if c.Appetite.Sample < 0 || c.Appetite.Sample > 1 {
|
||||
c.Appetite.Sample = 0.001
|
||||
}
|
||||
// The reference must have been folded at least once and the distribution
|
||||
// must hold enough scores for its tail to mean anything, so the floor is
|
||||
// expressed in windows rather than as a bare count.
|
||||
if c.Appetite.Warm < 2*c.Window {
|
||||
c.Appetite.Warm = 8 * c.Window
|
||||
}
|
||||
if c.Weight <= 0 {
|
||||
c.Weight = 0.2
|
||||
}
|
||||
if c.Severity == "" {
|
||||
c.Severity = types.SeverityMedium
|
||||
}
|
||||
if c.Action == "" {
|
||||
c.Action = types.ActionReview
|
||||
}
|
||||
if c.MaxOrgs <= 0 {
|
||||
c.MaxOrgs = 256
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
var (
|
||||
// ErrWindows is returned when the aggregate store does not keep the windows
|
||||
// the inventory reads. Failing at construction is the point: the alternative
|
||||
// is a model that runs, alerts, and has been reading zero for every feature.
|
||||
ErrWindows = errors.New("anomaly: aggregate store is missing a window the inventory reads")
|
||||
// ErrAction is returned for a configured action above the ceiling.
|
||||
ErrAction = errors.New("anomaly: action exceeds the ceiling for model evidence")
|
||||
// ErrSnapshot is returned for state that cannot be trusted.
|
||||
ErrSnapshot = errors.New("anomaly: snapshot rejected")
|
||||
)
|
||||
|
||||
// Store holds one model per tenant and scores transactions against it.
|
||||
//
|
||||
// Models are per tenant and never shared. A model learned across tenants would
|
||||
// carry one customer's behaviour into another's score, which is a disclosure of
|
||||
// the first tenant's activity however small — and the tenant boundary here is
|
||||
// physical, so the model must not be the one place it is not. The tree geometry
|
||||
// is per tenant too, so probing one tenant reveals nothing about where another's
|
||||
// regions lie.
|
||||
//
|
||||
// Safe for concurrent use.
|
||||
type Store struct {
|
||||
cfg Config
|
||||
vel *velocity.Store
|
||||
mu sync.RWMutex
|
||||
orgs map[string]*model
|
||||
}
|
||||
|
||||
type model struct {
|
||||
mu sync.Mutex
|
||||
trees []*tree
|
||||
seed uint64
|
||||
seen int // learned into the open window
|
||||
learned int64 // learned ever
|
||||
hist [histBuckets]float64
|
||||
cut float64
|
||||
scored int64
|
||||
alerted int64
|
||||
refused map[string]int64
|
||||
blind [Dims]int64
|
||||
sample []Sampled
|
||||
at int
|
||||
updated time.Time
|
||||
}
|
||||
|
||||
// New builds a Store reading aggregates from vel.
|
||||
//
|
||||
// It fails rather than degrades when vel does not keep a window the inventory
|
||||
// reads, and when the configured action exceeds the ceiling.
|
||||
func New(cfg Config, vel *velocity.Store) (*Store, error) {
|
||||
if vel == nil {
|
||||
return nil, fmt.Errorf("%w: no aggregate store", ErrWindows)
|
||||
}
|
||||
cfg = cfg.withDefaults()
|
||||
if types.ActionRank(cfg.Action) > types.ActionRank(types.ActionCeiling) {
|
||||
return nil, fmt.Errorf("%w: %q above %q", ErrAction, cfg.Action, types.ActionCeiling)
|
||||
}
|
||||
kept := map[string]bool{}
|
||||
for _, w := range vel.Windows() {
|
||||
kept[w.Name] = true
|
||||
}
|
||||
for _, f := range Inventory() {
|
||||
if f.Window != "" && !kept[f.Window] {
|
||||
return nil, fmt.Errorf("%w: feature %q reads %q", ErrWindows, f.Name, f.Window)
|
||||
}
|
||||
}
|
||||
if cfg.Seed == 0 {
|
||||
var b [8]byte
|
||||
if _, err := crypto.Read(b[:]); err != nil {
|
||||
return nil, fmt.Errorf("anomaly: seed: %w", err)
|
||||
}
|
||||
cfg.Seed = binary.LittleEndian.Uint64(b[:])
|
||||
}
|
||||
return &Store{cfg: cfg, vel: vel, orgs: map[string]*model{}}, nil
|
||||
}
|
||||
|
||||
// Config returns the configuration in force, with the seed withheld.
|
||||
func (s *Store) Config() Config {
|
||||
c := s.cfg
|
||||
c.Seed = 0
|
||||
return c
|
||||
}
|
||||
|
||||
// Assess scores one transaction, lets the model learn from it, and returns the
|
||||
// evidence when it alerts. It satisfies the engine's Scorer.
|
||||
//
|
||||
// ok is false whenever the model has nothing to contribute — because it is still
|
||||
// warming, because the features were unusable, because the score is inside the
|
||||
// appetite, or because it is in shadow. The caller evaluates on rules alone in
|
||||
// every one of those cases; none of them is a verdict of normal, and the counts
|
||||
// behind each are in State so that the difference is legible.
|
||||
//
|
||||
// The transaction must already have been recorded to the aggregate store. Reading
|
||||
// aggregates that include it is deliberate: the numbers quoted in the alert are
|
||||
// then the same ones an investigator sees when they look at the account, and
|
||||
// every baseline in the feature set has the transaction removed from it
|
||||
// arithmetically so that nothing is measured against itself.
|
||||
func (s *Store) Assess(tx types.Transaction, entity types.Entity) (types.RuleHit, bool) {
|
||||
a := s.judge(tx, true)
|
||||
if !a.Alert {
|
||||
return types.RuleHit{}, false
|
||||
}
|
||||
return types.RuleHit{
|
||||
Rule: types.Rule{
|
||||
ID: RuleID,
|
||||
OrgID: tx.OrgID,
|
||||
Name: RuleName,
|
||||
Severity: s.cfg.Severity,
|
||||
// Evidence scales with how unusual the transaction is, so a
|
||||
// borderline score contributes less than a plain one through the
|
||||
// same weight-of-evidence sum the rules use. There is no second
|
||||
// scoring path.
|
||||
Weight: s.cfg.Weight * a.Score,
|
||||
Action: s.cfg.Action,
|
||||
Enabled: true,
|
||||
},
|
||||
Match: true,
|
||||
Causes: a.Causes,
|
||||
}, true
|
||||
}
|
||||
|
||||
// Inspect scores one transaction without learning from it, moving any counter, or
|
||||
// touching the aggregate store. It is how a candidate is tried against a
|
||||
// tenant's real behaviour before anything depends on the answer, and it is the
|
||||
// model's analogue of testing a rule.
|
||||
//
|
||||
// Because it records nothing, the aggregates it reads do not include the
|
||||
// candidate: its numbers are the tenant's history as it stands. That is the
|
||||
// question worth asking of a hypothetical, and it is one transaction away from
|
||||
// what the live path would compute for the same input.
|
||||
func (s *Store) Inspect(tx types.Transaction, entity types.Entity) Assessment {
|
||||
return s.judge(tx, false)
|
||||
}
|
||||
|
||||
// Assessment is the model's full verdict on one transaction.
|
||||
type Assessment struct {
|
||||
// Scored is false when the model declined; Reason says which refusal.
|
||||
Scored bool `json:"scored"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
// Score is the anomaly score in [0,1]: 0 where the tenant's recent
|
||||
// behaviour is densest, 1 where there is none of it.
|
||||
Score float64 `json:"score"`
|
||||
// Cut is the threshold in force, derived from the appetite.
|
||||
Cut float64 `json:"cut"`
|
||||
// Alert is whether this becomes evidence. False in shadow however high the
|
||||
// score, and Shadow says so.
|
||||
Alert bool `json:"alert"`
|
||||
Shadow bool `json:"shadow"`
|
||||
// Causes is the per-feature attribution, ordered by contribution.
|
||||
Causes []types.Cause `json:"causes,omitempty"`
|
||||
// Values is every coordinate, including the ones that contributed nothing,
|
||||
// so a reviewer sees what the model read and not only what it concluded.
|
||||
Values []Value `json:"values,omitempty"`
|
||||
}
|
||||
|
||||
// Value is one coordinate as the model read it.
|
||||
type Value struct {
|
||||
Feature string `json:"feature"`
|
||||
X float64 `json:"x"`
|
||||
Observed float64 `json:"observed"`
|
||||
Baseline float64 `json:"baseline"`
|
||||
Unit string `json:"unit"`
|
||||
Blind bool `json:"blind"`
|
||||
}
|
||||
|
||||
// Sampled is one transaction retained below the line: scored, not alerted, and
|
||||
// selected for review so that what the model misses can be measured instead of
|
||||
// assumed.
|
||||
type Sampled struct {
|
||||
TxID string `json:"tx_id"`
|
||||
At time.Time `json:"at"`
|
||||
Score float64 `json:"score"`
|
||||
Cut float64 `json:"cut"`
|
||||
Top string `json:"top"`
|
||||
}
|
||||
|
||||
func (s *Store) judge(tx types.Transaction, learn bool) Assessment {
|
||||
inv := Inventory()
|
||||
if account(tx) == "" {
|
||||
m := s.model(tx.OrgID)
|
||||
m.mu.Lock()
|
||||
m.refused[ReasonUnidentified]++
|
||||
m.mu.Unlock()
|
||||
return Assessment{Shadow: s.cfg.Shadow, Reason: ReasonUnidentified}
|
||||
}
|
||||
p := project(tx.USD,
|
||||
s.vel.Observe(velocity.Key{OrgID: tx.OrgID, Kind: AxisAccount, Value: account(tx)}),
|
||||
pairObs(s.vel, tx),
|
||||
deviceObs(s.vel, tx),
|
||||
)
|
||||
|
||||
a := Assessment{Shadow: s.cfg.Shadow, Values: values(p, inv)}
|
||||
m := s.model(tx.OrgID)
|
||||
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
if !p.usable() {
|
||||
m.refused[ReasonUnusable]++
|
||||
a.Reason = ReasonUnusable
|
||||
return a
|
||||
}
|
||||
for i := range p.Blind {
|
||||
if p.Blind[i] && learn {
|
||||
m.blind[i]++
|
||||
}
|
||||
}
|
||||
|
||||
a.Score = m.score(p.X[:], s.cfg)
|
||||
a.Cut = m.cut
|
||||
warm := m.learned >= int64(s.cfg.Appetite.Warm)
|
||||
|
||||
if learn {
|
||||
m.learn(p.X[:], s.cfg)
|
||||
m.updated = time.Now().UTC()
|
||||
}
|
||||
if !warm {
|
||||
if learn {
|
||||
m.refused[ReasonWarming]++
|
||||
}
|
||||
a.Reason = ReasonWarming
|
||||
return a
|
||||
}
|
||||
|
||||
a.Scored = true
|
||||
if learn {
|
||||
m.scored++
|
||||
}
|
||||
|
||||
// Strictly above: a score equal to the cut is inside the appetite. The cut
|
||||
// is the upper edge of the bucket that exhausts the review budget, so the
|
||||
// realised share stays at or below the stated one rather than above it.
|
||||
if a.Score > m.cut {
|
||||
if learn {
|
||||
m.alerted++
|
||||
}
|
||||
a.Causes = m.explain(p, a.Score, s.cfg, inv)
|
||||
a.Alert = !s.cfg.Shadow
|
||||
return a
|
||||
}
|
||||
if learn && belowLine(tx.ID, s.cfg.Appetite.Sample) {
|
||||
m.keep(Sampled{TxID: tx.ID, At: tx.Timestamp, Score: a.Score, Cut: m.cut, Top: inv[p.far(inv)].Name})
|
||||
}
|
||||
return a
|
||||
}
|
||||
|
||||
func pairObs(vel *velocity.Store, tx types.Transaction) []velocity.Observation {
|
||||
if tx.Counterparty == "" || account(tx) == "" {
|
||||
return nil
|
||||
}
|
||||
return vel.Observe(velocity.Key{OrgID: tx.OrgID, Kind: AxisPair, Value: account(tx) + "\x1f" + tx.Counterparty})
|
||||
}
|
||||
|
||||
func deviceObs(vel *velocity.Store, tx types.Transaction) []velocity.Observation {
|
||||
if tx.DeviceFingerprint == "" {
|
||||
return nil
|
||||
}
|
||||
return vel.Observe(velocity.Key{OrgID: tx.OrgID, Kind: AxisDevice, Value: tx.DeviceFingerprint})
|
||||
}
|
||||
|
||||
func values(p Point, inv [Dims]Feature) []Value {
|
||||
out := make([]Value, 0, Dims)
|
||||
for i := range inv {
|
||||
out = append(out, Value{
|
||||
Feature: inv[i].Name, X: p.X[i], Observed: p.Raw[i],
|
||||
Baseline: p.Base[i], Unit: inv[i].Unit, Blind: p.Blind[i],
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// score is the anomaly score for x against the reference window: the share of
|
||||
// the forest that found x in a region emptier than uniform.
|
||||
//
|
||||
// Each tree votes on its own, in [0,1], and the votes are averaged. A tree whose
|
||||
// region holds at least the mass a uniform spread would put there votes zero; one
|
||||
// whose region is empty votes one; in between the vote is how far short the
|
||||
// region falls. The number that comes out says something a reviewer can hold on
|
||||
// to — nineteen of twenty-five independently drawn partitions of this tenant's
|
||||
// behaviour put this transaction where that behaviour does not go.
|
||||
//
|
||||
// The published algorithm sums the raw masses and ranks by the total. That is
|
||||
// sound for ranking and wrong here, and the reason is worth recording because it
|
||||
// is not obvious: mass*2^depth is heavy-tailed, so a single tree that happens to
|
||||
// find the point in a dense region returns a value many times the uniform one and
|
||||
// swamps every tree that found emptiness. Ranking survives it; a calibrated score
|
||||
// does not, and a calibrated score is what an appetite expressed as a quantile
|
||||
// needs. Clamping each tree's contribution before averaging costs nothing and
|
||||
// bounds any one tree's influence to 1/trees.
|
||||
//
|
||||
// Each tree is measured against its OWN root mass rather than the nominal window
|
||||
// size, which makes the score self-calibrating: a forest still filling its
|
||||
// reference is compared to the reference it actually has, so the score does not
|
||||
// carry a warm-up bias that decays over the first few thousand transactions. It
|
||||
// also means the window size governs only how fast the model forgets, and nothing
|
||||
// about what a score means.
|
||||
func (m *model) score(x []float64, cfg Config) float64 {
|
||||
var vote float64
|
||||
var voters int
|
||||
for _, t := range m.trees {
|
||||
scale := t.ref[0]
|
||||
if scale <= 0 {
|
||||
continue // no reference yet: this tree has no opinion
|
||||
}
|
||||
voters++
|
||||
if mass := t.mass(x, cfg.Depth, 0.1*scale); mass < scale {
|
||||
vote += 1 - mass/scale
|
||||
}
|
||||
}
|
||||
if voters == 0 {
|
||||
// Nothing learned. Unknown is not normal, so this reads as maximally
|
||||
// unusual rather than as clean — and the threshold starts above every
|
||||
// possible score, so it still cannot alert.
|
||||
return 1
|
||||
}
|
||||
return vote / float64(voters)
|
||||
}
|
||||
|
||||
func (m *model) learn(x []float64, cfg Config) {
|
||||
for _, t := range m.trees {
|
||||
t.learn(x, cfg.Depth)
|
||||
}
|
||||
m.learned++
|
||||
m.seen++
|
||||
// The distribution is only fed once a reference exists, or it would be a
|
||||
// record of the cold model scoring everything as maximally unusual and the
|
||||
// threshold derived from it would be meaningless.
|
||||
if m.learned > int64(cfg.Window) {
|
||||
m.hist[bucket(m.score(x, cfg))]++
|
||||
}
|
||||
if m.seen >= cfg.Window {
|
||||
for _, t := range m.trees {
|
||||
t.roll(cfg.Blend)
|
||||
}
|
||||
m.seen = 0
|
||||
// The distribution forgets at the same rate as the trees, so the
|
||||
// threshold describes the same stretch of the stream the model does.
|
||||
for i := range m.hist {
|
||||
m.hist[i] *= 1 - cfg.Blend
|
||||
}
|
||||
m.cut = cut(m.hist[:], cfg.Appetite.Review)
|
||||
}
|
||||
}
|
||||
|
||||
func bucket(score float64) int {
|
||||
i := int(score * histBuckets)
|
||||
if i >= histBuckets {
|
||||
i = histBuckets - 1
|
||||
}
|
||||
if i < 0 {
|
||||
i = 0
|
||||
}
|
||||
return i
|
||||
}
|
||||
|
||||
// cut is the score above which alerting stays inside the appetite: the upper edge
|
||||
// of the bucket at which the review budget runs out.
|
||||
//
|
||||
// Taking the upper edge makes the realised share at most the stated one. That is
|
||||
// the direction a governed alert level has to err, and it has a consequence worth
|
||||
// stating plainly: if more of the stream than the appetite allows sits in the top
|
||||
// bucket, no threshold can honour the appetite, and this returns 1 — above every
|
||||
// possible score — so the model alerts on nothing rather than on everything. A
|
||||
// saturated model is a control that has stopped contributing, which is bad; a
|
||||
// model that floods the queue is a control that has stopped being read, which is
|
||||
// worse and is the failure the appetite exists to prevent. State reports the
|
||||
// saturation either way, because the one thing that must not happen is for it to
|
||||
// be invisible.
|
||||
func cut(hist []float64, review float64) float64 {
|
||||
var total float64
|
||||
for _, c := range hist {
|
||||
total += c
|
||||
}
|
||||
if total <= 0 {
|
||||
return 1
|
||||
}
|
||||
budget := review * total
|
||||
var acc float64
|
||||
for i := len(hist) - 1; i >= 0; i-- {
|
||||
acc += hist[i]
|
||||
if acc > budget {
|
||||
return float64(i+1) / float64(len(hist))
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// explain attributes an alert to the features that produced it.
|
||||
//
|
||||
// For each coordinate the score is recomputed with that coordinate moved to its
|
||||
// feature's neutral value and every other coordinate held. The drop is what that
|
||||
// feature contributed. This is a counterfactual on the model that raised the
|
||||
// alert, not a surrogate fitted to imitate it, so the number is exact with
|
||||
// respect to the thing being explained — which is the whole reason the detector
|
||||
// is a tree and not a network, and it is why the answer survives being
|
||||
// challenged. The sentence it produces is the one a supervisor asks for: had this
|
||||
// customer's daily count been their own average rather than eleven times it, the
|
||||
// score would have been 0.31 instead of 0.88.
|
||||
//
|
||||
// Cost falls on alerts only, not on the stream: Dims+1 traversals, at the
|
||||
// appetite's share of transactions.
|
||||
//
|
||||
// A share of zero across every cause is a real answer and not a failure. It means
|
||||
// no single feature accounts for the score and the conjunction does — repairing
|
||||
// any one coordinate alone leaves the point just as isolated. The causes are then
|
||||
// ordered by distance from neutral, which still names the features involved, and
|
||||
// the counterfactual score printed against each shows why none of them is
|
||||
// individually to blame.
|
||||
func (m *model) explain(p Point, score float64, cfg Config, inv [Dims]Feature) []types.Cause {
|
||||
x := p.X // array: mutated by value, the caller's point is untouched
|
||||
var without, drop [Dims]float64
|
||||
var total float64
|
||||
for i := range inv {
|
||||
without[i] = score
|
||||
if x[i] == inv[i].Neutral {
|
||||
continue
|
||||
}
|
||||
keep := x[i]
|
||||
x[i] = inv[i].Neutral
|
||||
without[i] = m.score(x[:], cfg)
|
||||
x[i] = keep
|
||||
if d := score - without[i]; d > 0 {
|
||||
drop[i] = d
|
||||
total += d
|
||||
}
|
||||
}
|
||||
|
||||
causes := make([]types.Cause, 0, Dims)
|
||||
for i := range inv {
|
||||
if x[i] == inv[i].Neutral {
|
||||
continue // contributed nothing: it is already at neutral
|
||||
}
|
||||
c := types.Cause{
|
||||
Feature: inv[i].Name, Typology: inv[i].Typology,
|
||||
Indicator: inv[i].Indicator, Citation: inv[i].Citation,
|
||||
Severity: inv[i].Severity, Unit: inv[i].Unit,
|
||||
Observed: p.Raw[i], Baseline: p.Base[i], Without: without[i],
|
||||
}
|
||||
if total > 0 {
|
||||
c.Share = drop[i] / total
|
||||
}
|
||||
causes = append(causes, c)
|
||||
}
|
||||
if total > 0 {
|
||||
sort.SliceStable(causes, func(a, b int) bool { return causes[a].Share > causes[b].Share })
|
||||
return causes
|
||||
}
|
||||
byDistance(causes, p, inv)
|
||||
return causes
|
||||
}
|
||||
|
||||
func byDistance(causes []types.Cause, p Point, inv [Dims]Feature) {
|
||||
away := map[string]float64{}
|
||||
for i := range inv {
|
||||
d := p.X[i] - inv[i].Neutral
|
||||
if d < 0 {
|
||||
d = -d
|
||||
}
|
||||
away[inv[i].Name] = d
|
||||
}
|
||||
sort.SliceStable(causes, func(a, b int) bool { return away[causes[a].Feature] > away[causes[b].Feature] })
|
||||
}
|
||||
|
||||
// belowLine selects a transaction for review below the line by hash of its
|
||||
// identifier.
|
||||
//
|
||||
// Hashing rather than drawing at random costs nothing and buys two things: the
|
||||
// same transaction is always either in the sample or out of it, so a reviewer
|
||||
// holding the appetite can verify that the sample was not steered; and there is
|
||||
// no shared random state on the scoring path.
|
||||
func belowLine(txID string, rate float64) bool {
|
||||
if rate <= 0 || txID == "" {
|
||||
return false
|
||||
}
|
||||
var h uint32 = 2166136261
|
||||
for i := 0; i < len(txID); i++ {
|
||||
h ^= uint32(txID[i])
|
||||
h *= 16777619
|
||||
}
|
||||
const span = 1 << 20
|
||||
return float64(h%span) < rate*span
|
||||
}
|
||||
|
||||
func (m *model) keep(s Sampled) {
|
||||
if len(m.sample) < cap(m.sample) {
|
||||
m.sample = append(m.sample, s)
|
||||
return
|
||||
}
|
||||
if cap(m.sample) == 0 {
|
||||
return
|
||||
}
|
||||
m.sample[m.at] = s
|
||||
m.at = (m.at + 1) % cap(m.sample)
|
||||
}
|
||||
|
||||
// sampleDepth is how many below-the-line transactions are retained per tenant.
|
||||
// It is a window onto the sample, not the sample itself: durable retention for a
|
||||
// review cycle is the caller's, since only the caller knows the retention period
|
||||
// it owes.
|
||||
const sampleDepth = 256
|
||||
|
||||
func (s *Store) model(orgID string) *model {
|
||||
s.mu.RLock()
|
||||
m := s.orgs[orgID]
|
||||
s.mu.RUnlock()
|
||||
if m != nil {
|
||||
return m
|
||||
}
|
||||
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if m = s.orgs[orgID]; m != nil {
|
||||
return m
|
||||
}
|
||||
if len(s.orgs) >= s.cfg.MaxOrgs {
|
||||
s.evict()
|
||||
}
|
||||
m = s.plant(orgID, mix(s.cfg.Seed, orgID))
|
||||
s.orgs[orgID] = m
|
||||
return m
|
||||
}
|
||||
|
||||
func (s *Store) plant(orgID string, seed uint64) *model {
|
||||
rng := rand.New(rand.NewPCG(seed, seed^0x9e3779b97f4a7c15))
|
||||
m := &model{
|
||||
trees: make([]*tree, s.cfg.Trees),
|
||||
seed: seed,
|
||||
refused: map[string]int64{ReasonWarming: 0, ReasonUnusable: 0, ReasonUnidentified: 0},
|
||||
sample: make([]Sampled, 0, sampleDepth),
|
||||
cut: 1, // admit nothing until a distribution exists
|
||||
updated: time.Now().UTC(),
|
||||
}
|
||||
for i := range m.trees {
|
||||
m.trees[i] = plant(rng, Dims, s.cfg.Depth)
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// evict drops the least recently used tenant's model. Caller holds s.mu.
|
||||
func (s *Store) evict() {
|
||||
var oldest string
|
||||
var at time.Time
|
||||
first := true
|
||||
for id, m := range s.orgs {
|
||||
m.mu.Lock()
|
||||
u := m.updated
|
||||
m.mu.Unlock()
|
||||
if first || u.Before(at) {
|
||||
oldest, at, first = id, u, false
|
||||
}
|
||||
}
|
||||
if oldest != "" {
|
||||
delete(s.orgs, oldest)
|
||||
}
|
||||
}
|
||||
|
||||
// mix derives a tenant's seed from the store's seed and the tenant's identity, so
|
||||
// that no two tenants share a geometry and neither can be derived from the other
|
||||
// without the store's seed. SplitMix64 finalisation over an FNV-1a of the id.
|
||||
func mix(seed uint64, orgID string) uint64 {
|
||||
var h uint64 = 14695981039346656037
|
||||
for i := 0; i < len(orgID); i++ {
|
||||
h ^= uint64(orgID[i])
|
||||
h *= 1099511628211
|
||||
}
|
||||
z := seed ^ h
|
||||
z += 0x9e3779b97f4a7c15
|
||||
z = (z ^ (z >> 30)) * 0xbf58476d1ce4e5b9
|
||||
z = (z ^ (z >> 27)) * 0x94d049bb133111eb
|
||||
return z ^ (z >> 31)
|
||||
}
|
||||
|
||||
// State is what a review of the model reads. It covers one tenant only: a caller
|
||||
// scoped to a tenant cannot learn another's volumes, alert rate or behaviour from
|
||||
// it.
|
||||
type State struct {
|
||||
Config Config `json:"config"`
|
||||
Inventory [Dims]Feature `json:"inventory"`
|
||||
Digest string `json:"digest"`
|
||||
Org string `json:"org"`
|
||||
Learned int64 `json:"learned"`
|
||||
Warm bool `json:"warm"`
|
||||
Cut float64 `json:"cut"`
|
||||
Scored int64 `json:"scored"`
|
||||
Alerted int64 `json:"alerted"`
|
||||
// Realised is the share of scored transactions that alerted, against
|
||||
// Config.Appetite.Review which is the share intended. The two being
|
||||
// readable side by side is what makes the appetite a measured commitment
|
||||
// rather than a stated one.
|
||||
Realised float64 `json:"realised"`
|
||||
// Saturated means the appetite cannot be honoured by any threshold because
|
||||
// too much of the stream scores in the top bucket, so the model is alerting
|
||||
// on nothing. It is the one state that must never be mistaken for quiet.
|
||||
Saturated bool `json:"saturated"`
|
||||
// Refused counts transactions the model would not score, by reason. None of
|
||||
// these was examined by the model; all of them were examined by the rules.
|
||||
Refused map[string]int64 `json:"refused"`
|
||||
// Blind counts, per feature, how often it took its neutral value for want of
|
||||
// data. A feature blind on most traffic is not contributing whatever the
|
||||
// inventory claims for it.
|
||||
Blind map[string]int64 `json:"blind"`
|
||||
// Distribution is the score distribution the threshold is cut from, in 32
|
||||
// bands of 1/32.
|
||||
Distribution []float64 `json:"distribution"`
|
||||
// Sample is the below-the-line window: transactions scored, not alerted, and
|
||||
// retained for review so the miss rate can be measured.
|
||||
Sample []Sampled `json:"sample"`
|
||||
}
|
||||
|
||||
// State reports the model for one tenant.
|
||||
func (s *Store) State(orgID string) State {
|
||||
inv := Inventory()
|
||||
st := State{
|
||||
Config: s.Config(), Inventory: inv, Digest: s.Digest(), Org: orgID,
|
||||
Cut: 1, Refused: map[string]int64{}, Blind: map[string]int64{},
|
||||
Distribution: make([]float64, 32),
|
||||
}
|
||||
|
||||
s.mu.RLock()
|
||||
m := s.orgs[orgID]
|
||||
s.mu.RUnlock()
|
||||
if m == nil {
|
||||
return st
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
st.Learned, st.Cut, st.Scored, st.Alerted = m.learned, m.cut, m.scored, m.alerted
|
||||
st.Warm = m.learned >= int64(s.cfg.Appetite.Warm)
|
||||
st.Saturated = st.Warm && m.cut >= 1
|
||||
if m.scored > 0 {
|
||||
st.Realised = float64(m.alerted) / float64(m.scored)
|
||||
}
|
||||
for k, v := range m.refused {
|
||||
st.Refused[k] = v
|
||||
}
|
||||
for i := range inv {
|
||||
st.Blind[inv[i].Name] = m.blind[i]
|
||||
}
|
||||
band := histBuckets / len(st.Distribution)
|
||||
for i, c := range m.hist {
|
||||
st.Distribution[i/band] += c
|
||||
}
|
||||
st.Sample = append([]Sampled(nil), m.sample...)
|
||||
return st
|
||||
}
|
||||
|
||||
// Digest identifies the model's shape: the inventory in order and the detector's
|
||||
// geometry parameters. Learned state is only meaningful against the shape that
|
||||
// produced it, so the digest is what Restore checks.
|
||||
func (s *Store) Digest() string {
|
||||
h := sha256.New()
|
||||
fmt.Fprintf(h, "v%d|%d|%d|%d|%g|", snapshotVersion, Dims, s.cfg.Trees, s.cfg.Depth, s.cfg.Blend)
|
||||
fmt.Fprintf(h, "%d|", s.cfg.Window)
|
||||
for _, f := range Inventory() {
|
||||
fmt.Fprintf(h, "%s:%s:%g|", f.Name, f.Window, f.Neutral)
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil))
|
||||
}
|
||||
|
||||
// Snapshot is one tenant's learned state, in a form that can be stored and
|
||||
// restored.
|
||||
//
|
||||
// It carries the masses and the seed, never the geometry: geometry is a pure
|
||||
// function of the seed, so it is regenerated on restore and cannot be supplied.
|
||||
// That removes the sharpest edge a persisted model has — state that describes
|
||||
// where the regions are, rather than only how full they are.
|
||||
//
|
||||
// Restoring is what keeps a restart from being a blind window. The model fails
|
||||
// closed while warming, so a process that comes back with nothing learned
|
||||
// declines to score for the whole warm period, and a control that is off for
|
||||
// however long that takes is a control that was off.
|
||||
type Snapshot struct {
|
||||
Version int `json:"version"`
|
||||
Digest string `json:"digest"`
|
||||
OrgID string `json:"org_id"`
|
||||
Seed uint64 `json:"seed"`
|
||||
Learned int64 `json:"learned"`
|
||||
Seen int `json:"seen"`
|
||||
Cut float64 `json:"cut"`
|
||||
Ref [][]float64 `json:"ref"`
|
||||
Cur [][]float64 `json:"cur"`
|
||||
Hist []float64 `json:"hist"`
|
||||
}
|
||||
|
||||
// Snapshot returns one tenant's learned state, or false when that tenant has
|
||||
// none.
|
||||
func (s *Store) Snapshot(orgID string) (Snapshot, bool) {
|
||||
s.mu.RLock()
|
||||
m := s.orgs[orgID]
|
||||
s.mu.RUnlock()
|
||||
if m == nil {
|
||||
return Snapshot{}, false
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
snap := Snapshot{
|
||||
Version: snapshotVersion, Digest: s.Digest(), OrgID: orgID, Seed: m.seed,
|
||||
Learned: m.learned, Seen: m.seen, Cut: m.cut,
|
||||
Ref: make([][]float64, len(m.trees)),
|
||||
Cur: make([][]float64, len(m.trees)),
|
||||
Hist: append([]float64(nil), m.hist[:]...),
|
||||
}
|
||||
for i, t := range m.trees {
|
||||
snap.Ref[i] = append([]float64(nil), t.ref...)
|
||||
snap.Cur[i] = append([]float64(nil), t.cur...)
|
||||
}
|
||||
return snap, true
|
||||
}
|
||||
|
||||
// Restore installs learned state for one tenant, replacing whatever that tenant
|
||||
// had.
|
||||
//
|
||||
// Every check here exists because a snapshot is state the model will treat as its
|
||||
// own memory of the tenant's behaviour. State that is merely stale makes the
|
||||
// model wrong; state that has been chosen makes it wrong in a direction someone
|
||||
// picked — a region can be filled until activity inside it reads as ordinary. So
|
||||
// the shape must match, the tenant must match, and the masses must satisfy the
|
||||
// invariant the algorithm cannot avoid producing: a region's mass is the sum of
|
||||
// its two halves, because every point increments both a node and its parent, and
|
||||
// folding a window is linear in the masses. An array that fails it was not
|
||||
// produced by this algorithm.
|
||||
//
|
||||
// These are guards, not authentication. Anyone able to substitute a snapshot that
|
||||
// satisfies the invariant can still shift the model, so the caller owes the
|
||||
// snapshot integrity — it belongs where the tenant's own data belongs, and sealed
|
||||
// if it travels.
|
||||
func (s *Store) Restore(snap Snapshot) error {
|
||||
switch {
|
||||
case snap.Version != snapshotVersion:
|
||||
return fmt.Errorf("%w: version %d, want %d", ErrSnapshot, snap.Version, snapshotVersion)
|
||||
case snap.OrgID == "":
|
||||
return fmt.Errorf("%w: no tenant", ErrSnapshot)
|
||||
case snap.Digest != s.Digest():
|
||||
return fmt.Errorf("%w: shape does not match the running inventory", ErrSnapshot)
|
||||
case len(snap.Ref) != s.cfg.Trees || len(snap.Cur) != s.cfg.Trees:
|
||||
return fmt.Errorf("%w: %d trees, want %d", ErrSnapshot, len(snap.Ref), s.cfg.Trees)
|
||||
case len(snap.Hist) != histBuckets:
|
||||
return fmt.Errorf("%w: distribution has %d buckets, want %d", ErrSnapshot, len(snap.Hist), histBuckets)
|
||||
case snap.Learned < 0 || snap.Seen < 0 || snap.Seen > s.cfg.Window:
|
||||
return fmt.Errorf("%w: window position %d of %d after %d", ErrSnapshot, snap.Seen, s.cfg.Window, snap.Learned)
|
||||
case !finite(snap.Cut) || snap.Cut < 0 || snap.Cut > 1:
|
||||
return fmt.Errorf("%w: threshold %v outside [0,1]", ErrSnapshot, snap.Cut)
|
||||
}
|
||||
for _, c := range snap.Hist {
|
||||
if !finite(c) || c < 0 {
|
||||
return fmt.Errorf("%w: distribution holds %v", ErrSnapshot, c)
|
||||
}
|
||||
}
|
||||
|
||||
m := s.plant(snap.OrgID, snap.Seed)
|
||||
nodes := 1<<(s.cfg.Depth+1) - 1
|
||||
for i, t := range m.trees {
|
||||
if len(snap.Ref[i]) != nodes || len(snap.Cur[i]) != nodes {
|
||||
return fmt.Errorf("%w: tree %d has %d nodes, want %d", ErrSnapshot, i, len(snap.Ref[i]), nodes)
|
||||
}
|
||||
copy(t.ref, snap.Ref[i])
|
||||
copy(t.cur, snap.Cur[i])
|
||||
if !t.sound(s.cfg.Depth) {
|
||||
return fmt.Errorf("%w: tree %d does not satisfy the mass invariant", ErrSnapshot, i)
|
||||
}
|
||||
}
|
||||
m.learned, m.seen, m.cut = snap.Learned, snap.Seen, snap.Cut
|
||||
copy(m.hist[:], snap.Hist)
|
||||
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if _, held := s.orgs[snap.OrgID]; !held && len(s.orgs) >= s.cfg.MaxOrgs {
|
||||
s.evict()
|
||||
}
|
||||
s.orgs[snap.OrgID] = m
|
||||
return nil
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,373 @@
|
||||
package anomaly
|
||||
|
||||
import (
|
||||
"math"
|
||||
|
||||
"github.com/luxfi/aml/pkg/types"
|
||||
"github.com/luxfi/aml/pkg/velocity"
|
||||
)
|
||||
|
||||
// Dims is the number of dimensions in the model space, one per Feature. It is a
|
||||
// constant so a point is an array rather than a slice and scoring allocates
|
||||
// nothing.
|
||||
const Dims = 9
|
||||
|
||||
// Feature names one dimension of the model space and records the obligation it
|
||||
// serves.
|
||||
//
|
||||
// Where a rule maps a risk indicator to a typology, a model maps a typology to
|
||||
// the data features that express it, so the evidence of risk coverage is a
|
||||
// maintained mapping from typology to feature. Inventory is that mapping, and it
|
||||
// is code rather than a document because a mapping kept beside the model cannot
|
||||
// drift away from what the model actually reads.
|
||||
//
|
||||
// Every field is here because a reviewer asks for it. Typology is the pattern.
|
||||
// Indicator is the supervisor's own words for the thing being looked for.
|
||||
// Citation is where those words come from, so the claim is checkable rather than
|
||||
// asserted. Unit is how to read the raw number, which is what turns a coordinate
|
||||
// into a sentence an investigator can put in a case file.
|
||||
type Feature struct {
|
||||
Name string `json:"name"`
|
||||
Window string `json:"window,omitempty"`
|
||||
Typology string `json:"typology"`
|
||||
Indicator string `json:"indicator"`
|
||||
Citation string `json:"citation"`
|
||||
Severity string `json:"severity"`
|
||||
Unit string `json:"unit"`
|
||||
Neutral float64 `json:"neutral"`
|
||||
}
|
||||
|
||||
// Inventory is the typology-to-feature mapping the model is built on. The order
|
||||
// is the coordinate order of the model space and is therefore part of the
|
||||
// model's identity: adding, removing or reordering an entry invalidates learned
|
||||
// state, which Digest enforces.
|
||||
//
|
||||
// Two properties are deliberate throughout. Every feature is dimensionless, so
|
||||
// no coordinate carries a currency or a count that would have to be rescaled per
|
||||
// tenant. And every ratio is measured against the entity's own history rather
|
||||
// than a fixed limit, because the statutory test is consistency with what is
|
||||
// known about THIS customer, not comparison to a global number. The density
|
||||
// model over those ratios is what supplies the second axis a supervisor asks
|
||||
// about — how unusual a given deviation is among the institution's customers —
|
||||
// so intra-customer deviation and peer-group comparison come out of one
|
||||
// mechanism rather than two.
|
||||
func Inventory() [Dims]Feature {
|
||||
return [Dims]Feature{{
|
||||
Name: "amount",
|
||||
Window: "30d",
|
||||
Typology: "unusually large transaction",
|
||||
Indicator: "a transaction unusually large relative to the customer's own pattern",
|
||||
Citation: "Directive (EU) 2018/843, Art. 1(10)(b) replacing Art. 18(2) AMLD4; EBA/GL/2021/02, Guideline 4.60(a)",
|
||||
Severity: types.SeverityHigh,
|
||||
Unit: "multiples of the customer's own mean transaction value",
|
||||
Neutral: 0.5,
|
||||
}, {
|
||||
Name: "count",
|
||||
Window: "24h",
|
||||
Typology: "unusual pattern of activity",
|
||||
Indicator: "deviation in the frequency of transactions",
|
||||
Citation: "Directive (EU) 2018/843, Art. 1(10)(b) replacing Art. 18(2) AMLD4; EBA/GL/2021/02, Guideline 4.60(a)",
|
||||
Severity: types.SeverityMedium,
|
||||
Unit: "multiples of the customer's own transactions per active day",
|
||||
Neutral: 0.5,
|
||||
}, {
|
||||
Name: "volume",
|
||||
Window: "24h",
|
||||
Typology: "unusually large aggregate",
|
||||
Indicator: "deviation in amount taken over a window rather than per transaction",
|
||||
Citation: "Directive (EU) 2018/843, Art. 1(10)(b) replacing Art. 18(2) AMLD4; Regulation (EU) 2024/1624, Art. 69(2)",
|
||||
Severity: types.SeverityHigh,
|
||||
Unit: "multiples of the customer's own value moved per active day",
|
||||
Neutral: 0.5,
|
||||
}, {
|
||||
Name: "burst",
|
||||
Window: "1h",
|
||||
Typology: "unusual pattern of activity",
|
||||
Indicator: "successive transactions with no obvious economic rationale",
|
||||
Citation: "EBA/GL/2021/02, Guideline 4.60(a)",
|
||||
Severity: types.SeverityMedium,
|
||||
Unit: "share of the day's transactions falling in one hour",
|
||||
Neutral: 0,
|
||||
}, {
|
||||
Name: "subthreshold",
|
||||
Window: "7d",
|
||||
Typology: "structuring",
|
||||
Indicator: "transactions split to circumvent reporting limits",
|
||||
Citation: "EBA/GL/2021/02, Guideline 4.60(a); Regulation (EU) 2024/1624, Art. 19(9)",
|
||||
Severity: types.SeverityHigh,
|
||||
Unit: "share of transactions falling just below the reporting threshold",
|
||||
Neutral: 0,
|
||||
}, {
|
||||
Name: "subvalue",
|
||||
Window: "7d",
|
||||
Typology: "structuring",
|
||||
Indicator: "the link between several transactions kept below a reporting limit",
|
||||
Citation: "Regulation (EU) 2024/1624, Art. 69(2); EBA/GL/2021/02, Guideline 4.60(a)",
|
||||
Severity: types.SeverityHigh,
|
||||
Unit: "share of value moved just below the reporting threshold",
|
||||
Neutral: 0,
|
||||
}, {
|
||||
Name: "counterparty",
|
||||
Window: "30d",
|
||||
Typology: "unfamiliar counterparty",
|
||||
Indicator: "the origin and destination of the funds",
|
||||
Citation: "Regulation (EU) 2024/1624, Art. 34(2); Regulation (EU) 2024/1624, Art. 69(2)",
|
||||
Severity: types.SeverityMedium,
|
||||
Unit: "prior transactions between this customer and this counterparty",
|
||||
Neutral: 0,
|
||||
}, {
|
||||
Name: "device",
|
||||
Window: "24h",
|
||||
Typology: "network of connected persons",
|
||||
Indicator: "activity across a network of connected persons rather than one customer",
|
||||
Citation: "JMLSG Guidance Part I, para 5.7.5",
|
||||
Severity: types.SeverityMedium,
|
||||
Unit: "ratio of the device's transactions to this account's",
|
||||
Neutral: 0.5,
|
||||
}, {
|
||||
Name: "novelty",
|
||||
Window: "30d",
|
||||
Typology: "no established pattern",
|
||||
Indicator: "activity with no prior pattern for it to be consistent with",
|
||||
Citation: "MLR 2017 (SI 2017/692), reg. 28(11)(a)-(b); Regulation (EU) 2024/1624, Art. 26(1)",
|
||||
Severity: types.SeverityLow,
|
||||
Unit: "prior transactions by this account in the window",
|
||||
Neutral: 0,
|
||||
}}
|
||||
}
|
||||
|
||||
// Aggregation axes. A deviation is meaningless without saying whose: the axis is
|
||||
// what the sliding aggregates are grouped by. The pair axis is the customer and
|
||||
// one counterparty together, because "unfamiliar" is a fact about the
|
||||
// relationship and not about either party alone. The device axis is what
|
||||
// surfaces several nominally unrelated customers acting as one.
|
||||
const (
|
||||
AxisAccount = "account"
|
||||
AxisPair = "pair"
|
||||
AxisDevice = "device"
|
||||
)
|
||||
|
||||
// Keys returns the aggregate keys a transaction contributes to, and is the one
|
||||
// definition of those keys: the ingest path records to exactly these and the
|
||||
// model reads exactly these, so a key can never be written on one axis and read
|
||||
// on another. Axes whose identifier the transaction does not carry are omitted
|
||||
// rather than keyed on the empty string, which would pool every anonymous
|
||||
// transaction in the tenant into one aggregate.
|
||||
func Keys(tx types.Transaction) []velocity.Key {
|
||||
keys := make([]velocity.Key, 0, 3)
|
||||
if v := account(tx); v != "" {
|
||||
keys = append(keys, velocity.Key{OrgID: tx.OrgID, Kind: AxisAccount, Value: v})
|
||||
}
|
||||
if tx.Counterparty != "" && account(tx) != "" {
|
||||
keys = append(keys, velocity.Key{OrgID: tx.OrgID, Kind: AxisPair, Value: account(tx) + "\x1f" + tx.Counterparty})
|
||||
}
|
||||
if tx.DeviceFingerprint != "" {
|
||||
keys = append(keys, velocity.Key{OrgID: tx.OrgID, Kind: AxisDevice, Value: tx.DeviceFingerprint})
|
||||
}
|
||||
return keys
|
||||
}
|
||||
|
||||
// account is the identifier the customer's own aggregates are kept under,
|
||||
// falling back to the user when no account is carried. Monitoring is required at
|
||||
// several levels of aggregation and this is the lowest one the transaction
|
||||
// always identifies.
|
||||
func account(tx types.Transaction) string {
|
||||
if tx.AccountID != "" {
|
||||
return tx.AccountID
|
||||
}
|
||||
return tx.UserID
|
||||
}
|
||||
|
||||
// Point is one transaction expressed in the model's space, carrying alongside
|
||||
// each coordinate the number it was computed from and the baseline it was
|
||||
// measured against. Those two are not diagnostics — they are what the alert
|
||||
// quotes, so the explanation is the same arithmetic the score came from rather
|
||||
// than a second story told about it.
|
||||
type Point struct {
|
||||
X [Dims]float64
|
||||
Raw [Dims]float64
|
||||
Base [Dims]float64
|
||||
// Blind marks a coordinate that could not be computed from the data
|
||||
// available and took its neutral value instead. A model silently reading
|
||||
// neutral for a dimension it never has data for is indistinguishable from a
|
||||
// model reading a genuine absence of risk, so the difference is counted and
|
||||
// reported.
|
||||
Blind [Dims]bool
|
||||
}
|
||||
|
||||
// project maps one transaction's aggregates into the model space.
|
||||
//
|
||||
// It is a pure function of the aggregates: the caller owns the store and the
|
||||
// ordering, so every feature here is an ordinary testable expression over
|
||||
// numbers. usd is the transaction's value normalised at ingest, which is also
|
||||
// what the aggregates hold, so a ratio between them carries no exchange-rate
|
||||
// assumption.
|
||||
func project(usd float64, acct, pair, device []velocity.Observation) Point {
|
||||
var p Point
|
||||
inv := Inventory()
|
||||
|
||||
day := pick(acct, "24h")
|
||||
hour := pick(acct, "1h")
|
||||
week := pick(acct, "7d")
|
||||
month := pick(acct, "30d")
|
||||
|
||||
// amount: this transaction against what this customer moves in an average
|
||||
// transaction, on the days it is active. Dividing instead by thirty calendar
|
||||
// days would manufacture a deviation every time an occasional customer
|
||||
// transacts at all.
|
||||
perTx := over(month.Sum-usd, month.Count-1)
|
||||
p.set(0, ratio(usd, perTx), usd, perTx, perTx <= 0, inv)
|
||||
|
||||
// count: today's transactions against this customer's own rate per active
|
||||
// day. The transaction being scored is removed from the baseline so it
|
||||
// cannot raise the bar it is measured against.
|
||||
rate := over(float64(month.Count-1), month.Days)
|
||||
p.set(1, ratio(float64(day.Count), rate), float64(day.Count), rate, rate <= 0, inv)
|
||||
|
||||
// volume: today's value against this customer's own value per active day.
|
||||
// Separate from count because the same aggregate reached by more payments
|
||||
// and by larger ones are different typologies.
|
||||
flow := over(month.Sum-usd, month.Days)
|
||||
p.set(2, ratio(day.Sum, flow), day.Sum, flow, flow <= 0, inv)
|
||||
|
||||
// burst: concentration of the day's activity into this hour. Undefined below
|
||||
// two transactions in the day — one transaction is not a burst, and reading
|
||||
// it as complete concentration would flag every quiet customer.
|
||||
burst := 0.0
|
||||
blind := day.Count < 2
|
||||
if !blind {
|
||||
burst = share(float64(hour.Count), float64(day.Count))
|
||||
}
|
||||
p.set(3, burst, float64(hour.Count), float64(day.Count), blind, inv)
|
||||
|
||||
// subthreshold and subvalue: the structuring pair. The first is how much of
|
||||
// the activity sits just under the reporting limit, the second how much of
|
||||
// the money does. Both are needed because one payment under the limit among
|
||||
// fifty is not the pattern, and fifty of them is nothing else.
|
||||
p.set(4, share(float64(week.Near), float64(week.Count)), float64(week.Near), float64(week.Count), week.Count == 0, inv)
|
||||
p.set(5, share(week.NearSum, week.Sum), week.NearSum, week.Sum, week.Sum <= 0, inv)
|
||||
|
||||
// counterparty: how unfamiliar this counterparty is to this customer. Prior
|
||||
// dealings decay the signal hyperbolically — the first repeat matters far
|
||||
// more than the fortieth.
|
||||
prior := float64(pick(pair, "30d").Count - 1)
|
||||
if prior < 0 {
|
||||
prior = 0
|
||||
}
|
||||
p.set(6, 1/(1+prior), prior, 0, len(pair) == 0, inv)
|
||||
|
||||
// device: whether the device is transacting for more accounts than this one,
|
||||
// or this account is spread across more devices than usual. Both directions
|
||||
// matter, so the ratio is two-sided around its neutral value.
|
||||
dev := pick(device, "24h")
|
||||
p.set(7, ratio(float64(dev.Count), float64(day.Count)), float64(dev.Count), float64(day.Count), len(device) == 0 || day.Count == 0, inv)
|
||||
|
||||
// novelty: how little history this account has. It is what lets the model
|
||||
// join newness to size, which is a different finding from either alone.
|
||||
seen := float64(month.Count - 1)
|
||||
if seen < 0 {
|
||||
seen = 0
|
||||
}
|
||||
p.set(8, 1/(1+seen), seen, 0, false, inv)
|
||||
|
||||
return p
|
||||
}
|
||||
|
||||
// set writes one coordinate, substituting the feature's neutral value where the
|
||||
// data could not support it.
|
||||
//
|
||||
// It substitutes for a KNOWN absence — no baseline yet, too few transactions to
|
||||
// speak of a share — and marks the coordinate blind. It does not repair a
|
||||
// coordinate that arrived as something other than a number in [0,1]: that would
|
||||
// be an arithmetic path nobody anticipated, and quietly rewriting it to neutral
|
||||
// would score a broken point as ordinary. Point.usable is the single place that
|
||||
// decides what the trees will accept, and an unusable point is refused there.
|
||||
func (p *Point) set(i int, x, raw, base float64, blind bool, inv [Dims]Feature) {
|
||||
if blind {
|
||||
x = inv[i].Neutral
|
||||
}
|
||||
p.X[i], p.Raw[i], p.Base[i], p.Blind[i] = x, raw, base, blind
|
||||
}
|
||||
|
||||
// pick returns the observation for a named window, or the zero observation when
|
||||
// the store does not keep it.
|
||||
func pick(obs []velocity.Observation, window string) velocity.Observation {
|
||||
for _, o := range obs {
|
||||
if o.Window == window {
|
||||
return o
|
||||
}
|
||||
}
|
||||
return velocity.Observation{}
|
||||
}
|
||||
|
||||
// over is a mean that reports no baseline rather than an infinite one when there
|
||||
// is nothing to average over.
|
||||
func over(total float64, n int) float64 {
|
||||
if n <= 0 || total <= 0 {
|
||||
return 0
|
||||
}
|
||||
return total / float64(n)
|
||||
}
|
||||
|
||||
// ratio maps a value measured against a baseline onto [0,1], putting the
|
||||
// baseline itself exactly at the centre.
|
||||
//
|
||||
// It is r/(1+r) applied to the value's multiple of its baseline, so the model
|
||||
// space reads symmetrically in both directions: the customer's own average sits
|
||||
// at 0.5, nine times it at 0.9, a ninth of it at 0.1. Normality at the centre is
|
||||
// what makes a coordinate legible on its own and gives the counterfactual in
|
||||
// Store.explain a value to substitute that means something.
|
||||
//
|
||||
// The multiple comes from velocity.Deviation, which answers zero when there is
|
||||
// no baseline to compare against. A first transaction therefore reads as exactly
|
||||
// average rather than as infinitely unusual: a new customer is not suspicious
|
||||
// for being new.
|
||||
func ratio(value, baseline float64) float64 {
|
||||
r := 1 + velocity.Deviation(value, baseline)
|
||||
switch {
|
||||
case math.IsNaN(r), r <= 0:
|
||||
return 0
|
||||
case math.IsInf(r, 1):
|
||||
return 1
|
||||
}
|
||||
return r / (1 + r)
|
||||
}
|
||||
|
||||
// share is a fraction of a whole, clamped and safe on an empty whole.
|
||||
func share(part, whole float64) float64 {
|
||||
if !(whole > 0) || !(part > 0) {
|
||||
return 0
|
||||
}
|
||||
if part >= whole {
|
||||
return 1
|
||||
}
|
||||
return part / whole
|
||||
}
|
||||
|
||||
// usable reports whether a point can be fed to the trees.
|
||||
//
|
||||
// A coordinate outside the unit cube or not a number would corrupt every
|
||||
// subsequent score, because the masses it lands on are shared with every other
|
||||
// point: one poisoned update is not a bad answer for one transaction, it is a
|
||||
// quietly wrong model from then on. So an unusable point is neither scored nor
|
||||
// learned, and the refusal is counted.
|
||||
func (p Point) usable() bool {
|
||||
for _, v := range p.X {
|
||||
if !finite(v) || v < 0 || v > 1 {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// far is the coordinate furthest from its neutral value, which is the feature to
|
||||
// name when a point has to be described in one word.
|
||||
func (p Point) far(inv [Dims]Feature) int {
|
||||
best, at := -1.0, 0
|
||||
for i := range p.X {
|
||||
if d := math.Abs(p.X[i] - inv[i].Neutral); d > best {
|
||||
best, at = d, i
|
||||
}
|
||||
}
|
||||
return at
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
package anomaly
|
||||
|
||||
import (
|
||||
"math"
|
||||
"math/rand/v2"
|
||||
)
|
||||
|
||||
// A half-space tree is a binary tree over the unit cube built BEFORE any data
|
||||
// arrives: at each node a dimension is drawn at random and split at the midpoint
|
||||
// of that node's range. Nothing about the split depends on the stream, which is
|
||||
// the property the whole design rests on — there is no training pass, no sample
|
||||
// to retain, and no retraining job, so the model is a set of counters over a
|
||||
// fixed geometry.
|
||||
//
|
||||
// Each node holds two masses: ref, how many points fell in this region during
|
||||
// the reference window, and cur, how many are falling in it now. A point is
|
||||
// scored against ref and learned into cur. When a window closes, cur folds into
|
||||
// ref and a new window starts.
|
||||
//
|
||||
// Scoring descends the point's path and returns the mass of the deepest region
|
||||
// reached, scaled by 2^depth. The scaling is what makes the number comparable
|
||||
// across depths: halving a region halves its expected mass, so mass*2^depth is
|
||||
// flat at uniform density and falls toward zero as the region empties. A point
|
||||
// in a region emptier than uniform is the anomaly.
|
||||
//
|
||||
// Tan, Ting & Liu, "Fast Anomaly Detection for Streaming Data", IJCAI 2011.
|
||||
|
||||
// tree is one half-space tree, flattened into arrays indexed by heap position:
|
||||
// node i has children 2i+1 and 2i+2. The flat layout costs no pointers, no
|
||||
// allocation per node, and keeps a path walk inside a handful of cache lines.
|
||||
type tree struct {
|
||||
// dim and split describe the geometry and never change after planting.
|
||||
dim []uint8
|
||||
split []float64
|
||||
// ref is the mass of the reference window, which is what points are scored
|
||||
// against. cur is the mass accumulating in the window now open.
|
||||
ref []float64
|
||||
cur []float64
|
||||
}
|
||||
|
||||
// plant grows a tree of the given depth over dims dimensions, drawing the
|
||||
// geometry from rng and nothing else.
|
||||
//
|
||||
// The work range per dimension follows the paper: a random offset s in (0,1)
|
||||
// with half-width 2*max(s, 1-s). That places the unit interval strictly inside
|
||||
// the range, so no point can fall outside the tree, and it puts the root split
|
||||
// in that dimension exactly at s — which is what makes two trees planted from
|
||||
// different draws disagree, and disagreement is what a forest averages over.
|
||||
func plant(rng *rand.Rand, dims, depth int) *tree {
|
||||
n := 1<<(depth+1) - 1
|
||||
t := &tree{
|
||||
dim: make([]uint8, n),
|
||||
split: make([]float64, n),
|
||||
ref: make([]float64, n),
|
||||
cur: make([]float64, n),
|
||||
}
|
||||
lo := make([]float64, dims)
|
||||
hi := make([]float64, dims)
|
||||
for q := range lo {
|
||||
s := rng.Float64()
|
||||
w := 2 * math.Max(s, 1-s)
|
||||
lo[q], hi[q] = s-w, s+w
|
||||
}
|
||||
t.grow(rng, 0, 0, depth, lo, hi)
|
||||
return t
|
||||
}
|
||||
|
||||
// grow fills node i and everything below it.
|
||||
func (t *tree) grow(rng *rand.Rand, i, at, depth int, lo, hi []float64) {
|
||||
if at == depth {
|
||||
return // external node: no split to record
|
||||
}
|
||||
q := rng.IntN(len(lo))
|
||||
p := (lo[q] + hi[q]) / 2
|
||||
t.dim[i] = uint8(q)
|
||||
t.split[i] = p
|
||||
|
||||
keep := hi[q]
|
||||
hi[q] = p
|
||||
t.grow(rng, 2*i+1, at+1, depth, lo, hi)
|
||||
hi[q] = keep
|
||||
|
||||
keep = lo[q]
|
||||
lo[q] = p
|
||||
t.grow(rng, 2*i+2, at+1, depth, lo, hi)
|
||||
lo[q] = keep
|
||||
}
|
||||
|
||||
// child returns the heap index of the child x descends into.
|
||||
func (t *tree) child(i int, x []float64) int {
|
||||
if x[t.dim[i]] < t.split[i] {
|
||||
return 2*i + 1
|
||||
}
|
||||
return 2*i + 2
|
||||
}
|
||||
|
||||
// learn adds one point to the open window, incrementing every region on its
|
||||
// path. Cost is depth+1 increments and no allocation, which is what "O(1)
|
||||
// amortised update" means in practice.
|
||||
func (t *tree) learn(x []float64, depth int) {
|
||||
i := 0
|
||||
for at := 0; ; at++ {
|
||||
t.cur[i]++
|
||||
if at == depth {
|
||||
return
|
||||
}
|
||||
i = t.child(i, x)
|
||||
}
|
||||
}
|
||||
|
||||
// mass scores x against the reference window: the mass of the deepest region
|
||||
// reached, scaled by 2^depth.
|
||||
//
|
||||
// The descent stops early once a region's reference mass falls to limit, and
|
||||
// that early stop is doing real work rather than saving time. Below a handful of
|
||||
// points the mass of a region says nothing about its density — it is noise — and
|
||||
// multiplying noise by 2^depth would amplify exactly the thing that should be
|
||||
// ignored. Stopping puts the answer at the last depth where the count still
|
||||
// meant something.
|
||||
func (t *tree) mass(x []float64, depth int, limit float64) float64 {
|
||||
i := 0
|
||||
for at := 0; ; at++ {
|
||||
if at == depth || t.ref[i] <= limit {
|
||||
return t.ref[i] * float64(uint64(1)<<uint(at))
|
||||
}
|
||||
i = t.child(i, x)
|
||||
}
|
||||
}
|
||||
|
||||
// roll closes the open window: the mass it accumulated folds into the reference
|
||||
// at rate blend, and a new window starts empty.
|
||||
//
|
||||
// blend = 1 is the published algorithm, where the new window simply replaces the
|
||||
// reference. Below 1 it is an exponential blend, and the default is below 1 for
|
||||
// one reason: with a straight replacement, anyone who can submit transactions
|
||||
// can make their own behaviour normal by sending one window's worth of it. A
|
||||
// blend of b stretches the reference over roughly 1/b windows without costing a
|
||||
// byte more memory, so the same attack needs sustained volume against the whole
|
||||
// tenant rather than a burst. Adaptation to genuine drift is slower by the same
|
||||
// factor, which is the trade being made and the reason the number is a
|
||||
// parameter.
|
||||
func (t *tree) roll(blend float64) {
|
||||
for i := range t.ref {
|
||||
t.ref[i] += blend * (t.cur[i] - t.ref[i])
|
||||
t.cur[i] = 0
|
||||
}
|
||||
}
|
||||
|
||||
// sound reports whether the masses are usable: finite, non-negative, and
|
||||
// internally consistent.
|
||||
//
|
||||
// The consistency check is the useful one. Every point increments each node on
|
||||
// its path, so a parent's mass is exactly the sum of its children's, and the
|
||||
// blend is linear so folding preserves that. Any mass array that does not
|
||||
// satisfy it was not produced by this algorithm. That makes the invariant a
|
||||
// cheap guard on restored state — not authentication, which the caller owes, but
|
||||
// enough that a mass array cannot be bent into a shape that hides a region
|
||||
// without the bend being visible.
|
||||
func (t *tree) sound(depth int) bool {
|
||||
internal := 1<<depth - 1
|
||||
for i := range t.ref {
|
||||
if !finite(t.ref[i]) || t.ref[i] < 0 || !finite(t.cur[i]) || t.cur[i] < 0 {
|
||||
return false
|
||||
}
|
||||
if i < internal {
|
||||
if !within(t.ref[i], t.ref[2*i+1]+t.ref[2*i+2]) {
|
||||
return false
|
||||
}
|
||||
if !within(t.cur[i], t.cur[2*i+1]+t.cur[2*i+2]) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func finite(v float64) bool { return !math.IsNaN(v) && !math.IsInf(v, 0) }
|
||||
|
||||
// within compares two masses that arithmetic should have made equal, allowing
|
||||
// for the rounding a long chain of blends accumulates.
|
||||
func within(a, b float64) bool {
|
||||
d := math.Abs(a - b)
|
||||
return d <= 1e-6*math.Max(1, math.Max(math.Abs(a), math.Abs(b)))
|
||||
}
|
||||
@@ -0,0 +1,289 @@
|
||||
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
|
||||
// Use of this source code is governed by the Apache 2.0
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/aml/pkg/replay"
|
||||
"github.com/luxfi/aml/pkg/retention"
|
||||
"github.com/luxfi/aml/pkg/token"
|
||||
"github.com/luxfi/aml/pkg/types"
|
||||
)
|
||||
|
||||
// The record plane is where the ledger, the tokeniser and the engine meet. Each
|
||||
// of the three knows nothing of the other two: retention holds records and does
|
||||
// not know they are sealed, token seals bytes and does not know they are records,
|
||||
// replay reads events and cannot write. This file is the only place that joins
|
||||
// them, which is why it is the only place that has to be right about all three.
|
||||
|
||||
var (
|
||||
// errNoRecords is the refusal that keeps the engine from processing a
|
||||
// transaction it cannot record.
|
||||
errNoRecords = errors.New("record plane is not available")
|
||||
errNoParty = errors.New("record names no party")
|
||||
errFull = errors.New("history is full")
|
||||
)
|
||||
|
||||
// maxHistory bounds a replay. The ledger holds five years of records, and a
|
||||
// replay is an operator action, so it reads a bounded prefix rather than however
|
||||
// much a five-year ledger happens to contain.
|
||||
const maxHistory = 100_000
|
||||
|
||||
// maxSample bounds a caller-supplied history.
|
||||
const maxSample = 1000
|
||||
|
||||
// maxReason bounds the recorded reason for a refusal.
|
||||
const maxReason = 512
|
||||
|
||||
// vault is the org's tokeniser, or the reason there is none. A missing key is a
|
||||
// refusal and never a pass-through: the alternative is a clean receipt for a
|
||||
// transaction nobody can produce a record of.
|
||||
func (h *Handler) vault(orgID string) (*token.Vault, error) {
|
||||
if h.Records == nil || h.Keys == nil {
|
||||
return nil, errNoRecords
|
||||
}
|
||||
v, err := h.Keys.Org(orgID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %w", errNoRecords, err)
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
// slot binds a sealed body to the record that holds it. Both halves are
|
||||
// unsealed metadata, so the binding is recomputable when the record is read and a
|
||||
// body cannot be moved to another record or another class.
|
||||
func slot(class retention.Class, ref string) string {
|
||||
return string(class) + ":" + ref
|
||||
}
|
||||
|
||||
// parties are the keys a record is found by. Direct identifiers do not reach the
|
||||
// ledger in the clear: what is indexed is the pseudonym, deterministic within
|
||||
// this org and unrelated to the same person in any other.
|
||||
func parties(v *token.Vault, of map[token.Domain]string) ([]string, error) {
|
||||
out := make([]string, 0, len(of))
|
||||
for _, d := range []token.Domain{token.DomainSubject, token.DomainName, token.DomainAccount, token.DomainWallet} {
|
||||
value := strings.TrimSpace(of[d])
|
||||
if value == "" {
|
||||
continue
|
||||
}
|
||||
p, err := v.Pseudonym(d, value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, p)
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil, errNoParty
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// seal marshals and seals what a record has to be able to reconstruct. The whole
|
||||
// body is sealed as one piece, which is what keeps protection at rest from
|
||||
// becoming redaction: nothing is dropped and nothing is masked.
|
||||
func seal(v *token.Vault, class retention.Class, ref string, of any) ([]byte, error) {
|
||||
plain, err := json.Marshal(of)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("marshal record body: %w", err)
|
||||
}
|
||||
return v.Seal(slot(class, ref), plain)
|
||||
}
|
||||
|
||||
// retain writes the record for a transaction the engine has just evaluated.
|
||||
//
|
||||
// A blocked transaction is a refusal to carry out a transaction, and its
|
||||
// retention runs from the date of refusal rather than from the end of any
|
||||
// relationship (AMLR Art. 77(3) third trigger) — so it is retained as a refusal
|
||||
// with the rules that caused it, not as an ordinary transaction record.
|
||||
func (h *Handler) retain(v *token.Vault, tx types.Transaction, entity types.Entity, relationship string, alerts []types.Alert, action string) (string, error) {
|
||||
party, err := parties(v, map[token.Domain]string{
|
||||
token.DomainSubject: tx.UserID,
|
||||
token.DomainName: entity.Name,
|
||||
token.DomainAccount: tx.AccountID,
|
||||
})
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if tx.Counterparty != "" {
|
||||
other, err := v.Pseudonym(token.DomainSubject, tx.Counterparty)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
party = append(party, other)
|
||||
}
|
||||
|
||||
record := retention.Record{
|
||||
Org: tx.OrgID,
|
||||
Ref: tx.ID,
|
||||
Parties: party,
|
||||
Occurred: tx.Timestamp,
|
||||
}
|
||||
switch action {
|
||||
case types.ActionBlock:
|
||||
record.Class = retention.ClassRefusal
|
||||
record.Trigger = retention.TriggerRefusal
|
||||
record.Reason = refusalReason(alerts)
|
||||
default:
|
||||
record.Class = retention.ClassTransaction
|
||||
if relationship != "" {
|
||||
record.Trigger = retention.TriggerRelationshipEnd
|
||||
record.Relationship = relationship
|
||||
} else {
|
||||
record.Trigger = retention.TriggerOccasional
|
||||
}
|
||||
}
|
||||
|
||||
body, err := seal(v, record.Class, record.Ref, types.EvalContext{Tx: tx, Entity: entity})
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
record.Body = body
|
||||
|
||||
return h.Records.Retain(record)
|
||||
}
|
||||
|
||||
// refusalReason names the rules that refused the transaction, because "refused"
|
||||
// on its own does not answer why the firm refrained.
|
||||
func refusalReason(alerts []types.Alert) string {
|
||||
names := make([]string, 0, len(alerts))
|
||||
for _, a := range alerts {
|
||||
if a.ActionTaken == types.ActionBlock {
|
||||
names = append(names, a.RuleName)
|
||||
}
|
||||
}
|
||||
if len(names) == 0 {
|
||||
return "blocked on the aggregate score"
|
||||
}
|
||||
reason := "blocked by " + strings.Join(names, ", ")
|
||||
if len(reason) > maxReason {
|
||||
reason = reason[:maxReason]
|
||||
}
|
||||
return reason
|
||||
}
|
||||
|
||||
// history reads the org's retained transactions back into replayable events.
|
||||
//
|
||||
// The bodies are sealed, so this is where the vault opens them, under the
|
||||
// monitoring purpose — testing a typology before activation is monitoring work
|
||||
// and not a commercial read (AMLD4 Art. 41(2)). A body that does not open is an
|
||||
// error and not a skipped event: a replay over a silently shortened history is
|
||||
// the same lie as a replay over an empty one.
|
||||
func (h *Handler) history(orgID string, v *token.Vault) (replay.Slice, error) {
|
||||
judged, err := h.dispositions(orgID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var out replay.Slice
|
||||
err = h.Records.Each(retention.PurposeMonitoring, orgID, retention.ClassTransaction, func(r retention.Record) error {
|
||||
if len(out) >= maxHistory {
|
||||
return errFull
|
||||
}
|
||||
plain, err := v.Open(slot(r.Class, r.Ref), r.Body)
|
||||
if err != nil {
|
||||
return fmt.Errorf("record %s: %w", r.ID, err)
|
||||
}
|
||||
var ctx types.EvalContext
|
||||
if err := json.Unmarshal(plain, &ctx); err != nil {
|
||||
return fmt.Errorf("record %s: %w", r.ID, err)
|
||||
}
|
||||
|
||||
event := replay.Event{Tx: ctx.Tx, Entity: ctx.Entity}
|
||||
for _, a := range h.Alerts.ByTx(orgID, ctx.Tx.ID) {
|
||||
event.Alerted = append(event.Alerted, a.RuleID)
|
||||
if d, ok := judged[a.ID]; ok {
|
||||
event.Disposition = d
|
||||
}
|
||||
}
|
||||
out = append(out, event)
|
||||
return nil
|
||||
})
|
||||
if err != nil && !errors.Is(err, errFull) {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// dispositions is what humans concluded, keyed by the alert they concluded it
|
||||
// about. It comes from the retained assessments, which is the only place a
|
||||
// decision not to report is recorded.
|
||||
func (h *Handler) dispositions(orgID string) (map[string]replay.Disposition, error) {
|
||||
out := make(map[string]replay.Disposition)
|
||||
err := h.Records.Each(retention.PurposeMonitoring, orgID, retention.ClassAssessment, func(r retention.Record) error {
|
||||
if r.Assessment == nil {
|
||||
return nil
|
||||
}
|
||||
d := replay.Unproductive
|
||||
if r.Assessment.Result == retention.Reported {
|
||||
d = replay.Productive
|
||||
}
|
||||
for _, alert := range r.Assessment.Alerts {
|
||||
out[alert] = d
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// assess retains the Art. 69(2) assessment behind a case resolution: the
|
||||
// information and circumstances considered, and the result, whether or not it
|
||||
// produced a report (AMLR Art. 77(1)(b)). A dismissal is a retained decision.
|
||||
func (h *Handler) assess(v *token.Vault, orgID string, c *types.Case, in resolution) (string, error) {
|
||||
party := make([]string, 0, len(c.EntityIDs))
|
||||
for _, entity := range c.EntityIDs {
|
||||
if strings.TrimSpace(entity) == "" {
|
||||
continue
|
||||
}
|
||||
p, err := v.Pseudonym(token.DomainSubject, entity)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
party = append(party, p)
|
||||
}
|
||||
if len(party) == 0 {
|
||||
return "", errNoParty
|
||||
}
|
||||
|
||||
result := retention.NotReported
|
||||
if in.Resolution == types.ResolutionSARFiled {
|
||||
result = retention.Reported
|
||||
}
|
||||
decided := time.Now().UTC()
|
||||
|
||||
body, err := seal(v, retention.ClassAssessment, c.ID, map[string]any{
|
||||
"case": c,
|
||||
"resolution": in.Resolution,
|
||||
})
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
return h.Records.Retain(retention.Record{
|
||||
Org: orgID,
|
||||
Class: retention.ClassAssessment,
|
||||
Trigger: retention.TriggerOccasional,
|
||||
Ref: c.ID,
|
||||
Parties: party,
|
||||
Occurred: decided,
|
||||
Body: body,
|
||||
Assessment: &retention.Assessment{
|
||||
Alerts: c.AlertIDs,
|
||||
Case: c.ID,
|
||||
Considered: in.Considered,
|
||||
Result: result,
|
||||
Rationale: in.Rationale,
|
||||
By: in.By,
|
||||
At: decided,
|
||||
},
|
||||
})
|
||||
}
|
||||
+11
-8
@@ -7,6 +7,9 @@ import (
|
||||
"github.com/luxfi/aml/pkg/types"
|
||||
)
|
||||
|
||||
// acme is the org every record in these tests belongs to.
|
||||
const acme = "acme"
|
||||
|
||||
// TestAlertStoreEviction (RED-08) verifies that the alert store evicts oldest
|
||||
// entries when capacity is exceeded.
|
||||
func TestAlertStoreEviction(t *testing.T) {
|
||||
@@ -15,7 +18,7 @@ func TestAlertStoreEviction(t *testing.T) {
|
||||
// Add 200 items — store should evict down to ~90 (evicts 10% = 10 entries per trigger).
|
||||
for i := 0; i < 200; i++ {
|
||||
txID := fmt.Sprintf("tx-%d", i)
|
||||
store.Add(txID, []types.Alert{{ID: fmt.Sprintf("alert-%d", i)}})
|
||||
store.Add(txID, []types.Alert{{ID: fmt.Sprintf("alert-%d", i), OrgID: acme}})
|
||||
}
|
||||
|
||||
size := store.Len()
|
||||
@@ -24,12 +27,12 @@ func TestAlertStoreEviction(t *testing.T) {
|
||||
}
|
||||
|
||||
// Oldest entries should be evicted.
|
||||
if alerts := store.ByTx("tx-0"); len(alerts) > 0 {
|
||||
if alerts := store.ByTx(acme, "tx-0"); len(alerts) > 0 {
|
||||
t.Error("tx-0 should have been evicted")
|
||||
}
|
||||
|
||||
// Most recent entries should still exist.
|
||||
if alerts := store.ByTx("tx-199"); len(alerts) == 0 {
|
||||
if alerts := store.ByTx(acme, "tx-199"); len(alerts) == 0 {
|
||||
t.Error("tx-199 should still be present")
|
||||
}
|
||||
}
|
||||
@@ -40,7 +43,7 @@ func TestAlertStoreNoEvictionUnderCapacity(t *testing.T) {
|
||||
store := NewAlertStoreWithMax(200)
|
||||
|
||||
for i := 0; i < 50; i++ {
|
||||
store.Add(fmt.Sprintf("tx-%d", i), []types.Alert{{ID: fmt.Sprintf("a-%d", i)}})
|
||||
store.Add(fmt.Sprintf("tx-%d", i), []types.Alert{{ID: fmt.Sprintf("a-%d", i), OrgID: acme}})
|
||||
}
|
||||
|
||||
if store.Len() != 50 {
|
||||
@@ -49,7 +52,7 @@ func TestAlertStoreNoEvictionUnderCapacity(t *testing.T) {
|
||||
|
||||
// All entries should be present.
|
||||
for i := 0; i < 50; i++ {
|
||||
if alerts := store.ByTx(fmt.Sprintf("tx-%d", i)); len(alerts) == 0 {
|
||||
if alerts := store.ByTx(acme, fmt.Sprintf("tx-%d", i)); len(alerts) == 0 {
|
||||
t.Errorf("tx-%d should be present", i)
|
||||
}
|
||||
}
|
||||
@@ -60,14 +63,14 @@ func TestAlertStoreNoEvictionUnderCapacity(t *testing.T) {
|
||||
func TestAlertStoreDuplicateTxID(t *testing.T) {
|
||||
store := NewAlertStoreWithMax(100)
|
||||
|
||||
store.Add("tx-1", []types.Alert{{ID: "a1"}})
|
||||
store.Add("tx-1", []types.Alert{{ID: "a2"}})
|
||||
store.Add("tx-1", []types.Alert{{ID: "a1", OrgID: acme}})
|
||||
store.Add("tx-1", []types.Alert{{ID: "a2", OrgID: acme}})
|
||||
|
||||
if store.Len() != 1 {
|
||||
t.Errorf("store size = %d, want 1 (same tx_id)", store.Len())
|
||||
}
|
||||
|
||||
alerts := store.ByTx("tx-1")
|
||||
alerts := store.ByTx(acme, "tx-1")
|
||||
if len(alerts) != 2 {
|
||||
t.Errorf("alert count = %d, want 2", len(alerts))
|
||||
}
|
||||
|
||||
+16
-3
@@ -203,8 +203,20 @@ func (s *Store) Assign(caseID, assigneeID string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Resolve closes a case with a resolution.
|
||||
func (s *Store) Resolve(caseID, resolution, authorID string) error {
|
||||
// Resolve closes a case with a resolution, against the retained assessment that
|
||||
// decided it.
|
||||
//
|
||||
// The assessment id is required. Closing a case is a decision about whether to
|
||||
// report, and that decision is retained whether or not it produced one
|
||||
// (Regulation (EU) 2024/1624 Art. 77(1)(b)); where the decision is not to
|
||||
// report, the reasons must be documented and kept with the internal suspicion
|
||||
// report (JMLSG 6.32). So a dismissal here cannot be a row that quietly changes
|
||||
// state: without the record of the decision there is no closure.
|
||||
func (s *Store) Resolve(caseID, resolution, authorID, assessment string) error {
|
||||
if assessment == "" {
|
||||
return ErrNoAssessment
|
||||
}
|
||||
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
|
||||
@@ -216,6 +228,7 @@ func (s *Store) Resolve(caseID, resolution, authorID string) error {
|
||||
now := time.Now().UTC()
|
||||
c.Status = types.CaseClosed
|
||||
c.Resolution = resolution
|
||||
c.Assessment = assessment
|
||||
c.ClosedAt = &now
|
||||
c.UpdatedAt = now
|
||||
|
||||
@@ -224,7 +237,7 @@ func (s *Store) Resolve(caseID, resolution, authorID string) error {
|
||||
CaseID: caseID,
|
||||
AuthorID: authorID,
|
||||
Kind: types.EventStatusChange,
|
||||
Body: "closed: " + resolution,
|
||||
Body: "closed: " + resolution + " (assessment " + assessment + ")",
|
||||
CreatedAt: now,
|
||||
})
|
||||
|
||||
|
||||
@@ -147,7 +147,7 @@ func TestResolve(t *testing.T) {
|
||||
s := NewStore()
|
||||
c := s.Create("org1", types.SeverityHigh, nil, nil)
|
||||
|
||||
err := s.Resolve(c.ID, types.ResolutionSARFiled, "user1")
|
||||
err := s.Resolve(c.ID, types.ResolutionSARFiled, "user1", "assessment-1")
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve error: %v", err)
|
||||
}
|
||||
@@ -184,7 +184,7 @@ func TestCaseStoreEviction(t *testing.T) {
|
||||
// Create and immediately close 15 cases.
|
||||
for i := 0; i < 15; i++ {
|
||||
c := s.Create("org1", types.SeverityLow, nil, nil)
|
||||
s.Resolve(c.ID, types.ResolutionCleared, "system")
|
||||
s.Resolve(c.ID, types.ResolutionCleared, "system", "assessment-1")
|
||||
}
|
||||
|
||||
// Wait for closed cases to become stale.
|
||||
@@ -208,7 +208,7 @@ func TestCaseStoreEvictExpiredDirect(t *testing.T) {
|
||||
for i := 0; i < 5; i++ {
|
||||
c := s.Create("org1", types.SeverityLow, nil, nil)
|
||||
if i < 3 {
|
||||
s.Resolve(c.ID, types.ResolutionCleared, "system")
|
||||
s.Resolve(c.ID, types.ResolutionCleared, "system", "assessment-1")
|
||||
} else {
|
||||
openIDs = append(openIDs, c.ID)
|
||||
}
|
||||
|
||||
@@ -4,3 +4,7 @@ import "errors"
|
||||
|
||||
// ErrNotFound is returned when a case is not found.
|
||||
var ErrNotFound = errors.New("case not found")
|
||||
|
||||
// ErrNoAssessment is returned when a case would be closed without the retained
|
||||
// decision that closed it.
|
||||
var ErrNoAssessment = errors.New("case cannot be closed without a retained assessment")
|
||||
|
||||
@@ -96,7 +96,7 @@ func TestCheckEscalation_NotBreached(t *testing.T) {
|
||||
func TestCheckEscalation_AlreadyClosed(t *testing.T) {
|
||||
s := NewStore()
|
||||
c := s.Create("org1", types.SeverityCritical, nil, nil)
|
||||
_ = s.Resolve(c.ID, "cleared", "analyst")
|
||||
_ = s.Resolve(c.ID, "cleared", "analyst", "assessment-1")
|
||||
|
||||
config := DefaultTriageConfig()
|
||||
escalated := s.CheckEscalation(c, config)
|
||||
|
||||
+82
-13
@@ -5,7 +5,9 @@ package engine
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
@@ -17,6 +19,9 @@ type Engine struct {
|
||||
evaluator *Evaluator
|
||||
mu sync.RWMutex
|
||||
rules []types.Rule
|
||||
scorer Scorer
|
||||
faults atomic.Int64
|
||||
rejects atomic.Int64
|
||||
}
|
||||
|
||||
// New creates an Engine with the given rules.
|
||||
@@ -55,6 +60,71 @@ func (e *Engine) Evaluator() *Evaluator {
|
||||
return e.evaluator
|
||||
}
|
||||
|
||||
// SetScorer installs the Scorer consulted alongside the rules, or nil to run on
|
||||
// rules alone.
|
||||
func (e *Engine) SetScorer(s Scorer) {
|
||||
e.mu.Lock()
|
||||
e.scorer = s
|
||||
e.mu.Unlock()
|
||||
}
|
||||
|
||||
// ScorerFaults is the number of times the Scorer failed and its evidence was
|
||||
// discarded — a panic, a negative weight, or a weight that is not a number.
|
||||
//
|
||||
// It has to be readable. A Scorer that faults on every transaction contributes
|
||||
// nothing while the engine keeps answering, and the only difference between that
|
||||
// and a Scorer finding nothing to report is this counter.
|
||||
func (e *Engine) ScorerFaults() int64 { return e.faults.Load() + e.rejects.Load() }
|
||||
|
||||
// assess consults the Scorer, and confines it.
|
||||
//
|
||||
// Three things a Scorer must not be able to do, each enforced here rather than
|
||||
// trusted to the Scorer, because the property has to hold for one that is wrong
|
||||
// as well as one that is honest:
|
||||
//
|
||||
// It must not take the rule plane down. The transaction has already been
|
||||
// evaluated against every rule by the time this runs, and losing that verdict to
|
||||
// a fault in the statistical plane would turn a degraded control into no control
|
||||
// — so a panic is contained, counted, and the rules' answer stands.
|
||||
//
|
||||
// It must not weaken a verdict. Weight-of-evidence is a sum, so a negative weight
|
||||
// would subtract from a score built by the rules, and a model would be able to
|
||||
// argue a transaction down. Non-negative weights are already required of rules
|
||||
// for that reason; a computed weight is held to it too, along with the NaN a
|
||||
// computation can produce and a constant cannot.
|
||||
//
|
||||
// It must not act. Its action is capped at types.ActionCeiling, so the strongest
|
||||
// thing a statistical judgement can do is put a transaction in front of a person.
|
||||
func (e *Engine) assess(tx types.Transaction, entity types.Entity) (hit types.RuleHit, ok bool) {
|
||||
e.mu.RLock()
|
||||
sc := e.scorer
|
||||
e.mu.RUnlock()
|
||||
if sc == nil {
|
||||
return types.RuleHit{}, false
|
||||
}
|
||||
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
e.faults.Add(1)
|
||||
hit, ok = types.RuleHit{}, false
|
||||
}
|
||||
}()
|
||||
|
||||
hit, ok = sc.Assess(tx, entity)
|
||||
if !ok {
|
||||
return types.RuleHit{}, false
|
||||
}
|
||||
if hit.Rule.Weight < 0 || math.IsNaN(hit.Rule.Weight) || math.IsInf(hit.Rule.Weight, 0) {
|
||||
e.rejects.Add(1)
|
||||
return types.RuleHit{}, false
|
||||
}
|
||||
if types.ActionRank(hit.Rule.Action) > types.ActionRank(types.ActionCeiling) {
|
||||
hit.Rule.Action = types.ActionCeiling
|
||||
}
|
||||
hit.Match = true
|
||||
return hit, true
|
||||
}
|
||||
|
||||
// Evaluate runs all rules against a transaction and its primary entity.
|
||||
// Returns the computed alerts, the aggregate score, and the enforcement action.
|
||||
func (e *Engine) Evaluate(tx types.Transaction, entity types.Entity) ([]types.Alert, float64, string) {
|
||||
@@ -68,6 +138,15 @@ func (e *Engine) Evaluate(tx types.Transaction, entity types.Entity) ([]types.Al
|
||||
}
|
||||
|
||||
hits := e.evaluator.EvalAll(rules, ctx)
|
||||
|
||||
// The Scorer is consulted after the rules and its evidence joins theirs, so
|
||||
// the statistical plane can only add to what the rules found. It can raise a
|
||||
// transaction no rule matched, which is the point of having it; it cannot
|
||||
// lower one they did.
|
||||
if hit, ok := e.assess(tx, entity); ok {
|
||||
hits = append(hits, hit)
|
||||
}
|
||||
|
||||
if len(hits) == 0 {
|
||||
return nil, 0, types.ActionAllow
|
||||
}
|
||||
@@ -86,6 +165,7 @@ func (e *Engine) Evaluate(tx types.Transaction, entity types.Entity) ([]types.Al
|
||||
Severity: h.Rule.Severity,
|
||||
Score: breakdown[h.Rule.ID],
|
||||
ScoreBreakdown: breakdown,
|
||||
Causes: h.Causes,
|
||||
ActionTaken: h.Rule.Action,
|
||||
CreatedAt: now,
|
||||
UpdatedAt: now,
|
||||
@@ -96,23 +176,12 @@ func (e *Engine) Evaluate(tx types.Transaction, entity types.Entity) ([]types.Al
|
||||
return alerts, score, action
|
||||
}
|
||||
|
||||
// resolveAction picks the most severe action from a set of alerts.
|
||||
// Priority: block > report > review > flag > allow.
|
||||
// resolveAction picks the most demanding action from a set of alerts.
|
||||
func resolveAction(alerts []types.Alert) string {
|
||||
priority := map[string]int{
|
||||
types.ActionBlock: 4,
|
||||
types.ActionReport: 3,
|
||||
types.ActionReview: 2,
|
||||
types.ActionFlag: 1,
|
||||
types.ActionAllow: 0,
|
||||
}
|
||||
|
||||
best := types.ActionAllow
|
||||
bestP := 0
|
||||
for _, a := range alerts {
|
||||
if p, ok := priority[a.ActionTaken]; ok && p > bestP {
|
||||
if types.ActionRank(a.ActionTaken) > types.ActionRank(best) {
|
||||
best = a.ActionTaken
|
||||
bestP = p
|
||||
}
|
||||
}
|
||||
return best
|
||||
|
||||
@@ -0,0 +1,237 @@
|
||||
package engine
|
||||
|
||||
import (
|
||||
"math"
|
||||
"testing"
|
||||
|
||||
"github.com/luxfi/aml/pkg/types"
|
||||
)
|
||||
|
||||
// scorer is a Scorer that returns exactly what a test tells it to, including the
|
||||
// things an honest one never would.
|
||||
type scorer struct {
|
||||
hit types.RuleHit
|
||||
ok bool
|
||||
panic bool
|
||||
}
|
||||
|
||||
func (s scorer) Assess(types.Transaction, types.Entity) (types.RuleHit, bool) {
|
||||
if s.panic {
|
||||
panic("scorer fault")
|
||||
}
|
||||
return s.hit, s.ok
|
||||
}
|
||||
|
||||
func evidence(weight float64, action string) types.RuleHit {
|
||||
return types.RuleHit{
|
||||
Rule: types.Rule{
|
||||
ID: "anomaly", Name: "Unusual for this customer",
|
||||
Severity: types.SeverityMedium, Weight: weight, Action: action, Enabled: true,
|
||||
},
|
||||
Match: true,
|
||||
}
|
||||
}
|
||||
|
||||
func flagRule() types.Rule {
|
||||
return types.Rule{
|
||||
ID: "r-flag", Name: "flag", DSL: `tx.Notional > 100`,
|
||||
Severity: types.SeverityMedium, Weight: 0.3, Action: types.ActionFlag, Enabled: true,
|
||||
}
|
||||
}
|
||||
|
||||
var tx = types.Transaction{ID: "t1", OrgID: "acme", UserID: "u1", Notional: 500}
|
||||
|
||||
// A fault in the statistical plane must not take the rule plane with it. Every
|
||||
// rule has already been evaluated by the time the Scorer runs, and losing that
|
||||
// verdict to a panic would turn a degraded control into no control.
|
||||
func TestPanickingScorerDoesNotLoseTheRulesVerdict(t *testing.T) {
|
||||
e := New([]types.Rule{flagRule()})
|
||||
e.SetScorer(scorer{panic: true})
|
||||
|
||||
alerts, score, action := e.Evaluate(tx, types.Entity{})
|
||||
if action != types.ActionFlag {
|
||||
t.Fatalf("action = %q, want the rule's %q", action, types.ActionFlag)
|
||||
}
|
||||
if len(alerts) != 1 || score <= 0 {
|
||||
t.Fatalf("rule verdict lost: %d alerts, score %v", len(alerts), score)
|
||||
}
|
||||
// The fault must be countable. A Scorer that faults on every transaction
|
||||
// contributes nothing while the engine keeps answering, and this counter is
|
||||
// the only difference between that and one that finds nothing to report.
|
||||
if e.ScorerFaults() != 1 {
|
||||
t.Fatalf("faults = %d, want 1", e.ScorerFaults())
|
||||
}
|
||||
}
|
||||
|
||||
// Weight-of-evidence is a sum, so a negative weight would let a model argue a
|
||||
// transaction down below what the rules found. Rules are already forbidden
|
||||
// negative weights; a computed weight is held to the same rule, and so are the
|
||||
// values only a computation can produce.
|
||||
func TestScorerCannotWeakenAVerdict(t *testing.T) {
|
||||
base := New([]types.Rule{flagRule()})
|
||||
_, want, wantAction := base.Evaluate(tx, types.Entity{})
|
||||
|
||||
for name, weight := range map[string]float64{
|
||||
"negative": -5,
|
||||
"nan": math.NaN(),
|
||||
"+infinity": math.Inf(1),
|
||||
"-infinity": math.Inf(-1),
|
||||
} {
|
||||
e := New([]types.Rule{flagRule()})
|
||||
e.SetScorer(scorer{hit: evidence(weight, types.ActionReview), ok: true})
|
||||
alerts, got, action := e.Evaluate(tx, types.Entity{})
|
||||
|
||||
if got < want {
|
||||
t.Errorf("%s weight lowered the score from %v to %v", name, want, got)
|
||||
}
|
||||
if math.IsNaN(got) {
|
||||
t.Errorf("%s weight produced a score that is not a number", name)
|
||||
}
|
||||
if types.ActionRank(action) < types.ActionRank(wantAction) {
|
||||
t.Errorf("%s weight weakened the action from %q to %q", name, wantAction, action)
|
||||
}
|
||||
if len(alerts) != 1 {
|
||||
t.Errorf("%s weight: %d alerts, want the rule's 1", name, len(alerts))
|
||||
}
|
||||
if e.ScorerFaults() != 1 {
|
||||
t.Errorf("%s weight was not counted as a fault", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The model summons a person; it does not act. Whatever a Scorer asks for, the
|
||||
// strongest thing it can obtain is a review — enforced on the Scorer's output
|
||||
// rather than trusted to the Scorer, because the property has to hold for one
|
||||
// that is wrong as well as one that is honest.
|
||||
func TestScorerActionIsCappedAtTheCeiling(t *testing.T) {
|
||||
for _, asked := range []string{types.ActionBlock, types.ActionReport} {
|
||||
e := New(nil)
|
||||
e.SetScorer(scorer{hit: evidence(0.9, asked), ok: true})
|
||||
alerts, _, action := e.Evaluate(tx, types.Entity{})
|
||||
if action != types.ActionCeiling {
|
||||
t.Errorf("scorer asked for %q and got %q, ceiling is %q", asked, action, types.ActionCeiling)
|
||||
}
|
||||
if len(alerts) != 1 || alerts[0].ActionTaken != types.ActionCeiling {
|
||||
t.Errorf("scorer asked for %q and the alert carries %q", asked, alerts[0].ActionTaken)
|
||||
}
|
||||
}
|
||||
// At or below the ceiling it gets what it asked for.
|
||||
for _, asked := range []string{types.ActionFlag, types.ActionReview} {
|
||||
e := New(nil)
|
||||
e.SetScorer(scorer{hit: evidence(0.9, asked), ok: true})
|
||||
if _, _, action := e.Evaluate(tx, types.Entity{}); action != asked {
|
||||
t.Errorf("scorer asked for %q and got %q", asked, action)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The statistical plane may raise a transaction no rule matched — that is the
|
||||
// point of having it — and may never lower one they did.
|
||||
func TestScorerOnlyAdds(t *testing.T) {
|
||||
rules := []types.Rule{flagRule()}
|
||||
plain := New(rules)
|
||||
_, want, wantAction := plain.Evaluate(tx, types.Entity{})
|
||||
|
||||
e := New(rules)
|
||||
e.SetScorer(scorer{hit: evidence(0.2, types.ActionReview), ok: true})
|
||||
alerts, got, action := e.Evaluate(tx, types.Entity{})
|
||||
if got <= want {
|
||||
t.Fatalf("evidence did not add: %v -> %v", want, got)
|
||||
}
|
||||
if types.ActionRank(action) < types.ActionRank(wantAction) {
|
||||
t.Fatalf("evidence weakened the action: %q -> %q", wantAction, action)
|
||||
}
|
||||
if len(alerts) != 2 {
|
||||
t.Fatalf("%d alerts, want the rule's and the model's", len(alerts))
|
||||
}
|
||||
|
||||
// With no rules at all the model can still raise a transaction.
|
||||
only := New(nil)
|
||||
only.SetScorer(scorer{hit: evidence(0.2, types.ActionReview), ok: true})
|
||||
if _, score, action := only.Evaluate(tx, types.Entity{}); score <= 0 || action != types.ActionReview {
|
||||
t.Fatalf("model alone produced score %v action %q", score, action)
|
||||
}
|
||||
}
|
||||
|
||||
// Declining to score contributes nothing at all — not an alert with no weight,
|
||||
// which downstream would read as a verdict of normal.
|
||||
func TestScorerDecliningContributesNothing(t *testing.T) {
|
||||
e := New(nil)
|
||||
e.SetScorer(scorer{hit: evidence(0.9, types.ActionReview), ok: false})
|
||||
alerts, score, action := e.Evaluate(tx, types.Entity{})
|
||||
if len(alerts) != 0 || score != 0 || action != types.ActionAllow {
|
||||
t.Fatalf("a declining scorer produced %d alerts, score %v, action %q", len(alerts), score, action)
|
||||
}
|
||||
if e.ScorerFaults() != 0 {
|
||||
t.Fatal("declining to score was counted as a fault")
|
||||
}
|
||||
}
|
||||
|
||||
// A model's alert is a number until it carries its reasons, so the attribution has
|
||||
// to survive the trip into the alert an investigator opens.
|
||||
func TestCausesReachTheAlert(t *testing.T) {
|
||||
hit := evidence(0.2, types.ActionReview)
|
||||
hit.Causes = []types.Cause{{
|
||||
Feature: "subthreshold", Typology: "structuring",
|
||||
Indicator: "transactions split to circumvent reporting limits",
|
||||
Citation: "EBA/GL/2021/02, Guideline 4.60(a)", Severity: types.SeverityHigh,
|
||||
Unit: "share of transactions falling just below the reporting threshold",
|
||||
Observed: 11, Baseline: 11, Without: 0.31, Share: 0.62,
|
||||
}}
|
||||
e := New(nil)
|
||||
e.SetScorer(scorer{hit: hit, ok: true})
|
||||
|
||||
alerts, _, _ := e.Evaluate(tx, types.Entity{})
|
||||
if len(alerts) != 1 {
|
||||
t.Fatalf("%d alerts", len(alerts))
|
||||
}
|
||||
got := alerts[0].Causes
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("alert carries %d causes", len(got))
|
||||
}
|
||||
if got[0].Feature != "subthreshold" || got[0].Citation == "" || got[0].Typology == "" {
|
||||
t.Fatalf("attribution arrived incomplete: %+v", got[0])
|
||||
}
|
||||
// A rule alert explains itself and must not acquire causes it never had.
|
||||
rules := New([]types.Rule{flagRule()})
|
||||
ruleAlerts, _, _ := rules.Evaluate(tx, types.Entity{})
|
||||
if len(ruleAlerts[0].Causes) != 0 {
|
||||
t.Fatal("a rule alert carries model attribution")
|
||||
}
|
||||
}
|
||||
|
||||
// Once a weight is computed rather than read from a constant, arithmetic can
|
||||
// produce a value that is not a number, and NaN compares false against every
|
||||
// bound — so the obvious two-comparison clamp passes it straight through, where it
|
||||
// becomes a transaction's risk score and fails to marshal into the response.
|
||||
func TestScoreClampHandlesNotANumber(t *testing.T) {
|
||||
if got := clamp01(math.NaN()); math.IsNaN(got) {
|
||||
t.Fatal("clamp01 passed NaN through as a risk score")
|
||||
}
|
||||
if got := clamp01(math.NaN()); got != 1 {
|
||||
t.Fatalf("clamp01(NaN) = %v, want the top of the range so a broken score reaches a person", got)
|
||||
}
|
||||
if clamp01(math.Inf(1)) != 1 || clamp01(math.Inf(-1)) != 0 {
|
||||
t.Fatal("clamp01 does not bound the infinities")
|
||||
}
|
||||
if clamp01(0.4) != 0.4 {
|
||||
t.Fatal("clamp01 altered a value already in range")
|
||||
}
|
||||
}
|
||||
|
||||
// No Scorer at all must leave the engine exactly as it was.
|
||||
func TestNoScorerChangesNothing(t *testing.T) {
|
||||
rules := []types.Rule{flagRule()}
|
||||
a := New(rules)
|
||||
wantAlerts, want, wantAction := a.Evaluate(tx, types.Entity{})
|
||||
|
||||
b := New(rules)
|
||||
b.SetScorer(nil)
|
||||
gotAlerts, got, gotAction := b.Evaluate(tx, types.Entity{})
|
||||
if got != want || gotAction != wantAction || len(gotAlerts) != len(wantAlerts) {
|
||||
t.Fatalf("installing a nil scorer changed the verdict: %v/%q vs %v/%q", got, gotAction, want, wantAction)
|
||||
}
|
||||
if b.ScorerFaults() != 0 {
|
||||
t.Fatal("a nil scorer produced faults")
|
||||
}
|
||||
}
|
||||
+34
-6
@@ -1,6 +1,10 @@
|
||||
package engine
|
||||
|
||||
import "github.com/luxfi/aml/pkg/types"
|
||||
import (
|
||||
"math"
|
||||
|
||||
"github.com/luxfi/aml/pkg/types"
|
||||
)
|
||||
|
||||
// severityMultiplier returns a weight multiplier for a severity level.
|
||||
func severityMultiplier(severity string) float64 {
|
||||
@@ -19,7 +23,18 @@ func severityMultiplier(severity string) float64 {
|
||||
}
|
||||
|
||||
// clamp01 clamps v to [0, 1].
|
||||
//
|
||||
// Not a number clamps to 1, not to 0. A score arrives here from arithmetic over
|
||||
// weights, and once a Scorer computes a weight rather than reading a constant one,
|
||||
// arithmetic can produce NaN. NaN compares false against every bound, so the
|
||||
// obvious two-comparison clamp returns it unchanged: it would then travel as a
|
||||
// transaction's risk score, fail to marshal into the response, and read as
|
||||
// harmless in any test written with < or >. Sending it to the top of the range
|
||||
// puts a broken score in front of a person.
|
||||
func clamp01(v float64) float64 {
|
||||
if math.IsNaN(v) {
|
||||
return 1
|
||||
}
|
||||
if v < 0 {
|
||||
return 0
|
||||
}
|
||||
@@ -45,9 +60,22 @@ func Score(hits []types.RuleHit) (float64, map[string]float64) {
|
||||
return clamp01(total), breakdown
|
||||
}
|
||||
|
||||
// ScorerPlugin is the interface for pluggable ML scoring.
|
||||
// v1 uses weight-of-evidence only. This interface is the hook point
|
||||
// for later Hanzo Zen gateway integration.
|
||||
type ScorerPlugin interface {
|
||||
Score(tx types.Transaction, entity types.Entity, hits []types.RuleHit) (float64, map[string]float64, error)
|
||||
// Scorer contributes evidence the rule library cannot express: a statistical
|
||||
// judgement about whether behaviour is unusual for the entity, where a rule can
|
||||
// only ask whether it matches a pattern someone has already named.
|
||||
//
|
||||
// The evidence is an ordinary RuleHit, so it is scored by the same
|
||||
// weight-of-evidence sum, becomes the same Alert, and reaches the same review
|
||||
// queue. Nothing downstream distinguishes it, which is deliberate: an alert a
|
||||
// model raised is reviewed by a person on the same terms as any other, and there
|
||||
// is one scoring path rather than two.
|
||||
//
|
||||
// ok is false whenever the Scorer has nothing to contribute — it cannot score, or
|
||||
// it can and the transaction is unremarkable. The engine then evaluates on rules
|
||||
// alone. A Scorer must not signal "normal" by returning a hit with no weight, and
|
||||
// the engine must not treat ok=false as a clean result; a Scorer owes its own
|
||||
// account of what it declined to score, because that count is the difference
|
||||
// between a control that is quiet and one that is not running.
|
||||
type Scorer interface {
|
||||
Assess(tx types.Transaction, entity types.Entity) (types.RuleHit, bool)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
// Package reference holds the published data a rule needs but cannot derive:
|
||||
// currency conversion rates and jurisdiction risk listings.
|
||||
//
|
||||
// Both carry the date they were current at, and both refuse to answer rather than
|
||||
// guess. That is the difference between a control and the appearance of one: a
|
||||
// conversion table that passes an unknown currency through unchanged moves every
|
||||
// threshold by the exchange rate, and a jurisdiction list that answers "not
|
||||
// listed" because it was never loaded is indistinguishable from a clean world.
|
||||
package reference
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Rates converts amounts into USD from a table current at a stated instant.
|
||||
type Rates struct {
|
||||
// AsOf is when the table was current. It is required: a rate with no date
|
||||
// cannot be assessed for staleness, and a stale rate silently moves every
|
||||
// threshold expressed in USD.
|
||||
AsOf time.Time
|
||||
// USDPer maps an ISO 4217 code to the USD value of one unit.
|
||||
USDPer map[string]float64
|
||||
// MaxAge is how old the table may be before conversion fails. Zero means no
|
||||
// limit, which is appropriate only where every amount is already USD.
|
||||
MaxAge time.Duration
|
||||
// Now supplies the current instant; nil means the wall clock.
|
||||
Now func() time.Time
|
||||
}
|
||||
|
||||
// USD converts an amount to USD.
|
||||
//
|
||||
// An unknown currency is an error. Returning the amount unchanged — the
|
||||
// conservative-looking choice — is not conservative: a currency worth more than a
|
||||
// dollar is understated, so 10,000 units of a currency worth three dollars is
|
||||
// assessed as 10,000 and sits below a 10,000-dollar reporting threshold that it
|
||||
// is in truth three times over.
|
||||
func (r Rates) USD(_ context.Context, amount float64, currency string) (float64, error) {
|
||||
code := strings.ToUpper(strings.TrimSpace(currency))
|
||||
if code == "" || code == "USD" {
|
||||
return amount, nil
|
||||
}
|
||||
rate, ok := r.USDPer[code]
|
||||
if !ok {
|
||||
return 0, fmt.Errorf("reference: no rate for currency %q, so its USD value is unknown", code)
|
||||
}
|
||||
if rate <= 0 {
|
||||
return 0, fmt.Errorf("reference: rate for %q is %v, which is not a price", code, rate)
|
||||
}
|
||||
if r.MaxAge > 0 {
|
||||
if age := r.now().Sub(r.AsOf); age > r.MaxAge {
|
||||
return 0, fmt.Errorf("reference: rate table is %s old, older than the %s limit", age.Truncate(time.Hour), r.MaxAge)
|
||||
}
|
||||
}
|
||||
return amount * rate, nil
|
||||
}
|
||||
|
||||
func (r Rates) now() time.Time {
|
||||
if r.Now != nil {
|
||||
return r.Now()
|
||||
}
|
||||
return time.Now().UTC()
|
||||
}
|
||||
|
||||
// Jurisdictions is a published listing of higher-risk countries.
|
||||
//
|
||||
// The two tiers mirror what standard-setters publish: countries for which
|
||||
// countermeasures are called for, and countries under increased monitoring. They
|
||||
// are separate because the required response differs, and collapsing them into one
|
||||
// "risky" flag loses the distinction a rule needs to choose between blocking and
|
||||
// enhanced review.
|
||||
//
|
||||
// Membership is not compiled in. The lists change several times a year, a
|
||||
// compiled-in copy is wrong within months of a release, and a wrong list in a
|
||||
// compliance product is worse than an absent one because it is trusted. The
|
||||
// operator loads the current lists and states the date they were current at.
|
||||
type Jurisdictions struct {
|
||||
AsOf time.Time
|
||||
// Action holds the codes calling for countermeasures.
|
||||
Action []string
|
||||
// Monitoring holds the codes under increased monitoring.
|
||||
Monitoring []string
|
||||
}
|
||||
|
||||
// Jurisdiction returns the tier for an ISO 3166-1 alpha-2 code: the empty string
|
||||
// if the country is on neither list.
|
||||
//
|
||||
// It returns an error when no listing has been loaded. An empty listing would
|
||||
// answer "not listed" for every country, which reads as a world with no high-risk
|
||||
// jurisdictions in it, and no operator would notice.
|
||||
func (j Jurisdictions) Jurisdiction(code string) (string, error) {
|
||||
if len(j.Action) == 0 && len(j.Monitoring) == 0 {
|
||||
return "", fmt.Errorf("reference: no jurisdiction listing loaded, so no country can be assessed")
|
||||
}
|
||||
if j.AsOf.IsZero() {
|
||||
return "", fmt.Errorf("reference: jurisdiction listing has no date, so its currency cannot be assessed")
|
||||
}
|
||||
want := strings.ToUpper(strings.TrimSpace(code))
|
||||
if want == "" {
|
||||
return "", nil
|
||||
}
|
||||
for _, c := range j.Action {
|
||||
if strings.EqualFold(strings.TrimSpace(c), want) {
|
||||
return "action", nil
|
||||
}
|
||||
}
|
||||
for _, c := range j.Monitoring {
|
||||
if strings.EqualFold(strings.TrimSpace(c), want) {
|
||||
return "monitoring", nil
|
||||
}
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// Age reports how old the listing is, for a readiness check.
|
||||
func (j Jurisdictions) Age(now time.Time) time.Duration {
|
||||
if j.AsOf.IsZero() {
|
||||
return 0
|
||||
}
|
||||
return now.Sub(j.AsOf)
|
||||
}
|
||||
@@ -0,0 +1,327 @@
|
||||
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
|
||||
// Use of this source code is governed by the Apache 2.0
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// Package replay runs a rule over real history and reports what it would have
|
||||
// done, before anyone activates it.
|
||||
//
|
||||
// JMLSG 5.7.18 requires functionality to implement and test new typologies
|
||||
// before live activation, and the FCA treats retiring a rule set as a governed
|
||||
// act that must be justified against performance outcomes including alert
|
||||
// intelligence-value and the false-positive proportion (FCG 3.2.5A). Those two
|
||||
// requirements are one question — how many alerts would this raise, on what, and
|
||||
// how does that compare with the rule it replaces — and this package answers it.
|
||||
//
|
||||
// # Dry run, structurally
|
||||
//
|
||||
// A sandbox that can mutate live state is not a sandbox. This package reaches
|
||||
// the engine through [Evaluator], whose one method evaluates and returns hits,
|
||||
// and history through [History], whose one method iterates. Neither can write:
|
||||
// there is no store here, no case, no alert and no webhook, and the package
|
||||
// imports nothing that has one.
|
||||
//
|
||||
// The evaluation itself is the engine's own, not a copy of it. A sandbox with its
|
||||
// own evaluator can disagree with production about what a rule does, which makes
|
||||
// its answer worthless; the only difference between a replay and the live path is
|
||||
// that the rule set is the candidate and nothing is written.
|
||||
//
|
||||
// # Refusals
|
||||
//
|
||||
// An empty history is refused rather than reported as zero alerts, because "no
|
||||
// alerts" is exactly how a quiet rule looks, and activating a rule on the
|
||||
// strength of an empty replay is the failure this package exists to prevent.
|
||||
// An evaluation error is refused for the same reason: a count taken from a rule
|
||||
// that did not evaluate is not a count.
|
||||
package replay
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/aml/pkg/types"
|
||||
)
|
||||
|
||||
// Errors.
|
||||
var (
|
||||
ErrNoEvaluator = errors.New("replay: no evaluator")
|
||||
ErrNoHistory = errors.New("replay: no history")
|
||||
ErrNoRule = errors.New("replay: candidate has no expression")
|
||||
ErrEmpty = errors.New("replay: history is empty, so a replay proves nothing")
|
||||
ErrEval = errors.New("replay: rule did not evaluate over history")
|
||||
)
|
||||
|
||||
// listed is how many transaction ids an outcome or a delta carries. The counts
|
||||
// are exact whatever this is; the lists are evidence for a human, and a human
|
||||
// reading a decision does not need the millionth id. A list shorter than its
|
||||
// count was cut here.
|
||||
const listed = 1000
|
||||
|
||||
// Evaluator evaluates a rule set against one event. *engine.Evaluator satisfies
|
||||
// it, which is the whole point: the candidate is scored by the code that scores
|
||||
// production.
|
||||
type Evaluator interface {
|
||||
EvalAll(rules []types.Rule, ctx types.EvalContext) []types.RuleHit
|
||||
}
|
||||
|
||||
// Disposition is what a human concluded about an event that alerted. It is what
|
||||
// makes a false-positive proportion measured rather than guessed.
|
||||
type Disposition string
|
||||
|
||||
const (
|
||||
// Unjudged is an event nobody has concluded anything about yet.
|
||||
Unjudged Disposition = ""
|
||||
// Productive is an alert that led somewhere: escalated, or reported.
|
||||
Productive Disposition = "productive"
|
||||
// Unproductive is an alert dismissed as not suspicious.
|
||||
Unproductive Disposition = "unproductive"
|
||||
)
|
||||
|
||||
// Event is one historical transaction, the entity as it stood, and what the live
|
||||
// system did with it.
|
||||
type Event struct {
|
||||
Tx types.Transaction `json:"tx"`
|
||||
Entity types.Entity `json:"entity"`
|
||||
// Alerted is the rule ids that actually alerted when this was live. It is
|
||||
// how a replayed count is compared against an observed one.
|
||||
Alerted []string `json:"alerted,omitempty"`
|
||||
// Disposition is the conclusion a human reached about this event. It attaches
|
||||
// to the event and not to each alert, so where several rules alerted on one
|
||||
// event every one of them inherits it.
|
||||
Disposition Disposition `json:"disposition,omitempty"`
|
||||
}
|
||||
|
||||
// History is a replayable window of real events.
|
||||
type History interface {
|
||||
// Each visits every event. It stops and returns the callback's error.
|
||||
Each(func(Event) error) error
|
||||
}
|
||||
|
||||
// Slice is a History held in memory.
|
||||
type Slice []Event
|
||||
|
||||
// Each visits every event in order.
|
||||
func (s Slice) Each(fn func(Event) error) error {
|
||||
for _, e := range s {
|
||||
if err := fn(e); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Outcome is what one rule would have done over the replayed history.
|
||||
type Outcome struct {
|
||||
Rule string `json:"rule"`
|
||||
// Alerts is how many events the rule would raise an alert on, and Txs is on
|
||||
// what. Txs is cut at a bound; Alerts never is.
|
||||
Alerts int `json:"alerts"`
|
||||
Txs []string `json:"txs,omitempty"`
|
||||
// Observed is how many events actually recorded an alert from this rule when
|
||||
// it was live. For a new rule it is zero, and a replayed count far from an
|
||||
// observed one on an existing rule means the rule or the history has moved.
|
||||
Observed int `json:"observed"`
|
||||
// Judged is how many of the replayed alerts fall on events a human has
|
||||
// concluded something about. The proportions below are over Judged, not over
|
||||
// Alerts.
|
||||
Judged int `json:"judged"`
|
||||
Productive int `json:"productive"`
|
||||
Unproductive int `json:"unproductive"`
|
||||
// FalsePositive and Intelligence are the two outcomes the FCA asks for when a
|
||||
// rule set is retired. They are absent, not zero, when nothing was judged: an
|
||||
// unmeasured proportion reported as 0.0 reads as a perfect rule.
|
||||
FalsePositive *float64 `json:"false_positive_proportion,omitempty"`
|
||||
Intelligence *float64 `json:"intelligence_value,omitempty"`
|
||||
}
|
||||
|
||||
// Delta is the candidate against the rule it replaces.
|
||||
type Delta struct {
|
||||
// Added is what the candidate alerts on and the incumbent does not — the new
|
||||
// coverage, and the new volume.
|
||||
Added []string `json:"added,omitempty"`
|
||||
// Dropped is what the incumbent alerts on and the candidate does not — the
|
||||
// coverage being given up, which is what a decommissioning has to justify.
|
||||
Dropped []string `json:"dropped,omitempty"`
|
||||
// Kept is what both alert on.
|
||||
Kept []string `json:"kept,omitempty"`
|
||||
Sizes struct {
|
||||
Added int `json:"added"`
|
||||
Dropped int `json:"dropped"`
|
||||
Kept int `json:"kept"`
|
||||
} `json:"counts"`
|
||||
}
|
||||
|
||||
// Report is the answer: what the candidate would have done, what the incumbent
|
||||
// did, and the difference.
|
||||
type Report struct {
|
||||
// Events is how many events were replayed, and From/To the period they span.
|
||||
// A window is empty when the events carry no timestamps, which is how a
|
||||
// caller-supplied sample is distinguishable from real history.
|
||||
Events int `json:"events"`
|
||||
From time.Time `json:"from,omitzero"`
|
||||
To time.Time `json:"to,omitzero"`
|
||||
|
||||
Candidate Outcome `json:"candidate"`
|
||||
Incumbent *Outcome `json:"incumbent,omitempty"`
|
||||
Delta *Delta `json:"delta,omitempty"`
|
||||
}
|
||||
|
||||
// Run replays the candidate over the history, and the incumbent alongside it
|
||||
// when one is given.
|
||||
//
|
||||
// Both rules are evaluated as activated whatever their enabled flag says: the
|
||||
// question is what happens on activation, and the flag governs the live path
|
||||
// rather than the question. Nothing is written anywhere.
|
||||
func Run(e Evaluator, h History, candidate types.Rule, incumbent *types.Rule) (Report, error) {
|
||||
if e == nil {
|
||||
return Report{}, ErrNoEvaluator
|
||||
}
|
||||
if h == nil {
|
||||
return Report{}, ErrNoHistory
|
||||
}
|
||||
if strings.TrimSpace(candidate.DSL) == "" {
|
||||
return Report{}, ErrNoRule
|
||||
}
|
||||
|
||||
cand := activated(candidate)
|
||||
report := Report{Candidate: Outcome{Rule: name(cand)}}
|
||||
|
||||
var prev *types.Rule
|
||||
if incumbent != nil {
|
||||
if strings.TrimSpace(incumbent.DSL) == "" {
|
||||
return Report{}, fmt.Errorf("%w: incumbent %s", ErrNoRule, name(*incumbent))
|
||||
}
|
||||
p := activated(*incumbent)
|
||||
prev = &p
|
||||
report.Incumbent = &Outcome{Rule: name(p)}
|
||||
report.Delta = &Delta{}
|
||||
}
|
||||
|
||||
err := h.Each(func(ev Event) error {
|
||||
report.Events++
|
||||
observe(&report, ev)
|
||||
|
||||
hit, err := fires(e, cand, ev)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if hit {
|
||||
record(&report.Candidate, ev)
|
||||
}
|
||||
if prev == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
was, err := fires(e, *prev, ev)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if was {
|
||||
record(report.Incumbent, ev)
|
||||
}
|
||||
switch {
|
||||
case hit && was:
|
||||
report.Delta.Kept = add(report.Delta.Kept, ev.Tx.ID)
|
||||
report.Delta.Sizes.Kept++
|
||||
case hit:
|
||||
report.Delta.Added = add(report.Delta.Added, ev.Tx.ID)
|
||||
report.Delta.Sizes.Added++
|
||||
case was:
|
||||
report.Delta.Dropped = add(report.Delta.Dropped, ev.Tx.ID)
|
||||
report.Delta.Sizes.Dropped++
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return Report{}, err
|
||||
}
|
||||
if report.Events == 0 {
|
||||
return Report{}, ErrEmpty
|
||||
}
|
||||
|
||||
proportions(&report.Candidate)
|
||||
if report.Incumbent != nil {
|
||||
proportions(report.Incumbent)
|
||||
}
|
||||
return report, nil
|
||||
}
|
||||
|
||||
// fires reports whether a rule would alert on an event, through the engine's own
|
||||
// evaluation. An evaluation error is the caller's answer, not a zero.
|
||||
func fires(e Evaluator, r types.Rule, ev Event) (bool, error) {
|
||||
hits := e.EvalAll([]types.Rule{r}, types.EvalContext{Tx: ev.Tx, Entity: ev.Entity})
|
||||
for _, h := range hits {
|
||||
if h.EvalErr != "" {
|
||||
return false, fmt.Errorf("%w: %s on transaction %s: %s", ErrEval, name(r), ev.Tx.ID, h.EvalErr)
|
||||
}
|
||||
}
|
||||
return len(hits) > 0, nil
|
||||
}
|
||||
|
||||
// activated is the rule as it would be if it were live. The jurisdiction and
|
||||
// asset-class filters are untouched, because the live path applies them too.
|
||||
func activated(r types.Rule) types.Rule {
|
||||
r.Enabled = true
|
||||
return r
|
||||
}
|
||||
|
||||
func name(r types.Rule) string {
|
||||
if r.ID != "" {
|
||||
return r.ID
|
||||
}
|
||||
return r.Name
|
||||
}
|
||||
|
||||
// observe counts what the live system actually did, so a replayed count can be
|
||||
// held against it.
|
||||
func observe(report *Report, ev Event) {
|
||||
if ts := ev.Tx.Timestamp; !ts.IsZero() {
|
||||
if report.From.IsZero() || ts.Before(report.From) {
|
||||
report.From = ts.UTC()
|
||||
}
|
||||
if ts.After(report.To) {
|
||||
report.To = ts.UTC()
|
||||
}
|
||||
}
|
||||
for _, id := range ev.Alerted {
|
||||
if id == report.Candidate.Rule {
|
||||
report.Candidate.Observed++
|
||||
}
|
||||
if report.Incumbent != nil && id == report.Incumbent.Rule {
|
||||
report.Incumbent.Observed++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func record(o *Outcome, ev Event) {
|
||||
o.Alerts++
|
||||
o.Txs = add(o.Txs, ev.Tx.ID)
|
||||
switch ev.Disposition {
|
||||
case Productive:
|
||||
o.Judged++
|
||||
o.Productive++
|
||||
case Unproductive:
|
||||
o.Judged++
|
||||
o.Unproductive++
|
||||
}
|
||||
}
|
||||
|
||||
// proportions computes the two numbers the FCA asks for, and leaves them absent
|
||||
// when there is nothing to compute them from.
|
||||
func proportions(o *Outcome) {
|
||||
if o.Judged == 0 {
|
||||
return
|
||||
}
|
||||
fp := float64(o.Unproductive) / float64(o.Judged)
|
||||
value := float64(o.Productive) / float64(o.Judged)
|
||||
o.FalsePositive = &fp
|
||||
o.Intelligence = &value
|
||||
}
|
||||
|
||||
func add(ids []string, id string) []string {
|
||||
if len(ids) >= listed {
|
||||
return ids
|
||||
}
|
||||
return append(ids, id)
|
||||
}
|
||||
@@ -0,0 +1,435 @@
|
||||
package replay
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/aml/pkg/types"
|
||||
)
|
||||
|
||||
// evaluator stands in for the engine's evaluator with the same contract: a
|
||||
// disabled rule is not evaluated, a broken rule yields a hit carrying its error,
|
||||
// and a match yields a hit. The real *engine.Evaluator is exercised against this
|
||||
// package at the wiring point, in pkg/api, where the two are compiled together.
|
||||
type evaluator struct {
|
||||
fire func(types.Rule, types.EvalContext) (bool, error)
|
||||
}
|
||||
|
||||
func (e evaluator) EvalAll(rules []types.Rule, ctx types.EvalContext) []types.RuleHit {
|
||||
var hits []types.RuleHit
|
||||
for _, r := range rules {
|
||||
if !r.Enabled {
|
||||
continue
|
||||
}
|
||||
ok, err := e.fire(r, ctx)
|
||||
if err != nil {
|
||||
hits = append(hits, types.RuleHit{Rule: r, Match: true, EvalErr: err.Error()})
|
||||
continue
|
||||
}
|
||||
if ok {
|
||||
hits = append(hits, types.RuleHit{Rule: r, Match: true})
|
||||
}
|
||||
}
|
||||
return hits
|
||||
}
|
||||
|
||||
// above fires a rule whose DSL names a threshold the notional passes.
|
||||
func above() evaluator {
|
||||
return evaluator{fire: func(r types.Rule, ctx types.EvalContext) (bool, error) {
|
||||
var threshold float64
|
||||
if _, err := fmt.Sscanf(r.DSL, "notional > %f", &threshold); err != nil {
|
||||
return false, fmt.Errorf("cannot compile %q", r.DSL)
|
||||
}
|
||||
return ctx.Tx.Notional > threshold, nil
|
||||
}}
|
||||
}
|
||||
|
||||
// selects fires each rule on a named set of transactions, which is how two
|
||||
// partially overlapping rules are stated without inventing a DSL.
|
||||
func selects(set map[string]map[string]bool) evaluator {
|
||||
return evaluator{fire: func(r types.Rule, ctx types.EvalContext) (bool, error) {
|
||||
if r.DSL == "" {
|
||||
return false, errors.New("empty expression")
|
||||
}
|
||||
return set[r.ID][ctx.Tx.ID], nil
|
||||
}}
|
||||
}
|
||||
|
||||
func rule(id, dsl string) types.Rule {
|
||||
return types.Rule{ID: id, Name: id, DSL: dsl, Enabled: true}
|
||||
}
|
||||
|
||||
func event(id string, notional float64, d Disposition, alerted ...string) Event {
|
||||
return Event{
|
||||
Tx: types.Transaction{
|
||||
ID: id,
|
||||
Notional: notional,
|
||||
Timestamp: time.Date(2026, time.March, 1, 12, 0, 0, 0, time.UTC),
|
||||
},
|
||||
Alerted: alerted,
|
||||
Disposition: d,
|
||||
}
|
||||
}
|
||||
|
||||
// TestEmptyHistoryIsRefused is the sharpest fail-closed property here. A replay
|
||||
// over nothing looks exactly like a quiet rule, and a rule activated on that
|
||||
// evidence is the failure this package exists to prevent.
|
||||
func TestEmptyHistoryIsRefused(t *testing.T) {
|
||||
for name, h := range map[string]History{
|
||||
"nil slice": Slice(nil),
|
||||
"empty slice": Slice{},
|
||||
} {
|
||||
report, err := Run(above(), h, rule("cand", "notional > 100"), nil)
|
||||
if !errors.Is(err, ErrEmpty) {
|
||||
t.Errorf("%s: err = %v, want ErrEmpty", name, err)
|
||||
}
|
||||
if report.Events != 0 || report.Candidate.Alerts != 0 {
|
||||
t.Errorf("%s: refused run still reported %+v", name, report)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestQuietRuleOverRealHistoryIsNotAnError: zero alerts over a non-empty history
|
||||
// is an answer, and the distinction from the empty case is the whole point.
|
||||
func TestQuietRuleOverRealHistoryIsNotAnError(t *testing.T) {
|
||||
h := Slice{event("tx-1", 10, Unjudged), event("tx-2", 20, Unjudged)}
|
||||
|
||||
report, err := Run(above(), h, rule("cand", "notional > 1000000"), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if report.Events != 2 {
|
||||
t.Errorf("events = %d, want 2", report.Events)
|
||||
}
|
||||
if report.Candidate.Alerts != 0 {
|
||||
t.Errorf("alerts = %d, want 0", report.Candidate.Alerts)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRefusesWithoutItsParts.
|
||||
func TestRefusesWithoutItsParts(t *testing.T) {
|
||||
h := Slice{event("tx-1", 10, Unjudged)}
|
||||
good := rule("cand", "notional > 5")
|
||||
|
||||
if _, err := Run(nil, h, good, nil); !errors.Is(err, ErrNoEvaluator) {
|
||||
t.Errorf("no evaluator: err = %v", err)
|
||||
}
|
||||
if _, err := Run(above(), nil, good, nil); !errors.Is(err, ErrNoHistory) {
|
||||
t.Errorf("no history: err = %v", err)
|
||||
}
|
||||
for _, dsl := range []string{"", " ", "\t\n"} {
|
||||
if _, err := Run(above(), h, rule("cand", dsl), nil); !errors.Is(err, ErrNoRule) {
|
||||
t.Errorf("candidate DSL %q: err = %v, want ErrNoRule", dsl, err)
|
||||
}
|
||||
}
|
||||
blank := rule("old", "")
|
||||
if _, err := Run(above(), h, good, &blank); !errors.Is(err, ErrNoRule) {
|
||||
t.Errorf("incumbent with no DSL: err = %v, want ErrNoRule", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCountsAlertsAndWhatTheyFellOn answers "how many, and on what".
|
||||
func TestCountsAlertsAndWhatTheyFellOn(t *testing.T) {
|
||||
h := Slice{
|
||||
event("tx-1", 15000, Unjudged),
|
||||
event("tx-2", 500, Unjudged),
|
||||
event("tx-3", 11000, Unjudged),
|
||||
}
|
||||
|
||||
report, err := Run(above(), h, rule("ctr", "notional > 10000"), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if report.Candidate.Alerts != 2 {
|
||||
t.Fatalf("alerts = %d, want 2", report.Candidate.Alerts)
|
||||
}
|
||||
if got := report.Candidate.Txs; len(got) != 2 || got[0] != "tx-1" || got[1] != "tx-3" {
|
||||
t.Errorf("txs = %v, want [tx-1 tx-3]", got)
|
||||
}
|
||||
if report.Candidate.Rule != "ctr" {
|
||||
t.Errorf("rule = %q, want ctr", report.Candidate.Rule)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCandidateIsEvaluatedAsActivated: a rule that is not enabled yet is exactly
|
||||
// the rule an operator wants to test, so the flag must not silently turn the
|
||||
// answer into zero.
|
||||
func TestCandidateIsEvaluatedAsActivated(t *testing.T) {
|
||||
h := Slice{event("tx-1", 15000, Unjudged)}
|
||||
|
||||
candidate := rule("ctr", "notional > 10000")
|
||||
candidate.Enabled = false
|
||||
incumbent := rule("old", "notional > 9000")
|
||||
incumbent.Enabled = false
|
||||
|
||||
report, err := Run(above(), h, candidate, &incumbent)
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if report.Candidate.Alerts != 1 {
|
||||
t.Errorf("candidate alerts = %d, want 1", report.Candidate.Alerts)
|
||||
}
|
||||
if report.Incumbent.Alerts != 1 {
|
||||
t.Errorf("incumbent alerts = %d, want 1", report.Incumbent.Alerts)
|
||||
}
|
||||
// The caller's rule is not modified by having been replayed.
|
||||
if candidate.Enabled || incumbent.Enabled {
|
||||
t.Error("Run enabled the caller's rule")
|
||||
}
|
||||
}
|
||||
|
||||
// TestDeltaAgainstTheRuleItReplaces is the comparison a decommissioning has to
|
||||
// justify: what is gained, what is given up, what is unchanged.
|
||||
func TestDeltaAgainstTheRuleItReplaces(t *testing.T) {
|
||||
h := Slice{
|
||||
event("both", 20000, Unjudged),
|
||||
event("added", 9800, Unjudged),
|
||||
event("dropped", 9600, Unjudged),
|
||||
event("neither", 100, Unjudged),
|
||||
}
|
||||
|
||||
// Two rules that overlap partially, which one threshold cannot express: the
|
||||
// candidate picks up something new and gives something up.
|
||||
which := selects(map[string]map[string]bool{
|
||||
"new": {"both": true, "added": true},
|
||||
"old": {"both": true, "dropped": true},
|
||||
})
|
||||
candidate := rule("new", "typology a")
|
||||
incumbent := rule("old", "typology b")
|
||||
|
||||
report, err := Run(which, h, candidate, &incumbent)
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if report.Delta == nil {
|
||||
t.Fatal("no delta against an incumbent")
|
||||
}
|
||||
if got := report.Delta.Kept; len(got) != 1 || got[0] != "both" {
|
||||
t.Errorf("kept = %v, want [both]", got)
|
||||
}
|
||||
if got := report.Delta.Added; len(got) != 1 || got[0] != "added" {
|
||||
t.Errorf("added = %v, want [added]", got)
|
||||
}
|
||||
if got := report.Delta.Dropped; len(got) != 1 || got[0] != "dropped" {
|
||||
t.Errorf("dropped = %v, want [dropped]", got)
|
||||
}
|
||||
if s := report.Delta.Sizes; s.Kept != 1 || s.Added != 1 || s.Dropped != 1 {
|
||||
t.Errorf("counts = %+v, want 1/1/1", s)
|
||||
}
|
||||
if report.Candidate.Alerts != 2 || report.Incumbent.Alerts != 2 {
|
||||
t.Errorf("alerts candidate=%d incumbent=%d, want 2/2", report.Candidate.Alerts, report.Incumbent.Alerts)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNoDeltaWithoutAnIncumbent: nothing is being replaced, so there is nothing
|
||||
// to compare and the report says so rather than inventing a baseline.
|
||||
func TestNoDeltaWithoutAnIncumbent(t *testing.T) {
|
||||
h := Slice{event("tx-1", 15000, Unjudged)}
|
||||
|
||||
report, err := Run(above(), h, rule("ctr", "notional > 10000"), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if report.Incumbent != nil || report.Delta != nil {
|
||||
t.Errorf("invented a baseline: incumbent=%+v delta=%+v", report.Incumbent, report.Delta)
|
||||
}
|
||||
}
|
||||
|
||||
// TestProportionsAreMeasuredOrAbsent: the two numbers the FCA asks for are
|
||||
// computed over judged alerts, and reported as absent — never as zero — when
|
||||
// nothing has been judged. A false-positive proportion of 0.0 reads as a perfect
|
||||
// rule, which is the wrong thing to tell someone about to activate it.
|
||||
func TestProportionsAreMeasuredOrAbsent(t *testing.T) {
|
||||
judged := Slice{
|
||||
event("a", 15000, Unproductive),
|
||||
event("b", 15000, Unproductive),
|
||||
event("c", 15000, Unproductive),
|
||||
event("d", 15000, Productive),
|
||||
event("e", 15000, Unjudged),
|
||||
event("f", 100, Productive), // does not alert, so does not count
|
||||
}
|
||||
|
||||
report, err := Run(above(), judged, rule("ctr", "notional > 10000"), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
o := report.Candidate
|
||||
if o.Alerts != 5 {
|
||||
t.Fatalf("alerts = %d, want 5", o.Alerts)
|
||||
}
|
||||
if o.Judged != 4 || o.Unproductive != 3 || o.Productive != 1 {
|
||||
t.Fatalf("judged=%d unproductive=%d productive=%d, want 4/3/1", o.Judged, o.Unproductive, o.Productive)
|
||||
}
|
||||
if o.FalsePositive == nil || *o.FalsePositive != 0.75 {
|
||||
t.Errorf("false positive proportion = %v, want 0.75", o.FalsePositive)
|
||||
}
|
||||
if o.Intelligence == nil || *o.Intelligence != 0.25 {
|
||||
t.Errorf("intelligence value = %v, want 0.25", o.Intelligence)
|
||||
}
|
||||
|
||||
unjudged := Slice{event("a", 15000, Unjudged), event("b", 15000, Unjudged)}
|
||||
report, err = Run(above(), unjudged, rule("ctr", "notional > 10000"), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if report.Candidate.Alerts != 2 {
|
||||
t.Fatalf("alerts = %d, want 2", report.Candidate.Alerts)
|
||||
}
|
||||
if report.Candidate.FalsePositive != nil || report.Candidate.Intelligence != nil {
|
||||
t.Errorf("unmeasured proportions reported as %v and %v, want absent",
|
||||
report.Candidate.FalsePositive, report.Candidate.Intelligence)
|
||||
}
|
||||
|
||||
// And absent survives the wire: no zero appears in the JSON.
|
||||
encoded, err := json.Marshal(report.Candidate)
|
||||
if err != nil {
|
||||
t.Fatalf("Marshal: %v", err)
|
||||
}
|
||||
for _, key := range []string{"false_positive_proportion", "intelligence_value"} {
|
||||
if bytes.Contains(encoded, []byte(key)) {
|
||||
t.Errorf("%s present in %s", key, encoded)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestEvaluationErrorRefusesTheRun: a rule that did not evaluate has no count,
|
||||
// and reporting the alerts it managed before failing would be a partial answer
|
||||
// presented as a whole one.
|
||||
func TestEvaluationErrorRefusesTheRun(t *testing.T) {
|
||||
h := Slice{
|
||||
event("tx-1", 15000, Unjudged),
|
||||
event("tx-2", 15000, Unjudged),
|
||||
}
|
||||
|
||||
report, err := Run(above(), h, rule("broken", "this is not a threshold"), nil)
|
||||
if !errors.Is(err, ErrEval) {
|
||||
t.Fatalf("err = %v, want ErrEval", err)
|
||||
}
|
||||
if report.Events != 0 || report.Candidate.Alerts != 0 {
|
||||
t.Fatalf("failed run reported %+v", report)
|
||||
}
|
||||
|
||||
// The same for the incumbent: half a comparison is not a comparison.
|
||||
broken := rule("brokenOld", "not a threshold either")
|
||||
if _, err := Run(above(), h, rule("ctr", "notional > 10000"), &broken); !errors.Is(err, ErrEval) {
|
||||
t.Errorf("broken incumbent: err = %v, want ErrEval", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestObservedComesFromHistory: what the live system actually did, so a replayed
|
||||
// count can be held against it.
|
||||
func TestObservedComesFromHistory(t *testing.T) {
|
||||
h := Slice{
|
||||
event("tx-1", 15000, Unjudged, "old", "other"),
|
||||
event("tx-2", 15000, Unjudged, "old"),
|
||||
event("tx-3", 100, Unjudged, "other"),
|
||||
}
|
||||
|
||||
incumbent := rule("old", "notional > 10000")
|
||||
report, err := Run(above(), h, rule("new", "notional > 12000"), &incumbent)
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if report.Incumbent.Observed != 2 {
|
||||
t.Errorf("incumbent observed = %d, want 2", report.Incumbent.Observed)
|
||||
}
|
||||
if report.Candidate.Observed != 0 {
|
||||
t.Errorf("candidate observed = %d, want 0 (it was never live)", report.Candidate.Observed)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWindowSpansTheHistory: the period replayed is reported, so nobody mistakes
|
||||
// a handful of synthetic events for ninety days of traffic.
|
||||
func TestWindowSpansTheHistory(t *testing.T) {
|
||||
first := time.Date(2026, time.January, 2, 0, 0, 0, 0, time.UTC)
|
||||
last := time.Date(2026, time.June, 30, 0, 0, 0, 0, time.UTC)
|
||||
h := Slice{
|
||||
{Tx: types.Transaction{ID: "b", Notional: 15000, Timestamp: last}},
|
||||
{Tx: types.Transaction{ID: "a", Notional: 15000, Timestamp: first}},
|
||||
}
|
||||
|
||||
report, err := Run(above(), h, rule("ctr", "notional > 10000"), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if !report.From.Equal(first) || !report.To.Equal(last) {
|
||||
t.Errorf("window %s..%s, want %s..%s", report.From, report.To, first, last)
|
||||
}
|
||||
|
||||
// A sample with no timestamps has no window, and says so.
|
||||
sample := Slice{{Tx: types.Transaction{ID: "synthetic", Notional: 15000}}}
|
||||
report, err = Run(above(), sample, rule("ctr", "notional > 10000"), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if !report.From.IsZero() || !report.To.IsZero() {
|
||||
t.Errorf("synthetic sample claims a window %s..%s", report.From, report.To)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHistoryIsUnchanged: a sandbox that can mutate live state is not a sandbox,
|
||||
// and the history is the closest thing to live state a replay can reach.
|
||||
func TestHistoryIsUnchanged(t *testing.T) {
|
||||
h := Slice{
|
||||
event("tx-1", 15000, Unproductive, "old"),
|
||||
event("tx-2", 100, Productive, "other"),
|
||||
}
|
||||
before, err := json.Marshal(h)
|
||||
if err != nil {
|
||||
t.Fatalf("Marshal: %v", err)
|
||||
}
|
||||
|
||||
incumbent := rule("old", "notional > 9000")
|
||||
if _, err := Run(above(), h, rule("new", "notional > 10000"), &incumbent); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
|
||||
after, err := json.Marshal(h)
|
||||
if err != nil {
|
||||
t.Fatalf("Marshal: %v", err)
|
||||
}
|
||||
if string(before) != string(after) {
|
||||
t.Fatalf("history changed:\nbefore %s\nafter %s", before, after)
|
||||
}
|
||||
}
|
||||
|
||||
// TestListsAreBoundedAndCountsAreNot: the evidence a human reads is cut, the
|
||||
// number they decide on is not.
|
||||
func TestListsAreBoundedAndCountsAreNot(t *testing.T) {
|
||||
h := make(Slice, listed+500)
|
||||
for i := range h {
|
||||
h[i] = event(fmt.Sprintf("tx-%d", i), 15000, Unjudged)
|
||||
}
|
||||
|
||||
report, err := Run(above(), h, rule("ctr", "notional > 10000"), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if report.Candidate.Alerts != len(h) {
|
||||
t.Errorf("alerts = %d, want %d", report.Candidate.Alerts, len(h))
|
||||
}
|
||||
if len(report.Candidate.Txs) != listed {
|
||||
t.Errorf("listed %d transactions, want %d", len(report.Candidate.Txs), listed)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHistoryErrorSurfaces: a history that cannot be read is not an empty
|
||||
// history, and the difference must not be swallowed.
|
||||
func TestHistoryErrorSurfaces(t *testing.T) {
|
||||
unreadable := errors.New("sealed record does not open")
|
||||
report, err := Run(above(), broken{unreadable}, rule("ctr", "notional > 10000"), nil)
|
||||
if !errors.Is(err, unreadable) {
|
||||
t.Fatalf("err = %v, want the history's own error", err)
|
||||
}
|
||||
if report.Events != 0 {
|
||||
t.Errorf("failed read reported %d events", report.Events)
|
||||
}
|
||||
}
|
||||
|
||||
type broken struct{ err error }
|
||||
|
||||
func (b broken) Each(func(Event) error) error { return b.err }
|
||||
@@ -0,0 +1,29 @@
|
||||
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
|
||||
// Use of this source code is governed by the Apache 2.0
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package retention
|
||||
|
||||
import (
|
||||
"log"
|
||||
"time"
|
||||
|
||||
"github.com/hanzoai/base/core"
|
||||
)
|
||||
|
||||
// Cron registers the daily disposal at 03:30 UTC.
|
||||
//
|
||||
// A run that cannot prove what it destroyed reports nothing as disposed and logs
|
||||
// the failure, and the next run tries again. Deletion is an obligation with a
|
||||
// deadline measured in years, so a day of retries is safe where a false "done"
|
||||
// is not.
|
||||
func Cron(app core.App, l *Ledger) {
|
||||
app.Cron().Add("aml-retention-dispose", "30 3 * * *", func() {
|
||||
d, err := l.Dispose(time.Now().UTC())
|
||||
if err != nil {
|
||||
log.Printf("[aml-retention] disposal not proven, nothing reported as disposed: %v", err)
|
||||
return
|
||||
}
|
||||
log.Printf("[aml-retention] examined %d records, disposed %d", d.Examined, len(d.Disposed))
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,480 @@
|
||||
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
|
||||
// Use of this source code is governed by the Apache 2.0
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package retention
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// Ledger holds an org's retained records, the party index the five-year
|
||||
// lookback is answered from, and the relationship index the clock cascades
|
||||
// through.
|
||||
type Ledger struct {
|
||||
mu sync.RWMutex
|
||||
records map[string]*Record
|
||||
// parties maps org and party to the records that name that party. AMLR
|
||||
// Art. 78 must be answered "fully and speedily", which is an index and not a
|
||||
// scan: a lookback examines one party's records, never the ledger.
|
||||
parties map[string][]string
|
||||
// inside maps a relationship to the records retained inside it, so ending the
|
||||
// relationship starts their clocks too.
|
||||
inside map[string][]string
|
||||
}
|
||||
|
||||
// New returns an empty ledger.
|
||||
func New() *Ledger {
|
||||
return &Ledger{
|
||||
records: make(map[string]*Record),
|
||||
parties: make(map[string][]string),
|
||||
inside: make(map[string][]string),
|
||||
}
|
||||
}
|
||||
|
||||
func partyKey(org, party string) string { return org + "\x00" + party }
|
||||
|
||||
// Retain writes a record and returns its id.
|
||||
//
|
||||
// The caller supplies what happened; the ledger supplies the clock. Which
|
||||
// Art. 77(3) trigger applies is the caller's statement of fact — this is a
|
||||
// refusal, this is an occasional transaction, this sits inside that relationship
|
||||
// — and where the clock starts follows from it:
|
||||
//
|
||||
// - a refusal and an occasional transaction start at the event;
|
||||
// - a record inside a relationship inherits that relationship's clock, which
|
||||
// starts when the relationship ends;
|
||||
// - a business relationship itself has no clock until [Ledger.Close].
|
||||
//
|
||||
// So Record.Start is not an input. A record arriving with one set is refused,
|
||||
// because a caller that can name its own expiry can name one that never comes.
|
||||
func (l *Ledger) Retain(r Record) (string, error) {
|
||||
if r.Org == "" {
|
||||
return "", ErrOrg
|
||||
}
|
||||
if !r.Start.IsZero() {
|
||||
return "", fmt.Errorf("%w: the clock is the ledger's to start", ErrClock)
|
||||
}
|
||||
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
|
||||
switch r.Trigger {
|
||||
case TriggerRefusal, TriggerOccasional:
|
||||
if r.Relationship != "" {
|
||||
return "", fmt.Errorf("%w: %s does not run inside a relationship", ErrTrigger, r.Trigger)
|
||||
}
|
||||
r.Start = r.Occurred
|
||||
case TriggerRelationshipEnd:
|
||||
if r.Relationship == "" {
|
||||
// The relationship itself. Its clock starts when it ends.
|
||||
if r.Class != ClassRelationship {
|
||||
return "", fmt.Errorf("%w: %s needs the relationship it is retained inside", ErrRelationship, r.Class)
|
||||
}
|
||||
break
|
||||
}
|
||||
rel, ok := l.records[r.Relationship]
|
||||
if !ok || rel.Org != r.Org || rel.Class != ClassRelationship {
|
||||
return "", fmt.Errorf("%w: %s", ErrRelationship, r.Relationship)
|
||||
}
|
||||
r.Start = rel.Start
|
||||
default:
|
||||
return "", fmt.Errorf("%w: %q", ErrTrigger, r.Trigger)
|
||||
}
|
||||
|
||||
if err := r.validate(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
r.ID = uuid.NewString()
|
||||
r.Written = time.Now().UTC()
|
||||
// One representation of a moment, everywhere in the ledger.
|
||||
r.Occurred = r.Occurred.UTC()
|
||||
if !r.Start.IsZero() {
|
||||
r.Start = r.Start.UTC()
|
||||
}
|
||||
|
||||
stored := r.clone()
|
||||
l.records[r.ID] = &stored
|
||||
for _, p := range r.Parties {
|
||||
k := partyKey(r.Org, p)
|
||||
l.parties[k] = append(l.parties[k], r.ID)
|
||||
}
|
||||
if r.Relationship != "" {
|
||||
l.inside[r.Relationship] = append(l.inside[r.Relationship], r.ID)
|
||||
}
|
||||
return r.ID, nil
|
||||
}
|
||||
|
||||
// Close ends a business relationship and starts the retention clock on it and on
|
||||
// every record retained inside it. It returns how many records' clocks started,
|
||||
// the relationship included: a caller that expects the cascade can check it.
|
||||
func (l *Ledger) Close(org, id string, at time.Time) (int, error) {
|
||||
if org == "" {
|
||||
return 0, ErrOrg
|
||||
}
|
||||
if at.IsZero() {
|
||||
return 0, ErrOccurred
|
||||
}
|
||||
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
|
||||
rel, ok := l.records[id]
|
||||
if !ok || rel.Org != org || rel.Class != ClassRelationship {
|
||||
return 0, fmt.Errorf("%w: %s", ErrRelationship, id)
|
||||
}
|
||||
if !rel.Ended.IsZero() {
|
||||
return 0, fmt.Errorf("%w: %s ended %s", ErrClosed, id, rel.Ended.UTC().Format(time.RFC3339))
|
||||
}
|
||||
if at.Before(rel.Occurred) {
|
||||
return 0, fmt.Errorf("%w: %s cannot end before it began", ErrOccurred, id)
|
||||
}
|
||||
|
||||
at = at.UTC()
|
||||
rel.Ended = at
|
||||
rel.Start = at
|
||||
started := 1
|
||||
|
||||
for _, child := range l.inside[id] {
|
||||
r, ok := l.records[child]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
// Only a clock that has not started; an extended or already-started
|
||||
// record is never moved backwards.
|
||||
if r.Start.IsZero() {
|
||||
r.Start = at
|
||||
started++
|
||||
}
|
||||
}
|
||||
return started, nil
|
||||
}
|
||||
|
||||
// Extend applies a case-by-case extension of a retention period, capped at five
|
||||
// further years (AMLD4 Art. 40(1)). It is refused without a reason and a
|
||||
// decider, because "case by case" means somebody decided this case, and refused
|
||||
// outright if the ask exceeds the cap rather than silently shortened to it.
|
||||
func (l *Ledger) Extend(org, id string, by time.Duration, reason, who string) error {
|
||||
if org == "" {
|
||||
return ErrOrg
|
||||
}
|
||||
if by <= 0 || reason == "" || who == "" {
|
||||
return fmt.Errorf("%w: needs a positive period, a reason and a decider", ErrExtension)
|
||||
}
|
||||
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
|
||||
r, ok := l.records[id]
|
||||
if !ok || r.Org != org {
|
||||
return fmt.Errorf("%w: %s", ErrNotFound, id)
|
||||
}
|
||||
if r.Start.IsZero() {
|
||||
return fmt.Errorf("%w: %s", ErrClock, id)
|
||||
}
|
||||
if r.Extended != nil {
|
||||
return fmt.Errorf("%w: %s is already extended", ErrExtension, id)
|
||||
}
|
||||
limit := r.Start.AddDate(Period+Extension, 0, 0)
|
||||
if r.Start.AddDate(Period, 0, 0).Add(by).After(limit) {
|
||||
return fmt.Errorf("%w: %s past %s", ErrExtension, by, limit.UTC().Format(time.RFC3339))
|
||||
}
|
||||
|
||||
r.Extended = &Extended{By: by, Reason: reason, Who: who, At: time.Now().UTC()}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Get returns one record for a permitted purpose. The returned record is a copy:
|
||||
// a reader cannot reach back into the ledger and unmake part of it.
|
||||
func (l *Ledger) Get(p Purpose, org, id string) (Record, error) {
|
||||
if !p.permitted() {
|
||||
return Record{}, fmt.Errorf("%w: %q", ErrPurpose, p)
|
||||
}
|
||||
if org == "" {
|
||||
return Record{}, ErrOrg
|
||||
}
|
||||
|
||||
l.mu.RLock()
|
||||
defer l.mu.RUnlock()
|
||||
|
||||
r, ok := l.records[id]
|
||||
if !ok || r.Org != org {
|
||||
return Record{}, fmt.Errorf("%w: %s", ErrNotFound, id)
|
||||
}
|
||||
return r.clone(), nil
|
||||
}
|
||||
|
||||
// Each visits an org's records of one class, oldest event first, for a permitted
|
||||
// purpose. An empty class visits every class. Iteration is ordered so that a
|
||||
// file produced from it is reproducible.
|
||||
func (l *Ledger) Each(p Purpose, org string, c Class, fn func(Record) error) error {
|
||||
if !p.permitted() {
|
||||
return fmt.Errorf("%w: %q", ErrPurpose, p)
|
||||
}
|
||||
if org == "" {
|
||||
return ErrOrg
|
||||
}
|
||||
|
||||
// The order is settled first, on the cheap fields, and each record is copied
|
||||
// as it is visited. A walk therefore does not double the ledger in memory,
|
||||
// and it does not hold the lock across the caller's callback.
|
||||
type key struct {
|
||||
id string
|
||||
at time.Time
|
||||
}
|
||||
|
||||
l.mu.RLock()
|
||||
keys := make([]key, 0, len(l.records))
|
||||
for id, r := range l.records {
|
||||
if r.Org != org || (c != "" && r.Class != c) {
|
||||
continue
|
||||
}
|
||||
keys = append(keys, key{id: id, at: r.Occurred})
|
||||
}
|
||||
l.mu.RUnlock()
|
||||
|
||||
slices.SortFunc(keys, func(a, b key) int {
|
||||
if d := a.at.Compare(b.at); d != 0 {
|
||||
return d
|
||||
}
|
||||
return strings.Compare(a.id, b.id)
|
||||
})
|
||||
|
||||
for _, k := range keys {
|
||||
l.mu.RLock()
|
||||
stored, ok := l.records[k.id]
|
||||
var r Record
|
||||
if ok {
|
||||
r = stored.clone()
|
||||
}
|
||||
l.mu.RUnlock()
|
||||
if !ok {
|
||||
// Disposed between settling the order and reading it. A destroyed
|
||||
// record is not visited.
|
||||
continue
|
||||
}
|
||||
if err := fn(r); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Answer answers AMLR Art. 78: whether a business relationship with a named
|
||||
// person is or was maintained within the prior five years, and its nature. It
|
||||
// does not echo the party back, because the caller asked about a name and the
|
||||
// ledger only ever saw a pseudonym.
|
||||
type Answer struct {
|
||||
// Maintained is the answer: is, or was within the window.
|
||||
Maintained bool `json:"maintained"`
|
||||
// Current distinguishes "is" from "was".
|
||||
Current bool `json:"current"`
|
||||
// Natures is the nature of each relationship found, which Art. 78 requires
|
||||
// alongside its existence.
|
||||
Natures []string `json:"natures,omitempty"`
|
||||
// Records are the relationship records behind the answer, so the file can be
|
||||
// produced from it.
|
||||
Records []string `json:"records,omitempty"`
|
||||
From time.Time `json:"from"`
|
||||
To time.Time `json:"to"`
|
||||
// Examined is how many of this party's records were looked at. It is the
|
||||
// evidence that the answer came from an index and not from a scan: it does
|
||||
// not grow with the size of the ledger.
|
||||
Examined int `json:"examined"`
|
||||
}
|
||||
|
||||
// Lookback answers Art. 78 for one party. The party is the pseudonym the records
|
||||
// were indexed under, so the question is answerable without the ledger holding
|
||||
// the name in the clear.
|
||||
func (l *Ledger) Lookback(p Purpose, org, party string, now time.Time) (Answer, error) {
|
||||
if !p.permitted() {
|
||||
return Answer{}, fmt.Errorf("%w: %q", ErrPurpose, p)
|
||||
}
|
||||
if org == "" {
|
||||
return Answer{}, ErrOrg
|
||||
}
|
||||
if party == "" {
|
||||
return Answer{}, ErrParties
|
||||
}
|
||||
if err := checkNow(now); err != nil {
|
||||
return Answer{}, err
|
||||
}
|
||||
|
||||
now = now.UTC()
|
||||
answer := Answer{
|
||||
From: now.AddDate(-Period, 0, 0),
|
||||
To: now,
|
||||
}
|
||||
|
||||
l.mu.RLock()
|
||||
defer l.mu.RUnlock()
|
||||
|
||||
ids := l.parties[partyKey(org, party)]
|
||||
answer.Examined = len(ids)
|
||||
|
||||
natures := make(map[string]bool)
|
||||
for _, id := range ids {
|
||||
r, ok := l.records[id]
|
||||
if !ok || r.Class != ClassRelationship {
|
||||
continue
|
||||
}
|
||||
// Open now, or ended inside the window.
|
||||
current := r.Ended.IsZero()
|
||||
if !current && r.Ended.Before(answer.From) {
|
||||
continue
|
||||
}
|
||||
if r.Occurred.After(now) {
|
||||
continue
|
||||
}
|
||||
answer.Maintained = true
|
||||
answer.Current = answer.Current || current
|
||||
answer.Records = append(answer.Records, id)
|
||||
if r.Nature != "" {
|
||||
natures[r.Nature] = true
|
||||
}
|
||||
}
|
||||
|
||||
answer.Natures = make([]string, 0, len(natures))
|
||||
for n := range natures {
|
||||
answer.Natures = append(answer.Natures, n)
|
||||
}
|
||||
slices.Sort(answer.Natures)
|
||||
slices.Sort(answer.Records)
|
||||
if len(answer.Natures) == 0 {
|
||||
answer.Natures = nil
|
||||
}
|
||||
return answer, nil
|
||||
}
|
||||
|
||||
// Disposal is what a disposal run proves. Its fields are meaningful only when
|
||||
// Dispose returned no error: a run that cannot verify its own effect reports the
|
||||
// error and no success.
|
||||
type Disposal struct {
|
||||
At time.Time `json:"at"`
|
||||
Examined int `json:"examined"`
|
||||
Disposed []string `json:"disposed,omitempty"`
|
||||
}
|
||||
|
||||
// Dispose destroys every record whose retention period has expired, together
|
||||
// with every index entry that referenced it, and proves it before returning.
|
||||
//
|
||||
// Fail-closed in three ways. It refuses a date the caller's own clock has not
|
||||
// reached, so a clock lie cannot bring destruction forward. It verifies its own
|
||||
// post-conditions and returns an error rather than a count if any of them fails.
|
||||
// And it deletes whole records only — there is no partial disposal, which is
|
||||
// what keeps deletion-on-expiry from becoming redaction.
|
||||
func (l *Ledger) Dispose(now time.Time) (Disposal, error) {
|
||||
if err := checkNow(now); err != nil {
|
||||
return Disposal{}, err
|
||||
}
|
||||
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
|
||||
d := Disposal{At: now.UTC(), Examined: len(l.records)}
|
||||
|
||||
var doomed []string
|
||||
for id, r := range l.records {
|
||||
if r.Expired(now) {
|
||||
doomed = append(doomed, id)
|
||||
}
|
||||
}
|
||||
slices.Sort(doomed)
|
||||
|
||||
for _, id := range doomed {
|
||||
r := l.records[id]
|
||||
delete(l.records, id)
|
||||
for _, party := range r.Parties {
|
||||
k := partyKey(r.Org, party)
|
||||
if rest := drop(l.parties[k], id); len(rest) == 0 {
|
||||
delete(l.parties, k)
|
||||
} else {
|
||||
l.parties[k] = rest
|
||||
}
|
||||
}
|
||||
if r.Relationship != "" {
|
||||
if rest := drop(l.inside[r.Relationship], id); len(rest) == 0 {
|
||||
delete(l.inside, r.Relationship)
|
||||
} else {
|
||||
l.inside[r.Relationship] = rest
|
||||
}
|
||||
}
|
||||
// A disposed relationship stops indexing what was retained inside it.
|
||||
// Anything of its own that outlives it did so by extension, on purpose.
|
||||
delete(l.inside, id)
|
||||
}
|
||||
d.Disposed = doomed
|
||||
|
||||
if err := l.proveLocked(doomed, now); err != nil {
|
||||
return Disposal{}, err
|
||||
}
|
||||
return d, nil
|
||||
}
|
||||
|
||||
// proveLocked is the post-condition of a disposal run. It is the difference
|
||||
// between "the delete statement ran" and "the right records are gone".
|
||||
func (l *Ledger) proveLocked(disposed []string, now time.Time) error {
|
||||
gone := make(map[string]bool, len(disposed))
|
||||
for _, id := range disposed {
|
||||
gone[id] = true
|
||||
}
|
||||
|
||||
for _, id := range disposed {
|
||||
if _, ok := l.records[id]; ok {
|
||||
return fmt.Errorf("%w: %s survived disposal", ErrDisposal, id)
|
||||
}
|
||||
if _, ok := l.inside[id]; ok {
|
||||
return fmt.Errorf("%w: %s still indexes records retained inside it", ErrDisposal, id)
|
||||
}
|
||||
}
|
||||
for _, ids := range l.parties {
|
||||
for _, id := range ids {
|
||||
if gone[id] {
|
||||
return fmt.Errorf("%w: %s is still indexed under a party", ErrDisposal, id)
|
||||
}
|
||||
if _, ok := l.records[id]; !ok {
|
||||
return fmt.Errorf("%w: party index references vanished record %s", ErrDisposal, id)
|
||||
}
|
||||
}
|
||||
}
|
||||
for rel, ids := range l.inside {
|
||||
if _, ok := l.records[rel]; !ok {
|
||||
return fmt.Errorf("%w: relationship index references vanished record %s", ErrDisposal, rel)
|
||||
}
|
||||
for _, id := range ids {
|
||||
if _, ok := l.records[id]; !ok {
|
||||
return fmt.Errorf("%w: relationship %s references vanished record %s", ErrDisposal, rel, id)
|
||||
}
|
||||
}
|
||||
}
|
||||
for id, r := range l.records {
|
||||
if r.Expired(now) {
|
||||
return fmt.Errorf("%w: %s is expired and was not disposed", ErrDisposal, id)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func drop(ids []string, id string) []string {
|
||||
out := ids[:0:0]
|
||||
for _, existing := range ids {
|
||||
if existing != id {
|
||||
out = append(out, existing)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Len is how many records the ledger holds.
|
||||
func (l *Ledger) Len() int {
|
||||
l.mu.RLock()
|
||||
defer l.mu.RUnlock()
|
||||
return len(l.records)
|
||||
}
|
||||
@@ -0,0 +1,835 @@
|
||||
package retention
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
const org = "acme"
|
||||
|
||||
// base is fixed for the run so that ago(n) is one moment however often it is
|
||||
// called, and an expiry computed in an assertion is the expiry the ledger stored.
|
||||
var base = time.Now().UTC().Truncate(time.Second)
|
||||
|
||||
func ago(years int) time.Time { return base.AddDate(-years, 0, 0) }
|
||||
|
||||
func body(s string) []byte { return []byte(`{"tx":"` + s + `"}`) }
|
||||
|
||||
// The four shapes a caller retains, as helpers. The ledger has one write method
|
||||
// that takes a whole record; these keep the tests reading like the instrument.
|
||||
|
||||
func open(l *Ledger, o, nature string, parties []string, at time.Time) (string, error) {
|
||||
return l.Retain(Record{
|
||||
Org: o, Class: ClassRelationship, Trigger: TriggerRelationshipEnd,
|
||||
Nature: nature, Parties: parties, Occurred: at, Body: body("rel"),
|
||||
})
|
||||
}
|
||||
|
||||
func transact(l *Ledger, o, rel, ref string, parties []string, at time.Time, b []byte) (string, error) {
|
||||
trigger := TriggerOccasional
|
||||
if rel != "" {
|
||||
trigger = TriggerRelationshipEnd
|
||||
}
|
||||
return l.Retain(Record{
|
||||
Org: o, Class: ClassTransaction, Trigger: trigger, Relationship: rel,
|
||||
Ref: ref, Parties: parties, Occurred: at, Body: b,
|
||||
})
|
||||
}
|
||||
|
||||
func refuse(l *Ledger, o, ref string, parties []string, at time.Time, reason string) (string, error) {
|
||||
return l.Retain(Record{
|
||||
Org: o, Class: ClassRefusal, Trigger: TriggerRefusal,
|
||||
Ref: ref, Parties: parties, Occurred: at, Reason: reason, Body: body("refused"),
|
||||
})
|
||||
}
|
||||
|
||||
func assess(l *Ledger, o, rel, ref string, parties []string, a Assessment) (string, error) {
|
||||
trigger := TriggerOccasional
|
||||
if rel != "" {
|
||||
trigger = TriggerRelationshipEnd
|
||||
}
|
||||
return l.Retain(Record{
|
||||
Org: o, Class: ClassAssessment, Trigger: trigger, Relationship: rel,
|
||||
Ref: ref, Parties: parties, Occurred: a.At, Assessment: &a, Body: body("assessed"),
|
||||
})
|
||||
}
|
||||
|
||||
func mustGet(t *testing.T, l *Ledger, id string) Record {
|
||||
t.Helper()
|
||||
r, err := l.Get(PurposeInvestigation, org, id)
|
||||
if err != nil {
|
||||
t.Fatalf("Get(%s): %v", id, err)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// TestThreeTriggers is AMLR Art. 77(3) in full: five years from the end of the
|
||||
// relationship, from the occasional transaction, or from the date of refusal.
|
||||
// The third is the one implementations miss, so it is asserted like the others.
|
||||
func TestThreeTriggers(t *testing.T) {
|
||||
l := New()
|
||||
|
||||
// From the occasional transaction.
|
||||
occasional, err := transact(l, org, "", "tx-occasional", []string{"subject:p1"}, ago(4), body("occ"))
|
||||
if err != nil {
|
||||
t.Fatalf("occasional transaction: %v", err)
|
||||
}
|
||||
r := mustGet(t, l, occasional)
|
||||
if r.Trigger != TriggerOccasional {
|
||||
t.Errorf("occasional trigger = %q", r.Trigger)
|
||||
}
|
||||
if want := ago(4).AddDate(Period, 0, 0); !r.Expiry().Equal(want) {
|
||||
t.Errorf("occasional expiry = %s, want %s", r.Expiry(), want)
|
||||
}
|
||||
|
||||
// From the date of refusal.
|
||||
refused, err := refuse(l, org, "tx-refused", []string{"subject:p2"}, ago(4), "identification impossible")
|
||||
if err != nil {
|
||||
t.Fatalf("refusal: %v", err)
|
||||
}
|
||||
r = mustGet(t, l, refused)
|
||||
if r.Trigger != TriggerRefusal || r.Class != ClassRefusal {
|
||||
t.Errorf("refusal trigger/class = %q/%q", r.Trigger, r.Class)
|
||||
}
|
||||
if r.Reason != "identification impossible" {
|
||||
t.Errorf("refusal reason = %q", r.Reason)
|
||||
}
|
||||
if want := ago(4).AddDate(Period, 0, 0); !r.Expiry().Equal(want) {
|
||||
t.Errorf("refusal expiry = %s, want %s", r.Expiry(), want)
|
||||
}
|
||||
|
||||
// From the end of the business relationship.
|
||||
rel, err := open(l, org, "payments", []string{"subject:p3"}, ago(9))
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
if got := mustGet(t, l, rel).Expiry(); !got.IsZero() {
|
||||
t.Fatalf("open relationship expires at %s, want never", got)
|
||||
}
|
||||
if _, err := l.Close(org, rel, ago(4)); err != nil {
|
||||
t.Fatalf("Close: %v", err)
|
||||
}
|
||||
r = mustGet(t, l, rel)
|
||||
if r.Trigger != TriggerRelationshipEnd {
|
||||
t.Errorf("relationship trigger = %q", r.Trigger)
|
||||
}
|
||||
if want := ago(4).AddDate(Period, 0, 0); !r.Expiry().Equal(want) {
|
||||
t.Errorf("relationship expiry = %s, want %s", r.Expiry(), want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRefusalNeedsAReason: a refusal with no reason answers nothing about why
|
||||
// the firm refrained, which is the point of retaining it.
|
||||
func TestRefusalNeedsAReason(t *testing.T) {
|
||||
l := New()
|
||||
if _, err := refuse(l, org, "tx-1", []string{"subject:p1"}, ago(1), ""); !errors.Is(err, ErrReason) {
|
||||
t.Fatalf("err = %v, want ErrReason", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTheClockIsNotTheCallersToSet: a caller that can name its own expiry can
|
||||
// name one that never arrives, so Retain refuses a record that brings a clock.
|
||||
func TestTheClockIsNotTheCallersToSet(t *testing.T) {
|
||||
l := New()
|
||||
_, err := l.Retain(Record{
|
||||
Org: org, Class: ClassRefusal, Trigger: TriggerRefusal, Ref: "tx-1",
|
||||
Parties: []string{"subject:p1"}, Occurred: ago(6), Reason: "refused",
|
||||
Body: body("x"),
|
||||
// Six years ago, so the record would arrive already expired; a hundred
|
||||
// years hence and it would never expire. Neither is the caller's to say.
|
||||
Start: base.AddDate(100, 0, 0),
|
||||
})
|
||||
if !errors.Is(err, ErrClock) {
|
||||
t.Fatalf("err = %v, want ErrClock", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTriggerAndRelationshipMustAgree: a trigger is a statement about where the
|
||||
// period runs from, so it cannot contradict where the record sits.
|
||||
func TestTriggerAndRelationshipMustAgree(t *testing.T) {
|
||||
l := New()
|
||||
rel, err := open(l, org, "payments", []string{"subject:p1"}, ago(3))
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
|
||||
// An occasional transaction is by definition outside a relationship.
|
||||
_, err = l.Retain(Record{
|
||||
Org: org, Class: ClassTransaction, Trigger: TriggerOccasional, Relationship: rel,
|
||||
Ref: "tx-1", Parties: []string{"subject:p1"}, Occurred: ago(1), Body: body("x"),
|
||||
})
|
||||
if !errors.Is(err, ErrTrigger) {
|
||||
t.Errorf("occasional inside a relationship: err = %v, want ErrTrigger", err)
|
||||
}
|
||||
|
||||
// Only a relationship record may run from a relationship end it does not name.
|
||||
_, err = l.Retain(Record{
|
||||
Org: org, Class: ClassTransaction, Trigger: TriggerRelationshipEnd,
|
||||
Ref: "tx-2", Parties: []string{"subject:p1"}, Occurred: ago(1), Body: body("x"),
|
||||
})
|
||||
if !errors.Is(err, ErrRelationship) {
|
||||
t.Errorf("relationship-end with no relationship: err = %v, want ErrRelationship", err)
|
||||
}
|
||||
|
||||
// And an unknown trigger is not defaulted into one that works.
|
||||
_, err = l.Retain(Record{
|
||||
Org: org, Class: ClassTransaction, Trigger: Trigger("whenever"),
|
||||
Ref: "tx-3", Parties: []string{"subject:p1"}, Occurred: ago(1), Body: body("x"),
|
||||
})
|
||||
if !errors.Is(err, ErrTrigger) {
|
||||
t.Errorf("unknown trigger: err = %v, want ErrTrigger", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestClockCascadesOnClose: a record retained inside a relationship cannot know
|
||||
// its expiry when it is written, because the period runs from the end of the
|
||||
// relationship. Closing the relationship starts every one of those clocks.
|
||||
func TestClockCascadesOnClose(t *testing.T) {
|
||||
l := New()
|
||||
|
||||
rel, err := open(l, org, "brokerage", []string{"subject:p1"}, ago(9))
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
|
||||
var inside []string
|
||||
for i := range 3 {
|
||||
id, err := transact(l, org, rel, fmt.Sprint("tx-", i), []string{"subject:p1"}, ago(8-i), body(fmt.Sprint("tx", i)))
|
||||
if err != nil {
|
||||
t.Fatalf("transaction: %v", err)
|
||||
}
|
||||
inside = append(inside, id)
|
||||
if got := mustGet(t, l, id).Expiry(); !got.IsZero() {
|
||||
t.Fatalf("in-relationship record expires at %s while the relationship is open", got)
|
||||
}
|
||||
}
|
||||
|
||||
started, err := l.Close(org, rel, ago(4))
|
||||
if err != nil {
|
||||
t.Fatalf("Close: %v", err)
|
||||
}
|
||||
if want := len(inside) + 1; started != want {
|
||||
t.Errorf("Close started %d clocks, want %d", started, want)
|
||||
}
|
||||
|
||||
end := ago(4)
|
||||
for _, id := range inside {
|
||||
r := mustGet(t, l, id)
|
||||
if !r.Start.Equal(end) {
|
||||
t.Errorf("%s clock started %s, want %s", id, r.Start, end)
|
||||
}
|
||||
if want := end.AddDate(Period, 0, 0); !r.Expiry().Equal(want) {
|
||||
t.Errorf("%s expiry = %s, want %s", id, r.Expiry(), want)
|
||||
}
|
||||
}
|
||||
|
||||
// A record written into an already-closed relationship inherits its clock
|
||||
// rather than starting a new one.
|
||||
late, err := transact(l, org, rel, "tx-late", []string{"subject:p1"}, ago(5), body("late"))
|
||||
if err != nil {
|
||||
t.Fatalf("transaction after close: %v", err)
|
||||
}
|
||||
if got := mustGet(t, l, late).Start; !got.Equal(end) {
|
||||
t.Errorf("late record clock = %s, want %s", got, end)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCloseRefusals: a relationship closes once, in its own org, and not before
|
||||
// it opened.
|
||||
func TestCloseRefusals(t *testing.T) {
|
||||
l := New()
|
||||
rel, err := open(l, org, "payments", []string{"subject:p1"}, ago(3))
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
|
||||
if _, err := l.Close("other", rel, ago(1)); !errors.Is(err, ErrRelationship) {
|
||||
t.Errorf("cross-org close: err = %v, want ErrRelationship", err)
|
||||
}
|
||||
if _, err := l.Close(org, "no-such-id", ago(1)); !errors.Is(err, ErrRelationship) {
|
||||
t.Errorf("unknown close: err = %v, want ErrRelationship", err)
|
||||
}
|
||||
if _, err := l.Close(org, rel, ago(5)); !errors.Is(err, ErrOccurred) {
|
||||
t.Errorf("close before open: err = %v, want ErrOccurred", err)
|
||||
}
|
||||
|
||||
if _, err := l.Close(org, rel, ago(1)); err != nil {
|
||||
t.Fatalf("Close: %v", err)
|
||||
}
|
||||
if _, err := l.Close(org, rel, ago(1)); !errors.Is(err, ErrClosed) {
|
||||
t.Errorf("second close: err = %v, want ErrClosed", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDisposeDestroysExpiredAndProvesIt: the whole record and every index entry
|
||||
// that referenced it, and nothing that is still within its period.
|
||||
func TestDisposeDestroysExpiredAndProvesIt(t *testing.T) {
|
||||
l := New()
|
||||
|
||||
expired, err := refuse(l, org, "tx-old", []string{"subject:gone", "name:gone"}, ago(6), "refused")
|
||||
if err != nil {
|
||||
t.Fatalf("refusal: %v", err)
|
||||
}
|
||||
live, err := refuse(l, org, "tx-new", []string{"subject:kept"}, ago(1), "refused")
|
||||
if err != nil {
|
||||
t.Fatalf("refusal: %v", err)
|
||||
}
|
||||
relationship, err := open(l, org, "payments", []string{"subject:open"}, ago(20))
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
|
||||
d, err := l.Dispose(time.Now().UTC())
|
||||
if err != nil {
|
||||
t.Fatalf("Dispose: %v", err)
|
||||
}
|
||||
if len(d.Disposed) != 1 || d.Disposed[0] != expired {
|
||||
t.Fatalf("disposed = %v, want [%s]", d.Disposed, expired)
|
||||
}
|
||||
if d.Examined != 3 {
|
||||
t.Errorf("examined = %d, want 3", d.Examined)
|
||||
}
|
||||
|
||||
if _, err := l.Get(PurposeInvestigation, org, expired); !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("expired record still readable: err = %v", err)
|
||||
}
|
||||
if _, err := l.Get(PurposeInvestigation, org, live); err != nil {
|
||||
t.Errorf("record within its period was destroyed: %v", err)
|
||||
}
|
||||
if _, err := l.Get(PurposeInvestigation, org, relationship); err != nil {
|
||||
t.Errorf("open relationship was destroyed: %v", err)
|
||||
}
|
||||
|
||||
// The index entries went with it, so nothing can find a destroyed record.
|
||||
for _, party := range []string{"subject:gone", "name:gone"} {
|
||||
a, err := l.Lookback(PurposeDisclosure, org, party, time.Now().UTC())
|
||||
if err != nil {
|
||||
t.Fatalf("Lookback: %v", err)
|
||||
}
|
||||
if a.Examined != 0 {
|
||||
t.Errorf("party %q still indexes %d records", party, a.Examined)
|
||||
}
|
||||
}
|
||||
|
||||
// A second run has nothing left to do and still proves it.
|
||||
again, err := l.Dispose(time.Now().UTC())
|
||||
if err != nil {
|
||||
t.Fatalf("second Dispose: %v", err)
|
||||
}
|
||||
if len(again.Disposed) != 0 {
|
||||
t.Errorf("second run disposed %v", again.Disposed)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDisposeCascadeLeavesNoDanglingIndex: closing a relationship expires it and
|
||||
// everything inside it together, and the relationship index goes with them.
|
||||
func TestDisposeCascadeLeavesNoDanglingIndex(t *testing.T) {
|
||||
l := New()
|
||||
rel, err := open(l, org, "payments", []string{"subject:p1"}, ago(12))
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
if _, err := transact(l, org, rel, "tx-1", []string{"subject:p1"}, ago(11), body("tx")); err != nil {
|
||||
t.Fatalf("transaction: %v", err)
|
||||
}
|
||||
if _, err := l.Close(org, rel, ago(6)); err != nil {
|
||||
t.Fatalf("Close: %v", err)
|
||||
}
|
||||
|
||||
d, err := l.Dispose(time.Now().UTC())
|
||||
if err != nil {
|
||||
t.Fatalf("Dispose: %v", err)
|
||||
}
|
||||
if len(d.Disposed) != 2 {
|
||||
t.Fatalf("disposed %d records, want 2", len(d.Disposed))
|
||||
}
|
||||
if l.Len() != 0 {
|
||||
t.Errorf("ledger holds %d records, want 0", l.Len())
|
||||
}
|
||||
if len(l.parties) != 0 || len(l.inside) != 0 {
|
||||
t.Errorf("indexes left behind: %d party keys, %d relationships", len(l.parties), len(l.inside))
|
||||
}
|
||||
}
|
||||
|
||||
// TestDisposeRefusesTheFuture: a disposal run cannot be talked into early
|
||||
// destruction by a date the clock has not reached.
|
||||
func TestDisposeRefusesTheFuture(t *testing.T) {
|
||||
l := New()
|
||||
id, err := refuse(l, org, "tx-1", []string{"subject:p1"}, ago(1), "refused")
|
||||
if err != nil {
|
||||
t.Fatalf("refusal: %v", err)
|
||||
}
|
||||
|
||||
if _, err := l.Dispose(time.Now().UTC().AddDate(10, 0, 0)); !errors.Is(err, ErrFuture) {
|
||||
t.Fatalf("err = %v, want ErrFuture", err)
|
||||
}
|
||||
if _, err := l.Get(PurposeInvestigation, org, id); err != nil {
|
||||
t.Fatalf("record destroyed by a future date: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDisposeReportsNoSuccessWhenProofFails: the run must not report a count it
|
||||
// cannot stand behind. The ledger is corrupted deliberately here — an index entry
|
||||
// pointing at a record that is not there — because that is the shape of the bug a
|
||||
// disposal job hides when it counts only its own delete statements.
|
||||
func TestDisposeReportsNoSuccessWhenProofFails(t *testing.T) {
|
||||
l := New()
|
||||
if _, err := refuse(l, org, "tx-1", []string{"subject:p1"}, ago(1), "refused"); err != nil {
|
||||
t.Fatalf("refusal: %v", err)
|
||||
}
|
||||
l.parties[partyKey(org, "subject:ghost")] = []string{"a-record-that-is-not-there"}
|
||||
|
||||
d, err := l.Dispose(time.Now().UTC())
|
||||
if !errors.Is(err, ErrDisposal) {
|
||||
t.Fatalf("err = %v, want ErrDisposal", err)
|
||||
}
|
||||
if d.Examined != 0 || len(d.Disposed) != 0 {
|
||||
t.Fatalf("failed run reported work: %+v", d)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLookbackAnswersArticle78: is or was a relationship maintained with this
|
||||
// party in the prior five years, and what was its nature.
|
||||
func TestLookbackAnswersArticle78(t *testing.T) {
|
||||
l := New()
|
||||
now := time.Now().UTC()
|
||||
|
||||
// Ended inside the window.
|
||||
recent, err := open(l, org, "payments", []string{"name:ivan petrov"}, ago(9))
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
if _, err := l.Close(org, recent, ago(2)); err != nil {
|
||||
t.Fatalf("Close: %v", err)
|
||||
}
|
||||
|
||||
a, err := l.Lookback(PurposeDisclosure, org, "name:ivan petrov", now)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookback: %v", err)
|
||||
}
|
||||
if !a.Maintained {
|
||||
t.Error("relationship ended two years ago was not found")
|
||||
}
|
||||
if a.Current {
|
||||
t.Error("a closed relationship reported as current")
|
||||
}
|
||||
if len(a.Natures) != 1 || a.Natures[0] != "payments" {
|
||||
t.Errorf("natures = %v, want [payments]", a.Natures)
|
||||
}
|
||||
if len(a.Records) != 1 || a.Records[0] != recent {
|
||||
t.Errorf("records = %v, want [%s]", a.Records, recent)
|
||||
}
|
||||
if !a.From.Equal(now.AddDate(-Period, 0, 0)) || !a.To.Equal(now) {
|
||||
t.Errorf("window %s..%s is not the prior five years", a.From, a.To)
|
||||
}
|
||||
|
||||
// Still open: the answer is "is", not "was".
|
||||
if _, err := open(l, org, "custody", []string{"name:open party"}, ago(8)); err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
a, err = l.Lookback(PurposeDisclosure, org, "name:open party", now)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookback: %v", err)
|
||||
}
|
||||
if !a.Maintained || !a.Current {
|
||||
t.Errorf("open relationship: maintained=%v current=%v", a.Maintained, a.Current)
|
||||
}
|
||||
|
||||
// Ended before the window: not within the prior five years.
|
||||
old, err := open(l, org, "payments", []string{"name:old party"}, ago(20))
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
if _, err := l.Close(org, old, ago(7)); err != nil {
|
||||
t.Fatalf("Close: %v", err)
|
||||
}
|
||||
a, err = l.Lookback(PurposeDisclosure, org, "name:old party", now)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookback: %v", err)
|
||||
}
|
||||
if a.Maintained {
|
||||
t.Error("a relationship that ended seven years ago is inside the five-year window")
|
||||
}
|
||||
if a.Examined != 1 {
|
||||
t.Errorf("examined = %d, want 1 (the record exists, it is out of window)", a.Examined)
|
||||
}
|
||||
|
||||
// A party with nothing on file.
|
||||
a, err = l.Lookback(PurposeDisclosure, org, "name:stranger", now)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookback: %v", err)
|
||||
}
|
||||
if a.Maintained || a.Examined != 0 || len(a.Records) != 0 {
|
||||
t.Errorf("stranger: %+v", a)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLookbackIgnoresRecordsThatAreNotRelationships: Art. 78 asks about business
|
||||
// relationships, and a transaction with a party is not one.
|
||||
func TestLookbackIgnoresRecordsThatAreNotRelationships(t *testing.T) {
|
||||
l := New()
|
||||
if _, err := transact(l, org, "", "tx-1", []string{"name:one off"}, ago(1), body("x")); err != nil {
|
||||
t.Fatalf("transaction: %v", err)
|
||||
}
|
||||
|
||||
a, err := l.Lookback(PurposeDisclosure, org, "name:one off", time.Now().UTC())
|
||||
if err != nil {
|
||||
t.Fatalf("Lookback: %v", err)
|
||||
}
|
||||
if a.Maintained {
|
||||
t.Error("an occasional transaction was reported as a business relationship")
|
||||
}
|
||||
if a.Examined != 1 {
|
||||
t.Errorf("examined = %d, want 1", a.Examined)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLookbackIsIndexedNotScanned: the answer must be fully and speedily
|
||||
// available, so the work a lookback does is proportional to one party's records
|
||||
// and not to the ledger. Examined is that proof and it stays flat.
|
||||
func TestLookbackIsIndexedNotScanned(t *testing.T) {
|
||||
l := New()
|
||||
now := time.Now().UTC()
|
||||
|
||||
if _, err := open(l, org, "payments", []string{"name:target"}, ago(3)); err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
|
||||
before, err := l.Lookback(PurposeDisclosure, org, "name:target", now)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookback: %v", err)
|
||||
}
|
||||
|
||||
for i := range 5000 {
|
||||
ref := fmt.Sprintf("tx-noise-%d", i)
|
||||
if _, err := refuse(l, org, ref, []string{"subject:noise-" + ref}, ago(1), "refused"); err != nil {
|
||||
t.Fatalf("refusal: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
after, err := l.Lookback(PurposeDisclosure, org, "name:target", now)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookback: %v", err)
|
||||
}
|
||||
if after.Examined != before.Examined || after.Examined != 1 {
|
||||
t.Fatalf("examined went from %d to %d as the ledger grew to %d records",
|
||||
before.Examined, after.Examined, l.Len())
|
||||
}
|
||||
if !after.Maintained {
|
||||
t.Error("answer changed as unrelated records were added")
|
||||
}
|
||||
}
|
||||
|
||||
// TestOrgBoundary: no read crosses it, in either direction, on any surface.
|
||||
func TestOrgBoundary(t *testing.T) {
|
||||
l := New()
|
||||
const other = "beta"
|
||||
|
||||
mine, err := open(l, org, "payments", []string{"name:shared party"}, ago(2))
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
|
||||
if _, err := l.Get(PurposeInvestigation, other, mine); !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("cross-org Get: err = %v, want ErrNotFound", err)
|
||||
}
|
||||
a, err := l.Lookback(PurposeDisclosure, other, "name:shared party", time.Now().UTC())
|
||||
if err != nil {
|
||||
t.Fatalf("Lookback: %v", err)
|
||||
}
|
||||
if a.Maintained || a.Examined != 0 {
|
||||
t.Errorf("cross-org lookback saw %d records: %+v", a.Examined, a)
|
||||
}
|
||||
|
||||
seen := 0
|
||||
if err := l.Each(PurposeMonitoring, other, "", func(Record) error { seen++; return nil }); err != nil {
|
||||
t.Fatalf("Each: %v", err)
|
||||
}
|
||||
if seen != 0 {
|
||||
t.Errorf("cross-org Each saw %d records", seen)
|
||||
}
|
||||
if err := l.Extend(other, mine, time.Hour, "reason", "who"); err == nil {
|
||||
t.Error("cross-org Extend succeeded")
|
||||
}
|
||||
// A transaction cannot be filed into another org's relationship.
|
||||
if _, err := transact(l, other, mine, "tx-1", []string{"subject:p"}, ago(1), body("x")); !errors.Is(err, ErrRelationship) {
|
||||
t.Errorf("cross-org transaction: err = %v, want ErrRelationship", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadsRefuseOtherPurposes: retained personal data is for preventing money
|
||||
// laundering and terrorist financing, so a commercial purpose gets no data.
|
||||
func TestReadsRefuseOtherPurposes(t *testing.T) {
|
||||
l := New()
|
||||
id, err := refuse(l, org, "tx-1", []string{"subject:p1"}, ago(1), "refused")
|
||||
if err != nil {
|
||||
t.Fatalf("refusal: %v", err)
|
||||
}
|
||||
|
||||
for _, p := range []Purpose{"marketing", "credit scoring", "model training", ""} {
|
||||
if _, err := l.Get(p, org, id); !errors.Is(err, ErrPurpose) {
|
||||
t.Errorf("Get(%q): err = %v, want ErrPurpose", p, err)
|
||||
}
|
||||
if err := l.Each(p, org, "", func(Record) error { return nil }); !errors.Is(err, ErrPurpose) {
|
||||
t.Errorf("Each(%q): err = %v, want ErrPurpose", p, err)
|
||||
}
|
||||
if _, err := l.Lookback(p, org, "subject:p1", time.Now().UTC()); !errors.Is(err, ErrPurpose) {
|
||||
t.Errorf("Lookback(%q): err = %v, want ErrPurpose", p, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestRecordsAreNotRedactableThroughAReader: the ledger hands out copies, so a
|
||||
// reader that alters what it was given has not altered the retained record.
|
||||
// Redaction is prohibited, and here it is also not reachable.
|
||||
func TestRecordsAreNotRedactableThroughAReader(t *testing.T) {
|
||||
l := New()
|
||||
id, err := assess(l, org, "", "case-1", []string{"subject:p1"}, Assessment{
|
||||
Considered: []string{"velocity", "profile"},
|
||||
Result: NotReported,
|
||||
Rationale: "consistent with the profile on file",
|
||||
By: "mlro",
|
||||
At: ago(1),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("assessment: %v", err)
|
||||
}
|
||||
|
||||
got := mustGet(t, l, id)
|
||||
got.Body[0] = 'X'
|
||||
got.Parties[0] = "subject:someone-else"
|
||||
got.Assessment.Rationale = ""
|
||||
got.Assessment.Considered = nil
|
||||
got.Nature = "rewritten"
|
||||
|
||||
after := mustGet(t, l, id)
|
||||
if after.Body[0] == 'X' {
|
||||
t.Error("a reader rewrote the retained body")
|
||||
}
|
||||
if after.Parties[0] != "subject:p1" {
|
||||
t.Errorf("a reader rewrote the party index key: %q", after.Parties[0])
|
||||
}
|
||||
if after.Assessment.Rationale == "" || len(after.Assessment.Considered) != 2 {
|
||||
t.Error("a reader redacted the retained assessment")
|
||||
}
|
||||
|
||||
// The same through a walk, which is the other way out of the ledger.
|
||||
if err := l.Each(PurposeInvestigation, org, "", func(r Record) error {
|
||||
r.Body[0] = 'Y'
|
||||
r.Assessment.Rationale = "rewritten"
|
||||
return nil
|
||||
}); err != nil {
|
||||
t.Fatalf("Each: %v", err)
|
||||
}
|
||||
walked := mustGet(t, l, id)
|
||||
if walked.Body[0] == 'Y' || walked.Assessment.Rationale == "rewritten" {
|
||||
t.Error("a walk rewrote the retained record")
|
||||
}
|
||||
}
|
||||
|
||||
// TestExtendRefusals: an extension is a decision somebody made about one case,
|
||||
// within five further years, once.
|
||||
func TestExtendRefusals(t *testing.T) {
|
||||
l := New()
|
||||
id, err := refuse(l, org, "tx-1", []string{"subject:p1"}, ago(1), "refused")
|
||||
if err != nil {
|
||||
t.Fatalf("refusal: %v", err)
|
||||
}
|
||||
relationship, err := open(l, org, "payments", []string{"subject:p2"}, ago(1))
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
|
||||
for name, call := range map[string]error{
|
||||
"no reason": l.Extend(org, id, time.Hour, "", "mlro"),
|
||||
"no decider": l.Extend(org, id, time.Hour, "reason", ""),
|
||||
"zero period": l.Extend(org, id, 0, "reason", "mlro"),
|
||||
"negative": l.Extend(org, id, -time.Hour, "reason", "mlro"),
|
||||
"over the cap": l.Extend(org, id, 6*365*24*time.Hour, "reason", "mlro"),
|
||||
"clock unstarted": l.Extend(org, relationship, time.Hour, "reason", "mlro"),
|
||||
} {
|
||||
if call == nil {
|
||||
t.Errorf("%s: extension accepted", name)
|
||||
}
|
||||
}
|
||||
if err := l.Extend(org, "no-such-record", time.Hour, "reason", "mlro"); !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("unknown record: err = %v, want ErrNotFound", err)
|
||||
}
|
||||
|
||||
if err := l.Extend(org, id, 365*24*time.Hour, "live investigation", "mlro"); err != nil {
|
||||
t.Fatalf("Extend: %v", err)
|
||||
}
|
||||
if err := l.Extend(org, id, time.Hour, "again", "mlro"); !errors.Is(err, ErrExtension) {
|
||||
t.Errorf("second extension: err = %v, want ErrExtension", err)
|
||||
}
|
||||
|
||||
r := mustGet(t, l, id)
|
||||
if r.Extended == nil || r.Extended.Who != "mlro" || r.Extended.Reason != "live investigation" {
|
||||
t.Fatalf("extension not recorded: %+v", r.Extended)
|
||||
}
|
||||
if want := r.Start.AddDate(Period, 0, 0).Add(365 * 24 * time.Hour); !r.Expiry().Equal(want) {
|
||||
t.Errorf("expiry = %s, want %s", r.Expiry(), want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestExtensionSurvivesDisposal: an extended record is not swept up on the date
|
||||
// it would otherwise have expired.
|
||||
func TestExtensionSurvivesDisposal(t *testing.T) {
|
||||
l := New()
|
||||
id, err := refuse(l, org, "tx-1", []string{"subject:p1"}, ago(6), "refused")
|
||||
if err != nil {
|
||||
t.Fatalf("refusal: %v", err)
|
||||
}
|
||||
if err := l.Extend(org, id, 2*365*24*time.Hour, "live investigation", "mlro"); err != nil {
|
||||
t.Fatalf("Extend: %v", err)
|
||||
}
|
||||
|
||||
d, err := l.Dispose(time.Now().UTC())
|
||||
if err != nil {
|
||||
t.Fatalf("Dispose: %v", err)
|
||||
}
|
||||
if len(d.Disposed) != 0 {
|
||||
t.Fatalf("disposed an extended record: %v", d.Disposed)
|
||||
}
|
||||
if _, err := l.Get(PurposeInvestigation, org, id); err != nil {
|
||||
t.Fatalf("extended record destroyed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWriteRefusals: what the ledger will not accept, because a record that
|
||||
// cannot answer for itself is not worth retaining.
|
||||
func TestWriteRefusals(t *testing.T) {
|
||||
l := New()
|
||||
rel, err := open(l, org, "payments", []string{"subject:p1"}, ago(1))
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
|
||||
if _, err := transact(l, "", "", "tx-1", []string{"subject:p1"}, ago(1), body("x")); !errors.Is(err, ErrOrg) {
|
||||
t.Errorf("no org: err = %v, want ErrOrg", err)
|
||||
}
|
||||
if _, err := transact(l, org, "", "tx-1", nil, ago(1), body("x")); !errors.Is(err, ErrParties) {
|
||||
t.Errorf("no parties: err = %v, want ErrParties", err)
|
||||
}
|
||||
if _, err := transact(l, org, "", "tx-1", []string{""}, ago(1), body("x")); !errors.Is(err, ErrParties) {
|
||||
t.Errorf("empty party: err = %v, want ErrParties", err)
|
||||
}
|
||||
// MLR reg. 40(2)(b): the record must be enough to reconstruct the transaction.
|
||||
if _, err := transact(l, org, "", "tx-1", []string{"subject:p1"}, ago(1), nil); !errors.Is(err, ErrBody) {
|
||||
t.Errorf("no body: err = %v, want ErrBody", err)
|
||||
}
|
||||
if _, err := transact(l, org, "", "", []string{"subject:p1"}, ago(1), body("x")); !errors.Is(err, ErrRef) {
|
||||
t.Errorf("no reference: err = %v, want ErrRef", err)
|
||||
}
|
||||
if _, err := transact(l, org, "", "tx-1", []string{"subject:p1"}, time.Time{}, body("x")); !errors.Is(err, ErrOccurred) {
|
||||
t.Errorf("no event date: err = %v, want ErrOccurred", err)
|
||||
}
|
||||
if _, err := open(l, org, "", []string{"subject:p1"}, ago(1)); !errors.Is(err, ErrNature) {
|
||||
t.Errorf("no nature: err = %v, want ErrNature", err)
|
||||
}
|
||||
if _, err := transact(l, org, "not-a-relationship", "tx-1", []string{"subject:p1"}, ago(1), body("x")); !errors.Is(err, ErrRelationship) {
|
||||
t.Errorf("unknown relationship: err = %v, want ErrRelationship", err)
|
||||
}
|
||||
// A transaction record is not a container for other records.
|
||||
tx, err := transact(l, org, rel, "tx-2", []string{"subject:p1"}, ago(1), body("x"))
|
||||
if err != nil {
|
||||
t.Fatalf("transaction: %v", err)
|
||||
}
|
||||
if _, err := transact(l, org, tx, "tx-3", []string{"subject:p1"}, ago(1), body("x")); !errors.Is(err, ErrRelationship) {
|
||||
t.Errorf("transaction as relationship: err = %v, want ErrRelationship", err)
|
||||
}
|
||||
if _, err := assess(l, org, "", "case-1", []string{"subject:p1"}, Assessment{
|
||||
Result: NotReported, At: ago(1), // no rationale, nothing considered, nobody decided
|
||||
}); !errors.Is(err, ErrAssessment) {
|
||||
t.Errorf("incomplete assessment: err = %v, want ErrAssessment", err)
|
||||
}
|
||||
if _, err := l.Retain(Record{
|
||||
Org: org, Class: Class("guesswork"), Trigger: TriggerRefusal, Ref: "x",
|
||||
Parties: []string{"subject:p1"}, Occurred: ago(1), Body: body("x"),
|
||||
}); !errors.Is(err, ErrClass) {
|
||||
t.Errorf("unknown class: err = %v, want ErrClass", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestEachIsOrderedOldestFirst: a file produced from the ledger is reproducible.
|
||||
func TestEachIsOrderedOldestFirst(t *testing.T) {
|
||||
l := New()
|
||||
for i, years := range []int{1, 4, 2, 3} {
|
||||
if _, err := refuse(l, org, fmt.Sprint("tx-", i), []string{"subject:p1"}, ago(years), "refused"); err != nil {
|
||||
t.Fatalf("refusal: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
var seen []time.Time
|
||||
if err := l.Each(PurposeRetention, org, ClassRefusal, func(r Record) error {
|
||||
seen = append(seen, r.Occurred)
|
||||
return nil
|
||||
}); err != nil {
|
||||
t.Fatalf("Each: %v", err)
|
||||
}
|
||||
if len(seen) != 4 {
|
||||
t.Fatalf("saw %d records, want 4", len(seen))
|
||||
}
|
||||
for i := 1; i < len(seen); i++ {
|
||||
if seen[i].Before(seen[i-1]) {
|
||||
t.Fatalf("out of order at %d: %s before %s", i, seen[i], seen[i-1])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestEachFiltersByClass: the class filter is a filter, and an empty one means
|
||||
// every class.
|
||||
func TestEachFiltersByClass(t *testing.T) {
|
||||
l := New()
|
||||
if _, err := refuse(l, org, "tx-1", []string{"subject:p1"}, ago(1), "refused"); err != nil {
|
||||
t.Fatalf("refusal: %v", err)
|
||||
}
|
||||
if _, err := transact(l, org, "", "tx-2", []string{"subject:p1"}, ago(1), body("x")); err != nil {
|
||||
t.Fatalf("transaction: %v", err)
|
||||
}
|
||||
|
||||
count := func(c Class) int {
|
||||
n := 0
|
||||
if err := l.Each(PurposeMonitoring, org, c, func(Record) error { n++; return nil }); err != nil {
|
||||
t.Fatalf("Each(%q): %v", c, err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
if got := count(ClassRefusal); got != 1 {
|
||||
t.Errorf("refusals = %d, want 1", got)
|
||||
}
|
||||
if got := count(ClassTransaction); got != 1 {
|
||||
t.Errorf("transactions = %d, want 1", got)
|
||||
}
|
||||
if got := count(""); got != 2 {
|
||||
t.Errorf("all classes = %d, want 2", got)
|
||||
}
|
||||
if got := count(ClassAssessment); got != 0 {
|
||||
t.Errorf("assessments = %d, want 0", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestEachStopsOnError: a caller can abandon the walk, and the error is its own.
|
||||
func TestEachStopsOnError(t *testing.T) {
|
||||
l := New()
|
||||
for i := range 3 {
|
||||
if _, err := refuse(l, org, fmt.Sprint("tx-", i), []string{"subject:p1"}, ago(1), "refused"); err != nil {
|
||||
t.Fatalf("refusal: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
stop := errors.New("enough")
|
||||
seen := 0
|
||||
err := l.Each(PurposeMonitoring, org, "", func(Record) error {
|
||||
seen++
|
||||
return stop
|
||||
})
|
||||
if !errors.Is(err, stop) {
|
||||
t.Fatalf("err = %v, want the caller's error", err)
|
||||
}
|
||||
if seen != 1 {
|
||||
t.Errorf("visited %d records after the error, want 1", seen)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,393 @@
|
||||
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
|
||||
// Use of this source code is governed by the Apache 2.0
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// Package retention keeps what the instruments require kept, for as long as they
|
||||
// require it, and destroys it when they require that.
|
||||
//
|
||||
// # Three clocks
|
||||
//
|
||||
// A retention period runs five years from the end of the business relationship,
|
||||
// from the occasional transaction, or from the date of refusal to enter a
|
||||
// relationship or carry out a transaction (Regulation (EU) 2024/1624
|
||||
// Art. 77(3)). The third trigger is the one implementations miss, so [Refusal]
|
||||
// is a first-class record here and a refused transaction is retained exactly
|
||||
// like a completed one.
|
||||
//
|
||||
// A record retained inside a relationship cannot know its own expiry when it is
|
||||
// written, because the clock starts at the end of the relationship. So
|
||||
// [Ledger.Close] starts the clock on the relationship and on everything retained
|
||||
// inside it, and until then those records have no expiry at all.
|
||||
//
|
||||
// # Deletion on expiry against no redaction
|
||||
//
|
||||
// Two requirements pull against each other. Records must not be redacted
|
||||
// (AMLR Art. 77(1) final subparagraph); personal data must be deleted when the
|
||||
// retention period expires, subject to a case-by-case extension of at most five
|
||||
// further years (Directive (EU) 2015/849 Art. 40(1) second subparagraph).
|
||||
//
|
||||
// They are reconciled by the unit of disposal: the whole record, never a field.
|
||||
// A retained record is append-only for its whole life — this package exposes no
|
||||
// operation that removes, masks or rewrites any part of one, so redaction is not
|
||||
// a policy that could be broken but an operation that does not exist. Reads hand
|
||||
// out deep copies for the same reason: a reader cannot reach back into the
|
||||
// ledger and unmake part of a record. At expiry the record is destroyed in full,
|
||||
// with every index entry that referenced it, and [Ledger.Dispose] proves that
|
||||
// before it reports success. There is no state in which a record exists
|
||||
// partially.
|
||||
//
|
||||
// Sealing is not redaction. A sealed body (see pkg/token) holds the whole record
|
||||
// and is reversible under the org's key, so the transaction can still be
|
||||
// reconstructed (MLR 2017 reg. 40(2)(b)). Nothing is dropped, so nothing is
|
||||
// redacted.
|
||||
//
|
||||
// # Purpose
|
||||
//
|
||||
// Retained personal data may be processed only to prevent money laundering and
|
||||
// terrorist financing; processing for commercial purposes is prohibited
|
||||
// (AMLD4 Art. 41(2)). Every read therefore takes a [Purpose] from a closed set,
|
||||
// and a purpose outside that set is refused rather than logged.
|
||||
package retention
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The retention arithmetic, in years, each from its own instrument.
|
||||
const (
|
||||
// Period is the retention period. AMLR Art. 77(3).
|
||||
Period = 5
|
||||
// Extension is the cap on a case-by-case extension of it.
|
||||
// AMLD4 Art. 40(1) second subparagraph.
|
||||
Extension = 5
|
||||
// Ceiling is how long records of a transaction occurring during a business
|
||||
// relationship need be kept, counted from the transaction. MLR 2017
|
||||
// reg. 40(4).
|
||||
Ceiling = 10
|
||||
)
|
||||
|
||||
// skew is how far ahead of the caller's clock a supplied "now" may be before it
|
||||
// is refused. Disposal is destruction: it does not run on a future date.
|
||||
const skew = time.Minute
|
||||
|
||||
// Errors. Each is a refusal, and none of them is recoverable by proceeding.
|
||||
var (
|
||||
ErrOrg = errors.New("retention: empty org")
|
||||
ErrClass = errors.New("retention: unknown class")
|
||||
ErrTrigger = errors.New("retention: unknown trigger")
|
||||
ErrParties = errors.New("retention: record names no party")
|
||||
ErrNature = errors.New("retention: relationship has no nature")
|
||||
ErrReason = errors.New("retention: refusal has no reason")
|
||||
ErrBody = errors.New("retention: record has no body to reconstruct from")
|
||||
ErrRef = errors.New("retention: record has no reference to what it retains")
|
||||
ErrOccurred = errors.New("retention: record has no event date")
|
||||
ErrClock = errors.New("retention: retention clock has not started")
|
||||
ErrAssessment = errors.New("retention: incomplete Art. 69(2) assessment")
|
||||
ErrRelationship = errors.New("retention: no such relationship")
|
||||
ErrClosed = errors.New("retention: relationship is already closed")
|
||||
ErrNotFound = errors.New("retention: no such record")
|
||||
ErrPurpose = errors.New("retention: purpose is not money laundering or terrorist financing prevention")
|
||||
ErrFuture = errors.New("retention: refusing to act on a future date")
|
||||
ErrExtension = errors.New("retention: extension exceeds the five further years allowed")
|
||||
ErrDisposal = errors.New("retention: disposal could not be proven")
|
||||
)
|
||||
|
||||
// Class is what a record is.
|
||||
type Class string
|
||||
|
||||
const (
|
||||
// ClassRelationship is a business relationship. Its retention clock starts
|
||||
// when it ends.
|
||||
ClassRelationship Class = "relationship"
|
||||
// ClassTransaction is a transaction, retained in enough detail to be
|
||||
// reconstructed. MLR 2017 reg. 40(2)(b).
|
||||
ClassTransaction Class = "transaction"
|
||||
// ClassAssessment is the Art. 69(2) assessment — retained whether or not it
|
||||
// produced a report. AMLR Art. 77(1)(b).
|
||||
ClassAssessment Class = "assessment"
|
||||
// ClassRefusal is a refusal to enter a relationship or carry out a
|
||||
// transaction. AMLR Art. 77(3) third trigger.
|
||||
ClassRefusal Class = "refusal"
|
||||
)
|
||||
|
||||
// Trigger is which Art. 77(3) event starts the retention clock.
|
||||
type Trigger string
|
||||
|
||||
const (
|
||||
// TriggerRelationshipEnd counts five years from the end of the relationship.
|
||||
TriggerRelationshipEnd Trigger = "relationship_end"
|
||||
// TriggerOccasional counts five years from the occasional transaction.
|
||||
TriggerOccasional Trigger = "occasional_transaction"
|
||||
// TriggerRefusal counts five years from the date of refusal.
|
||||
TriggerRefusal Trigger = "refusal"
|
||||
)
|
||||
|
||||
// Result is what an assessment concluded.
|
||||
type Result string
|
||||
|
||||
const (
|
||||
// Reported means a report went to the FIU.
|
||||
Reported Result = "reported"
|
||||
// NotReported means it did not, which is a retained decision and not a
|
||||
// deleted row. AMLR Art. 77(1)(b); JMLSG 6.32; JMLSG 8.6 on information not
|
||||
// acted upon.
|
||||
NotReported Result = "not_reported"
|
||||
)
|
||||
|
||||
// Purpose is why a retained record is being read. The set is closed: retained
|
||||
// personal data may be processed only to prevent money laundering and terrorist
|
||||
// financing (AMLD4 Art. 41(2)), so a purpose outside it has no representation
|
||||
// that the ledger will accept.
|
||||
type Purpose string
|
||||
|
||||
const (
|
||||
// PurposeMonitoring is ongoing monitoring of the business relationship.
|
||||
PurposeMonitoring Purpose = "monitoring"
|
||||
// PurposeInvestigation is case work, up to and including a report.
|
||||
PurposeInvestigation Purpose = "investigation"
|
||||
// PurposeDisclosure is answering the FIU or a competent authority.
|
||||
PurposeDisclosure Purpose = "disclosure"
|
||||
// PurposeRetention is administering the retention period itself.
|
||||
PurposeRetention Purpose = "retention"
|
||||
)
|
||||
|
||||
func (p Purpose) permitted() bool {
|
||||
switch p {
|
||||
case PurposeMonitoring, PurposeInvestigation, PurposeDisclosure, PurposeRetention:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Assessment is the Art. 69(2) record: the information and circumstances
|
||||
// considered, and the result. It is retained whether or not it produced a
|
||||
// report, and the reasons are required either way — a dismissed alert is a
|
||||
// retained decision with its rationale.
|
||||
type Assessment struct {
|
||||
// Alerts and Case are what was assessed.
|
||||
Alerts []string `json:"alerts,omitempty"`
|
||||
Case string `json:"case,omitempty"`
|
||||
// Considered is the information and circumstances that were considered.
|
||||
Considered []string `json:"considered"`
|
||||
// Result is what was concluded.
|
||||
Result Result `json:"result"`
|
||||
// Rationale is why. Required for both results: AMLR Art. 77(1)(b) retains
|
||||
// the result, and JMLSG 6.32 requires the reasons for not reporting.
|
||||
Rationale string `json:"rationale"`
|
||||
// By and At are who decided and when. An assessment is a person's decision.
|
||||
By string `json:"by"`
|
||||
At time.Time `json:"at"`
|
||||
}
|
||||
|
||||
func (a Assessment) validate() error {
|
||||
switch a.Result {
|
||||
case Reported, NotReported:
|
||||
default:
|
||||
return fmt.Errorf("%w: result %q", ErrAssessment, a.Result)
|
||||
}
|
||||
if len(a.Considered) == 0 {
|
||||
return fmt.Errorf("%w: nothing recorded as considered", ErrAssessment)
|
||||
}
|
||||
if a.Rationale == "" {
|
||||
return fmt.Errorf("%w: no rationale for %q", ErrAssessment, a.Result)
|
||||
}
|
||||
if a.By == "" {
|
||||
return fmt.Errorf("%w: no decider", ErrAssessment)
|
||||
}
|
||||
if a.At.IsZero() {
|
||||
return fmt.Errorf("%w: no decision date", ErrAssessment)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a Assessment) clone() Assessment {
|
||||
out := a
|
||||
out.Alerts = append([]string(nil), a.Alerts...)
|
||||
out.Considered = append([]string(nil), a.Considered...)
|
||||
return out
|
||||
}
|
||||
|
||||
// Extended is a case-by-case extension of a retention period. AMLD4 Art. 40(1)
|
||||
// permits a further period of at most five years, decided case by case, so an
|
||||
// extension carries who decided it and why or it is refused.
|
||||
type Extended struct {
|
||||
By time.Duration `json:"by"`
|
||||
Reason string `json:"reason"`
|
||||
Who string `json:"who"`
|
||||
At time.Time `json:"at"`
|
||||
}
|
||||
|
||||
// Record is one retained record. It is append-only: nothing in this package
|
||||
// rewrites a field of a written record except to start its retention clock and
|
||||
// to extend its period, both of which only ever move the expiry later.
|
||||
type Record struct {
|
||||
ID string `json:"id"`
|
||||
Org string `json:"org"`
|
||||
Class Class `json:"class"`
|
||||
|
||||
// Parties are every party the record concerns, and the keys the five-year
|
||||
// lookback is indexed by. They are pseudonyms in production: the ledger
|
||||
// never needs the cleartext to answer Art. 78.
|
||||
Parties []string `json:"parties"`
|
||||
|
||||
// Ref is the caller's own reference for what is retained — a transaction id, a
|
||||
// case id. It is held in the clear, so it must be a synthetic reference and
|
||||
// never a direct identifier: it is the handle that ties a retained record to
|
||||
// the system it came from, and what a sealed body is bound to.
|
||||
Ref string `json:"ref,omitempty"`
|
||||
|
||||
// Nature is the nature of the business relationship, which Art. 78 requires
|
||||
// to be reportable alongside its existence.
|
||||
Nature string `json:"nature,omitempty"`
|
||||
|
||||
// Reason is why a transaction or relationship was refused.
|
||||
Reason string `json:"reason,omitempty"`
|
||||
|
||||
// Relationship is the relationship this record is retained inside. Empty for
|
||||
// an occasional transaction or a refusal, which have their own clocks.
|
||||
Relationship string `json:"relationship,omitempty"`
|
||||
|
||||
// Occurred is when the underlying event happened, or when the relationship
|
||||
// opened. Distinct from Start: an in-relationship transaction occurs years
|
||||
// before its retention clock starts.
|
||||
Occurred time.Time `json:"occurred"`
|
||||
// Ended is when a relationship ended. Zero while it is open.
|
||||
Ended time.Time `json:"ended,omitzero"`
|
||||
|
||||
Trigger Trigger `json:"trigger"`
|
||||
// Start is when the retention clock started. Zero means it has not: an open
|
||||
// relationship, and everything retained inside it, has no expiry.
|
||||
Start time.Time `json:"start,omitzero"`
|
||||
Extended *Extended `json:"extended,omitempty"`
|
||||
|
||||
// Body is the record itself, opaque to the ledger and sealed by the caller.
|
||||
// It is what reconstructs the transaction, so it is never partial.
|
||||
Body []byte `json:"body,omitempty"`
|
||||
|
||||
// Assessment is set on ClassAssessment records.
|
||||
Assessment *Assessment `json:"assessment,omitempty"`
|
||||
|
||||
Written time.Time `json:"written"`
|
||||
}
|
||||
|
||||
// Expiry is when the record must be destroyed. Zero means the retention clock
|
||||
// has not started, so the record has no expiry — an open business relationship
|
||||
// is retained for as long as it is open, because the period runs from its end.
|
||||
func (r Record) Expiry() time.Time {
|
||||
if r.Start.IsZero() {
|
||||
return time.Time{}
|
||||
}
|
||||
|
||||
// The five years the AMLR requires. This is a floor: nothing below shortens it.
|
||||
floor := r.Start.AddDate(Period, 0, 0)
|
||||
|
||||
at := floor
|
||||
if r.Extended != nil {
|
||||
at = at.Add(r.Extended.By)
|
||||
}
|
||||
// A case-by-case extension cannot exceed five further years.
|
||||
if limit := r.Start.AddDate(Period+Extension, 0, 0); at.After(limit) {
|
||||
at = limit
|
||||
}
|
||||
// MLR reg. 40(4): records of a transaction occurring during a relationship
|
||||
// need not be kept beyond ten years from the transaction. Applied only to an
|
||||
// extension, never below the floor — where the two instruments disagree the
|
||||
// mandatory floor wins, because destroying a record an instrument requires
|
||||
// kept is the worse of the two failures. A UK-only deployment wanting the
|
||||
// full effect of reg. 40(4) needs a compliance decision, not a code change.
|
||||
if r.Class == ClassTransaction && !r.Occurred.IsZero() {
|
||||
if c := r.Occurred.AddDate(Ceiling, 0, 0); at.After(c) && !c.Before(floor) {
|
||||
at = c
|
||||
}
|
||||
}
|
||||
return at
|
||||
}
|
||||
|
||||
// Expired reports whether the retention period has run out.
|
||||
func (r Record) Expired(now time.Time) bool {
|
||||
e := r.Expiry()
|
||||
return !e.IsZero() && !e.After(now)
|
||||
}
|
||||
|
||||
// clone deep-copies a record. Reads hand out clones so that no reader can reach
|
||||
// into the ledger and unmake part of a retained record.
|
||||
func (r Record) clone() Record {
|
||||
out := r
|
||||
out.Parties = append([]string(nil), r.Parties...)
|
||||
out.Body = append([]byte(nil), r.Body...)
|
||||
if r.Extended != nil {
|
||||
e := *r.Extended
|
||||
out.Extended = &e
|
||||
}
|
||||
if r.Assessment != nil {
|
||||
a := r.Assessment.clone()
|
||||
out.Assessment = &a
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (r Record) validate() error {
|
||||
if r.Org == "" {
|
||||
return ErrOrg
|
||||
}
|
||||
switch r.Class {
|
||||
case ClassRelationship, ClassTransaction, ClassAssessment, ClassRefusal:
|
||||
default:
|
||||
return fmt.Errorf("%w: %q", ErrClass, r.Class)
|
||||
}
|
||||
switch r.Trigger {
|
||||
case TriggerRelationshipEnd, TriggerOccasional, TriggerRefusal:
|
||||
default:
|
||||
return fmt.Errorf("%w: %q", ErrTrigger, r.Trigger)
|
||||
}
|
||||
if len(r.Parties) == 0 {
|
||||
return ErrParties
|
||||
}
|
||||
for _, p := range r.Parties {
|
||||
if p == "" {
|
||||
return ErrParties
|
||||
}
|
||||
}
|
||||
if r.Occurred.IsZero() {
|
||||
return ErrOccurred
|
||||
}
|
||||
|
||||
switch r.Class {
|
||||
case ClassRelationship:
|
||||
if r.Nature == "" {
|
||||
return ErrNature
|
||||
}
|
||||
case ClassTransaction:
|
||||
if len(r.Body) == 0 {
|
||||
return ErrBody
|
||||
}
|
||||
// A transaction record with no reference cannot be tied back to the
|
||||
// transaction it is supposed to reconstruct.
|
||||
if r.Ref == "" {
|
||||
return ErrRef
|
||||
}
|
||||
case ClassRefusal:
|
||||
if r.Reason == "" {
|
||||
return ErrReason
|
||||
}
|
||||
case ClassAssessment:
|
||||
if r.Assessment == nil {
|
||||
return fmt.Errorf("%w: no assessment", ErrAssessment)
|
||||
}
|
||||
if err := r.Assessment.validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// checkNow refuses a date the caller's own clock has not reached. Disposal is
|
||||
// destruction, and a lie about the time must not be able to bring it forward.
|
||||
func checkNow(now time.Time) error {
|
||||
if now.After(time.Now().UTC().Add(skew)) {
|
||||
return fmt.Errorf("%w: %s", ErrFuture, now.UTC().Format(time.RFC3339))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,209 @@
|
||||
package retention
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func date(y int, m time.Month, d int) time.Time {
|
||||
return time.Date(y, m, d, 0, 0, 0, 0, time.UTC)
|
||||
}
|
||||
|
||||
// TestExpiryIsFiveCalendarYears: the period is five years, not 1825 days. The
|
||||
// dates here span two leap years, so a duration constant gets this wrong.
|
||||
func TestExpiryIsFiveCalendarYears(t *testing.T) {
|
||||
for _, c := range []struct{ start, want time.Time }{
|
||||
{date(2020, time.January, 1), date(2025, time.January, 1)},
|
||||
{date(2020, time.February, 29), date(2025, time.March, 1)},
|
||||
{date(2021, time.June, 30), date(2026, time.June, 30)},
|
||||
} {
|
||||
r := Record{Class: ClassRefusal, Trigger: TriggerRefusal, Start: c.start}
|
||||
if got := r.Expiry(); !got.Equal(c.want) {
|
||||
t.Errorf("start %s: expiry = %s, want %s",
|
||||
c.start.Format(time.DateOnly), got.Format(time.DateOnly), c.want.Format(time.DateOnly))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestClockNotStartedNeverExpires: an open business relationship has no expiry,
|
||||
// because the five years run from the end of it. A record with no clock must
|
||||
// never be swept up by a disposal run.
|
||||
func TestClockNotStartedNeverExpires(t *testing.T) {
|
||||
r := Record{Class: ClassRelationship, Trigger: TriggerRelationshipEnd, Occurred: date(1999, time.January, 1)}
|
||||
|
||||
if got := r.Expiry(); !got.IsZero() {
|
||||
t.Fatalf("expiry = %s, want zero", got)
|
||||
}
|
||||
if r.Expired(time.Now().UTC().AddDate(500, 0, 0)) {
|
||||
t.Fatal("a relationship with no end date expired")
|
||||
}
|
||||
}
|
||||
|
||||
// TestExpiredBoundary: the record is expired on the expiry date, not the day
|
||||
// after. Deletion is due when the period has run out.
|
||||
func TestExpiredBoundary(t *testing.T) {
|
||||
start := date(2020, time.January, 1)
|
||||
r := Record{Class: ClassRefusal, Trigger: TriggerRefusal, Start: start}
|
||||
expiry := date(2025, time.January, 1)
|
||||
|
||||
if r.Expired(expiry.Add(-time.Nanosecond)) {
|
||||
t.Error("expired one instant early")
|
||||
}
|
||||
if !r.Expired(expiry) {
|
||||
t.Error("not expired on the expiry date")
|
||||
}
|
||||
if !r.Expired(expiry.Add(time.Hour)) {
|
||||
t.Error("not expired after the expiry date")
|
||||
}
|
||||
}
|
||||
|
||||
// TestExtensionCappedAtFiveFurtherYears: whatever an extension claims, the
|
||||
// expiry never passes ten years from the trigger.
|
||||
func TestExtensionCappedAtFiveFurtherYears(t *testing.T) {
|
||||
start := date(2020, time.January, 1)
|
||||
r := Record{
|
||||
Class: ClassRefusal,
|
||||
Trigger: TriggerRefusal,
|
||||
Start: start,
|
||||
Extended: &Extended{By: 40 * 365 * 24 * time.Hour, Reason: "forged", Who: "nobody"},
|
||||
}
|
||||
|
||||
want := date(2030, time.January, 1)
|
||||
if got := r.Expiry(); !got.Equal(want) {
|
||||
t.Fatalf("expiry = %s, want %s (five years plus five further)",
|
||||
got.Format(time.DateOnly), want.Format(time.DateOnly))
|
||||
}
|
||||
}
|
||||
|
||||
// TestExtensionMovesExpiry: an extension inside the cap does what it says.
|
||||
func TestExtensionMovesExpiry(t *testing.T) {
|
||||
start := date(2020, time.January, 1)
|
||||
plain := Record{Class: ClassRefusal, Trigger: TriggerRefusal, Start: start}
|
||||
extended := plain
|
||||
extended.Extended = &Extended{By: 365 * 24 * time.Hour, Reason: "live investigation", Who: "mlro"}
|
||||
|
||||
if !extended.Expiry().After(plain.Expiry()) {
|
||||
t.Fatalf("extended expiry %s is not after %s", extended.Expiry(), plain.Expiry())
|
||||
}
|
||||
if got, want := extended.Expiry(), plain.Expiry().Add(365*24*time.Hour); !got.Equal(want) {
|
||||
t.Fatalf("expiry = %s, want %s", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTransactionCeilingBoundsExtensionsNotTheFloor is the reg. 40(4) ten-year
|
||||
// ceiling, and the resolution of its conflict with the AMLR: it can shorten an
|
||||
// extension, and it can never shorten the five years the AMLR requires from the
|
||||
// end of the relationship.
|
||||
func TestTransactionCeilingBoundsExtensionsNotTheFloor(t *testing.T) {
|
||||
// Transaction in 2019, relationship ended 2022. Floor 2027, extension to
|
||||
// 2032, ceiling 2029 — the ceiling bites.
|
||||
bites := Record{
|
||||
Class: ClassTransaction,
|
||||
Trigger: TriggerRelationshipEnd,
|
||||
Occurred: date(2019, time.January, 1),
|
||||
Start: date(2022, time.January, 1),
|
||||
Extended: &Extended{By: 5 * 365 * 24 * time.Hour, Reason: "r", Who: "w"},
|
||||
}
|
||||
if got, want := bites.Expiry(), date(2029, time.January, 1); !got.Equal(want) {
|
||||
t.Errorf("ceiling did not bind the extension: expiry = %s, want %s",
|
||||
got.Format(time.DateOnly), want.Format(time.DateOnly))
|
||||
}
|
||||
|
||||
// Transaction in 2010, relationship ended 2024. Ceiling 2020 is before the
|
||||
// mandatory floor 2029, so the floor wins and nothing is destroyed early.
|
||||
floorWins := Record{
|
||||
Class: ClassTransaction,
|
||||
Trigger: TriggerRelationshipEnd,
|
||||
Occurred: date(2010, time.January, 1),
|
||||
Start: date(2024, time.January, 1),
|
||||
}
|
||||
if got, want := floorWins.Expiry(), date(2029, time.January, 1); !got.Equal(want) {
|
||||
t.Errorf("ceiling cut below the mandatory period: expiry = %s, want %s",
|
||||
got.Format(time.DateOnly), want.Format(time.DateOnly))
|
||||
}
|
||||
|
||||
// And it does not touch other classes: a refusal is not an in-relationship
|
||||
// transaction record.
|
||||
refusal := Record{
|
||||
Class: ClassRefusal,
|
||||
Trigger: TriggerRefusal,
|
||||
Occurred: date(2010, time.January, 1),
|
||||
Start: date(2024, time.January, 1),
|
||||
}
|
||||
if got, want := refusal.Expiry(), date(2029, time.January, 1); !got.Equal(want) {
|
||||
t.Errorf("refusal expiry = %s, want %s", got.Format(time.DateOnly), want.Format(time.DateOnly))
|
||||
}
|
||||
}
|
||||
|
||||
// TestAssessmentRequiresItsParts is the record most systems miss: a dismissed
|
||||
// alert is a retained decision with its rationale. An assessment missing any
|
||||
// part of Art. 69(2) is refused, so it cannot be retained as an empty gesture.
|
||||
func TestAssessmentRequiresItsParts(t *testing.T) {
|
||||
whole := Assessment{
|
||||
Considered: []string{"velocity over 30 days", "customer profile"},
|
||||
Result: NotReported,
|
||||
Rationale: "salary payments consistent with the profile on file",
|
||||
By: "mlro",
|
||||
At: time.Now().UTC(),
|
||||
}
|
||||
if err := whole.validate(); err != nil {
|
||||
t.Fatalf("complete assessment refused: %v", err)
|
||||
}
|
||||
|
||||
for name, broken := range map[string]func(*Assessment){
|
||||
"no result": func(a *Assessment) { a.Result = "" },
|
||||
"bad result": func(a *Assessment) { a.Result = "maybe" },
|
||||
"nothing considered": func(a *Assessment) { a.Considered = nil },
|
||||
"no rationale": func(a *Assessment) { a.Rationale = "" },
|
||||
"no decider": func(a *Assessment) { a.By = "" },
|
||||
"no date": func(a *Assessment) { a.At = time.Time{} },
|
||||
} {
|
||||
a := whole.clone()
|
||||
broken(&a)
|
||||
if err := a.validate(); !errors.Is(err, ErrAssessment) {
|
||||
t.Errorf("%s: err = %v, want ErrAssessment", name, err)
|
||||
}
|
||||
}
|
||||
|
||||
// A report needs its reasons too: AMLR Art. 77(1)(b) retains the result
|
||||
// either way.
|
||||
reported := whole.clone()
|
||||
reported.Result = Reported
|
||||
reported.Rationale = ""
|
||||
if err := reported.validate(); !errors.Is(err, ErrAssessment) {
|
||||
t.Errorf("reported without rationale: err = %v, want ErrAssessment", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCheckNowRefusesTheFuture: destruction does not run on a date the clock has
|
||||
// not reached, so a clock lie cannot bring it forward.
|
||||
func TestCheckNowRefusesTheFuture(t *testing.T) {
|
||||
if err := checkNow(time.Now().UTC()); err != nil {
|
||||
t.Errorf("now refused: %v", err)
|
||||
}
|
||||
if err := checkNow(time.Now().UTC().Add(30 * time.Second)); err != nil {
|
||||
t.Errorf("now within clock skew refused: %v", err)
|
||||
}
|
||||
if err := checkNow(time.Now().UTC().Add(time.Hour)); !errors.Is(err, ErrFuture) {
|
||||
t.Errorf("an hour ahead: err = %v, want ErrFuture", err)
|
||||
}
|
||||
if err := checkNow(time.Now().UTC().AddDate(6, 0, 0)); !errors.Is(err, ErrFuture) {
|
||||
t.Errorf("six years ahead: err = %v, want ErrFuture", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPurposeIsClosed: only money laundering and terrorist financing prevention.
|
||||
// Anything else has no representation the ledger accepts.
|
||||
func TestPurposeIsClosed(t *testing.T) {
|
||||
for _, p := range []Purpose{PurposeMonitoring, PurposeInvestigation, PurposeDisclosure, PurposeRetention} {
|
||||
if !p.permitted() {
|
||||
t.Errorf("%q should be permitted", p)
|
||||
}
|
||||
}
|
||||
for _, p := range []Purpose{"", "marketing", "analytics", "credit scoring", "training", "MONITORING"} {
|
||||
if p.permitted() {
|
||||
t.Errorf("%q should not be permitted", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,292 @@
|
||||
// Copyright 2024-2026 Lux Partners Limited. All rights reserved.
|
||||
// Use of this source code is governed by the Apache 2.0
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// Package token protects direct identifiers at rest while keeping the engine
|
||||
// able to correlate.
|
||||
//
|
||||
// Two operations over one key schedule:
|
||||
//
|
||||
// - [Vault.Pseudonym] is a deterministic, org-scoped, domain-separated token.
|
||||
// It is what the engine indexes and correlates on, so a retained record can
|
||||
// be found by party without the party's name being held in the clear.
|
||||
// - [Vault.Seal] is authenticated encryption of a whole retained record. It is
|
||||
// reversible under the org's key, which is what keeps a tokenised record
|
||||
// able to reconstruct the transaction (MLR 2017 reg. 40(2)(b)) and able to
|
||||
// be re-screened when a new designation enters into force
|
||||
// (EBA/GL/2024/15 §4.1.4 para 16(a)).
|
||||
//
|
||||
// The requirement served is purpose limitation — retained personal data may be
|
||||
// processed only to prevent money laundering and terrorist financing, and
|
||||
// processing for commercial purposes is prohibited (Directive (EU) 2015/849
|
||||
// Art. 41(2)) — together with data minimisation. It is not obfuscation for its
|
||||
// own sake, and it is deliberately not one-way: a hash-only design can neither
|
||||
// reconstruct a transaction nor re-screen a customer base against a new
|
||||
// designation, so it would fail the obligations it is supposed to be protecting.
|
||||
//
|
||||
// Determinism is per org and never across orgs. The org is the HKDF salt, so
|
||||
// one value under two orgs yields two unrelated pseudonyms: a cross-tenant join
|
||||
// is not merely disallowed, it is not computable (HIP-0302).
|
||||
//
|
||||
// Key material comes from KMS. [Source] is the seam; there is no literal key,
|
||||
// no default and no plaintext path. A vault without key material refuses.
|
||||
package token
|
||||
|
||||
import (
|
||||
"crypto/aes"
|
||||
"crypto/cipher"
|
||||
"crypto/hkdf"
|
||||
"crypto/hmac"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"golang.org/x/text/unicode/norm"
|
||||
)
|
||||
|
||||
// Errors. Every one of them is a refusal: nothing here degrades to plaintext.
|
||||
var (
|
||||
ErrNoKey = errors.New("token: no key material")
|
||||
ErrWeakKey = errors.New("token: key material is unusable")
|
||||
ErrOrg = errors.New("token: empty org")
|
||||
ErrDomain = errors.New("token: unknown domain")
|
||||
ErrEmpty = errors.New("token: empty value")
|
||||
ErrSealed = errors.New("token: sealed record does not open")
|
||||
ErrNoSource = errors.New("token: no key source")
|
||||
)
|
||||
|
||||
// keyLen is the length of every derived key, and the minimum length of the root
|
||||
// material they are derived from.
|
||||
const keyLen = 32
|
||||
|
||||
// pseudonymLen is how much of the MAC a pseudonym carries. 16 bytes is a
|
||||
// 128-bit token: the birthday bound is 2^64 values per org and domain, which no
|
||||
// customer base approaches, and a shorter token is easier to hold in a case file.
|
||||
const pseudonymLen = 16
|
||||
|
||||
// Domain names the kind of identifier a token stands for. Each domain has its
|
||||
// own key, so a token minted in one domain cannot be joined against another.
|
||||
type Domain string
|
||||
|
||||
const (
|
||||
// DomainSubject is the party identifier the firm itself holds — a user id,
|
||||
// an external id, a counterparty reference.
|
||||
DomainSubject Domain = "subject"
|
||||
// DomainName is a natural or legal person's name.
|
||||
DomainName Domain = "name"
|
||||
// DomainAccount is an account identifier.
|
||||
DomainAccount Domain = "account"
|
||||
// DomainWallet is a distributed-ledger address. EBA/GL/2024/15 §4.1.4
|
||||
// para 17(c) makes these screening fields where the lists carry them.
|
||||
DomainWallet Domain = "wallet"
|
||||
// DomainDevice is a device fingerprint.
|
||||
DomainDevice Domain = "device"
|
||||
// DomainNetwork is a network address.
|
||||
DomainNetwork Domain = "network"
|
||||
)
|
||||
|
||||
// domains is every domain a vault derives a key for. A domain absent here has
|
||||
// no key and is refused rather than defaulted.
|
||||
var domains = []Domain{
|
||||
DomainSubject, DomainName, DomainAccount,
|
||||
DomainWallet, DomainDevice, DomainNetwork,
|
||||
}
|
||||
|
||||
// canonical is the form a domain's values are tokenised in, and reports whether
|
||||
// the domain is known. Determinism is only meaningful modulo a canonical form,
|
||||
// so the form belongs to the domain rather than to each caller.
|
||||
func canonical(d Domain, v string) (string, bool) {
|
||||
v = norm.NFKC.String(strings.TrimSpace(v))
|
||||
switch d {
|
||||
case DomainName:
|
||||
// A name is not an identifier: case and internal spacing carry nothing.
|
||||
// Homoglyphs and transliterations survive this and are the fuzzy
|
||||
// matcher's problem, not the tokeniser's.
|
||||
return strings.Join(strings.Fields(strings.ToLower(v)), " "), true
|
||||
case DomainSubject, DomainAccount, DomainWallet, DomainDevice, DomainNetwork:
|
||||
// Opaque handles. Case can distinguish two real identifiers — base58
|
||||
// ledger addresses are the clear case — so it is preserved.
|
||||
return v, true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// Vault holds one org's derived keys.
|
||||
type Vault struct {
|
||||
org string
|
||||
seal cipher.AEAD
|
||||
keys map[Domain][]byte
|
||||
}
|
||||
|
||||
// New derives an org's vault from root key material. It refuses an empty org,
|
||||
// and refuses material shorter than 32 bytes or entirely zero: a tokeniser
|
||||
// without a key refuses, it does not pass plaintext through.
|
||||
func New(org string, root []byte) (*Vault, error) {
|
||||
if org == "" {
|
||||
return nil, ErrOrg
|
||||
}
|
||||
if len(root) == 0 {
|
||||
return nil, ErrNoKey
|
||||
}
|
||||
if len(root) < keyLen {
|
||||
return nil, fmt.Errorf("%w: %d bytes, need %d", ErrWeakKey, len(root), keyLen)
|
||||
}
|
||||
if allZero(root) {
|
||||
return nil, fmt.Errorf("%w: all-zero material is a placeholder, not a key", ErrWeakKey)
|
||||
}
|
||||
|
||||
salt := []byte("aml/token/org/" + org)
|
||||
keys := make(map[Domain][]byte, len(domains))
|
||||
for _, d := range domains {
|
||||
k, err := hkdf.Key(sha256.New, root, salt, "pseudonym/"+string(d), keyLen)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("token: derive %s: %w", d, err)
|
||||
}
|
||||
keys[d] = k
|
||||
}
|
||||
|
||||
sk, err := hkdf.Key(sha256.New, root, salt, "seal", keyLen)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("token: derive seal: %w", err)
|
||||
}
|
||||
block, err := aes.NewCipher(sk)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("token: cipher: %w", err)
|
||||
}
|
||||
aead, err := cipher.NewGCM(block)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("token: gcm: %w", err)
|
||||
}
|
||||
|
||||
return &Vault{org: org, seal: aead, keys: keys}, nil
|
||||
}
|
||||
|
||||
func allZero(b []byte) bool {
|
||||
for _, c := range b {
|
||||
if c != 0 {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// Pseudonym returns the token for a value: deterministic within this org and
|
||||
// domain, unrelated to the same value under any other org or domain.
|
||||
func (v *Vault) Pseudonym(d Domain, value string) (string, error) {
|
||||
c, ok := canonical(d, value)
|
||||
if !ok {
|
||||
return "", fmt.Errorf("%w: %q", ErrDomain, d)
|
||||
}
|
||||
if c == "" {
|
||||
return "", ErrEmpty
|
||||
}
|
||||
mac := hmac.New(sha256.New, v.keys[d])
|
||||
mac.Write([]byte(c))
|
||||
return string(d) + ":" + hex.EncodeToString(mac.Sum(nil)[:pseudonymLen]), nil
|
||||
}
|
||||
|
||||
// Seal encrypts a retained record. slot binds the ciphertext to where it is
|
||||
// stored and the org is bound with it, so a sealed record neither opens under
|
||||
// another org nor moves to another slot.
|
||||
func (v *Vault) Seal(slot string, plain []byte) ([]byte, error) {
|
||||
nonce := make([]byte, v.seal.NonceSize())
|
||||
if _, err := rand.Read(nonce); err != nil {
|
||||
return nil, fmt.Errorf("token: nonce: %w", err)
|
||||
}
|
||||
return v.seal.Seal(nonce, nonce, plain, v.aad(slot)), nil
|
||||
}
|
||||
|
||||
// Open decrypts a record sealed for this org and slot.
|
||||
func (v *Vault) Open(slot string, sealed []byte) ([]byte, error) {
|
||||
n := v.seal.NonceSize()
|
||||
if len(sealed) < n+v.seal.Overhead() {
|
||||
return nil, fmt.Errorf("%w: %d bytes", ErrSealed, len(sealed))
|
||||
}
|
||||
out, err := v.seal.Open(nil, sealed[:n], sealed[n:], v.aad(slot))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %s", ErrSealed, slot)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (v *Vault) aad(slot string) []byte {
|
||||
return []byte("aml/token/v1\x00" + v.org + "\x00" + slot)
|
||||
}
|
||||
|
||||
// Source supplies root key material for an org.
|
||||
//
|
||||
// The engine does not talk to KMS itself: the platform (credz, or a KMSSecret
|
||||
// synced into the pod) puts the material where a Source reads it. What the
|
||||
// engine needs is one secret of at least 32 random bytes per deployment,
|
||||
// rotated by the platform. The org parameter is here so that per-org material —
|
||||
// where one org's keys must be independently revocable — is a different Source
|
||||
// and not a different design.
|
||||
type Source func(org string) ([]byte, error)
|
||||
|
||||
// Env reads root key material from a process environment variable, which is
|
||||
// where KMS-mounted material lands. Hex encoded, at least 32 bytes. There is no
|
||||
// default and no literal: unset, malformed, short or all-zero is a refusal.
|
||||
func Env(name string) Source {
|
||||
return func(string) ([]byte, error) {
|
||||
raw := strings.TrimSpace(os.Getenv(name))
|
||||
if raw == "" {
|
||||
return nil, fmt.Errorf("%w: %s is not set", ErrNoKey, name)
|
||||
}
|
||||
b, err := hex.DecodeString(raw)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %s is not hex", ErrWeakKey, name)
|
||||
}
|
||||
if len(b) < keyLen {
|
||||
return nil, fmt.Errorf("%w: %s is %d bytes, need %d", ErrWeakKey, name, len(b), keyLen)
|
||||
}
|
||||
if allZero(b) {
|
||||
return nil, fmt.Errorf("%w: %s is all zero", ErrWeakKey, name)
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
}
|
||||
|
||||
// Keyring holds one vault per org, derived on demand from a Source. Failures
|
||||
// are not cached: material that a KMS sync has not delivered yet must be
|
||||
// refused now and honoured later.
|
||||
type Keyring struct {
|
||||
src Source
|
||||
mu sync.Mutex
|
||||
vaults map[string]*Vault
|
||||
}
|
||||
|
||||
// NewKeyring returns a keyring over a Source.
|
||||
func NewKeyring(src Source) *Keyring {
|
||||
return &Keyring{src: src, vaults: make(map[string]*Vault)}
|
||||
}
|
||||
|
||||
// Org returns the org's vault.
|
||||
func (k *Keyring) Org(org string) (*Vault, error) {
|
||||
if org == "" {
|
||||
return nil, ErrOrg
|
||||
}
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
|
||||
if v, ok := k.vaults[org]; ok {
|
||||
return v, nil
|
||||
}
|
||||
if k.src == nil {
|
||||
return nil, ErrNoSource
|
||||
}
|
||||
root, err := k.src(org)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
v, err := New(org, root)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
k.vaults[org] = v
|
||||
return v, nil
|
||||
}
|
||||
@@ -0,0 +1,342 @@
|
||||
package token
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// rootA and rootB are two distinct 32-byte roots. Fixed, so a failure is a
|
||||
// failure of the construction and not of the day's randomness.
|
||||
var (
|
||||
rootA = mustHex("6c75782d616d6c2d746f6b656e2d726f6f742d612d33326279746573212121")
|
||||
rootB = mustHex("6c75782d616d6c2d746f6b656e2d726f6f742d622d33326279746573212121")
|
||||
)
|
||||
|
||||
func mustHex(s string) []byte {
|
||||
for len(s) < keyLen*2 {
|
||||
s += "0f"
|
||||
}
|
||||
b, err := hex.DecodeString(s)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func vault(t *testing.T, org string, root []byte) *Vault {
|
||||
t.Helper()
|
||||
v, err := New(org, root)
|
||||
if err != nil {
|
||||
t.Fatalf("New(%q): %v", org, err)
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// TestPseudonymDeterministic is the property correlation rests on: the same
|
||||
// value under the same org and domain is always the same token, so two records
|
||||
// about one party can be found together.
|
||||
func TestPseudonymDeterministic(t *testing.T) {
|
||||
v := vault(t, "acme", rootA)
|
||||
|
||||
first, err := v.Pseudonym(DomainName, "Ivan Petrov")
|
||||
if err != nil {
|
||||
t.Fatalf("Pseudonym: %v", err)
|
||||
}
|
||||
second, err := v.Pseudonym(DomainName, "Ivan Petrov")
|
||||
if err != nil {
|
||||
t.Fatalf("Pseudonym: %v", err)
|
||||
}
|
||||
if first != second {
|
||||
t.Fatalf("not deterministic: %q != %q", first, second)
|
||||
}
|
||||
|
||||
// A second vault over the same root and org must agree, or a restart loses
|
||||
// every correlation the index holds.
|
||||
if again, err := vault(t, "acme", rootA).Pseudonym(DomainName, "Ivan Petrov"); err != nil || again != first {
|
||||
t.Fatalf("not stable across vaults: %q vs %q (err %v)", again, first, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPseudonymNeverCorrelatesAcrossOrgs is the cross-tenant guarantee: the same
|
||||
// person under two orgs is two unrelated tokens, so there is no join to make.
|
||||
func TestPseudonymNeverCorrelatesAcrossOrgs(t *testing.T) {
|
||||
const name = "Ivan Petrov"
|
||||
|
||||
// Same root, different org: the org is the salt, so the keys differ.
|
||||
a, err := vault(t, "acme", rootA).Pseudonym(DomainName, name)
|
||||
if err != nil {
|
||||
t.Fatalf("acme: %v", err)
|
||||
}
|
||||
b, err := vault(t, "beta", rootA).Pseudonym(DomainName, name)
|
||||
if err != nil {
|
||||
t.Fatalf("beta: %v", err)
|
||||
}
|
||||
if a == b {
|
||||
t.Fatalf("cross-org join is computable: %q == %q", a, b)
|
||||
}
|
||||
|
||||
// Different root as well, for the per-org-material Source shape.
|
||||
c, err := vault(t, "beta", rootB).Pseudonym(DomainName, name)
|
||||
if err != nil {
|
||||
t.Fatalf("beta/rootB: %v", err)
|
||||
}
|
||||
if c == a || c == b {
|
||||
t.Fatalf("distinct material still correlates: %q %q %q", a, b, c)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPseudonymDomainSeparated stops an account token being compared against a
|
||||
// name token: a match across domains would be a fabricated link.
|
||||
func TestPseudonymDomainSeparated(t *testing.T) {
|
||||
v := vault(t, "acme", rootA)
|
||||
const value = "same-string"
|
||||
|
||||
seen := make(map[string]Domain, len(domains))
|
||||
for _, d := range domains {
|
||||
p, err := v.Pseudonym(d, value)
|
||||
if err != nil {
|
||||
t.Fatalf("Pseudonym(%s): %v", d, err)
|
||||
}
|
||||
if !strings.HasPrefix(p, string(d)+":") {
|
||||
t.Errorf("%s token %q does not name its domain", d, p)
|
||||
}
|
||||
if prev, dup := seen[p]; dup {
|
||||
t.Errorf("domains %s and %s collide on %q", prev, d, p)
|
||||
}
|
||||
seen[p] = d
|
||||
}
|
||||
}
|
||||
|
||||
// TestPseudonymUnknownDomainRefused: an invented domain has no key, and
|
||||
// defaulting it would silently put two kinds of identifier in one namespace.
|
||||
func TestPseudonymUnknownDomainRefused(t *testing.T) {
|
||||
v := vault(t, "acme", rootA)
|
||||
if _, err := v.Pseudonym(Domain("passport"), "X1234"); !errors.Is(err, ErrDomain) {
|
||||
t.Fatalf("unknown domain: err = %v, want ErrDomain", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPseudonymEmptyValueRefused: an empty identifier is not an identifier, and
|
||||
// indexing one would make every party with a missing field the same party.
|
||||
func TestPseudonymEmptyValueRefused(t *testing.T) {
|
||||
v := vault(t, "acme", rootA)
|
||||
for _, value := range []string{"", " ", "\t\n", " "} {
|
||||
if _, err := v.Pseudonym(DomainName, value); !errors.Is(err, ErrEmpty) {
|
||||
t.Errorf("Pseudonym(%q): err = %v, want ErrEmpty", value, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestCanonicalForm pins what a domain treats as the same value. Names fold;
|
||||
// opaque handles do not, because case distinguishes real base58 addresses.
|
||||
func TestCanonicalForm(t *testing.T) {
|
||||
v := vault(t, "acme", rootA)
|
||||
|
||||
same := func(d Domain, a, b string) {
|
||||
t.Helper()
|
||||
pa, err := v.Pseudonym(d, a)
|
||||
if err != nil {
|
||||
t.Fatalf("%s %q: %v", d, a, err)
|
||||
}
|
||||
pb, err := v.Pseudonym(d, b)
|
||||
if err != nil {
|
||||
t.Fatalf("%s %q: %v", d, b, err)
|
||||
}
|
||||
if pa != pb {
|
||||
t.Errorf("%s: %q and %q should be one value", d, a, b)
|
||||
}
|
||||
}
|
||||
differ := func(d Domain, a, b string) {
|
||||
t.Helper()
|
||||
pa, _ := v.Pseudonym(d, a)
|
||||
pb, _ := v.Pseudonym(d, b)
|
||||
if pa == pb {
|
||||
t.Errorf("%s: %q and %q should be two values", d, a, b)
|
||||
}
|
||||
}
|
||||
|
||||
same(DomainName, "Ivan Petrov", " ivan PETROV ")
|
||||
same(DomainName, "Ｉvan Petrov", "Ivan Petrov") // NFKC fullwidth I
|
||||
differ(DomainName, "Ivan Petrov", "Ivana Petrov")
|
||||
|
||||
same(DomainWallet, " 1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa ", "1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa")
|
||||
differ(DomainWallet, "1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa", "1a1zp1ep5qgefi2dmptftl5slmv7divfna")
|
||||
differ(DomainAccount, "GB33BUKB20201555555555", "gb33bukb20201555555555")
|
||||
}
|
||||
|
||||
// TestNoKeyRefuses: a tokeniser without usable material must refuse. Every one
|
||||
// of these inputs is a way a deployment ends up with no real key.
|
||||
func TestNoKeyRefuses(t *testing.T) {
|
||||
if _, err := New("acme", nil); !errors.Is(err, ErrNoKey) {
|
||||
t.Errorf("nil root: err = %v, want ErrNoKey", err)
|
||||
}
|
||||
if _, err := New("acme", []byte{}); !errors.Is(err, ErrNoKey) {
|
||||
t.Errorf("empty root: err = %v, want ErrNoKey", err)
|
||||
}
|
||||
if _, err := New("acme", bytes.Repeat([]byte{7}, keyLen-1)); !errors.Is(err, ErrWeakKey) {
|
||||
t.Errorf("short root: err = %v, want ErrWeakKey", err)
|
||||
}
|
||||
if _, err := New("acme", make([]byte, keyLen)); !errors.Is(err, ErrWeakKey) {
|
||||
t.Errorf("all-zero root: err = %v, want ErrWeakKey", err)
|
||||
}
|
||||
if _, err := New("", rootA); !errors.Is(err, ErrOrg) {
|
||||
t.Errorf("empty org: err = %v, want ErrOrg", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestEnvHasNoDefault: the key comes from KMS-mounted material, so an unset or
|
||||
// placeholder variable is a refusal and never a fallback value.
|
||||
func TestEnvHasNoDefault(t *testing.T) {
|
||||
const name = "AML_TOKEN_KEY_TEST"
|
||||
src := Env(name)
|
||||
|
||||
t.Setenv(name, "")
|
||||
if _, err := src("acme"); !errors.Is(err, ErrNoKey) {
|
||||
t.Errorf("unset: err = %v, want ErrNoKey", err)
|
||||
}
|
||||
|
||||
for _, bad := range []string{"not-hex-at-all", hex.EncodeToString(make([]byte, keyLen)), "0a0b0c"} {
|
||||
t.Setenv(name, bad)
|
||||
if _, err := src("acme"); !errors.Is(err, ErrWeakKey) {
|
||||
t.Errorf("Env(%q): err = %v, want ErrWeakKey", bad, err)
|
||||
}
|
||||
}
|
||||
|
||||
t.Setenv(name, hex.EncodeToString(rootA))
|
||||
got, err := src("acme")
|
||||
if err != nil || !bytes.Equal(got, rootA) {
|
||||
t.Fatalf("Env: got %x err %v, want %x", got, err, rootA)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSealRoundTrip: the record comes back whole, which is what "not redacted"
|
||||
// and "reconstruct the transaction" require of the storage layer.
|
||||
func TestSealRoundTrip(t *testing.T) {
|
||||
v := vault(t, "acme", rootA)
|
||||
plain := []byte(`{"tx":{"id":"tx-1","notional":15000},"entity":{"name":"Ivan Petrov"}}`)
|
||||
|
||||
sealed, err := v.Seal("record:r-1", plain)
|
||||
if err != nil {
|
||||
t.Fatalf("Seal: %v", err)
|
||||
}
|
||||
if bytes.Contains(sealed, []byte("Ivan")) {
|
||||
t.Fatal("sealed record carries the plaintext")
|
||||
}
|
||||
|
||||
out, err := v.Open("record:r-1", sealed)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
if !bytes.Equal(out, plain) {
|
||||
t.Fatalf("Open = %q, want %q", out, plain)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSealIsFresh: two seals of one record differ, so ciphertext equality never
|
||||
// discloses that two records hold the same body.
|
||||
func TestSealIsFresh(t *testing.T) {
|
||||
v := vault(t, "acme", rootA)
|
||||
plain := []byte("same body")
|
||||
|
||||
first, err := v.Seal("record:r-1", plain)
|
||||
if err != nil {
|
||||
t.Fatalf("Seal: %v", err)
|
||||
}
|
||||
second, err := v.Seal("record:r-1", plain)
|
||||
if err != nil {
|
||||
t.Fatalf("Seal: %v", err)
|
||||
}
|
||||
if bytes.Equal(first, second) {
|
||||
t.Fatal("seal is deterministic: nonce is not fresh")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSealedRecordDoesNotOpenElsewhere: not under another org, not in another
|
||||
// slot, not after a single flipped bit.
|
||||
func TestSealedRecordDoesNotOpenElsewhere(t *testing.T) {
|
||||
acme := vault(t, "acme", rootA)
|
||||
beta := vault(t, "beta", rootA) // same root, other org
|
||||
|
||||
sealed, err := acme.Seal("record:r-1", []byte("body"))
|
||||
if err != nil {
|
||||
t.Fatalf("Seal: %v", err)
|
||||
}
|
||||
|
||||
if _, err := beta.Open("record:r-1", sealed); !errors.Is(err, ErrSealed) {
|
||||
t.Errorf("cross-org open: err = %v, want ErrSealed", err)
|
||||
}
|
||||
if _, err := acme.Open("record:r-2", sealed); !errors.Is(err, ErrSealed) {
|
||||
t.Errorf("cross-slot open: err = %v, want ErrSealed", err)
|
||||
}
|
||||
|
||||
tampered := bytes.Clone(sealed)
|
||||
tampered[len(tampered)-1] ^= 1
|
||||
if _, err := acme.Open("record:r-1", tampered); !errors.Is(err, ErrSealed) {
|
||||
t.Errorf("tampered open: err = %v, want ErrSealed", err)
|
||||
}
|
||||
if _, err := acme.Open("record:r-1", sealed[:acme.seal.NonceSize()]); !errors.Is(err, ErrSealed) {
|
||||
t.Errorf("truncated open: err = %v, want ErrSealed", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAADBindsOrgAndSlot pins both halves of the associated data. The org half
|
||||
// is defence in depth — per-org key derivation already makes a cross-org open
|
||||
// fail — so only this test can catch its removal.
|
||||
func TestAADBindsOrgAndSlot(t *testing.T) {
|
||||
v := vault(t, "acme", rootA)
|
||||
aad := string(v.aad("record:r-1"))
|
||||
|
||||
if !strings.Contains(aad, "acme") {
|
||||
t.Errorf("aad %q does not bind the org", aad)
|
||||
}
|
||||
if !strings.Contains(aad, "record:r-1") {
|
||||
t.Errorf("aad %q does not bind the slot", aad)
|
||||
}
|
||||
}
|
||||
|
||||
// TestKeyringCachesVaultsNotFailures: a key a KMS sync has not delivered yet
|
||||
// must be refused now and honoured when it lands, so a failed lookup cannot be
|
||||
// remembered as "this org has no key".
|
||||
func TestKeyringCachesVaultsNotFailures(t *testing.T) {
|
||||
var calls int
|
||||
var fail = true
|
||||
ring := NewKeyring(func(org string) ([]byte, error) {
|
||||
calls++
|
||||
if fail {
|
||||
return nil, ErrNoKey
|
||||
}
|
||||
return rootA, nil
|
||||
})
|
||||
|
||||
if _, err := ring.Org("acme"); !errors.Is(err, ErrNoKey) {
|
||||
t.Fatalf("first: err = %v, want ErrNoKey", err)
|
||||
}
|
||||
fail = false
|
||||
first, err := ring.Org("acme")
|
||||
if err != nil {
|
||||
t.Fatalf("second: %v", err)
|
||||
}
|
||||
second, err := ring.Org("acme")
|
||||
if err != nil {
|
||||
t.Fatalf("third: %v", err)
|
||||
}
|
||||
if first != second {
|
||||
t.Error("vault not cached: derived twice for one org")
|
||||
}
|
||||
if calls != 2 {
|
||||
t.Errorf("source called %d times, want 2 (one refusal, one success)", calls)
|
||||
}
|
||||
}
|
||||
|
||||
// TestKeyringWithoutSourceRefuses: no seam wired is no key, not a pass-through.
|
||||
func TestKeyringWithoutSourceRefuses(t *testing.T) {
|
||||
if _, err := NewKeyring(nil).Org("acme"); !errors.Is(err, ErrNoSource) {
|
||||
t.Fatalf("nil source: err = %v, want ErrNoSource", err)
|
||||
}
|
||||
if _, err := NewKeyring(Env("AML_TOKEN_KEY_ABSENT")).Org(""); !errors.Is(err, ErrOrg) {
|
||||
t.Fatalf("empty org: err = %v, want ErrOrg", err)
|
||||
}
|
||||
}
|
||||
+80
-1
@@ -24,6 +24,36 @@ const (
|
||||
ActionReport = "report"
|
||||
)
|
||||
|
||||
// ActionCeiling is the strongest action that evidence from a statistical model
|
||||
// may ask for: it escalates to a person, and does not block a payment or file a
|
||||
// report on its own.
|
||||
//
|
||||
// Two reasons, both external to this codebase. A risk score must be overridable
|
||||
// by a person with a documented rationale, which only means something if the
|
||||
// score has not already acted. And the weighting behind a decision must not be
|
||||
// driven by a single factor, which a model's output would be if it could act
|
||||
// alone.
|
||||
const ActionCeiling = ActionReview
|
||||
|
||||
// ActionRank orders actions by how much they demand, so that "the strongest of
|
||||
// these" and "no stronger than this" are one comparison rather than two tables.
|
||||
// An unknown action ranks lowest: an action nobody defined cannot be allowed to
|
||||
// outrank one that was.
|
||||
func ActionRank(action string) int {
|
||||
switch action {
|
||||
case ActionBlock:
|
||||
return 4
|
||||
case ActionReport:
|
||||
return 3
|
||||
case ActionReview:
|
||||
return 2
|
||||
case ActionFlag:
|
||||
return 1
|
||||
default:
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
// Case statuses.
|
||||
const (
|
||||
CaseOpen = "open"
|
||||
@@ -104,6 +134,16 @@ type Transaction struct {
|
||||
Qty float64 `json:"qty"`
|
||||
Notional float64 `json:"notional"`
|
||||
Currency string `json:"currency"`
|
||||
// USD is Notional converted at ingest, and it is set by the engine on the
|
||||
// way in — a value supplied by a client is overwritten.
|
||||
//
|
||||
// It is frozen at ingest rather than converted on read for two reasons. A
|
||||
// threshold applied today over a window of past transactions must not move
|
||||
// because a rate moved, or the same history answers a reporting question
|
||||
// differently on two days. And every aggregate and every ratio computed
|
||||
// downstream is then in one unit, so no comparison carries a silent
|
||||
// assumption about which currency it is in.
|
||||
USD float64 `json:"usd"`
|
||||
Counterparty string `json:"counterparty,omitempty"`
|
||||
IPAddress string `json:"ip_address,omitempty"`
|
||||
DeviceFingerprint string `json:"device_fingerprint,omitempty"`
|
||||
@@ -159,7 +199,10 @@ type Alert struct {
|
||||
Severity string `json:"severity"`
|
||||
Score float64 `json:"score"`
|
||||
ScoreBreakdown map[string]float64 `json:"score_breakdown,omitempty"`
|
||||
ActionTaken string `json:"action_taken"`
|
||||
// Causes carries the per-feature explanation when a model raised this
|
||||
// alert. Empty for a rule alert, whose expression is its explanation.
|
||||
Causes []Cause `json:"causes,omitempty"`
|
||||
ActionTaken string `json:"action_taken"`
|
||||
ReviewedBy string `json:"reviewed_by,omitempty"`
|
||||
ReviewedAt *time.Time `json:"reviewed_at,omitempty"`
|
||||
Decision string `json:"decision,omitempty"`
|
||||
@@ -180,6 +223,9 @@ type Case struct {
|
||||
OpenedAt time.Time `json:"opened_at"`
|
||||
ClosedAt *time.Time `json:"closed_at,omitempty"`
|
||||
Resolution string `json:"resolution,omitempty"`
|
||||
// Assessment is the retained Art. 69(2) decision that closed the case — the
|
||||
// information considered, the result and the reasons. A closed case has one.
|
||||
Assessment string `json:"assessment,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
}
|
||||
@@ -238,12 +284,45 @@ type Webhook struct {
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
}
|
||||
|
||||
// Cause is one feature's contribution to an alert a model raised.
|
||||
//
|
||||
// Where a rule is its own explanation — the expression says what fired — a
|
||||
// model's output is a number, and a number on its own cannot go to an
|
||||
// investigator. What is owed instead is the features that influenced the output,
|
||||
// the risk associated with each, and the risk indicators behind them, and a Cause
|
||||
// is exactly that owed set for one feature.
|
||||
//
|
||||
// Observed and Baseline are the arithmetic the score came from, in the terms Unit
|
||||
// names, so the explanation is the same calculation rather than a second account
|
||||
// of it. Without is the score the model would have produced had this feature been
|
||||
// unremarkable, holding everything else: it is a counterfactual on the model
|
||||
// itself and it is what makes the contribution a measurement rather than an
|
||||
// attribution scheme's opinion.
|
||||
type Cause struct {
|
||||
Feature string `json:"feature"`
|
||||
Typology string `json:"typology"`
|
||||
Indicator string `json:"indicator"`
|
||||
Citation string `json:"citation"`
|
||||
Severity string `json:"severity"`
|
||||
Unit string `json:"unit"`
|
||||
Observed float64 `json:"observed"`
|
||||
Baseline float64 `json:"baseline"`
|
||||
Without float64 `json:"without"`
|
||||
// Share is this feature's part of the score, in [0,1]. Zero across every
|
||||
// cause means no single feature accounts for the alert and the combination
|
||||
// does; the causes are then ordered by how far each sits from unremarkable.
|
||||
Share float64 `json:"share"`
|
||||
}
|
||||
|
||||
// RuleHit is the result of evaluating a single rule against a transaction.
|
||||
type RuleHit struct {
|
||||
Rule Rule
|
||||
Match bool
|
||||
Score float64
|
||||
EvalErr string // non-empty if the rule evaluation failed (fail-closed)
|
||||
// Causes explains a hit that came from a model rather than an expression.
|
||||
// Empty for rules, which explain themselves.
|
||||
Causes []Cause
|
||||
}
|
||||
|
||||
// EvalContext is the evaluation context passed to expr-lang rules.
|
||||
|
||||
@@ -384,6 +384,13 @@ func (s *Store) Late() int64 {
|
||||
return s.late
|
||||
}
|
||||
|
||||
// Windows are the windows this Store keeps, so a caller that reads observations
|
||||
// by name can check at construction that the names it needs exist rather than
|
||||
// silently reading zero for a window nobody configured.
|
||||
func (s *Store) Windows() []Window {
|
||||
return append([]Window(nil), s.cfg.Windows...)
|
||||
}
|
||||
|
||||
// Keys is the number of live keys, for capacity monitoring against MaxKeys.
|
||||
func (s *Store) Keys() int {
|
||||
n := 0
|
||||
|
||||
Reference in New Issue
Block a user