mirror of
https://github.com/luxfi/aml.git
synced 2026-09-24 00:25:20 +00:00
Each of these is a control that worked and could be made not to, by a caller holding an ordinary token for its own tenant. Every one carries a test, and every one carries a mutation in tools/mutate.py, because a bound with no test is a bound that comes back. THE EXPENSIVE PLANE COULD TAKE THE MACHINE. A model search replays a tenant's history through up to MaxTrials detectors and a fit replays it through one; both were registered plain. Four bounds now, each doing one thing and nothing else: one study per tenant (api.gate), a deadline derived from the request's own context so a client that goes away cancels the work (api.within, 2 min), a process budget whose tokens ARE the trial workers and are taken BEFORE the history is read (topology.Budget, half the cores), and a clamp on the width one request may name (MaxWorkers). The gate is a map keyed by the tenant with nothing ever evicted from it, so one institution's study is never another's refusal; the budget is the one bound that is not per tenant, deliberately, because the CPU is shared however it is modelled and waiting for the machine is not the same as being refused because of somebody else. A rule replay is admitted the same way. Fit now records what it cost, as a run already did. THE DECIDER WAS FREE TEXT. Every plane asks a governed act for a reason and a decider so a supervisor can be told who turned a control off, and the answer was whatever the caller typed. types.Decider is a distinct type carrying `json:"-"`, so no wire format has a name for one, and pkg/api/typed.go writes the AUTHENTICATED SUBJECT onto every field of that type after the body and after the path. Identity now yields a Caller — two projections of one verified credential, the tenant that indexes every store and the subject that names a decision, both from the same signature. A credential naming nobody still authenticates, because reads and ingest are the tenant's work and not a person's, and refuses every operation that would record a decision. A TENANT COULD STOP ITS OWN PAYMENTS, AND THE RETRY DOUBLE-COUNTED. suppress.Cover runs once per activation on the ingest path and failed closed past MaxCandidates, which h.monitor turned into a 503 on every transaction. It degrades now: the answer is over what was read and says so (Cover.Partial, Activation.Unchecked), and the crowding is refused at DECLARATION instead (MaxInForce), where a refusal costs one operator request rather than every payment. The retry that followed such a 503 wrote a second activation for one firing, which counted in the streak, so a declared repetition policy fired on a repeat that never happened — a payment blocked because the client retried. An activation naming a transaction is now identified by (tenant, transaction, rule, subject), hashed so no customer value is in a key. The retained record is idempotent by the same reasoning: token.Vault.Fingerprint names the plaintext deterministically, since a seal draws a fresh nonce and sealed bytes never match; and the transaction now carries ONE clock, its own. The reception clock ingest stamped made every retry a different fact under one id, which no retry could ever clear. A genuine conflict answers 409 rather than 503, because 503 reads as "retry" and a client retrying a permanent refusal retries forever. A RUNG'S COUNT WAS THE LIMIT OF AN UNBOUNDED READ. It becomes a store LIMIT on the ingest path, so it is bounded at declaration (MaxCount, MaxWithin) and clamped again where it is used, because a bound introduced after a row was written does not reach that row. THE CATALOG STORED PAYLOAD VALUES AFTER ALL. A minimum IS a value, exactly, at any volume, and at a count of one so are the sum and the mean. Declared numbers are the transaction model's own and keep their moments; a custom number is whatever the institution put under its own key and gets the sketch and nothing else, absent rather than zero. And the vocabulary itself is now bounded PER TENANT, in the accumulator and in the rows alike: MaxKeys bounds one payload used as a bag, it does not bound a tenant sending a different key on every transaction, and that growth was in the memory every other institution's ingest runs in. Reaching the bound degrades only the tenant that reached it, refuses no payload, and publishes what it turned away. AN ADOPTED CONTROL COULD GO QUIET WITH NOBODY DECIDING IT SHOULD. The fit is durable and the adoption is durable, but what adoption DOES is install learned state into a model in memory — and this is one replica with a Recreate strategy, so a rollout empties it, and the store holds a bounded number of tenants' models, so an idle institution's can be dropped for a busy one's. Either way the tenant returns to warming, which reports no alerts, which is what a quiet institution also reports. The model now asks the model plane what this tenant last adopted when it plants a model for it, and State publishes when the model started and whether it started from an adoption, so a reset is visible instead of inferred from a silence. Co-authored-by: Hanzo Dev <[email protected]>
389 lines
14 KiB
Go
389 lines
14 KiB
Go
package suppress
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"os"
|
|
"strings"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/luxfi/aml/internal/instance"
|
|
)
|
|
|
|
const (
|
|
acme = "hanzo/acme"
|
|
rival = "hanzo/rival"
|
|
// other is the SAME org name under a different brand — the case a bare-org
|
|
// regression collides into one tenant.
|
|
other = "zoo/acme"
|
|
)
|
|
|
|
var noon = time.Date(2026, 3, 1, 12, 0, 0, 0, time.UTC)
|
|
|
|
func shelf(t *testing.T) *Shelf {
|
|
t.Helper()
|
|
app := instance.New(t)
|
|
t.Cleanup(app.Cleanup)
|
|
if err := Ensure(app); err != nil {
|
|
t.Fatalf("ensure: %v", err)
|
|
}
|
|
s := NewBase(app)
|
|
s.Now = func() time.Time { return noon }
|
|
return s
|
|
}
|
|
|
|
func suppressed(t *testing.T, s *Shelf, org string, in *SuppressIn) *Suppression {
|
|
t.Helper()
|
|
if in.Reason == "" {
|
|
in.Reason = "known merchant settlement pattern, reviewed 2026-02"
|
|
}
|
|
if in.By == "" {
|
|
in.By = "a.mensah"
|
|
}
|
|
out, err := s.Suppress(context.Background(), org, in)
|
|
if err != nil {
|
|
t.Fatalf("suppress: %v", err)
|
|
}
|
|
return out
|
|
}
|
|
|
|
// TestAKillSwitchIsNotASuppression. A row naming neither a rule nor a subject
|
|
// silences the whole monitoring programme and is indistinguishable in the store
|
|
// from an ordinary narrow one.
|
|
func TestAKillSwitchIsNotASuppression(t *testing.T) {
|
|
s := shelf(t)
|
|
_, err := s.Suppress(context.Background(), acme, &SuppressIn{Reason: "too noisy", By: "a.mensah"})
|
|
if !errors.Is(err, ErrBroad) {
|
|
t.Fatalf("a suppression covering everything must be refused, got %v", err)
|
|
}
|
|
}
|
|
|
|
// TestEverySuppressionCarriesADecision.
|
|
func TestEverySuppressionCarriesADecision(t *testing.T) {
|
|
s := shelf(t)
|
|
ctx := context.Background()
|
|
if _, err := s.Suppress(ctx, acme, &SuppressIn{Rule: "r1", By: "a"}); !errors.Is(err, ErrReason) {
|
|
t.Errorf("no reason must be refused, got %v", err)
|
|
}
|
|
if _, err := s.Suppress(ctx, acme, &SuppressIn{Rule: "r1", Reason: "r"}); !errors.Is(err, ErrDecider) {
|
|
t.Errorf("no decider must be refused, got %v", err)
|
|
}
|
|
sup := suppressed(t, s, acme, &SuppressIn{Rule: "r1"})
|
|
if _, err := s.Lift(ctx, acme, &LiftIn{ID: sup.ID, By: "a"}); !errors.Is(err, ErrReason) {
|
|
t.Errorf("lifting without a reason must be refused, got %v", err)
|
|
}
|
|
if _, err := s.Lift(ctx, acme, &LiftIn{ID: sup.ID, Reason: "r"}); !errors.Is(err, ErrDecider) {
|
|
t.Errorf("lifting without a decider must be refused, got %v", err)
|
|
}
|
|
}
|
|
|
|
// TestASubjectNeedsItsAxis: a value with no kind covers "any subject whose value
|
|
// happens to be this string", which spans accounts and addresses at once.
|
|
func TestASubjectNeedsItsAxis(t *testing.T) {
|
|
s := shelf(t)
|
|
_, err := s.Suppress(context.Background(), acme, &SuppressIn{Value: "acct-1", Reason: "r", By: "a"})
|
|
if !errors.Is(err, ErrSubject) {
|
|
t.Fatalf("a subject value with no kind must be refused, got %v", err)
|
|
}
|
|
}
|
|
|
|
// TestNarrowestCoverWins. A tenant that suppressed a whole rule and then took a
|
|
// specific decision about one account must see the specific one named, because
|
|
// that is the decision a reviewer will be asked about.
|
|
func TestNarrowestCoverWins(t *testing.T) {
|
|
s := shelf(t)
|
|
ctx := context.Background()
|
|
broad := suppressed(t, s, acme, &SuppressIn{Rule: "r1"})
|
|
narrow := suppressed(t, s, acme, &SuppressIn{Rule: "r1", Kind: "account", Value: "acct-1"})
|
|
|
|
cover, err := s.Cover(ctx, acme, &CoverIn{Rule: "r1", Kind: "account", Value: "acct-1"})
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if !cover.Covered || cover.Suppression.ID != narrow.ID {
|
|
t.Fatalf("the specific decision must be the one named, got %+v", cover.Suppression)
|
|
}
|
|
|
|
cover, err = s.Cover(ctx, acme, &CoverIn{Rule: "r1", Kind: "account", Value: "acct-2"})
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if !cover.Covered || cover.Suppression.ID != broad.ID {
|
|
t.Fatalf("another account falls to the rule-wide decision, got %+v", cover.Suppression)
|
|
}
|
|
|
|
cover, err = s.Cover(ctx, acme, &CoverIn{Rule: "r2", Kind: "account", Value: "acct-1"})
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if cover.Covered {
|
|
t.Fatalf("a suppression on r1 must not cover r2: %+v", cover.Suppression)
|
|
}
|
|
}
|
|
|
|
// TestWindowsAndLifts. Both stop a suppression applying; neither destroys it.
|
|
func TestWindowsAndLifts(t *testing.T) {
|
|
s := shelf(t)
|
|
ctx := context.Background()
|
|
windowed := suppressed(t, s, acme, &SuppressIn{Rule: "r1", Until: noon.Add(time.Hour)})
|
|
|
|
if c, _ := s.Cover(ctx, acme, &CoverIn{Rule: "r1"}); !c.Covered {
|
|
t.Fatal("a suppression inside its window must cover")
|
|
}
|
|
s.Now = func() time.Time { return noon.Add(2 * time.Hour) }
|
|
if c, _ := s.Cover(ctx, acme, &CoverIn{Rule: "r1"}); c.Covered {
|
|
t.Fatal("a suppression past its window must not cover")
|
|
}
|
|
s.Now = func() time.Time { return noon }
|
|
|
|
lifted, err := s.Lift(ctx, acme, &LiftIn{ID: windowed.ID, Reason: "volume reviewed", By: "r.okafor"})
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if lifted.LiftedBy != "r.okafor" || lifted.LiftWhy != "volume reviewed" || lifted.Lifted.IsZero() {
|
|
t.Fatalf("a lift must name its decider and its reason: %+v", lifted)
|
|
}
|
|
if _, err := s.Lift(ctx, acme, &LiftIn{ID: windowed.ID, Reason: "again", By: "r.okafor"}); !errors.Is(err, ErrLifted) {
|
|
t.Fatalf("lifting twice must be refused, got %v", err)
|
|
}
|
|
|
|
ledger, err := s.Ledger(ctx, acme, &LedgerIn{})
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if len(ledger.Suppressions) != 1 {
|
|
t.Fatalf("a lift destroyed the record: %+v", ledger.Suppressions)
|
|
}
|
|
if ledger.Suppressions[0].By != "a.mensah" {
|
|
t.Fatalf("the original decider did not survive the lift: %+v", ledger.Suppressions[0])
|
|
}
|
|
inForce, err := s.Ledger(ctx, acme, &LedgerIn{InForce: true})
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if len(inForce.Suppressions) != 0 {
|
|
t.Fatalf("a lifted suppression is not in force: %+v", inForce.Suppressions)
|
|
}
|
|
}
|
|
|
|
// TestClosedWindowRefused: a suppression that could never apply is an operator
|
|
// error, and accepting it hides the mistake until somebody wonders why the alerts
|
|
// never stopped.
|
|
func TestClosedWindowRefused(t *testing.T) {
|
|
s := shelf(t)
|
|
_, err := s.Suppress(context.Background(), acme, &SuppressIn{
|
|
Rule: "r1", Reason: "r", By: "a", Until: noon.Add(-time.Hour),
|
|
})
|
|
if !errors.Is(err, ErrWindow) {
|
|
t.Fatalf("a window that has already closed must be refused, got %v", err)
|
|
}
|
|
}
|
|
|
|
// TestTenantIsolation over every operation, including the same org name under a
|
|
// second brand.
|
|
func TestTenantIsolation(t *testing.T) {
|
|
s := shelf(t)
|
|
ctx := context.Background()
|
|
mine := suppressed(t, s, acme, &SuppressIn{Rule: "r1", Kind: "account", Value: "acct-1"})
|
|
|
|
for _, stranger := range []string{other, rival} {
|
|
c, err := s.Cover(ctx, stranger, &CoverIn{Rule: "r1", Kind: "account", Value: "acct-1"})
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if c.Covered {
|
|
t.Errorf("%s is covered by %s's suppression", stranger, acme)
|
|
}
|
|
ledger, err := s.Ledger(ctx, stranger, &LedgerIn{})
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if len(ledger.Suppressions) != 0 {
|
|
t.Errorf("%s can read %s's suppressions", stranger, acme)
|
|
}
|
|
if _, err := s.Lift(ctx, stranger, &LiftIn{ID: mine.ID, Reason: "r", By: "b"}); !errors.Is(err, ErrNotHere) {
|
|
t.Errorf("%s can lift %s's suppression: %v", stranger, acme, err)
|
|
}
|
|
}
|
|
if c, _ := s.Cover(ctx, acme, &CoverIn{Rule: "r1", Kind: "account", Value: "acct-1"}); !c.Covered {
|
|
t.Fatal("another tenant lifted this suppression")
|
|
}
|
|
}
|
|
|
|
// TestBareOrgIsRefused at the boundary of every operation.
|
|
func TestBareOrgIsRefused(t *testing.T) {
|
|
s := shelf(t)
|
|
ctx := context.Background()
|
|
for _, bare := range []string{"acme", "", "unknown/acme"} {
|
|
if _, err := s.Suppress(ctx, bare, &SuppressIn{Rule: "r1", Reason: "r", By: "a"}); err == nil {
|
|
t.Errorf("Suppress accepted %q as a tenant", bare)
|
|
}
|
|
if _, err := s.Cover(ctx, bare, &CoverIn{Rule: "r1"}); err == nil {
|
|
t.Errorf("Cover accepted %q as a tenant", bare)
|
|
}
|
|
if _, err := s.Ledger(ctx, bare, &LedgerIn{}); err == nil {
|
|
t.Errorf("Ledger accepted %q as a tenant", bare)
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestNothingDeletes: silence is governed, so the record of it is not disposable
|
|
// by the same knob that created it.
|
|
func TestNothingDeletes(t *testing.T) {
|
|
for _, name := range []string{"suppress.go", "shelf.go"} {
|
|
raw, err := os.ReadFile(name)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if strings.Contains(string(raw), ".Delete(") {
|
|
t.Errorf("%s calls Delete: disposal is pkg/retention's decision and nobody else's", name)
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestRestart: a suppression that vanished on a rollout turns a governed silence
|
|
// into a surprise, and one that outlived its record is worse.
|
|
func TestRestart(t *testing.T) {
|
|
first := instance.New(t)
|
|
if err := Ensure(first); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
before := NewBase(first)
|
|
before.Now = func() time.Time { return noon }
|
|
mine := suppressed(t, before, acme, &SuppressIn{Rule: "r1", Kind: "account", Value: "acct-1"})
|
|
suppressed(t, before, other, &SuppressIn{Rule: "r1", Kind: "account", Value: "acct-1"})
|
|
|
|
second := instance.Restart(t, first)
|
|
if err := Ensure(second); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
after := NewBase(second)
|
|
after.Now = func() time.Time { return noon }
|
|
|
|
c, err := after.Cover(context.Background(), acme, &CoverIn{Rule: "r1", Kind: "account", Value: "acct-1"})
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if !c.Covered || c.Suppression.ID != mine.ID {
|
|
t.Fatalf("the suppression did not survive the restart: %+v", c)
|
|
}
|
|
if c.Suppression.Reason == "" || c.Suppression.By == "" {
|
|
t.Fatalf("the decision behind it did not survive: %+v", c.Suppression)
|
|
}
|
|
|
|
ledger, err := after.Ledger(context.Background(), acme, &LedgerIn{})
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if len(ledger.Suppressions) != 1 {
|
|
t.Fatalf("the tenant boundary did not survive the restart: %+v", ledger.Suppressions)
|
|
}
|
|
if c, _ := after.Cover(context.Background(), rival, &CoverIn{Rule: "r1", Kind: "account", Value: "acct-1"}); c.Covered {
|
|
t.Fatal("a third tenant is covered after the restart")
|
|
}
|
|
}
|
|
|
|
// TestCrowdingIsRefusedWhereItCostsARequest.
|
|
//
|
|
// The bound on how many suppressions bear on one rule is enforced at
|
|
// DECLARATION, where a refusal costs one operator request. The same crowding met
|
|
// on the ingest path would cost a payment: Cover runs once per activation and its
|
|
// caller cannot process a transaction it cannot record, so an error there is a
|
|
// tenant-triggerable outage of that tenant's own payments — and the retry of a
|
|
// failed ingest is what turns one incomplete read into a double-counted one.
|
|
func TestCrowdingIsRefusedWhereItCostsARequest(t *testing.T) {
|
|
s := shelf(t)
|
|
s.inForceMax = 3
|
|
ctx := context.Background()
|
|
|
|
for i := range 3 {
|
|
suppressed(t, s, acme, &SuppressIn{Rule: "ctr", Kind: "account", Value: string(rune('a' + i))})
|
|
}
|
|
_, err := s.Suppress(ctx, acme, &SuppressIn{
|
|
Rule: "ctr", Kind: "account", Value: "one-too-many",
|
|
Reason: "another one", By: "a.mensah",
|
|
})
|
|
if !errors.Is(err, ErrCrowded) {
|
|
t.Fatalf("declaring past the bound: %v, want ErrCrowded", err)
|
|
}
|
|
|
|
// A rule that is not crowded is unaffected, and so is another tenant. A bound
|
|
// one institution reached must never refuse another's work.
|
|
suppressed(t, s, acme, &SuppressIn{Rule: "structuring", Kind: "account", Value: "acct-1"})
|
|
for i := range 3 {
|
|
suppressed(t, s, rival, &SuppressIn{Rule: "ctr", Kind: "account", Value: string(rune('a' + i))})
|
|
}
|
|
|
|
// Lifting makes room: the bound counts what is IN FORCE, not what the ledger
|
|
// holds, so an institution is never refused for its own history.
|
|
ledger, err := s.Ledger(ctx, acme, &LedgerIn{Rule: "ctr"})
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if _, err := s.Lift(ctx, acme, &LiftIn{ID: ledger.Suppressions[0].ID, Reason: "reviewed", By: "a.mensah"}); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if _, err := s.Suppress(ctx, acme, &SuppressIn{
|
|
Rule: "ctr", Kind: "account", Value: "room-now", Reason: "another one", By: "a.mensah",
|
|
}); err != nil {
|
|
t.Fatalf("lifting one did not make room: %v", err)
|
|
}
|
|
}
|
|
|
|
// TestACrowdedCoverDegradesAndSaysSo.
|
|
//
|
|
// The read bound is the second line, and reaching it must not fail the ingest
|
|
// path. The answer is over what was read and it is MARKED, so an incomplete
|
|
// governance answer is visible rather than absorbed. Degrading in this direction
|
|
// produces an alert nobody wanted; refusing produces silence, and silence is what
|
|
// an unmonitored institution also looks like.
|
|
func TestACrowdedCoverDegradesAndSaysSo(t *testing.T) {
|
|
s := shelf(t)
|
|
s.candidates, s.inForceMax = 2, 100
|
|
ctx := context.Background()
|
|
|
|
for i := range 4 {
|
|
suppressed(t, s, acme, &SuppressIn{Rule: "ctr", Kind: "account", Value: string(rune('a' + i))})
|
|
}
|
|
cover, err := s.Cover(ctx, acme, &CoverIn{Rule: "ctr", Kind: "account", Value: "a", At: noon})
|
|
if err != nil {
|
|
t.Fatalf("a crowded cover check must degrade, not refuse: %v", err)
|
|
}
|
|
if !cover.Partial {
|
|
t.Fatal("the answer was over a page of the candidates and did not say so")
|
|
}
|
|
|
|
// A tenant under the bound gets a complete answer, so Partial is a real
|
|
// finding and not something every answer carries.
|
|
suppressed(t, s, rival, &SuppressIn{Rule: "ctr", Kind: "account", Value: "acct-1"})
|
|
clean, err := s.Cover(ctx, rival, &CoverIn{Rule: "ctr", Kind: "account", Value: "acct-1", At: noon})
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if clean.Partial || !clean.Covered {
|
|
t.Fatalf("an uncrowded tenant: %+v", clean)
|
|
}
|
|
}
|
|
|
|
// TestCoverNeverRefusesForVolume reads the source: the ingest path's one question
|
|
// must have no error return that a tenant can reach by declaring rows. The
|
|
// property is easy to reintroduce by hand, and a comment does not hold it.
|
|
func TestCoverNeverRefusesForVolume(t *testing.T) {
|
|
raw, err := os.ReadFile("shelf.go")
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
body, _, found := strings.Cut(string(raw), "// better breaks the tie")
|
|
if !found {
|
|
t.Fatal("could not find the end of Cover in the source")
|
|
}
|
|
_, cover, found := strings.Cut(body, "func (s *Shelf) Cover(")
|
|
if !found {
|
|
t.Fatal("could not find Cover in the source")
|
|
}
|
|
if strings.Contains(cover, "ErrStore, s.maxCandidates()") || strings.Contains(cover, "cannot be answered completely") {
|
|
t.Fatal("Cover refuses on volume again: a tenant can stop its own ingest by declaring suppressions")
|
|
}
|
|
}
|