reference: the lookup data a decision consults, at its own prefix and inside its own ceiling

The versioned reference plane — throwaway-inbox domains, hosting and Tor ranges,
issuer-prefix structure, delegated autonomous-system numbers, and how current the
designation lists the screening engine holds actually are. Freshness is on the
wire because a stale list answers "not listed" for everything and reads exactly
like a clean world; a set that never loaded REFUSES rather than answering clean.
The shared baseline has no tenant column at all, so a cross-tenant write is
unrepresentable; a tenant's own allow and deny entries live in that
organisation's own file. Lawfulness is a field: every fetched source states the
basis its bytes reach a tenant under, and the three sources we may not hold —
politically-exposed persons, issuer identification, commercial network
reputation — are declared as seams that refuse, naming the licence we lack.

Four things are different from the version this replaces.

/v1/reference, NOT /v1/ml/reference. openapi.Product is the second path segment,
and the published tag and the per-product count in openapi/floor.json are both
derived from it — so under /v1/ml these six operations WERE the model-serving
product's, in its tag and its total, one app over from the risk model plane that
had just been moved off /v1/ml for that exact reason. The test that was supposed
to hold the address compared the served paths against the literal
/v1/ml/reference, so it agreed with whatever the constant said;
TestThisAppAnswersUnderItsOwnName states the property against the app's own
identity instead and fails under any other product's prefix.

The per-organisation ceiling is MEASURED. It was a count of 10,000 entries per
set with the byte figure derived from it as maxKey+maxNote+128, and the
derivation left out the implicit index over PRIMARY KEY ("set", key) — a second
copy of the two widest columns on every row — plus page slack and at-rest
encryption. A worst-case row measures 1,952 bytes against the 1,152 published, so
the stated 128 MiB was really 204 MiB on the one volume every organisation's
store shares. The budget is now stated in bytes, the entry count is its quotient,
and TestOneOrgsOverridesCostWhatTheyArePublishedToCost puts worst-case rows on a
real file and fails if one costs more than the published figure.

Every term of a row is bounded. `by` — the writer recorded for an adverse-action
input — was a term of that product and nothing bounded it: at a 2 KiB writer a
row measures 4,683 bytes and at 8 KiB, 10,144, which is 1,064 MiB against a
published 128. IAM mints a UUID so no live principal is near it, which is exactly
why it survived review; a term that is small by luck is not a ceiling. maxActor
bounds it at the one door that writes a row, refused rather than trimmed, the
same rule the key and the note follow.

One sentence, once. "This set has never loaded" was written out four times, in
build, in answer, in the consulted row and in the set view. The behaviour is held
on the wire already — TestOverrideSurvivesAnUnloadedBaseline fails if any of them
stops saying it — so this is four copies of one fact collapsed to one constant,
not a hole being closed.

The manifest row is added beside risk and takes nothing from it: /v1/risk is
live and untouched. floor.json rises 1678 -> 1682 paths with reference at 6 and
ml unchanged.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>

apps/reference/Makefile

@@ -0,0 +1,8 @@
+# Generated by plugin/gen-app-cmds. DO NOT EDIT.
+#
+# The build contract is mk/plugin.mk — one file carrying every target an app
+# needs: build, test, vet, openapi, clean. This names the app(s) this package
+# backs and includes it. Written from the same apps.Wire() parse that writes
+# plugin/<app>/main.go, so an app cannot have a main and no Makefile.
+APPS := reference
+include ../../mk/plugin.mk

apps/reference/derive.go

@@ -0,0 +1,288 @@
+package reference
+
+// derive.go computes the two sets nobody publishes for us: the structural card
+// table, and the browser identities the fleet has seen under enough separate
+// organisations that no single one of them could have produced the observation.
+//
+// THE BOUNDARY THIS FILE DEFENDS. Everything a tenant reads from the shared
+// baseline must be either PUBLIC (someone else published it under a licence) or
+// AGGREGATE (a statistic that no single organisation could have produced alone).
+// The device set is the only place we compute the second kind, so the whole
+// argument lives here and is enforced twice — in the statement's HAVING and again
+// on the way out — because a row written before a gate existed is still a row.
+//
+// The floor is deliberately high. Publishing "this browser was seen by two
+// organisations" tells the first of them something about the second; publishing
+// "this browser was seen by twenty-five" tells them about a population. The
+// second is a statistic and the first is a disclosure, and the line between them
+// is the only thing standing between a shared baseline and a data-sharing
+// agreement nobody signed.
+
+import (
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// The k-anonymity floor for anything derived from fleet traffic.
+//
+// Orgs is the number of DISTINCT organisations that must have contributed
+// before a key may be published. Twenty-five is chosen so that an adversary who
+// controls several organisations still cannot attribute the row: to learn about
+// one contributor you must know every other contributor's value.
+//
+// Rows bounds the other direction. Twenty-five organisations contributing one
+// observation each is twenty-five readable facts, not a population.
+const (
+	Orgs = 25
+	Rows = 1000
+)
+
+// Publishable is the k-anonymity gate as a PURE PREDICATE. It is exported
+// because it is the ONE definition: the statement that computes a derived set
+// binds these numbers, the reader re-checks them, and any other plane that
+// publishes a cross-organisation aggregate must call this rather than restate
+// the constants — two spellings of a floor is one floor that can drift, and the
+// drift always favours the weaker.
+func Publishable(orgs uint32, n uint64) bool { return orgs >= Orgs && n >= Rows }
+
+// producer is everything a local source needs: a bounded context, the instant
+// the run is dated at, and a reader for the shared event plane. The reader is a
+// function rather than a package call so a test can produce a set without a
+// warehouse.
+type producer struct {
+	ctx   context.Context
+	now   time.Time
+	query func(ctx context.Context, q string, args ...any) ([]map[string]any, error)
+}
+
+// deviceWindow is how far back the device aggregate looks. Thirty days is long
+// enough for a shared browser to accumulate the twenty-five organisations the
+// floor requires, and short enough that a device that stopped being shared
+// leaves the set.
+const deviceWindow = 30 * 24 * time.Hour
+
+// deviceStatement is the ONE statement that computes the device aggregate, and
+// it is a PACKAGE CONSTANT: nothing a caller sends reaches it, as an identifier
+// or otherwise. Its five placeholders bind, in order, the window start, the
+// window end, the reserved anonymous tenant, the organisation floor and the
+// observation floor.
+//
+// The anonymous lane is excluded at the source. The reserved `$public` tenant is
+// where the event door files credential-less writes, so counting it would let an
+// unauthenticated stranger push any browser identity over the floor and into
+// every tenant's baseline. Excluding it in the FROM rather than in a later
+// filter is the difference between a refusal and a place to forget.
+const deviceStatement = `
+	SELECT id, orgs, n FROM (
+	  SELECT anonymous_id AS id, uniqExact(org) AS orgs, count() AS n
+	  FROM event.event
+	  WHERE time >= ? AND time < ? AND anonymous_id != '' AND org != ?
+	  GROUP BY anonymous_id
+	)
+	WHERE orgs >= ? AND n >= ?
+	ORDER BY orgs DESC, id
+	LIMIT ` + deviceCap + deviceBudget
+
+// deviceCap bounds the result so a warehouse that suddenly matches everything
+// costs a truncated set rather than the process. It is a string constant spliced
+// into the statement because a LIMIT cannot bind, and it is not reachable from
+// any caller.
+const deviceCap = "50000"
+
+// deviceBudget bounds what the aggregation may SPEND, which the LIMIT does not.
+//
+// The LIMIT applies to the outer select — the rows that survive the floor — and
+// says nothing about the inner GROUP BY, which visits every distinct browser
+// identity the whole fleet saw in [deviceWindow] before a single row is filtered.
+// On a busy event plane that is a grouping over hundreds of millions of keys, and
+// it runs against the one warehouse analytics, insights, sentry, commerce and
+// gateway usage all share, roughly daily and unattended. Housekeeping for one
+// reference set must not be able to stall the store every other plane reads from.
+//
+// So the statement states its own budget: it spills to disk rather than growing,
+// it stops rather than spilling forever, and it gives up rather than running past
+// the window it is allowed. Exceeding a budget fails THIS take, which is already
+// a case this plane handles — the previous version stands and ages out visibly.
+const deviceBudget = `
+	SETTINGS max_execution_time = 300,
+	         max_memory_usage = 4000000000,
+	         max_bytes_before_external_group_by = 2000000000,
+	         max_bytes_before_external_sort = 2000000000`
+
+// publicTenant is the reserved org the event door files credential-less writes
+// under (apps/analytics/event.go). It is not a customer and it never
+// contributes to an aggregate.
+const publicTenant = "$public"
+
+// produceDevice computes the browser identities the fleet sees across many
+// organisations.
+//
+// THE KEY IS A DIGEST, NEVER THE IDENTIFIER. The shared table holds
+// sha256(anonymous_id) and the lookup path digests the caller's value the same
+// way, so no browser identifier is ever written into a store every tenant reads.
+// The digest is not salted and does not need to be: the input is a
+// client-minted opaque identifier with the entropy of a UUID, so there is no
+// dictionary to run against it — and the k-anonymity floor means the only
+// identifiers published at all are ones twenty-five organisations already share.
+//
+// The count is BUCKETED rather than exact. "Seen by 25 to 99 organisations" is
+// the signal a rule wants; the exact number is a fingerprint of the population
+// and buys nothing.
+func produceDevice(p producer) ([]Entry, error) {
+	if p.query == nil {
+		return nil, fmt.Errorf("reference: the device aggregate needs the event plane, which is not connected")
+	}
+	from := p.now.Add(-deviceWindow).UTC()
+	rows, err := p.query(p.ctx, deviceStatement, from, p.now.UTC(), publicTenant, Orgs, Rows)
+	if err != nil {
+		return nil, fmt.Errorf("reference: device aggregate: %w", err)
+	}
+	out := make([]Entry, 0, len(rows))
+	for _, r := range rows {
+		id, _ := r["id"].(string)
+		orgs := count32(r["orgs"])
+		n := count64(r["n"])
+		// The SECOND gate. The statement already refused anything below the floor;
+		// this refuses a row that predates the gate, or one a future edit lets
+		// through, because the cost of the two disagreeing is borne by the tenant
+		// whose data leaks rather than by whoever edited the statement.
+		if id == "" || !Publishable(orgs, n) {
+			continue
+		}
+		sum := sha256.Sum256([]byte(id))
+		out = append(out, Entry{
+			Key:   hex.EncodeToString(sum[:]),
+			Value: map[string]string{"class": "shared", "orgs": bucket(orgs)},
+			Orgs:  orgs,
+			N:     n,
+		})
+	}
+	return out, nil
+}
+
+// bucket renders an organisation count as the band a rule should read. Bands
+// rather than numbers: the exact count of organisations sharing one browser is
+// a more precise description of the population than anyone needs.
+func bucket(orgs uint32) string {
+	switch {
+	case orgs >= 1000:
+		return "1000+"
+	case orgs >= 100:
+		return "100-999"
+	default:
+		return "25-99"
+	}
+}
+
+// count32 and count64 read a warehouse count column, which arrives as whichever
+// numeric type the driver chose.
+func count32(v any) uint32 { return uint32(count64(v)) }
+
+func count64(v any) uint64 {
+	switch n := v.(type) {
+	case uint64:
+		return n
+	case uint32:
+		return uint64(n)
+	case int64:
+		if n < 0 {
+			return 0
+		}
+		return uint64(n)
+	case int:
+		if n < 0 {
+			return 0
+		}
+		return uint64(n)
+	case float64:
+		if n < 0 {
+			return 0
+		}
+		return uint64(n)
+	case string:
+		u, err := strconv.ParseUint(strings.TrimSpace(n), 10, 64)
+		if err != nil {
+			return 0
+		}
+		return u
+	default:
+		return 0
+	}
+}
+
+// scheme is one card scheme and the issuer identification number prefixes it
+// publishes, with the account lengths it issues at.
+type scheme struct {
+	name     string
+	prefixes []string
+	lengths  []int
+}
+
+// schemes is the structural card table: which scheme an issuer identification
+// number belongs to, from the major industry identifier of ISO/IEC 7812 and the
+// prefix ranges each scheme publishes.
+//
+// This is COMPUTED rather than downloaded on purpose, and it is the one set
+// where that is the right answer. These prefixes are structural facts that have
+// been stable for decades, they are published by the schemes themselves, and no
+// database is licensed to state them. What a licensed database adds — the
+// institution behind a prefix, its country, whether the product is debit,
+// credit or prepaid — is exactly what the issuer seam declares we do not have.
+var schemes = []scheme{
+	{"visa", []string{"4"}, []int{13, 16, 19}},
+	{"mastercard", []string{"51", "52", "53", "54", "55"}, []int{16}},
+	{"mastercard", span2221to2720(), []int{16}},
+	{"amex", []string{"34", "37"}, []int{15}},
+	{"discover", []string{"6011", "65"}, []int{16, 19}},
+	{"discover", spanOf(644, 649), []int{16, 19}},
+	{"jcb", spanOf(3528, 3589), []int{16, 19}},
+	{"unionpay", []string{"62", "81"}, []int{16, 17, 18, 19}},
+	{"diners", []string{"36", "38", "39"}, []int{14, 16, 19}},
+	{"diners", spanOf(300, 305), []int{14, 16, 19}},
+	{"maestro", []string{"5018", "5020", "5038", "5893", "6304", "6759", "6761", "6762", "6763"}, []int{12, 13, 14, 15, 16, 17, 18, 19}},
+	{"mir", spanOf(2200, 2204), []int{16, 17, 18, 19}},
+	{"elo", []string{"4011", "4312", "4389", "4514", "4573", "5041", "5066", "5090", "6277", "6362", "6363", "6504", "6505", "6516", "6550"}, []int{16}},
+	{"troy", []string{"9792"}, []int{16}},
+	{"rupay", []string{"60", "6521", "6522", "8171", "8172"}, []int{16}},
+}
+
+// spanOf renders an inclusive numeric prefix range as its literal members.
+func spanOf(lo, hi int) []string {
+	out := make([]string, 0, hi-lo+1)
+	for n := lo; n <= hi; n++ {
+		out = append(out, strconv.Itoa(n))
+	}
+	return out
+}
+
+// span2221to2720 is the Mastercard two-series. It is expressed as its 500
+// members rather than as a range test so that every entry in the set is a
+// literal prefix and one matcher serves the whole set.
+func span2221to2720() []string { return spanOf(2221, 2720) }
+
+// produceBIN renders the structural table into entries. It takes no producer
+// input: the answer is the same on every run, which is why the version digest
+// over it is stable and a refresh that changes nothing is visibly a refresh
+// that changed nothing.
+func produceBIN(producer) ([]Entry, error) {
+	out := make([]Entry, 0, 1024)
+	for _, s := range schemes {
+		lengths := make([]string, 0, len(s.lengths))
+		for _, n := range s.lengths {
+			lengths = append(lengths, strconv.Itoa(n))
+		}
+		joined := strings.Join(lengths, ",")
+		for _, p := range s.prefixes {
+			out = append(out, Entry{
+				Key:   p,
+				Value: map[string]string{"scheme": s.name, "lengths": joined},
+			})
+		}
+	}
+	return out, nil
+}

apps/reference/fetch.go

@@ -0,0 +1,179 @@
+package reference
+
+// fetch.go takes one publisher's bytes. It is the same shape luxfi/aml
+// pkg/screen arrived at for the sanctions lists, for the same reasons, and the
+// reasons are worth restating because each one is a measured failure:
+//
+//   - RETRY TRANSPORT, NEVER A PARSE. A publisher that just refused a TLS
+//     handshake will serve the file correctly seconds later; a publisher whose
+//     schema changed will fail identically on every attempt, so retrying it only
+//     delays the refusal.
+//   - A TRUNCATED LIST IS THE DANGEROUS FAILURE. It parses. It yields a shorter
+//     list of members, every one of them correct, and nothing anywhere reports a
+//     problem — so a response that reaches the read limit is an error rather
+//     than a shorter set.
+//   - THE DIGEST IS NOT DECORATION. It is what tells a refresh that changed
+//     nothing from a refresh that did not run, and it is what makes ingest
+//     idempotent: the version IS the content.
+
+import (
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"net/http"
+	"net/netip"
+	"strings"
+	"time"
+)
+
+const (
+	// maxBody bounds one download, and it is the bound that decides how much the
+	// PARSE may cost — a parser turns bytes into entries one at a time, so
+	// whatever arrives here is allocated before any later gate can look at it.
+	//
+	// Measured: the largest source in the catalog is AWS's ip-ranges.json at 2.6
+	// MB, and the next is under a tenth of that. Sixteen is six times the largest
+	// and leaves room for years of growth; the previous sixty-four was twenty-five
+	// times it, and twenty-five times a few megabytes of adversarial one-token
+	// lines is a heap this one-replica deployment does not have. A publisher that
+	// outgrows this refuses on its own row and ages out visibly, which is a
+	// condition an operator can see and raise — unlike an OOM.
+	maxBody = 16 << 20
+	// attempts is how many times a publisher is asked before its source is
+	// recorded failed.
+	attempts = 3
+	// backoff grows per attempt: a publisher that just failed under load is not
+	// helped by being asked again immediately.
+	backoff = 5 * time.Second
+	// fetchTimeout bounds one attempt end to end.
+	fetchTimeout = 2 * time.Minute
+)
+
+// download takes one URL and returns its bytes. It is a value rather than a
+// direct call so the retry behaviour — which decides whether a blip costs a day
+// of freshness — is testable without a network.
+type download func(ctx context.Context, url string) ([]byte, error)
+
+// wire is the real downloader.
+func wire(ctx context.Context, url string) ([]byte, error) {
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return nil, err
+	}
+	req.Header.Set("User-Agent", "hanzo-reference/1")
+	client := &http.Client{Timeout: fetchTimeout, CheckRedirect: hop}
+	resp, err := client.Do(req)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return nil, fmt.Errorf("HTTP %d from %s", resp.StatusCode, url)
+	}
+	body, err := io.ReadAll(io.LimitReader(resp.Body, maxBody))
+	if err != nil {
+		return nil, err
+	}
+	if len(body) == maxBody {
+		return nil, fmt.Errorf("%s reached the %d byte limit and may be truncated", url, maxBody)
+	}
+	return body, nil
+}
+
+// hop decides whether a publisher's redirect may be followed.
+//
+// Every Origin in the catalog is an HTTPS constant, so the ONE thing about the
+// address this process cannot state in code is where a redirect goes. This runs
+// inside the cluster, so "wherever the publisher says" reaches the pod network
+// and the instance metadata address; and a hop to http:// hands the whole
+// baseline every tenant's decisions read to anyone on the path.
+//
+// So a hop keeps the two properties the origin already had — TLS, and a
+// destination outside this network — and is refused otherwise. The literal
+// address forms are what a redirect can carry without a lookup; a name that
+// RESOLVES inward is not caught here and does not need to be, because it is the
+// publisher's own DNS and the same trust as the bytes themselves.
+func hop(req *http.Request, via []*http.Request) error {
+	if len(via) >= 5 {
+		return fmt.Errorf("%s redirected %d times", via[0].URL, len(via))
+	}
+	if req.URL.Scheme != "https" {
+		return fmt.Errorf("%s redirected to %s, and a reference source is fetched over TLS or not at all", via[0].URL, req.URL.Scheme)
+	}
+	if inward(req.URL.Hostname()) {
+		return fmt.Errorf("%s redirected to %s, which is inside this network", via[0].URL, req.URL.Host)
+	}
+	return nil
+}
+
+// inward reports whether a host is a literal address this process should never
+// be sent to: private, loopback, link-local (the instance metadata address is
+// one), unspecified or multicast. A name is not judged here — resolving it is the
+// publisher's own DNS.
+func inward(host string) bool {
+	a, err := netip.ParseAddr(strings.Trim(host, "[]"))
+	if err != nil {
+		return false
+	}
+	a = a.Unmap()
+	return a.IsPrivate() || !a.IsGlobalUnicast()
+}
+
+// pull downloads one source and parses it, retrying transport failures.
+func pull(ctx context.Context, get download, src Source, wait func(context.Context, time.Duration) error) ([]Entry, error) {
+	var last error
+	for attempt := 1; attempt <= attempts; attempt++ {
+		body, err := get(ctx, src.Origin)
+		if err != nil {
+			last = fmt.Errorf("attempt %d of %d: %w", attempt, attempts, err)
+			// The context ending is the deployment shutting down or giving up, not the
+			// publisher failing, so there is nothing to retry into.
+			if ctx.Err() != nil {
+				return nil, last
+			}
+			if attempt < attempts {
+				if werr := wait(ctx, time.Duration(attempt)*backoff); werr != nil {
+					return nil, last
+				}
+				continue
+			}
+			return nil, last
+		}
+		return src.parse(body)
+	}
+	return nil, last
+}
+
+// pause waits, or returns early when the context ends.
+func pause(ctx context.Context, d time.Duration) error {
+	t := time.NewTimer(d)
+	defer t.Stop()
+	select {
+	case <-t.C:
+		return nil
+	case <-ctx.Done():
+		return ctx.Err()
+	}
+}
+
+// digest is the content address of a parsed source: the version.
+//
+// It is computed over the SORTED, canonical rendering of the entries rather
+// than over the downloaded bytes, and that is the whole point. A publisher who
+// reorders their file, or adds a timestamp comment, has not changed the set —
+// digesting the bytes would mint a new version and re-ingest every row for a
+// change that is not one. Digesting the meaning makes the version an identity:
+// the same set is the same version, whoever fetched it and whenever.
+func digest(entries []Entry) string {
+	h := sha256.New()
+	for _, e := range order(entries) {
+		fmt.Fprintf(h, "%s\x00", e.Key)
+		for _, k := range keys(e.Value) {
+			fmt.Fprintf(h, "%s\x01%s\x02", k, e.Value[k])
+		}
+		fmt.Fprintf(h, "%g\x03%d\x04%d\x1e", e.Score, e.Orgs, e.N)
+	}
+	return hex.EncodeToString(h.Sum(nil))
+}

apps/reference/live_test.go

@@ -0,0 +1,55 @@
+package reference
+
+// live_test.go takes every published source for real.
+//
+// It is env-gated because it reaches ten third-party publishers and a unit suite
+// must not depend on them. It is here anyway because the failure it catches is
+// the one nothing else can: a publisher who changes a column keeps serving 200,
+// the parser keeps returning entries, and the set silently becomes a shorter
+// list of correct members. Only real bytes prove otherwise.
+//
+//	REFERENCE_LIVE=1 go test -tags sqlite_fts5 -run Live ./apps/reference/
+
+import (
+	"context"
+	"os"
+	"testing"
+	"time"
+)
+
+func TestLiveSourcesStillParse(t *testing.T) {
+	if os.Getenv("REFERENCE_LIVE") == "" {
+		t.Skip("set REFERENCE_LIVE=1 to take every published source for real")
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
+	defer cancel()
+
+	for _, set := range Catalog() {
+		if set.Kind != KindFetch {
+			continue
+		}
+		for _, src := range set.Sources {
+			t.Run(set.Name+"/"+src.Name, func(t *testing.T) {
+				got, err := pull(ctx, wire, src, pause)
+				if err != nil {
+					t.Fatalf("%s: %v", src.Origin, err)
+				}
+				if len(got) == 0 {
+					t.Fatalf("%s parsed to nothing", src.Origin)
+				}
+				// A key with no value is a row the parser read halfway.
+				for _, e := range got[:min(len(got), 50)] {
+					if e.Key == "" {
+						t.Fatalf("%s produced an entry with no key", src.Origin)
+					}
+				}
+				// The digest is stable across two parses of the same bytes, which is
+				// what makes a re-ingest a no-op.
+				if digest(got) == "" {
+					t.Fatal("no digest")
+				}
+				t.Logf("%-14s %-13s %6d entries  %s", set.Name, src.Name, len(got), digest(got)[:12])
+			})
+		}
+	}
+}

apps/reference/override.go

@@ -0,0 +1,289 @@
+package reference
+
+// override.go is a tenant's own say over the shared baseline: the entries this
+// organisation allows or denies whatever the published data says.
+//
+// ISOLATION IS PHYSICAL, NOT PREDICATED. An override lives in the
+// organisation's OWN SQLite file, reached through cloud.OrgNamespace — the one
+// door a validated org walks through — so a distinct organisation is a distinct
+// file and a query in one cannot reach another's rows. There is no `org` column
+// to forget in a WHERE clause, because there is no shared table. This is the
+// same physical isolation every other per-entity store in the binary has
+// (HIP-0302), and it is why the cross-tenant write in this design is
+// unrepresentable rather than merely refused.
+//
+// The wire agrees with the store: no In struct carries a scope, an org or a
+// tenant field, so a caller cannot even NAME another organisation. The
+// namespace is minted from the validated principal and from nothing else.
+//
+// THE BOUND IS PER TENANT. An organisation may hold maxOverrides entries per
+// set, and the write past that is refused. A shared cap would let one
+// organisation's list quietly evict another's; a per-tenant cap means a tenant
+// can only ever degrade itself.
+
+import (
+	"database/sql"
+	"fmt"
+	"strings"
+	"time"
+)
+
+// ── the bound, in the dimension that binds ───────────────────────────────────
+//
+// A CAP ON A COUNT OF CALLER-SIZED VALUES IS NOT A BOUND. This budget used to be
+// stated the other way round — 10,000 entries per set, with the byte figure
+// DERIVED from that as maxKey+maxNote+128 — and the derivation was wrong in two
+// independent ways at once, which is what a derivation nobody measures does:
+//
+//   - IT LEFT OUT THE INDEX. `PRIMARY KEY ("set", key)` gives the dialect an
+//     implicit index carrying the set and the key AGAIN, so the widest row costs
+//     its own bytes plus a second copy of the two widest columns, plus the page
+//     slack and the at-rest encryption every row on this store pays. A real
+//     worst-case row MEASURES 1,952 bytes against the 1,152 published — so the
+//     stated 128 MiB per organisation was really 204 MiB, on the ONE volume every
+//     organisation's store shares.
+//   - IT LEFT THE WRITER UNBOUNDED. `by` is a term of the row and nothing bounded
+//     it, so the product it appears in was not a bound at all: at a 2 KiB writer
+//     the row measures 4,683 bytes and at 8 KiB it measures 10,144 — 1,064 MiB
+//     against a published 128. IAM mints a UUID, so no live principal is anywhere
+//     near it; that is exactly why it survived review. The term was small by luck
+//     rather than by construction, and luck is not a ceiling.
+//
+// So: the budget is stated in BYTES, the entry count is DERIVED from it, every
+// term of a row is refused past its own bound at the one door that writes one,
+// and the row cost the budget divides by is MEASURED against a real file
+// ([TestOneOrgsOverridesCostWhatTheyArePublishedToCost]). There is no second
+// spelling of the bound left to drift.
+
+// ownBudget is what ONE organisation's overrides may cost ON DISK, across every
+// set in the catalog.
+//
+// BYTES and not rows, because bytes is the dimension that is actually shared:
+// every organisation's store is a file on the one volume this deployment mounts,
+// so a per-tenant bound expressed in rows is a tenant filling the disk the other
+// tenants' overrides live on — a cross-tenant failure wearing a filesystem.
+const ownBudget = 128 << 20
+
+// maxActor bounds, IN BYTES, the writer recorded on a row.
+//
+// It is here rather than at the wire op because it is a term of [rowBytes], and
+// the row is what the budget is about: bounded at the one door that writes one,
+// the bound holds for every path that reaches the store and not only for the path
+// a reviewer happened to read. 128 bytes is three times the UUID IAM mints.
+const maxActor = 128
+
+// rowBytes is what ONE worst-case override costs on an organisation's own file:
+// every column at its own bound, the implicit index over the primary key, the
+// page slack and the encryption. It is MEASURED and then published here, in that
+// order — [TestOneOrgsOverridesCostWhatTheyArePublishedToCost] fills a real store
+// with worst-case rows and fails if one costs more than this.
+//
+// 2,560 leaves about a fifth over the measurement, which is the room a schema
+// that gains a bounded column has before the ceiling has to be restated.
+const rowBytes = 2560
+
+// maxOwn is [ownBudget] in ENTRIES PER SET, and it is the bound the write door
+// enforces. DERIVED, so count × cost IS the byte budget and there is exactly one
+// number to change.
+//
+// It divides by the catalog because the budget is per ORGANISATION and an
+// organisation may hold entries in every set: publishing a new set therefore
+// costs every organisation some of its per-set allowance, which is the honest
+// arithmetic rather than a surprise on the volume.
+func maxOwn() int { return ownBudget / (len(Catalog()) * rowBytes) }
+
+// The two verdicts an override can carry. An override is a DECISION, unlike a
+// baseline entry, which carries facts and leaves the decision to policy — the
+// tenant is the only party entitled to say "for us, this one is fine".
+const (
+	Allow = "allow"
+	Deny  = "deny"
+)
+
+// verdictOK is the closed vocabulary. Anything else is refused at the door: a
+// free-text verdict cannot be counted, tested or acted on.
+func verdictOK(v string) bool { return v == Allow || v == Deny }
+
+// overrides is one organisation's override store.
+type overrides struct{ db *sql.DB }
+
+// openOverrides migrates and wraps a freshly opened per-org database.
+//
+// `set` is quoted because it is a keyword in the dialect and the word is the
+// one the wire uses; renaming the column would put a second name on one concept
+// for the sake of the parser.
+func openOverrides(db *sql.DB) (*overrides, error) {
+	const schema = `
+CREATE TABLE IF NOT EXISTS override (
+    "set"   TEXT NOT NULL,
+    key     TEXT NOT NULL,
+    verdict TEXT NOT NULL,
+    note    TEXT NOT NULL DEFAULT '',
+    at      TEXT NOT NULL,
+    by      TEXT NOT NULL DEFAULT '',
+    PRIMARY KEY ("set", key)
+);`
+	if _, err := db.Exec(schema); err != nil {
+		return nil, fmt.Errorf("reference: migrate overrides: %w", err)
+	}
+	return &overrides{db: db}, nil
+}
+
+func (o *overrides) Close() error { return o.db.Close() }
+
+// ReferenceOverride is one entry a tenant laid over the baseline.
+type ReferenceOverride struct {
+	// Key is the member this organisation is speaking about.
+	Key string `json:"key"`
+	// Verdict is allow or deny.
+	Verdict string `json:"verdict"`
+	// Note is why, in the operator's own words. Optional, and bounded.
+	Note string `json:"note,omitempty"`
+	// At is when it was written, RFC 3339.
+	At string `json:"at"`
+	// By is who wrote it.
+	By string `json:"by,omitempty"`
+}
+
+// count is how many entries this organisation holds in one set.
+func (o *overrides) count(set string) (int, error) {
+	var n int
+	err := o.db.QueryRow(`SELECT count(*) FROM override WHERE "set" = ?`, set).Scan(&n)
+	return n, err
+}
+
+// put writes entries, replacing an existing key. Idempotent on (set, key): the
+// same entry written twice is one entry.
+//
+// The whole batch is one transaction, so a batch that would cross the per-set
+// bound writes nothing rather than half of itself — a half-applied deny list is
+// worse than a refused one, because nobody can tell which half applied.
+func (o *overrides) put(set string, in []ReferenceOverride, by string, now time.Time) (int, error) {
+	if len(in) == 0 {
+		return 0, nil
+	}
+	// The writer is a term of [rowBytes], so it is bounded HERE — at the one door
+	// that writes a row — and refused rather than trimmed, the same rule the key and
+	// the note follow. An identity this long cannot be one IAM minted, so refusing
+	// is the reading an operator can act on; silently cutting it would shorten the
+	// attribution on an adverse-action record instead.
+	if len(by) > maxActor {
+		return 0, fmt.Errorf("the writer on this row is %d bytes and the bound is %d; a stored writer is a term of the %d MiB every organisation may hold, so it is refused rather than shortened",
+			len(by), maxActor, ownBudget>>20)
+	}
+	tx, err := o.db.Begin()
+	if err != nil {
+		return 0, err
+	}
+	defer func() { _ = tx.Rollback() }()
+
+	var held int
+	if err := tx.QueryRow(`SELECT count(*) FROM override WHERE "set" = ?`, set).Scan(&held); err != nil {
+		return 0, err
+	}
+	var fresh int
+	for _, e := range in {
+		var exists int
+		if err := tx.QueryRow(`SELECT count(*) FROM override WHERE "set" = ? AND key = ?`, set, e.Key).Scan(&exists); err != nil {
+			return 0, err
+		}
+		if exists == 0 {
+			fresh++
+		}
+	}
+	if bound := maxOwn(); held+fresh > bound {
+		return 0, fmt.Errorf("this org already holds %d overrides in %q and the bound is %d, which is the %d MiB one organisation may hold divided by the %d sets it may hold them in and the %d bytes one costs",
+			held, set, bound, ownBudget>>20, len(Catalog()), rowBytes)
+	}
+
+	stamp := now.UTC().Format(time.RFC3339)
+	for _, e := range in {
+		if _, err := tx.Exec(
+			`INSERT INTO override ("set", key, verdict, note, at, by) VALUES (?, ?, ?, ?, ?, ?)
+			 ON CONFLICT("set", key) DO UPDATE SET verdict = excluded.verdict, note = excluded.note, at = excluded.at, by = excluded.by`,
+			set, e.Key, e.Verdict, e.Note, stamp, by); err != nil {
+			return 0, err
+		}
+	}
+	if err := tx.Commit(); err != nil {
+		return 0, err
+	}
+	return len(in), nil
+}
+
+// clear removes one entry, reporting whether there was one.
+func (o *overrides) clear(set, key string) (bool, error) {
+	res, err := o.db.Exec(`DELETE FROM override WHERE "set" = ? AND key = ?`, set, key)
+	if err != nil {
+		return false, err
+	}
+	n, err := res.RowsAffected()
+	return n > 0, err
+}
+
+// list pages one set's entries in key order.
+func (o *overrides) list(set, after string, limit int) ([]ReferenceOverride, error) {
+	rows, err := o.db.Query(
+		`SELECT key, verdict, note, at, by FROM override WHERE "set" = ? AND key > ? ORDER BY key LIMIT ?`,
+		set, after, limit)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	out := make([]ReferenceOverride, 0, limit)
+	for rows.Next() {
+		var e ReferenceOverride
+		if err := rows.Scan(&e.Key, &e.Verdict, &e.Note, &e.At, &e.By); err != nil {
+			return nil, err
+		}
+		out = append(out, e)
+	}
+	return out, rows.Err()
+}
+
+// pick resolves the MOST SPECIFIC override among a bounded candidate list.
+//
+// The candidates come from the SAME function the baseline lookup uses
+// (candidates, in resolve.go), most specific first, so an override and a
+// baseline entry are matched by one rule. Two matchers would mean a tenant's
+// deny of tempbox.example covering mail.tempbox.example in the baseline's sense
+// but not in their own, which is a surprise nobody can debug.
+//
+// One query with a bounded IN list, never a scan of the tenant's whole set.
+func (o *overrides) pick(set string, candidates []string) (ReferenceOverride, bool, error) {
+	if len(candidates) == 0 {
+		return ReferenceOverride{}, false, nil
+	}
+	args := make([]any, 0, len(candidates)+1)
+	args = append(args, set)
+	holes := make([]string, 0, len(candidates))
+	for _, c := range candidates {
+		holes = append(holes, "?")
+		args = append(args, c)
+	}
+	rows, err := o.db.Query(
+		`SELECT key, verdict, note, at, by FROM override WHERE "set" = ? AND key IN (`+strings.Join(holes, ",")+`)`,
+		args...)
+	if err != nil {
+		return ReferenceOverride{}, false, err
+	}
+	defer rows.Close()
+	found := map[string]ReferenceOverride{}
+	for rows.Next() {
+		var e ReferenceOverride
+		if err := rows.Scan(&e.Key, &e.Verdict, &e.Note, &e.At, &e.By); err != nil {
+			return ReferenceOverride{}, false, err
+		}
+		found[e.Key] = e
+	}
+	if err := rows.Err(); err != nil {
+		return ReferenceOverride{}, false, err
+	}
+	// candidates is ordered most specific first, so the first hit is the answer.
+	for _, c := range candidates {
+		if e, ok := found[c]; ok {
+			return e, true, nil
+		}
+	}
+	return ReferenceOverride{}, false, nil
+}

apps/reference/refresh_test.go

@@ -0,0 +1,532 @@
+package reference
+
+// refresh_test.go holds the four properties of a REFRESH — which version prune
+// spares, what a take that changed size is allowed to do, what a receipt with
+// nothing in it becomes, and when the first take happens — against a warehouse
+// (warehouse_test.go) rather than against the text of a statement.
+//
+// The distinction is the point. Every one of these was a call site that
+// disagreed with a statement constant that was itself correct, so a test reading
+// only the constant passed while the plane did the opposite.
+
+import (
+	"context"
+	"fmt"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/hanzoai/cloud"
+	luxlog "github.com/luxfi/log"
+)
+
+// plant builds the service the way Mount does, minus the routes and minus the
+// background loop, with a downloader the test controls. It is how a refresh is
+// driven without a network and without an HTTP surface.
+func plant(t *testing.T, get download) *cloud.Service[state] {
+	t.Helper()
+	deps := cloud.Deps{Logger: luxlog.New("test"), DataDir: t.TempDir(), Brand: "hanzo"}
+	base := cloud.NewBase(deps, subsystem)
+	own := cloud.NewOrgStore[*overrides](base, subsystem, openOverrides)
+	s := &cloud.Service[state]{
+		Base: base,
+		State: state{
+			plane: newPlane(),
+			own:   own,
+			get:   get,
+			sync:  own.Sync,
+			now:   func() time.Time { return time.Now().UTC() },
+			work:  &work{stop: make(chan struct{})},
+		},
+	}
+	t.Cleanup(func() { _ = own.CloseAll() })
+	return s
+}
+
+// serves is a downloader that answers every URL with the same bytes.
+func serves(body *string) download {
+	return func(context.Context, string) ([]byte, error) { return []byte(*body), nil }
+}
+
+// list renders n disposable domains, which is what the `domain` set's one
+// publisher serves and what parseLines reads.
+func list(n int) string {
+	out := make([]string, 0, n)
+	for i := range n {
+		out = append(out, fmt.Sprintf("tempbox%03d.example", i))
+	}
+	return strings.Join(out, "\n")
+}
+
+// domainSet is the catalog's `domain` set: one publisher, one parser, fetched.
+func domainSet(t *testing.T) Set {
+	t.Helper()
+	set, ok := byName("domain")
+	if !ok {
+		t.Fatal("no domain set")
+	}
+	return set
+}
+
+// TestPruneSparesTheVersionADecisionMayStillCite.
+//
+// The whole plane is sold on one promise: a decision records the version it
+// consulted and an auditor resolves that string back to what the version
+// contained. A refresh happens at some instant; a decision taken a moment before
+// it names the version that was current a moment before it. So a prune that
+// spares only the NEW version deletes the rows behind every citation in that
+// window, and the promise is void for exactly the decisions most likely to be
+// disputed.
+//
+// The call site passed the current version for BOTH of the statement's two
+// placeholders, so `version != ? AND version != ?` spared one version, not two —
+// while the statement constant, and the only test over it, said two.
+func TestPruneSparesTheVersionADecisionMayStillCite(t *testing.T) {
+	w := newWarehouse()
+	w.use(t)
+	body := list(10)
+	s := plant(t, serves(&body))
+	set := domainSet(t)
+	ctx := context.Background()
+
+	took := func() string {
+		t.Helper()
+		got, err := take(ctx, s, set, nil, false)
+		if err != nil {
+			t.Fatalf("take: %v", err)
+		}
+		if got[0].Refusal != "" {
+			t.Fatalf("take refused: %s", got[0].Refusal)
+		}
+		return got[0].Version
+	}
+
+	v1 := took()
+	body = list(11)
+	v2 := took()
+	body = list(12)
+	v3 := took()
+	if v1 == v2 || v2 == v3 {
+		t.Fatalf("three different lists must be three versions: %s %s %s", v1, v2, v3)
+	}
+
+	if n := len(w.rows("domain", "disposable", v3)); n != 12 {
+		t.Errorf("the current version holds %d rows, want 12", n)
+	}
+	if n := len(w.rows("domain", "disposable", v2)); n != 11 {
+		t.Errorf("the PREVIOUS version holds %d rows, want 11 — a decision taken a moment before the refresh cites it, and an auditor asking what it contained deserves an answer", n)
+	}
+	if n := len(w.rows("domain", "disposable", v1)); n != 0 {
+		t.Errorf("the version before last still holds %d rows; two is the whole history the membership keeps", n)
+	}
+
+	// The manifest is a record and is never pruned: every version still resolves
+	// to a publisher, a licence and a date.
+	for _, v := range []string{v1, v2, v3} {
+		if _, ok := w.held("domain", "disposable", v); !ok {
+			t.Errorf("version %s lost its manifest row", v)
+		}
+	}
+
+	// And the two versions named in the last prune are DIFFERENT ones.
+	prunes := w.prunes()
+	if len(prunes) == 0 {
+		t.Fatal("no prune ran")
+	}
+	last := prunes[len(prunes)-1]
+	if last[2] == last[3] {
+		t.Errorf("prune spared %q twice; two placeholders bound to one string spare one version", last[2])
+	}
+	if last[2] != v3 || last[3] != v2 {
+		t.Errorf("prune spared (%q, %q), want the current and the one it replaced (%q, %q)", last[2], last[3], v3, v2)
+	}
+}
+
+// TestSweepOldSparesTheOneItReplaced states the same property on the pure
+// function, so the pair a take carries — current, and the one it superseded — is
+// pinned independently of the warehouse.
+func TestSweepOldSparesTheOneItReplaced(t *testing.T) {
+	var got [][4]string
+	drop := func(_ context.Context, set, source, keep, alsoKeep string) error {
+		got = append(got, [4]string{set, source, keep, alsoKeep})
+		return nil
+	}
+	was := map[string]held{"disposable": {Version: "v1", Keys: 10}}
+	now := []version{{Set: "domain", Source: "disposable", Version: "v2", Keys: 11}}
+	if err := sweepOld(context.Background(), drop, "domain", was, now); err != nil {
+		t.Fatalf("sweepOld: %v", err)
+	}
+	if len(got) != 1 || got[0] != [4]string{"domain", "disposable", "v2", "v1"} {
+		t.Fatalf("sweepOld dropped %v, want one call sparing v2 and v1", got)
+	}
+
+	// A source taken for the FIRST time has no previous version, and an empty
+	// second value spares nothing extra — which is correct, there is nothing extra.
+	got = nil
+	if err := sweepOld(context.Background(), drop, "domain", map[string]held{}, now); err != nil {
+		t.Fatalf("sweepOld: %v", err)
+	}
+	if len(got) != 1 || got[0][3] != "" {
+		t.Fatalf("a first take spared %v; there is no previous version to spare", got)
+	}
+}
+
+// TestASourceThatSwungIsRefusedAndThePreviousVersionStands.
+//
+// The empty take was already an error and the truncated download already an
+// error. Between them sat the dangerous case: a publisher serving a VALID,
+// parseable list at a fraction or a multiple of its previous size, landing
+// silently as the new baseline every organisation's decisions read. A list that
+// shrank answers "not listed" for everything it lost and is indistinguishable
+// from a clean world — the exact failure this plane exists to make visible.
+func TestASourceThatSwungIsRefusedAndThePreviousVersionStands(t *testing.T) {
+	w := newWarehouse()
+	w.use(t)
+	body := list(100)
+	s := plant(t, serves(&body))
+	set := domainSet(t)
+	ctx := context.Background()
+
+	first, err := take(ctx, s, set, nil, false)
+	if err != nil {
+		t.Fatalf("first take: %v", err)
+	}
+	full := first[0].Version
+	if first[0].Keys != 100 {
+		t.Fatalf("first take landed %d", first[0].Keys)
+	}
+
+	// A tenth of the list. Parses cleanly, and is refused.
+	body = list(10)
+	shrunk, err := take(ctx, s, set, nil, false)
+	if err != nil {
+		t.Fatalf("shrunk take: %v", err)
+	}
+	if shrunk[0].Refusal == "" {
+		t.Fatalf("a list at a tenth of its size landed silently: %+v", shrunk[0])
+	}
+	if got := s.State.plane.get("domain"); got == nil || len(got.byKey) != 100 {
+		t.Fatalf("the refused take changed the live snapshot: %v", got)
+	}
+	if _, ok := w.held("domain", "disposable", full); !ok {
+		t.Error("the previous version must stand")
+	}
+
+	// Ten times the list is refused for the same reason, in the other direction.
+	body = list(1000)
+	grown, err := take(ctx, s, set, nil, false)
+	if err != nil {
+		t.Fatalf("grown take: %v", err)
+	}
+	if grown[0].Refusal == "" {
+		t.Fatalf("a list at ten times its size landed silently: %+v", grown[0])
+	}
+
+	// Ordinary movement is not a swing: published lists do move.
+	body = list(150)
+	moved, err := take(ctx, s, set, nil, false)
+	if err != nil {
+		t.Fatalf("moved take: %v", err)
+	}
+	if moved[0].Refusal != "" || moved[0].Keys != 150 {
+		t.Fatalf("ordinary growth must land: %+v", moved[0])
+	}
+
+	// And the operator has a lever: force says the change is real.
+	body = list(10)
+	forced, err := take(ctx, s, set, nil, true)
+	if err != nil {
+		t.Fatalf("forced take: %v", err)
+	}
+	if forced[0].Refusal != "" || forced[0].Keys != 10 {
+		t.Fatalf("force must accept a real change: %+v", forced[0])
+	}
+}
+
+// TestAPoisonedDisposableListDoesNotLand.
+//
+// The size gate catches a list that arrives at a fraction or a multiple of
+// itself. It cannot catch ONE added row, and one added row is the whole attack on
+// the one source in this catalog with no pin, no signature and no digest: the
+// disposable list is fetched from a public repository, and "gmail.com is
+// disposable" refuses a large share of every tenant's legitimate signups at once.
+// A publisher naming a mailbox provider is wrong about something we can check, so
+// the take is refused whole and the previous version stands.
+func TestAPoisonedDisposableListDoesNotLand(t *testing.T) {
+	w := newWarehouse()
+	w.use(t)
+	body := list(50)
+	s := plant(t, serves(&body))
+	set := domainSet(t)
+	ctx := context.Background()
+
+	first, err := take(ctx, s, set, nil, false)
+	if err != nil {
+		t.Fatalf("first take: %v", err)
+	}
+	good := first[0].Version
+
+	// One row added, everything else identical.
+	body = list(50) + "\nGmail.com\n"
+	poisoned, err := take(ctx, s, set, nil, false)
+	if err != nil {
+		t.Fatalf("poisoned take: %v", err)
+	}
+	if poisoned[0].Refusal == "" {
+		t.Fatalf("a list naming a mailbox provider landed: %+v", poisoned[0])
+	}
+	if got := s.State.plane.get("domain"); got == nil || len(got.byKey) != 50 {
+		t.Fatalf("the poisoned take reached the live snapshot: %v", got)
+	}
+	if _, ok := w.held("domain", "disposable", good); !ok {
+		t.Error("the previous version must stand")
+	}
+	// Even forced: force accepts a size change somebody vouched for, not a list
+	// that is wrong about a fact.
+	if forced, err := take(ctx, s, set, nil, true); err != nil || forced[0].Refusal == "" {
+		t.Errorf("force must not land a poisoned list: %+v %v", forced, err)
+	}
+}
+
+func TestSwungMeasuresBothDirections(t *testing.T) {
+	for _, c := range []struct {
+		was, now uint64
+		want     bool
+	}{
+		{0, 5000, false},  // a first take has nothing to compare against
+		{100, 100, false}, // unchanged
+		{100, 399, false}, // inside the bound
+		{100, 401, true},  // past it, growing
+		{100, 26, false},  // inside the bound
+		{100, 24, true},   // past it, shrinking
+		{100, 0, true},    // gone
+	} {
+		if got := swung(c.was, c.now); got != c.want {
+			t.Errorf("swung(%d, %d) = %v, want %v", c.was, c.now, got, c.want)
+		}
+	}
+}
+
+// TestAReceiptWithNothingInItIsNotAFreshList.
+//
+// A set of kind attest holds no membership here: the screening engine holds it
+// and this plane records the engine's load receipt. That makes the receipt the
+// ONLY evidence there is, and it was written through verbatim — a loader posting
+// {"source":"OFAC","version":"","keys":0} made the sanction set report a current,
+// non-stale version composed as "OFAC@", so the compliance freshness signal said
+// the designation lists were current at the moment the loader served nothing.
+// The field's own doc comment names this failure; the code did not check for it.
+func TestAReceiptWithNothingInItIsNotAFreshList(t *testing.T) {
+	w := newWarehouse()
+	w.use(t)
+	body := ""
+	s := plant(t, serves(&body))
+	set, ok := byName("sanction")
+	if !ok {
+		t.Fatal("no sanction set")
+	}
+	ctx := context.Background()
+
+	// A load that designated nobody, and one that names no version.
+	got, err := take(ctx, s, set, []ReferenceReceipt{
+		{Source: "OFAC", Version: "2026-08-01", Keys: 0},
+		{Source: "UN", Version: "", Keys: 12000},
+	}, false)
+	if err != nil {
+		t.Fatalf("take: %v", err)
+	}
+	for _, r := range got {
+		if r.Refusal == "" {
+			t.Errorf("%s: a receipt with no designations or no version was recorded as a successful load: %+v", r.Source, r)
+		}
+	}
+	if v, ok := w.held("sanction", "OFAC", "2026-08-01"); !ok || v.Status == statusReady {
+		t.Errorf("a zero-designation load is durable as a refusal, not as a ready version: %+v", v)
+	}
+	// The set therefore still has no version, and says so rather than reporting a
+	// current one composed of nothing.
+	view := project(set, s.State.plane.get("sanction"), s.State.now())
+	if view.Version != "" {
+		t.Errorf("the set reports version %q from receipts that carried nothing", view.Version)
+	}
+	if view.Refusal == "" {
+		t.Error("a set with no ready version must refuse")
+	}
+
+	// A real receipt is recorded, ready, and names its freshness.
+	if _, err := take(ctx, s, set, []ReferenceReceipt{{Source: "OFAC", Version: "sha-abc", Keys: 12000}}, false); err != nil {
+		t.Fatalf("take: %v", err)
+	}
+	v, ok := w.held("sanction", "OFAC", "sha-abc")
+	if !ok || v.Status != statusReady || v.Keys != 12000 {
+		t.Fatalf("a real load must be ready: %+v", v)
+	}
+}
+
+func TestUnattestedNamesWhyAReceiptIsNotEvidence(t *testing.T) {
+	for _, c := range []struct {
+		name string
+		r    ReferenceReceipt
+		want bool
+	}{
+		{"whole", ReferenceReceipt{Version: "v", Keys: 1}, false},
+		{"no designations", ReferenceReceipt{Version: "v", Keys: 0}, true},
+		{"negative", ReferenceReceipt{Version: "v", Keys: -1}, true},
+		{"no version", ReferenceReceipt{Version: " ", Keys: 1}, true},
+		{"the loader said so", ReferenceReceipt{Version: "v", Keys: 1, Refusal: "connection reset"}, true},
+	} {
+		if got := unattested(c.r) != ""; got != c.want {
+			t.Errorf("%s: unattested = %v, want %v", c.name, got, c.want)
+		}
+	}
+}
+
+// TestColdStartTakesRatherThanRefusingForTheWholeBeat.
+//
+// Into an EMPTY warehouse — a first deploy, a wipe, a migration — the first
+// hydrate legitimately finds nothing and succeeds. With the take only on the
+// beat, every set then answered "this set has never loaded" for a whole [beat]
+// while the only operator lever was one hand-made refresh call per set. A control
+// that is not there for six hours after a deploy is not a control.
+func TestColdStartTakesRatherThanRefusingForTheWholeBeat(t *testing.T) {
+	w := newWarehouse()
+	w.use(t)
+	body := list(20)
+	s := plant(t, serves(&body))
+
+	// Nothing has ever loaded, and the plane says so.
+	if got := s.State.plane.get("domain"); got != nil {
+		t.Fatalf("a fresh plane already holds %v", got)
+	}
+
+	s.State.work.done.Add(1)
+	go tend(s)
+	t.Cleanup(func() { close(s.State.work.stop); s.State.work.done.Wait() })
+
+	// Well inside `settle`, let alone `beat`.
+	deadline := time.Now().Add(5 * time.Second)
+	for {
+		got := s.State.plane.get("domain")
+		if got != nil && len(got.byKey) == 20 {
+			break
+		}
+		if time.Now().After(deadline) {
+			t.Fatalf("the plane did not take a set at cold start; it refuses until the %s beat: %v", beat, got)
+		}
+		time.Sleep(5 * time.Millisecond)
+	}
+	if v, _ := project(domainSet(t), s.State.plane.get("domain"), s.State.now()), 0; v.Refusal != "" {
+		t.Errorf("a set taken at cold start still refuses: %s", v.Refusal)
+	}
+}
+
+// TestASweepTakesOnlyWhatIsDue: the cold start and the beat run the SAME pass, so
+// a set that is current is not re-taken by either.
+func TestASweepTakesOnlyWhatIsDue(t *testing.T) {
+	w := newWarehouse()
+	w.use(t)
+	body := list(20)
+	s := plant(t, serves(&body))
+
+	sweep(s)
+	first := s.State.plane.get("domain")
+	if first == nil || len(first.byKey) != 20 {
+		t.Fatalf("the first sweep did not take the set: %v", first)
+	}
+	wrote := w.wrote
+
+	// Nothing has aged, so the second sweep writes no entry rows.
+	sweep(s)
+	if w.wrote != wrote {
+		t.Errorf("a sweep re-landed %d rows for a set that is still current", w.wrote-wrote)
+	}
+}
+
+// TestASourceCannotLandWhatNoLookupCouldReach.
+//
+// [maxKey] is refused at the wire in both directions — a key looked up, a key
+// written as an override — and a MEMBER arriving from a publisher is the same
+// value from the third side. A member longer than the bound is one no lookup can
+// ever match, so landing it costs the warehouse, every hydrate and the snapshot
+// every request reads, and buys nothing.
+//
+// Refused WHOLE, like the mailbox-provider gate: a list quietly missing the rows
+// we dropped answers "not listed" for them and reads exactly like a clean world.
+func TestASourceCannotLandWhatNoLookupCouldReach(t *testing.T) {
+	w := newWarehouse()
+	w.use(t)
+	body := list(50)
+	s := plant(t, serves(&body))
+	set := domainSet(t)
+	ctx := context.Background()
+
+	first, err := take(ctx, s, set, nil, false)
+	if err != nil {
+		t.Fatalf("first take: %v", err)
+	}
+	good := first[0].Version
+
+	// One member past the door, everything else ordinary.
+	body = list(50) + "\n" + strings.Repeat("a.", maxKey) + "example\n"
+	huge, err := take(ctx, s, set, nil, false)
+	if err != nil {
+		t.Fatalf("take: %v", err)
+	}
+	if huge[0].Refusal == "" {
+		t.Fatalf("a member no lookup could reach landed in the baseline every org reads: %+v", huge[0])
+	}
+	if got := s.State.plane.get("domain"); got == nil || len(got.byKey) != 50 {
+		t.Fatalf("the refused take reached the live snapshot: %v", got)
+	}
+	if _, ok := w.held("domain", "disposable", good); !ok {
+		t.Error("the previous version must stand")
+	}
+	// Force is the lever for a size somebody vouched for, not for a member the
+	// lookup door would refuse anyway.
+	if forced, err := take(ctx, s, set, nil, true); err != nil || forced[0].Refusal == "" {
+		t.Errorf("force must not land an unreachable member: %+v %v", forced, err)
+	}
+
+	// A member AT the bound is a member, and lands.
+	at := strings.Repeat("b", maxKey-len(".example")) + ".example"
+	if len(at) != maxKey {
+		t.Fatalf("the sample member is %d bytes, not the bound", len(at))
+	}
+	body = list(50) + "\n" + at + "\n"
+	ok2, err := take(ctx, s, set, nil, false)
+	if err != nil || ok2[0].Refusal != "" || ok2[0].Keys != 51 {
+		t.Fatalf("a member at the bound must land: %+v %v", ok2[0], err)
+	}
+}
+
+// TestASourceCannotLandMoreMembersThanTheDoorAdmits.
+//
+// [swing] bounds how far a take may move from the version it REPLACES, and a
+// first take has no previous version to be measured against — which, after a
+// cold start into an empty warehouse, is every take. So the publisher's end of
+// the resolve amplifier was open: one source, one refresh, however many members
+// [maxBody] admits, all of them into the snapshot this one-replica deployment
+// keeps in memory for every request.
+func TestASourceCannotLandMoreMembersThanTheDoorAdmits(t *testing.T) {
+	s := plant(t, serves(new(string)))
+
+	// One string, shared by every element: this test is about the COUNT, and
+	// materialising a million distinct keys would only measure the test.
+	flood := make([]Entry, maxMembers+1)
+	for i := range flood {
+		flood[i] = Entry{Key: "member.example"}
+	}
+	src := Source{Name: "flood", Origin: "local", Basis: GrantOwn, Terms: "a test",
+		produce: func(producer) ([]Entry, error) { return flood, nil }}
+
+	if _, err := gather(context.Background(), s, src, s.State.now()); err == nil {
+		t.Fatalf("a source landed %d members with nothing to measure it against", len(flood))
+	}
+	// And the bound admits what a publisher plausibly serves: the largest list in
+	// this catalog is a ninth of it.
+	fine := Source{Name: "fine", Origin: "local", Basis: GrantOwn, Terms: "a test",
+		produce: func(producer) ([]Entry, error) { return flood[:maxMembers], nil }}
+	if _, err := gather(context.Background(), s, fine, s.State.now()); err != nil {
+		t.Errorf("a source at the bound must be admitted: %v", err)
+	}
+}

apps/reference/resolve.go

@@ -0,0 +1,426 @@
+package reference
+
+// resolve.go answers the only question this plane exists to answer: what is
+// known about this key, which version said so, and how old is that version.
+//
+// TWO LOOKUPS, ALWAYS IN THIS ORDER: the caller's OWN override first, then the
+// shared baseline. First hit wins. An organisation's say about its own world
+// beats a published list, and that is the whole of the precedence rule — there
+// is no third tier and no merge.
+//
+// ONE MATCHER FOR BOTH. The candidate keys an override is looked up by are the
+// SAME candidates the baseline is looked up by, produced by one function, most
+// specific first. Two matchers would mean a tenant's deny of tempbox.example
+// covering mail.tempbox.example in the baseline's sense and not in their own.
+//
+// SILENCE IS NEVER CLEAN. An answer carries Refusal when the set could not be
+// consulted — never loaded, unlicensed, held elsewhere — and a caller that reads
+// Hit==false without reading Refusal is reading "we have no idea" as "not
+// listed". That distinction is the difference between a control and the
+// appearance of one, and it is why every field below exists.
+//
+// STALENESS IS A SIGNAL, NOT AN ERROR. A set past its freshness bound still
+// answers, and says so, because yesterday's list beats no list — but a decision
+// that consulted a three-week-old disposable-domain list should be able to know
+// that it did.
+
+import (
+	"net/netip"
+	"regexp"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+// scanned is one entry of a set that cannot be addressed by key: a compiled
+// crawler pattern, or a delegated number range.
+type scanned struct {
+	entry  Entry
+	re     *regexp.Regexp
+	lo, hi uint64
+}
+
+// snap is one set, resolved and immutable. It is built once per refresh and
+// swapped in whole, so a reader either sees the old version or the new one and
+// never a half-built map.
+type snap struct {
+	set     Set
+	version string
+	took    []version
+	asOf    time.Time
+	fetched time.Time
+	byKey   map[string]Entry
+	scan    []scanned
+	any     *regexp.Regexp
+	refusal string
+}
+
+// stale reports whether this set is past its freshness bound.
+func (s *snap) stale(now time.Time) bool {
+	if s == nil || s.asOf.IsZero() {
+		return true
+	}
+	if s.set.MaxAge <= 0 {
+		return false
+	}
+	return now.Sub(s.asOf) > s.set.MaxAge
+}
+
+// age is how long since the OLDEST contributing publisher was current. The
+// oldest and not the newest: a set is exactly as fresh as its weakest source,
+// and reporting the newest would let one daily-updating publisher hide three
+// that stopped answering months ago.
+func (s *snap) age(now time.Time) time.Duration {
+	if s == nil || s.asOf.IsZero() {
+		return 0
+	}
+	return now.Sub(s.asOf)
+}
+
+// neverLoaded is the ONE sentence an unloaded set says, and it is one sentence
+// because it is one fact reached from four places: a snapshot built with no ready
+// source ([build]), an answer for a set the plane holds NOTHING for ([answer]),
+// the consulted row beside it (resolve), and the set view (project). It was
+// written out four times. The BEHAVIOUR is held on the wire already —
+// [TestOverrideSurvivesAnUnloadedBaseline] fails if any of them stops saying it —
+// so this is not a hole being closed; it is four copies of one sentence collapsed
+// so they cannot come to say four different things.
+//
+// The state it names is the ordinary one, not an exotic one: this deployment runs
+// one replica with a recreate rollout, so EVERY rollout starts with the plane
+// holding nothing at all until the first hydrate lands. That is why "unloaded"
+// answers rather than erroring, and why the answer has to carry the reason.
+const neverLoaded = "this set has never loaded, so it cannot tell a clean key from an unknown one"
+
+// build assembles a set's snapshot from its sources' current versions and their
+// entries. A set with no ready source is not an empty set — it is a set that
+// refuses, and the refusal names why.
+func build(set Set, took []version, entries []Entry) *snap {
+	s := &snap{set: set, took: took, byKey: map[string]Entry{}}
+
+	if set.Kind == KindSeam {
+		s.refusal = set.Refusal
+		return s
+	}
+	if len(took) == 0 {
+		s.refusal = neverLoaded
+		return s
+	}
+
+	sort.Slice(took, func(i, j int) bool { return took[i].Source < took[j].Source })
+	var parts []string
+	for i, v := range took {
+		parts = append(parts, v.Source+"@"+v.Version)
+		if i == 0 || v.AsOf.Before(s.asOf) {
+			s.asOf = v.AsOf
+		}
+		if v.Fetched.After(s.fetched) {
+			s.fetched = v.Fetched
+		}
+	}
+	// The set's version is the composition of its sources' versions, so a
+	// decision can name ONE string and an auditor can resolve it back to exactly
+	// which publisher contributed what.
+	s.version = strings.Join(parts, "+")
+
+	if set.Kind == KindAttest {
+		// The membership is held by the component that screens against it. This
+		// plane knows how fresh that component's lists are and nothing else, and
+		// saying so is more useful than a second copy that would drift.
+		s.refusal = "membership of this set is held by the component that screens against it; this plane reports its freshness"
+		return s
+	}
+
+	switch set.Match {
+	case MatchPattern:
+		var alts []string
+		for _, e := range entries {
+			re, err := regexp.Compile(e.Key)
+			if err != nil {
+				// A pattern that will not compile is dropped rather than failing the
+				// whole set: one bad row from a publisher must not silence the rest.
+				continue
+			}
+			s.scan = append(s.scan, scanned{entry: e, re: re})
+			alts = append(alts, "(?:"+e.Key+")")
+		}
+		// One alternation over every pattern is the fast reject. Most traffic
+		// matches nothing, and a single automaton pass answers that in one walk of
+		// the input instead of one walk per pattern.
+		if len(alts) > 0 {
+			if re, err := regexp.Compile(strings.Join(alts, "|")); err == nil {
+				s.any = re
+			}
+		}
+	case MatchRange:
+		for _, e := range entries {
+			lo, hi, ok := span(e.Key)
+			if !ok {
+				continue
+			}
+			s.scan = append(s.scan, scanned{entry: e, lo: lo, hi: hi})
+		}
+	default:
+		for _, e := range entries {
+			s.byKey[e.Key] = e
+		}
+	}
+
+	if len(entries) == 0 && !set.emptyIsFact() {
+		s.refusal = "this set loaded no entries, which no published list is — the fetch or the parse is wrong"
+	}
+	return s
+}
+
+// candidates renders the bounded, most-specific-first list of keys a lookup
+// should try. It is the ONE place a set's matching shape becomes concrete, and
+// both the override store and the baseline snapshot consult it.
+//
+// Every case is bounded in COUNT by the key's own length — at most one key for an
+// exact set, one per label for a hostname, eight for a card number, 129 for an
+// address — and every case is bounded in BYTES by that length too, which is the
+// property that matters and the one this function used to lack.
+//
+// A DOMAIN SUFFIX IS A SLICE, NEVER A JOIN. Splitting a host into labels and
+// re-joining each tail allocates a fresh copy of every suffix: an L-label host
+// costs O(L * len(key)) bytes, so one 8 KB dotted key materialised 16 MB and one
+// [maxKey]-bounded resolve call of [maxKeys] such keys was 1.7 GB in a single
+// request — on a one-replica deployment, an OOM every product on the host shares.
+// A Go string is immutable, so host[o:] is the SAME bytes with a different header:
+// walking the dot offsets gives the identical suffixes in O(L) headers over one
+// backing array. The bound at the door ([maxKey], reference.go) and this shape are
+// the two halves of one property — the door refuses a key no published list could
+// carry, and this makes the work linear in whatever the door admits.
+func candidates(set Set, key string) []string {
+	key = strings.TrimSpace(key)
+	if key == "" {
+		return nil
+	}
+	switch set.Match {
+	case MatchDomain:
+		host := strings.ToLower(strings.Trim(key, "."))
+		if at := strings.LastIndexByte(host, '@'); at >= 0 {
+			host = host[at+1:]
+		}
+		// Every suffix that still has two labels in it, most specific first. A bare
+		// public suffix is deliberately not a candidate: a deny on ".example" would
+		// be a deny on a registry, not on a member.
+		out := make([]string, 0, 8)
+		for o := 0; ; {
+			rest := host[o:]
+			dot := strings.IndexByte(rest, '.')
+			if dot < 0 {
+				break
+			}
+			out = append(out, rest)
+			o += dot + 1
+		}
+		if len(out) == 0 {
+			out = append(out, host)
+		}
+		return out
+	case MatchDigits:
+		digits := strings.Map(func(r rune) rune {
+			if r >= '0' && r <= '9' {
+				return r
+			}
+			return -1
+		}, key)
+		if digits == "" {
+			return nil
+		}
+		// An issuer identification number is at most eight digits; taking prefixes
+		// longer than that from a full card number would be looking up the account.
+		n := min(len(digits), 8)
+		out := make([]string, 0, n)
+		for i := n; i >= 1; i-- {
+			out = append(out, digits[:i])
+		}
+		return out
+	case MatchNet:
+		addr, err := netip.ParseAddr(strings.Trim(key, "[]"))
+		if err != nil {
+			if p, ok := prefix(key); ok {
+				return []string{p.String()}
+			}
+			return nil
+		}
+		addr = addr.Unmap()
+		out := make([]string, 0, addr.BitLen()+1)
+		for bits := addr.BitLen(); bits >= 0; bits-- {
+			out = append(out, netip.PrefixFrom(addr, bits).Masked().String())
+		}
+		return out
+	default:
+		return []string{key}
+	}
+}
+
+// look resolves a key against one built snapshot. It returns the entry and the
+// candidate that matched, so the answer can say WHICH published member covered
+// the key — a deny on tempbox.example is a different fact from a deny on
+// mail.tempbox.example and an operator has to be able to tell them apart.
+func (s *snap) look(key string) (Entry, string, bool) {
+	if s == nil {
+		return Entry{}, "", false
+	}
+	switch s.set.Match {
+	case MatchPattern:
+		if s.any != nil && !s.any.MatchString(key) {
+			return Entry{}, "", false
+		}
+		for _, c := range s.scan {
+			if c.re != nil && c.re.MatchString(key) {
+				return c.entry, c.entry.Key, true
+			}
+		}
+		return Entry{}, "", false
+	case MatchRange:
+		n, err := strconv.ParseUint(strings.TrimSpace(strings.TrimPrefix(strings.ToUpper(key), "AS")), 10, 64)
+		if err != nil {
+			return Entry{}, "", false
+		}
+		for _, c := range s.scan {
+			if n >= c.lo && n <= c.hi {
+				return c.entry, c.entry.Key, true
+			}
+		}
+		return Entry{}, "", false
+	default:
+		for _, c := range candidates(s.set, key) {
+			if e, ok := s.byKey[c]; ok {
+				return e, c, true
+			}
+		}
+		return Entry{}, "", false
+	}
+}
+
+// plane holds the built snapshots. The map is replaced per set under a write
+// lock and every snap is immutable, so a reader takes a pointer and is never
+// racing a build.
+//
+// Bounded by the CATALOG, which is code: there is one snapshot per declared set
+// and no caller can mint another. That is the difference between this and a
+// per-tenant cache — there is no key an adversary controls.
+type plane struct {
+	mu   sync.RWMutex
+	snap map[string]*snap
+}
+
+func newPlane() *plane { return &plane{snap: map[string]*snap{}} }
+
+func (p *plane) get(set string) *snap {
+	p.mu.RLock()
+	defer p.mu.RUnlock()
+	return p.snap[set]
+}
+
+func (p *plane) put(name string, s *snap) {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	p.snap[name] = s
+}
+
+// all returns every built snapshot, in catalog order.
+func (p *plane) all() []*snap {
+	p.mu.RLock()
+	defer p.mu.RUnlock()
+	out := make([]*snap, 0, len(p.snap))
+	for _, s := range Catalog() {
+		if got, ok := p.snap[s.Name]; ok {
+			out = append(out, got)
+		}
+	}
+	return out
+}
+
+// ReferenceAnswer is what the plane says about one key in one set — including, always,
+// which version said it.
+type ReferenceAnswer struct {
+	// Set is the set consulted.
+	Set string `json:"set"`
+	// Key is the key as asked.
+	Key string `json:"key"`
+	// Hit is whether the key is a member. It is meaningful ONLY when Refusal is
+	// empty: false with a refusal means the set could not be consulted, which is
+	// not the same as the key being clean.
+	Hit bool `json:"hit"`
+	// From is override or baseline — which plane answered.
+	From string `json:"from,omitempty"`
+	// Matched is the member that covered the key, which for a domain or a network
+	// is the enclosing entry rather than the key itself.
+	Matched string `json:"matched,omitempty"`
+	// Verdict is the tenant's own allow or deny, present only for an override.
+	// The baseline never carries one: it states facts and leaves the decision to
+	// the caller's policy.
+	Verdict string `json:"verdict,omitempty"`
+	// Value is what the publisher says about the member — class, operator,
+	// scheme, region.
+	Value map[string]string `json:"value,omitempty"`
+	// Score is the published risk weight where the source expresses one.
+	Score float64 `json:"score,omitempty"`
+	// Version is the exact baseline version consulted, composed of each
+	// contributing publisher and its content digest. It is what makes a decision
+	// reproducible: an auditor takes this string and knows precisely what was
+	// consulted.
+	Version string `json:"version,omitempty"`
+	// AsOf is when the oldest contributing publisher was current, RFC 3339.
+	AsOf string `json:"asOf,omitempty"`
+	// Age is how old that is, as a duration.
+	Age string `json:"age,omitempty"`
+	// Stale is whether the set is past its freshness bound. A stale set still
+	// answers — yesterday's list beats none — and this is how a decision knows it
+	// leaned on one.
+	Stale bool `json:"stale,omitempty"`
+	// Refusal is why the set could not be consulted, when it could not: never
+	// loaded, held elsewhere, or a source we hold no licence for. Non-empty means
+	// Hit must not be read as an answer.
+	Refusal string `json:"refusal,omitempty"`
+}
+
+// answer resolves one key: the tenant's override first, then the baseline. own
+// is the tenant's store, or nil for a caller with none yet — which is not an
+// error, it is an organisation that has never written one.
+func answer(set Set, s *snap, own *overrides, key string, now time.Time) (ReferenceAnswer, error) {
+	a := ReferenceAnswer{Set: set.Name, Key: key}
+	if s != nil {
+		a.Version, a.Stale = s.version, s.stale(now)
+		if !s.asOf.IsZero() {
+			a.AsOf = s.asOf.UTC().Format(time.RFC3339)
+			a.Age = s.age(now).Truncate(time.Minute).String()
+		}
+		a.Refusal = s.refusal
+	} else {
+		a.Refusal = neverLoaded
+	}
+
+	// The override is consulted even for a seam or an unloaded set, and
+	// deliberately: an organisation's own deny list is the one thing that still
+	// works when the published source does not, and refusing to read it because
+	// the baseline is unavailable would take away the only control left.
+	if own != nil {
+		if e, ok, err := own.pick(set.Name, candidates(set, key)); err != nil {
+			return ReferenceAnswer{}, err
+		} else if ok {
+			a.Hit, a.From, a.Matched, a.Verdict = true, "override", e.Key, e.Verdict
+			if e.Note != "" {
+				a.Value = map[string]string{"note": e.Note}
+			}
+			a.Refusal = ""
+			return a, nil
+		}
+	}
+
+	if a.Refusal != "" {
+		return a, nil
+	}
+	if e, matched, ok := s.look(key); ok {
+		a.Hit, a.From, a.Matched, a.Value, a.Score = true, "baseline", matched, e.Value, e.Score
+	} else {
+		a.From = "baseline"
+	}
+	return a, nil
+}

apps/reference/set.go

@@ -0,0 +1,379 @@
+package reference
+
+// set.go is the CATALOG: every reference set this plane can answer from, where
+// each one comes from, and under what terms we are allowed to hold it.
+//
+// The catalog is code, not configuration. A set that exists is a set someone
+// wrote a parser and a licence line for, and a set whose source we may not
+// redistribute is DECLARED here as a seam rather than quietly omitted — an
+// absent set and an unlicensed one look identical from the outside, and only one
+// of them is a decision.
+//
+// THE UNIT OF VERSION AND FRESHNESS IS THE SOURCE, NOT THE SET. A set is the
+// union of its sources, and each source carries its own version, its own as-of
+// and its own failure. That is not an implementation detail: `net` draws on
+// eight publishers, and a set-wide version would make one publisher's outage
+// either block every other publisher's update or silently shrink the set. Per
+// source, a publisher that stops answering ages out visibly on its own row while
+// the rest stay current — which is the same shape luxfi/aml pkg/screen arrived at
+// for the four sanctions publishers, for the same reason.
+
+import (
+	"time"
+)
+
+// Kind is how a set's baseline comes to exist. Four kinds and no more, because
+// each one implies a different answer to "what does silence mean here".
+type Kind string
+
+const (
+	// KindFetch is downloaded from a published source. An EMPTY fetch set is a
+	// failure, never a fact: no publisher's list of disposable domains or hosting
+	// ranges is empty, so zero entries means the fetch or the parse is wrong.
+	KindFetch Kind = "fetch"
+	// KindLocal is computed here — a structural table that follows from a
+	// published standard, or an aggregate over fleet traffic. An empty local set
+	// IS a fact: "no device is shared across enough organisations to publish" is a
+	// true statement about the world.
+	KindLocal Kind = "local"
+	// KindAttest is held by the component that screens against it. This plane
+	// records that component's load receipt and answers ONLY freshness — never
+	// membership, because a second copy of a sanctions list is a second thing to
+	// keep current and the two would disagree on the day it mattered.
+	KindAttest Kind = "attest"
+	// KindSeam is declared and NOT held: the source needs a licence we do not
+	// have. Every lookup against it refuses. A seam is louder than an omission,
+	// which is the whole reason it is in the catalog.
+	KindSeam Kind = "seam"
+)
+
+// Match is how a key is tested against a set's entries. Five matchers, each a
+// pure function over a built snapshot (see resolve.go).
+type Match string
+
+const (
+	// MatchExact is equality on the normalised key: an ASN, a device digest, a
+	// publisher name.
+	MatchExact Match = "exact"
+	// MatchDomain walks a hostname up its labels — mail.tempbox.example matches an
+	// entry for tempbox.example — because a disposable provider's subdomains are
+	// disposable too.
+	MatchDomain Match = "domain"
+	// MatchNet is longest-prefix on an IP address against CIDR entries.
+	MatchNet Match = "net"
+	// MatchDigits is longest numeric prefix, which is how an issuer identification
+	// number addresses a card scheme.
+	MatchDigits Match = "digits"
+	// MatchPattern tests the key against each entry as a regular expression,
+	// which is how a crawler declares itself in a user-agent string.
+	MatchPattern Match = "pattern"
+	// MatchRange is containment in a closed numeric interval, which is how a
+	// number registry delegates autonomous system numbers in blocks.
+	MatchRange Match = "range"
+)
+
+// Set is one published reference set: what it holds, how fresh it has to be, and
+// where its entries lawfully come from.
+type Set struct {
+	// Name is the address: /v1/reference/<name>. One word, lower case.
+	Name string
+	// Kind decides what an empty set means and whether membership is held here.
+	Kind Kind
+	// What is one sentence an operator can read.
+	What string
+	// Match is how a key is tested against this set's entries.
+	Match Match
+	// MaxAge is how old a source's newest successful load may be before this set
+	// is STALE. Past it the set still answers, and every answer says so — a stale
+	// list answers "not listed" for everything and reads exactly like a clean
+	// world, which is the failure this whole plane exists to make visible.
+	MaxAge time.Duration
+	// Sources are the publishers this set draws on. Empty for local and seam sets.
+	Sources []Source
+	// Refusal is why a seam set cannot be consulted. Non-empty ONLY for KindSeam,
+	// and it names the licence we do not hold rather than saying "unavailable".
+	Refusal string
+}
+
+// Grant is the BASIS on which a source's data may reach a tenant through this
+// plane. It is a closed vocabulary rather than free text, and that is the whole
+// point of the type.
+//
+// Terms used to be the only field, and it carried both kinds of sentence at
+// once: "CC0-1.0" (a licence) and "operator-published range list" (a description
+// of where a file came from). The gate over it could only ask whether the string
+// was non-empty, so an unlicensed source wearing a licence field passed — the
+// mirror image of the seam argument this plane is built on, where an unlicensed
+// set REFUSES precisely because an absent one and an unlicensed one look
+// identical from the outside.
+//
+// Splitting the kind from the citation makes the position machine-checkable and
+// puts it on the wire, so which sources rest on a licence and which rest on an
+// operator's own publication is an audit anyone can run rather than a judgement
+// buried in a string.
+type Grant string
+
+const (
+	// GrantLicence — the publisher states an explicit licence that permits
+	// redistribution. Terms names it: CC0-1.0, MIT, CC BY 3.0 US.
+	GrantLicence Grant = "licence"
+	// GrantRegistry — the registry of record publishes the data for anyone to
+	// consult, which is what a registry is for. Terms names the registry.
+	GrantRegistry Grant = "registry"
+	// GrantOperator — an operator's machine-readable statement about its OWN
+	// network, published so third parties can filter and route by it. It is NOT a
+	// licence and this value does not claim one: it says the data is a list of
+	// factual prefixes the operator publishes for exactly this use, and Terms names
+	// the publication. Stating that plainly is what lets someone review it.
+	GrantOperator Grant = "operator"
+	// GrantOwn — computed here, from a published standard or from fleet aggregates
+	// that clear the k-anonymity floor. Nothing of anyone else's is redistributed.
+	GrantOwn Grant = "own"
+	// GrantNone — nothing reaches a tenant through this source at all: the
+	// membership is held by the component that screens against it and this plane
+	// carries only its freshness. The only honest basis for a set of kind attest.
+	GrantNone Grant = "none"
+)
+
+// grants is the vocabulary as a set, so the gate has ONE definition to check
+// against and a new value cannot be introduced by spelling it.
+var grants = map[Grant]bool{GrantLicence: true, GrantRegistry: true, GrantOperator: true, GrantOwn: true, GrantNone: true}
+
+// Redistributes reports whether this basis lets a publisher's bytes reach a
+// tenant. It is the predicate a fetched source must satisfy.
+func (g Grant) Redistributes() bool {
+	return g == GrantLicence || g == GrantRegistry || g == GrantOperator
+}
+
+// Source is one publisher of one set: where it is, on what basis we may pass it
+// on, and how its bytes become entries.
+type Source struct {
+	// Name is the publisher, stable across versions — it is the key freshness is
+	// tracked per.
+	Name string
+	// Origin is the exact URL fetched, so an auditor can take the same bytes.
+	Origin string
+	// Basis is the KIND of permission this data reaches a tenant under, from a
+	// closed vocabulary. Required: the zero value is not a basis, and the catalog
+	// gate refuses it.
+	Basis Grant
+	// Terms is the CITATION the basis points at — the licence identifier, the
+	// registry, or the operator publication. Required: a basis with nothing behind
+	// it is an assertion.
+	Terms string
+	// parse turns the fetched bytes into entries. Nil for a local source, whose
+	// entries come from produce.
+	parse func([]byte) ([]Entry, error)
+	// produce computes a local source's entries. Nil for a fetched source.
+	produce func(ctx producer) ([]Entry, error)
+}
+
+// Entry is one member of a set: the key, the facts the publisher states about
+// it, and — for a set derived from fleet traffic — the two counts that prove the
+// row could not have come from one organisation.
+type Entry struct {
+	// Key is the normalised member: a domain, a CIDR, an IIN prefix, a digest.
+	Key string
+	// Value is what the publisher says about it — class, operator, region,
+	// scheme. Facts, never a verdict: the verdict is the caller's policy, and a
+	// baseline that shipped verdicts would be making every tenant's policy for it.
+	Value map[string]string
+	// Score is a risk weight in [0,1] where the source expresses one, else zero.
+	Score float64
+	// Orgs and N are the k-anonymity evidence for a derived entry: how many
+	// distinct organisations and how many observations produced it. Zero for a
+	// published source, where the evidence is the licence instead.
+	Orgs uint32
+	N    uint64
+}
+
+// Freshness bounds. A day for the lists that move daily, a week for the ones
+// that move monthly. They are separate constants rather than one because the
+// question "is this stale" has a different honest answer per publisher cadence.
+const (
+	daily   = 36 * time.Hour
+	weekly  = 8 * 24 * time.Hour
+	monthly = 40 * 24 * time.Hour
+)
+
+// Catalog is every set, in a stable order. It is the ONE declaration: the routes
+// project it, the refresh walks it, and a lookup for a name not in it is a 404
+// rather than an invented empty answer.
+//
+// LAWFULNESS IS A FIELD, NOT A FOOTNOTE. Every fetched source states the terms
+// it is redistributed under, and every source we would want but may not have is
+// present as a seam naming the licence we lack. The three seams below are the
+// honest state of the art: politically-exposed-person listings, issuer
+// identification tables and commercial network reputation are all sold, and the
+// free copies in circulation are either non-commercial-only or of unstated
+// provenance. Embedding one of those would put a licence breach inside a
+// compliance product.
+func Catalog() []Set {
+	return []Set{
+		{
+			Name:   "domain",
+			Kind:   KindFetch,
+			What:   "Email domains that hand out throwaway inboxes.",
+			Match:  MatchDomain,
+			MaxAge: weekly,
+			Sources: []Source{{
+				Name:   "disposable",
+				Origin: "https://raw.githubusercontent.com/disposable-email-domains/disposable-email-domains/main/disposable_email_blocklist.conf",
+				Basis:  GrantLicence,
+				Terms:  "CC0-1.0",
+				parse:  parseDisposable,
+			}},
+		},
+		{
+			Name:   "net",
+			Kind:   KindFetch,
+			What:   "IP ranges with a known character: cloud and hosting estates, Tor exits, and the addresses no public host may use.",
+			Match:  MatchNet,
+			MaxAge: weekly,
+			Sources: []Source{
+				{Name: "aws", Origin: "https://ip-ranges.amazonaws.com/ip-ranges.json", Basis: GrantOperator, Terms: "Amazon's own ip-ranges.json, published machine-readable so third parties can filter and route by it; no separate licence is stated and none is claimed here", parse: parseAWS},
+				{Name: "gcp", Origin: "https://www.gstatic.com/ipranges/cloud.json", Basis: GrantOperator, Terms: "Google's own cloud.json, published machine-readable so third parties can filter and route by it; no separate licence is stated and none is claimed here", parse: parseGCP},
+				{Name: "oracle", Origin: "https://docs.oracle.com/iaas/tools/public_ip_ranges.json", Basis: GrantOperator, Terms: "Oracle's own public_ip_ranges.json, published machine-readable so third parties can filter and route by it; no separate licence is stated and none is claimed here", parse: parseOracle},
+				{Name: "fastly", Origin: "https://api.fastly.com/public-ip-list", Basis: GrantOperator, Terms: "Fastly's own public IP list API, published so third parties can filter and route by it; no separate licence is stated and none is claimed here", parse: parseFastly},
+				{Name: "cloudflare4", Origin: "https://www.cloudflare.com/ips-v4", Basis: GrantOperator, Terms: "Cloudflare's own published IPv4 range list, served for third parties to allow traffic by; no separate licence is stated and none is claimed here", parse: parseCIDRs("cloudflare", "hosting", "cloudflare")},
+				{Name: "cloudflare6", Origin: "https://www.cloudflare.com/ips-v6", Basis: GrantOperator, Terms: "Cloudflare's own published IPv6 range list, served for third parties to allow traffic by; no separate licence is stated and none is claimed here", parse: parseCIDRs("cloudflare", "hosting", "cloudflare")},
+				{Name: "linode", Origin: "https://geoip.linode.com/", Basis: GrantOperator, Terms: "RFC 8805 geofeed self-published by the operator, which exists so third parties can read it; no separate licence is stated and none is claimed here", parse: parseLinode},
+				{Name: "digitalocean", Origin: "https://digitalocean.com/geo/google.csv", Basis: GrantOperator, Terms: "RFC 8805 geofeed self-published by the operator, which exists so third parties can read it; no separate licence is stated and none is claimed here", parse: parseDigitalOcean},
+				{Name: "tor", Origin: "https://check.torproject.org/torbulkexitlist", Basis: GrantLicence, Terms: "CC BY 3.0 US — the Tor Project publishes the bulk exit list for operator use under its site licence", parse: parseTor},
+				{Name: "reserved4", Origin: "https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry-1.csv", Basis: GrantRegistry, Terms: "IANA IPv4 Special-Purpose Address Registry, the registry of record", parse: parseSpecial},
+				{Name: "reserved6", Origin: "https://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry-1.csv", Basis: GrantRegistry, Terms: "IANA IPv6 Special-Purpose Address Registry, the registry of record", parse: parseSpecial},
+			},
+		},
+		{
+			Name:   "crawler",
+			Kind:   KindFetch,
+			What:   "User-agent patterns by which an automated client declares itself.",
+			Match:  MatchPattern,
+			MaxAge: monthly,
+			Sources: []Source{{
+				Name:   "patterns",
+				Origin: "https://raw.githubusercontent.com/monperrus/crawler-user-agents/master/crawler-user-agents.json",
+				Basis:  GrantLicence,
+				Terms:  "MIT",
+				parse:  parseCrawlers,
+			}},
+		},
+		{
+			Name:   "asn",
+			Kind:   KindFetch,
+			What:   "Autonomous system numbers and the registry each block was delegated to.",
+			Match:  MatchRange,
+			MaxAge: monthly,
+			Sources: []Source{{
+				Name:   "iana",
+				Origin: "https://www.iana.org/assignments/as-numbers/as-numbers-1.csv",
+				Basis:  GrantRegistry,
+				Terms:  "IANA Autonomous System Number Registry, the registry of record",
+				parse:  parseASN,
+			}},
+		},
+		{
+			Name:   "bin",
+			Kind:   KindLocal,
+			What:   "Issuer identification number prefixes and the card scheme each one belongs to.",
+			Match:  MatchDigits,
+			MaxAge: monthly,
+			Sources: []Source{{
+				Name:    "structure",
+				Origin:  "ISO/IEC 7812 major industry identifier and the schemes' own published prefix ranges",
+				Basis:   GrantOwn,
+				Terms:   "computed here from structural facts; no issuer database is licensed or held",
+				produce: produceBIN,
+			}},
+		},
+		{
+			Name:   "device",
+			Kind:   KindLocal,
+			What:   "Browser identities seen under enough separate organisations that no single one of them could have produced the observation.",
+			Match:  MatchExact,
+			MaxAge: daily,
+			Sources: []Source{{
+				Name:    "fleet",
+				Origin:  "aggregate over the shared event plane",
+				Basis:   GrantOwn,
+				Terms:   "our own aggregate, published only above the k-anonymity floor",
+				produce: produceDevice,
+			}},
+		},
+		{
+			Name:   "sanction",
+			Kind:   KindAttest,
+			What:   "Freshness of the designation lists the screening engine holds — which publisher, how many designations, how long ago.",
+			Match:  MatchExact,
+			MaxAge: daily,
+			Sources: []Source{
+				{Name: "OFAC", Origin: "luxfi/aml pkg/screen", Basis: GrantNone, Terms: "receipt only; the designations stay with the engine that screens"},
+				{Name: "UN", Origin: "luxfi/aml pkg/screen", Basis: GrantNone, Terms: "receipt only; the designations stay with the engine that screens"},
+				{Name: "EU", Origin: "luxfi/aml pkg/screen", Basis: GrantNone, Terms: "receipt only; the designations stay with the engine that screens"},
+				{Name: "OFSI", Origin: "luxfi/aml pkg/screen", Basis: GrantNone, Terms: "receipt only; the designations stay with the engine that screens"},
+			},
+		},
+		{
+			Name:   "jurisdiction",
+			Kind:   KindAttest,
+			What:   "Freshness of the higher-risk country listing the screening engine evaluates against.",
+			Match:  MatchExact,
+			MaxAge: monthly,
+			Sources: []Source{
+				{Name: "listing", Origin: "luxfi/aml pkg/reference", Basis: GrantNone, Terms: "receipt only; the listing stays with the engine that evaluates it"},
+			},
+		},
+		{
+			Name:    "pep",
+			Kind:    KindSeam,
+			What:    "Politically exposed persons and their close associates.",
+			Match:   MatchExact,
+			MaxAge:  weekly,
+			Refusal: "no politically-exposed-person listing is held: the comprehensive ones are sold under commercial terms and the open one is licensed for non-commercial use only. Screening against this set would be screening against nothing, so it refuses instead.",
+		},
+		{
+			Name:    "issuer",
+			Kind:    KindSeam,
+			What:    "Card issuer identity behind an issuer identification number — institution, country, product and funding type.",
+			Match:   MatchDigits,
+			MaxAge:  monthly,
+			Refusal: "no issuer identification database is held: the authoritative tables are licensed by the card schemes and the freely circulating copies state no provenance. The scheme a prefix belongs to is structural and is answered by the bin set; the institution behind it is not.",
+		},
+		{
+			Name:    "reputation",
+			Kind:    KindSeam,
+			What:    "Commercial network reputation — per-address and per-autonomous-system abuse scoring.",
+			Match:   MatchNet,
+			MaxAge:  daily,
+			Refusal: "no commercial network reputation feed is held. Deriving one from our own traffic needs an address-to-autonomous-system map, and every map we can reach carries redistribution terms we have not accepted. The net set answers what an operator publishes about its own estate; it does not score behaviour.",
+		},
+	}
+}
+
+// byName resolves a set by its address. A name the catalog does not carry is not
+// an empty set — it is a name this plane never published.
+func byName(name string) (Set, bool) {
+	for _, s := range Catalog() {
+		if s.Name == name {
+			return s, true
+		}
+	}
+	return Set{}, false
+}
+
+// source resolves one publisher within a set.
+func (s Set) source(name string) (Source, bool) {
+	for _, src := range s.Sources {
+		if src.Name == name {
+			return src, true
+		}
+	}
+	return Source{}, false
+}
+
+// emptyIsFact reports whether a set with zero entries is telling the truth.
+//
+// It is derived from Kind rather than declared per set, so the two cannot
+// disagree: a downloaded list is never legitimately empty, and a computed one
+// frequently is.
+func (s Set) emptyIsFact() bool { return s.Kind != KindFetch }

apps/reference/source.go

@@ -0,0 +1,477 @@
+package reference
+
+// source.go turns a publisher's bytes into entries. One parser per publisher and
+// no default: luxfi/aml pkg/screen fell back to one parser for four different
+// list formats on the grounds that they looked similar, and three of the four
+// then failed silently every night for months. A format this file does not know
+// is an error, never an empty list.
+//
+// Every parser here is TOTAL over its input in one direction only: it refuses
+// bytes it cannot read, and it skips individual rows it cannot read while
+// counting them, so a publisher who changes one column does not turn the whole
+// set into silence.
+
+import (
+	"encoding/csv"
+	"encoding/json"
+	"fmt"
+	"net/netip"
+	"strconv"
+	"strings"
+)
+
+// maxEntries bounds one source's contribution, so a publisher who starts
+// serving something enormous costs a refusal rather than the process.
+const maxEntries = 200_000
+
+// errEmpty is what every fetched parser returns for a source that yielded
+// nothing. It is an error rather than an empty slice because no publisher's list
+// of disposable domains, hosting ranges or crawler patterns is empty — zero
+// entries means the fetch or the parse is wrong, and the previous version must
+// stand.
+func errEmpty(name string) error {
+	return fmt.Errorf("reference: %s parsed to no entries, which no published list is", name)
+}
+
+// parseLines reads a newline-delimited list, ignoring blanks and # comments. The
+// value is the same for every member, because a bare list states membership and
+// nothing else.
+func parseLines(source string, value map[string]string) func([]byte) ([]Entry, error) {
+	return func(body []byte) ([]Entry, error) {
+		out := make([]Entry, 0, 1024)
+		for _, line := range strings.Split(string(body), "\n") {
+			line = strings.ToLower(strings.TrimSpace(line))
+			if line == "" || strings.HasPrefix(line, "#") {
+				continue
+			}
+			if len(out) >= maxEntries {
+				return nil, fmt.Errorf("reference: %s carries more than %d entries", source, maxEntries)
+			}
+			out = append(out, Entry{Key: line, Value: value})
+		}
+		if len(out) == 0 {
+			return nil, errEmpty(source)
+		}
+		return out, nil
+	}
+}
+
+// mailbox is the closed list of domains that are NOT disposable-inbox providers
+// and that no disposable-domain list may name.
+//
+// It exists because the disposable list is fetched from a public repository with
+// no pin, no signature and no digest to check against — the honest state of that
+// source — so anyone who can land a commit on it can change what every
+// organisation's decisions read. The size gate (swung) catches a list that
+// arrives at a fraction or a multiple of itself; it cannot catch ONE added row,
+// and one added row is the whole attack: "gmail.com is disposable" refuses a
+// large share of every tenant's legitimate signups at once.
+//
+// Bounded coverage, stated plainly: this is the blast radius of that one row, not
+// a proof the list is honest. A publisher naming one of these is wrong about
+// something we know, so the whole take is refused and the previous version stands
+// — a poisoned list must not land at all rather than land minus the row we
+// happened to recognise.
+var mailbox = map[string]bool{
+	"gmail.com": true, "googlemail.com": true,
+	"outlook.com": true, "hotmail.com": true, "live.com": true, "msn.com": true,
+	"yahoo.com": true, "ymail.com": true,
+	"icloud.com": true, "me.com": true, "mac.com": true,
+	"aol.com": true, "gmx.com": true, "gmx.net": true, "mail.com": true,
+	"proton.me": true, "protonmail.com": true, "pm.me": true,
+	"zoho.com": true, "fastmail.com": true, "yandex.ru": true, "qq.com": true,
+	"163.com": true, "126.com": true, "naver.com": true, "web.de": true,
+}
+
+// parseDisposable reads the throwaway-inbox list and refuses a take that names a
+// domain we know is a mailbox provider.
+func parseDisposable(body []byte) ([]Entry, error) {
+	entries, err := parseLines("disposable", map[string]string{"class": "disposable"})(body)
+	if err != nil {
+		return nil, err
+	}
+	for _, e := range entries {
+		if mailbox[e.Key] {
+			return nil, fmt.Errorf("reference: the disposable list names %q, which hands out mailboxes rather than throwaway inboxes; the whole take is refused rather than landing a list that is wrong about something we can check", e.Key)
+		}
+	}
+	return entries, nil
+}
+
+// prefix normalises one CIDR (or a bare address, which is its own /32 or /128)
+// into the canonical masked form entries are keyed by. A bare address is
+// accepted because two of the publishers here list addresses rather than blocks.
+func prefix(s string) (netip.Prefix, bool) {
+	s = strings.TrimSpace(s)
+	if s == "" {
+		return netip.Prefix{}, false
+	}
+	if p, err := netip.ParsePrefix(s); err == nil {
+		return p.Masked(), true
+	}
+	if a, err := netip.ParseAddr(s); err == nil {
+		return netip.PrefixFrom(a, a.BitLen()), true
+	}
+	return netip.Prefix{}, false
+}
+
+// block builds one network entry. class is what the address IS (hosting, tor,
+// reserved) and operator is who runs it — two separate facts, because "this is a
+// datacentre" and "this is Amazon" are different inputs to a rule.
+func block(p netip.Prefix, class, operator, region string) Entry {
+	v := map[string]string{"class": class}
+	if operator != "" {
+		v["operator"] = operator
+	}
+	if region != "" {
+		v["region"] = region
+	}
+	return Entry{Key: p.String(), Value: v}
+}
+
+// parseCIDRs reads a newline-delimited CIDR list.
+func parseCIDRs(source, class, operator string) func([]byte) ([]Entry, error) {
+	return func(body []byte) ([]Entry, error) {
+		out := make([]Entry, 0, 256)
+		for _, line := range strings.Split(string(body), "\n") {
+			line = strings.TrimSpace(line)
+			if line == "" || strings.HasPrefix(line, "#") {
+				continue
+			}
+			p, ok := prefix(line)
+			if !ok {
+				continue
+			}
+			out = append(out, block(p, class, operator, ""))
+		}
+		if len(out) == 0 {
+			return nil, errEmpty(source)
+		}
+		return out, nil
+	}
+}
+
+// parseTor reads the bulk exit list: one exit address per line. An exit address
+// is not hosting — it is an address whose traffic arrived through a network
+// designed to detach it from its origin, which is a different fact and gets its
+// own class.
+func parseTor(body []byte) ([]Entry, error) {
+	out := make([]Entry, 0, 2048)
+	for _, line := range strings.Split(string(body), "\n") {
+		p, ok := prefix(line)
+		if !ok {
+			continue
+		}
+		out = append(out, block(p, "tor", "tor", ""))
+	}
+	if len(out) == 0 {
+		return nil, errEmpty("tor")
+	}
+	return out, nil
+}
+
+func parseAWS(body []byte) ([]Entry, error) {
+	var doc struct {
+		Prefixes []struct {
+			IP      string `json:"ip_prefix"`
+			Region  string `json:"region"`
+			Service string `json:"service"`
+		} `json:"prefixes"`
+		V6 []struct {
+			IP      string `json:"ipv6_prefix"`
+			Region  string `json:"region"`
+			Service string `json:"service"`
+		} `json:"ipv6_prefixes"`
+	}
+	if err := json.Unmarshal(body, &doc); err != nil {
+		return nil, fmt.Errorf("reference: aws ranges: %w", err)
+	}
+	out := make([]Entry, 0, len(doc.Prefixes)+len(doc.V6))
+	// The AMAZON service row is the union of every other row, so keeping only it
+	// gives one entry per block instead of four saying the same thing.
+	for _, p := range doc.Prefixes {
+		if p.Service != "AMAZON" {
+			continue
+		}
+		if q, ok := prefix(p.IP); ok {
+			out = append(out, block(q, "hosting", "aws", p.Region))
+		}
+	}
+	for _, p := range doc.V6 {
+		if p.Service != "AMAZON" {
+			continue
+		}
+		if q, ok := prefix(p.IP); ok {
+			out = append(out, block(q, "hosting", "aws", p.Region))
+		}
+	}
+	if len(out) == 0 {
+		return nil, errEmpty("aws")
+	}
+	return out, nil
+}
+
+func parseGCP(body []byte) ([]Entry, error) {
+	var doc struct {
+		Prefixes []struct {
+			V4    string `json:"ipv4Prefix"`
+			V6    string `json:"ipv6Prefix"`
+			Scope string `json:"scope"`
+		} `json:"prefixes"`
+	}
+	if err := json.Unmarshal(body, &doc); err != nil {
+		return nil, fmt.Errorf("reference: gcp ranges: %w", err)
+	}
+	out := make([]Entry, 0, len(doc.Prefixes))
+	for _, p := range doc.Prefixes {
+		for _, raw := range []string{p.V4, p.V6} {
+			if q, ok := prefix(raw); ok {
+				out = append(out, block(q, "hosting", "gcp", p.Scope))
+			}
+		}
+	}
+	if len(out) == 0 {
+		return nil, errEmpty("gcp")
+	}
+	return out, nil
+}
+
+func parseOracle(body []byte) ([]Entry, error) {
+	var doc struct {
+		Regions []struct {
+			Region string `json:"region"`
+			CIDRs  []struct {
+				CIDR string `json:"cidr"`
+			} `json:"cidrs"`
+		} `json:"regions"`
+	}
+	if err := json.Unmarshal(body, &doc); err != nil {
+		return nil, fmt.Errorf("reference: oracle ranges: %w", err)
+	}
+	var out []Entry
+	for _, r := range doc.Regions {
+		for _, c := range r.CIDRs {
+			if q, ok := prefix(c.CIDR); ok {
+				out = append(out, block(q, "hosting", "oracle", r.Region))
+			}
+		}
+	}
+	if len(out) == 0 {
+		return nil, errEmpty("oracle")
+	}
+	return out, nil
+}
+
+func parseFastly(body []byte) ([]Entry, error) {
+	var doc struct {
+		V4 []string `json:"addresses"`
+		V6 []string `json:"ipv6_addresses"`
+	}
+	if err := json.Unmarshal(body, &doc); err != nil {
+		return nil, fmt.Errorf("reference: fastly ranges: %w", err)
+	}
+	out := make([]Entry, 0, len(doc.V4)+len(doc.V6))
+	for _, raw := range append(append([]string{}, doc.V4...), doc.V6...) {
+		if q, ok := prefix(raw); ok {
+			out = append(out, block(q, "hosting", "fastly", ""))
+		}
+	}
+	if len(out) == 0 {
+		return nil, errEmpty("fastly")
+	}
+	return out, nil
+}
+
+// geofeed reads the RFC 8805 self-published geofeed both Linode and
+// DigitalOcean serve: prefix, country, region, city, postal code. Only the first
+// three columns are kept — a city is not a risk signal and a postal code is
+// closer to personal data than to one.
+func geofeed(source, operator string) func([]byte) ([]Entry, error) {
+	return func(body []byte) ([]Entry, error) {
+		var out []Entry
+		for _, line := range strings.Split(string(body), "\n") {
+			line = strings.TrimSpace(line)
+			if line == "" || strings.HasPrefix(line, "#") {
+				continue
+			}
+			cols := strings.Split(line, ",")
+			p, ok := prefix(cols[0])
+			if !ok {
+				continue
+			}
+			region := ""
+			if len(cols) > 1 {
+				region = strings.TrimSpace(cols[1])
+			}
+			out = append(out, block(p, "hosting", operator, region))
+		}
+		if len(out) == 0 {
+			return nil, errEmpty(source)
+		}
+		return out, nil
+	}
+}
+
+var (
+	parseLinode       = geofeed("linode", "linode")
+	parseDigitalOcean = geofeed("digitalocean", "digitalocean")
+)
+
+// parseSpecial reads an IANA special-purpose address registry. These are the
+// blocks no public host may legitimately be reached at, so an inbound
+// connection claiming one is a claim about the world that is not true.
+//
+// A footnote marker rides on some blocks ("192.0.0.0/29[2]") and one row can
+// carry several blocks in one field, so the cell is split and each part is
+// stripped before parsing.
+func parseSpecial(body []byte) ([]Entry, error) {
+	r := csv.NewReader(strings.NewReader(string(body)))
+	r.FieldsPerRecord = -1
+	r.LazyQuotes = true
+	rows, err := r.ReadAll()
+	if err != nil {
+		return nil, fmt.Errorf("reference: iana special registry: %w", err)
+	}
+	var out []Entry
+	for i, row := range rows {
+		if i == 0 || len(row) < 2 {
+			continue
+		}
+		name := strings.Trim(strings.TrimSpace(row[1]), `"`)
+		for _, part := range strings.Split(row[0], ",") {
+			if cut := strings.IndexByte(part, '['); cut >= 0 {
+				part = part[:cut]
+			}
+			p, ok := prefix(part)
+			if !ok {
+				continue
+			}
+			e := block(p, "reserved", "iana", "")
+			e.Value["name"] = name
+			out = append(out, e)
+		}
+	}
+	if len(out) == 0 {
+		return nil, errEmpty("iana special registry")
+	}
+	return out, nil
+}
+
+// parseASN reads the IANA autonomous system number registry. Rows address a
+// RANGE of numbers ("1877-1901"), so entries are keyed by the range and matched
+// numerically — see MatchRange.
+//
+// What this set answers is narrow and worth being precise about: whether a
+// number has been delegated at all, and to which regional registry. It does NOT
+// answer whether the operator behind it is trustworthy; that is the reputation
+// seam.
+func parseASN(body []byte) ([]Entry, error) {
+	r := csv.NewReader(strings.NewReader(string(body)))
+	r.FieldsPerRecord = -1
+	r.LazyQuotes = true
+	rows, err := r.ReadAll()
+	if err != nil {
+		return nil, fmt.Errorf("reference: iana as-numbers: %w", err)
+	}
+	var out []Entry
+	for i, row := range rows {
+		if i == 0 || len(row) < 2 {
+			continue
+		}
+		lo, hi, ok := span(row[0])
+		if !ok {
+			continue
+		}
+		desc := strings.TrimSpace(row[1])
+		out = append(out, Entry{
+			Key: fmt.Sprintf("%d-%d", lo, hi),
+			Value: map[string]string{
+				"registry": registry(desc),
+				"status":   status(desc),
+			},
+		})
+	}
+	if len(out) == 0 {
+		return nil, errEmpty("iana as-numbers")
+	}
+	return out, nil
+}
+
+// span reads "1877-1901" or "0" into a closed numeric interval.
+func span(s string) (lo, hi uint64, ok bool) {
+	s = strings.TrimSpace(s)
+	a, b, dash := strings.Cut(s, "-")
+	lo, err := strconv.ParseUint(strings.TrimSpace(a), 10, 32)
+	if err != nil {
+		return 0, 0, false
+	}
+	if !dash {
+		return lo, lo, true
+	}
+	hi, err = strconv.ParseUint(strings.TrimSpace(b), 10, 32)
+	if err != nil {
+		return 0, 0, false
+	}
+	if hi < lo {
+		return 0, 0, false
+	}
+	return lo, hi, true
+}
+
+// registry names the regional registry a block was delegated to, from the
+// registry's own description ("Assigned by ARIN").
+func registry(desc string) string {
+	for _, rir := range []string{"ARIN", "RIPE NCC", "APNIC", "LACNIC", "AFRINIC"} {
+		if strings.Contains(strings.ToUpper(desc), rir) {
+			return strings.ToLower(strings.ReplaceAll(rir, " NCC", ""))
+		}
+	}
+	return ""
+}
+
+// status is whether the block is delegated, reserved or held back. An
+// autonomous system number that IANA has not delegated cannot legitimately
+// appear in routing, so a claim naming one is false on its face.
+func status(desc string) string {
+	d := strings.ToLower(desc)
+	switch {
+	case strings.Contains(d, "assigned by"):
+		return "delegated"
+	case strings.Contains(d, "reserved"):
+		return "reserved"
+	case strings.Contains(d, "unallocated"), strings.Contains(d, "available"):
+		return "unallocated"
+	default:
+		return "other"
+	}
+}
+
+// parseCrawlers reads the crawler-user-agents catalogue. Its `pattern` field is
+// a regular expression by design, so it is carried through as one and compiled
+// when the snapshot is built — rewriting it as a literal would drop the
+// alternations and anchors the publisher put there on purpose.
+func parseCrawlers(body []byte) ([]Entry, error) {
+	var doc []struct {
+		Pattern string `json:"pattern"`
+		URL     string `json:"url"`
+	}
+	if err := json.Unmarshal(body, &doc); err != nil {
+		return nil, fmt.Errorf("reference: crawler patterns: %w", err)
+	}
+	out := make([]Entry, 0, len(doc))
+	for _, c := range doc {
+		p := strings.TrimSpace(c.Pattern)
+		if p == "" {
+			continue
+		}
+		e := Entry{Key: p, Value: map[string]string{"class": "crawler"}}
+		if c.URL != "" {
+			e.Value["about"] = c.URL
+		}
+		out = append(out, e)
+	}
+	if len(out) == 0 {
+		return nil, errEmpty("crawler patterns")
+	}
+	return out, nil
+}

apps/reference/store.go

@@ -0,0 +1,543 @@
+package reference
+
+// store.go is the durable baseline: two tables in the shared warehouse, and the
+// ingest that fills them.
+//
+// THE TABLES HAVE NO TENANT COLUMN, AND THAT IS THE SECURITY ARGUMENT. Read the
+// DDL below as the proof rather than as schema: there is nowhere in either shape
+// for an organisation to go. A tenant's own allow and deny entries live
+// somewhere else entirely — one SQLite file per organisation, opened through
+// cloud.OrgNamespace (override.go) — so the two planes are not two rows in one
+// table separated by a predicate, they are two different stores. A bug that
+// leaked one into the other would have to open another organisation's file,
+// which is the same isolation every other per-entity store in this binary
+// already has. There is no `scope` column to get wrong.
+//
+// WHAT MAY GO IN HERE. Data someone else published under a licence we hold, and
+// aggregates over fleet traffic that pass the k-anonymity floor in derive.go.
+// Nothing else. reference_test.go holds both halves of that to a test.
+//
+// IDEMPOTENT BY CONTENT ADDRESS. A version IS the digest of the entries, so
+// fetching the same set twice produces the same version, writes the same primary
+// keys, and a ReplacingMergeTree collapses them. Re-running an ingest is not
+// merely safe, it is a no-op that says so.
+//
+// RESUMABLE BY CURSOR, CORRECT BY MERGE. Entries land in sorted chunks and the
+// manifest's `landed` counter advances after each one, so a run that dies at
+// chunk k resumes at chunk k. The counter is an OPTIMISATION: because the
+// version is content-addressed, a resumed run re-derives the identical sorted
+// list, so re-writing a chunk that already landed produces the same rows the
+// merge already deduplicates. The cursor saves the work; the primary key
+// guarantees the answer.
+
+import (
+	"context"
+	"fmt"
+	"sort"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/hanzoai/cloud/apps/datastore"
+)
+
+const (
+	// sourceTable holds one row per (set, source, version): what was taken, from
+	// where, under what terms, when, and how much of it landed.
+	sourceTable = "hanzo.reference_source"
+	// entryTable holds the members of every version.
+	entryTable = "hanzo.reference_entry"
+	// chunk is how many entries land per statement. Big enough that a 65,000-entry
+	// list is a handful of round trips, small enough that a failure costs one.
+	chunk = 5000
+	// maxRead bounds a version read, so a source that grew past its own cap
+	// cannot make the snapshot build unbounded.
+	maxRead = maxEntries
+	// ddlTimeout bounds the idempotent table bootstrap.
+	ddlTimeout = 10 * time.Second
+)
+
+// Statuses a version passes through. `ready` is the only one a snapshot builds
+// from: a half-landed version must never answer, because a set that is missing
+// its tail answers "not listed" for everything in it.
+// `refused` is the third: a take that produced no answerable version, recorded so
+// the failure is durable and the source's row shows why. Because only `ready` is
+// read back (currentStatement), recording a refusal leaves the previous ready
+// version current — which is the correct outcome for a publisher that stopped
+// answering, and the one that ages out visibly instead of silently emptying a set.
+const (
+	statusIngest  = "ingesting"
+	statusReady   = "ready"
+	statusRefused = "refused"
+)
+
+const createDatabase = `CREATE DATABASE IF NOT EXISTS hanzo`
+
+// createSource declares the version manifest. It carries NO TTL on purpose.
+// Provenance is a record: a decision names the version it consulted, and that
+// name has to keep resolving to a publisher, a licence and a date long after the
+// bulk membership of that version has been pruned.
+const createSource = `CREATE TABLE IF NOT EXISTS hanzo.reference_source (
+  set      LowCardinality(String),
+  source   LowCardinality(String),
+  version  String,
+  origin   String,
+  terms    String,
+  as_of    DateTime,
+  fetched  DateTime,
+  keys     UInt64,
+  landed   UInt64,
+  status   LowCardinality(String),
+  refusal  String,
+  at       DateTime
+) ENGINE = ReplacingMergeTree(at)
+ORDER BY (set, source, version)`
+
+// createEntry declares the membership. Also no TTL: a table TTL is a clock
+// nobody can hold, and a version's rows must live exactly as long as that
+// version is current or one behind it. Superseded versions are pruned by the
+// ingest that supersedes them, which is the only moment anything knows.
+const createEntry = `CREATE TABLE IF NOT EXISTS hanzo.reference_entry (
+  set     LowCardinality(String),
+  source  LowCardinality(String),
+  version String,
+  key     String,
+  value   Map(String, String),
+  score   Float64,
+  orgs    UInt32,
+  n       UInt64,
+  at      DateTime
+) ENGINE = ReplacingMergeTree(at)
+ORDER BY (set, source, version, key)`
+
+// The warehouse as VALUES, on the same terms as apps/analytics/warehouse.go:
+// production is always the ONE datastore client, and a test substitutes them to
+// drive the durable half — the version manifest, the resume cursor, the prune —
+// without standing up a store. They are the only door this package reaches the
+// warehouse through, so there is one place to substitute and no second path that
+// could stay real while these are faked.
+var (
+	storeReady = datastore.Ready
+	storeQuery = datastore.Query
+	storeExec  = datastore.Exec
+)
+
+// tableMu guards the lazy bootstrap. Only SUCCESS is latched: a failed DDL
+// leaves the flag false so the next call retries. sync.Once is wrong here — it
+// would cache the failure forever, and the warehouse connects asynchronously so
+// the first attempt usually happens before it is up.
+var (
+	tableMu    sync.Mutex
+	tableReady bool
+)
+
+// ensure creates the database and both tables exactly once successfully.
+func ensure(ctx context.Context) error {
+	tableMu.Lock()
+	defer tableMu.Unlock()
+	if tableReady {
+		return nil
+	}
+	if !storeReady() {
+		return fmt.Errorf("reference: the warehouse is not connected")
+	}
+	for _, stmt := range []string{createDatabase, createSource, createEntry} {
+		if err := storeExec(ctx, stmt); err != nil {
+			return fmt.Errorf("reference: bootstrap: %w", err)
+		}
+	}
+	tableReady = true
+	return nil
+}
+
+// version is one taken version of one source: the manifest row.
+type version struct {
+	Set     string
+	Source  string
+	Version string
+	Origin  string
+	Terms   string
+	AsOf    time.Time
+	Fetched time.Time
+	Keys    uint64
+	Landed  uint64
+	Status  string
+	Refusal string
+}
+
+// ready reports whether this version may be answered from.
+func (v version) ready() bool { return v.Status == statusReady && v.Landed >= v.Keys }
+
+// current reads the newest ready version of every source, one row each.
+//
+// LIMIT 1 BY is what keeps this bounded as history accumulates: without it the
+// read grows with every version ever taken, which is the read amplification a
+// single-pod warehouse cannot afford.
+const currentStatement = `SELECT set, source, version, origin, terms, as_of, fetched, keys, landed, status, refusal
+FROM ` + sourceTable + ` FINAL
+WHERE status = ?
+ORDER BY set, source, fetched DESC
+LIMIT 1 BY set, source`
+
+func current(ctx context.Context) ([]version, error) {
+	if err := ensure(ctx); err != nil {
+		return nil, err
+	}
+	rows, err := storeQuery(ctx, currentStatement, statusReady)
+	if err != nil {
+		return nil, fmt.Errorf("reference: read versions: %w", err)
+	}
+	out := make([]version, 0, len(rows))
+	for _, r := range rows {
+		out = append(out, asVersion(r))
+	}
+	return out, nil
+}
+
+// taken reads one specific version's manifest row, which is how a resumed ingest
+// learns where it stopped.
+const takenStatement = `SELECT set, source, version, origin, terms, as_of, fetched, keys, landed, status, refusal
+FROM ` + sourceTable + ` FINAL
+WHERE set = ? AND source = ? AND version = ?`
+
+func taken(ctx context.Context, set, source, ver string) (version, bool, error) {
+	rows, err := storeQuery(ctx, takenStatement, set, source, ver)
+	if err != nil {
+		return version{}, false, fmt.Errorf("reference: read version: %w", err)
+	}
+	if len(rows) == 0 {
+		return version{}, false, nil
+	}
+	return asVersion(rows[0]), true, nil
+}
+
+func asVersion(r map[string]any) version {
+	return version{
+		Set:     text(r["set"]),
+		Source:  text(r["source"]),
+		Version: text(r["version"]),
+		Origin:  text(r["origin"]),
+		Terms:   text(r["terms"]),
+		AsOf:    when(r["as_of"]),
+		Fetched: when(r["fetched"]),
+		Keys:    count64(r["keys"]),
+		Landed:  count64(r["landed"]),
+		Status:  text(r["status"]),
+		Refusal: text(r["refusal"]),
+	}
+}
+
+const markStatement = `INSERT INTO ` + sourceTable +
+	` (set, source, version, origin, terms, as_of, fetched, keys, landed, status, refusal, at)
+VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`
+
+// mark writes the manifest row. ReplacingMergeTree(at) on (set, source,
+// version) means the newest write of one version wins, so a status change is an
+// insert rather than an update — the plane has one way to write a row.
+func mark(ctx context.Context, v version, at time.Time) error {
+	return storeExec(ctx, markStatement,
+		v.Set, v.Source, v.Version, v.Origin, v.Terms,
+		v.AsOf.UTC(), v.Fetched.UTC(), v.Keys, v.Landed, v.Status, v.Refusal, at.UTC())
+}
+
+// landed is the outcome of an ingest: which version is current now, and whether
+// this run actually changed anything.
+type landing struct {
+	Version   string
+	Keys      int
+	Wrote     int
+	Unchanged bool
+	Resumed   bool
+}
+
+// ingest lands one source's entries as a version, idempotently and resumably.
+//
+// The order of operations is the design: content-address FIRST, so a re-run of
+// an unchanged source is answered before a single row is written; then mark the
+// version `ingesting` so a crash leaves a visible half-version rather than a
+// silent gap; then land the chunks, advancing the cursor; then mark `ready`,
+// which is the moment the version becomes answerable; then prune what it
+// superseded.
+func ingest(ctx context.Context, set string, src Source, entries []Entry, asOf, now time.Time) (landing, error) {
+	if err := ensure(ctx); err != nil {
+		return landing{}, err
+	}
+	sorted := order(entries)
+	ver := digest(sorted)
+
+	prior, found, err := taken(ctx, set, src.Name, ver)
+	if err != nil {
+		return landing{}, err
+	}
+	if found && prior.ready() {
+		// The same set, already landed. Re-stamp the manifest so a refresh that
+		// changed nothing is distinguishable from a refresh that did not run — the
+		// distinction luxfi/aml pkg/screen's Fitness.Digest exists to make — and
+		// write not one entry row.
+		prior.Fetched = now
+		prior.AsOf = asOf
+		if err := mark(ctx, prior, now); err != nil {
+			return landing{}, err
+		}
+		return landing{Version: ver, Keys: len(sorted), Unchanged: true}, nil
+	}
+
+	from := 0
+	resumed := false
+	if found && prior.Landed > 0 && prior.Landed < uint64(len(sorted)) {
+		from = int(prior.Landed)
+		resumed = true
+	}
+
+	v := version{
+		Set: set, Source: src.Name, Version: ver,
+		Origin: src.Origin, Terms: src.Terms,
+		AsOf: asOf, Fetched: now,
+		Keys: uint64(len(sorted)), Landed: uint64(from), Status: statusIngest,
+	}
+	if err := mark(ctx, v, now); err != nil {
+		return landing{}, err
+	}
+
+	wrote := 0
+	for i := from; i < len(sorted); i += chunk {
+		end := min(i+chunk, len(sorted))
+		if err := land(ctx, set, src.Name, ver, sorted[i:end], now); err != nil {
+			return landing{}, err
+		}
+		wrote += end - i
+		v.Landed = uint64(end)
+		if err := mark(ctx, v, now); err != nil {
+			return landing{}, err
+		}
+	}
+
+	v.Status = statusReady
+	v.Landed = uint64(len(sorted))
+	if err := mark(ctx, v, now); err != nil {
+		return landing{}, err
+	}
+	return landing{Version: ver, Keys: len(sorted), Wrote: wrote, Resumed: resumed}, nil
+}
+
+// insert builds one chunk's statement and its bound arguments. PURE, so the one
+// place a value could become part of a statement instead of a parameter is
+// testable without a warehouse: the only interpolation is the repeated
+// placeholder tuple, generated from the chunk length, and it carries no input.
+func insert(set, source, ver string, entries []Entry, at time.Time) (string, []any) {
+	if len(entries) == 0 {
+		return "", nil
+	}
+	const cols = "(set, source, version, key, value, score, orgs, n, at)"
+	tuples := make([]string, 0, len(entries))
+	args := make([]any, 0, len(entries)*9)
+	for _, e := range entries {
+		tuples = append(tuples, "(?, ?, ?, ?, ?, ?, ?, ?, ?)")
+		args = append(args, set, source, ver, e.Key, e.Value, e.Score, e.Orgs, e.N, at.UTC())
+	}
+	return "INSERT INTO " + entryTable + " " + cols + " VALUES " + strings.Join(tuples, ", "), args
+}
+
+// land writes one chunk.
+func land(ctx context.Context, set, source, ver string, entries []Entry, at time.Time) error {
+	stmt, args := insert(set, source, ver, entries, at)
+	if stmt == "" {
+		return nil
+	}
+	if err := storeExec(ctx, stmt, args...); err != nil {
+		return fmt.Errorf("reference: land %s/%s: %w", set, source, err)
+	}
+	return nil
+}
+
+const readStatement = `SELECT key, value, score, orgs, n
+FROM ` + entryTable + ` FINAL
+WHERE set = ? AND source = ? AND version = ?
+ORDER BY key
+LIMIT ` + maxReadLiteral
+
+// maxReadLiteral is maxRead as the literal the statement carries. A LIMIT cannot
+// bind, so it is spelled once here and nothing a caller sends reaches it.
+const maxReadLiteral = "200000"
+
+// read materialises one version's entries.
+func read(ctx context.Context, set, source, ver string) ([]Entry, error) {
+	rows, err := storeQuery(ctx, readStatement, set, source, ver)
+	if err != nil {
+		return nil, fmt.Errorf("reference: read %s/%s: %w", set, source, err)
+	}
+	out := make([]Entry, 0, len(rows))
+	for _, r := range rows {
+		out = append(out, Entry{
+			Key:   text(r["key"]),
+			Value: pairs(r["value"]),
+			Score: number(r["score"]),
+			Orgs:  count32(r["orgs"]),
+			N:     count64(r["n"]),
+		})
+	}
+	return out, nil
+}
+
+const pruneStatement = `ALTER TABLE ` + entryTable +
+	` DELETE WHERE set = ? AND source = ? AND version != ? AND version != ?`
+
+// prune drops the membership of every version but the two named.
+//
+// Non-fatal by construction. A failed prune leaves history, which costs storage
+// and nothing else; treating it as an ingest failure would turn a housekeeping
+// problem into a freshness one.
+func prune(ctx context.Context, set, source, keep, alsoKeep string) error {
+	return storeExec(ctx, pruneStatement, set, source, keep, alsoKeep)
+}
+
+// dropper is prune as a value, so what a take supersedes can be decided and
+// tested without a warehouse to delete from.
+type dropper func(ctx context.Context, set, source, keep, alsoKeep string) error
+
+// sweepOld drops the membership of every version of every source but the CURRENT
+// one and the one it REPLACED.
+//
+// The previous version is kept deliberately, and `was` is why this function
+// exists rather than a loop at the call site: the current snapshot knows what is
+// current now and cannot know what was current a moment ago, so the pair has to
+// be carried in. A call site that passed the current version for both spared
+// exactly one version — the statement's two placeholders bound to the same
+// string — and quietly deleted the rows behind every citation taken in the
+// window before a refresh, which is the audit answer this whole plane is sold on.
+//
+// A source taken for the first time has no previous version; `was` carries no
+// entry for it, `alsoKeep` is the empty string, and a version that is not the
+// empty string spares nothing extra — which is correct, there is nothing extra.
+func sweepOld(ctx context.Context, drop dropper, set string, was map[string]held, now []version) error {
+	var first error
+	for _, v := range now {
+		if err := drop(ctx, set, v.Source, v.Version, was[v.Source].Version); err != nil && first == nil {
+			first = err
+		}
+	}
+	return first
+}
+
+// held is what this plane holds for one source at one instant: the version and
+// how many members it carries. It is captured BEFORE a take so the take can be
+// compared with what it replaces — the previous version is what prune spares
+// (sweepOld) and the previous size is what the swing gate measures against
+// (swung).
+type held struct {
+	Version string
+	Keys    uint64
+}
+
+// holding reads what a snapshot currently holds, per source.
+func holding(s *snap) map[string]held {
+	out := map[string]held{}
+	if s == nil {
+		return out
+	}
+	for _, v := range s.took {
+		out[v.Source] = held{Version: v.Version, Keys: v.Keys}
+	}
+	return out
+}
+
+// swing is how far a source's size may move in ONE take before the take is
+// refused and the previous version is left standing.
+//
+// The empty case was already an error and the truncation case already an error,
+// and between them sat the dangerous one: a publisher serving a valid, parseable
+// list at a tenth or ten times its previous size. Both directions are refused
+// because both are silent. A list that shrank answers "not listed" for everything
+// it lost and reads exactly like a clean world; a list that exploded puts members
+// nobody vetted into the baseline every organisation's decisions read.
+//
+// Four rather than two: published lists do move, and a bound that fires on
+// ordinary growth is a bound an operator learns to force through.
+const swing = 4
+
+// swung reports whether a take moved a source's size past [swing], in either
+// direction. A source taken for the first time has nothing to compare against and
+// is never refused.
+func swung(was, now uint64) bool {
+	if was == 0 {
+		return false
+	}
+	return now*swing < was || now > was*swing
+}
+
+// order sorts entries by key. It is the canonical order the digest is taken over
+// and the order chunks land in, so a resumed run and a fresh run agree on which
+// entry is the nth.
+func order(entries []Entry) []Entry {
+	out := make([]Entry, len(entries))
+	copy(out, entries)
+	sort.Slice(out, func(i, j int) bool { return out[i].Key < out[j].Key })
+	return out
+}
+
+// keys returns a value map's keys in sorted order, so the digest of an entry
+// does not depend on Go's map iteration.
+func keys(m map[string]string) []string {
+	out := make([]string, 0, len(m))
+	for k := range m {
+		out = append(out, k)
+	}
+	sort.Strings(out)
+	return out
+}
+
+// text, when, number and pairs read a warehouse column into the type this
+// package wants, tolerating whichever concrete type the driver chose.
+func text(v any) string {
+	switch s := v.(type) {
+	case string:
+		return s
+	case *string:
+		if s == nil {
+			return ""
+		}
+		return *s
+	default:
+		return ""
+	}
+}
+
+func when(v any) time.Time {
+	switch t := v.(type) {
+	case time.Time:
+		return t.UTC()
+	case *time.Time:
+		if t == nil {
+			return time.Time{}
+		}
+		return t.UTC()
+	default:
+		return time.Time{}
+	}
+}
+
+func number(v any) float64 {
+	switch f := v.(type) {
+	case float64:
+		return f
+	case float32:
+		return float64(f)
+	default:
+		return float64(count64(v))
+	}
+}
+
+func pairs(v any) map[string]string {
+	switch m := v.(type) {
+	case map[string]string:
+		return m
+	case map[string]any:
+		out := make(map[string]string, len(m))
+		for k, val := range m {
+			out[k] = text(val)
+		}
+		return out
+	default:
+		return map[string]string{}
+	}
+}

apps/reference/tenant_test.go

@@ -0,0 +1,707 @@
+package reference
+
+// tenant_test.go exercises the whole surface over real HTTP, with real per-org
+// stores on disk, because the two properties it holds are properties of the
+// WIRE and not of a function:
+//
+//   - one organisation's overrides can never reach another's, and there is no
+//     request shape that names another organisation at all;
+//   - an organisation's own say beats the shared baseline, and clearing it
+//     restores the baseline's answer.
+
+import (
+	"bytes"
+	"encoding/json"
+	"errors"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"runtime"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/hanzoai/cek"
+	"github.com/hanzoai/cloud"
+	"github.com/hanzoai/cloud/apps/principal"
+	"github.com/hanzoai/namespace"
+	luxlog "github.com/luxfi/log"
+	fiber "github.com/zap-proto/fiber/v3"
+	"github.com/zap-proto/zip"
+)
+
+const testTimeout = 30 * time.Second
+
+// TestMain seeds a cek master key so the encrypted-at-rest per-org store opens
+// on an encryption-capable build.
+func TestMain(m *testing.M) {
+	if _, err := cek.SetDevMaster(); err != nil {
+		panic(err)
+	}
+	os.Exit(m.Run())
+}
+
+// mount brings the plane up on a bare app with a temp data dir and no
+// warehouse — every test here is about tenancy, precedence and bounds, and none
+// of them needs one.
+//
+// NO WAREHOUSE IS LOAD-BEARING, not incidental: with one, the first hydrate
+// succeeds and the mount sweeps, which would send this suite to eleven
+// publishers over the real network. The check is here rather than in a comment
+// so combining the two fails loudly instead of quietly dialling out.
+func mount(t *testing.T) *zip.App {
+	t.Helper()
+	if storeReady() {
+		t.Fatal("mount() is the no-warehouse harness; a test that wants one builds its own service (see plant)")
+	}
+	app := zip.New(zip.Config{Logger: luxlog.New("test")})
+	if err := Mount(app, cloud.Deps{Logger: luxlog.New("test"), DataDir: t.TempDir(), Brand: "hanzo"}); err != nil {
+		t.Fatalf("Mount: %v", err)
+	}
+	t.Cleanup(func() { _ = Shutdown() })
+	return app
+}
+
+// seed installs a baseline snapshot directly, as a successful refresh would.
+func seed(t *testing.T, name string, entries []Entry) {
+	t.Helper()
+	set, ok := byName(name)
+	if !ok {
+		t.Fatalf("no set %q", name)
+	}
+	mounted.State.plane.put(name, loaded(set, entries))
+}
+
+func call(t *testing.T, app *zip.App, method, path, org string, body any) (int, map[string]any) {
+	t.Helper()
+	t.Helper()
+	var r io.Reader
+	if body != nil {
+		b, _ := json.Marshal(body)
+		r = bytes.NewReader(b)
+	}
+	rq := httptest.NewRequest(method, path, r)
+	if body != nil {
+		rq.Header.Set("Content-Type", "application/json")
+	}
+	if org != "" {
+		rq.Header.Set("X-Org-Id", org)
+		rq.Header.Set("X-User-Id", "u_"+org) // a validated principal (principal.Org gate)
+	}
+	resp, err := app.Fiber().Test(rq, fiber.TestConfig{Timeout: testTimeout, FailOnTimeout: true})
+	if err != nil {
+		t.Fatalf("%s %s: %v", method, path, err)
+	}
+	defer func() { _ = resp.Body.Close() }()
+	raw, _ := io.ReadAll(resp.Body)
+	var out map[string]any
+	_ = json.Unmarshal(raw, &out)
+	return resp.StatusCode, out
+}
+
+// answersFor pulls the answer for one key out of a resolve response.
+func answersFor(t *testing.T, body map[string]any, key string) map[string]any {
+	t.Helper()
+	list, _ := body["answers"].([]any)
+	for _, a := range list {
+		m, _ := a.(map[string]any)
+		if m["key"] == key {
+			return m
+		}
+	}
+	t.Fatalf("no answer for %q in %v", key, body)
+	return nil
+}
+
+// ── isolation ────────────────────────────────────────────────────────────────
+
+// TestOverridesNeverLeaveTheirOrg is the invariant. Two organisations write
+// opposite overrides on the same key; neither sees the other's entry in any
+// read, and each one's resolve gets its own verdict.
+func TestOverridesNeverLeaveTheirOrg(t *testing.T) {
+	app := mount(t)
+	seed(t, "domain", []Entry{{Key: "tempbox.example", Value: map[string]string{"class": "disposable"}}})
+
+	if code, _ := call(t, app, http.MethodPut, "/v1/reference/domain", "acme",
+		map[string]any{"entries": []any{map[string]any{"key": "tempbox.example", "verdict": Allow, "note": "acme trusts it"}}}); code != 200 {
+		t.Fatalf("acme write: %d", code)
+	}
+	if code, _ := call(t, app, http.MethodPut, "/v1/reference/domain", "globex",
+		map[string]any{"entries": []any{map[string]any{"key": "partner.globex", "verdict": Deny, "note": "globex denies it"}}}); code != 200 {
+		t.Fatalf("globex write: %d", code)
+	}
+
+	// Each org's listing contains only its own entry.
+	for org, want := range map[string]string{"acme": "tempbox.example", "globex": "partner.globex"} {
+		code, body := call(t, app, http.MethodGet, "/v1/reference/domain", org, nil)
+		if code != 200 {
+			t.Fatalf("%s read: %d", org, code)
+		}
+		list, _ := body["overrides"].([]any)
+		if len(list) != 1 {
+			t.Fatalf("%s sees %d overrides, want exactly its own: %v", org, len(list), list)
+		}
+		got, _ := list[0].(map[string]any)
+		if got["key"] != want {
+			t.Errorf("%s sees %v, want %q — another org's entry is visible", org, got["key"], want)
+		}
+		if note, _ := got["note"].(string); org == "acme" && note != "acme trusts it" {
+			t.Errorf("acme's note reads %q", note)
+		}
+	}
+
+	// And the resolve answers differ by caller, on the same key, at the same
+	// instant, against the same baseline.
+	_, acme := call(t, app, http.MethodPost, "/v1/reference/resolve", "acme",
+		map[string]any{"sets": []string{"domain"}, "keys": []string{"user@tempbox.example"}})
+	a := answersFor(t, acme, "user@tempbox.example")
+	if a["from"] != "override" || a["verdict"] != Allow {
+		t.Errorf("acme's own allow must win: %v", a)
+	}
+	_, globex := call(t, app, http.MethodPost, "/v1/reference/resolve", "globex",
+		map[string]any{"sets": []string{"domain"}, "keys": []string{"user@tempbox.example"}})
+	g := answersFor(t, globex, "user@tempbox.example")
+	if g["from"] != "baseline" {
+		t.Errorf("globex has no override on that key and must read the baseline: %v", g)
+	}
+	if g["verdict"] != nil {
+		t.Errorf("the baseline states facts and never a verdict: %v", g)
+	}
+
+	// A third organisation, which never wrote anything, sees an empty list and
+	// the baseline's own answer.
+	code, body := call(t, app, http.MethodGet, "/v1/reference/domain", "initech", nil)
+	if code != 200 {
+		t.Fatalf("initech read: %d", code)
+	}
+	if list, _ := body["overrides"].([]any); len(list) != 0 {
+		t.Errorf("an org that wrote nothing holds %v", list)
+	}
+	if set, _ := body["set"].(map[string]any); set["overrides"] != float64(0) {
+		t.Errorf("initech's override count is %v", set["overrides"])
+	}
+}
+
+// TestNoRequestShapeNamesAnotherOrg holds the structural half: the write and
+// clear inputs carry NO field an organisation could be named in, so a caller
+// cannot even express the cross-tenant write. Sending one anyway changes
+// nothing, because there is nowhere for it to bind.
+func TestNoRequestShapeNamesAnotherOrg(t *testing.T) {
+	forbidden := map[string]bool{"org": true, "owner": true, "tenant": true, "scope": true, "project": true, "brand": true, "account": true}
+	for _, in := range []any{SetReferenceIn{}, ClearReferenceIn{}, ResolveReferenceIn{}, ReferenceIn{}, RefreshReferenceIn{}, ReferenceOverrideIn{}, ReferenceReceipt{}} {
+		for _, f := range wireNames(in) {
+			if forbidden[f] {
+				t.Errorf("%T carries a %q field; a cross-tenant write must be inexpressible, not merely refused", in, f)
+			}
+		}
+	}
+
+	app := mount(t)
+	seed(t, "domain", []Entry{{Key: "tempbox.example"}})
+	// Try to aim a write at another org anyway, every way the wire allows.
+	for _, body := range []map[string]any{
+		{"org": "globex", "entries": []any{map[string]any{"key": "a.example", "verdict": Deny}}},
+		{"scope": "globex", "owner": "globex", "tenant": "globex", "entries": []any{map[string]any{"key": "b.example", "verdict": Deny}}},
+	} {
+		if code, _ := call(t, app, http.MethodPut, "/v1/reference/domain?org=globex&scope=globex", "acme", body); code != 200 {
+			t.Fatalf("write: %d", code)
+		}
+	}
+	// globex holds nothing; acme holds both.
+	if _, g := call(t, app, http.MethodGet, "/v1/reference/domain", "globex", nil); len(g["overrides"].([]any)) != 0 {
+		t.Fatalf("a smuggled org landed in globex: %v", g["overrides"])
+	}
+	_, a := call(t, app, http.MethodGet, "/v1/reference/domain", "acme", nil)
+	if len(a["overrides"].([]any)) != 2 {
+		t.Fatalf("the writes did not land under the caller: %v", a["overrides"])
+	}
+}
+
+// wireNames lists the json field names of a wire struct.
+func wireNames(v any) []string {
+	b, err := json.Marshal(v)
+	if err != nil {
+		return nil
+	}
+	var m map[string]any
+	if err := json.Unmarshal(b, &m); err != nil {
+		return nil
+	}
+	out := make([]string, 0, len(m))
+	for k := range m {
+		out = append(out, k)
+	}
+	return out
+}
+
+// TestUnvalidatedCallerIsRefused: no principal, no organisation, no answer.
+// Every route fails closed from the same line.
+func TestUnvalidatedCallerIsRefused(t *testing.T) {
+	app := mount(t)
+	for _, c := range []struct {
+		method, path string
+		body         any
+	}{
+		{http.MethodGet, "/v1/reference", nil},
+		{http.MethodGet, "/v1/reference/domain", nil},
+		{http.MethodPut, "/v1/reference/domain", map[string]any{"entries": []any{map[string]any{"key": "a.example", "verdict": Deny}}}},
+		{http.MethodDelete, "/v1/reference/domain?key=a.example", nil},
+		{http.MethodPost, "/v1/reference/resolve", map[string]any{"keys": []string{"a.example"}}},
+	} {
+		if code, _ := call(t, app, c.method, c.path, "", c.body); code != http.StatusForbidden {
+			t.Errorf("%s %s with no principal answered %d, want 403", c.method, c.path, code)
+		}
+	}
+}
+
+// TestRefreshIsPlatformWork: writing the baseline every organisation reads is
+// gated to the platform's own identity, so no tenant can move another tenant's
+// world.
+func TestRefreshIsPlatformWork(t *testing.T) {
+	app := mount(t)
+	if code, _ := call(t, app, http.MethodPost, "/v1/reference/refresh", "acme", map[string]any{"set": "domain"}); code != http.StatusForbidden {
+		t.Errorf("a tenant refreshing the shared baseline answered %d, want 403", code)
+	}
+
+	// TWO ADMIN SCOPES, AND ONLY ONE OF THEM IS THIS ONE. An org admin is admin OF
+	// THEIR OWN ORG — self-service, org-scoped, not platform-privileged. This route
+	// writes the baseline EVERY org reads, so admitting the org-scoped bit here
+	// would let any customer's own administrator rewrite every other customer's
+	// reference data: the conflation IS the privilege escalation.
+	refresh := func(hdr map[string]string) int {
+		t.Helper()
+		rq := httptest.NewRequest(http.MethodPost, "/v1/reference/refresh",
+			strings.NewReader(`{"set":"domain"}`))
+		rq.Header.Set("Content-Type", "application/json")
+		rq.Header.Set("X-Org-Id", "acme")
+		rq.Header.Set("X-User-Id", "u_acme")
+		for k, v := range hdr {
+			rq.Header.Set(k, v)
+		}
+		resp, err := app.Fiber().Test(rq, fiber.TestConfig{Timeout: testTimeout, FailOnTimeout: true})
+		if err != nil {
+			t.Fatalf("refresh: %v", err)
+		}
+		_ = resp.Body.Close()
+		return resp.StatusCode
+	}
+	if code := refresh(map[string]string{"X-User-IsOrgAdmin": "true"}); code != http.StatusForbidden {
+		t.Errorf("an admin of their OWN org refreshing the shared baseline answered %d, want 403", code)
+	}
+	// And platform sudo is not refused by the gate. It stops at the warehouse this
+	// harness deliberately does not have, which is the next check and not this one.
+	if code := refresh(map[string]string{"X-User-IsAdmin": "true"}); code == http.StatusForbidden {
+		t.Error("SuperAdmin was refused by the gate meant to admit exactly it")
+	}
+}
+
+// ── precedence ───────────────────────────────────────────────────────────────
+
+// TestOverrideBeatsBaselineAndClearingRestoresIt is the precedence rule end to
+// end: override first, baseline second, first hit wins — and a removal can only
+// ever restore the baseline's own answer, never delete a published member.
+func TestOverrideBeatsBaselineAndClearingRestoresIt(t *testing.T) {
+	app := mount(t)
+	seed(t, "domain", []Entry{{Key: "tempbox.example", Value: map[string]string{"class": "disposable"}}})
+
+	// Before: the baseline answers.
+	_, before := call(t, app, http.MethodPost, "/v1/reference/resolve", "acme",
+		map[string]any{"sets": []string{"domain"}, "keys": []string{"user@tempbox.example"}})
+	if a := answersFor(t, before, "user@tempbox.example"); a["from"] != "baseline" || a["hit"] != true {
+		t.Fatalf("baseline should answer first: %v", a)
+	}
+
+	// An allow over it wins.
+	call(t, app, http.MethodPut, "/v1/reference/domain", "acme",
+		map[string]any{"entries": []any{map[string]any{"key": "tempbox.example", "verdict": Allow}}})
+	_, during := call(t, app, http.MethodPost, "/v1/reference/resolve", "acme",
+		map[string]any{"sets": []string{"domain"}, "keys": []string{"user@tempbox.example"}})
+	a := answersFor(t, during, "user@tempbox.example")
+	if a["from"] != "override" || a["verdict"] != Allow {
+		t.Fatalf("the org's own allow must beat the published list: %v", a)
+	}
+
+	// Clearing it restores the baseline, which was never touched.
+	code, cleared := call(t, app, http.MethodDelete, "/v1/reference/domain?key=tempbox.example", "acme", nil)
+	if code != 200 || cleared["cleared"] != true {
+		t.Fatalf("clear answered %d %v", code, cleared)
+	}
+	_, after := call(t, app, http.MethodPost, "/v1/reference/resolve", "acme",
+		map[string]any{"sets": []string{"domain"}, "keys": []string{"user@tempbox.example"}})
+	if a := answersFor(t, after, "user@tempbox.example"); a["from"] != "baseline" || a["hit"] != true {
+		t.Fatalf("the baseline must be intact after a removal: %v", a)
+	}
+}
+
+// TestOverrideIsMatchedTheSameWayTheBaselineIs: a deny on an apex covers its
+// subdomains, and a deny on a block covers its addresses — because both planes
+// go through one candidate function.
+func TestOverrideIsMatchedTheSameWayTheBaselineIs(t *testing.T) {
+	app := mount(t)
+	seed(t, "domain", []Entry{{Key: "other.example"}})
+	seed(t, "net", []Entry{{Key: "10.0.0.0/8", Value: map[string]string{"class": "hosting"}}})
+
+	call(t, app, http.MethodPut, "/v1/reference/domain", "acme",
+		map[string]any{"entries": []any{map[string]any{"key": "partner.example", "verdict": Allow}}})
+	call(t, app, http.MethodPut, "/v1/reference/net", "acme",
+		map[string]any{"entries": []any{map[string]any{"key": "203.0.113.0/24", "verdict": Deny}}})
+
+	_, body := call(t, app, http.MethodPost, "/v1/reference/resolve", "acme",
+		map[string]any{"sets": []string{"domain"}, "keys": []string{"bob@mail.partner.example"}})
+	if a := answersFor(t, body, "bob@mail.partner.example"); a["from"] != "override" || a["matched"] != "partner.example" {
+		t.Errorf("an override on the apex must cover a subdomain: %v", a)
+	}
+	_, body = call(t, app, http.MethodPost, "/v1/reference/resolve", "acme",
+		map[string]any{"sets": []string{"net"}, "keys": []string{"203.0.113.9"}})
+	if a := answersFor(t, body, "203.0.113.9"); a["from"] != "override" || a["matched"] != "203.0.113.0/24" {
+		t.Errorf("an override on a block must cover an address in it: %v", a)
+	}
+}
+
+// TestOverrideSurvivesAnUnloadedBaseline: an organisation's own deny list is the
+// one control that still works when the published source does not, so it is
+// consulted even when the set refuses.
+func TestOverrideSurvivesAnUnloadedBaseline(t *testing.T) {
+	app := mount(t)
+	call(t, app, http.MethodPut, "/v1/reference/domain", "acme",
+		map[string]any{"entries": []any{map[string]any{"key": "bad.example", "verdict": Deny}}})
+
+	_, body := call(t, app, http.MethodPost, "/v1/reference/resolve", "acme",
+		map[string]any{"sets": []string{"domain"}, "keys": []string{"x@bad.example", "x@unknown.example"}})
+	if a := answersFor(t, body, "x@bad.example"); a["from"] != "override" || a["verdict"] != Deny || a["refusal"] != nil {
+		t.Errorf("an override answers even with no baseline loaded: %v", a)
+	}
+	// And a key the override does not cover reports the refusal rather than
+	// reading as clean.
+	a := answersFor(t, body, "x@unknown.example")
+	if a["hit"] == true {
+		t.Errorf("an unloaded set cannot hit: %v", a)
+	}
+	if a["refusal"] == nil {
+		t.Errorf("a miss on an unloaded set must carry the refusal: %v", a)
+	}
+	if refused, _ := body["refused"].([]any); len(refused) == 0 {
+		t.Errorf("the response must name the sets that could not be consulted: %v", body)
+	}
+}
+
+// TestResolveNamesEveryVersionConsulted: the whole point of versioning is that a
+// decision can record exactly what it leaned on.
+func TestResolveNamesEveryVersionConsulted(t *testing.T) {
+	app := mount(t)
+	seed(t, "domain", []Entry{{Key: "tempbox.example"}})
+	seed(t, "net", []Entry{{Key: "10.0.0.0/8"}})
+
+	_, body := call(t, app, http.MethodPost, "/v1/reference/resolve", "acme",
+		map[string]any{"sets": []string{"domain", "net", "pep"}, "keys": []string{"user@tempbox.example"}})
+	consulted, _ := body["consulted"].([]any)
+	if len(consulted) != 3 {
+		t.Fatalf("want a version line per consulted set, got %v", consulted)
+	}
+	seen := map[string]map[string]any{}
+	for _, c := range consulted {
+		m, _ := c.(map[string]any)
+		seen[m["set"].(string)] = m
+	}
+	for _, name := range []string{"domain", "net"} {
+		if v, _ := seen[name]["version"].(string); v == "" {
+			t.Errorf("%s consulted with no version named: %v", name, seen[name])
+		}
+		if seen[name]["asOf"] == nil {
+			t.Errorf("%s consulted with no as-of: %v", name, seen[name])
+		}
+	}
+	if seen["pep"]["refusal"] == nil {
+		t.Errorf("an unlicensed set must be named as refused: %v", seen["pep"])
+	}
+}
+
+// TestMisspeltSetIsRefusedRatherThanSkipped: a caller who names a set that does
+// not exist and gets a silent pass has been told the key is clean by a set that
+// was never consulted.
+func TestMisspeltSetIsRefusedRatherThanSkipped(t *testing.T) {
+	app := mount(t)
+	if code, _ := call(t, app, http.MethodPost, "/v1/reference/resolve", "acme",
+		map[string]any{"sets": []string{"domians"}, "keys": []string{"a.example"}}); code != http.StatusNotFound {
+		t.Errorf("a misspelt set answered %d, want 404", code)
+	}
+	if code, _ := call(t, app, http.MethodGet, "/v1/reference/nosuchset", "acme", nil); code != http.StatusNotFound {
+		t.Errorf("an unknown set answered %d, want 404", code)
+	}
+}
+
+// ── bounds ───────────────────────────────────────────────────────────────────
+
+// TestBoundsAreRefusals: a lookup cannot be turned into a scan, and a batch that
+// would cross the per-set bound writes nothing rather than half of itself.
+func TestBoundsAreRefusals(t *testing.T) {
+	app := mount(t)
+
+	tooMany := make([]string, maxKeys+1)
+	for i := range tooMany {
+		tooMany[i] = "a.example"
+	}
+	if code, _ := call(t, app, http.MethodPost, "/v1/reference/resolve", "acme",
+		map[string]any{"keys": tooMany}); code != http.StatusBadRequest {
+		t.Errorf("an oversized resolve answered %d, want 400", code)
+	}
+	if code, _ := call(t, app, http.MethodPost, "/v1/reference/resolve", "acme",
+		map[string]any{"keys": []string{}}); code != http.StatusBadRequest {
+		t.Errorf("an empty resolve answered %d, want 400", code)
+	}
+	if code, _ := call(t, app, http.MethodPut, "/v1/reference/domain", "acme",
+		map[string]any{"entries": []any{map[string]any{"key": "a.example", "verdict": "maybe"}}}); code != http.StatusBadRequest {
+		t.Errorf("a verdict outside the vocabulary answered %d, want 400", code)
+	}
+}
+
+// TestOneRequestCannotSpendTheProcess is the ship-blocker, over the wire.
+//
+// Two amplifiers composed. A key had a COUNT bound and no BYTE bound, so one
+// 8 KB dotted key materialised every suffix of itself — O(labels x bytes) — and
+// [maxKeys] of them allocated 1.7 GB inside a single authenticated request. And
+// `sets` had no bound and no dedupe, so naming one set N times ran N times the
+// answers, multiplying whatever the first amplifier cost. On the one-replica
+// deployment this plane ships on, that is one request away from taking down every
+// other product in the process.
+//
+// Measured on the fix: the same call allocates what its own body weighs and is
+// refused. The assertion is on the allocation as well as the status, because a
+// 400 arrived at after doing the work is not a bound.
+func TestOneRequestCannotSpendTheProcess(t *testing.T) {
+	app := mount(t)
+	seed(t, "domain", []Entry{{Key: "tempbox.example"}})
+
+	long := strings.Repeat("a.", 4096) + "example"
+	keys := make([]string, maxKeys)
+	for i := range keys {
+		keys[i] = long
+	}
+	var before, after runtime.MemStats
+	runtime.GC()
+	runtime.ReadMemStats(&before)
+	code, _ := call(t, app, http.MethodPost, "/v1/reference/resolve", "acme",
+		map[string]any{"sets": []string{"domain"}, "keys": keys})
+	runtime.ReadMemStats(&after)
+	if code != http.StatusBadRequest {
+		t.Errorf("%d keys of %d bytes answered %d, want 400", len(keys), len(long), code)
+	}
+	if grew := (after.TotalAlloc - before.TotalAlloc) >> 20; grew > 64 {
+		t.Errorf("one refused request allocated %d MiB; a bound reached after the work is not a bound", grew)
+	}
+
+	// The same key, written as an override, is the other half of the same door.
+	if code, _ := call(t, app, http.MethodPut, "/v1/reference/domain", "acme",
+		map[string]any{"entries": []any{map[string]any{"key": long, "verdict": Deny}}}); code != http.StatusBadRequest {
+		t.Errorf("an over-long override key answered %d, want 400", code)
+	}
+	// A removal takes its key in the query string, where the transport's own
+	// header buffer refuses anything really enormous before this plane sees it —
+	// so the case that matters is the one that gets through: past maxKey, inside
+	// the buffer.
+	overLong := strings.Repeat("d.", (maxKey+8)/2) + "example"
+	if code, _ := call(t, app, http.MethodDelete, "/v1/reference/domain?key="+overLong, "acme", nil); code != http.StatusBadRequest {
+		t.Errorf("an over-long clear key of %d bytes answered %d, want 400", len(overLong), code)
+	}
+	// The page cursor is the last KEY of the previous page, so it crosses the same
+	// door: every door is the same door.
+	if code, _ := call(t, app, http.MethodGet, "/v1/reference/domain?after="+overLong, "acme", nil); code != http.StatusBadRequest {
+		t.Errorf("an over-long page cursor of %d bytes answered %d, want 400", len(overLong), code)
+	}
+
+	// And the bound refuses nothing a caller legitimately asks about: the longest
+	// address RFC 5321 permits still resolves.
+	legit := strings.Repeat("a", 64) + "@" + strings.Repeat("b", 61) + "." + strings.Repeat("c", 61) + "." + strings.Repeat("d", 61) + ".example"
+	if len(legit) > maxKey {
+		t.Fatalf("the sample address is %d bytes, past the bound itself", len(legit))
+	}
+	if code, _ := call(t, app, http.MethodPost, "/v1/reference/resolve", "acme",
+		map[string]any{"sets": []string{"domain"}, "keys": []string{legit}}); code != 200 {
+		t.Errorf("a %d-byte address answered %d; the bound must refuse nothing real", len(legit), code)
+	}
+
+	// The second amplifier: more names than there are sets is refused outright,
+	// and a set named twice is consulted once.
+	dup := make([]string, len(Catalog())+1)
+	for i := range dup {
+		dup[i] = "domain"
+	}
+	if code, _ := call(t, app, http.MethodPost, "/v1/reference/resolve", "acme",
+		map[string]any{"sets": dup, "keys": []string{"x@tempbox.example"}}); code != http.StatusBadRequest {
+		t.Errorf("naming %d sets when the plane publishes %d answered %d, want 400", len(dup), len(Catalog()), code)
+	}
+	code, body := call(t, app, http.MethodPost, "/v1/reference/resolve", "acme",
+		map[string]any{"sets": []string{"domain", "domain", "domain"}, "keys": []string{"x@tempbox.example"}})
+	if code != 200 {
+		t.Fatalf("a repeated set answered %d", code)
+	}
+	if answers, _ := body["answers"].([]any); len(answers) != 1 {
+		t.Errorf("one set named three times produced %d answers; a duplicate is a redundancy, not more work", len(answers))
+	}
+	if consulted, _ := body["consulted"].([]any); len(consulted) != 1 {
+		t.Errorf("one set named three times was consulted %d times", len(consulted))
+	}
+}
+
+// TestAnOverrideCannotFillTheVolumeEveryOrgSharesOn is the other ship-blocker.
+//
+// maxOverrides bounds one organisation's entries per set, which is a bound on
+// ROWS and was not a bound on BYTES: with no length on the key, one entry could
+// be the whole request body, so 10,000 entries x 11 sets was gigabytes of
+// attacker-chosen data on the one volume every other organisation's store lives
+// on. A per-tenant count bound only isolates tenants once one row is bounded.
+func TestAnOverrideCannotFillTheVolumeEveryOrgSharesOn(t *testing.T) {
+	app := mount(t)
+
+	big := strings.Repeat("x", 1<<20)
+	code, _ := call(t, app, http.MethodPut, "/v1/reference/domain", "acme",
+		map[string]any{"entries": []any{map[string]any{"key": big, "verdict": Deny}}})
+	if code != http.StatusBadRequest {
+		t.Fatalf("a 1 MiB override key answered %d, want 400", code)
+	}
+	// Nothing landed: a refused write writes nothing.
+	_, read := call(t, app, http.MethodGet, "/v1/reference/domain", "acme", nil)
+	if list, _ := read["overrides"].([]any); len(list) != 0 {
+		t.Fatalf("a refused write left %d entries behind", len(list))
+	}
+
+	// A note past its bound is refused rather than silently cut: an operator's
+	// stated reason for an adverse action is not a field to trim.
+	if code, _ := call(t, app, http.MethodPut, "/v1/reference/domain", "acme",
+		map[string]any{"entries": []any{map[string]any{"key": "a.example", "verdict": Deny, "note": strings.Repeat("n", maxNote+1)}}}); code != http.StatusBadRequest {
+		t.Errorf("an over-long note answered %d, want 400", code)
+	}
+
+	// And a key at the bound is accepted and stored whole.
+	ok := strings.Repeat("k", maxKey-len(".example")) + ".example"
+	if code, _ := call(t, app, http.MethodPut, "/v1/reference/domain", "acme",
+		map[string]any{"entries": []any{map[string]any{"key": ok, "verdict": Deny}}}); code != 200 {
+		t.Fatalf("a %d-byte key answered %d, want 200", len(ok), code)
+	}
+	_, read = call(t, app, http.MethodGet, "/v1/reference/domain", "acme", nil)
+	list, _ := read["overrides"].([]any)
+	if len(list) != 1 {
+		t.Fatalf("want the one accepted entry, got %v", list)
+	}
+	if got, _ := list[0].(map[string]any)["key"].(string); len(got) != len(ok) {
+		t.Errorf("stored key is %d bytes, wrote %d — a key must never be trimmed", len(got), len(ok))
+	}
+}
+
+// TestAnAcknowledgedOverrideIsDurable: an override is a record, not a cache — it
+// is why a signup was refused. This deployment runs ONE replica with a recreate
+// rollout, so an unshipped write is lost by the next deploy, and a control an
+// operator believes is in force and is not is worse than one they know is absent.
+// So the write is acknowledged only once the store is fenced to its durable
+// object, exactly as apps/research and apps/books do it.
+func TestAnAcknowledgedOverrideIsDurable(t *testing.T) {
+	app := mount(t)
+
+	// This replica does not hold the organisation's write lease.
+	mounted.State.sync = func(namespace.Namespace) (bool, error) { return false, nil }
+	if code, _ := call(t, app, http.MethodPut, "/v1/reference/domain", "acme",
+		map[string]any{"entries": []any{map[string]any{"key": "a.example", "verdict": Deny}}}); code != http.StatusServiceUnavailable {
+		t.Errorf("a write that could not be made durable answered %d, want 503", code)
+	}
+
+	// The ship errors outright.
+	mounted.State.sync = func(namespace.Namespace) (bool, error) { return false, errors.New("object store unreachable") }
+	if code, _ := call(t, app, http.MethodPut, "/v1/reference/domain", "acme",
+		map[string]any{"entries": []any{map[string]any{"key": "b.example", "verdict": Deny}}}); code != http.StatusServiceUnavailable {
+		t.Errorf("a write whose ship failed answered %d, want 503", code)
+	}
+
+	// Acknowledged: the write lands, and so does the removal that follows it.
+	shipped := 0
+	mounted.State.sync = func(namespace.Namespace) (bool, error) { shipped++; return true, nil }
+	if code, _ := call(t, app, http.MethodPut, "/v1/reference/domain", "acme",
+		map[string]any{"entries": []any{map[string]any{"key": "c.example", "verdict": Deny}}}); code != 200 {
+		t.Fatalf("an acknowledged write answered %d", code)
+	}
+	if code, cleared := call(t, app, http.MethodDelete, "/v1/reference/domain?key=c.example", "acme", nil); code != 200 || cleared["cleared"] != true {
+		t.Fatalf("clear answered %d %v", code, cleared)
+	}
+	if shipped != 2 {
+		t.Errorf("the store shipped %d times for a write and a removal; both are writes", shipped)
+	}
+}
+
+// TestThisAppOwnsOnlyItsOwnLeaf: /v1/ml is a SHARED parent — a model-serving
+// plane answers on /v1/ml/models and a dataset plane on /v1/ml/datasets in the
+// same process — so a middleware installed there by this app would run inside two
+// other products' request paths, decided by nothing but mount order.
+func TestThisAppOwnsOnlyItsOwnLeaf(t *testing.T) {
+	app := mount(t)
+	var sawPrincipal bool
+	app.Fiber().Get("/v1/ml/models", func(c fiber.Ctx) error {
+		_, sawPrincipal = principal.OrgFrom(c.Context())
+		return c.SendString("neighbour")
+	})
+	rq := httptest.NewRequest(http.MethodGet, "/v1/ml/models", nil)
+	rq.Header.Set("X-Org-Id", "acme")
+	rq.Header.Set("X-User-Id", "u_acme")
+	resp, err := app.Fiber().Test(rq, fiber.TestConfig{Timeout: testTimeout, FailOnTimeout: true})
+	if err != nil {
+		t.Fatalf("neighbour: %v", err)
+	}
+	_ = resp.Body.Close()
+	if sawPrincipal {
+		t.Error("this app's bridge ran for a neighbouring app's route; an app owns its own leaf and nothing above it")
+	}
+	// And it still runs for every route this app DOES own — which the whole
+	// tenancy suite above depends on, and this states outright.
+	if code, _ := call(t, app, http.MethodGet, "/v1/reference", "acme", nil); code != 200 {
+		t.Errorf("this app's own collection route answered %d", code)
+	}
+}
+
+// TestTheSetListReportsStaleAndRefused: the two ways this plane can be quietly
+// wrong are reported rather than inferred.
+func TestTheSetListReportsStaleAndRefused(t *testing.T) {
+	app := mount(t)
+	seed(t, "domain", []Entry{{Key: "tempbox.example"}})
+
+	code, body := call(t, app, http.MethodGet, "/v1/reference", "acme", nil)
+	if code != 200 {
+		t.Fatalf("list: %d", code)
+	}
+	sets, _ := body["sets"].([]any)
+	if len(sets) != len(Catalog()) {
+		t.Fatalf("want %d sets, got %d", len(Catalog()), len(sets))
+	}
+	refused, _ := body["refused"].([]any)
+	names := map[string]bool{}
+	for _, r := range refused {
+		names[r.(string)] = true
+	}
+	for _, want := range []string{"pep", "issuer", "reputation", "net"} {
+		if !names[want] {
+			t.Errorf("%q should be reported as refused (unlicensed, or never loaded): %v", want, refused)
+		}
+	}
+	if names["domain"] {
+		t.Errorf("a loaded set must not be refused: %v", refused)
+	}
+	// Every set states its terms so an operator can audit the licences from the
+	// wire alone.
+	for _, s := range sets {
+		m, _ := s.(map[string]any)
+		if m["kind"] == string(KindSeam) {
+			if m["refusal"] == nil {
+				t.Errorf("%v is a seam with no reason on the wire", m["set"])
+			}
+			continue
+		}
+		srcs, _ := m["sources"].([]any)
+		if len(srcs) == 0 {
+			t.Errorf("%v publishes no sources", m["set"])
+		}
+		for _, s := range srcs {
+			sm, _ := s.(map[string]any)
+			if terms, _ := sm["terms"].(string); terms == "" {
+				t.Errorf("%v/%v states no terms on the wire", m["set"], sm["source"])
+			}
+		}
+	}
+}

apps/reference/typed_wire_test.go

@@ -0,0 +1,159 @@
+package reference
+
+import (
+	"sort"
+	"strings"
+	"testing"
+
+	"github.com/hanzoai/cloud/openapi"
+)
+
+// This file makes "every route is a typed op" a GATE instead of a paragraph.
+// Prose cannot fail: a route added tomorrow as a raw func(*zip.Ctx) error would
+// leave the claim standing and the route invisible to every projection — no
+// schema, no description, no MCP tool, no CLI command, no SDK method.
+
+// untypedByDesign is the CLOSED list of operations that are not typed ops. It is
+// EMPTY, and that is the claim: this surface has no route that cannot be
+// expressed as one.
+var untypedByDesign = map[string]string{}
+
+// ops reads BOTH projections of the live router at their one shared address
+// form: what the document says is served, and which of those carry a typed
+// registry entry.
+func surface(t *testing.T) (served map[string]bool, typed map[string]string) {
+	t.Helper()
+	app := mount(t)
+	doc, err := openapi.Spec(app, openapi.Info{Title: "reference", Version: "v1"})
+	if err != nil {
+		t.Fatalf("spec: %v", err)
+	}
+	reg, err := openapi.Typed(app)
+	if err != nil {
+		t.Fatalf("typed registry: %v", err)
+	}
+	served, typed = map[string]bool{}, map[string]string{}
+	for path, item := range doc.Paths {
+		for method := range item {
+			served[strings.ToUpper(method)+" "+path] = true
+		}
+	}
+	for key, op := range reg.Ops {
+		typed[key] = op.Description
+	}
+	return served, typed
+}
+
+// TestEveryRouteIsTyped fails when an operation is neither a typed op nor named
+// above, so the next route added here is typed by default.
+func TestEveryRouteIsTyped(t *testing.T) {
+	served, typed := surface(t)
+
+	var untyped []string
+	for key := range served {
+		if _, ok := typed[key]; ok {
+			continue
+		}
+		if _, named := untypedByDesign[key]; named {
+			continue
+		}
+		untyped = append(untyped, key)
+	}
+	if len(untyped) > 0 {
+		sort.Strings(untyped)
+		t.Errorf("operation(s) with no registry entry and no reason: %s\n"+
+			"A route that is not a typed op has no schema, no prose, no MCP tool, no CLI command and no SDK method.",
+			strings.Join(untyped, ", "))
+	}
+	if got, want := len(typed)+len(untypedByDesign), len(served); got != want {
+		t.Errorf("typed(%d) + named(%d) = %d, served = %d — the ledgers must partition the surface",
+			len(typed), len(untypedByDesign), got, want)
+	}
+	// The MEASURED surface, so the prose cannot drift from the binary.
+	if len(served) != 6 || len(typed) != 6 {
+		t.Errorf("served = %d (want 6), typed = %d (want 6)", len(served), len(typed))
+	}
+}
+
+// TestEveryTypedOpIsDescribed proves the lifted prose reached the binary. That
+// prose IS the product surface: it becomes the OpenAPI description AND the MCP
+// tool description a model reads to pick the tool.
+func TestEveryTypedOpIsDescribed(t *testing.T) {
+	_, typed := surface(t)
+	if len(typed) == 0 {
+		t.Fatal("no typed ops in the registry at all")
+	}
+	for key, desc := range typed {
+		if strings.TrimSpace(desc) == "" {
+			t.Errorf("%s has no description — run: go generate -run zipdoc ./apps/reference/...", key)
+		}
+	}
+}
+
+// TestTheSurfaceIsTheDeclaredPrefix holds that this app answers on exactly one
+// address family and takes nothing that belongs to the ml app one prefix over —
+// the manifest routes /v1/reference here and /v1/ml/models and /v1/ml/health
+// there, and zip refuses two owners for one path at compose time.
+func TestTheSurfaceIsTheDeclaredPrefix(t *testing.T) {
+	served, _ := surface(t)
+	for key := range served {
+		_, path, _ := strings.Cut(key, " ")
+		if !strings.HasPrefix(path, "/v1/reference") {
+			t.Errorf("%s is outside this app's declared prefix", key)
+		}
+	}
+	for _, foreign := range []string{"GET /v1/ml/models", "GET /v1/ml/health"} {
+		if served[foreign] {
+			t.Errorf("this app serves %s, which belongs to the ml app", foreign)
+		}
+	}
+}
+
+// TestThisAppAnswersUnderItsOwnName states the prefix property INDEPENDENTLY of
+// the constants that build it, which is the only way a prefix test can catch the
+// defect this app shipped with.
+//
+// Every op here used to answer at /v1/ml/reference and carry ml's OpenAPI tag —
+// inside the model-SERVING product's prefix, one app over from the risk model
+// plane that had just been moved OFF /v1/ml for that exact reason. The test that
+// was supposed to hold the address could not: it compared the served paths against
+// the literal /v1/ml/reference, so it agreed with whatever the constant said and
+// would agree again with the next wrong prefix.
+//
+// So the statement here is about the app's IDENTITY and not about its routes: the
+// PRODUCT every served path resolves to must be this app's own name — the same
+// word that names its per-organisation store file and its manifest row.
+//
+// ONE CHECK COVERS THREE THINGS, because in this fleet they are one thing.
+// openapi.Product is the fleet's own definition — the second path segment — and
+// it is what the published tag and the per-product count in openapi/floor.json
+// are BOTH derived from (openapi.From sets op.Tags to it; Fold then overwrites a
+// typed op's own tags with the path's). So `zip.WithTags` does not decide the
+// published tag and asserting it separately would assert nothing: the path is the
+// only thing to get right, and getting it wrong moved six operations into ml's
+// tag and ml's count at once.
+func TestThisAppAnswersUnderItsOwnName(t *testing.T) {
+	app := mount(t)
+	doc, err := openapi.Spec(app, openapi.Info{Title: "reference", Version: "v1"})
+	if err != nil {
+		t.Fatalf("spec: %v", err)
+	}
+	if len(doc.Paths) == 0 {
+		t.Fatal("no paths at all — this test would pass vacuously")
+	}
+	for path, item := range doc.Paths {
+		if got := openapi.Product(path); got != subsystem {
+			t.Errorf("%s is product %q and this app is %q — an op under another product's prefix IS that "+
+				"product's surface: its tag, its published count and its documentation, whatever this package "+
+				"calls itself", path, got, subsystem)
+		}
+		// And the tag the document actually publishes is that product, so an operator
+		// reading the tag reads the owner.
+		for method, op := range item {
+			if len(op.Tags) == 0 || op.Tags[0] != openapi.Product(path) {
+				t.Errorf("%s %s publishes tags %v, which is not the product its path resolves to (%q)",
+					strings.ToUpper(method), path, op.Tags, openapi.Product(path))
+			}
+		}
+	}
+}

apps/reference/warehouse_test.go

@@ -0,0 +1,196 @@
+package reference
+
+// warehouse_test.go is a warehouse small enough to reason about and faithful
+// enough to hold the durable half to account.
+//
+// It exists because three of this plane's properties are properties of the
+// SEQUENCE of statements a refresh issues, not of any function: which version
+// prune spares, whether a take that shrank is allowed to land, and whether a
+// receipt with nothing in it becomes a current version. Every one of those was
+// asserted before by reading the statement CONSTANT, which is the shape of a test
+// that cannot fail — the constant was right and the call site was wrong, and a
+// test that reads only the constant passes either way.
+//
+// So it answers the six statements this package issues, and nothing else. An
+// unrecognised statement is a test failure rather than an empty result set: a
+// fake that silently answers nothing to a statement it does not know is the same
+// toothless test in another costume.
+
+import (
+	"context"
+	"fmt"
+	"sort"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+)
+
+// warehouse is the in-memory stand-in: the two tables, and a log of every
+// statement issued so a test can assert on the sequence as well as the outcome.
+type warehouse struct {
+	mu sync.Mutex
+	// manifest is hanzo.reference_source keyed the way its ORDER BY keys it.
+	manifest map[[3]string]version
+	// members is hanzo.reference_entry, keyed by (set, source, version).
+	members map[[3]string][]Entry
+	// pruned records every prune, in order, as (set, source, keep, alsoKeep).
+	pruned [][4]string
+	// wrote counts entry-row writes, so "unchanged writes no rows" stays provable.
+	wrote int
+	// down makes every statement fail, which is a warehouse that is not up.
+	down bool
+}
+
+func newWarehouse() *warehouse {
+	return &warehouse{manifest: map[[3]string]version{}, members: map[[3]string][]Entry{}}
+}
+
+// use substitutes this warehouse for the real one for the life of the test, and
+// resets the package's DDL latch so each test bootstraps against its own store.
+//
+// It must be called BEFORE mount: the restore is registered first and cleanup is
+// last-in-first-out, so the mounted service's Shutdown runs — and its background
+// loop stops touching these values — before they are put back.
+func (w *warehouse) use(t *testing.T) {
+	t.Helper()
+	ready, query, exec := storeReady, storeQuery, storeExec
+	tableMu.Lock()
+	tableReady = false
+	tableMu.Unlock()
+	t.Cleanup(func() {
+		storeReady, storeQuery, storeExec = ready, query, exec
+		tableMu.Lock()
+		tableReady = false
+		tableMu.Unlock()
+	})
+	storeReady = func() bool { return !w.isDown() }
+	storeQuery = w.query
+	storeExec = w.exec
+}
+
+func (w *warehouse) isDown() bool {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	return w.down
+}
+
+func (w *warehouse) exec(_ context.Context, stmt string, args ...any) error {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	if w.down {
+		return fmt.Errorf("warehouse down")
+	}
+	switch {
+	case strings.HasPrefix(stmt, "CREATE"):
+		return nil
+	case strings.HasPrefix(stmt, markStatement[:40]):
+		v := version{
+			Set: args[0].(string), Source: args[1].(string), Version: args[2].(string),
+			Origin: args[3].(string), Terms: args[4].(string),
+			AsOf: args[5].(time.Time), Fetched: args[6].(time.Time),
+			Keys: args[7].(uint64), Landed: args[8].(uint64),
+			Status: args[9].(string), Refusal: args[10].(string),
+		}
+		w.manifest[[3]string{v.Set, v.Source, v.Version}] = v
+		return nil
+	case strings.HasPrefix(stmt, "INSERT INTO "+entryTable):
+		// Nine bound values per row, in the order insert() renders them.
+		for i := 0; i+9 <= len(args); i += 9 {
+			k := [3]string{args[i].(string), args[i+1].(string), args[i+2].(string)}
+			w.members[k] = append(w.members[k], Entry{
+				Key: args[i+3].(string), Value: args[i+4].(map[string]string),
+				Score: args[i+5].(float64), Orgs: args[i+6].(uint32), N: args[i+7].(uint64),
+			})
+			w.wrote++
+		}
+		return nil
+	case strings.HasPrefix(stmt, "ALTER TABLE "+entryTable):
+		set, source, keep, alsoKeep := args[0].(string), args[1].(string), args[2].(string), args[3].(string)
+		w.pruned = append(w.pruned, [4]string{set, source, keep, alsoKeep})
+		for k := range w.members {
+			if k[0] == set && k[1] == source && k[2] != keep && k[2] != alsoKeep {
+				delete(w.members, k)
+			}
+		}
+		return nil
+	}
+	return fmt.Errorf("warehouse: no statement like %.60q", stmt)
+}
+
+func (w *warehouse) query(_ context.Context, q string, args ...any) ([]map[string]any, error) {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	if w.down {
+		return nil, fmt.Errorf("warehouse down")
+	}
+	switch {
+	case strings.HasPrefix(q, currentStatement[:40]):
+		want := args[0].(string)
+		newest := map[[2]string]version{}
+		for _, v := range w.manifest {
+			if v.Status != want {
+				continue
+			}
+			k := [2]string{v.Set, v.Source}
+			if held, ok := newest[k]; !ok || v.Fetched.After(held.Fetched) {
+				newest[k] = v
+			}
+		}
+		out := make([]map[string]any, 0, len(newest))
+		for _, v := range newest {
+			out = append(out, rowOf(v))
+		}
+		sort.Slice(out, func(i, j int) bool {
+			return out[i]["set"].(string)+out[i]["source"].(string) < out[j]["set"].(string)+out[j]["source"].(string)
+		})
+		return out, nil
+	case strings.HasPrefix(q, takenStatement[:40]):
+		v, ok := w.manifest[[3]string{args[0].(string), args[1].(string), args[2].(string)}]
+		if !ok {
+			return nil, nil
+		}
+		return []map[string]any{rowOf(v)}, nil
+	case strings.HasPrefix(q, readStatement[:30]):
+		got := w.members[[3]string{args[0].(string), args[1].(string), args[2].(string)}]
+		out := make([]map[string]any, 0, len(got))
+		for _, e := range got {
+			out = append(out, map[string]any{"key": e.Key, "value": e.Value, "score": e.Score, "orgs": e.Orgs, "n": e.N})
+		}
+		return out, nil
+	}
+	return nil, fmt.Errorf("warehouse: no query like %.60q", q)
+}
+
+func rowOf(v version) map[string]any {
+	return map[string]any{
+		"set": v.Set, "source": v.Source, "version": v.Version,
+		"origin": v.Origin, "terms": v.Terms,
+		"as_of": v.AsOf, "fetched": v.Fetched,
+		"keys": v.Keys, "landed": v.Landed,
+		"status": v.Status, "refusal": v.Refusal,
+	}
+}
+
+// held reads one manifest row back.
+func (w *warehouse) held(set, source, ver string) (version, bool) {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	v, ok := w.manifest[[3]string{set, source, ver}]
+	return v, ok
+}
+
+// rows reads one version's membership back — the thing prune deletes and an
+// auditor asks for.
+func (w *warehouse) rows(set, source, ver string) []Entry {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	return w.members[[3]string{set, source, ver}]
+}
+
+// prunes copies the prune log.
+func (w *warehouse) prunes() [][4]string {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	return append([][4]string{}, w.pruned...)
+}

apps/reference/zipdoc_gen.go

@@ -0,0 +1,157 @@
+// Code generated by zipdoc; DO NOT EDIT.
+
+package reference
+
+import (
+	"encoding/json"
+
+	"github.com/zap-proto/zip"
+)
+
+func init() {
+	zip.Describe("DELETE /v1/reference/:set", zip.Doc{
+		Description: "Removes one of your organisation's overrides.\n\nIt removes an entry your organisation wrote, never a baseline member: the\npublished set is not writable from here, so a removal can only ever restore\nthe baseline's own answer.",
+		Fields: map[string]string{
+			"ClearReferenceOut.cleared":   "Cleared is false when your org held no such override — which is not an\nerror, it is the honest answer to a removal that had nothing to remove.",
+			"ClearReferenceOut.key":       "Key is the entry named.",
+			"ClearReferenceOut.overrides": "Overrides is how many your org still holds in this set.",
+			"ClearReferenceOut.set":       "Set is the set cleared in.",
+		},
+		Example: json.RawMessage(`{"set":"domain","key":"partner.example"}`),
+	})
+	zip.Describe("GET /v1/reference", zip.Doc{
+		Description: "Lists every set this plane publishes, with its version and how\nfresh it is.\n\nRead the Stale and Refused lists first: they are the two ways this plane can\nbe quietly wrong, and they are reported rather than inferred. A set in\nRefused answers nothing — it has never loaded, it is held by another\ncomponent, or it names a source we hold no licence for.",
+		Fields: map[string]string{
+			"ReferenceSet.age":         "Age is how long ago that was.",
+			"ReferenceSet.asOf":        "AsOf is when the OLDEST contributing publisher was current, RFC 3339. The\noldest and not the newest: a set is exactly as fresh as its weakest source.",
+			"ReferenceSet.keys":        "Keys is how many members the baseline carries.",
+			"ReferenceSet.kind":        "Kind is how the baseline comes to exist: fetch (downloaded from a\npublisher), local (computed here), attest (held by the component that\nscreens against it, freshness reported), or seam (declared and NOT held,\nbecause the source needs a licence we do not have).",
+			"ReferenceSet.match":       "Match is how a key is tested: exact, domain, net, digits, pattern or range.",
+			"ReferenceSet.maxAge":      "MaxAge is how old this set may be before it is stale.",
+			"ReferenceSet.overrides":   "Overrides is how many entries YOUR org has laid over this baseline.",
+			"ReferenceSet.refusal":     "Refusal names why the set cannot be relied on, when it cannot: never\nloaded, held elsewhere, or a licence we do not hold. Non-empty means a\nlookup against this set will not answer, rather than answering clean.",
+			"ReferenceSet.set":         "Set is the name this set is addressed by.",
+			"ReferenceSet.sources":     "Sources is each contributing publisher, its licence and its own freshness.",
+			"ReferenceSet.stale":       "Stale is whether it is past that bound. A stale set still answers and says\nso, because yesterday's list beats none.",
+			"ReferenceSet.version":     "Version is the exact baseline consulted — every contributing publisher and\nits content digest. A decision records this and an auditor resolves it back.",
+			"ReferenceSet.what":        "What the set holds, in one sentence.",
+			"ReferenceSetsOut.refused": "Refused names the sets that cannot be consulted at all. A key checked\nagainst one of these is UNKNOWN, not clean.",
+			"ReferenceSetsOut.sets":    "Sets is the whole catalog, in a stable order.",
+			"ReferenceSetsOut.stale":   "Stale names the sets past their freshness bound — the list to alarm on.",
+			"ReferenceSource.asOf":     "AsOf is when this publisher was current, RFC 3339.",
+			"ReferenceSource.basis":    "Basis is the KIND of permission this publisher's data reaches you under:\nlicence (an explicit grant), registry (the registry of record publishing for\nanyone to consult), operator (an operator's own machine-readable statement\nabout its own network, published for third parties to filter by — not a\nlicence, and not claimed as one), own (computed here), or none (nothing\nreaches you: the membership is held by the component that screens against\nit). It is on the wire so the licence position is an audit you can run.",
+			"ReferenceSource.keys":     "Keys is how many members this publisher contributed.",
+			"ReferenceSource.origin":   "Origin is exactly where it was taken from, so it can be taken again.",
+			"ReferenceSource.refusal":  "Refusal is why this publisher's last take failed, if it did. The set keeps\nits previous version of this source and ages out visibly rather than\nsilently shrinking.",
+			"ReferenceSource.source":   "Source is the publisher.",
+			"ReferenceSource.terms":    "Terms is the CITATION that basis points at — the licence identifier, the\nregistry, or the operator publication. A source with no stated terms is not\nin the catalog.",
+			"ReferenceSource.version":  "Version is the content digest of what this publisher last supplied. Two\nrefreshes that agree on it took the same data.",
+		},
+		Example: json.RawMessage(`{}`),
+	})
+	zip.Describe("GET /v1/reference/:set", zip.Doc{
+		Description: "Reference describes one set and lists your org's overrides in it.\n\nThe set half is public data about a published list — its version, its\npublishers, their licences and how current each one is. The overrides half is\nyours alone: it is read from your organisation's own store, and no other\norganisation's entries can appear in it.",
+		Fields: map[string]string{
+			"ReferenceIn.after":         "After pages the override listing: the last key of the previous page.",
+			"ReferenceIn.limit":         "Limit caps the override listing: default 200, maximum 1000.",
+			"ReferenceOut.next":         "Next is the key to page from, empty when this is the last page.",
+			"ReferenceOut.overrides":    "Overrides is YOUR org's entries over that baseline, in key order. They are\nheld in your organisation's own store and are not visible to any other.",
+			"ReferenceOut.set":          "Set is the published set: its version, its freshness and its sources.",
+			"ReferenceOverride.at":      "At is when it was written, RFC 3339.",
+			"ReferenceOverride.by":      "By is who wrote it.",
+			"ReferenceOverride.key":     "Key is the member this organisation is speaking about.",
+			"ReferenceOverride.note":    "Note is why, in the operator's own words. Optional, and bounded.",
+			"ReferenceOverride.verdict": "Verdict is allow or deny.",
+			"ReferenceSet.age":          "Age is how long ago that was.",
+			"ReferenceSet.asOf":         "AsOf is when the OLDEST contributing publisher was current, RFC 3339. The\noldest and not the newest: a set is exactly as fresh as its weakest source.",
+			"ReferenceSet.keys":         "Keys is how many members the baseline carries.",
+			"ReferenceSet.kind":         "Kind is how the baseline comes to exist: fetch (downloaded from a\npublisher), local (computed here), attest (held by the component that\nscreens against it, freshness reported), or seam (declared and NOT held,\nbecause the source needs a licence we do not have).",
+			"ReferenceSet.match":        "Match is how a key is tested: exact, domain, net, digits, pattern or range.",
+			"ReferenceSet.maxAge":       "MaxAge is how old this set may be before it is stale.",
+			"ReferenceSet.overrides":    "Overrides is how many entries YOUR org has laid over this baseline.",
+			"ReferenceSet.refusal":      "Refusal names why the set cannot be relied on, when it cannot: never\nloaded, held elsewhere, or a licence we do not hold. Non-empty means a\nlookup against this set will not answer, rather than answering clean.",
+			"ReferenceSet.set":          "Set is the name this set is addressed by.",
+			"ReferenceSet.sources":      "Sources is each contributing publisher, its licence and its own freshness.",
+			"ReferenceSet.stale":        "Stale is whether it is past that bound. A stale set still answers and says\nso, because yesterday's list beats none.",
+			"ReferenceSet.version":      "Version is the exact baseline consulted — every contributing publisher and\nits content digest. A decision records this and an auditor resolves it back.",
+			"ReferenceSet.what":         "What the set holds, in one sentence.",
+			"ReferenceSource.asOf":      "AsOf is when this publisher was current, RFC 3339.",
+			"ReferenceSource.basis":     "Basis is the KIND of permission this publisher's data reaches you under:\nlicence (an explicit grant), registry (the registry of record publishing for\nanyone to consult), operator (an operator's own machine-readable statement\nabout its own network, published for third parties to filter by — not a\nlicence, and not claimed as one), own (computed here), or none (nothing\nreaches you: the membership is held by the component that screens against\nit). It is on the wire so the licence position is an audit you can run.",
+			"ReferenceSource.keys":      "Keys is how many members this publisher contributed.",
+			"ReferenceSource.origin":    "Origin is exactly where it was taken from, so it can be taken again.",
+			"ReferenceSource.refusal":   "Refusal is why this publisher's last take failed, if it did. The set keeps\nits previous version of this source and ages out visibly rather than\nsilently shrinking.",
+			"ReferenceSource.source":    "Source is the publisher.",
+			"ReferenceSource.terms":     "Terms is the CITATION that basis points at — the licence identifier, the\nregistry, or the operator publication. A source with no stated terms is not\nin the catalog.",
+			"ReferenceSource.version":   "Version is the content digest of what this publisher last supplied. Two\nrefreshes that agree on it took the same data.",
+		},
+		Example: json.RawMessage(`{"set":"domain","limit":50}`),
+	})
+	zip.Describe("POST /v1/reference/refresh", zip.Doc{
+		Description: "Takes a new version of one set. SuperAdmin only.\n\nIt is platform work, not tenant work: it writes the shared baseline every\norganisation reads, so it is gated to the platform's own identity. Nothing\nhere can write an organisation's overrides, and nothing an organisation sends\ncan reach this route.\n\nIdempotent. A version is the content digest of what was taken, so refreshing\nan unchanged publisher writes no rows and reports unchanged. Resumable: a run\nthat died half-way is continued from where it stopped rather than restarted.\n\nA set whose source needs a licence we do not hold is refused with the reason,\nrather than being quietly skipped.",
+		Fields: map[string]string{
+			"ReferenceReceipt.asOf":       "AsOf is when the load happened, RFC 3339. Absent is dated on arrival, which\ncan only make the list look older than it is.",
+			"ReferenceReceipt.keys":       "Keys is how many designations that load carried. Zero from a publisher who\ndesignates somebody is a failed load wearing a successful one's clothes,\nand belongs in Refusal instead.",
+			"ReferenceReceipt.refusal":    "Refusal is why the load failed, when it did.",
+			"ReferenceReceipt.source":     "Source is the publisher this receipt is for.",
+			"ReferenceReceipt.version":    "Version is the digest of what that publisher supplied, so a refresh that\nchanged nothing can be told from a refresh that did not run.",
+			"ReferenceTaken.keys":         "Keys is how many members it carries.",
+			"ReferenceTaken.refusal":      "Refusal is why this publisher contributed nothing, if it did not. The set\nkeeps its previous version of this source rather than shrinking.",
+			"ReferenceTaken.resumed":      "Resumed is true when this run continued a version a previous run left\nhalf-landed.",
+			"ReferenceTaken.source":       "Source is the publisher.",
+			"ReferenceTaken.unchanged":    "Unchanged is true when the publisher's data was byte-for-byte the set we\nalready held.",
+			"ReferenceTaken.version":      "Version is the content digest that landed.",
+			"ReferenceTaken.wrote":        "Wrote is how many rows this run actually wrote. Zero with Unchanged means\nthe publisher served the same set again.",
+			"RefreshReferenceIn.force":    "Force accepts a take whose size moved past the change bound. A publisher\nserving a tenth or ten times its previous list is refused by default and the\nprevious version is left standing; this is the operator saying the change is\nreal. It cannot make an empty, truncated or unparseable take land — those are\nerrors, not magnitudes.",
+			"RefreshReferenceIn.receipts": "Receipts are supplied by the component that holds the membership, for a set\nof kind attest. They are refused on any other kind, and a set of kind attest\nis refused without them: this plane never invents a freshness it did not\nobserve.",
+			"RefreshReferenceIn.set":      "Set is the set to refresh.",
+			"RefreshReferenceOut.set":     "Set is the set refreshed.",
+			"RefreshReferenceOut.stale":   "Stale is whether it is STILL past its freshness bound after the refresh,\nwhich is what a publisher that has stopped answering looks like.",
+			"RefreshReferenceOut.took":    "Took is what each publisher contributed.",
+			"RefreshReferenceOut.version": "Version is the set's new composed version.",
+		},
+		Example: json.RawMessage(`{"set":"domain"}`),
+	})
+	zip.Describe("POST /v1/reference/resolve", zip.Doc{
+		Description: "Looks keys up against the reference plane.\n\nYour organisation's own overrides are consulted FIRST and win outright; the\nshared baseline answers everything they do not cover. Every answer names the\nversion that produced it, when that version was current and whether it is\nstale, so a decision can record exactly what it consulted.\n\nRead Refusal before reading Hit. A set that has never loaded, one held by the\ncomponent that screens against it, and one whose source needs a licence we do\nnot hold all answer with a refusal — and a miss on a refusing set means\nnothing is known, not that the key is clean.",
+		Fields: map[string]string{
+			"ReferenceAnswer.age":           "Age is how old that is, as a duration.",
+			"ReferenceAnswer.asOf":          "AsOf is when the oldest contributing publisher was current, RFC 3339.",
+			"ReferenceAnswer.from":          "From is override or baseline — which plane answered.",
+			"ReferenceAnswer.hit":           "Hit is whether the key is a member. It is meaningful ONLY when Refusal is\nempty: false with a refusal means the set could not be consulted, which is\nnot the same as the key being clean.",
+			"ReferenceAnswer.key":           "Key is the key as asked.",
+			"ReferenceAnswer.matched":       "Matched is the member that covered the key, which for a domain or a network\nis the enclosing entry rather than the key itself.",
+			"ReferenceAnswer.refusal":       "Refusal is why the set could not be consulted, when it could not: never\nloaded, held elsewhere, or a source we hold no licence for. Non-empty means\nHit must not be read as an answer.",
+			"ReferenceAnswer.score":         "Score is the published risk weight where the source expresses one.",
+			"ReferenceAnswer.set":           "Set is the set consulted.",
+			"ReferenceAnswer.stale":         "Stale is whether the set is past its freshness bound. A stale set still\nanswers — yesterday's list beats none — and this is how a decision knows it\nleaned on one.",
+			"ReferenceAnswer.value":         "Value is what the publisher says about the member — class, operator,\nscheme, region.",
+			"ReferenceAnswer.verdict":       "Verdict is the tenant's own allow or deny, present only for an override.\nThe baseline never carries one: it states facts and leaves the decision to\nthe caller's policy.",
+			"ReferenceAnswer.version":       "Version is the exact baseline version consulted, composed of each\ncontributing publisher and its content digest. It is what makes a decision\nreproducible: an auditor takes this string and knows precisely what was\nconsulted.",
+			"ReferenceVersion.asOf":         "AsOf is when the oldest of them was current, RFC 3339.",
+			"ReferenceVersion.refusal":      "Refusal is why it could not be consulted, when it could not.",
+			"ReferenceVersion.set":          "Set is the set.",
+			"ReferenceVersion.stale":        "Stale is whether it is past its freshness bound.",
+			"ReferenceVersion.version":      "Version is every contributing publisher and its content digest.",
+			"ResolveReferenceIn.keys":       "Keys are the values to look up, at most 100 per call: email addresses or\ndomains, IP addresses, card prefixes, user-agent strings, autonomous system\nnumbers, device digests.",
+			"ResolveReferenceIn.sets":       "Sets narrows which sets to consult. Empty consults every set whose matcher\ncan read the keys given.",
+			"ResolveReferenceOut.answers":   "Answers is one entry per (set, key) consulted.",
+			"ResolveReferenceOut.consulted": "Consulted names the version of every set that took part, so a decision can\nrecord precisely what it leaned on. Record this with the decision: it is\nwhat makes the decision reproducible a year later.",
+			"ResolveReferenceOut.refused":   "Refused names the consulted sets that could not answer at all. A key that\nmissed in one of these is UNKNOWN, not clean.",
+			"ResolveReferenceOut.stale":     "Stale names the consulted sets past their freshness bound. Staleness is\nitself a risk signal — a decision taken against a three-week-old list is a\nweaker decision, and this is how it knows.",
+		},
+		Example: json.RawMessage(`{"sets":["domain","net"],"keys":["user@tempbox.example","3.5.140.1"]}`),
+	})
+	zip.Describe("PUT /v1/reference/:set", zip.Doc{
+		Description: "Writes your organisation's own allow and deny entries over a set.\n\nIdempotent on the key: writing the same entry twice is one entry, and writing\nit again replaces the verdict and the note. The whole batch is one\ntransaction, so a batch that would cross the per-set bound writes nothing\nrather than half of itself — a half-applied deny list is worse than a refused\none, because nobody can tell which half applied.\n\nYour entries are held in your organisation's own store and are never visible\nto another organisation, and they never change what any other organisation\nsees. The shared baseline is not writable from here at all.",
+		Fields: map[string]string{
+			"ReferenceOverrideIn.key":     "Key is the member: a domain, a CIDR or address, an issuer prefix, a\ndevice digest. It is matched the same way the baseline is, so a deny on\ntempbox.example also covers mail.tempbox.example.",
+			"ReferenceOverrideIn.note":    "Note is why, in your own words. Optional, bounded to 512 bytes.",
+			"ReferenceOverrideIn.verdict": "Verdict is allow or deny, and nothing else. An override is a decision —\nunlike a baseline entry, which states facts and leaves the decision to your\npolicy — because your organisation is the only party entitled to say \"for\nus, this one is fine\".",
+			"SetReferenceIn.entries":      "Entries are the overrides to write, up to 1000 per call.",
+			"SetReferenceOut.overrides":   "Overrides is how many your org now holds in this set.",
+			"SetReferenceOut.set":         "Set is the set written in.",
+			"SetReferenceOut.written":     "Written is how many entries this call wrote.",
+		},
+		Example: json.RawMessage(`{"set":"domain","entries":[{"key":"partner.example","verdict":"allow","note":"our reseller"}]}`),
+	})
+}

manifest/apps.go

@@ -137,6 +137,19 @@ var Apps = []App{
 	// two owners for one prefix at compose time, which checks it rather than
 	// trusting it.
 	{Name: "dataset", Prefixes: []string{"/v1/ml/datasets"}},
+	// reference is the LOOKUP DATA a decision consults — throwaway-inbox domains,
+	// hosting and Tor ranges, issuer-prefix structure, and how current the
+	// designation lists are. A separate row from risk because it is separate state:
+	// risk holds one model per organisation in memory, this holds versioned sets
+	// every organisation reads the same copy of, and neither can be rebuilt from the
+	// other.
+	//
+	// /v1/reference, NOT /v1/ml/reference. It was written as a leaf under /v1/ml,
+	// which would have put six reference operations inside the model-SERVING
+	// product's prefix and its OpenAPI tag — the same collision the row above
+	// records having already been moved off /v1/ml to escape. A prefix that has to
+	// be explained twice was the wrong prefix once.
+	{Name: "reference", Prefixes: []string{"/v1/reference"}},
 	{Name: "usage", Prefixes: []string{"/v1/usage"}},
 	{Name: "leaderboard", Prefixes: []string{"/v1/usage/activity", "/v1/usage/leaderboard", "/v1/usage/rollup/backfill"}},
 	{Name: "crm", Prefixes: []string{"/v1/crm"}},

manifest/order_test.go

@@ -23,7 +23,7 @@ var frozen = []string{
 	"dns", "domain", "prompts", "agents", "link", "wallets",
 	"x402", "deploy", "functions", "tracker", "templates", "blueprint",
 	"framework", "knowledge", "help", "content", "catalogsync", "webhooks",
-	"ml", "risk", "dataset", "usage", "leaderboard", "crm", "marketing", "ads",
+	"ml", "risk", "dataset", "reference", "usage", "leaderboard", "crm", "marketing", "ads",
 	"campaign", "validators", "social", "analytics", "git", "sync",
 	"visor", "venue", "captable", "code", "zt", "share",
 	"dataroom", "graph", "security", "integrations", "destinations", "cloudflare",

openapi.yaml

@@ -1096,6 +1096,23 @@ components:
         symbol:
           type: string
       type: object
+    ClearReferenceOut:
+      properties:
+        cleared:
+          description: |-
+            Cleared is false when your org held no such override — which is not an
+            error, it is the honest answer to a removal that had nothing to remove.
+          type: boolean
+        key:
+          description: Key is the entry named.
+          type: string
+        overrides:
+          description: Overrides is how many your org still holds in this set.
+          type: integer
+        set:
+          description: Set is the set cleared in.
+          type: string
+      type: object
     ClipBody:
       properties:
         bytes:
@@ -4614,6 +4631,366 @@ components:
         seats:
           type: integer
       type: object
+    ReferenceAnswer:
+      properties:
+        age:
+          description: Age is how old that is, as a duration.
+          type: string
+        asOf:
+          description: AsOf is when the oldest contributing publisher was current,
+            RFC 3339.
+          type: string
+        from:
+          description: From is override or baseline — which plane answered.
+          type: string
+        hit:
+          description: |-
+            Hit is whether the key is a member. It is meaningful ONLY when Refusal is
+            empty: false with a refusal means the set could not be consulted, which is
+            not the same as the key being clean.
+          type: boolean
+        key:
+          description: Key is the key as asked.
+          type: string
+        matched:
+          description: |-
+            Matched is the member that covered the key, which for a domain or a network
+            is the enclosing entry rather than the key itself.
+          type: string
+        refusal:
+          description: |-
+            Refusal is why the set could not be consulted, when it could not: never
+            loaded, held elsewhere, or a source we hold no licence for. Non-empty means
+            Hit must not be read as an answer.
+          type: string
+        score:
+          description: Score is the published risk weight where the source expresses
+            one.
+          type: number
+        set:
+          description: Set is the set consulted.
+          type: string
+        stale:
+          description: |-
+            Stale is whether the set is past its freshness bound. A stale set still
+            answers — yesterday's list beats none — and this is how a decision knows it
+            leaned on one.
+          type: boolean
+        value:
+          additionalProperties:
+            type: string
+          description: |-
+            Value is what the publisher says about the member — class, operator,
+            scheme, region.
+          type: object
+        verdict:
+          description: |-
+            Verdict is the tenant's own allow or deny, present only for an override.
+            The baseline never carries one: it states facts and leaves the decision to
+            the caller's policy.
+          type: string
+        version:
+          description: |-
+            Version is the exact baseline version consulted, composed of each
+            contributing publisher and its content digest. It is what makes a decision
+            reproducible: an auditor takes this string and knows precisely what was
+            consulted.
+          type: string
+      type: object
+    ReferenceOut:
+      properties:
+        next:
+          description: Next is the key to page from, empty when this is the last page.
+          type: string
+        overrides:
+          description: |-
+            Overrides is YOUR org's entries over that baseline, in key order. They are
+            held in your organisation's own store and are not visible to any other.
+          items:
+            $ref: '#/components/schemas/ReferenceOverride'
+          type: array
+        set:
+          $ref: '#/components/schemas/ReferenceSet'
+          description: 'Set is the published set: its version, its freshness and its
+            sources.'
+      type: object
+    ReferenceOverride:
+      properties:
+        at:
+          description: At is when it was written, RFC 3339.
+          type: string
+        by:
+          description: By is who wrote it.
+          type: string
+        key:
+          description: Key is the member this organisation is speaking about.
+          type: string
+        note:
+          description: Note is why, in the operator's own words. Optional, and bounded.
+          type: string
+        verdict:
+          description: Verdict is allow or deny.
+          type: string
+      type: object
+    ReferenceOverrideIn:
+      properties:
+        key:
+          description: |-
+            Key is the member: a domain, a CIDR or address, an issuer prefix, a
+            device digest. It is matched the same way the baseline is, so a deny on
+            tempbox.example also covers mail.tempbox.example.
+          type: string
+        note:
+          description: Note is why, in your own words. Optional, bounded to 512 bytes.
+          type: string
+        verdict:
+          description: |-
+            Verdict is allow or deny, and nothing else. An override is a decision —
+            unlike a baseline entry, which states facts and leaves the decision to your
+            policy — because your organisation is the only party entitled to say "for
+            us, this one is fine".
+          type: string
+      type: object
+    ReferenceReceipt:
+      properties:
+        asOf:
+          description: |-
+            AsOf is when the load happened, RFC 3339. Absent is dated on arrival, which
+            can only make the list look older than it is.
+          type: string
+        keys:
+          description: |-
+            Keys is how many designations that load carried. Zero from a publisher who
+            designates somebody is a failed load wearing a successful one's clothes,
+            and belongs in Refusal instead.
+          type: integer
+        refusal:
+          description: Refusal is why the load failed, when it did.
+          type: string
+        source:
+          description: Source is the publisher this receipt is for.
+          type: string
+        version:
+          description: |-
+            Version is the digest of what that publisher supplied, so a refresh that
+            changed nothing can be told from a refresh that did not run.
+          type: string
+      type: object
+    ReferenceSet:
+      properties:
+        age:
+          description: Age is how long ago that was.
+          type: string
+        asOf:
+          description: |-
+            AsOf is when the OLDEST contributing publisher was current, RFC 3339. The
+            oldest and not the newest: a set is exactly as fresh as its weakest source.
+          type: string
+        keys:
+          description: Keys is how many members the baseline carries.
+          type: integer
+        kind:
+          description: |-
+            Kind is how the baseline comes to exist: fetch (downloaded from a
+            publisher), local (computed here), attest (held by the component that
+            screens against it, freshness reported), or seam (declared and NOT held,
+            because the source needs a licence we do not have).
+          type: string
+        match:
+          description: 'Match is how a key is tested: exact, domain, net, digits,
+            pattern or range.'
+          type: string
+        maxAge:
+          description: MaxAge is how old this set may be before it is stale.
+          type: string
+        overrides:
+          description: Overrides is how many entries YOUR org has laid over this baseline.
+          type: integer
+        refusal:
+          description: |-
+            Refusal names why the set cannot be relied on, when it cannot: never
+            loaded, held elsewhere, or a licence we do not hold. Non-empty means a
+            lookup against this set will not answer, rather than answering clean.
+          type: string
+        set:
+          description: Set is the name this set is addressed by.
+          type: string
+        sources:
+          description: Sources is each contributing publisher, its licence and its
+            own freshness.
+          items:
+            $ref: '#/components/schemas/ReferenceSource'
+          type: array
+        stale:
+          description: |-
+            Stale is whether it is past that bound. A stale set still answers and says
+            so, because yesterday's list beats none.
+          type: boolean
+        version:
+          description: |-
+            Version is the exact baseline consulted — every contributing publisher and
+            its content digest. A decision records this and an auditor resolves it back.
+          type: string
+        what:
+          description: What the set holds, in one sentence.
+          type: string
+      type: object
+    ReferenceSetsOut:
+      properties:
+        refused:
+          description: |-
+            Refused names the sets that cannot be consulted at all. A key checked
+            against one of these is UNKNOWN, not clean.
+          items:
+            type: string
+          type: array
+        sets:
+          description: Sets is the whole catalog, in a stable order.
+          items:
+            $ref: '#/components/schemas/ReferenceSet'
+          type: array
+        stale:
+          description: Stale names the sets past their freshness bound — the list
+            to alarm on.
+          items:
+            type: string
+          type: array
+      type: object
+    ReferenceSource:
+      properties:
+        asOf:
+          description: AsOf is when this publisher was current, RFC 3339.
+          type: string
+        basis:
+          description: |-
+            Basis is the KIND of permission this publisher's data reaches you under:
+            licence (an explicit grant), registry (the registry of record publishing for
+            anyone to consult), operator (an operator's own machine-readable statement
+            about its own network, published for third parties to filter by — not a
+            licence, and not claimed as one), own (computed here), or none (nothing
+            reaches you: the membership is held by the component that screens against
+            it). It is on the wire so the licence position is an audit you can run.
+          type: string
+        keys:
+          description: Keys is how many members this publisher contributed.
+          type: integer
+        origin:
+          description: Origin is exactly where it was taken from, so it can be taken
+            again.
+          type: string
+        refusal:
+          description: |-
+            Refusal is why this publisher's last take failed, if it did. The set keeps
+            its previous version of this source and ages out visibly rather than
+            silently shrinking.
+          type: string
+        source:
+          description: Source is the publisher.
+          type: string
+        terms:
+          description: |-
+            Terms is the CITATION that basis points at — the licence identifier, the
+            registry, or the operator publication. A source with no stated terms is not
+            in the catalog.
+          type: string
+        version:
+          description: |-
+            Version is the content digest of what this publisher last supplied. Two
+            refreshes that agree on it took the same data.
+          type: string
+      type: object
+    ReferenceTaken:
+      properties:
+        keys:
+          description: Keys is how many members it carries.
+          type: integer
+        refusal:
+          description: |-
+            Refusal is why this publisher contributed nothing, if it did not. The set
+            keeps its previous version of this source rather than shrinking.
+          type: string
+        resumed:
+          description: |-
+            Resumed is true when this run continued a version a previous run left
+            half-landed.
+          type: boolean
+        source:
+          description: Source is the publisher.
+          type: string
+        unchanged:
+          description: |-
+            Unchanged is true when the publisher's data was byte-for-byte the set we
+            already held.
+          type: boolean
+        version:
+          description: Version is the content digest that landed.
+          type: string
+        wrote:
+          description: |-
+            Wrote is how many rows this run actually wrote. Zero with Unchanged means
+            the publisher served the same set again.
+          type: integer
+      type: object
+    ReferenceVersion:
+      properties:
+        asOf:
+          description: AsOf is when the oldest of them was current, RFC 3339.
+          type: string
+        refusal:
+          description: Refusal is why it could not be consulted, when it could not.
+          type: string
+        set:
+          description: Set is the set.
+          type: string
+        stale:
+          description: Stale is whether it is past its freshness bound.
+          type: boolean
+        version:
+          description: Version is every contributing publisher and its content digest.
+          type: string
+      type: object
+    RefreshReferenceIn:
+      properties:
+        force:
+          description: |-
+            Force accepts a take whose size moved past the change bound. A publisher
+            serving a tenth or ten times its previous list is refused by default and the
+            previous version is left standing; this is the operator saying the change is
+            real. It cannot make an empty, truncated or unparseable take land — those are
+            errors, not magnitudes.
+          type: boolean
+        receipts:
+          description: |-
+            Receipts are supplied by the component that holds the membership, for a set
+            of kind attest. They are refused on any other kind, and a set of kind attest
+            is refused without them: this plane never invents a freshness it did not
+            observe.
+          items:
+            $ref: '#/components/schemas/ReferenceReceipt'
+          type: array
+        set:
+          description: Set is the set to refresh.
+          type: string
+      type: object
+    RefreshReferenceOut:
+      properties:
+        set:
+          description: Set is the set refreshed.
+          type: string
+        stale:
+          description: |-
+            Stale is whether it is STILL past its freshness bound after the refresh,
+            which is what a publisher that has stopped answering looks like.
+          type: boolean
+        took:
+          description: Took is what each publisher contributed.
+          items:
+            $ref: '#/components/schemas/ReferenceTaken'
+          type: array
+        version:
+          description: Version is the set's new composed version.
+          type: string
+      type: object
     Registration:
       properties:
         createdAt:
@@ -4693,6 +5070,55 @@ components:
         projects:
           type: integer
       type: object
+    ResolveReferenceIn:
+      properties:
+        keys:
+          description: |-
+            Keys are the values to look up, at most 100 per call: email addresses or
+            domains, IP addresses, card prefixes, user-agent strings, autonomous system
+            numbers, device digests.
+          items:
+            type: string
+          type: array
+        sets:
+          description: |-
+            Sets narrows which sets to consult. Empty consults every set whose matcher
+            can read the keys given.
+          items:
+            type: string
+          type: array
+      type: object
+    ResolveReferenceOut:
+      properties:
+        answers:
+          description: Answers is one entry per (set, key) consulted.
+          items:
+            $ref: '#/components/schemas/ReferenceAnswer'
+          type: array
+        consulted:
+          description: |-
+            Consulted names the version of every set that took part, so a decision can
+            record precisely what it leaned on. Record this with the decision: it is
+            what makes the decision reproducible a year later.
+          items:
+            $ref: '#/components/schemas/ReferenceVersion'
+          type: array
+        refused:
+          description: |-
+            Refused names the consulted sets that could not answer at all. A key that
+            missed in one of these is UNKNOWN, not clean.
+          items:
+            type: string
+          type: array
+        stale:
+          description: |-
+            Stale names the consulted sets past their freshness bound. Staleness is
+            itself a risk signal — a decision taken against a three-week-old list is a
+            weaker decision, and this is how it knows.
+          items:
+            type: string
+          type: array
+      type: object
     Result:
       properties:
         host:
@@ -5300,6 +5726,26 @@ components:
         waitlistMode:
           type: boolean
       type: object
+    SetReferenceIn:
+      properties:
+        entries:
+          description: Entries are the overrides to write, up to 1000 per call.
+          items:
+            $ref: '#/components/schemas/ReferenceOverrideIn'
+          type: array
+      type: object
+    SetReferenceOut:
+      properties:
+        overrides:
+          description: Overrides is how many your org now holds in this set.
+          type: integer
+        set:
+          description: Set is the set written in.
+          type: string
+        written:
+          description: Written is how many entries this call wrote.
+          type: integer
+      type: object
     SharePolicy:
       properties:
         revenueShareBps:
@@ -85681,6 +86127,222 @@ paths:
       tags:
       - rag
       x-app: github.com/hanzoai/ai
+  /v1/reference:
+    get:
+      description: |-
+        Lists every set this plane publishes, with its version and how
+        fresh it is.
+
+        Read the Stale and Refused lists first: they are the two ways this plane can
+        be quietly wrong, and they are reported rather than inferred. A set in
+        Refused answers nothing — it has never loaded, it is held by another
+        component, or it names a source we hold no licence for.
+      operationId: referenceSets
+      responses:
+        "200":
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/ReferenceSetsOut'
+          description: ok
+      summary: Lists every set this plane publishes, with its version and how fresh
+        it is.
+      tags:
+      - reference
+      x-app: reference
+  /v1/reference/{set}:
+    delete:
+      description: |-
+        Removes one of your organisation's overrides.
+
+        It removes an entry your organisation wrote, never a baseline member: the
+        published set is not writable from here, so a removal can only ever restore
+        the baseline's own answer.
+      operationId: referenceClear
+      parameters:
+      - example: domain
+        in: path
+        name: set
+        required: true
+        schema:
+          type: string
+      - example: partner.example
+        in: query
+        name: key
+        required: false
+        schema:
+          type: string
+      responses:
+        "200":
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/ClearReferenceOut'
+          description: ok
+      summary: Removes one of your organisation's overrides.
+      tags:
+      - reference
+      x-app: reference
+    get:
+      description: |-
+        Reference describes one set and lists your org's overrides in it.
+
+        The set half is public data about a published list — its version, its
+        publishers, their licences and how current each one is. The overrides half is
+        yours alone: it is read from your organisation's own store, and no other
+        organisation's entries can appear in it.
+      operationId: referenceSet
+      parameters:
+      - example: domain
+        in: path
+        name: set
+        required: true
+        schema:
+          type: string
+      - description: 'After pages the override listing: the last key of the previous
+          page.'
+        in: query
+        name: after
+        required: false
+        schema:
+          type: string
+      - description: 'Limit caps the override listing: default 200, maximum 1000.'
+        example: 50
+        in: query
+        name: limit
+        required: false
+        schema:
+          type: integer
+      responses:
+        "200":
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/ReferenceOut'
+          description: ok
+      summary: Reference describes one set and lists your org's overrides in it.
+      tags:
+      - reference
+      x-app: reference
+    put:
+      description: |-
+        Writes your organisation's own allow and deny entries over a set.
+
+        Idempotent on the key: writing the same entry twice is one entry, and writing
+        it again replaces the verdict and the note. The whole batch is one
+        transaction, so a batch that would cross the per-set bound writes nothing
+        rather than half of itself — a half-applied deny list is worse than a refused
+        one, because nobody can tell which half applied.
+
+        Your entries are held in your organisation's own store and are never visible
+        to another organisation, and they never change what any other organisation
+        sees. The shared baseline is not writable from here at all.
+      operationId: referenceOverride
+      parameters:
+      - example: domain
+        in: path
+        name: set
+        required: true
+        schema:
+          type: string
+      requestBody:
+        content:
+          application/json:
+            example:
+              entries:
+              - key: partner.example
+                note: our reseller
+                verdict: allow
+              set: domain
+            schema:
+              $ref: '#/components/schemas/SetReferenceIn'
+        required: true
+      responses:
+        "200":
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/SetReferenceOut'
+          description: ok
+      summary: Writes your organisation's own allow and deny entries over a set.
+      tags:
+      - reference
+      x-app: reference
+  /v1/reference/refresh:
+    post:
+      description: |-
+        Takes a new version of one set. SuperAdmin only.
+
+        It is platform work, not tenant work: it writes the shared baseline every
+        organisation reads, so it is gated to the platform's own identity. Nothing
+        here can write an organisation's overrides, and nothing an organisation sends
+        can reach this route.
+
+        Idempotent. A version is the content digest of what was taken, so refreshing
+        an unchanged publisher writes no rows and reports unchanged. Resumable: a run
+        that died half-way is continued from where it stopped rather than restarted.
+
+        A set whose source needs a licence we do not hold is refused with the reason,
+        rather than being quietly skipped.
+      operationId: referenceRefresh
+      requestBody:
+        content:
+          application/json:
+            example:
+              set: domain
+            schema:
+              $ref: '#/components/schemas/RefreshReferenceIn'
+        required: true
+      responses:
+        "200":
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/RefreshReferenceOut'
+          description: ok
+      summary: Takes a new version of one set.
+      tags:
+      - reference
+      x-app: reference
+  /v1/reference/resolve:
+    post:
+      description: |-
+        Looks keys up against the reference plane.
+
+        Your organisation's own overrides are consulted FIRST and win outright; the
+        shared baseline answers everything they do not cover. Every answer names the
+        version that produced it, when that version was current and whether it is
+        stale, so a decision can record exactly what it consulted.
+
+        Read Refusal before reading Hit. A set that has never loaded, one held by the
+        component that screens against it, and one whose source needs a licence we do
+        not hold all answer with a refusal — and a miss on a refusing set means
+        nothing is known, not that the key is clean.
+      operationId: referenceResolve
+      requestBody:
+        content:
+          application/json:
+            example:
+              keys:
+              - user@tempbox.example
+              - 3.5.140.1
+              sets:
+              - domain
+              - net
+            schema:
+              $ref: '#/components/schemas/ResolveReferenceIn'
+        required: true
+      responses:
+        "200":
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/ResolveReferenceOut'
+          description: ok
+      summary: Looks keys up against the reference plane.
+      tags:
+      - reference
+      x-app: reference
   /v1/referrals:
     get:
       description: |-
@@ -94901,6 +95563,12 @@ tags:
     a cloud binary with the money, ingest and telemetry callbacks cloud BUILDS but
     cannot INSTALL.
   name: rag
+- description: 'Package reference is the lookup data a risk decision needs but cannot
+    derive: which email domains hand out throwaway inboxes, which addresses belong
+    to a datacentre or a Tor exit, which card scheme an issuer prefix belongs to,
+    which browsers the fleet sees everywhere, and how current the designation lists
+    the screening engine holds actually are.'
+  name: reference
 - description: Package referrals is credit for both sides when someone you refer actually
     spends.
   name: referrals

openapi/floor.json

@@ -1,6 +1,6 @@
 {
-  "paths": 1683,
-  "operations": 2335,
+  "paths": 1687,
+  "operations": 2341,
   "products": {
     "admin": 87,
     "ads": 7,
@@ -135,6 +135,7 @@
     "query": 1,
     "query_multiple": 1,
     "rag": 5,
+    "reference": 6,
     "referrals": 2,
     "registry": 6,
     "releases": 1,

plugin/reference/main.go

@@ -0,0 +1,28 @@
+package main
+
+import (
+	"fmt"
+	"os"
+
+	"github.com/hanzoai/cloud"
+	"github.com/hanzoai/cloud/apps/reference"
+)
+
+// Standalone entry for the reference app.
+//
+// This is the app's OWN composition root: it links only its own subsystem and
+// the cloud request tier, never the whole fleet, so the build is this one app
+// and not the union the fused binary was. The light host loads it as a plugin;
+// run directly it serves standalone. Its OpenAPI subset comes from
+// `reference openapi`. Hand-owned — edit the spec below directly.
+func main() {
+	if err := cloud.Listen([]cloud.Plugin{{
+		Name:     "reference",
+		Price:    cloud.Free,
+		Mount:    reference.Mount,
+		Shutdown: cloud.CtxShutdown(reference.Shutdown),
+	}}, []string{"reference"}); err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+}

plugin/reference/openapi.json

@@ -0,0 +1,740 @@
+{
+  "openapi": "3.1.0",
+  "info": {
+    "title": "Hanzo Cloud API",
+    "description": "Package reference is the lookup data a risk decision needs but cannot derive: which email domains hand out throwaway inboxes, which addresses belong to a datacentre or a Tor exit, which card scheme an issuer prefix belongs to, which browsers the fleet sees everywhere, and how current the designation lists the screening engine holds actually are.",
+    "version": "v1"
+  },
+  "servers": [
+    {
+      "url": "https://api.hanzo.ai"
+    }
+  ],
+  "tags": [
+    {
+      "name": "reference"
+    }
+  ],
+  "paths": {
+    "/v1/reference": {
+      "get": {
+        "operationId": "referenceSets",
+        "summary": "Lists every set this plane publishes, with its version and how fresh it is.",
+        "description": "Lists every set this plane publishes, with its version and how\nfresh it is.\n\nRead the Stale and Refused lists first: they are the two ways this plane can\nbe quietly wrong, and they are reported rather than inferred. A set in\nRefused answers nothing — it has never loaded, it is held by another\ncomponent, or it names a source we hold no licence for.",
+        "tags": [
+          "reference"
+        ],
+        "responses": {
+          "200": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/ReferenceSetsOut"
+                }
+              }
+            },
+            "description": "ok"
+          }
+        }
+      }
+    },
+    "/v1/reference/refresh": {
+      "post": {
+        "operationId": "referenceRefresh",
+        "summary": "Takes a new version of one set.",
+        "description": "Takes a new version of one set. SuperAdmin only.\n\nIt is platform work, not tenant work: it writes the shared baseline every\norganisation reads, so it is gated to the platform's own identity. Nothing\nhere can write an organisation's overrides, and nothing an organisation sends\ncan reach this route.\n\nIdempotent. A version is the content digest of what was taken, so refreshing\nan unchanged publisher writes no rows and reports unchanged. Resumable: a run\nthat died half-way is continued from where it stopped rather than restarted.\n\nA set whose source needs a licence we do not hold is refused with the reason,\nrather than being quietly skipped.",
+        "tags": [
+          "reference"
+        ],
+        "requestBody": {
+          "content": {
+            "application/json": {
+              "example": {
+                "set": "domain"
+              },
+              "schema": {
+                "$ref": "#/components/schemas/RefreshReferenceIn"
+              }
+            }
+          },
+          "required": true
+        },
+        "responses": {
+          "200": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/RefreshReferenceOut"
+                }
+              }
+            },
+            "description": "ok"
+          }
+        }
+      }
+    },
+    "/v1/reference/resolve": {
+      "post": {
+        "operationId": "referenceResolve",
+        "summary": "Looks keys up against the reference plane.",
+        "description": "Looks keys up against the reference plane.\n\nYour organisation's own overrides are consulted FIRST and win outright; the\nshared baseline answers everything they do not cover. Every answer names the\nversion that produced it, when that version was current and whether it is\nstale, so a decision can record exactly what it consulted.\n\nRead Refusal before reading Hit. A set that has never loaded, one held by the\ncomponent that screens against it, and one whose source needs a licence we do\nnot hold all answer with a refusal — and a miss on a refusing set means\nnothing is known, not that the key is clean.",
+        "tags": [
+          "reference"
+        ],
+        "requestBody": {
+          "content": {
+            "application/json": {
+              "example": {
+                "keys": [
+                  "user@tempbox.example",
+                  "3.5.140.1"
+                ],
+                "sets": [
+                  "domain",
+                  "net"
+                ]
+              },
+              "schema": {
+                "$ref": "#/components/schemas/ResolveReferenceIn"
+              }
+            }
+          },
+          "required": true
+        },
+        "responses": {
+          "200": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/ResolveReferenceOut"
+                }
+              }
+            },
+            "description": "ok"
+          }
+        }
+      }
+    },
+    "/v1/reference/{set}": {
+      "delete": {
+        "operationId": "referenceClear",
+        "summary": "Removes one of your organisation's overrides.",
+        "description": "Removes one of your organisation's overrides.\n\nIt removes an entry your organisation wrote, never a baseline member: the\npublished set is not writable from here, so a removal can only ever restore\nthe baseline's own answer.",
+        "tags": [
+          "reference"
+        ],
+        "parameters": [
+          {
+            "name": "set",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string"
+            },
+            "example": "domain"
+          },
+          {
+            "name": "key",
+            "in": "query",
+            "required": false,
+            "schema": {
+              "type": "string"
+            },
+            "example": "partner.example"
+          }
+        ],
+        "responses": {
+          "200": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/ClearReferenceOut"
+                }
+              }
+            },
+            "description": "ok"
+          }
+        }
+      },
+      "get": {
+        "operationId": "referenceSet",
+        "summary": "Reference describes one set and lists your org's overrides in it.",
+        "description": "Reference describes one set and lists your org's overrides in it.\n\nThe set half is public data about a published list — its version, its\npublishers, their licences and how current each one is. The overrides half is\nyours alone: it is read from your organisation's own store, and no other\norganisation's entries can appear in it.",
+        "tags": [
+          "reference"
+        ],
+        "parameters": [
+          {
+            "name": "set",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string"
+            },
+            "example": "domain"
+          },
+          {
+            "name": "after",
+            "in": "query",
+            "required": false,
+            "description": "After pages the override listing: the last key of the previous page.",
+            "schema": {
+              "type": "string"
+            }
+          },
+          {
+            "name": "limit",
+            "in": "query",
+            "required": false,
+            "description": "Limit caps the override listing: default 200, maximum 1000.",
+            "schema": {
+              "type": "integer"
+            },
+            "example": 50
+          }
+        ],
+        "responses": {
+          "200": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/ReferenceOut"
+                }
+              }
+            },
+            "description": "ok"
+          }
+        }
+      },
+      "put": {
+        "operationId": "referenceOverride",
+        "summary": "Writes your organisation's own allow and deny entries over a set.",
+        "description": "Writes your organisation's own allow and deny entries over a set.\n\nIdempotent on the key: writing the same entry twice is one entry, and writing\nit again replaces the verdict and the note. The whole batch is one\ntransaction, so a batch that would cross the per-set bound writes nothing\nrather than half of itself — a half-applied deny list is worse than a refused\none, because nobody can tell which half applied.\n\nYour entries are held in your organisation's own store and are never visible\nto another organisation, and they never change what any other organisation\nsees. The shared baseline is not writable from here at all.",
+        "tags": [
+          "reference"
+        ],
+        "parameters": [
+          {
+            "name": "set",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string"
+            },
+            "example": "domain"
+          }
+        ],
+        "requestBody": {
+          "content": {
+            "application/json": {
+              "example": {
+                "entries": [
+                  {
+                    "key": "partner.example",
+                    "note": "our reseller",
+                    "verdict": "allow"
+                  }
+                ],
+                "set": "domain"
+              },
+              "schema": {
+                "$ref": "#/components/schemas/SetReferenceIn"
+              }
+            }
+          },
+          "required": true
+        },
+        "responses": {
+          "200": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/SetReferenceOut"
+                }
+              }
+            },
+            "description": "ok"
+          }
+        }
+      }
+    }
+  },
+  "components": {
+    "schemas": {
+      "ClearReferenceOut": {
+        "properties": {
+          "cleared": {
+            "description": "Cleared is false when your org held no such override — which is not an\nerror, it is the honest answer to a removal that had nothing to remove.",
+            "type": "boolean"
+          },
+          "key": {
+            "description": "Key is the entry named.",
+            "type": "string"
+          },
+          "overrides": {
+            "description": "Overrides is how many your org still holds in this set.",
+            "type": "integer"
+          },
+          "set": {
+            "description": "Set is the set cleared in.",
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "ReferenceAnswer": {
+        "properties": {
+          "age": {
+            "description": "Age is how old that is, as a duration.",
+            "type": "string"
+          },
+          "asOf": {
+            "description": "AsOf is when the oldest contributing publisher was current, RFC 3339.",
+            "type": "string"
+          },
+          "from": {
+            "description": "From is override or baseline — which plane answered.",
+            "type": "string"
+          },
+          "hit": {
+            "description": "Hit is whether the key is a member. It is meaningful ONLY when Refusal is\nempty: false with a refusal means the set could not be consulted, which is\nnot the same as the key being clean.",
+            "type": "boolean"
+          },
+          "key": {
+            "description": "Key is the key as asked.",
+            "type": "string"
+          },
+          "matched": {
+            "description": "Matched is the member that covered the key, which for a domain or a network\nis the enclosing entry rather than the key itself.",
+            "type": "string"
+          },
+          "refusal": {
+            "description": "Refusal is why the set could not be consulted, when it could not: never\nloaded, held elsewhere, or a source we hold no licence for. Non-empty means\nHit must not be read as an answer.",
+            "type": "string"
+          },
+          "score": {
+            "description": "Score is the published risk weight where the source expresses one.",
+            "type": "number"
+          },
+          "set": {
+            "description": "Set is the set consulted.",
+            "type": "string"
+          },
+          "stale": {
+            "description": "Stale is whether the set is past its freshness bound. A stale set still\nanswers — yesterday's list beats none — and this is how a decision knows it\nleaned on one.",
+            "type": "boolean"
+          },
+          "value": {
+            "additionalProperties": {
+              "type": "string"
+            },
+            "description": "Value is what the publisher says about the member — class, operator,\nscheme, region.",
+            "type": "object"
+          },
+          "verdict": {
+            "description": "Verdict is the tenant's own allow or deny, present only for an override.\nThe baseline never carries one: it states facts and leaves the decision to\nthe caller's policy.",
+            "type": "string"
+          },
+          "version": {
+            "description": "Version is the exact baseline version consulted, composed of each\ncontributing publisher and its content digest. It is what makes a decision\nreproducible: an auditor takes this string and knows precisely what was\nconsulted.",
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "ReferenceOut": {
+        "properties": {
+          "next": {
+            "description": "Next is the key to page from, empty when this is the last page.",
+            "type": "string"
+          },
+          "overrides": {
+            "description": "Overrides is YOUR org's entries over that baseline, in key order. They are\nheld in your organisation's own store and are not visible to any other.",
+            "items": {
+              "$ref": "#/components/schemas/ReferenceOverride"
+            },
+            "type": "array"
+          },
+          "set": {
+            "$ref": "#/components/schemas/ReferenceSet",
+            "description": "Set is the published set: its version, its freshness and its sources."
+          }
+        },
+        "type": "object"
+      },
+      "ReferenceOverride": {
+        "properties": {
+          "at": {
+            "description": "At is when it was written, RFC 3339.",
+            "type": "string"
+          },
+          "by": {
+            "description": "By is who wrote it.",
+            "type": "string"
+          },
+          "key": {
+            "description": "Key is the member this organisation is speaking about.",
+            "type": "string"
+          },
+          "note": {
+            "description": "Note is why, in the operator's own words. Optional, and bounded.",
+            "type": "string"
+          },
+          "verdict": {
+            "description": "Verdict is allow or deny.",
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "ReferenceOverrideIn": {
+        "properties": {
+          "key": {
+            "description": "Key is the member: a domain, a CIDR or address, an issuer prefix, a\ndevice digest. It is matched the same way the baseline is, so a deny on\ntempbox.example also covers mail.tempbox.example.",
+            "type": "string"
+          },
+          "note": {
+            "description": "Note is why, in your own words. Optional, bounded to 512 bytes.",
+            "type": "string"
+          },
+          "verdict": {
+            "description": "Verdict is allow or deny, and nothing else. An override is a decision —\nunlike a baseline entry, which states facts and leaves the decision to your\npolicy — because your organisation is the only party entitled to say \"for\nus, this one is fine\".",
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "ReferenceReceipt": {
+        "properties": {
+          "asOf": {
+            "description": "AsOf is when the load happened, RFC 3339. Absent is dated on arrival, which\ncan only make the list look older than it is.",
+            "type": "string"
+          },
+          "keys": {
+            "description": "Keys is how many designations that load carried. Zero from a publisher who\ndesignates somebody is a failed load wearing a successful one's clothes,\nand belongs in Refusal instead.",
+            "type": "integer"
+          },
+          "refusal": {
+            "description": "Refusal is why the load failed, when it did.",
+            "type": "string"
+          },
+          "source": {
+            "description": "Source is the publisher this receipt is for.",
+            "type": "string"
+          },
+          "version": {
+            "description": "Version is the digest of what that publisher supplied, so a refresh that\nchanged nothing can be told from a refresh that did not run.",
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "ReferenceSet": {
+        "properties": {
+          "age": {
+            "description": "Age is how long ago that was.",
+            "type": "string"
+          },
+          "asOf": {
+            "description": "AsOf is when the OLDEST contributing publisher was current, RFC 3339. The\noldest and not the newest: a set is exactly as fresh as its weakest source.",
+            "type": "string"
+          },
+          "keys": {
+            "description": "Keys is how many members the baseline carries.",
+            "type": "integer"
+          },
+          "kind": {
+            "description": "Kind is how the baseline comes to exist: fetch (downloaded from a\npublisher), local (computed here), attest (held by the component that\nscreens against it, freshness reported), or seam (declared and NOT held,\nbecause the source needs a licence we do not have).",
+            "type": "string"
+          },
+          "match": {
+            "description": "Match is how a key is tested: exact, domain, net, digits, pattern or range.",
+            "type": "string"
+          },
+          "maxAge": {
+            "description": "MaxAge is how old this set may be before it is stale.",
+            "type": "string"
+          },
+          "overrides": {
+            "description": "Overrides is how many entries YOUR org has laid over this baseline.",
+            "type": "integer"
+          },
+          "refusal": {
+            "description": "Refusal names why the set cannot be relied on, when it cannot: never\nloaded, held elsewhere, or a licence we do not hold. Non-empty means a\nlookup against this set will not answer, rather than answering clean.",
+            "type": "string"
+          },
+          "set": {
+            "description": "Set is the name this set is addressed by.",
+            "type": "string"
+          },
+          "sources": {
+            "description": "Sources is each contributing publisher, its licence and its own freshness.",
+            "items": {
+              "$ref": "#/components/schemas/ReferenceSource"
+            },
+            "type": "array"
+          },
+          "stale": {
+            "description": "Stale is whether it is past that bound. A stale set still answers and says\nso, because yesterday's list beats none.",
+            "type": "boolean"
+          },
+          "version": {
+            "description": "Version is the exact baseline consulted — every contributing publisher and\nits content digest. A decision records this and an auditor resolves it back.",
+            "type": "string"
+          },
+          "what": {
+            "description": "What the set holds, in one sentence.",
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "ReferenceSetsOut": {
+        "properties": {
+          "refused": {
+            "description": "Refused names the sets that cannot be consulted at all. A key checked\nagainst one of these is UNKNOWN, not clean.",
+            "items": {
+              "type": "string"
+            },
+            "type": "array"
+          },
+          "sets": {
+            "description": "Sets is the whole catalog, in a stable order.",
+            "items": {
+              "$ref": "#/components/schemas/ReferenceSet"
+            },
+            "type": "array"
+          },
+          "stale": {
+            "description": "Stale names the sets past their freshness bound — the list to alarm on.",
+            "items": {
+              "type": "string"
+            },
+            "type": "array"
+          }
+        },
+        "type": "object"
+      },
+      "ReferenceSource": {
+        "properties": {
+          "asOf": {
+            "description": "AsOf is when this publisher was current, RFC 3339.",
+            "type": "string"
+          },
+          "basis": {
+            "description": "Basis is the KIND of permission this publisher's data reaches you under:\nlicence (an explicit grant), registry (the registry of record publishing for\nanyone to consult), operator (an operator's own machine-readable statement\nabout its own network, published for third parties to filter by — not a\nlicence, and not claimed as one), own (computed here), or none (nothing\nreaches you: the membership is held by the component that screens against\nit). It is on the wire so the licence position is an audit you can run.",
+            "type": "string"
+          },
+          "keys": {
+            "description": "Keys is how many members this publisher contributed.",
+            "type": "integer"
+          },
+          "origin": {
+            "description": "Origin is exactly where it was taken from, so it can be taken again.",
+            "type": "string"
+          },
+          "refusal": {
+            "description": "Refusal is why this publisher's last take failed, if it did. The set keeps\nits previous version of this source and ages out visibly rather than\nsilently shrinking.",
+            "type": "string"
+          },
+          "source": {
+            "description": "Source is the publisher.",
+            "type": "string"
+          },
+          "terms": {
+            "description": "Terms is the CITATION that basis points at — the licence identifier, the\nregistry, or the operator publication. A source with no stated terms is not\nin the catalog.",
+            "type": "string"
+          },
+          "version": {
+            "description": "Version is the content digest of what this publisher last supplied. Two\nrefreshes that agree on it took the same data.",
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "ReferenceTaken": {
+        "properties": {
+          "keys": {
+            "description": "Keys is how many members it carries.",
+            "type": "integer"
+          },
+          "refusal": {
+            "description": "Refusal is why this publisher contributed nothing, if it did not. The set\nkeeps its previous version of this source rather than shrinking.",
+            "type": "string"
+          },
+          "resumed": {
+            "description": "Resumed is true when this run continued a version a previous run left\nhalf-landed.",
+            "type": "boolean"
+          },
+          "source": {
+            "description": "Source is the publisher.",
+            "type": "string"
+          },
+          "unchanged": {
+            "description": "Unchanged is true when the publisher's data was byte-for-byte the set we\nalready held.",
+            "type": "boolean"
+          },
+          "version": {
+            "description": "Version is the content digest that landed.",
+            "type": "string"
+          },
+          "wrote": {
+            "description": "Wrote is how many rows this run actually wrote. Zero with Unchanged means\nthe publisher served the same set again.",
+            "type": "integer"
+          }
+        },
+        "type": "object"
+      },
+      "ReferenceVersion": {
+        "properties": {
+          "asOf": {
+            "description": "AsOf is when the oldest of them was current, RFC 3339.",
+            "type": "string"
+          },
+          "refusal": {
+            "description": "Refusal is why it could not be consulted, when it could not.",
+            "type": "string"
+          },
+          "set": {
+            "description": "Set is the set.",
+            "type": "string"
+          },
+          "stale": {
+            "description": "Stale is whether it is past its freshness bound.",
+            "type": "boolean"
+          },
+          "version": {
+            "description": "Version is every contributing publisher and its content digest.",
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "RefreshReferenceIn": {
+        "properties": {
+          "force": {
+            "description": "Force accepts a take whose size moved past the change bound. A publisher\nserving a tenth or ten times its previous list is refused by default and the\nprevious version is left standing; this is the operator saying the change is\nreal. It cannot make an empty, truncated or unparseable take land — those are\nerrors, not magnitudes.",
+            "type": "boolean"
+          },
+          "receipts": {
+            "description": "Receipts are supplied by the component that holds the membership, for a set\nof kind attest. They are refused on any other kind, and a set of kind attest\nis refused without them: this plane never invents a freshness it did not\nobserve.",
+            "items": {
+              "$ref": "#/components/schemas/ReferenceReceipt"
+            },
+            "type": "array"
+          },
+          "set": {
+            "description": "Set is the set to refresh.",
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "RefreshReferenceOut": {
+        "properties": {
+          "set": {
+            "description": "Set is the set refreshed.",
+            "type": "string"
+          },
+          "stale": {
+            "description": "Stale is whether it is STILL past its freshness bound after the refresh,\nwhich is what a publisher that has stopped answering looks like.",
+            "type": "boolean"
+          },
+          "took": {
+            "description": "Took is what each publisher contributed.",
+            "items": {
+              "$ref": "#/components/schemas/ReferenceTaken"
+            },
+            "type": "array"
+          },
+          "version": {
+            "description": "Version is the set's new composed version.",
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "ResolveReferenceIn": {
+        "properties": {
+          "keys": {
+            "description": "Keys are the values to look up, at most 100 per call: email addresses or\ndomains, IP addresses, card prefixes, user-agent strings, autonomous system\nnumbers, device digests.",
+            "items": {
+              "type": "string"
+            },
+            "type": "array"
+          },
+          "sets": {
+            "description": "Sets narrows which sets to consult. Empty consults every set whose matcher\ncan read the keys given.",
+            "items": {
+              "type": "string"
+            },
+            "type": "array"
+          }
+        },
+        "type": "object"
+      },
+      "ResolveReferenceOut": {
+        "properties": {
+          "answers": {
+            "description": "Answers is one entry per (set, key) consulted.",
+            "items": {
+              "$ref": "#/components/schemas/ReferenceAnswer"
+            },
+            "type": "array"
+          },
+          "consulted": {
+            "description": "Consulted names the version of every set that took part, so a decision can\nrecord precisely what it leaned on. Record this with the decision: it is\nwhat makes the decision reproducible a year later.",
+            "items": {
+              "$ref": "#/components/schemas/ReferenceVersion"
+            },
+            "type": "array"
+          },
+          "refused": {
+            "description": "Refused names the consulted sets that could not answer at all. A key that\nmissed in one of these is UNKNOWN, not clean.",
+            "items": {
+              "type": "string"
+            },
+            "type": "array"
+          },
+          "stale": {
+            "description": "Stale names the consulted sets past their freshness bound. Staleness is\nitself a risk signal — a decision taken against a three-week-old list is a\nweaker decision, and this is how it knows.",
+            "items": {
+              "type": "string"
+            },
+            "type": "array"
+          }
+        },
+        "type": "object"
+      },
+      "SetReferenceIn": {
+        "properties": {
+          "entries": {
+            "description": "Entries are the overrides to write, up to 1000 per call.",
+            "items": {
+              "$ref": "#/components/schemas/ReferenceOverrideIn"
+            },
+            "type": "array"
+          }
+        },
+        "type": "object"
+      },
+      "SetReferenceOut": {
+        "properties": {
+          "overrides": {
+            "description": "Overrides is how many your org now holds in this set.",
+            "type": "integer"
+          },
+          "set": {
+            "description": "Set is the set written in.",
+            "type": "string"
+          },
+          "written": {
+            "description": "Written is how many entries this call wrote.",
+            "type": "integer"
+          }
+        },
+        "type": "object"
+      }
+    }
+  }
+}

typed_request_gate_test.go

@@ -65,6 +65,14 @@ var allowedRequestUses = map[string]string{
 		"modeled as an In field, because zip binds an In field from the BODY too and this route has never " +
 		"accepted an account there. authWrite fails closed off the HTTP path: no request, no attested " +
 		"caller, no mutation.",
+	"apps/reference/reference.go": "actor / the refresh gate — two facts beyond the org, both identity. " +
+		"An override is an adverse-action input (it is why a signup was refused), so the row records the " +
+		"validated user id who wrote it, which principal.OrgFrom does not carry; and refreshing the shared " +
+		"baseline every org reads is platform work, gated on the SuperAdmin claim (X-User-IsAdmin), which " +
+		"must never become an In field a caller could assert for itself. The TENANT is resolved with " +
+		"principal.OrgFrom (nsOf, right beside them) and never through the request, so no read or write is " +
+		"scoped by anything the request carries. The stored writer is bounded in BYTES at the same door " +
+		"(maxActor), because it is a term of a published per-organisation byte ceiling.",
 	"apps/provisioning/typed.go": "tenantOf — the provisioning control plane ALLOCATES and DESTROYS real " +
 		"backend resources, and its tenant is not the org principal.OrgFrom carries. Two facts differ, both " +
 		"live: the org is folded through namespace.Sanitize (the slug every physical name, S3 " +