Compare commits

...
Author SHA1 Message Date
hanzo-dev fbd68ebff0 dataset: the leaves landed in main at the old address; say so where the address is decided
They were never RELEASED — api.hanzo.ai/v1/openapi.json carries four /v1/ml paths
and none of them is a dataset — so the move is still free. The note said "nothing
had published these five leaves", which was true when it was written and stopped
being true when main took the branch. A comment that describes a state the tree has
left is the thing this file keeps being wrong about.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-03 10:26:23 -07:00
hanzo-dev a19b2f15bc Merge remote-tracking branch 'origin/main' into blue/dataset-fix2
# Conflicts:
#	manifest/apps.go
#	spend.go

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-03 10:24:45 -07:00
hanzo-dev 671e87bf2b dataset: a metered surface needs standing, or it is free the moment enforcement lands
plugin/dataset declares Price: cloud.Metered — correctly, a materialisation is a
bounded scan of the one warehouse every product on api.hanzo.ai shares — and the
app was absent from meteredApps. SpendGate resolves the trees it requires standing
for from that list through the manifest, so this surface would have charged 10c per
materialisation at its own meter while being exempt from the gate that refuses a
caller with no balance: money moving inside handlers the gate never guards.

It is the exact drift TestMeteredSurfacesRequireStanding was written for, and the
gate caught it the first time the app could be compiled — the previous commit's
cloud.Serve fix is what let this one be seen at all. Mutation-tested: with the
entry removed the gate names the app and fails.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-03 10:21:13 -07:00
hanzo-dev 9e3ad1f360 dataset: address the risk product's record plane at /v1/risk, not /v1/ml
/v1/ml IS MODEL SERVING, AND IT IS LIVE WITH CUSTOMERS. Four paths on
api.hanzo.ai today — InferenceServices and predict. These five leaves own
hanzo.risk_dataset and hanzo.risk_row and read hanzo.risk_feature: they are the
risk product's record plane, and a dataset of risk features is not a thing you
serve. Addressed under /v1/ml they made one prefix mean both "the models you
serve" and "the rows a model learned from" — the exact ambiguity the risk row was
split out of /v1/ml to end, reintroduced one level down by an app that was written
before /v1/risk existed.

Nothing had published them, so the address moved rather than being kept for the
sake of an unused spelling. It could not have moved later: a published address is
permanent, and this one would have been wrong forever.

The operation ids and the tag moved with it — mlExportDataset became
riskExportDataset — because an SDK method called mlExportDataset sitting beside a
live mlPredict names two products as one, and the Go type names are the schema
names in every generated client.

A SEPARATE APP FROM risk, still. The two hold different tenancy models and
different failure domains: risk's boundary is in-process per-org counters and its
failure domain is this process, this plane's is a qualified `<brand>/<org>` key
over the columnar store. One package holding two tenancy models is the shape a
privilege bug grows in.

NO RISK ROUTE MOVES. `risk` keeps /v1/risk outright and its row is untouched. The
nesting is the routing rule, not a collision — manifest.OwnerOf and the router both
take the LONGEST match, which is how `storage` already holds /v1/s3/buckets against
provisioning's /v1/s3. TestEveryServedPathReachesTheAppThatServesIt reports the
same 1670-of-1682 before and after, so all five leaves reach `dataset` at the new
address and nothing else changed hands. Enumerating risk's leaves to make room was
the other option and was rejected: a missing leaf in a hand-written list is a live
404.

Two gates, both mutation-tested (drift reintroduced, named test RED, reverted,
green): TestTheSurfaceIsExactlyWhatItSays with TestTheManifestNamesWhatTheRouterServes
catch a leaf drifting back under /v1/ml, and TestEveryOpIsNamedTaggedAndDescribed
now pins the tag as well as the id prefix — it checked the id and let any tag
through.

Also in this commit, both found by regenerating rather than by reading:

  plugin/dataset/main.go called cloud.Serve, which main renamed to cloud.Listen.
  The app could not BUILD on current main, so its spec subset could not be
  projected — the drift gate's own regeneration step was failing shut.

  plugin/dataset/mcp.json is deleted. It was the last mcp.json in the fleet — 118
  apps carry openapi.json, one carried this — because the catalogue was retired as
  a second source for a fact the child already knows (plugin/embed.go: o11y's held
  12 tools while its binary served 365). Nothing reads it, and it was already
  stale, still naming the seven ops ml*.

  openapi/floor.json carried ml=14: my merge took main's path and operation totals
  but the products map auto-merged to this branch's side, where the seven dataset
  ops were still counted under `ml`. Restored to main's 7. The ratchet raised risk
  10 -> 17 and the totals 1678/2328 -> 1683/2335 on its own, which is +7 operations
  across +5 paths — the move, with nothing gained or lost.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-03 10:17:40 -07:00
hanzo-dev b851446a2a dataset: bound the bytes, not the count, and say what the bound excluded
A COUNT OVER CALLER-SIZED VALUES IS NOT A BOUND. maxRows capped how many rows a
materialisation holds and `page` capped how many an export returns, and neither
bounded a single byte. A row's coordinates are float64s this plane counts, but its
SUBJECT is a string this plane does not write: the rollup lifts it from
`distinct_id`, `session_id` and `user_id`, which arrive on /v1/event from the
caller. distinct_id is capped at 256 bytes on the anonymous lane and REPLACED by
the token's own subject on the signed one — but session_id, which the `session`
rollup files as a subject verbatim, is capped nowhere. So "200k rows of ten
float64s plus a subject key is tens of megabytes" and "eight jobs is a few hundred
megabytes" were arithmetic over an unknown, and one tenant's traffic decided the
real number for the whole process.

maxSubjectBytes bounds the one caller-sized value a row carries, at 256 — what the
identified lane already states for a subject, for the reason that carries over
unchanged: a minted id is a uuid, the value is KEYED, and something longer is not
an id. With it, count times max IS the byte bound, and every byte figure the
package states is now DERIVED from it rather than written down beside it:
maxRowBytes, maxResidentBytes, maxProcessBytes, maxPageBytes. The stale prose
claims are gone rather than corrected — that was the other spelling.

ONE ENFORCEMENT POINT. `representable` is the predicate and the only place it is
written. It is applied on the way OUT of the source, which is the way IN to this
process and to the rows table, so no read path needs a second check: an export
page is bounded because every row it can return already came through it.

AND THE DEGRADATION IS NAMED. A bound that quietly drops rows is worse than no
bound — the dataset that comes back looks complete, and a model fitted on it is
blind to a population nobody can see was missing. The census measures both halves
on ONE pass (conditional aggregates, not a filter), the excluded subject count
rides on the version, the manifest, the lineage and the wire, and it is in the
source fingerprint, so `reproducible` is measured over it too: a window that grew
a subject too large to carry is a window that MOVED, and lineage now says so.

census and facts cannot disagree about the population, because the predicate is
one expression used by both — measured against, then read with. Two spellings
would sample the share from rows the read never returned.

Three gates, each mutation-tested (defect reintroduced, named test RED, reverted,
green):

  TestEveryReadOfTheSourceIsBoundedInBytes         the package's own AST — a
    function that reads the source without the bound fails, the same shape as the
    admission gate beside it, because it is the same failure: a new op skipping a
    property nobody checks. Carries the same anti-vacuity floor.
  TestAnUnrepresentableSubjectIsExcludedAndCounted end to end: the bound binds,
    the representable rows all survive, the count is on the version AND the
    lineage, and it is falsifiable.
  TestTheByteBoundIsDerivedFromTheValueBound       the arithmetic, so no byte
    figure can be asserted independently again.

The fake store EVALUATES the bound rather than ignoring it — a fixture that
ignored it would let every test above pass with the predicate deleted. It also now
binds `?` positionally across the WHOLE statement, as a driver does, which is what
the census's conditional aggregates require.

TestTheTenantLeadsEveryPredicate was reading args[0] and calling it the tenant.
That was a positional coincidence, true only while no statement carried
placeholders before its WHERE; it now reads the argument bound to the leading
`org = ?` itself, which is the property it always meant.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-03 10:11:54 -07:00
hanzo-dev adaf4254d2 Merge remote-tracking branch 'origin/main' into blue/dataset-fix2
# Conflicts:
#	manifest/apps.go
#	manifest/order_test.go
#	middleware_identity.go
#	openapi/floor.json

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-03 09:59:47 -07:00
18 changed files with 1475 additions and 1128 deletions
+55 -35
View File
@@ -1,4 +1,4 @@
// Package dataset is the per-org dataset plane of /v1/ml: a dataset is a
// Package dataset is the per-org dataset plane of /v1/risk: a dataset is a
// VERSIONED, IMMUTABLE snapshot of one tenant's own event surface, and this is
// where it is declared, materialised, described, exported and disposed of.
//
@@ -35,9 +35,24 @@
// the store, so it restarts empty and a restart loses nothing but the jobs in
// flight — which is exactly what a plane holding the record of what a model
// trained on must do, and exactly what a process pinned to one replica for its
// in-memory forests cannot promise. Its surface is five leaves under /v1/ml that
// no other app claims; zip refuses two owners for one prefix at compose time, so
// that is checked rather than agreed.
// in-memory forests cannot promise.
//
// WHY IT IS ADDRESSED UNDER /v1/risk. It owns hanzo.risk_dataset and hanzo.risk_row
// and reads hanzo.risk_feature: it is the risk product's record plane, so it is at
// that product's address. It was written at /v1/ml/datasets before /v1/risk existed,
// and /v1/ml is model SERVING — InferenceServices and predict, live with customers.
// A dataset of risk features is not a thing you serve, and one prefix meaning both
// "models you serve" and "the rows a model learned from" is the ambiguity the risk
// row was split out of /v1/ml to end. These five leaves landed in main at the old
// address but were never RELEASED — api.hanzo.ai/v1/openapi.json carries four
// /v1/ml paths and none of them is a dataset — so the address moved while moving
// it was still free. A published address is permanent.
//
// It is a SEPARATE app from risk, not a folded-in one, because the two hold
// different tenancy models and different failure domains — risk's boundary is
// in-process per-org counters, this plane's is a qualified `<brand>/<org>` key over
// the columnar store. The prefixes nest and that is the routing rule, not a
// collision: manifest.OwnerOf and the router both take the LONGEST match.
//
// WHAT IS PER PROCESS, SAID PLAINLY. Every read, every declaration and every
// disposal is a pure function of the store and answers identically from any
@@ -133,7 +148,7 @@ type scan struct {
since time.Time
}
// Mount wires the dataset leaves of /v1/ml onto app.
// Mount wires the dataset leaves of /v1/risk onto app.
//
// EVERY INHERITED CAPABILITY IS WIRED HERE, EXPLICITLY. Being embedded in cloud
// makes each one AVAILABLE; none is automatic:
@@ -216,14 +231,14 @@ func mount(p *plane, app cloud.Router) error {
//
// app.Use installs the tenant bridge ONCE PER DECLARED PREFIX — the scope
// reads manifest.Apps for that — so the middleware lands exactly on
// /v1/ml/datasets and nowhere else. Grouping at /v1/ml to get a shorter leaf
// would install it across a subtree this app does not own, which is the escape
// the scope exists to refuse.
// /v1/risk/datasets and nowhere else. Grouping at /v1/risk to get a shorter leaf
// would install it across a subtree this app does not own — the risk plane's own
// live leaves — which is the escape the scope exists to refuse.
//
// cloud.ZipApp recovers the typed-op registry, which the Router interface does
// not carry, and the ops register at ABSOLUTE paths on it. That is what keeps
// the published address exactly `/v1/ml/datasets` — a group root composes to
// `/v1/ml/datasets/`, and a trailing slash in the document is a trailing slash
// the published address exactly `/v1/risk/datasets` — a group root composes to
// `/v1/risk/datasets/`, and a trailing slash in the document is a trailing slash
// in every generated SDK.
//
// Bridge FIRST, before any leaf: a typed op receives only a context, and this
@@ -241,35 +256,35 @@ func mount(p *plane, app cloud.Router) error {
app.Use(cloud.DenyEnvelope())
o := ops{p: p}
zip.Post(z, "/v1/ml/datasets", o.create,
zip.WithOperationID("mlCreateDataset"),
zip.Post(z, "/v1/risk/datasets", o.create,
zip.WithOperationID("riskCreateDataset"),
zip.WithSummary("Declare the next version of a dataset"),
zip.WithTags("ml"))
zip.Get(z, "/v1/ml/datasets", o.list,
zip.WithOperationID("mlDatasets"),
zip.WithTags("risk"))
zip.Get(z, "/v1/risk/datasets", o.list,
zip.WithOperationID("riskDatasets"),
zip.WithSummary("List this org's datasets"),
zip.WithTags("ml"))
zip.Get(z, "/v1/ml/datasets/:name", o.describe,
zip.WithOperationID("mlDataset"),
zip.WithTags("risk"))
zip.Get(z, "/v1/risk/datasets/:name", o.describe,
zip.WithOperationID("riskDataset"),
zip.WithSummary("Describe every version of one dataset"),
zip.WithTags("ml"))
zip.Delete(z, "/v1/ml/datasets/:name", o.dispose,
zip.WithOperationID("mlDeleteDataset"),
zip.WithTags("risk"))
zip.Delete(z, "/v1/risk/datasets/:name", o.dispose,
zip.WithOperationID("riskDeleteDataset"),
zip.WithSummary("Dispose of one dataset and every version of it"),
zip.WithTags("ml"))
zip.Post(z, "/v1/ml/datasets/:name/materialize", o.materialize,
zip.WithOperationID("mlMaterializeDataset"),
zip.WithTags("risk"))
zip.Post(z, "/v1/risk/datasets/:name/materialize", o.materialize,
zip.WithOperationID("riskMaterializeDataset"),
zip.WithSummary("Materialise the declared version into immutable rows"),
zip.WithStatus(202),
zip.WithTags("ml"))
zip.Get(z, "/v1/ml/datasets/:name/lineage", o.lineage,
zip.WithOperationID("mlDatasetLineage"),
zip.WithTags("risk"))
zip.Get(z, "/v1/risk/datasets/:name/lineage", o.lineage,
zip.WithOperationID("riskDatasetLineage"),
zip.WithSummary("Show where a version's rows came from, and whether that can still be demonstrated"),
zip.WithTags("ml"))
zip.Get(z, "/v1/ml/datasets/:name/export", o.export,
zip.WithOperationID("mlExportDataset"),
zip.WithTags("risk"))
zip.Get(z, "/v1/risk/datasets/:name/export", o.export,
zip.WithOperationID("riskExportDataset"),
zip.WithSummary("Read a version's rows back, one page at a time"),
zip.WithTags("ml"))
zip.WithTags("risk"))
return nil
}
@@ -330,10 +345,15 @@ func (p *plane) who(ctx context.Context) (caller, error) {
// warehouse scans against a single stateful store, and a thousand times [maxRows]
// rows resident in one process. Both are fleet resources no tenant owns.
//
// Eight jobs of at most 200k rows is a few hundred megabytes and eight concurrent
// scans, which the store carries and this process survives. The ceiling is stated
// here rather than inferred from a pool size so that raising it is a decision
// somebody made.
// Eight jobs of at most [maxRows] rows is [maxProcessBytes] resident and eight
// concurrent scans, which the store carries and this process survives. That first
// figure is a COMPUTED one, and it did not used to be: this comment read "a few
// hundred megabytes" while a row's subject was a string the caller sized, so the
// real number was whatever one tenant's `session_id` made it. See
// [maxSubjectBytes] — the count is only a byte bound because the value is bounded.
//
// The ceiling is stated here rather than inferred from a pool size so that raising
// it is a decision somebody made.
const maxJobs = 8
// admit is THE door to the source surface: it prices the act at the meter, takes
+182 -73
View File
@@ -164,8 +164,8 @@ func surface(f *fake, org string, subjects, per int) {
// to nothing so the window is admitted in full. The window ends just past the
// last bucket so the derived cuts (70% and 85% of the window by time) fall INSIDE
// the data and all three splits are populated.
func declared(name string) mlDatasetSpec {
return mlDatasetSpec{
func declared(name string) riskDatasetSpec {
return riskDatasetSpec{
Name: name,
Kind: kindPerson,
From: origin0.Add(-time.Hour).Format(time.RFC3339),
@@ -177,15 +177,15 @@ func declared(name string) mlDatasetSpec {
// settled waits for a materialisation to reach a terminal state, then returns the
// version as `describe` reports it.
func settled(t *testing.T, app *zip.App, org, name string) mlDataset {
func settled(t *testing.T, app *zip.App, org, name string) riskDataset {
t.Helper()
deadline := time.Now().Add(10 * time.Second)
for {
code, body := do(t, app, http.MethodGet, "/v1/ml/datasets/"+name, org, nil)
code, body := do(t, app, http.MethodGet, "/v1/risk/datasets/"+name, org, nil)
if code != http.StatusOK {
t.Fatalf("describe %s: %d (%s)", name, code, body)
}
var v mlDatasetVersions
var v riskDatasetVersions
if err := json.Unmarshal(body, &v); err != nil {
t.Fatalf("describe %s: %v", name, err)
}
@@ -204,12 +204,12 @@ func settled(t *testing.T, app *zip.App, org, name string) mlDataset {
}
// build declares and materialises one dataset, and returns the settled version.
func build(t *testing.T, app *zip.App, org string, in mlDatasetSpec) mlDataset {
func build(t *testing.T, app *zip.App, org string, in riskDatasetSpec) riskDataset {
t.Helper()
if code, body := do(t, app, http.MethodPost, "/v1/ml/datasets", org, in); code != http.StatusOK {
if code, body := do(t, app, http.MethodPost, "/v1/risk/datasets", org, in); code != http.StatusOK {
t.Fatalf("declare: %d (%s)", code, body)
}
if code, body := do(t, app, http.MethodPost, "/v1/ml/datasets/"+in.Name+"/materialize", org, nil); code != http.StatusAccepted {
if code, body := do(t, app, http.MethodPost, "/v1/risk/datasets/"+in.Name+"/materialize", org, nil); code != http.StatusAccepted {
t.Fatalf("materialize: want 202, got %d (%s)", code, body)
}
return settled(t, app, org, in.Name)
@@ -245,11 +245,11 @@ func TestForeignOrgSeesNothing(t *testing.T) {
// A list is only ever this org's.
for _, org := range []string{"acme", "globex"} {
code, body := do(t, app, http.MethodGet, "/v1/ml/datasets", org, nil)
code, body := do(t, app, http.MethodGet, "/v1/risk/datasets", org, nil)
if code != http.StatusOK {
t.Fatalf("list %s: %d (%s)", org, code, body)
}
var list mlDatasetList
var list riskDatasetList
if err := json.Unmarshal(body, &list); err != nil {
t.Fatal(err)
}
@@ -259,11 +259,11 @@ func TestForeignOrgSeesNothing(t *testing.T) {
}
// An export never crosses. acme's rows are all acme's subjects.
code, body := do(t, app, http.MethodGet, "/v1/ml/datasets/shared/export?limit=1000", "acme", nil)
code, body := do(t, app, http.MethodGet, "/v1/risk/datasets/shared/export?limit=1000", "acme", nil)
if code != http.StatusOK {
t.Fatalf("export: %d (%s)", code, body)
}
var page mlDatasetRows
var page riskDatasetRows
if err := json.Unmarshal(body, &page); err != nil {
t.Fatal(err)
}
@@ -278,7 +278,7 @@ func TestForeignOrgSeesNothing(t *testing.T) {
// A disposal is per tenant: acme drops its own and globex still has its own,
// with every row intact.
if code, body := do(t, app, http.MethodDelete, "/v1/ml/datasets/shared", "acme", nil); code != http.StatusOK {
if code, body := do(t, app, http.MethodDelete, "/v1/risk/datasets/shared", "acme", nil); code != http.StatusOK {
t.Fatalf("dispose: %d (%s)", code, body)
}
// acme's BYTES are gone and its RECORD remains: the version reads `disposed`
@@ -287,10 +287,10 @@ func TestForeignOrgSeesNothing(t *testing.T) {
if got := describe(t, app, "acme", "shared"); got.Items[0].Status != statusDisposed {
t.Fatalf("acme's disposed dataset reads %q", got.Items[0].Status)
}
if code, _ := do(t, app, http.MethodGet, "/v1/ml/datasets/shared/export", "acme", nil); code != http.StatusConflict {
if code, _ := do(t, app, http.MethodGet, "/v1/risk/datasets/shared/export", "acme", nil); code != http.StatusConflict {
t.Fatalf("acme exported a disposed dataset: %d", code)
}
code, body = do(t, app, http.MethodGet, "/v1/ml/datasets/shared/export?limit=1000", "globex", nil)
code, body = do(t, app, http.MethodGet, "/v1/risk/datasets/shared/export?limit=1000", "globex", nil)
if code != http.StatusOK {
t.Fatalf("globex export after acme's disposal: %d (%s)", code, body)
}
@@ -311,13 +311,13 @@ func TestNoPrincipalReachesNothing(t *testing.T) {
method, path string
body any
}{
{http.MethodPost, "/v1/ml/datasets", declared("x")},
{http.MethodGet, "/v1/ml/datasets", nil},
{http.MethodGet, "/v1/ml/datasets/x", nil},
{http.MethodPost, "/v1/ml/datasets/x/materialize", nil},
{http.MethodGet, "/v1/ml/datasets/x/lineage", nil},
{http.MethodGet, "/v1/ml/datasets/x/export", nil},
{http.MethodDelete, "/v1/ml/datasets/x", nil},
{http.MethodPost, "/v1/risk/datasets", declared("x")},
{http.MethodGet, "/v1/risk/datasets", nil},
{http.MethodGet, "/v1/risk/datasets/x", nil},
{http.MethodPost, "/v1/risk/datasets/x/materialize", nil},
{http.MethodGet, "/v1/risk/datasets/x/lineage", nil},
{http.MethodGet, "/v1/risk/datasets/x/export", nil},
{http.MethodDelete, "/v1/risk/datasets/x", nil},
} {
code, body := do(t, app, tc.method, tc.path, "", tc.body)
if code != http.StatusForbidden {
@@ -334,7 +334,7 @@ func TestNoPrincipalReachesNothing(t *testing.T) {
// are not an organisation's.
func TestTheAnonymousLaneIsNotATenant(t *testing.T) {
app := mountHTTP(t, newPlane(&fake{}))
code, body := do(t, app, http.MethodGet, "/v1/ml/datasets", tenant.Public, nil)
code, body := do(t, app, http.MethodGet, "/v1/risk/datasets", tenant.Public, nil)
if code != http.StatusForbidden {
t.Fatalf("the anonymous lane reached the plane: %d (%s)", code, body)
}
@@ -458,7 +458,7 @@ func TestAPublishedVersionCannotBeMutated(t *testing.T) {
}
// The door refuses a second materialisation of a published version.
code, body := do(t, app, http.MethodPost, "/v1/ml/datasets/d/materialize", "one", nil)
code, body := do(t, app, http.MethodPost, "/v1/risk/datasets/d/materialize", "one", nil)
if code != http.StatusConflict {
t.Fatalf("re-materialising a published version: want 409, got %d (%s)", code, body)
}
@@ -514,7 +514,7 @@ func TestAnIncompleteAttemptIsNeverReadable(t *testing.T) {
p := newPlane(f)
app := mountHTTP(t, p)
if code, body := do(t, app, http.MethodPost, "/v1/ml/datasets", "one", declared("d")); code != http.StatusOK {
if code, body := do(t, app, http.MethodPost, "/v1/risk/datasets", "one", declared("d")); code != http.StatusOK {
t.Fatalf("declare: %d (%s)", code, body)
}
// The attempt is recorded, and then the process dies — which is exactly the
@@ -529,14 +529,14 @@ func TestAnIncompleteAttemptIsNeverReadable(t *testing.T) {
t.Fatal(err)
}
code, body := do(t, app, http.MethodPost, "/v1/ml/datasets/d/materialize", "one", nil)
code, body := do(t, app, http.MethodPost, "/v1/risk/datasets/d/materialize", "one", nil)
if code != http.StatusConflict {
t.Fatalf("re-attempting an incomplete version: want 409, got %d (%s)", code, body)
}
if !strings.Contains(string(body), "did not complete") {
t.Fatalf("the refusal does not name the state: %s", body)
}
if code, body := do(t, app, http.MethodGet, "/v1/ml/datasets/d/export", "one", nil); code != http.StatusConflict {
if code, body := do(t, app, http.MethodGet, "/v1/risk/datasets/d/export", "one", nil); code != http.StatusConflict {
t.Fatalf("exporting an unpublished version: want 409, got %d (%s)", code, body)
}
}
@@ -575,7 +575,7 @@ func TestARestartChangesNothing(t *testing.T) {
func TestAnUnreachableStoreRefusesRatherThanAnswersEmpty(t *testing.T) {
f := &fake{down: true}
app := mountHTTP(t, newPlane(f))
code, body := do(t, app, http.MethodGet, "/v1/ml/datasets", "one", nil)
code, body := do(t, app, http.MethodGet, "/v1/risk/datasets", "one", nil)
if code != http.StatusServiceUnavailable {
t.Fatalf("want 503 from a dead store, got %d (%s)", code, body)
}
@@ -595,12 +595,12 @@ func TestTheStoresOwnWordsNeverReachTheCaller(t *testing.T) {
app := mountHTTP(t, newPlane(f))
for _, tc := range []struct{ method, path string }{
{http.MethodGet, "/v1/ml/datasets"},
{http.MethodGet, "/v1/ml/datasets/d"},
{http.MethodGet, "/v1/ml/datasets/d/export"},
{http.MethodGet, "/v1/ml/datasets/d/lineage"},
{http.MethodDelete, "/v1/ml/datasets/d"},
{http.MethodPost, "/v1/ml/datasets/d/materialize"},
{http.MethodGet, "/v1/risk/datasets"},
{http.MethodGet, "/v1/risk/datasets/d"},
{http.MethodGet, "/v1/risk/datasets/d/export"},
{http.MethodGet, "/v1/risk/datasets/d/lineage"},
{http.MethodDelete, "/v1/risk/datasets/d"},
{http.MethodPost, "/v1/risk/datasets/d/materialize"},
} {
code, body := do(t, app, tc.method, tc.path, "one", nil)
if code == http.StatusOK {
@@ -613,7 +613,7 @@ func TestTheStoresOwnWordsNeverReachTheCaller(t *testing.T) {
}
}
// A declare fails the same way, and writes nothing.
code, body := do(t, app, http.MethodPost, "/v1/ml/datasets", "one", declared("d"))
code, body := do(t, app, http.MethodPost, "/v1/risk/datasets", "one", declared("d"))
if code == http.StatusOK || strings.Contains(string(body), "clickhouse") {
t.Fatalf("declare over a failing store: %d (%s)", code, body)
}
@@ -640,7 +640,7 @@ func TestARowFromAnotherTenantIsRefusedNotFiltered(t *testing.T) {
f.leak = true
f.mu.Unlock()
code, body := do(t, app, http.MethodGet, "/v1/ml/datasets", "acme", nil)
code, body := do(t, app, http.MethodGet, "/v1/risk/datasets", "acme", nil)
if code == http.StatusOK {
t.Fatalf("acme was served globex's register: %s", body)
}
@@ -682,7 +682,7 @@ func TestTheHorizonExcludesTheImmatureTail(t *testing.T) {
}
app := mountHTTP(t, newPlane(f))
in := mlDatasetSpec{
in := riskDatasetSpec{
Name: "mature", Kind: kindPerson,
From: now.Add(-(days + 1) * 24 * time.Hour).Format(time.RFC3339),
To: now.Add(time.Hour).Format(time.RFC3339),
@@ -712,11 +712,11 @@ func TestTheHorizonExcludesTheImmatureTail(t *testing.T) {
func TestAWindowYoungerThanItsHorizonIsRefusedAtTheDoor(t *testing.T) {
app := mountHTTP(t, newPlane(&fake{}))
now := time.Now().UTC()
in := mlDatasetSpec{
in := riskDatasetSpec{
Name: "young", From: now.Add(-2 * time.Hour).Format(time.RFC3339),
To: now.Format(time.RFC3339), Horizon: 30,
}
code, body := do(t, app, http.MethodPost, "/v1/ml/datasets", "one", in)
code, body := do(t, app, http.MethodPost, "/v1/risk/datasets", "one", in)
if code != http.StatusBadRequest {
t.Fatalf("want 400, got %d (%s)", code, body)
}
@@ -781,14 +781,14 @@ func TestOneMaterializationPerOrg(t *testing.T) {
p := newPlane(f)
app := mountHTTP(t, p)
if code, body := do(t, app, http.MethodPost, "/v1/ml/datasets", "one", declared("d")); code != http.StatusOK {
if code, body := do(t, app, http.MethodPost, "/v1/risk/datasets", "one", declared("d")); code != http.StatusOK {
t.Fatalf("declare: %d (%s)", code, body)
}
// Hold the slot the way a running job does, then ask again.
if _, err := p.claim(mustKey("one"), "d", 1); err != nil {
t.Fatalf("the slot was already held: %v", err)
}
code, body := do(t, app, http.MethodPost, "/v1/ml/datasets/d/materialize", "one", nil)
code, body := do(t, app, http.MethodPost, "/v1/risk/datasets/d/materialize", "one", nil)
if code != http.StatusConflict {
t.Fatalf("a second materialisation: want 409, got %d (%s)", code, body)
}
@@ -826,10 +826,10 @@ func TestTheProcessIsBoundedAcrossTenantsToo(t *testing.T) {
// A tenant holding no slot of its own still cannot start one: the resource it
// would spend is the plane's, not its own.
surface(f, "late", 3, 3)
if code, body := do(t, app, http.MethodPost, "/v1/ml/datasets", "late", declared("d")); code != http.StatusOK {
if code, body := do(t, app, http.MethodPost, "/v1/risk/datasets", "late", declared("d")); code != http.StatusOK {
t.Fatalf("declare: %d (%s)", code, body)
}
code, body := do(t, app, http.MethodPost, "/v1/ml/datasets/d/materialize", "late", nil)
code, body := do(t, app, http.MethodPost, "/v1/risk/datasets/d/materialize", "late", nil)
if code != http.StatusServiceUnavailable {
t.Fatalf("a full plane: want 503, got %d (%s)", code, body)
}
@@ -842,7 +842,7 @@ func TestTheProcessIsBoundedAcrossTenantsToo(t *testing.T) {
// would have left this caller a version it could never build, and the ceiling
// would destroy work rather than defer it.
p.release(mustKey("org00"))
if code, body := do(t, app, http.MethodPost, "/v1/ml/datasets/d/materialize", "late", nil); code != http.StatusAccepted {
if code, body := do(t, app, http.MethodPost, "/v1/risk/datasets/d/materialize", "late", nil); code != http.StatusAccepted {
t.Fatalf("materialising once the plane freed: want 202, got %d (%s)", code, body)
}
got := settled(t, app, "late", "d")
@@ -861,14 +861,14 @@ func TestADeadStoreIsNeverReportedAsTheCallersMistake(t *testing.T) {
app := mountHTTP(t, newPlane(f))
build(t, app, "one", declared("d"))
if code, body := do(t, app, http.MethodGet, "/v1/ml/datasets/d/export?split=holdout", "one", nil); code != http.StatusBadRequest {
if code, body := do(t, app, http.MethodGet, "/v1/risk/datasets/d/export?split=holdout", "one", nil); code != http.StatusBadRequest {
t.Fatalf("an unknown split: want 400, got %d (%s)", code, body)
}
f.mu.Lock()
f.down = true
f.mu.Unlock()
code, body := do(t, app, http.MethodGet, "/v1/ml/datasets/d/export?split=train", "one", nil)
code, body := do(t, app, http.MethodGet, "/v1/risk/datasets/d/export?split=train", "one", nil)
if code != http.StatusServiceUnavailable {
t.Fatalf("an export over a dead store: want 503, got %d (%s)", code, body)
}
@@ -946,7 +946,7 @@ func TestLineageIsAdmittedLikeTheScanItIs(t *testing.T) {
t.Fatalf("the slot was already held: %v", err)
}
f.forget()
code, body := do(t, app, http.MethodGet, "/v1/ml/datasets/d/lineage", "one", nil)
code, body := do(t, app, http.MethodGet, "/v1/risk/datasets/d/lineage", "one", nil)
if code != http.StatusConflict {
t.Fatalf("a second concurrent scan: want 409, got %d (%s)", code, body)
}
@@ -967,7 +967,7 @@ func TestLineageIsAdmittedLikeTheScanItIs(t *testing.T) {
t.Fatalf("filling slot %d: %v", i, err)
}
}
code, body = do(t, app, http.MethodGet, "/v1/ml/datasets/d/lineage", "one", nil)
code, body = do(t, app, http.MethodGet, "/v1/risk/datasets/d/lineage", "one", nil)
if code != http.StatusServiceUnavailable {
t.Fatalf("a full plane: want 503, got %d (%s)", code, body)
}
@@ -981,7 +981,7 @@ func TestLineageIsAdmittedLikeTheScanItIs(t *testing.T) {
l.gate = metering.ErrInsufficientBalance
l.mu.Unlock()
f.forget()
code, body = do(t, app, http.MethodGet, "/v1/ml/datasets/d/lineage", "one", nil)
code, body = do(t, app, http.MethodGet, "/v1/risk/datasets/d/lineage", "one", nil)
if code != http.StatusPaymentRequired {
t.Fatalf("an unfunded lineage: want 402, got %d (%s)", code, body)
}
@@ -1011,7 +1011,7 @@ func TestTheBillersOwnWordsNeverReachTheCaller(t *testing.T) {
l.mu.Lock()
l.gate = nil
l.mu.Unlock()
if code, body := do(t, app, http.MethodPost, "/v1/ml/datasets", "one", declared("d")); code != http.StatusOK {
if code, body := do(t, app, http.MethodPost, "/v1/risk/datasets", "one", declared("d")); code != http.StatusOK {
t.Fatalf("declare: %d (%s)", code, body)
}
l.mu.Lock()
@@ -1024,8 +1024,8 @@ func TestTheBillersOwnWordsNeverReachTheCaller(t *testing.T) {
path string
body any
}{
{"materialize", http.MethodPost, "/v1/ml/datasets/d/materialize", nil},
{"create", http.MethodPost, "/v1/ml/datasets", declared("other")},
{"materialize", http.MethodPost, "/v1/risk/datasets/d/materialize", nil},
{"create", http.MethodPost, "/v1/risk/datasets", declared("other")},
} {
code, body := do(t, app, probe.method, probe.path, "one", probe.body)
if strings.Contains(string(body), "10.43.7.19") || strings.Contains(string(body), "commerce") {
@@ -1052,7 +1052,7 @@ func TestADeclarationIsGatedTooEvenThoughItIsFree(t *testing.T) {
twin(f, "one")
l := &ledger{}
app := mountHTTP(t, newPlaneBilled(f, l))
if code, body := do(t, app, http.MethodPost, "/v1/ml/datasets", "one", declared("d")); code != http.StatusOK {
if code, body := do(t, app, http.MethodPost, "/v1/risk/datasets", "one", declared("d")); code != http.StatusOK {
t.Fatalf("declare: %d (%s)", code, body)
}
l.mu.Lock()
@@ -1079,7 +1079,7 @@ func TestATimedOutJobRecordsWhyItTimedOut(t *testing.T) {
twin(f, "one")
p := newPlane(f)
app := mountHTTP(t, p)
if code, body := do(t, app, http.MethodPost, "/v1/ml/datasets", "one", declared("d")); code != http.StatusOK {
if code, body := do(t, app, http.MethodPost, "/v1/risk/datasets", "one", declared("d")); code != http.StatusOK {
t.Fatalf("declare: %d (%s)", code, body)
}
e, ok, err := p.latest(context.Background(), mustKey("one"), "d")
@@ -1135,7 +1135,7 @@ func TestAVersionNumberIsNeverReusedAcrossADisposal(t *testing.T) {
t.Fatalf("version 3 does not carry its own digest")
}
if code, body := do(t, app, http.MethodDelete, "/v1/ml/datasets/orders", "one", nil); code != http.StatusOK {
if code, body := do(t, app, http.MethodDelete, "/v1/risk/datasets/orders", "one", nil); code != http.StatusOK {
t.Fatalf("dispose: %d (%s)", code, body)
}
// Every version reads disposed, and the record of each is still there — the
@@ -1149,7 +1149,7 @@ func TestAVersionNumberIsNeverReusedAcrossADisposal(t *testing.T) {
t.Fatalf("version %d reads %q after a disposal", v.Version, v.Status)
}
}
if code, body := do(t, app, http.MethodGet, "/v1/ml/datasets/orders/export", "one", nil); code != http.StatusConflict {
if code, body := do(t, app, http.MethodGet, "/v1/risk/datasets/orders/export", "one", nil); code != http.StatusConflict {
t.Fatalf("a disposed dataset still exports: %d (%s)", code, body)
}
f.mu.Lock()
@@ -1179,12 +1179,12 @@ func TestADisposalIsIdempotent(t *testing.T) {
app := mountHTTP(t, newPlane(f))
build(t, app, "one", declared("orders"))
if code, body := do(t, app, http.MethodDelete, "/v1/ml/datasets/orders", "one", nil); code != http.StatusOK {
if code, body := do(t, app, http.MethodDelete, "/v1/risk/datasets/orders", "one", nil); code != http.StatusOK {
t.Fatalf("dispose: %d (%s)", code, body)
}
when := describeVersion(t, app, "one", "orders", 1).At
time.Sleep(1100 * time.Millisecond) // the register's instant has one-second resolution
if code, body := do(t, app, http.MethodDelete, "/v1/ml/datasets/orders", "one", nil); code != http.StatusOK {
if code, body := do(t, app, http.MethodDelete, "/v1/risk/datasets/orders", "one", nil); code != http.StatusOK {
t.Fatalf("a repeat disposal: %d (%s)", code, body)
}
if again := describeVersion(t, app, "one", "orders", 1).At; again != when {
@@ -1254,19 +1254,19 @@ func TestATokenFromAnotherBrandIsNotThisBrandsTenant(t *testing.T) {
app := mountHTTP(t, newPlane(f))
// This deployment's own brand vouched: served.
if code, body := vouched(t, app, http.MethodGet, "/v1/ml/datasets", "one", brand, nil); code != http.StatusOK {
if code, body := vouched(t, app, http.MethodGet, "/v1/risk/datasets", "one", brand, nil); code != http.StatusOK {
t.Fatalf("this brand's own principal was refused: %d (%s)", code, body)
}
// Another brand's IAM vouched: refused, and refused before any statement.
f.forget()
code, body := vouched(t, app, http.MethodGet, "/v1/ml/datasets", "one", "lux", nil)
code, body := vouched(t, app, http.MethodGet, "/v1/risk/datasets", "one", "lux", nil)
if code != http.StatusForbidden {
t.Fatalf("a lux-issued principal read a hanzo tenant: %d (%s)", code, body)
}
for _, c := range f.seen() {
t.Fatalf("a cross-brand caller reached the store: %s", c.Stmt)
}
if code, body := vouched(t, app, http.MethodPost, "/v1/ml/datasets", "one", "lux", declared("d")); code != http.StatusForbidden {
if code, body := vouched(t, app, http.MethodPost, "/v1/risk/datasets", "one", "lux", declared("d")); code != http.StatusForbidden {
t.Fatalf("a lux-issued principal declared into a hanzo tenant: %d (%s)", code, body)
}
}
@@ -1304,13 +1304,13 @@ func mustKey(org string) tenant.Key {
return k
}
func exported(t *testing.T, app *zip.App, org, name string) []mlDatasetRow {
func exported(t *testing.T, app *zip.App, org, name string) []riskDatasetRow {
t.Helper()
code, body := do(t, app, http.MethodGet, "/v1/ml/datasets/"+name+"/export?limit=5000", org, nil)
code, body := do(t, app, http.MethodGet, "/v1/risk/datasets/"+name+"/export?limit=5000", org, nil)
if code != http.StatusOK {
t.Fatalf("export %s: %d (%s)", name, code, body)
}
var page mlDatasetRows
var page riskDatasetRows
if err := json.Unmarshal(body, &page); err != nil {
t.Fatalf("export %s: %v", name, err)
}
@@ -1318,13 +1318,13 @@ func exported(t *testing.T, app *zip.App, org, name string) []mlDatasetRow {
}
// describe reads a dataset's whole version history, newest first.
func describe(t *testing.T, app *zip.App, org, name string) mlDatasetVersions {
func describe(t *testing.T, app *zip.App, org, name string) riskDatasetVersions {
t.Helper()
code, body := do(t, app, http.MethodGet, "/v1/ml/datasets/"+name, org, nil)
code, body := do(t, app, http.MethodGet, "/v1/risk/datasets/"+name, org, nil)
if code != http.StatusOK {
t.Fatalf("describe %s: %d (%s)", name, code, body)
}
var v mlDatasetVersions
var v riskDatasetVersions
if err := json.Unmarshal(body, &v); err != nil {
t.Fatal(err)
}
@@ -1334,7 +1334,7 @@ func describe(t *testing.T, app *zip.App, org, name string) mlDatasetVersions {
return v
}
func describeVersion(t *testing.T, app *zip.App, org, name string, version int) mlDataset {
func describeVersion(t *testing.T, app *zip.App, org, name string, version int) riskDataset {
t.Helper()
v := describe(t, app, org, name)
for _, e := range v.Items {
@@ -1343,24 +1343,24 @@ func describeVersion(t *testing.T, app *zip.App, org, name string, version int)
}
}
t.Fatalf("version %d of %s is not in the description", version, name)
return mlDataset{}
return riskDataset{}
}
func lineageOf(t *testing.T, app *zip.App, org, name string, version int) mlLineage {
func lineageOf(t *testing.T, app *zip.App, org, name string, version int) riskLineage {
t.Helper()
code, body := do(t, app, http.MethodGet,
fmt.Sprintf("/v1/ml/datasets/%s/lineage?version=%d", name, version), org, nil)
fmt.Sprintf("/v1/risk/datasets/%s/lineage?version=%d", name, version), org, nil)
if code != http.StatusOK {
t.Fatalf("lineage %s: %d (%s)", name, code, body)
}
var out mlLineage
var out riskLineage
if err := json.Unmarshal(body, &out); err != nil {
t.Fatal(err)
}
return out
}
func subjectsOf(rows []mlDatasetRow) map[string]bool {
func subjectsOf(rows []riskDatasetRow) map[string]bool {
out := map[string]bool{}
for _, r := range rows {
out[r.Subject] = true
@@ -1379,3 +1379,112 @@ func equalSets(a, b map[string]bool) bool {
}
return true
}
// ── a count over caller-sized values is not a bound ──────────────────────────
// TestAnUnrepresentableSubjectIsExcludedAndCounted is the byte bound, end to end.
//
// THE DEFECT IT CLOSES. [maxRows] caps how many rows a materialisation holds and
// [page] caps how many an export returns. Neither bounded a single byte: a row's
// coordinates are float64s this plane counts, but its SUBJECT is a string this
// plane does not write — the rollup lifts it from `distinct_id`, `session_id` and
// `user_id`, which arrive on /v1/event from the caller, and `session_id` is length-
// capped nowhere on the way in. So one tenant's traffic decided what "200k rows"
// and "a few hundred megabytes" meant, and the published figures were arithmetic
// over an unknown.
//
// AND THE DEGRADATION IS NAMED. A bound that quietly drops rows is worse than no
// bound: the dataset that comes back looks complete, and a model fitted on it is
// blind to a population nobody can see was missing. So the excluded subjects are
// COUNTED, on the same pass, and the count rides on the version, the lineage and
// the wire.
func TestAnUnrepresentableSubjectIsExcludedAndCounted(t *testing.T) {
f := &fake{}
twin(f, "one")
k := mustKey("one")
// Two subjects no dataset can carry, in the SAME window as the representable
// ones, each in several buckets — so a plane that merely deduplicated would look
// the same as one that bounded.
huge := []string{strings.Repeat("a", maxSubjectBytes), strings.Repeat("b", maxSubjectBytes*40)}
for i, s := range huge {
for j := range 3 {
f.feature = append(f.feature, featRow{
Org: k.String(), Kind: kindPerson, Subject: s,
Bucket: origin0.Add(time.Duration(i*3+j) * time.Hour),
Value: map[string]float64{
"events": 1, "sessions": 1, "distincts": 1, "paths": 1,
"errors": 0, "calls": 1, "failures": 0, "tokens": 1, "spend_nano": 1, "ips": 1,
},
})
}
}
app := mountHTTP(t, newPlane(f))
got := build(t, app, "one", declared("d"))
if got.Status != statusReady {
t.Fatalf("refused: %s", got.Refusal)
}
// 1. NOT SILENT. The version states how many subjects it could not represent.
// `kindPerson` is 6 bytes, so the first oversized subject is 6+256 and the
// second 6+10240 — both past the bound, and both counted.
if got.Oversize != len(huge) {
t.Errorf("the version reports %d unrepresentable subjects, want %d — a bound that binds silently is indistinguishable from a tenant with no data", got.Oversize, len(huge))
}
// 2. THE BOUND BINDS. No row of an oversized subject reached the dataset, so
// every row in it is inside [maxRowBytes] and page*maxRowBytes bounds an export.
rows := exported(t, app, "one", "d")
if len(rows) == 0 {
t.Fatal("no rows at all; this test would prove nothing")
}
for _, r := range rows {
if n := len(r.Kind) + len(r.Subject); n > maxSubjectBytes {
t.Errorf("an exported row carries a %d-byte subject identity, past the %d-byte bound: %q", n, maxSubjectBytes, r.Subject)
}
}
// 3. THE REPRESENTABLE ROWS ARE ALL STILL THERE. A bound that also dropped
// legitimate rows would satisfy (1) and (2) and be a worse defect. twin() lays
// down 20 subjects across 3 buckets each.
if got.Counts.Subjects != 20 {
t.Errorf("%d representable subjects survived, want 20 — the bound excluded rows it should have carried", got.Counts.Subjects)
}
// 4. READABLE WHERE AN OPERATOR LOOKS, and part of what "reproducible" means.
var lin riskLineage
code, body := do(t, app, http.MethodGet, "/v1/risk/datasets/d/lineage", "one", nil)
if code != http.StatusOK {
t.Fatalf("lineage: %d (%s)", code, body)
}
if err := json.Unmarshal(body, &lin); err != nil {
t.Fatalf("lineage: %v", err)
}
if lin.Oversize != len(huge) {
t.Errorf("lineage reports %d unrepresentable subjects, want %d", lin.Oversize, len(huge))
}
if !lin.Reproducible {
t.Errorf("a version whose source has not moved is not reproducible: %s", lin.Refusal)
}
// 5. AND IT IS FALSIFIABLE. One more oversized subject is a source that MOVED,
// even though nothing representable changed — so the drift report covers the
// population the bound excludes rather than ignoring it.
f.feature = append(f.feature, featRow{
Org: k.String(), Kind: kindPerson, Subject: strings.Repeat("c", maxSubjectBytes+1),
Bucket: origin0.Add(time.Hour),
Value: map[string]float64{"events": 1},
})
code, body = do(t, app, http.MethodGet, "/v1/risk/datasets/d/lineage", "one", nil)
if code != http.StatusOK {
t.Fatalf("lineage: %d (%s)", code, body)
}
lin = riskLineage{}
if err := json.Unmarshal(body, &lin); err != nil {
t.Fatalf("lineage: %v", err)
}
if lin.Reproducible {
t.Error("the source grew a subject this version could not carry and lineage still certifies it re-derivable")
}
}
+94 -34
View File
@@ -209,51 +209,88 @@ var termRe = regexp.MustCompile(`^([a-z_]+) (=|>=|<) \?$`)
// the share.
var sampleRe = regexp.MustCompile(`^cityHash64\(\?, subject\) % (\d+) < \?$`)
// sizeRe is the row-size bound — [representable]. It consumes one argument, and
// the fake EVALUATES it: a store that ignored it would hand back the oversized
// subjects the plane believes it excluded, so every byte-bound test below would
// pass with the bound deleted. That is the exact failure mode a fixture is for.
var sizeRe = regexp.MustCompile(`^length\(subject_kind\) \+ length\(subject\) <= \?$`)
// sizeCol is the synthetic column the size bound reads. The source readers put the
// subject identity's byte length in the row under this name, so the bound is
// evaluated by the same generic [match] as every other conjunct rather than by a
// special case that could drift from it.
const sizeCol = "__subject_bytes__"
// pred is a parsed WHERE clause: its conjuncts, the arguments THEY were parsed
// against, how many of those were consumed, and the seeded sample if one is present.
//
// args is deliberately not the caller's whole argument list. A driver binds `?`
// POSITIONALLY across the entire statement, so a statement carrying placeholders
// in its SELECT list — the census, whose aggregates are conditional — offsets
// every WHERE argument. The fake has to do the same or it would compare `org = ?`
// against the first aggregate's bound and match nothing.
type pred struct {
terms []term
args []any
used int
seed string
shared int
}
// where extracts the conjuncts and the arguments they consume. It FAILS on a
// conjunct it does not recognise rather than skipping it — a skipped predicate is
// a predicate that was never tested.
func where(s string, args []any) ([]term, int, string, int, error) {
func where(s string, args []any) (pred, error) {
i := strings.Index(s, " WHERE ")
if i < 0 {
return nil, 0, "", 0, nil
return pred{args: args, shared: shareDenominator}, nil
}
// Every placeholder BEFORE the WHERE belongs to the SELECT list, and consumed
// its argument there.
lead := strings.Count(s[:i], "?")
if lead > len(args) {
return pred{}, fmt.Errorf("fake store: %d placeholders precede the WHERE but only %d arguments were bound", lead, len(args))
}
p := pred{args: args[lead:], shared: shareDenominator}
rest := s[i+len(" WHERE "):]
for _, stop := range []string{" GROUP BY ", " ORDER BY ", " LIMIT "} {
if j := strings.Index(rest, stop); j >= 0 {
rest = rest[:j]
}
}
var (
out []term
used int
seed string
shared = shareDenominator
)
for _, part := range strings.Split(rest, " AND ") {
part = strings.TrimSpace(part)
if m := termRe.FindStringSubmatch(part); m != nil {
out = append(out, term{col: m[1], op: m[2]})
used++
p.terms = append(p.terms, term{col: m[1], op: m[2]})
p.used++
continue
}
if sizeRe.MatchString(part) {
p.terms = append(p.terms, term{col: sizeCol, op: "<="})
p.used++
continue
}
if sampleRe.MatchString(part) {
if used+1 >= len(args) {
return nil, 0, "", 0, fmt.Errorf("fake store: the sample predicate has no arguments")
if p.used+1 >= len(p.args) {
return pred{}, fmt.Errorf("fake store: the sample predicate has no arguments")
}
seed = fmt.Sprint(args[used])
shared = int(asFloat(args[used+1]))
used += 2
p.seed = fmt.Sprint(p.args[p.used])
p.shared = int(asFloat(p.args[p.used+1]))
p.used += 2
continue
}
return nil, 0, "", 0, fmt.Errorf("fake store: unrecognised predicate %q", part)
return pred{}, fmt.Errorf("fake store: unrecognised predicate %q", part)
}
return out, used, seed, shared, nil
return p, nil
}
// match evaluates the conjuncts against one row.
func match(terms []term, args []any, row map[string]any) bool {
for i, t := range terms {
got, want := row[t.col], args[i]
func match(p pred, row map[string]any) bool {
for i, t := range p.terms {
if i >= len(p.args) {
return false
}
got, want := row[t.col], p.args[i]
switch t.op {
case "=":
if fmt.Sprint(got) != fmt.Sprint(want) {
@@ -267,6 +304,10 @@ func match(terms []term, args []any, row map[string]any) bool {
if geq(got, want) {
return false
}
case "<=":
if asFloat(got) > asFloat(want) {
return false
}
}
}
return true
@@ -289,9 +330,11 @@ func render(v any) string {
func asFloat(v any) float64 { return number(v) }
// tail reads the trailing LIMIT / OFFSET arguments a statement declared.
func tail(s string, args []any, used int) (limit, offset int) {
// tail reads the trailing LIMIT / OFFSET arguments a statement declared. They are
// read from the predicate's own argument slice, after what the conjuncts consumed.
func tail(s string, p pred) (limit, offset int) {
limit, offset = -1, 0
args, used := p.args, p.used
if strings.Contains(s, " LIMIT ? OFFSET ?") {
if used+1 < len(args) {
limit, offset = int(asFloat(args[used])), int(asFloat(args[used+1]))
@@ -317,7 +360,7 @@ func tail(s string, args []any, used int) (limit, offset int) {
// the engine behaviour a published version's immutability rests on, so it is
// reproduced here rather than assumed.
func (f *fake) readManifest(s string, args []any) ([]map[string]any, error) {
terms, used, _, _, err := where(s, args)
p, err := where(s, args)
if err != nil {
return nil, err
}
@@ -341,7 +384,7 @@ func (f *fake) readManifest(s string, args []any) ([]map[string]any, error) {
out := []map[string]any{}
for _, key := range order {
r := best[key]
if f.leak || match(terms, args, r) {
if f.leak || match(p, r) {
out = append(out, r)
}
}
@@ -370,25 +413,25 @@ func (f *fake) readManifest(s string, args []any) ([]map[string]any, error) {
}
out = kept
}
if limit, _ := tail(s, args, used); limit >= 0 && len(out) > limit {
if limit, _ := tail(s, p); limit >= 0 && len(out) > limit {
out = out[:limit]
}
return out, nil
}
func (f *fake) readRows(s string, args []any) ([]map[string]any, error) {
terms, used, _, _, err := where(s, args)
p, err := where(s, args)
if err != nil {
return nil, err
}
out := []map[string]any{}
for _, r := range f.rows {
if f.leak || match(terms, args, r) {
if f.leak || match(p, r) {
out = append(out, r)
}
}
sort.SliceStable(out, func(i, j int) bool { return fmt.Sprint(out[i]["id"]) < fmt.Sprint(out[j]["id"]) })
limit, offset := tail(s, args, used)
limit, offset := tail(s, p)
if offset > len(out) {
return []map[string]any{}, nil
}
@@ -402,26 +445,42 @@ func (f *fake) readRows(s string, args []any) ([]map[string]any, error) {
// readSource serves both shapes the plane reads from the source: the census
// aggregate and the grouped fact read.
func (f *fake) readSource(s string, args []any) ([]map[string]any, error) {
terms, used, seed, shared, err := where(s, args)
p, err := where(s, args)
if err != nil {
return nil, err
}
// sizeCol is what [representable] reads. It is computed here, from the row, so
// the fake evaluates the real byte length rather than trusting the plane.
var kept []featRow
for _, r := range f.feature {
row := map[string]any{"org": r.Org, "subject_kind": r.Kind, "subject": r.Subject, "bucket": r.Bucket}
if !match(terms, args, row) {
row := map[string]any{
"org": r.Org, "subject_kind": r.Kind, "subject": r.Subject, "bucket": r.Bucket,
sizeCol: len(r.Kind) + len(r.Subject),
}
if !match(p, row) {
continue
}
if shared < shareDenominator && bucketOf(seed, r.Subject) >= shared {
if p.shared < shareDenominator && bucketOf(p.seed, r.Subject) >= p.shared {
continue
}
kept = append(kept, r)
}
if strings.Contains(s, "uniqExact") {
// THE CENSUS, whose aggregates are CONDITIONAL: the size bound sits inside them
// rather than in the WHERE, because the count of what the bound EXCLUDES is one
// of the things being measured. The fake honours that split — it evaluates the
// same predicate per aggregate — so a census that stopped conditioning, or
// stopped counting the excluded, changes what these tests see.
fits := func(r featRow) bool { return len(r.Kind)+len(r.Subject) <= maxSubjectBytes }
seen := map[string]bool{}
subjects := map[string]bool{}
oversize := map[string]bool{}
var first, last time.Time
for _, r := range kept {
if !fits(r) {
oversize[r.Kind+"\x00"+r.Subject] = true
continue
}
seen[r.Kind+"\x00"+r.Subject+"\x00"+r.Bucket.String()] = true
subjects[r.Kind+"\x00"+r.Subject] = true
if first.IsZero() || r.Bucket.Before(first) {
@@ -433,7 +492,8 @@ func (f *fake) readSource(s string, args []any) ([]map[string]any, error) {
}
return []map[string]any{{
"rows": uint64(len(seen)), "subjects": uint64(len(subjects)),
"first": first, "last": last,
"oversize": uint64(len(oversize)),
"first": first, "last": last,
}}, nil
}
@@ -454,7 +514,7 @@ func (f *fake) readSource(s string, args []any) ([]map[string]any, error) {
}
out = append(out, row)
}
if limit, _ := tail(s, args, used); limit >= 0 && len(out) > limit {
if limit, _ := tail(s, p); limit >= 0 && len(out) > limit {
out = out[:limit]
}
return out, nil
+30 -17
View File
@@ -317,6 +317,7 @@ func (p *plane) build(ctx context.Context, a scan, e entry) (entry, error) {
e.Digest = digest(s, e.Version, rows)
e.Share = share
e.Truncated = hitLimit
e.Oversize = seen.Oversize
e.Status = statusReady
e.Refusal = ""
e.At = time.Now().UTC()
@@ -329,6 +330,7 @@ func (p *plane) build(ctx context.Context, a scan, e entry) (entry, error) {
First: stamp(seen.First),
Last: stamp(seen.Last),
Share: share,
Oversize: seen.Oversize,
Retention: p.retention(ctx),
})
@@ -344,14 +346,19 @@ func (p *plane) build(ctx context.Context, a scan, e entry) (entry, error) {
// on the manifest so the claim "these rows came from that window of that plane"
// can be CHECKED later rather than believed.
type origin struct {
Table string `json:"table"`
From string `json:"from"`
To string `json:"to"`
Rows int `json:"rows"`
Subjects int `json:"subjects"`
First string `json:"first"`
Last string `json:"last"`
Share int `json:"share"`
Table string `json:"table"`
From string `json:"from"`
To string `json:"to"`
Rows int `json:"rows"`
Subjects int `json:"subjects"`
First string `json:"first"`
Last string `json:"last"`
Share int `json:"share"`
// Oversize is how many subjects the window held that no dataset can represent.
// It is IN THE FINGERPRINT because it is part of what the source looked like: a
// window whose unrepresentable population changed is a window that changed, and
// the reproducibility check is exact agreement over everything recorded here.
Oversize int `json:"oversize"`
Retention string `json:"retention"`
}
@@ -388,16 +395,16 @@ func readOrigin(s string) (origin, error) {
// source that now starts later has expired its older rows; one that now starts
// EARLIER has been backfilled. Both mean the window no longer holds what this
// version was built from, and only the first was noticed before.
func (p *plane) lineage(ctx context.Context, a scan, e entry) (mlLineage, error) {
func (p *plane) lineage(ctx context.Context, a scan, e entry) (riskLineage, error) {
o, err := readOrigin(e.Source)
if err != nil {
return mlLineage{}, err
return riskLineage{}, err
}
s, err := e.Spec.spec()
if err != nil {
return mlLineage{}, err
return riskLineage{}, err
}
out := mlLineage{
out := riskLineage{
Dataset: e.Name,
Version: e.Version,
Source: o.Table,
@@ -406,17 +413,18 @@ func (p *plane) lineage(ctx context.Context, a scan, e entry) (mlLineage, error)
Rows: o.Rows,
Subjects: o.Subjects,
Share: o.Share,
Oversize: o.Oversize,
Digest: e.Digest,
Retention: o.Retention,
}
until, err := instant("to", o.To)
if err != nil {
return mlLineage{}, err
return riskLineage{}, err
}
now, err := p.census(ctx, a, s, until)
if err != nil {
return mlLineage{}, err
return riskLineage{}, err
}
out.Holds = now.Rows
out.Refusal = drift(o, now)
@@ -425,9 +433,11 @@ func (p *plane) lineage(ctx context.Context, a scan, e entry) (mlLineage, error)
}
// drift names the first way the source no longer holds what a version was built
// from, or "" when it holds exactly that. It is a total comparison of the four
// measurements the manifest records, so "reproducible" means all four agree and
// nothing else.
// from, or "" when it holds exactly that. It is a total comparison of EVERY
// measurement the manifest records, so "reproducible" means all of them agree and
// nothing else — including the unrepresentable population, because a window whose
// oversize count moved is a window whose contents moved, and a claim that is not
// checked is not a claim.
func drift(o origin, now census) string {
switch {
case now.Rows == 0:
@@ -438,6 +448,9 @@ func drift(o origin, now census) string {
case now.Subjects != o.Subjects:
return fmt.Sprintf("the source now holds %d subjects for this window where this version was built from %d",
now.Subjects, o.Subjects)
case now.Oversize != o.Oversize:
return fmt.Sprintf("the source now holds %d subjects too large to represent for this window where this version was built alongside %d",
now.Oversize, o.Oversize)
case stamp(now.First) != o.First:
return fmt.Sprintf("the source now starts at %s where this version was built from %s", stamp(now.First), o.First)
case stamp(now.Last) != o.Last:
+69 -5
View File
@@ -157,7 +157,8 @@ const manifestDDL = `
unproductive UInt64,
horizon UInt32,
share UInt16,
truncated UInt8
truncated UInt8,
oversize UInt64
) ENGINE = ReplacingMergeTree(seq)
PARTITION BY (org, name)
ORDER BY (org, name, version)`
@@ -220,6 +221,10 @@ type entry struct {
Horizon int
Share int
Truncated bool
// Oversize is how many of this window's subjects could not be represented —
// their subject identity exceeded [maxSubjectBytes]. Recorded so a bound that
// bound is a fact on the version rather than a difference nobody can see.
Oversize int
}
// manifestColumns is the fixed column list of every manifest write and read, in
@@ -228,6 +233,7 @@ var manifestColumns = []string{
"org", "name", "version", "seq", "at", "by", "status", "refusal",
"spec", "source", "digest", "rows", "train", "val", "test", "subjects",
"judged", "productive", "unproductive", "horizon", "share", "truncated",
"oversize",
}
// put writes manifest rows. It is the ONLY writer of hanzo.risk_dataset.
@@ -258,7 +264,7 @@ func (p *plane) put(ctx context.Context, k tenant.Key, es ...entry) error {
e.Spec.canon(), e.Source, e.Digest,
uint64(e.Counts.Rows), uint64(e.Counts.Train), uint64(e.Counts.Val), uint64(e.Counts.Test),
uint64(e.Counts.Subjects), uint64(e.Counts.Judged), uint64(e.Counts.Productive), uint64(e.Counts.Unproductive),
uint32(e.Horizon), uint16(e.Share), boolByte(e.Truncated))
uint32(e.Horizon), uint16(e.Share), boolByte(e.Truncated), uint64(e.Oversize))
}
stmt := "INSERT INTO " + manifestTable + " (" + strings.Join(manifestColumns, ", ") + ") VALUES " +
strings.TrimSuffix(strings.Repeat(group+", ", len(es)), ", ")
@@ -367,6 +373,7 @@ func (p *plane) read(ctx context.Context, k tenant.Key, stmt string, args ...any
Horizon: int(number(r["horizon"])),
Share: int(number(r["share"])),
Truncated: number(r["truncated"]) != 0,
Oversize: int(number(r["oversize"])),
}
if e.Spec, err = decode(text(r["spec"])); err != nil {
return nil, fmt.Errorf("dataset %q version %d: %w", e.Name, e.Version, err)
@@ -564,11 +571,48 @@ type census struct {
Subjects int
First time.Time
Last time.Time
// Oversize is how many subjects the window holds that CANNOT be represented in
// a dataset — their subject identity is longer than [maxSubjectBytes]. It is
// measured on the same pass that measures the rest, so naming it costs nothing
// and omitting it would make the bound silent.
Oversize int
}
// representable is the predicate a source row must satisfy to FIT IN A DATASET:
// its subject identity is at most [maxSubjectBytes]. It returns the SQL and its
// one bound argument, and it is the only place either is written.
//
// IT IS ONE EXPRESSION BECAUSE TWO STATEMENTS DEPEND ON IT AGREEING WITH ITSELF.
// [plane.census] counts what satisfies it AND what does not, in one pass; [plane.facts]
// returns only what satisfies it. Were the two spelled separately, the share would
// be computed against a population the read then did not return, and a capped
// dataset would sample from rows that were never eligible — a reproducibility bug
// with nothing pointing at it.
//
// It is applied on the way OUT of the source, which is the way IN to this process
// and to the rows table, so it is the single point at which [maxRowBytes] becomes
// true of every row this plane holds, writes or returns.
func representable() (string, any) {
return "length(subject_kind) + length(subject) <= ?", maxSubjectBytes
}
// oversized is representable's complement: the rows the bound EXCLUDES. It exists
// so the exclusion can be COUNTED rather than merely happening — a bound that
// binds silently is indistinguishable from a tenant with no data, which is the one
// reading an operator must never have to guess at.
func oversized() (string, any) {
pred, arg := representable()
return "NOT (" + pred + ")", arg
}
// sourceWhere is THE source predicate: org first and bound, then the kind, then
// the window. The order follows the source table's own sort key
// (org, subject_kind, subject, bucket), so the read is a prefix scan.
//
// The row-size bound is NOT here, deliberately: the census must see the rows it
// excludes in order to count them, so [representable] is applied per-aggregate
// there and as a conjunct in [plane.facts]. Both name it; the AST gate
// [TestEveryReadOfTheSourceIsBoundedInBytes] refuses a reader that names neither.
func sourceWhere(k tenant.Key, s spec, until time.Time) (string, []any) {
where := []string{"org = ?"}
args := []any{k.String()}
@@ -602,9 +646,22 @@ func (p *plane) census(ctx context.Context, a scan, s spec, until time.Time) (ce
return census{}, errStore
}
where, args := sourceWhere(k, s, until)
raw, err := p.store.Query(ctx,
"SELECT uniqExact((subject_kind, subject, bucket)) AS rows, uniqExact((subject_kind, subject)) AS subjects,"+
" min(bucket) AS first, max(bucket) AS last FROM "+sourceTable+" WHERE "+where, args...)
// The measurement is CONDITIONAL rather than filtered, so one pass answers both
// halves: what this window contributes to a dataset, and what it holds that no
// dataset can carry. The extent is measured over the representable rows only —
// it is the extent of the thing being built, and comparing a version against a
// window whose edges were set by a row the version never held is a drift report
// that fires on nothing anybody can act on.
fits, bound := representable()
over, _ := oversized()
stmt := "SELECT uniqExactIf((subject_kind, subject, bucket), " + fits + ") AS rows," +
" uniqExactIf((subject_kind, subject), " + fits + ") AS subjects," +
" uniqExactIf((subject_kind, subject), " + over + ") AS oversize," +
" minIf(bucket, " + fits + ") AS first, maxIf(bucket, " + fits + ") AS last" +
" FROM " + sourceTable + " WHERE " + where
// The aggregates precede the WHERE in the statement, so their arguments precede
// the predicate's. Five occurrences, one value.
raw, err := p.store.Query(ctx, stmt, append([]any{bound, bound, bound, bound, bound}, args...)...)
if err != nil {
return census{}, fmt.Errorf("%w: measure the source: %v", errStore, err)
}
@@ -616,6 +673,7 @@ func (p *plane) census(ctx context.Context, a scan, s spec, until time.Time) (ce
Subjects: int(number(raw[0]["subjects"])),
First: when(raw[0]["first"]),
Last: when(raw[0]["last"]),
Oversize: int(number(raw[0]["oversize"])),
}, nil
}
@@ -639,6 +697,12 @@ func (p *plane) facts(ctx context.Context, a scan, s spec, until time.Time, shar
return nil, false, errStore
}
where, args := sourceWhere(k, s, until)
// THE ROW-SIZE BOUND, and the reason [maxRows] means anything. It is the SAME
// expression the census counted with, so the population the share was chosen
// against is exactly the population this returns.
fits, bound := representable()
where += " AND " + fits
args = append(args, bound)
if share < shareDenominator {
where += fmt.Sprintf(" AND cityHash64(?, subject) %% %d < ?", shareDenominator)
args = append(args, s.Seed, uint32(share))
+108 -6
View File
@@ -35,7 +35,7 @@ func exercise(t *testing.T) ([]call, *fake) {
}
exported(t, app, "one", "d")
lineageOf(t, app, "one", "d", built.Version)
if code, body := do(t, app, http.MethodDelete, "/v1/ml/datasets/d", "one", nil); code != 200 {
if code, body := do(t, app, http.MethodDelete, "/v1/risk/datasets/d", "one", nil); code != 200 {
t.Fatalf("dispose: %d (%s)", code, body)
}
seen := f.seen()
@@ -98,8 +98,15 @@ func TestTheTenantLeadsEveryPredicate(t *testing.T) {
if !strings.HasPrefix(c.Stmt[i+len(" WHERE "):], "org = ?") {
t.Errorf("a read whose predicate does not OPEN with org: %s", c.Stmt)
}
if len(c.Args) == 0 || c.Args[0] != key {
t.Errorf("a read that does not bind the tenant first: %s %v", c.Stmt, c.Args)
// The argument bound to THAT placeholder is the tenant, which is not the same
// claim as "the first argument is the tenant". A driver binds `?` positionally
// across the WHOLE statement, so a statement carrying placeholders in its
// SELECT list — the census, whose aggregates are conditional on the row-size
// bound — offsets every WHERE argument. Reading args[0] there measured an
// aggregate's bound and called it the tenant.
lead := strings.Count(c.Stmt[:i], "?")
if lead >= len(c.Args) || c.Args[lead] != key {
t.Errorf("a read whose leading org predicate does not bind the tenant: %s %v (arg %d)", c.Stmt, c.Args, lead)
}
}
if checked < 8 {
@@ -149,7 +156,7 @@ func TestADatasetNameCannotCarrySyntax(t *testing.T) {
} {
in := declared("placeholder")
in.Name = bad
code, body := do(t, app, http.MethodPost, "/v1/ml/datasets", "one", in)
code, body := do(t, app, http.MethodPost, "/v1/risk/datasets", "one", in)
if code != http.StatusBadRequest {
t.Errorf("name %q was admitted with %d (%s)", bad, code, body)
}
@@ -157,10 +164,10 @@ func TestADatasetNameCannotCarrySyntax(t *testing.T) {
// Case is NORMALISED, not refused: one dataset has one name, so `Orders` and
// `orders` are the same dataset rather than two that differ by a shift key.
in := declared("MixedCase")
if code, body := do(t, app, http.MethodPost, "/v1/ml/datasets", "one", in); code != http.StatusOK {
if code, body := do(t, app, http.MethodPost, "/v1/risk/datasets", "one", in); code != http.StatusOK {
t.Fatalf("a mixed-case name was refused: %d (%s)", code, body)
}
if code, body := do(t, app, http.MethodGet, "/v1/ml/datasets/mixedcase", "one", nil); code != http.StatusOK {
if code, body := do(t, app, http.MethodGet, "/v1/risk/datasets/mixedcase", "one", nil); code != http.StatusOK {
t.Fatalf("the normalised name does not address the dataset: %d (%s)", code, body)
}
}
@@ -265,6 +272,101 @@ func TestEveryReadOfTheSourceTakesAnAdmission(t *testing.T) {
// rather than a hole in the test above.
const exempt = "retention"
// TestEveryReadOfTheSourceIsBoundedInBytes is the SECOND half of the admission
// property, and it exists because the first half was satisfiable by a read that
// was still unbounded.
//
// A [scan] proves a read was priced, counted and limited to [maxRows] ROWS. A row
// count bounds bytes only if a row's size is bounded, and a row's subject is a
// string this plane does not write: the rollup lifts it from `distinct_id`,
// `session_id` and `user_id`, which arrive on /v1/event from the caller.
// `session_id` — the whole subject of the `session` rollup — is capped nowhere on
// the way in. So "200k rows" and "eight jobs" were counts over caller-sized values,
// and one tenant's traffic decided how many bytes either meant.
//
// [representable] is the bound, and this is the gate that keeps a future reader
// from being added without it. It is the SAME shape as the admission gate above —
// the package's own AST — because the failure it prevents is the same one: a new
// op that reads the source and skips a property nobody is checking. `lineage` is
// how that happened the first time.
func TestEveryReadOfTheSourceIsBoundedInBytes(t *testing.T) {
fset := token.NewFileSet()
pkg, err := parser.ParseDir(fset, ".", func(fi os.FileInfo) bool {
return !strings.HasSuffix(fi.Name(), "_test.go")
}, 0)
if err != nil {
t.Fatalf("parse: %v", err)
}
var checked int
for _, p := range pkg {
for name, file := range p.Files {
for _, d := range file.Decls {
fn, ok := d.(*ast.FuncDecl)
if !ok || fn.Body == nil {
continue
}
var names, asks, bounds bool
ast.Inspect(fn.Body, func(n ast.Node) bool {
switch v := n.(type) {
case *ast.Ident:
names = names || v.Name == "sourceTable"
// Either spelling of the bound counts: the read applies
// [representable], the census additionally counts what
// [oversized] excludes.
bounds = bounds || v.Name == "representable" || v.Name == "oversized"
case *ast.SelectorExpr:
asks = asks || v.Sel.Name == "Query" || v.Sel.Name == "Exec"
}
return true
})
if !names || !asks || fn.Name.Name == exempt {
continue
}
checked++
if !bounds {
t.Errorf("%s: %s reads the source without the row-size bound — a row cap over caller-sized subjects bounds the COUNT and not the bytes",
name, fn.Name.Name)
}
}
}
}
// A gate that examined nothing passes. This is the same anti-vacuity floor the
// admission gate carries, and for the same reason: both walk the same set of
// functions, so a change that hides them from one hides them from both.
if checked < 2 {
t.Fatalf("only %d functions read the source; this test no longer covers the reads", checked)
}
}
// TestTheByteBoundIsDerivedFromTheValueBound pins the arithmetic that makes a count
// mean a size. Every figure the package states about memory or response size is
// computed from [maxSubjectBytes] and len(dims); this fails if any of them is ever
// written down independently again, which is how the wrong one got there before.
func TestTheByteBoundIsDerivedFromTheValueBound(t *testing.T) {
if got, want := maxRowBytes, maxSubjectBytes+8*len(dims)+fixedRowBytes; got != want {
t.Errorf("maxRowBytes = %d, want %d — it must be derived, not asserted", got, want)
}
if got, want := maxResidentBytes, maxRows*maxRowBytes; got != want {
t.Errorf("maxResidentBytes = %d, want %d", got, want)
}
if got, want := maxProcessBytes, maxJobs*maxResidentBytes; got != want {
t.Errorf("maxProcessBytes = %d, want %d", got, want)
}
if got, want := maxPageBytes, page*maxRowBytes; got != want {
t.Errorf("maxPageBytes = %d, want %d", got, want)
}
// The bound has to actually bound something. A subject cap at or above the row
// count's own scale would make the product meaningless again.
if maxSubjectBytes <= 0 {
t.Fatal("maxSubjectBytes must be positive, or the product below is not a bound")
}
// And the whole point: the resident ceiling is a real number an operator can
// hold in their head, not "a few hundred megabytes" of unknown provenance.
if maxProcessBytes > 8<<30 {
t.Errorf("maxProcessBytes = %d bytes, which is past anything this process can hold; one of the counts is too high", maxProcessBytes)
}
}
// TestAnUnkeyedCallReachesNoStatement. Every entry point takes a tenant.Key, and
// the zero key — the value of a Key that was never minted — refuses before any
// statement is built.
+63 -6
View File
@@ -104,13 +104,40 @@ const (
// as "no rows" and is indistinguishable from a quiet tenant.
minWindow = time.Minute
// maxRows caps one materialisation. The rows are held in this process to
// assign the splits and compute the digest — the two things that CANNOT be
// done in the store without duplicating the definition of a split — so the cap
// is a memory bound as well as a scan bound: 200k rows of ten float64s plus a
// subject key is tens of megabytes, which one queued job at a time can afford.
// maxRows caps HOW MANY rows one materialisation holds. The rows are held in
// this process to assign the splits and compute the digest — the two things
// that CANNOT be done in the store without duplicating the definition of a
// split — so a count on its own is a scan bound and NOTHING ELSE. What bounds
// the memory is this count times [maxRowBytes], and that product is a bound
// only because [maxSubjectBytes] exists.
maxRows = 200_000
// maxSubjectBytes bounds the ONE caller-sized value a row carries: the subject
// identity, `subject_kind` + `subject`.
//
// A COUNT OVER CALLER-SIZED VALUES IS NOT A BOUND. Every other part of a row is
// fixed by this package — the kind comes from a closed set, the coordinates are
// at most len(dims) float64s, the instant is an instant. The subject is not: the
// rollup that writes the source lifts it from `distinct_id`, `session_id` and
// `user_id`, which arrive on /v1/event from the caller. `distinct_id` is capped
// at 256 bytes on the anonymous lane and REPLACED by the token's own subject on
// the signed one, but `session_id` — which the `session` rollup files as a
// subject verbatim — is capped nowhere. So "200k rows is tens of megabytes" and
// "eight jobs is a few hundred megabytes" were arithmetic over an unknown, and
// one tenant's traffic decided the real number.
//
// 256 bytes is what the identified lane already states for a subject, for the
// reason that carries over unchanged: a minted id is a uuid (36 bytes), the
// value is KEYED — uniqExact over it is the whole point of the surface — and
// something longer is not an id. With the bound in place, count times max IS the
// byte bound, which is the only form in which either number means anything.
//
// It is enforced ONCE, at [representable], on the way OUT of the source and
// therefore on the way IN to this process and to the rows table. There is no
// second spelling downstream: every row this plane holds, writes or returns came
// through that predicate.
maxSubjectBytes = 256
// maxHorizon bounds the maturity wait. A year is past every dispute window
// that exists; beyond it the horizon is excluding data for no reason anybody
// can name.
@@ -128,6 +155,36 @@ const (
maxSeed = 128
)
// The BYTE bounds, every one of them DERIVED. A byte ceiling written down beside a
// count is two numbers nothing keeps in agreement, and the one that was wrong here
// was always the byte one. These are computed from [maxSubjectBytes] and len(dims),
// so raising either moves them and no comment goes stale.
//
// They are vars and not consts only because len(dims) is a slice length. Nothing
// assigns them; [TestTheByteBoundIsDerivedFromTheValueBound] pins the arithmetic.
var (
// maxRowBytes is one row's ceiling: its subject identity, its coordinates, and
// the fixed remainder (the derived id, the instant, the split). It is what makes
// every count below convertible into a size.
maxRowBytes = maxSubjectBytes + 8*len(dims) + fixedRowBytes
// maxResidentBytes is what ONE materialisation can hold in this process, and
// maxProcessBytes is what all [maxJobs] of them can hold at once. This is the
// claim that used to be a guess.
maxResidentBytes = maxRows * maxRowBytes
maxProcessBytes = maxJobs * maxResidentBytes
// maxPageBytes is the largest export response. The rows table holds only
// subjects that passed [representable] on the way in, so the page count times
// the row ceiling bounds the body — no second check on the read path.
maxPageBytes = page * maxRowBytes
)
// fixedRowBytes is the part of a row this package fixes: a 64-hex derived id, an
// RFC-3339 instant, a split name, and the JSON punctuation around them. Rounded up;
// it is a ceiling, not a measurement.
const fixedRowBytes = 192
// name is what a dataset may be called: lower-case, digits and single hyphens,
// starting with a letter. It is BOUND everywhere it is used — including in the
// partition expression, which takes values — so this refusal is about the name
@@ -275,7 +332,7 @@ func (s spec) columns() []string {
// `now` is passed rather than read so the horizon arithmetic is testable, which
// is the whole of R1: a maturity rule that cannot be tested is a maturity rule
// nobody has checked.
func normalize(in mlDatasetSpec, now time.Time) (spec, error) {
func normalize(in riskDatasetSpec, now time.Time) (spec, error) {
var s spec
s.Name = strings.ToLower(strings.TrimSpace(in.Name))
+19 -19
View File
@@ -14,8 +14,8 @@ import (
var day = 24 * time.Hour
func window(from, to time.Time) mlDatasetSpec {
return mlDatasetSpec{Name: "d", From: from.Format(time.RFC3339), To: to.Format(time.RFC3339)}
func window(from, to time.Time) riskDatasetSpec {
return riskDatasetSpec{Name: "d", From: from.Format(time.RFC3339), To: to.Format(time.RFC3339)}
}
// TestNormalizeRefusesEveryUnboundedOrUnreadableSpec. Each case below is either a
@@ -25,38 +25,38 @@ func TestNormalizeRefusesEveryUnboundedOrUnreadableSpec(t *testing.T) {
now := time.Date(2026, 6, 1, 0, 0, 0, 0, time.UTC)
ok := window(now.Add(-30*day), now.Add(-1*day))
mutate := func(f func(*mlDatasetSpec)) mlDatasetSpec {
mutate := func(f func(*riskDatasetSpec)) riskDatasetSpec {
in := ok
f(&in)
return in
}
for _, tc := range []struct {
name string
in mlDatasetSpec
in riskDatasetSpec
says string
}{
{"no name", mutate(func(s *mlDatasetSpec) { s.Name = "" }), "dataset name"},
{"a name with syntax in it", mutate(func(s *mlDatasetSpec) { s.Name = "a;b" }), "dataset name"},
{"an unknown subject kind", mutate(func(s *mlDatasetSpec) { s.Kind = "device" }), "subject kind"},
{"an unpublished dim", mutate(func(s *mlDatasetSpec) { s.Dims = []string{"events", "secrets"} }), "not a published dim"},
{"an unparseable window", mutate(func(s *mlDatasetSpec) { s.From = "yesterday" }), "RFC 3339"},
{"an inverted window", mutate(func(s *mlDatasetSpec) { s.From, s.To = s.To, s.From }), "empty or inverted"},
{"an empty window", mutate(func(s *mlDatasetSpec) { s.To = s.From }), "empty or inverted"},
{"no name", mutate(func(s *riskDatasetSpec) { s.Name = "" }), "dataset name"},
{"a name with syntax in it", mutate(func(s *riskDatasetSpec) { s.Name = "a;b" }), "dataset name"},
{"an unknown subject kind", mutate(func(s *riskDatasetSpec) { s.Kind = "device" }), "subject kind"},
{"an unpublished dim", mutate(func(s *riskDatasetSpec) { s.Dims = []string{"events", "secrets"} }), "not a published dim"},
{"an unparseable window", mutate(func(s *riskDatasetSpec) { s.From = "yesterday" }), "RFC 3339"},
{"an inverted window", mutate(func(s *riskDatasetSpec) { s.From, s.To = s.To, s.From }), "empty or inverted"},
{"an empty window", mutate(func(s *riskDatasetSpec) { s.To = s.From }), "empty or inverted"},
{"a window past the source's retention", window(now.Add(-401*day), now), "retention of the source"},
{"a negative horizon", mutate(func(s *mlDatasetSpec) { s.Horizon = -1 }), "between 0 and"},
{"a horizon past a year", mutate(func(s *mlDatasetSpec) { s.Horizon = 366 }), "between 0 and"},
{"a window younger than its horizon", mutate(func(s *mlDatasetSpec) { s.Horizon = 60 }), "maturity horizon"},
{"one cut", mutate(func(s *mlDatasetSpec) { s.Cuts = []string{now.Format(time.RFC3339)} }), "exactly two"},
{"three cuts", mutate(func(s *mlDatasetSpec) {
{"a negative horizon", mutate(func(s *riskDatasetSpec) { s.Horizon = -1 }), "between 0 and"},
{"a horizon past a year", mutate(func(s *riskDatasetSpec) { s.Horizon = 366 }), "between 0 and"},
{"a window younger than its horizon", mutate(func(s *riskDatasetSpec) { s.Horizon = 60 }), "maturity horizon"},
{"one cut", mutate(func(s *riskDatasetSpec) { s.Cuts = []string{now.Format(time.RFC3339)} }), "exactly two"},
{"three cuts", mutate(func(s *riskDatasetSpec) {
s.Cuts = []string{now.Format(time.RFC3339), now.Format(time.RFC3339), now.Format(time.RFC3339)}
}), "exactly two"},
{"cuts out of order", mutate(func(s *mlDatasetSpec) {
{"cuts out of order", mutate(func(s *riskDatasetSpec) {
s.Cuts = []string{now.Add(-5 * day).Format(time.RFC3339), now.Add(-20 * day).Format(time.RFC3339)}
}), "strictly increase"},
{"a cut outside the window", mutate(func(s *mlDatasetSpec) {
{"a cut outside the window", mutate(func(s *riskDatasetSpec) {
s.Cuts = []string{now.Add(-20 * day).Format(time.RFC3339), now.Add(5 * day).Format(time.RFC3339)}
}), "inside the window"},
{"an oversized seed", mutate(func(s *mlDatasetSpec) { s.Seed = strings.Repeat("s", maxSeed+1) }), "at most"},
{"an oversized seed", mutate(func(s *riskDatasetSpec) { s.Seed = strings.Repeat("s", maxSeed+1) }), "at most"},
} {
t.Run(tc.name, func(t *testing.T) {
_, err := normalize(tc.in, now)
+74 -54
View File
@@ -26,7 +26,7 @@ type ops struct{ p *plane }
// ── what a caller sends ──────────────────────────────────────────────────────
// mlDatasetSpec is the whole of what a dataset IS: a bound query over this org's
// riskDatasetSpec is the whole of what a dataset IS: a bound query over this org's
// own feature surface, a maturity horizon, where the splits cut, and the seed
// that decides membership. Declaring one mints the next VERSION; it never
// rewrites an existing one.
@@ -34,7 +34,7 @@ type ops struct{ p *plane }
// Nothing here becomes a SQL identifier. Dims resolve through the published
// allowlist to fixed columns, the kind is checked against a closed set, and every
// remaining value binds.
type mlDatasetSpec struct {
type riskDatasetSpec struct {
// Name identifies the dataset across its versions: lower-case letters, digits
// and hyphens, starting with a letter.
Name string `json:"name"`
@@ -70,46 +70,46 @@ type mlDatasetSpec struct {
Rows int `json:"rows,omitempty"`
}
// mlDatasetsIn takes nothing off the wire. The whole input is the caller's
// riskDatasetsIn takes nothing off the wire. The whole input is the caller's
// validated principal, which is what decides whose datasets these are.
type mlDatasetsIn struct{}
type riskDatasetsIn struct{}
// mlDatasetRef addresses one dataset by name. The name is the path segment: the
// riskDatasetRef addresses one dataset by name. The name is the path segment: the
// URL is the addressing authority, so it binds from there whatever a body says.
type mlDatasetRef struct {
type riskDatasetRef struct {
// Name is the dataset, from the path.
Name string `json:"name"`
}
// mlMaterializeIn asks for the declared version to be built. It carries only the
// riskMaterializeIn asks for the declared version to be built. It carries only the
// name because a materialisation always targets the version that is DECLARED —
// there is no version to choose, and offering one would suggest a published
// version could be rebuilt.
type mlMaterializeIn struct {
type riskMaterializeIn struct {
// Name is the dataset, from the path.
Name string `json:"name"`
}
// mlDisposeIn asks for a whole dataset to be disposed of. It carries no version,
// riskDisposeIn asks for a whole dataset to be disposed of. It carries no version,
// because disposal is per DATASET: the register row and the bytes go together, in
// one partition drop that cannot name another tenant.
type mlDisposeIn struct {
type riskDisposeIn struct {
// Name is the dataset, from the path.
Name string `json:"name"`
}
// mlLineageIn addresses one version's lineage.
type mlLineageIn struct {
// riskLineageIn addresses one version's lineage.
type riskLineageIn struct {
// Name is the dataset, from the path.
Name string `json:"name"`
// Version is the version to trace. Zero takes the newest published one.
Version int `json:"version,omitempty"`
}
// mlExportIn reads a published version's rows back, one page at a time. The page
// riskExportIn reads a published version's rows back, one page at a time. The page
// is bounded by the plane, not by the caller: an export is a read of the same
// store every other tenant is using.
type mlExportIn struct {
type riskExportIn struct {
// Name is the dataset, from the path.
Name string `json:"name"`
// Version is the version to read. Zero takes the newest published one.
@@ -126,10 +126,10 @@ type mlExportIn struct {
// ── what a caller gets ───────────────────────────────────────────────────────
// mlDataset is one version of one dataset. A version is the unit of citation: a
// riskDataset is one version of one dataset. A version is the unit of citation: a
// model names the dataset AND the version AND the digest, or it has not said what
// it was fitted on.
type mlDataset struct {
type riskDataset struct {
// Name and Version identify the version.
Name string `json:"name"`
Version int `json:"version"`
@@ -150,9 +150,9 @@ type mlDataset struct {
// one spec agree on it or the plane says they do not.
Digest string `json:"digest,omitempty"`
// Spec is the bound query this version was built from, exactly as recorded.
Spec mlDatasetSpec `json:"spec"`
Spec riskDatasetSpec `json:"spec"`
// Counts is how the rows fall across the splits.
Counts mlSplitCounts `json:"counts"`
Counts riskSplitCounts `json:"counts"`
// Share is the fraction of the window's subjects admitted, in thousandths.
// 1000 means the whole window fitted under the cap; anything less means the
// version is a reproducible sample and says by how much.
@@ -161,10 +161,19 @@ type mlDataset struct {
// trailing subject is dropped whole when that happens, because half a subject
// on one side of a split is exactly the leak the grouping prevents.
Truncated bool `json:"truncated,omitempty"`
// Oversize is how many of the window's subjects this version could NOT carry
// because their subject identity exceeds the plane's per-subject byte bound.
//
// It is on the wire, not only in a log, because it is the one degradation a
// caller cannot otherwise detect: the rows that are here look complete, and a
// dataset silently missing a population is a model silently blind to it.
// Non-zero does not make a version invalid — it makes it a version whose
// coverage is STATED. Zero is the normal case and omits.
Oversize int `json:"oversize,omitempty"`
}
// mlSplitCounts is how a version's rows fall, and how much of it is judged.
type mlSplitCounts struct {
// riskSplitCounts is how a version's rows fall, and how much of it is judged.
type riskSplitCounts struct {
Rows int `json:"rows"`
Train int `json:"train"`
Val int `json:"val"`
@@ -183,27 +192,27 @@ type mlSplitCounts struct {
Unproductive int `json:"unproductive"`
}
// mlDatasetList is every dataset this org holds, newest version first.
type mlDatasetList struct {
// riskDatasetList is every dataset this org holds, newest version first.
type riskDatasetList struct {
// Items is one entry per dataset, carrying its newest version. Never null: an
// org that has declared nothing gets an empty array.
Items []mlDataset `json:"items"`
Items []riskDataset `json:"items"`
}
// mlDatasetVersions is every version of one dataset, newest first. The whole
// riskDatasetVersions is every version of one dataset, newest first. The whole
// history is returned because the point of a version is that the old ones are
// still there: a model fitted last quarter cites one of them.
type mlDatasetVersions struct {
type riskDatasetVersions struct {
Name string `json:"name"`
Items []mlDataset `json:"items"`
Items []riskDataset `json:"items"`
}
// mlLineage is where a version's rows came from, and whether that can still be
// riskLineage is where a version's rows came from, and whether that can still be
// DEMONSTRATED. Reproducible is measured by asking the source the same question
// again — it is false when the source has since expired the window, which is a
// fact about the plane rather than a failure, and hiding it would make every
// lineage claim unfalsifiable.
type mlLineage struct {
type riskLineage struct {
Dataset string `json:"dataset"`
Version int `json:"version"`
// Source is the plane the rows were derived from.
@@ -218,6 +227,10 @@ type mlLineage struct {
Subjects int `json:"subjects"`
// Share is the fraction of subjects admitted, in thousandths.
Share int `json:"share"`
// Oversize is how many subjects the window held that were too large to
// represent when this version was built. It is part of the fingerprint, so it
// is part of what "reproducible" is measured over.
Oversize int `json:"oversize,omitempty"`
// Holds is what the source holds for the same window NOW. The difference
// between it and Rows is the whole of the reproducibility claim.
Holds int `json:"holds"`
@@ -234,8 +247,8 @@ type mlLineage struct {
Refusal string `json:"refusal,omitempty"`
}
// mlDatasetRow is one row of a published version.
type mlDatasetRow struct {
// riskDatasetRow is one row of a published version.
type riskDatasetRow struct {
// ID names the row forever. It is DERIVED from the row's own subject and
// instant, not allocated, so two materialisations of the same fact agree on it
// without coordinating.
@@ -251,8 +264,8 @@ type mlDatasetRow struct {
Point []float64 `json:"point"`
}
// mlDatasetRows is one page of a version's rows.
type mlDatasetRows struct {
// riskDatasetRows is one page of a version's rows.
type riskDatasetRows struct {
Dataset string `json:"dataset"`
Version int `json:"version"`
// Digest is the version's fingerprint. An exported page that did not carry it
@@ -265,12 +278,12 @@ type mlDatasetRows struct {
Offset int `json:"offset"`
Limit int `json:"limit"`
// Rows is the page. Never null.
Rows []mlDatasetRow `json:"rows"`
Rows []riskDatasetRow `json:"rows"`
}
// mlDisposal is what a disposal removed. A retention action answers with what it
// riskDisposal is what a disposal removed. A retention action answers with what it
// destroyed, because "204 No Content" is a poor reply to "prove you deleted it".
type mlDisposal struct {
type riskDisposal struct {
Dataset string `json:"dataset"`
// Versions is how many versions went, and Rows how many rows they held between
// them, as the register recorded them.
@@ -293,7 +306,7 @@ type mlDisposal struct {
// their own limits. Every refusal names which bound it hit.
//
// Example: {"name": "signups", "kind": "person", "from": "2026-01-01T00:00:00Z", "to": "2026-04-01T00:00:00Z", "horizon": 14}
func (o ops) create(ctx context.Context, in *mlDatasetSpec) (*mlDataset, error) {
func (o ops) create(ctx context.Context, in *riskDatasetSpec) (*riskDataset, error) {
c, err := o.p.who(ctx)
if err != nil {
return nil, err
@@ -316,7 +329,7 @@ func (o ops) create(ctx context.Context, in *mlDatasetSpec) (*mlDataset, error)
// has declared none gets an empty list; a store that cannot be reached gets a
// refusal, never an empty list, because the two read identically and only one of
// them is true.
func (o ops) list(ctx context.Context, _ *mlDatasetsIn) (*mlDatasetList, error) {
func (o ops) list(ctx context.Context, _ *riskDatasetsIn) (*riskDatasetList, error) {
c, err := o.p.who(ctx)
if err != nil {
return nil, err
@@ -328,7 +341,7 @@ func (o ops) list(ctx context.Context, _ *mlDatasetsIn) (*mlDatasetList, error)
if err != nil {
return nil, o.p.gap(err)
}
out := &mlDatasetList{Items: []mlDataset{}}
out := &riskDatasetList{Items: []riskDataset{}}
seen := map[string]bool{}
for _, e := range all {
if seen[e.Name] {
@@ -348,7 +361,7 @@ func (o ops) list(ctx context.Context, _ *mlDatasetsIn) (*mlDatasetList, error)
// probe learns nothing about another tenant's datasets.
//
// Example: {"name": "signups"}
func (o ops) describe(ctx context.Context, in *mlDatasetRef) (*mlDatasetVersions, error) {
func (o ops) describe(ctx context.Context, in *riskDatasetRef) (*riskDatasetVersions, error) {
c, err := o.p.who(ctx)
if err != nil {
return nil, err
@@ -363,7 +376,7 @@ func (o ops) describe(ctx context.Context, in *mlDatasetRef) (*mlDatasetVersions
if len(es) == 0 {
return nil, zip.ErrNotFound("no such dataset")
}
out := &mlDatasetVersions{Name: es[0].Name, Items: make([]mlDataset, 0, len(es))}
out := &riskDatasetVersions{Name: es[0].Name, Items: make([]riskDataset, 0, len(es))}
for _, e := range es {
out.Items = append(out.Items, *o.view(c, e))
}
@@ -387,7 +400,7 @@ func (o ops) describe(ctx context.Context, in *mlDatasetRef) (*mlDatasetVersions
// moving source honestly is.
//
// Example: {"name": "signups"}
func (o ops) materialize(ctx context.Context, in *mlMaterializeIn) (*mlDataset, error) {
func (o ops) materialize(ctx context.Context, in *riskMaterializeIn) (*riskDataset, error) {
c, err := o.p.who(ctx)
if err != nil {
return nil, err
@@ -421,7 +434,7 @@ func (o ops) materialize(ctx context.Context, in *mlMaterializeIn) (*mlDataset,
// this plane's own deadline rather than the caller's patience.
//
// Example: {"name": "signups", "version": 1}
func (o ops) lineage(ctx context.Context, in *mlLineageIn) (*mlLineage, error) {
func (o ops) lineage(ctx context.Context, in *riskLineageIn) (*riskLineage, error) {
c, err := o.p.who(ctx)
if err != nil {
return nil, err
@@ -460,7 +473,7 @@ func (o ops) lineage(ctx context.Context, in *mlLineageIn) (*mlLineage, error) {
// the dataset.
//
// Example: {"name": "signups", "version": 1, "split": "train", "limit": 500}
func (o ops) export(ctx context.Context, in *mlExportIn) (*mlDatasetRows, error) {
func (o ops) export(ctx context.Context, in *riskExportIn) (*riskDatasetRows, error) {
c, err := o.p.who(ctx)
if err != nil {
return nil, err
@@ -491,17 +504,17 @@ func (o ops) export(ctx context.Context, in *mlExportIn) (*mlDatasetRows, error)
if err != nil {
return nil, o.p.gap(err)
}
out := &mlDatasetRows{
out := &riskDatasetRows{
Dataset: e.Name,
Version: e.Version,
Digest: e.Digest,
Dims: e.Spec.Dims,
Offset: offset,
Limit: limit,
Rows: make([]mlDatasetRow, 0, len(rows)),
Rows: make([]riskDatasetRow, 0, len(rows)),
}
for _, r := range rows {
out.Rows = append(out.Rows, mlDatasetRow{
out.Rows = append(out.Rows, riskDatasetRow{
ID: r.ID,
Split: splitName(r.Split),
Kind: r.Kind,
@@ -513,8 +526,14 @@ func (o ops) export(ctx context.Context, in *mlExportIn) (*mlDatasetRows, error)
return out, nil
}
// page is the largest export page. An export is a read of the same store every
// other tenant is using, so the page size is the plane's to set.
// page is the largest export page, in ROWS. An export is a read of the same store
// every other tenant is using, so the page size is the plane's to set.
//
// It bounds the response BODY — at most [maxPageBytes] — only because every row in
// the rows table came through [representable] on the way in, so a row's subject is
// at most [maxSubjectBytes]. A page count over rows carrying caller-sized strings
// would bound the row count and nothing else. There is deliberately no second
// size check on this read path: the bound is enforced where rows ENTER, once.
const page = 5_000
// DeleteDataset disposes of one dataset and every version of it: the rows are
@@ -537,7 +556,7 @@ const page = 5_000
// cited has no rows once this returns, and every read of it says so.
//
// Example: {"name": "signups"}
func (o ops) dispose(ctx context.Context, in *mlDisposeIn) (*mlDisposal, error) {
func (o ops) dispose(ctx context.Context, in *riskDisposeIn) (*riskDisposal, error) {
c, err := o.p.who(ctx)
if err != nil {
return nil, err
@@ -558,7 +577,7 @@ func (o ops) dispose(ctx context.Context, in *mlDisposeIn) (*mlDisposal, error)
// the register has already marked disposed.
return nil, zip.ErrConflict("a materialisation of this dataset is running; it must finish before the dataset can be disposed of")
}
out := &mlDisposal{Dataset: es[0].Name, Versions: len(es)}
out := &riskDisposal{Dataset: es[0].Name, Versions: len(es)}
for _, e := range es {
out.Rows += e.Counts.Rows
}
@@ -619,9 +638,9 @@ func (o ops) published(ctx context.Context, c caller, name string, version int)
// view projects a register entry onto the wire. It is the ONE projection, so the
// list, the description and the two mutations cannot describe a version
// differently.
func (o ops) view(c caller, e entry) *mlDataset {
func (o ops) view(c caller, e entry) *riskDataset {
held, running := o.p.running(c.key)
return &mlDataset{
return &riskDataset{
Name: e.Name,
Version: e.Version,
At: stamp(e.At),
@@ -630,7 +649,7 @@ func (o ops) view(c caller, e entry) *mlDataset {
Running: running && held.name == e.Name && held.version == e.Version,
Refusal: e.Refusal,
Digest: e.Digest,
Spec: mlDatasetSpec{
Spec: riskDatasetSpec{
Name: e.Spec.Name,
Kind: e.Spec.Kind,
Dims: e.Spec.Dims,
@@ -641,7 +660,7 @@ func (o ops) view(c caller, e entry) *mlDataset {
Seed: e.Spec.Seed,
Rows: e.Spec.Rows,
},
Counts: mlSplitCounts{
Counts: riskSplitCounts{
Rows: e.Counts.Rows,
Train: e.Counts.Train,
Val: e.Counts.Val,
@@ -653,5 +672,6 @@ func (o ops) view(c caller, e entry) *mlDataset {
},
Share: e.Share,
Truncated: e.Truncated,
Oversize: e.Oversize,
}
}
+19 -9
View File
@@ -77,13 +77,13 @@ func TestEveryRouteIsATypedOp(t *testing.T) {
// generated artefacts at once and must be a deliberate edit here.
func TestTheSurfaceIsExactlyWhatItSays(t *testing.T) {
want := []string{
"DELETE /v1/ml/datasets/{name}",
"GET /v1/ml/datasets",
"GET /v1/ml/datasets/{name}",
"GET /v1/ml/datasets/{name}/export",
"GET /v1/ml/datasets/{name}/lineage",
"POST /v1/ml/datasets",
"POST /v1/ml/datasets/{name}/materialize",
"DELETE /v1/risk/datasets/{name}",
"GET /v1/risk/datasets",
"GET /v1/risk/datasets/{name}",
"GET /v1/risk/datasets/{name}/export",
"GET /v1/risk/datasets/{name}/lineage",
"POST /v1/risk/datasets",
"POST /v1/risk/datasets/{name}/materialize",
}
served, _ := projections(t)
var got []string
@@ -113,12 +113,22 @@ func TestEveryOpIsNamedTaggedAndDescribed(t *testing.T) {
t.Errorf("operation id %q is claimed by both %s and %s", op.OperationID, prev, key)
}
ids[op.OperationID] = key
if !strings.HasPrefix(op.OperationID, "ml") {
t.Errorf("%s is called %q; every leaf of this plane belongs to the ml face", key, op.OperationID)
// The name and the tag both say `risk`, because that is the product these
// leaves belong to they own hanzo.risk_dataset and read hanzo.risk_feature.
// They were `ml*` while the plane was addressed at /v1/ml/datasets, and /v1/ml
// is model SERVING: an SDK method called mlExportDataset beside a live
// mlPredict names two products as one.
if !strings.HasPrefix(op.OperationID, "risk") {
t.Errorf("%s is called %q; every leaf of this plane belongs to the risk face", key, op.OperationID)
}
if len(op.Tags) == 0 {
t.Errorf("%s carries no tag", key)
}
for _, tag := range op.Tags {
if tag != "risk" {
t.Errorf("%s is tagged %q; a dataset of risk features is not part of the %s product", key, tag, tag)
}
}
if strings.TrimSpace(op.Summary) == "" {
t.Errorf("%s has no summary", key)
}
+125 -120
View File
@@ -9,157 +9,162 @@ import (
)
func init() {
zip.Describe("DELETE /v1/ml/datasets/:name", zip.Doc{
zip.Describe("DELETE /v1/risk/datasets/:name", zip.Doc{
Description: "Disposes of one dataset and every version of it: the rows are\ndropped and the register is marked with what went.\n\nThis is the ONLY expiry in this plane. Neither table carries a TTL, deliberately:\na table TTL is a fleet-wide clock no tenant can hold longer or shorten, which is\nthe opposite of a retention decision belonging to the tenant whose records they\nare. The drop is a partition drop on (org, dataset), so the tenant is the first\ncomponent of the thing being dropped and a disposal cannot be spelled across one.\n\nThe BYTES are what goes. The register keeps one `disposed` row per version — the\nname, the number, the spec, the digest and who disposed of it when — for two\nreasons: a retention obligation is answered by a record of the deletion, not by\nsilence; and version numbers must stay monotone, so that after `orders` is\ndisposed of and declared again the next version is 4 and not 1. A number that\ncould be reused would make every citation of `orders v3` ambiguous forever.\n\nIt is not reversible and there is no soft state in between. A version a model\ncited has no rows once this returns, and every read of it says so.",
Fields: map[string]string{
"mlDisposal.versions": "Versions is how many versions went, and Rows how many rows they held between\nthem, as the register recorded them.",
"mlDisposeIn.name": "Name is the dataset, from the path.",
"riskDisposal.versions": "Versions is how many versions went, and Rows how many rows they held between\nthem, as the register recorded them.",
"riskDisposeIn.name": "Name is the dataset, from the path.",
},
Example: json.RawMessage(`{"name":"signups"}`),
})
zip.Describe("GET /v1/ml/datasets", zip.Doc{
zip.Describe("GET /v1/risk/datasets", zip.Doc{
Description: "Datasets lists this org's datasets, each with its newest version. An org that\nhas declared none gets an empty list; a store that cannot be reached gets a\nrefusal, never an empty list, because the two read identically and only one of\nthem is true.",
Fields: map[string]string{
"mlDataset.at": "At is when this version last changed state, and By who.",
"mlDataset.counts": "Counts is how the rows fall across the splits.",
"mlDataset.digest": "Digest fingerprints the SPEC and the ROWS together. Two materialisations of\none spec agree on it or the plane says they do not.",
"mlDataset.name": "Name and Version identify the version.",
"mlDataset.refusal": "Refusal names why there are no bytes, when there are none.",
"mlDataset.running": "Running is true while THIS process is materialising the version. A version\nthat is `materializing` and not running was started by a process that is\ngone — two states the register cannot tell apart, because a register cannot\nknow which processes are alive.",
"mlDataset.share": "Share is the fraction of the window's subjects admitted, in thousandths.\n1000 means the whole window fitted under the cap; anything less means the\nversion is a reproducible sample and says by how much.",
"mlDataset.spec": "Spec is the bound query this version was built from, exactly as recorded.",
"mlDataset.status": "Status is declared, materializing, ready or refused. Only `ready` has bytes,\nand `ready` is terminal: a published version is never rewritten.",
"mlDataset.truncated": "Truncated is true when the row cap bound before the window ran out. The\ntrailing subject is dropped whole when that happens, because half a subject\non one side of a split is exactly the leak the grouping prevents.",
"mlDatasetList.items": "Items is one entry per dataset, carrying its newest version. Never null: an\norg that has declared nothing gets an empty array.",
"mlDatasetSpec.cuts": "Cuts are the two RFC 3339 instants dividing train | val | test. Omit them to\ntake 70% and 85% of the window by time. Splitting is TEMPORAL and then\ngrouped by subject — a random split puts one device on both sides of the\nline and the model memorises the entity instead of the behaviour.",
"mlDatasetSpec.dims": "Dims are the coordinates to carry, by published name. Empty takes the whole\nsurface. They are stored in the plane's own order, never the order given, so\ntwo requests naming the same dims produce identical rows.",
"mlDatasetSpec.from": "From and To bound the event window, half-open, RFC 3339. The window may not\nbe longer than the source's own retention: past that, its older half is\nalready gone and the dataset would silently be shorter than it says.",
"mlDatasetSpec.horizon": "Horizon is how many days a row must have aged before it may be admitted. It\nis what keeps a fact that was not yet knowable at scoring time out of a\ntraining set: a chargeback lands 30 to 120 days after the transaction it\ncondemns, so 120 for the payment lane and 14 for signup abuse. Zero admits\nthe whole window and is honest only where the outcome is immediate.",
"mlDatasetSpec.kind": "Kind narrows to one subject kind — person, session or account. Empty takes\nevery kind.",
"mlDatasetSpec.name": "Name identifies the dataset across its versions: lower-case letters, digits\nand hyphens, starting with a letter.",
"mlDatasetSpec.rows": "Rows caps the materialisation. Zero takes the plane's own bound.",
"mlDatasetSpec.seed": "Seed decides WHICH subjects are admitted when the window holds more rows\nthan the cap allows. It is recorded on the version, so a capped dataset is\nreproducible rather than being whichever rows the store returned first.\nOmit it to seed from the dataset's name.",
"mlSplitCounts.judged": "Judged is how many rows carry a disposition. It is zero until a label plane\nwrites one, and reporting it plainly is what lets a model plane refuse to\nrank rather than name a winner it cannot justify.",
"mlSplitCounts.productive": "Productive and Unproductive are the two judged classes, so the imbalance is\nvisible before anyone trains on it.",
"mlSplitCounts.subjects": "Subjects is how many distinct subjects the rows belong to. Every row of one\nsubject is in ONE split, so this is the real sample size — the row count\nflatters it whenever a subject is active.",
"riskDataset.at": "At is when this version last changed state, and By who.",
"riskDataset.counts": "Counts is how the rows fall across the splits.",
"riskDataset.digest": "Digest fingerprints the SPEC and the ROWS together. Two materialisations of\none spec agree on it or the plane says they do not.",
"riskDataset.name": "Name and Version identify the version.",
"riskDataset.oversize": "Oversize is how many of the window's subjects this version could NOT carry\nbecause their subject identity exceeds the plane's per-subject byte bound.\n\nIt is on the wire, not only in a log, because it is the one degradation a\ncaller cannot otherwise detect: the rows that are here look complete, and a\ndataset silently missing a population is a model silently blind to it.\nNon-zero does not make a version invalid — it makes it a version whose\ncoverage is STATED. Zero is the normal case and omits.",
"riskDataset.refusal": "Refusal names why there are no bytes, when there are none.",
"riskDataset.running": "Running is true while THIS process is materialising the version. A version\nthat is `materializing` and not running was started by a process that is\ngone — two states the register cannot tell apart, because a register cannot\nknow which processes are alive.",
"riskDataset.share": "Share is the fraction of the window's subjects admitted, in thousandths.\n1000 means the whole window fitted under the cap; anything less means the\nversion is a reproducible sample and says by how much.",
"riskDataset.spec": "Spec is the bound query this version was built from, exactly as recorded.",
"riskDataset.status": "Status is declared, materializing, ready or refused. Only `ready` has bytes,\nand `ready` is terminal: a published version is never rewritten.",
"riskDataset.truncated": "Truncated is true when the row cap bound before the window ran out. The\ntrailing subject is dropped whole when that happens, because half a subject\non one side of a split is exactly the leak the grouping prevents.",
"riskDatasetList.items": "Items is one entry per dataset, carrying its newest version. Never null: an\norg that has declared nothing gets an empty array.",
"riskDatasetSpec.cuts": "Cuts are the two RFC 3339 instants dividing train | val | test. Omit them to\ntake 70% and 85% of the window by time. Splitting is TEMPORAL and then\ngrouped by subject — a random split puts one device on both sides of the\nline and the model memorises the entity instead of the behaviour.",
"riskDatasetSpec.dims": "Dims are the coordinates to carry, by published name. Empty takes the whole\nsurface. They are stored in the plane's own order, never the order given, so\ntwo requests naming the same dims produce identical rows.",
"riskDatasetSpec.from": "From and To bound the event window, half-open, RFC 3339. The window may not\nbe longer than the source's own retention: past that, its older half is\nalready gone and the dataset would silently be shorter than it says.",
"riskDatasetSpec.horizon": "Horizon is how many days a row must have aged before it may be admitted. It\nis what keeps a fact that was not yet knowable at scoring time out of a\ntraining set: a chargeback lands 30 to 120 days after the transaction it\ncondemns, so 120 for the payment lane and 14 for signup abuse. Zero admits\nthe whole window and is honest only where the outcome is immediate.",
"riskDatasetSpec.kind": "Kind narrows to one subject kind — person, session or account. Empty takes\nevery kind.",
"riskDatasetSpec.name": "Name identifies the dataset across its versions: lower-case letters, digits\nand hyphens, starting with a letter.",
"riskDatasetSpec.rows": "Rows caps the materialisation. Zero takes the plane's own bound.",
"riskDatasetSpec.seed": "Seed decides WHICH subjects are admitted when the window holds more rows\nthan the cap allows. It is recorded on the version, so a capped dataset is\nreproducible rather than being whichever rows the store returned first.\nOmit it to seed from the dataset's name.",
"riskSplitCounts.judged": "Judged is how many rows carry a disposition. It is zero until a label plane\nwrites one, and reporting it plainly is what lets a model plane refuse to\nrank rather than name a winner it cannot justify.",
"riskSplitCounts.productive": "Productive and Unproductive are the two judged classes, so the imbalance is\nvisible before anyone trains on it.",
"riskSplitCounts.subjects": "Subjects is how many distinct subjects the rows belong to. Every row of one\nsubject is in ONE split, so this is the real sample size — the row count\nflatters it whenever a subject is active.",
},
})
zip.Describe("GET /v1/ml/datasets/:name", zip.Doc{
zip.Describe("GET /v1/risk/datasets/:name", zip.Doc{
Description: "Dataset describes every version of one dataset, newest first — the whole\nhistory, because the point of a version is that the older ones are still there\nand a model fitted last quarter cites one of them.\n\nA name this org does not own answers 404, exactly as an unknown name does, so a\nprobe learns nothing about another tenant's datasets.",
Fields: map[string]string{
"mlDataset.at": "At is when this version last changed state, and By who.",
"mlDataset.counts": "Counts is how the rows fall across the splits.",
"mlDataset.digest": "Digest fingerprints the SPEC and the ROWS together. Two materialisations of\none spec agree on it or the plane says they do not.",
"mlDataset.name": "Name and Version identify the version.",
"mlDataset.refusal": "Refusal names why there are no bytes, when there are none.",
"mlDataset.running": "Running is true while THIS process is materialising the version. A version\nthat is `materializing` and not running was started by a process that is\ngone — two states the register cannot tell apart, because a register cannot\nknow which processes are alive.",
"mlDataset.share": "Share is the fraction of the window's subjects admitted, in thousandths.\n1000 means the whole window fitted under the cap; anything less means the\nversion is a reproducible sample and says by how much.",
"mlDataset.spec": "Spec is the bound query this version was built from, exactly as recorded.",
"mlDataset.status": "Status is declared, materializing, ready or refused. Only `ready` has bytes,\nand `ready` is terminal: a published version is never rewritten.",
"mlDataset.truncated": "Truncated is true when the row cap bound before the window ran out. The\ntrailing subject is dropped whole when that happens, because half a subject\non one side of a split is exactly the leak the grouping prevents.",
"mlDatasetRef.name": "Name is the dataset, from the path.",
"mlDatasetSpec.cuts": "Cuts are the two RFC 3339 instants dividing train | val | test. Omit them to\ntake 70% and 85% of the window by time. Splitting is TEMPORAL and then\ngrouped by subject — a random split puts one device on both sides of the\nline and the model memorises the entity instead of the behaviour.",
"mlDatasetSpec.dims": "Dims are the coordinates to carry, by published name. Empty takes the whole\nsurface. They are stored in the plane's own order, never the order given, so\ntwo requests naming the same dims produce identical rows.",
"mlDatasetSpec.from": "From and To bound the event window, half-open, RFC 3339. The window may not\nbe longer than the source's own retention: past that, its older half is\nalready gone and the dataset would silently be shorter than it says.",
"mlDatasetSpec.horizon": "Horizon is how many days a row must have aged before it may be admitted. It\nis what keeps a fact that was not yet knowable at scoring time out of a\ntraining set: a chargeback lands 30 to 120 days after the transaction it\ncondemns, so 120 for the payment lane and 14 for signup abuse. Zero admits\nthe whole window and is honest only where the outcome is immediate.",
"mlDatasetSpec.kind": "Kind narrows to one subject kind — person, session or account. Empty takes\nevery kind.",
"mlDatasetSpec.name": "Name identifies the dataset across its versions: lower-case letters, digits\nand hyphens, starting with a letter.",
"mlDatasetSpec.rows": "Rows caps the materialisation. Zero takes the plane's own bound.",
"mlDatasetSpec.seed": "Seed decides WHICH subjects are admitted when the window holds more rows\nthan the cap allows. It is recorded on the version, so a capped dataset is\nreproducible rather than being whichever rows the store returned first.\nOmit it to seed from the dataset's name.",
"mlSplitCounts.judged": "Judged is how many rows carry a disposition. It is zero until a label plane\nwrites one, and reporting it plainly is what lets a model plane refuse to\nrank rather than name a winner it cannot justify.",
"mlSplitCounts.productive": "Productive and Unproductive are the two judged classes, so the imbalance is\nvisible before anyone trains on it.",
"mlSplitCounts.subjects": "Subjects is how many distinct subjects the rows belong to. Every row of one\nsubject is in ONE split, so this is the real sample size — the row count\nflatters it whenever a subject is active.",
"riskDataset.at": "At is when this version last changed state, and By who.",
"riskDataset.counts": "Counts is how the rows fall across the splits.",
"riskDataset.digest": "Digest fingerprints the SPEC and the ROWS together. Two materialisations of\none spec agree on it or the plane says they do not.",
"riskDataset.name": "Name and Version identify the version.",
"riskDataset.oversize": "Oversize is how many of the window's subjects this version could NOT carry\nbecause their subject identity exceeds the plane's per-subject byte bound.\n\nIt is on the wire, not only in a log, because it is the one degradation a\ncaller cannot otherwise detect: the rows that are here look complete, and a\ndataset silently missing a population is a model silently blind to it.\nNon-zero does not make a version invalid — it makes it a version whose\ncoverage is STATED. Zero is the normal case and omits.",
"riskDataset.refusal": "Refusal names why there are no bytes, when there are none.",
"riskDataset.running": "Running is true while THIS process is materialising the version. A version\nthat is `materializing` and not running was started by a process that is\ngone — two states the register cannot tell apart, because a register cannot\nknow which processes are alive.",
"riskDataset.share": "Share is the fraction of the window's subjects admitted, in thousandths.\n1000 means the whole window fitted under the cap; anything less means the\nversion is a reproducible sample and says by how much.",
"riskDataset.spec": "Spec is the bound query this version was built from, exactly as recorded.",
"riskDataset.status": "Status is declared, materializing, ready or refused. Only `ready` has bytes,\nand `ready` is terminal: a published version is never rewritten.",
"riskDataset.truncated": "Truncated is true when the row cap bound before the window ran out. The\ntrailing subject is dropped whole when that happens, because half a subject\non one side of a split is exactly the leak the grouping prevents.",
"riskDatasetRef.name": "Name is the dataset, from the path.",
"riskDatasetSpec.cuts": "Cuts are the two RFC 3339 instants dividing train | val | test. Omit them to\ntake 70% and 85% of the window by time. Splitting is TEMPORAL and then\ngrouped by subject — a random split puts one device on both sides of the\nline and the model memorises the entity instead of the behaviour.",
"riskDatasetSpec.dims": "Dims are the coordinates to carry, by published name. Empty takes the whole\nsurface. They are stored in the plane's own order, never the order given, so\ntwo requests naming the same dims produce identical rows.",
"riskDatasetSpec.from": "From and To bound the event window, half-open, RFC 3339. The window may not\nbe longer than the source's own retention: past that, its older half is\nalready gone and the dataset would silently be shorter than it says.",
"riskDatasetSpec.horizon": "Horizon is how many days a row must have aged before it may be admitted. It\nis what keeps a fact that was not yet knowable at scoring time out of a\ntraining set: a chargeback lands 30 to 120 days after the transaction it\ncondemns, so 120 for the payment lane and 14 for signup abuse. Zero admits\nthe whole window and is honest only where the outcome is immediate.",
"riskDatasetSpec.kind": "Kind narrows to one subject kind — person, session or account. Empty takes\nevery kind.",
"riskDatasetSpec.name": "Name identifies the dataset across its versions: lower-case letters, digits\nand hyphens, starting with a letter.",
"riskDatasetSpec.rows": "Rows caps the materialisation. Zero takes the plane's own bound.",
"riskDatasetSpec.seed": "Seed decides WHICH subjects are admitted when the window holds more rows\nthan the cap allows. It is recorded on the version, so a capped dataset is\nreproducible rather than being whichever rows the store returned first.\nOmit it to seed from the dataset's name.",
"riskSplitCounts.judged": "Judged is how many rows carry a disposition. It is zero until a label plane\nwrites one, and reporting it plainly is what lets a model plane refuse to\nrank rather than name a winner it cannot justify.",
"riskSplitCounts.productive": "Productive and Unproductive are the two judged classes, so the imbalance is\nvisible before anyone trains on it.",
"riskSplitCounts.subjects": "Subjects is how many distinct subjects the rows belong to. Every row of one\nsubject is in ONE split, so this is the real sample size — the row count\nflatters it whenever a subject is active.",
},
Example: json.RawMessage(`{"name":"signups"}`),
})
zip.Describe("GET /v1/ml/datasets/:name/export", zip.Doc{
zip.Describe("GET /v1/risk/datasets/:name/export", zip.Doc{
Description: "Reads a published version's rows back, one bounded page at a\ntime, in the version's own stable row order.\n\nOnly a published version can be exported. Rows written by an attempt that never\ncompleted are inert — no register row names them — and they are disposed of with\nthe dataset.",
Fields: map[string]string{
"mlDatasetRow.at": "At is the row's instant.",
"mlDatasetRow.id": "ID names the row forever. It is DERIVED from the row's own subject and\ninstant, not allocated, so two materialisations of the same fact agree on it\nwithout coordinating.",
"mlDatasetRow.kind": "Kind and Subject name whose row this is.",
"mlDatasetRow.point": "Point is the coordinates, in the order the version's spec names its dims.",
"mlDatasetRow.split": "Split is train, val or test.",
"mlDatasetRows.digest": "Digest is the version's fingerprint. An exported page that did not carry it\nwould be bytes with no way to say which dataset they are.",
"mlDatasetRows.dims": "Dims names what each coordinate of Point means, in Point's own order.",
"mlDatasetRows.offset": "Offset and Limit are the page actually served, which may be smaller than the\none asked for.",
"mlDatasetRows.rows": "Rows is the page. Never null.",
"mlExportIn.limit": "Limit is how many rows to return. Zero and anything above the plane's bound\ntake the bound.",
"mlExportIn.name": "Name is the dataset, from the path.",
"mlExportIn.offset": "Offset is where the page starts, in the version's own row order (by id,\nwhich is derived from the row and therefore stable forever).",
"mlExportIn.split": "Split narrows to train, val or test. Empty reads every split.",
"mlExportIn.version": "Version is the version to read. Zero takes the newest published one.",
"riskDatasetRow.at": "At is the row's instant.",
"riskDatasetRow.id": "ID names the row forever. It is DERIVED from the row's own subject and\ninstant, not allocated, so two materialisations of the same fact agree on it\nwithout coordinating.",
"riskDatasetRow.kind": "Kind and Subject name whose row this is.",
"riskDatasetRow.point": "Point is the coordinates, in the order the version's spec names its dims.",
"riskDatasetRow.split": "Split is train, val or test.",
"riskDatasetRows.digest": "Digest is the version's fingerprint. An exported page that did not carry it\nwould be bytes with no way to say which dataset they are.",
"riskDatasetRows.dims": "Dims names what each coordinate of Point means, in Point's own order.",
"riskDatasetRows.offset": "Offset and Limit are the page actually served, which may be smaller than the\none asked for.",
"riskDatasetRows.rows": "Rows is the page. Never null.",
"riskExportIn.limit": "Limit is how many rows to return. Zero and anything above the plane's bound\ntake the bound.",
"riskExportIn.name": "Name is the dataset, from the path.",
"riskExportIn.offset": "Offset is where the page starts, in the version's own row order (by id,\nwhich is derived from the row and therefore stable forever).",
"riskExportIn.split": "Split narrows to train, val or test. Empty reads every split.",
"riskExportIn.version": "Version is the version to read. Zero takes the newest published one.",
},
Example: json.RawMessage(`{"name":"signups","version":1,"split":"train","limit":500}`),
})
zip.Describe("GET /v1/ml/datasets/:name/lineage", zip.Doc{
zip.Describe("GET /v1/risk/datasets/:name/lineage", zip.Doc{
Description: "Shows where a version's rows came from and whether that can\nstill be demonstrated.\n\nThe answer is MEASURED, not recalled: the plane asks the source the same\nbounded question again and compares it to the fingerprint taken when the\nversion was built. Anything but exact agreement is reported as drift — the\nsource is fed by a rollup that runs behind the events, so \"it holds more now\"\nis the ordinary case and it means re-running the spec would not reproduce this\nversion. An admitted gap is actionable; an unfalsifiable claim is not.\n\nIT IS A PRICED, BOUNDED READ, because it is the same statement a\nmaterialisation is charged for: an exact distinct-count over up to 400 days of\nthis org's feature surface. It takes the org's ONE source-scan slot, so a\ntenant looping it spends one scan and not a thousand; it counts against the\nplane's ceiling, so the fleet's warehouse is bounded too; and it runs under\nthis plane's own deadline rather than the caller's patience.",
Fields: map[string]string{
"mlLineage.digest": "Digest is the version's fingerprint, repeated here so a lineage answer is\nself-contained.",
"mlLineage.from": "From and To are the window actually read — To is the window's end pulled\nback by the maturity horizon, which is usually earlier than the spec's.",
"mlLineage.holds": "Holds is what the source holds for the same window NOW. The difference\nbetween it and Rows is the whole of the reproducibility claim.",
"mlLineage.reproducible": "Reproducible is true when the source still holds what this version was built\nfrom. Refusal says why not, when it is false.",
"mlLineage.retention": "Retention is the source's own expiry rule as the store reports it, read at\nmaterialisation time rather than assumed. A source whose retention is\nshorter than this window cannot re-derive it.",
"mlLineage.rows": "Rows and Subjects are what the source held for that window at\nmaterialisation time.",
"mlLineage.share": "Share is the fraction of subjects admitted, in thousandths.",
"mlLineage.source": "Source is the plane the rows were derived from.",
"mlLineageIn.name": "Name is the dataset, from the path.",
"mlLineageIn.version": "Version is the version to trace. Zero takes the newest published one.",
"riskLineage.digest": "Digest is the version's fingerprint, repeated here so a lineage answer is\nself-contained.",
"riskLineage.from": "From and To are the window actually read — To is the window's end pulled\nback by the maturity horizon, which is usually earlier than the spec's.",
"riskLineage.holds": "Holds is what the source holds for the same window NOW. The difference\nbetween it and Rows is the whole of the reproducibility claim.",
"riskLineage.oversize": "Oversize is how many subjects the window held that were too large to\nrepresent when this version was built. It is part of the fingerprint, so it\nis part of what \"reproducible\" is measured over.",
"riskLineage.reproducible": "Reproducible is true when the source still holds what this version was built\nfrom. Refusal says why not, when it is false.",
"riskLineage.retention": "Retention is the source's own expiry rule as the store reports it, read at\nmaterialisation time rather than assumed. A source whose retention is\nshorter than this window cannot re-derive it.",
"riskLineage.rows": "Rows and Subjects are what the source held for that window at\nmaterialisation time.",
"riskLineage.share": "Share is the fraction of subjects admitted, in thousandths.",
"riskLineage.source": "Source is the plane the rows were derived from.",
"riskLineageIn.name": "Name is the dataset, from the path.",
"riskLineageIn.version": "Version is the version to trace. Zero takes the newest published one.",
},
Example: json.RawMessage(`{"name":"signups","version":1}`),
})
zip.Describe("POST /v1/ml/datasets", zip.Doc{
zip.Describe("POST /v1/risk/datasets", zip.Doc{
Description: "Declares the next version of a dataset from a bound query over\nthis org's own feature surface.\n\nIt mints a VERSION and writes no rows: a version is declared, then materialised\nonce, then never rewritten. Version numbers are monotone and never reused, so\n\"version 3 of signups\" means one thing forever — which is the whole reason a\nmodel can cite one.\n\nThe window is bounded by the source's retention, the horizon by a year, the\nrows by the plane's cap, and the number of datasets and versions per org by\ntheir own limits. Every refusal names which bound it hit.",
Fields: map[string]string{
"mlDataset.at": "At is when this version last changed state, and By who.",
"mlDataset.counts": "Counts is how the rows fall across the splits.",
"mlDataset.digest": "Digest fingerprints the SPEC and the ROWS together. Two materialisations of\none spec agree on it or the plane says they do not.",
"mlDataset.name": "Name and Version identify the version.",
"mlDataset.refusal": "Refusal names why there are no bytes, when there are none.",
"mlDataset.running": "Running is true while THIS process is materialising the version. A version\nthat is `materializing` and not running was started by a process that is\ngone — two states the register cannot tell apart, because a register cannot\nknow which processes are alive.",
"mlDataset.share": "Share is the fraction of the window's subjects admitted, in thousandths.\n1000 means the whole window fitted under the cap; anything less means the\nversion is a reproducible sample and says by how much.",
"mlDataset.spec": "Spec is the bound query this version was built from, exactly as recorded.",
"mlDataset.status": "Status is declared, materializing, ready or refused. Only `ready` has bytes,\nand `ready` is terminal: a published version is never rewritten.",
"mlDataset.truncated": "Truncated is true when the row cap bound before the window ran out. The\ntrailing subject is dropped whole when that happens, because half a subject\non one side of a split is exactly the leak the grouping prevents.",
"mlDatasetSpec.cuts": "Cuts are the two RFC 3339 instants dividing train | val | test. Omit them to\ntake 70% and 85% of the window by time. Splitting is TEMPORAL and then\ngrouped by subject — a random split puts one device on both sides of the\nline and the model memorises the entity instead of the behaviour.",
"mlDatasetSpec.dims": "Dims are the coordinates to carry, by published name. Empty takes the whole\nsurface. They are stored in the plane's own order, never the order given, so\ntwo requests naming the same dims produce identical rows.",
"mlDatasetSpec.from": "From and To bound the event window, half-open, RFC 3339. The window may not\nbe longer than the source's own retention: past that, its older half is\nalready gone and the dataset would silently be shorter than it says.",
"mlDatasetSpec.horizon": "Horizon is how many days a row must have aged before it may be admitted. It\nis what keeps a fact that was not yet knowable at scoring time out of a\ntraining set: a chargeback lands 30 to 120 days after the transaction it\ncondemns, so 120 for the payment lane and 14 for signup abuse. Zero admits\nthe whole window and is honest only where the outcome is immediate.",
"mlDatasetSpec.kind": "Kind narrows to one subject kind — person, session or account. Empty takes\nevery kind.",
"mlDatasetSpec.name": "Name identifies the dataset across its versions: lower-case letters, digits\nand hyphens, starting with a letter.",
"mlDatasetSpec.rows": "Rows caps the materialisation. Zero takes the plane's own bound.",
"mlDatasetSpec.seed": "Seed decides WHICH subjects are admitted when the window holds more rows\nthan the cap allows. It is recorded on the version, so a capped dataset is\nreproducible rather than being whichever rows the store returned first.\nOmit it to seed from the dataset's name.",
"mlSplitCounts.judged": "Judged is how many rows carry a disposition. It is zero until a label plane\nwrites one, and reporting it plainly is what lets a model plane refuse to\nrank rather than name a winner it cannot justify.",
"mlSplitCounts.productive": "Productive and Unproductive are the two judged classes, so the imbalance is\nvisible before anyone trains on it.",
"mlSplitCounts.subjects": "Subjects is how many distinct subjects the rows belong to. Every row of one\nsubject is in ONE split, so this is the real sample size — the row count\nflatters it whenever a subject is active.",
"riskDataset.at": "At is when this version last changed state, and By who.",
"riskDataset.counts": "Counts is how the rows fall across the splits.",
"riskDataset.digest": "Digest fingerprints the SPEC and the ROWS together. Two materialisations of\none spec agree on it or the plane says they do not.",
"riskDataset.name": "Name and Version identify the version.",
"riskDataset.oversize": "Oversize is how many of the window's subjects this version could NOT carry\nbecause their subject identity exceeds the plane's per-subject byte bound.\n\nIt is on the wire, not only in a log, because it is the one degradation a\ncaller cannot otherwise detect: the rows that are here look complete, and a\ndataset silently missing a population is a model silently blind to it.\nNon-zero does not make a version invalid — it makes it a version whose\ncoverage is STATED. Zero is the normal case and omits.",
"riskDataset.refusal": "Refusal names why there are no bytes, when there are none.",
"riskDataset.running": "Running is true while THIS process is materialising the version. A version\nthat is `materializing` and not running was started by a process that is\ngone — two states the register cannot tell apart, because a register cannot\nknow which processes are alive.",
"riskDataset.share": "Share is the fraction of the window's subjects admitted, in thousandths.\n1000 means the whole window fitted under the cap; anything less means the\nversion is a reproducible sample and says by how much.",
"riskDataset.spec": "Spec is the bound query this version was built from, exactly as recorded.",
"riskDataset.status": "Status is declared, materializing, ready or refused. Only `ready` has bytes,\nand `ready` is terminal: a published version is never rewritten.",
"riskDataset.truncated": "Truncated is true when the row cap bound before the window ran out. The\ntrailing subject is dropped whole when that happens, because half a subject\non one side of a split is exactly the leak the grouping prevents.",
"riskDatasetSpec.cuts": "Cuts are the two RFC 3339 instants dividing train | val | test. Omit them to\ntake 70% and 85% of the window by time. Splitting is TEMPORAL and then\ngrouped by subject — a random split puts one device on both sides of the\nline and the model memorises the entity instead of the behaviour.",
"riskDatasetSpec.dims": "Dims are the coordinates to carry, by published name. Empty takes the whole\nsurface. They are stored in the plane's own order, never the order given, so\ntwo requests naming the same dims produce identical rows.",
"riskDatasetSpec.from": "From and To bound the event window, half-open, RFC 3339. The window may not\nbe longer than the source's own retention: past that, its older half is\nalready gone and the dataset would silently be shorter than it says.",
"riskDatasetSpec.horizon": "Horizon is how many days a row must have aged before it may be admitted. It\nis what keeps a fact that was not yet knowable at scoring time out of a\ntraining set: a chargeback lands 30 to 120 days after the transaction it\ncondemns, so 120 for the payment lane and 14 for signup abuse. Zero admits\nthe whole window and is honest only where the outcome is immediate.",
"riskDatasetSpec.kind": "Kind narrows to one subject kind — person, session or account. Empty takes\nevery kind.",
"riskDatasetSpec.name": "Name identifies the dataset across its versions: lower-case letters, digits\nand hyphens, starting with a letter.",
"riskDatasetSpec.rows": "Rows caps the materialisation. Zero takes the plane's own bound.",
"riskDatasetSpec.seed": "Seed decides WHICH subjects are admitted when the window holds more rows\nthan the cap allows. It is recorded on the version, so a capped dataset is\nreproducible rather than being whichever rows the store returned first.\nOmit it to seed from the dataset's name.",
"riskSplitCounts.judged": "Judged is how many rows carry a disposition. It is zero until a label plane\nwrites one, and reporting it plainly is what lets a model plane refuse to\nrank rather than name a winner it cannot justify.",
"riskSplitCounts.productive": "Productive and Unproductive are the two judged classes, so the imbalance is\nvisible before anyone trains on it.",
"riskSplitCounts.subjects": "Subjects is how many distinct subjects the rows belong to. Every row of one\nsubject is in ONE split, so this is the real sample size — the row count\nflatters it whenever a subject is active.",
},
Example: json.RawMessage(`{"name":"signups","kind":"person","from":"2026-01-01T00:00:00Z","to":"2026-04-01T00:00:00Z","horizon":14}`),
})
zip.Describe("POST /v1/ml/datasets/:name/materialize", zip.Doc{
zip.Describe("POST /v1/risk/datasets/:name/materialize", zip.Doc{
Description: "Builds the declared version into immutable rows and answers\n202 as soon as the attempt is on record.\n\nIt never holds the request open for the work: a materialisation is a bounded\nwarehouse scan, and letting an HTTP client's timeout be a data plane's timeout\nis how one tenant's retry loop becomes everyone's outage. ONE materialisation\nruns per org at a time; a second is refused rather than queued, because a queue\nadmits the same work later and the honest answer to \"again\" while one is\nrunning is that one is running.\n\nOnly a DECLARED version is admitted. A published version is immutable, and a\nversion whose earlier attempt did not complete is never re-attempted — that\nwould union two runs' rows under one number and make the digest a lie. In both\ncases the answer is to declare a new version, which is what a second run over a\nmoving source honestly is.",
Fields: map[string]string{
"mlDataset.at": "At is when this version last changed state, and By who.",
"mlDataset.counts": "Counts is how the rows fall across the splits.",
"mlDataset.digest": "Digest fingerprints the SPEC and the ROWS together. Two materialisations of\none spec agree on it or the plane says they do not.",
"mlDataset.name": "Name and Version identify the version.",
"mlDataset.refusal": "Refusal names why there are no bytes, when there are none.",
"mlDataset.running": "Running is true while THIS process is materialising the version. A version\nthat is `materializing` and not running was started by a process that is\ngone — two states the register cannot tell apart, because a register cannot\nknow which processes are alive.",
"mlDataset.share": "Share is the fraction of the window's subjects admitted, in thousandths.\n1000 means the whole window fitted under the cap; anything less means the\nversion is a reproducible sample and says by how much.",
"mlDataset.spec": "Spec is the bound query this version was built from, exactly as recorded.",
"mlDataset.status": "Status is declared, materializing, ready or refused. Only `ready` has bytes,\nand `ready` is terminal: a published version is never rewritten.",
"mlDataset.truncated": "Truncated is true when the row cap bound before the window ran out. The\ntrailing subject is dropped whole when that happens, because half a subject\non one side of a split is exactly the leak the grouping prevents.",
"mlDatasetSpec.cuts": "Cuts are the two RFC 3339 instants dividing train | val | test. Omit them to\ntake 70% and 85% of the window by time. Splitting is TEMPORAL and then\ngrouped by subject — a random split puts one device on both sides of the\nline and the model memorises the entity instead of the behaviour.",
"mlDatasetSpec.dims": "Dims are the coordinates to carry, by published name. Empty takes the whole\nsurface. They are stored in the plane's own order, never the order given, so\ntwo requests naming the same dims produce identical rows.",
"mlDatasetSpec.from": "From and To bound the event window, half-open, RFC 3339. The window may not\nbe longer than the source's own retention: past that, its older half is\nalready gone and the dataset would silently be shorter than it says.",
"mlDatasetSpec.horizon": "Horizon is how many days a row must have aged before it may be admitted. It\nis what keeps a fact that was not yet knowable at scoring time out of a\ntraining set: a chargeback lands 30 to 120 days after the transaction it\ncondemns, so 120 for the payment lane and 14 for signup abuse. Zero admits\nthe whole window and is honest only where the outcome is immediate.",
"mlDatasetSpec.kind": "Kind narrows to one subject kind — person, session or account. Empty takes\nevery kind.",
"mlDatasetSpec.name": "Name identifies the dataset across its versions: lower-case letters, digits\nand hyphens, starting with a letter.",
"mlDatasetSpec.rows": "Rows caps the materialisation. Zero takes the plane's own bound.",
"mlDatasetSpec.seed": "Seed decides WHICH subjects are admitted when the window holds more rows\nthan the cap allows. It is recorded on the version, so a capped dataset is\nreproducible rather than being whichever rows the store returned first.\nOmit it to seed from the dataset's name.",
"mlMaterializeIn.name": "Name is the dataset, from the path.",
"mlSplitCounts.judged": "Judged is how many rows carry a disposition. It is zero until a label plane\nwrites one, and reporting it plainly is what lets a model plane refuse to\nrank rather than name a winner it cannot justify.",
"mlSplitCounts.productive": "Productive and Unproductive are the two judged classes, so the imbalance is\nvisible before anyone trains on it.",
"mlSplitCounts.subjects": "Subjects is how many distinct subjects the rows belong to. Every row of one\nsubject is in ONE split, so this is the real sample size — the row count\nflatters it whenever a subject is active.",
"riskDataset.at": "At is when this version last changed state, and By who.",
"riskDataset.counts": "Counts is how the rows fall across the splits.",
"riskDataset.digest": "Digest fingerprints the SPEC and the ROWS together. Two materialisations of\none spec agree on it or the plane says they do not.",
"riskDataset.name": "Name and Version identify the version.",
"riskDataset.oversize": "Oversize is how many of the window's subjects this version could NOT carry\nbecause their subject identity exceeds the plane's per-subject byte bound.\n\nIt is on the wire, not only in a log, because it is the one degradation a\ncaller cannot otherwise detect: the rows that are here look complete, and a\ndataset silently missing a population is a model silently blind to it.\nNon-zero does not make a version invalid — it makes it a version whose\ncoverage is STATED. Zero is the normal case and omits.",
"riskDataset.refusal": "Refusal names why there are no bytes, when there are none.",
"riskDataset.running": "Running is true while THIS process is materialising the version. A version\nthat is `materializing` and not running was started by a process that is\ngone — two states the register cannot tell apart, because a register cannot\nknow which processes are alive.",
"riskDataset.share": "Share is the fraction of the window's subjects admitted, in thousandths.\n1000 means the whole window fitted under the cap; anything less means the\nversion is a reproducible sample and says by how much.",
"riskDataset.spec": "Spec is the bound query this version was built from, exactly as recorded.",
"riskDataset.status": "Status is declared, materializing, ready or refused. Only `ready` has bytes,\nand `ready` is terminal: a published version is never rewritten.",
"riskDataset.truncated": "Truncated is true when the row cap bound before the window ran out. The\ntrailing subject is dropped whole when that happens, because half a subject\non one side of a split is exactly the leak the grouping prevents.",
"riskDatasetSpec.cuts": "Cuts are the two RFC 3339 instants dividing train | val | test. Omit them to\ntake 70% and 85% of the window by time. Splitting is TEMPORAL and then\ngrouped by subject — a random split puts one device on both sides of the\nline and the model memorises the entity instead of the behaviour.",
"riskDatasetSpec.dims": "Dims are the coordinates to carry, by published name. Empty takes the whole\nsurface. They are stored in the plane's own order, never the order given, so\ntwo requests naming the same dims produce identical rows.",
"riskDatasetSpec.from": "From and To bound the event window, half-open, RFC 3339. The window may not\nbe longer than the source's own retention: past that, its older half is\nalready gone and the dataset would silently be shorter than it says.",
"riskDatasetSpec.horizon": "Horizon is how many days a row must have aged before it may be admitted. It\nis what keeps a fact that was not yet knowable at scoring time out of a\ntraining set: a chargeback lands 30 to 120 days after the transaction it\ncondemns, so 120 for the payment lane and 14 for signup abuse. Zero admits\nthe whole window and is honest only where the outcome is immediate.",
"riskDatasetSpec.kind": "Kind narrows to one subject kind — person, session or account. Empty takes\nevery kind.",
"riskDatasetSpec.name": "Name identifies the dataset across its versions: lower-case letters, digits\nand hyphens, starting with a letter.",
"riskDatasetSpec.rows": "Rows caps the materialisation. Zero takes the plane's own bound.",
"riskDatasetSpec.seed": "Seed decides WHICH subjects are admitted when the window holds more rows\nthan the cap allows. It is recorded on the version, so a capped dataset is\nreproducible rather than being whichever rows the store returned first.\nOmit it to seed from the dataset's name.",
"riskMaterializeIn.name": "Name is the dataset, from the path.",
"riskSplitCounts.judged": "Judged is how many rows carry a disposition. It is zero until a label plane\nwrites one, and reporting it plainly is what lets a model plane refuse to\nrank rather than name a winner it cannot justify.",
"riskSplitCounts.productive": "Productive and Unproductive are the two judged classes, so the imbalance is\nvisible before anyone trains on it.",
"riskSplitCounts.subjects": "Subjects is how many distinct subjects the rows belong to. Every row of one\nsubject is in ONE split, so this is the real sample size — the row count\nflatters it whenever a subject is active.",
},
Example: json.RawMessage(`{"name":"signups"}`),
})
+17 -10
View File
@@ -127,16 +127,23 @@ var Apps = []App{
// /v1/risk/health is this app's own REAL probe (OwnsHealth), which the generic
// always-ok liveness route would otherwise shadow.
{Name: "risk", Prefixes: []string{"/v1/risk"}},
// The dataset plane sits BESIDE ml under /v1/ml rather than inside it, because
// the two share a face and nothing else. ml is a Kubernetes CRD bridge whose
// tenant boundary is a per-org NAMESPACE and whose failure domain is the
// cluster; this is a warehouse-backed record plane whose tenant boundary is a
// qualified `<brand>/<org>` KEY and whose failure domain is the columnar store.
// One package holding two tenancy models is the shape a privilege bug grows in,
// so they are two rows claiming two disjoint sets of leaves — and zip refuses
// two owners for one prefix at compose time, which checks it rather than
// trusting it.
{Name: "dataset", Prefixes: []string{"/v1/ml/datasets"}},
// dataset is the risk product's RECORD plane, and it is addressed UNDER /v1/risk
// because that is the product it belongs to: it owns hanzo.risk_dataset and
// hanzo.risk_row and reads hanzo.risk_feature. It was written at /v1/ml/datasets
// before /v1/risk existed — and /v1/ml is model SERVING, live with customers, so
// that address named the wrong product. Nothing published it, so it moved.
//
// A SEPARATE ROW FROM `risk`, not a folded-in one, because the two hold different
// tenancy models and different failure domains: risk's tenant boundary is
// in-process per-org counters and its failure domain is this process, while this
// plane's boundary is a qualified `<brand>/<org>` KEY and its failure domain is
// the columnar store. One package holding two tenancy models is the shape a
// privilege bug grows in.
//
// The nesting is the routing rule, not a collision: [OwnerOf] and the router both
// take the LONGEST match, which is how `storage` already holds /v1/s3/buckets
// against provisioning's /v1/s3. No risk route moves.
{Name: "dataset", Prefixes: []string{"/v1/risk/datasets"}},
{Name: "usage", Prefixes: []string{"/v1/usage"}},
{Name: "leaderboard", Prefixes: []string{"/v1/usage/activity", "/v1/usage/leaderboard", "/v1/usage/rollup/backfill"}},
{Name: "crm", Prefixes: []string{"/v1/crm"}},
+564 -547
View File
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -112,7 +112,7 @@
"mesh": 1,
"messages": 2,
"metrics": 4,
"ml": 14,
"ml": 7,
"models": 3,
"mq": 15,
"networks": 3,
@@ -141,7 +141,7 @@
"rerank": 1,
"research": 8,
"responses": 1,
"risk": 10,
"risk": 17,
"router": 40,
"run": 1,
"runner": 3,
+2 -2
View File
@@ -21,12 +21,12 @@ import (
// cost nothing and declare nothing — Price is what ONE request to this surface
// costs at the edge gate, and the per-op fee is stated in the app.
//
// It does not own health. Serve's generic /v1/dataset/health liveness route is
// It does not own health. Listen's generic /v1/dataset/health liveness route is
// the honest answer for this plane: everything it knows is in the store, so a
// probe of its own would report the store's reachability, which is already what
// every op reports in band as a 503 rather than as an empty answer.
func main() {
if err := cloud.Serve([]cloud.Plugin{{
if err := cloud.Listen([]cloud.Plugin{{
Name: "dataset",
Price: cloud.Metered,
Mount: dataset.Mount,
-145
View File
@@ -1,145 +0,0 @@
[
{
"description": "Declares the next version of a dataset from a bound query over\nthis org's own feature surface.\n\nIt mints a VERSION and writes no rows: a version is declared, then materialised\nonce, then never rewritten. Version numbers are monotone and never reused, so\n\"version 3 of signups\" means one thing forever — which is the whole reason a\nmodel can cite one.\n\nThe window is bounded by the source's retention, the horizon by a year, the\nrows by the plane's cap, and the number of datasets and versions per org by\ntheir own limits. Every refusal names which bound it hit.",
"inputSchema": {
"properties": {
"cuts": {
"description": "Cuts are the two RFC 3339 instants dividing train | val | test. Omit them to\ntake 70% and 85% of the window by time. Splitting is TEMPORAL and then\ngrouped by subject — a random split puts one device on both sides of the\nline and the model memorises the entity instead of the behaviour.",
"items": {
"type": "string"
},
"type": "array"
},
"dims": {
"description": "Dims are the coordinates to carry, by published name. Empty takes the whole\nsurface. They are stored in the plane's own order, never the order given, so\ntwo requests naming the same dims produce identical rows.",
"items": {
"type": "string"
},
"type": "array"
},
"from": {
"description": "From and To bound the event window, half-open, RFC 3339. The window may not\nbe longer than the source's own retention: past that, its older half is\nalready gone and the dataset would silently be shorter than it says.",
"type": "string"
},
"horizon": {
"description": "Horizon is how many days a row must have aged before it may be admitted. It\nis what keeps a fact that was not yet knowable at scoring time out of a\ntraining set: a chargeback lands 30 to 120 days after the transaction it\ncondemns, so 120 for the payment lane and 14 for signup abuse. Zero admits\nthe whole window and is honest only where the outcome is immediate.",
"type": "integer"
},
"kind": {
"description": "Kind narrows to one subject kind — person, session or account. Empty takes\nevery kind.",
"type": "string"
},
"name": {
"description": "Name identifies the dataset across its versions: lower-case letters, digits\nand hyphens, starting with a letter.",
"type": "string"
},
"rows": {
"description": "Rows caps the materialisation. Zero takes the plane's own bound.",
"type": "integer"
},
"seed": {
"description": "Seed decides WHICH subjects are admitted when the window holds more rows\nthan the cap allows. It is recorded on the version, so a capped dataset is\nreproducible rather than being whichever rows the store returned first.\nOmit it to seed from the dataset's name.",
"type": "string"
},
"to": {
"type": "string"
}
},
"type": "object"
},
"name": "mlCreateDataset"
},
{
"description": "Dataset describes every version of one dataset, newest first — the whole\nhistory, because the point of a version is that the older ones are still there\nand a model fitted last quarter cites one of them.\n\nA name this org does not own answers 404, exactly as an unknown name does, so a\nprobe learns nothing about another tenant's datasets.",
"inputSchema": {
"properties": {
"name": {
"description": "Name is the dataset, from the path.",
"type": "string"
}
},
"type": "object"
},
"name": "mlDataset"
},
{
"description": "Shows where a version's rows came from and whether that can\nstill be demonstrated.\n\nThe answer is MEASURED, not recalled: the plane asks the source the same\nbounded question again and compares it to the fingerprint taken when the\nversion was built. Anything but exact agreement is reported as drift — the\nsource is fed by a rollup that runs behind the events, so \"it holds more now\"\nis the ordinary case and it means re-running the spec would not reproduce this\nversion. An admitted gap is actionable; an unfalsifiable claim is not.\n\nIT IS A PRICED, BOUNDED READ, because it is the same statement a\nmaterialisation is charged for: an exact distinct-count over up to 400 days of\nthis org's feature surface. It takes the org's ONE source-scan slot, so a\ntenant looping it spends one scan and not a thousand; it counts against the\nplane's ceiling, so the fleet's warehouse is bounded too; and it runs under\nthis plane's own deadline rather than the caller's patience.",
"inputSchema": {
"properties": {
"name": {
"description": "Name is the dataset, from the path.",
"type": "string"
},
"version": {
"description": "Version is the version to trace. Zero takes the newest published one.",
"type": "integer"
}
},
"type": "object"
},
"name": "mlDatasetLineage"
},
{
"description": "Datasets lists this org's datasets, each with its newest version. An org that\nhas declared none gets an empty list; a store that cannot be reached gets a\nrefusal, never an empty list, because the two read identically and only one of\nthem is true.",
"inputSchema": {
"properties": {},
"type": "object"
},
"name": "mlDatasets"
},
{
"description": "Disposes of one dataset and every version of it: the rows are\ndropped and the register is marked with what went.\n\nThis is the ONLY expiry in this plane. Neither table carries a TTL, deliberately:\na table TTL is a fleet-wide clock no tenant can hold longer or shorten, which is\nthe opposite of a retention decision belonging to the tenant whose records they\nare. The drop is a partition drop on (org, dataset), so the tenant is the first\ncomponent of the thing being dropped and a disposal cannot be spelled across one.\n\nThe BYTES are what goes. The register keeps one `disposed` row per version — the\nname, the number, the spec, the digest and who disposed of it when — for two\nreasons: a retention obligation is answered by a record of the deletion, not by\nsilence; and version numbers must stay monotone, so that after `orders` is\ndisposed of and declared again the next version is 4 and not 1. A number that\ncould be reused would make every citation of `orders v3` ambiguous forever.\n\nIt is not reversible and there is no soft state in between. A version a model\ncited has no rows once this returns, and every read of it says so.",
"inputSchema": {
"properties": {
"name": {
"description": "Name is the dataset, from the path.",
"type": "string"
}
},
"type": "object"
},
"name": "mlDeleteDataset"
},
{
"description": "Reads a published version's rows back, one bounded page at a\ntime, in the version's own stable row order.\n\nOnly a published version can be exported. Rows written by an attempt that never\ncompleted are inert — no register row names them — and they are disposed of with\nthe dataset.",
"inputSchema": {
"properties": {
"limit": {
"description": "Limit is how many rows to return. Zero and anything above the plane's bound\ntake the bound.",
"type": "integer"
},
"name": {
"description": "Name is the dataset, from the path.",
"type": "string"
},
"offset": {
"description": "Offset is where the page starts, in the version's own row order (by id,\nwhich is derived from the row and therefore stable forever).",
"type": "integer"
},
"split": {
"description": "Split narrows to train, val or test. Empty reads every split.",
"type": "string"
},
"version": {
"description": "Version is the version to read. Zero takes the newest published one.",
"type": "integer"
}
},
"type": "object"
},
"name": "mlExportDataset"
},
{
"description": "Builds the declared version into immutable rows and answers\n202 as soon as the attempt is on record.\n\nIt never holds the request open for the work: a materialisation is a bounded\nwarehouse scan, and letting an HTTP client's timeout be a data plane's timeout\nis how one tenant's retry loop becomes everyone's outage. ONE materialisation\nruns per org at a time; a second is refused rather than queued, because a queue\nadmits the same work later and the honest answer to \"again\" while one is\nrunning is that one is running.\n\nOnly a DECLARED version is admitted. A published version is immutable, and a\nversion whose earlier attempt did not complete is never re-attempted — that\nwould union two runs' rows under one number and make the digest a lie. In both\ncases the answer is to declare a new version, which is what a second run over a\nmoving source honestly is.",
"inputSchema": {
"properties": {
"name": {
"description": "Name is the dataset, from the path.",
"type": "string"
}
},
"type": "object"
},
"name": "mlMaterializeDataset"
}
]
+51 -43
View File
@@ -2,7 +2,7 @@
"openapi": "3.1.0",
"info": {
"title": "Hanzo Cloud API",
"description": "Package dataset is the per-org dataset plane of /v1/ml: a dataset is a VERSIONED, IMMUTABLE snapshot of one tenant's own event surface, and this is where it is declared, materialised, described, exported and disposed of.",
"description": "Package dataset is the per-org dataset plane of /v1/risk: a dataset is a VERSIONED, IMMUTABLE snapshot of one tenant's own event surface, and this is where it is declared, materialised, described, exported and disposed of.",
"version": "v1"
},
"servers": [
@@ -12,24 +12,24 @@
],
"tags": [
{
"name": "ml"
"name": "risk"
}
],
"paths": {
"/v1/ml/datasets": {
"/v1/risk/datasets": {
"get": {
"operationId": "mlDatasets",
"operationId": "riskDatasets",
"summary": "List this org's datasets",
"description": "Datasets lists this org's datasets, each with its newest version. An org that\nhas declared none gets an empty list; a store that cannot be reached gets a\nrefusal, never an empty list, because the two read identically and only one of\nthem is true.",
"tags": [
"ml"
"risk"
],
"responses": {
"200": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/mlDatasetList"
"$ref": "#/components/schemas/riskDatasetList"
}
}
},
@@ -38,11 +38,11 @@
}
},
"post": {
"operationId": "mlCreateDataset",
"operationId": "riskCreateDataset",
"summary": "Declare the next version of a dataset",
"description": "Declares the next version of a dataset from a bound query over\nthis org's own feature surface.\n\nIt mints a VERSION and writes no rows: a version is declared, then materialised\nonce, then never rewritten. Version numbers are monotone and never reused, so\n\"version 3 of signups\" means one thing forever — which is the whole reason a\nmodel can cite one.\n\nThe window is bounded by the source's retention, the horizon by a year, the\nrows by the plane's cap, and the number of datasets and versions per org by\ntheir own limits. Every refusal names which bound it hit.",
"tags": [
"ml"
"risk"
],
"requestBody": {
"content": {
@@ -55,7 +55,7 @@
"to": "2026-04-01T00:00:00Z"
},
"schema": {
"$ref": "#/components/schemas/mlDatasetSpec"
"$ref": "#/components/schemas/riskDatasetSpec"
}
}
},
@@ -66,7 +66,7 @@
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/mlDataset"
"$ref": "#/components/schemas/riskDataset"
}
}
},
@@ -75,13 +75,13 @@
}
}
},
"/v1/ml/datasets/{name}": {
"/v1/risk/datasets/{name}": {
"delete": {
"operationId": "mlDeleteDataset",
"operationId": "riskDeleteDataset",
"summary": "Dispose of one dataset and every version of it",
"description": "Disposes of one dataset and every version of it: the rows are\ndropped and the register is marked with what went.\n\nThis is the ONLY expiry in this plane. Neither table carries a TTL, deliberately:\na table TTL is a fleet-wide clock no tenant can hold longer or shorten, which is\nthe opposite of a retention decision belonging to the tenant whose records they\nare. The drop is a partition drop on (org, dataset), so the tenant is the first\ncomponent of the thing being dropped and a disposal cannot be spelled across one.\n\nThe BYTES are what goes. The register keeps one `disposed` row per version — the\nname, the number, the spec, the digest and who disposed of it when — for two\nreasons: a retention obligation is answered by a record of the deletion, not by\nsilence; and version numbers must stay monotone, so that after `orders` is\ndisposed of and declared again the next version is 4 and not 1. A number that\ncould be reused would make every citation of `orders v3` ambiguous forever.\n\nIt is not reversible and there is no soft state in between. A version a model\ncited has no rows once this returns, and every read of it says so.",
"tags": [
"ml"
"risk"
],
"parameters": [
{
@@ -100,7 +100,7 @@
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/mlDisposal"
"$ref": "#/components/schemas/riskDisposal"
}
}
},
@@ -109,11 +109,11 @@
}
},
"get": {
"operationId": "mlDataset",
"operationId": "riskDataset",
"summary": "Describe every version of one dataset",
"description": "Dataset describes every version of one dataset, newest first — the whole\nhistory, because the point of a version is that the older ones are still there\nand a model fitted last quarter cites one of them.\n\nA name this org does not own answers 404, exactly as an unknown name does, so a\nprobe learns nothing about another tenant's datasets.",
"tags": [
"ml"
"risk"
],
"parameters": [
{
@@ -132,7 +132,7 @@
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/mlDatasetVersions"
"$ref": "#/components/schemas/riskDatasetVersions"
}
}
},
@@ -141,13 +141,13 @@
}
}
},
"/v1/ml/datasets/{name}/export": {
"/v1/risk/datasets/{name}/export": {
"get": {
"operationId": "mlExportDataset",
"operationId": "riskExportDataset",
"summary": "Read a version's rows back, one page at a time",
"description": "Reads a published version's rows back, one bounded page at a\ntime, in the version's own stable row order.\n\nOnly a published version can be exported. Rows written by an attempt that never\ncompleted are inert — no register row names them — and they are disposed of with\nthe dataset.",
"tags": [
"ml"
"risk"
],
"parameters": [
{
@@ -205,7 +205,7 @@
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/mlDatasetRows"
"$ref": "#/components/schemas/riskDatasetRows"
}
}
},
@@ -214,13 +214,13 @@
}
}
},
"/v1/ml/datasets/{name}/lineage": {
"/v1/risk/datasets/{name}/lineage": {
"get": {
"operationId": "mlDatasetLineage",
"operationId": "riskDatasetLineage",
"summary": "Show where a version's rows came from, and whether that can still be demonstrated",
"description": "Shows where a version's rows came from and whether that can\nstill be demonstrated.\n\nThe answer is MEASURED, not recalled: the plane asks the source the same\nbounded question again and compares it to the fingerprint taken when the\nversion was built. Anything but exact agreement is reported as drift — the\nsource is fed by a rollup that runs behind the events, so \"it holds more now\"\nis the ordinary case and it means re-running the spec would not reproduce this\nversion. An admitted gap is actionable; an unfalsifiable claim is not.\n\nIT IS A PRICED, BOUNDED READ, because it is the same statement a\nmaterialisation is charged for: an exact distinct-count over up to 400 days of\nthis org's feature surface. It takes the org's ONE source-scan slot, so a\ntenant looping it spends one scan and not a thousand; it counts against the\nplane's ceiling, so the fleet's warehouse is bounded too; and it runs under\nthis plane's own deadline rather than the caller's patience.",
"tags": [
"ml"
"risk"
],
"parameters": [
{
@@ -249,7 +249,7 @@
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/mlLineage"
"$ref": "#/components/schemas/riskLineage"
}
}
},
@@ -258,13 +258,13 @@
}
}
},
"/v1/ml/datasets/{name}/materialize": {
"/v1/risk/datasets/{name}/materialize": {
"post": {
"operationId": "mlMaterializeDataset",
"operationId": "riskMaterializeDataset",
"summary": "Materialise the declared version into immutable rows",
"description": "Builds the declared version into immutable rows and answers\n202 as soon as the attempt is on record.\n\nIt never holds the request open for the work: a materialisation is a bounded\nwarehouse scan, and letting an HTTP client's timeout be a data plane's timeout\nis how one tenant's retry loop becomes everyone's outage. ONE materialisation\nruns per org at a time; a second is refused rather than queued, because a queue\nadmits the same work later and the honest answer to \"again\" while one is\nrunning is that one is running.\n\nOnly a DECLARED version is admitted. A published version is immutable, and a\nversion whose earlier attempt did not complete is never re-attempted — that\nwould union two runs' rows under one number and make the digest a lie. In both\ncases the answer is to declare a new version, which is what a second run over a\nmoving source honestly is.",
"tags": [
"ml"
"risk"
],
"parameters": [
{
@@ -283,7 +283,7 @@
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/mlDataset"
"$ref": "#/components/schemas/riskDataset"
}
}
},
@@ -295,7 +295,7 @@
},
"components": {
"schemas": {
"mlDataset": {
"riskDataset": {
"properties": {
"at": {
"description": "At is when this version last changed state, and By who.",
@@ -305,7 +305,7 @@
"type": "string"
},
"counts": {
"$ref": "#/components/schemas/mlSplitCounts",
"$ref": "#/components/schemas/riskSplitCounts",
"description": "Counts is how the rows fall across the splits."
},
"digest": {
@@ -316,6 +316,10 @@
"description": "Name and Version identify the version.",
"type": "string"
},
"oversize": {
"description": "Oversize is how many of the window's subjects this version could NOT carry\nbecause their subject identity exceeds the plane's per-subject byte bound.\n\nIt is on the wire, not only in a log, because it is the one degradation a\ncaller cannot otherwise detect: the rows that are here look complete, and a\ndataset silently missing a population is a model silently blind to it.\nNon-zero does not make a version invalid — it makes it a version whose\ncoverage is STATED. Zero is the normal case and omits.",
"type": "integer"
},
"refusal": {
"description": "Refusal names why there are no bytes, when there are none.",
"type": "string"
@@ -329,7 +333,7 @@
"type": "integer"
},
"spec": {
"$ref": "#/components/schemas/mlDatasetSpec",
"$ref": "#/components/schemas/riskDatasetSpec",
"description": "Spec is the bound query this version was built from, exactly as recorded."
},
"status": {
@@ -346,19 +350,19 @@
},
"type": "object"
},
"mlDatasetList": {
"riskDatasetList": {
"properties": {
"items": {
"description": "Items is one entry per dataset, carrying its newest version. Never null: an\norg that has declared nothing gets an empty array.",
"items": {
"$ref": "#/components/schemas/mlDataset"
"$ref": "#/components/schemas/riskDataset"
},
"type": "array"
}
},
"type": "object"
},
"mlDatasetRow": {
"riskDatasetRow": {
"properties": {
"at": {
"description": "At is the row's instant.",
@@ -389,7 +393,7 @@
},
"type": "object"
},
"mlDatasetRows": {
"riskDatasetRows": {
"properties": {
"dataset": {
"type": "string"
@@ -415,7 +419,7 @@
"rows": {
"description": "Rows is the page. Never null.",
"items": {
"$ref": "#/components/schemas/mlDatasetRow"
"$ref": "#/components/schemas/riskDatasetRow"
},
"type": "array"
},
@@ -425,7 +429,7 @@
},
"type": "object"
},
"mlDatasetSpec": {
"riskDatasetSpec": {
"properties": {
"cuts": {
"description": "Cuts are the two RFC 3339 instants dividing train | val | test. Omit them to\ntake 70% and 85% of the window by time. Splitting is TEMPORAL and then\ngrouped by subject — a random split puts one device on both sides of the\nline and the model memorises the entity instead of the behaviour.",
@@ -471,11 +475,11 @@
},
"type": "object"
},
"mlDatasetVersions": {
"riskDatasetVersions": {
"properties": {
"items": {
"items": {
"$ref": "#/components/schemas/mlDataset"
"$ref": "#/components/schemas/riskDataset"
},
"type": "array"
},
@@ -485,7 +489,7 @@
},
"type": "object"
},
"mlDisposal": {
"riskDisposal": {
"properties": {
"dataset": {
"type": "string"
@@ -500,7 +504,7 @@
},
"type": "object"
},
"mlLineage": {
"riskLineage": {
"properties": {
"dataset": {
"type": "string"
@@ -517,6 +521,10 @@
"description": "Holds is what the source holds for the same window NOW. The difference\nbetween it and Rows is the whole of the reproducibility claim.",
"type": "integer"
},
"oversize": {
"description": "Oversize is how many subjects the window held that were too large to\nrepresent when this version was built. It is part of the fingerprint, so it\nis part of what \"reproducible\" is measured over.",
"type": "integer"
},
"refusal": {
"type": "string"
},
@@ -552,7 +560,7 @@
},
"type": "object"
},
"mlSplitCounts": {
"riskSplitCounts": {
"properties": {
"judged": {
"description": "Judged is how many rows carry a disposition. It is zero until a label plane\nwrites one, and reporting it plainly is what lets a model plane refuse to\nrank rather than name a winner it cannot justify.",
+1 -1
View File
@@ -279,7 +279,7 @@ var meteredApps = []string{
"automations", // per-run automation fee.
"cloudflare", // Workers AI + provisioning.
"content", // studio renders (GPU).
"dataset", // the scan that materialises a set, priced per source row read.
"dataset", // materialising a version: a bounded scan of the shared warehouse.
"flow", // flow executions.
"functions", // serverless invoke.
"ml", // predict + train (compute).