zip refuses a program whose middleware has no routes beneath it, and the refusal
was right about three surfaces here. Nothing in this repo called Build() from a
test, so the refusal could only ever surface as a startup panic — or, for a
subsystem no test drove, as silence.
auditlog and catalog declared Bridge (and audit's noStore) on a group at their
own prefix while declaring the op on the App with its WHOLE path. The op is
therefore that group's SIBLING, not its child, so the group's subtree was empty
and the middleware could not run. Both packages' entire test suites were
panicking out of app.Test, which builds. Use is the verb that says the true
thing, and it needs no second copy of the prefix: cloud's scope bounds it to the
subsystem's declared subtrees, and a bare *zip.App treats root middleware as
live. The ops keep their exact paths — moving them onto the group with an empty
leaf would publish /v1/audit/ and /v1/catalog/, which neither API has served.
zen was worse and not the same defect. Its Claim gates ai's "/v1", it declares no
prefix of its own (Coresident), and plugin/zen fed manifest.PrefixesFor("zen") —
a ROUTING answer — into cloud.Plugin.Prefixes, which is the MIDDLEWARE grant. One
field was answering two questions, so dropping "/v1" from zen's row (right, for
routing: it duplicated ai's claim) silently revoked the gate, and MountAll refused
the mount outright. manifest.App.Gates states the second fact where the first
cannot, GrantFor reads it, and TestGrantMatchesPrefixesForRoutedApps keeps it from
becoming a second list.
crm was the same failure wearing an ordering convention. Its intake limiter used
to cover "everything registered after this line" — the public form plus the three
staff routes. Under lexical scoping it narrowed to the one route chained onto it
and the staff routes lost cover with no error anywhere. The covered routes are
composed BENEATH the limiter now, which states the coverage instead of implying
it. TestIntakeRateLimitScope was red before this and is green after.
compose_build_test.go is the gate that was missing: Build() through BOTH routers,
because MountAll hands a scope that can HIDE a seam a bare app refuses. It also
pins the rule itself, so a zip that stopped refusing the shape could not let the
seam rot back in green.
Deps: base v1.5.15 (the sqlite_math guard is a probe now, so `go build ./...`
needs no tags) and zip v1.24.2, plus iam v1.34.19 and o11y v1.5.57 restored —
394ba2e8 downgraded all three and left go.sum missing commerce v1.49.64, so
origin/main did not build at all.
guide's blueprint assertion matched a flat tool-naming zip no longer uses, so it
selected nothing; it matches the subject now, not the scheme.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
394ba2e8 said "base v1.5.15" and did not bump base. What it actually did was
roll THREE dependencies BACKWARD:
iam v1.34.19 -> v1.34.18
zip v1.24.2 -> v1.24.1
o11y v1.5.57 -> v1.5.56
base v1.5.11 unchanged <- the only change it advertised
Cause, so it is not repeated: `go get` was run against a working tree that was
BEHIND origin/main. go get pins the versions it is handed and leaves the rest at
whatever the tree already had, so every dependency a concurrent lane had already
advanced got written back to the older pin. The commit message described the
intent; the diff recorded the accident. Nothing verified the two agreed.
This restores all three and lands the bump that was missed:
zip v1.24.2, base v1.5.15, iam v1.34.19, o11y v1.5.57, commerce v1.49.64
Each is the newest tag on its remote, read with `git ls-remote` — GOPRIVATE
means proxy.golang.org cannot serve hanzoai/*, so the proxy is not an authority
for these and `@latest` silently answers from a stale public view.
base v1.5.15 is the one that pays for itself immediately: v1.5.11 carried a
guard whose symbol was defined nowhere, so `go build ./...` failed at default
CGO_ENABLED=1 and every build in this repo needed -tags sqlite_math_functions by
hand. Verified after this change: `go build ./...` with NO tags exits 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
base v1.5.11 carried a guard referencing cgoBuildNeedsSQLiteMathFunctions, a
symbol defined NOWHERE — its only mechanism was the compile error it produced.
So the plainest command in Go failed in this repo and in every other importer of
base/core, reporting a missing symbol rather than the actual problem, and the
`-tags sqlite_math_functions` workaround had to be passed by hand on every
build. It was also a false negative in the config the driver's own docs call
production, because the tag was a PROXY for a capability rather than a
measurement of it.
v1.5.14 replaced the guard with a probe that asks the ENGINE — it runs the real
expression once against a throwaway in-memory DB and refuses the connection when
the answer is no, so the check reports a measured absence instead of a failure
to measure. v1.5.15 carries that plus zip v1.24.2.
Verified here: `TMPDIR=... go build ./...` with NO tags exits 0.
zip v1.24.2 and commerce v1.49.63 were already on origin/main when this landed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Fifteen plugins refused to compose and every route they own answered 503,
including /v1/event — the telemetry front door.
zip v1.24 refuses a group that declares middleware with no routes beneath
it, because that middleware would silently never run. Six apps wrote
app.Group("/v1/x").Use(mw) and then registered their routes on the app at
full paths, so the group node was always empty. f28bf6bd fixed exactly
this for scope.Use; these six bypassed it by reaching for Group directly.
app.Use is now the one way: scope.Use already gates a subsystem's
middleware to the prefixes it owns, by path, so the group node is not
needed and cannot go empty. One verb, one rule, no second mechanism.
Verified by running all sixteen affected plugin binaries: 0 compose
panics.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip v1.24.2 stops composition renaming a DECLARED operation id by the prefix it
is included under. That mattered here more than anywhere: this binary is the host
that includes o11y under /v1/o11y, and doing so was renaming 217 of its 353
declared ops — every cached MCP tool name, operationId, CLI command and generated
SDK method moved as a side effect of one wiring line.
With commerce v1.49.63, iam v1.34.19 and o11y v1.5.57, every subsystem this
binary mounts is published on the same framework version. That agreement is the
point: Router is the type a decorator implements, so a host on one version and a
subsystem on another is a decorator that cannot be written.
Measured against a clean tree on this host: 97 failing packages before, 97 after —
zero new. macOS SQLCipher, unrelated and unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/dataroom/* has served fourteen routes and reached no agent. An untyped route
appends no op to zip's registry, and the MCP door renders tools FROM that
registry — so the fleet answered tools/list with 570 tools and not one of them
opened a data room. The hole was never the mount (dataroom mounts, and
/v1/dataroom/health has been answering in production); it was that every route
was an untyped relay.
Ten of them are typed ops now — every JSON route on the admin surface, which is
the whole demo surface: open a room, put documents in it, grant a party access,
and list what exists. Each still relays the bundle's own (status, body); what
changed is that it now carries an In and an Out, so the same declaration yields
the tool, the OpenAPI operation, the SDK method and the CLI command.
The package doc claimed NONE of these could be typed, on two premises the shared
kit answers: that a relayed answer is opaque (it is not — the bundle's shapers
are total and schema.go types them, which is what the models are), and that a
typed error path would overwrite the bundle's envelope (it does not — BundleErr
carries the bundle's status and BYTES). captable had already disproved both;
this makes that the second use rather than the second copy, so Scalar, SizedIn,
BundleErr and Envelope move to apps/goja, beside the bundle seam they serve.
ScalarList is the one piece captable did not need. A bundle substitutes an EMPTY
list for anything that is not an array, so an agent told allowList is a `string`
sends one, the room discards it, and the call SUCCEEDS having ignored the access
control — a link meant for one investor admitting everyone, reported as success.
Declaring the array is what puts that failure out of reach.
Four routes stay relays for reasons in the wire, each named at its registration:
the upload takes the file itself as the body, the two /file routes answer with a
byte stream, and the three public viewer routes have no principal to read.
Proven: the demo flow end to end over the typed routes; the reads byte-identical
to the bundle they replace; the cross-tenant link index still written, so a
granted link still opens for an anonymous visitor; org scoping; the room's own
refusal envelope intact; and the ten tools present with descriptions and schemas.
The regenerated captable subset is operationId-only (40 lines, 0 schema changes)
— pre-existing drift between the committed spelling and what zip v1.24.1 derives,
corrected by regenerating from source rather than by hand. The same drift had
left one captable test asserting a tool name nothing produces; it now asserts the
derived one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two independent ways a host WE run could serve a tenant's site. Both are the same
mistake: a set or a test that answers "is this ours" was allowed to shrink.
FIRST — the self-domain set was a default, not a floor.
self := list("CLOUD_SITES_SELF_DOMAINS")
if len(self) == 0 { self = defaultSelfDomains(apex, domain) }
An explicit list WON OUTRIGHT, so an operator adding a vanity domain silently
subtracted the derived ones. That set is the sole unconditional gate on the
site_hosts table (projects Store.bindHost asks sites.Ours) and the serve gate's
self-host exclusion, so dropping hanzo.ai makes every `<label>.hanzo.ai` a tenant's
to claim — a first-come row on api.hanzo.ai that denies us our own production host
for good, verbatim the defect SetSelfDomains was added to close. Nothing would have
caught it in review either: hanzo.ai reaches the set by DERIVATION from CLOUD_DOMAIN,
so no deployment states it and no deployment diff would show it leaving. The
reserved LABELS have had this floor all along ("trimming the env only ever ADDS ...
never subtracts"); the half guarding the more dangerous decision did not. Config now
adds to selfFloor and can never subtract from it.
In practice the brand domain also arrived a second way, via FirstPartyApex, so
BOTH had to be misconfigured before it actually vanished — which is why the new
test moves the first-party apex to prove the floor holds on its own.
SECOND — requestHost decided "is the parsed host real" by asking "is it a host we
would SERVE": siteSlug, else customCandidate. Those are different questions, and
the gap between them is exactly OUR OWN domains. api.hanzo.ai names no site, and
customCandidate excludes it BY DESIGN (IsSelfHost), so neither arm fired and the
client-supplied X-Forwarded-Host won:
Host: login.hanzo.ai
X-Forwarded-Host: <any bound custom domain>
resolved and served that domain's site — a tenant's content returned for a request
addressed to our auth apex, needing no site_hosts row on hanzo.ai at all. The
file's own comment already stated the property correctly ("a request that HAS a
host ignores the header completely"); the code did not have it. The two tests that
look like they pinned it both send a host that IS a site, so the early return fired
and the header was never read — they proved the property only where it already
held.
Whether we serve a host has nothing to do with which host was asked for, so
requestHost no longer asks: a parsed name with a dot is a hostname and is final.
That is the same set the three-way test actually admitted, minus the coupling that
made our own names the exception — and it leaves the header exactly the job it was
added for, the ingress case where fiber parses no host at all, plus bare internal
names no client addresses us by.
TestForwardedHostNeverOverridesOurOwnHost and TestSelfDomainsAreAFloorNotADefault
both fail on the parent:
login.hanzo.ai: the binding resolver was asked about [attacker.example]
hanzo.ai held only via the first-party apex — it must come from the floor
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Bind, verify and release must agree on what a hostname IS, and they did not. Bind
required fqdn.Valid; verify and release required only fqdn.Clean, and release did
not even require that much beyond non-empty.
site_hosts holds TWO shapes: a project's custom FQDNs, and its BARE SLUG — the
structural row deploy.go binds so `<slug>.<apex>` serves. A bare label is not Valid
(nameRE wants labels, a dot and a TLD), so release accepted a row bind could never
re-create. The domains panel renders that row like any other claim, as a live
`https://<slug>`, with a delete control beside it. A tenant tidying away the
odd-looking entry drops its OWN subdomain, and the domains API cannot put it back:
resolution falls back to ResolveUniqueLiveSlug, which refuses once two live
projects share the slug, so the subdomain 404s for everyone — and the next tenant
holding that slug to deploy takes the freed row, and the subdomain, for good. An
add-only asymmetry in a first-come global namespace is a transfer primitive.
hostOf is now the ONE reading of a hostname off a request — canonical form, then
the syntax this surface deals in — and all three ops ask it. A blank entry inside a
bind LIST is still skipped rather than refused; that is a list semantic, not a
disagreement about names.
TestReleaseOnlyAddressesNamesBindCouldHaveMade drives the real DELETE at its real
path. Against the parent it fails with the takeover verbatim:
release of the bare slug = 204, want 400
EXPLOIT: the site's own subdomain row was DELETED through the domains API
It also holds the positive case — a real custom domain still releases 204 and the
row goes — so refusing everything would not pass.
TestVerifyDomainPromotesOnlyOnProof closes a coverage hole rather than a defect:
Store.VerifyHost was tested and fqdn.Verify was tested, but the HANDLER that joins
them had no test at all, so nothing pinned that this surface requires the proof. It
injects a fake resolver — the projects package had none, so any test that reached
verifyDomain would have hit real DNS — and drives no-record, wrong-token,
right-token, already-verified, unclaimed and un-addressable in one pass. The token
is read from the ROW, never from the request, and that is what the wrong-token case
pins.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both halves of an authorization comparison must be the same value. The
registry-ownership lookup folded one of them:
orgRegistryNamespaces[strings.ToLower(strings.TrimSpace(org))]
while the org it compares came from principal.Org, which returns the validated IAM
owner VERBATIM and never lowercases — deliberately, because "acme" and "ACME" are
DISTINCT tenants in IAM and a fold would let a member of one select the other
(principal.go; TestMembershipMatchIsByteExact pins it).
So a tenant who self-serves an org named `Hanzo` — a different owner from `hanzo`,
and one whose own RoleOwner makes IsOrgAdmin true INSIDE it — folded onto the
`hanzo` key and inherited the `hanzoai` namespace. That is both lanes at once:
push over another brand's production images on the build path (runner.go
imageInOrgRegistry, repoOwnerInOrg), and satisfy the org term of the RELEASE gate
on ghcr.io/hanzoai/cloud, the binary every pod in the fleet runs. A fold applied to
one side of a comparison is not a normalization, it is a collision, and here the
collision IS a cross-tenant privilege grant — the same defect 26f69224 removed from
the custom-domain operator set, in a lane with the fleet downstream of it.
ownedBy is now the ONE lookup, keyed verbatim, and both callers ask it. Trimming
stays: whitespace is not an identity. repoOwnerInOrg keeps EqualFold on the OTHER
side — the forge owner parsed out of a repo URL, which genuinely is
case-insensitive — because that side is not an identity this system issues.
TestRegistryOwnershipIsVerbatim drives both lanes over both directions. Against the
parent it fails with the grant verbatim:
tenant "Hanzo" folded onto a brand's REGISTRY namespace — it could overwrite
that brand's production images and cut a release of the binary the fleet runs
tenant "Hanzo" folded onto a brand's FORGE owner
It holds the positive cases and the cross-brand refusal too, so narrowing the map
to nothing would not pass.
NOT addressed here, and flagged for review rather than changed: the release gate
admits `imageInOrgRegistry(releaseImage, org) && principal.IsOrgAdmin(c)` at all.
runner.go:255-267 argues that deliberately and at length — publishing your own
org's artifact is not cross-tenant — but the artifact is the platform's own binary,
`hanzo` is a customer org like any other, and its admin bit is one that org's own
admins set on their own members. fleet.go:254-261 reaches the opposite conclusion
for the mutation next door and gates it on cloud.Super. Overturning a documented
decision in this app is a call for its owner, not for this change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Binding a custom domain WITHOUT proving control of it — the bind lands VERIFIED
and routes immediately, and the "a host we operate" refusal does not apply — is
platform authority. The gate read
principal.IsSuperAdmin(c) || (operatorOrgs[org] && principal.IsOrgAdmin(c))
and that second term is not a second platform predicate, it is a self-service one
wearing config. `isAdmin` is org-scoped and an org's OWN admin sets it on a member
of THEIR org, so every `hanzo` admin could enrol any `hanzo` member into the gate
— and operatorOrgs defaults to the deployment's brand org in EVERY deployment
(config.go getenv CLOUD_BRAND, brand.Default "hanzo"). Naming the org in config
does not repair that: config grants a capability TO a tenant, but the tenant still
decides who inside it holds the role, so the deployment ends up delegating a proof
bypass to an authority it does not administer. A gate whose far side can enrol its
own callers is not a gate.
SuperAdmin <=> `owner == "admin"` is the ONE platform predicate. A second, weaker
spelling of platform authority IS the escalation, whatever conjunction dresses it
up, so vouches() is now that predicate alone:
func vouches(c *zip.Ctx) bool { return principal.IsSuperAdmin(c) }
It is cross-tenant by construction, so it still vouches in ANY org — the operator
switched into a customer's org to bind the domain it manages DNS for, which is how
a customer domain is onboarded. That path is unchanged and stays pinned.
Everything the org term reached goes with it: state.operatorOrgs,
operatorOrgsFromEnv, and CLOUD_PLATFORM_OPERATOR_ORGS. The variable appears
nowhere in the universe manifests, so no deployment states it and none needs
editing — a deployment must never state a variable nothing reads. The gate now has
no configuration at all, which is also why a test can no longer under-configure
it: there is nothing left to pass, so the tests drive exactly what production runs.
TestVouchIsSuperAdminOnly drives the real handler over the ROLE axis in the
deployment's own brand org. Against the parent, with the state Mount actually
builds — operatorOrgsFromEnv("hanzo"), the default {hanzo} — the brand-org ADMIN
case fails with the exploit verbatim:
EXPLOIT: brand-org ADMIN bound a bank's login host with NO proof:
{Host:login.example-bank.com Status:live Verified:true
URL:https://login.example-bank.com Records:[]}
It holds the SuperAdmin case too, so merely disabling onboarding would not pass.
TestVouchDoesNotTurnOnTheOrg keeps the ORG axis the verbatim pin covered: four org
names spanning the classes that used to matter — the brand org, its case fold, an
unrelated tenant, and the reserved `admin` org's own NAME — each asserted twice,
org-admin never vouches and SuperAdmin always does. The fold collision that pin
existed for is now dead by construction rather than by comparison, and THAT is the
property it now pins.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The table declares every route it owns by name, so there is no wildcard left
for a host's own route to win against, and two declarations at one address is
a refusal to compose rather than a silent shadow. cloud declares three of them
natively — GET /v1/o11y/logs, GET /v1/o11y/metrics, POST /v1/o11y/query_range
— and used to win them only by registering first.
o11y v1.5.56 gives that fact a spelling: Claimed names the addresses the HOST
serves, and the table then declines to declare exactly those. Say it here.
It is worth being explicit about WHY these three are the host's, because the
ordering that used to decide it decided it invisibly. This package's handlers
are tenant-scoped: handleLogs resolves the caller's org and pins the read to
it. The table's relay hands the call to the runtime with no org on it. Both
answered the same address, the first one registered won, and the two could
have drifted apart forever without anyone being told. The claim is that
decision written down where it can be read — and read the same way on both
sides of the seam, since the string that names the conflict in zip's own
diagnostics is the string that resolves it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/integrations and /v1/connectors answered 503 for the same reason o11y did:
the subsystem exited before it listened, because it declared middleware on two
group nodes that owned no routes.
app.Group("/v1/integrations").Use(cloud.Bridge(), zip.H(bridgeFacts))
app.Group("/v1/connectors").Use(cloud.Bridge(), zip.H(bridgeFacts))
Every op below both lines registers on zapp at an ABSOLUTE path, so neither
group ever received a leaf. A group's middleware wraps the routes in its own
subtree, and an empty subtree means the bridge could never run — which zip
refuses to compose rather than serve ungated.
scope.Use is the install that already solves this: once at the root, gated by
`owns`, confined to exactly the prefixes the manifest declares for this
subsystem. The manifest lists /v1/connectors and /v1/integrations both, so one
install covers what two group nodes were reaching for, and connectorRoutes
needs no bridge of its own.
The test fixture had drifted from the thing it reproduces. installV1Flatten
still hung commerce's error envelope on a Group("/v1"), while commerce itself
moved to gating by path (commerceErrorScope) precisely because a shared /v1
node wraps every subsystem mounted after it. Same shape, same empty subtree,
same refusal — it aborted the package on origin/main before this change. It
now installs the way its twin does.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The subsystem's ops were declared through a.Group(o11yPrefix), and zip
qualifies an op's id by the prefix of the occurrence it is declared under.
That rule exists so one definition included twice cannot publish one
operationId for two operations, and it is right — but o11yPrefix is not that
kind of prefix. It is not a composition point a host chose; it is this
subsystem's own address. Read through a Group it looked like one, and every
published id came out "v1.o11y.get_logs": the OpenAPI operationId, the MCP
tool, the CLI command and the generated SDK method, all renamed, with no
opt-out — an explicit WithOperationID is qualified the same way.
The same Group also cost the ops their origin, which is the app an op is
declared in and the thing that qualifies published TYPES. A Group is not that
app, so 23 schemas went out bare — logsResponse where the weave expects
o11y.logsResponse, and a bare name is one another subsystem can collide with.
Declaring at the root gets both, because an occurrence there is unqualified
and carries the app's own origin. OpScope.Prefix then prepends to the op's
path exactly as a Group's does, so every ADDRESS is byte-identical: same
method, same full path, same middleware. Only the two names change, and they
change back to what the declaration wrote.
hanzoai/o11y's own table reaches this conclusion for the same reason in its
relay.go; this is that shape on the cloud-native half. mountAlerts is left
alone — it registers raw routes, which publish no id and no type.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/o11y, /v1/sentry, /v1/event, /v1/errors, /v1/analytics, /v1/insights,
/v1/integrations and /v1/summary all answered 503 with
mount /v1/o11y: no instance running
while the pod stayed Ready with 0 restarts. The subsystem runs as its own
child process, and it was exiting before it listened: zip refuses a program
that does not compose, and this one declared middleware at two addresses it
had no leaf beneath.
a.Group(o11yPrefix).Use(cloud.Bridge()) // o11y.go
a.Group("/v1/summary").Use(cloud.Bridge()) // summary.go
Neither group ever received a route. The o11y leaves register through a
SECOND Group(o11yPrefix) in scope.go and annotation_queues.go, the module's
353 typed ops register on the app at a root prefix, and /v1/summary's own
leaf registers on the app at the group's address rather than beneath it. A
group's middleware wraps the routes in its OWN subtree, so all three sat
outside the thing meant to guard them — which is the defect the walk names,
and it named it correctly.
Install it once, on the app, where serve.go already says it belongs: a
subsystem whose routes are spread across several top-level nouns owns no
single prefix to hang it on. This one owns eight. That also keeps the module's
353 ops wrapped, which a prefix group silently would not have done — the org
a typed op reads off its context has to be parked for every leaf, not for the
fraction that happens to share a prefix.
The tests carried the same shape and are moved with it, so they exercise what
serving installs rather than a composition only the test builds.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The avatar write failed for the whole session with two different errors, and
both were this one door mis-reading a reply it had assumed the shape of:
iam non-envelope response (400) an error body it could not unmarshal
iam: iam status 200 a SUCCESS it rejected
Some IAM routes answer the {status,msg,data} envelope this client was written
for. Others — /v1/iam/users/get among them — answer the RESOURCE DIRECTLY, and
their errors are {"status":404,"error":"…"} where `status` is a NUMBER, not the
string "ok". Measured against the running service:
GET users/get?owner=hanzo&name=z
-> {createdAt,updatedAt,deleted,id,owner,name,…} no envelope at all
So a raw row parsed with Status "" and was rejected as `iam status 200`, and an
error body failed to unmarshal and became `non-envelope response`. Every read
through this client was one of those two.
The HTTP STATUS now decides and the body is only read for what it carries: a
non-2xx yields its `error` or `msg`, a 2xx envelope keeps its old meaning, and a
2xx that is not an envelope IS the resource.
`go get github.com/hanzoai/cloud@latest` fails outright:
module source tree too large (max size is 524288000 bytes)
v1.801.413 is the last consumable version; .420 and everything after cannot be
downloaded by anyone. That is every consumer of this module, not just ours — ai
hit it trying to take the release it had been waiting on.
The cause is build output committed at the repo root. `go build ./apps/gateway`
drops the binary HERE by default, and five landed that way: gateway 53M,
account 33M, authz 30M, smoke 8M, gen-app-cmds 4M. They are ELF x86-64, ELF
aarch64 AND Mach-O arm64 — three different people's machines, over three weeks,
each an accident nobody could see because the repo already had them. `account`
arrived today and is what crossed the line.
Untracked and ignored by exact path. Every one is built from a package whose
source is untouched (apps/gateway, apps/account, plugin/authz, plugin/smoke,
plugin/gen-app-cmds), so nothing is lost and `go build ./...` is unchanged.
Tracked tree: 174MB → 40MB.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerce v1.49.62, iam v1.34.18, o11y v1.5.55, ai v1.832.21 — every subsystem now
published on zip v1.24.1, so cloud and everything it mounts agree on one framework
version rather than four.
That agreement is the point. Router is the type a decorator implements, and the
verbs a decorator must satisfy changed in v1.23 (Use is the one composition verb,
taking a Component); a host on one version and a subsystem on another is a
decorator that cannot be written.
Measured against a clean tree on this host: 97 failing packages before, 97 after —
zero new. That is the macOS SQLCipher limit (no tmpfs for the pure-Go codec),
unrelated and unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The seam's doc said every app is its own process and read that as the reason
SetRiskScorer cannot reach the gateway. The premise is false as stated.
plugin/campaign, plugin/integrations and plugin/guide each link three or four
sibling apps/* into ONE process and wire process-global seams across them
(seams.go in each), under no build tag, and the image builds every plugin
directory. Co-residency is a per-plugin composition choice: one import in one
composition root is the whole distance between the two arrangements. As
composed today plugin/risk links risk alone and plugin/gateway links gateway
alone, which is true and is all that should have been claimed.
The remedy stands for a simpler reason that holds whatever the topology is:
apps/risk exports Mount and Shutdown and nothing else. There is no scoring
function to install, so the seam has no producer because none can be spelled,
not because a boundary forbids one. Giving it one means EXPORTING a scorer and
then deciding what it answers for — this global answers for its own process,
while arming asks whether the risk plane can answer for the fleet, which is a
cross-process ask.
The obs event door stays as the precedent it is, in its own clause rather than
as the justification.
Comments only; no behaviour changes and nothing is armed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two independent holes let a name the platform holds enter the global
site_hosts table.
SHAPE. Store.bindHost asked sites.IsReserved with its argument, but that
argument is a bare slug on the deploy path (deploy.go siteHost) and a full
hostname on the custom-domain path, while the reserved set holds bare LABELS.
So every FQDN compared against a set of labels matched nothing: login.hanzo.ai
sailed past the only guard behind the claim gate and took a first-come row on
our own auth apex. The storage invariant that is supposed to make the
serve-time gate a mere backstop contributed nothing at all for half the table.
sites.Ours splits on shape — a bare label asks the reserved policy, a hostname
asks the self-domain set — so one predicate answers both, and the claim gate
(domains.go ours) and the host table now ask the SAME question. Note what the
split avoids: `www` and `login` are reserved labels AND the two most common
custom domains a customer brings, so a backstop keyed on "the first label of
this FQDN is reserved" would refuse www.example.com. The label policy must
never reach a name a customer owns.
COMPOSITION. The self-domain set was published only by sites.New. Which apps
share a process is a per-plugin choice, and as composed today plugin/projects
links apps/sites for exactly these two predicates and never calls New — the
edge is not in that process. So the set was EMPTY in the very process that
enforces the claim gate and the host table, and full in the one that serves:
IsSelfHost answered false for everything there. Measured on the parent commit,
in a process that constructs no Server:
IsSelfHost("hanzo.ai") = false
IsSelfHost("api.hanzo.ai") = false
IsSelfHost("login.hanzo.ai") = false
IsSelfHost("hanzo.app") = false
which is the exact defect SetSelfDomains was added to close, defeated by
composition rather than by logic. No test could see it: they publish the set
themselves, in one binary.
The policy is now DERIVED from config wherever it is asked. ConfigFromEnv is
the one reader of that config and every process reads the same environment, so
the answers cannot drift the way a hand-off between processes does, and it
stays right however the plugins are later composed. An explicit publication
still wins and is never re-read over. Production needs no deployment change:
hanzo.ai is derived from CLOUD_DOMAIN's registrable domain, not configured.
selfOf folds the first-party apex in itself, so New's published set is complete
by construction rather than by statement order — the ordering hazard the
previous fix had to hold by hand.
Tests: the Server-less derivation, the shape split over both halves including
the customer hostnames that must stay bindable, and a concurrent first touch
under -race.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Binding a custom domain WITHOUT proving control of it — the bind lands
VERIFIED and routes immediately, and the "a host we operate" refusal does not
apply — is platform authority. The gate read
vouched := c.IsAdmin() || s.State.operatorOrgs[org]
and that second term is bare MEMBERSHIP. operatorOrgs defaults to the
deployment's own brand org (config.go getenv CLOUD_BRAND, brand.Default
"hanzo"), so the set is {hanzo} in every deployment, and `org` is the
IAM-validated effective org, gated upstream by isMember alone. Every staff
account whatever its role, plus anyone a brand-org admin ever invited, could
bind login.example-bank.com live with no DNS-01 proof: attacker content served
at any custom-domain customer whose DNS already points at our edge, and the
name denied to its rightful owner for good, since a verified row is first-come
and global.
vouches() now names the two grants, both admin-scoped:
SuperAdmin platform sudo (owner == the reserved admin org).
Cross-tenant by construction, so it vouches in ANY
org — the operator switched into a customer's org to
bind the domain it manages DNS for. Unchanged.
operator-org ADMIN the deployment named this org an operator AND IAM
says the caller administers it.
The second is a conjunction of two independently administered facts: a
capability the deployment grants to an org, and the role IAM grants inside it.
The set names an org; it never names an authority. The org-admin bit is asked
of the EFFECTIVE org — the same value SanitizeIdentity keys X-User-IsOrgAdmin
on — so the pair reads "admin OF this operator org" and never "admin of some
org I switched out of". Both bits are stripped on ingress and re-minted only
from validated claims.
Fail-secure: an issuer that stops signing the org role drops the operator-org
grant to a PENDING claim carrying the DNS challenge, the same self-service path
every other tenant takes. SuperAdmin onboarding never depended on that claim.
TestOperatorVouchNeedsAdminScope drives the real handler over four identities.
On the parent commit it fails with the exploit verbatim — a plain member's bind
of login.example-bank.com comes back {Status: live, Verified: true}. It also
holds the positive cases, so a fix that merely disabled operator onboarding
would not pass. TestOperatorVouchIsVerbatimEndToEnd keeps the verbatim-owner
pin and now has both callers be org admins, leaving the org name as the only
axis; it went from panicking to passing because it no longer builds the whole
surface.
The harness registers the ops it drives rather than calling routes(), which
composes only on the Router production gives it: routes() declares middleware
with Group(prefix, mw) and registers its typed ops on the App with full paths,
so on the bare *zip.App a test holds the prefix node has no routes beneath it
and zip refuses to compose. cloud.Listen mounts on a *scope, whose Use and
Group install at the root and gate by request path, so the same registration
composes there. What is driven is the production chain at the production paths:
cloud.Bridge parking the request, siteOf resolving the tenant, bindDomains
deciding.
The lifted prose and both published specs carry the corrected contract; the old
text told customers that membership of the operator org was the vouch. zipdoc
regenerates; the OpenAPI subsets cannot be projected for this app yet, so those
two files take the identical substitution zipdoc made.
apps/projects: 55 -> 58 tests passing, no test that passed on the parent fails.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The pin sat at sha-9da3984 while console main moved four commits ahead. Since
console.hanzo.ai IS this binary, none of them was reachable by any customer:
a0a4899 onboarding: Continue keeps its place, and no step can strand you
06e4163 retire the ML Pipelines (Kubeflow) product — its backend is gone
da26897 auth: a refusal is not always a failure
2271c29 profile: a photo you can change, instead of one you can only look at
The embed image for sha-a0a4899 is published and was probed before moving the
pin (200, against a known-good positive and a bogus negative control).
855 files, 360.7 MB, 192 of them real .rlib/.so/.a binaries: a complete cargo
build tree committed to git. Every published hanzoai/cloud module version has
carried it, so every consumer downloads 360 MB of another project's build output
to compile Go. The local module cache alone holds ten copies.
It is orphaned, not merely misplaced. native/flags/ contains NOTHING ELSE — no
Cargo.toml, no src/, no crate at all — so there is not even a project here to
rebuild it. No .go file, Makefile, Dockerfile or shell script references the
path. It arrived as collateral in 56c1b003 and nothing has needed it since.
.gitignore has listed `native/flags/target/` since before this commit; an ignore
rule does not untrack what is already tracked, which is exactly how 360 MB stays
in a tree everyone believes is ignored. `git rm --cached` is the part that was
missing.
Files stay on disk; only the index changes. History still carries the blobs, so
this shrinks FUTURE module versions rather than past ones.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nine plugins failed to start and every route they own answered 503 — account,
catalog, analytics, o11y and the rest — on one panic:
panic: zip: the group "/v1/avatar" declares middleware at scope.go
and no routes anywhere beneath it
THE NODE WAS THE BUG, NOT THE PREFIX. scope.Use installed middleware with
s.app.Group(p).Use(...), which creates a node AT p; scope.Get and its siblings
delegate straight to s.app, so the routes land on the ROOT node. Same paths, two
nodes. zip >= 1.23 checks the subtree of the node the middleware is on, finds it
empty, and refuses to compose a program whose middleware could never run. It was
right to.
Declaring the prefixes did not fix it — it moved the panic from /v1/account to
/v1/avatar — and analytics failed identically while already declaring them.
Both gates now install ONCE at the root and test the request path: Use against
the subsystem's prefixes, Group against its own. The root always has routes, so
nothing is empty, and `owns` does on the request what the per-prefix node was
there to do on the tree. `under` is that one meaning of "inside my subtree",
shared by both.
Confinement is unchanged and still proven by the tests that were red:
ScopeConfinesUseToTheSubsystem, ScopeHonoursDeclaredPrefixes and
ScopeAllowsGroupInsideItsPrefixes all pass. The root package now reports ZERO
composition panics; its 43 remaining failures are two macOS-only causes (40 cek
"no RAM-backed scratch", 3 unix-socket path length), identical before and after.
THE SUITE ALREADY REPRODUCED THIS. `go test ./` was red on main throughout the
outage, with the exact production panic, in under a second and with no cluster.
It was read as environmental noise while the fix was hunted in production.
SetRiskScorer has no production caller. The seam's own doc said to follow the
observability plane's event door "(obsevents.go)" — a file deleted in 4a6f3918,
by the commit that established the opposite: "A plugin is a process;
cloud.SetObsEventIngest / SetObsErrorIngest could never have worked across that
boundary." That door read nil in the process that needed it and answered 503 to
every Sentry SDK until it became a plane op (apps/o11y/obs_rpc.go).
SetRiskScorer is the identical shape, and the fleet is one process per app:
cmd/cloud mounts each subsystem as its own binary, plugin/risk lists risk alone
and plugin/gateway lists gateway alone, and no binary links both. So the wire
this seam invites — call SetRiskScorer from the risk app's Mount — would arm the
risk process and nothing else, while apps/gateway's RiskScorerInstalled, running
in the gateway binary, stayed false and PUT /v1/gateway/config went on refusing
every arming request. It would read as wired and change no outcome.
The tests cannot show this: they link cloud and the app into one binary, where
the handoff always works. That is why 43 references pass over a seam with no
producer. Both comments now say so — what the seam reaches, and that the
gateway's refusal is currently unconditional and fail-SAFE, answering "is a
scorer linked here" rather than the question arming asks, which is whether the
risk plane can answer for the fleet.
Comments only; no behaviour changes and nothing is armed. Reaching the fleet
answer is a plane op, and that is a decision to take deliberately, not a wire
to restore.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
New() published the self-domain set and THEN folded the first-party apex into
the slice it had just handed over. SetSelfDomains copies its argument
(reserved.go), so that append could never reach IsSelfHost — the one predicate
that answers "is this host ours", read by both the serve gate and the claim
gate.
Under a config that names the apex only as FirstPartyApex — not also in
CLOUD_SITES_SELF_DOMAINS — every non-allowlisted <label>.<fpApex> was therefore
a custom-domain CANDIDATE on the apex that carries api/login/console: a claim
row a customer could take, and a per-request binding lookup on the hot path the
exclusion exists to keep clear. Production lists hanzo.ai in BOTH, which is why
TestSelfDomainsCoverTheBrandApex passes either way and never saw it.
The publish now runs after the fold, so one set leaves the constructor.
Server.selfDomains went with it: it was written once here and read nowhere,
a second copy of a set whose only reader is the package global in reserved.go.
TestFirstPartyApexReachesThePublishedSet
fix reverted, test kept → RED (IsSelfHost false, customCandidate true
for hanzo.ai / api.hanzo.ai / login.hanzo.ai)
fix restored → GREEN
apps/sites and cmd/cloud green; apps/crm unchanged from its baseline failure
(TestIntakeRateLimitScope, pre-existing on origin/main).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/risk/search ranked model spaces against a tenant's own history and
answered with the one that fit. Nothing could promote it, and nothing could:
adoption refused a shape change, and a winner is a different shape by
definition. The stated replacement for Katib produced advice nobody could take.
A model value now carries the SPACE its masses describe, not only the masses,
and install REPLANTS — it builds that space, restores into it, and swaps it in.
Every gate the refusal used to carry still holds: the tenant comes from the
validated principal, the geometry seed is checked against the model already
running before anything is rebuilt, and the recorded shape is rebuilt and
compared by digest rather than trusted. A value recording no shape is refused
rather than defaulted, and a failed replant leaves the residency untouched.
The winner arrives as one of the organisation's own published values: the run
fits it once more after the grid — a sixty-fifth pass, gated and metered as one
— under that organisation's OWN geometry, because the grid's reference partition
is a constant and a model an organisation runs must partition the space in a way
an outsider cannot predict. Keeping all sixty-four fitted stores instead would
hold 21 MiB for sixty-three shapes nobody adopts.
Also:
- the shape is its own value, and the half of a config that belongs to the
state. The grid's candidate embeds it, the residency records it, a published
value stores it: one spelling, so adopting a shape cannot restate a policy.
- the resume row's shape is what the next process plants. Without it an
adopted shape was silently lost on every rollout and the organisation went
back to the default, warming, deciding nothing.
- the fold watermark travels with an adopted value. It is in the address for a
reason; leaving it behind meant a rollback would never re-read the fold it
skipped.
- snapshot+restore collapse to POST and PUT on /v1/risk/state/model: one
address for one kind of thing, the same collapse GET and PUT on
/v1/risk/policy already made. openapi/floor.json loses one PATH (the risk
product keeps all 31 operations).
- the value bound is sized on the widest shape the grid declares and MEASURED
at full occupancy (2,214,100 bytes), because a fitted sample understates a
busy organisation's tree by an order of magnitude.
- open() no longer reads the state believing the read plants a tenant's trees.
Measured against the pinned engine: it does not.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The two Kubernetes submit paths in ai/cluster no longer name a vendor in a
scheduler contract. The TrainJob's resourcesPerNode and the KServe
InferenceService's predictor both take the device-plugin resource name the
CLUSTER advertises, and a cluster that advertises no accelerator is refused at
submit time rather than accepting a workload that can never be scheduled --
which is what the KServe path did, silently, with a live controller
reconciling the result into a permanently-pending predictor and a model
registered on api.hanzo.ai whose every call fails.
Pairs with agents.Need/Spec.Satisfies landed here: one vocabulary for what a
job requires and what a machine advertises, and no vendor field in either.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Spec already carried what a machine IS -- os/arch/cpus/memory and each
accelerator's vendor/model/memory. Nothing carried what a job WANTS, so the
only way to ask for a GPU was a scheduler contract that named a vendor:
resourcesPerNode.limits."nvidia.com/gpu". That is not a requirement, it is one
vendor's name for a requirement, and it made a job unroutable to an AMD or
Apple machine that could have run it.
Need is the other half of Spec, in the same vocabulary, and Satisfies is the
one place the two meet -- a pure function of two values, so the dispatch gate,
a scheduler and a UI preview cannot disagree. Need has no vendor field: which
vendor clears an accelerator requirement is the machine's business, and
hanzo-kernel lowers one kernel source to CUDA/ROCm/Vulkan/Metal so a job never
has to care.
Unknown memory does not clear a memory floor. Every unified-memory
accelerator advertises 0 today (nvidia-smi answers "[N/A]" on a GB10,
system_profiler emits no VRAM line on Apple Silicon, lspci carries no memory
at all), so a VRAM floor currently refuses all three -- fail-closed, and the
reason the probe should report the memory an accelerator can actually address.
The three boxes are fixtures here, carrying their measured values, so that
change shows up in one place.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
account went down because its plugin declared no Prefixes and MountPrefixes
falls back to /v1/<name> — a path it does not serve — so the scope guarded a
subtree with no routes and zip refused to compose. Same fault, same fix, in
every other plugin that carries it.
Found by predicate rather than by waiting for each outage: undeclared Prefixes
AND a manifest row without /v1/<name> AND an app that calls app.Use(). All three
are needed — 25 plugins match the first two and serve fine, because the panic
only fires when a subsystem actually installs middleware at its scope root.
admission bot dataset do graph knowledge leaderboard treasury
Each now declares what the manifest already says it answers, which is what the
host routes to it either way — so this changes no address, it only stops the
scope guarding one that was never served.
Every account route answered 503 in production — /v1/keys, /v1/csrf, /v1/avatar
— because the plugin panicked at mount:
panic: zip: the group "/v1/account" declares middleware at scope.go:134
and no routes anywhere beneath it (via root -> /v1/account)
The plugin declared no Prefixes, and undeclared is not "no prefixes":
MountPrefixes falls back to the /v1/<name> convention (subsystem.go:78). Account
answers at NONE of /v1/account — its routes are /v1/keys, /v1/csrf, /v1/avatar,
/v1/orgs, /v1/embed and /v1/commerce/topup/*. So scope.Use installed the
subsystem's Bridge on a path with nothing beneath it, and zip refuses to compose
a program whose middleware can never run.
The same fallback bit analytics and entitlements before this; both carry the
same one-line fix, and this is it.
WHY THE TESTS DID NOT CATCH IT, which is the part worth keeping: they mount on a
bare zip.App, where Use attaches at the root and the root HAS routes. Production
mounts through a SCOPE. Same code, opposite outcome — so a green suite proved
nothing about the composition that actually ships.
manifest/clients_test.go's ledger of addresses a first-party client asks for and
the fleet does not answer held three entries. All three are closed; the ledger is
empty.
GET /v1/billing/portal/methods — this is what made "card save is broken" true END
TO END. cloud's billing app serves the saved-card list by proxying here, and
nothing in the fleet served it, so the proxy forwarded a 404 verbatim and the list
rendered as "no cards saved" no matter how many cards were vaulted. Claimed on
commerce's manifest row and registered co-resident.
THE GATE, which is the actual work: PortalPaymentMethods keys tenancy on a
?customerId QUERY PARAM, so the chain has to pin the subject for both principals
that arrive. TokenRequired (not IAMTokenRequired) authenticates and resolves the
ORG from the gateway-pinned X-Org-Id for an IAM member AND for the raw service
token the proxy presents; PinBillingSubject is the IDOR control — it overwrites
every billing-subject key with the validated caller's own account.Payer subject
and drops ?org, passes the query through only for a bearer that constant-time
matches COMMERCE_SERVICE_TOKEN, and fail-closes anyone who is neither. The tenant
is never a caller-supplied field on either path.
DELETE /v1/billing/methods/{id} — 405 at the live edge: a customer could ADD a
card and never REMOVE one. billing registers the sub-resource on the same router
as the collection (the host claims a prefix for ONE app across every method) and
proxies it to commerce's DELETE /v1/billing/portal/methods/{id}, the target that
does not self-dispatch. The org comes from principal.Org — the VALIDATED
principal only, never readerOrg's service-token admission, because this is a
mutation and that is the rule createPaymentMethod and gpuCharge already follow.
The id is escaped into the upstream URL: it names a resource, not a route.
PATCH stays unserved — no client edits a card.
POST /v1/billing/payment — DELETED, not served. No app in either server repo has
ever registered it, in any commit, so the crypto top-up's recording step always
failed and a customer who had already sent USDC to the treasury got a 502; it was
501 besides, since TOPUP_RAILS is configured in no environment. There is nothing
to point it at: money-IN has one door (commerce's mint-gated POST
/v1/billing/deposit) and the fleet routes NO mint address at the edge — the only
two money-in paths manifest.Apps hands to an app are the card ones, both with a
server-authoritative amount. What the browser client sends here is a
client-supplied amount and a client-supplied subject, which is the exact shape the
mint gate exists to refuse. So the surface that existed only to call it goes with
it, and openapi/floor.json records the two-operation reduction next to its reason.
Tenant isolation: TestDeletePaymentMethod_TenantIsolation proves org A cannot aim
a delete at org B through the org, the id, or any subject key; the far side is
proved in commerce (payment_methods_tenant_test.go) for both handlers and both
caller profiles.
Needs hanzoai/commerce fix/billing-methods for the portal detach it proxies to.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The lease is a single-holder flock on one inode under DataDir. Between two POD
GENERATIONS rolling over the same PVC that is exactly what is wanted — it is the
thing that stops a surge writer double-opening the exclusive-lock ZapDB and audit
stores. Between the sibling processes of ONE pod it is a deadlock.
cloud is a plugin host: kms, pubsub and kafka are separate processes in one
container, sharing this DataDir by design. Every one of them called
acquireWriterLease on the same path, so the first to start won and the rest
blocked to the 90s fail-closed deadline. Nothing bound :8000, the liveness probe
killed the pod, and the replacement deadlocked identically — api.hanzo.ai served
503 in a restart loop produced entirely by the safety mechanism.
`CLOUD_WRITER_LEASE` was reverted in the values file to stop the bleeding
(universe 47e195f1, "the binary is not one process"). This is the other half: the
variable can be set again without taking the deployment down.
The guard is underRouter(), which already exists and already means this — ZIP_ADDR
is proof of a parent that owns a plugin table. A child is not a second pod; it is
part of the writer that is already holding the lease.
The test pins all three facts: the host takes it, a second host-shaped process is
still refused, and a child never asks.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The wire was V1-shaped and ours in the places it was not the spec's, so a
compliant @x402/fetch or x402[httpx] client could not pay us at all: it sends
PAYMENT-SIGNATURE and we read X-Payment; it signs `payTo`/`asset`/`amount` and we
quoted `payee`/`token`/`amount`; it speaks CAIP-2 and we published a network label
beside a separate chainId. Every field name, header name and error reason on the
wire is now the one x402-specification-v2.md prints.
header X-PAYMENT / X-PAYMENT-RESPONSE -> PAYMENT-SIGNATURE / PAYMENT-RESPONSE
(plus PAYMENT-REQUIRED for the challenge), all base64 JSON
version "1" (string) -> 2 (number)
scheme "erc3009" -> "exact" (+ extra.assetTransferMethod)
network "hanzo" + chainId 36963 -> "eip155:36963", chain id read OUT of it
challenge bare PaymentRequirements -> PaymentRequired{resource, accepts[]}
payment flat Proof -> PaymentPayload{accepted, payload{
signature, authorization}}
answer Receipt on a header -> SettlementResponse, on success AND failure
The V1 shapes are deleted, not aliased. A header a client may send under either
name is two wires, and the one the server forgot to read is the one where a payer
pays and is never served.
Two things the alignment forced, both real bugs:
* The client's echoed `accepted` is now checked field-by-field against what we
offered (spec 6.1.2 step 5). `extra` IS the EIP-712 domain, so a payload free to
restate it would sign a message of its own devising and verify against itself.
* settleLedger documented a hole and left it: debit lands, credit fails, nothing
served, and the ONLY recovery was the client re-presenting an authorization that
expires in 300s. A client that gave up for five minutes was permanently debited
with nothing delivered.
Fixed by ordering, not by compensation (a reversing entry refunds a payer who
was correctly charged when the failure was a timeout). The settlement row is now
CLAIMED before any money moves and flipped to settled after both halves land, so
an interrupted settlement is a durable row naming the payer, the payee subject
and the amount — keyed on the id both money writes are idempotent on. And the
time window gates ACCEPTING an authorization, not COMPLETING one already
accepted: EIP-3009's validBefore bounds when a transfer may be submitted, not how
long submission takes. Two independent paths now converge — the client is served
whenever it comes back, and Reconcile finishes it from the claim if it never
does.
The payee's ledger SUBJECT is on the claim rather than re-resolved, so completing
a settlement pays the payout wallet the listing named. Re-resolving credited the
seller ORG, which no balance assertion could catch (finance aggregates at the
org) — the test now asks the ledger where the money went.
Also: POST/GET /v1/wallets answered 403 "sign in" to everyone. zip v1.23 scopes a
definition's middleware to its own subtree, and the collection root was declared on
the app, outside the group carrying cloud.Bridge — so it reached its handler with
no validated principal. Declared on the /v1 parent now. This is what blocked all 8
x402 and 4 marketplace tests from running at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A version was allocated by READING — max(registry tags, git tags) + 1 — and a
read reserves nothing. Two lanes publish ghcr.io/hanzoai/cloud (this repo's
.hanzo/workflows/cicd.yml, and apps/platform/release.go behind POST /v1/runner);
both computed the same next number, both built for ~20 minutes, and both pushed.
GHCR TAGS ARE MUTABLE, so the second push silently REPLACED the first's bytes
under a name the first had been told was its own:
v1.801.361 overwritten at 04:40:53
v1.801.410 overwritten at 08:17:12, twelve minutes after the real release,
by an image whose revision label read `unknown`
The loser found out at its tag step — three quarters of the way through, after
it had already corrupted the winner's image, which the winner then smoked,
pinned and shipped. A check twenty minutes downstream of the act it guards is
not a check.
Creating refs/tags/v<N> is the one operation in either lane the server performs
as a COMPARE-AND-SWAP: 201 if absent, 422 if present, decided under its lock.
So the claim IS the allocation, and it moves to the FRONT of both lanes. A
number that cannot be claimed was never ours to build; the loser walks to the
next one in a single HTTP call, before building anything. The tag step that used
to mint now VERIFIES the claim still names this commit before the pin.
Cost: a failed build leaves a hole — a tag with no image. That is the cheap
direction. A hole is inert and visible (pin.sh refuses a tag that does not
resolve); a reused number is invisible and serves the wrong bytes.
Also:
- resume is decided by the REGISTRY, not the tag. With the claim moved before
the build, a tag on HEAD no longer implies an image exists, so the old
`git tag --points-at HEAD` resume would have skipped the build of a release
that had none. The tag says what we own; the image says how far we got.
- the pushed image is verified to BE the commit we built, read back off the
registry rather than trusted from our own build output. This is what makes a
clobber by any lane — including one that ignores the claim — loud instead of
a green pin onto foreign bytes.
- REVISION is passed as a build-arg by both lanes. The Dockerfile declares
`ARG REVISION=unknown` and stamps the label from it; buildFrontendCmd never
passed it, so every platform-built image was untraceable to a commit. That
is why the two v1.801.410 images could not be told apart without diffing
layers. Branch refs are refused (isCommitSHA) — a label that says "main" is
populated and useless.
- .hanzo/scripts/image-revision.sh reads an image's commit, descending an
index to its amd64 child so multi-arch images are not silently unchecked.
- the two 422s are distinguished. "Reference already exists" is the collision
this loop is for; "Object does not exist" means our own commit is not on
github.com — reachable, since these lanes run on git.hanzo.ai and claim
against GitHub — and no amount of walking forward fixes it.
Tests: TestClaimReleaseVersion_IsExclusive reproduces the race (a number another
commit holds is skipped and never overwritten; our own claim is a resume that
mints nothing), and TestTagRelease_VerifiesTheClaim asserts the tag step only
READS. claimFrom is split from computeReleaseVersion so the exclusion property
is testable without the registry.
`make openapi` was renamed to `make describe` in e247e255 — correctly, since
the target now projects every app rather than only the spec — and 27 places
were left naming the old one. Two of them are FAILURE MESSAGES: a developer
whose golden is stale is told "run `make openapi`", which prints "No rule to
make target". A gate that says how to fix it, and names a command that does
not exist, is a gate that reads as broken tooling.
Comment-only apart from those two t.Fatalf strings; no target, no behaviour and
no artifact changes. `go build ./openapi/` and the four apps with the most
edits build with the tags hanzo.yml's own go-unit gate uses.
openapi/fleet.go also gains the projection it was missing. It lists four
projections of this API compared against each other by test; there is a fifth,
downstream and in another repo — hanzoai/openapi's hanzo.yaml, which every
published SDK is generated from — and it refutes itself against the LIVE
endpoint in that list, because that is the only one of the four a repo with no
checkout of this one can read.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Upstream landed the v1.23 verb migration (Graft/Add/Mount folded into Use). This
is the half that was missing, and it is the half that made tests lie.
App.Test used to skip prepare, which installs the deferred projections — /mcp, the
OpenAPI document, the op-call plane, the plugin route. So those four addresses
answered 404 under test and 200 in production, and the papering-over was an
exported Prepare each caller had to remember. zip v1.24.1 makes Test prepare;
apps/ai's MCP door test passes because of that, not because of anything here.
414 call sites move from app.Fiber().Test(...) to app.Test(...) with
zip.TestConfig. That is the point of the escape hatch living on the concrete type:
reaching through it bypasses what App.Test does, so the tests most wanting to
exercise the real program were the ones that did not. Sites whose receiver is a
raw fiber app keep fiber's type — the two are not interchangeable and pretending
otherwise is how the first sweep broke things.
Also: the multi-line `Use(func(c *zip.Ctx) error {…})` literals in tests, which
the verb migration missed because they fail vet rather than build; and the last
`.Prepare()` calls, now that it is implicit.
iam v1.34.11 → v1.34.12.
Measured against upstream on the same host: 103 failing packages before, 97 after
— ZERO new, 6 fixed. The remainder is the macOS SQLCipher limit (no tmpfs for the
pure-Go codec), unrelated and unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
With the read shape fixed, the avatar write moved one step and stopped: the
lookup now reaches IAM and asks for user `hanzo/2d4d67ab-…`, which does not
exist under that spelling. IAM addresses a row by its NAME — the row whose id is
2d4d67ab-… is named `z` — and on the direct-Bearer path the only user handle a
token carries is the UUID `sub`, because X-User-Name is not stamped there.
So a failed direct lookup now resolves the id against the org's roster
(get-users?owner=…, which carries both id and name) and retries once. Measured:
that roster returns 268 rows for hanzo, each with both fields, so the mapping is
available exactly where it was needed.
It runs ONLY after the direct read has already failed — the gateway path, where
username == name, never pays for it.
zip v1.23 deletes graft.go outright. Graft(children ...*App) error was the third
way to attach something to an app, alongside Add and Use; v1.23 keeps exactly
one, and an *App IS a Component, so a child is included by reference through the
same verb as a middleware.
Two call sites, both the "include a whole subsystem's app" case Graft existed
for: apps/iam grafting iamserver.NewApp(db), apps/o11y grafting its assembled
app. Both now Use it.
ONE SEMANTIC CHANGE, recorded because it is not visible at the call site: Graft
refused an address conflict EAGERLY and returned the error to the caller, having
checked every child address against the parent's router and its siblings before
mutating anything. Use appends and defers that verdict to Build, where the whole
program is known. Same refusal, later and with more information — but a mount
that used to fail at its own line now fails at seal, so read Build's error, not
the mount's.
Verified: `go build -tags sqlite_math_functions ./...` returns ZERO across the
whole tree with iam v1.34.11, o11y v1.5.54 and commerce v1.49.61 — the three
dependencies that had to publish first. Graft appears in no .go file in any repo
in the estate.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Cloud pinned zip v1.18.23 while latest was v1.23.0 — five minor versions of
drift on the framework every subsystem composes through. v1.23 removes the
second and third ways to attach something to an app and leaves exactly one:
Use(cs ...Component), where a Component is a Handler or an *App included by
reference.
Cloud's own four adaptations:
- Router.Use widens from ...zip.Handler to ...zip.Component, mirroring
zip.Router exactly. That mirroring is load-bearing: ZipApp's type switch asks
whether a *zip.App satisfies this interface, and a narrower Use made that case
IMPOSSIBLE — the compiler rejected the switch outright rather than silently
taking the wrong branch.
- scope.Use forwards Components unchanged, still once per declared prefix.
- (*zip.App).Add is gone. zip.Load already returns the leaf *App and an *App IS
a Component, so both call sites capture the leaf and Use it. The eager and
lazy rungs of the plugin ladder keep their existing error handling; only the
attach verb changed.
- Five bare closures passed to Use now go through zip.H. Go will not implicitly
convert func(c *zip.Ctx) error to an interface that only the named Handler
type implements. Route methods still take ...Handler, so no route registration
changed — only Use sites.
Verified: with zip at v1.23.0, `go build -tags sqlite_math_functions ./...`
reports ZERO errors from apps/, cmd/, clients/ or internal/. The only remaining
failures are in three dependencies that must publish first — commerce
(mintRouter implements the old Use, and reaches Fiber() through the Router
interface, which no longer exposes it), iam and o11y (both call the now-
unexported app.Prepare; the public replacement is app.Build, which returns an
error Prepare did not). Those are in flight; this commit is the cloud half and
does not build green until they land.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The site edge is mounted in two processes — the light router that owns the
public port (cmd/cloud) and cloud.Listen in every per-app child — and each
resolved sites.Config for itself, from two spellings of the first-party keys
(CLOUD_SITES_FIRSTPARTY_* against CLOUD_SITES_FIRST_PARTY_*) with two sets of
defaults. Neither spelling is set in production, so the two processes booted
different policy off their DEFAULTS alone: the router resolved no first-party
apex, so hanzo.ai never entered its self-domain set, so every hanzo.ai host —
api.hanzo.ai included — was a custom-domain candidate and took the per-request
binding lookup the self-domain exclusion exists to keep off that path.
The reserved denylist was NOT affected. Both spellings read the same
CLOUD_SITES_RESERVED key, the operator value only ever ADDS via
SetReservedExtra, and the labels an attacker wants are baked into reserved.go —
so an empty value cannot un-reserve anything. Measured at the live edge:
www/api/app/admin/stg/login/wallet.hanzo.app all fall through with no
X-Hanzo-Site, against quest.hanzo.app which answers with one.
The env keys and their defaults now live in apps/sites, the package that owns
the type, and both call sites read that one function. Config keeps only Domain,
which feeds the self-domain set.
Also removes exports with no caller in this repo or any that import it:
OKList/OKRaw (envelope.go, whose note about a clients/admin/core delegator was
stale — that package is gone), BrandInfo, DegradedNames/IsDegraded (the release
smoke reads /v1/health over HTTP; the in-process accessors had only tests, whose
coverage of the live Degraded/Degradations pair is kept), and the four inert
Stage-0 control-plane fields with their NODE_ID/PEERS/ROLE/CONTROL_PLANE_QUORUM
reads.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every op on /v1/engine was dialling a refused connection in production. The
upstream default named :1234; svc/engine in namespace hanzo exposes ONE port and
it is 36900 (name http, port 36900, target 36900), and the cloud Deployment sets
no ENGINE_UPSTREAM — so the wrong default WAS the production value. status
answered reachable:false, and models, model and system answered 503. The
subsystem was honest about being unreachable, which is why nothing alarmed: a
truthful report of a broken configuration reads exactly like a runtime that is
down.
1234 is standalone `hanzo serve`'s default. 36900 is the port the engine binds as
the node's engine, and 36900 is what is deployed — so the comment claiming 1234
was "the in-cluster Service of the engine deployment" described a process this
cluster does not run. The 1234-vs-36900 confusion is already on record from the
desktop build, where a frontend discovered models at one port while the engine
that answers ran at the other; this is the same mistake on the cloud side.
MEASURED against the deployed engine over a port-forward to svc/engine:36900,
which answers exactly the three endpoints this plane calls, all 200 —
GET /health ("OK"), GET /v1/models (the model list, with a loaded model carrying
"status":"loaded") and GET /v1/system/info (the SystemInfo document: os, kernel,
cpu, memory). So the upstream is present and correct and only the port was wrong;
this is one token, not a redesign.
ENGINE_UPSTREAM stays the override, and 1234 remains right where it is right — a
dev box running `hanzo serve`. Behaviour only: no route, no schema, no operation
id and no published byte changes, and regenerating the subset produces no diff.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An operation id is the generated SDK METHOD NAME and the CLI COMMAND, and a
summary is what the MCP tool list shows a model choosing between tools. This
plane stated neither, so both were defaults, and both defaults were wrong in a
way only a caller sees.
zip derives an unstated id from the path, so the four ops published
`get_v1_engine_status`, `get_v1_engine_models`, `get_v1_engine_model`,
`get_v1_engine_system` — path mangling where the rest of the fleet publishes the
product and the noun. The risk product's thirty-one operations are `riskScore`,
`riskState`, `riskDatasets`, `riskLabelCoverage`; these are now `engineStatus`,
`engineModels`, `engineModel`, `engineSystem`, which is also the rule this
package already applied to its own SCHEMA names and only to those.
A summary defaults to the first sentence of the Go doc comment, and a Go doc
comment opens with the Go IDENTIFIER — so the published summaries read "Status
reports whether the engine deployment is reachable", "Models lists the models
the engine serves", "Model reads one model's load state". A Go symbol name was
the first word a CLI user, an SDK reader and a model picking a tool saw. Each op
now states a summary written in the imperative for the person calling it, and
the doc comment stays a Go doc comment that zipdoc still lifts as the
description: two audiences, two sentences, one declaration.
FORWARDS-ONLY, and it costs nothing: this plane has no customers on it. No
alias, no redirect, no compat shim.
Regenerating from source changes exactly four operation ids. No path is added or
removed, no (path, method) pair moves, no schema changes, and openapi/floor.json
is byte-identical because the operation count did not move.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
openapi/floor.json is the ratchet: "a published surface may grow and may not
quietly shrink". It carries a global path and operation count AND a per-PRODUCT
count, and the per-product half is the half that binds — floor.go compares
`f.Products[p]` against the regenerated count and refuses a product that lost
operations. The file's own code states the failure mode out loud: "an absent floor
makes every shrink legal".
The merge 94df22e1 resolved this generated file to
`{"paths":1684,"operations":2336}`: the entire `products` map — 180 entries —
GONE, and both global counts rolled backwards. Every commit before it carries the
map. So on main right now every product may shrink without the gate saying
anything, and the global floor sits five paths and five operations below the
surface actually published.
IT ALREADY COST SOMETHING, within hours. Deleting the /v1/train facade removed ten
operations, and a ten-operation shrink is exactly what this ratchet exists to make
an author state deliberately — the dataset move had to lower `ml: 14 -> 7` in the
same commit that moved the routes, and said so. The train deletion needed no such
line, because there was no per-product floor left to lower. The gate was not
merely stale; it was switched off, and a shrink walked through it unremarked.
Restoring it is `make describe` and nothing else: 1689 paths, 2341 operations, 178
products. `train` is simply absent from the restored map rather than floored at
zero, because the map is regenerated from the document that exists — the ratchet
resumes from the current truth, which is the only honest baseline available once
the old one has been discarded.
Committed on its own because it belongs to no surface change: these counts are
identical with or without the engine renaming beside it, since an operation id
does not move a count.
The lesson is the fleet's own about derived artifacts, with a sharper edge: two
derived files can agree with each other while both are wrong, and a MERGE is a
third way for a derived file to become wrong — resolved by hand toward whichever
side the conflict presented. A ratchet resolved toward the weaker side does not
look stale. It looks fine, and it is off.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/label addressed /v1/ml/labels, apps/reference addressed /v1/ml/reference and
apps/dataset addressed /v1/ml/datasets. All three are the RISK product — the
labels a decision is adjudicated with, the lookup data it consults, and the
snapshot its model was fitted on — and all three were corrected to /v1/risk/*.
The reason is the same each time: openapi.Product takes a product from the FIRST
/v1 segment and a per-op tag cannot override it, so an address IS a published
product membership, and /v1/ml is the live serving product. Filing a second
product there makes /v1/ml/models mean two things at once.
apps/dataset's address_test.go says the third time "stops being a recollection
and becomes this gate" — and builds it, for apps/dataset. A per-app gate cannot
catch the fourth time, because the fourth time happens in a fifth app that does
not have one. So the invariant is stated once, on the side every app must pass
through: the ROUTING GRANT. A plane cannot publish under /v1/ml without a
manifest row saying so, which makes the row the one place the mistake is always
visible.
It refuses a second OWNER, not a second app, and that boundary is measured
rather than assumed: fourteen products here are answered by more than one app and
27 apps publish into more than one product, so neither "one app per product" nor
"one product per app" is a fleet invariant, and asserting either would invent a
rule the fleet does not keep. What all three mistakes actually broke is narrower
and true — the serving prefix has one owner.
The second test is the half the ai incident argues for: a gate that only refuses
intruders stays green when the owner itself vanishes. Narrowing ai's row to
/v1/ai once took the entire inference surface off the wire with every probe
green, so the owner's own leaves are asserted individually — a count cannot say
WHICH address stopped routing.
BOTH DIRECTIONS ARE MUTATION-PROVEN, because a gate nobody made fail is a gate
nobody knows works. Re-addressing dataset to /v1/ml/datasets fails with the
ROUTING GRANT message naming the app, the prefix and the product it would join;
narrowing ml's row to /v1/mlops fails BOTH tests — the non-vacuity check (which
also proves `under` is segment-aware, since /v1/mlops is correctly not under
/v1/ml) and each unrouted leaf by name.
/v1/train is deliberately NOT fenced: it was just deleted because the Kubeflow
CRDs behind it are not served. A gate must fence prefixes that exist, and naming
a deleted one would trip the non-vacuity check while saying nothing true.
It also does not judge the NAME. `ml` being the vague word is why this keeps
happening, but renaming a prefix that is published is a wire change with its own
cost; until that is worth paying the ambiguity is fenced, not resolved.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/train/* was a thin proxy onto two Kubeflow CRDs that this cluster does not
serve: trainer.kubeflow.org/trainjobs and kubeflow.org/{experiments,trials}.
GET /v1/train/health answers 503 degraded in production right now
({"crds":{"experiments":false,"trainjobs":false},"status":"degraded"}) with
nothing in cloud changed — the CRDs were retired underneath it. Ten operations
go, and with them the degraded door.
Measured before deleting, cluster-wide: zero TrainJobs, zero Experiments, zero
Trials, and no ClusterTrainingRuntime for a TrainJob to reference. The katib
half could not have worked at all — katib's admission webhook requires the
namespace label katib.kubeflow.org/metrics-collector-injection=enabled,
ensureNamespace writes only {managed-by, hanzo.ai/org}, and no namespace in the
cluster carries it. So POST /v1/train/experiments took the billing gate, created
an Experiment, and katib never admitted a Trial.
There were also TWO doors onto one TrainJob CRD: this one and the hanzoai/ai
broker at /v1/finetune/*, which has the product around it (presets, HF pickers,
status polling, deploy-to-serving). One door survives.
KServe STAYS. /v1/ml/models is the only path in the estate that serves a
classical artifact end to end, and it is proven: POST /v1/ml/models with an
sklearn joblib -> 201, the storage-initializer pulls the model, then POST
/v1/ml/models/{name}/predict -> 200 with correct predictions on the
kserve-mlserver runtime. GET /v1/ml/health is 200 today. Its runtime-capacity
clause and every serving test are untouched.
openapi/floor.json needs NO edit on this base: the shrink guard is a FLOOR, and
main's committed floor (1684 paths / 2336 operations) is already below the
post-deletion document, measured at 1689 / 2341. The guard still bites — raising
the floor above the real count fails TestFleetIsTheWeaveOfItsApps with the same
"THE PUBLISHED SURFACE SHRANK" report, which is how these numbers were read.
(An earlier pass here lowered a floor that ALSO carried a per-product map; main
has since dropped that map, so the rebase takes main's shape unchanged.)
The billing-gate integration tests keep
their coverage by exercising the surviving create (POST /v1/ml/models) — the
gate is the shared create() body, not a per-kind one.
Also repointed every pointer that named the deleted route, so none dangles:
apps/engine's intentRefused reason and LLM.md (now /v1/finetune/jobs), spend.go's
routing-union example, apps/platform/drift.go's analogy, and the mutation in
scripts/mutate.py whose anchor line and target test are both gone (it would have
reported ANCHOR-MISS).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
getUser sent the `<owner>/<name>` composite as `id`, and IAM wants the two as
separate fields. Measured against the running service with the console client:
?id=hanzo/2d4d67ab-… 400 field "owner" is required
?id=hanzo/z 400 field "owner" is required
?owner=hanzo&name=z 200
So NO caller of this ever read a user row. The avatar write is where it became
visible — "photo stored but the profile could not be updated", with
`iam non-envelope response (400)` behind it — but moveUserToOrg carries the same
fault silently, and its failure mode is worse: the whole-row re-submit it feeds
would have nothing to re-submit.
`name` is the USERNAME (the row's own `name`, "z"), never the UUID that `sub`
carries — the row this now reads has id 2d4d67ab-… and name z, which is why
addressing it by the uuid found nothing.
The composite is KEPT for the one caller that has no owner to send: a first-run,
org-less user, where resolving their authoritative (owner, name) is the entire
purpose of the read. Refusing that here would break onboarding to fix the avatar.
The fake IAM now insists on the same shape, so a client that regresses to `id`
for a caller that HAS an owner fails in the suite instead of in production.
A sibling dissolved PUT /v1/risk/state/appetite while this branch was open (the
decision regime has one address now), so typed.go conflicted on the op that used to
sit between `state` and `snapshot`. Resolved by taking MAIN'S typed.go whole and
re-applying this branch's five changes onto it, rather than by editing the conflict
markers: the incoming change deletes an op, and a hunk-level resolution is how a
deletion gets silently un-deleted.
zipdoc_gen.go is generated — resolved by regenerating from the merged source, never
by merging the artifact. Same for openapi.yaml and plugin/risk/openapi.json.
Verified after the merge: 11 operations before and after, floor.json identical to
main (risk 31, paths 1695, operations 2351), and main's own new gates — verb_test
and learn_cost_test — pass beside this branch's.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Regenerating every subset from source is the drift gate, and it turns up drift in two
apps beyond the ones whose prose this branch fixes. Both are real, and both are in the
direction of the published document being AHEAD of the code: it advertises money fields
the runtime stopped returning.
7b4ddd9e removed the mint from the referral read and 3a8be85b removed it from the promo
redemption. Both regenerated their zipdoc_gen.go, so the PROSE moved; neither
regenerated plugin/<app>/openapi.json, so the SCHEMAS did not. The result is 15
referral fields and 4 promo fields published as response properties no handler can
populate -- refereeGrantCents, referrerGrantCents, creditsEarnedCents, refereeBonusCents,
referrerBonusCents, grantedCents, creditsCents, the credited counters and the txn ids on
the referral side; creditCents and creditEntryId, plus the plan and seats request fields,
on the promo side. apps/referrals' assertNoMoneyKeys already fails the runtime response
if any of them reappears, so source and test agreed with each other and only the artifact
disagreed. Two fields the code does return -- Redemption.discountCents and
sweepResult.qualified -- were missing for the same reason.
Every SDK, the MCP tool list and the CLI are projections of this file, so those were dead
money fields in every generated client's types.
The operation-count ratchet did not and could not catch it: openapi/floor.json counts
paths, operations and per-product operations, and none of those move when a response
schema loses a field. It ratchets UP here -- billing 26 -> 27, operations 2351 -> 2352 --
for the newly described GET /v1/billing/tier.
Proved structurally rather than by diff, over the parsed documents with description and
summary elided at every depth: across openapi.yaml the only identity change is that one
operation GAINED, no operation was lost, no operationId moved, and all 22 contract
differences are the marketing and referrals schema fields named above. The commerce prose
entry contributes none of them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The repo's own gates could not pass on main. Both failures are the same shape --
prose that drifted from the routes it describes -- and both are fixed at the one
place the prose is written.
GET /v1/billing/tier published an operationId and nothing else, so openapi.Complete
refused commerce's document outright and `make -f mk/fleet.mk surface-check` died
there. Its handler is commerce's, in another module, and its path reaches the router
through a table rather than a literal, so there is no doc comment here for zipdoc to
lift -- which is what describe.go exists for, and what every sibling on that same
registration loop already uses. The entry states what the caller gets, that the
subject keys are pinned to the validated caller before the handler runs, that tier
is derived from active and trialing subscriptions, and the two rules a reader
otherwise gets wrong: gate on effectiveAvailable rather than prepaidAvailable,
because granted credits spend too and an account funded only by a grant reads zero
prepaid while holding real spendable credit; and a subscription-store error answers
500 rather than downgrading a paid subscriber to free.
POST /finance/starter was the inverse -- prose describing an operation that no
longer exists. The op, its middleware and its tests went with the automatic
money-mint (41b23f12) and the lift was never rerun, so commerce's registry still
explained a route the router does not carry. Regenerating drops it; nothing here
re-adds it.
POST /sites/resolve and /sites/resolve-org are live typed ops in
apps/projects/sites_rpc.go whose doc comments had never been lifted at all.
Verified per package, the way the generator loads: zipdoc -check is clean across all
100 directories that declare it, where commerce and projects were stale.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
THE MODEL WAS A PLACE. The `model` table is keyed on the tenant and written ON
CONFLICT DO UPDATE, so it is ONE CELL per organisation and every write destroys
the state before it. That single fact was three separate open problems: there was
nothing to roll back TO, so rollback needed an op that shipped the masses out to
the caller and an op that took them back in; two fitted models could not both be
named, so champion-and-challenger had nowhere to live; and no decision could say
which model produced it.
A content address ends all three. The value's name is a pure function of what
makes two models answer the same event differently — the shape, the geometry seed,
the position in the window, the threshold, the masses as IEEE-754 bits, and the
FOLD WATERMARK, which is not redundant with the mass count: two models with
identical masses reached by different routes disagree about what is left to fold,
and one will re-teach history the other will not.
THE IN-PROCESS MODEL STAYS MUTABLE, and that is the design. Measured: one encoded
value is 466 KiB at the reachable shape — 25 trees of 511 nodes over two windows,
every mass carrying a full mantissa because folding blends them. The sweep writes
every 500 events or every 30 seconds, so "a value per write" is 56 MiB an hour per
active organisation to record a counter going up. So identity is a SUCCESSION OF
STATES: the `model` row stays a place and its whole job is resuming a killed
process, and `published` is append-only, addressed by content, retained under a
budget stated in BYTES (10 values, following policyBudget's reasoning).
WHAT THIS DELETES. riskSnapshotBody is gone from the wire in BOTH directions, with
its two conversions. Publishing answers with a NAME; adopting takes one. The
masses never leave the organisation's own store, so the caller stops being the
custodian of a tenant's model — the engine's own Restore asks for exactly this
("it belongs where the tenant's own data belongs, and sealed if it travels"). Two
tests changed from refusing a threat to proving it unreachable: a caller can no
longer describe a model instead of naming one, so there is no body to compose and
no seed to choose. Rollback is naming a prior address; what the working state
descends from is DERIVED from its mass count, so rolling backward is right for
free where a stored head pointer is exactly what would fall out of step.
A SCORE NOW NAMES THE MODEL SPACE IT RAN IN. The shape is cached on the residency
and read in the same critical section as the verdict, so it costs nothing and
cannot cite a space the score did not run in. It is deliberately NOT an address:
the masses at the instant of a score are counters between two published values, so
citing one would claim that value produced this score. The shape, the policy
version and the event's own time are what IS true, and the value history's clock
brackets the decision from there.
ISOLATION, MEASURED RATHER THAN ASSERTED. The address deliberately omits the
organisation — a name that must be unguessable is obscurity, not isolation. Two
mutations were run and two drafts of the test comment were wrong before they were:
the per-organisation FILE and the `tenant = ?` row predicate are two layers and
EITHER ONE ALONE HOLDS, so a foreign org handed a real address resolves nothing
through all three doors with either layer removed. The single-layer regression is
caught where the file layer is already absent by design — two brands' identically
named organisations share one file — and that test fails on that one mutation
alone. The finding underneath inverts the obvious reading: the per-org file is not
what makes this isolated; the row predicate holds on its own.
11 operations before, 11 after: no path moved and the floor is untouched. The two
paths SHOULD collapse into one (/v1/risk/state/model, POST to publish and PUT to
adopt) — that is the prefix plane's call, not this commit's.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/risk/learn recorded a batch, trained on it AND answered the model's
verdict on every event. Three things under one name: you could not observe
without training, and could not train without being answered.
An observation is a VALUE the plane records; learning is a TRANSFORMATION over
observations; a verdict is a QUERY against the result. POST /v1/risk/score is
already the query and is already pure. So learn drops the verdict and answers
`learned` — how many events the model actually learned from.
WHAT IT COST TO CARRY. plane.learn called the engine twice per event, Inspect for
the response's verdict and Assess for the counters, and both enter the engine's
judge: two projections of the point at three aggregate reads each, and two walks
of the forest. Above the cut both also ran the counterfactual attribution, a
further walk per dimension over nine dimensions. Assess's own return was
discarded, so the attribution was computed twice and thrown away once.
MEASURED, and stated as measured (learn_cost_test.go, BenchmarkLearn):
before after
batch 8 31.5 µs/event 27.8 µs/event -12%
batch 128 20.0 µs/event 17.5 µs/event -12%
batch 128 6717 allocs 6169 allocs -8%
A tenth, not a half: the durable record and the aggregates are the larger part of
what a caller waits for, and the attribution is reached only by the share of the
stream the appetite admits — one per cent by default.
NOTHING DEPENDED ON THE SYNCHRONOUS VERDICT. No CLI command references risk, no
SDK carries a risk client, and the one in-process consumer of a verdict is
cloud.Decide, whose scorer is never installed outside tests (SetRiskScorer has no
non-test caller), so every question it asks answers {allow, scorer-absent}. The
live plane reports one resident model built once. Observe-and-judge in one round
trip is now a COMPOSITION and the published prose says which order: score first,
then learn, so the verdict is the model's opinion of an event it has not yet
learned from.
A DUPLICATE IS NOW WHOLLY INERT. It moved nothing before and was still judged;
now it costs no model work at all, is not counted in `learned`, and is not
metered — the meter runs on what was DONE, which is this app's own stated rule
for the gate/meter pair. The gate still bounds the batch the caller stated,
because how much of it is new is unknowable until the record is written.
TestPlane_TheVerbsStaySeparate (verb_test.go) walks the package and fails if
learn calls Inspect or score calls Assess. It is structural because the braid is
invisible behaviourally: with Inspect put back, every other test in this package
still passes.
zipdoc regenerated; plugin/risk/openapi.json and openapi.yaml rewoven
mechanically (make -f mk/fleet.mk openapi-weave, exit 0). riskScoreOut, riskCause
and riskValue stay in the document — score still answers them; they are simply no
longer reachable from riskLearnOut. Package green under -race.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/risk/state answered seven kinds of thing at once and PUT
/v1/risk/state/appetite answered all of them back to a call that changes three
numbers. This lands the half whose home already exists: the decision regime is
read and written at ONE address, /v1/risk/policy, and both verbs answer
riskPolicyOut. The published surface loses a path and keeps its operation count.
plane.appetite no longer computes the model value — r.mod.State and r.vel.strain
are gone from the policy write. What the model IS stays for the model-value track;
the telemetry kinds (refusals by reason, blind by feature, the realised share,
saturation, aggregate strain) stay on GET /v1/risk/state until they are emitted on
the fleet's one metric road, because deleting them before that would remove an
organisation's only view of its own refusals.
Per-organisation isolation is proven through the new address by the two policy
tenancy tests. Mutation table in the commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The regime was written at PUT /v1/risk/state/appetite and read at GET
/v1/risk/policy — two addresses for one plane, the writing one named after the
mutable spot the regime happened to sit in. `state` is literally the word for
that spot; a regime is not a spot. It is three numbers an organisation adopted,
at a time, by a named identity, kept under a version forever after.
Worse than the second address is what the write ANSWERED: riskModelState, fifteen
fields describing the whole MODEL — its shape, what it had learned, whether it
was warm, the threshold in force, the realised share beside the stated one, every
refusal by reason, every feature that read blind, the fold coverage of the event
surface, and the aggregate strain. A call that changes three numbers knows none
of that. It was reporting a model it happened to be holding a lock on, which is
the same braid, one level up, as the regime living on the learned state's row —
the defect the versioned policy record was cut to fix.
So the plane has ONE address and both verbs answer riskPolicyOut. plane.appetite
returns the VERSION it left in force and nothing else: the two calls to
r.mod.State and r.vel.strain are gone from the policy write, so a policy write no
longer computes the model value at all. policyOut is the one projection both verbs
render through, and the version in force is a PARAMETER to it — a write knows what
it enacted from inside the lock it enacted under, and re-reading it there would let
two concurrent restatements each report the other's version.
There is no `minted` flag. plane.enact is idempotent on the regime, so a
restatement answers the version already in force and equality of the VALUE is the
signal. A boolean about the operation is a second thing to keep true beside the
value that already says it.
The published surface SHRINKS by one path and holds its operation count: the write
joined the address that already existed instead of keeping a second one. Mechanically:
paths 1696 → 1695 (-/v1/risk/state/appetite)
operations 2351 → 2351 (PUT moved onto /v1/risk/policy)
openapi/floor.json is lowered in this commit because the weave refuses a shrink
otherwise, which is the gate working: a reduction is reviewed next to its reason.
The op's published schema description loses 27 lines of riskModelState /
riskSurface / riskAggregates prose it had no business publishing.
Nothing consumed the old address. Swept every checkout under ~/work: the only
references outside apps/risk's own tests were the derived specs. Live, PUT
/v1/risk/state/appetite is routed and GET /v1/risk/policy is 404 — the policy
plane is merged but not yet deployed — so this is the one moment the consolidation
costs a deployed client nothing.
Per-organisation isolation is untouched and proven through the NEW address:
TestPolicy_HistoryIsPerTenantOverTheWire and
TestPolicy_TwoBrandsShareAFileAndNotAHistory both state their regimes at PUT
/v1/risk/policy and both still see nothing of the other organisation.
Mutation table — each named test RED under the defect, GREEN after revert:
M1 re-register PUT /v1/risk/state/appetite TestPolicy_HasOneAddress RED
M2 put a model field back on the answer TestPolicy_AWriteAnswersThePolicyAndNotTheModel RED
Verified: go build -tags "sqlite_purego sqlite_math_functions sqlite_fts5" ./... =
0; go vet apps/risk = 0; go test apps/risk = ok; -race = ok (23s); zipdoc -check
./apps/risk/ = 0. Fleet-wide zipdoc -check reports the same 16 stale/missing
generated files before and after this change (measured by stashing it), so this
adds no drift: apps/risk is not among them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/iam/* was forwarded to a separate identity origin by an edge in cloud:
a tenant gate, an org pin, a read/write allowlist and a sign-in passthrough,
all re-deciding what IAM already decides for itself.
It mounted on `!cfg.Enabled("iam") && iamHost() != ""`. An empty enable list
mounts everything (Config.Enabled), and no deployment in the fleet sets one,
so Enabled("iam") is true everywhere and the condition never held: the iam
app owns /v1/iam through its graft, on every deployment, and has since the
graft landed. The edge answered no request anywhere.
Deleting it removes the second reader of the identity address and the second
place a tenant scope is decided. The prefixes are unchanged — manifest's iam
row already routes them — and the org pin the edge applied is IAM's own
authorize, in the process that owns the store.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Removing CLOUD_ENABLE/--enable (ecafb31c) deleted the flag from the binary but
left three launchers passing it. Go's default flag.CommandLine is ExitOnError,
so this is not a warning:
$ ./bin/cloud --enable=iam
flag provided but not defined: -enable
`make run` and the documented compose quickstart both died before boot, and the
four README brand examples were copy-paste instructions to do the same. The
removal was deliberate and test-enforced (cmd/cloud/mount_test.go asserts the
flag's absence) — the launchers were simply never updated with it. My miss.
RUN_ENABLE is renamed RUN_PLUGINS because it never mounted anything: it is the
list of plugin BINARIES to build so local dev does not build all 106. The host
mounts what manifest.Apps lists and resolves each plugin as a file beside
itself; a lazy one with no binary never starts, a Required one fails loudly.
Keeping the name would have said the binary takes a mount list, which is the
thing that took devnet down twice.
compose.yml's HANZO_ENABLE goes with it. Its environment: block is inert
anyway — HANZO_BRAND/DOMAIN/DATA_DIR have no Go read sites and work only
because the same file passes the flags, which main.go forward() republishes as
CLOUD_*. That is a separate cleanup, not this one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/risk answered ONE refusal in TWO shapes, with the same status and the same
sentence in both. Measured on this package's priced ops:
this app's own copy of the rule 403 {"status":403,"error":"no validated principal"}
the fleet's money door 403 {"error":{"code":"forbidden","message":"no validated principal"}}
The nested one is canonical — it is what the edge gate and every other Hanzo
surface emit, so a client that reads `error.code` reads it everywhere. The flat one
was zip rendering a returned error, which meant `error.code` was absent from
exactly one product's refusals.
[ops.gate] held its own `if ledger == ""` copy of the rule from before the fleet
door had one. [cloud.ResourceMeter.Gate] now refuses an empty org above both of its
branches as [cloud.ErrNoLedger], and [cloud.denial] renders that as the 403 with
this exact sentence, so the copy decided nothing except the shape. Deleted; the
twenty-four lines of prose explaining a defect that is fixed elsewhere are replaced
by a citation of where.
Deleting it exposed a residual the report did not name, because a status assertion
cannot see a shape: POST /v1/risk/search reaches [ops.admit] BEFORE it prices
anything, so its refusal came from [tenantOf] — still flat. Eight ops nested and
one flat is worse than nine flat, so tenantOf's no-principal branch answers with
the same fleet value. A malformed org is a different fact and keeps its own
sentence.
And the hole the deletion could have opened, closed: ResourceMeter.Gate returns
EARLY at zero cost, before its own empty-org refusal, and the price is an operator
knob where 0 is legal. So "the money door refuses an unidentified caller" holds
only while somebody is charged; the app's deleted copy had covered that by
accident, running before the price was computed. tenantOf covers it on purpose, and
a test sets the price to zero and requires the same 403.
E1 risk holds its own copy again TestPricedOps_RefuseAnUnidentifiedCallerInTheFleetsOwnEnvelope RED
E2 tenantOf answers flat again TestPricedOps_RefuseAnUnidentifiedCallerInTheFleetsOwnEnvelope RED
E3 tenantOf answers flat again TestPricedOps_RefuseAnUnidentifiedCallerEvenWhenTheOperatorPricesThemAtZero RED
E4 the fleet door stops refusing TestPricedOps_ResolveTheTenantBeforeTheyAskForMoney RED
E5 identity takes a money status TestPricedOps_ResolveTheTenantBeforeTheyAskForMoney RED
All GREEN after revert. E4 and E5 mutate cloud's own money door, which is where
gate_order_test.go's mutation note now points — the note said "delete the
empty-ledger guard from ops.gate and every op reports 503", and that had silently
stopped being true.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The label row said a specific prefix MUST be registered before the bare /v1/risk
or "every label op lands on the decision plane", and cited
TestSpecificPrefixesPrecede as the gate that pinned it. Neither half was true.
No test of that name exists anywhere in the tree, and the rule is not the
router's: registering the bare /v1/risk FIRST and all three specific prefixes
after it, the live router still delivers /v1/risk/labels to label,
/v1/risk/reference to reference and /v1/risk/datasets to dataset, and the router
oracle stays green. zip resolves nested static prefixes by specificity; mount
order decides only between EQUAL patterns, which is what the ai-before-zen note
is actually about.
Says that, and names the gate that does hold —
TestEveryServedPathReachesTheAppThatServesIt builds the real router from these
rows and asks it, per published path, which app receives the request. An oracle
over every row at once is why no row needs a rule of its own to remember.
Prose only; no row moves and no behaviour changes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The dataset plane addressed /v1/ml/datasets, beside the KServe model-SERVING
plane's own /v1/ml/health and /v1/ml/models — one prefix, two products.
openapi.Product reads an operation's product off the FIRST /v1 segment and
nothing else, so `ml` published 14 operations that were seven serving ops with
customers on them and seven dataset ops whose rows feed the risk model, which
learns in-process from the org's own events and is never served by KServe.
Moves the five paths to /v1/risk/datasets and renames the seven operation ids
and nine schema names to the risk face, so the SDK method, the CLI command and
the generated type each name the product they belong to. Fourteen of the fifteen
Go types take the bare risk<Noun> name; the dispose pair carries the noun it
disposes of, because apps/label already publishes riskDisposeIn for LABEL
disposal and the fleet's schema namespace is flat.
The tag was never the product: openapi.Fold assigns op.Tags from the router
projection, so zip.WithTags("ml") had been inert. It now reads "risk" too, so a
declaration and a projection do not disagree.
/v1/ml is untouched. Its four serving paths (7300 bytes) and its three schemas
(1533 bytes) are byte-identical, and regenerating the fleet document from source
moves exactly the seven dataset operations and the nine dataset schemas: 2350
operations, 1695 paths and 2060 schemas before and after, with every other
triple unchanged.
No alias and no redirect from the old prefix, because nothing calls it: a sweep
of every git checkout under work/hanzo finds /v1/ml/datasets named only in this
repo's own generated artifacts and prose. No client, SDK, CLI or MCP consumer
names it.
The floor ratchet refuses a shrink, so `ml: 14 -> 7` is lowered here in the same
commit that moves the routes; `risk` rises 23 -> 30 on the same regeneration.
address_test.go makes the address a gate rather than a third recollection after
label and reference: the product, the operation-id prefix and the schema prefix
are each asserted off the published projection, and each assertion fails when
the projection is empty, so none of the three can pass by examining nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
kserve admits an InferenceService whose model format no ClusterServingRuntime
supports and then never schedules it, so /v1/ml/health answered 200 while every
deploy hung. A served CRD is not capacity, and the probe read only the CRD.
health now takes the cluster-scoped coordinate the plane needs at least one of
(the zero GVR for a plane with no such fact — training carries its own images)
and reports the count as its own field. An unreadable list reports the read
error instead, because a missing grant is a broken probe and not an empty
cluster, and the two call for different acts.
scripts/mutate.py carries four rows: dropping the clause, folding the read error
into the count as zero, asking capacity of training, and reading the runtime kind
at the InferenceService's v1beta1 instead of v1alpha1.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The bounds on `review` and `sample` were spelled twice — once at [ops.appetite]
and once in [admitRegime]. Collapsing them to one spelling is right. But the
SURVIVING spelling had no test: admitRegime was made to `return nil`
unconditionally and the WHOLE package stayed green, so the published contract
(`review ∈ (0, 0.5]`, `sample ∈ [0, 1]`) was held by code that could be deleted
without one failure. Same shape as a control switching itself off, one layer down:
the bound is present and nothing measures it.
Both halves are asserted, because a refusal that half-applies is worse than no
bound: six out-of-contract appetites are refused 400 with the field named, AND the
regime in force is untouched afterwards — no version minted, no history row, the
model still deciding under 0.02/0.10. Over the WIRE, because that is where the
contract is published and where the op's deleted copy used to answer.
A1 admitRegime returns nil for every regime named test RED, whole package RED
A2 plane.enact stops calling admitRegime named test RED, whole package RED
Both GREEN after revert.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An organisation that took its model out of shadow BEFORE the model had learned
anything was told live=true and had nothing written down. The regime lived on the
same row as the learned state, and that row's writer declines to write while the
snapshot holds no learned mass — correctly, because there is no state to lose. So
PUT /v1/risk/state/appetite answered 200, reported live, and persisted nothing;
this binary deploys Recreate at one replica, so the next rollout rebuilt from
defaultConfig — shadow — and the model decided nothing. No error, no log, nothing
to alert on: a model silently disarmed, on a routed door.
Two conflicts, both resolved by REGENERATING rather than choosing a side:
openapi.yaml and openapi/floor.json are derived, and the branch was cut when the
fleet had 1684 paths. `make -C apps/risk describe` + the weave produce 1696 paths
and risk 24 operations — main's 1695/23 plus this branch's one op, GET
/v1/risk/policy. apps/risk/zipdoc_gen.go and plugin/risk/openapi.json regenerate
byte-identical to the branch's, so the projection did not drift.
One semantic conflict: `plane.close` gained the shutdown window on main
(cda5a031), so the branch's four `close()` calls in policy_test.go take
context.Background() like the other eighteen in the package.
Mutation table re-run against THIS merge, since main moved under the branch —
each named test RED under the defect, GREEN after revert:
M1 plane.enact writes no durable row TestPolicy_GoingLiveSurvivesTheRollout RED
M2 verdict drops the version citation TestPolicy_EveryScoreCitesTheRegimeItWasDecidedUnder RED
M3 no value-equality short circuit TestPolicy_ARestatementOfTheSameRegimeMintsNoVersion RED
M4 the 24-per-24h bound deleted TestPolicy_TheRateBoundBindsAndIsNamed RED
M5 a pre-existing regime is not adopted TestPolicy_ARegimePredatingTheRecordIsAdopted RED
M6 the history read loses its predicate TestPolicy_TwoBrandsShareAFileAndNotAHistory RED
M7 retention stops disposing TestPolicy_RetentionIsCountedAndNotSilent RED
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Four defects on the LIVE /v1/risk plane, each closed so the wrong thing is
unrepresentable rather than merely unlikely, plus the harness rows that hold
them.
THE ORG STORE. CloseAll emptied its maps, which made "closed" indistinguishable
from "nothing opened yet" — so a request still in flight during a rollout
reached For() after shutdown and opened the file again, re-hydrating it and
re-claiming the fence lease the SUCCESSOR pod was claiming. Closing is now
terminal and every door answers ErrStoreClosed.
THE ROLLOUT. close() was `stop(); wg.Wait()` with every save queued BEHIND it.
The wait was unbounded inside a 30-second window while ONE search's durable
Sync is bounded at 30 seconds on its own, so the process was killed with not
one tenant's model written down — every tenant back to warming, and a warming
model refuses to score, which reads as clean. The saves now run
unconditionally; the drain gets what is left of the caller's own window; a
drain that did not finish is ErrDrainIncomplete and not a silence.
THE SEARCH BOUND. "ONE RUN PER TENANT" checked the slot and set it two
warehouse operations later, so sixteen concurrent callers each rolled the
tenant's source planes and read its whole history before any of them claimed
anything — and the ledger gate was the LAST thing in the sequence, so all of it
was free. The slot is claimed atomically with the check, and a search is priced
in TWO halves, each before the half it prices: the surface on the window the
caller stated, the grid on the measured history.
THE STRAIN REPORT. velocity caps PER SHARD, so the store drops a tenant's
subjects long before the flat census notices: 200 subjects in, the store holds
198, and the report said forgotten=0, saturated=false. Two of that
organisation's own subjects read as "has done nothing" and nothing said so.
reconcile now measures the store/census shortfall as a high-water mark, and the
report describes the bound that actually binds.
THE RESIDENT BOUND had no test at all: evict() could be made to return nil —
the bound fully disarmed, an OOM on a one-replica Recreate deployment — with
the whole suite green. Its operating point is now four properties held
together: SERVED past the bound, BOUNDED at it, LOUD on the probe, LOSSLESS on
the way out.
Mutation-proved: scripts/mutate.py "risk:" — eighteen rows, each reintroducing
one defect and going red on the named test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The ceiling was the wrong way round. maxOverrides was a chosen number (10,000)
and the byte figure was computed beside it from `row` — the SUM OF THE WIRE
TERMS, 1,152 bytes. A row does not cost what it carries: measured on a real
per-org store, the widest row this door admits costs 1,952 bytes once the
implicit index over the primary key, page slack and the encryption are counted.
So the published 120 MiB per-organisation ceiling was understating the truth by
1.69x, on the ONE volume every organisation's store shares.
Inverted, so the bound is in the dimension that binds:
ownBudget = 128 MiB the primary figure — what one org may occupy
rowBytes = 2048 MEASURED (1,952) plus a page of room
maxOverrides() = ownBudget / (sets x rowBytes) = 5,957 per set
The count is now the division rather than a number standing next to one, so
count x rowBytes IS the byte bound. Adding a set lowers the count instead of
quietly multiplying the volume.
`row` is renamed `stated` and keeps its old meaning — the WIRE width — because
the two figures are different quantities and conflating them is what caused
this. TestTheVolumeOneOrgMayOccupyIsStated now checks the quotient (one more
entry per set must not fit) instead of pinning a constant, and
TestOneOrgsOverridesCostWhatTheyArePublishedToCost fills a real store with
worst-case rows and fails if one costs more than the figure the count is divided
from — so rowBytes can never drift from a store again.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The header claimed "nothing held the mechanism at all — the whole suite stayed
green". That was true when it was written and is not true now: hold_test.go
landed the cross-tenant reclaim tests on main, and MEASURED with
scripts/mutate.py, TestEviction_IsCountedOnTheProbe and
TestEviction_WritesTheVictimsStateDownFirst kill all three mutations that disarm
this bound. A test whose stated reason for existing is false is a test the next
reader deletes for the wrong reason, or keeps for one.
What it actually adds is two things the pair cannot see: the bound asserted at
every one of maxResident+8 arrivals rather than once just past it, and a lossless
leg with no t.Skip exit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The surface read — rolling up to four source planes into the organisation's own
feature surface and reading the window back — ran BEFORE ANY GATE AT ALL. A
caller with no balance drove the whole warehouse cost, was refused at the very
end, and paid for none of it, as often as it cared to ask. Two rows: the gate
moved back below the work it prices, and the surface half deleted so only the
grid is paid for. Each goes red on the test that names the property.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Sixteen rows for the org-store lifecycle, the rollout, the search bound, the
strain report and the resident bound — so every assertion this cycle added is
held by a mutation that breaks the thing it guards, in the harness rather than in
a transcript.
AND THE HARNESS COULD DESTROY ITS OWN EVIDENCE. The restore is in a finally,
which does not run when the process is killed — a CI timeout, a ^C — so an
interrupted run leaves the mutant in the tree and the original in .mutbak. The
next run then copied over that backup, destroying the only clean copy, and
reported ANCHOR-MISS on a tree it had silently corrupted. That happened here and
cost a file. A leftover backup is now a hard refusal that says how to recover.
One row is written against the CONDITION rather than as an early return, because
`return nil` after the guard is unreachable code: go vet rejects it, the row
scores NO-COMPILE, and a mutant that never ran proves nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The run slot now bounds one tenant's CONCURRENCY. Nothing bounded its RATE: the
setup ran before any gate at all — the ledger check was the LAST step of begin —
so a caller with no balance drove a roll of up to four source planes and a
full-window read on the one warehouse every tenant shares, was refused at the
end, and paid for none of it, as often as it cared to ask.
Pricing the grid on its upper bound would close the same hole and leave no
viable operating point: the upper bound is maxHistory x the whole grid whatever
that organisation's history holds, so a tenant with two hundred events would be
refused unless it could cover twenty thousand.
So a search is priced TWICE, each half before the half it prices. The surface
from the WINDOW, which the caller states and which is therefore a number before
anything runs — the same unit and the same price ops.features already pays for
the same work, now spelled once in windowScreens. The grid from the MEASURED
history, where it already was. Each meters on what it actually did, so a run
cancelled by a rollout is still billed for the trials that ran.
The plane takes ONE money seam (charge: gate n, get the meter for what was done)
instead of a gate parameter and a book parameter, which is what makes "gate
before, meter after" the same shape at both halves rather than a convention.
And the debit test now waits for BOTH debits. With a synchronous surface debit
landing first, waiting for one was satisfied without the background meter ever
running — the fixture would have made the property it exists for unobservable.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
velocity applies its cap PER SHARD — MaxKeys/shards+1, which is FIVE keys against
a published census ceiling of 320 — and subjects hash unevenly, so a shard fills
long before the total does and the store drops that shard's least-recently-used
key. The census is a flat count and saw none of it. Measured:
subjects=200 storeKeys=198 censusLen=200 forgotten=0 SATURATED=false
subjects=320 storeKeys=290 censusLen=320 forgotten=0 SATURATED=true
At 200 subjects two of that organisation's own subjects are gone and the model
reports itself healthy; each one reads as "has done nothing", scores as
unremarkable, and raises nothing. At 320 the state is right by luck and the COUNT
an operator acts on still says zero while thirty subjects are gone. The published
8 MiB budget buys 320 subjects and the bound that binds starts biting near 200 —
a bound stated in one dimension and enforced in another.
reconcile already read the store's own key count and threw the comparison away.
It now measures the shortfall against the census under ONE lock, so the two
numbers describe one moment, and keeps it as a HIGH-WATER MARK: a dropped subject
is re-admitted the moment it is active again, which closes the shortfall but does
not un-blind the window in which that subject read as inactive. A gauge would
report that a control which switched itself off never did.
ALSO: the resident bound had no guard at all. `evict` could be made never to trip
and the whole suite stayed green — the bound fully disarmed, residents growing
without limit, which is 64 x (8 MiB of rings + its model) against a 9 GiB
GOMEMLIMIT on a ONE-replica Recreate deployment. A bigger constant was never the
fix; the operating point is four properties at once, and each is a way to be
wrong: SERVED past the bound (a refusal is a cliff, not an operating point),
BOUNDED at every step, LOUD on the probe (eviction is lossless, so the count is
the only sign), LOSSLESS (otherwise one tenant's arrival costs another its model,
which is the cross-tenant defect wearing a capacity hat).
Mutation-proved: scripts/mutate.py "strain"/"bound" — seven rows. Dropping the
store's half of Saturated reports "the store holds 142 of this organisation's 144
subjects and strain reports saturated=false". Refusing past the bound reports
"organisation 64 of 72 was refused a residency". Dropping the evicted tenant's
save reports "an evicted organisation came back having learned 0 of 120".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TWO DEFECTS, both "a bound that is not on the dimension that matters".
THE ROLLOUT LOST EVERY TENANT'S MODEL. close() was `p.stop(); p.wg.Wait()` with
every save BEHIND that wait, and the wait had no bound at all — while Shutdown
was handed the shutdown window as a context and threw it away. The process gets
30 seconds (serve.go) inside a 60-second grace period, and ONE search finishing
after cancellation writes its result to a shelf whose durable Sync is bounded at
durableOpTimeout — thirty seconds, the whole window, on its own. So the wait
outlived the window, the pod was killed, and not one resident model had been
written down. Every tenant returned to warming, from one tenant's search, once
per deploy — and a warming model REFUSES to score, which reads as clean to
anything that does not check the refusal.
The saves now run unconditionally and never behind the drain; background work
gets whatever is left of the caller's own window, bounded by it and by
drainBudget; and a drain that did not finish is ErrDrainIncomplete — a named
state joined into the error, with the count of models written down anyway, rather
than a silence.
THE SEARCH BOUND BOUNDED NOTHING EXPENSIVE. begin() checked "is a search already
running for this organisation?" and set the flag that answers it two warehouse
operations later — check-then-act with the entire cost of a search setup in the
window. Measured: 16 concurrent callers for ONE organisation were all accepted,
all rolled that tenant's source planes, and all read its whole history, against
the one warehouse every tenant shares; 16 background grids then raced for one
shelf row. The slot is now claimed atomically with the check that grants it,
before any of that work, and released on every path that does not reach the run.
claim/settle/unclaim are one fact in one place — the background goroutine's
hand-rolled delete is gone with them.
Mutation-proved: scripts/mutate.py "rollout"/"search" — six rows. The original
close() ordering does not merely fail, it HANGS to the test timeout, which is the
production SIGKILL. Making the saves conditional on a clean drain reports "tenant
A came back having learned 0 of 300". Restoring the check-then-act reports "16 of
16 concurrent searches were accepted".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
OrgStore.CloseAll closed every handle and then installed fresh empty maps, which
made "closed" indistinguishable from "nothing opened yet". Every one of the
fifteen callers is a Shutdown path, so a request still in flight during a rollout
reached For() after it and opened the file again — on a durable deployment
re-hydrating it and re-claiming the fence lease the SUCCESSOR pod was claiming at
that moment. Two live writers for one org, through a handle nothing thought was
reachable.
Closing is now terminal: the flag is set before the maps are cleared, so there is
no window in which the store is empty and still openable, and every door that can
open a file answers ErrStoreClosed. An arrival after shutdown is a fact worth
surfacing, so it is an error rather than a silent no-op.
The risk plane reaches this path by construction — its own close() empties the
resident map, so an in-flight score rebuilds a residency and asks for the shelf.
Mutation-proved: scripts/mutate.py "orgstore" — three rows, one per half of the
mechanism (the flag, the For guard, the Each guard). Each goes red on
TestOrgStoreCloseAllIsTerminal at a distinct assertion.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ownVolume() is what one org may occupy on the volume every org's SQLite file
shares: maxOverrides x len(Catalog()) x row, where row = maxKey + maxNote + 128.
The key and the note were bounded at the wire door. The third term was not: the
writer is actor()'s reading of the X-User-Id the request carries, so it was a
caller-sized value stored on up to 10,000 rows in every set the catalog
publishes. A count over caller-sized values is not a byte bound, and the stated
120 MiB ceiling was a figure nothing held to.
maxActor = 128 (three times the UUID IAM mints), enforced at the store door
rather than the wire op — the row is what the budget is about, so bounding it
where a row is written covers every path that reaches the store and not only the
one a reviewer read. row now DERIVES from it, so raising the bound moves the
ceiling instead of silently detaching it.
Refused, never shortened: the writer is who an adverse action is attributed to,
and a truncated identity names someone who does not exist. errActor separates
the two refusals put() can produce — the row cap is a 409 (the org already holds
what it holds), an over-long writer is a 400 (a value that arrived on this call).
TestTheWriterOnARowIsBoundedLikeEveryOtherTerm rides the wire, because the
defect was in what the wire hands the store: a unit test on put() would have
passed against a handler that never bounded the header. call() is now callAs()
with the default principal, so there is one request builder and not two.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/ is where mounted, billable subsystems live: 116 of its 139 directories
declare `func Mount(`. apps/money declares none, serves zero routes, and is 216
non-test lines that already wrap github.com/hanzoai/money (imported as `hz`) to
pin ONE unit — creditUSD, dollars at 18 decimals. That is a library, and filing
it under apps/ said it was a product.
Pure path move: apps/money -> money. No API change, 44 import lines rewritten,
0 residual references. Full tree builds (`go build -tags sqlite_math_functions
./...` rc=0).
Not deleted and not folded upstream: hanzoai/money is the general library
(multi-currency, rates, min/max/sum) and this is the platform's credit-asset
facade over it. creditUSD is Hanzo-specific and does not belong in a public
money library. One library, one facade, each in the right place.
Unrelated, found while verifying: `go build ./...` fails on
hanzoai/base@v1.5.11/core/sqlite_math_required.go (undefined
cgoBuildNeedsSQLiteMathFunctions) on a CLEAN tree too — it needs
-tags sqlite_math_functions. The default build command does not work in this
repo and that is worth fixing separately.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ResourceMeter.Gate is the one pre-create gate all 23 priced surfaces call, and
handed an empty org it asked the money plane to price a spend for a nameless
subject. The money plane then answered a question about IDENTITY in the
vocabulary of MONEY, in two measured shapes:
co-resident ledger — metering refuses an empty org fail-closed, which is not a
4xx, so the wire's fallback renders 503 "Billing temporarily unavailable":
the caller is told the biller is broken.
peer ledger — gatePeer ships AuthorizeIn{Subject:""}, commerce's own
validate:"required" rejects it, and because that refusal IS a 4xx the wire
preserves it verbatim: 400 `field "subject" is required` — a field that
appears in no published request schema on any of these surfaces, so no
caller can ever satisfy it.
apps/risk already carried this guard at its own door; the same disagreement is
reachable wherever a handler's tenant check and principal.Ledger differ.
provisioning is the live instance: create() admits an admin with no org, and an
org over MaxOrgLen, through tenant() — and Ledger answers "" for both. Its seven
routes serve on api.hanzo.ai today.
An empty org is now refused as ErrNoLedger before either branch runs, and denial
— the one decision both renderings read — renders it as the tenant gate's own
403 rather than as a fault of the biller. This changes no allow/deny outcome on
a named caller: an empty org already failed on both branches. It does close a
worse case than the wrong status, proven by the mutation below: under fail-OPEN
with the biller down, an unidentified caller was allowed, so the surface was
free rather than gated.
Mutation-proven via scripts/mutate.py:
gate: ask the money plane to price a spend for a nameless subject KILLED
gate: render an identity refusal as a fault of the biller KILLED
risk: widen the empty-ledger guard until it refuses every caller KILLED
Two rows are deliberately absent and say so in place: "guard below fail-open"
is a semantic no-op (fail-open lives inside Authorize/gatePeer, so no placement
in Gate can follow it), and "delete ops.gate's guard" now SURVIVES — with Gate
refusing, the risk-level guard is no longer load-bearing for the status or the
sentence, only for the envelope.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
plugin/<app>/mcp.json stopped being a generated artifact when the door started
asking each subsystem for its tools at the moment it is asked (package fleet,
mk/plugin.mk describe). The committed copies were the exact hazard that change
removed: plugin/o11y/mcp.json held 12 tools while the o11y binary served 365,
and nothing compared them.
plugin/dataset/mcp.json was the last one left — no go:embed names it, no Go
source opens it, and every remaining mention in the tree is prose recording that
the mechanism was retired. Removing it leaves 0 of 124 app dirs carrying one, so
"the tool catalogue is not an artifact" is a property of the tree rather than a
sentence in a comment.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The last attempt at this served a plane op from deps.AI, which in the `ai`
process is the HTTP gateway to api.hanzo.ai — so it added a hop instead of
removing one, and was reverted. The premise was right and the mechanism was
wrong. zip states the mechanism itself:
"Because it is an ordinary route on the app, it rides EVERY transport the
app Listens on with no extra wiring — ZAP over a unix socket is simply the
address the caller dialed."
`ai` already answers its whole /v1 surface on $ZIP_RUNTIME_DIR/ai.sock. Nothing
in hanzoai/ai has to change and no second inference API has to exist: a sibling
speaks the SAME OpenAI-compatible wire to the SAME routes, over the socket.
before sibling --HTTPS--> Cloudflare --> ingress --> api.hanzo.ai --> ai
after sibling --unix----> ai
AIHTTPOn / AIHTTPM2MOn state the TRANSPORT separately from the address, so
there is still ONE inference client; only the route it travels differs. The
M2M token exchange keeps the default transport — IAM is a different peer and
naming it is a separate question, still open.
aiRoute is the one decision both pickers share, so completions and embeddings
cannot disagree about where the peer is. It is answered by WHAT THE PROCESS IS:
!Enabled(ai) means this process does not carry the app, which is exactly when
`ai` is a sibling. The ai process and the host keep the configured address —
routing inference back through the picker there would be a self-call, and
TestTheAIProcessDoesNotDialItself pins it.
The socket transport WAKES the peer through the same reach() every plane call
uses, so a cold lazy app is "not started yet" rather than "not deployed here".
CLOUD_AI_ZAP_ADDR is deleted with its field: it made a deployment state where
its own code lives, which is the thing being removed. CLOUD_AI_BASE_URL now
only describes inference that is genuinely elsewhere — and on the Hanzo
deployment it is no longer consulted, because that pod carries `ai`.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The app was renamed to /v1/risk/reference — openapi.Product reads the product
off the first /v1 segment, and these six operations are the risk product's —
but the committed projection and the woven document were generated before the
rename and still published /v1/ml/reference. manifest's
TestEveryServedPathReachesTheAppThatServesIt is the gate that caught it: four
paths the fleet published and routed to the /v1 catch-all instead.
Regenerated plugin/reference/openapi.json from the app's own live router and
re-wove openapi.yaml from the subsets. The delta is exactly the four reference
paths; no neighbour moved.
plugin/reference/mcp.json goes with it. The tool catalogue stopped being a
generated artifact when the door started asking each subsystem for its tools at
the moment it is asked (package fleet) — 122 of 124 apps ship no such file, and
a stale one beside a regenerated projection is a second answer to a question
that has one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The coordinates are the part worth proving, and ai's own tests cannot reach them.
ai declares the store structurally (object.SecretStore) precisely so it needs no
KMS dependency — which means its tests can only use a fake, and a fake proves the
logic while proving nothing about where the bytes land.
A bare ref resolves to path "/", which fileOrg treats as the FACADE and lands in
the deployment's system partition. The REST surface folds the caller's org and
lands the same name at /orgs/{org}. Those are different databases. Write through
one door and read through the other and the secret is simply not there: no error,
no warning, just a key the gateway cannot find. This asserts the admin seal and
the completion-path resolve use one door.
Second test pins the migration's direction: env serves until a key moves into KMS,
and KMS wins after. Without it "migrate the key" could be a no-op that looks
complete.
Verified on Linux, where libsqlcipher is linked and the store actually opens.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The commit claimed a sibling would stop reaching `ai` through Cloudflare. It
does not. In the `ai` process cfg.Enabled("ai") is true, so deps.AI is the HTTP
gateway pointed at https://api.hanzo.ai/v1 — the pod's own public address — and
the plane handler I added served inference THROUGH it. The path became
sibling --UDS--> ai process --HTTPS--> api.hanzo.ai --> airouters
where before it was one HTTPS call. Strictly worse, and the opposite of what
the message said.
The premise was right and the wiring was wrong: `ai` exposes its inference as
HTTP ROUTES (airouters), with no in-process ChatCompletion to call, so serving
the plane op from deps.AI just re-entered the transport. Closing this for real
means an in-process entry point in hanzoai/ai that the plane op can call
without a socket — a change in that module, not a rewiring in this one.
Reverted whole rather than patched so main does not carry a half-measure that
reads, from the commit log, as if the round trip were gone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`ai` is a plugin of this same binary running as its own process, and the only
way a sibling could get a completion was its PUBLIC address. So the pod left
through Cloudflare, performed a client_credentials exchange to authenticate to
its own deployment, and came back — to reach code one socket away. Its own
startup line said so:
deps.AI (completions) -> HTTP gateway (IAM M2M)
base_url https://api.hanzo.ai/v1 <- the pod's OWN public address
token_url http://iam.hanzo.svc/... <- a token to call itself
Every part of that was a consequence of addressing a peer by URL. plane.AIChat
and plane.AIEmbed are addressed by APP NAME — zip.SocketPath resolves it and
reach() starts the app if it is not listening — which is how the meter already
debits commerce and how the gate already reads the ledger. There is no address
to configure, no credential to mint, and no second answer to where `ai` is.
Which client a process gets is decided by WHAT IT IS, not by an env:
!cfg.Enabled("ai") means this process does not carry the app, which is exactly
when `ai` is a sibling. The process that IS `ai` (and the host, which carries
everything) falls through to the real transport — asking the plane there would
be this process calling itself, and the test pins that.
CLOUD_AI_ZAP_ADDR is deleted with its field: it was the previous attempt at
this, and it still made a deployment state where its own code lives. The
gateway stays for inference that is genuinely REMOTE — a different fact, not a
second way to reach the same thing.
Billing is deliberately not on the servant side: the caller reserves before it
asks and settles on the reported usage, so pricing it again there would bill
one completion twice. MaxTokens rides the request so the ceiling the caller
reserved against is the ceiling the servant honors.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The upload reached S3 and the profile did not move: 502 "photo stored but the
profile could not be updated", and the log said why — `iam non-envelope
response (400)` for id hanzo/2d4d67ab-….
IAM parses a user id as `<owner>/<name>` through GetOwnerAndNameFromId, and on
the direct-Bearer path X-User-Id is a UUID subject, so `<owner>/<uuid>` names no
user. keyID() is the composite that resolves — the key ops (mint/revoke) already
use it for exactly this reason, and the comment explaining it sits in the file I
edited. On the gateway path username == name, so the two are the same value and
nothing changes there.
Found by uploading a real PNG to production rather than by reading the route
back. The honest failure is what made it findable in one step: the handler
reported that the bytes had landed and the record had not, instead of a 500 or a
success the user could not see.
The branch carried a second one: a root `cloud.Qualify` returning `type Tenant
string`. Two spellings of the key the dataset plane writes, the risk plane reads
and this plane joins on — and the two did not agree.
It did not canonicalise the brand. A deployment started with CLOUD_BRAND=Hanzo
filed `Hanzo/acme` while apps/dataset asked for `hanzo/acme`. Silent,
permanent, no error: one business, two key spaces.
It did not ask the registry. A brand nothing vouches for minted a key the
writers of these surfaces refuse to produce.
`Tenant` is a STRING type, so it can be written as a literal in any package and
decoded straight out of a request body. `tenant.Key` is a struct with one
unexported field: no literal, no JSON, no caller-asserted tenancy. Only
tenant.Mint and tenant.Of produce one.
tenant.Of additionally compares the token's verified issuer brand against the
deployment's — a token minted by lux.id cannot become `hanzo/acme`.
So the root file goes and apps/label takes tenant.Key/tenant.Of throughout. The
door shrinks: tenantOf no longer re-checks the shape it just minted, because the
mint is the check.
Mutation-tested at the ONE place the property now lives: drop the case fold in
tenant.canon and TestTheMintAgreesWithTheWriterOfTheSourceTable reports
`Mint("Hanzo","acme") = "Hanzo/acme"` against a writer that writes `hanzo/acme`,
RED; restored, green. 67 label tests pass on the ported key.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Eight of the ten declared /v1/risk paths answered a 400 that named a field the
caller cannot send. Measured on api.hanzo.ai with each operation's own declared
body: score, learn, features, state, state/appetite, state/snapshot,
state/restore and search/{id} all returned
{"error":{"code":"","message":"field \"subject\" is required"}}
`subject` is plane.AuthorizeIn's field, not any risk operation's. It appears in
no published request schema on this surface, so no caller could ever satisfy it
— a door that answers and cannot be opened.
ops.gate re-derived the caller's ledger with principal.Ledger, which answers ""
for exactly the requests the tenant gate refuses (it composes the same Validated
check), and then handed that empty ledger to the money plane. Asked to price a
spend for a nameless subject, the money plane answered in the vocabulary of
money about a question of identity, in two shapes:
co-resident ledger — metering refuses an empty org fail-closed, which is not a
4xx, so the money wire's fallback renders 503 "Billing temporarily
unavailable": the caller is told the biller is broken.
peer ledger (what deploys) — the gate ships AuthorizeIn{Subject:""} over the
internal plane, commerce's own `validate:"required"` rejects it, and because
that refusal IS a 4xx the money wire preserves it verbatim as the 400 above.
ops.search never had the defect, for the one reason that it reaches ops.admit
before it prices anything. This makes that ordering general: an empty ledger is
an identity refusal, answered with the tenant gate's own sentence from the one
function that owns it, before the money plane is asked anything.
The route itself was never missing. GET /v1/risk/score is a 405 here — the route
is declared, the verb is not — and the report of a 404 traces to the deployed
edge flattening that to plain-text "not found", byte-identical to an unregistered
path. Named in the report; not reachable from this suite.
TestTypedOpsRefuseAnUnvalidatedPrincipal asserted this exact property and passed
throughout, because it mounts deps with no metering client: with no client the
money gate is a no-op and the op fell through to the honest 403 the test wanted.
The new tests mount through mountBilled, the only fixture in which the ordering
is observable. A companion assertion counting calls to the ledger fake was
written, found unfalsifiable — an empty org is refused inside AuthorizeVerdict
before any HTTP request, so the counter reads zero either way — and deleted
rather than shipped; the reason is recorded where it would have gone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`oversize` is on the wire, so it is in the document: the app's own subset, the
prose map and the woven openapi.yaml every SDK repo pulls. Generated by
`make -C apps/dataset describe` + `make -f mk/fleet.mk openapi-weave OUT=openapi.yaml`
— no hand edit. No new operation, so the floor is unmoved.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A COUNT OVER CALLER-SIZED VALUES IS NOT A BOUND. maxRows capped how many rows a
materialisation holds and `page` capped how many an export returns, and neither
bounded a single byte. A row's coordinates are float64s this plane counts, but its
SUBJECT is a string this plane does not write: the rollup lifts it from
`distinct_id`, `session_id` and `user_id`, which arrive on /v1/event from the
caller. distinct_id is capped at 256 bytes on the anonymous lane and REPLACED by
the token's own subject on the signed one — but session_id, which the `session`
rollup files as a subject verbatim, is capped nowhere. So "200k rows of ten
float64s plus a subject key is tens of megabytes" and "eight jobs is a few hundred
megabytes" were arithmetic over an unknown, and one tenant's traffic decided the
real number for the whole process.
maxSubjectBytes bounds the one caller-sized value a row carries, at 256 — what the
identified lane already states for a subject, for the reason that carries over
unchanged: a minted id is a uuid, the value is KEYED, and something longer is not
an id. With it, count times max IS the byte bound, and every byte figure the
package states is now DERIVED from it rather than written down beside it:
maxRowBytes, maxResidentBytes, maxProcessBytes, maxPageBytes. The stale prose
claims are gone rather than corrected — that was the other spelling.
ONE ENFORCEMENT POINT. `representable` is the predicate and the only place it is
written. It is applied on the way OUT of the source, which is the way IN to this
process and to the rows table, so no read path needs a second check: an export
page is bounded because every row it can return already came through it.
AND THE DEGRADATION IS NAMED. A bound that quietly drops rows is worse than no
bound — the dataset that comes back looks complete, and a model fitted on it is
blind to a population nobody can see was missing. The census measures both halves
on ONE pass (conditional aggregates, not a filter), the excluded subject count
rides on the version, the manifest, the lineage and the wire, and it is in the
source fingerprint, so `reproducible` is measured over it too: a window that grew
a subject too large to carry is a window that MOVED, and lineage now says so.
census and facts cannot disagree about the population, because the predicate is
one expression used by both — measured against, then read with. Two spellings
would sample the share from rows the read never returned.
Three gates, each mutation-tested (defect reintroduced, named test RED, reverted,
green):
TestEveryReadOfTheSourceIsBoundedInBytes the package's own AST — a
function that reads the source without the bound fails, the same shape as the
admission gate beside it, because it is the same failure: a new op skipping a
property nobody checks. Carries the same anti-vacuity floor.
TestAnUnrepresentableSubjectIsExcludedAndCounted end to end: the bound binds,
the representable rows all survive, the count is on the version AND the
lineage, and it is falsifiable.
TestTheByteBoundIsDerivedFromTheValueBound the arithmetic, so no byte
figure can be asserted independently again.
The fake store EVALUATES the bound rather than ignoring it — a fixture that
ignored it would let every test above pass with the predicate deleted. It also now
binds `?` positionally across the WHOLE statement, as a driver does, which is what
the census's conditional aggregates require.
TestTheTenantLeadsEveryPredicate was reading args[0] and calling it the tenant.
That was a positional coincidence, true only while no statement carried
placeholders before its WHERE; it now reads the argument bound to the leading
`org = ?` itself, which is the property it always meant.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
(cherry picked from commit b851446a2a)
CLOUD_ENABLE / --enable named the subsystems to mount. The binary already
knows: manifest.Apps is the host's set, and a plugin IS its app (Listen sets
cfg.Enable from what plugin/<app>/main.go was built as — that stays; it is
the binary stating itself, not a deployment restating it).
A second source of truth can only add disagreement, and it did. Both devnet
outages on 2026-08-02 were this list: one named "plans", an app the manifest
does not have, and one omitted "kms", the credential broker every other child
pulls its data-plane key from — so every child failed closed at its first
store open. Production has never set it.
Removing the input removes the failure class and five branches with it: the
broker precondition, the unknown-name guard, and three `on != nil` selections
that only existed to police a list nobody should have been writing. Empty
already meant all, which is what production runs.
Values move in the same change — devnet and testnet drop the list, so no
deployment is left naming a variable the binary no longer reads. Docs and
cloud-probe.sh follow; the probe had been STRIPPING the variable, so it
already agreed.
TestTheAppSetIsNotNamedTwice replaces TestAnAllowlistWithoutTheBrokerIsRefused
— that test policed the list, and the guard is now that no source states the
set again.
Also fixes a pre-existing red on main, unrelated to this: apps/catalog's
cloud.Request call site was never added to allowedRequestUses, so the escape
hatch ratchet failed on clean origin/main. It is a legitimate use (the
published corpus reads as PublicOrg for everyone, so the tenant is re-pointed
while the caller's authority travels whole) and is now recorded with that
reason.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An hourly goroutine in authors deposited credit with nobody in the loop, and a
GET on three surfaces accrued-and-paid on read. Both are gone, along with the
capability that made them one line each.
- apps/authors/scheduler.go, sweepAndPayout, autoPayoutAuthor — the unattended
hourly accrue+pay loop, default ON, no env var, no route, no human.
- the lazy sweep on GET /v1/authors, /v1/affiliates, /v1/affiliates/me and
/me/earnings. Reads read; the admin POST sweep still accrues.
- payout settlement in both programs. A payout RECORDS what is owed, for every
method including credits; a human settles it. Accrual — the product — stays.
- treasury.Reserve/Credit returned backed=true when unmounted, and `mounted` is
a package global, so in one-binary-per-app it was ALWAYS nil in callers: every
"reserve-backed" payout was an unbacked mint that logged itself as reserved.
With settlement gone it has no callers, so it is deleted rather than fixed.
- POST /v1/admin/credits — a second admin mint with no cap and no positivity
check, whose audit did not fail closed. core.ApplyGrant is the one door: it
caps, rejects non-positive amounts, checks the org, and refuses without a
durable audit store. The relay and its wire client are deleted.
- payout.Client.Deposit, the ONE money-in primitive all three programs shared,
and the deposit method on each program's seam. The seams now carry a single
read, matching referrals: reviving a mint has to start by re-declaring the
capability, in front of a test that says no.
- the published POST /finance/starter op, advertising a grant deleted in
41b23f12.
openapi.yaml, plugin/admin/openapi.json and openapi/floor.json drop the deleted
route in this commit, so the reduction is reviewed next to its reason.
Tests assert the guarantee rather than the old behaviour: a GET grants nothing
and the ledger receives zero deposits, proven at the wire against a commerce stub
that fails on any write.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/ws/query_progress is in openapi.yaml and in plugin/o11y/openapi.json, so it is a
method in every generated SDK — and no manifest row named it, so the composed
binary routed it nowhere. Its own prose says the address "was unreachable from the
composed binary until the route table named it, because the old wildcard covered
only the o11y prefix"; the wildcard went and the row never grew the prefix back.
TestEveryServedPathReachesTheAppThatServesIt has been RED on main for this one
path. It reads 1673-of-1684 now against 1672-of-1684 before, and the recorded
ledger is unchanged — nothing else changed hands.
Mutation-tested: prefix removed, the named test reports UNREACHABLE and fails;
restored, green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both avatar routes answered 404 in production. The routes were registered and
the handlers were fine; the HOST routes by manifest prefix, and account's row
did not name /v1/avatar — so `ai`, which owns the /v1 remainder, won it.
manifest's own router test said so in as many words on the tree that shipped:
UNREACHABLE: account /v1/avatar -> ai
UNREACHABLE: account /v1/avatar/{org}/{user}/{digest} -> ai
I did not run it. Targeted package tests passed and I shipped on those, which
is how a route can be complete, tested, deployed and unreachable at once.
With the prefix named, both paths mount to account (zip mounting … addr=account)
and the ledger drops from three unreachable paths to one — o11y
/ws/query_progress, which is not this change's and stays as it was.
iamurl.go opens by saying the IAM-address policy "existed as three inlined
copies ... so there is exactly one now". There were four, and they had grown
three different fallbacks:
iamurl.go IAM_URL, else the public issuer (the policy)
auth_apikey.go IAM_URL, else IAM_INTERNAL_URL, else NOTHING
apps/account/iam.go IAM_URL, else a hardcoded cluster address
apps/platform re-read IAM_URL to answer a different question
A copy of a policy does not disagree until one is edited. These had already
diverged: auth_apikey read a fifth env name no cloud deployment sets
(IAM_INTERNAL_URL is on admin-guard and chat only), so a single-process deploy
that knew only its issuer resolved "" and API-key auth stayed silently
unconfigured; apps/account reached a cluster address that a non-cluster deploy
does not have.
iamurl.go now answers both questions the estate actually asks, over ONE env
read: IAMBase() for the address, IAMExternal() for whether a separate IAM is
named. They are separate because conflating them IS the bug — a deployment
with only a public issuer resolves a real address while naming no external
IAM, and a caller inferring one from the other reaches for a store that is
not there. TestIAMAddressHasOneReader walks the tree and fails on a second
reader, so this cannot drift back.
Also: pickCompletionsClient preferred the PUBLIC gateway over the in-cluster
peer. `ai` is a plugin of this same binary running as its own process, so that
ordering sent the pod out through Cloudflare and back, minting an OAuth token
to authenticate to its own deployment, to reach code one socket away. The peer
is now first. Ordered, not gated: nothing sets CLOUD_AI_ZAP_ADDR today, so
this is inert until an address is named and the gateway keeps answering.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
main was pinned to hanzoai/ai v1.832.18, which is RETRACTED. That version seals
the secret MASK as a provider key: the admin API returns "***" for a stored
secret, so saving a provider form without touching the key field seals the literal
"***" into KMS under the provider's own name — and because ai resolves KMS-first,
the store then answers "***" for every read, outranking the env var that was
serving the real key. The provider stops authenticating while its row still looks
correct. v1.832.19 carries the guard.
.19 also brings: secrets resolved from the EMBEDDED in-process KMS (this binary's
own apps/kms) instead of over HTTP to the standalone deployment, which had never
worked — 404 on the path it used, 401 on the correct one; /v1/provider-flags
renamed to /v1/models/providers and derived from the served catalog, so the
provider set and the model list cannot disagree; and a model family is now
controlled from admin.hanzo.ai rather than only from deployment env.
apps/referrals did not compile on main: payout.Deposit gained a required `ref`
(commerce guards on it so a retried payout credits AT MOST ONCE) and this caller
was not updated with it. A referral pays TWO wallets, so the referral id alone
would name both and commerce would dedupe the second against the first — the
referee's bonus would silently never land. bonusRef(id, side) makes each credit
its own event while staying stable across retries.
The published spec is regenerated: /v1/provider-flags is gone from openapi.yaml
and plugin/ai/openapi.json (the .18 pin never regenerated it, so the drift gate
was red), and the retired provider-flags product is dropped from the floor.
Verified on Linux (macOS has no tmpfs for the pure-Go SQLCipher codec, so the
store-backed suites cannot run there): apps/referrals, apps/ai, apps/kms and
openapi all green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The plane fix shipped and /v1/catalog answered 500 "index: no org on the call"
— the public browse reaching the index as nobody.
cloud.For() states a caller, but zip's forwardIdentity prefers an INBOUND
request over a stated one ("an inbound request always wins over what it said"),
and a typed handler's ctx carries the in-flight request. So For() was silently
overridden and the call went out as whoever asked. For a signed-out visitor
that is nobody, and the index refuses a call with no org — correctly.
cloud.As() is the form for this: it re-points the tenant on a context with no
request behind it, which is the one place zip reads what we stated. The caller's
authority still travels whole; only the tenant moves — which is the point, since
the published corpus is read as PublicOrg by everyone, signed in or not.
Caught by asking production rather than by trusting the deploy: the previous
commit was verified live and answered 500, not rows.
payout.Deposit sent no idempotency key at all, so every retry of an author
royalty or affiliate commission credited the wallet again. Commerce now
requires the key (it is the reference to the money event), and both callers
already hold the right value: the payout row's own id.
- Deposit takes ref and sends it as X-Idempotency-Key, via a post helper
kept beside do because only a WRITE carries a reference — a read has no
event to name.
- ErrNoRef refuses an unnamed deposit HERE rather than at commerce, so the
failure names the caller's missing value instead of arriving as a status,
and a new payout path cannot quietly ship without one.
- authors and affiliates pass "payout:"+payoutID through their commerce
seams.
Tests: an unnamed payout never reaches commerce (the fake server records
that it was not called), and the ref travels as the key commerce guards on.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/referrals ran a "lazy qualify sweep" before listing, and that sweep
reached commerce.Deposit. Loading your own referrals page minted platform
credit — $10 to you, $5 to your referee — on a read. The middleware in front of
the prefix gated writes by asking `method != POST`, so it waved the GET through;
the two safeties behind it did not hold either. treasury.Reserve returns
backed=true when treasury is unmounted, so the "backed by the reserve fund"
claim was a passthrough in any deploy without it, and the at-most-once latch
bounds the mint per referral, not in total.
The precedent is set twice on main: 41b23f12 deleted the $5 starter grant and
45b3b5cf deleted finance.Deposit along with its imports. Same here. The deposit
path is DELETED, not disabled — a flag-disabled money mint is one flag from an
enabled one.
What goes: the two bonus constants, the ledger currency + grant:referral tag,
grant(), the treasury reservation and its import, LatchCredit, SetTxns, the
grant/txn/credited_at columns, the `credited` status, and every cents field on
the wire. A field that can only ever report zero is a lie about what the surface
does, so creditsEarnedCents and the two bonus amounts go rather than freeze at 0.
The commerce seam keeps ONE method, spendCents — it is a question, not an
instruction — and TestCommerceSeamIsReadOnly fails if it grows a write.
What stays, because it is the actual product: who referred whom, the stable
code, the share link, and qualification. Qualification is a WRITE, so it now
happens only on POST /v1/admin/referrals/sweep. A GET reports; it does not
transition.
The gate is fixed at the defect class, not the instance: it waves through GET
and HEAD by name and requires an org for every other verb, including ones this
package does not serve. "Not POST" meaning "harmless" is the reasoning that let
a read reach a deposit.
What a qualified referral is WORTH is not this package's question. That is an
affiliate payable in hanzoai/commerce, settled by wire or to a connected wallet
— never minted as platform credit.
Tests: a GET in the exact state that used to pay leaves the row byte-identical
and touches the money plane zero times; the real payout client against a stub
commerce that fails on any write proves zero deposits at the wire.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The reservation landed with a 32768 constant standing in for "the most a
completion can be". That number is wrong per model by construction: it caps a
1M-context model at whatever was typed, and it is one release out of date the
moment a model ships. This estate has already paid for that shape twice —
ai/model's name-matching table gave deepseek-v4-pro 16384 and 402'd every long
prompt, and glm-5.2 dead-ended /compact on a stale 16K fallback. Both were
fixed by moving the number into models.yaml. This does the same for billing.
- completionCeiling(model) resolves through SetCompletionCeiling, installed
in apps/ai from ModelConfig.MaxOutput (ai v1.832.18), falling back to the
model's context window — still a true bound, since prompt + completion can
never exceed it. The constant survives only as a FLOOR for a model the
catalog does not declare, and is documented as never a per-model answer.
- the seam exists because hanzoai/ai/controllers imports hanzoai/cloud, so
the catalog is a CYCLE from cloud's root, not merely weight. apps/ai links
both, which is where every other cross-module hook is installed.
- atMost no longer writes the ceiling onto the request. What we reserve is a
billing fact; req.MaxTokens is the CALLER's, and forwarding a limit they
never asked for silently truncates their answer. The reservation is sound
regardless — a model cannot exceed its own max output — so the bound holds
whether or not we restate it on the wire. Only a caller-set MaxTokens now
reaches the provider.
TestCeilingComesFromTheModelNotAConstant pins all four: a 1M model reserves
>=1M, an undeclared model takes the floor, a caller's MaxTokens wins, and
atMost never mutates it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzo.app's Community page rendered "ERROR: CATALOG: 503" under an otherwise
fully-drawn page. api.hanzo.ai/v1/catalog answered
{"status":503,"error":"catalog: index not mounted"} on EVERY request, and
nothing was down.
browse() guarded on index.Ready(), and Ready() reports whether the index is
mounted IN THIS BINARY — its own doc says so. That was true and harmless while
cloud was one fused process. It stopped being harmless when the fleet became
one process per app: `index` and `catalog` are two manifest rows, index.Mount
is only ever called by plugin/index, so inside the catalog process that global
is nil and always will be. An in-process dependency survived a process split,
and the only symptom was a status code.
The index is now ASKED, on the internal plane, exactly as visor asks tasks for
its activities and as commerce serves its ledger. Not opened: the index store is
one encrypted SQLite with a single writer (MaxOpenConns(1), keyed through cek),
so a second process opening the same file to read it is the collision, not the
cure.
Both legs are kept and neither is dead: a fused binary that mounted both apps
still reads in-process, because a wire hop to something in the same address
space is pointless; the deployed fleet takes the plane. The write side does not
move — Reconcile stays in the process that owns the file.
apps/search carries the same dependency and is mounted by no plugin at all, so
it is unreachable rather than broken. Left alone here; naming it so the next
reader does not mistake this fix for having covered it.
Moving imageType into internal/magic took the only use of "bytes" in files.go
with it and left the import behind — which does not compile, and it is what
turned the release train red twice (the image job exited 1 at ~6m18s and no
cloud image published, while v1.801.399 had built fine from the commit before).
It was invisible locally for a reason worth writing down: `go build ./apps/team`
on macOS fails first on hanzoai/base's `cgo && !sqlite_math_functions` guard, so
the package was never type-checked and my error hid behind someone else's. The
build CI and the Dockerfile use is CGO_ENABLED=0 — under that, the package
compiles and the mistake is immediate.
Verified the way CI does: CGO_ENABLED=0 go build ./... and go vet ./... both
clean. The apps/team test failures that remain on this machine are all one
environmental cause (75 of 75: "cek: no RAM-backed scratch", macOS has no
/dev/shm) and fail at store setup before any assertion.
The drift gate regenerates every app's spec subset FROM SOURCE and fails on any
diff, so adding /v1/avatar without re-running it turned CI red — the release
train's image job exited 1 after 6m29s and no cloud image was published.
Regenerated, not hand-edited: zipdoc lifts the doc comments into zipdoc_gen.go,
`describe` projects account's own subset, and the weave proves the fleet spec
equals the sum of the subsets. floor.json moves by exactly what was added —
+2 paths, +2 operations, one new `avatar` product with 2.
zip dials every plugin with zaphttp's DEFAULT transport, whose readTimeout is
30 seconds, and nothing overrode it. That deadline is armed ONCE, just before
the response head is read, and for a STREAMED response it is never re-armed:
the head arrives, SetBodyStream returns, and the whole body must then land
within what is left of the original 30 seconds.
So it is not an idle timeout. It is a hard cap on the TOTAL DURATION of a
response — applied to the plane whose responses are model completions, because
`ai` owns the /v1 remainder (/v1/chat/completions and the rest of the
OpenAI-compatible surface) and runs as a plugin behind this wire.
Every completion longer than 30 seconds was cut:
- streamed: the body stops mid-token at ~30.1s with NO finish_reason, which
every layer downstream reads as a complete answer. Measured against the
live gateway, a long-page request to claude-opus-4.8 returned 7056 bytes,
no </html>, total 30.1s — while the identical request straight to the
upstream streamed 389s and finished properly at 15661 bytes. Reproduced
in-cluster too, so it was never the edge.
- unstreamed: 502 `zaphttp: read response: read unix …: i/o timeout`.
This is why hanzo.app's builder could produce a small landing page but not a
real app: an app is just a longer generation, and the truncation was silent. It
also made a slow-to-first-token model look broken rather than slow — enso
spends ~19s thinking, so most of its budget was gone before it emitted a token.
The ZAP scheme is re-registered before any plugin mounts (the transport is
resolved at dial time from a process-global registry, so a later registration
would leave already-dialed plugins on the default). BOTH halves are supplied: a
Dial-only Transport would silently remove the host's ability to LISTEN on its
own default scheme.
15 minutes, not zero: zero is no deadline at all and a wedged plugin would hold
the connection forever. The honest shape is an idle timeout — time BETWEEN
frames — which this transport does not offer, so it stays a total-duration cap
set past anything a real completion reaches.
There was no way to set one. IAM carries an `avatar` on every user row and the
console renders it, but the only writers were federation (a GitHub avatar_url,
an OIDC `picture` claim) and SCIM — so a user who signed up with a password had
a monogram and no way to replace it, and the console's Profile card answered the
attempt with "Edit in IAM", which links to an IAM that cannot do it either.
Production agreed: /v1/avatar was a 404 while /v1/keys was a 403.
POST /v1/avatar stores the image and records its URL on the caller's IAM row, so
every surface that already reads `avatar` picks it up with no further call.
GET /v1/avatar/:org/:user/:digest serves it.
Storage is deps.VFS — the existing S3 seam (SeaweedFS via clients/s3vfs), which
was chosen for exactly this: "an adapter+crypto is needless complexity for small
avatars". No new store.
Three properties make it safe to serve an uploaded file back from an API origin:
- THE FORMAT IS DECIDED BY THE BYTES. png/jpeg/gif/webp by magic number;
anything else is 415 on the way in and 404 on the way out. A filename and a
part Content-Type are the client's to choose, so neither may decide what
this origin later serves — an SVG is a program, not a picture.
- THE ADDRESS IS THE CONTENT. The key ends in the sha256 of the bytes, so a
new photo is a new URL rather than a stale cache of the old face, and the
read caches for a year. A replaced photo is deliberately NOT deleted: the
old URL is already inside issued tokens and rendered pages, and an object
store costs bytes where a broken face costs a person their profile.
- THE READ TAKES NO CREDENTIALS, AND MUST NOT. Its whole job is to be an <img>
from console.hanzo.ai, a different origin that sends no cookies and cannot
set a header. So the 64 hex of sha256 IS the capability — producible only by
someone who already has the image. The org and user segments are REFUSED
unless they are plain identifiers rather than sanitized: apps/team's seg()
folds "a/b" and "a_b" onto one key, and a fold in a tenancy key is two
identities sharing an address.
internal/magic is the one magic-byte allow-list, now shared with apps/team's
files plane instead of copied. (The three `seg` functions are NOT duplicates —
same name, three different concepts — so they stay where they are.)
The oversize test mounts the app at production's 16 MiB edge body limit. Left at
zip's 4 MiB default the framework refuses the request first and the handler's
own 413 is unreachable and untested — the shape of the bug where studio's 4K
sources could not enqueue.
POST /v1/marketing/promos/:code/redeem was a self-service money mint. The only
gate was tenant(ctx) -- ANY validated principal. `plan` and `seats` came off the
REQUEST BODY unvalidated and were multiplied into a finance.Deposit, so
{"plan":"team","seats":10} deposited 10 x $179.10 = $1,791.00 of real spendable
credit into the caller's own org. Nothing ever collected the charge the discount
was supposedly against; the charge was computed, returned to the caller, and
discarded. `instrument` was the anti-farming key, but instrumentUsed("")
returned false, so OMITTING the field skipped the guard entirely. With the
1,000-org cap, open signup and a personal org per account, that is ~$1.79M of
self-serve credit. The seed shipped active=1 in v1.801.398, which is live.
THE CAMPAIGN WAS NEVER AUTHORIZED. It became live because a schema migration
INSERTed it on every boot -- a business decision arriving as a side effect of a
code change. The seed is deleted, and because deleting an INSERT does nothing
for a database that already ran it, migratePromos now DELETEs the row on every
boot. Redemption history is deliberately KEPT: it is the evidence of what
happened while the campaign was live, and destroying it would destroy the audit
trail exactly when it matters.
CREDIT INTO AN ORG IS AN ADMIN DECISION -- deliberate, through the admin
surface, against an auditable ledger. So the deposit is not fixed, it is GONE,
along with the finance/money/types imports that made it reachable: reviving a
mint here would have to start by reviving an import. This follows 41b23f12,
which deleted the automatic $5 starter grant rather than switching it off, for
the same reason -- a money-mint left disabled is one flag away from enabled.
The subsystem now ships OFF (campaignsLive=false, read from no env var, no
platform switch, no column; TestCampaignsShipOff asserts the shipped value).
The guards are hardened anyway, so a REVIVED campaign cannot resurrect the hole:
- Plan is DERIVED from the org's live ACTIVE/TRIALING paid subscription via
cloud.PlanChecker -- the same seam SpendGate resolves -- and RedeemInput no
longer HAS plan/seats fields. A field that does not exist cannot be trusted
by the next reader. No qualifying subscription means no redemption.
- FAIL CLOSED on an unreadable plan authority. SpendGate deliberately fails
OPEN on this same read, because refusing on an outage 402s every paying
customer at once. Here the asymmetry inverts: an outage must not be able to
manufacture a claim that money is later granted against.
- instrumentUsed("") now returns TRUE. An absent instrument is not evidence of
a fresh card, it is the absence of evidence, and the partial unique index
(WHERE instrument <> '') means the database will not catch it either.
- maxClaimCents bounds every recorded claim, checked under the same lock as
every other guard, and REFUSES rather than clamps -- a silent clamp would
record a wrong figure and hide the bug that produced it.
- Seats is the single-seat floor, never a caller-supplied multiplier.
Redemption.CreditCents/CreditEntryID become DiscountCents: the row records a
claim, not a balance, and a field named for credit that credits nothing is how
the next bug gets written.
Tests drive the shipped routes through the real router, because the hole was a
handler that trusted its input -- a store-level test would have proved the store
fine and missed it. The hardening tests force a campaign live (revivedPromoRoutes)
to answer the question that matters if the decision is ever reversed. Proven: the
exploit body cannot move the recorded figure; a live ledger receives ZERO
deposits; a redemption while closed is refused; nothing is seeded; migrate purges
an already-seeded row while preserving its redemptions.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A prepaid gate reads a SETTLED balance and a completion's cost is not known
until it finishes. Two ways an org spent money it did not have sat between
those facts:
- the gate priced only EstTokens(prompt), so the completion was never
covered. A 1c org asking for a million-token completion was allowed and
settled at $2.00, ending at -$1.99.
- N calls in flight each read the balance before any of them debited, so
each was authorized for the whole of it. Twenty simultaneous callers
against 5c spent 20x the balance.
meteredAI now commits a call's worst case before weighing it and releases
that commitment when the debit reaches the ledger:
- types.ChatRequest gains MaxTokens, and atMost() resolves the ceiling ONTO
the request so the transport forwards the very number the gate priced.
Reserving a ceiling nobody enforces leaves the completion just as unfunded,
only less visibly, so clients/aihttp sends it on both the buffered and the
streamed path.
- commitments tracks per-org committed-but-unsettled cents. commit() returns
the RUNNING TOTAL, which is what the balance must cover, so a second
concurrent caller must clear the first one's commitment. Nothing else has
to know reservations exist.
- the release runs inside the recording goroutine (meterUsage's new posted
hook, threaded through the peer path too). Releasing when the call returns
would let the next gate read a balance that still contains money already
being spent — the very window this closes. It runs on every exit: a hold
that leaks is a paying customer locked out of their own balance.
Holds stay per-pod. apps/finance owns the settled truth and says so
("transient holds are the caller's in-pod concern, never persisted here"),
so this never becomes a second ledger.
MeterUsage keeps its signature — one debit verb, twelve callers untouched.
Tests drive the real meteredAI against a wallet whose balance MOVES; the
existing fixtures answer every gate from a fixed body, which is why neither
gap showed up before. A barrier holds the concurrent callers at the balance
read so the TOCTOU is deterministic rather than a race won by luck.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The starter grant minted $5 into a wallet from middleware, on first credential
contact, with no human in the loop. Credit into an org is an ADMIN decision --
made deliberately, through the admin surface, against an auditable ledger --
so an automatic path that creates money is not a feature to fix but a mechanism
to remove.
DELETED RATHER THAN SWITCHED OFF. A disabled money-mint is one flag away from
an enabled one, and the flag is the kind of thing a later reader flips to
"unblock" something. There is no starter code left to re-enable: the middleware,
its mount in serve.go, the cross-process plane op (finance_starter / StarterIn /
Granted) that let a non-ledger binary ask for it, and their tests are gone.
Note this also removes the shared-signup-org exclusion that lived in the gate.
It was sound anti-abuse for a grant that no longer exists, and keeping half a
mechanism to guard the other half is how dead code survives.
The paywall consequence is deliberate and is NOT taken here: SpendGate stays
behind its kill switch. With no automatic funding, enforcing it 402s every new
account from its first request -- an honest paywall, and a product decision that
deserves its own change rather than arriving as a side effect of this one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
sha-846069c predates the fix. The console SPA is baked into this binary by
//go:embed, so nothing about a console release reaches production until this
line moves — which is exactly what the pin is for, and why it is a sha and not
`:latest`.
sha-9da3984 carries three commits: the token store moved off sessionStorage
(a second tab started signed OUT while the first was still signed in), the
landing CTA starts the sign-in instead of routing to a page that asks again,
and the @hanzo/iam bump to 0.21.6 without which the static export dies on
`ReferenceError: sessionStorage is not defined` while prerendering
/auth/callback.
Verified before pinning: the console CI run for 9da3984 is green and pushed
console-embed:sha-9da3984-amd64. The previous run, on b9d31aa, was RED — so the
image for the storage fix alone never existed, and pinning to it would have
failed the CONSOLE-GATE here rather than shipping anything.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two corrections to what shipped in v1.801.397.
TIER. It was registered on the bare public chain plans uses, but GetTier opens
with middleware.GetOrganization, so it panicked on a nil interface conversion and
answered 500 — the route went from unreachable to reachable-and-broken. A tier is
org state; the org has to be resolved first. Moved onto the billingRead loop,
which supplies the IAM leg its six siblings already rely on.
ROUNDING. The balance now reads (502 -> 200, $149,913.08), but it rounds UP, not
down as the comment claimed. hanzoai/decimal's Rescale rounds half-away-from-zero
(decimal.go:145) and Minor() is a Rescale — measured live, …078983985999994361
served 14991308 cents, a tenth of a cent above the true balance. The comment is
corrected rather than the behaviour: this number is a display, nothing is billed
from it, and the spend gate reads the exact decimal itself. A debit that must not
overstate has to round down deliberately instead of reusing this.
apps/ai carries the same "truncated toward zero" claim on the same call and is
wrong the same way. Noted in place; its gate compares > 0, so a sub-cent rounding
cannot change its verdict.
GET /v1/billing/balance answered 502 "billing upstream unreachable" on every
call and the console rendered "Unavailable" while the ledger held the money.
Nothing upstream was involved — that label is wrong and it is why this looked
like a connectivity problem for days. plane.Money.Minor() REFUSES a value finer
than a cent rather than round behind the caller, and the ledger keeps eighteen
decimals because per-token charges are routinely finer than a cent. So a REAL
balance broke the read: live the org held $149,913.078983985999994361 and the
error was "is finer than its minor unit; round explicitly". It got worse as
usage accumulated, since a longer history makes a sub-cent tail likelier.
Minor()'s own doc says a caller that wants a rounded figure — "a display, a
summary" — should round explicitly, where the choice is visible. This view is
exactly that and never did. It now rounds DOWN, the same choice apps/ai
documents for the same value: a displayed balance must never exceed what the
account can actually spend, and truncation understates by under a cent. Nothing
is billed from this number.
Also reverts a wrong fix from earlier in this session: registering balance
co-resident in apps/commerce. The manifest gives /v1/billing/balance to the
`billing` app, not commerce, so that registration was in a subtree it does not
own and could never have run.
Two things this exposed, both left alone deliberately:
- The 502's message names an upstream that is not in the path. Renaming it is
a behaviour change to an error contract callers may match on.
- apps/billing's test binary needs libsqlcipher and cannot compile on a
laptop, so `go build` passing there proves nothing about the test file. That
masked a first version of this patch which called a Minor() that does not
exist on the local money type; caught by type-checking with go vet instead.
GET /v1/billing/balance answered 502 "billing upstream unreachable" on every
call, and the console renders that as "Unavailable" while the ledger holds real
money.
The cause is the one apps/account/billing_coresident.go already documents in its
header: co-resident there is NO standalone commerce — the in-cluster service
selects the cloud pods themselves — so the /v1/billing/* bridge forwards to a
default base that is the public edge, and the read re-enters the same bridge in
an unbounded self-dispatch loop. Six sibling reads (invoices, subscriptions,
alerts, payouts, settings, credits) were already registered co-resident to
shadow that wildcard. Balance was not one of them.
It now registers through the identical chain — RequestContext, IAMTokenRequired,
PinBillingSubject — so the subject pin is byte-for-byte what the bridge applied
and a read scopes to exactly the account the spend gate debits, never wider.
balance/all rides the same prefix, since a prefix owns its whole subtree.
The apps/commerce test binary does not compile on a laptop (hanzoai/base needs
libsqlcipher, which the build image links and macOS does not); verified by
stashing that the identical failure predates this change.
Every AI request logged
tier_cache: Commerce lookup failed for key=...: commerce returned 404
(defaulting to zen-free)
and that default is 60 rpm against 500 for pro, 2000 for team, 50000 for
enterprise. So the failure did not give anything away — it silently served every
PAYING customer the most restrictive tier in the table.
Cause is the exclusive-subtree rule again: account-bridge owns /v1/billing, and
commerce's row named seventeen sibling paths but not tier, so
GET /v1/billing/tier never reached the handler that answers it
(commerce api/billing/handlers.go:43 registers it; live it 404s).
The UNREACHABLE ledger in manifest/router_test.go did not catch this and is not
wrong to have missed it: it fires on paths the fleet PUBLISHES, and cloud does
not publish tier — commerce serves it and only the ai router calls it. A
cross-binary caller is outside what that gate can see from here.
The pinned console-embed was sha-147ecd3, built 2026-07-27. console main is 63
commits ahead of it, so a week of console work — including the onboarding fix
below — has never reached production: the pin does not track main BY DESIGN (a
floating tag made a release silently bake the PREVIOUS console), so it only moves
when someone moves it, and nobody had.
sha-846069c is console main HEAD. Verified published before pinning:
ghcr.io/hanzoai/console-embed:sha-846069c-amd64 resolves to
sha256:f2849a31b082da0d690bc5403a41ad995624fdfd6595089008821c07718a9255, and
console-embed:latest now points at the same digest. Probed with all four Accept
types against a bogus-tag control — a two-header probe returns a FALSE 404 on an
image that is present, which is how a healthy registry can be misread as an
outage.
What this carries to production, beyond 62 other commits: a refused org create no
longer reads as a complaint about the NAME. /v1/iam/onboard answers 409 for two
opposite reasons — the first-run gate (this account already admins an org;
founding a second would orphan it) and a name genuinely held by another tenant —
and the console showed the server's organization-level message immediately after
the customer typed a name. It now reads the account to tell the two apart, and
offers the way into the org the identity is actually in instead of dead-ending on
a form that can never submit.
cloud pinned ai v1.832.16 while five commits sat on ai's main untagged, so a fix
merged hours ago was in no release and no binary. That is the third instance
today of merged-and-unshipped, each with a different cause: an image published
before its own fix landed, a build job silently skipped by a stale generated-doc
gate, and now a module change in no tag at all. The symptom is identical from
outside — the code is right and production is wrong — which is what makes it
expensive to notice.
Carries: the ok-with-no-user key refusal (IAM answering status=ok with a null
user used to fall through to a bare "invalid API key" that named no cause), the
openrouter seed URL carrying a /v1 the family appends itself, a family's kms://
key resolved before it becomes an Authorization header, family control from
admin.hanzo.ai rather than env alone, and /v1/provider-flags becoming
/v1/models/providers derived from the served catalog.
No release cut here. The next cloud build carries it: cloud is one replica with
strategy Recreate, so every release is a total outage of inference, billing and
auth, and a dependency bump does not justify one on its own.
The site edge has never run in production. It was mounted in serve.go — the
FUSED composition root — but the image runs cmd/cloud (the light router) on the
public port with one process per app beside it, and none of those is serve.go.
So every <slug>.hanzo.app fell through to the console SPA, and the whole /v1
surface answered on the customer's own hostname.
Proven at the pod rather than inferred: the process holding :8000 is /cloud (a
separate 23MB binary from /plugins), and `grep -c sites_resolve /cloud` is 0 —
the code was not in the running binary at all, even at the tag that contains it.
The two earlier fixes this session were both real and both invisible for this
reason: the host resolution fix (8b729f8ef) and the plane resolver (5f0b74fe9)
were compiled into a composition root nothing runs.
The router "deliberately links none of" the fleet's package graph, and that
holds: `go list -deps ./cmd/cloud` still contains ZERO of the root package.
apps/sites is a leaf, and the cross-app call is made here with zip.DialApp —
the same door wake.go already uses to publish one op without cloud.Plane().
apps/sites exports the wire types so the caller restates no mapping.
Mounted BEFORE webui, which owns "/" for every unclaimed path and would
otherwise answer first for every site host.
The test reads run()'s own source for the call site, because the defect was
never a logic error — the package was always correct — it was a middleware that
ran nowhere. My first version called mountSites directly and PASSED with the
call site deleted, which is exactly the mistake this file is about. Both failure
modes are now negative-controlled: no call at all, and mounted after the console.
Every published site served the console SPA. The host fix in 8b729f8ef was
necessary and not sufficient: with the host resolving correctly the edge still
found no site, because sites.SetResolver writes a PACKAGE-LEVEL registry inside
`projects` and the edge middleware reads it inside whichever process fronts
:8000. The pod boots ~25 single-app processes ("enabled":["<one>"], 25 distinct,
zero multi-app — measured on the live pod), so those are never the same process
and the registry is always nil where it is consulted.
A nil registry is a clean MISS, not a fault. So every lookup failed silently,
every request fell through to the API pipeline, and no error was logged anywhere
because nothing had failed. Proven at the pod with the ingress bypassed:
wget --header="Host: app.maxpower.hanzo.app" http://127.0.0.1:8000/
-> <title>Hanzo Cloud Console
for a site that is genuinely published and has its own Ingress.
The fix is the seam this repo already uses for exactly this shape:
FinanceScopeRules is on the plane, in its own words, because "the READER is a
cloud EDGE middleware" and the fact belongs to another app. projects now
publishes sites_resolve / sites_resolve_org from the one process that owns the
store, and the edge falls back to them. Co-residence still wins with no hop —
currentResolver prefers the in-process registry and only then asks.
Not-found stays a clean 404; a failure to ASK stays an error, so the edge can
render 503. Collapsing those would serve 404s for live customer sites during any
transient failure of the owning app, which is indistinguishable from deletion.
The comment on SetResolver said "until it is set, every site request is an honest
404 (the projects subsystem is not mounted)". That premise was the bug: projects
IS mounted, just elsewhere, and 404 is not honest when the site exists.
Negative-controlled: removing the fallback fails the new test with the
fall-through this commit is named for. The second test pins that a co-resident
store is still used without the hop.
The relational engine wrapped an identifier in the dialect delimiter but wrote
the body verbatim, so a delimiter inside the identifier closed the quote the
writer opened and the rest executed — Desc("name`,(subquery)--") emitted two
quoted identifiers plus bare SQL rather than one identifier. xorm v1.4.5 doubles
the delimiter, the standard SQL identifier escape, so a name that smuggled one
becomes a single identifier that does not exist and fails closed.
orm is the only place that version is chosen — consumers name hanzoai/orm and
the engine arrives underneath — so this is the hop that makes it live here.
go list -m confirms xorm resolves to v1.4.5 in this build.
Bump only. The four failing packages are the standing pre-existing set
(apps/iam, apps/plan, apps/pricing, manifest); the bump adds none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
plugin/dataset/main.go called cloud.Serve, which does not exist. Every other
plugin calls cloud.Listen. The image build enumerates the manifest and compiles
each plugin in turn, so this failed the whole build at the dataset step:
plugin/dataset/main.go:29:18: undefined: cloud.Serve
That means no cloud image has been buildable since ae3a30994 landed. `go build
./...` at the repo root does not catch it — the plugin mains are only reached by
the Dockerfile's per-plugin loop, so the gap between "compiles locally" and
"produces an image" is exactly one word wide.
Every published site served the CONSOLE, and mounted the whole cloud API under
the customer's own hostname. Measured live 2026-08-03:
quest.hanzo.app/ -> <title>Hanzo Cloud Console
quest.hanzo.app/v1/billing/plans -> 200
fiber parses the request URI once, and behind the ingress the parsed host is
empty — so siteSlug("") failed, customCandidate("") failed, and every request
fell through to c.Continue() into the API pipeline. The site edge was mounted and
configured correctly the whole time; it just never learned which host was asked
for. Same accessor and same failure as commerce's tenant resolver, in a second
codebase, the same night.
The parsed host still WINS whenever it names something this server serves — that
ordering is the security property, not a detail, because the host picks the ORG
here and a client able to override a real host could serve itself another
tenant's site. X-Forwarded-Host is consulted only when the parsed host is not a
host we can serve, which is the ingress case and never a direct request.
TestMiddlewareTenantKeyedByHostNotPath (pre-existing, unchanged) still passes.
Negative-controlled: reverting to Hostname() alone fails the new test with the
fall-through this commit is named for. One correction on the way: the first
version of the new test used req.Host="" to express "no parsed host" — httptest
synthesizes "localhost" for that, so it proved nothing until measured.
studio could not enqueue a single render. `POST /v1/tasks/.../activities` answered
403 "identity required", so thirteen jobs sat `queued` in its worklog for up to
nineteen hours while both GPUs polled an empty namespace every two seconds and
reported themselves healthy. Nothing in the queue, the worker logs, or the studio
UI said why.
The cause is one branch. homeOrg already knows that a machine cannot mis-attribute
its org the way a human choosing an app can — that is why the KMS sync identity
reads `owner`. But KMS is recognised by audience, which works only because its
client id is DERIVED from its org ("<org>-platform-kms"). No other app's id is,
so every other client_credentials principal fell through to the estate rule, found
the empty `orgs` a machine correctly carries, and resolved nothing. SanitizeIdentity
then minted X-User-Id with no X-Org-Id, and each org gate refused it.
So the recognition comes from the token's SHAPE, which a human token cannot wear: in
a client_credentials token the client IS the subject — IAM sets sub to "<org>/<app>"
— and azp equals the sole audience, because the app asked for a token for itself. A
human's subject is the user and azp is whichever app they signed in through, which is
the exact mis-attribution homeOrg exists to prevent. Every field read is IAM-signed;
none is a header a caller sets.
This does not widen who may cross tenants. It resolves an org for a principal that
has exactly one and can no more choose it than an sk- key can: `owner` is the
application's own organization, and getting the token requires that application's
client secret. A human with no `orgs` still resolves nothing and still fails closed —
tested, along with each half of the shape, because a partial match is a human token
that merely resembles a machine.
An organisation that took its model out of shadow BEFORE the model had learned
anything was told live=true and had nothing written down. The regime lived on the
same row as the learned state, and that row's writer declines to write while the
snapshot holds no learned mass — correctly, because there is no state to lose. So
PUT /v1/risk/state/appetite answered 200, reported live, and persisted nothing;
this binary deploys Recreate at one replica, so the next rollout rebuilt from
defaultConfig — shadow — and the model decided nothing. No error, no log, nothing
to alert on. A model silently disarmed, on a routed door.
The two facts are decomplected. The regime is now its own append-only versioned
record on the tenant's own shelf, written BEFORE anything in memory moves, so a
policy that cannot be written down is refused rather than answered from state the
next rollout will undo.
A regime is a VALUE: a version is minted only when the numbers CHANGE, so a
version means "the Nth distinct policy this organisation adopted" rather than "the
Nth time somebody pressed save", and a client that restates its config on every
deploy is free rather than the cheapest way to fill a disk.
Every score now cites the version it was decided under. Cut is derived from the
appetite that version states, so without the citation a restated appetite made
every earlier decision unreconstructible — the threshold it was measured against
no longer existed anywhere. GET /v1/risk/policy reads the history back.
Bounds, both per tenant and both on the tenant's own table:
RATE at most 24 distinct regimes per rolling 24h, refused past it with the
organisation's own bound named and the regime in force untouched.
TOTAL 381 versions, which IS 256 KiB divided by a measured worst-case row.
At the ceiling the oldest is disposed of and the number disposed of is
REPORTED, derived from the lowest surviving version so it cannot drift.
A regime that predates this record is ADOPTED as version 1 on first residency.
Without that, resolving the regime only from the new record would have returned
every already-live organisation to shadow on the first rollout after this ships —
causing the very defect being fixed, to every tenant at once.
The bounds on review and sample had two spellings, one at the op and one in the
plane. The op's copy is deleted; admitRegime is the one door, at the same
strictness the published contract always had.
Two test gates widened, because both could have been passed by looking at
nothing:
- TestOps_EveryOpIsAdmittedAndPriced parsed typed.go alone, so an op declared in
any other file was admitted and priced by nobody's assertion. It now parses
the package, matches on the RECEIVER TYPE (the plane carries score/learn/
state/appetite too), and fails when a registered op is declared nowhere.
- the per-tenant history assertion over the wire cannot observe the query's
tenant predicate: two orgs are two FILES. The predicate is load-bearing where
two brands share one file, and TestPolicy_TwoBrandsShareAFileAndNotAHistory is
the test that fails when it is dropped.
Eleven mutations applied, each named test red under the defect and green after
revert.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
THE ADDRESS IS THE PRODUCT. openapi.Fold takes an operation's product tag from
the first /v1 segment of its path and nothing else (openapi.Product); a per-op
zip.WithTags names a different axis and cannot override it. The seven ops were
addressed /v1/ml/labels, so seven compliance operations — their own writers
(commerce adjudicates the dispute, the compliance face closes the case, an
analyst files the review), their own five-year retention floor, their own
per-tenant file — would have been published as part of the KServe model-SERVING
product, which is four paths and live with customers on it. Nothing in the fleet
would have said so: the floor ratchet reads `ml: 7 -> 14` as growth, because it
refuses a shrink and only a shrink. It is the same mistake apps/risk's own
manifest row already records having made and corrected once, one layer up.
So: /v1/risk/labels, tag risk, every operation id and schema name risk-prefixed
(riskLabelEvent and not riskEvent — apps/risk publishes a riskEvent already, and
it is a scored decision rather than a judged one). floor.json returns ml to 7 and
raises risk to 17. The manifest row precedes risk, whose prefix is the bare
/v1/risk, and TestEveryServedPathReachesTheAppThatServesIt proves all six paths
reach label over the real fleet router rather than a comment claiming they do.
address_test.go walks the live projection — the same openapi.FleetSpec that writes
the committed subset — so a route re-addressed into somebody else's product fails
at the plane.
A BOUND ON COUNT OVER CALLER-SIZED VALUES IS NOT A BOUND. maxResolve capped a
resolve at 500 named events and nothing capped a subject: the rows were bounded
and the bytes were bounded only by the edge's BodyLimit, which is a fact about the
deployment. Each subject is then amplified below the door — a dedupe key, a
grouping key, one bound parameter per event in a statement against a single-writer
file. The write door had the ceiling all along (admit, subjectMax); the read doors,
added after, did not, and nothing compared them.
There is now ONE spelling of each ceiling — admitSubject, admitKind, admitSource,
admitEvidence, and instantMax inside stamp(), which is the one parser every time
field passes — and every door asks it. So `count × ceiling` IS the byte bound of
everything this plane binds, holds and stores. An unknown kind or source is
refused on the READ path too: it can only ever match zero rows, so refusing says
so instead of charging for the scan. bound_test.go proves it twice: reflect walks
every In type and fails on a caller-sized field with no declared ceiling (the
structural half — a new field cannot arrive unbounded), and every declared ceiling
is refused over the wire with a refusal that does not carry the value back.
A LITIGATION HOLD THAT ARRIVES MID-SWEEP KEEPS THE RECORD IN BOTH PLANES OR IN
NEITHER. dispose sweeps the derived copy FIRST so nothing is orphaned in the
warehouse, then deletes from the record re-asserting `hold = 0`. That protected
the record and silently corrupted the copy: a record the delete declines to remove
has already been swept, its seq is behind the delivery cursor, and deliver() asks
the cursor rather than the world — so no retry re-sends it, pending() answers zero,
and the row is present in the compliance record and permanently absent from the
answer key a training join reads. A missing fraud label reads as an honest
customer, and the row is the one somebody is litigating. remove() now reports what
it kept, the sweep writes those back from the record, a repair that fails refuses
the request rather than acknowledging a short copy, and `restored` is a NAMED state
on the response. `disposed` counts what was disposed of rather than what was
identified: a compliance report that says it deleted a record it is still holding
is the wrong answer to the only question the report is asked.
THE PUBLISHED PRECEDENCE RULE NAMES THE FIELD THE RESOLVER READS. The op exists so
a caller holding a contested resolution can reproduce it, and its second term said
`seen` while stronger() compares `knowable`. The two are equal for a live pipeline
and differ for exactly the backfilled history the derivation exists to hold back,
so a caller reproducing the rule got a different winner and no way to see why. The
test counted the terms, which made the only property that matters unobservable; it
now pins each term to the field at its position.
Also: plugin/label/mcp.json is deleted. It is the only mcp.json in the tree, no
app on main has one, the generator that wrote them was retired with its gate
(manifest/mcp_test.go says so), and it declared seven tools under the old
operation ids with nothing left to regenerate or compare it.
25 mutants in scripts/mutate.py, 11 of them new, 25 KILLED: each reintroduces one
of these defects and the named test goes RED.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
resolve could spend the whole process on one authenticated request. Two
amplifiers composed. A key had a count bound and no BYTE bound, and the domain
matcher split a host into labels and re-joined every tail, so an L-label host
allocated a copy of each of its L suffixes: one 8 KB dotted key materialised
16.8 MB and 100 of them 1.7 GB, measured over the router. And `sets` had no
bound and no dedupe, so naming one set N times ran N times the answers.
- maxKey bounds one key in bytes at every door it crosses — looked up,
written, removed — and REFUSES rather than truncating, because a shortened
key is a different key.
- a domain suffix is now a slice of the host, not a join of its labels: the
same answers in O(L) headers over one backing array.
- a call may name each published set once, and a set named twice is consulted
once.
The override write had the same hole from the other side: maxOverrides bounded
rows and nothing bounded a row, so 10,000 entries x 11 sets was gigabytes of
attacker-chosen bytes on the one volume every other organisation's store lives
on. The same maxKey closes it; an over-long note is refused rather than trimmed;
and what one organisation may occupy on that volume is now a figure the code
computes and a test pins, so raising rows, a key, a note or the catalog is an act
with its consequence next to it.
An override is a record, so it is now shipped before it is acknowledged, the way
apps/research and apps/books do it — this deployment is one replica with a
recreate rollout, and an unshipped write is a control an operator believes is in
force and is not.
prune spared ONE version: the call site passed the current version for both of
the statement's two placeholders, deleting the rows behind every citation taken
in the window before a refresh. What a take supersedes is now decided by
sweepOld over what the plane held before it, and proved against a warehouse
rather than against the text of the statement.
The publisher's end of the same amplifier is closed with the same door. A take
is refused whole if it carries a member longer than maxKey — one no lookup could
ever reach, so it is only weight in the warehouse, in every hydrate and in the
snapshot every request reads — or more members than a published set holds: swing
measures GROWTH against the version a take replaces, and a first take has nothing
to measure against, which after a cold start is every take. maxBody comes down to
six times the largest source in the catalog (measured: 2.6 MB), because the parse
allocates before any later gate can look at what it made. A publisher's redirect
must keep the two properties its origin already had, TLS and a destination
outside this network: this process runs in the cluster, where "wherever the
publisher says" reaches the pod network and the metadata address.
Also: a take whose size swings past 4x is refused and the previous version
stands (force is the operator lever); the disposable list is refused whole if it
names a mailbox provider, which is the one-row attack the size gate cannot see
on the one unpinned source; an attest receipt with no version or no designations
is recorded as a refusal instead of a current, fresh list; the plane sweeps at
cold start instead of refusing every set for six hours after a deploy; the one
cross-fleet aggregation states its own memory and time budget; every source
states a typed redistribution Basis from a closed vocabulary, on the wire, so
the licence position is an audit rather than a sentence; refresh reads the ONE
SuperAdmin predicate rather than restating it, and an admin of their own org is
refused there, because this route writes the baseline every org reads; and the
bridge sits on this app's own leaf, not on the /v1/ml parent two other products
answer under.
65 tests green under -race (one skipped: it dials the real publishers). Every fix
has a regression test that fails when that fix alone is reverted: 24 mutants, 24
killed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/ml/reference publishes ten sets — disposable email domains, hosting and
Tor address ranges, crawler user-agent patterns, delegated autonomous system
numbers, card-scheme prefixes, browsers the fleet sees everywhere, and the
freshness of the designation lists the screening engine holds. Six typed ops.
The unit of version and freshness is the SOURCE, not the set, so one publisher's
outage neither blocks the others' updates nor silently shrinks the set. A
version IS the content digest of the sorted entries, which makes a re-take of an
unchanged publisher a no-op that says so, and a half-landed version resumable
from its cursor without depending on it — the primary key already deduplicates.
Two planes, two stores. The baseline tables carry no tenant column, so a
cross-tenant write is unrepresentable rather than refused; a tenant's own
allow and deny entries live in that organisation's own store. Resolution is
override then baseline, both through one candidate function.
The baseline carries only published data under terms we hold — every source
states its licence — and aggregates above a k-anonymity floor no single
organisation can reach. Sources we may not redistribute are declared seams that
refuse, because an absent set and an unlicensed one look identical from outside.
A set that never loaded refuses rather than answering "not listed", and a stale
set answers and says so: every answer carries the version, its as-of, its age
and whether it is past the bound.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/ml/labels is the answer key the model plane cannot build for itself: what
turned out to be fraud, who said so, and when they could first have said it.
Chargeoff, dispute, case, refund, review and the below-the-line sample all file
into one append-only record per tenant — its own encrypted SQLite file, no org
column, so a cross-tenant read is not forbidden but inexpressible — with a
derived ClickHouse copy for joining at training scale.
Seven typed zip ops, so the REST route, the OpenAPI operation, the MCP tool, the
CLI command and every SDK method are projections of one declaration: mlLabel,
mlLabels, mlResolveLabels, mlLabelCoverage, mlLabelVocabulary, mlDisposeLabels,
mlHoldLabels.
THREE PROPERTIES, EACH THE POINT OF THE PLANE, NONE OF WHICH THE FIRST CUT HAD.
DURABILITY. Nothing in the package called OrgStore.Sync, so the only ship was
CloseAll on a graceful shutdown. cloud deploys strategy Recreate at one replica:
an ungraceful termination lost every acknowledged record since process start,
and the successor hydrated the older durable snapshot OVER the local file — an
acknowledged compliance record was not merely at risk, it was overwritten by an
older copy of the tenant's own history. state.ship is now the ship-before-ack
step every write path calls before it answers, and an unacked ship fails the
request rather than acknowledging a divergent local copy. That covers BOTH
shapes: a replica that never held the lease (ErrNotOwner) and one deposed
between the write and the ship, whose fenced Put is refused at a stale round
with no error at all. The two sibling durable planes hold the same contract
(apps/research shipFor, apps/books shipLedger).
DELIVERY. The cursor was the pair (wrote, id) over a write clock truncated to
the second and a content digest — an order the writer never took. A write that
commits after a concurrent delivery has read, whose digest sorts lower inside
the same second, was already behind the mark: never mirrored, unreachable by any
retry, and pending() answered zero because it asked the same predicate. A hole
in the answer key reads as an honest customer. The cursor is now the store's own
AUTOINCREMENT position, allocated inside the insert on the single connection
every statement for a tenant runs on (sqlpool.Single), so cursor order is commit
order by construction.
LEAKAGE. `seen` is whatever the caller sent, bounded only by At <= Seen <=
now+skew, and nothing tied it to any fact the server observed — a dispute filed
today with seen == at was knowable a year before the record existed, and a
backtest standing two days after the event resolved it. Fact.Knowable is derived
server-side as the later of `seen` and the clock at the write, it is the only
time visible() and stronger() read, and it is a column in the derived copy so
the warehouse applies the same predicate the record plane does. A live pipeline
is unaffected: Wrote is within minutes of Seen and the derivation changes
nothing.
Also: the coverage window now ends where maturity begins, so the gate on
training no longer answers zero on its own defaults; Group returns the whole
matured cohort so `matured` counts what matured and `unlabelled` says why
`judged` is low; a litigation hold is a fact about the record with its own op
that can also release one, instead of a digest-excluded flag silently dropped on
any record that already existed; the warehouse partitions by month like every
other table in the fleet rather than by tenant, whose cardinality is the
customer count; and resolve answers one row per distinct event, so an event
named twice is no longer its own conflict.
Every one of those carries a mutant in scripts/mutate.py that reverts it and a
test that goes RED when it does: 14 rows, 14 killed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An org name is unique within an issuer and not across issuers: acme on
hanzo.id and acme on zoo.ngo are two unrelated businesses. A per-org SQLite
file never confuses them, because DataDir is per deployment and the brand is
the directory the file is in. A COLUMN in the shared columnar warehouse has no
such directory — org = 'acme' there is a predicate over both — so every row a
brand-shared table carries has to be keyed on <brand>/<org> and every read of
it has to bind the same form.
Qualify joins SanitizeOrg and OrgNamespace as the third org-naming door and the
only one for a shared plane. Qualified is derived from Qualify rather than
restated, so a change to the shape cannot leave a validator behind. Tenant
carries no JSON tags and has no constructor but Qualify, so no In struct can
decode one and no caller can assert a tenant for itself.
The brand half comes from Deps.Brand, never a header: a caller that can choose
its brand has chosen which tenant space its org lands in.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-02 11:27:31 -07:00
1449 changed files with 41653 additions and 12468 deletions
# TWO DIFFERENT FAILURES SHARE THIS STATUS, and treating them alike
# would turn one of them into a ten-attempt loop that ends in the
# wrong diagnosis. "Reference already exists" is the collision this
# loop is for. "Object does not exist" means OUR OWN COMMIT is not
# on github.com — which is a live possibility, because this workflow
# runs on git.hanzo.ai and claims against GitHub, so a commit that
# reached the forge and not the mirror lands exactly here. It is not
# a name to skip past; it is a repo that has not been published, and
# the next number would fail identically.
WHY=$(jq -r '.message // empty' /tmp/claim.json)
if [ "$WHY" != "Reference already exists" ]; then
echo "::error::claiming v${CAND} was refused with: ${WHY:-unknown}. If this is 'Object does not exist', commit ${SHA} is on the forge but not on github.com — the release lane claims versions against GitHub, so the mirror must carry the commit first."
exit 1
fi
# Taken. By us, or by somebody else? The distinction is the whole
# difference between a resume and a collision, and it is one GET.
echo "v$NEXT is already published from our sha — skipping the build"
RESUMED=1
else
echo "::error::v$NEXT is a version this run OWNS (tag at ${SHA}) but the registry already serves bytes built from '${REV:-an unlabelled commit}'. Another lane pushed onto a name it did not hold. Nothing here may overwrite it — publish the intended bytes under a new number and delete the foreign image."
exit 1
fi
elif [ "$CODE" != "404" ]; then
echo "::error::manifest probe for v$NEXT returned $CODE — cannot tell whether the name is free"
exit 1
fi
echo "version=$NEXT" >> "$GITHUB_OUTPUT"
echo "highest seen v$LAST (registry + git tags) -> building v$NEXT (document sha256:$SPEC_SHA)"
echo "resumed=$RESUMED" >> "$GITHUB_OUTPUT"
echo "highest seen v$LAST (registry + git tags) -> v$NEXT, claimed at ${SHA} (document sha256:$SPEC_SHA)"
- uses:docker/setup-buildx-action@v3
if:steps.ver.outputs.resumed == '0'
@@ -379,8 +475,17 @@ jobs:
# VERSION is what the binary reports as X-Api-Version. Without it the
# ldflag falls back to the `dev` default and a released image cannot
# say which release it is — and every car below keys off that header.
# REVISION is passed as a BUILD-ARG, not only as a label, because the
# Dockerfile declares `ARG REVISION=unknown` and stamps the label from
# it. A builder that sets only the outside label leaves that ARG at its
# default, and a builder that sets neither publishes an image whose
# commit is unrecoverable — which is precisely what the platform lane
# did for every image it ever pushed. Feeding the ARG makes the label
# truthful no matter which builder runs the Dockerfile, instead of
# truthful only in the lane that remembers to override it afterwards.
echo "::error::${img} resolves, but the bytes behind that tag were built from '${REV:-an unlabelled commit}', not ${{ github.sha }}. Another lane pushed over the tag this run owns. Nothing downstream may pin it."
exit 1
fi
echo "$img is built from ${{ github.sha }} — tag, commit and image agree"
# THE SMOKE GATE. Boot the image that was actually pushed and require it to
# reach "zip listening" without a crash signature, on release.go's boot env
@@ -460,37 +587,32 @@ jobs:
steps:
- uses:actions/checkout@v4
# The receipt, minted only now: build pushed it, smoke proved it boots. A
# tag can therefore never name an image that did not start.
# THE TAG IS NOT MINTED HERE ANY MORE — it was claimed before the build, as
# the compare-and-swap that made this version this run's to build (see the
# `image` job's claim step). Minting it here was the bug: for the whole
# length of a build, a number was "taken" only in the sense that a lane
# INTENDED to take it, and two lanes intending the same number both pushed
# images before either reached this step. The winner's tag then named the
# loser's bytes, and the loser died here — after the damage.
#
# 422 USED TO BE A HARD ERROR, and that is exactly what made a resume
# impossible: the second run of a release whose fanout failed died here
# instead of continuing. A 422 whose ref already points at OUR sha is this
# same release, already receipted — idempotent success. A 422 pointing
# anywhere else is a genuine collision and still fails.
- name:Tag the release
# What remains is the assertion that nothing moved underneath us. A git tag
# is not mutable by accident, so this is expected to be quiet; it is here
# because the one thing worse than a moved tag is a moved tag that shipped.
- name:The claimed tag still names this commit
env:
GH_PAT:${{ secrets.GH_PAT }}
run:|
set -euo pipefail
TAG="v${{ needs.image.outputs.version }}"
CODE=$(curl -s -o /tmp/tag.json -w '%{http_code}' -X POST \
Description:"Lists the accepted (chain, token, treasury) triples, so a browser can\nrender \"send USDC here\" without the addresses being baked into its bundle.\n\nThis exists because the console previously gated its top-up UI on\nNEXT_PUBLIC_HANZO_HUSD_ADDRESS/_TREASURY — build-time constants. Enabling a rail\ntherefore meant rebuilding and redeploying the frontend, and with them unset the\nUI reported \"not available yet\" no matter what the server could actually accept.\nServing the set at runtime keeps ONE source of truth (the server's config) and\nlets a rail be switched on without shipping a bundle.\n\nEverything here is public on-chain data; no secret is exposed, and the set is\nempty on a deployment that accepts no crypto rail.",
Fields:map[string]string{
"railList.rails":"Rails is every (chain, token, treasury) triple this deployment accepts.",
"railView.chain":"Chain is the human chain name, e.g. \"Base\".",
"railView.chainId":"ChainID is the EIP-155 chain id the wallet must be on.",
"railView.decimals":"Decimals is the token's decimal places — 6 for USDC, 18 for an 18-decimal\ntoken. Cents are derived per-rail from it.",
"railView.id":"ID is the stable rail id to name when submitting a transfer, e.g. \"base-usdc\".",
"railView.symbol":"Symbol is the display symbol, e.g. \"USDC\".",
"railView.token":"Token is the ERC-20 contract address to transfer.",
"railView.treasury":"Treasury is the address on this chain to send funds to.",
Description:"Streams a stored photo. No credentials — see the file header.",
})
zip.Describe("GET /v1/csrf",zip.Doc{
Description:"IssueCSRFToken mints the anti-CSRF token a browser echoes as X-CSRF-Token on\nevery money write (mint/revoke a key, top up, onboard, and the billing/commerce\nwrite verbs). The token is bound to the caller's validated identity and expires,\nso one minted for one identity cannot authorize a write as another.\n\nIt is answered no-store, so it is never cached by a shared proxy. This is the\nsame-origin endpoint the embedded console reads — the Same-Origin Policy is what\nstops a cross-site page from reading the response and forging a write.",
@@ -61,19 +51,6 @@ func init() {
"apiKeyList.keys":"Keys is every key the caller holds, at most one per type.",
Description:"Credits the caller's org for a stablecoin transfer they already sent\nto the treasury. It reads the receipt from that rail's chain, confirms a mined,\nsuccessful ERC-20 Transfer to the rail's treasury, derives USD cents from the\non-chain value using the token's own decimals, records the credit, and returns\nthe amount plus the new balance.\n\nThe credited amount is the ON-CHAIN value, never a number the caller sends, and\nthe credit lands on the caller's own validated org — there is no way to name a\nthird-party subject. Nothing is credited that the chain did not confirm: a\nmissing, failed or non-matching transaction is refused, and a deployment with no\npayment rail enabled says so rather than inventing a credit.",
Fields:map[string]string{
"walletTopupReq.fromAddress":"FromAddress is the wallet the transfer was sent from. Optional; when given it\nmust match the transfer's on-chain sender.",
"walletTopupReq.rail":"Which accepted rail the transfer was sent on, e.g. \"base-usdc\". The client\nnames it rather than the server guessing from the tx: the same address can\nexist on several chains, so inferring would risk crediting against the wrong\ntreasury. It may be omitted only while exactly one rail is enabled.",
"walletTopupReq.txHash":"TxHash is the hash of the ERC-20 transfer that was already sent to the rail's\ntreasury. The receipt is read from that chain; nothing is credited that the\nchain did not confirm.",
"walletTopupResp.balance":"Balance is the org's new USD-ledger balance in cents. Best-effort: a read\nfailure reports 0, and the credit has already landed either way.",
"walletTopupResp.creditedCents":"CreditedCents is the USD credit recorded, derived from the ON-CHAIN value\nusing the token's own decimals — never a client-supplied number.",
"walletTopupResp.status":"Status is how commerce recorded the payment.",
"walletTopupResp.txHash":"TxHash is the transfer that was credited.",
Description:"Creates — or rotates — the caller's API key of the requested type and\nreturns it ONCE. A real IAM failure surfaces as 502, never a fabricated key.\n\nRotating is what creating means here: a user holds one key per type, so the\nendpoint is idempotent by (caller, type) and the superseded credential stops\nworking. Two live secrets for one user would make \"revoke my key\" a lie.",
Description:"Mints credit for one org. It is the ONE admin mint surface, and it\ndoes NOT mint in-process: it forwards the request to commerce's already-mint-gated\nPOST /v1/billing/credits, authenticated by the service token and scoped to the\ntarget org, then writes one tamper-evident compliance record. Commerce stays the sole\ncredit ledger; this is a thin, audited relay so there is exactly one place credit is\ncreated.\n\nThe body is commerce's OWN CreateCreditGrant contract, forwarded whole — every field\nit carries reaches commerce. The only two this layer reads are the target org (`org`,\nor `user` as the org-pool alias), which selects the namespace commerce's EdgeAuth\ntrusts, and `idempotencyKey`, which makes a double-clicked grant credit once.\n\nA FAILED grant is audited too, with the request body attached: an attempted mint is\nexactly as interesting to a compliance auditor as a successful one.",
Description:"Onboards a hosted service, or edits one, so a new host comes under the\nlaunch gate WITHOUT a redeploy. Re-registering an existing service PRESERVES its live\nswitch — editing the hosts of a service that is already open must not silently close\nit again.",
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
hanzo-dev
zeekayandhanzo-dev