POST /v1/lsp — one door, because there is one value: a language server
rooted at a workspace, asked about a position. code (static index) and
lsp (live server) are two reads of the ONE checkout.
{repo, rev?, path, line, character, method} → hover | definition |
references | typeDefinition | implementation | documentSymbol |
completion | diagnostics
Positions are the LSP's and pass through untouched: 0-based line, 0-based
UTF-16 character. Re-basing them would corrupt every multi-byte line, and
the callers already speak LSP.
Isolation is structural, not checked. The org comes from the validated
principal and is BOTH the pool key and the OWNER SEGMENT of the git URL;
a caller supplies only a repo slug. There is no input from which one
tenant can name another tenant's repository.
Scripts-off by default. A dependency fetch that runs dependency-authored
code is RCE triggered by whatever the caller asked us to check out, and it
buys nothing — servers resolve from source. So: npm ci --ignore-scripts,
cargo fetch (not build), go mod download; the python fetch builds sdists
and therefore does not run. rust-analyzer is additionally told not to run
build.rs or expand proc macros, because otherwise the server does at load
time exactly what the fetch was chosen to avoid. One predicate, one place.
The deployed worker must still be sandboxed — see fetchable's comment.
Language table ported verbatim from hanzo-tools-lsp (same binaries, same
argv, same root markers); install_cmd deliberately dropped — a worker that
can npm install -g at request time is one an attacker can make write to
its own filesystem.
server.go is the testable core: Content-Length JSON-RPC with a SINGLE
reader goroutine demultiplexing responses, server→client requests and
notifications. The python tool reads inline from each call, which drops
every frame that is not the awaited response — which is why it cannot
report diagnostics, and why an unanswered client/registerCapability
deadlocks it. 24 tests drive a fake server over in-process pipes; no
toolchain, no network.
Two bugs the tests found, both real in production:
- Close() wrote a polite shutdown unconditionally, so a server that had
stopped reading its stdin blocked it forever — holding a pool slot and,
at Shutdown, the whole binary.
- rel() compared a symlink-resolved root against an unresolved target, so
any data dir with a symlink component (/var on a Mac, a mounted volume)
made every location "outside" the checkout and handed the caller the
worker's absolute path instead of a repo-relative one.
Metered, not Free: cost is the COLD start (checkout + fetch + first index),
not the query. Warm point queries are recorded and free, so the pricing
does not teach callers to re-key their workspace. Gate before the work.
apps/code has NO checkout to reuse — it indexes files POSTed to it and says
so. apps/deploy has the only working-tree checkout and is not importable
(it would link k8s into this binary). This is therefore a second one,
following deploy's invocation and hardened env exactly; the fix is hoisting
it into the root cloud package, not made here.
Mount order: apps.Wire() and its integers are gone. manifest.Apps slice
position IS the order, so lsp's row sits after code's, with order_test's
frozen sequence updated in the same commit.
The comment blamed apps/answer for exposing no constructor. That is true and
it is not the reason.
Research carries an explicit 25-cent per-answer fee (apps/answer/mode.go),
charged through Bill.Gate on the request path where a payer has been resolved
and can be refused. A tool call has no payer, so a direct seam to the engine
would be an unbilled 25-cent operation an agent may invoke in a loop — free
inference, reached by the exact route this codebase keeps closing.
That the package makes it awkward is not an accident to route around: Params is
built from request-scoped billing context and Sink's methods are unexported, so
the money gate is structurally hard to bypass. Wiring it properly means an entry
that takes a payer and charges it — a billing decision, not an adapter.
web_search and fetch_url are different in KIND, not merely cheaper: their HTTP
routes gate on AUTHENTICATION, and the agent request reaching the tool was
already authenticated and metered at /v1/responses. In-process use matches how
they are reached over HTTP. deep_research does not.
A unix socket address is capped near a hundred bytes, and t.TempDir embeds
the test's own long name — on darwin the bind failed on a discarded goroutine
error and every dial refused, so the suite reported a phantom (socket never
began listening) instead of the truth. A short anonymous dir keeps the
address inside the cap on every platform; on a Mac without a mounted tmpfs
the tests now fail for the honest reason instead — the pure-Go SQLCipher
codec refuses to decrypt to persistent storage, which is the fail-closed
property it exists for.
The probe was a closure marshalling a map — a route and nothing else, in none
of the five projections. It is now a typed op whose answer is byte-identical
to the map it replaces (the struct's field order mirrors the map's sorted
keys), pinned by test, and whose prose rides the handler's doc comment the
way every typed op's does — so its openapi.Describe entry is gone.
Every other raw registration now states its survival in one sentence. The
webhook intake speaks the provider's protocol (HMAC over the raw payload),
the invoice PDF serves bytes, the fail-closed wildcard must shadow every
method under every prefix, and the rest obey the module-handler rule, stated
once above Mount: a handler living in hanzoai/commerce behind unexported
internals cannot ride a typed op, and a typed twin would be a second
implementation of the same money move — the drift payments.go exists to
prevent. The typed door onto a module rail is that file's exported-core
pattern, which is module work.
The eight operations that said nothing about themselves — wire, the crypto
rail, and the saved-card family — now state their gate, their tenant scope
and what they fail closed on, which is what let the commerce subset
regenerate at all. That regeneration also repaired two stale artifacts the
committed subset carried: GET /v1/billing/methods claimed its portal
sibling's operationId (the duplicate the weave gate died on), and the two
saved-card POSTs were absent outright.
The fleet golden and the floor now agree with the module: the deposit proxy
and the webhook relay were removed upstream (deposits are commerce's own
rails; the real receiver is /v1/billing/webhooks/:provider), so the golden
drops those four operations, the commerce floor is hand-lowered to four, and
the dead prose goes with them.
commerce v1.50.7 added PUT /_/commerce/providers/{name} — the verb behind "a payment
rail can be turned on and off" — and the dep bump landed without prose for it, so
`commerce describe` refused and the app could not project its own document. The
subset is fail-closed, so nothing was written and the drift stayed invisible until
the next regeneration asked.
It gets the sentence, and the same gate its GET twin states: the tenant comes from
the IAM owner claim and nowhere else, so a cross-tenant write is not expressible;
404 for an unknown provider is byte-identical to the cross-tenant probe's answer. The
one thing worth saying that the shape does not: this owns a single bit and never the
credential, which is why a rail can come back with the same stored secret.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
analytics' subset caught up in the regeneration and started publishing GET
/v1/event.js — an address the fleet then routed to ai, because a prefix owns a
SEGMENT subtree and "/v1/event" does not cover "/v1/event.js". It fell through to
ai's "/v1" and the router oracle said so: the fleet published a path it delivers
somewhere else.
The address is analytics' own — its binary projected it from its own router. So it
gets its own entry, which is what "deeper than the sibling that currently wins"
means for a path whose distinguishing character is a dot.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The weave was red on main and nothing downstream could move: openapi/weave_test.go
is the SOLE writer of openapi.yaml, so the served document, the CLI, every SDK, MCP
and the docs were all frozen behind
operationId "get_v1_billing_portal_methods" is claimed by both
"GET /v1/billing/methods" and "GET /v1/billing/portal/methods"
The derivation was never at fault. operationID(method, path) walks every segment and
yields two distinct ids for those two paths, and it has verified uniqueness inside
From since 2026-07-27. The subset that claimed otherwise was written on 2026-08-04 —
so no run of `commerce describe` at any version could have emitted it. It was
hand-edited: d7024e88 moved the three saved-card verbs to this app and wrote the two
new paths into plugin/commerce/openapi.json by copying the /v1/billing/portal/methods
block, operationId and prose together. The copied description then described itself
("the SERVICE-TOKEN face of the same list a customer reads at /v1/billing/methods",
published AT /v1/billing/methods).
It was hand-edited because it could not be generated: `commerce describe` refused,
and still refused here, with eight operations saying nothing about themselves. So the
fix is the prose, and the file follows from it.
- Eight operations get the sentence they owe a caller: the customer saved-card
family (GET/POST/DELETE /v1/billing/methods), its portal POST, and the top-up
rails nobody had described at all — wire instructions, crypto options, the
deposit mint and the deposit poll. Five of them were not in the published
document in any form.
- The two portal twins stop claiming a proxy that no longer exists. Both families
are served in this process; they are two addresses because they admit two
principals, not because either forwards to the other (apps/billing/billing.go
says the same at the spot the hop used to sit).
- mount.go said the opposite of the manifest — that /v1/billing/methods belongs to
billing and "a registration here is unreachable in the fleet", beside a live
registration of it. manifest.Apps gives commerce both prefixes and withholds them
from billing. Left alone, that comment invites deleting a route that works.
- /v1/commerce/{deposits,deposits/:id/confirm,deposits/:id/status,webhooks/:provider}
lose their prose. The broker-dealer proxy behind them was deleted and the manifest
stopped naming them; what was left was prose for routes nothing serves, which
renders nowhere and reads in source as though it were live. openapi/floor.json
drops commerce 8 → 4 in the same commit, which is where the ratchet asks for the
reason to be.
AND THE ASSUMPTION THAT LET IT LAND IS NOW A CHECK. Weave did refuse the document —
but a collision inside ONE part reaches it with no app attached, so it could name the
two addresses and nothing else, and which of 123 subsets shipped them was a search.
openapi.Subsets is holding the app's name when it decodes the bytes, so it asks there
whether the part is injective — the same uniqueOperationIDs, one statement of the
rule, asked once per part and once over the composition, because neither fact implies
the other. TestSubsetsRefusesAnAppWhoseOwnIDsCollide fails without it.
Regenerating also lands 13 subsets that had gone stale behind the red gate, and drops
zipdoc entries that were lifting commercemid.RequestContext's doc comment — a
paragraph about mint-gated context locals — as eight routes' descriptions.
plugin/iam is deliberately NOT regenerated: its current projection introduces a
schema "Role" that means something different from framework's ({role, user}), which
Weave refuses and which needs a rename in one of those two apps. That is a separate
defect this one merely uncovered.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Carries the P0 fix a live-browser pass proved: the assistant's streamed
completion posted with no Authorization header, was refused, and the card
then blamed the user's session. The release routes every self-reading
stream through the client's one authorized door, moves preferences onto
cloud's /v1/prefs, tells the truth on a 401, and mounts exactly one
composer per viewport.
ai v1.832.30 declares web_search / fetch_url / deep_research but holds no
backend for the two this host serves. agent/builtin_tool/web must stay a leaf
package (object imports agent, agent imports the registry), and websearch lives
here in any case — so this is where the seam closes, beside the balance and tier
readers, for the same reason they are here: this package links both sides and
the host does not.
In-process, never over api.hanzo.ai. The edge validates a CUSTOMER credential
and answers 401 to a service; routing our own calls back through it is what once
fail-closed every completion at 503 on a perfectly healthy pod.
deep_research is deliberately NOT installed yet. apps/answer builds its Params
from unexported fields and exposes no constructor, so wiring it means giving
that package an entry point rather than reaching into it from here. Until then
the tool reports that it is UNAVAILABLE IN THIS DEPLOYMENT — which is the honest
answer and specifically not an empty result: an agent told "no results"
concludes the web holds nothing on the subject and answers from memory in a
confident voice.
fetch_url needs nothing here — ai installs its own crawl at bootstrap.
/v1/event.js resolved the anonymous id itself, out of localStorage alone. That is
ORIGIN-scoped, so it never saw the cookie @hanzo/event shares across *.hanzo.ai
and never saw the id hz.js had already left behind under its own key: an origin
carrying only this tag was a separate population, and one visitor was two or
three people depending on which snippet a surface happened to load.
anon.js is that chain, vendored BYTE-FOR-BYTE from @hanzo/event
(hanzoai/ui pkgs/event/src/anon.js), and tag.go now serves it with the tag as one
asset inside one wrapper — so the door holds no second implementation, and
neither half leaves a name on the page it is pasted into. tag.js calls hzAnonId
and no longer names an identity key at all.
Resolution is cookie · localStorage hz_anon_id · localStorage hz_id · in-memory ·
mint: every id already in a browser is adopted, and only a browser holding none
is given a new one.
TestTagBehavior now runs the COMPOSED asset rather than tag.js, because the chain
is half of what ships and the other half no longer runs alone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The bank details were stored and the rail still answered 'Wire transfer not
configured'. Both doors reach the same KMS store keyed by (path, name, env), so
a read at a path nothing writes is indistinguishable from a bank nobody entered
— it would have stayed silent indefinitely.
cloud writes every in-process secret under /orgs/{org} (apps/destinations,
apps/integrations, credz all build that shape, and the REST surface folds writes
under it from the validated org claim). commerce alone spelled it
/tenants/hanzo/wire. v1.50.7 adopts the host's convention and drops the
hardcoded org: the brand serving the page now decides whose bank is read.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The extension renders structured result widgets (comparison, steps, stats,
timeline, entity, definition) but nothing emitted them: /v1/ask's stream is
status|sources|text|follow_ups|done and the system prompts never mentioned the
format. The client was ready and the producer was missing.
widgetRule teaches the one format both modes share, so search and research
cannot drift apart. It says three things beyond the schema:
- Only when the question's SHAPE calls for one, at most two, most answers none.
A widget on every answer is a worse document, not a better one.
- The PROSE MUST STAND ALONE. The client validates every block and drops
anything malformed — a ragged table, an unknown kind, a field of the wrong
type — keeping the prose. An answer that leaned on a widget to be complete
would read as a hole exactly when validation refused one.
- DATA ONLY, NEVER HTML. This model reads the open web, so every page it
fetches is a potential injection source. Markup it authored would be a path
into the extension's origin, which holds the user's session. The renderer
owns the shapes and escapes every field; the model owns only content.
widget_rule_test.go is the closest thing to a shared type across the two repos.
The renderer lives in the extension, so a typo here fails nothing: it produces
answers whose widgets silently never appear, which looks exactly like a model
choosing not to emit one. The test parses every example out of the prompt and
holds it to the shape the client accepts — one example per kind, every required
field present, the comparison example rectangular (the client drops ragged
tables rather than padding them), and both modes carrying the rule.
Negative control run: renaming the steps example's field to "items" fails with
`kind "steps" example omits required field "steps"`.
Pre-existing and unrelated: the root package does not build right now
(middleware_spend.go, another session's in-flight work). apps/answer builds and
tests clean.
No release has pinned since v1.801.454. .455 and .457 built, published, and
were never referenced by anything — the fleet has been serving .454 while
three versions sat in the registry.
The rollout car reads UNIVERSE_PIN_TOKEN from
/v1/kms/orgs/<org>/secrets/deploy/... and the broker has no such route. The
store root is derived from the validated claim, which is exactly what makes
another tenant's secret unnameable rather than merely refused, so there is no
org in a KMS path and never was. Measured against kms.hanzo.ai:
/v1/kms/auth/login 401 (route exists)
/v1/kms/orgs/hanzo/secrets/deploy/UNIVERSE_PIN_TOKEN 404
/v1/kms/secrets/deploy/UNIVERSE_PIN_TOKEN 403 (route exists)
fanout reads FLEET_DISPATCH_TOKEN through the same wrong shape and would have
failed the same way the moment rollout stopped failing first.
WHY IT WAS INVISIBLE, WHICH IS THE HALF WORTH FIXING. The step is
`set -euo pipefail` and the read is `TOKEN=$(curl -fsS ... | jq ...)`. curl
-f exits 22 on the 404, pipefail propagates it, set -e aborts the assignment —
so the authored `::error::UNIVERSE_PIN_TOKEN missing in KMS ...` is
unreachable code. Every one of these releases died with a bare `exitcode
'22': failure` and never printed the sentence written to explain it. Three
guards in this file were dead the same way; the two KMS logins are fixed here
too, and `receipt`'s create-or-update on a tag with no release yet is the
third (left alone — it is a different car and a different bug).
A guard that cannot run is worse than no guard: it reads as diagnosis
already handled.
KMS_ORG goes with the paths. The org is the credential's, so a variable that
named it was a knob that decided nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The corpus has two sources and both were in-process globals the catalog
process cannot see. The index write was one (previous commit). This is the
other, and it failed more quietly still.
projects.LiveSites answers nil when its package is unmounted, on the stated
reasoning that a deployment which hosts no sites is not an error. That is
true of a DEPLOYMENT and false of a PROCESS: catalog, index and projects are
three separate apps and therefore three separate processes, so in the
catalog process nil never meant "nothing is serving" — it meant "you asked
the wrong half of the fleet". nil and empty are the same answer, so the
corpus was published with no sites in it and nothing anywhere said so.
What that costs is the whole `site` kind: every demo URL, the lineage that
files a remix under community instead of leaving it looking like one of our
starters, and the deployed demos the template lane is mostly made of. The
repos alone would have made a catalog of source with nothing live in it.
plane sites_live, and it takes no org — the same shape as
sites_resolve beside it. This is THE cross-org read; the rule
that makes it safe (public, live, not hidden) is applied in the
query by the app that owns the store, so there is no tenant here
for a caller to widen into.
projects Ready(), so a caller can finally tell "nothing is serving" from
"ask the process that owns the store" — the distinction
LiveSites cannot make, because it answers nil for both.
catalog serving(), the third seam with the same two legs as lexical and
write.
Same test file, same reason: the split is what production runs, so the test
reaches a real peer over a real socket.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud.SetRiskScorer has held the seam and the whole fail policy since it was
written and has never had a producer. It could not have one: the model is
in-process mutable state, so exactly one binary may hold it, and the pod forks
one process per app. Every gate in the fleet therefore read a nil scorer, took
the absent exemption, and allowed unscored — fleet-wide, silently. The
observability plane's event door was the same shape and learned it the
expensive way; obsevents.go is gone and apps/o11y/obs_rpc.go replaced it.
So the scorer is REACHED rather than linked. apps/risk publishes risk_decide on
its own socket, commerce installs a plane client as the seam's first producer,
and POST /v1/billing/topup/token — the one self-serve credit door, whose mint
authority is a settled card charge — asks it before the charge.
The tenant is minted from the PLANE CALLER, never from the body. RiskDecideIn
carries no org and cannot: a model is trained on one organisation's own
behaviour, so naming which model answers would be the only cross-tenant read
this plane has to offer. The HTTP mint (tenantOf) reads a principal parked on a
request, which a plane call does not have, so it would fail closed on every
call; planeTenant qualifies cloud.Who(ctx).Org with the deployment's own brand
instead, and qualify() refuses an empty org.
The gate states Privileged explicitly. cloud.Privileged() matches IAM and KMS
paths and not this one, so the default would be the fail-OPEN branch — a scorer
outage waving through the one route that mints spendable balance. With the bit
set, a scorer that is present and cannot answer refuses and the caller retries;
a scorer that is NOT DEPLOYED still allows, which is the exemption cloud's fail
policy already carries and the plane client is careful to preserve: over a
socket "not deployed" would otherwise arrive as a failed call and take the
closed branch, so the socket is probed, ErrNoPeer is read as absence, and the
lazy child is woken off the request path rather than inside a 150ms budget.
A REFUSAL CARRIES NO SCORE. The engine assigns one before it checks whether the
model has warmed, so a declining model returns a populated, meaningless number;
publishing it would turn "no opinion" into "this is fine". And an alert is a
REVIEW, never a block: cloud's own vocabulary says a statistical judgement may
reach review and no further on its own, and this model is exactly that.
IT SHIPS IN SHADOW. No org is armed, no regime is changed, and a model nobody
has reviewed is in shadow — where alert is forced false however high the score.
So every legitimate top-up proceeds today and every decision is on the record
with the shape and policy version that produced it. The subject kinds move to
the call contract for the same reason the signal names are there: two spellings
of "account" would not read as a disagreement, they would namespace one subject
into two.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The published catalog has been empty since catalog and index became two
plugin rows, and not because a sync failed to run. It ran every hour, read
GitHub and the sites table correctly, assembled the whole corpus — and then
handed it to index.Reconcile, which serves out of the index's own
process-level global. In the catalog process that global is nil and always
will be. So the corpus was never written a single time.
The READ leg was given a plane op when the split happened; the write leg was
deliberately left in-process, on the reasoning that the store has one writer
and it lives where the file lives. That property is right and the conclusion
was wrong: the writer is still one and still the index's, whether the corpus
reaches it through a function call or a socket. Handing it over does not add
a second writer — a second process OPENING that SQLite would, and neither
leg does.
Fixing the read alone therefore could not have shown a row. It turned a
503 into 200 {"data":[],"total":0}: from a page that said it was broken to
one that said the fleet had built nothing. hanzo.app's /community and
/templates have rendered that empty ever since.
plane index_reconcile, the mirror of index_query, with the corpus
relayed verbatim for the same reason the read's rows are raw.
index publishes it beside the read; the swap still executes here,
in the process that owns the file. query_rpc.go -> rpc.go, the
name the other multi-op peers already use.
catalog write(), the mirror of lexical(): in-process first, then the
plane. Both legs now go through the GENERATED index client, so
the app name, the op and the In/Out pair are fixed to each
other by the compiler instead of at run time.
The suite could not have caught this: every existing catalog test mounts the
index and the lens on ONE app, which is the topology this deployment stopped
having — index.Ready() is true there, so the plane leg was never executed by
a test at all. split_test.go models the split instead and reaches a real peer
over a real socket. It fails on the old code with "index: not mounted".
A silent write failure outlives a loud read failure, and it is the harder one
to see: an error names itself, an empty page looks like an answer.
Also commits plane/team, generator output for an op added without a
regenerate in c8b7c31.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The provider list that decides which rails pay.hanzo.ai offers was readable and
not writable, so card, Apple Pay, Google Pay, Cash App, ACH, wire and crypto were
whatever happened to be in the tenant row and moving one meant editing the
database. v1.50.5 adds the write verb, one rail per call — a whole-list PUT built
from what the admin read projection exposes would write every provider back with
an empty KMS path and silently disconnect all of them from their credentials.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/commands serves zip.CommandsFromSpec over the rendered document —
the same function the CLI's tree is derived from, on the same bytes every
SDK is generated from. No new registry, no generator, no build step: a
route registered this morning is a command this afternoon.
serve() gains one line, so both document sources light up at once — Mount
for an app binary, MountFleet for the front door that answers api.hanzo.ai.
Total, and unfiltered by caller on purpose. zip has no per-op scope; it has
Authorizer, which decides on the DECODED INPUT of every op. Permission is a
fact about an input, so any list filtered here would be a second static
claim the Authorizer is free to contradict — wrong in the direction that
hides working functionality from people who have access.
Two things the design did not know, both measured here:
The size. The projection is 2,344,651 bytes, 454,881 gzipped — 1.32x
smaller than the document, not the 4.5x/107 KB the design quotes. That
figure was taken from hanzoai/cli's spec/products.json, a different
artifact with every description, summary and operationId stripped. This
one keeps the prose, and the prose is most of it (Description alone is
1,028,067 bytes). Recorded in command.go rather than fixed by forking
zip.Command into a trimmed wire shape.
The order. 41 of the 2,323 commands share zip's (Service, Name) sort key
— `mq streams-delete` is claimed by three — and sortCommands is unstable,
so the tie fell to a map walk. Two replicas weaving one document served
identical content under different ETags, which is a full re-download on
every conditional request that lands on a different pod. order() completes
the key with (Method, Path).
openapi.Door names what serve registers. Three gates needed exactly that
fact and each had written the one literal that was true when it was written
— Complete skipping a description no app can own, cmd/cloud exempting the
doors from a scoped deployment's surface, manifest refusing an app row that
claims one. The second door made all three wrong the same afternoon.
Pins: every served command is a route the document carries; the served
bytes are CommandsFromSpec of openapi.yaml exactly; two mounts of one
document agree on bytes AND ETag; a conditional request gets 304 and no
body; the projection is smaller than what it projects. Plus the door itself
answering on the host's own spec() mount.
TestFleetIsTheWeaveOfItsApps and two cmd/cloud gates are red on this tree
already, over a duplicate operationId between GET /v1/billing/methods and
GET /v1/billing/portal/methods. Not touched, not absorbed: openapi.yaml
therefore does not yet carry /v1/commands, and the full-fleet version of
the door test belongs in the commit that fixes the weave.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The image job reserves a version by creating refs/tags/v<N> AT THE COMMIT on
github.com. A ref can only point at an object that is there, so for a commit
github has never seen the claim answers 404 — 'Object does not exist' — and
refuses to build.
It routinely has not seen it. CI runs on git.hanzo.ai, which is canonical and
where the push lands; github is fed by a push mirror on an EIGHT-HOUR interval,
and the claim runs seconds after the push. The object the tag must name is
normally hours away, so the failure is not a permissions problem that looks like
a race — it is a race that looks like a permissions problem, and it is why four
days of releases stacked up behind one commit.
The commit is now published to github before the claim reads it, on a ref of its
own: main is the mirror's to move and the two lineages do diverge, while this
step's only job is to make the object exist.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
metric v1.10.0 moved the OTel bridge out of its root package, and zip took
the bump, so the framework and everything that inherits it now link no
OpenTelemetry SDK at all. Cloud's own meter pipeline still imports the SDK
directly and deliberately — it feeds the status page until zip's rows are
proven where the gauges read, and it is deleted with that proof, not before.
Moving /v1/billing/methods in-process (it had been forwarded through a base URL
this deployment never sets, so a signed-in customer got 401 listing their own
cards) quietly changed what an ANONYMOUS caller gets: the proxy answered 401 and
the co-resident chain answers 403, because PinBillingSubject refused both of its
cases with ErrForbidden.
Those two cases are different answers and must not share a status. A service
token is a credential: presenting one and omitting X-Org-Id is an authenticated
request that names no scope, and 403 is right. Presenting nothing is not signed
in. The difference is load-bearing on the customer path — a browser
re-authenticates on 401 and merely reports 403 — so an expired session on the
saved-cards screen showed a permission error instead of sending the customer to
sign in. The billing app it moved from answered 401 on purpose; its test said so
in as many words, and that test was deleted with the proxy.
So the tests move with the route, which is how this surfaced at all:
apps/commerce gains the assertion the billing app used to carry — anonymous
GET and POST answer 401, and a 404 fails loudly rather than passing as
"refused", since an unmounted route also declines every request.
apps/billing loses seven tests that pinned a proxy that no longer exists, the
three handlers behind them (paymentMethods, createPaymentMethod,
deletePaymentMethod — defined, registered nowhere, dead since the move), and
an openapi.Describe for DELETE /v1/billing/methods/{id}, an address this app
published and did not serve. That last one is the exact publish/serve
disagreement manifest.Apps exists to catch.
Also merges the GitHub lineage, which had diverged: 4 commits there (console
embed pinned by semver, platform forge sibling, the ai subscriber-scoring fix)
against 19 here. The version claim mints its tag on github.com, so a commit that
only ever reached the forge cannot be claimed — which is what the image job hit
after the containment fix let it run at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every team surface resolved its caller by decoding an HS256 token itself, so
"who is calling" was answered in six places against one signing key. This adds
identity (apps/team/account.go): ONE seam that resolves a caller, and the only
place a credential's algorithm is routed on. account, typed, files, billing,
collab and the transactor hold the seam instead of the secret, and three of them
stop importing the condemned package — account.go is the only reader left inside
apps/team.
Verification, tenancy and authorization are three questions, answered separately:
- VERIFICATION is cloud's own IAM validator, narrowed. A signature from a
trusted issuer says IAM minted the token, never that it was minted FOR this
surface — IAM's signer emits the same claims into an access token and an
id_token but for aud/tokenType/nonce. A session door must say which it
means, so the lane takes access tokens whose audience this deployment NAMES.
The boundary's no-audience-gate posture is right for an API door and wrong
here; the divergence is stated at the pin.
- TENANCY is the home org from the signed membership set, never `owner`.
`owner` carries the application's org, so it is chosen by whichever app the
caller authenticated through, and a lane reading it scopes every store query
to an org the caller selected. No membership set means no home, which is
also every machine credential — a team session is a person's.
- IDENTITY is the `sub` claim, resolved through the store. The canonical user
id falls back to preferred_username, and an account id derives from a UUID
verbatim, so a token with no sub whose username is a colleague's account uuid
resolved to the colleague. Subject-only, confirmed against the rows a login
created.
An IAM credential never leaves the seam: it is an estate-wide bearer held in an
HttpOnly cookie so page JS cannot read it, and the account RPC echoes a caller's
token back to page JS.
Workspace authorization on the IAM lane is the membership rows (admit) — the
server decides, the caller signs nothing. The transactor keeps its path-borne
workspace token and gains no ambient lane: a WebSocket is exempt from CORS, so a
cookie-borne credential would make the Origin list the data plane's only access
control, and that list no longer carries a wildcard either.
The existing credential answers first on every carrier, so a client that has one
behaves exactly as it did and the new lane serves only a browser holding nothing
else. meet decides a room join and does not own the workspace rows, so team
publishes them on the internal plane and meet asks, off the boundary's own
attestation rather than off headers a client can set. analytics is untouched:
its trust order already resolves a validated IAM bearer ahead of the team token.
The HS256 arm is deleted when login mints IAM-only and front/love/
analytics-collector verify IAM; getWorkspaceInfo is the one surface that still
needs a client change first.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Carries the converged console main: the Recent + Suggested strip on Models,
the topbar slimmed to navigation (environment, theme and alerts fold into
the account menu and drawer; production is the default environment), the
cross-app launcher restored, and the assistant defaulting to Enso. The pin
names the image GHCR serves — probed 200 with a negative control — and the
sha is the forge lineage's, which is the lineage that builds.
The framework boundary now exports a server span and a request record per
request to the o11y ears this binary already runs (planesink: 4317 spans,
4318 logs), shaped by zip against the collector's own receivers — no SDK,
nothing any plugin inherits. The address is the switch: O11Y_SPANS_ADDR and
O11Y_LOGS_ADDR turn each signal on, and O11Y_METRICS_ADDR — the name the
manifest already states — is now genuinely read (v1.24.x read a name the
manifest had stopped matching and reached the right ear by loopback
coincidence).
Every cloud release since 62590c3 — nine consecutive commits, four days — was
blocked in the `containment` job, and nothing about those commits caused it:
go: github.com/hanzoai/zen@v1.4.10: invalid version: git ls-remote ...
remote: Repository not found.
The token was valid and non-empty. hanzoai/zen simply has one collaborator where
its sibling modules — ai, commerce, orm, account — have fifteen, so the build
identity could read every private module in the graph except that one. GitHub
answers 404 for "private and not yours" and for "does not exist" alike, so the
error could not distinguish an ACL from a deletion. An ACL drifting beside the
code stopped the fleet, and the earlier green runs only hid it behind a warm
module cache.
So the build no longer asks GitHub for code that is ours. The module PATH stays
github.com/hanzoai/* — that is the package's name, not its address — while git
dials git.hanzo.ai, which is canonical anyway. go.sum is untouched and still
decides: the forge mirrors the same objects, the zip hashes to the committed h1:
line, and a forge serving different bytes would fail the build instead of
shipping them. Proved cold, with no GitHub credential present at all. GH_PAT
remains the fallback for anything the forge has not mirrored, and the four
modules it was missing (goauthorizenet, namespace, sendgrid-go, commerce) are
mirrored now.
Riding along, because it could not ship until the train moved: commerce v1.50.3
carries the wire rail reading the deployment's bank through the host's secret
plane, the payer reference on the wire memo, and the saved-card and tier guards.
Its work had landed on the lineage the history rewrite replaced, invisible to
the module path everything actually resolves; it is replayed onto the published
history, with thirdparty/square taken from the published side because fitKey()
is applied at all three Square boundaries there and carries tests.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The pin named a sha, then a digest. Both say which bytes built the bundle
but not which console RELEASE it is, so "what console is in cloud
v1.801.N" needed a second lookup, and the last six bumps read like
receipts for nothing.
The builder already publishes both: sha-<sha7>-amd64 on every main push,
and the bare semver on a cut v* tag. v8.5.37 is the tag for console
origin/main's tip.
This SUPERSEDES the digest pinned by #383 rather than reverting it: that
pin shipped the anonymous-visitor entry fix, and v8.5.37 contains that
commit (b1a76fff2a, verified ancestor) plus the toast render-loop fix --
the OAuth return stacked twelve identical cards and now raises one. It is
a strict forward move, not a swap.
Both images were pulled in-cluster with the fleet's own credentials before
this was written, because pinning an image the registry cannot serve has no
rollback path: v8.5.37 is 2383695 bytes, the superseded digest 2375128.
The discipline shifts rather than disappears. A digest cannot be re-pushed
to different bytes; a semver tag can, and :v8.4.118 was. So the rule is now
that a cut tag is never re-pointed -- cut the next patch instead.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The recipe handed each app 41000+index*10 and that was accidental complexity.
The question is "does this binary compose". Answering it does not require owning
a port namespace, a stride, or an index.
It also answered WRONG. A second run inside sixty seconds collided with the first
run's sockets in TIME_WAIT — which `ss -lnt` does not show, so the ports looked
free — and reported up to SIXTEEN healthy apps as DIED. A check that invents
failures gets ignored exactly as fast as one that misses them, and this check
exists because fifteen plugins reached production without one.
:0 on all four listeners deletes the bookkeeping, the stride, the TIME_WAIT
window and the concurrency cap in one move. The kernel already allocates ports
correctly; the recipe just had to stop doing it by hand.
Measured on this tree, same command, minutes apart:
before >> compose FAILED: 16 of 120 (15 of them "address already in use")
after >> compose FAILED: 1 of 120
The one is `kafka`, which fail-closes when no broker answers at
nats://127.0.0.1:4222. That is the app being honest about a missing dependency,
it reproduces identically on unmodified main, and it is not a composition fault.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.801.426 was pinned, rolled out and served traffic while the job meant to build
it sat Failed. The image had been built from OLDER source: two fixes reported as
shipped were not in the running binary, and nothing detected it, because there
was no way to ask the process what commit it was. /v1/health returned
{"status":"ok"} and nothing else, /v1/version was a 404, and no revision variable
existed in the Go source. Establishing the truth took exec-ing into the pod,
running the image's own /admin binary and reading its panic.
ONE fact, stamped ONCE, read through ONE function:
-X github.com/hanzoai/cloud.revision=<40-hex>
reported on the health payload every one of these processes already serves —
/v1/health on the product API, and /healthz, /readyz, /health on the ops listener
(CLOUD_HEALTH_LISTEN, the unauthenticated in-cluster read). No new route, no new
port, no new env var. The two surfaces built their bodies separately — one a map,
one a fixed byte string — which is exactly how a field comes to exist on one and
not the other, so both now build from healthBody and a literal can no longer miss
an addition.
MEASURED, on plugin/base linked with the flags the Dockerfile passes:
:18110 /v1/health {"revision":"d25b0f5e70f79bfb04ec60b1f535f20db9a62062",...}
:18112 /healthz {"revision":"d25b0f5e70f79bfb04ec60b1f535f20db9a62062",...}
:18112 /readyz {"revision":"d25b0f5e70f79bfb04ec60b1f535f20db9a62062",...}
and, linked with no -X at all, {"revision":"unknown"} on all three.
UNKNOWN IS A VALUE, NOT A BLANK. Only a full 40-hex lowercase object name is
reported; an unexpanded ${REVISION}, a branch name, "dev", a short sha and the
empty string all read "unknown" — measured on a real binary stamped `main`, which
serves "unknown". A value that is NEARLY a commit is worse than none, because
someone acts on it. That rule is cloud.IsCommit, and it is the rule the BUILDER
already applied before passing build-arg:REVISION, so apps/platform's private
isCommitSHA copy is deleted and calls it: the two ends of that wire cannot drift
into disagreeing about what they hand each other.
THE STAMP NOW REACHES THE STAGE THAT SHIPS. `ARG REVISION` existed already — in
the FINAL stage, feeding the OCI label, invisible to `go build`, because an ARG
is per-stage. The wire was connected at one end. Worse, the one -X that did exist
reaches nothing: cmd/cloud does not link the root package, so
`-X …cloud.Version=` on /cloud has always been dropped — measured, the flag is in
that binary's `go version -m` record and the value is nowhere in its bytes. The
PLUGINS serve /v1/health and they carried no -X whatsoever, so stamping only the
entrypoint would have left the answering process mute. Both facts now go into one
GO_LDFLAGS used by every binary in the image.
`-X` on a symbol that does not resolve is SILENTLY DROPPED, and under the rule
above a dropped stamp reads as the legitimate "unknown" — invisible, exactly like
the image revision LABEL that has read `unknown` in this fleet with nobody
noticing. Two things close it: the image greps its own linked binaries for the sha
(`go version -m` is not a witness — it echoes the flag that was REQUESTED, present
even when the symbol was never set), and version_test.go LINKS a real binary and
asks the process. The first draft of that test read the child's EXIT CODE, and a
deliberately mis-named symbol sailed through it green, because a test that skips
itself exits 0 like one that passes; it reads the child's OUTPUT now, and the
mutation fails it.
The Dockerfile's ARG sits as late as it can, below `COPY . .` — everything from
there down is already re-keyed by any source change, so a per-commit value costs
nothing, while the same value in scope above would re-key `go mod download` and
turn every build into a full one.
`make` builds report "unknown" on a dirty tree, correctly: REVISION comes from
`git describe --always --abbrev=40 --match='' --dirty`, and `-dirty` is not a
40-hex name, so an uncommitted tree cannot name a commit whose source is not what
was built — the same lie, locally.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three defects shipped today because the only thing that could see them was a
running binary and nothing ran one. Each is now the smallest program that
reproduces it.
scope_group_test.go — the group, which broke twice in one day, in opposite
directions:
- TestGroupWithLaterUseComposes. `g := app.Group(p); g.Use(mw)` while the
routes register through ZipApp on the ROOT. Group returning the raw router
hangs middleware on a node whose subtree is necessarily empty; zip refuses
that at boot, which is what crash-looped fifteen plugins. Asserted with
zip.App.Build — Listen minus the sockets, verdict returned rather than
thrown. RED against the raw-router Group with the exact production text.
- TestGroupPrefixesWhatRegistersThroughIt. The opposite kind: a program that
composes perfectly and answers somewhere else. A child Group must prefix
what registers through it down BOTH paths — the route methods and OpScope,
because `zip.Get(app.Group("/v1"), "/bots", h)` is a real idiom here — and
must leave an absolute path at the subsystem root alone. Asserted on the
composed route table, since no compose check can see a route that merely
MOVED. RED separately against each half of the fix.
- TestGroupUseOutsideThePrefixesFailsTheMount / ...IsAllowed. Confinement
through the door Group opened, and its limit: middleware at a prefix the
subsystem does not own installs nothing and fails the mount, while a BARE
group there is ordinary — a prefix is just a path.
plugin_surface_test.go — the two drifts, DERIVED from the specs the mains
declare (go/ast) and the routes the committed projections hold, so a subsystem
added tomorrow is checked tomorrow:
- TestHealthOwnershipMatchesWhatIsRegistered. OwnsHealth is a claim with two
halves. Claimed falsely, one address is declared twice and zip refuses the
program — reverting plugin/authz/main.go reproduces the crash verbatim:
`GET /v1/authz/health: declared by "authz" at serve.go:324 and by "authz" at
serve/mount.go:31`. Claimed while owning nothing, the address silently 404s.
The second half asks the manifest which health address is this app's rather
than assuming /v1/<name>/health — plan answers /v1/plans, storage /v1/s3.
- TestDeclaredPrefixesCoverTheSurface. A grant a main WRITES must cover the
surface it serves. One-directional on purpose: containment, never equality,
or deploy's 14-leaf row makes its own /v1/deploy bridge an escape. It checks
the half a document can answer; middleware has no address, so the other half
is `make compose`, which runs the binaries.
plugin/bot/main.go — bot declared OwnsHealth: true and registers no health
route anywhere, so the field's only effect was to suppress serve.go's generic
route and leave GET /v1/bot/health answering 404. Measured before: 404. After:
{"service":"bot","status":"ok"}. Nothing changes in the fleet, where /v1/bot/health
routes to `runtime`; this is the standalone binary's own liveness answer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The awk class was [a-zA-Z_-]+, which excludes digits, so `make help` silently
omitted e2e and e2e-ui — two targets that have existed all along and that nobody
browsing help could discover. One character.
Found by running `make help` in console and reading the output against the file,
which is the only way this shows up: the recipe succeeds either way.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The timeout is the cost and it is paid per app, so one-at-a-time made this most of
an hour for 120 apps. A check that takes an hour gets skipped, and a check that
gets skipped is how fifteen crash-looping plugins reached production in the first
place — the mechanism has to be cheap enough that it is actually used.
Each app already had its own data dir and its own port block, so nothing contends;
xargs -P just stops them queueing. Failures go to files rather than racing onto
stdout, and the summary names how many of how many failed. Measured: 120 apps in
minutes rather than ~50.
First full run on this tree: `>> compose: 120 apps boot`.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Aliases, not recipes: dev -> run, lint -> vet. Both targets already existed
and already did the right thing; only the names the rest of the fleet uses
were absent, so there is still exactly one recipe behind each.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip refuses to compose a program whose middleware could never run, and it
refuses at BOOT. So on v1.801.425/.426 fifteen plugins compiled, linked, vetted
and passed unit tests, then crash-looped in production. Nothing in this Makefile
could have caught that: the only thing that finds a compose panic is running the
binary.
SURVIVAL is the signal, and it is the only honest one. A compose panic is fatal,
so a process still alive when the timeout kills it (rc 124) composed. Grepping
the log for a success line does NOT work — `"message":"zip new"` is printed
BEFORE composition, and reading it as a pass is exactly how a broken build got
reported as shipped twice in one day.
Each app gets a writable data dir and its OWN four ports, because without them it
dies on `mkdir /var/lib/cloud/orgs` or on binding :8080/:9653/:9090/:8081 long
before it reaches the router — and an early death looks like silence, which reads
as a pass. That mistake is why a "16 binaries, 0 panics" check was worthless: the
binaries had exited before composing.
It reuses `apps`, so there is one way to build an app binary and no second
mechanism. `clean` takes the scratch dir with it.
Proven both directions on this commit:
GREEN make compose APPS="admin prefs" -> >> compose: 2 apps boot
RED inject `ZipApp(app).Group("/v1/prefs/zzz").Use(Bridge())` into prefs
-> PANIC prefs
zip: the group "/v1/prefs/zzz" declares middleware at
prefs/prefs.go:147 and no routes anywhere beneath it
-> >> compose FAILED, make exits 1
(The injection is needed because every real call site is already fixed — deploy's
was corrected by bd2c284e, so it can no longer reproduce the fault.)
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/console d761fbc. Clicking "Sign in" on cloud.hanzo.ai appeared to do
nothing: console.hanzo.ai/ served a SECOND copy of the Hanzo Cloud marketing page
wearing the byte-identical @hanzogui/shell header, so the click landed on a page
indistinguishable from the one it left and read as a re-render. Reaching hanzo.id
took three clicks, two of them through pages that only asked "did you mean it?".
The console is the application; the marketing face is cloud.hanzo.ai. `/` is no
longer a special surface — an anonymous visitor STARTS the authorize hop.
Pinned by digest, not the usual sha-<sha7>-amd64 tag: that build published to
:latest, which is the moving target this file already warns about. The digest
names these bytes and no other.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The billing gate died of a peer call that was an HTTP URL. COMMERCE_URL defaulted
to the public api.hanzo.ai edge — which is THIS binary — so the /v1/billing/*
forwarder re-entered itself, and apps/commerce/transport still carries the scar
tissue: a whole app republished as an http.Handler, every edge middleware re-run
per peer read, and a goroutine-keyed depth counter (maxDepth = 8) to stop the
recursion it cannot otherwise prevent. A call by name cannot express that mistake,
because there is no address to point at the wrong thing.
An app already declares everything a caller needs:
zip.Post[plane.BalanceIn, plane.Balance](cloud.Plane(), "/finance/balance",
planeBalance, zip.WithOperationID(plane.FinanceBalance))
— the app, the op, the request type and the response type, in one expression.
Cloud already projects that registry as OpenAPI, a CLI, an MCP tool list and a
routing declaration. A typed Go client for a peer call is ONE MORE PROJECTION of
it, which is why it is generated here rather than hand-written once per caller.
commerce.FinanceBalance(ctx, &plane.BalanceIn{Currency: "usd"})
plane/gen emits one package per peer (14 apps, 28 ops) holding ONLY request and
response types and call stubs. What it buys is a check no care buys today: a
hand-written peer call is four independent facts that must agree at RUN time, and
nothing stops pairing commerce's op with iam's name, or BalanceIn with Txns. The
wrapper fixes all four to each other where they are declared.
THE CLIENT HALF MOVED TO THE LEAF, and that is what makes any of this possible.
package cloud is itself a caller — the edge rate-limiter reads finance_scope_rules
— so a client that imported cloud could never be imported BY cloud, and the one
call that most needed to stop being a URL is the one the mechanism could not have
expressed. Ask and everything under it now live in package plane; cloud keeps the
server half (Plane, ServePlane) because binding a socket reports itself to o11y.
cloud.Ask stays as a forwarder, so the 44 existing call sites do not move and
there is still exactly one implementation.
It does not drag the peer's tree: plane/commerce is 355 packages against
apps/commerce's 1231, and imports zero apps/ packages — one more than the leaf it
needs. An importable client that linked the implementation would have rebuilt the
problem with extra steps.
Generated FROM SOURCE, judged BY THE RUNNING REGISTRY. zipdoc already reads these
same call sites; reading source buys hermeticity a mount cannot (no store opened,
no boot order, no app that must come up before it can be described). zip's rule —
project from the live router, never the AST — is about a host discovering a plugin
it does not build, and it still binds: plane_registry_test.go mounts commerce and
asserts the generated surface IS the live plane registry, so the generator never
gets to be quietly wrong. Reading the AST's index expression alone had already
been quietly wrong once — treasury spells its registration with inferred type
arguments, so its only op was dropped; types.Info.Instances sees both spellings.
Proven against the real thing, not a fake. plane_client_test.go mounts commerce as
a plugin process does, binds its plane socket as Serve does, and calls the
generated function: commerce answers amount="0" currency="USD" over the socket. A
cold peer with no router answers ErrNoPeer naming the app — a named absence, never
a timeout a caller would have to guess at.
Three root-package call sites converted, including both money ops — the prepaid
gate and the meter now reach commerce as commerce.FinanceAuthorize and
commerce.FinanceRecord. Those are the imports that were structurally impossible
before, so they are the proof the direction is real.
Full suite: 133 failing test names / 26 packages, byte-identical to the same
measurement on origin/main. Regression set EMPTY. go vet ./... exit 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/graph was never a graph database. Its package doc says so: "chain data:
your block indexers and how far each has caught up, plus the on-chain price
feeds." It proxies luxfi/indexer (explorer REST) and luxfi/graph (GraphQL) --
"graph" here meant GraphQL, not a property graph.
Both upstreams already mount under /v1/explorer (client.go: graphd default
prefix), so the name follows the contract the app already speaks rather than
inventing one.
Wire unchanged: the app keeps /v1/indexers and /v1/oracles, keeps its frozen
mount position, and the woven document moves only x-app and the tag prose.
References to luxfi/graph -- GRAPH_URL, graphQLPath, the upstream log key --
stay, because those name the upstream, not this app.
/v1/graph is now free for the embedded per-org graph layer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The self-hosted-git allowance existed, was correct in intent, and could never
match. `selfGitHost` was `deps.Domain` verbatim — "api.hanzo.ai" — while
hostAllowed grants that host or a SUBDOMAIN of it. The forge is "git.hanzo.ai":
a sibling. So every native build was refused with
repo.url host "git.hanzo.ai" is not an allowed git provider
and the estate fell back to building from GitHub, which is exactly what broke
when hanzoai/cloud moved to hanzo-inc/cloud and the build credential lost access.
The code's own comment already said APEX; only the assignment disagreed.
selfGitHost is now the registrable apex ("hanzo.ai"), so every sibling the
deployment owns — git., ci., cd. — is a trusted build source by construction,
for hanzo.ai, lux.network, zoo.network and any white-label domain, with no
per-brand list to maintain. publicsuffix rather than "last two labels" so a
multi-label suffix (co.uk) yields the registrable domain and not the suffix
itself, which would trust every domain under it.
Proven both directions: TestSelfForgeIsAllowedFromTheApex admits
git/ci/cd.hanzo.ai and still REFUSES git.evil.com;
TestOldDomainVerbatimRefusedTheForge pins the old behaviour as the defect, so
this cannot silently regress.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/runtime was never a second product. It was the transport to @hanzo/bot —
the TS service that executes a run — plus a relay of that service's own ops
paths at /v1/bot/*. What separated it from apps/bots was a LANGUAGE boundary,
Go surface and TS executor, wearing the shape of a product boundary. Both
answer for the same thing: a bot doing your work on a real desktop.
So they merge. runtime.go becomes transport.go, ops.go becomes relay.go,
bots.Mount mounts both faces, and the manifest holds one row for the pair.
THE WIRE DOES NOT MOVE. Every path this fleet serves is the path it served
before — the whole diff to openapi.yaml is `x-app: runtime` -> `x-app: bots`
on the seven relayed operations. No CLI, SDK, MCP tool or doc regenerates to
a different address, because none of them has a different address to go to.
Two things the merge had to earn rather than assume:
- apps/coding also dispatches to that executor, so it followed the transport
from apps/runtime to apps/bots. That is not new coupling wearing a new
name: coding runs its tasks ON the bot runtime, which is what the import
now says out loud.
- the typed-or-named gate was TWO gates, one per old package, each blind to
half of what is now one surface. They are one gate over the whole product,
and mountWith mounts the relay so the gate can actually see it. Two gates
stapled together would have kept passing while measuring nothing.
/v1/bot is still shared with apps/bot, whose three deeper prefixes win on it by
specificity. That sharing is the remaining defect and it is not this commit's
to fix: apps/bot's product is connected machines, not a bot, and it is the one
that has to vacate.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip exported TWO things named Mount and they were unrelated:
zip/remote.go func Mount(prefix, addr string, decl ...Declaration) (*App, error)
112 subsystems func Mount(app cloud.Router, deps cloud.Deps) error
The second is this repo's registration contract, now enforced by the compiler
through cloud.MountFunc. The first was a leftover — one of five composition
verbs (Listen/Mount/Add/Graft/Use), the other four dissolved into Use, and this
one survived only by being the one that pointed at another process.
Renamed upstream to Proxy, which is what it BUILDS: every route it registers
runs one handler whose whole body is forward() to the given address. It is also
a noun, which a function returning a value should be. Not Remote — zip.Remote
already exists and is the CALLER's side; this is the SERVER's side, a stand-in
inside this program's own routing table.
Source-compatible here: cloud had no zip.Mount call site, only the one comment
in cmd/cloud, which now names the function that exists.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`type Store` is declared 40 times under apps/ and 35 of those are structurally
IDENTICAL — `struct { db *sql.DB }`. That number is a trap and I am not acting
on it: in Go, git.Store and bot.Store are different types because they are in
different packages, each carries its own app's schema in its own methods, and
"consolidating" a one-field handle wrapper would produce `type Store struct{
shared.DB }` — a rename that moves no code and removes no duplication. 35 of 40
share a shape; ~0 share a MEANING. Optimising that number would be pure damage.
The duplication the number was pointing at is one level down, in the
CONSTRUCTOR, and it is exact. 33 stores open themselves with byte-identical
code modulo their own name:
db, err := cek.Open(namespace.System(), "<app>", dir)
if err != nil { return nil, fmt.Errorf("open <app> store: %w", err) }
sqlpool.Single(db)
Those three lines are a PAIR that nothing paired. sqlpool.Single's own doc says
a two-statement read-modify-write (tracker's per-project issue number, agents'
MAX(seq)+1) is atomic ONLY because no second connection can interleave — so the
cap is a correctness requirement, and it was a separate call every caller had to
remember. 33 remembered.
apps/framework did not. It opens a cek database through an engine OpenDB
callback and never capped it, so its DocType store has been running with an
uncapped pool. That is the defect, and it is the one a 33-way copied prologue
exists to produce: the rule holds until someone writes the 34th store.
sqlpool.Open is the rule moved INSIDE the opener, so there is nothing left to
forget — the same argument as cloud.App being the only way to obtain an app.
36 call sites converted, framework included; the stores that open differently
(git takes a *sql.DB from the OrgStore cache, others carry extra migrations)
still do, because they are different and the point is not uniformity.
Measured, not assumed: shapes compared by AST field set, not by name. Of the
other collision families the brief named, Result (9 decls) and Config (10) have
ZERO structurally identical pairs, and Client has exactly one family of 3
(`base string; http *http.Client; token string`). Those are Go's package
qualifier working, and they were left alone.
Regression set EMPTY by name: all 15 failures in the touched packages
(TestDocTypeAndDocumentRoundTrip, TestForkCreatesProjectFromTemplate, the Redeem
family, …) fail identically on forge/main.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
All five diverged from the canonical Mount signature, and the divergence was
invisible because nothing checked it. Now MountFunc does (see the parent commit),
so these are what the compiler demanded.
agent, ai, commerce took *zip.App, so they had to be wired through the
old App field — which ALSO granted app-wide middleware.
They now take cloud.Router and reach the concrete app
through cloud.ZipApp, the named hole for it.
agent installs no middleware at all (hanzoai/agent
calls Use nowhere) and now mounts SCOPED: it had the
whole binary's gate to buy a typed-op registry.
ai and commerce genuinely do gate everything —
commerce wraps all of /v1 — and now DECLARE it as
Plugin.Global at their composition root, where a
capability belongs, instead of implying it with a
parameter type.
dataroom imported cloud as `hcloud`. The alias is cosmetic to
the compiler and fatal to every grep-based check —
which is how these five stayed invisible. One name.
treasury/anchor not a Mount, but the same disease one level down and
the cleanest Pike case in the repo: status() took
ledger.Backend — ELEVEN methods — to call Root and
nothing else. The parameter said the anchor could
accrue revenue, seed the reserve, debit a program and
rewrite the revenue-share policy. It can do none of
those. It now takes a one-method `rooted`, declared by
the consumer, which both backends satisfy without a
line of change.
RETRACTED: rollingcap's `func Mount(_ cloud.Router, _ cloud.Deps) error` was
reported to me as the fifth violation, possibly dead. It is neither. It installs
the rolling AI-spend cap reader — live, and load-bearing — and `_` is CORRECT Go
for parameters it genuinely does not read. It already IS a MountFunc; the
compiler says so. Naming those parameters to satisfy a text pattern would make
the code worse. Left exactly as it is.
Regression set EMPTY by name (148 failing tests before, the same 148 after).
apps/commerce's TestBalanceCents and TestInProcessClient fail identically on the
parent commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two names in this file each meant two things, and both cost the fleet something
real.
1. Plugin had TWO mount fields with TWO signatures
Mount MountFunc // func(Router, Deps) error
App func(*zip.App, Deps) error // "for a subsystem that gates everything"
App braided a POLICY question (may this subsystem install middleware over the
whole binary) into a TYPE question (what shape is its entry point). Because the
grant carried its own signature, a subsystem could take the grant just to get
the concrete *zip.App — and three did. agent, ai and commerce each held app-wide
middleware authority they never asked for, purely because their Mount named a
concrete type. hanzoai/agent calls Use nowhere; agent's grant bought it nothing
and risked everything after it in the mount order.
Nothing reported this. The field's own doc claimed "apps.TestWireFrozen fails on
a new one"; no such test exists anywhere in this repo — grep it. A contract
stated in a comment and violated five times means nothing is checking it.
Unbraided: App becomes `Global bool`, and every subsystem — scoped or global —
goes through spec.Mount. The grant now decides only WHICH Router arrives (the
bare app, or a scope bound to declared prefixes). One shape, so MountFunc is the
whole enforcement and it is the compiler: at 123 composition roots a divergent
Mount cannot be assigned and cannot link. It found a sixth violator I had missed
by grep on the first build — cloud.MountMetrics, whose doc says "adapts
hanzoai/metrics into a MountFunc" while its signature took *zip.App. It is one
now, and mounts SCOPED (hanzoai/metrics installs no middleware either).
The concrete app stays reachable through ZipApp, the named hole that already
existed for exactly this and reports nil rather than pretending.
2. Router restated zip.Router instead of embedding it
Ten method lines were COPIED here. Copying an interface makes cloud a second
place zip's routing surface is defined, and the two agree only while someone
keeps them agreeing. When zip v1.23 widened one signature (Use took Component,
not Handler), every implementor that had spelled the methods out had to move in
lockstep — which is what stalled v1.19+ adoption across the fleet.
Embedded, a zip routing change costs this file zero edits, and Fiber() +
Plugins() are visibly what cloud ADDS rather than being buried among ten lines
cloud merely echoes. It carries zip.OpTarget along, which is a gain, not a
widening: scope, *zip.App and commerce's mintRouter all already have OpScope,
and a Router that IS an OpTarget is one zip.Get[In, Out] accepts directly.
Regression set EMPTY, compared BY NAME against a pristine worktree at the same
commit: 148 failing tests across 29 packages before, the identical 148 after
(comm -13 on the sorted name sets is empty in both directions).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerceTierToLadder allow-listed only `starter`->trial and `pro`/`enterprise`
->paid, and folded EVERYTHING ELSE to free. The plans commerce actually sells
are go ($9), dev ($19), pro, max, team and business — so a subscriber on
go/dev/max/team/business was scored free, then refused every SKU carrying a
trial or paid floor AND throttled by the free-tier flash cap on top. We took
their money and shut the door.
v1.832.29 inverts the default: a plan is PAID unless it is explicitly free
("", "free", or a `*-free` suffix), with starter/trial the one middle rung.
That direction is safe because A TIER IS NOT A PAYMENT. filter_balance.go has
no exemptions and fails closed — "nothing runs on credit it has not been funded
for" — so an unrecognized plan reaching `paid` still cannot spend a cent it has
not been funded. The OLD default was the dangerous one: it silently cost
revenue every time anyone added a plan slug. Now, add a plan and it works.
Both tests had encoded the defect as intent ("developer"->free,
"mystery"->free) and were corrected to the real plan list. 22 packages ok,
0 FAIL at the module.
Also carries the zen seed fix (v1.832.24) and the object TestMain fix
(v1.832.27), which took that package from ZERO tests executing to 153 PASS —
TestMain called os.Exit(0) before m.Run(). That immediately surfaced a
crawl-storage default still naming the retired MinIO Service, a correction that
had been made twice and could never take effect.
Verified: `go build ./...` clean at this pin.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A signed-in customer got 401 listing their own cards, and the checkout's
prefill failed on every load: cloud's billing app forwarded /v1/billing/methods
to commerce over HTTP, and that proxy is unconfigured here
(CLOUD_COMMERCE_HTTP_URL is unset), so the customer address for saved cards has
never worked on this deployment.
An internal HTTP hop to a service compiled into the same binary is the wrong
shape whatever its config, so the three verbs move to the commerce app and are
served in-process on the same pinned-subject chain as their portal twins — the
gate that keeps a caller inside its own account whatever it sends. The prefix
moves with them, because the router must deliver where the handler lives.
Also drops /v1/commerce/deposits and /v1/commerce/webhooks from the manifest
and the published document: the broker-dealer proxy behind them is deleted, and
a manifest that claims an address nothing serves is how a path silently routes
to the wrong app.
The routing oracle is what found all of it — both halves, in both directions.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Per-org KMS hydration runs only under KMS_ENABLED, unset here, so the rail
answered "not configured" with every field stored correctly. v1.50.1 falls
back to deployment env, which universe now supplies from commerce-secrets.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`p.Status = "error"` on a non-live completion was UNCONDITIONAL, so "this project
is broken" was reachable from any deployment row the org could name — including
one that had already been superseded.
Found by triggering it on hanzo-ai, not by reading it. v5 was live with all 8401
objects correct in the bucket; completing v3 — an older probe deployment left
queued during the Sites-plane migration — as `error` flipped the PROJECT to
"error", and the sites edge stopped serving. Nothing was wrong with the site. The
bytes never moved. A status field on a superseded row took the host down, and
re-completing v5 as live brought it straight back.
The ordinary production shape is the same bug with worse timing: a rebuild that
fails while the PREVIOUS build is still live. The old content is still being
served and the site is up, yet the project gets marked broken — and the "report a
failed build" step that every CI workflow carries (so a dead build cannot leave a
deployment queued forever) is exactly the thing that would send it.
The rule is now named rather than inlined: failureOwnsProject(currentDeploy,
deployID). Error propagates to the project only when the deployment IS the one the
project points at, or when it points at nothing — a first deploy that fails leaves
a project that has genuinely never served, and that one should read as error. The
deployment row is still error in every case and LifecycleDeployFailed still fires,
so the failure stays visible where it belongs: on the deployment, not on the health
of a site that is up.
The `live` branch already recorded p.CurrentDeploy, so the information needed to
make this distinction was there the whole time and simply was not consulted.
TestFailureOwnsProject pins all four cases. Mutation-checked: restoring the old
`return true` fails exactly the two that matter — the superseded deployment and
the failed rebuild — so the test cannot pass against the behaviour it exists to
prevent.
(TestDeleteStays204WithNoBody fails on darwin both with and without this change:
the SQLCipher codec refuses to decrypt without tmpfs, and macOS has no /dev/shm.
Pre-existing and unrelated.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A plan-refused caller was told to request limited-preview access — a
door that cannot open when the plan is the blocker. Each family gate
now speaks its own sentence: upgrade for the subscription floor, paid
capacity for the funding floor, request-access only where a grant is
truly what is missing.
The edge resolved an extension-less path to two candidates — the exact key and
`<rel>/index.html` — and Next.js `output: export` writes neither for a normal
route. Without `trailingSlash` it emits the FLAT file `pricing.html`, so the
homepage served and every other route 404'd.
Measured on the hanzo.ai export (759 pages, 8402 objects) published to
hanzo-ai.hanzo.app:
/ 200 /pricing 404 (/pricing.html 200)
/zen 404 /zen.html 200
/zen/models 404 /zen/models.html 200
That is the shape of a site that reports `status: live`, serves its homepage,
and is unusable — the failure is invisible from the deploy and from the root URL.
`<rel>.html` is added as a middle candidate. Both spellings are legitimate and
both are now tried: Hugo, Jekyll, Vite MPA and `trailingSlash: true` emit the
directory-index form, Next's default emits the flat form. The alternative was
setting `trailingSlash: true` in every repo, which pushes a server limitation
onto each site and rewrites every canonical URL to work around it — this is one
extra HEAD miss on the paths using the other convention, in one place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The embedded family catalog no longer gates any enso SKU, so discovery
stops advertising a waitlist and ai serves every caller without a grant.
The app builder's default model refused everyone outside the launch org;
now it answers.
The envelope's RAM-backed guard now recognises a native macOS tmpfs
(sudo mount_tmpfs <dir>, then HANZO_SQLITE_RAMFS_DIR=<dir>), so the
cek-backed suites can run on a Mac instead of failing on every commit.
Fail-closed is unchanged: anything statfs cannot verify as tmpfs is
still refused.
Record.NeedCommit carried a xorm-style `db:"index"`, and dbx reads that tag
as the column NAME — so the column was called `index` while
ScanNeedCommitRecords queries `need_commit`. Live in this pod: 'no such
column: need_commit' every five minutes, and the record-chain commit task had
never committed a record.
Also carries a guard for the class, placed in routers rather than object
because object's TestMain os.Exit(0)s that package without a seeded database —
a test there reports ok without ever running.
hanzo_service_up is now recorded a second time, through Metrics() — the
registry the framework exports natively — under the exact series name and
label the meter pipeline has always published. Two producers, one probe, one
truth: every rule and reader sees an uninterrupted series whichever road the
sample travelled, which is what lets the old road be retired without a gap
once every reader is confirmed on this one.
The verdict is recorded where it is decided: the probe client's transport,
which already tells each target's reason on change. A nil registry skips
recording — a test that mounts probes without one measures the probing, not
the export.
zip moves to v1.24.6 for Metrics(), which also brings the native span and log
export and the convention that finds a collector with no configuration.
hanzoai/ai is a library this binary links, not a service of its own, so the
crawl4ai leg it dropped stays live in production until this pin moves. v1.832.25
points /v1/crawl at apps/crawl.Fetch — the same guarded dialer the answer
engine's read stage uses, refusing non-public addresses on every hop including
redirects — instead of dialling crawl.hanzo.svc.cluster.local:11235, a name that
does not resolve and returned success:false for every request.
The survey shipped with the loop steerable by the pages it read. Its decision
prompt carries titles and URLs crawled from hosts we do not control, so the move
that comes back is partly authored by them — and nothing checked it. Three
consequences, all reachable by getting one page into the ranked set:
a `read` URL was fetched whether or not the survey had ever gathered it, so a
page could point the cluster's egress at any address and carry the user's
question there in a query string, then have the answer filed in the tenant's
corpus;
a round's `queries` list was unbounded — mode.maxQueries bounded round zero and
nothing else — so one move could issue hundreds of serial meta-searches from a
shared egress that is already bot-challenged;
page text was spliced into the numbered source block unfenced, so a body
containing `[9] Title\nURL\nbody` became a source indistinguishable from a real
one, and nothing validated that the URLs the answer cited were sources at all.
CONTAINMENT, NOT INSTRUCTION. A prompt is advice to a model. These are properties
of the text:
`read` is intersected with the pool the survey itself gathered, and `queries`
and `read` are clipped where the untrusted value crosses into the loop;
ground.go fences each source with a per-request nonce the page cannot know,
because it was written before the request existed;
every markdown link in the answer is checked against the gathered set — on the
stream and in `done`, through the same function — so a citation always points at
a page THIS request fetched. An ungrounded link keeps its text and loses its
target: the sentence still reads.
Two bounds that could not bind now do. tokenCeiling measured the survey's own
subtotal while the plan and the synthesis — the expensive calls — sat outside it,
so research's 400k ceiling was ~50x the reachable total; it now takes the
request's running total. And plan, survey and synthesis shared one 300s clock
with no reservation, so a survey that spent it handed a full corpus to a
completion that could not start and the caller got "the model is unavailable"
after five minutes; the gather now gets 70% and synthesis keeps the rest.
Also fixed, and each one a bug on its own:
unread MARKED every URL a move named while read FETCHED only the first six, so
a move naming ten pages blacklisted four without ever fetching one — and the
round then tripped saturation and ended the survey early, losing evidence on
exactly the runs working hardest. The cap now comes before the marking.
Source.Text carries the fetched page and Source.Snippet stays the search
summary. Reading no longer overwrites what the wire shows, so a research answer
stops re-sending up to a megabyte of duplicate page text across twelve
snapshots, and one frame per round is now the whole story.
A round's queries run concurrently. Serially, at 12s each, a six-query round
spent most of the answer's wall clock waiting.
An unreadable move gets ONE stricter reprompt. A model that opens with "Sure!
Let me look at the JVM next" collapsed research into a single pass, and nothing
downstream could tell that from a model that judged the evidence complete.
A client that hung up ends the run: one research answer costs five minutes,
three dozen fetches and up to eight completions, and a closed tab bought all of
it. The plan's topic headings now reach the client as a planning detail, which
is what makes a three-minute wait legible. Crawl and completion failures are
logged — a degradation nobody can see is a degradation nobody can fix.
The package doc claimed METERED ONCE. It is not true: build.go hands this engine
a metered AI plane, so every internal completion also debits the payer per token
alongside the flat fee. Which layer should price /v1/ask is an open decision,
recorded rather than claimed away.
The SearchEvent union is unchanged — same variants, same keys, same order.
v1.801.445 failed to build, twice, on `go mod download`:
github.com/hanzoai/commerce@v1.49.67: reading go.mod at revision
v1.49.67: unknown revision v1.49.67
v1.49.67 is a PHANTOM. It exists on no remote: github.com/hanzoai/commerce
carries exactly two tag refs (v1.50.0), and git.hanzo.ai/hanzoai/commerce
lists ...v1.49.65, v1.49.66, v1.49.68 — .67 is skipped. It survives only in
warm module caches, which is why .442/.443/.444 built with the same pin: the
BuildKit `cloud-gomod-v4` cache is PER NODE, .444 landed on a node that still
held it (runner-pool-32g-3mn0fk) and .445 landed on one that did not
(runner-pool-32g-3mnls1). The release train was therefore passing by luck of
placement, and any cold node broke it — exactly the failure the Dockerfile
comment above `go mod download` anticipates.
Nothing here can be fixed by retrying. The Dockerfile sets
GOPRIVATE=github.com/hanzoai/* with GOPROXY=...,direct, so hanzoai modules
resolve DIRECT FROM GITHUB, and v1.50.0 is the only commerce version GitHub
has. This is an upgrade, not a workaround: commerce was re-published under
MIT OR Apache-2.0 and cut v1.50.0, and go.mod simply had not caught up.
Verified rather than assumed: `go build ./...` is clean against v1.50.0, and
v1.50.0 carries the same 1875 dirs and 8 billing packages as .67 including the
Cloudflare 502 work (thirdparty/cloudflare, api/billing) — the ~134 fewer .go
files are the tests/enterprise the OSS publish strips.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.832.24 stops seeding an admin/zen provider row that pointed at do-ai's base
https://inference.do-ai.run/v1. familyProvider reads that row as an operator
override of ZEN_URL, and the family paths append their own /v1, so every zen
catalog refresh hit .../v1/v1/models and 404'd — no zen SKU was listed or
served on api.hanzo.ai, once a minute, silently. Sibling enso was never seeded
and never broke.
The seed also self-heals ProviderUrl on every boot and re-creates a deleted
row, so this was not fixable in the database; v1.832.24 drops the seed entry
AND prunes the existing row, scoped to the exact stale shape.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
money.Amount.Minor() rescales, and hanzoai/decimal's Rescale rounds
HALF-AWAY-FROM-ZERO (decimal.go:145). Both balance call sites carried a
comment saying it "truncates toward zero". It never did.
So a balance of 4.995 USD read as 500 cents. hanzoai/ai's transaction check
is `avail < priceCents`, and apps/billing/gpu_charge.go is
`available < req.AmountCents` — both admit a 500-cent charge against a
balance that cannot cover it. The debit that follows is exact, so the
difference lands as a negative balance nobody authorized. Half a cent, every
time, with no error to find.
plane.Money.FloorMinor is that rounding made once, where the wire conversion
already lives, for every caller that COMPARES rather than debits. Minor()
refusing a sub-unit amount is still right for a debit and is untouched.
big.Int.Div is Euclidean, so a negative balance floors away from zero rather
than drifting back toward solvency the way truncation would, and the scale
comes from the currency exactly as Minor() takes it — cents are a property
of USD, not of money.
The guard that existed for this GREPPED ai.go for `a.Minor()` and for the
sentence claiming truncation, so it confirmed the wrong claim was still
written down. A test that reads the source cannot notice the sentence is
false. plane/money_test.go asserts the arithmetic instead — including the
4.995 case that was admitted, the real prod balance, and both signs. What is
left in apps/ai is the one thing only that package can say: that its gate
still asks for the floored figure.
Also corrects apps/billing/balance.go's comment calling this number "a
DISPLAY, nothing is billed from it". It is `available` on
/v1/billing/balance, which is what ai's balance gate reads over the S2S HTTP
path, and what gpu_charge.go compares against a GPU's price.
Verified: money.Amount.Minor()=500 vs FloorMinor()=499 for 4.995.
apps/billing's TestGPUCharge_* fail identically at origin/main -- they refuse
to run without a tmpfs (/dev/shm), which macOS has not; unrelated to this.
Deep research was the one thing the answer engine could not do. `research` planned
sub-queries, searched them ONCE, read six pages, and wrote. A question whose answer
is only visible after the first round of reading — which is what "research" means —
got a search with a longer prompt.
SURVEY is the value that was missing: search and read applied to a plan, and
ITERATED. Each round asks the model for one compact JSON move (`next`, `queries`,
`read`, `done`), runs it, and discards the prose that came with it — gathering is
not writing. No tool plane is needed or used, and a model that answers in prose
ends the survey instead of derailing it.
`rounds == 0` is the single pass, byte for byte. search/news are untouched: same
queries, same one-per-host set, same 90s, same 2c. One code path, parameterized by
a mode value — not a second engine, and not a second route. /v1/ask stays the one
door and `mode` stays a value handed to it.
Bounded four ways, because an unbounded agent loop cannot be gated by Bill.Gate
before it runs and an ungated loop on a per-org ledger is a money bug:
rounds the mode's budget, hard-capped at maxRounds=8
deadline per-mode wall clock (search 90s, research 300s)
tokenCeiling per-mode token spend (120k / 400k)
saturation a round that found no new source and read no new page
Saturation replaces the `len(srcs) >= maxSources` guard the design called for.
That test cannot do the job it was written for: rank() already caps the set at
maxSources, so it fires on the FIRST productive round and collapses research back
into the single pass the survey exists to iterate. "No new evidence arrived" is
the bound that was meant, it cannot be satisfied vacuously, and it also terminates
a model that keeps proposing a query it has already run.
Every exit lands on the same return, and the caller always goes on to synthesize
whatever was gathered — the envelope reaches `done` from every path.
The contract did not move. status | sources | text | follow_ups | done, four
stages, no fifth: the round's next step rides as `status{planning, detail}`, and
per-page reading progress as `status{reading, detail:host}` — which the union has
always declared and nothing used to emit. `sources` stays a CUMULATIVE snapshot
because all three SDK consumers replace their list on it.
Alongside, four things the port made necessary:
rank takes a per-host cap. One page per host is right for a six-source answer,
where breadth is the value, and wrong for research, where three pages from
an authoritative domain are the point. hostCap<=1 reproduces the old set.
rank cleans `[PDF]`/`(Official Site)` furniture off a title so a citation reads
as a document name — keeping the original when a title is entirely
bracketed, which would otherwise cite the page by bare hostname.
read takes the url list and the clip from its caller instead of a mode's top-N,
and reports progress per source. An iterated survey clips to 3000 runes:
many sources x a long page is the one way this loop could overrun a window.
synth streams through a joiner that holds a markdown link until its closing
paren, so a citation never renders as `[Rich Hickey](htt` and rewrite
itself. Delivery only — the finished text is identical.
research is repriced 10c -> 25c. It now gathers across rounds, decides between
them, and reads more than once; the price follows the work.
Not ported, deliberately: code interpreter, X search, chart artifacts, images, and
every external-SaaS leg (Tavily, Exa, Firecrawl, Notte, Supadata, Daytona). Search
is apps/websearch in-process and keyless; reading is apps/crawl, SSRF-guarded, with
its archive->fetch->headless ladder already standing in for Exa->Firecrawl->metadata.
Provider reasoning tokens are never mapped to `text`: every consumer appends delta
into answer and would corrupt both the answer and the persisted training sample.
Round decisions want temperature 0. cloud.ChatRequest carries no temperature field
and inventing one would fork the AI contract for a single caller, so the prompt
buys the determinism instead. Flagged, not faked.
Tests: survey_test.go proves rounds==0 is the old loop, that the gather actually
iterates (a later round searches and reads what round zero did not, and round
zero's page survives the re-rank), that the plan is carried verbatim into every
round, that snapshots are cumulative, that a round's prose never reaches the
answer, and that each of the five exits leaves the frame sequence intact.
stream_test.go adds the wire ordering invariant and proves the server never emits
the union's `error` variant.
v1.832.24 stops seeding an admin/zen provider row that pointed at do-ai's base
https://inference.do-ai.run/v1. familyProvider reads that row as an operator
override of ZEN_URL, and the family paths append their own /v1, so every zen
catalog refresh hit .../v1/v1/models and 404'd — no zen SKU was listed or
served on api.hanzo.ai, once a minute, silently. Sibling enso was never seeded
and never broke.
The seed also self-heals ProviderUrl on every boot and re-creates a deleted
row, so this was not fixable in the database; v1.832.24 drops the seed entry
AND prunes the existing row, scoped to the exact stale shape.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The rail's "could not generate a deposit address" was a 502, and Cloudflare
replaces an origin 502 with its own HTML interstitial — so the customer read
"request failed" while a clear JSON message sat at the origin, unreachable.
v1.49.68 answers 503 (which passes through, as the wire rail's own 503
already proves) with a sentence naming the rail, saying it is temporary, and
pointing at the alternatives.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`g := r.Group(p); g.Use(mw)` is the form that panicked — the group carried
middleware, the routes were registered on the root, and zip refused to compose
a guard that could never run. It is fixed, and it was the only one of the three
Use idioms with no test. Eleven production apps write it (books, captable,
commerce, company, compliance, dataroom, framework, git, legal, risk,
validators); the tests covered the other two.
Reverting scope.Group to hand back the raw router reproduces the original panic
verbatim and turns TestGroupThenUseGuardsOnlyItsSubtree red, which is the point:
asserting only that the panic is gone would also pass on a scope that silently
dropped the middleware, and a dead guard is worse than a panic because the panic
is honest. Each case asserts both halves — the guarded path answers 401, its
unguarded sibling answers 200 — across three subsystems, one with declared
prefixes and one nesting a group inside a group.
Addressing is asked separately from gating, because a gated route answers 401
whether or not it exists, so a test that asked both at once could not tell "the
route is where I said" from "the guard refused a 404".
And the residue is written down: a scope installs at the root, so its middleware
runs for what follows it. Use before the routes guards them; Use after them
guards nothing, and it composes either way because the root always has routes.
apps/company depends on exactly that — its fundraise/deck leaf sits above
g.Use(limitBody) on purpose, since the deck is document bytes and a JSON body
cap on it would be wrong. That deliberate exemption rested on an unwritten rule.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
INERT IN PRODUCTION. CLOUD_WRITER_LEASE is unset everywhere and this change
does not set it. With it unset every path here is a no-op, the lock file is
never created, and boot is byte-identical to what is running now. The env flip
and the strategy change remain a separate, human-gated step. Nothing to watch
on deploy.
The lease states a POD-level fact — "this pod owns this volume" — with a
PROCESS-level primitive, flock(2) on {DataDir}/.writer.lock. Those are the same
sentence only when exactly one process per pod reaches for the lock, and cloud
stopped being one process: cmd/cloud is a router that spawns every subsystem as
its own child, all sharing one DataDir.
The acquire lived in cloud.Listen — the body every PLUGIN binary runs, and the
one thing the router never calls. So turning the lease on aimed a single-holder
lock at the siblings instead of at the other pod. kms won the flock; pubsub and
kafka waited out the 90s fail-closed budget; nothing bound :8000; the liveness
probe killed the pod and its replacement deadlocked identically. api.hanzo.ai
served 503 from 08:52Z to 08:56Z on 2026-08-04.
The repair after that (aa416cc6) made the children skip the lock and stopped
the deadlock. It also left the lock in nobody's hands, because the router does
not run that code at all — an interlock that logs as armed and holds nothing.
That is the worse of the two states: the deadlock announced itself, whereas a
lock held by nobody is quiet until someone believes the log line and switches
to RollingUpdate. With S3_ADMIN_* armed the hydrate path renames over the live
DB, so the quiet failure is the expensive one.
So the duty is decided once, in internal/writerlease, from the only thing that
can distinguish these processes — their position in the tree:
Take the pod root, which takes the lock BEFORE it spawns anything and
releases it AFTER the last child is gone
Inherit a child, which is handed the answer and never contends
Off no lease configured — today, and correct under Recreate
cmd/cloud takes it before its mount loops and stamps CLOUD_WRITER_LEASE_HELD
with its own pid; zip already spawns children with append(os.Environ(), …), so
each is born knowing the volume is claimed. The stamp carries the pid rather
than a bare flag so a child CHECKS it — it counts only when it names that
child's own parent — which is what stops a stray value in a manifest from
talking a fresh pod root out of taking the lock. cloud.Listen makes the SAME
call: in the fleet it inherits, and run standalone (`hanzo kms` on its own
volume) it is a pod root and takes the lease itself. One rule, one reader of
CLOUD_WRITER_LEASE; the Config bool is gone, since a bool parsed per process
is true in all of them alike, which was the original misreading.
Also: Acquire rechecks that the inode it locked is still the file at that path,
because flock locks an inode and an unlink+recreate under it yields two holders
who both believe they are alone.
Tests run real processes, since the defect was a property of the process tree:
TestLegacyRule_SiblingsDeadlock reproduces the incident — 3 subsystems, one
DataDir, nobody above them: 1 serves, 2 spend the whole budget waiting for
a handoff that cannot come
TestFixedRule_SiblingsAllServe same topology, root holds first: all 3 serve,
duty=inherit, zero contention
TestFixed_VolumeStillDefended the one that fails against main as it stands
("a second pod OPENED the volume while this pod holds the lease") — proves
the fix did not simply disconnect the alarm
TestStampCannotBeForgedByConfig a hand-placed stamp cannot disarm a pod root
go build ./... green; go test ./... introduces no new failures (29 packages
fail on origin/main before and after, all unrelated).
Follow-ups NOT in this change, deliberately: flock is a lie on NFS/CIFS, so a
shared-filesystem refusal is still worth having (see blue/writer-ha); and
cto/writer-interlock-honest argues the exclusive-store premise is now stale,
which is a question about whether to keep the mechanism at all, not about
whether it should be correct.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Not crashing was not the fix. The ai router maps ANY non-2xx from
/v1/billing/tier to TierZenFree, so the 400 that replaced the 500 downgrades a
paying customer exactly as the crash did — 60 rpm against 500 for pro, silently,
with no error the customer or we would ever see. The tier has to RESOLVE.
Its caller is a service, not a person, and the two resolve an org by different
doors. IAMTokenRequired admits only a gateway-validated user identity (ownerID
AND X-User-Id AND email) and deliberately falls through on a bare X-Org-Id,
because admitting on that alone once let an off-gateway caller name any victim
org. The router sends precisely that shape — verified in ratelimit.go, a service
bearer plus X-Org-Id and no user — so it arrived with a nil org.
TokenRequired is the door for a caller that IS a credential: it verifies the
service token first and only then calls ensureIAMOrg to resolve the header.
Credential before trust. Same reason the catalog CRUD and the recharge poke each
carry their own TokenRequired rather than riding an IAM chain.
The commerce v1.49.67 handler fix stays and is still right: a tier that genuinely
cannot be read is refused rather than answered Free. This makes it readable.
The saved-card family had no way in. A customer could list cards and remove
one; adding required POST /v1/billing/methods, which the billing app
forwards to commerce's /v1/billing/methods — an address commerce does not
serve co-resident, and which the in-cluster `commerce` Service resolves back
to these pods, so the forward re-enters the forwarder. It never got that
far: CLOUD_COMMERCE_HTTP_URL is unset, so the proxy is unconfigured and
every saved-card call, GET and POST alike, answers 501 "billing is not
configured". Measured live on pay.hanzo.ai. Nothing could bill a monthly
plan to a card on file because no card could be put on file.
The portal face is where its siblings already live, so the POST goes there
too: no HTTP hop, nothing to self-dispatch, same gate. commerce's
CreatePaymentMethod vaults the Square nonce as a reusable card-on-file and
stores the billing address with it — that vault is what a renewal charges.
PinBillingSubject is load-bearing here rather than decorative: this handler
takes its subject from the BODY (customerId), and the pin rewrites the
subject keys there while preserving card/type/sourceId, so a caller can only
attach a card to its own account whatever the body claims.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The route registration landed in v1.801.440 and /v1/billing/tier still answered
500: the chain was never the problem, the handler was. GetTier type-asserted the
organization, and IAMTokenRequired resolves one only from a gateway-validated
user identity — so every S2S caller, which is this route's main caller, arrived
with nil. Fixed upstream where the handler lives, not worked around here.
Outside apps/iam no app file imports what a bearer is hand-rolled from
(crypto/hmac, a JWT library) or joins apps/team/token, except the pinned
entries: foreign auth wires (LiveKit, SigV4, webhook schemes, the mpc ring),
non-identity seals (OAuth state, unsubscribe links, a KDF), and the condemned
team token with its complete reader set, which only shrinks. The root's
deleted second authority failed permissively — key confusion, machine
principals as humans — and apps/ had no guard against growing another.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud/apps/<name> is plugin-mode wiring for github.com/hanzoai/<name>; the
product owns the functionality and ships its own standalone daemon. The
boundary now counts itself: an app either mounts its product or holds a pin
naming which debt it is (mismatched / unwired / unextracted), and the lists
only ratchet down. Measured at pin time: 140 apps — 16 mounted, 1 mismatched
(deploy mounts hanzoai/cd), 27 unwired, 96 unextracted.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The status page published three incidents for services that were up. Not one
bug — one structural defect: a hand-maintained address table drifted from the
fleet, and the prober faithfully published three wrong addresses. iam was
probed on a path it serves on a different port; kms was probed at a deployment
scaled to zero years after the fold into cloud; hanzo-mpc named a Service that
selects nothing.
fleetTargets is now the one registry of where a service answers, and both
availability reads take their address from it. iam is probed at the OIDC
discovery document — what its own readiness reads, and the first thing every
client fetches, so answering it means a customer can sign in. kms is probed at
cloud's embedded /v1/kms/health, which still fails closed without the master
key, so the API can be up while KMS honestly is not. hanzo-mpc is removed
rather than retargeted: the real ring answers 307 to every path including
absent ones — a probe that can only say yes trades a false outage for a false
all-clear.
The second address table is deleted. productmap synthesized addresses by
convention — port 80, try /health then /healthz — wrong for 19 of 27 products
and burning two timeouts per miss. An unwatched workload now has no address
and is probed not at all.
A failure names its target, address and reason, edge-triggered: one line when
it breaks, one when it recovers, and a changed reason reports again because
that is a different chase.
Verified from inside the live pod: the shipped list answers 20/20. A red test
turned green with it — the scoped-status read that blocked a full second
dialing a service that was never there.
/v1/search and /v1/vector belong to provisioning; product owns exactly
four routes inside them. Hanging requireKey on the two parent groups
claimed both subtrees, so the confinement gate refused the boot — and it
was right to: in the unified binary that middleware would have gated
provisioning's routes with product's key.
The credential is a REQUEST fact, so each op now declares it: keyedIn
carries the Authorization header as a typed input field (zip's stated
replacement for exactly this middleware) and requireKey opens every
handler. Same statuses in the same order — unset key 503s, wrong key
401s, search and vector keys never cross — and the four addresses are
byte-identical; the document gains only the header parameter each op
always required but never published.
Verified: apps/product suite green, bin/product boots (was the one
compose failure in 120), openapi weave green with no path added, moved
or removed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every program now emits its request log natively — method, path, status,
duration, trace and span, the caller when the environment parked one — through
its own logger, so the per-app Logger install is deleted along with the three
test copies of it. The framework also propagates trace context and serves
/metrics from the one registry.
Kept deliberately: the OTel meter pipeline (metrics_http.go, installMeter) and
the span path. The datastore pipeline they feed is what /v1/summary and
availability READ, and the framework's export has not yet been proven to land
where those readers look. Two instruments briefly is safe; a blinded status
page is not. The compiler found the callers a text search missed — the span
middleware records the RED metrics — which is exactly why they stay until the
export path is measured.
Also carries luxfi/metric v1.9.1 transitively pinned by zip — the registry
that records by default.
failRedirect is the one place a failed OAuth return lands, and it logged
nothing, so every failure was indistinguishable from a request that never
arrived. It now logs the opaque public reason together with a precise
internal cause; the browser still gets exactly one reason, so no oracle is
offered to a caller probing states.
verify() wraps errBadState with that cause. errors.Is still matches, so
control flow is unchanged. The distinction it makes visible is between a
state signed by a different key -- what happens when the signing key is
absent and each boot invents its own, breaking every flow that spans a
restart -- and one that simply expired.
Folds the two ad-hoc warns into the funnel, carrying org in the cause so
no detail is lost.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The module-side registrations (commerce v1.49.65+) serve nothing in the
fleet by themselves — commerce's api.Route() bundle is never compiled here,
and the host hands an app only the prefixes its manifest row names. So the
four rails get what every self-service billing address before them got:
co-resident registrations on the pinned-subject IAM chain, and their
prefixes named deeper than the bare /v1/billing stem nobody claims.
- GET /v1/billing/wire — the serving brand's receiving bank
details, caller's billing key in the payment reference (attribution is
why it sits on the pinned chain; an unpinned wire is unattributable).
- GET /v1/billing/crypto/options — chains+tokens from the live MPC
processor; the pay SPA's asset picker renders exactly this.
- POST /v1/billing/crypto/deposit — per-payer custody address via the
signer fleet's /keygen; payer is the PINNED subject, never a body value;
open intents are reused so a refresh cannot spray keygens.
- GET /v1/billing/crypto/deposit/:id — caller-scoped intent state.
Nothing mints on any of these: wire settles via the admin wire/credit verb
on bank receipt, crypto via the chain watcher on real confirmations.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
geometry is a closed sum and family derives from the geometry's own type, so
"half-space parameters attached to a transformer" does not compile and this
package carries no check for it. family leads the content address, domain
separated, because every term after it is one family's arithmetic — two
families' numerically identical masses can no longer be named as one value.
The detector seam is the six methods the learner already calls. The half-space
counters are the first implementation and behave as before. legacy() is the one
remaining braid, at the disk boundary, and refuses a family the resume column
cannot carry rather than recording a shape for a model that is not one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.49.65's crypto rail called an MPC API the deployed fleet never served
(vaults routes, all 404). v1.49.66's GenerateAddress speaks the live
luxfi/mpc contract: POST /keygen {"org_id"} → all-chain addresses in one
threshold keygen. Env to arm it: MPC_ENDPOINT + MPC_API_KEY (KMS-synced
into commerce-secrets).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestTheThreeAddressesAreToldApartNotShared has been red on main since
"identity is the composer's" moved cloud.Bridge out of every subsystem and into
cloud.App, which installs it at the root ahead of every typed route. surfaceApp
was not updated, so it mounted o11y onto a bare zip.App with no Bridge — and a
typed op with no Bridge has no validated org on its context, so it answers 403
to everything, including the RED read this test asks for its OWN datastore
refusal.
That reads like a live outage of the o11y surface and is not one: the real
process composes through cloud.App. A harness that stops composing the way the
process does does not measure the process, and this one was reporting a
middleware it never installed as a broken handler.
scopeApp already says exactly this, one file over. surfaceApp is where it was
missed. Package goes green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
event.trace is the summary the read plane resolves a trace id against, and it
is fed by the SPAN WRITER — one partial row per trace per batch, folded by an
AggregatingMergeTree — not by a materialized view. hanzoai/o11y's own driver
does exactly this (pkg/datastoretraces/writer.go, traceSQLTmpl). This sink is
the production span writer and implemented event.span without it, so the
summary stayed empty while 710k spans piled up beside it.
An empty summary is not a missing feature, it is a silent outage.
TraceTimeRangeFinder resolves every trace_id predicate against this table
first, and a lookup that returns no row makes the querier SHORT-CIRCUIT THE
TRACE QUERY TO EMPTY (pkg/querier/builder_query.go, narrowWindowByTraceID) —
a missing summary row means "no spans exist", which is the one thing it did
not mean. The detail read, the waterfall, the flamegraph and every funnel
answered empty over a complete span table.
So: traceRowsOf folds the rows the span writer ALREADY built — one row-building
path, not two, with the four column indices pinned by a test so a reorder goes
red instead of corrupting the summary. end is max(start+duration), not
max(start): the longest span need not start last. The summary write fails SOFT
— a derived rollup must never be able to take down the fact ingest it is
derived from.
And GET /v1/o11y/traces, the org's trace LIST: the one address in the family
the module leaves open. It declares the detail (/traces/{traceId}), the field
catalog and three per-trace projections, every one of which needs an id this
read is where you get — the detail was reachable only by someone who already
knew the answer. Claiming the collection and nothing under it keeps that a
composition rather than a second declaration at an address that has an owner.
Typed, declared at the ROOT at its full path so zipdoc can resolve it, org
pinned as a bound parameter on the table's leading sort key, no admin
widening: a trace list is a tenant's records, not a rollup over them. It
aggregates on read because a merge is asynchronous and never a promise —
skipping that reports a BATCH as a trace.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Brings GET /v1/billing/wire (brand receiving-bank details, payer-attributed
reference), GET /v1/billing/crypto/options and POST /v1/billing/crypto/deposit
(per-payer MPC custody addresses) onto the served billing surface, plus the
$5 pay-as-you-go floor (topupBounds min 100→500 cents). v1.49.65 is the merge
of the forge lineage (zip v1.24.2, these rails) with the GitHub lineage
(dual-license, billing payment/invoice cores) — both remotes now converge.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The gate refused ANY apps/* import from cmd/cloud as a stand-in for "links no
subsystem graph". The stand-in went wrong the moment an app's EDGE had to run in
this process: cmd/cloud/sites.go serves <slug>.hanzo.app, and it must, because
the image runs this binary on the public port — a published site mounted anywhere
else is served nowhere, which is the outage that put it there.
MEASURED on origin/main: cmd/cloud links 392 packages, inside the ~395 the gate's
own prose names, and apps/sites pulls 2 cloud packages. Nothing grew. The gate has
been red for a day over a graph that never changed — and because every later car
declares `needs:` on it, that is the entire release train stopped by its own
approximation. Live is v1.801.431 while tags reach v1.801.436.
So the exception is NAMED with its reason, the discipline go-unit's -skip list
already follows, and the property it approximates is measured directly beside it.
Two checks, because they are two questions: the name catches a subsystem leaking
in, the count catches an ALLOWED leaf that quietly grew a graph. A widened pattern
would have answered only the first and silently given up the second.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud.App(name, cfg, deps, tools) now builds every program — the host and all
125 plugin programs — and installs the canonical chain through identity in one
place. Identify states the order invariant once: the boundary that mints the
validated org runs before the enrichment that parks it, and neither exists
without the other. Middleware order at the host is proven byte-identical
before and after (18 entries), so a refused request is still audited.
Every subsystem self-install of the enrichment is deleted — 74 sites across 67
packages. A subsystem asserting identity for itself repeats a claim it cannot
check; two of those copies sat on nodes owning no routes, which zip refuses to
compose, and that is the outage that took /v1/o11y and /v1/integrations down.
The childless spelling is gone everywhere: a gate passed TO Group is installed
at the root bounded by its prefix, and the two-step form that slipped past
that (a bare Group then Use on it) is collapsed at its last holdouts —
referrals' three gates and the audit fixtures.
Tests stop rebuilding the composer by hand: each package that mounts on a
bare app owns one compose helper, so anonymous cases still refuse and
principal-carrying cases reach the handler exactly as production does.
Also fixed, found by the constructor's own test: the markdown negotiation
replaced the Vary header CORS had written, so every CORS response on every
program advertised Vary: Accept — a shared cache could hand one origin's body
to a different origin. The later writer is additive now.
Census: zero non-test enrichment installs under apps/, zero childless chains.
Build and vet clean over the whole tree in the shipping mode.
The workspaces + members tables were 443 lines of hand-written database/sql:
positional placeholders, a hand-rolled scanWorkspace whose Scan matched wsCols by
POSITION, and a prefixed() that rebuilt the projection as a string. They now go
through hanzoai/orm's relational plane — orm.Select/orm.Typed for the reads, the
dbx builder for the writes.
Not orm.Model. That is the document plane over the JSON _entities table, and
these tables are typed and indexed with composite uniqueness ((owner_org, slug),
(workspace_id, user_id)) that a json_extract store cannot hold; orm's own guidance
names this file as a dbx target for exactly that reason.
The handle still comes from cek and orm is ADAPTED onto it. orm's SQLiteDBConfig
carries no master key, so orm.OpenSQLite would have written this store — every
workspace, membership and display name — as a plaintext `SQLite format 3` file.
That is the regression apps/iam documents removing, and it is not reintroduced
here: encryption at rest is a property of who opens the file.
The positional Scan is gone with it. wsCols and the struct now agree by `db` tag,
so a column added without a field is a mapping that does not resolve rather than a
silent shift of every value one position left.
THREE STATEMENTS STAY VERBATIM, each because the builder cannot say the thing that
makes it correct, and each now running through orm's own NewQuery rather than a
database/sql handle — so the file has ONE data path:
migrate — CreateUniqueIndex emits neither IF NOT EXISTS nor a WHERE, so it can
express neither the idempotence nor the PARTIAL (owner <> '') index; dbx.Sync
writes no indexes at all.
the EnsureWorkspace create — dbx's Upsert only ever emits DO UPDATE SET, and
there is no seam for a conflict target carrying the partial index's WHERE. As a
DO UPDATE the meaning INVERTS: the racing loser would overwrite the winner
instead of yielding to it, and the converge-to-one-workspace property is gone.
AddMember — Upsert fans EVERY inserted column into the SET list, so a re-invite
would overwrite joined_at and is_bot. joined_at is the order GuestRank ranks by
and the guest cap admits by, so that is not a cosmetic overwrite: it silently
reshuffles which guests keep access.
TestAddMemberPreservesJoinOrderAndBotFlag pins that last one at the four columns
the statement treats differently. Swapping in db.Upsert turns it red on three of
them (joined_at, is_bot, display_name) — checked, not assumed.
The test fixtures moved to the builder too, so no `?` placeholder survives in the
package.
apps/team + apps/analytics + apps/meet green, CGO_ENABLED=0 -tags sqlite_fts5,
against the ten pre-existing zip-compose failures already red on this commit's
parent.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zen composing was never the same as zen running, and only the first had been
fixed. It is Coresident: it answers no path and works by wrapping ai's "/v1",
routing zen-SKU requests and Next()ing the rest. A middleware cannot be a
separate process, so the light host deliberately skips Coresident apps
(cmd/cloud mount returns early) and plugin/zen exists only for the gen-app-cmds
bijection. plugin/ai linked only ai. The Claim was therefore mounted in NO binary
the fleet runs — which cmd/cloud had already recorded as "zen's child never saw a
request" — so every zen SKU served through ai's catch-all with zen's gate and its
meter never consulted.
zen mounts FIRST in plugin/ai, ahead of the greedy All("/v1/*") ai registers:
zip scopes middleware to the entries that follow it, so a Claim installed after
that catch-all would sit behind the very route it exists to gate.
MEASURED, on the real zen.Mount through the real MountAll:
zen5 -> 402, reached host catch-all = false (claimed and gated)
gpt-4o -> 200, reached host catch-all = true (falls through untouched)
Both halves matter. The first is the gate doing its job; the second is the proof
zen did not become a wall in front of the whole model API.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.4.4 billed every streamed call from an accumulator no chunk had touched.
c.SendStreamWriter hands its closure to fasthttp, which spawns it on its own
goroutine (stream.go:43) and returns immediately, so the recordCtx that followed
read usage.tally() before a byte arrived: completion -> 0, cached -> 0 (the whole
prompt re-billed as fresh) and ResponseID -> "", which is the join key /v1/feedback
and the learning ledger need to tie streamed traffic to its own feedback.
Reproduced here before bumping, on the exact shape: `go test -race` reports the
write from the writer goroutine against the handler's read, and the value visible
at metering time is 0 where the streamed total is 50. It is not merely racy — the
closure genuinely has not run, so a mutex would silence -race and change no number.
Both same-dialect paths were affected (proxy.go stream, sse.go anthropic-native);
streamTranslated, buffered and ultra meter synchronously and were always correct.
The drift runs BOTH ways, which is why this is not just our lost revenue:
answer-heavy shapes under-collect 62-85%, and cache-heavy short answers OVERCHARGE
a live ledger by up to 4x.
v1.4.9 was cut for this — the fix existed on zen's main and sat in no released
tag, so every consumer was still on the defect. MEASURED at the boundary:
v1.4.8 fails `go test -race` at proxy.go:509; v1.4.9 exits 0 with no races.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The rebase produced a program that does not compile — `undefined: sv1`, so the whole
apps/commerce test binary failed to build, a package that was GREEN on main.
Both sides fixed the same zip v1.24 composition rule and picked different halves of
it. This branch dropped the /v1 group and moved the middleware to Use; main kept the
group and moved the ROUTE onto it, so the chain guards something. The auto-merge took
this branch's deletion of `sv1 := app.Group("/v1")` together with main's
`sv1.Get("/store/current", ...)` that still names it.
Main's shape is the better statement of what the test is for: the case is an envelope
that must apply to commerce's own route and must NOT clobber a sibling's, and putting
commerce's route back on the group is what production does (Mount's storeV1 group).
So this file returns to main's version byte for byte — `git diff a4a1f998 --
apps/commerce/errorscope_test.go` is empty — and the branch keeps no opinion about it.
No commerce money logic is touched, on this branch or in this commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The zipdoc -check gate at the top of `make test` was red on three packages, so the
suite aborted before a single test ran. The lift is regenerated from source; all 99
directories carrying the directive now answer -check clean.
What the regeneration is, measured rather than assumed:
marketing prose only. Operation set identical, field-doc set identical.
account prose, plus doc comments for onboardResp.accessKey/accessSecret.
projects prose, plus a doc comment for projectsProject.key, plus the lift for
POST /projects/resolve-key (an internal cloud.Plane op whose file
landed without a regeneration).
The extra entries are DESCRIPTIONS for fields that already exist: AccessKey and
AccessSecret are account.go:604-605 and Key is projects.go:159, all three already
carrying the comments lifted here. zipdoc supplies prose; a schema property comes
from struct reflection at describe time. So this adds documentation to the surface
and no field to it.
THE PUBLISHED ARTIFACTS ARE DELIBERATELY NOT INCLUDED. Regenerating
plugin/{account,marketing,projects}/openapi.json from this same source does NOT
come out prose-only, and it is not this commit's business to land it quietly:
- plugin/account: onboardResp gains accessKey + accessSecret as published
properties, and THREE operationIds are renamed — post_v1_orgs -> v1.post_orgs,
post_v1_keys -> v1.post_keys, delete_v1_keys -> v1.delete_keys.
- plugin/projects: projectsProject gains a published `key` property.
The rename is zip's scheme change, not ours, and the fleet is mid-migration: the
committed account subset still carries the old form while marketing and projects
already carry the new one. Every SDK is generated from these files, so completing
that migration renames methods for callers and is one deliberate fleet-wide act
with an owner behind it, not a side effect of unblocking a test gate. The same
change is what TestTargetOpsProjectEverywhere has been red about (it asserts
post_v1_agents_targets and zip now emits v1.agents.post_targets).
Verified identical on a pristine a4a1f998 worktree, so none of the drift above is
this branch's: it is the published document catching up to source that already
landed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
main answered the red ratchet by RECORDING apps/risk/policy_wire.go in
allowedRequestUses (f345ac70). This branch answered it by removing the second call
site instead — caller() moved beside ops.gate, which already reads that same
X-User-Id header, so the package reaches for the raw request in one file and the
pin stays one entry.
Both landed, so the rebase left both: the entry describes a call site that is no
longer there. The gate catches that in the direction people forget — it walks
allowedRequestUses and fails on any file that no longer calls cloud.Request, so a
pin cannot outlive the code it justifies. Removing it is what makes the ratchet
green, not an exemption.
Net effect against main: the escape hatch does not grow. apps/risk/typed.go states
both reasons on the one entry it already had.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/networks answered 403 "X-Org-Id required" to a validated caller. Bridge was
installed on the group, twice — ng.Use and mg.Use — and middleware on a group wraps
what is composed beneath it. The two collection ROOTS are declared on the App with
their whole path on purpose: joining "/v1/networks" with an empty leaf yields
"/v1/networks/", a different address from the one they have always served. So the
group's Bridge covered /v1/networks/routers and /:id and missed /v1/networks itself,
the op read no parked org, and the tenant gate refused a request that was fine.
One Use replaces both. On a scope it is bounded by the prefixes the manifest already
declares for zt (/v1/networks and /v1/mesh/services), which is every route here and
nothing else; on a bare app — what this package's tests mount on — it is app-wide,
which is what the tests want. The groups stay as what they are, path prefixes.
TestBridgeIsInstalledOnEveryPrefix already stated this and had been red: it asserts a
validated caller is SERVED on all four routes and an anonymous one still refused.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two more of the shape the previous commit fixed in nineteen places, found by the
gate rather than by reading: apps/deploy panicked the fleet's own surface-check
(`the group "/v1/deploy" declares middleware at deploy.go:316 and no routes
anywhere beneath it`), and apps/agents failed 54 of its own tests.
deploy is the clearer one. deploy.go built app.Group(dashPrefix) to hold
cloud.Bridge + bounce, and dashboard.go builds ANOTHER app.Group(dashPrefix) to
hold the routes. Group returns a new definition per call, so those are two nodes
at one path: the middleware sat on the empty one and NEITHER the bridge NOR the
sign-in bounce ever ran for a single route of that surface.
agents is the same mistake with a subtler tell, because its group is not empty —
/metrics, /activity and the :ref leaves ARE beneath it, so nothing panicked. The
rest of the surface is not: the collection root, mountSessions and mountTargets
all register on the Router by absolute path. So Bridge parked no org for
/v1/agents/targets or /v1/agents/sessions, and every op under them answered 403
"X-Org-Id required" to a request that carried one.
It only ever showed up in tests, and that is the part worth keeping in mind:
Serve installs a Bridge app-wide, so serving was unaffected and only a bare Mount
could see the hole. TestHTTPTargetRejectsOversizeGPUList is the example — it
asserts 400 for an oversize GPU list, got the 403 first, and so had never once
exercised the bound it is named after. It passes now for its stated reason.
No bound widens. Every prefix either subsystem declares is under its own group's
path (manifest/apps.go), so a scope confines this to exactly what the group named
and the plugin binaries serve nothing else.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestRequestEscapeHatchIsPinned was red on main: apps/risk/policy_wire.go called
cloud.Request from a second file in a package that already had one, and the pin knew
nothing about it.
The fact caller() needs is the validated user id (X-User-Id) a policy version is
recorded against, and nothing parks that in the context — principal parks the org and
validated-ness, so principal.OrgFrom and principal.ValidatedFrom cannot answer it, and
an attribution the caller could state in a body is not an attribution. So the request
is genuinely required. What was not required is a second call site for it: ops.gate,
in typed.go, already reads that exact header for the meter's actor.
caller moves there, beside gate. Two functions, one seam, one pin — which is the shape
the map's own entries describe for wallets, tools, deploy, usage and o11y, and the
reason it is a per-FILE list. The pin's existing apps/risk/typed.go entry now states
both reasons; no entry is added, and the escape hatch does not grow to make a test
green. The function itself moves byte for byte.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nineteen sites carried the same defect scope.Use had, at the other door: a group
created with middleware, its routes registered on the App with their whole paths.
Middleware on a group wraps what is composed BENEATH that group, so every one of
those groups was empty — the middleware never ran, and since zip v1.20 the program
does not compose at all, which is a panic at mount for the plugin and an aborted
test binary for the package.
TWO GATES HAD SILENTLY STOPPED RUNNING, and that is the part that matters more than
the panic. apps/referrals put requireOrgOnWrite on a /v1/referrals group and
requireAdmin on each of the two /v1/admin/referrals boards, and registered all three
leaves on the App: the write gate never ran on POST /v1/referrals/claim and the admin
gate never ran on EITHER board. apps/team installed Bridge on one /v1/team group
while every file builds its own group from the same constant — the same routing
subtree, a different definition — so the bots, files, blob and cookie planes ran with
no validated org and answered 403 to valid requests (eight tests, all of which were
unreachable behind the package's own panic).
The identity and error-shape middleware moves to Use, which a scope bounds to the
prefixes the manifest declares for the subsystem and which is app-wide under a bare
Mount — admin, admission, auditlog, catalog, guide, integrations (x2), label,
marketing, prefs, projects, team, templates. referrals keeps three different bounds
and states each as the predicate a group prefix was standing in for (under), the same
shape apps/commerce already uses for its error envelope. o11y installs on its own
App, which its routes are already beneath. In team the install moves ABOVE the group
it used to sit on: fiber runs middleware in registration order, so one added after a
subtree never wraps it.
Five test harnesses mirrored the invalid shape on a bare app and move the same way.
No address changes and no gate widens: every bound here is the one the group prefix
named.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The door description still promised "NO CREDENTIAL IS ALSO ADMITTED ... filed
under the reserved $public tenant" — and it goes into the OpenAPI document
customers read. The code has refused since the key work landed: 401
ingest_key_required with nothing presented, 403 ingest_key_unknown for a
credential that names no project. A contract that promises the opposite of the
code is worse than no contract, because a client writes against it.
The projection survives for the reduced principal it was narrowed for — a
workspace token writing into its own org.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The escape-hatch ratchet has been failing on origin/main, naming
apps/risk/policy_wire.go and then apps/commerce/invoices.go. Both are
legitimate; neither was recorded, so the guard could not tell them from a new
one and main stayed red.
A red main is not a cosmetic problem here. Three of today's near-misses got as
far as they did because "the tests pass" had stopped meaning anything, and the
next reader has to re-derive whether each failure is theirs. So the fix is to
answer the pin, not to loosen it.
apps/risk/policy_wire.go caller() reads c.User() for an attributable policy
record. No ctx helper answers it — OrgFrom gives
the tenant, not WHO changed the appetite bounds.
Off the HTTP path it returns empty and plane.enact
refuses, so a change is never recorded anonymously.
apps/commerce/invoices.go eventsFrom/kmsFrom lift two request-scoped side
channels out of c.Locals(), which no ctx helper
exposes. Both optional by design: a missing
analytics collector must never fail a money move,
and a missing KMS client is the dev/test posture.
The root package is green again.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
event.fact is not a rename of event.event, it is a merge — five signals in one
table. Moving the source without naming the signal counted errors and spans as
device observations, and this is the one cross-tenant reader, so a phantom
identity inflates both k-anonymity floors: 425 identities unfiltered against
423 real ones on the live plane.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
config.go read CLOUD_CORS_ORIGINS and fell back to GATEWAY_CORS_ORIGINS, the
pre-rename gateway spelling, "shared with the gateway so both trust boundaries
agree on one list". Measured across every chart in universe, GATEWAY_CORS_ORIGINS
is set ZERO times — by cloud's chart, by the gateway's, by anyone — so the shared
name agreed on nothing. cloud's own values file sets CLOUD_CORS_ORIGINS.
Two spellings for one security-relevant allowlist is a way to half-configure it:
set the dead name and the browser silently gets no ACAO. No test pinned the
fallback, and the direction of failure on removal is restrictive — an origin that
is not listed is refused, never admitted.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
docs/consolidation.md was the execution queue for merging the standalone Go
services into one binary. Nothing references it, and every mechanism its Method
section instructs the next wave to use has since been deleted:
apps/apps.go no such file
apps.Wire no such symbol
cloud.Register no such symbol (only unrelated seam registrars remain)
"order < 150" there are no orders; manifest/apps.go is the table
clients/<svc>/Mount an app is plugin/<name>/main.go, one binary per app
Its inventory has drifted the same way — "37 native apps/*" against today's 120
manifest rows, three of the apps it names (paassvc, console, prompt) no longer
exist, "Wave 1 — THIS build" shipped long ago, and its CGO_ENABLED=0 "production
parity" rule is the opposite of what the Dockerfile does for plugins
(CGO_ENABLED=1 + libsqlite3 + sqlite_fts5 + sqlite_math_functions). A doc that
tells the next engineer to write code against a registration mechanism that does
not compile is worse than no doc.
The durable half — which tiers stay out of this binary and why — is not lost: the
edge/data-plane split is in README.md, and the isolation reason for identity is
stated where it binds, in apps/iam/iam.go. If that table is wanted as a fleet-wide
statement it belongs in LLM.md, which is the one doc this repo maintains, not in a
migration queue.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
scripts/smoke-runtime.sh said it was "intended for CI smoke gating", and CI
never called it — nothing in this repo, in .hanzo/workflows, in the Makefile or
in universe named it. The role it claimed is already served twice over, by
things that ARE invoked: `make smoke` runs plugin/smoke, the release gate's
functional prober, and `make e2e` runs e2e/run.sh, which builds the host and its
plugins and boots them for the Playwright suite.
It was also the weakest of the three. Five hardcoded probes against
plugin/smoke's one read per subsystem, with none of the contract that makes that
prober worth running — no 402-on-a-read rule, no 5xx rule, no authed/tolerant
distinction. And it reached for `curl -fsS`, the flag that turns a failing probe
into a silent one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
migration/ + plugin/migrate-pg-to-sqlite ported a `hanzo_cloud` PostgreSQL
database into per-org SQLite. That database belonged to the deployed cloud-api
PYTHON service, which this repo replaced; the runbook the tool cites
(CLAUDE_PG_TO_SQLITE_MIGRATION.md) no longer exists, no Makefile target, CI job,
compose file, chart or sibling repo invokes it, and it is absent from
manifest/apps.go so the image never built it. Its only two mentions were the
generator's exemption list and a comment naming it as an example. A one-shot
import for a service nobody runs is not part of v1.
The bijection test derives tool-ness by PARSING for a cloud.Listen call rather
than matching a name list, so it needed nothing; only gen-app-cmds' notApps map
did.
LLM.md's release section described an architecture that was deleted with the
fused binary: a `hanzo.yml binaries:` lane publishing ./cmd/cloud to S3, every
app resolving to "the multi-call binary with a different --enable", pinned by
TestRemote_DedicatedBeatsMultiCall. hanzo.yml now states NO binaries lane and
gives the reason; remote() looks up exactly name/os/arch with no multi-call
fallback; and the cited test was replaced by TestRemote_NoMultiCallFallback,
which asserts the opposite of what the doc claimed. The line numbers it quoted
had come to land on the comment saying the lane was removed. Corrected to what
the code does, and CLOUD_PLUGINS named for what it is: a supported input with no
producer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The catalog shipped 66 entries and 45 of them pointed at repositories that do
not exist. github.com/hanzo-templates/<slug> 404s for all of them, authenticated
or not, and their demo URLs 404 too — a gallery where two thirds of 'fork this'
leads nowhere.
The work was never missing, only misfiled. Those templates live in hanzo-apps
under a template- prefix, which is the org convention for a static site; the
catalog was written against a hanzo-templates layout that only 21 of them ever
moved to. 66 template-* repos in hanzo-apps, 66 entries here — the sets match.
21 kept real in hanzo-templates (expo-*, flutter*, swiftui*, desktop-*)
42 fixed repointed to hanzo-apps/template-<slug>
3 dropped innovise, kalli, unfixed — no repo in any org, and a catalog entry
that cannot be forked is worse than an absent one
All 63 remaining sources verified 200 against the GitHub API before this landed,
not assumed from the naming rule.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The plane's product-event table was renamed event.event -> event.fact. Writers
moved; these four readers did not, and nothing errored because the old table
still exists — it just stopped receiving rows on 2026-08-02. A frozen table
answers every query, so the failure was stale numbers, not an outage: the GTM
funnel was still reporting $13.37 of revenue from July.
event.fact holds every signal in one table (act, clip, error, log, span), so
the rename alone is not the fix. Each read pins signal='act', the predicate
apps/analytics already carries in scope(). Without it the funnel counts 85
visitors where 83 acted, and risk folds a person's error rows into things
they did.
risk binds the signal as a literal rather than a positional arg: all four
rollups share one 5-arg storeExec call, and the SQL already spells 'person'
and 'session' the same way.
Verified against the live warehouse — every query returns growing data where
it returned frozen, and apps/risk tests pass.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/o11y went native and now declares all 367 of its routes by name.
There is no wildcard left for a host route to shadow, so the three
addresses cloud also declared stopped being a silent precedence artefact
and became a refusal to compose — the plugin crash-looped and every
/v1/o11y read 503'd.
o11y.Claimed(...) made it boot, but a claim SUPPRESSES the module's
declaration. It did not resolve the collision, it just picked a winner and
wrote the choice down: each of the three quietly cost the fleet the
module's real read at that address, and one of them was cloud's forward
into a runtime path that no longer exists. What the addresses needed was
to be told apart.
GET /v1/o11y/logs — DELETED. Not a stub (it was a real two-view read
over event.log/event.span) but it had NO caller: the console reaches
logs through the query engine, and nothing in cloud calls it either.
An address nobody calls is not a contract. The module's real
log-record read answers there now.
GET /v1/o11y/metrics — MOVED to /v1/o11y/product/metrics. The module's
is the metric-NAME CATALOG; ours is one product's RED window keyed by
?product=. Two questions, so two names — ours says which one it
answers, and the module keeps the bare name. Live in the console
(App Platform drawer + canvas sparklines); patched there on the same
branch.
POST /v1/o11y/query_range — DELETED, along with POST /v1/o11y/query and
the whole of query.go. Both pinned /api/v3/<resource> to reach the v3
engine. The runtime registers every route at its full public path and
dropped prefix-stripping, so it serves no /api/* route at all, and
queryRangeV3 has no caller left — the forward reached the terminal /*
catch-all, not an engine. A route that forwards to an address nothing
serves is dead. The module's v5 querier answers there now.
Cloud therefore claims nothing: o11y.Mount(a) takes no options and the
boot log reads claimed:0. A host that has to name the addresses it takes
is a host that took addresses it did not own.
STILL OWED, and it is a console change: the trace/log explorers send a v3
composite that v5 refuses with unknown field "queryType". They were
already broken before this (the v3 pin reached the catch-all), so nothing
regressed — an opaque non-answer became an honest 400. The v5 migration
is recorded in apps/o11y/LLM.md.
The route table is unchanged in size: 389 method+path pairs before and
after. Only two lines move — /v1/o11y/product/metrics arrives and the dead
POST /v1/o11y/query leaves — because the module reclaimed exactly the
addresses cloud vacated.
openapi.yaml is the whole-fleet weave and cannot be regenerated per app,
so it also catches up on two other lanes' already-committed subsets
(billing +5 paths, payments +2) that it was stale against.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
IAM's error envelope rode on HTTP 200, so an SDK checking the transport before
the body read a refused signup as a successful one. The envelope is unchanged;
the status now agrees with it.
cloud's own IAM client is unaffected by construction: iamClient.do parses the
envelope whatever the status and decides on status != "ok", so it reached the
same verdict before and after. The bump is what makes the two agree at the wire
as well as in the body.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The unclassified branch rendered err.Error() verbatim as the 500 body, so a
customer's first fault told them about ZapDB migrations, CLOUD_KMS_MASTER_KEY_REF,
http://iam.hanzo.svc and features that are "not yet implemented". That text is
written for an operator; putting it on the wire published our internals to
whoever tripped it.
The rule is provenance, not status. An *HTTPError or a *fiber.Error was
constructed by a handler that chose a status AND a sentence — a decision the seam
does not second-guess, which is what keeps "Billing temporarily unavailable"
readable. An error that chose neither now renders a stable sentence naming the
request id instead.
The detail is not lost, it is relocated: ErrorHandler logs every 5xx whole, with
method, path and the X-Request-Id the response carries, so the log line and the
response a caller holds identify each other. The wrapped chain reaches the
operator, which is more than the client ever saw, since the client only ever got
the outermost sentence.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Federated sign-up files a brand-new user under the sign-up application's own
organization (iam internal/oidc/federation.go: `org := app.Organization`), so
the first request a new customer ever makes already carries an X-Org-Id. onboard
read that as "already has an org": the first org they asked for took the
ADDITIONAL branch, which creates an org and leaves the founder outside it, and
`personal: true` answered 409 "you already have an organization" — true of the
landing org, and useless to someone thirty seconds past signing up.
The orgs a sign-up can land in are exactly the ones this package already refuses
to hand to a customer (onboarding.go's reservedOrgs). One list, one fact, asked
twice: an org no customer may CREATE is one no customer can be said to OWN.
Naming the set rather than a single brand constant keeps a white-labelled
deployment correct, where the landing org is that brand's own.
Standing beats the landing. A SuperAdmin IS a member of the reserved `admin`
org, so treating that as a landing and moving them out would strip the privilege
it defines; an org admin therefore always counts as owning their org, read from
the authoritative IAM row rather than a header, since the answer is the one that
MOVES a user. A caller already in a real tenant is spared the read entirely, so
an invited member creating a second org is never yanked out of the team that
invited them.
The provisioning response now carries the credential it minted. IAM stores the
argon2id digest and blanks the plaintext, so the secret is readable exactly once
— in the answer to the call that mints it. Dropping it left a customer holding
an account whose credential had been issued and could never be obtained; a
replay, which mints nothing, reveals nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Merging the three analytics branches leaves publicTenant named where it no
longer exists: the heatmap tests normalize under it, and five comments describe
a lane the key work deleted. The tests take a real org like the rest of the
file; the comments say what the door does now — the org is the reduced
principal's own, resolved from its credential.
apps/reference read event.event, which stopped receiving rows on 2026-08-02
while the plane moved to event.fact. Columns are identical, so the device
aggregate was grouping a frozen table.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The golden is DERIVED from plugin/*/openapi.json, and the merge of origin/main
into forge/main brought a subset the forge-side golden predates: 1969abfc gave
POST /v1/runner's description its release: true SUPERADMIN paragraph. Weaving
the committed subsets reproduces every other byte, so the whole delta is those
three lines of prose.
No route moves — 1695 paths before and after, none added, none removed, and the
floor ratchet is untouched at 179 products. TestFleetIsTheWeaveOfItsApps and
TestTheServedDocumentIsTheArtifact both read this file and both go green.
Only the weave step ran. `make describe` also regenerates the subsets from
code, and that step is broken on BOTH mains independently of this merge — forge
fails first in zipdoc on apps/o11y/annotation_queues.go (which origin's dc013156
fixes), and past that the authors app is refused for installing middleware at
/v1/admin/authors, outside the /v1/authors prefix it owns. Neither is this
merge's to decide, and neither touches the derivation performed here.
zip removed App.Shadow — the verb that let an op be declared in one scope and
answered in another. Cloud never referenced it, so nothing here changes shape.
o11y v1.5.58 carries the same removal plus one narrowing of under().
Proven route-neutral: the woven fleet document is byte-identical before and
after (1695 paths, 3676621 bytes, cmp clean), so no SDK repo sees a route move.
Suite: 161 packages ok, 143 failing tests, and the failing set is byte-identical
by NAME to main's — regression set empty.
`release: true` on POST /v1/runner publishes releaseImage — ghcr.io/hanzoai/cloud,
the binary every service in every org runs. The gate read
principal.IsSuperAdmin(c) ||
(imageInOrgRegistry(releaseImage, org) && principal.IsOrgAdmin(c))
and the second term is not a narrower platform predicate, it is a self-service one
wearing a registry map. `isAdmin` is org-scoped and an org's OWN admin sets it on a
member of THEIR org, so every `hanzo` admin could enrol any `hanzo` member into the
gate — and `hanzo` is the deployment's brand org in every deployment, and the one
orgRegistryNamespaces gives `hanzoai`. So the far side of the gate chose its own
callers, and what it chose them for was the tag the whole fleet rolls onto at the
next reconcile: iam, kms, gateway and every customer app, in every tenant.
Owning the registry namespace is the right bound for an ordinary PUSH, and it stays
there. A push lands ONE tenant's artifact in the namespace that tenant owns, so the
caller's own org is exactly what should bound it. A release lands OURS on everyone,
so no property of the caller's own org can be what admits it. The two lanes part
company at that one fact, and nowhere else on this endpoint.
SuperAdmin <=> `owner == "admin"` is the ONE platform predicate, so the gate is now
cloud.Super and nothing conjoined to it:
func mayRelease(c *zip.Ctx) error {
if !cloud.Super.Admits(cloud.AuthorityOf(c)) { ... }
}
Validated is part of that scope, which collapses the MACHINE arm into the same
expression rather than a second rule beside it: PLATFORM_BUILD_CALLBACK_TOKEN mints
no principal, so a leaked build token still enqueues an ordinary build and still
cannot cut a release — the property TestRunnerRelease_SharedTokenCannotRelease
already pinned, now held by the scope itself.
READING a release took the same org-scoped gate (mayReadReleases), mirrored by
comment. Both are now the one function, so "the 202 hands back an id its caller may
ask about" is true by construction. Nothing was widened: the published contract for
both GETs already read "SuperAdmin only — this is the platform's own publishing
record, not a tenant surface", and the code did not do that. The /v1/runner prose
said cutting a release was "IAM's decision alone", which was true of the credential
and silent on the scope; it now names SuperAdmin.
The production release path does not go through this gate at all — a merge to
cloud's own main dispatches launchRelease in-process (push.go, isReleasePush), so
what tightens here is only the hand-cut, and the CLI cannot even send the flag
(cli.BuildReq has no Release field).
TestReleaseSurfacesTakeOneAuthority drives the REAL handlers across all three doors
— cut, list, read-one — over the role axis, and asserts they agree principal for
principal. Against the parent the brand-org ADMIN case fails with the escalation
verbatim, on every door:
cutting a release: admitted=true, want false (HTTP 502 — resolve main: ...)
listing releases: admitted=true, want false (HTTP 200 — {"data":[]})
reading one release: admitted=true, want false (HTTP 404 — no such release ...)
The 502 is the tell: the seams are stubbed to 500, so a request only reaches the
pipeline by having been authorized. It holds the SuperAdmin row on all three doors
too, so merely disabling the path would not pass, and TestRunnerRelease_-
OwningOrgAdminRefused pins the cut on its own.
It replaces TestReadingAReleaseIsNotStricterThanCuttingOne, which read runner.go
and release.go as TEXT and asserted both mentioned the same predicate NAMES.
Spelling was all it could ever see: it was green while both surfaces contradicted
the published contract, and it would have stayed green had the two admitted
different callers under the same names.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every release since v1.801.427 has failed. The image build runs
'go generate -run zipdoc ./...' and it exits 1:
zipdoc: apps/o11y/annotation_queues.go:94:10: zip.Get: cannot resolve the path
prefix of the router this op registers on
That is why seven plugins — templates, integrations, admin, guide, marketing,
prefs, o11y — answer 503 'no instance running' in production while the fix for
them sat on main unable to ship.
The cause was under{}, introduced to declare o11y's typed ops at the ROOT so
their ids stay unqualified and their schemas keep the app's origin. The goal is
right; the mechanism was a composite literal, and zipdoc resolves a router's
prefix only from a .Group() call or a variable assigned from one. A custom
OpTarget is invisible to it, and zipdoc treats what it cannot resolve as an error
rather than assuming an empty prefix — correctly, since a wrong prefix files
prose under the wrong identity.
Same intent, statically visible: register on the app with the full path,
o11yPrefix+"/reviews". The router is the *App (root, no prefix) and the path is a
constant expression the type checker folds, so zipdoc reads both. Declaration
stays at the root, so the ids and schema names under{} protected are unchanged.
Addresses are byte-identical — the regenerated zipdoc_gen.go changes by pure
ADDITION: GET /v1/o11y/sessions is documented now, because zipdoc could not reach
it before. under.go is deleted; nothing else used it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A project mints a publishable key at create — the ONE thing that attributes its
site's events. The key resolves to (org, project), so it names the site as well
as the tenant, and a key no project holds resolves to nothing: delete the
project and recording stops, structurally.
Removes the two attribution paths that were not that:
- the keyless lane. A beacon carrying no credential was ACCEPTED into a
reserved `$public` tenant and answered {"accepted":1}. No org could read
that partition, so every such caller lost everything it sent behind a 200.
Three first-party properties shipped keyless without one failed build.
handle now refuses: 401 ingest_key_required with no credential, 403
ingest_key_unknown with one that names no project.
- the site-host carve. A POST to <slug>.hanzo.app routed into the anonymous
lane with a host-derived tenant — a second mechanism, and the one that could
not be checked, since that middleware runs before the identity boundary. A
site's beacon carries its project key to the ingest door instead.
The projection, its bounds and the opt-out gate survive for the reduced lane (a
team guest writes into the org that invited it, at reduced capability).
CLOUD_ANALYTICS_PUBLIC_CAPTURE gated the deleted lane and is gone with it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Group handed back the raw zip router, so `g := app.Group(p); g.Use(mw)` walked
past every gate in this file and hung middleware on a node whose subtree is
necessarily empty — typed ops register through ZipApp, on the ROOT, so a
subsystem's routes never land beneath the group. zip v1.24 refuses to compose
that, correctly, and crash-looped fifteen plugins on v1.801.425/.426.
This file's own header said that idiom "needs no policing: the group already
bounds it". It bounds the MIDDLEWARE and says nothing about where the ROUTES
went, which is the half that mattered.
Group now returns a child scope, so all three idioms are ONE install — at the
root, gated by path — and no node is left that can be empty. The child carries
`at`, its path prefix, because a group PREFIXES what is registered through it:
without that, `zip.Get(app.Group("/v1"), "/bots", h)` (bots, entitlements)
registers /bots — a route silently MOVED, which still composes, so no compose
check could have caught it. OpScope carries the same prefix for the same reason.
Also, three subsystems that were escaping their bounds silently, because a bare
Group used to skip the check entirely:
- team answers /collaborator; the manifest says so and the plugin did not.
- label answers /v1/risk/labels; same.
- zt installed one bridge at /v1/mesh, a level ABOVE the only path it
serves there. One app.Use, gated by scope to what zt declares.
And OwnsHealth on authz/domain/experiments/metrics, which serve their own
health — so serve.go's generic liveness route was a second declaration of it.
Verified three ways, because a weaker check let a broken build reach production
twice today. The binaries were exiting on `mkdir /var/lib/cloud/orgs: permission
denied` BEFORE composing, and I read that silence as a pass:
1. survival — a compose panic is fatal, so rc 124 under timeout is the only
honest signal; "zip new" is logged before composition and proves nothing.
17/17 affected plugins survive, each with private ports and a writable dir.
2. route projection — `describe` diffed before vs after across 34 plugins:
34 identical, 0 changed. deploy/label/referrals now project where they
previously panicked. This is what caught the moved-route defect above.
3. go test -run 'Scope|Mount|Prefix|Route' green.
o11y is NOT fixed here: its three routes (logs, metrics, query_range) are
duplicate declarations against upstream o11y@v1.5.55, and zip dedupes on the
resolved path, so moving the node cannot help. Separate change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The anonymous lane projected the property bag down to the @hanzo/observe
annotation and dropped everything else, so a $click crossed it naming WHICH
element was clicked and never where on the page it sat. Element identity cannot
be drawn as a heat map, and logged-out traffic is the bulk of what one is made
of — so the position survived only on the signed-in lane, for a minority of
clicks.
The position is the second declared family. It does not get the annotation's
free ride: nothing lifts these into a column, so each really does add a key to
the attributes dictionary. What bounds it is that the set is CLOSED and spelled
by this server — five keys, never a caller's own vocabulary, values that are
numbers and a boolean, so the dictionary grows by five and stops.
boundedPosition filters rather than clamps, for the same reason
boundedAnnotation does: a clamped coordinate is a click somewhere the visitor
did not click, and a heat map is a picture of exactly that.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/event.js is the install path for a surface with no bundler:
<script defer src="https://api.hanzo.ai/v1/event.js" data-key="pk-…"></script>
One line, and the same line for a Hanzo property and for a customer's own
page. It autocaptures pageviews (initial and SPA) and uncaught errors onto
the canonical {batch:[…]} wire, carrying the key as a bearer on fetch and as
?ingest_key on the sendBeacon drain that cannot set a header. Identity uses
@hanzo/event's own storage keys and session TTL, so a page carrying both
clients resolves to one person.
WITHOUT A KEY IT SENDS NOTHING. The keyless beacons this replaces named
their site in a body field and carried no credential, so their events were
accepted 200 into $public — a reserved tenant the owning org cannot read.
The tenant comes from the publishable key, never from a body, so an unkeyed
page is silent rather than reporting success into a tenant nobody reads.
It is served beside the door because a tag that drifts from its wire is a
tag that 400s: asset and ingest ship in one binary and version together.
openapi grows the response half it had deferred until a route asked: Bytes
declares a body under the media type the handler sets, mirroring Binary on
the request side, so the document states JavaScript instead of claiming
JSON. Register copied the body half field by field and would have dropped
it; the copy now carries it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip refuses a program whose middleware has no routes beneath it, and the refusal
was right about three surfaces here. Nothing in this repo called Build() from a
test, so the refusal could only ever surface as a startup panic — or, for a
subsystem no test drove, as silence.
auditlog and catalog declared Bridge (and audit's noStore) on a group at their
own prefix while declaring the op on the App with its WHOLE path. The op is
therefore that group's SIBLING, not its child, so the group's subtree was empty
and the middleware could not run. Both packages' entire test suites were
panicking out of app.Test, which builds. Use is the verb that says the true
thing, and it needs no second copy of the prefix: cloud's scope bounds it to the
subsystem's declared subtrees, and a bare *zip.App treats root middleware as
live. The ops keep their exact paths — moving them onto the group with an empty
leaf would publish /v1/audit/ and /v1/catalog/, which neither API has served.
zen was worse and not the same defect. Its Claim gates ai's "/v1", it declares no
prefix of its own (Coresident), and plugin/zen fed manifest.PrefixesFor("zen") —
a ROUTING answer — into cloud.Plugin.Prefixes, which is the MIDDLEWARE grant. One
field was answering two questions, so dropping "/v1" from zen's row (right, for
routing: it duplicated ai's claim) silently revoked the gate, and MountAll refused
the mount outright. manifest.App.Gates states the second fact where the first
cannot, GrantFor reads it, and TestGrantMatchesPrefixesForRoutedApps keeps it from
becoming a second list.
crm was the same failure wearing an ordering convention. Its intake limiter used
to cover "everything registered after this line" — the public form plus the three
staff routes. Under lexical scoping it narrowed to the one route chained onto it
and the staff routes lost cover with no error anywhere. The covered routes are
composed BENEATH the limiter now, which states the coverage instead of implying
it. TestIntakeRateLimitScope was red before this and is green after.
compose_build_test.go is the gate that was missing: Build() through BOTH routers,
because MountAll hands a scope that can HIDE a seam a bare app refuses. It also
pins the rule itself, so a zip that stopped refusing the shape could not let the
seam rot back in green.
Deps: base v1.5.15 (the sqlite_math guard is a probe now, so `go build ./...`
needs no tags) and zip v1.24.2, plus iam v1.34.19 and o11y v1.5.57 restored —
394ba2e8 downgraded all three and left go.sum missing commerce v1.49.64, so
origin/main did not build at all.
guide's blueprint assertion matched a flat tool-naming zip no longer uses, so it
selected nothing; it matches the subject now, not the scheme.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
394ba2e8 said "base v1.5.15" and did not bump base. What it actually did was
roll THREE dependencies BACKWARD:
iam v1.34.19 -> v1.34.18
zip v1.24.2 -> v1.24.1
o11y v1.5.57 -> v1.5.56
base v1.5.11 unchanged <- the only change it advertised
Cause, so it is not repeated: `go get` was run against a working tree that was
BEHIND origin/main. go get pins the versions it is handed and leaves the rest at
whatever the tree already had, so every dependency a concurrent lane had already
advanced got written back to the older pin. The commit message described the
intent; the diff recorded the accident. Nothing verified the two agreed.
This restores all three and lands the bump that was missed:
zip v1.24.2, base v1.5.15, iam v1.34.19, o11y v1.5.57, commerce v1.49.64
Each is the newest tag on its remote, read with `git ls-remote` — GOPRIVATE
means proxy.golang.org cannot serve hanzoai/*, so the proxy is not an authority
for these and `@latest` silently answers from a stale public view.
base v1.5.15 is the one that pays for itself immediately: v1.5.11 carried a
guard whose symbol was defined nowhere, so `go build ./...` failed at default
CGO_ENABLED=1 and every build in this repo needed -tags sqlite_math_functions by
hand. Verified after this change: `go build ./...` with NO tags exits 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestMergedAndChainedAreIndistinguishable... pinned a fact so that a zip bump
which changed it would fail here. A zip bump changed it, so it failed — this is
the test working, not breaking.
The old fact: a duplicate registration and a middleware chain produce the SAME
observable route, so the generator must never read the handler count. From zip
v1.24 the duplicate half is not constructible at all — a second registration of
one pattern is REFUSED at composition time instead of merged, and the refusal
arrives as a panic out of Registry(), which took the whole package down before
any assertion could run.
The conclusion is unchanged and now rests on something stronger: a handler count
above one can ONLY be a chain, because a collision can no longer reach the
registry. Both halves are still pinned — the chain projects to exactly one
operation, and the duplicate is still refused — so if either moves, the
generator's assumption is forced back into review.
The chain's shape is now MEASURED rather than assumed. zip registers a GET as
GET plus an automatic HEAD companion, so one registration is two route entries;
the old test asserted one and would have failed on that alone. The GET is what
carries the chain and the HEAD is fiber's own, dropped from the projection.
openapi.yaml is regenerated because this branch adds six payment/invoice path
keys. Nothing is removed and no other app's operations move.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`tools/list` at api.hanzo.ai answered 570 tools and not one of them could take
money. Searching payment / charge / checkout / invoice returned nothing; the only
"pay" hit was an admin payout. An agent could incorporate a company, issue its
cap table, open a data room and sign a contract, and then had no way to be paid
for any of it.
The rails were never the problem. Square is live in the commerce module compiled
into this binary — official SDK, real charges, per-org KMS credentials, first in
the fiat priority list — and POST /v1/billing/topup/token has been charging cards
through it all along. What was missing was SHAPE: that route is a raw
func(*zip.Ctx) error, so it is a route and nothing else. No registry entry means
no OpenAPI operation, no MCP tool, no SDK method, no CLI command. The same was
true of the whole invoice lifecycle, of which this binary mounted only the list
and the PDF — an org could read invoices it had no way to create.
So commerce v1.49.64 lifted the money logic into cores that take values instead
of requests, and this adds seven typed ops over them:
takePayment POST /v1/payments
getPayment GET /v1/payments/:id
raiseInvoice POST /v1/billing/invoices
getInvoice GET /v1/billing/invoices/:id
issueInvoice POST /v1/billing/invoices/:id/issue
collectInvoice POST /v1/billing/invoices/:id/collect
voidInvoice POST /v1/billing/invoices/:id/void
They are TYPED, which is the whole point: each publishes a real JSON Schema with
per-field prose lifted from the handler's doc comment, down to a $defs for an
invoice line item. A payment tool with no parameters is useless to an agent — it
can be listed and never called — and this estate already has 469 untyped writes.
These add none.
THERE IS STILL ONE WAY TO TAKE A PAYMENT. Every op delegates to the same core
commerce's own HTTP handlers now delegate to, so the server-side amount bounds,
the idempotency derivation, the processor selection and the ledger credit are
shared rather than reimplemented. A second charge path would be a second set of
bounds to drift and a second idempotency key to disagree, which is a double
charge waiting for the right retry.
IDENTITY IS NEVER AN INPUT. The paying org is read from the validated principal
cloud.Bridge parks on the context; there is no org field and no subject field on
any of these ops, so a caller cannot steer money to an account it did not prove.
Mode is not an input either — sandbox versus live follows the org's credentials,
and the answer STATES which bucket it credited, so a sandbox receipt can never be
read as live money. Tests pin both: the schemas must not publish org/subject/test,
and every op must refuse a caller with no validated org rather than defaulting.
Also fixes errorscope_test's own empty group node — it declared middleware on a
/v1 group and registered every route on the app, so from zip v1.24 the group
guarded nothing and Registry() panicked before any assertion ran. The store route
moves onto the group, which is what production does and what makes the test a
valid program; its address and every expectation are unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
base v1.5.11 carried a guard referencing cgoBuildNeedsSQLiteMathFunctions, a
symbol defined NOWHERE — its only mechanism was the compile error it produced.
So the plainest command in Go failed in this repo and in every other importer of
base/core, reporting a missing symbol rather than the actual problem, and the
`-tags sqlite_math_functions` workaround had to be passed by hand on every
build. It was also a false negative in the config the driver's own docs call
production, because the tag was a PROXY for a capability rather than a
measurement of it.
v1.5.14 replaced the guard with a probe that asks the ENGINE — it runs the real
expression once against a throwaway in-memory DB and refuses the connection when
the answer is no, so the check reports a measured absence instead of a failure
to measure. v1.5.15 carries that plus zip v1.24.2.
Verified here: `TMPDIR=... go build ./...` with NO tags exits 0.
zip v1.24.2 and commerce v1.49.63 were already on origin/main when this landed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Fifteen plugins refused to compose and every route they own answered 503,
including /v1/event — the telemetry front door.
zip v1.24 refuses a group that declares middleware with no routes beneath
it, because that middleware would silently never run. Six apps wrote
app.Group("/v1/x").Use(mw) and then registered their routes on the app at
full paths, so the group node was always empty. f28bf6bd fixed exactly
this for scope.Use; these six bypassed it by reaching for Group directly.
app.Use is now the one way: scope.Use already gates a subsystem's
middleware to the prefixes it owns, by path, so the group node is not
needed and cannot go empty. One verb, one rule, no second mechanism.
Verified by running all sixteen affected plugin binaries: 0 compose
panics.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
tools/list proves an agent is TOLD a data room can be opened. It does not prove
one can open it, and the gap between those is a real failure mode with its own
shape: over MCP there is no URL, so zip passes the arguments object as the body
with a NIL path map, and an op whose address reaches it only from the path is
addressable over REST and NOWHERE else. Every existing test here drives HTTP,
where the path always binds, so all of them would stay green while the agent got
not-found — and three of these ops carry an id.
So the demo is driven the way the agent will drive it: open a room, grant a party
access naming the room by argument alone, read it back by id, and list it. The
negative half is what makes the positive half mean anything — the same read with
no id must NOT resolve a room, or 'found' proves only that something answered.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Tranche D listed dataroom 17. It is 10 typed and 7 refused now, and the note
records the two things a later reader needs: that the shared bundle-backed kit
(Scalar, SizedIn, BundleErr, Envelope) moved to apps/goja on its SECOND use
rather than becoming a second copy, and that ScalarList exists because a wrong
type on an access-control list is a SILENT failure — the room discards a
non-array and reports success, so a link meant for one investor would admit
everyone.
The tranche's remaining-count column is left alone: it already disagrees with
its own row (104 declared, 51 remaining after pricing/ml/automations/compliance
were struck without it), so reconciling it here would be a second lane's edit
wearing this one's commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip v1.24.2 stops composition renaming a DECLARED operation id by the prefix it
is included under. That mattered here more than anywhere: this binary is the host
that includes o11y under /v1/o11y, and doing so was renaming 217 of its 353
declared ops — every cached MCP tool name, operationId, CLI command and generated
SDK method moved as a side effect of one wiring line.
With commerce v1.49.63, iam v1.34.19 and o11y v1.5.57, every subsystem this
binary mounts is published on the same framework version. That agreement is the
point: Router is the type a decorator implements, so a host on one version and a
subsystem on another is a decorator that cannot be written.
Measured against a clean tree on this host: 97 failing packages before, 97 after —
zero new. macOS SQLCipher, unrelated and unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/dataroom/* has served fourteen routes and reached no agent. An untyped route
appends no op to zip's registry, and the MCP door renders tools FROM that
registry — so the fleet answered tools/list with 570 tools and not one of them
opened a data room. The hole was never the mount (dataroom mounts, and
/v1/dataroom/health has been answering in production); it was that every route
was an untyped relay.
Ten of them are typed ops now — every JSON route on the admin surface, which is
the whole demo surface: open a room, put documents in it, grant a party access,
and list what exists. Each still relays the bundle's own (status, body); what
changed is that it now carries an In and an Out, so the same declaration yields
the tool, the OpenAPI operation, the SDK method and the CLI command.
The package doc claimed NONE of these could be typed, on two premises the shared
kit answers: that a relayed answer is opaque (it is not — the bundle's shapers
are total and schema.go types them, which is what the models are), and that a
typed error path would overwrite the bundle's envelope (it does not — BundleErr
carries the bundle's status and BYTES). captable had already disproved both;
this makes that the second use rather than the second copy, so Scalar, SizedIn,
BundleErr and Envelope move to apps/goja, beside the bundle seam they serve.
ScalarList is the one piece captable did not need. A bundle substitutes an EMPTY
list for anything that is not an array, so an agent told allowList is a `string`
sends one, the room discards it, and the call SUCCEEDS having ignored the access
control — a link meant for one investor admitting everyone, reported as success.
Declaring the array is what puts that failure out of reach.
Four routes stay relays for reasons in the wire, each named at its registration:
the upload takes the file itself as the body, the two /file routes answer with a
byte stream, and the three public viewer routes have no principal to read.
Proven: the demo flow end to end over the typed routes; the reads byte-identical
to the bundle they replace; the cross-tenant link index still written, so a
granted link still opens for an anonymous visitor; org scoping; the room's own
refusal envelope intact; and the ten tools present with descriptions and schemas.
The regenerated captable subset is operationId-only (40 lines, 0 schema changes)
— pre-existing drift between the committed spelling and what zip v1.24.1 derives,
corrected by regenerating from source rather than by hand. The same drift had
left one captable test asserting a tool name nothing produces; it now asserts the
derived one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two independent ways a host WE run could serve a tenant's site. Both are the same
mistake: a set or a test that answers "is this ours" was allowed to shrink.
FIRST — the self-domain set was a default, not a floor.
self := list("CLOUD_SITES_SELF_DOMAINS")
if len(self) == 0 { self = defaultSelfDomains(apex, domain) }
An explicit list WON OUTRIGHT, so an operator adding a vanity domain silently
subtracted the derived ones. That set is the sole unconditional gate on the
site_hosts table (projects Store.bindHost asks sites.Ours) and the serve gate's
self-host exclusion, so dropping hanzo.ai makes every `<label>.hanzo.ai` a tenant's
to claim — a first-come row on api.hanzo.ai that denies us our own production host
for good, verbatim the defect SetSelfDomains was added to close. Nothing would have
caught it in review either: hanzo.ai reaches the set by DERIVATION from CLOUD_DOMAIN,
so no deployment states it and no deployment diff would show it leaving. The
reserved LABELS have had this floor all along ("trimming the env only ever ADDS ...
never subtracts"); the half guarding the more dangerous decision did not. Config now
adds to selfFloor and can never subtract from it.
In practice the brand domain also arrived a second way, via FirstPartyApex, so
BOTH had to be misconfigured before it actually vanished — which is why the new
test moves the first-party apex to prove the floor holds on its own.
SECOND — requestHost decided "is the parsed host real" by asking "is it a host we
would SERVE": siteSlug, else customCandidate. Those are different questions, and
the gap between them is exactly OUR OWN domains. api.hanzo.ai names no site, and
customCandidate excludes it BY DESIGN (IsSelfHost), so neither arm fired and the
client-supplied X-Forwarded-Host won:
Host: login.hanzo.ai
X-Forwarded-Host: <any bound custom domain>
resolved and served that domain's site — a tenant's content returned for a request
addressed to our auth apex, needing no site_hosts row on hanzo.ai at all. The
file's own comment already stated the property correctly ("a request that HAS a
host ignores the header completely"); the code did not have it. The two tests that
look like they pinned it both send a host that IS a site, so the early return fired
and the header was never read — they proved the property only where it already
held.
Whether we serve a host has nothing to do with which host was asked for, so
requestHost no longer asks: a parsed name with a dot is a hostname and is final.
That is the same set the three-way test actually admitted, minus the coupling that
made our own names the exception — and it leaves the header exactly the job it was
added for, the ingress case where fiber parses no host at all, plus bare internal
names no client addresses us by.
TestForwardedHostNeverOverridesOurOwnHost and TestSelfDomainsAreAFloorNotADefault
both fail on the parent:
login.hanzo.ai: the binding resolver was asked about [attacker.example]
hanzo.ai held only via the first-party apex — it must come from the floor
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Bind, verify and release must agree on what a hostname IS, and they did not. Bind
required fqdn.Valid; verify and release required only fqdn.Clean, and release did
not even require that much beyond non-empty.
site_hosts holds TWO shapes: a project's custom FQDNs, and its BARE SLUG — the
structural row deploy.go binds so `<slug>.<apex>` serves. A bare label is not Valid
(nameRE wants labels, a dot and a TLD), so release accepted a row bind could never
re-create. The domains panel renders that row like any other claim, as a live
`https://<slug>`, with a delete control beside it. A tenant tidying away the
odd-looking entry drops its OWN subdomain, and the domains API cannot put it back:
resolution falls back to ResolveUniqueLiveSlug, which refuses once two live
projects share the slug, so the subdomain 404s for everyone — and the next tenant
holding that slug to deploy takes the freed row, and the subdomain, for good. An
add-only asymmetry in a first-come global namespace is a transfer primitive.
hostOf is now the ONE reading of a hostname off a request — canonical form, then
the syntax this surface deals in — and all three ops ask it. A blank entry inside a
bind LIST is still skipped rather than refused; that is a list semantic, not a
disagreement about names.
TestReleaseOnlyAddressesNamesBindCouldHaveMade drives the real DELETE at its real
path. Against the parent it fails with the takeover verbatim:
release of the bare slug = 204, want 400
EXPLOIT: the site's own subdomain row was DELETED through the domains API
It also holds the positive case — a real custom domain still releases 204 and the
row goes — so refusing everything would not pass.
TestVerifyDomainPromotesOnlyOnProof closes a coverage hole rather than a defect:
Store.VerifyHost was tested and fqdn.Verify was tested, but the HANDLER that joins
them had no test at all, so nothing pinned that this surface requires the proof. It
injects a fake resolver — the projects package had none, so any test that reached
verifyDomain would have hit real DNS — and drives no-record, wrong-token,
right-token, already-verified, unclaimed and un-addressable in one pass. The token
is read from the ROW, never from the request, and that is what the wrong-token case
pins.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both halves of an authorization comparison must be the same value. The
registry-ownership lookup folded one of them:
orgRegistryNamespaces[strings.ToLower(strings.TrimSpace(org))]
while the org it compares came from principal.Org, which returns the validated IAM
owner VERBATIM and never lowercases — deliberately, because "acme" and "ACME" are
DISTINCT tenants in IAM and a fold would let a member of one select the other
(principal.go; TestMembershipMatchIsByteExact pins it).
So a tenant who self-serves an org named `Hanzo` — a different owner from `hanzo`,
and one whose own RoleOwner makes IsOrgAdmin true INSIDE it — folded onto the
`hanzo` key and inherited the `hanzoai` namespace. That is both lanes at once:
push over another brand's production images on the build path (runner.go
imageInOrgRegistry, repoOwnerInOrg), and satisfy the org term of the RELEASE gate
on ghcr.io/hanzoai/cloud, the binary every pod in the fleet runs. A fold applied to
one side of a comparison is not a normalization, it is a collision, and here the
collision IS a cross-tenant privilege grant — the same defect 26f69224 removed from
the custom-domain operator set, in a lane with the fleet downstream of it.
ownedBy is now the ONE lookup, keyed verbatim, and both callers ask it. Trimming
stays: whitespace is not an identity. repoOwnerInOrg keeps EqualFold on the OTHER
side — the forge owner parsed out of a repo URL, which genuinely is
case-insensitive — because that side is not an identity this system issues.
TestRegistryOwnershipIsVerbatim drives both lanes over both directions. Against the
parent it fails with the grant verbatim:
tenant "Hanzo" folded onto a brand's REGISTRY namespace — it could overwrite
that brand's production images and cut a release of the binary the fleet runs
tenant "Hanzo" folded onto a brand's FORGE owner
It holds the positive cases and the cross-brand refusal too, so narrowing the map
to nothing would not pass.
NOT addressed here, and flagged for review rather than changed: the release gate
admits `imageInOrgRegistry(releaseImage, org) && principal.IsOrgAdmin(c)` at all.
runner.go:255-267 argues that deliberately and at length — publishing your own
org's artifact is not cross-tenant — but the artifact is the platform's own binary,
`hanzo` is a customer org like any other, and its admin bit is one that org's own
admins set on their own members. fleet.go:254-261 reaches the opposite conclusion
for the mutation next door and gates it on cloud.Super. Overturning a documented
decision in this app is a call for its owner, not for this change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Binding a custom domain WITHOUT proving control of it — the bind lands VERIFIED
and routes immediately, and the "a host we operate" refusal does not apply — is
platform authority. The gate read
principal.IsSuperAdmin(c) || (operatorOrgs[org] && principal.IsOrgAdmin(c))
and that second term is not a second platform predicate, it is a self-service one
wearing config. `isAdmin` is org-scoped and an org's OWN admin sets it on a member
of THEIR org, so every `hanzo` admin could enrol any `hanzo` member into the gate
— and operatorOrgs defaults to the deployment's brand org in EVERY deployment
(config.go getenv CLOUD_BRAND, brand.Default "hanzo"). Naming the org in config
does not repair that: config grants a capability TO a tenant, but the tenant still
decides who inside it holds the role, so the deployment ends up delegating a proof
bypass to an authority it does not administer. A gate whose far side can enrol its
own callers is not a gate.
SuperAdmin <=> `owner == "admin"` is the ONE platform predicate. A second, weaker
spelling of platform authority IS the escalation, whatever conjunction dresses it
up, so vouches() is now that predicate alone:
func vouches(c *zip.Ctx) bool { return principal.IsSuperAdmin(c) }
It is cross-tenant by construction, so it still vouches in ANY org — the operator
switched into a customer's org to bind the domain it manages DNS for, which is how
a customer domain is onboarded. That path is unchanged and stays pinned.
Everything the org term reached goes with it: state.operatorOrgs,
operatorOrgsFromEnv, and CLOUD_PLATFORM_OPERATOR_ORGS. The variable appears
nowhere in the universe manifests, so no deployment states it and none needs
editing — a deployment must never state a variable nothing reads. The gate now has
no configuration at all, which is also why a test can no longer under-configure
it: there is nothing left to pass, so the tests drive exactly what production runs.
TestVouchIsSuperAdminOnly drives the real handler over the ROLE axis in the
deployment's own brand org. Against the parent, with the state Mount actually
builds — operatorOrgsFromEnv("hanzo"), the default {hanzo} — the brand-org ADMIN
case fails with the exploit verbatim:
EXPLOIT: brand-org ADMIN bound a bank's login host with NO proof:
{Host:login.example-bank.com Status:live Verified:true
URL:https://login.example-bank.com Records:[]}
It holds the SuperAdmin case too, so merely disabling onboarding would not pass.
TestVouchDoesNotTurnOnTheOrg keeps the ORG axis the verbatim pin covered: four org
names spanning the classes that used to matter — the brand org, its case fold, an
unrelated tenant, and the reserved `admin` org's own NAME — each asserted twice,
org-admin never vouches and SuperAdmin always does. The fold collision that pin
existed for is now dead by construction rather than by comparison, and THAT is the
property it now pins.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The table declares every route it owns by name, so there is no wildcard left
for a host's own route to win against, and two declarations at one address is
a refusal to compose rather than a silent shadow. cloud declares three of them
natively — GET /v1/o11y/logs, GET /v1/o11y/metrics, POST /v1/o11y/query_range
— and used to win them only by registering first.
o11y v1.5.56 gives that fact a spelling: Claimed names the addresses the HOST
serves, and the table then declines to declare exactly those. Say it here.
It is worth being explicit about WHY these three are the host's, because the
ordering that used to decide it decided it invisibly. This package's handlers
are tenant-scoped: handleLogs resolves the caller's org and pins the read to
it. The table's relay hands the call to the runtime with no org on it. Both
answered the same address, the first one registered won, and the two could
have drifted apart forever without anyone being told. The claim is that
decision written down where it can be read — and read the same way on both
sides of the seam, since the string that names the conflict in zip's own
diagnostics is the string that resolves it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/integrations and /v1/connectors answered 503 for the same reason o11y did:
the subsystem exited before it listened, because it declared middleware on two
group nodes that owned no routes.
app.Group("/v1/integrations").Use(cloud.Bridge(), zip.H(bridgeFacts))
app.Group("/v1/connectors").Use(cloud.Bridge(), zip.H(bridgeFacts))
Every op below both lines registers on zapp at an ABSOLUTE path, so neither
group ever received a leaf. A group's middleware wraps the routes in its own
subtree, and an empty subtree means the bridge could never run — which zip
refuses to compose rather than serve ungated.
scope.Use is the install that already solves this: once at the root, gated by
`owns`, confined to exactly the prefixes the manifest declares for this
subsystem. The manifest lists /v1/connectors and /v1/integrations both, so one
install covers what two group nodes were reaching for, and connectorRoutes
needs no bridge of its own.
The test fixture had drifted from the thing it reproduces. installV1Flatten
still hung commerce's error envelope on a Group("/v1"), while commerce itself
moved to gating by path (commerceErrorScope) precisely because a shared /v1
node wraps every subsystem mounted after it. Same shape, same empty subtree,
same refusal — it aborted the package on origin/main before this change. It
now installs the way its twin does.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The subsystem's ops were declared through a.Group(o11yPrefix), and zip
qualifies an op's id by the prefix of the occurrence it is declared under.
That rule exists so one definition included twice cannot publish one
operationId for two operations, and it is right — but o11yPrefix is not that
kind of prefix. It is not a composition point a host chose; it is this
subsystem's own address. Read through a Group it looked like one, and every
published id came out "v1.o11y.get_logs": the OpenAPI operationId, the MCP
tool, the CLI command and the generated SDK method, all renamed, with no
opt-out — an explicit WithOperationID is qualified the same way.
The same Group also cost the ops their origin, which is the app an op is
declared in and the thing that qualifies published TYPES. A Group is not that
app, so 23 schemas went out bare — logsResponse where the weave expects
o11y.logsResponse, and a bare name is one another subsystem can collide with.
Declaring at the root gets both, because an occurrence there is unqualified
and carries the app's own origin. OpScope.Prefix then prepends to the op's
path exactly as a Group's does, so every ADDRESS is byte-identical: same
method, same full path, same middleware. Only the two names change, and they
change back to what the declaration wrote.
hanzoai/o11y's own table reaches this conclusion for the same reason in its
relay.go; this is that shape on the cloud-native half. mountAlerts is left
alone — it registers raw routes, which publish no id and no type.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/o11y, /v1/sentry, /v1/event, /v1/errors, /v1/analytics, /v1/insights,
/v1/integrations and /v1/summary all answered 503 with
mount /v1/o11y: no instance running
while the pod stayed Ready with 0 restarts. The subsystem runs as its own
child process, and it was exiting before it listened: zip refuses a program
that does not compose, and this one declared middleware at two addresses it
had no leaf beneath.
a.Group(o11yPrefix).Use(cloud.Bridge()) // o11y.go
a.Group("/v1/summary").Use(cloud.Bridge()) // summary.go
Neither group ever received a route. The o11y leaves register through a
SECOND Group(o11yPrefix) in scope.go and annotation_queues.go, the module's
353 typed ops register on the app at a root prefix, and /v1/summary's own
leaf registers on the app at the group's address rather than beneath it. A
group's middleware wraps the routes in its OWN subtree, so all three sat
outside the thing meant to guard them — which is the defect the walk names,
and it named it correctly.
Install it once, on the app, where serve.go already says it belongs: a
subsystem whose routes are spread across several top-level nouns owns no
single prefix to hang it on. This one owns eight. That also keeps the module's
353 ops wrapped, which a prefix group silently would not have done — the org
a typed op reads off its context has to be parked for every leaf, not for the
fraction that happens to share a prefix.
The tests carried the same shape and are moved with it, so they exercise what
serving installs rather than a composition only the test builds.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The avatar write failed for the whole session with two different errors, and
both were this one door mis-reading a reply it had assumed the shape of:
iam non-envelope response (400) an error body it could not unmarshal
iam: iam status 200 a SUCCESS it rejected
Some IAM routes answer the {status,msg,data} envelope this client was written
for. Others — /v1/iam/users/get among them — answer the RESOURCE DIRECTLY, and
their errors are {"status":404,"error":"…"} where `status` is a NUMBER, not the
string "ok". Measured against the running service:
GET users/get?owner=hanzo&name=z
-> {createdAt,updatedAt,deleted,id,owner,name,…} no envelope at all
So a raw row parsed with Status "" and was rejected as `iam status 200`, and an
error body failed to unmarshal and became `non-envelope response`. Every read
through this client was one of those two.
The HTTP STATUS now decides and the body is only read for what it carries: a
non-2xx yields its `error` or `msg`, a 2xx envelope keeps its old meaning, and a
2xx that is not an envelope IS the resource.
`go get github.com/hanzoai/cloud@latest` fails outright:
module source tree too large (max size is 524288000 bytes)
v1.801.413 is the last consumable version; .420 and everything after cannot be
downloaded by anyone. That is every consumer of this module, not just ours — ai
hit it trying to take the release it had been waiting on.
The cause is build output committed at the repo root. `go build ./apps/gateway`
drops the binary HERE by default, and five landed that way: gateway 53M,
account 33M, authz 30M, smoke 8M, gen-app-cmds 4M. They are ELF x86-64, ELF
aarch64 AND Mach-O arm64 — three different people's machines, over three weeks,
each an accident nobody could see because the repo already had them. `account`
arrived today and is what crossed the line.
Untracked and ignored by exact path. Every one is built from a package whose
source is untouched (apps/gateway, apps/account, plugin/authz, plugin/smoke,
plugin/gen-app-cmds), so nothing is lost and `go build ./...` is unchanged.
Tracked tree: 174MB → 40MB.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerce v1.49.62, iam v1.34.18, o11y v1.5.55, ai v1.832.21 — every subsystem now
published on zip v1.24.1, so cloud and everything it mounts agree on one framework
version rather than four.
That agreement is the point. Router is the type a decorator implements, and the
verbs a decorator must satisfy changed in v1.23 (Use is the one composition verb,
taking a Component); a host on one version and a subsystem on another is a
decorator that cannot be written.
Measured against a clean tree on this host: 97 failing packages before, 97 after —
zero new. That is the macOS SQLCipher limit (no tmpfs for the pure-Go codec),
unrelated and unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The seam's doc said every app is its own process and read that as the reason
SetRiskScorer cannot reach the gateway. The premise is false as stated.
plugin/campaign, plugin/integrations and plugin/guide each link three or four
sibling apps/* into ONE process and wire process-global seams across them
(seams.go in each), under no build tag, and the image builds every plugin
directory. Co-residency is a per-plugin composition choice: one import in one
composition root is the whole distance between the two arrangements. As
composed today plugin/risk links risk alone and plugin/gateway links gateway
alone, which is true and is all that should have been claimed.
The remedy stands for a simpler reason that holds whatever the topology is:
apps/risk exports Mount and Shutdown and nothing else. There is no scoring
function to install, so the seam has no producer because none can be spelled,
not because a boundary forbids one. Giving it one means EXPORTING a scorer and
then deciding what it answers for — this global answers for its own process,
while arming asks whether the risk plane can answer for the fleet, which is a
cross-process ask.
The obs event door stays as the precedent it is, in its own clause rather than
as the justification.
Comments only; no behaviour changes and nothing is armed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two independent holes let a name the platform holds enter the global
site_hosts table.
SHAPE. Store.bindHost asked sites.IsReserved with its argument, but that
argument is a bare slug on the deploy path (deploy.go siteHost) and a full
hostname on the custom-domain path, while the reserved set holds bare LABELS.
So every FQDN compared against a set of labels matched nothing: login.hanzo.ai
sailed past the only guard behind the claim gate and took a first-come row on
our own auth apex. The storage invariant that is supposed to make the
serve-time gate a mere backstop contributed nothing at all for half the table.
sites.Ours splits on shape — a bare label asks the reserved policy, a hostname
asks the self-domain set — so one predicate answers both, and the claim gate
(domains.go ours) and the host table now ask the SAME question. Note what the
split avoids: `www` and `login` are reserved labels AND the two most common
custom domains a customer brings, so a backstop keyed on "the first label of
this FQDN is reserved" would refuse www.example.com. The label policy must
never reach a name a customer owns.
COMPOSITION. The self-domain set was published only by sites.New. Which apps
share a process is a per-plugin choice, and as composed today plugin/projects
links apps/sites for exactly these two predicates and never calls New — the
edge is not in that process. So the set was EMPTY in the very process that
enforces the claim gate and the host table, and full in the one that serves:
IsSelfHost answered false for everything there. Measured on the parent commit,
in a process that constructs no Server:
IsSelfHost("hanzo.ai") = false
IsSelfHost("api.hanzo.ai") = false
IsSelfHost("login.hanzo.ai") = false
IsSelfHost("hanzo.app") = false
which is the exact defect SetSelfDomains was added to close, defeated by
composition rather than by logic. No test could see it: they publish the set
themselves, in one binary.
The policy is now DERIVED from config wherever it is asked. ConfigFromEnv is
the one reader of that config and every process reads the same environment, so
the answers cannot drift the way a hand-off between processes does, and it
stays right however the plugins are later composed. An explicit publication
still wins and is never re-read over. Production needs no deployment change:
hanzo.ai is derived from CLOUD_DOMAIN's registrable domain, not configured.
selfOf folds the first-party apex in itself, so New's published set is complete
by construction rather than by statement order — the ordering hazard the
previous fix had to hold by hand.
Tests: the Server-less derivation, the shape split over both halves including
the customer hostnames that must stay bindable, and a concurrent first touch
under -race.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Binding a custom domain WITHOUT proving control of it — the bind lands
VERIFIED and routes immediately, and the "a host we operate" refusal does not
apply — is platform authority. The gate read
vouched := c.IsAdmin() || s.State.operatorOrgs[org]
and that second term is bare MEMBERSHIP. operatorOrgs defaults to the
deployment's own brand org (config.go getenv CLOUD_BRAND, brand.Default
"hanzo"), so the set is {hanzo} in every deployment, and `org` is the
IAM-validated effective org, gated upstream by isMember alone. Every staff
account whatever its role, plus anyone a brand-org admin ever invited, could
bind login.example-bank.com live with no DNS-01 proof: attacker content served
at any custom-domain customer whose DNS already points at our edge, and the
name denied to its rightful owner for good, since a verified row is first-come
and global.
vouches() now names the two grants, both admin-scoped:
SuperAdmin platform sudo (owner == the reserved admin org).
Cross-tenant by construction, so it vouches in ANY
org — the operator switched into a customer's org to
bind the domain it manages DNS for. Unchanged.
operator-org ADMIN the deployment named this org an operator AND IAM
says the caller administers it.
The second is a conjunction of two independently administered facts: a
capability the deployment grants to an org, and the role IAM grants inside it.
The set names an org; it never names an authority. The org-admin bit is asked
of the EFFECTIVE org — the same value SanitizeIdentity keys X-User-IsOrgAdmin
on — so the pair reads "admin OF this operator org" and never "admin of some
org I switched out of". Both bits are stripped on ingress and re-minted only
from validated claims.
Fail-secure: an issuer that stops signing the org role drops the operator-org
grant to a PENDING claim carrying the DNS challenge, the same self-service path
every other tenant takes. SuperAdmin onboarding never depended on that claim.
TestOperatorVouchNeedsAdminScope drives the real handler over four identities.
On the parent commit it fails with the exploit verbatim — a plain member's bind
of login.example-bank.com comes back {Status: live, Verified: true}. It also
holds the positive cases, so a fix that merely disabled operator onboarding
would not pass. TestOperatorVouchIsVerbatimEndToEnd keeps the verbatim-owner
pin and now has both callers be org admins, leaving the org name as the only
axis; it went from panicking to passing because it no longer builds the whole
surface.
The harness registers the ops it drives rather than calling routes(), which
composes only on the Router production gives it: routes() declares middleware
with Group(prefix, mw) and registers its typed ops on the App with full paths,
so on the bare *zip.App a test holds the prefix node has no routes beneath it
and zip refuses to compose. cloud.Listen mounts on a *scope, whose Use and
Group install at the root and gate by request path, so the same registration
composes there. What is driven is the production chain at the production paths:
cloud.Bridge parking the request, siteOf resolving the tenant, bindDomains
deciding.
The lifted prose and both published specs carry the corrected contract; the old
text told customers that membership of the operator org was the vouch. zipdoc
regenerates; the OpenAPI subsets cannot be projected for this app yet, so those
two files take the identical substitution zipdoc made.
apps/projects: 55 -> 58 tests passing, no test that passed on the parent fails.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The pin sat at sha-9da3984 while console main moved four commits ahead. Since
console.hanzo.ai IS this binary, none of them was reachable by any customer:
a0a4899 onboarding: Continue keeps its place, and no step can strand you
06e4163 retire the ML Pipelines (Kubeflow) product — its backend is gone
da26897 auth: a refusal is not always a failure
2271c29 profile: a photo you can change, instead of one you can only look at
The embed image for sha-a0a4899 is published and was probed before moving the
pin (200, against a known-good positive and a bogus negative control).
855 files, 360.7 MB, 192 of them real .rlib/.so/.a binaries: a complete cargo
build tree committed to git. Every published hanzoai/cloud module version has
carried it, so every consumer downloads 360 MB of another project's build output
to compile Go. The local module cache alone holds ten copies.
It is orphaned, not merely misplaced. native/flags/ contains NOTHING ELSE — no
Cargo.toml, no src/, no crate at all — so there is not even a project here to
rebuild it. No .go file, Makefile, Dockerfile or shell script references the
path. It arrived as collateral in 56c1b003 and nothing has needed it since.
.gitignore has listed `native/flags/target/` since before this commit; an ignore
rule does not untrack what is already tracked, which is exactly how 360 MB stays
in a tree everyone believes is ignored. `git rm --cached` is the part that was
missing.
Files stay on disk; only the index changes. History still carries the blobs, so
this shrinks FUTURE module versions rather than past ones.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nine plugins failed to start and every route they own answered 503 — account,
catalog, analytics, o11y and the rest — on one panic:
panic: zip: the group "/v1/avatar" declares middleware at scope.go
and no routes anywhere beneath it
THE NODE WAS THE BUG, NOT THE PREFIX. scope.Use installed middleware with
s.app.Group(p).Use(...), which creates a node AT p; scope.Get and its siblings
delegate straight to s.app, so the routes land on the ROOT node. Same paths, two
nodes. zip >= 1.23 checks the subtree of the node the middleware is on, finds it
empty, and refuses to compose a program whose middleware could never run. It was
right to.
Declaring the prefixes did not fix it — it moved the panic from /v1/account to
/v1/avatar — and analytics failed identically while already declaring them.
Both gates now install ONCE at the root and test the request path: Use against
the subsystem's prefixes, Group against its own. The root always has routes, so
nothing is empty, and `owns` does on the request what the per-prefix node was
there to do on the tree. `under` is that one meaning of "inside my subtree",
shared by both.
Confinement is unchanged and still proven by the tests that were red:
ScopeConfinesUseToTheSubsystem, ScopeHonoursDeclaredPrefixes and
ScopeAllowsGroupInsideItsPrefixes all pass. The root package now reports ZERO
composition panics; its 43 remaining failures are two macOS-only causes (40 cek
"no RAM-backed scratch", 3 unix-socket path length), identical before and after.
THE SUITE ALREADY REPRODUCED THIS. `go test ./` was red on main throughout the
outage, with the exact production panic, in under a second and with no cluster.
It was read as environmental noise while the fix was hunted in production.
SetRiskScorer has no production caller. The seam's own doc said to follow the
observability plane's event door "(obsevents.go)" — a file deleted in 4a6f3918,
by the commit that established the opposite: "A plugin is a process;
cloud.SetObsEventIngest / SetObsErrorIngest could never have worked across that
boundary." That door read nil in the process that needed it and answered 503 to
every Sentry SDK until it became a plane op (apps/o11y/obs_rpc.go).
SetRiskScorer is the identical shape, and the fleet is one process per app:
cmd/cloud mounts each subsystem as its own binary, plugin/risk lists risk alone
and plugin/gateway lists gateway alone, and no binary links both. So the wire
this seam invites — call SetRiskScorer from the risk app's Mount — would arm the
risk process and nothing else, while apps/gateway's RiskScorerInstalled, running
in the gateway binary, stayed false and PUT /v1/gateway/config went on refusing
every arming request. It would read as wired and change no outcome.
The tests cannot show this: they link cloud and the app into one binary, where
the handoff always works. That is why 43 references pass over a seam with no
producer. Both comments now say so — what the seam reaches, and that the
gateway's refusal is currently unconditional and fail-SAFE, answering "is a
scorer linked here" rather than the question arming asks, which is whether the
risk plane can answer for the fleet.
Comments only; no behaviour changes and nothing is armed. Reaching the fleet
answer is a plane op, and that is a decision to take deliberately, not a wire
to restore.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
New() published the self-domain set and THEN folded the first-party apex into
the slice it had just handed over. SetSelfDomains copies its argument
(reserved.go), so that append could never reach IsSelfHost — the one predicate
that answers "is this host ours", read by both the serve gate and the claim
gate.
Under a config that names the apex only as FirstPartyApex — not also in
CLOUD_SITES_SELF_DOMAINS — every non-allowlisted <label>.<fpApex> was therefore
a custom-domain CANDIDATE on the apex that carries api/login/console: a claim
row a customer could take, and a per-request binding lookup on the hot path the
exclusion exists to keep clear. Production lists hanzo.ai in BOTH, which is why
TestSelfDomainsCoverTheBrandApex passes either way and never saw it.
The publish now runs after the fold, so one set leaves the constructor.
Server.selfDomains went with it: it was written once here and read nowhere,
a second copy of a set whose only reader is the package global in reserved.go.
TestFirstPartyApexReachesThePublishedSet
fix reverted, test kept → RED (IsSelfHost false, customCandidate true
for hanzo.ai / api.hanzo.ai / login.hanzo.ai)
fix restored → GREEN
apps/sites and cmd/cloud green; apps/crm unchanged from its baseline failure
(TestIntakeRateLimitScope, pre-existing on origin/main).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/risk/search ranked model spaces against a tenant's own history and
answered with the one that fit. Nothing could promote it, and nothing could:
adoption refused a shape change, and a winner is a different shape by
definition. The stated replacement for Katib produced advice nobody could take.
A model value now carries the SPACE its masses describe, not only the masses,
and install REPLANTS — it builds that space, restores into it, and swaps it in.
Every gate the refusal used to carry still holds: the tenant comes from the
validated principal, the geometry seed is checked against the model already
running before anything is rebuilt, and the recorded shape is rebuilt and
compared by digest rather than trusted. A value recording no shape is refused
rather than defaulted, and a failed replant leaves the residency untouched.
The winner arrives as one of the organisation's own published values: the run
fits it once more after the grid — a sixty-fifth pass, gated and metered as one
— under that organisation's OWN geometry, because the grid's reference partition
is a constant and a model an organisation runs must partition the space in a way
an outsider cannot predict. Keeping all sixty-four fitted stores instead would
hold 21 MiB for sixty-three shapes nobody adopts.
Also:
- the shape is its own value, and the half of a config that belongs to the
state. The grid's candidate embeds it, the residency records it, a published
value stores it: one spelling, so adopting a shape cannot restate a policy.
- the resume row's shape is what the next process plants. Without it an
adopted shape was silently lost on every rollout and the organisation went
back to the default, warming, deciding nothing.
- the fold watermark travels with an adopted value. It is in the address for a
reason; leaving it behind meant a rollback would never re-read the fold it
skipped.
- snapshot+restore collapse to POST and PUT on /v1/risk/state/model: one
address for one kind of thing, the same collapse GET and PUT on
/v1/risk/policy already made. openapi/floor.json loses one PATH (the risk
product keeps all 31 operations).
- the value bound is sized on the widest shape the grid declares and MEASURED
at full occupancy (2,214,100 bytes), because a fitted sample understates a
busy organisation's tree by an order of magnitude.
- open() no longer reads the state believing the read plants a tenant's trees.
Measured against the pinned engine: it does not.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The two Kubernetes submit paths in ai/cluster no longer name a vendor in a
scheduler contract. The TrainJob's resourcesPerNode and the KServe
InferenceService's predictor both take the device-plugin resource name the
CLUSTER advertises, and a cluster that advertises no accelerator is refused at
submit time rather than accepting a workload that can never be scheduled --
which is what the KServe path did, silently, with a live controller
reconciling the result into a permanently-pending predictor and a model
registered on api.hanzo.ai whose every call fails.
Pairs with agents.Need/Spec.Satisfies landed here: one vocabulary for what a
job requires and what a machine advertises, and no vendor field in either.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Spec already carried what a machine IS -- os/arch/cpus/memory and each
accelerator's vendor/model/memory. Nothing carried what a job WANTS, so the
only way to ask for a GPU was a scheduler contract that named a vendor:
resourcesPerNode.limits."nvidia.com/gpu". That is not a requirement, it is one
vendor's name for a requirement, and it made a job unroutable to an AMD or
Apple machine that could have run it.
Need is the other half of Spec, in the same vocabulary, and Satisfies is the
one place the two meet -- a pure function of two values, so the dispatch gate,
a scheduler and a UI preview cannot disagree. Need has no vendor field: which
vendor clears an accelerator requirement is the machine's business, and
hanzo-kernel lowers one kernel source to CUDA/ROCm/Vulkan/Metal so a job never
has to care.
Unknown memory does not clear a memory floor. Every unified-memory
accelerator advertises 0 today (nvidia-smi answers "[N/A]" on a GB10,
system_profiler emits no VRAM line on Apple Silicon, lspci carries no memory
at all), so a VRAM floor currently refuses all three -- fail-closed, and the
reason the probe should report the memory an accelerator can actually address.
The three boxes are fixtures here, carrying their measured values, so that
change shows up in one place.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
account went down because its plugin declared no Prefixes and MountPrefixes
falls back to /v1/<name> — a path it does not serve — so the scope guarded a
subtree with no routes and zip refused to compose. Same fault, same fix, in
every other plugin that carries it.
Found by predicate rather than by waiting for each outage: undeclared Prefixes
AND a manifest row without /v1/<name> AND an app that calls app.Use(). All three
are needed — 25 plugins match the first two and serve fine, because the panic
only fires when a subsystem actually installs middleware at its scope root.
admission bot dataset do graph knowledge leaderboard treasury
Each now declares what the manifest already says it answers, which is what the
host routes to it either way — so this changes no address, it only stops the
scope guarding one that was never served.
Every account route answered 503 in production — /v1/keys, /v1/csrf, /v1/avatar
— because the plugin panicked at mount:
panic: zip: the group "/v1/account" declares middleware at scope.go:134
and no routes anywhere beneath it (via root -> /v1/account)
The plugin declared no Prefixes, and undeclared is not "no prefixes":
MountPrefixes falls back to the /v1/<name> convention (subsystem.go:78). Account
answers at NONE of /v1/account — its routes are /v1/keys, /v1/csrf, /v1/avatar,
/v1/orgs, /v1/embed and /v1/commerce/topup/*. So scope.Use installed the
subsystem's Bridge on a path with nothing beneath it, and zip refuses to compose
a program whose middleware can never run.
The same fallback bit analytics and entitlements before this; both carry the
same one-line fix, and this is it.
WHY THE TESTS DID NOT CATCH IT, which is the part worth keeping: they mount on a
bare zip.App, where Use attaches at the root and the root HAS routes. Production
mounts through a SCOPE. Same code, opposite outcome — so a green suite proved
nothing about the composition that actually ships.
manifest/clients_test.go's ledger of addresses a first-party client asks for and
the fleet does not answer held three entries. All three are closed; the ledger is
empty.
GET /v1/billing/portal/methods — this is what made "card save is broken" true END
TO END. cloud's billing app serves the saved-card list by proxying here, and
nothing in the fleet served it, so the proxy forwarded a 404 verbatim and the list
rendered as "no cards saved" no matter how many cards were vaulted. Claimed on
commerce's manifest row and registered co-resident.
THE GATE, which is the actual work: PortalPaymentMethods keys tenancy on a
?customerId QUERY PARAM, so the chain has to pin the subject for both principals
that arrive. TokenRequired (not IAMTokenRequired) authenticates and resolves the
ORG from the gateway-pinned X-Org-Id for an IAM member AND for the raw service
token the proxy presents; PinBillingSubject is the IDOR control — it overwrites
every billing-subject key with the validated caller's own account.Payer subject
and drops ?org, passes the query through only for a bearer that constant-time
matches COMMERCE_SERVICE_TOKEN, and fail-closes anyone who is neither. The tenant
is never a caller-supplied field on either path.
DELETE /v1/billing/methods/{id} — 405 at the live edge: a customer could ADD a
card and never REMOVE one. billing registers the sub-resource on the same router
as the collection (the host claims a prefix for ONE app across every method) and
proxies it to commerce's DELETE /v1/billing/portal/methods/{id}, the target that
does not self-dispatch. The org comes from principal.Org — the VALIDATED
principal only, never readerOrg's service-token admission, because this is a
mutation and that is the rule createPaymentMethod and gpuCharge already follow.
The id is escaped into the upstream URL: it names a resource, not a route.
PATCH stays unserved — no client edits a card.
POST /v1/billing/payment — DELETED, not served. No app in either server repo has
ever registered it, in any commit, so the crypto top-up's recording step always
failed and a customer who had already sent USDC to the treasury got a 502; it was
501 besides, since TOPUP_RAILS is configured in no environment. There is nothing
to point it at: money-IN has one door (commerce's mint-gated POST
/v1/billing/deposit) and the fleet routes NO mint address at the edge — the only
two money-in paths manifest.Apps hands to an app are the card ones, both with a
server-authoritative amount. What the browser client sends here is a
client-supplied amount and a client-supplied subject, which is the exact shape the
mint gate exists to refuse. So the surface that existed only to call it goes with
it, and openapi/floor.json records the two-operation reduction next to its reason.
Tenant isolation: TestDeletePaymentMethod_TenantIsolation proves org A cannot aim
a delete at org B through the org, the id, or any subject key; the far side is
proved in commerce (payment_methods_tenant_test.go) for both handlers and both
caller profiles.
Needs hanzoai/commerce fix/billing-methods for the portal detach it proxies to.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The lease is a single-holder flock on one inode under DataDir. Between two POD
GENERATIONS rolling over the same PVC that is exactly what is wanted — it is the
thing that stops a surge writer double-opening the exclusive-lock ZapDB and audit
stores. Between the sibling processes of ONE pod it is a deadlock.
cloud is a plugin host: kms, pubsub and kafka are separate processes in one
container, sharing this DataDir by design. Every one of them called
acquireWriterLease on the same path, so the first to start won and the rest
blocked to the 90s fail-closed deadline. Nothing bound :8000, the liveness probe
killed the pod, and the replacement deadlocked identically — api.hanzo.ai served
503 in a restart loop produced entirely by the safety mechanism.
`CLOUD_WRITER_LEASE` was reverted in the values file to stop the bleeding
(universe 47e195f1, "the binary is not one process"). This is the other half: the
variable can be set again without taking the deployment down.
The guard is underRouter(), which already exists and already means this — ZIP_ADDR
is proof of a parent that owns a plugin table. A child is not a second pod; it is
part of the writer that is already holding the lease.
The test pins all three facts: the host takes it, a second host-shaped process is
still refused, and a child never asks.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The wire was V1-shaped and ours in the places it was not the spec's, so a
compliant @x402/fetch or x402[httpx] client could not pay us at all: it sends
PAYMENT-SIGNATURE and we read X-Payment; it signs `payTo`/`asset`/`amount` and we
quoted `payee`/`token`/`amount`; it speaks CAIP-2 and we published a network label
beside a separate chainId. Every field name, header name and error reason on the
wire is now the one x402-specification-v2.md prints.
header X-PAYMENT / X-PAYMENT-RESPONSE -> PAYMENT-SIGNATURE / PAYMENT-RESPONSE
(plus PAYMENT-REQUIRED for the challenge), all base64 JSON
version "1" (string) -> 2 (number)
scheme "erc3009" -> "exact" (+ extra.assetTransferMethod)
network "hanzo" + chainId 36963 -> "eip155:36963", chain id read OUT of it
challenge bare PaymentRequirements -> PaymentRequired{resource, accepts[]}
payment flat Proof -> PaymentPayload{accepted, payload{
signature, authorization}}
answer Receipt on a header -> SettlementResponse, on success AND failure
The V1 shapes are deleted, not aliased. A header a client may send under either
name is two wires, and the one the server forgot to read is the one where a payer
pays and is never served.
Two things the alignment forced, both real bugs:
* The client's echoed `accepted` is now checked field-by-field against what we
offered (spec 6.1.2 step 5). `extra` IS the EIP-712 domain, so a payload free to
restate it would sign a message of its own devising and verify against itself.
* settleLedger documented a hole and left it: debit lands, credit fails, nothing
served, and the ONLY recovery was the client re-presenting an authorization that
expires in 300s. A client that gave up for five minutes was permanently debited
with nothing delivered.
Fixed by ordering, not by compensation (a reversing entry refunds a payer who
was correctly charged when the failure was a timeout). The settlement row is now
CLAIMED before any money moves and flipped to settled after both halves land, so
an interrupted settlement is a durable row naming the payer, the payee subject
and the amount — keyed on the id both money writes are idempotent on. And the
time window gates ACCEPTING an authorization, not COMPLETING one already
accepted: EIP-3009's validBefore bounds when a transfer may be submitted, not how
long submission takes. Two independent paths now converge — the client is served
whenever it comes back, and Reconcile finishes it from the claim if it never
does.
The payee's ledger SUBJECT is on the claim rather than re-resolved, so completing
a settlement pays the payout wallet the listing named. Re-resolving credited the
seller ORG, which no balance assertion could catch (finance aggregates at the
org) — the test now asks the ledger where the money went.
Also: POST/GET /v1/wallets answered 403 "sign in" to everyone. zip v1.23 scopes a
definition's middleware to its own subtree, and the collection root was declared on
the app, outside the group carrying cloud.Bridge — so it reached its handler with
no validated principal. Declared on the /v1 parent now. This is what blocked all 8
x402 and 4 marketplace tests from running at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A version was allocated by READING — max(registry tags, git tags) + 1 — and a
read reserves nothing. Two lanes publish ghcr.io/hanzoai/cloud (this repo's
.hanzo/workflows/cicd.yml, and apps/platform/release.go behind POST /v1/runner);
both computed the same next number, both built for ~20 minutes, and both pushed.
GHCR TAGS ARE MUTABLE, so the second push silently REPLACED the first's bytes
under a name the first had been told was its own:
v1.801.361 overwritten at 04:40:53
v1.801.410 overwritten at 08:17:12, twelve minutes after the real release,
by an image whose revision label read `unknown`
The loser found out at its tag step — three quarters of the way through, after
it had already corrupted the winner's image, which the winner then smoked,
pinned and shipped. A check twenty minutes downstream of the act it guards is
not a check.
Creating refs/tags/v<N> is the one operation in either lane the server performs
as a COMPARE-AND-SWAP: 201 if absent, 422 if present, decided under its lock.
So the claim IS the allocation, and it moves to the FRONT of both lanes. A
number that cannot be claimed was never ours to build; the loser walks to the
next one in a single HTTP call, before building anything. The tag step that used
to mint now VERIFIES the claim still names this commit before the pin.
Cost: a failed build leaves a hole — a tag with no image. That is the cheap
direction. A hole is inert and visible (pin.sh refuses a tag that does not
resolve); a reused number is invisible and serves the wrong bytes.
Also:
- resume is decided by the REGISTRY, not the tag. With the claim moved before
the build, a tag on HEAD no longer implies an image exists, so the old
`git tag --points-at HEAD` resume would have skipped the build of a release
that had none. The tag says what we own; the image says how far we got.
- the pushed image is verified to BE the commit we built, read back off the
registry rather than trusted from our own build output. This is what makes a
clobber by any lane — including one that ignores the claim — loud instead of
a green pin onto foreign bytes.
- REVISION is passed as a build-arg by both lanes. The Dockerfile declares
`ARG REVISION=unknown` and stamps the label from it; buildFrontendCmd never
passed it, so every platform-built image was untraceable to a commit. That
is why the two v1.801.410 images could not be told apart without diffing
layers. Branch refs are refused (isCommitSHA) — a label that says "main" is
populated and useless.
- .hanzo/scripts/image-revision.sh reads an image's commit, descending an
index to its amd64 child so multi-arch images are not silently unchecked.
- the two 422s are distinguished. "Reference already exists" is the collision
this loop is for; "Object does not exist" means our own commit is not on
github.com — reachable, since these lanes run on git.hanzo.ai and claim
against GitHub — and no amount of walking forward fixes it.
Tests: TestClaimReleaseVersion_IsExclusive reproduces the race (a number another
commit holds is skipped and never overwritten; our own claim is a resume that
mints nothing), and TestTagRelease_VerifiesTheClaim asserts the tag step only
READS. claimFrom is split from computeReleaseVersion so the exclusion property
is testable without the registry.
`make openapi` was renamed to `make describe` in e247e255 — correctly, since
the target now projects every app rather than only the spec — and 27 places
were left naming the old one. Two of them are FAILURE MESSAGES: a developer
whose golden is stale is told "run `make openapi`", which prints "No rule to
make target". A gate that says how to fix it, and names a command that does
not exist, is a gate that reads as broken tooling.
Comment-only apart from those two t.Fatalf strings; no target, no behaviour and
no artifact changes. `go build ./openapi/` and the four apps with the most
edits build with the tags hanzo.yml's own go-unit gate uses.
openapi/fleet.go also gains the projection it was missing. It lists four
projections of this API compared against each other by test; there is a fifth,
downstream and in another repo — hanzoai/openapi's hanzo.yaml, which every
published SDK is generated from — and it refutes itself against the LIVE
endpoint in that list, because that is the only one of the four a repo with no
checkout of this one can read.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Upstream landed the v1.23 verb migration (Graft/Add/Mount folded into Use). This
is the half that was missing, and it is the half that made tests lie.
App.Test used to skip prepare, which installs the deferred projections — /mcp, the
OpenAPI document, the op-call plane, the plugin route. So those four addresses
answered 404 under test and 200 in production, and the papering-over was an
exported Prepare each caller had to remember. zip v1.24.1 makes Test prepare;
apps/ai's MCP door test passes because of that, not because of anything here.
414 call sites move from app.Fiber().Test(...) to app.Test(...) with
zip.TestConfig. That is the point of the escape hatch living on the concrete type:
reaching through it bypasses what App.Test does, so the tests most wanting to
exercise the real program were the ones that did not. Sites whose receiver is a
raw fiber app keep fiber's type — the two are not interchangeable and pretending
otherwise is how the first sweep broke things.
Also: the multi-line `Use(func(c *zip.Ctx) error {…})` literals in tests, which
the verb migration missed because they fail vet rather than build; and the last
`.Prepare()` calls, now that it is implicit.
iam v1.34.11 → v1.34.12.
Measured against upstream on the same host: 103 failing packages before, 97 after
— ZERO new, 6 fixed. The remainder is the macOS SQLCipher limit (no tmpfs for the
pure-Go codec), unrelated and unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
With the read shape fixed, the avatar write moved one step and stopped: the
lookup now reaches IAM and asks for user `hanzo/2d4d67ab-…`, which does not
exist under that spelling. IAM addresses a row by its NAME — the row whose id is
2d4d67ab-… is named `z` — and on the direct-Bearer path the only user handle a
token carries is the UUID `sub`, because X-User-Name is not stamped there.
So a failed direct lookup now resolves the id against the org's roster
(get-users?owner=…, which carries both id and name) and retries once. Measured:
that roster returns 268 rows for hanzo, each with both fields, so the mapping is
available exactly where it was needed.
It runs ONLY after the direct read has already failed — the gateway path, where
username == name, never pays for it.
zip v1.23 deletes graft.go outright. Graft(children ...*App) error was the third
way to attach something to an app, alongside Add and Use; v1.23 keeps exactly
one, and an *App IS a Component, so a child is included by reference through the
same verb as a middleware.
Two call sites, both the "include a whole subsystem's app" case Graft existed
for: apps/iam grafting iamserver.NewApp(db), apps/o11y grafting its assembled
app. Both now Use it.
ONE SEMANTIC CHANGE, recorded because it is not visible at the call site: Graft
refused an address conflict EAGERLY and returned the error to the caller, having
checked every child address against the parent's router and its siblings before
mutating anything. Use appends and defers that verdict to Build, where the whole
program is known. Same refusal, later and with more information — but a mount
that used to fail at its own line now fails at seal, so read Build's error, not
the mount's.
Verified: `go build -tags sqlite_math_functions ./...` returns ZERO across the
whole tree with iam v1.34.11, o11y v1.5.54 and commerce v1.49.61 — the three
dependencies that had to publish first. Graft appears in no .go file in any repo
in the estate.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Cloud pinned zip v1.18.23 while latest was v1.23.0 — five minor versions of
drift on the framework every subsystem composes through. v1.23 removes the
second and third ways to attach something to an app and leaves exactly one:
Use(cs ...Component), where a Component is a Handler or an *App included by
reference.
Cloud's own four adaptations:
- Router.Use widens from ...zip.Handler to ...zip.Component, mirroring
zip.Router exactly. That mirroring is load-bearing: ZipApp's type switch asks
whether a *zip.App satisfies this interface, and a narrower Use made that case
IMPOSSIBLE — the compiler rejected the switch outright rather than silently
taking the wrong branch.
- scope.Use forwards Components unchanged, still once per declared prefix.
- (*zip.App).Add is gone. zip.Load already returns the leaf *App and an *App IS
a Component, so both call sites capture the leaf and Use it. The eager and
lazy rungs of the plugin ladder keep their existing error handling; only the
attach verb changed.
- Five bare closures passed to Use now go through zip.H. Go will not implicitly
convert func(c *zip.Ctx) error to an interface that only the named Handler
type implements. Route methods still take ...Handler, so no route registration
changed — only Use sites.
Verified: with zip at v1.23.0, `go build -tags sqlite_math_functions ./...`
reports ZERO errors from apps/, cmd/, clients/ or internal/. The only remaining
failures are in three dependencies that must publish first — commerce
(mintRouter implements the old Use, and reaches Fiber() through the Router
interface, which no longer exposes it), iam and o11y (both call the now-
unexported app.Prepare; the public replacement is app.Build, which returns an
error Prepare did not). Those are in flight; this commit is the cloud half and
does not build green until they land.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The site edge is mounted in two processes — the light router that owns the
public port (cmd/cloud) and cloud.Listen in every per-app child — and each
resolved sites.Config for itself, from two spellings of the first-party keys
(CLOUD_SITES_FIRSTPARTY_* against CLOUD_SITES_FIRST_PARTY_*) with two sets of
defaults. Neither spelling is set in production, so the two processes booted
different policy off their DEFAULTS alone: the router resolved no first-party
apex, so hanzo.ai never entered its self-domain set, so every hanzo.ai host —
api.hanzo.ai included — was a custom-domain candidate and took the per-request
binding lookup the self-domain exclusion exists to keep off that path.
The reserved denylist was NOT affected. Both spellings read the same
CLOUD_SITES_RESERVED key, the operator value only ever ADDS via
SetReservedExtra, and the labels an attacker wants are baked into reserved.go —
so an empty value cannot un-reserve anything. Measured at the live edge:
www/api/app/admin/stg/login/wallet.hanzo.app all fall through with no
X-Hanzo-Site, against quest.hanzo.app which answers with one.
The env keys and their defaults now live in apps/sites, the package that owns
the type, and both call sites read that one function. Config keeps only Domain,
which feeds the self-domain set.
Also removes exports with no caller in this repo or any that import it:
OKList/OKRaw (envelope.go, whose note about a clients/admin/core delegator was
stale — that package is gone), BrandInfo, DegradedNames/IsDegraded (the release
smoke reads /v1/health over HTTP; the in-process accessors had only tests, whose
coverage of the live Degraded/Degradations pair is kept), and the four inert
Stage-0 control-plane fields with their NODE_ID/PEERS/ROLE/CONTROL_PLANE_QUORUM
reads.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every op on /v1/engine was dialling a refused connection in production. The
upstream default named :1234; svc/engine in namespace hanzo exposes ONE port and
it is 36900 (name http, port 36900, target 36900), and the cloud Deployment sets
no ENGINE_UPSTREAM — so the wrong default WAS the production value. status
answered reachable:false, and models, model and system answered 503. The
subsystem was honest about being unreachable, which is why nothing alarmed: a
truthful report of a broken configuration reads exactly like a runtime that is
down.
1234 is standalone `hanzo serve`'s default. 36900 is the port the engine binds as
the node's engine, and 36900 is what is deployed — so the comment claiming 1234
was "the in-cluster Service of the engine deployment" described a process this
cluster does not run. The 1234-vs-36900 confusion is already on record from the
desktop build, where a frontend discovered models at one port while the engine
that answers ran at the other; this is the same mistake on the cloud side.
MEASURED against the deployed engine over a port-forward to svc/engine:36900,
which answers exactly the three endpoints this plane calls, all 200 —
GET /health ("OK"), GET /v1/models (the model list, with a loaded model carrying
"status":"loaded") and GET /v1/system/info (the SystemInfo document: os, kernel,
cpu, memory). So the upstream is present and correct and only the port was wrong;
this is one token, not a redesign.
ENGINE_UPSTREAM stays the override, and 1234 remains right where it is right — a
dev box running `hanzo serve`. Behaviour only: no route, no schema, no operation
id and no published byte changes, and regenerating the subset produces no diff.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An operation id is the generated SDK METHOD NAME and the CLI COMMAND, and a
summary is what the MCP tool list shows a model choosing between tools. This
plane stated neither, so both were defaults, and both defaults were wrong in a
way only a caller sees.
zip derives an unstated id from the path, so the four ops published
`get_v1_engine_status`, `get_v1_engine_models`, `get_v1_engine_model`,
`get_v1_engine_system` — path mangling where the rest of the fleet publishes the
product and the noun. The risk product's thirty-one operations are `riskScore`,
`riskState`, `riskDatasets`, `riskLabelCoverage`; these are now `engineStatus`,
`engineModels`, `engineModel`, `engineSystem`, which is also the rule this
package already applied to its own SCHEMA names and only to those.
A summary defaults to the first sentence of the Go doc comment, and a Go doc
comment opens with the Go IDENTIFIER — so the published summaries read "Status
reports whether the engine deployment is reachable", "Models lists the models
the engine serves", "Model reads one model's load state". A Go symbol name was
the first word a CLI user, an SDK reader and a model picking a tool saw. Each op
now states a summary written in the imperative for the person calling it, and
the doc comment stays a Go doc comment that zipdoc still lifts as the
description: two audiences, two sentences, one declaration.
FORWARDS-ONLY, and it costs nothing: this plane has no customers on it. No
alias, no redirect, no compat shim.
Regenerating from source changes exactly four operation ids. No path is added or
removed, no (path, method) pair moves, no schema changes, and openapi/floor.json
is byte-identical because the operation count did not move.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
openapi/floor.json is the ratchet: "a published surface may grow and may not
quietly shrink". It carries a global path and operation count AND a per-PRODUCT
count, and the per-product half is the half that binds — floor.go compares
`f.Products[p]` against the regenerated count and refuses a product that lost
operations. The file's own code states the failure mode out loud: "an absent floor
makes every shrink legal".
The merge 94df22e1 resolved this generated file to
`{"paths":1684,"operations":2336}`: the entire `products` map — 180 entries —
GONE, and both global counts rolled backwards. Every commit before it carries the
map. So on main right now every product may shrink without the gate saying
anything, and the global floor sits five paths and five operations below the
surface actually published.
IT ALREADY COST SOMETHING, within hours. Deleting the /v1/train facade removed ten
operations, and a ten-operation shrink is exactly what this ratchet exists to make
an author state deliberately — the dataset move had to lower `ml: 14 -> 7` in the
same commit that moved the routes, and said so. The train deletion needed no such
line, because there was no per-product floor left to lower. The gate was not
merely stale; it was switched off, and a shrink walked through it unremarked.
Restoring it is `make describe` and nothing else: 1689 paths, 2341 operations, 178
products. `train` is simply absent from the restored map rather than floored at
zero, because the map is regenerated from the document that exists — the ratchet
resumes from the current truth, which is the only honest baseline available once
the old one has been discarded.
Committed on its own because it belongs to no surface change: these counts are
identical with or without the engine renaming beside it, since an operation id
does not move a count.
The lesson is the fleet's own about derived artifacts, with a sharper edge: two
derived files can agree with each other while both are wrong, and a MERGE is a
third way for a derived file to become wrong — resolved by hand toward whichever
side the conflict presented. A ratchet resolved toward the weaker side does not
look stale. It looks fine, and it is off.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/label addressed /v1/ml/labels, apps/reference addressed /v1/ml/reference and
apps/dataset addressed /v1/ml/datasets. All three are the RISK product — the
labels a decision is adjudicated with, the lookup data it consults, and the
snapshot its model was fitted on — and all three were corrected to /v1/risk/*.
The reason is the same each time: openapi.Product takes a product from the FIRST
/v1 segment and a per-op tag cannot override it, so an address IS a published
product membership, and /v1/ml is the live serving product. Filing a second
product there makes /v1/ml/models mean two things at once.
apps/dataset's address_test.go says the third time "stops being a recollection
and becomes this gate" — and builds it, for apps/dataset. A per-app gate cannot
catch the fourth time, because the fourth time happens in a fifth app that does
not have one. So the invariant is stated once, on the side every app must pass
through: the ROUTING GRANT. A plane cannot publish under /v1/ml without a
manifest row saying so, which makes the row the one place the mistake is always
visible.
It refuses a second OWNER, not a second app, and that boundary is measured
rather than assumed: fourteen products here are answered by more than one app and
27 apps publish into more than one product, so neither "one app per product" nor
"one product per app" is a fleet invariant, and asserting either would invent a
rule the fleet does not keep. What all three mistakes actually broke is narrower
and true — the serving prefix has one owner.
The second test is the half the ai incident argues for: a gate that only refuses
intruders stays green when the owner itself vanishes. Narrowing ai's row to
/v1/ai once took the entire inference surface off the wire with every probe
green, so the owner's own leaves are asserted individually — a count cannot say
WHICH address stopped routing.
BOTH DIRECTIONS ARE MUTATION-PROVEN, because a gate nobody made fail is a gate
nobody knows works. Re-addressing dataset to /v1/ml/datasets fails with the
ROUTING GRANT message naming the app, the prefix and the product it would join;
narrowing ml's row to /v1/mlops fails BOTH tests — the non-vacuity check (which
also proves `under` is segment-aware, since /v1/mlops is correctly not under
/v1/ml) and each unrouted leaf by name.
/v1/train is deliberately NOT fenced: it was just deleted because the Kubeflow
CRDs behind it are not served. A gate must fence prefixes that exist, and naming
a deleted one would trip the non-vacuity check while saying nothing true.
It also does not judge the NAME. `ml` being the vague word is why this keeps
happening, but renaming a prefix that is published is a wire change with its own
cost; until that is worth paying the ambiguity is fenced, not resolved.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/train/* was a thin proxy onto two Kubeflow CRDs that this cluster does not
serve: trainer.kubeflow.org/trainjobs and kubeflow.org/{experiments,trials}.
GET /v1/train/health answers 503 degraded in production right now
({"crds":{"experiments":false,"trainjobs":false},"status":"degraded"}) with
nothing in cloud changed — the CRDs were retired underneath it. Ten operations
go, and with them the degraded door.
Measured before deleting, cluster-wide: zero TrainJobs, zero Experiments, zero
Trials, and no ClusterTrainingRuntime for a TrainJob to reference. The katib
half could not have worked at all — katib's admission webhook requires the
namespace label katib.kubeflow.org/metrics-collector-injection=enabled,
ensureNamespace writes only {managed-by, hanzo.ai/org}, and no namespace in the
cluster carries it. So POST /v1/train/experiments took the billing gate, created
an Experiment, and katib never admitted a Trial.
There were also TWO doors onto one TrainJob CRD: this one and the hanzoai/ai
broker at /v1/finetune/*, which has the product around it (presets, HF pickers,
status polling, deploy-to-serving). One door survives.
KServe STAYS. /v1/ml/models is the only path in the estate that serves a
classical artifact end to end, and it is proven: POST /v1/ml/models with an
sklearn joblib -> 201, the storage-initializer pulls the model, then POST
/v1/ml/models/{name}/predict -> 200 with correct predictions on the
kserve-mlserver runtime. GET /v1/ml/health is 200 today. Its runtime-capacity
clause and every serving test are untouched.
openapi/floor.json needs NO edit on this base: the shrink guard is a FLOOR, and
main's committed floor (1684 paths / 2336 operations) is already below the
post-deletion document, measured at 1689 / 2341. The guard still bites — raising
the floor above the real count fails TestFleetIsTheWeaveOfItsApps with the same
"THE PUBLISHED SURFACE SHRANK" report, which is how these numbers were read.
(An earlier pass here lowered a floor that ALSO carried a per-product map; main
has since dropped that map, so the rebase takes main's shape unchanged.)
The billing-gate integration tests keep
their coverage by exercising the surviving create (POST /v1/ml/models) — the
gate is the shared create() body, not a per-kind one.
Also repointed every pointer that named the deleted route, so none dangles:
apps/engine's intentRefused reason and LLM.md (now /v1/finetune/jobs), spend.go's
routing-union example, apps/platform/drift.go's analogy, and the mutation in
scripts/mutate.py whose anchor line and target test are both gone (it would have
reported ANCHOR-MISS).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
getUser sent the `<owner>/<name>` composite as `id`, and IAM wants the two as
separate fields. Measured against the running service with the console client:
?id=hanzo/2d4d67ab-… 400 field "owner" is required
?id=hanzo/z 400 field "owner" is required
?owner=hanzo&name=z 200
So NO caller of this ever read a user row. The avatar write is where it became
visible — "photo stored but the profile could not be updated", with
`iam non-envelope response (400)` behind it — but moveUserToOrg carries the same
fault silently, and its failure mode is worse: the whole-row re-submit it feeds
would have nothing to re-submit.
`name` is the USERNAME (the row's own `name`, "z"), never the UUID that `sub`
carries — the row this now reads has id 2d4d67ab-… and name z, which is why
addressing it by the uuid found nothing.
The composite is KEPT for the one caller that has no owner to send: a first-run,
org-less user, where resolving their authoritative (owner, name) is the entire
purpose of the read. Refusing that here would break onboarding to fix the avatar.
The fake IAM now insists on the same shape, so a client that regresses to `id`
for a caller that HAS an owner fails in the suite instead of in production.
A sibling dissolved PUT /v1/risk/state/appetite while this branch was open (the
decision regime has one address now), so typed.go conflicted on the op that used to
sit between `state` and `snapshot`. Resolved by taking MAIN'S typed.go whole and
re-applying this branch's five changes onto it, rather than by editing the conflict
markers: the incoming change deletes an op, and a hunk-level resolution is how a
deletion gets silently un-deleted.
zipdoc_gen.go is generated — resolved by regenerating from the merged source, never
by merging the artifact. Same for openapi.yaml and plugin/risk/openapi.json.
Verified after the merge: 11 operations before and after, floor.json identical to
main (risk 31, paths 1695, operations 2351), and main's own new gates — verb_test
and learn_cost_test — pass beside this branch's.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Regenerating every subset from source is the drift gate, and it turns up drift in two
apps beyond the ones whose prose this branch fixes. Both are real, and both are in the
direction of the published document being AHEAD of the code: it advertises money fields
the runtime stopped returning.
7b4ddd9e removed the mint from the referral read and 3a8be85b removed it from the promo
redemption. Both regenerated their zipdoc_gen.go, so the PROSE moved; neither
regenerated plugin/<app>/openapi.json, so the SCHEMAS did not. The result is 15
referral fields and 4 promo fields published as response properties no handler can
populate -- refereeGrantCents, referrerGrantCents, creditsEarnedCents, refereeBonusCents,
referrerBonusCents, grantedCents, creditsCents, the credited counters and the txn ids on
the referral side; creditCents and creditEntryId, plus the plan and seats request fields,
on the promo side. apps/referrals' assertNoMoneyKeys already fails the runtime response
if any of them reappears, so source and test agreed with each other and only the artifact
disagreed. Two fields the code does return -- Redemption.discountCents and
sweepResult.qualified -- were missing for the same reason.
Every SDK, the MCP tool list and the CLI are projections of this file, so those were dead
money fields in every generated client's types.
The operation-count ratchet did not and could not catch it: openapi/floor.json counts
paths, operations and per-product operations, and none of those move when a response
schema loses a field. It ratchets UP here -- billing 26 -> 27, operations 2351 -> 2352 --
for the newly described GET /v1/billing/tier.
Proved structurally rather than by diff, over the parsed documents with description and
summary elided at every depth: across openapi.yaml the only identity change is that one
operation GAINED, no operation was lost, no operationId moved, and all 22 contract
differences are the marketing and referrals schema fields named above. The commerce prose
entry contributes none of them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The repo's own gates could not pass on main. Both failures are the same shape --
prose that drifted from the routes it describes -- and both are fixed at the one
place the prose is written.
GET /v1/billing/tier published an operationId and nothing else, so openapi.Complete
refused commerce's document outright and `make -f mk/fleet.mk surface-check` died
there. Its handler is commerce's, in another module, and its path reaches the router
through a table rather than a literal, so there is no doc comment here for zipdoc to
lift -- which is what describe.go exists for, and what every sibling on that same
registration loop already uses. The entry states what the caller gets, that the
subject keys are pinned to the validated caller before the handler runs, that tier
is derived from active and trialing subscriptions, and the two rules a reader
otherwise gets wrong: gate on effectiveAvailable rather than prepaidAvailable,
because granted credits spend too and an account funded only by a grant reads zero
prepaid while holding real spendable credit; and a subscription-store error answers
500 rather than downgrading a paid subscriber to free.
POST /finance/starter was the inverse -- prose describing an operation that no
longer exists. The op, its middleware and its tests went with the automatic
money-mint (41b23f12) and the lift was never rerun, so commerce's registry still
explained a route the router does not carry. Regenerating drops it; nothing here
re-adds it.
POST /sites/resolve and /sites/resolve-org are live typed ops in
apps/projects/sites_rpc.go whose doc comments had never been lifted at all.
Verified per package, the way the generator loads: zipdoc -check is clean across all
100 directories that declare it, where commerce and projects were stale.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
THE MODEL WAS A PLACE. The `model` table is keyed on the tenant and written ON
CONFLICT DO UPDATE, so it is ONE CELL per organisation and every write destroys
the state before it. That single fact was three separate open problems: there was
nothing to roll back TO, so rollback needed an op that shipped the masses out to
the caller and an op that took them back in; two fitted models could not both be
named, so champion-and-challenger had nowhere to live; and no decision could say
which model produced it.
A content address ends all three. The value's name is a pure function of what
makes two models answer the same event differently — the shape, the geometry seed,
the position in the window, the threshold, the masses as IEEE-754 bits, and the
FOLD WATERMARK, which is not redundant with the mass count: two models with
identical masses reached by different routes disagree about what is left to fold,
and one will re-teach history the other will not.
THE IN-PROCESS MODEL STAYS MUTABLE, and that is the design. Measured: one encoded
value is 466 KiB at the reachable shape — 25 trees of 511 nodes over two windows,
every mass carrying a full mantissa because folding blends them. The sweep writes
every 500 events or every 30 seconds, so "a value per write" is 56 MiB an hour per
active organisation to record a counter going up. So identity is a SUCCESSION OF
STATES: the `model` row stays a place and its whole job is resuming a killed
process, and `published` is append-only, addressed by content, retained under a
budget stated in BYTES (10 values, following policyBudget's reasoning).
WHAT THIS DELETES. riskSnapshotBody is gone from the wire in BOTH directions, with
its two conversions. Publishing answers with a NAME; adopting takes one. The
masses never leave the organisation's own store, so the caller stops being the
custodian of a tenant's model — the engine's own Restore asks for exactly this
("it belongs where the tenant's own data belongs, and sealed if it travels"). Two
tests changed from refusing a threat to proving it unreachable: a caller can no
longer describe a model instead of naming one, so there is no body to compose and
no seed to choose. Rollback is naming a prior address; what the working state
descends from is DERIVED from its mass count, so rolling backward is right for
free where a stored head pointer is exactly what would fall out of step.
A SCORE NOW NAMES THE MODEL SPACE IT RAN IN. The shape is cached on the residency
and read in the same critical section as the verdict, so it costs nothing and
cannot cite a space the score did not run in. It is deliberately NOT an address:
the masses at the instant of a score are counters between two published values, so
citing one would claim that value produced this score. The shape, the policy
version and the event's own time are what IS true, and the value history's clock
brackets the decision from there.
ISOLATION, MEASURED RATHER THAN ASSERTED. The address deliberately omits the
organisation — a name that must be unguessable is obscurity, not isolation. Two
mutations were run and two drafts of the test comment were wrong before they were:
the per-organisation FILE and the `tenant = ?` row predicate are two layers and
EITHER ONE ALONE HOLDS, so a foreign org handed a real address resolves nothing
through all three doors with either layer removed. The single-layer regression is
caught where the file layer is already absent by design — two brands' identically
named organisations share one file — and that test fails on that one mutation
alone. The finding underneath inverts the obvious reading: the per-org file is not
what makes this isolated; the row predicate holds on its own.
11 operations before, 11 after: no path moved and the floor is untouched. The two
paths SHOULD collapse into one (/v1/risk/state/model, POST to publish and PUT to
adopt) — that is the prefix plane's call, not this commit's.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/risk/learn recorded a batch, trained on it AND answered the model's
verdict on every event. Three things under one name: you could not observe
without training, and could not train without being answered.
An observation is a VALUE the plane records; learning is a TRANSFORMATION over
observations; a verdict is a QUERY against the result. POST /v1/risk/score is
already the query and is already pure. So learn drops the verdict and answers
`learned` — how many events the model actually learned from.
WHAT IT COST TO CARRY. plane.learn called the engine twice per event, Inspect for
the response's verdict and Assess for the counters, and both enter the engine's
judge: two projections of the point at three aggregate reads each, and two walks
of the forest. Above the cut both also ran the counterfactual attribution, a
further walk per dimension over nine dimensions. Assess's own return was
discarded, so the attribution was computed twice and thrown away once.
MEASURED, and stated as measured (learn_cost_test.go, BenchmarkLearn):
before after
batch 8 31.5 µs/event 27.8 µs/event -12%
batch 128 20.0 µs/event 17.5 µs/event -12%
batch 128 6717 allocs 6169 allocs -8%
A tenth, not a half: the durable record and the aggregates are the larger part of
what a caller waits for, and the attribution is reached only by the share of the
stream the appetite admits — one per cent by default.
NOTHING DEPENDED ON THE SYNCHRONOUS VERDICT. No CLI command references risk, no
SDK carries a risk client, and the one in-process consumer of a verdict is
cloud.Decide, whose scorer is never installed outside tests (SetRiskScorer has no
non-test caller), so every question it asks answers {allow, scorer-absent}. The
live plane reports one resident model built once. Observe-and-judge in one round
trip is now a COMPOSITION and the published prose says which order: score first,
then learn, so the verdict is the model's opinion of an event it has not yet
learned from.
A DUPLICATE IS NOW WHOLLY INERT. It moved nothing before and was still judged;
now it costs no model work at all, is not counted in `learned`, and is not
metered — the meter runs on what was DONE, which is this app's own stated rule
for the gate/meter pair. The gate still bounds the batch the caller stated,
because how much of it is new is unknowable until the record is written.
TestPlane_TheVerbsStaySeparate (verb_test.go) walks the package and fails if
learn calls Inspect or score calls Assess. It is structural because the braid is
invisible behaviourally: with Inspect put back, every other test in this package
still passes.
zipdoc regenerated; plugin/risk/openapi.json and openapi.yaml rewoven
mechanically (make -f mk/fleet.mk openapi-weave, exit 0). riskScoreOut, riskCause
and riskValue stay in the document — score still answers them; they are simply no
longer reachable from riskLearnOut. Package green under -race.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/risk/state answered seven kinds of thing at once and PUT
/v1/risk/state/appetite answered all of them back to a call that changes three
numbers. This lands the half whose home already exists: the decision regime is
read and written at ONE address, /v1/risk/policy, and both verbs answer
riskPolicyOut. The published surface loses a path and keeps its operation count.
plane.appetite no longer computes the model value — r.mod.State and r.vel.strain
are gone from the policy write. What the model IS stays for the model-value track;
the telemetry kinds (refusals by reason, blind by feature, the realised share,
saturation, aggregate strain) stay on GET /v1/risk/state until they are emitted on
the fleet's one metric road, because deleting them before that would remove an
organisation's only view of its own refusals.
Per-organisation isolation is proven through the new address by the two policy
tenancy tests. Mutation table in the commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The regime was written at PUT /v1/risk/state/appetite and read at GET
/v1/risk/policy — two addresses for one plane, the writing one named after the
mutable spot the regime happened to sit in. `state` is literally the word for
that spot; a regime is not a spot. It is three numbers an organisation adopted,
at a time, by a named identity, kept under a version forever after.
Worse than the second address is what the write ANSWERED: riskModelState, fifteen
fields describing the whole MODEL — its shape, what it had learned, whether it
was warm, the threshold in force, the realised share beside the stated one, every
refusal by reason, every feature that read blind, the fold coverage of the event
surface, and the aggregate strain. A call that changes three numbers knows none
of that. It was reporting a model it happened to be holding a lock on, which is
the same braid, one level up, as the regime living on the learned state's row —
the defect the versioned policy record was cut to fix.
So the plane has ONE address and both verbs answer riskPolicyOut. plane.appetite
returns the VERSION it left in force and nothing else: the two calls to
r.mod.State and r.vel.strain are gone from the policy write, so a policy write no
longer computes the model value at all. policyOut is the one projection both verbs
render through, and the version in force is a PARAMETER to it — a write knows what
it enacted from inside the lock it enacted under, and re-reading it there would let
two concurrent restatements each report the other's version.
There is no `minted` flag. plane.enact is idempotent on the regime, so a
restatement answers the version already in force and equality of the VALUE is the
signal. A boolean about the operation is a second thing to keep true beside the
value that already says it.
The published surface SHRINKS by one path and holds its operation count: the write
joined the address that already existed instead of keeping a second one. Mechanically:
paths 1696 → 1695 (-/v1/risk/state/appetite)
operations 2351 → 2351 (PUT moved onto /v1/risk/policy)
openapi/floor.json is lowered in this commit because the weave refuses a shrink
otherwise, which is the gate working: a reduction is reviewed next to its reason.
The op's published schema description loses 27 lines of riskModelState /
riskSurface / riskAggregates prose it had no business publishing.
Nothing consumed the old address. Swept every checkout under ~/work: the only
references outside apps/risk's own tests were the derived specs. Live, PUT
/v1/risk/state/appetite is routed and GET /v1/risk/policy is 404 — the policy
plane is merged but not yet deployed — so this is the one moment the consolidation
costs a deployed client nothing.
Per-organisation isolation is untouched and proven through the NEW address:
TestPolicy_HistoryIsPerTenantOverTheWire and
TestPolicy_TwoBrandsShareAFileAndNotAHistory both state their regimes at PUT
/v1/risk/policy and both still see nothing of the other organisation.
Mutation table — each named test RED under the defect, GREEN after revert:
M1 re-register PUT /v1/risk/state/appetite TestPolicy_HasOneAddress RED
M2 put a model field back on the answer TestPolicy_AWriteAnswersThePolicyAndNotTheModel RED
Verified: go build -tags "sqlite_purego sqlite_math_functions sqlite_fts5" ./... =
0; go vet apps/risk = 0; go test apps/risk = ok; -race = ok (23s); zipdoc -check
./apps/risk/ = 0. Fleet-wide zipdoc -check reports the same 16 stale/missing
generated files before and after this change (measured by stashing it), so this
adds no drift: apps/risk is not among them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/iam/* was forwarded to a separate identity origin by an edge in cloud:
a tenant gate, an org pin, a read/write allowlist and a sign-in passthrough,
all re-deciding what IAM already decides for itself.
It mounted on `!cfg.Enabled("iam") && iamHost() != ""`. An empty enable list
mounts everything (Config.Enabled), and no deployment in the fleet sets one,
so Enabled("iam") is true everywhere and the condition never held: the iam
app owns /v1/iam through its graft, on every deployment, and has since the
graft landed. The edge answered no request anywhere.
Deleting it removes the second reader of the identity address and the second
place a tenant scope is decided. The prefixes are unchanged — manifest's iam
row already routes them — and the org pin the edge applied is IAM's own
authorize, in the process that owns the store.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Removing CLOUD_ENABLE/--enable (ecafb31c) deleted the flag from the binary but
left three launchers passing it. Go's default flag.CommandLine is ExitOnError,
so this is not a warning:
$ ./bin/cloud --enable=iam
flag provided but not defined: -enable
`make run` and the documented compose quickstart both died before boot, and the
four README brand examples were copy-paste instructions to do the same. The
removal was deliberate and test-enforced (cmd/cloud/mount_test.go asserts the
flag's absence) — the launchers were simply never updated with it. My miss.
RUN_ENABLE is renamed RUN_PLUGINS because it never mounted anything: it is the
list of plugin BINARIES to build so local dev does not build all 106. The host
mounts what manifest.Apps lists and resolves each plugin as a file beside
itself; a lazy one with no binary never starts, a Required one fails loudly.
Keeping the name would have said the binary takes a mount list, which is the
thing that took devnet down twice.
compose.yml's HANZO_ENABLE goes with it. Its environment: block is inert
anyway — HANZO_BRAND/DOMAIN/DATA_DIR have no Go read sites and work only
because the same file passes the flags, which main.go forward() republishes as
CLOUD_*. That is a separate cleanup, not this one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/risk answered ONE refusal in TWO shapes, with the same status and the same
sentence in both. Measured on this package's priced ops:
this app's own copy of the rule 403 {"status":403,"error":"no validated principal"}
the fleet's money door 403 {"error":{"code":"forbidden","message":"no validated principal"}}
The nested one is canonical — it is what the edge gate and every other Hanzo
surface emit, so a client that reads `error.code` reads it everywhere. The flat one
was zip rendering a returned error, which meant `error.code` was absent from
exactly one product's refusals.
[ops.gate] held its own `if ledger == ""` copy of the rule from before the fleet
door had one. [cloud.ResourceMeter.Gate] now refuses an empty org above both of its
branches as [cloud.ErrNoLedger], and [cloud.denial] renders that as the 403 with
this exact sentence, so the copy decided nothing except the shape. Deleted; the
twenty-four lines of prose explaining a defect that is fixed elsewhere are replaced
by a citation of where.
Deleting it exposed a residual the report did not name, because a status assertion
cannot see a shape: POST /v1/risk/search reaches [ops.admit] BEFORE it prices
anything, so its refusal came from [tenantOf] — still flat. Eight ops nested and
one flat is worse than nine flat, so tenantOf's no-principal branch answers with
the same fleet value. A malformed org is a different fact and keeps its own
sentence.
And the hole the deletion could have opened, closed: ResourceMeter.Gate returns
EARLY at zero cost, before its own empty-org refusal, and the price is an operator
knob where 0 is legal. So "the money door refuses an unidentified caller" holds
only while somebody is charged; the app's deleted copy had covered that by
accident, running before the price was computed. tenantOf covers it on purpose, and
a test sets the price to zero and requires the same 403.
E1 risk holds its own copy again TestPricedOps_RefuseAnUnidentifiedCallerInTheFleetsOwnEnvelope RED
E2 tenantOf answers flat again TestPricedOps_RefuseAnUnidentifiedCallerInTheFleetsOwnEnvelope RED
E3 tenantOf answers flat again TestPricedOps_RefuseAnUnidentifiedCallerEvenWhenTheOperatorPricesThemAtZero RED
E4 the fleet door stops refusing TestPricedOps_ResolveTheTenantBeforeTheyAskForMoney RED
E5 identity takes a money status TestPricedOps_ResolveTheTenantBeforeTheyAskForMoney RED
All GREEN after revert. E4 and E5 mutate cloud's own money door, which is where
gate_order_test.go's mutation note now points — the note said "delete the
empty-ledger guard from ops.gate and every op reports 503", and that had silently
stopped being true.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The label row said a specific prefix MUST be registered before the bare /v1/risk
or "every label op lands on the decision plane", and cited
TestSpecificPrefixesPrecede as the gate that pinned it. Neither half was true.
No test of that name exists anywhere in the tree, and the rule is not the
router's: registering the bare /v1/risk FIRST and all three specific prefixes
after it, the live router still delivers /v1/risk/labels to label,
/v1/risk/reference to reference and /v1/risk/datasets to dataset, and the router
oracle stays green. zip resolves nested static prefixes by specificity; mount
order decides only between EQUAL patterns, which is what the ai-before-zen note
is actually about.
Says that, and names the gate that does hold —
TestEveryServedPathReachesTheAppThatServesIt builds the real router from these
rows and asks it, per published path, which app receives the request. An oracle
over every row at once is why no row needs a rule of its own to remember.
Prose only; no row moves and no behaviour changes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The dataset plane addressed /v1/ml/datasets, beside the KServe model-SERVING
plane's own /v1/ml/health and /v1/ml/models — one prefix, two products.
openapi.Product reads an operation's product off the FIRST /v1 segment and
nothing else, so `ml` published 14 operations that were seven serving ops with
customers on them and seven dataset ops whose rows feed the risk model, which
learns in-process from the org's own events and is never served by KServe.
Moves the five paths to /v1/risk/datasets and renames the seven operation ids
and nine schema names to the risk face, so the SDK method, the CLI command and
the generated type each name the product they belong to. Fourteen of the fifteen
Go types take the bare risk<Noun> name; the dispose pair carries the noun it
disposes of, because apps/label already publishes riskDisposeIn for LABEL
disposal and the fleet's schema namespace is flat.
The tag was never the product: openapi.Fold assigns op.Tags from the router
projection, so zip.WithTags("ml") had been inert. It now reads "risk" too, so a
declaration and a projection do not disagree.
/v1/ml is untouched. Its four serving paths (7300 bytes) and its three schemas
(1533 bytes) are byte-identical, and regenerating the fleet document from source
moves exactly the seven dataset operations and the nine dataset schemas: 2350
operations, 1695 paths and 2060 schemas before and after, with every other
triple unchanged.
No alias and no redirect from the old prefix, because nothing calls it: a sweep
of every git checkout under work/hanzo finds /v1/ml/datasets named only in this
repo's own generated artifacts and prose. No client, SDK, CLI or MCP consumer
names it.
The floor ratchet refuses a shrink, so `ml: 14 -> 7` is lowered here in the same
commit that moves the routes; `risk` rises 23 -> 30 on the same regeneration.
address_test.go makes the address a gate rather than a third recollection after
label and reference: the product, the operation-id prefix and the schema prefix
are each asserted off the published projection, and each assertion fails when
the projection is empty, so none of the three can pass by examining nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
kserve admits an InferenceService whose model format no ClusterServingRuntime
supports and then never schedules it, so /v1/ml/health answered 200 while every
deploy hung. A served CRD is not capacity, and the probe read only the CRD.
health now takes the cluster-scoped coordinate the plane needs at least one of
(the zero GVR for a plane with no such fact — training carries its own images)
and reports the count as its own field. An unreadable list reports the read
error instead, because a missing grant is a broken probe and not an empty
cluster, and the two call for different acts.
scripts/mutate.py carries four rows: dropping the clause, folding the read error
into the count as zero, asking capacity of training, and reading the runtime kind
at the InferenceService's v1beta1 instead of v1alpha1.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The bounds on `review` and `sample` were spelled twice — once at [ops.appetite]
and once in [admitRegime]. Collapsing them to one spelling is right. But the
SURVIVING spelling had no test: admitRegime was made to `return nil`
unconditionally and the WHOLE package stayed green, so the published contract
(`review ∈ (0, 0.5]`, `sample ∈ [0, 1]`) was held by code that could be deleted
without one failure. Same shape as a control switching itself off, one layer down:
the bound is present and nothing measures it.
Both halves are asserted, because a refusal that half-applies is worse than no
bound: six out-of-contract appetites are refused 400 with the field named, AND the
regime in force is untouched afterwards — no version minted, no history row, the
model still deciding under 0.02/0.10. Over the WIRE, because that is where the
contract is published and where the op's deleted copy used to answer.
A1 admitRegime returns nil for every regime named test RED, whole package RED
A2 plane.enact stops calling admitRegime named test RED, whole package RED
Both GREEN after revert.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An organisation that took its model out of shadow BEFORE the model had learned
anything was told live=true and had nothing written down. The regime lived on the
same row as the learned state, and that row's writer declines to write while the
snapshot holds no learned mass — correctly, because there is no state to lose. So
PUT /v1/risk/state/appetite answered 200, reported live, and persisted nothing;
this binary deploys Recreate at one replica, so the next rollout rebuilt from
defaultConfig — shadow — and the model decided nothing. No error, no log, nothing
to alert on: a model silently disarmed, on a routed door.
Two conflicts, both resolved by REGENERATING rather than choosing a side:
openapi.yaml and openapi/floor.json are derived, and the branch was cut when the
fleet had 1684 paths. `make -C apps/risk describe` + the weave produce 1696 paths
and risk 24 operations — main's 1695/23 plus this branch's one op, GET
/v1/risk/policy. apps/risk/zipdoc_gen.go and plugin/risk/openapi.json regenerate
byte-identical to the branch's, so the projection did not drift.
One semantic conflict: `plane.close` gained the shutdown window on main
(cda5a031), so the branch's four `close()` calls in policy_test.go take
context.Background() like the other eighteen in the package.
Mutation table re-run against THIS merge, since main moved under the branch —
each named test RED under the defect, GREEN after revert:
M1 plane.enact writes no durable row TestPolicy_GoingLiveSurvivesTheRollout RED
M2 verdict drops the version citation TestPolicy_EveryScoreCitesTheRegimeItWasDecidedUnder RED
M3 no value-equality short circuit TestPolicy_ARestatementOfTheSameRegimeMintsNoVersion RED
M4 the 24-per-24h bound deleted TestPolicy_TheRateBoundBindsAndIsNamed RED
M5 a pre-existing regime is not adopted TestPolicy_ARegimePredatingTheRecordIsAdopted RED
M6 the history read loses its predicate TestPolicy_TwoBrandsShareAFileAndNotAHistory RED
M7 retention stops disposing TestPolicy_RetentionIsCountedAndNotSilent RED
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Four defects on the LIVE /v1/risk plane, each closed so the wrong thing is
unrepresentable rather than merely unlikely, plus the harness rows that hold
them.
THE ORG STORE. CloseAll emptied its maps, which made "closed" indistinguishable
from "nothing opened yet" — so a request still in flight during a rollout
reached For() after shutdown and opened the file again, re-hydrating it and
re-claiming the fence lease the SUCCESSOR pod was claiming. Closing is now
terminal and every door answers ErrStoreClosed.
THE ROLLOUT. close() was `stop(); wg.Wait()` with every save queued BEHIND it.
The wait was unbounded inside a 30-second window while ONE search's durable
Sync is bounded at 30 seconds on its own, so the process was killed with not
one tenant's model written down — every tenant back to warming, and a warming
model refuses to score, which reads as clean. The saves now run
unconditionally; the drain gets what is left of the caller's own window; a
drain that did not finish is ErrDrainIncomplete and not a silence.
THE SEARCH BOUND. "ONE RUN PER TENANT" checked the slot and set it two
warehouse operations later, so sixteen concurrent callers each rolled the
tenant's source planes and read its whole history before any of them claimed
anything — and the ledger gate was the LAST thing in the sequence, so all of it
was free. The slot is claimed atomically with the check, and a search is priced
in TWO halves, each before the half it prices: the surface on the window the
caller stated, the grid on the measured history.
THE STRAIN REPORT. velocity caps PER SHARD, so the store drops a tenant's
subjects long before the flat census notices: 200 subjects in, the store holds
198, and the report said forgotten=0, saturated=false. Two of that
organisation's own subjects read as "has done nothing" and nothing said so.
reconcile now measures the store/census shortfall as a high-water mark, and the
report describes the bound that actually binds.
THE RESIDENT BOUND had no test at all: evict() could be made to return nil —
the bound fully disarmed, an OOM on a one-replica Recreate deployment — with
the whole suite green. Its operating point is now four properties held
together: SERVED past the bound, BOUNDED at it, LOUD on the probe, LOSSLESS on
the way out.
Mutation-proved: scripts/mutate.py "risk:" — eighteen rows, each reintroducing
one defect and going red on the named test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The ceiling was the wrong way round. maxOverrides was a chosen number (10,000)
and the byte figure was computed beside it from `row` — the SUM OF THE WIRE
TERMS, 1,152 bytes. A row does not cost what it carries: measured on a real
per-org store, the widest row this door admits costs 1,952 bytes once the
implicit index over the primary key, page slack and the encryption are counted.
So the published 120 MiB per-organisation ceiling was understating the truth by
1.69x, on the ONE volume every organisation's store shares.
Inverted, so the bound is in the dimension that binds:
ownBudget = 128 MiB the primary figure — what one org may occupy
rowBytes = 2048 MEASURED (1,952) plus a page of room
maxOverrides() = ownBudget / (sets x rowBytes) = 5,957 per set
The count is now the division rather than a number standing next to one, so
count x rowBytes IS the byte bound. Adding a set lowers the count instead of
quietly multiplying the volume.
`row` is renamed `stated` and keeps its old meaning — the WIRE width — because
the two figures are different quantities and conflating them is what caused
this. TestTheVolumeOneOrgMayOccupyIsStated now checks the quotient (one more
entry per set must not fit) instead of pinning a constant, and
TestOneOrgsOverridesCostWhatTheyArePublishedToCost fills a real store with
worst-case rows and fails if one costs more than the figure the count is divided
from — so rowBytes can never drift from a store again.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The header claimed "nothing held the mechanism at all — the whole suite stayed
green". That was true when it was written and is not true now: hold_test.go
landed the cross-tenant reclaim tests on main, and MEASURED with
scripts/mutate.py, TestEviction_IsCountedOnTheProbe and
TestEviction_WritesTheVictimsStateDownFirst kill all three mutations that disarm
this bound. A test whose stated reason for existing is false is a test the next
reader deletes for the wrong reason, or keeps for one.
What it actually adds is two things the pair cannot see: the bound asserted at
every one of maxResident+8 arrivals rather than once just past it, and a lossless
leg with no t.Skip exit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The surface read — rolling up to four source planes into the organisation's own
feature surface and reading the window back — ran BEFORE ANY GATE AT ALL. A
caller with no balance drove the whole warehouse cost, was refused at the very
end, and paid for none of it, as often as it cared to ask. Two rows: the gate
moved back below the work it prices, and the surface half deleted so only the
grid is paid for. Each goes red on the test that names the property.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Sixteen rows for the org-store lifecycle, the rollout, the search bound, the
strain report and the resident bound — so every assertion this cycle added is
held by a mutation that breaks the thing it guards, in the harness rather than in
a transcript.
AND THE HARNESS COULD DESTROY ITS OWN EVIDENCE. The restore is in a finally,
which does not run when the process is killed — a CI timeout, a ^C — so an
interrupted run leaves the mutant in the tree and the original in .mutbak. The
next run then copied over that backup, destroying the only clean copy, and
reported ANCHOR-MISS on a tree it had silently corrupted. That happened here and
cost a file. A leftover backup is now a hard refusal that says how to recover.
One row is written against the CONDITION rather than as an early return, because
`return nil` after the guard is unreachable code: go vet rejects it, the row
scores NO-COMPILE, and a mutant that never ran proves nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The run slot now bounds one tenant's CONCURRENCY. Nothing bounded its RATE: the
setup ran before any gate at all — the ledger check was the LAST step of begin —
so a caller with no balance drove a roll of up to four source planes and a
full-window read on the one warehouse every tenant shares, was refused at the
end, and paid for none of it, as often as it cared to ask.
Pricing the grid on its upper bound would close the same hole and leave no
viable operating point: the upper bound is maxHistory x the whole grid whatever
that organisation's history holds, so a tenant with two hundred events would be
refused unless it could cover twenty thousand.
So a search is priced TWICE, each half before the half it prices. The surface
from the WINDOW, which the caller states and which is therefore a number before
anything runs — the same unit and the same price ops.features already pays for
the same work, now spelled once in windowScreens. The grid from the MEASURED
history, where it already was. Each meters on what it actually did, so a run
cancelled by a rollout is still billed for the trials that ran.
The plane takes ONE money seam (charge: gate n, get the meter for what was done)
instead of a gate parameter and a book parameter, which is what makes "gate
before, meter after" the same shape at both halves rather than a convention.
And the debit test now waits for BOTH debits. With a synchronous surface debit
landing first, waiting for one was satisfied without the background meter ever
running — the fixture would have made the property it exists for unobservable.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
velocity applies its cap PER SHARD — MaxKeys/shards+1, which is FIVE keys against
a published census ceiling of 320 — and subjects hash unevenly, so a shard fills
long before the total does and the store drops that shard's least-recently-used
key. The census is a flat count and saw none of it. Measured:
subjects=200 storeKeys=198 censusLen=200 forgotten=0 SATURATED=false
subjects=320 storeKeys=290 censusLen=320 forgotten=0 SATURATED=true
At 200 subjects two of that organisation's own subjects are gone and the model
reports itself healthy; each one reads as "has done nothing", scores as
unremarkable, and raises nothing. At 320 the state is right by luck and the COUNT
an operator acts on still says zero while thirty subjects are gone. The published
8 MiB budget buys 320 subjects and the bound that binds starts biting near 200 —
a bound stated in one dimension and enforced in another.
reconcile already read the store's own key count and threw the comparison away.
It now measures the shortfall against the census under ONE lock, so the two
numbers describe one moment, and keeps it as a HIGH-WATER MARK: a dropped subject
is re-admitted the moment it is active again, which closes the shortfall but does
not un-blind the window in which that subject read as inactive. A gauge would
report that a control which switched itself off never did.
ALSO: the resident bound had no guard at all. `evict` could be made never to trip
and the whole suite stayed green — the bound fully disarmed, residents growing
without limit, which is 64 x (8 MiB of rings + its model) against a 9 GiB
GOMEMLIMIT on a ONE-replica Recreate deployment. A bigger constant was never the
fix; the operating point is four properties at once, and each is a way to be
wrong: SERVED past the bound (a refusal is a cliff, not an operating point),
BOUNDED at every step, LOUD on the probe (eviction is lossless, so the count is
the only sign), LOSSLESS (otherwise one tenant's arrival costs another its model,
which is the cross-tenant defect wearing a capacity hat).
Mutation-proved: scripts/mutate.py "strain"/"bound" — seven rows. Dropping the
store's half of Saturated reports "the store holds 142 of this organisation's 144
subjects and strain reports saturated=false". Refusing past the bound reports
"organisation 64 of 72 was refused a residency". Dropping the evicted tenant's
save reports "an evicted organisation came back having learned 0 of 120".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TWO DEFECTS, both "a bound that is not on the dimension that matters".
THE ROLLOUT LOST EVERY TENANT'S MODEL. close() was `p.stop(); p.wg.Wait()` with
every save BEHIND that wait, and the wait had no bound at all — while Shutdown
was handed the shutdown window as a context and threw it away. The process gets
30 seconds (serve.go) inside a 60-second grace period, and ONE search finishing
after cancellation writes its result to a shelf whose durable Sync is bounded at
durableOpTimeout — thirty seconds, the whole window, on its own. So the wait
outlived the window, the pod was killed, and not one resident model had been
written down. Every tenant returned to warming, from one tenant's search, once
per deploy — and a warming model REFUSES to score, which reads as clean to
anything that does not check the refusal.
The saves now run unconditionally and never behind the drain; background work
gets whatever is left of the caller's own window, bounded by it and by
drainBudget; and a drain that did not finish is ErrDrainIncomplete — a named
state joined into the error, with the count of models written down anyway, rather
than a silence.
THE SEARCH BOUND BOUNDED NOTHING EXPENSIVE. begin() checked "is a search already
running for this organisation?" and set the flag that answers it two warehouse
operations later — check-then-act with the entire cost of a search setup in the
window. Measured: 16 concurrent callers for ONE organisation were all accepted,
all rolled that tenant's source planes, and all read its whole history, against
the one warehouse every tenant shares; 16 background grids then raced for one
shelf row. The slot is now claimed atomically with the check that grants it,
before any of that work, and released on every path that does not reach the run.
claim/settle/unclaim are one fact in one place — the background goroutine's
hand-rolled delete is gone with them.
Mutation-proved: scripts/mutate.py "rollout"/"search" — six rows. The original
close() ordering does not merely fail, it HANGS to the test timeout, which is the
production SIGKILL. Making the saves conditional on a clean drain reports "tenant
A came back having learned 0 of 300". Restoring the check-then-act reports "16 of
16 concurrent searches were accepted".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
OrgStore.CloseAll closed every handle and then installed fresh empty maps, which
made "closed" indistinguishable from "nothing opened yet". Every one of the
fifteen callers is a Shutdown path, so a request still in flight during a rollout
reached For() after it and opened the file again — on a durable deployment
re-hydrating it and re-claiming the fence lease the SUCCESSOR pod was claiming at
that moment. Two live writers for one org, through a handle nothing thought was
reachable.
Closing is now terminal: the flag is set before the maps are cleared, so there is
no window in which the store is empty and still openable, and every door that can
open a file answers ErrStoreClosed. An arrival after shutdown is a fact worth
surfacing, so it is an error rather than a silent no-op.
The risk plane reaches this path by construction — its own close() empties the
resident map, so an in-flight score rebuilds a residency and asks for the shelf.
Mutation-proved: scripts/mutate.py "orgstore" — three rows, one per half of the
mechanism (the flag, the For guard, the Each guard). Each goes red on
TestOrgStoreCloseAllIsTerminal at a distinct assertion.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ownVolume() is what one org may occupy on the volume every org's SQLite file
shares: maxOverrides x len(Catalog()) x row, where row = maxKey + maxNote + 128.
The key and the note were bounded at the wire door. The third term was not: the
writer is actor()'s reading of the X-User-Id the request carries, so it was a
caller-sized value stored on up to 10,000 rows in every set the catalog
publishes. A count over caller-sized values is not a byte bound, and the stated
120 MiB ceiling was a figure nothing held to.
maxActor = 128 (three times the UUID IAM mints), enforced at the store door
rather than the wire op — the row is what the budget is about, so bounding it
where a row is written covers every path that reaches the store and not only the
one a reviewer read. row now DERIVES from it, so raising the bound moves the
ceiling instead of silently detaching it.
Refused, never shortened: the writer is who an adverse action is attributed to,
and a truncated identity names someone who does not exist. errActor separates
the two refusals put() can produce — the row cap is a 409 (the org already holds
what it holds), an over-long writer is a 400 (a value that arrived on this call).
TestTheWriterOnARowIsBoundedLikeEveryOtherTerm rides the wire, because the
defect was in what the wire hands the store: a unit test on put() would have
passed against a handler that never bounded the header. call() is now callAs()
with the default principal, so there is one request builder and not two.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/ is where mounted, billable subsystems live: 116 of its 139 directories
declare `func Mount(`. apps/money declares none, serves zero routes, and is 216
non-test lines that already wrap github.com/hanzoai/money (imported as `hz`) to
pin ONE unit — creditUSD, dollars at 18 decimals. That is a library, and filing
it under apps/ said it was a product.
Pure path move: apps/money -> money. No API change, 44 import lines rewritten,
0 residual references. Full tree builds (`go build -tags sqlite_math_functions
./...` rc=0).
Not deleted and not folded upstream: hanzoai/money is the general library
(multi-currency, rates, min/max/sum) and this is the platform's credit-asset
facade over it. creditUSD is Hanzo-specific and does not belong in a public
money library. One library, one facade, each in the right place.
Unrelated, found while verifying: `go build ./...` fails on
hanzoai/base@v1.5.11/core/sqlite_math_required.go (undefined
cgoBuildNeedsSQLiteMathFunctions) on a CLEAN tree too — it needs
-tags sqlite_math_functions. The default build command does not work in this
repo and that is worth fixing separately.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ResourceMeter.Gate is the one pre-create gate all 23 priced surfaces call, and
handed an empty org it asked the money plane to price a spend for a nameless
subject. The money plane then answered a question about IDENTITY in the
vocabulary of MONEY, in two measured shapes:
co-resident ledger — metering refuses an empty org fail-closed, which is not a
4xx, so the wire's fallback renders 503 "Billing temporarily unavailable":
the caller is told the biller is broken.
peer ledger — gatePeer ships AuthorizeIn{Subject:""}, commerce's own
validate:"required" rejects it, and because that refusal IS a 4xx the wire
preserves it verbatim: 400 `field "subject" is required` — a field that
appears in no published request schema on any of these surfaces, so no
caller can ever satisfy it.
apps/risk already carried this guard at its own door; the same disagreement is
reachable wherever a handler's tenant check and principal.Ledger differ.
provisioning is the live instance: create() admits an admin with no org, and an
org over MaxOrgLen, through tenant() — and Ledger answers "" for both. Its seven
routes serve on api.hanzo.ai today.
An empty org is now refused as ErrNoLedger before either branch runs, and denial
— the one decision both renderings read — renders it as the tenant gate's own
403 rather than as a fault of the biller. This changes no allow/deny outcome on
a named caller: an empty org already failed on both branches. It does close a
worse case than the wrong status, proven by the mutation below: under fail-OPEN
with the biller down, an unidentified caller was allowed, so the surface was
free rather than gated.
Mutation-proven via scripts/mutate.py:
gate: ask the money plane to price a spend for a nameless subject KILLED
gate: render an identity refusal as a fault of the biller KILLED
risk: widen the empty-ledger guard until it refuses every caller KILLED
Two rows are deliberately absent and say so in place: "guard below fail-open"
is a semantic no-op (fail-open lives inside Authorize/gatePeer, so no placement
in Gate can follow it), and "delete ops.gate's guard" now SURVIVES — with Gate
refusing, the risk-level guard is no longer load-bearing for the status or the
sentence, only for the envelope.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
plugin/<app>/mcp.json stopped being a generated artifact when the door started
asking each subsystem for its tools at the moment it is asked (package fleet,
mk/plugin.mk describe). The committed copies were the exact hazard that change
removed: plugin/o11y/mcp.json held 12 tools while the o11y binary served 365,
and nothing compared them.
plugin/dataset/mcp.json was the last one left — no go:embed names it, no Go
source opens it, and every remaining mention in the tree is prose recording that
the mechanism was retired. Removing it leaves 0 of 124 app dirs carrying one, so
"the tool catalogue is not an artifact" is a property of the tree rather than a
sentence in a comment.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The last attempt at this served a plane op from deps.AI, which in the `ai`
process is the HTTP gateway to api.hanzo.ai — so it added a hop instead of
removing one, and was reverted. The premise was right and the mechanism was
wrong. zip states the mechanism itself:
"Because it is an ordinary route on the app, it rides EVERY transport the
app Listens on with no extra wiring — ZAP over a unix socket is simply the
address the caller dialed."
`ai` already answers its whole /v1 surface on $ZIP_RUNTIME_DIR/ai.sock. Nothing
in hanzoai/ai has to change and no second inference API has to exist: a sibling
speaks the SAME OpenAI-compatible wire to the SAME routes, over the socket.
before sibling --HTTPS--> Cloudflare --> ingress --> api.hanzo.ai --> ai
after sibling --unix----> ai
AIHTTPOn / AIHTTPM2MOn state the TRANSPORT separately from the address, so
there is still ONE inference client; only the route it travels differs. The
M2M token exchange keeps the default transport — IAM is a different peer and
naming it is a separate question, still open.
aiRoute is the one decision both pickers share, so completions and embeddings
cannot disagree about where the peer is. It is answered by WHAT THE PROCESS IS:
!Enabled(ai) means this process does not carry the app, which is exactly when
`ai` is a sibling. The ai process and the host keep the configured address —
routing inference back through the picker there would be a self-call, and
TestTheAIProcessDoesNotDialItself pins it.
The socket transport WAKES the peer through the same reach() every plane call
uses, so a cold lazy app is "not started yet" rather than "not deployed here".
CLOUD_AI_ZAP_ADDR is deleted with its field: it made a deployment state where
its own code lives, which is the thing being removed. CLOUD_AI_BASE_URL now
only describes inference that is genuinely elsewhere — and on the Hanzo
deployment it is no longer consulted, because that pod carries `ai`.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The app was renamed to /v1/risk/reference — openapi.Product reads the product
off the first /v1 segment, and these six operations are the risk product's —
but the committed projection and the woven document were generated before the
rename and still published /v1/ml/reference. manifest's
TestEveryServedPathReachesTheAppThatServesIt is the gate that caught it: four
paths the fleet published and routed to the /v1 catch-all instead.
Regenerated plugin/reference/openapi.json from the app's own live router and
re-wove openapi.yaml from the subsets. The delta is exactly the four reference
paths; no neighbour moved.
plugin/reference/mcp.json goes with it. The tool catalogue stopped being a
generated artifact when the door started asking each subsystem for its tools at
the moment it is asked (package fleet) — 122 of 124 apps ship no such file, and
a stale one beside a regenerated projection is a second answer to a question
that has one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The coordinates are the part worth proving, and ai's own tests cannot reach them.
ai declares the store structurally (object.SecretStore) precisely so it needs no
KMS dependency — which means its tests can only use a fake, and a fake proves the
logic while proving nothing about where the bytes land.
A bare ref resolves to path "/", which fileOrg treats as the FACADE and lands in
the deployment's system partition. The REST surface folds the caller's org and
lands the same name at /orgs/{org}. Those are different databases. Write through
one door and read through the other and the secret is simply not there: no error,
no warning, just a key the gateway cannot find. This asserts the admin seal and
the completion-path resolve use one door.
Second test pins the migration's direction: env serves until a key moves into KMS,
and KMS wins after. Without it "migrate the key" could be a no-op that looks
complete.
Verified on Linux, where libsqlcipher is linked and the store actually opens.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The commit claimed a sibling would stop reaching `ai` through Cloudflare. It
does not. In the `ai` process cfg.Enabled("ai") is true, so deps.AI is the HTTP
gateway pointed at https://api.hanzo.ai/v1 — the pod's own public address — and
the plane handler I added served inference THROUGH it. The path became
sibling --UDS--> ai process --HTTPS--> api.hanzo.ai --> airouters
where before it was one HTTPS call. Strictly worse, and the opposite of what
the message said.
The premise was right and the wiring was wrong: `ai` exposes its inference as
HTTP ROUTES (airouters), with no in-process ChatCompletion to call, so serving
the plane op from deps.AI just re-entered the transport. Closing this for real
means an in-process entry point in hanzoai/ai that the plane op can call
without a socket — a change in that module, not a rewiring in this one.
Reverted whole rather than patched so main does not carry a half-measure that
reads, from the commit log, as if the round trip were gone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`ai` is a plugin of this same binary running as its own process, and the only
way a sibling could get a completion was its PUBLIC address. So the pod left
through Cloudflare, performed a client_credentials exchange to authenticate to
its own deployment, and came back — to reach code one socket away. Its own
startup line said so:
deps.AI (completions) -> HTTP gateway (IAM M2M)
base_url https://api.hanzo.ai/v1 <- the pod's OWN public address
token_url http://iam.hanzo.svc/... <- a token to call itself
Every part of that was a consequence of addressing a peer by URL. plane.AIChat
and plane.AIEmbed are addressed by APP NAME — zip.SocketPath resolves it and
reach() starts the app if it is not listening — which is how the meter already
debits commerce and how the gate already reads the ledger. There is no address
to configure, no credential to mint, and no second answer to where `ai` is.
Which client a process gets is decided by WHAT IT IS, not by an env:
!cfg.Enabled("ai") means this process does not carry the app, which is exactly
when `ai` is a sibling. The process that IS `ai` (and the host, which carries
everything) falls through to the real transport — asking the plane there would
be this process calling itself, and the test pins that.
CLOUD_AI_ZAP_ADDR is deleted with its field: it was the previous attempt at
this, and it still made a deployment state where its own code lives. The
gateway stays for inference that is genuinely REMOTE — a different fact, not a
second way to reach the same thing.
Billing is deliberately not on the servant side: the caller reserves before it
asks and settles on the reported usage, so pricing it again there would bill
one completion twice. MaxTokens rides the request so the ceiling the caller
reserved against is the ceiling the servant honors.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The upload reached S3 and the profile did not move: 502 "photo stored but the
profile could not be updated", and the log said why — `iam non-envelope
response (400)` for id hanzo/2d4d67ab-….
IAM parses a user id as `<owner>/<name>` through GetOwnerAndNameFromId, and on
the direct-Bearer path X-User-Id is a UUID subject, so `<owner>/<uuid>` names no
user. keyID() is the composite that resolves — the key ops (mint/revoke) already
use it for exactly this reason, and the comment explaining it sits in the file I
edited. On the gateway path username == name, so the two are the same value and
nothing changes there.
Found by uploading a real PNG to production rather than by reading the route
back. The honest failure is what made it findable in one step: the handler
reported that the bytes had landed and the record had not, instead of a 500 or a
success the user could not see.
The branch carried a second one: a root `cloud.Qualify` returning `type Tenant
string`. Two spellings of the key the dataset plane writes, the risk plane reads
and this plane joins on — and the two did not agree.
It did not canonicalise the brand. A deployment started with CLOUD_BRAND=Hanzo
filed `Hanzo/acme` while apps/dataset asked for `hanzo/acme`. Silent,
permanent, no error: one business, two key spaces.
It did not ask the registry. A brand nothing vouches for minted a key the
writers of these surfaces refuse to produce.
`Tenant` is a STRING type, so it can be written as a literal in any package and
decoded straight out of a request body. `tenant.Key` is a struct with one
unexported field: no literal, no JSON, no caller-asserted tenancy. Only
tenant.Mint and tenant.Of produce one.
tenant.Of additionally compares the token's verified issuer brand against the
deployment's — a token minted by lux.id cannot become `hanzo/acme`.
So the root file goes and apps/label takes tenant.Key/tenant.Of throughout. The
door shrinks: tenantOf no longer re-checks the shape it just minted, because the
mint is the check.
Mutation-tested at the ONE place the property now lives: drop the case fold in
tenant.canon and TestTheMintAgreesWithTheWriterOfTheSourceTable reports
`Mint("Hanzo","acme") = "Hanzo/acme"` against a writer that writes `hanzo/acme`,
RED; restored, green. 67 label tests pass on the ported key.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Eight of the ten declared /v1/risk paths answered a 400 that named a field the
caller cannot send. Measured on api.hanzo.ai with each operation's own declared
body: score, learn, features, state, state/appetite, state/snapshot,
state/restore and search/{id} all returned
{"error":{"code":"","message":"field \"subject\" is required"}}
`subject` is plane.AuthorizeIn's field, not any risk operation's. It appears in
no published request schema on this surface, so no caller could ever satisfy it
— a door that answers and cannot be opened.
ops.gate re-derived the caller's ledger with principal.Ledger, which answers ""
for exactly the requests the tenant gate refuses (it composes the same Validated
check), and then handed that empty ledger to the money plane. Asked to price a
spend for a nameless subject, the money plane answered in the vocabulary of
money about a question of identity, in two shapes:
co-resident ledger — metering refuses an empty org fail-closed, which is not a
4xx, so the money wire's fallback renders 503 "Billing temporarily
unavailable": the caller is told the biller is broken.
peer ledger (what deploys) — the gate ships AuthorizeIn{Subject:""} over the
internal plane, commerce's own `validate:"required"` rejects it, and because
that refusal IS a 4xx the money wire preserves it verbatim as the 400 above.
ops.search never had the defect, for the one reason that it reaches ops.admit
before it prices anything. This makes that ordering general: an empty ledger is
an identity refusal, answered with the tenant gate's own sentence from the one
function that owns it, before the money plane is asked anything.
The route itself was never missing. GET /v1/risk/score is a 405 here — the route
is declared, the verb is not — and the report of a 404 traces to the deployed
edge flattening that to plain-text "not found", byte-identical to an unregistered
path. Named in the report; not reachable from this suite.
TestTypedOpsRefuseAnUnvalidatedPrincipal asserted this exact property and passed
throughout, because it mounts deps with no metering client: with no client the
money gate is a no-op and the op fell through to the honest 403 the test wanted.
The new tests mount through mountBilled, the only fixture in which the ordering
is observable. A companion assertion counting calls to the ledger fake was
written, found unfalsifiable — an empty org is refused inside AuthorizeVerdict
before any HTTP request, so the counter reads zero either way — and deleted
rather than shipped; the reason is recorded where it would have gone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`oversize` is on the wire, so it is in the document: the app's own subset, the
prose map and the woven openapi.yaml every SDK repo pulls. Generated by
`make -C apps/dataset describe` + `make -f mk/fleet.mk openapi-weave OUT=openapi.yaml`
— no hand edit. No new operation, so the floor is unmoved.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A COUNT OVER CALLER-SIZED VALUES IS NOT A BOUND. maxRows capped how many rows a
materialisation holds and `page` capped how many an export returns, and neither
bounded a single byte. A row's coordinates are float64s this plane counts, but its
SUBJECT is a string this plane does not write: the rollup lifts it from
`distinct_id`, `session_id` and `user_id`, which arrive on /v1/event from the
caller. distinct_id is capped at 256 bytes on the anonymous lane and REPLACED by
the token's own subject on the signed one — but session_id, which the `session`
rollup files as a subject verbatim, is capped nowhere. So "200k rows of ten
float64s plus a subject key is tens of megabytes" and "eight jobs is a few hundred
megabytes" were arithmetic over an unknown, and one tenant's traffic decided the
real number for the whole process.
maxSubjectBytes bounds the one caller-sized value a row carries, at 256 — what the
identified lane already states for a subject, for the reason that carries over
unchanged: a minted id is a uuid, the value is KEYED, and something longer is not
an id. With it, count times max IS the byte bound, and every byte figure the
package states is now DERIVED from it rather than written down beside it:
maxRowBytes, maxResidentBytes, maxProcessBytes, maxPageBytes. The stale prose
claims are gone rather than corrected — that was the other spelling.
ONE ENFORCEMENT POINT. `representable` is the predicate and the only place it is
written. It is applied on the way OUT of the source, which is the way IN to this
process and to the rows table, so no read path needs a second check: an export
page is bounded because every row it can return already came through it.
AND THE DEGRADATION IS NAMED. A bound that quietly drops rows is worse than no
bound — the dataset that comes back looks complete, and a model fitted on it is
blind to a population nobody can see was missing. The census measures both halves
on ONE pass (conditional aggregates, not a filter), the excluded subject count
rides on the version, the manifest, the lineage and the wire, and it is in the
source fingerprint, so `reproducible` is measured over it too: a window that grew
a subject too large to carry is a window that MOVED, and lineage now says so.
census and facts cannot disagree about the population, because the predicate is
one expression used by both — measured against, then read with. Two spellings
would sample the share from rows the read never returned.
Three gates, each mutation-tested (defect reintroduced, named test RED, reverted,
green):
TestEveryReadOfTheSourceIsBoundedInBytes the package's own AST — a
function that reads the source without the bound fails, the same shape as the
admission gate beside it, because it is the same failure: a new op skipping a
property nobody checks. Carries the same anti-vacuity floor.
TestAnUnrepresentableSubjectIsExcludedAndCounted end to end: the bound binds,
the representable rows all survive, the count is on the version AND the
lineage, and it is falsifiable.
TestTheByteBoundIsDerivedFromTheValueBound the arithmetic, so no byte
figure can be asserted independently again.
The fake store EVALUATES the bound rather than ignoring it — a fixture that
ignored it would let every test above pass with the predicate deleted. It also now
binds `?` positionally across the WHOLE statement, as a driver does, which is what
the census's conditional aggregates require.
TestTheTenantLeadsEveryPredicate was reading args[0] and calling it the tenant.
That was a positional coincidence, true only while no statement carried
placeholders before its WHERE; it now reads the argument bound to the leading
`org = ?` itself, which is the property it always meant.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
(cherry picked from commit b851446a2a)
CLOUD_ENABLE / --enable named the subsystems to mount. The binary already
knows: manifest.Apps is the host's set, and a plugin IS its app (Listen sets
cfg.Enable from what plugin/<app>/main.go was built as — that stays; it is
the binary stating itself, not a deployment restating it).
A second source of truth can only add disagreement, and it did. Both devnet
outages on 2026-08-02 were this list: one named "plans", an app the manifest
does not have, and one omitted "kms", the credential broker every other child
pulls its data-plane key from — so every child failed closed at its first
store open. Production has never set it.
Removing the input removes the failure class and five branches with it: the
broker precondition, the unknown-name guard, and three `on != nil` selections
that only existed to police a list nobody should have been writing. Empty
already meant all, which is what production runs.
Values move in the same change — devnet and testnet drop the list, so no
deployment is left naming a variable the binary no longer reads. Docs and
cloud-probe.sh follow; the probe had been STRIPPING the variable, so it
already agreed.
TestTheAppSetIsNotNamedTwice replaces TestAnAllowlistWithoutTheBrokerIsRefused
— that test policed the list, and the guard is now that no source states the
set again.
Also fixes a pre-existing red on main, unrelated to this: apps/catalog's
cloud.Request call site was never added to allowedRequestUses, so the escape
hatch ratchet failed on clean origin/main. It is a legitimate use (the
published corpus reads as PublicOrg for everyone, so the tenant is re-pointed
while the caller's authority travels whole) and is now recorded with that
reason.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An hourly goroutine in authors deposited credit with nobody in the loop, and a
GET on three surfaces accrued-and-paid on read. Both are gone, along with the
capability that made them one line each.
- apps/authors/scheduler.go, sweepAndPayout, autoPayoutAuthor — the unattended
hourly accrue+pay loop, default ON, no env var, no route, no human.
- the lazy sweep on GET /v1/authors, /v1/affiliates, /v1/affiliates/me and
/me/earnings. Reads read; the admin POST sweep still accrues.
- payout settlement in both programs. A payout RECORDS what is owed, for every
method including credits; a human settles it. Accrual — the product — stays.
- treasury.Reserve/Credit returned backed=true when unmounted, and `mounted` is
a package global, so in one-binary-per-app it was ALWAYS nil in callers: every
"reserve-backed" payout was an unbacked mint that logged itself as reserved.
With settlement gone it has no callers, so it is deleted rather than fixed.
- POST /v1/admin/credits — a second admin mint with no cap and no positivity
check, whose audit did not fail closed. core.ApplyGrant is the one door: it
caps, rejects non-positive amounts, checks the org, and refuses without a
durable audit store. The relay and its wire client are deleted.
- payout.Client.Deposit, the ONE money-in primitive all three programs shared,
and the deposit method on each program's seam. The seams now carry a single
read, matching referrals: reviving a mint has to start by re-declaring the
capability, in front of a test that says no.
- the published POST /finance/starter op, advertising a grant deleted in
41b23f12.
openapi.yaml, plugin/admin/openapi.json and openapi/floor.json drop the deleted
route in this commit, so the reduction is reviewed next to its reason.
Tests assert the guarantee rather than the old behaviour: a GET grants nothing
and the ledger receives zero deposits, proven at the wire against a commerce stub
that fails on any write.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/ws/query_progress is in openapi.yaml and in plugin/o11y/openapi.json, so it is a
method in every generated SDK — and no manifest row named it, so the composed
binary routed it nowhere. Its own prose says the address "was unreachable from the
composed binary until the route table named it, because the old wildcard covered
only the o11y prefix"; the wildcard went and the row never grew the prefix back.
TestEveryServedPathReachesTheAppThatServesIt has been RED on main for this one
path. It reads 1673-of-1684 now against 1672-of-1684 before, and the recorded
ledger is unchanged — nothing else changed hands.
Mutation-tested: prefix removed, the named test reports UNREACHABLE and fails;
restored, green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both avatar routes answered 404 in production. The routes were registered and
the handlers were fine; the HOST routes by manifest prefix, and account's row
did not name /v1/avatar — so `ai`, which owns the /v1 remainder, won it.
manifest's own router test said so in as many words on the tree that shipped:
UNREACHABLE: account /v1/avatar -> ai
UNREACHABLE: account /v1/avatar/{org}/{user}/{digest} -> ai
I did not run it. Targeted package tests passed and I shipped on those, which
is how a route can be complete, tested, deployed and unreachable at once.
With the prefix named, both paths mount to account (zip mounting … addr=account)
and the ledger drops from three unreachable paths to one — o11y
/ws/query_progress, which is not this change's and stays as it was.
iamurl.go opens by saying the IAM-address policy "existed as three inlined
copies ... so there is exactly one now". There were four, and they had grown
three different fallbacks:
iamurl.go IAM_URL, else the public issuer (the policy)
auth_apikey.go IAM_URL, else IAM_INTERNAL_URL, else NOTHING
apps/account/iam.go IAM_URL, else a hardcoded cluster address
apps/platform re-read IAM_URL to answer a different question
A copy of a policy does not disagree until one is edited. These had already
diverged: auth_apikey read a fifth env name no cloud deployment sets
(IAM_INTERNAL_URL is on admin-guard and chat only), so a single-process deploy
that knew only its issuer resolved "" and API-key auth stayed silently
unconfigured; apps/account reached a cluster address that a non-cluster deploy
does not have.
iamurl.go now answers both questions the estate actually asks, over ONE env
read: IAMBase() for the address, IAMExternal() for whether a separate IAM is
named. They are separate because conflating them IS the bug — a deployment
with only a public issuer resolves a real address while naming no external
IAM, and a caller inferring one from the other reaches for a store that is
not there. TestIAMAddressHasOneReader walks the tree and fails on a second
reader, so this cannot drift back.
Also: pickCompletionsClient preferred the PUBLIC gateway over the in-cluster
peer. `ai` is a plugin of this same binary running as its own process, so that
ordering sent the pod out through Cloudflare and back, minting an OAuth token
to authenticate to its own deployment, to reach code one socket away. The peer
is now first. Ordered, not gated: nothing sets CLOUD_AI_ZAP_ADDR today, so
this is inert until an address is named and the gateway keeps answering.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
main was pinned to hanzoai/ai v1.832.18, which is RETRACTED. That version seals
the secret MASK as a provider key: the admin API returns "***" for a stored
secret, so saving a provider form without touching the key field seals the literal
"***" into KMS under the provider's own name — and because ai resolves KMS-first,
the store then answers "***" for every read, outranking the env var that was
serving the real key. The provider stops authenticating while its row still looks
correct. v1.832.19 carries the guard.
.19 also brings: secrets resolved from the EMBEDDED in-process KMS (this binary's
own apps/kms) instead of over HTTP to the standalone deployment, which had never
worked — 404 on the path it used, 401 on the correct one; /v1/provider-flags
renamed to /v1/models/providers and derived from the served catalog, so the
provider set and the model list cannot disagree; and a model family is now
controlled from admin.hanzo.ai rather than only from deployment env.
apps/referrals did not compile on main: payout.Deposit gained a required `ref`
(commerce guards on it so a retried payout credits AT MOST ONCE) and this caller
was not updated with it. A referral pays TWO wallets, so the referral id alone
would name both and commerce would dedupe the second against the first — the
referee's bonus would silently never land. bonusRef(id, side) makes each credit
its own event while staying stable across retries.
The published spec is regenerated: /v1/provider-flags is gone from openapi.yaml
and plugin/ai/openapi.json (the .18 pin never regenerated it, so the drift gate
was red), and the retired provider-flags product is dropped from the floor.
Verified on Linux (macOS has no tmpfs for the pure-Go SQLCipher codec, so the
store-backed suites cannot run there): apps/referrals, apps/ai, apps/kms and
openapi all green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The plane fix shipped and /v1/catalog answered 500 "index: no org on the call"
— the public browse reaching the index as nobody.
cloud.For() states a caller, but zip's forwardIdentity prefers an INBOUND
request over a stated one ("an inbound request always wins over what it said"),
and a typed handler's ctx carries the in-flight request. So For() was silently
overridden and the call went out as whoever asked. For a signed-out visitor
that is nobody, and the index refuses a call with no org — correctly.
cloud.As() is the form for this: it re-points the tenant on a context with no
request behind it, which is the one place zip reads what we stated. The caller's
authority still travels whole; only the tenant moves — which is the point, since
the published corpus is read as PublicOrg by everyone, signed in or not.
Caught by asking production rather than by trusting the deploy: the previous
commit was verified live and answered 500, not rows.
payout.Deposit sent no idempotency key at all, so every retry of an author
royalty or affiliate commission credited the wallet again. Commerce now
requires the key (it is the reference to the money event), and both callers
already hold the right value: the payout row's own id.
- Deposit takes ref and sends it as X-Idempotency-Key, via a post helper
kept beside do because only a WRITE carries a reference — a read has no
event to name.
- ErrNoRef refuses an unnamed deposit HERE rather than at commerce, so the
failure names the caller's missing value instead of arriving as a status,
and a new payout path cannot quietly ship without one.
- authors and affiliates pass "payout:"+payoutID through their commerce
seams.
Tests: an unnamed payout never reaches commerce (the fake server records
that it was not called), and the ref travels as the key commerce guards on.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/referrals ran a "lazy qualify sweep" before listing, and that sweep
reached commerce.Deposit. Loading your own referrals page minted platform
credit — $10 to you, $5 to your referee — on a read. The middleware in front of
the prefix gated writes by asking `method != POST`, so it waved the GET through;
the two safeties behind it did not hold either. treasury.Reserve returns
backed=true when treasury is unmounted, so the "backed by the reserve fund"
claim was a passthrough in any deploy without it, and the at-most-once latch
bounds the mint per referral, not in total.
The precedent is set twice on main: 41b23f12 deleted the $5 starter grant and
45b3b5cf deleted finance.Deposit along with its imports. Same here. The deposit
path is DELETED, not disabled — a flag-disabled money mint is one flag from an
enabled one.
What goes: the two bonus constants, the ledger currency + grant:referral tag,
grant(), the treasury reservation and its import, LatchCredit, SetTxns, the
grant/txn/credited_at columns, the `credited` status, and every cents field on
the wire. A field that can only ever report zero is a lie about what the surface
does, so creditsEarnedCents and the two bonus amounts go rather than freeze at 0.
The commerce seam keeps ONE method, spendCents — it is a question, not an
instruction — and TestCommerceSeamIsReadOnly fails if it grows a write.
What stays, because it is the actual product: who referred whom, the stable
code, the share link, and qualification. Qualification is a WRITE, so it now
happens only on POST /v1/admin/referrals/sweep. A GET reports; it does not
transition.
The gate is fixed at the defect class, not the instance: it waves through GET
and HEAD by name and requires an org for every other verb, including ones this
package does not serve. "Not POST" meaning "harmless" is the reasoning that let
a read reach a deposit.
What a qualified referral is WORTH is not this package's question. That is an
affiliate payable in hanzoai/commerce, settled by wire or to a connected wallet
— never minted as platform credit.
Tests: a GET in the exact state that used to pay leaves the row byte-identical
and touches the money plane zero times; the real payout client against a stub
commerce that fails on any write proves zero deposits at the wire.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The reservation landed with a 32768 constant standing in for "the most a
completion can be". That number is wrong per model by construction: it caps a
1M-context model at whatever was typed, and it is one release out of date the
moment a model ships. This estate has already paid for that shape twice —
ai/model's name-matching table gave deepseek-v4-pro 16384 and 402'd every long
prompt, and glm-5.2 dead-ended /compact on a stale 16K fallback. Both were
fixed by moving the number into models.yaml. This does the same for billing.
- completionCeiling(model) resolves through SetCompletionCeiling, installed
in apps/ai from ModelConfig.MaxOutput (ai v1.832.18), falling back to the
model's context window — still a true bound, since prompt + completion can
never exceed it. The constant survives only as a FLOOR for a model the
catalog does not declare, and is documented as never a per-model answer.
- the seam exists because hanzoai/ai/controllers imports hanzoai/cloud, so
the catalog is a CYCLE from cloud's root, not merely weight. apps/ai links
both, which is where every other cross-module hook is installed.
- atMost no longer writes the ceiling onto the request. What we reserve is a
billing fact; req.MaxTokens is the CALLER's, and forwarding a limit they
never asked for silently truncates their answer. The reservation is sound
regardless — a model cannot exceed its own max output — so the bound holds
whether or not we restate it on the wire. Only a caller-set MaxTokens now
reaches the provider.
TestCeilingComesFromTheModelNotAConstant pins all four: a 1M model reserves
>=1M, an undeclared model takes the floor, a caller's MaxTokens wins, and
atMost never mutates it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzo.app's Community page rendered "ERROR: CATALOG: 503" under an otherwise
fully-drawn page. api.hanzo.ai/v1/catalog answered
{"status":503,"error":"catalog: index not mounted"} on EVERY request, and
nothing was down.
browse() guarded on index.Ready(), and Ready() reports whether the index is
mounted IN THIS BINARY — its own doc says so. That was true and harmless while
cloud was one fused process. It stopped being harmless when the fleet became
one process per app: `index` and `catalog` are two manifest rows, index.Mount
is only ever called by plugin/index, so inside the catalog process that global
is nil and always will be. An in-process dependency survived a process split,
and the only symptom was a status code.
The index is now ASKED, on the internal plane, exactly as visor asks tasks for
its activities and as commerce serves its ledger. Not opened: the index store is
one encrypted SQLite with a single writer (MaxOpenConns(1), keyed through cek),
so a second process opening the same file to read it is the collision, not the
cure.
Both legs are kept and neither is dead: a fused binary that mounted both apps
still reads in-process, because a wire hop to something in the same address
space is pointless; the deployed fleet takes the plane. The write side does not
move — Reconcile stays in the process that owns the file.
apps/search carries the same dependency and is mounted by no plugin at all, so
it is unreachable rather than broken. Left alone here; naming it so the next
reader does not mistake this fix for having covered it.
Moving imageType into internal/magic took the only use of "bytes" in files.go
with it and left the import behind — which does not compile, and it is what
turned the release train red twice (the image job exited 1 at ~6m18s and no
cloud image published, while v1.801.399 had built fine from the commit before).
It was invisible locally for a reason worth writing down: `go build ./apps/team`
on macOS fails first on hanzoai/base's `cgo && !sqlite_math_functions` guard, so
the package was never type-checked and my error hid behind someone else's. The
build CI and the Dockerfile use is CGO_ENABLED=0 — under that, the package
compiles and the mistake is immediate.
Verified the way CI does: CGO_ENABLED=0 go build ./... and go vet ./... both
clean. The apps/team test failures that remain on this machine are all one
environmental cause (75 of 75: "cek: no RAM-backed scratch", macOS has no
/dev/shm) and fail at store setup before any assertion.
The drift gate regenerates every app's spec subset FROM SOURCE and fails on any
diff, so adding /v1/avatar without re-running it turned CI red — the release
train's image job exited 1 after 6m29s and no cloud image was published.
Regenerated, not hand-edited: zipdoc lifts the doc comments into zipdoc_gen.go,
`describe` projects account's own subset, and the weave proves the fleet spec
equals the sum of the subsets. floor.json moves by exactly what was added —
+2 paths, +2 operations, one new `avatar` product with 2.
zip dials every plugin with zaphttp's DEFAULT transport, whose readTimeout is
30 seconds, and nothing overrode it. That deadline is armed ONCE, just before
the response head is read, and for a STREAMED response it is never re-armed:
the head arrives, SetBodyStream returns, and the whole body must then land
within what is left of the original 30 seconds.
So it is not an idle timeout. It is a hard cap on the TOTAL DURATION of a
response — applied to the plane whose responses are model completions, because
`ai` owns the /v1 remainder (/v1/chat/completions and the rest of the
OpenAI-compatible surface) and runs as a plugin behind this wire.
Every completion longer than 30 seconds was cut:
- streamed: the body stops mid-token at ~30.1s with NO finish_reason, which
every layer downstream reads as a complete answer. Measured against the
live gateway, a long-page request to claude-opus-4.8 returned 7056 bytes,
no </html>, total 30.1s — while the identical request straight to the
upstream streamed 389s and finished properly at 15661 bytes. Reproduced
in-cluster too, so it was never the edge.
- unstreamed: 502 `zaphttp: read response: read unix …: i/o timeout`.
This is why hanzo.app's builder could produce a small landing page but not a
real app: an app is just a longer generation, and the truncation was silent. It
also made a slow-to-first-token model look broken rather than slow — enso
spends ~19s thinking, so most of its budget was gone before it emitted a token.
The ZAP scheme is re-registered before any plugin mounts (the transport is
resolved at dial time from a process-global registry, so a later registration
would leave already-dialed plugins on the default). BOTH halves are supplied: a
Dial-only Transport would silently remove the host's ability to LISTEN on its
own default scheme.
15 minutes, not zero: zero is no deadline at all and a wedged plugin would hold
the connection forever. The honest shape is an idle timeout — time BETWEEN
frames — which this transport does not offer, so it stays a total-duration cap
set past anything a real completion reaches.
There was no way to set one. IAM carries an `avatar` on every user row and the
console renders it, but the only writers were federation (a GitHub avatar_url,
an OIDC `picture` claim) and SCIM — so a user who signed up with a password had
a monogram and no way to replace it, and the console's Profile card answered the
attempt with "Edit in IAM", which links to an IAM that cannot do it either.
Production agreed: /v1/avatar was a 404 while /v1/keys was a 403.
POST /v1/avatar stores the image and records its URL on the caller's IAM row, so
every surface that already reads `avatar` picks it up with no further call.
GET /v1/avatar/:org/:user/:digest serves it.
Storage is deps.VFS — the existing S3 seam (SeaweedFS via clients/s3vfs), which
was chosen for exactly this: "an adapter+crypto is needless complexity for small
avatars". No new store.
Three properties make it safe to serve an uploaded file back from an API origin:
- THE FORMAT IS DECIDED BY THE BYTES. png/jpeg/gif/webp by magic number;
anything else is 415 on the way in and 404 on the way out. A filename and a
part Content-Type are the client's to choose, so neither may decide what
this origin later serves — an SVG is a program, not a picture.
- THE ADDRESS IS THE CONTENT. The key ends in the sha256 of the bytes, so a
new photo is a new URL rather than a stale cache of the old face, and the
read caches for a year. A replaced photo is deliberately NOT deleted: the
old URL is already inside issued tokens and rendered pages, and an object
store costs bytes where a broken face costs a person their profile.
- THE READ TAKES NO CREDENTIALS, AND MUST NOT. Its whole job is to be an <img>
from console.hanzo.ai, a different origin that sends no cookies and cannot
set a header. So the 64 hex of sha256 IS the capability — producible only by
someone who already has the image. The org and user segments are REFUSED
unless they are plain identifiers rather than sanitized: apps/team's seg()
folds "a/b" and "a_b" onto one key, and a fold in a tenancy key is two
identities sharing an address.
internal/magic is the one magic-byte allow-list, now shared with apps/team's
files plane instead of copied. (The three `seg` functions are NOT duplicates —
same name, three different concepts — so they stay where they are.)
The oversize test mounts the app at production's 16 MiB edge body limit. Left at
zip's 4 MiB default the framework refuses the request first and the handler's
own 413 is unreachable and untested — the shape of the bug where studio's 4K
sources could not enqueue.
POST /v1/marketing/promos/:code/redeem was a self-service money mint. The only
gate was tenant(ctx) -- ANY validated principal. `plan` and `seats` came off the
REQUEST BODY unvalidated and were multiplied into a finance.Deposit, so
{"plan":"team","seats":10} deposited 10 x $179.10 = $1,791.00 of real spendable
credit into the caller's own org. Nothing ever collected the charge the discount
was supposedly against; the charge was computed, returned to the caller, and
discarded. `instrument` was the anti-farming key, but instrumentUsed("")
returned false, so OMITTING the field skipped the guard entirely. With the
1,000-org cap, open signup and a personal org per account, that is ~$1.79M of
self-serve credit. The seed shipped active=1 in v1.801.398, which is live.
THE CAMPAIGN WAS NEVER AUTHORIZED. It became live because a schema migration
INSERTed it on every boot -- a business decision arriving as a side effect of a
code change. The seed is deleted, and because deleting an INSERT does nothing
for a database that already ran it, migratePromos now DELETEs the row on every
boot. Redemption history is deliberately KEPT: it is the evidence of what
happened while the campaign was live, and destroying it would destroy the audit
trail exactly when it matters.
CREDIT INTO AN ORG IS AN ADMIN DECISION -- deliberate, through the admin
surface, against an auditable ledger. So the deposit is not fixed, it is GONE,
along with the finance/money/types imports that made it reachable: reviving a
mint here would have to start by reviving an import. This follows 41b23f12,
which deleted the automatic $5 starter grant rather than switching it off, for
the same reason -- a money-mint left disabled is one flag away from enabled.
The subsystem now ships OFF (campaignsLive=false, read from no env var, no
platform switch, no column; TestCampaignsShipOff asserts the shipped value).
The guards are hardened anyway, so a REVIVED campaign cannot resurrect the hole:
- Plan is DERIVED from the org's live ACTIVE/TRIALING paid subscription via
cloud.PlanChecker -- the same seam SpendGate resolves -- and RedeemInput no
longer HAS plan/seats fields. A field that does not exist cannot be trusted
by the next reader. No qualifying subscription means no redemption.
- FAIL CLOSED on an unreadable plan authority. SpendGate deliberately fails
OPEN on this same read, because refusing on an outage 402s every paying
customer at once. Here the asymmetry inverts: an outage must not be able to
manufacture a claim that money is later granted against.
- instrumentUsed("") now returns TRUE. An absent instrument is not evidence of
a fresh card, it is the absence of evidence, and the partial unique index
(WHERE instrument <> '') means the database will not catch it either.
- maxClaimCents bounds every recorded claim, checked under the same lock as
every other guard, and REFUSES rather than clamps -- a silent clamp would
record a wrong figure and hide the bug that produced it.
- Seats is the single-seat floor, never a caller-supplied multiplier.
Redemption.CreditCents/CreditEntryID become DiscountCents: the row records a
claim, not a balance, and a field named for credit that credits nothing is how
the next bug gets written.
Tests drive the shipped routes through the real router, because the hole was a
handler that trusted its input -- a store-level test would have proved the store
fine and missed it. The hardening tests force a campaign live (revivedPromoRoutes)
to answer the question that matters if the decision is ever reversed. Proven: the
exploit body cannot move the recorded figure; a live ledger receives ZERO
deposits; a redemption while closed is refused; nothing is seeded; migrate purges
an already-seeded row while preserving its redemptions.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A prepaid gate reads a SETTLED balance and a completion's cost is not known
until it finishes. Two ways an org spent money it did not have sat between
those facts:
- the gate priced only EstTokens(prompt), so the completion was never
covered. A 1c org asking for a million-token completion was allowed and
settled at $2.00, ending at -$1.99.
- N calls in flight each read the balance before any of them debited, so
each was authorized for the whole of it. Twenty simultaneous callers
against 5c spent 20x the balance.
meteredAI now commits a call's worst case before weighing it and releases
that commitment when the debit reaches the ledger:
- types.ChatRequest gains MaxTokens, and atMost() resolves the ceiling ONTO
the request so the transport forwards the very number the gate priced.
Reserving a ceiling nobody enforces leaves the completion just as unfunded,
only less visibly, so clients/aihttp sends it on both the buffered and the
streamed path.
- commitments tracks per-org committed-but-unsettled cents. commit() returns
the RUNNING TOTAL, which is what the balance must cover, so a second
concurrent caller must clear the first one's commitment. Nothing else has
to know reservations exist.
- the release runs inside the recording goroutine (meterUsage's new posted
hook, threaded through the peer path too). Releasing when the call returns
would let the next gate read a balance that still contains money already
being spent — the very window this closes. It runs on every exit: a hold
that leaks is a paying customer locked out of their own balance.
Holds stay per-pod. apps/finance owns the settled truth and says so
("transient holds are the caller's in-pod concern, never persisted here"),
so this never becomes a second ledger.
MeterUsage keeps its signature — one debit verb, twelve callers untouched.
Tests drive the real meteredAI against a wallet whose balance MOVES; the
existing fixtures answer every gate from a fixed body, which is why neither
gap showed up before. A barrier holds the concurrent callers at the balance
read so the TOCTOU is deterministic rather than a race won by luck.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The starter grant minted $5 into a wallet from middleware, on first credential
contact, with no human in the loop. Credit into an org is an ADMIN decision --
made deliberately, through the admin surface, against an auditable ledger --
so an automatic path that creates money is not a feature to fix but a mechanism
to remove.
DELETED RATHER THAN SWITCHED OFF. A disabled money-mint is one flag away from
an enabled one, and the flag is the kind of thing a later reader flips to
"unblock" something. There is no starter code left to re-enable: the middleware,
its mount in serve.go, the cross-process plane op (finance_starter / StarterIn /
Granted) that let a non-ledger binary ask for it, and their tests are gone.
Note this also removes the shared-signup-org exclusion that lived in the gate.
It was sound anti-abuse for a grant that no longer exists, and keeping half a
mechanism to guard the other half is how dead code survives.
The paywall consequence is deliberate and is NOT taken here: SpendGate stays
behind its kill switch. With no automatic funding, enforcing it 402s every new
account from its first request -- an honest paywall, and a product decision that
deserves its own change rather than arriving as a side effect of this one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
sha-846069c predates the fix. The console SPA is baked into this binary by
//go:embed, so nothing about a console release reaches production until this
line moves — which is exactly what the pin is for, and why it is a sha and not
`:latest`.
sha-9da3984 carries three commits: the token store moved off sessionStorage
(a second tab started signed OUT while the first was still signed in), the
landing CTA starts the sign-in instead of routing to a page that asks again,
and the @hanzo/iam bump to 0.21.6 without which the static export dies on
`ReferenceError: sessionStorage is not defined` while prerendering
/auth/callback.
Verified before pinning: the console CI run for 9da3984 is green and pushed
console-embed:sha-9da3984-amd64. The previous run, on b9d31aa, was RED — so the
image for the storage fix alone never existed, and pinning to it would have
failed the CONSOLE-GATE here rather than shipping anything.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two corrections to what shipped in v1.801.397.
TIER. It was registered on the bare public chain plans uses, but GetTier opens
with middleware.GetOrganization, so it panicked on a nil interface conversion and
answered 500 — the route went from unreachable to reachable-and-broken. A tier is
org state; the org has to be resolved first. Moved onto the billingRead loop,
which supplies the IAM leg its six siblings already rely on.
ROUNDING. The balance now reads (502 -> 200, $149,913.08), but it rounds UP, not
down as the comment claimed. hanzoai/decimal's Rescale rounds half-away-from-zero
(decimal.go:145) and Minor() is a Rescale — measured live, …078983985999994361
served 14991308 cents, a tenth of a cent above the true balance. The comment is
corrected rather than the behaviour: this number is a display, nothing is billed
from it, and the spend gate reads the exact decimal itself. A debit that must not
overstate has to round down deliberately instead of reusing this.
apps/ai carries the same "truncated toward zero" claim on the same call and is
wrong the same way. Noted in place; its gate compares > 0, so a sub-cent rounding
cannot change its verdict.
GET /v1/billing/balance answered 502 "billing upstream unreachable" on every
call and the console rendered "Unavailable" while the ledger held the money.
Nothing upstream was involved — that label is wrong and it is why this looked
like a connectivity problem for days. plane.Money.Minor() REFUSES a value finer
than a cent rather than round behind the caller, and the ledger keeps eighteen
decimals because per-token charges are routinely finer than a cent. So a REAL
balance broke the read: live the org held $149,913.078983985999994361 and the
error was "is finer than its minor unit; round explicitly". It got worse as
usage accumulated, since a longer history makes a sub-cent tail likelier.
Minor()'s own doc says a caller that wants a rounded figure — "a display, a
summary" — should round explicitly, where the choice is visible. This view is
exactly that and never did. It now rounds DOWN, the same choice apps/ai
documents for the same value: a displayed balance must never exceed what the
account can actually spend, and truncation understates by under a cent. Nothing
is billed from this number.
Also reverts a wrong fix from earlier in this session: registering balance
co-resident in apps/commerce. The manifest gives /v1/billing/balance to the
`billing` app, not commerce, so that registration was in a subtree it does not
own and could never have run.
Two things this exposed, both left alone deliberately:
- The 502's message names an upstream that is not in the path. Renaming it is
a behaviour change to an error contract callers may match on.
- apps/billing's test binary needs libsqlcipher and cannot compile on a
laptop, so `go build` passing there proves nothing about the test file. That
masked a first version of this patch which called a Minor() that does not
exist on the local money type; caught by type-checking with go vet instead.
GET /v1/billing/balance answered 502 "billing upstream unreachable" on every
call, and the console renders that as "Unavailable" while the ledger holds real
money.
The cause is the one apps/account/billing_coresident.go already documents in its
header: co-resident there is NO standalone commerce — the in-cluster service
selects the cloud pods themselves — so the /v1/billing/* bridge forwards to a
default base that is the public edge, and the read re-enters the same bridge in
an unbounded self-dispatch loop. Six sibling reads (invoices, subscriptions,
alerts, payouts, settings, credits) were already registered co-resident to
shadow that wildcard. Balance was not one of them.
It now registers through the identical chain — RequestContext, IAMTokenRequired,
PinBillingSubject — so the subject pin is byte-for-byte what the bridge applied
and a read scopes to exactly the account the spend gate debits, never wider.
balance/all rides the same prefix, since a prefix owns its whole subtree.
The apps/commerce test binary does not compile on a laptop (hanzoai/base needs
libsqlcipher, which the build image links and macOS does not); verified by
stashing that the identical failure predates this change.
Every AI request logged
tier_cache: Commerce lookup failed for key=...: commerce returned 404
(defaulting to zen-free)
and that default is 60 rpm against 500 for pro, 2000 for team, 50000 for
enterprise. So the failure did not give anything away — it silently served every
PAYING customer the most restrictive tier in the table.
Cause is the exclusive-subtree rule again: account-bridge owns /v1/billing, and
commerce's row named seventeen sibling paths but not tier, so
GET /v1/billing/tier never reached the handler that answers it
(commerce api/billing/handlers.go:43 registers it; live it 404s).
The UNREACHABLE ledger in manifest/router_test.go did not catch this and is not
wrong to have missed it: it fires on paths the fleet PUBLISHES, and cloud does
not publish tier — commerce serves it and only the ai router calls it. A
cross-binary caller is outside what that gate can see from here.
The pinned console-embed was sha-147ecd3, built 2026-07-27. console main is 63
commits ahead of it, so a week of console work — including the onboarding fix
below — has never reached production: the pin does not track main BY DESIGN (a
floating tag made a release silently bake the PREVIOUS console), so it only moves
when someone moves it, and nobody had.
sha-846069c is console main HEAD. Verified published before pinning:
ghcr.io/hanzoai/console-embed:sha-846069c-amd64 resolves to
sha256:f2849a31b082da0d690bc5403a41ad995624fdfd6595089008821c07718a9255, and
console-embed:latest now points at the same digest. Probed with all four Accept
types against a bogus-tag control — a two-header probe returns a FALSE 404 on an
image that is present, which is how a healthy registry can be misread as an
outage.
What this carries to production, beyond 62 other commits: a refused org create no
longer reads as a complaint about the NAME. /v1/iam/onboard answers 409 for two
opposite reasons — the first-run gate (this account already admins an org;
founding a second would orphan it) and a name genuinely held by another tenant —
and the console showed the server's organization-level message immediately after
the customer typed a name. It now reads the account to tell the two apart, and
offers the way into the org the identity is actually in instead of dead-ending on
a form that can never submit.
cloud pinned ai v1.832.16 while five commits sat on ai's main untagged, so a fix
merged hours ago was in no release and no binary. That is the third instance
today of merged-and-unshipped, each with a different cause: an image published
before its own fix landed, a build job silently skipped by a stale generated-doc
gate, and now a module change in no tag at all. The symptom is identical from
outside — the code is right and production is wrong — which is what makes it
expensive to notice.
Carries: the ok-with-no-user key refusal (IAM answering status=ok with a null
user used to fall through to a bare "invalid API key" that named no cause), the
openrouter seed URL carrying a /v1 the family appends itself, a family's kms://
key resolved before it becomes an Authorization header, family control from
admin.hanzo.ai rather than env alone, and /v1/provider-flags becoming
/v1/models/providers derived from the served catalog.
No release cut here. The next cloud build carries it: cloud is one replica with
strategy Recreate, so every release is a total outage of inference, billing and
auth, and a dependency bump does not justify one on its own.
The site edge has never run in production. It was mounted in serve.go — the
FUSED composition root — but the image runs cmd/cloud (the light router) on the
public port with one process per app beside it, and none of those is serve.go.
So every <slug>.hanzo.app fell through to the console SPA, and the whole /v1
surface answered on the customer's own hostname.
Proven at the pod rather than inferred: the process holding :8000 is /cloud (a
separate 23MB binary from /plugins), and `grep -c sites_resolve /cloud` is 0 —
the code was not in the running binary at all, even at the tag that contains it.
The two earlier fixes this session were both real and both invisible for this
reason: the host resolution fix (8b729f8ef) and the plane resolver (5f0b74fe9)
were compiled into a composition root nothing runs.
The router "deliberately links none of" the fleet's package graph, and that
holds: `go list -deps ./cmd/cloud` still contains ZERO of the root package.
apps/sites is a leaf, and the cross-app call is made here with zip.DialApp —
the same door wake.go already uses to publish one op without cloud.Plane().
apps/sites exports the wire types so the caller restates no mapping.
Mounted BEFORE webui, which owns "/" for every unclaimed path and would
otherwise answer first for every site host.
The test reads run()'s own source for the call site, because the defect was
never a logic error — the package was always correct — it was a middleware that
ran nowhere. My first version called mountSites directly and PASSED with the
call site deleted, which is exactly the mistake this file is about. Both failure
modes are now negative-controlled: no call at all, and mounted after the console.
Every published site served the console SPA. The host fix in 8b729f8ef was
necessary and not sufficient: with the host resolving correctly the edge still
found no site, because sites.SetResolver writes a PACKAGE-LEVEL registry inside
`projects` and the edge middleware reads it inside whichever process fronts
:8000. The pod boots ~25 single-app processes ("enabled":["<one>"], 25 distinct,
zero multi-app — measured on the live pod), so those are never the same process
and the registry is always nil where it is consulted.
A nil registry is a clean MISS, not a fault. So every lookup failed silently,
every request fell through to the API pipeline, and no error was logged anywhere
because nothing had failed. Proven at the pod with the ingress bypassed:
wget --header="Host: app.maxpower.hanzo.app" http://127.0.0.1:8000/
-> <title>Hanzo Cloud Console
for a site that is genuinely published and has its own Ingress.
The fix is the seam this repo already uses for exactly this shape:
FinanceScopeRules is on the plane, in its own words, because "the READER is a
cloud EDGE middleware" and the fact belongs to another app. projects now
publishes sites_resolve / sites_resolve_org from the one process that owns the
store, and the edge falls back to them. Co-residence still wins with no hop —
currentResolver prefers the in-process registry and only then asks.
Not-found stays a clean 404; a failure to ASK stays an error, so the edge can
render 503. Collapsing those would serve 404s for live customer sites during any
transient failure of the owning app, which is indistinguishable from deletion.
The comment on SetResolver said "until it is set, every site request is an honest
404 (the projects subsystem is not mounted)". That premise was the bug: projects
IS mounted, just elsewhere, and 404 is not honest when the site exists.
Negative-controlled: removing the fallback fails the new test with the
fall-through this commit is named for. The second test pins that a co-resident
store is still used without the hop.
The relational engine wrapped an identifier in the dialect delimiter but wrote
the body verbatim, so a delimiter inside the identifier closed the quote the
writer opened and the rest executed — Desc("name`,(subquery)--") emitted two
quoted identifiers plus bare SQL rather than one identifier. xorm v1.4.5 doubles
the delimiter, the standard SQL identifier escape, so a name that smuggled one
becomes a single identifier that does not exist and fails closed.
orm is the only place that version is chosen — consumers name hanzoai/orm and
the engine arrives underneath — so this is the hop that makes it live here.
go list -m confirms xorm resolves to v1.4.5 in this build.
Bump only. The four failing packages are the standing pre-existing set
(apps/iam, apps/plan, apps/pricing, manifest); the bump adds none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
plugin/dataset/main.go called cloud.Serve, which does not exist. Every other
plugin calls cloud.Listen. The image build enumerates the manifest and compiles
each plugin in turn, so this failed the whole build at the dataset step:
plugin/dataset/main.go:29:18: undefined: cloud.Serve
That means no cloud image has been buildable since ae3a30994 landed. `go build
./...` at the repo root does not catch it — the plugin mains are only reached by
the Dockerfile's per-plugin loop, so the gap between "compiles locally" and
"produces an image" is exactly one word wide.
Every published site served the CONSOLE, and mounted the whole cloud API under
the customer's own hostname. Measured live 2026-08-03:
quest.hanzo.app/ -> <title>Hanzo Cloud Console
quest.hanzo.app/v1/billing/plans -> 200
fiber parses the request URI once, and behind the ingress the parsed host is
empty — so siteSlug("") failed, customCandidate("") failed, and every request
fell through to c.Continue() into the API pipeline. The site edge was mounted and
configured correctly the whole time; it just never learned which host was asked
for. Same accessor and same failure as commerce's tenant resolver, in a second
codebase, the same night.
The parsed host still WINS whenever it names something this server serves — that
ordering is the security property, not a detail, because the host picks the ORG
here and a client able to override a real host could serve itself another
tenant's site. X-Forwarded-Host is consulted only when the parsed host is not a
host we can serve, which is the ingress case and never a direct request.
TestMiddlewareTenantKeyedByHostNotPath (pre-existing, unchanged) still passes.
Negative-controlled: reverting to Hostname() alone fails the new test with the
fall-through this commit is named for. One correction on the way: the first
version of the new test used req.Host="" to express "no parsed host" — httptest
synthesizes "localhost" for that, so it proved nothing until measured.
studio could not enqueue a single render. `POST /v1/tasks/.../activities` answered
403 "identity required", so thirteen jobs sat `queued` in its worklog for up to
nineteen hours while both GPUs polled an empty namespace every two seconds and
reported themselves healthy. Nothing in the queue, the worker logs, or the studio
UI said why.
The cause is one branch. homeOrg already knows that a machine cannot mis-attribute
its org the way a human choosing an app can — that is why the KMS sync identity
reads `owner`. But KMS is recognised by audience, which works only because its
client id is DERIVED from its org ("<org>-platform-kms"). No other app's id is,
so every other client_credentials principal fell through to the estate rule, found
the empty `orgs` a machine correctly carries, and resolved nothing. SanitizeIdentity
then minted X-User-Id with no X-Org-Id, and each org gate refused it.
So the recognition comes from the token's SHAPE, which a human token cannot wear: in
a client_credentials token the client IS the subject — IAM sets sub to "<org>/<app>"
— and azp equals the sole audience, because the app asked for a token for itself. A
human's subject is the user and azp is whichever app they signed in through, which is
the exact mis-attribution homeOrg exists to prevent. Every field read is IAM-signed;
none is a header a caller sets.
This does not widen who may cross tenants. It resolves an org for a principal that
has exactly one and can no more choose it than an sk- key can: `owner` is the
application's own organization, and getting the token requires that application's
client secret. A human with no `orgs` still resolves nothing and still fails closed —
tested, along with each half of the shape, because a partial match is a human token
that merely resembles a machine.
An organisation that took its model out of shadow BEFORE the model had learned
anything was told live=true and had nothing written down. The regime lived on the
same row as the learned state, and that row's writer declines to write while the
snapshot holds no learned mass — correctly, because there is no state to lose. So
PUT /v1/risk/state/appetite answered 200, reported live, and persisted nothing;
this binary deploys Recreate at one replica, so the next rollout rebuilt from
defaultConfig — shadow — and the model decided nothing. No error, no log, nothing
to alert on. A model silently disarmed, on a routed door.
The two facts are decomplected. The regime is now its own append-only versioned
record on the tenant's own shelf, written BEFORE anything in memory moves, so a
policy that cannot be written down is refused rather than answered from state the
next rollout will undo.
A regime is a VALUE: a version is minted only when the numbers CHANGE, so a
version means "the Nth distinct policy this organisation adopted" rather than "the
Nth time somebody pressed save", and a client that restates its config on every
deploy is free rather than the cheapest way to fill a disk.
Every score now cites the version it was decided under. Cut is derived from the
appetite that version states, so without the citation a restated appetite made
every earlier decision unreconstructible — the threshold it was measured against
no longer existed anywhere. GET /v1/risk/policy reads the history back.
Bounds, both per tenant and both on the tenant's own table:
RATE at most 24 distinct regimes per rolling 24h, refused past it with the
organisation's own bound named and the regime in force untouched.
TOTAL 381 versions, which IS 256 KiB divided by a measured worst-case row.
At the ceiling the oldest is disposed of and the number disposed of is
REPORTED, derived from the lowest surviving version so it cannot drift.
A regime that predates this record is ADOPTED as version 1 on first residency.
Without that, resolving the regime only from the new record would have returned
every already-live organisation to shadow on the first rollout after this ships —
causing the very defect being fixed, to every tenant at once.
The bounds on review and sample had two spellings, one at the op and one in the
plane. The op's copy is deleted; admitRegime is the one door, at the same
strictness the published contract always had.
Two test gates widened, because both could have been passed by looking at
nothing:
- TestOps_EveryOpIsAdmittedAndPriced parsed typed.go alone, so an op declared in
any other file was admitted and priced by nobody's assertion. It now parses
the package, matches on the RECEIVER TYPE (the plane carries score/learn/
state/appetite too), and fails when a registered op is declared nowhere.
- the per-tenant history assertion over the wire cannot observe the query's
tenant predicate: two orgs are two FILES. The predicate is load-bearing where
two brands share one file, and TestPolicy_TwoBrandsShareAFileAndNotAHistory is
the test that fails when it is dropped.
Eleven mutations applied, each named test red under the defect and green after
revert.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
THE ADDRESS IS THE PRODUCT. openapi.Fold takes an operation's product tag from
the first /v1 segment of its path and nothing else (openapi.Product); a per-op
zip.WithTags names a different axis and cannot override it. The seven ops were
addressed /v1/ml/labels, so seven compliance operations — their own writers
(commerce adjudicates the dispute, the compliance face closes the case, an
analyst files the review), their own five-year retention floor, their own
per-tenant file — would have been published as part of the KServe model-SERVING
product, which is four paths and live with customers on it. Nothing in the fleet
would have said so: the floor ratchet reads `ml: 7 -> 14` as growth, because it
refuses a shrink and only a shrink. It is the same mistake apps/risk's own
manifest row already records having made and corrected once, one layer up.
So: /v1/risk/labels, tag risk, every operation id and schema name risk-prefixed
(riskLabelEvent and not riskEvent — apps/risk publishes a riskEvent already, and
it is a scored decision rather than a judged one). floor.json returns ml to 7 and
raises risk to 17. The manifest row precedes risk, whose prefix is the bare
/v1/risk, and TestEveryServedPathReachesTheAppThatServesIt proves all six paths
reach label over the real fleet router rather than a comment claiming they do.
address_test.go walks the live projection — the same openapi.FleetSpec that writes
the committed subset — so a route re-addressed into somebody else's product fails
at the plane.
A BOUND ON COUNT OVER CALLER-SIZED VALUES IS NOT A BOUND. maxResolve capped a
resolve at 500 named events and nothing capped a subject: the rows were bounded
and the bytes were bounded only by the edge's BodyLimit, which is a fact about the
deployment. Each subject is then amplified below the door — a dedupe key, a
grouping key, one bound parameter per event in a statement against a single-writer
file. The write door had the ceiling all along (admit, subjectMax); the read doors,
added after, did not, and nothing compared them.
There is now ONE spelling of each ceiling — admitSubject, admitKind, admitSource,
admitEvidence, and instantMax inside stamp(), which is the one parser every time
field passes — and every door asks it. So `count × ceiling` IS the byte bound of
everything this plane binds, holds and stores. An unknown kind or source is
refused on the READ path too: it can only ever match zero rows, so refusing says
so instead of charging for the scan. bound_test.go proves it twice: reflect walks
every In type and fails on a caller-sized field with no declared ceiling (the
structural half — a new field cannot arrive unbounded), and every declared ceiling
is refused over the wire with a refusal that does not carry the value back.
A LITIGATION HOLD THAT ARRIVES MID-SWEEP KEEPS THE RECORD IN BOTH PLANES OR IN
NEITHER. dispose sweeps the derived copy FIRST so nothing is orphaned in the
warehouse, then deletes from the record re-asserting `hold = 0`. That protected
the record and silently corrupted the copy: a record the delete declines to remove
has already been swept, its seq is behind the delivery cursor, and deliver() asks
the cursor rather than the world — so no retry re-sends it, pending() answers zero,
and the row is present in the compliance record and permanently absent from the
answer key a training join reads. A missing fraud label reads as an honest
customer, and the row is the one somebody is litigating. remove() now reports what
it kept, the sweep writes those back from the record, a repair that fails refuses
the request rather than acknowledging a short copy, and `restored` is a NAMED state
on the response. `disposed` counts what was disposed of rather than what was
identified: a compliance report that says it deleted a record it is still holding
is the wrong answer to the only question the report is asked.
THE PUBLISHED PRECEDENCE RULE NAMES THE FIELD THE RESOLVER READS. The op exists so
a caller holding a contested resolution can reproduce it, and its second term said
`seen` while stronger() compares `knowable`. The two are equal for a live pipeline
and differ for exactly the backfilled history the derivation exists to hold back,
so a caller reproducing the rule got a different winner and no way to see why. The
test counted the terms, which made the only property that matters unobservable; it
now pins each term to the field at its position.
Also: plugin/label/mcp.json is deleted. It is the only mcp.json in the tree, no
app on main has one, the generator that wrote them was retired with its gate
(manifest/mcp_test.go says so), and it declared seven tools under the old
operation ids with nothing left to regenerate or compare it.
25 mutants in scripts/mutate.py, 11 of them new, 25 KILLED: each reintroduces one
of these defects and the named test goes RED.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
plugin/dataset declares Price: cloud.Metered. TestMeteredSurfacesRequireStanding
(added on main after this branch forked) fails on a metered surface missing from
meteredApps: it would spend a provider's money with no standing check. The scan
that materialises a set is priced per source row read.
Also carries the wove openapi.yaml for the merged surface.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This binary links orm/db, whose toJSONFieldName lowercased the first rune of
each dot-segment and returned the rest untouched — so an all-lowercase payload
passed through byte for byte into json_extract(data, '$.%s'), a SQL string
literal a quote ends.
v0.6.19 keeps only identifier characters and drops the rest, rather than
escaping them: there is then no escaping to get wrong and no dialect to be right
about. It fails closed by construction — a name that had illegal characters
becomes a field that does not exist, so a filter on it matches no rows and an
ORDER BY on it sorts every row equally.
Bump only — no cloud source changes. The four failing packages are the standing
pre-existing set (apps/iam, apps/plan, apps/pricing, manifest); the bump adds
none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The resident bound is a count of tenants with an LRU over it, so one
organisation's arrival drops another's residency. Two properties make that a
capacity decision rather than one tenant unlearning another: the victim's state
is saved before the residency goes, and the reclaim is counted on the probe.
Both were prose. Removing either left the whole suite green.
Losing the save also leaves the slot the eviction took in p.opening held, so the
victim is locked out for the life of the process as well as returned to its last
save.
TestRings_SurviveAnEviction names the lossless property but reimplements it —
it deletes the map entry, bumps the counter and calls save itself — so it cannot
fail when the plane stops doing any of the three. The two new eviction tests
drive plane.resident past maxResident and read only what an operator can read.
Two further tests state the per-field byte cap and the forgotten-subject count
from the direction an operator reads them. Those bounds were already covered
(TestField_IsRefusedAtTheDoorAndNotTruncated,
TestRings_TheCeilingIsMeasuredNotAsserted) and no gap is claimed for them.
Test-only. Nothing is armed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two injections landed upstream today and this binary was still pinned behind
both of them, which is the only part that decides whether they are fixed in
production.
commerce v1.49.50 is the one that matters. Its generic REST list handler put
`?sort=` into ORDER BY json_extract(data, '$.%s') — and into data->>'%s' on the
postgres side — with no check. `a') , (select 1) --` closed the path and the
call. Ordering by a conditional turns ROW ORDER into a one-bit oracle, so it
read another kind's rows across the per-kind authorization boundary a column at
a time. The `data` column is a BLOB, which is why it read as harmless: a bare
LIKE against it matches nothing, and only cast(data as text) shows the leak.
ai v1.832.14 is the original sortField fix in GetDbQuery, and v1.832.15 covers
the two siblings that had copied the same unguarded ORDER BY into chat and
provider listing.
Bump only — no cloud source changes. money v0.2.5 rides along transitively. The
four failing packages are the standing pre-existing set (apps/iam, apps/plan,
apps/pricing, manifest); the bump adds none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
surface-check piped every step to /dev/null and then, on failure, said
'see the message above' — but there was no message above, because it had just
been thrown away. The weave's diagnostic names exactly which schema collided in
which two apps; discarding it is what made the o11y collision expensive to
diagnose.
Proven by the first run after the change: the failure had been reported as
'account cannot project its own document — an app that cannot describe itself is
the bug', which points at apps/account. The captured output says '/bin/sh: go:
not found'. The app was never the problem; the gate was blaming it for a missing
compiler.
Three sites, same shape: both describe loops and the weave. Output is captured
and printed only on failure, so success stays quiet.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The v1.5.51 bump covered the v5 statement builders. This covers the v3/v4 ones,
which are the live path for the older query API and were raw in three positions
rather than one: the ORDER BY direction (a bare string, so
"asc,(select count() from system.tables)" was a whole extra ordering term and
needed neither a quote nor a backtick), the column, and the attribute key in the
single quotes of the map accessors — that last one landing in the WHERE clause,
where it is a predicate the caller writes rather than an ordering oracle.
On the lines that build a filter the VALUE already went through
QuoteEscapedString and the KEY beside it went in bare, which is the same
asymmetry the original sortField bug had.
Bump only — no cloud source changes. The four failing packages are the standing
pre-existing set (apps/iam, apps/plan, apps/pricing, manifest); the bump adds
none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The embedded o11y query plane serves /v1/o11y out of this binary, so the
injection fixed in o11y 25bb4acfb0 was live here on v1.5.50: a telemetry field
key name — caller JSON on the query-range door, and replayed out of stored
dashboards and alert rules — reached ClickHouse unquoted in two positions a
bound parameter cannot occupy, the backtick-quoted identifier (SELECT alias,
GROUP BY, ORDER BY) and the single-quoted attribute accessor.
A metrics group-by name of
le`,(select/**/grouparray(name)/**/from/**/system.tables)/**/as/**/`x
emitted that subquery as an additional SELECT column, and its rows came back in
the response body. Metrics is the reachable one because a metric label is
free-form: an unrecognised name falls back to the labels column by design, so
there is no key allowlist for it to fail against.
Bump only — no cloud source changes. The four failing packages are the standing
pre-existing set (apps/iam, apps/plan, apps/pricing, manifest); the bump adds
none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
plugin/ai/openapi.json was last written at 3b50091c, with go.mod pinned to
hanzoai/ai v1.832.10. d5d768f1 took v1.832.12 for an unrelated key fix and
754fb821 took .13; neither regenerated a subset. Since then the published
document has been describing a router the binary no longer has.
v1.832.12 rewrote the ai door's resource table. Three resources moved address and
one was deleted:
/v1/ai/applications -> /v1/ai/deployments 8 operations
/v1/ai/sessions -> /v1/ai/signin-sessions 7
/v1/ai/users -> /v1/ai/usages/user-names 1
/v1/ai/users/table-infos -> /v1/ai/usages/by-user 1
/v1/ai/permissions deleted 6
The six deleted are GET and POST /v1/ai/permissions and GET, PUT, PATCH and
DELETE /v1/ai/permissions/{owner}/{name}. Every handler behind them was an iam.*
call to the IAM server — controllers/permission.go held no rows of its own — so
the address was a second door onto /v1/iam/permissions, which apps/iam serves and
still serves. Upstream deleted the controller with the routes and says so in the
same edit.
So openapi.yaml has been advertising 23 addresses that answer 404 and hiding 18
that are served: the 17 renamed above plus POST /v1/iam/oauth/device/info, whose
prose was written in b5d8f927 and has been waiting for a regeneration to reach the
document. Every SDK, the CLI's command tree and docs.hanzo.ai are projections of
this file, so a client generated from it calls /v1/ai/applications and gets
nothing.
floor.json comes down by six, which is the act it is designed for: the deletion is
upstream, deliberate and already shipped, so the number moves in the commit that
carries it rather than being absorbed by a re-measure. paths and operations follow
(-1 and -5 — the six net against iam's one), and iam goes up by one.
Forty-two other subsets change prose and schema only: a summary now leads with
what the operation does rather than with the Go identifier that implements it, and
visor's cluster list publishes the degraded field its response type already
carried.
Nothing in the router changed here. Regenerating a second time reproduces all 119
generated files byte for byte.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Complete judged an orphan by Product() — the first path segment after /v1/. A
product is not an app, and fourteen of them are answered by more than one,
because the manifest separates nested prefixes exactly as the router does.
/v1/s3 PROVISIONS an s3 resource (provisioning) while /v1/s3/buckets is the DATA
plane (storage), so storage was charged with provisioning's POST /v1/s3 — a
declaration provisioning both serves and describes — and could not project its
own document. surface-check died there, on an app with nothing wrong with it.
Attribute by manifest.OwnerOf instead: the longest-prefix rule the host itself
routes by, so the gate blames the app the request would actually reach. Asked
rather than re-derived, the two cannot drift.
It is passed IN rather than imported. manifest's own tests read this package to
pin the spec door's address, so an openapi that imported manifest back makes
manifest's test binary an import cycle — the compiler says so. nil is refused
rather than defaulted, because judging nothing is a gate that reports success on
every defect it exists to catch.
The host's own door is declined explicitly. owner answers a MANIFEST question,
and for /v1/openapi.json the manifest's answer is precisely the misroute
cmd/cloud's static route exists to correct; it is served by the host, mounted by
no app, and renders in the fleet document. Product() hid that by accident,
returning "" for any segment holding a dot.
The gate is unchanged where it matters. A declaration misfiled inside an app's
own prefix still fails — pinned here, and mutation-proven against apps/storage,
where an added GET /v1/s3/buckets/:bucket/files takes the run red while the
false positive stays gone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The admin read counted "active" and "trialing" alike when summing mrrCents off
commerce's subscriptions wire. commerce's own rollup counted active only, and
the warehouse SQL behind the SaaS board counted everything except trialing --
so past_due and unpaid were revenue there and nowhere else. Three definitions,
three different MRR numbers for the same account depending on which board you
opened, and no test anywhere asserted the two agreed.
commerce now owns the answer: subscription.Status.CountsTowardMRR. A trial is
not revenue, because nobody has been charged and the trial may end in a cancel
-- counting it lifts the board exactly when a promotion drives signups. This
surface asks that function instead of matching status strings.
Revenue and entitlement are separate questions and the loop still answers both.
A trialing subject is subscribed and still names its plan; it just contributes
no money yet.
The metrics SQL cannot call Go, so it spells the same predicate as
status = 'active' and says so where a reader will find it. That also drops
past_due and unpaid from the board's run-rate, which the Go side never counted.
Picks up commerce v1.49.48, which also stops overstating a quarterly plan by 3x
-- MRR now divides by IntervalCount, so the mrrCents this reads is already
right and nothing here re-derives it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The package, its directory, its plugin and its manifest row all said
"agentskills" — a compound naming the AUDIENCE (agents) alongside the thing
(skills). The thing is skills; every reader of this catalogue is an agent, so
the qualifier distinguishes it from nothing.
Renames the token only. The wire is untouched on purpose: the served addresses
stay /.well-known/agent-skills/... and the document schema stays
hanzo.agent-skills/v1, because those are an external discovery convention we
implement rather than a name we own. The manifest row's Prefixes are unchanged,
so nothing moves address — only the app's own name, its import path
(apps/skills), its plugin binary and its file names change.
Also fixes two comments still pointing at clients/agentskills, a path that
stopped existing when the subsystems moved to apps/.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ed4fbe5b corrected the auth line in all three brand catalogues — hk- is not a
key, so the sentence pointed a reader at a credential that resolves to nobody.
The correction is right. But SKILL.md is a GENERATED file whose bytes are
attested by a sha256 in index.json beside it, and the hand-edit changed the
bytes without recomputing the digest. All three brands went stale at once.
That is not cosmetic: the digest IS the integrity contract a client checks
after fetching a skill document, so every reader that verified was told the
catalogue had been tampered with. TestServeSkillDigest and TestCatalogIntegrity
were failing on main for exactly this, and they were right.
Recomputes the three digests and nothing else — the served bytes were already
correct, only the attestation had fallen behind them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`make -f mk/fleet.mk openapi-weave` has refused since 593aa309 regenerated the
o11y subset:
schema "Service" means different things in "ingress" and "o11y"
Six names carried two shapes across the fleet — `Service` and `TLSConfig` against
ingress, `Account` against books, `Channel` against content, `Event` against
analytics, `Host` against plugins — all six from hanzoai/o11y's internal type
packages. The weave is right to refuse: a generated SDK binds whichever shape the
merge reads last. But openapi.yaml is regenerated by ONE command for the whole
fleet, so the refusal did not merely hold o11y's typed ops out of the document —
it meant nobody could add or change ANY api surface and prove it.
Renaming the six upstream only reveals the seventh. They are ordinary words, five
other apps use them, and a type name that has to stay unique against every app in
the fleet is a name nobody can choose safely. The seam is the defect, not the
nouns.
apps/iam already answered this and there is exactly one mechanism: zip.Graft. A
grafted op carries Origin = the child's AppName, and zip qualifies every named
type it reaches as "<origin>.<Type>" — unconditionally, not on collision, so a
published name is never a function of who else is in the room. That is why
identity's 95 schemas are iam.* and have never collided with anything.
MountO11y now builds one zip.App{AppName: "o11y"}, mounts the whole observability
surface on it — cloud's own org-pinned reads and hanzoai/o11y's relay table
together — and grafts that into the host. ONE origin for the product: splitting it
down the module seam would namespace half of o11y's types and leave the other half
bare, which is two conventions for one thing.
make -f mk/fleet.mk openapi-weave ok
…and again, byte-identical (idempotent)
THE PUBLISHED SDK SURFACE MOVES. Schema names are part of the contract:
- 777 o11y schema names go from X to o11y.X. Against the openapi.yaml published
today, 24 of those are renames of names it already carries; the other 753
appear for the first time, because the weave has refused to publish them since
they arrived.
- NO other app's schema names change. Each of the six formerly-colliding bare
names stays with the app the published document already binds it to —
Service/TLSConfig ingress, Account books, Channel content, Event analytics,
Host plugins — with the same shape.
- Nothing outside apps/o11y changes to keep compiling. These are generated
schema keys, not Go identifiers; no test in this repo pins one of o11y's.
- The MCP tool list is untouched: plugin/o11y/mcp.json is byte-identical.
The rename is PURE, and measured rather than eyeballed. Against the subset that
PRISTINE main regenerates from the same source: all 389 (path, method,
operationId) triples identical, and all 389 operation objects byte-identical once
the o11y. prefix is undone — parameters, bodies, responses, tags, prose and x-app
included. The woven document resolves 2987 $refs with zero dangling.
Two facts the graft made local, both load-bearing:
ALL /v1/sentry/* stays on the HOST. zip.App.Declaration drops HEAD and OPTIONS
unconditionally — they are the shadows fiber generates for a GET and for CORS — so
a door opened with All cannot cross a graft intact, and OPTIONS is a method that
proxy genuinely answers and publishes as an operation. It is registered at the
same point in the same order, and costs nothing there: a wildcard proxy declares
no typed op and contributes no schema.
cloud and the module BOTH claim GET /v1/o11y/{logs,metrics} and POST
/v1/o11y/query_range. scope.go is the one owner — it pins the caller's org
server-side — and first-registered is what makes it the half that answers. That
used to depend on the host's global mount order; both halves now register on one
app, two adjacent lines apart. TestHostRoutesStillWinTheThreeSharedAddresses is
the gate, and it is mutation-proven: put o11y.Mount first and all three go red.
openapi/floor.json rises with the document — 1410 → 1679 paths, 1979 → 2333
operations, o11y 28 → 363, sentry 7 → 24 — and no product falls, which is what the
ratchet checked before writing it.
One document defect SURVIVES and is unchanged: at POST /v1/o11y/query_range the
router answers cloud's untyped builderQueryHandler while the document publishes
the module's typed contract, because only the module's half has a registry entry
to publish. It predates this and goes away with the console v3→v5 migration that
deletes cloud's half (apps/o11y/LLM.md).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/iam/oauth/device/info arrived with hanzoai/iam v1.34.0 -> v1.34.5
(85f5697c) and reached the router with no prose anywhere. It is an UNTYPED route
inside a GRAFTED app, so neither seam that normally carries prose reaches it:
zipdoc lifts doc comments off TYPED ops, and the upstream module ships no lift
for this one, so there is nothing in zip's extraction for a host to read.
openapi.Complete then refused the whole document — "1 operation(s) say nothing
about themselves" — so apps/iam could not describe itself at all, and the drift
gate stopped there without reaching any app after it.
openapi.Describe is the seam that refusal names for exactly this case. It is
additive metadata on a route the router already carries, so it cannot add, move
or rename an operation; it renders nothing if the address is not live. The
sentence is the upstream handler's own, restated where a host can reach it, and
it says the three things a caller cannot guess: the fields come off the pending
code's own application, the read is a POST because the user_code is a secret,
and every refusal is the same opaque one so a 40-bit code cannot be hunted.
The sentence belongs upstream, on the operation. When github.com/hanzoai/iam
gives the op its own prose, this goes away.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zap-proto/zip went v1.18.16 -> v1.18.23 (c516e5a3, then 85f5697c) and the doc
extraction changed with it: a lift no longer keeps the leading Go identifier,
so "RevokeKey revokes the caller's own API key" became "Revokes the caller's own
API key". The committed lifts were never regenerated against it, so `zipdoc
-check` — the per-package gate `make test` runs before the suite — was red in 53
of the 95 directories that carry the generate line.
This is `go generate -run zipdoc` and nothing else. Parsing both trees' ASTs
rather than diffing their text, it moves:
377 Description lifts losing their leading identifier
8 Fields lifts catching up to a doc comment their source already changed
86 routes gaining a lift at all — the bump also reads an UNTYPED route's
prose, which is the whole point of c516e5a3
and, in the same comparison:
0 Examples changed
0 routes lost a lift
0 Descriptions changed in any other way
A lift carries Description, Fields and Example and nothing else, so it can only
ever move PROSE — it cannot add, remove, rename or retype an operation. The
router remains the only thing that decides what is published.
51 of the 86 new lifts are the doc comment of cloud.Handle, not of the handler
it wraps: the extraction follows the Service-scoped binder rather than the
function passed to it. Every one of those routes already declares its prose with
openapi.Describe, which outranks a lift, so none of it reaches a published
document — verified: the string "Binds a Service-scoped handler" appears zero
times in plugin/*/openapi.json. It is inert, and it is what the generator emits;
editing a generated file to remove it would only turn the gate red again.
The published documents are deliberately NOT regenerated here. Doing so moves
real API surface that is nothing to do with this bump — plugin/ai alone renames
23 operations and adds 18 behind the ai v1.832.12 bump — and openapi-weave
refuses to run at all while six schema names mean two different things across
apps. Both need their own review.
Regenerating a second time produces zero further change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GetCert qualified the certificate id with the caller's tenant while
GetApplication used the platform partition, so a deployment whose IAM_ORG was not
"admin" resolved its application and then could not find that application's
certificate. With no cert there is no key to validate a bearer against, so the
process authenticates nothing — and the message named a missing certificate
rather than a lookup pointed at the wrong tenant.
It surfaced the moment identity moved in-process on devnet: /v1/models 503 with
`read cert "cert-hanzo" ... does not exist`, while the same cert answered 200 at
id=admin%2Fcert-hanzo. Production never hit it because it reads the standalone
IAM, whose store has been converged for months.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
main moved again while the suite ran. One conflict, modify/delete on
plugin/gateway/mcp.json: main deleted all 116 committed tool files because the
catalogue is a query now — the host asks each child over its ZAP socket rather
than reading an array some earlier build wrote down — and this work had
regenerated the gateway one.
The delete wins, and it takes plugin/risk/mcp.json with it. That file is the
same artifact class main just retired: nothing embeds it any more
(plugin/embed.go), nothing writes it any more (mk/plugin.mk, describe.go), and
leaving it would put back one of the 116 second sources whose whole defect was
that they could only be stale or accidentally correct.
Still nothing armed and nothing routed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
internal/org and apps/mpc each turned the master into a key themselves, so the
estate had three answers to one question and no test compared them.
The snapshot cipher derived HKDF(master; salt="hanzo/org-db/v1"; info=orgID) by
hand while the local file it snapshots was opened by cek. The two were supposed
to agree and nothing checked that they did — a comment's promise, which is the
same shape of bug as base's deleted core/encryption.go, where the claim was
simply false. Worse, keying on the org alone meant every database an org owned
sealed under ONE key: its settings snapshot opened its ledger. cek binds the
subsystem, so Seal/Open now take the name (ns, subsystem) the file was opened
under, and TestCipherKeyIsCEKDerived opens a sealed blob with a key derived
independently through cek — the equality is now a test, not a sentence.
apps/mpc ran HKDF(master; salt=orgSlug; info="cek-aes256gcm"), inlined verbatim
from hanzoai/kms/sdk/go. That repo is archived and read-only, so this is the
only live copy and there is nothing left to stay byte-identical with. Argon2id
stays exactly where it was: stretching a guessable passphrase and deriving a key
for a named thing are different jobs, and cek does the second one. The client
takes a namespace instead of a slug — cloud has one door where a string becomes
a tenant, and TestOnlyOrgnsBuildsANamespace is right that this package is not it.
Also drops the NewReplicator/WithEncryption aliases. Nothing in cloud shipped
through them; they only survived to let one test seal through replica.Cipher,
whose string-shaped Seal a namespace-keyed cipher cannot satisfy. That test is
replaced by TestDurableShipsCiphertext, which asserts the same property against
the path that actually ships: Sync writes a blob the marker does not appear in,
and a successor reads the row back out of it.
Nothing migrates. A database is born under its key or it does not exist.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Prefixes NEST. provisioning is metered and routed /v1/vector and /v1/search;
product is Free and routed the more specific /v1/vector/collections,
/v1/vector/stats, /v1/search/indexes and /v1/search/stats. Billable scanned the
metered trees with a bare HasPrefix, so all four of product's surfaces billed on
provisioning's standing — a Free product gated behind a balance.
The router resolves by longest prefix, so ownership does too: manifest.OwnerOf
answers which app actually serves a path, and if that app is not metered then
nothing is spent and nothing is owed. No new list to drift — the manifest already
owns routing, meteredApps already owns price.
Also deletes an invented history. The comment claimed "/v1/provisioning/, which
provisioning does not answer" had been in this file; `git log -S '"/v1/provisioning/"'
-- spend.go` returns exactly one commit, the one that wrote the claim. The real
defect was assuming provisioning's tree was /v1/provisioning when the manifest
routes it elsewhere, which is what the corrected text now says. A comment
describing a defect that never existed is the same failure the commit was
punishing.
Negative-controlled: removing the ownership check fails the new test on all four
paths. The 19 TestAudit_* failures in this package are pre-existing (verified by
stashing) and untouched.
vmproxy.go was the final VM reader in the repo. It spelled a vendor into the
route table (/v1/o11y/vm/{query,query_range}), admitted 19 exact PromQL
strings, and returned that store's Prometheus envelope byte-for-byte. The store
is gone, so the route named for it and speaking its wire is gone too — keeping
it would be a dependency's vocabulary outliving the dependency, which is the
thing metricsgauge.go already refused when it declined to keep the envelope.
Of the 19 queries, exactly 3 are still MEASURED. `up`, `sum(up)` and `count(up)`
were the platform-health board asking how much of the fleet is up now and
lately, and the fleet prober has been recording hanzo_service_up every 30s the
whole time. So that question is asked plainly instead: GET /v1/o11y/availability,
a typed op, platform-sudo as before, answering the instant inventory and the
trend in ONE read where the board used to make three PromQL round-trips.
The number changes and the endpoint does not hide it. `count(up)` counted every
scrape target VM federated — hundreds. `total` counts fleetTargets, a couple of
dozen services we chose to knock on. Per-replica identity stays gone, for the
reason status.go already recorded: a Service address is not a pod.
The other 16 have no native producer and are DELETED rather than answered
empty — verified, not assumed:
node_memory_* node-exporter, scraped per node
container_memory_working_set kubelet cAdvisor, scraped per pod
kube_deployment_* kube-state-metrics, scraped
kube_statefulset_* kube-state-metrics, scraped
lux_validator_* (4) ghcr.io/luxfi/monitoring /validator-exporter —
lux_network_* (6) binds :9101, renders a Prometheus exposition,
no push/OTLP/egress at all; a lux-side vmagent
scraped it and remote-wrote to the hub. That
write had already POSTed into a 404 for 1369
consecutive retries in July, so the board was
dark before this change.
ALERTS{alertstate="firing"} vmalert's remote-write of its own state, and
nothing else.
availability.go's header is the ledger: each dead producer by name, and what
would have to be MEASURED to get the panel back. Two of them should not come
back as metrics at all — the replica grid is the API server's object status and
should be watched at the source, and the firing set is rows in o11y's ruler
store, so it is a projection of rule state rather than a series to re-collect.
The lux boards come back when that exporter pushes to the ZAP metric receiver
the way this process does; gaugeSeries then works on those names unchanged.
Five of the console's 18 declared queries were already dead weight — the four
freeze-detection signals and the alert rollup were allowlisted here and fetched
by no panel.
Console impact. The client reads data.result[].value and never checks `status`,
so a reshaped 200 would parse to [] and paint a board of zeroes; a deleted route
throws and renders its error card. That is why this is a deletion.
MetricsModule/InfraMetrics (/metrics) KEEPS everything, move to the new op
StatusModule (/status) KEEPS everything but the Endpoint col
LuxNetworkModule (console.lux.cloud) goes fully dark; nothing can serve it
Also retires the prose VM left behind: telemetry.go and metrics_http.go claimed
metrics were collected by scrape because "every metric READER queries
VictoriaMetrics" — both false since the :9464 exposition was deleted. Every
reader now goes through metricsgauge.go. serviceUp is shared with summary.go
rather than copied, since it is the same value.
Gates: build, vet, ./apps/o11y/ + root tests, and cmd/cloud deps — all pass.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both changes were merged and neither was reachable: v1.832.10 and v1.49.46
predate them, so the fused binary kept resolving hk- through the old
dispatch no matter how many times main was rebuilt.
ai v1.832.11 makes isIAMApiKey recognize sk- and asks the provider table
first, so a retired prefix falls through to IAM and comes back key_unknown
with the one sentence a stranded holder can act on. v1.832.12 stops that
sentence's sibling — the pk- refusal — pointing at /api/models, which 404s.
commerce v1.49.47 drops the same dead prefix from its edge comments.
The dispatch order moved, so the hk- refusal has to be re-verified against
the rolled image rather than assumed: it reaches IAM by a different road now.
A valid, funded sk- key could not buy one token of inference. IAM resolved it
correctly — get-user?accessKey returned {"status":"ok","data":{"owner":…}} —
while POST /v1/chat/completions answered 401 "invalid API key" for the same key
in the same second.
Both answers were right. They were answering different questions, because the
inference path never asked IAM.
ai v1.832.10 is what this module pinned, and there isIAMApiKey(token) meant
strings.HasPrefix(token, "hk-"). Once hk- was retired estate-wide, IAM minted
only pk-/sk-, so every key a customer can hold missed that branch and fell to
the dispatch default — which read sk- as an UPSTREAM VENDOR key, looked it up in
the provider table, missed, and returned the bare authError("invalid API key").
The bare string with no ": %s" suffix is the fingerprint: it is the one refusal
on that path that carries no cause, because a provider-table miss has none to
give. IAM was never consulted, which is exactly why querying IAM directly
disagreed with the endpoint.
ai v1.832.11 fixed it by asking the STORE rather than the spelling: the provider
table first (an exact lookup, so it can never claim a key it does not hold), and
anything it misses is put to IAM, whose refusal names the cure where a provider
miss can only say "invalid". That fix has been tagged and unreachable since —
this module still pinned .10, and so did v1.801.375, so rolling that tag would
have shipped the same 401.
Nothing here weakens the door. The two shapes and their guards are unchanged and
still covered: pk- authenticates nothing (KeyWrongDoor, never a principal — it
ships in client JS), sk- keeps its same-tenant pin (KeyForeignUser still fires on
a cross-tenant reference), and an hk--shaped string is simply not a key and takes
the generic key_unknown path that renders "mint a new one at cloud.hanzo.ai/keys"
— no branch of its own, so the third family cannot come back.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
main moved eight commits while this was building, so it is integrated here
rather than pushed past. One conflict, in spend.go, and it is the good kind:
both sides were right about different things.
main replaced the hand-written meteredTrees path list with meteredApps — a set
of app NAMES resolved through manifest.PrefixesFor — because a copied routing
table goes stale, and that one had in six ways. This branch had added
"/v1/risk/" to the list main deleted.
The mechanism is main's and the fact is this branch's. plugin/risk declares
Price: cloud.Metered, and main's new TestMeteredSurfacesRequireStanding fails
in BOTH directions — a Metered surface missing from the list is a surface that
charges and cannot be gated. So risk is carried across as a name, and that test
passes because of it, not in spite of it.
Nothing is armed and nothing is routed. apps/risk is still mounted nowhere;
/v1/risk appears only as an abuse-gate exemption, a manifest entry and now a
standing requirement. The abuse gate stays shadow per org — edge.Store.Mode
returns live only on an exact match and is not inherited. /v1/ml is untouched.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The fleet's agent door answered from plugin/<app>/mcp.json: the tool array each
app's binary projected when it was BUILT, embedded by plugin/embed.go and handed
to zip as Plugin.Tools. 116 files, 49,865 lines, and a second source for a fact
every child already knows.
A second source can only be stale or accidentally correct. This one was stale in
the way no gate in this repository could see: o11y's 353 missing ops live in
github.com/hanzoai/o11y, so a go.mod bump in ANOTHER repo invalidated an artifact
in this one with nothing in the diff to say so. Regenerating it more often is not
the fix — a generator on a hook is still two sources with a race between them,
and the trigger is in a different repository. (593aa309 did regenerate it, which
is why the file reads 365 today. The next cross-repo bump silently un-fixes it.)
So the host asks. POST /v1/mcp is the HOST's own handler now (zip's is Disabled,
so exactly one handler holds the address). A tools/list forwards the CALLER's own
message to every composed subsystem's own /mcp over its private ZAP socket, in
parallel, and unions the replies — zip.App.Start resolves a cold child on the
same single-flighted path a prefix request takes, so the first list pays one
start per app and nothing after it does. A tools/call goes to the app that listed
the name, verbatim; the child's own registry decides whether the tool exists.
A SUBSYSTEM THAT DOES NOT ANSWER IS NAMED, in result._meta["hanzo.ai/unavailable"],
because a silently-short list and a stale file are the same defect: the caller
cannot tell an app that serves nothing from one that did not answer. Measured on
the built binaries — host + real o11y child, kms pointed at a dead address:
tools=364, unavailable=[{kms, connection refused}]
364 and not 365 because o11y projects get_v1_o11y_logs twice; the door serves the
first and logs the collision. THAT DUPLICATE IS WHY cmd/cloud's tests were red on
main — zip refused the Load ("tool is already served by plugin o11y"), a boot
failure. One name still has one owner; it is no longer fatal to the fleet.
Also gone with the mechanism they configured: manifest.App.Open and zip's
one-open-plugin rule. The host forwards the caller's own request to EVERY
subsystem now, so each answers for this caller out of its own rows, and being
asked per caller is no longer a privilege one app holds.
The release gate moved with the door. Car 3 compared the live tool count against
`jq -s length` over the committed files — both sides were the same bytes, so it
proved only that the image carried its own tree, and it passed while o11y's
catalogue held 12 of 365. It asks the better question now: did every subsystem
answer. A broken deployment used to match the files exactly.
plugin/<app>/openapi.json SURVIVES, for the one reason the catalogue could not:
the weave carries each subsystem's prose, and that prose is lifted from the app's
SOURCE at describe time (openapi.Synopsis). A running child has no comment to
read and would answer with its deployment's brand blurb, which the weave would
publish as the description of every product tag. Deleting that half waits on the
synopsis becoming a declared value.
Tests are against RUNNING subsystems (fleet/mcp_test.go): real zip children on
real ZAP sockets, exact sets, bodies never status codes. Mutation-checked three
ways — unmount a child, silence the outage report, ask only the first app — all
three go red.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nothing textually conflicted, so the merge could not see this: the branch was
written when hk- was one of the key shapes, and main dropped it from
APIKeyPrefixes in the meantime. isAPIKey("hk-...") is now false, so a validated
caller presenting one falls through credentialClass to session rather than
secret, and TestCredentialClass_ReadsTheCredentialNotTheClient failed on the
one subtest that asserted the old taxonomy.
Main is right and the branch is stale, so the three sites the branch
reintroduced are re-cast the way main re-cast every other one. The fixture is
kept rather than deleted, restated as the negative it now proves: an hk- string
contributes nothing to the classification, so putting hk- back into
APIKeyPrefixes turns this red.
No behaviour changes. Both lanes involved are attributed lanes — secret maps to
agent, session to human — and neither is the bot lane, so nothing is granted or
refused differently. The authentication path is untouched: hk- reaches no key
door and resolves to no principal, which is main's point.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud pinned cek v0.2.1. The releases after it were a trap: v0.2.2 shipped a
sidecar.go written against sqlitedrv APIs that hanzoai/sqlite v0.5.0 removed,
so anyone bumping cek and sqlite together got a module that does not compile —
and cloud now takes sqlite v0.5.0.
v0.2.3 is the release where the sidecar states its own scheme instead of
reaching into the driver for it, which is why it builds against v0.5.0 at all.
Verified: cek at v0.2.3 builds and its own suite passes on sqlite v0.5.0, and
cloud's suite is unchanged by the bump — the same 7 packages fail before and
after, none of them cek's.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
tools and automations both debited a unit on the way out and never gated on the
way in. An org at $0 ran either one unbounded and the ledger simply went
negative — the rate budget automations already had bounds the SHAPE of the spend,
never the amount. Both now gate the same unit they meter, from the same knob, so
what is authorized and what is charged cannot drift. automations gates in
startRun, the one choke point all four run-start paths pass through (manual, MCP,
trigger, cron), so the gate is not something three entrypoints walk around;
engineErr stops folding a 402 into "engine: …" 500, which told a customer their
automation was broken when they had run out of credit.
meteredTrees restated the fleet's routing table by hand and had gone stale in six
ways, each one a surface that charges and could not be gated: it said
"/v1/provisioning/", which nothing answers — the manifest routes provisioning at
/v1/{datastore,docdb,kv,search,sql,vector}, so every vector/sql/kv/docdb create,
the exact set the non-LLM billing gap was opened for, was not a billable path at
all. projects answers /v1/sites, venue answers /v1/cloud, tools answers
/v1/skills and /v1/plugins, and ten Metered surfaces were missing outright. It is
a set of NAMES now, resolved through manifest.PrefixesFor — which paths an app
answers is the manifest's fact, read from there rather than copied.
spend.go has cited TestMeteredSurfacesRequireStanding since the day the list was
written. It did not exist. That is the third comment in this repo caught
describing a gate nobody built, which is worse than no gate: it reads as
enforced, so nobody looks. It exists now, reads Price out of every
plugin/<name>/main.go the way TestPriceDeclared does, and fails in both
directions — a Metered surface missing from the list, and a listed surface that
is no longer Metered.
Also deletes automations' meterUnit, which had no callers.
On 08-01 api.hanzo.ai/v1/models and /v1/chat/completions answered 503
{"error":"mount /v1: no instance running"} for ~30 minutes while the pod
reported Ready with 0 restarts.
Nothing in that sentence was a bug in mount(). o11y v1.5.41 seized
:4317-:4319 from the `ai` child, the child's listen failed, and the host
degraded it to absent — which is the 2026-07-29 fix working exactly as
designed. The defect is that absence was reported in a FIELD and the probe
reads the STATUS CODE. `ai` owns the greedy "/v1", so every specifically
mounted prefix (/v1/sentry, /v1/o11y, /v1/commerce/tenant, /v1/admin/*)
kept answering from its own subsystem and only a path falling THROUGH to
`ai` showed it.
So split the question the two probes were sharing:
/healthz liveness. 200 while the process routes, unchanged. Restarting
cannot help — the cause is in the image or the config, so the
replacement fails identically — and it would take the console,
the log stream and 111 healthy siblings with it.
/readyz readiness. 503 when draining, or when a VITAL subsystem is
absent. New route; it did not exist on :8000 at all.
App.Vital is the other half of App.Required, kept separate because they
answer different questions. Required asks "may this process run", and its
own doc names "a pod that never goes Ready" as the one thing aborting buys
before rejecting the whole package. Vital buys exactly that and nothing
else: the pod stays up, keeps serving its siblings, and can still say why.
Required's bar is "serving without it is UNSAFE"; Vital's is narrower —
"serving without it is POINTLESS". Exactly one app qualifies, and "/v1" is
why: it is not a subsystem's prefix, it is the product API's remainder.
The timing is what makes a 503 cheap: a rollout whose image cannot start
`ai` never gets a Ready pod, so the Deployment stalls and the OLD pods keep
serving — the bad config stops at the first replica instead of reaching all
of them. Fleet-wide, endpoints empty and the product is down, but it was
already down; now kubectl says so.
A non-vital absence stays READY and is still reported. Pulling a pod
because one minor subsystem died turns a partial failure into a total one,
which is the same mistake as aborting, made later.
Also closes a contract drain.go describes and nothing delivered: it states
that SIGTERM flips /readyz to 503 so K8s marks the pod NotReady, and the
root package implements that on the ops listener at :9090 — but both k8s
probes point at :8000, where /readyz 404'd. The drain signal guarding the
M3 writer-election handoff was never read by anything. The host keeps its
own flag rather than borrowing cloud.SetDraining(): it does not link the
root package, and it is a different process anyway.
Tests reproduce the outage with the real app name and the real child-exec
path (/bin/false), and fail without the fix:
GET /readyz = 200, want 503.
The chart still probes /healthz for readiness and MUST NOT be repointed
until an image serving /readyz is rolled — a 404 fails readiness exactly as
hard as a wrong port, which is its own recorded outage.
All three white-label catalogs told agents an hk- key was accepted. It is
not — it resolves to nobody — so the sentence sent a reader to a credential
that cannot work. Names the secret sk- instead, and says plainly that a
publishable pk- identifies an org and never authenticates.
Found by a binary-safe grep: these files sit beside committed binaries whose
string tables contain the prefix, and the default grep skips those paths, so
a normal sweep reports the tree clean.
cloud already embeds IAM (apps/iam, grafted in-process) and prod already enables it,
but /v1/iam/.well-known/jwks answers {"keys":[]} because the store it opens is empty:
/var/lib/cloud/orgs/_platform/global.db, 4096 bytes. The real identity graph — 391
users, 134 orgs, 286 applications and the 10 signing certs — lives in the standalone
pod's /data/iam/iam.db. That is why every service still points at iam.hanzo.svc, and
why a single pod failing readiness takes identity down for the fleet.
It is not a file copy, because the stores differ three ways at once. The standalone
writes PLAINTEXT — the literal "SQLite format 3" header, which is the exposure
apps/iam/openStore exists to remove. The embed writes through cek, keyed from the
process master. And it opens exactly one namespace-derived path, so a file dropped
beside it is not a file it reads.
So the rows move, not the bytes: read the source with a plain driver (read-only and
immutable, so a live pod's WAL is never touched), write through the same cek.Open the
subsystem itself uses. One transaction — identity half-moved is worse than not moved.
Columns and DDL are read from the source rather than restated here, so a schema that
grows cannot silently lose its newest field. It refuses a source with no certs, and
refuses a non-empty destination unless told, because merging two identity stores is a
decision rather than a default.
Rehearsed against a verified 96MB snapshot of prod on a Linux box (the cek codec needs
RAM-backed scratch, so not on macOS): 5745 rows across 15 kinds written and verified
kind-by-kind, certs=10, and the destination reads back as ciphertext rather than
"SQLite format 3".
The :9464 Prometheus exposition existed for exactly one caller: a scraper that
filled VictoriaMetrics, because VictoriaMetrics was where every reader looked.
The readers moved to the telemetry store and the measurements now push
themselves there, so what is left is a listener whose only client no longer
exists. That is not a way out, it is an open port.
The registry STAYS. It stops being a published surface and becomes the buffer
the meter provider renders into and the push drains — which is why the otelprom
reader is untouched while its HTTP face is deleted. The exporter was never the
scrape; it is the encoder, and it is the one that already speaks the family
model the datastore receiver takes.
cloud.Metrics() goes with the listener it served. MetricGatherer stays and is
now the only door onto the registry: read what was measured, never register.
Note the :9090 ops /metrics is untouched — it is a hardcoded `cloud_up 1` stub
for liveness, not the registry, and it answers a different question.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The lifecycle defense: an edge traffic sensor, a per-credential abuse gate and
the identity boundary's own attestation. Compiled in and mounted, SHADOW per
org — edge.Store.Mode returns live only on an exact "live" match, is absent
from the inherited base, and is never set here, so nothing is armed by landing
it. /v1/risk stays unrouted; /v1/ml is untouched.
Four conflicts, each resolved on its merits rather than by side:
middleware_identity.go — an import collision where both sides were right.
main added namespace.Sanitize (OrgHasUnsafeRune, the cross-org fold refusal),
the branch added principal.Mint. Both symbols are live in the merged body, so
both imports stay; either blanket resolution drops a defense and the build.
middleware_ratelimit.go — the branch's code, main's reasoning. The branch
tests the ROUTER's path (RoutePath + underPrefix) instead of the raw spelling,
which is what stops /v1/billing/../v1/ai/chat from prefix-matching its way
into a rate-limit exemption. Its rationale, though, describes an in-process
HTTP self-dispatch that main has since replaced with a typed ZAP op, so the
comment kept is main's, which is the one that is true here.
plugin/o11y/main.go — complementary, not competing. The branch installs the
identity boundary and the abuse gate in this hand-written main; main added
cloud.ErrorHandler so a propagated refusal renders as its own status instead
of 500. Those two middlewares are precisely what emits refusals, so the
handler matters more after the branch lands, not less. Both kept.
openapi/floor.json — the pointwise maximum, which is what Floor.Raise means:
taking the lower side would silently undo the other's ratchet. Then
regenerated from source, so the regeneration is the truth.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two duplications, both of the same shape: cloud held a second copy of a fact
that another service owns.
PKCE (iam v1.34.5)
------------------
apps/deploy/login.go had pkceChallenge, whose own comment said it was
"byte-identical to IAM's own pkceChallenge", and apps/integrations had
twitterChallenge -- the same two statements again under a third name. iam is
the authorization server; it decides what a code_challenge is. It just moved
that derivation out of internal/oidc into pkg/pkce so a client can import it,
so both copies are deleted and both call sites use pkce.Challenge. The
iam-facing one also sends pkce.Method rather than its own "S256" literal, so
this client cannot ask for a method that server refuses. Twitter's call keeps
its own literal: that is X's protocol requirement, not iam's.
twitterVerifier stays exactly as it was -- it is a different thing (a
per-app-constant verifier, with the reasoning for why that is safe only for a
confidential client), and only the challenge transform was duplicated.
MRR (commerce v1.49.46)
-----------------------
apps/admin/commerce carried monthlyNormalized, a third copy of commerce's
interval normalization. It agreed with commerce's arithmetic and still got the
answer wrong, because it never read quantity at all: a 10-seat plan at
$20/seat reported $20 here and $200 in commerce's own rollup. commerce
invoices Price x quantity, so $20 was never the revenue -- it was the unit
price wearing the name MRR.
commerce now puts mrrCents on each subscription in /v1/billing/subscriptions,
computed by the one definition that also feeds its rollup and its event
stream. This is a display surface, so it reads that number. Price and Interval
are dropped from subscriptionsWire entirely -- not left in place unused, which
would invite the next reader to normalize them again.
Tests
-----
- deploy TestPKCEChallenge: deleted. It pinned the RFC 7636 Appendix B vector
against the local function that is gone; the vector is pinned in iam's
pkg/pkce, which also now checks the encoding is unpadded base64url. The
authorize-redirect test still proves the published challenge derives from
the stored verifier -- that part is this package's own and is kept.
- integrations: the "challenge must be the HASH of the verifier" assertion is
kept verbatim, repointed at pkce.Challenge.
- admin/commerce TestMonthlyNormalizedCents: deleted with the function it
pinned; that arithmetic is pinned in commerce's api/billing. Replaced with
five tests over a stub commerce that prove Plan SUMS mrrCents, ignores
canceled subscriptions, returns an honest pay-as-you-go zero, decodes the
mrrCents tag, and -- the case the old code failed -- reports 20000 for a
subscription whose price says 2000.
- apps/admin cockpit/finance stubs now serve mrrCents, because that is what
commerce serves. Their expected MRR figures are unchanged; the stubs were
simply a wire shape behind.
Failing-test set unchanged: the same 7 packages (apps/ai, apps/iam, apps/plan,
apps/pricing, cmd/cloud, manifest, openapi) fail on origin/main before this
change and after it. cek stays at v0.2.1 deliberately -- v0.2.2 adds a
sidecar.go built against hanzoai/sqlite v0.4.0 APIs and does not compile
against the v0.5.0 this module already pins.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The measurements were never the problem — the prober has recorded
hanzo_service_up for as long as it has existed. The problem was that the only
way to READ it was a PromQL instant query, so /v1/summary (PUBLIC,
unauthenticated) and /v1/o11y/status each held a VictoriaMetrics client, and VM
could not be removed without both going dark.
latestGauge answers one question — the newest value of a gauge, per label set —
against event.metric joined to event.series on the fingerprint, because a sample
carries no labels and a series carries no values. argMax over the window, not an
average: a service down for four of the last five minutes and up now must read
1, and any aggregate answers 0.8, a number true of nothing. Asking it in ONE
place is what stops the two callers growing two dialects of the same SQL.
⚠️ status.go loses per-replica identity, and that is a real loss rather than one
to paper over. It read `up{service=…}` — the SCRAPE's own metric, one series per
target, so instance/pod came free because something had visited each pod to
produce it. Nothing scrapes anything now. hanzo_service_up is recorded per
SERVICE: the prober asks a Service address whether the service answered, and a
Service address is not a replica. So the inventory is one row per service, keyed
by what was actually measured. Synthesising a pod name would invent a fact no
measurement supports. Wanting per-replica health back means MEASURING it — the
prober resolving endpoints and probing each — not deriving it.
Source says "datastore" now because it is. The dead instant-query decoder and
its envelope go with the caller that needed them; vmproxy.go is the last VM
holdout and is SuperAdmin-only.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
APIKeyPrefixes drops hk-, so a string with that prefix no longer reaches
isAPIKey and never travels to IAM's get-user door. It resolves to no
principal, exactly as any other unrecognized bearer does.
pk- is unchanged: it is still recognized, still short-circuited to nil in
validatedPrincipal, and still resolves through resolve-key to an org and
nothing else. Nothing here lets it authenticate.
The analytics free-text scrubber loses its hk alternative for the same
reason — the prefix names no credential, and the bearer/entropy/query-param
branches still cover every real one.
Fixtures that used an hk- value as a working key are re-cast to sk-;
enumerations of the family throughout the comments now read pk-/sk-. Those
enumerations also carried fw_ and hz_, which isAPIKey has not recognized
since they were dropped as never-minted entries, so they go too — a comment
that claims to describe isAPIKey should describe it.
plugin/dns/openapi.json is regenerated from apps/dns; openapi.yaml takes the
identical string. The remaining hk- in openapi.yaml and plugin/{iam,ai} is
projected from hanzoai/iam and hanzoai/ai and has to be ripped there.
Prometheus is gone. Metrics were the last signal still leaving this process by
being PULLED: a exposition on :9464 that something outside had to visit. That
was never a statement about metrics, it was a statement about where they were
KEPT — VictoriaMetrics, because every reader queried VictoriaMetrics. With one
telemetry store left, traces and logs already reach it in-process over ZAP and
metrics had no way there at all. cloud was the only service in the fleet whose
own measurements existed nowhere once the scrape stopped.
So metrics travel like their two siblings: gather the registry on a timer and
hand the batch straight to datastoremetrics.WriteMetrics, in-process. The
receiver on :4319 is for OTHER processes; this process owns the writer, so it
calls it rather than dialing its own socket to reach a function already in
scope — the same reasoning that makes traces in-process by default.
The registry stops being a published surface and becomes an internal buffer.
That is why this gathers Prometheus families instead of writing a second OTel
exporter: zapmetricreceiver.MetricBatch IS the family model — name, help, type,
labels, value, buckets, quantiles — and the meter provider already renders into
exactly that. One encoder between two spellings of one structure; a second one
would drift from it. MetricGatherer hands out a Gatherer, not the *Registry, so
a reader cannot become a registrant.
Tests drive the whole path: install the provider, record through the ordinary
OTel API, push, and assert what arrives — including that a ZERO gauge survives
the trip with its labels, because hanzo_service_up == 0 is a real negative
answer and a translation that dropped it would turn a down service into a
missing one. Every family shape is pinned, a failing store costs one skipped
push, and shutdown is idempotent.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
typed_wire_test.go asserted a flat 200 for an unparseable body. That was the old
contract — the receiver answered 200 to everything — so with the status code now
describing DELIVERY it read 503 (no egress configured under test) and failed CI.
The fact the test actually exists to pin is that a malformed payload is RECORDED
rather than REFUSED: a 4xx would make Alertmanager retry it forever. That is now
asserted directly, with an egress installed so delivery is not the variable
under test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v0.5.0 deleted the key-derivation and DEK-wrapping API (DeriveKey,
DeriveChildKey, NewDEK, WrapDEK, UnwrapDEK, PrincipalType, PrincipalAAD,
WithPrincipalKey). cloud stopped calling it in 4cb56f6b; commerce and
tasks were the two things left in the build graph that still did, and
both have now moved to cek + namespace:
commerce v1.49.45 per-org store keyed by cek.DeriveKey, placed by
namespace.Path; DEK sidecar, rewrap and the
plaintext->encrypted migration deleted
tasks v1.52.9 shards opened by cek.Open under the shared
orgs/<org>[/projects/<p>]/<ns>.db layout; dek.go
and tasksd upgrade deleted
cek goes v0.2.2 -> v0.2.1, which is a REQUIREMENT of v0.5.0, not a
preference: v0.2.2 added sidecar.go to read legacy wrapped-DEK databases,
and it is written against the very symbols v0.5.0 removed, so v0.2.2 does
not compile against v0.5.0 at all. Its exported API is identical to
v0.2.1 — the delta is that legacy read path plus its tests — and reading
databases written under the old scheme is the compatibility this
migration exists to remove.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.5.49 carried the request-target fix in ONE of five relay bodies. The other
four — infra, metrics, rules/alerts, integrations — still handed the embedded
runtime a request with RequestURI empty, so 94 of 367 routes answered 404 in this
binary while the standalone served them 200. That is the configuration cloud runs
in production, so the whole of infra, metrics, rules/alerts, integrations,
cloud_integrations, downtime_schedules, route_policies, metric_reduction_rules
and gateway ingestion_keys was unreachable through the unified door.
v1.5.50 collapses the four copies into the one relay and adds the reachability
census that measures it — registered and reachable are now the same set.
It also stops a restart from locking every principal out of o11y (localauthz now
rehydrates its tuples from the durable role rows), and stops a schedule-less
maintenance window from panicking inside encoding/json — a panic that in this
binary would take iam, commerce and ai down with it, and that only became
reachable now that the downtime routes are.
plugin/o11y is unchanged by the bump, verified by regenerating: the fix is in how
a relayed request is shaped, not in which routes exist.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A control that switches itself off has to be visible from OUTSIDE the process,
and "visible" has to mean a test can drive a tenant into the state and read it
back. The probe published `strained` and nothing held it: a constant zero would
have passed every assertion here.
Now one organisation is pushed past its own aggregate ceiling — where it starts
forgetting its least-recently-active subjects, and every velocity feature for
those subjects reads as inactive, which is exactly what a quiet subject looks
like — and the probe is read back for a non-zero count.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A figure in a comment is a claim. One scope at both its ceilings is 4.0 MiB of
the 128 MiB budget — 3% — not the vague fraction that was written there.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
resolve could spend the whole process on one authenticated request. Two
amplifiers composed. A key had a count bound and no BYTE bound, and the domain
matcher split a host into labels and re-joined every tail, so an L-label host
allocated a copy of each of its L suffixes: one 8 KB dotted key materialised
16.8 MB and 100 of them 1.7 GB, measured over the router. And `sets` had no
bound and no dedupe, so naming one set N times ran N times the answers.
- maxKey bounds one key in bytes at every door it crosses — looked up,
written, removed — and REFUSES rather than truncating, because a shortened
key is a different key.
- a domain suffix is now a slice of the host, not a join of its labels: the
same answers in O(L) headers over one backing array.
- a call may name each published set once, and a set named twice is consulted
once.
The override write had the same hole from the other side: maxOverrides bounded
rows and nothing bounded a row, so 10,000 entries x 11 sets was gigabytes of
attacker-chosen bytes on the one volume every other organisation's store lives
on. The same maxKey closes it; an over-long note is refused rather than trimmed;
and what one organisation may occupy on that volume is now a figure the code
computes and a test pins, so raising rows, a key, a note or the catalog is an act
with its consequence next to it.
An override is a record, so it is now shipped before it is acknowledged, the way
apps/research and apps/books do it — this deployment is one replica with a
recreate rollout, and an unshipped write is a control an operator believes is in
force and is not.
prune spared ONE version: the call site passed the current version for both of
the statement's two placeholders, deleting the rows behind every citation taken
in the window before a refresh. What a take supersedes is now decided by
sweepOld over what the plane held before it, and proved against a warehouse
rather than against the text of the statement.
The publisher's end of the same amplifier is closed with the same door. A take
is refused whole if it carries a member longer than maxKey — one no lookup could
ever reach, so it is only weight in the warehouse, in every hydrate and in the
snapshot every request reads — or more members than a published set holds: swing
measures GROWTH against the version a take replaces, and a first take has nothing
to measure against, which after a cold start is every take. maxBody comes down to
six times the largest source in the catalog (measured: 2.6 MB), because the parse
allocates before any later gate can look at what it made. A publisher's redirect
must keep the two properties its origin already had, TLS and a destination
outside this network: this process runs in the cluster, where "wherever the
publisher says" reaches the pod network and the metadata address.
Also: a take whose size swings past 4x is refused and the previous version
stands (force is the operator lever); the disposable list is refused whole if it
names a mailbox provider, which is the one-row attack the size gate cannot see
on the one unpinned source; an attest receipt with no version or no designations
is recorded as a refusal instead of a current, fresh list; the plane sweeps at
cold start instead of refusing every set for six hours after a deploy; the one
cross-fleet aggregation states its own memory and time budget; every source
states a typed redistribution Basis from a closed vocabulary, on the wire, so
the licence position is an audit rather than a sentence; refresh reads the ONE
SuperAdmin predicate rather than restating it, and an admin of their own org is
refused there, because this route writes the baseline every org reads; and the
bridge sits on this app's own leaf, not on the /v1/ml parent two other products
answer under.
65 tests green under -race (one skipped: it dials the real publishers). Every fix
has a regression test that fails when that fix alone is reverted: 24 mutants, 24
killed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/ml/reference publishes ten sets — disposable email domains, hosting and
Tor address ranges, crawler user-agent patterns, delegated autonomous system
numbers, card-scheme prefixes, browsers the fleet sees everywhere, and the
freshness of the designation lists the screening engine holds. Six typed ops.
The unit of version and freshness is the SOURCE, not the set, so one publisher's
outage neither blocks the others' updates nor silently shrinks the set. A
version IS the content digest of the sorted entries, which makes a re-take of an
unchanged publisher a no-op that says so, and a half-landed version resumable
from its cursor without depending on it — the primary key already deduplicates.
Two planes, two stores. The baseline tables carry no tenant column, so a
cross-tenant write is unrepresentable rather than refused; a tenant's own
allow and deny entries live in that organisation's own store. Resolution is
override then baseline, both through one candidate function.
The baseline carries only published data under terms we hold — every source
states its licence — and aggregates above a k-anonymity floor no single
organisation can reach. Sources we may not redistribute are declared seams that
refuse, because an absent set and an unlicensed one look identical from outside.
A set that never loaded refuses rather than answering "not listed", and a stale
set answers and says so: every answer carries the version, its as-of, its age
and whether it is past the bound.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Mutation testing found three assertions that survive the defect they name, which
means they were not holding anything:
THE FEATURES GATE was proved only by its METER. Pricing the gate at zero screens
left every assertion green — an op that bills the right number while admitting a
caller with no balance, which is the control off with the books looking perfect.
It is now proved as a bracket, the shape the search gate already used: a cent
under the measured cost refuses, exactly the cost admits.
THE SEARCH METER was proved on a run that RAN TO COMPLETION, where "what it was
admitted for" and "what it did" are the same number and no assertion can tell
them apart. It is now proved on a run a rollout cuts short — the case that
matters, since this binary deploys at one replica with the old pod stopped first,
and the case where metering at accept charges in full for a grid that never ran.
The arena check that needs a completed run is its own test now.
THE FOLD MARK's second step was invisible to every behavioural test, because
replayable applies the window a SECOND time in memory where the nanosecond still
exists, so the re-read row is dropped downstream and nothing upstream can see it.
It is asserted at the seam where it is real instead: a mark must still exclude
the bucket it names AFTER a round trip through tsLiteral, which is second-grained.
Also: the snapshot interval had no test at all — only the count watermark did.
sweep takes its schedule as a parameter, so newPlane states it at the call site
and the loop is measurable without a thirty-second wait. A model that learns a
little and then goes quiet never reaches the watermark, and that is the case the
interval exists for.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The scan is under the lock and the sort is not, so the two numbers that matter
are the report at a full table and an observation while it runs. Measured
rather than argued: 844us for the whole report over 4,096 callers, 313ns to
observe one request.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three findings, one root cause: the sensor and the enforcement were keyed on
the raw Authorization value, validated or not — a string the caller picks. A
held verdict was walked out of five times in five by editing that header; the
lane with no tenant could be filled on demand; and a caller could open one
table entry per request.
A CALLER IS A FACT WE STATED. Signal now carries two fingerprints of the same
credential under different trust: Cred, set only when the identity boundary
VALIDATED it, which is the only thing that can key a caller; and Presented,
whatever the request carried, counted only as spread because a wall of invalid
credentials from one address IS the stuffing signature. callerKey cannot see
the second one. Both are built in ONE place (observation), pinned by a test.
The scorer is asked about the same thing the sensor holds, so a refused caller
cannot be re-judged as somebody else by editing a header.
NOTHING IS REMOVED TO MAKE ROOM FOR SOMETHING ELSE. The reclaim pass that
dropped the oldest half is deleted, not guarded: a table reclaims keys that are
DEAD (unseen for a window, under no live verdict) and REFUSES what does not
fit. So a flood cannot erase a neighbour's counts, cannot release a held
verdict, and cannot overrun the ceiling — the previous rule skipped pinned keys
and then admitted anyway, which is how 25,000 held callers lived in a table
that published 20,000.
THE BOUND IS IN BYTES. A cap on the NUMBER of keys is not a bound when the
values behind them are not bounded, so every string that can enter an entry is
clamped at the door, every entry has a published worst-case size, and every
admission charges it against ONE process budget. Count x size IS the byte
bound; a test fills a table with worst-case values, measures it, and fails if
the published numbers understate it. The per-scope ceilings stay as the
FAIRNESS bound, so the anonymous lane cannot take the room the tenants need.
A BOUND THAT BINDS SAYS SO. Strain is graded — clear, full, refuse, blind —
carried on the observation so an unmeasured caller is not screened as a brand
new one, reported on the scope's own view, and announced once per rise rather
than once per request. The lane that has no tenant is readable at last: it is
named by the empty scope (?org=), which cannot collide with any tenant.
WHAT THE SENSOR CANNOT SEE IT DOES NOT INVENT. A request with no validated
credential and no client address has no identity; keying it under the empty
address would file the whole internet in one row, read as the worst stuffing
run ever recorded, and let one verdict refuse everybody. It is counted as
traffic, named blind, and nothing is held against it. This is the live shape
today: the balancer in front of the ingress is TCP with no PROXY protocol, so
no client address reaches this process at all.
THE LANE IS DERIVED FROM THE COUNTS, so it is computed inside the observation
that produced them. Passing it in meant stating it before those counts existed,
and every request ever counted landed in "unknown".
ABSENCE IS NOT SILENCE. The fail policy turns on two facts, the same rule
hanzoai/iam applies at its own gate: a privileged grant waits for a scorer that
is THERE and did not answer, and proceeds when there is no scorer at all — or
when every slot is held and nothing has come back for a stall, which is a
deadlocked scorer, not a queue. Without that second fact one hung goroutine
403s every armed org's key store until the pod restarts.
AN UNANSWERED SCREEN IS NOT A SALE. Only a scored verdict is billed; the count
is split into answered and unanswered so a scorer that has gone dark is a
number on the org's own report. A fact we do not have is ABSENT from the
signals, never empty — an empty string is a value a scorer can group by.
A SCOPE KEY NAMES THE ROUTE. canonicalService and the rate limiter's
exemptions read the router's path, not the spelling: "/V1/AI/chat" reached the
"/v1/ai/chat" handler while producing a different rate bucket and a different
spend-cap axis.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/ml/labels is the answer key the model plane cannot build for itself: what
turned out to be fraud, who said so, and when they could first have said it.
Chargeoff, dispute, case, refund, review and the below-the-line sample all file
into one append-only record per tenant — its own encrypted SQLite file, no org
column, so a cross-tenant read is not forbidden but inexpressible — with a
derived ClickHouse copy for joining at training scale.
Seven typed zip ops, so the REST route, the OpenAPI operation, the MCP tool, the
CLI command and every SDK method are projections of one declaration: mlLabel,
mlLabels, mlResolveLabels, mlLabelCoverage, mlLabelVocabulary, mlDisposeLabels,
mlHoldLabels.
THREE PROPERTIES, EACH THE POINT OF THE PLANE, NONE OF WHICH THE FIRST CUT HAD.
DURABILITY. Nothing in the package called OrgStore.Sync, so the only ship was
CloseAll on a graceful shutdown. cloud deploys strategy Recreate at one replica:
an ungraceful termination lost every acknowledged record since process start,
and the successor hydrated the older durable snapshot OVER the local file — an
acknowledged compliance record was not merely at risk, it was overwritten by an
older copy of the tenant's own history. state.ship is now the ship-before-ack
step every write path calls before it answers, and an unacked ship fails the
request rather than acknowledging a divergent local copy. That covers BOTH
shapes: a replica that never held the lease (ErrNotOwner) and one deposed
between the write and the ship, whose fenced Put is refused at a stale round
with no error at all. The two sibling durable planes hold the same contract
(apps/research shipFor, apps/books shipLedger).
DELIVERY. The cursor was the pair (wrote, id) over a write clock truncated to
the second and a content digest — an order the writer never took. A write that
commits after a concurrent delivery has read, whose digest sorts lower inside
the same second, was already behind the mark: never mirrored, unreachable by any
retry, and pending() answered zero because it asked the same predicate. A hole
in the answer key reads as an honest customer. The cursor is now the store's own
AUTOINCREMENT position, allocated inside the insert on the single connection
every statement for a tenant runs on (sqlpool.Single), so cursor order is commit
order by construction.
LEAKAGE. `seen` is whatever the caller sent, bounded only by At <= Seen <=
now+skew, and nothing tied it to any fact the server observed — a dispute filed
today with seen == at was knowable a year before the record existed, and a
backtest standing two days after the event resolved it. Fact.Knowable is derived
server-side as the later of `seen` and the clock at the write, it is the only
time visible() and stronger() read, and it is a column in the derived copy so
the warehouse applies the same predicate the record plane does. A live pipeline
is unaffected: Wrote is within minutes of Seen and the derivation changes
nothing.
Also: the coverage window now ends where maturity begins, so the gate on
training no longer answers zero on its own defaults; Group returns the whole
matured cohort so `matured` counts what matured and `unlabelled` says why
`judged` is low; a litigation hold is a fact about the record with its own op
that can also release one, instead of a digest-excluded flag silently dropped on
any record that already existed; the warehouse partitions by month like every
other table in the fleet rather than by tenant, whose cardinality is the
customer count; and resolve answers one row per distinct event, so an event
named twice is no longer its own conflict.
Every one of those carries a mutant in scripts/mutate.py that reverts it and a
test that goes RED when it does: 14 rows, 14 killed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An org name is unique within an issuer and not across issuers: acme on
hanzo.id and acme on zoo.ngo are two unrelated businesses. A per-org SQLite
file never confuses them, because DataDir is per deployment and the brand is
the directory the file is in. A COLUMN in the shared columnar warehouse has no
such directory — org = 'acme' there is a predicate over both — so every row a
brand-shared table carries has to be keyed on <brand>/<org> and every read of
it has to bind the same form.
Qualify joins SanitizeOrg and OrgNamespace as the third org-naming door and the
only one for a shared plane. Qualified is derived from Qualify rather than
restated, so a change to the shape cannot leave a validator behind. Tenant
carries no JSON tags and has no constructor but Qualify, so no In struct can
decode one and no caller can assert a tenant for itself.
The brand half comes from Deps.Brand, never a header: a caller that can choose
its brand has chosen which tenant space its org lands in.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/o11y calls o11y.Mount(a) unconditionally, which registers the module's 353
typed operations. plugin/o11y's committed projections were last written the day
BEFORE that call landed, and nothing regenerated them: mcp.json held 12 tools and
openapi.json 20 paths / 32 operations.
So the fleet MCP door at POST /v1/mcp offered agents twelve o11y tools — the
reviews queue and one summary — while the process behind it served three hundred
and sixty-five. The observability surface was absent from /v1/openapi.json and
from every SDK generated off it. Regenerating with the app's own `describe`
target gives 365 tools and 289 paths / 388 operations, and takes the fleet total
from 932 to 1285.
apps/o11y/zipdoc_gen.go was stale in the same window: it still carried
POST /obs/event/claim, a route that no longer exists, and the pre-rewrite prose
for POST /v1/o11y/alerts/:receiver, which now states that delivery is
synchronous. Both come back from source.
This is what `make -f mk/fleet.mk surface-check` exists to catch, and it did not,
because the gate has been failing earlier in its own loop on unrelated apps — a
red gate stops reporting anything new. Recorded rather than papered over: the
regeneration is verified idempotent (a second run reproduces it byte for byte),
and the remaining blockers are in other apps.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The daily recompute wrote hanzo.risk_baseline whether or not anything read it,
and nothing did — a warehouse cost paid every day and collected on never, which
is the same "declared but unwired" defect as a reader with no writer, inverted.
The catalogue gains a NETWORK lens beside its model and surface lenses: the
published bands over the same window, carrying no tenant and the same for every
caller, so an organisation's own surface has something to be read against. One
table, one reader, so the disclosure argument has one place to hold.
Two lenses can fail in one response — an unreachable warehouse fails both — so
the gap accumulates instead of reporting one outage as the other's.
The recording warehouse files baseline rows under the TABLE and not under a
tenant, because that table has no tenant to file them under. Keying them by the
bound org meant keying them by the window's start, which a test had to guess to
the second in order to see its own fixture.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
30497e64 capped the buildkitd volume because it is the one that grew. The
artifact job's /w and the smoke job's /data were left as `emptyDir: {}`, and
that is the same defect with a different volume name — which is why this states
the rule as a table rather than adding two more one-off assertions.
An uncapped emptyDir is charged to the NODE's ephemeral storage. It therefore
does not fail the pod that fills it: it fills the runner's rootfs, trips
DiskPressure, and the kubelet answers by evicting the pod's NEIGHBOURS. A job
can stay inside every resource limit it declares and still take down every other
job on the node, so the bound has to be on the VOLUME and not only on the
container — three of eight runners under DiskPressure, 70 finished build Jobs
still holding their pods, and a release Evicted mid-flight for "node was low on
resource: ephemeral-storage".
/w is not scratch: it is the clone, HOME, GOPATH, the npm cache and every
produced binary for every platform, for every recipe entry in turn, in one
volume — the same order of magnitude as the build cache, so the same half-a-node
bound. /data is genuinely small (one 120s boot test against a fresh SQLite) and
is capped anyway, because the failure it prevents is an image that loops writing
on boot, and that one is the neighbours' problem rather than its own.
The three consts sit together: "how much node disk may a job take" is one
question with three answers, not three unrelated numbers, and a reader who finds
one now finds all of them. TestBuildCacheIsCapped is folded into the table — it
was exactly the table's `build` row, and a rule with two gates is a rule that can
be half-updated. Its incident evidence moves into the table's doc. Proven by
uncapping /w: the artifact row fails with the volume, the wanted bound and what
it actually found.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both were lost the same way — a conflict resolved by taking one side whole,
where the correct answer was on both sides.
THE PLANE SINK STOPPED COUNTING. 30497e64's predecessor put
cloud.ObserveRows at datastoreSink.Insert, after Send, because that is the one
choke point every plane row this process writes passes through, and because
"event.span went to zero" is the sentence nobody could say for four and a half
months. fix/o11y-convergence deleted event_ingest.go — correctly; it was a
writer that had never once reached a table — and carried datastoreSink forward
into planesink.go from a branch point 32 commits behind, so the counter went
with the dead file. The merge was a modify/delete, git kept the deletion, and
nothing failed: metrics_plane.go still seeds event.span at zero, so the series
exists and reads zero, which is indistinguishable from the outage it was added
to detect. Restored at the same place, after Send, on the surviving sink.
THE NEWEST PLANE TYPE WAS OUTSIDE THE GATE. bf48977a dropped ObsClaimIn and
ObsClaimed from TestNoPlaneTypeCarriesAnUnencodableKind because the op was
retired, which is right, but fix/upstream-status-and-spend-alerts-rpc had added
plane.ScopeRules in the same window and it was never added to the walk. That
test exists because the failure is a property of the KIND — the next map added
to any of these types is another 24h door-shut-while-green — so a plane type the
walk does not reach is the one place the next one will appear. ScopeRules is
walked rather than ScopeRule: the walk descends a slice of structs, and the
reply is the only way the row type ever crosses.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The merge brought two implementations of the same rule together. main landed
mcpAlias (34554a19) five hours after feat/mcp-one-door was opened, both answering
the bare /mcp with a 308 onto /v1/mcp. Keeping both is not belt-and-braces: the
route wins over the terminal handler, so webui/mcp.go's FrameworkMCPPath branch
becomes unreachable code that its own tests still exercise in isolation — a rule
that passes its test and never runs.
mcpAlias goes, and the terminal handler stays, because only one of them is
self-scoping. app.All("/mcp") claims the path unconditionally, so the same call
inside cloud.Serve would hijack a PLUGIN's own zip door — a plugin serves MCP at
exactly FrameworkMCPPath, and an alias there redirects the door onto itself.
webui's handler is terminal: it sees only paths no route claimed, so the plugin
matches its real route and never reaches it, and the host, which moved its door
to MCPPath, does. Same behaviour on the host, no footgun on the child.
It is also the superset. mcpAlias answered /mcp only; mcpDoor also answers a
non-POST /v1/mcp with 405 + Allow: POST (Streamable HTTP's "no SSE stream" —
where a 404 says the door is absent) and gives both replies a JSON body, so a
caller reading bytes instead of following the hop still never gets HTML. And the
address is now one value, manifest.MCPPath, where main still had it as a literal
in cmd/cloud and a second literal in manifest/mcp_test.go.
main's test survives intact and gets stronger: app308 no longer calls a helper,
so it now proves the COMPOSED host redirects. Neutering mcpDoor reproduces the
original defect exactly — POST /mcp = 405, GET /mcp = 200 text/html, 3576 bytes,
the same byte count 34554a19 recorded from production.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
(cherry picked from commit c4c3f40986725d1e99b6050c09de2aa8dfb63d48)
The kind-walk gained ObsClaimIn/ObsClaimed when it was written; the
convergence lane then retired obs_event_claim along with the dead
LLM-obs write path it fed, so the walk referenced types that no longer
exist. Dropped from the list and the header note rewritten to say why:
the property this test guards is a property of the KIND, which is
exactly why it walks kinds instead of a list of types, and it outlives
any individual op.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The floor is a monotone history of what the surface published, per product; the
merge left it at main's totals while the woven document carries risk's ten. The
ratchet raises it from the regenerated document rather than from either side's
guess.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The rename to Listen reached every caller but this one, and the release has not
built since: the image builds each plugin main as its own binary, so it stopped at
plugin/tools/main.go:20:18: undefined: cloud.Serve
The call shape was already right — only the verb was left behind.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The learning ops move off /v1/ml, which a live product already owns: the
Kubernetes model-SERVING plane (InferenceServices, /v1/ml/models/{name}/predict)
is on it and has customers. /v1/ml/models would otherwise have meant "models you
serve" and "models that learn" at once. Nine typed ops move to /v1/risk/*, the
serving routes are untouched, and there is no alias — nothing shipped under the
old prefix to be compatible with.
A CAP ON A COUNT OF CALLER-SIZED VALUES IS NOT A BOUND. Every per-tenant ceiling
here was a COUNT multiplied by something a caller chose, so the published figures
understated the real cost by whatever the caller picked. maxField bounds, in
bytes, every string an observation carries; the ceilings are derived from it; and
the published figures are now MEASURED against a real worst case rather than
recomputed from the same formula.
A partly-applied bucket is no longer a state the fold can be in. ctx is consulted
at bucket boundaries only, so an interrupted fold resumes exactly where it
stopped instead of re-teaching what it already applied. The watermark steps by a
whole second because that is the finest value tsLiteral can bind — a nanosecond
mark was truncated back onto the bucket it meant to exclude.
The fold marks only what the surface can HOLD. horizon() is one function: the
rollup stops rollLag behind the present, so the fold reads and marks there too.
It used to mark `now`, which claimed buckets the rollup had not written yet — the
next rollup wrote them and the fold never came back, so every cycle dropped that
organisation's most recent history while the model reported itself warm.
A gap re-arms. Only a fold that SUCCEEDED is a fold that happened, so a warehouse
blip on a tenant's first touch no longer costs it the moat for the life of its
residency.
The recording warehouse now honours the bound window. A fake that ignored it made
every window assertion in this package a stub, and fixtures derive their bucket
stamps from horizon() rather than from `now` — a row stamped `now` is a row that
could not exist.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The one-shot seed of hanzo.usage_rollup_daily could never run. EnsureUsageRollup
creates the incremental view on the FIRST leaderboard or activity read; the view
starts capturing on the next ledger insert; and the handler then asked "does the
rollup have any rows at all?" — which is true forever after. Every non-forced
POST /v1/usage/rollup/backfill answered 409, and the only way past it was
?force=true, which the code itself documents as double-counting. So pre-view
history was never laid down.
Measured on the live datastore: the rollup holds 530 of hanzo.cloud_usage's
19,792 requests (2.7%) and 567 of its 266,088 cost cents (0.2%) — everything
from 2026-07-01 to 2026-07-27 is missing, plus most of 07-28, because that is
when the view was created. Every leaderboard and activity read is served from
the rollup, so all of them under-report by ~97%. Nothing errored: an MV is an
insert trigger, and history it never saw produces no row and no complaint.
The guard was asking the wrong question. It must count only the days the seed
would WRITE — `WHERE day < toDate(?)` — not the whole table. Then a rollup that
the live view is already filling forward does not block a seed of the days
behind it, while re-seeding a day already covered still refuses.
The two ranges also have to be the same set of days to be comparable: the seed
selects the ledger by `timestamp`, the guard counts the rollup by `day`. A
mid-day cutoff seeds a PARTIAL day that the view may also hold, and once both
land in a SummingMergeTree no count can separate them. rollupCutoff snaps the
bound to UTC midnight — the rollup's own grain — in one place, used by the seed
and the guard alike.
The bound stays one-sided: this seeds "everything before the view existed", once.
Widening the cutoff later is a real re-seed of covered days and is refused;
?force=true remains the deliberate override.
Tests pin the regression directly — a rollup in exactly the state the live view
leaves it (rows from its creation day onward, nothing before) must accept a seed
of the earlier days, and the guard must be the range query. Reverting the guard
to the whole-table count fails it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
MCP clients are configured with a HOST and reach for the door at /mcp. Nothing
claimed that path, so it fell through to the console catch-all and answered
twice-wrong on api.hanzo.ai:
POST /mcp -> 405 method not allowed (the SPA route is GET-only)
GET /mcp -> 200 text/html, 3576 B (the console shell)
Both read like a server that is up, which is how this survived: the host
answers, the door never opens. The 404 on mcp.hanzo.ai is the milder half of
the same bug — a 404 at least says no.
308 and not 301/302: only the permanent-redirect pair preserves the method and
the body, and MCP is a JSON-RPC POST. A 302 would arrive at /v1/mcp as a
bodiless GET — the same dead end, one hop further along.
It is an ALIAS, not a second handler. /v1/mcp stays the only place MCP is
served, so the tool list, the auth path and the transport cannot drift between
two doors. Registered before webui.Mount so it wins the path, and extracted as
mcpAlias() alongside spec()/health() so it is reachable from a test.
The test is the defect: neutering mcpAlias reproduces 405 and the 3576-byte
shell — the same byte count TestAnAbsentPrefixBeatsTheConsoleCatchAll already
records for an unclaimed path. Its third case asserts the target is real, which
caught that zip installs no door at all when the registry is empty (installMCP),
so an alias could otherwise have pointed at nothing and still gone green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The buildkitd worker cache and snapshots are the whole of a build's working set
and the only thing in that pod that grows without bound. An emptyDir with no
sizeLimit is charged to the NODE's ephemeral storage, so one overrun fills the
node rootfs and trips DiskPressure — and the kubelet answers that by evicting
every pod on the node, not the build that caused it.
Three of eight runners sat under DiskPressure with 70 finished build Jobs still
holding their pods and emptyDirs, and a release was Evicted mid-flight for
"node was low on resource: ephemeral-storage". A cap turns that into this pod
overrunning and being retried.
The same fix exists in infra/k8s/hanzo-build/image-build-job.yaml, which is a
hand-applied template deliberately kept out of every kustomization — no build
this cluster runs comes from it. buildJobSpec is what launches them, so the cap
has to be here to be real.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Serve and Listen were two names for one act. zip's App already calls it Listen —
`app.Listen(zapAddr, httpAddr)` — and this function's whole job is to build that
app and hand it its addresses, so calling it Serve made the entry point disagree
with the thing it enters. One verb, all the way down: a plugin's main says
cloud.Listen, cloud says app.Listen, and nothing has to be translated in a
reader's head on the way through.
117 composition roots move with it. ServePlane is untouched — it names a
different act (bind one app's own socket for the internal plane), and collapsing
it into this would be the opposite of the point.
Also fixes apps/iam's TestMain, which had gone red on every store test:
credz.Boot's last resort is cek.EnsureDevKey, and that DECLINES on a codec-linked
build by design — a build that can really encrypt must be handed a real key, not
invent one. So the throwaway goes in through the same door a deployment uses, and
only when nothing else supplied one. Six failures back to the one pre-existing
ratchet (iam serves 97 untyped ops against a budget of 88).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cek v0.2.1 derives every key from the master and the namespace, which is the
right steady state and cannot read a single file this fleet already has. Every
one of them was born under the wrapped-DEK scheme: a DEK from crypto/rand,
wrapped beside the database in a .dek sidecar. Random key material is not
reproducible by derivation, so a release on v0.2.1 meets those pages the way
SQLCipher meets a wrong key —
page 1: sqlcipher: wrong key or corrupted page
— across 211 org directories and every subsystem in each: kms, agents, finance,
code, treasury, crm, the audit log. That is not a migration, it is the data
becoming unreadable at the moment of a deploy.
v0.2.2 reads the sidecar when there is one and derives when there is not, so the
file says which scheme it belongs to. It writes nothing, so this changes no byte
on disk and stays reversible; new stores are still born derived and sidecar-free,
and the set the compat path governs only shrinks.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The port landed in three pieces across two commits and this is the piece that
matters at runtime: cmd/reach/main.go was committed and openapi/reach.py was
deleted, but .hanzo/workflows/cicd.yml still said
run: python3 openapi/reach.py openapi.yaml ...
so the reach job would have failed on a file that is no longer in the tree. LLM.md
likewise still named the Python path.
Both were staged and verified staged twice, and twice the commit carried only the
Go file. Whatever dropped them, the lesson is the one already written down: read
the committed value back, do not trust that `git add` stuck.
committed cicd.yml before: python3 openapi/reach.py (script absent -> fail)
committed cicd.yml after: go run ./cmd/reach
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6965c345 removed openapi/reach.py but did not carry cmd/reach/main.go, the
workflow change or the LLM.md line with it: a multi-path `git add` did not stage
the new file, and I committed without reading back what was staged. main was left
with the gate script deleted and nothing calling its replacement.
This is the other three quarters of that change:
cmd/reach/main.go the port
.hanzo/workflows/cicd.yml go run ./cmd/reach, plus setup-go and the
private-module env the containment job uses
LLM.md the gate table names the Go tool now
Verified before and after: go build ./cmd/reach and go vet pass, and the tool
agrees with the deleted Python exactly against production — 872 addresses,
646 literal, 226 parameterised, 0 dark.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Search, Crawl, Vector, Console and Managed Services had their prices stated
nowhere but hanzo.ai's components. That is how the same product came to be
advertised twice at different prices: one tab priced Hanzo Vector at Starter $29 /
Growth $299, another at Free $0 / Pro $25 / Business $99, both live, one saying 1M
vectors cost $29/mo and the other saying 1M vectors was free.
plans v1.4.11 carries the rate cards (every number copied from what those pages
already advertise — nothing offered changes); pricing v1.4.7 serves them. This is
the route.
DISPLAY rate cards: no entitlement or limit fields, so nothing can bill off them.
What a product COSTS and what a plan GRANTS are different questions, and conflating
them is how a catalog row becomes a self-serve mint.
Spec regenerated the same way as datastore — zipdoc first (a route with no
generated doc registers but does NOT project, which is why the first describe
produced nothing), then the subset in a Linux userspace, then the weave. The weave
also picks up three /v1/o11y/* paths that the committed o11y subset already had and
the golden did not: pre-existing drift, corrected rather than carried. floor.json
1397->1399 paths, 1966->1968 operations.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud is a Go service: 2284 .go files, no Python in the shipped image. The ONLY
thing that wanted a Python runtime was this gate, and on Ubuntu 24.04 that turned
into a dependency the runner refuses to install:
error: externally-managed-environment (PEP 668)
- To install Python packages system-wide, try apt install python3-xyz
So a check about ROUTING kept failing over package management — first as a
JSONDecodeError from a json.load "fallback" that could never parse YAML, then as
a refused pip, then as an apt step existing only to feed one script. Each fix was
smaller than the question of why it was Python at all.
gopkg.in/yaml.v3 is ALREADY a direct dependency, so the Go version provisions
nothing: no pip, no apt, no runner-image dependency, and the toolchain is the one
the release already builds with.
Behaviour preserved exactly, including the load-bearing parts:
- wildcard keys dropped, not filled
- the literal/parameterised asymmetry in isDark: ANY 404 condemns a literal
address, while a parameterised one is dark only on the exact router-miss body
(the rule that caught an edge worker whose 404 carried JSON)
- checkInstrument() still runs at measure time, not in a test file nobody runs
- the ratchet may still only SHRINK, and a healed line still fails the release
Verified by running BOTH against production, same arguments, minutes apart:
python3 openapi/reach.py 872 addresses (646 literal, 226 parameterised), 0 dark
go run ./cmd/reach 872 addresses (646 literal, 226 parameterised), 0 dark
reach.py is deleted rather than kept alongside: two implementations of one gate
is the drift this gate exists to catch.
The reach job gains setup-go and the same private-module env the containment job
uses, since github.com/hanzoai/* must resolve direct+authenticated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Records what the door work established, so the next 4xx here is diagnosed by
SHAPE rather than re-guessed:
{"status":"error","msg":…} gate() itself (fall-through route)
{"status":403,"error":…} a typed op — relay re-wrapped it
{"status":404,"error":"404 page not found"} the runtime's web provider, no path
The three probes are the control: mountHealth dispatches them and never calls
relay, so "probes 200, everything else 404" is a lost path, never auth.
Names every exempt op and the argument for it — the rule being this gate's own
purpose read backwards: it exists because the runtime trusts X-Org-Id as
gateway-minted, so an op whose gate reads no tenant from the request has nothing
for a forged tenant to reach. Exemption is not authorization; each still faces
its own credential one layer in.
States the SPA question rather than leaving it open: /v1/o11y is the API and only
the API — the module names all 367 routes so an unconverted one 404s instead of
falling through a wildcard, so /v1/o11y/ is a 404 and every answer on the prefix
is JSON. The console is o11y-site at o11y.hanzo.ai / obs.hanzo.ai behind
admin-guard.
AND FLAGS SOMETHING THE DOOR WORK UNCOVERED BUT DID NOT FIX. The section headed
"the o11y pin is BLOCKED at v1.5.34 — do not bump it alone" listed three
in-handler forwards that a bump would break. The pin is now v1.5.49 and those
forwards were never updated: "/api/sessions" is registered nowhere in the pinned
module (the list is /v1/o11y/llm/sessions) and "/api/v3" survives only in
parser_test.go. sessions.go and query.go therefore name routes the runtime no
longer serves. Both are org-gated, so neither is reachable anonymously and
neither appeared in the door probes. They need their own pass — and the fix is
to forward to the current spellings, not to re-pin.
Also notes why editing r.URL.Path is not a redirect against the embed:
adaptor.FiberApp routes on RequestURI, so a handler that rewrites URL.Path and
leaves RequestURI alone is routed by the original path.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
api.hanzo.ai/v1/o11y/version answered 403 and, once the gate was fixed, 404.
Two independent defects stacked at one seam; the second was invisible while the
first refused the request before it could reach anything.
1. THE GATE (already fixed here by 89ae5449, shipped now). Its exempt list named
/v1/o11y/api/v1/health and three /api/v2 siblings — the namespace hanzoai/o11y
stopped rewriting onto at v1.5.37. Four names, zero routes, so every public op
was refused. The answer is o11y.Anonymous now: the fact lives beside the routes.
2. THE PATH (o11y v1.5.49, picked up here). relay handed the runtime a request
built by http.NewRequest — a CLIENT request, whose RequestURI is empty by
design. The embedded runtime is adaptor.FiberApp, which copies RequestURI into
fasthttp verbatim, so the path was erased and every typed op fell through to
the console web provider's http.NotFound. That is the
404 {"status":404,"error":"404 page not found"} the door answered on /version,
/health, /global/config and /users/me the moment the gate stopped refusing
them. Only livez/healthz/readyz worked, because mountHealth dispatches those
itself and never calls relay.
WHY NEITHER WAS CAUGHT. red_forge_test.go calls gate() directly against a
backend that answers 200 to anything — it proves the predicate and cannot see
the chain, and it passed throughout. door_test.go exercises the REAL MountO11y
route table against a runtime that routes: anonymous /version and /health must
return the RUNTIME's bytes, tenant reads must still be refused with the DOOR's
own reason (not the runtime's 401, which would mean the request got through),
and the four dead /api/v1|v2 names must NOT be exempt. Verified by reverting
each fix in turn: with the dead list restored it fails with the exact production
body, 403 {"status":403,"error":"no validated principal"}.
THE SPA QUESTION, ANSWERED: /v1/o11y is the API, and only the API. There is no
catch-all under it — hanzoai/o11y v1.5.48 named all 367 routes precisely so an
unconverted route 404s instead of silently falling through — so /v1/o11y/ is a
404 and every answer on the prefix is JSON. The console is o11y-site at
o11y.hanzo.ai / obs.hanzo.ai behind admin-guard, which 302s a browser to
hanzo.id PKCE and 401s a machine. One door per concern: api.hanzo.ai serves the
operations, the site serves the app.
Pre-existing and NOT from this change: TestUntypedRoutesKeepTheirWire fails on
pristine origin/main (unparseable alert receipt = 503, want 200) — the alert
lane's, untouched here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
My previous commit added `python3 -m pip install pyyaml`, which the runner
rejects:
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install python3-xyz
The image is Ubuntu 24.04, where a system-wide pip install is blocked by PEP 668.
The error names its own remedy, so this uses it: python3-yaml IS PyYAML, and
apt-get is already how this fleet installs system packages (hanzoai/engine does
the same, unsudoed — the job runs as root).
Also drops `--quiet`, which was a mistake of mine: it hid the reason the step
failed, so the log showed only "exitcode 1" with no message. The step now prints
the installed version, so "PyYAML is present" is evidence rather than assumption.
The rest of the pipeline was unaffected — gate, cicd, containment, image and
rollout all passed on the previous run; reach was the first failing job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reach job failed with
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
which reads as a corrupt spec. It is not. reach.py did
try: import yaml # "the forge runner has it"
except ImportError: doc = json.load(open(spec))
The runner does NOT have it, and that fallback could never work: the script is
always called on openapi.yaml, and json.load on YAML fails exactly like the above.
So a missing package presented as a broken document, and the check it guards has
not actually run.
PyYAML is now installed in the job and REQUIRED by the script, which exits saying
so instead of mis-parsing.
That makes the check run — and it immediately found real drift: 5 published
addresses that api.hanzo.ai does not route. Three were the o11y probes:
/v1/o11y/api/v2/healthz 404
/v1/o11y/healthz 403 <- routed, auth-gated
Our o11y fork moved off /api/v2; openapi.yaml still advertised the old shape, so
the document promised three addresses that answer 404. Renamed to the paths that
exist. (Also what the house rule says: /v1/, never an /api/ prefix.)
reach.py before 5 dark
reach.py after 2 dark
⚠️ Still dark, and NOT fixed here because both need a product decision rather
than a transcription fix — either the route should exist or the document should
stop publishing it:
GET /v1/ai/mcp/tools 404
GET /v1/pricing/datastore 404 (/v1/pricing itself is 200)
The ratchet file openapi/unreachable.txt is empty, i.e. the standing policy is
zero dark addresses, so neither was silently allowlisted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every org store is answering 500 right now:
cloud: OrgDB pragma "PRAGMA busy_timeout=5000": file is not a database
Measured across two INDEPENDENT subsystems — /v1/agents/sessions and
/v1/tracker/projects. Keys are derived per (namespace, subsystem), so two
subsystems failing identically is not two corrupt files; it is one wrong MASTER.
cek is careful about the absent case: DeriveKey returns ErrNoMaster when no
master is installed, so "unset" fails loudly. What it cannot distinguish is a
master that is present and WRONG — and this package can hand it one. Branch 3 of
resolve() mints a random master when nothing is configured, which is exactly
right for a laptop and catastrophic for a deployment that has lost its KMS: every
file was encrypted under the key being replaced, so each opens as "file is not a
database" while the data sits intact and unreadable.
It is the same mistake the branch above already refuses for a launched child —
"Inventing a second key here is worse than not starting, because it succeeds" —
and the reasoning does not depend on there being a token. It depends on whether
anything preceded this process.
So ask that, and ask it of the disk: resolve() already takes dataDir. A random
master is honest over an EMPTY directory and refused over one already holding
databases, with an error naming the remedy rather than 500s thirty frames later.
No notion of "production" is required, which is why this is checkable at all.
An unreadable subtree is reported, never skipped: the point is to refuse when we
cannot be SURE the directory is empty. A missing directory is not an error — it
is the clearest possible "nothing preceded this process".
This is a fail-closed guard, not a diagnosis of the live incident: whether the
running deployment took this branch needs its boot log (credz reports posture and
`bootFrom`), which needs cluster access this session does not have. The guard is
correct either way.
The five failing tests in this package fail identically without this change —
macOS has no socket peer-credential lookup ("peer credentials unavailable on this
platform"). The two added here pass.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
github.com/hanzoai/plans embeds blockchain.json, which /v1/pricing/blockchain
serves. v1.4.9 adds the Indexer API (hanzo.ai advertises it; the catalog did not
publish it). v1.4.10 restores the Wallet API tiers the page actually shows —
Free 0/5K, Growth $49/250K, Scale $249/2.5M. The catalog's row had been
byte-identical to the NFT one, which is what an uncorrected copy-paste looks like.
This lands BEFORE hanzo.ai is wired to the live catalog, deliberately: the live
read replaces the page's fallback, so shipping the site first would have repriced
the Wallet API downward on the public pricing page for as long as cloud lagged.
The same module also carries the Go/Dev/Pro/Max ladder, so /v1/pricing/subscriptions
stops disagreeing with /v1/billing/plans about what is sold.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The embedded IAM is not the identity authority. It mounts, serves routes and
answers OIDC discovery — and its store is EMPTY:
[iam] "seed: read init_data.json: no such file" -> "iam seed skipped"
[iam] "iam embedded in-process"
/var/lib/cloud/iam/global.db is 32KB of schema, and the cloud pod's own JWKS
answers with ZERO keys where iam.hanzo.svc answers with nine. Every org, user,
key and signing cert lives in the standalone IAM's PVC — a different file, in a
different pod.
So resolving a key against that store resolves nothing, in every process: the iam
child reads the empty store directly, and every other child asks the iam child
over the plane and gets the same nothing back. API-key authentication has been
broken since v1.801.359. The HTTP call this replaced went to iam.hanzo.svc, which
is where the keys actually are.
The direction was right and the premise was not. Reading identity in-process
requires the identity to BE in the process, and consolidating it is a DATA
migration — the standalone IAM's store into the embedded one — not a transport
change. Until that happens the resolver talks to the service that holds the data.
What stays, because it was never about the transport: the plane-topology lesson
(each app is its own composition root and calls cloud.Serve, so a package-global
seam is nil in every process but its owner), apps/iam's TestMain booting the
data-plane key the way the binary does, and the cloud.Request gate no longer
walking .claude/worktrees.
What I should have done: I flagged twice that I could not positively test a real
key and shipped anyway. A resolver change is not verified by the absence of 401s
in a low-traffic window.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`campaignWhere` wrote its own predicate — org, time bounds, `utm_campaign` — and
no `signal` at all. On five tables that was harmless, because naming
`event.event` selected product events by construction. On the ONE fact table the
signal is a PREDICATE, and a predicate can be forgotten: this one was, so
`uniqExact(distinct_id)` and `sum(revenue)` ranged over every log line, span and
error in the window as well as the acts.
It is latent only because nothing carries a `utm_campaign` yet. The day a
campaign ships it stops being latent, and the first number anyone sees is wrong.
So the campaign read composes `eventsWhere` instead of restating it. Which rows
and which sort is ONE decision and now has ONE spelling; the campaign and the
optional variant are what this read adds. Arg order follows the composition —
[org, signal, start, end], then campaign, then variant — and everything
user-derived is still bound, never interpolated.
The regression test pins the composition rather than the string: the built
predicate must have `eventsWhere`'s output as a prefix, with its args leading.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzo.ai's Infrastructure tab renders a Hanzo Datastore section that fetches
/v1/pricing/datastore. Cloud never registered that route, so it 404d and the
component fell back to "Live pricing is temporarily unavailable. Contact sales for
current rates." Permanently — which is the worst version of that message: it reads
as a passing outage, so nobody investigates, and it tells a visitor to wait for
something that was never coming.
The data existed the whole time. datastore.json sat in hanzoai/pricing unembedded
and unserved; pricing v1.4.6 embeds it and the goja bundle serves it whole (it is
a rate card, not a plans list, so it is not wrapped like iam/base/paas). This adds
the injection and the route.
Verified the payload against the CONSUMER's own validator (hanzo.ai
DatastorePricing isDatastorePricing): 3 tiers, usage.storage.pricePerGBMonth and
usage.egress.public_internet.pricePerGB numeric, included[] an array.
Spec regenerated, which is the part that is easy to skip: the weave gate does NOT
catch a route added without regenerating its subset — both artifacts are derived,
so they agree with each other while both are wrong, and plugin/ingress lost eight
paths that way with `make test` green. `make describe` cannot run on macOS at all
(hanzoai/sqlite's isRAMBacked is `return false` off Linux, so the encrypted
catalog store refuses to open), so the pricing app was cross-compiled and
described in a Linux userspace, then woven. Golden diff is one added path and none
removed; floor.json ratchets 1396→1397 paths, 1965→1966 operations.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/ai fetched the IAM signing cert in a package init() and panicked when it
could not, so identity was a boot-ordered hard dependency of every route in the
binary. That took api.hanzo.ai down twice: 2026-07-27, when a manifest typo killed
IAM's pods, and 2026-08-02, when an IAM build answered no /healthz and its Service
held zero endpoints for 25 minutes. Neither time could the process recover once
IAM came back — the dial had already happened, before any subsystem mounted.
v1.832.10 resolves the cert lazily, retries a failure at most once per 5s, and
refuses requests with 503 at the door while it is unresolved. Never 401, which
blames the caller for our outage, and never served unauthenticated — the
fail-open that once left the service Running and answering 401 to everything.
decimal and money come along as ai's own transitive patches.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Arrival and delivery were the same word. This endpoint logged PAGE-DELIVERED,
answered 200 `ok`, and then failed to send — because the egress ran detached in
a goroutine the response never waited for, so the answer was written before the
send was attempted and could not possibly have been about it. Alertmanager
recorded Notify success over a silent pager.
Split the two facts. ALERT-RECEIVED is arrival; ALERT-DELIVERED /
ALERT-UNDELIVERED is egress. Delivery is synchronous under an 8s budget and THE
STATUS CODE REPORTS DELIVERY: nothing carried it means 503, so Alertmanager
retries and counts it. No egress configured is a failure, not a no-op — that
silent early return is how a whole deployment could page nobody forever.
Egress is a chain: the org's KMS-custodied Slack bot token first, then a plain
POST to CLOUD_ALERTS_WEBHOOK_URL, which needs no integrations peer, no org and
no connected workspace — it works in exactly the state that silenced the first.
A failure is recorded even when a later egress succeeds, so a broken egress
cannot hide behind a working one.
Then the reason no rule could have caught any of this: the data plane emitted
nothing. 904 metric names in the store and not one described a row moving.
hanzo_http_requests_total existed in code with ZERO callers, so /v1/event's 5xx
rate was unalertable while the Sentry envelope 503'd for a day. Wire it into the
one middleware that already has path, status and validated org in hand, and add
the missing measurements: rows landed per warehouse table, ingest admission
outcomes, plane sockets bound, cron fires against the engine's own promise,
memory against GOMEMLIMIT, and alert egress itself.
Every counter is SEEDED AT ZERO at boot. A counter first touched by its first
event has no series until that event, so an ingest path that never runs looks
exactly like one that was never built — which is how span ingest stayed dead for
four and a half months without a single rule being able to notice. Seeding is
what makes silence measurable.
Tests pin the new contract: undeliverable answers 503, no egress is a failure,
the fallback carries it while the Slack failure is still recorded, the chain
stops at the first success, delivery is synchronous, and PAGE-DELIVERED is gone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
api.hanzo.ai/v1/o11y/version answered 403 {"status":403,"error":"no validated
principal"}. So did /health, the three probes, sign-in and the shared-dashboard
reads. o11y.hanzo.ai answered all of them 200. The unified door refused exactly
what the standalone door served, which is why o11y still needed a door of its own.
The refusal was never o11y's. gate() here kept its own list of the paths that
need no principal, and the list named /v1/o11y/api/v1/health, /api/v2/healthz,
/api/v2/readyz and /api/v2/livez — the INTERNAL namespace hanzoai/o11y rewrote
onto until v1.5.37 deleted the rewrite. Four names, zero routes. The exemption
matched nothing, so the gate refused everything behind it, including every op
whose runtime gate is OpenAccess. LLM.md had listed the drift as a known trap
since v1.5.37; the pin reached v1.5.46 with the list still in place.
Two shapes at the door were the diagnosis, not the status codes. /v1/o11y/livez
returned the gate's own {"status":"error","msg":...} and /v1/o11y/version the
zip envelope {"status":403,"error":...} — the second is a typed op relaying to
the same gate and re-raising its {msg} through relay's refusal(). Same gate, two
hops. /v1/o11y/nonexistent 404s, which proves the wildcard is gone and the typed
table is live.
o11y.Anonymous is the fix: the fact lives beside the routes it describes, and
this file asks. The rule is this gate's own purpose read backwards — the gate
exists because the runtime trusts X-Org-Id as gateway-minted, so an op whose own
gate reads no tenant from the request has nothing for a forged tenant to reach,
and gating it can only remove an answer. The exempt set is the runtime's
OpenAccess routes plus the two public-dashboard reads it gates with
CheckWithoutClaims. Every read of a tenant's telemetry stays gated here AND at
the runtime: one rule enforced twice, not two rules. Exemption is not
authorization — the DSN key, the service-account key, the share's scope and the
session cookie are each still the op's own admission test, one layer in.
Also gone: isHealthPath, isErrorIngestPath and isSentryIngestPath, all three
copies of a route table one repo away from the routes; the five /v1/o11y/*
wildcard descriptions, whose address no longer exists; and the three probe
descriptions' dead /api/v2 spelling. The eleven escape hatches hanzoai/o11y
registers by hand now carry prose here, because Describe lives in this module
and o11y deliberately no longer imports it.
The published document still lacks o11y's 353 typed ops: regenerating
plugin/o11y/openapi.json now succeeds (389 operations, up from 34) but the weave
fails closed on six schema names that mean different things in o11y and in
ingress/books/content/analytics/plugins. Named in LLM.md; it is a rename in
hanzoai/o11y, not a change here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The fleet's agent door is composed, not written — zip projects every typed op
into a tool and the host serves the union — so there was never an aggregation
problem to solve and this adds no registry. What was missing is the ANSWER to
"what is on the door", which no process could give and no test asserted; an
inventory nobody can read is how a door serving nothing passes for a healthy
one, which is this estate's recurring defect and not a hypothetical.
ai owns it, in cloud/apps/ai rather than hanzoai/ai: hanzoai/ai imports
github.com/hanzoai/cloud, so the aggregate cannot live there without a cycle,
and a mounted module cannot see its siblings anyway. This package is already
the sibling that imports both.
One typed op, GET /v1/ai/mcp/tools (operationId aiMCPTools), reporting three
numbers that are three different questions:
published — every tool this BUILD can serve, from the same plugin/<app>/mcp.json
bytes the host hands zip at Load. 930 across 116 subsystems.
served — what THIS PROCESS's door actually composed, read from the live
composition (App.Plugins + App.MCPTools). The number that can be
zero while published is nine hundred.
local — the part of served this process registered itself.
Being a typed op, it is itself a tool on the door it describes — ai's first, and
the reason the fleet's total moves 929 -> 930. Nothing here declares that; zip
projects it, zipdoc lifts its prose, and the weave carries its schema.
The gate is the op's own — principal.ValidatedFrom, the bit cloud.Bridge parks —
so it holds identically over REST and MCP and fails closed off the HTTP path.
mcp_test.go proves it in both directions on the real op: anonymous over HTTP is
403 "sign in to read this deployment's MCP tool surface", anonymous over MCP is
isError with that same string, and validated over each answers the same body.
The tests compose the REAL fleet — every manifest row, its real catalogue, as
remote mounts so nothing is spawned — and read BODIES: 930 tools on one door,
no name with two owners, o11y unmounted leaves exactly its 12 behind by name and
by count, and the inventory equals what the door actually serves at three
different compositions. Both halves were mutation-checked: unmounting iam from
the composition turns the count test red (831 vs 930), and making the inventory
report the build instead of the door turns three assertions red.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The plane was five tables with an identical envelope — event.event, event.error,
event.log, event.span, event.metric. That is CONSISTENT and not UNIFIED: no
cross-signal question could be asked without a five-way UNION ALL, and a new
product meant a new table name, which is how a namespace grows a _v2.
Occurrences now land in ONE table, event.fact, with `signal` saying which sort
of thing happened. `signal` also leads the partition key, so per-signal
retention still drops whole parts, `WHERE signal='log'` still prunes to a fifth
of them (measured: Parts 1/5), and the Replacing collapse stays per-signal so an
id collision across two signals cannot delete a row.
event -> act a namespace cannot also be a member of itself
group -> issue it identifies an issue, and `group` is one letter from `groups`
site -> origin the place of the fault; `site` already names a published Site
level + severity_text + severity_number -> severity UInt8, the OTLP number
alone, with the word a read-time function of it
group0..group4 -> groups Map(type, key)
clip is the worked example of why a signal is a value and not a place: session
replay costs one signal value and two columns (object, bytes), reuses
duration/session_id/url, and adds no name to the namespace. The blob it indexes
stays in object storage — it is wrong for a bus message and wrong for a
warehouse row.
Write path: `table` is where a signal lands and how its row is built, and there
are exactly two because there are two grains — an occurrence happened, a sample
was measured. Five writers share one column list, one args builder and one
INSERT string, so they cannot drift into five shapes of one envelope.
landableSignals still DERIVES from writers, so the door and the store are one
edit. sample keeps no writer until event.sample has its name (o11y migration
0003): a door that accepts what the sink cannot land is a 200 that means
discarded.
Read path: `scope(org, signal)` is the mandatory leading predicate of every read
— the tenant first because it leads the sort key, the signal second because it
leads the partition key, both bound. Two lenses were hand-rolling `org = ?` and
would have silently read every signal as a product event.
The fact struct is flat because the table is flat; four optional bodies were the
right shape when each had its own table. Sparsity is not a cost — measured on
the live event.log (798,375 rows), an unpopulated column costs 515 bytes for the
whole table.
DDL is unchanged here and stays unchanged here: hanzoai/o11y owns it
(deploy/datastore/migrations/0002_event_fact.sql), which the pre-existing scan
over this package's sources enforces.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Six instances of one defect shipped: the billing surface dropped compound words
from its route names (payment-methods → methods, payment-config → settings,
auto-recharge → recharge, spend-alerts → alerts, credit-grants → credits, and
four more), cloud and commerce both converged, every published document agreed
with itself, every server test passed — and not one client followed. The
clients' own tests passed too, because a test that pins the URL its client sends
is a test of nothing: it agrees with the client while the client is wrong.
router_test.go asks whether every path an app PUBLISHES reaches that app, and it
is complete on that side. Nobody was asking the other question. manifest/
clients_test.go asks it: for every address a first-party client requests, is it
ROUTED (some row in manifest.Apps hands it to an app) and is it SERVED (that app
registers it, per the subset that app's own binary projects from its own
router). Neither side is derived from the other — `calls` is hand-authored
source, the router is built through the real zip.Load, and the subsets are
forced back to source by `make test`.
A second gate refuses the retired NAMES outright, in `calls` and in Apps. The
general one can only catch a compound name once somebody writes it down, and the
whole failure was that nobody did.
Proven by reverting each half. Point the clients back at the old names and four
failures name the client, the address and its replacement. Drop /v1/billing/
alerts from the commerce row and four more name the address, the app the fleet
delivers it to instead ("ai", the tail /v1 remainder) and the fact that it does
not register it. Both green with the fix.
Writing the ledger turned up three breaks nothing was watching, all the same
shape:
- GET /v1/billing/portal/methods. cloud's billing app serves the saved-card
list by proxying here, and NOTHING in the fleet serves it — so the list is
empty however many cards were vaulted. It was wrong twice: it also asked for
the retired /v1/billing/portal/payment-methods, which does not exist in the
commerce module this binary links. The name is fixed here. The missing owner
is NOT, and is recorded rather than guessed: PortalPaymentMethods keys
tenancy on a customerId query param, so the console chain cannot serve its
service-token caller and the S2S chain would let any authenticated browser
read another tenant's cards by naming their customerId. Choosing that gate
is an IDOR decision, not a route.
- DELETE/PATCH /v1/billing/methods/{id}. The billing app owns the prefix and
registers only the collection, so the live edge answers 405: a customer can
add a card and never remove one. Reaching the handlers means proxying an
address this app owns, which is the self-dispatch loop that produced the
depth-8 502s on top-up. Recorded.
- POST /v1/billing/payment. Never served by anything, ever. Recorded.
GET /v1/billing/credits WAS fixable and is fixed: the handler sat in the
vendored module with no app registering it, so the Credits tab read a 404 the
client swallowed into an empty list. Registered co-resident beside its siblings,
named on the commerce row, and now in the woven spec — so the SDKs get it too.
Minting is untouched and stays on the mint-gated POST /v1/billing/credit.
Regenerating the commerce subset also surfaced the same defect in the prose
layer, which is why `make -C apps/commerce describe` was already red on main:
POST /v1/store/token had no description because the description that exists was
keyed to /v1/store/storefront-token — an address the router has never
registered. Prose written for a route nobody serves is the identical mistake one
layer up. Re-keyed, so the gate is green again.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two money paths read "I could not reach the authority" as "nothing is priced",
and a priced tool was therefore served for free in both. The tests that caught it
say so plainly: a priced tool was SERVED with no payment rail, and a priced tool
was SERVED while its price was unknowable.
The shared mistake is that cloud.ErrNoPeer does not mean what both call sites
took it to mean. It is documented as "this app is not part of the deployment",
but reach() also returns it when it simply cannot get through and no router is
present to contradict it — and a killed peer leaves a socket file behind that
refuses every connection, so a marketplace that DIED is indistinguishable from a
fleet that never had one. Both layers were entitled by that ambiguity to guess,
and both guessed in the direction that gives the shop away.
x402 held an explicit ErrNoPeer-means-unpriced exception, four lines below the
rule it contradicts ("the PRICE is unknown ⇒ error, never free"). It is gone: an
unreached price is unknown, and unknown is never zero. The exception bought
nothing anyway — manifest/apps.go lists marketplace beside x402, so a rail
deployed without its table is a misconfiguration, and one that refuses loudly
costs less than one that quietly sells everything for nothing.
tools mapped the same error to ErrChargerUnset and then asked the LOCAL row
whether it was priced — but in a split fleet that row never carries a price,
because prices are marketplace listings living in another process. Silence read
as free. It now asks the table's owner instead: for sale means refuse, as the
outage it is; not priced, or no table in this deployment either, keeps the old
answer, so a deployment holding neither rail nor table still lets a tool's own
declaration have the last word.
The refusal rides the existing 424 rather than a 402 — no challenge can be issued
for a rail that is gone, and a client cannot satisfy one. A free tool is refused
too while the table is unreachable: the table is the only thing that can say
"free", and this is the direction that costs nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Components states which value each seam contributes — Register a *Schema, the
typed fold zip's JSON verbatim — and Fold is documented as a merge that must not
replace what Register already named. Project then undid both: nouns.into
marshalled every schema to canonical JSON and unmarshalled it back into a bare
any, so every *Schema that reached a document through Project came out the other
side as map[string]any. It ran unconditionally, with or without relays, which
means it was not a relay behaviour at all — it flattened the whole components
block of any document that merely passed through.
The canonical bytes were never the problem; they are the right way to compare two
claims to one name, since two trees that serialize identically describe the same
type. They stay the comparison. What changes is that the claimant's own value is
kept beside them and is what reaches the document, so the bytes decide conflicts
and the value decides the type. Published JSON is unchanged — the bytes came from
marshalling that value in the first place.
into can no longer fail, so it stops pretending it can.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nine admin tests asserted that money reconciles, and all nine failed with the
same signature: credits correct, spend $0, and every per-org revenue read folded
into "partial: one or more org revenue reads failed". Read as a product bug that
is every money panel in admin showing zero against real balances.
It was the fixtures. The billing route became /usage/rollup, and the rename
reached the client and the four other callers but not the four fake-commerce
stubs here, which still matched on HasSuffix(path, "/usage-rollup"). So Spend
requested a path the stub 404s, and the aggregators reported the outage they were
correctly told about. /balance was untouched, which is exactly why credits read
right and spend read nothing.
The stubs each document themselves as mimicking the live contract — one says
"Verified against live commerce" — while routing on a path that contract does not
serve. commerce registers Get("/usage/rollup") under /v1/billing; that is the
server these fakes stand in for.
No assertion moved. Every expected value is what it was; six strings changed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
basedb existed for exactly one reason: cek.Open chose a path and did not create
its parent, and on the pure-Go codec that does not fail the open — the database
is written back at CLOSE, so a missing directory loses every write of the session
after the caller has been told it had a store. cek v0.2.1 does the MkdirAll
itself, pinned by a round-trip test, so the wrapper is now a synonym. It is
deleted and its 70 call sites name cek directly.
The larger duplication was underneath it. Sixty stores each set
SetMaxOpenConns(1) and re-applied busy_timeout, journal_mode=WAL and
foreign_keys=ON by hand. hanzoai/sqlite already applies those, on EVERY
connection — which the hand-rolled db.Exec did not: a one-shot Exec lands on
whichever connection happens to serve it and is gone the moment that connection
is recycled. So the fifty-two copies were not merely a fact restated fifty-two
times, they were the weaker of the two mechanisms shadowing the stronger one.
They say nothing now, and sqlpool_test.go asserts the driver still delivers each
default, so the deletion goes red in one place instead of rotting in fifty.
What is NOT already universal is the pool cap. sqlite sets it on the envelope
path and not on the live-libsqlcipher path, which is the one the shipped image
builds — so the cap is real, and it is stated once, in sqlpool.Single. The
package imports nothing but database/sql, so every store can reach it.
Two databases were outside all of this and are not any more. team keeps its
pragmas, because a dynamic journal_mode and foreign_keys=OFF are an override
rather than a restatement. git's ssh-key registry was a bare sql.Open on a
hand-joined path — no namespace, no key, and consequently the only store in the
binary written to disk in plaintext. It opens through cek like everything else.
hanzoai/sqlite stays at v0.4.0. v0.5.0 deletes the DEK/principal API
(PrincipalType, NewDEK, WrapDEK, UnwrapDEK, PrincipalAAD, DeriveKey) that
hanzoai/commerce and hanzoai/tasks still compile against, and no published
version of either has migrated, so taking it breaks the build. cek v0.2.1 does
not want it either — it requires v0.4.0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The plan reconcile assigned SKU and Metadata, which the catalog does not publish,
so every reconcile cleared a stored SKU and replaced the Metadata map with nil.
planEqual compared neither, so whether a value survived depended on which other
fields happened to differ — same data, same code, two outcomes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
limitsEqual compared Limits by ADDRESS (every field is a *int), so the reconciling
seed would have rewritten every plan row on every boot — right values, but
'corrected' never reaching zero, which is the signal that says the catalog has
converged rather than being fixed over and over.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two defects, each fixed at its one seam.
402/403 -> 500 (#141). zip renders 500 for any error that is not a
*zip.HTTPError, so a clean refusal reached the console as an internal
fault: commerce answers 402 out of funds, the gate answers 403, a plane
callee answers either — and the UI drew a DEAD CARD, with no code to
branch on. The status always existed (ErrInsufficientBalance MEANS 402;
zip's callFault carries a callee's status across the plane intact); what
was missing was a seam to carry it to the wire, so ~20 handlers
remembered cloud.Denied and every other one 500'd.
errmap.go decides it ONCE — refused() is the single classifier, read by
both the app's renderer and the money wire's denial() — and Serve installs
it. A refusal is 4xx ONLY: a 5xx is the absence of a decision, and reading
an unreachable-commerce 502 as a refusal would answer the money wire with
a transport message where "Billing temporarily unavailable" belongs. The
body stays zip's own {status, code, error}; this fills `code` rather than
inventing a second envelope.
spend-alerts 502 recursion (#146). ScopeRateLimit read its config with a
service-token GET /v1/billing/alerts through the commerce transport, which
dispatches in-process by publishing the WHOLE shared app — so the fetch
re-ran the entire edge chain, including ScopeRateLimit, whose cache is
filled only AFTER the fetch returns and was therefore still cold. It asked
again, and again, to the depth guard: 502, ~135 per half hour on the live
pod, with the ceiling failing open throughout. A split deploy recursed the
same way over the network.
commerce owns the rows, so commerce answers for them: a new plane op
(plane.FinanceScopeRules, apps/commerce/scoperules_rpc.go) over the ZAP
socket, which carries that app's ops and no edge chain — the recursion is
structurally absent, not bounded. metering.Client.ScopeRules and the
/v1/billing/alerts path are deleted, so there is one way. The op runs the
same whole-org query as commerce's cap verdict, so a rate ceiling and a
spend cap can no longer bind on different rows.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestCanonicalDoorFallsThroughWhenObsPeerIsAbsent was written around two worlds —
a peer that claims the body, and a peer that is absent so the product wire takes
it. Only one of them was ever reachable: the claim's sink wrote to tables that do
not exist, so it declined every body it was ever offered, and the claim is now
deleted outright. A test that distinguishes a state from itself pins nothing.
The assertion it carries is the one worth keeping, and the parallel lane already
fixed it — the body lands ZERO facts and must be told so (400 unroutable_events)
rather than receipted 200. That is the mechanism that hid a four-and-a-half-month
span outage: a green check beside a total loss. It now asserts BOTH halves,
because either alone is a lie in one direction — a 400 with facts landed is as
wrong as a 200 with none.
So this keeps the assertion, drops the world, and renames to what it actually
pins: an obs-shaped body is ORDINARY at this door, refused on its own merits, and
the product wire beside it is untouched.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A convergence audit read a deployed vintage and reported seven /v1-plane
drifts. Four were already closed on main; three were real, and the largest was
a write path that had never once reached a table.
apps/o11y/event_ingest.go inserted UNQUALIFIED `traces` / `observations` /
`scores`. The DSN it opened (O11Y_DATASTORE_DSN) names no database — hanzo-ds
sets Auth.Database from the URL path and applies no fallback — so those names
resolved to `default`, which is EMPTY on the live datastore. The file said as
much about itself ("ASSUMED SCHEMA", ungrounded against a producer that ships
dist-only) and the datastore's query log holds no such INSERT in its whole
retained window: not failures, never attempted.
It was also the third claimant on one concept. LLM observability is READ off
gen_ai spans in event.span by the o11y runtime, and the eval product owns the
grounded projections — hanzo.eval_traces / hanzo.eval_scores, with DDL
apps/eval/telemetry.go creates itself. So the sink is deleted rather than
qualified: pointing a duplicate at the right database would have made a path
nothing produces for look canonical.
Deleting it retires the plane op it existed to serve (obs_event_claim), which
means POST /v1/event — the fleet's busiest door — no longer makes a
synchronous cross-process round-trip per event to be told "not mine". The
answer the caller gets is unchanged: the claim could only ever decline.
planesink.go keeps the shared datastoreSink, beside its one remaining caller,
and states every table fully qualified. Two pins hold the shape it depends on:
tables name their database (the mistake above, made unrepeatable), and no
plane writer binds ingested_at — event.span measures retention AND partitioning
from that column and defaults it to now64(3), so a writer that names it hands
the wire control of when its own row expires. apps/analytics states the same
rule for its four warehouse writers; this is that pin for these two.
manifest/apps.go drops /v1/insights/e. A prefix there is a claim that the app
ANSWERS the path, and analytics retired that door — a claim it 404s kept the
retirement invisible in the one table that states what the fleet serves. The
four comments still describing it as live are corrected with it.
Verified against the live plane before touching anything: the JetStream stream
is EVENT (4839 messages, 5 consumers, durable on a retain-class volume) and
EVENTS does not exist, so the constant is right as written and a rename would
have orphaned the data or 503'd the plane.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET https://api.hanzo.ai/mcp returned 200 text/html — the console SPA. A client
that checks a status code reads that as a healthy MCP endpoint. POST there
returned 405 "method not allowed" in the console's own voice, which reads as a
door that exists and was called wrong. Neither was true: nothing was mounted at
that path at all.
Registration order is not what did it. The zap-proto/fiber fork matches by
ServeMux-1.22 specificity, so a static /mcp beats the console's /* catch-all no
matter which registered first — proven by the same app answering POST /mcp with
a real tools/list when the door IS there. The path simply had no route in this
process, and the terminal handler could not tell a machine door from a
client-side console route, so it served the shell.
Why there was no route: the address was written down twice and only once.
cmd/cloud moved zip's door to /v1/mcp with a literal; webui's apiPrefixes — the
list of paths the SPA must never answer — did not know the framework default had
been vacated. Two copies of one fact, and the second was missing.
So the address is now ONE value, manifest.MCPPath, read by the app that serves
the door and by the front door that must refuse to answer it with HTML. The gate
in manifest/mcp_test.go reads it too, instead of restating the literal a third
time — a guard that spells its own address can pass while the door has moved.
/v1/mcp is canonical: /v1/<thing> is the house rule, it is what production
already serves, and being under /v1/ is what puts it in the namespace where an
unmatched sibling is a real 404 instead of HTML. Not /v1/ai/mcp — the door
serves the union of every subsystem's build-time catalogue (929 tools across 116
apps), so filing it under one subsystem would either shrink it or misname it.
The terminal handler now owes a caller two honest answers, and neither serves
MCP:
/mcp -> 308 to /v1/mcp, JSON body. 308 keeps method and body, so a POSTed
initialize or tools/list reaches the real door instead of the
shell. A signpost, not a second surface — nothing here has a tool
list.
/v1/mcp -> 405 + Allow: POST for a non-POST. zip registers only the JSON-RPC
POST; MCP Streamable HTTP says a door with no SSE stream answers
405, and the 404 it used to give says the door is absent — the
same lie pointing the other way.
Both rules are self-scoping because the handler is TERMINAL: it sees only paths
no route claimed. A plugin serving its own door at zip's default — which is
where a host forwards a composed tools/call, and why cloud.Serve must keep it —
matches a real route and never reaches them.
And the API-namespace 404 now runs BEFORE the static-console gate. WHAT the path
is decides the answer, not what verb a browser would have used: a POST to an
unmatched /v1 path was answered 405, telling a client the endpoint exists and it
used the wrong method. 405 is a claim about a door; only a door may make it.
Proven against the built host binary, by body:
POST /v1/mcp -> 200 application/json, 42 tools, and an initialize handshake
returning protocolVersion 2025-06-18
POST -L /mcp -> follows the hop, id echoed, lands on the tool list
GET /mcp -> 308, Location: /v1/mcp, no HTML
GET /v1/mcp -> 405, Allow: POST
/, /settings, /mcp-servers, /auth/callback -> 200 text/html, the console shell
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.49.39 alone would have shipped a HALF-MIGRATED catalog. Every stored plan row
carries Managed=true, which commerce's seed sets on its own writes, and the seed
skipped any Managed row — so a boot would have created `go` and `dev` while
leaving `pro` at $20 and `developer`/`plus` on sale beside them.
v1.49.40 gives the seed a real discriminator (AdminEdited, set only by the admin
CRUD) so it can correct rows it wrote itself, and archives what the catalog
stopped publishing. Rolling this binary is now the whole cutover: no SuperAdmin
bearer, no script run against production, no manual step to forget.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/event/{project}/envelope has answered 503 "error ingest unavailable"
for 24h+ with zero successes, and every signal said the path was healthy: the
o11y process was up, /var/lib/cloud/run/o11y.sock was bound, and obs_error_post
was registered on it. The tell was in the access log, not the error — dur_ms=0.
The request was never leaving analytics.
ObsErrorIn.Headers was a map[string]string. zapenc carries scalars, strings,
byte slices, structs, pointers and slices, and REFUSES anything else at encode
rather than dropping it, precisely so a field can never silently fail to arrive:
zip: encode obs_error_post: ObsErrorIn.Headers:
zapenc: map cannot cross the plane; give it a type that can
So zip.Call failed on the caller's side of the socket, Ask returned (nil, err),
and the handler's `err != nil || out == nil` branch reshaped it into the generic
503 that made this look like an unreachable peer. It was an unencodable argument.
Its sibling op on the SAME socket is why this hid so well. ObsClaimIn is two
scalar fields, so obs_event_claim encoded fine and POST /v1/event stayed 200
throughout — the door looked half-open, which pointed every investigation at
routing and at the socket. Binding that socket (f43e2b10) was necessary and did
not fix this: two independent faults on one path, and the first one masked the
second until it was fixed.
Headers is now []Header, a slice of structs — the shape zapenc already carries,
one complete ZAP message per element. Not a workaround for the map; the map was
never a wire type here.
The tests are the two halves of the rule. TestObsErrorInCrossesThePlane makes a
REAL crossing over a real socket and echoes a header back, so a dropped header
fails loudly rather than passing as an empty map; it fails on the old shape with
the encode error above. TestNoPlaneTypeCarriesAnUnencodableKind walks field
KINDS across the plane's types, because the fault is a property of the kind and
the next map added to any of them is this same outage. Nothing in the suite
could have caught it before: the op's own tests call the handler directly and
never encode.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/event answered 200 {"accepted":0,"dropped":1} for EVERY wire shape it
publishes when nothing could be attributed. The anonymous projection refuses a
kind it cannot name (publicKinds stores pageviews and errors; a log, a span and
an exception envelope are none of those) and said so in a receipt field no
client parses and no probe reads. A caller whose key was absent, revoked or
mistyped lost 100% of what it sent with a green check beside it — the mechanism
that hid an 88% log loss, a four-and-a-half-month span outage and a day of
missing Sentry traffic.
The rule is now exactly "did anything land", decided ONCE in `answer` (event.go)
— the tail every lane already reached:
accepted > 0 -> 200, unchanged, INCLUDING the partial batch. A batch is
never failed whole for its worst element.
nothing sent -> 200. Dropping nothing is not losing anything.
nothing landed -> 4xx naming the one thing the caller can fix, in
HTTPError.Code: 401 ingest_key_required (no credential),
403 insufficient_capability (a guest token that RESOLVED —
it has a key, so telling it to get one is a false
instruction), 400 unroutable_events (full capability, and
the body still named nothing storable).
DNT stays 200: an honored opt-out is the one total drop that is not a failure.
The o11y claim goes through the same receipt — it is where the missing logs and
spans were headed, so exempting it would leave the defect in the lane it cost
the most. obs_door_test.go asserted `want 200 via the product wire` for a body
that landed ZERO facts; it now asserts the truth.
A nonzero drop is visible: hanzo_ingest_dropped_total{org,source,reason} (what
an ingest-drop alert reads) plus a warn line naming tenant and door.
Shape dispatch is untouched — isInsightsWire still runs BEFORE the canonical
object branch, pinned directly at the decoder.
25 existing tests encoded the 200-while-storing-nothing as the contract. Each
keeps its capability assertion and swaps only the status observable.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Red held this plane on three ship-blockers, all of which were the same shape:
a property the package claimed and did not have.
EVERY READ OF THE SOURCE IS NOW ADMITTED. `lineage` re-ran the exact census a
materialisation is charged 10c for — an exact distinct-count over up to 400 days
of one tenant's feature surface — with no gate, no meter and no bound of any
kind, on a GET any authenticated caller could loop in parallel against the one
stateful store every product on api.hanzo.ai shares. It is not fixed by adding
three checks to that op: `census` and `facts` now take a `scan`, a value with no
exported field that exists only where `admit` returned one, so an unpriced
unbounded warehouse read is not a thing this package can spell. admit is the one
door — it prices at the meter, takes the tenant's single slot and one of the
plane's eight, and lineage additionally runs under the plane's own deadline
rather than the caller's patience. A test reads the package's own AST and fails
if any function that reads the source takes no admission.
A JOB CAN RECORD THAT IT RAN OUT OF TIME. The write that ended a job shared the
context of the work it was reporting on, so the one case a refusal exists for —
the fifteen-minute wall expiring — could never be written: the driver refused
before sending anything and the version sat in `materializing` with an EMPTY
refusal forever. The work's context and the record's are now two, and `record`
opens its own, so they cannot be fused again.
ONE MINT. hanzo.risk_feature.org is written by the rollup's `qualify`, which
lower-cases the brand and requires a registered one, and was read here by a mint
that only trimmed and accepted anything. CLOUD_BRAND=Hanzo therefore had the
writer filing rows under `hanzo/acme` while this plane asked for `Hanzo/acme`
and got nothing, forever, with no error anywhere. The mint canonicalises and
requires a brand the registry vouches for; a test pins it byte-for-byte against
the writer's algorithm over every brand and spelling, and Mount refuses at boot
rather than 403-ing every request.
Also closed, from the same review:
the biller's own words A gate that cannot be ASKED fails with the peer's
transport detail, which zip renders as the body of a
500. Both priced ops now refuse through cloud.Denied
— the fleet's one 402/503 contract — and the reason
goes to the log.
version numbers Disposal dropped the register, so the counter reset:
a second `orders` reached v3 again with different
bytes and nothing recorded that the first v3 existed.
Disposal now drops the BYTES and marks the register,
`disposed` outranking `ready` in the engine's own
version column, so numbering is monotone across a
disposal and "prove you deleted it" is answered by a
record instead of by silence.
a falsifiable claim lineage certified a window re-derivable whenever the
source had not SHRUNK — but the source is fed by a
rollup running behind the events, so holding MORE is
the ordinary case and re-running the spec would not
reproduce the digest. Reproducible now means exact
agreement on all four recorded measurements.
one brand's org cloud trusts every white-label brand's issuer, so a
lux.id token reached a hanzo deployment and minted
`hanzo/<org>` for it. The identity boundary now
stamps the brand it VERIFIED from `iss`, and the
mint refuses when the two facts disagree.
what is per process The package claimed two processes over one store
answer identically. Reads, declarations and disposals
do; ADMISSION does not, and that is now stated with
the deployment contract it implies rather than
claimed.
And one the review missed: the branch already failed the fleet's own
TestRequestEscapeHatchIsPinned, because `who` reads the request for the BILLING
identity — the ledger, the validated project, the attribution — which is
deliberately a different value from the tenant. The reason is now on the record
in allowedRequestUses, where the next reader will find it.
Every fix carries a regression test, and each was mutation-tested: with the fix
reverted, its test fails.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A dataset is a VALUE, not a query. Storing a spec and re-running it is the
design that guarantees irreproducibility: the source is a SummingMergeTree
whose parts merge, its retention drops the tail, and the rollup behind it can
be re-run — so the same question asked twice is two different answers, and a
model that cites "the query" has cited nothing. Here a dataset is declared as
a version, materialised once, fingerprinted over the spec AND the rows, and
never rewritten. That is the only form in which an audit can be answered.
Seven typed zip ops under /v1/ml/datasets — declare, list, describe,
materialise, lineage, export, dispose. One declaration each, so the route, the
OpenAPI operation, the MCP tool, the CLI verb and every generated SDK method
come from the same place and cannot drift.
The four properties, and where each is enforced:
tenancy every statement opens `org = ?` with a tenant.Key, which has no
exported field, cannot be written as a literal outside its own
package and cannot be decoded from a request body. The key also
leads both tables' sort keys and both partition expressions, so
a per-tenant read is a prefix scan and a disposal cannot be
spelled across a tenant. A row that comes back belonging to
someone else is a REFUSAL, not a filter.
immutability `ready` is the greatest rank of the ReplacingMergeTree version
column, so no later write of any other stage displaces a
published version; the door refuses any transition out of a
terminal state. Two layers, engine and door.
determinism splits are temporal cuts and then grouped by SUBJECT — a random
split puts one device on both sides of the line and the model
memorises the entity. Membership under the row cap is a seeded
sample of the subject, recorded on the manifest, so a capped
dataset reproduces instead of being whichever rows came back
first. Coordinates are hashed as IEEE-754 bits, not as text.
expiry NO table TTL, deliberately: a table TTL is a fleet-wide clock no
tenant can hold longer. Disposal is the tenant's own DROP
PARTITION on (org, name).
The maturity horizon is what keeps a training set from knowing the future: a
chargeback lands 30 to 120 days after the transaction it condemns, so a row is
admitted only once it has aged past the horizon its version declares. Lineage
is MEASURED rather than recalled — the plane re-asks the source and says
plainly when the window can no longer be re-derived, because an admitted gap is
actionable and an unfalsifiable claim is not.
Bounded so a build cannot become a DoS: the window by the source's retention,
the horizon at a year, rows at 200k, names and versions per org, ONE
materialisation per tenant and eight in the process, a fifteen-minute job wall,
a bounded export page, and the priced act metered at the same gate the fleet
uses. Materialising answers 202 as soon as the attempt is on record; a client's
timeout is never a data plane's timeout.
Its own app rather than a leaf of another: it shares no state with a scorer, so
it restarts empty and answers identically — which a process pinned to one
replica for its in-memory forests cannot promise, and which is exactly what a
plane holding the record of what a model trained on must do.
apps/tenant is its own package because the key is not any one subsystem's: the
dataset plane, the scoring plane and the compliance plane must agree on it byte
for byte, or a dataset is keyed differently from the model fitted on it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
FOUR MORE COPIES OF THE ADDRESS RULE. apps/security, apps/legal and
apps/compliance each carried their own clientIP — left-most X-Forwarded-For,
falling back to X-Real-Ip — and apps/plugin put the raw header straight into an
audit record's SourceIP. Every one of them writes the value into a durable audit
row, and both headers are written by the party being audited: an address the
subject chose is not evidence. They read cloud.ClientIP now, which is the one
rule, so there is one place to be right and one place to review.
THE DOCUMENT. plugin/gateway/{openapi,mcp}.json re-describe the config plane from
the typed ops — mode's prose says what it now does, and the traffic view carries
the tenant's own saturation count — and openapi.yaml is the weave of the subsets,
which this branch had left behind: it added GET /v1/gateway/traffic without
regenerating the golden, so the published document was one operation short of the
routes and openapi/floor.json still said gateway had two. The floor RAISES
(2 → 3, paths 1396 → 1397), which is the direction a new route is meant to move
it. `make -f mk/fleet.mk openapi-weave` is green and idempotent.
The o11y plugin's own chain test pins what the new boundary buys: a forged
X-User-Id and X-User-IsAdmin do not survive it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Six defects, and the first two are the same defect twice: state a caller can
grow, held in one place, bounded once for everybody.
THE AGGREGATES ARE PER TENANT (ring.go). One process-wide velocity.Store with a
~9,225-key cap over the whole fleet backed eight of the nine model dimensions.
A key is <org, axis, subject> and the subject comes off the wire, so the busiest
organisation evicted the quietest one's keys — and a tenant reading zero for
eight dimensions scores as unremarkable and raises nothing. No error, no log, no
alert: a cross-tenant denial of service costing the attacker ordinary use of
their own account. The fix is the shape, not a bigger number: a ring set belongs
to one resident, newRings is the only constructor in the package, and the bound
is a per-tenant byte budget, so a tenant that outgrows it evicts its OWN
least-recently-active subject. The process ceiling is the product of two stated
numbers (64 x 8 MiB) rather than a hope.
THE RINGS ARE DURABLE, as a projection rather than as state. This binary rolls
one replica at a time with the old pod stopped first, so in-memory aggregates
were lost on every rollout — the same silent quiet, arriving on a schedule. They
are now rebuilt deterministically from the tenant's own record of what it
taught, bounded by age AND by count, because every tenant's shelf shares one
volume and a record bounded only by age is one tenant filling another's disk.
TIME IS BOUNDED IN BOTH DIRECTIONS. `at` was parsed and believed. The rings
track a leading edge, so one event stamped in the future moved it there and
every real event for that subject was then older than every window: one request,
and that subject's velocity features read as nothing for as long as the state
lived. Per-subject detector evasion, free, for any authenticated caller. Refused
at the wire door with a 400 and refused again in the rings, so no path can
poison the edge.
SCORE AND LEARN ARE GATED AND METERED. The only Gate/Meter call site was the
search. The two ops an abuser would actually call in a loop were free, unbounded
compute against per-tenant model state and a per-tenant disk write — a denial of
service and lost revenue at the same time, and which one it is depends only on
who finds it first. The billable unit is a SCREEN: one event judged against an
organisation's own model. Scoring is one, a batch is one per event, a search is
one per candidate per event, priced from the measured size of the run rather
than as a flat fee. Each tenant's in-flight calls are bounded too, per tenant,
so at the bound that organisation is told to slow down and nobody else notices.
THE MOAT IS WIRED. rollup() was the only writer of hanzo.risk_feature and had no
production caller, so the per-org feature surface was a table nothing wrote, the
warm read nothing, and the whole thing was prose. plane.roll is that caller: one
window per source plane per tenant under a durable per-tenant watermark, because
the surface is a SummingMergeTree and rolling a window twice does not replace
it, it doubles that organisation's history. Windows are aligned to the surface's
own five-minute grain — a window that ends mid-bucket counts every uniqExact
column once per partial insert — and every surface read rolls first, so a search
cannot answer "your history is empty" while the fold that would have filled it
is still running.
ONE ORGANISATION, ONE VOTE. The k-anonymity floor counted CONTRIBUTORS: twenty
five organisations satisfied it while one supplied a million of the million-and
-twenty-four values, and the published network median was then that
organisation's own median, republished under a name that says it is everyone's.
Each organisation is now reduced to one number — its own median over its own
subjects — before any quantile is taken, so every contributor's weight is
exactly 1/orgs, which the floor bounds at 4%.
Three more found while proving the above:
A FOLD IS ADMITTED, NOT ASSUMED. A fold rolls four source planes and reads a
window, and one goroutine per residency is unbounded fan-out at the warehouse.
Tickets bound the concurrency; a tenant that finds none is not marked folded,
so its next touch retries — deferred and reported, never dropped and silent.
A HISTORY IS FOLDED ONCE. The fold watermark travels in the snapshot's own
row, so a model restored from its own state does not read the same thirty days
again. Without it, a resident bound a busy fleet hits routinely meant the
masses counted how often we evicted a tenant rather than what its traffic did.
Both watermark-driven folds serialise per tenant, because idempotent-under-a
-watermark is only true one at a time.
A METER OWNS THE STRINGS IT RETAINS (metering.Usage.Clone). A Usage built in a
handler carries zero-copy views into the reused request arena, and MeterUsage
records on a background goroutine — so the debit could marshal the NEXT
request's bytes onto this caller's row, on a connection two tenants took turns
on. Not a crash: a wrong record. Fixed once where the retention happens rather
than in each of eleven callers.
Nine new tests, each named for the defect it refutes and each mutation-proven:
remove the fix and the test that names it fails. zipdoc -check green, the woven
document regenerates from source, and the suite is green under -race.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three ways a caller was deciding its own case, and one shared table every
tenant could evict from.
THE ADDRESS. ClientIP took the LEFT-MOST X-Forwarded-For entry — the one the
client writes — so one host could present a million clients: it defeated the
per-IP edge limit keyed on it, put a chosen address into an audit row, and fed
the sensor's address table without bound from an unauthenticated request. The
rule is now the peer first (a direct caller cannot lie about its socket) and,
for a trusted peer, the right-most chain entry that is not one of our own hops.
Our hops are a CIDR set (CLOUD_TRUSTED_PROXIES, private space by default), not
a hop count: a count is a promise about topology that nothing enforces. All
X-Forwarded-For header lines are read, since a second line hides the first.
THE LANE. principalValidated read c.Org() || c.User(), and both are headers:
X-Org-Id survives the boundary on the anonymous path by design, and a
hand-written plugin process may have no boundary at all. Two headers plus an
sk--shaped string that never validated moved a bad bot into the AGENT lane. The
boundary now states what it minted (principal.Mint, a request-local slot no
client can write) and the classifier reads that. No attestation ⇒ anonymous.
The gate's tenant comes from the same place, so a forged org cannot write into
another tenant's sensor state or read its posture.
THE PATH. Privileged compared strings.HasPrefix against the raw c.Path() while
fiber routes case-insensitively and ignores a trailing slash, so /V1/KMS/...
reached the key store and matched no grant prefix — the scorer's silence
ALLOWED what fail-closed exists to refuse. cloud.RoutePath is fiber's own
detection path, and every security comparison runs against it on segment
boundaries (which also stops /v1/iam/signupfoo matching /v1/iam/signup).
THE TABLE. edge.Traffic held one map for the whole fleet under a global cap,
so the org that filled it evicted whoever was quietest and that victim's abuse
controls went quiet with no error. State is now per tenant with a per-tenant
bound, reached only by indexing tenants[org]: a cross-tenant read or eviction
is unwritable, not merely refused, and a tenant at its ceiling degrades itself
and reports it (TrafficView.Saturated). One reclaim policy in table[V] —
unexported map, cap as a constructor argument, a live verdict never dropped.
ARMING. Mode was seeded from the platform row for every org, so the one PUT
that arms the anonymous lane armed every tenant silently. It reads the org's
own row now; the platform row governs the lane with no tenant and nothing else.
And it is no longer self-service: writing it takes SuperAdmin whichever row it
lands on, because the subject of an abuse control must not be able to switch
the control off — an org-admin credential is what a stolen key buys.
Also: asking is bounded (MaxScorerCalls in flight, RefusalBusy past it, so a
stuck scorer cannot become an out-of-memory), and the o11y plugin installs the
identity boundary it was reading headers without.
Every fix carries a regression test, and each was mutation-proven: revert the
fix, exactly that test goes red.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
When a held verdict lapsed, the caller went back to being judged only by the
local pattern — so an attacker refused on evidence the SENSOR cannot see got a
minute of enforcement and then walked. The scorer reads the org's own history:
prior accounts on a device, a spend curve, a chargeback. None of that leaves a
trace in a rolling minute of request counts, so waiting for the pattern to
re-trip waits forever.
A lapsed hold now forces the question again on the next request. Once the scorer
allows the caller, the record is released so it stops forcing a screen — the
hold record is exactly the interval between 'a verdict ended' and 'the scorer
said it is fine now'.
The test backdates the deadline rather than sleeping a millisecond, because a
millisecond hold is a race and a test that can flake is a test that will.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
credentialClass read the Authorization header directly, which is a SECOND answer
to a question the identity boundary already answers. callerToken is the one
resolution SanitizeIdentity and CallerBearer share — Authorization bearer, then
X-Authorization, then Basic, then a session cookie — so a client authenticating
with any of the others validated upstream and was then counted here as anonymous,
pooling its traffic under its ADDRESS instead of under itself. That is precisely
the caller the sensor exists to tell apart from its neighbours, and it was the
one it could not see.
Now it reads callerToken, with X-Api-Key after it: that header is not in the
boundary's precedence but several SDKs send it, and a caller the boundary could
not identify is still a caller the sensor must distinguish from the next one.
Three spellings of one key now fingerprint to one caller, which is asserted.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
edge.Store.Mode resolved an empty org to shadow, which meant the anonymous lane
could never be armed — there was no org to arm, so the one lane the gate exists
for was permanently a sensor. It now resolves to the PLATFORM row, the same shape
PerIPRPM already has and for the same reason: a request with no tenant at
evaluation time is governed by the platform scope. A SuperAdmin arms it by
targeting the reserved admin org, whose row IS that scope, so this is one
mechanism rather than a second one for the case that has no tenant.
Also: a liveness probe is never a grant and is never screened, through ONE
predicate both the exemption and Privileged read, so the two cannot disagree
about what a probe is. Without it a fail-closed deployment answered 403 to its
own kubelet on GET /v1/kms/health. Read-only methods only, so a route cannot be
named into the exemption to dodge the gate.
And the exempt list is now only what it has to be: the appeal surface (/v1/risk,
because a refusal nobody can read the decision for is unappealable) and the
internal money plane. /v1/ml and /v1/aml come off it — they are ordinary
surfaces served by ordinary apps, and the gate calls the scorer as a Go function
rather than as a route, so watching them cannot re-enter anything.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two enforcement points now defend the lifecycle, and neither of them scores.
cloud.Decide (risk.go) is the ONE door to /v1/risk: the app that owns that
prefix hands its scoring function to the core with SetRiskScorer, the same
inversion SetObsEventIngest already uses because package cloud cannot import an
app. The fail policy lives in one function and nowhere else — an absent,
erroring, silent, out-of-vocabulary, panicking or over-budget scorer ALLOWS an
ordinary request and BLOCKS a privileged grant, and every answer names why it
was not scored, so an allow that happened because nobody was listening is never
recorded as clean.
AbuseGate sits between the rate limiter and the funding gates and keys on the
CREDENTIAL, which is what neither existing limiter can see: EdgeRateLimit keys
on client IP before identity, ScopeRateLimit on (org, project, service), so a
key lifted out of a CI log and used inside its org's normal ceiling is invisible
to both. It counts, classifies, asks and enforces; a non-allow verdict is held
for a minute so an attack costs one screen rather than one per request, and the
refusal is a 401/403 that AuditTrail already records — one event, one record.
Shadow per org by default, and arming is refused while no scorer is installed.
That is the line between a defense and an outage: failing closed on a component
that was never wired would answer 403 to a deployment's own key store.
Agency reads our own issuance and never the client's self-description. An
attributable machine credential is the agent lane whatever its user-agent
claims; a browser session is human; an unattributable caller is unknown until it
shows an abuse shape, and only then bot. Anonymous is not malicious.
The sensor lives in the leaf edge package so the middleware and /v1/gateway
share one object. Bounded by construction, org leading every key, credentials
present only as a keyed per-process fingerprint that cannot be tested against a
candidate key off-box.
GET /v1/gateway/traffic reports the caller's own lane split, denials, screens
and busiest credentials — the differentiator has to be visible to be worth
anything.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A new app, apps/risk, owning the native leaves of /v1/ml: the per-organisation
feature surface over the event surface every org already writes to, and the
per-organisation models trained on it.
The leaves are on this row and not on ml's because the model is IN-PROCESS
MUTABLE STATE. A manifest row is a binary; if one process learned and another
scored, the two would hold different mass counters and answer one question two
ways, with no error and no log. One owner of the state, one row. ml keeps the
kserve/Kubeflow bridge at /v1/ml/models and /v1/train/*, and longest-prefix
match separates them.
Training is in-process on half-space trees (luxfi/aml pkg/anomaly): no training
pass, no retained sample, no job — the geometry is built before data arrives and
the model IS a set of mass counters, per tenant, ~336 KB, snapshot-restorable on
the per-org encrypted store the rest of the fleet already uses. Attribution is a
counterfactual on the model that raised the alert, so there is no second
explainer to disagree with the scorer. Only four base-free packages are linked;
the image's sqlite gate is satisfied.
The boundary is three arguments, not one rule:
TYPE — the warehouse is reachable from one file, and every function in it that
builds a feature statement takes a tenant, which has no exported constructor
and is minted in one place from the validated principal.
STATEMENT — the tenant is the leading BOUND predicate of every statement; the
window is bound; every column and aggregation resolves through a fixed
allowlist. Nothing user-derived is ever an identifier.
BEHAVIOUR — an organisation naming another's subject reads zero rows, not a
refusal, because a refusal is a probe oracle.
The key is <brand>/<org>, so two brands' identically named organisations are two
tenants, two models and two sets of rows. The source planes are read with the
bare slug they carry and the surface is written with the qualified key, and the
direction is asserted rather than commented.
Cross-organisation learning is aggregate-only and it is ONE table with no tenant
column at all: four quantiles of one dimension over one day, published only when
at least twenty-five organisations and a thousand buckets contributed, enforced
in the statement and again on read. No model reads it. The anonymous lane is
refused at the mint, so there is no filter to forget.
Shadow is the default, per organisation, with stated appetite reported beside
realised. Shutdown snapshots every resident model: this binary rolls one replica
at a time, and a model not written down is a tenant returned to warming — and a
warming model refuses to score, which reads as clean to anything not checking
the refusal.
Nine typed zip ops and one route untyped by design (the real probe, whose 503
carries the report a typed error envelope would drop), held to a closed list by
a test. zipdoc -check green; the app's own subset and the woven openapi.yaml
regenerate from source; 39 tests green, and green under -race.
Eleven mutations proven: drop the org predicate, add an org column to the
baseline, key the model on the bare org, carry the bare org on the transaction,
fold two subject kinds onto one key, write the bare org into the surface, drop
the snapshot tenant check, lower the k-anonymity floor, admit the anonymous
lane, let a second file reach the warehouse. Each fails the test that names it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-01 16:36:29 -07:00
2261 changed files with 161563 additions and 58527 deletions
# TWO DIFFERENT FAILURES SHARE THIS STATUS, and treating them alike
# would turn one of them into a ten-attempt loop that ends in the
# wrong diagnosis. "Reference already exists" is the collision this
# loop is for. "Object does not exist" means OUR OWN COMMIT is not
# on github.com — which is a live possibility, because this workflow
# runs on git.hanzo.ai and claims against GitHub, so a commit that
# reached the forge and not the mirror lands exactly here. It is not
# a name to skip past; it is a repo that has not been published, and
# the next number would fail identically.
WHY=$(jq -r '.message // empty' /tmp/claim.json)
if [ "$WHY" != "Reference already exists" ]; then
echo "::error::claiming v${CAND} was refused with: ${WHY:-unknown}. If this is 'Object does not exist', commit ${SHA} is on the forge but not on github.com — the release lane claims versions against GitHub, so the mirror must carry the commit first."
exit 1
fi
# Taken. By us, or by somebody else? The distinction is the whole
# difference between a resume and a collision, and it is one GET.
echo "v$NEXT is already published from our sha — skipping the build"
RESUMED=1
else
echo "::error::v$NEXT is a version this run OWNS (tag at ${SHA}) but the registry already serves bytes built from '${REV:-an unlabelled commit}'. Another lane pushed onto a name it did not hold. Nothing here may overwrite it — publish the intended bytes under a new number and delete the foreign image."
exit 1
fi
elif [ "$CODE" != "404" ]; then
echo "::error::manifest probe for v$NEXT returned $CODE — cannot tell whether the name is free"
exit 1
fi
echo "version=$NEXT" >> "$GITHUB_OUTPUT"
echo "highest seen v$LAST (registry + git tags) -> building v$NEXT (document sha256:$SPEC_SHA)"
echo "resumed=$RESUMED" >> "$GITHUB_OUTPUT"
echo "highest seen v$LAST (registry + git tags) -> v$NEXT, claimed at ${SHA} (document sha256:$SPEC_SHA)"
- uses:docker/setup-buildx-action@v3
if:steps.ver.outputs.resumed == '0'
@@ -379,27 +540,59 @@ jobs:
# VERSION is what the binary reports as X-Api-Version. Without it the
# ldflag falls back to the `dev` default and a released image cannot
# say which release it is — and every car below keys off that header.
# REVISION is passed as a BUILD-ARG, not only as a label, because the
# Dockerfile declares `ARG REVISION=unknown` and stamps the label from
# it. A builder that sets only the outside label leaves that ARG at its
# default, and a builder that sets neither publishes an image whose
# commit is unrecoverable — which is precisely what the platform lane
# did for every image it ever pushed. Feeding the ARG makes the label
# truthful no matter which builder runs the Dockerfile, instead of
# truthful only in the lane that remembers to override it afterwards.
echo "::error::${img} resolves, but the bytes behind that tag were built from '${REV:-an unlabelled commit}', not ${{ github.sha }}. Another lane pushed over the tag this run owns. Nothing downstream may pin it."
exit 1
fi
echo "$img is built from ${{ github.sha }} — tag, commit and image agree"
# THE SMOKE GATE. Boot the image that was actually pushed and require it to
# reach "zip listening" without a crash signature, on release.go's boot env
@@ -460,37 +653,32 @@ jobs:
steps:
- uses:actions/checkout@v4
# The receipt, minted only now: build pushed it, smoke proved it boots. A
# tag can therefore never name an image that did not start.
# THE TAG IS NOT MINTED HERE ANY MORE — it was claimed before the build, as
# the compare-and-swap that made this version this run's to build (see the
# `image` job's claim step). Minting it here was the bug: for the whole
# length of a build, a number was "taken" only in the sense that a lane
# INTENDED to take it, and two lanes intending the same number both pushed
# images before either reached this step. The winner's tag then named the
# loser's bytes, and the loser died here — after the damage.
#
# 422 USED TO BE A HARD ERROR, and that is exactly what made a resume
# impossible: the second run of a release whose fanout failed died here
# instead of continuing. A 422 whose ref already points at OUR sha is this
# same release, already receipted — idempotent success. A 422 pointing
# anywhere else is a genuine collision and still fails.
- name:Tag the release
# What remains is the assertion that nothing moved underneath us. A git tag
# is not mutable by accident, so this is expected to be quiet; it is here
# because the one thing worse than a moved tag is a moved tag that shipped.
- name:The claimed tag still names this commit
env:
GH_PAT:${{ secrets.GH_PAT }}
run:|
set -euo pipefail
TAG="v${{ needs.image.outputs.version }}"
CODE=$(curl -s -o /tmp/tag.json -w '%{http_code}' -X POST \
echo "::error::FLEET_DISPATCH_TOKEN missing in KMS at ${KMS_ORG}/deploy (env ${KMS_SECRET_ENV}). Create it with contents:write + metadata:read on hanzoai/{python-sdk,js-sdk,java-sdk,cli} hanzo-go/sdk hanzo-rs/sdk hanzo-kotlin/sdk hanzo-cpp/sdk hanzo-docs/docs — six owners, so a fine-grained PAT cannot carry it; use a classic PAT with repo scope or a GitHub App installed on all six. Until it exists, every cloud release ships a document no client is regenerated from — which is the failure this car was built to end, so it fails rather than skipping."
echo "::error::FLEET_DISPATCH_TOKEN missing in KMS at deploy/ (env ${KMS_SECRET_ENV}, org from the KMS credential). Create it with contents:write + metadata:read on hanzoai/{python-sdk,js-sdk,java-sdk,cli} hanzo-go/sdk hanzo-rs/sdk hanzo-kotlin/sdk hanzo-cpp/sdk hanzo-docs/docs — six owners, so a fine-grained PAT cannot carry it; use a classic PAT with repo scope or a GitHub App installed on all six. Until it exists, every cloud release ships a document no client is regenerated from — which is the failure this car was built to end, so it fails rather than skipping."
["${n:-0}" -gt 1]||{echo"SKILLS-GATE FAIL: apps/skills/catalog holds the ${n:-0}-skill fallback — the overlay missed the //go:embed path";exit 1;};\
j="$(find /src/webui/dist -type f -name '*.js'| wc -l)";\
["$j" -gt 0]||{echo"CONSOLE-GATE FAIL: webui/dist carries no JavaScript — the overlay missed the //go:embed path and the image would ship the fallback shell";exit 1;};\
CGO_ENABLED=1 go build -tags "libsqlite3 sqlite_fts5 sqlite_math_functions" -ldflags="$GO_LDFLAGS" -o "/plugins/$p""./plugin/$p";\
done
# THE STAMP LANDED — asked of the ARTIFACT, not of the flag string.
#
# `-X` naming a path or symbol the linker cannot resolve is not an error: it is
# dropped, the build succeeds, and every binary then reports the entirely
# legitimate-looking "unknown" forever. A renamed package or variable would fail
# in exactly the one way nobody looks at, which is how this started.
#
# `go version -m` is NOT a witness — it echoes the -ldflags string that was
# REQUESTED, and that string is present even when the symbol was never set
# (measured on /cloud, whose Version stamp has been dropped all along). Only the
# linked bytes answer.
#
# strings|grep rather than a bare grep: grep treats binary input as non-text and
# its exit status there is not portable across implementations, so a plain
# `grep -qF` can report no match on a binary that demonstrably contains the sha.
# strings normalises to text lines first; binutils is already installed above.
#
# An image built with no REVISION is not a failure — it is a build that cannot
# name its commit, and it says so here and on every health response it serves.
RUNset -eu;\
if["$REVISION"="unknown"];then\
echo">> no REVISION build-arg: this image cannot name its commit, and every health response it serves will report revision=unknown";\
else\
strings -a /plugins/base | grep -qF "$REVISION"||{echo"FATAL: -X did not reach /plugins/base — github.com/hanzoai/cloud.revision was not resolved, so it was dropped and every health response would report 'unknown'";exit 1;};\
echo">> revision $REVISION linked into the plugins";\
fi
# Prove a SHIPPED sqlite-backed plugin binds sqlite3_* to libsqlcipher, not a
# plaintext libsqlite3. /plugins/base opens per-org stores under the SAME CGO=1 +
# libsqlite3 build every plugin above got, so it is a real witness for the set.
agentskills:## Regenerate the FULL agent-skills catalog into apps/agentskills/catalog (go:embed source) from the openapi SOT. OPENAPI_DIR=<path to openapi>.
skills:## Regenerate the FULL agent-skills catalog into apps/skills/catalog (go:embed source) from the openapi SOT. OPENAPI_DIR=<path to openapi>.
@test -f "$(OPENAPI_DIR)/skills.py"||{echo"openapi checkout not found at $(OPENAPI_DIR) — set OPENAPI_DIR=<path> or clone hanzoai/openapi";exit 1;}
# skills.py rewrites the whole catalog dir; the .gitignore keeps only the tiny
# `ai` fallback tracked, so the full set is embedded at build but never committed.
@echo ">> openapi.yaml — $$(grep -c '^ /' openapi.yaml) paths. The MCP tool list is NOT an artifact: POST /v1/mcp asks every subsystem."
test-cgo:## Prove the cgo build works too — forces the fork's pure-Go backend via -tags sqlite_purego so the embedded modernc importers don't double-register "sqlite".
Description:"RevokeKey revokes the caller's own API key of the requested class. The class is\nthe same field mint takes — `?type=publishable`, defaulting to secret — so\nrevoking the key that ships in a browser bundle does not sign its holder out of\ntheir own API: the other key keeps working.\n\nRevoking is how a key is replaced when it does not need replacing; minting the\nsame class again rotates it in one step. IAM drops the credential immediately,\nbut the gateway caches keys for a few minutes, so a request that beat the cache\nexpiry may still be served.\n\nFor callers written against the older shape, the class is also accepted in a JSON\nrequest body, read only when `?type=` is absent.",
Description:"Revokes the caller's own API key of the requested class. The class is\nthe same field mint takes — `?type=publishable`, defaulting to secret — so\nrevoking the key that ships in a browser bundle does not sign its holder out of\ntheir own API: the other key keeps working.\n\nRevoking is how a key is replaced when it does not need replacing; minting the\nsame class again rotates it in one step. IAM drops the credential immediately,\nbut the gateway caches keys for a few minutes, so a request that beat the cache\nexpiry may still be served.\n\nFor callers written against the older shape, the class is also accepted in a JSON\nrequest body, read only when `?type=` is absent.",
Fields:map[string]string{
"keyTypeIn.type":"Type is the key class to act on: \"secret\" (sk-, session-equivalent, belongs\non a server) or \"publishable\" (pk-, org-identifying, safe in a browser\nbundle). Omitted means secret, which is what every existing caller means.",
"revokedKey.ok":"OK is true when the key was revoked. A failure is an error status, never a\nfalse here.",
Description:"TopupRails lists the accepted (chain, token, treasury) triples, so a browser can\nrender \"send USDC here\" without the addresses being baked into its bundle.\n\nThis exists because the console previously gated its top-up UI on\nNEXT_PUBLIC_HANZO_HUSD_ADDRESS/_TREASURY — build-time constants. Enabling a rail\ntherefore meant rebuilding and redeploying the frontend, and with them unset the\nUI reported \"not available yet\" no matter what the server could actually accept.\nServing the set at runtime keeps ONE source of truth (the server's config) and\nlets a rail be switched on without shipping a bundle.\n\nEverything here is public on-chain data; no secret is exposed, and the set is\nempty on a deployment that accepts no crypto rail.",
Fields:map[string]string{
"railList.rails":"Rails is every (chain, token, treasury) triple this deployment accepts.",
"railView.chain":"Chain is the human chain name, e.g. \"Base\".",
"railView.chainId":"ChainID is the EIP-155 chain id the wallet must be on.",
"railView.decimals":"Decimals is the token's decimal places — 6 for USDC, 18 for an 18-decimal\ntoken. Cents are derived per-rail from it.",
"railView.id":"ID is the stable rail id to name when submitting a transfer, e.g. \"base-usdc\".",
"railView.symbol":"Symbol is the display symbol, e.g. \"USDC\".",
"railView.token":"Token is the ERC-20 contract address to transfer.",
"railView.treasury":"Treasury is the address on this chain to send funds to.",
Description:"Streams a stored photo. No credentials — see the file header.",
})
zip.Describe("GET /v1/csrf",zip.Doc{
Description:"IssueCSRFToken mints the anti-CSRF token a browser echoes as X-CSRF-Token on\nevery money write (mint/revoke a key, top up, onboard, and the billing/commerce\nwrite verbs). The token is bound to the caller's validated identity and expires,\nso one minted for one identity cannot authorize a write as another.\n\nIt is answered no-store, so it is never cached by a shared proxy. This is the\nsame-origin endpoint the embedded console reads — the Same-Origin Policy is what\nstops a cross-site page from reading the response and forging a write.",
@@ -39,7 +29,7 @@ func init() {
},
})
zip.Describe("GET /v1/embed",zip.Doc{
Description:"EmbedStatus reports whether one of this brand's shared embedded apps (cms, erp,\nhelp) may be framed by the caller and is actually running, so a console module\ncan choose between the embed and the provision panel.\n\nIt answers two questions the browser cannot answer for itself. ENTITLEMENT is\nserver-authoritative: each app is a single shared per-BRAND instance, so only a\nmember of the owning brand org — or a SuperAdmin — is given the embed URL; every\nother caller gets phase \"not-entitled\" and no URL. REACHABILITY is a probe of\nthat origin, which a cross-origin page cannot read for itself.\n\nThe probed host is always <app>.<this deployment's own brand domain>: no part of\nit comes from the request, so this can never be steered into probing an\narbitrary origin.",
Description:"Reports whether one of this brand's shared embedded apps (cms, erp,\nhelp) may be framed by the caller and is actually running, so a console module\ncan choose between the embed and the provision panel.\n\nIt answers two questions the browser cannot answer for itself. ENTITLEMENT is\nserver-authoritative: each app is a single shared per-BRAND instance, so only a\nmember of the owning brand org — or a SuperAdmin — is given the embed URL; every\nother caller gets phase \"not-entitled\" and no URL. REACHABILITY is a probe of\nthat origin, which a cross-origin page cannot read for itself.\n\nThe probed host is always <app>.<this deployment's own brand domain>: no part of\nit comes from the request, so this can never be steered into probing an\narbitrary origin.",
Fields:map[string]string{
"embedStatusReq.app":"App is the embedded app to report on: cms (Content Studio), erp or help.",
"embedStatusResp.app":"App is the app this verdict is about.",
@@ -52,7 +42,7 @@ func init() {
Example:json.RawMessage(`{"app":"cms"}`),
})
zip.Describe("GET /v1/keys",zip.Doc{
Description:"GetKey returns the caller's own API keys — every type they hold, read\nAUTHORITATIVELY from IAM rather than from the session claim, which lags a key\nminted moments ago. No secret material comes back: a secret key is represented\nby its prefix, and only a publishable key (public by construction) carries its\nfull value.\n\nA transient IAM read failure reports an empty set rather than a 5xx, so the\npage shows the honest empty state and never a fabricated key.",
Description:"Returns the caller's own API keys — every type they hold, read\nAUTHORITATIVELY from IAM rather than from the session claim, which lags a key\nminted moments ago. No secret material comes back: a secret key is represented\nby its prefix, and only a publishable key (public by construction) carries its\nfull value.\n\nA transient IAM read failure reports an empty set rather than a 5xx, so the\npage shows the honest empty state and never a fabricated key.",
Fields:map[string]string{
"apiKey.createdAt":"CreatedAt is when the key last changed, as IAM records it.",
"apiKey.key":"Key is the FULL value, and is present for a publishable key only: it is\npublic by construction and useless to its holder if it cannot be read back.",
@@ -61,21 +51,8 @@ func init() {
"apiKeyList.keys":"Keys is every key the caller holds, at most one per type.",
Description:"WalletTopup credits the caller's org for a stablecoin transfer they already sent\nto the treasury. It reads the receipt from that rail's chain, confirms a mined,\nsuccessful ERC-20 Transfer to the rail's treasury, derives USD cents from the\non-chain value using the token's own decimals, records the credit, and returns\nthe amount plus the new balance.\n\nThe credited amount is the ON-CHAIN value, never a number the caller sends, and\nthe credit lands on the caller's own validated org — there is no way to name a\nthird-party subject. Nothing is credited that the chain did not confirm: a\nmissing, failed or non-matching transaction is refused, and a deployment with no\npayment rail enabled says so rather than inventing a credit.",
Fields:map[string]string{
"walletTopupReq.fromAddress":"FromAddress is the wallet the transfer was sent from. Optional; when given it\nmust match the transfer's on-chain sender.",
"walletTopupReq.rail":"Which accepted rail the transfer was sent on, e.g. \"base-usdc\". The client\nnames it rather than the server guessing from the tx: the same address can\nexist on several chains, so inferring would risk crediting against the wrong\ntreasury. It may be omitted only while exactly one rail is enabled.",
"walletTopupReq.txHash":"TxHash is the hash of the ERC-20 transfer that was already sent to the rail's\ntreasury. The receipt is read from that chain; nothing is credited that the\nchain did not confirm.",
"walletTopupResp.balance":"Balance is the org's new USD-ledger balance in cents. Best-effort: a read\nfailure reports 0, and the credit has already landed either way.",
"walletTopupResp.creditedCents":"CreditedCents is the USD credit recorded, derived from the ON-CHAIN value\nusing the token's own decimals — never a client-supplied number.",
"walletTopupResp.status":"Status is how commerce recorded the payment.",
"walletTopupResp.txHash":"TxHash is the transfer that was credited.",
Description:"MintKey creates — or rotates — the caller's API key of the requested type and\nreturns it ONCE. A real IAM failure surfaces as 502, never a fabricated key.\n\nRotating is what creating means here: a user holds one key per type, so the\nendpoint is idempotent by (caller, type) and the superseded credential stops\nworking. Two live secrets for one user would make \"revoke my key\" a lie.",
Description:"Creates — or rotates — the caller's API key of the requested type and\nreturns it ONCE. A real IAM failure surfaces as 502, never a fabricated key.\n\nRotating is what creating means here: a user holds one key per type, so the\nendpoint is idempotent by (caller, type) and the superseded credential stops\nworking. Two live secrets for one user would make \"revoke my key\" a lie.",
Fields:map[string]string{
"keyTypeIn.type":"Type is the key class to act on: \"secret\" (sk-, session-equivalent, belongs\non a server) or \"publishable\" (pk-, org-identifying, safe in a browser\nbundle). Omitted means secret, which is what every existing caller means.",
"mintedKey.accessKey":"AccessKey is the same value under its predecessor name, carried so callers\nwritten against the older field keep working. One value, two names.",
Description:"Onboard creates the caller's organization. Two flows, keyed on whether the caller\nalready has a home org (mirrors app/onboard/route.ts):\n\n - FIRST-RUN (no owner): create + MOVE the user in as admin, so their next JWT\n carries the new owner and the cloud scopes everything to it.\n - ADDITIONAL (owner set): create the org but do NOT move the user — a move\n changes their IAM owner (stripping a SuperAdmin's status + orphaning their\n current org). They reach the new org via the OrgSwitcher, which re-scopes\n X-Org-Id without touching IAM membership. A personal-org request from someone\n who already has an org is meaningless → 409.",
Description:"Onboard creates the caller's organization. Two flows, keyed on whether the caller\nalready has a home org (mirrors app/onboard/route.ts):\n\n - FIRST-RUN (no home org): create + MOVE the user in as admin, so their next\n JWT carries the new owner and the cloud scopes everything to it. This is the\n path a fresh OAuth sign-up takes, from the sign-up application's org.\n - ADDITIONAL (owner set): create the org but do NOT move the user — a move\n changes their IAM owner (stripping a SuperAdmin's status + orphaning their\n current org). They reach the new org via the OrgSwitcher, which re-scopes\n X-Org-Id without touching IAM membership. A personal-org request from someone\n who already has an org is meaningless → 409.",
Fields:map[string]string{
"onboardReq.name":"Name is the organization's display name. Ignored when personal is true, which\nderives the name from the caller's own username instead.",
"onboardReq.personal":"Personal asks for the caller's own workspace: the name is derived from their\nusername and the slug auto-suffixes to stay unique. Meaningless — and refused\n— for a caller who already has an organization.",
"onboardResp.additional":"Additional is true when the caller already had an organization and this one\nwas created WITHOUT moving them into it — they reach it via the org switcher.",
"onboardResp.displayName":"DisplayName is the organization's human name.",
"onboardResp.org":"Org is the created organization's slug, which is what X-Org-Id carries.",
"onboardReq.name":"Name is the organization's display name. Ignored when personal is true, which\nderives the name from the caller's own username instead.",
"onboardReq.personal":"Personal asks for the caller's own workspace: the name is derived from their\nusername and the slug auto-suffixes to stay unique. Meaningless — and refused\n— for a caller who already has an organization.",
"onboardResp.accessKey":"AccessKey is the identifier of the org-scoped credential provisioning minted\nwith the organization. Present on a first run that actually minted one.",
"onboardResp.accessSecret":"AccessSecret is that credential's confidential half, returned ONCE — on the\nresponse that mints it and never again. IAM keeps only its argon2id digest\nand blanks the plaintext, so this is the single moment it exists in a form\nits owner can read; a replay of the same provision re-reveals nothing.",
"onboardResp.additional":"Additional is true when the caller already had an organization and this one\nwas created WITHOUT moving them into it — they reach it via the org switcher.",
"onboardResp.displayName":"DisplayName is the organization's human name.",
"onboardResp.org":"Org is the created organization's slug, which is what X-Org-Id carries.",
Description:"Mints credit for one org. It is the ONE admin mint surface, and it\ndoes NOT mint in-process: it forwards the request to commerce's already-mint-gated\nPOST /v1/billing/credits, authenticated by the service token and scoped to the\ntarget org, then writes one tamper-evident compliance record. Commerce stays the sole\ncredit ledger; this is a thin, audited relay so there is exactly one place credit is\ncreated.\n\nThe body is commerce's OWN CreateCreditGrant contract, forwarded whole — every field\nit carries reaches commerce. The only two this layer reads are the target org (`org`,\nor `user` as the org-pool alias), which selects the namespace commerce's EdgeAuth\ntrusts, and `idempotencyKey`, which makes a double-clicked grant credit once.\n\nA FAILED grant is audited too, with the request body attached: an attempted mint is\nexactly as interesting to a compliance auditor as a successful one.",
Description:"Onboards a hosted service, or edits one, so a new host comes under the\nlaunch gate WITHOUT a redeploy. Re-registering an existing service PRESERVES its live\nswitch — editing the hosts of a service that is already open must not silently close\nit again.",
Description:"WaitlistMode reports whether ONE host is currently gated by the launch waitlist.\nIt resolves the host to the service that governs it and reads that service's\nwaitlist switch, so a guard sitting in front of a hosted surface can decide in one\ncall whether to show the waitlist or the product. It answers for the ONE host\nasked about and never enumerates the registry, which is why it needs no\ncredential. It FAILS OPEN: an unregistered host, an unmounted registry and a store\nfault all answer known=false with mode=false, so a request is never gated pre-boot\nor on a registry fault.",
Description:"Reports whether ONE host is currently gated by the launch waitlist.\nIt resolves the host to the service that governs it and reads that service's\nwaitlist switch, so a guard sitting in front of a hosted surface can decide in one\ncall whether to show the waitlist or the product. It answers for the ONE host\nasked about and never enumerates the registry, which is why it needs no\ncredential. It FAILS OPEN: an unregistered host, an unmounted registry and a store\nfault all answer known=false with mode=false, so a request is never gated pre-boot\nor on a registry fault.",
Fields:map[string]string{
"waitlistModeView.host":"Host is the queried host, normalized (lowercased, port stripped).",
"waitlistModeView.known":"Known is false when no registered service claims this host, or when the\nregistry is unavailable — the guard then lets the request through, which is\nwhy the two cases answer alike.",
Description:"Binds a Service-scoped handler to a route: it adapts a\n`func(*Service[S], *zip.Ctx) error` to the plain `func(*zip.Ctx) error` the\nrouter takes, capturing s. One adapter, so packages write free-function\nhandlers and register them with `app.Get(\"/path\", cloud.Handle(s, myHandler))`.",
})
zip.Describe("PUT /v1/ads/campaigns/:id",zip.Doc{
Description:"Replaces the user-owned fields of one of the caller org's\ncampaigns and answers the stored row. It is a full replace, not a patch: every\nfield is written from the request, so an omitted one is cleared. externalId is\nlaunch-owned and is never touched here, so editing a campaign cannot break its\nlink to a live provider execution.",
Description:"DeleteAgent removes an agent and every run recorded against it. Answers 204.",
Description:"Removes an agent and every run recorded against it. Answers 204.",
Fields:map[string]string{
"agentRef.ref":"Ref is the agent's public id (the agent_… handle create and list return) or\nits org-unique name, from the path. Either resolves the same agent.",
Description:"DeleteTarget deregisters one machine. Only its owner, or an org admin, may\nremove it; an unknown id, a cross-org id and a machine owned by someone else\nall answer the same not-found, so a probe learns nothing about what exists.",
Description:"Deregisters one machine. Only its owner, or an org admin, may\nremove it; an unknown id, a cross-org id and a machine owned by someone else\nall answer the same not-found, so a probe learns nothing about what exists.",
Fields:map[string]string{
"targetDeleted.deleted":"Deleted is true when the target was removed.",
"targetDeleted.id":"ID is the target that was removed.",
@@ -26,20 +26,20 @@ func init() {
Example:json.RawMessage(`{"id":"tgt_1"}`),
})
zip.Describe("GET /v1/agents",zip.Doc{
Description:"ListAgents returns every agent defined in the caller's org, each with the\nnumber of runs recorded against it.",
Description:"Returns every agent defined in the caller's org, each with the\nnumber of runs recorded against it.",
Fields:map[string]string{
"agentList.agents":"Agents is the org's agents, each carrying its recorded run count.",
},
})
zip.Describe("GET /v1/agents/:ref",zip.Doc{
Description:"GetAgent returns one agent with its system prompt and its 20 most recent runs.\nThe ref is the agent's public id or its org-unique name — a created agent is\nimmediately gettable by whatever create handed back.",
Description:"Returns one agent with its system prompt and its 20 most recent runs.\nThe ref is the agent's public id or its org-unique name — a created agent is\nimmediately gettable by whatever create handed back.",
Fields:map[string]string{
"agentRef.ref":"Ref is the agent's public id (the agent_… handle create and list return) or\nits org-unique name, from the path. Either resolves the same agent.",
},
Example:json.RawMessage(`{"ref":"helper"}`),
})
zip.Describe("GET /v1/agents/:ref/runs",zip.Doc{
Description:"ListAgentRuns returns one agent's execution history, newest first — each run's\ninput, its output or its error, and how long it took. Every row is a run that\nactually happened.",
Description:"Returns one agent's execution history, newest first — each run's\ninput, its output or its error, and how long it took. Every row is a run that\nactually happened.",
Fields:map[string]string{
"runList.runs":"Runs is the agent's executions, newest first.",
"runsQuery.limit":"Limit caps how many runs come back, newest first. Absent, zero or out of\nrange (1..200) reads as 50.",
Description:"AgentActivity serves the org-wide recent-activity feed. Events are REAL: each\nrecorded run is an invoked (ok) or failed (error) event; each agent's own\ncreate/update timestamps are created/updated events. Merged, newest first,\ncapped. Nothing is invented — an org with no agents and no runs gets [].",
Description:"Serves the org-wide recent-activity feed. Events are REAL: each\nrecorded run is an invoked (ok) or failed (error) event; each agent's own\ncreate/update timestamps are created/updated events. Merged, newest first,\ncapped. Nothing is invented — an org with no agents and no runs gets [].",
Fields:map[string]string{
"activityFeed.activity":"Activity is the merged run/create/update events, newest first, capped at 50.",
"activityView.agent":"agent name",
@@ -57,14 +57,14 @@ func init() {
},
})
zip.Describe("GET /v1/agents/builds",zip.Doc{
Description:"ListBuilds returns the public index of every published build, most recently\nupdated first, so a gallery can link straight to the story behind each product.\nPUBLIC, no tenancy: publishing is the author's act, and only published root\nsessions appear here.",
Description:"Returns the public index of every published build, most recently\nupdated first, so a gallery can link straight to the story behind each product.\nPUBLIC, no tenancy: publishing is the author's act, and only published root\nsessions appear here.",
Fields:map[string]string{
"buildList.builds":"Builds is every published build, most recently updated first.",
"buildsQuery.limit":"Limit caps the page. Absent, zero or over 500 reads as 100.",
Description:"ReadBuild returns the readable build of one product: the agent session that\nproduced it, turn by turn — the prompts, the reasoning, the commits each turn\nproduced — plus the exact `git log` that re-derives every commit binding from\ngit itself, so nothing here has to be taken on trust.\n\nPUBLIC, no tenancy: it answers only for a session its author explicitly\npublished, which is what makes it safe to be anonymous. An unpublished session\nis invisible here no matter who asks; its owner reads it through the org-scoped\n/v1/agents/sessions routes, which need a validated principal.",
Description:"Returns the readable build of one product: the agent session that\nproduced it, turn by turn — the prompts, the reasoning, the commits each turn\nproduced — plus the exact `git log` that re-derives every commit binding from\ngit itself, so nothing here has to be taken on trust.\n\nPUBLIC, no tenancy: it answers only for a session its author explicitly\npublished, which is what makes it safe to be anonymous. An unpublished session\nis invisible here no matter who asks; its owner reads it through the org-scoped\n/v1/agents/sessions routes, which need a validated principal.",
Fields:map[string]string{
"buildRef.org":"Org is the org that published the build, from the path.",
"buildRef.project":"Project is the product's slug, from the path.",
Description:"AgentMetrics serves the invocations-over-time histogram for the org's Agents\ndashboard. Every point is a REAL count of recorded runs in that time bucket —\none series line per agent that ran in the window. The Resource Usage rollup is\nall-null because this store meters no CPU/memory/storage/cost; the console\nrenders those as \"—\" rather than a fabricated figure. No runs => empty series\n(an honest \"not connected / no activity yet\"), never a synthesized trend.",
Description:"Serves the invocations-over-time histogram for the org's Agents\ndashboard. Every point is a REAL count of recorded runs in that time bucket —\none series line per agent that ran in the window. The Resource Usage rollup is\nall-null because this store meters no CPU/memory/storage/cost; the console\nrenders those as \"—\" rather than a fabricated figure. No runs => empty series\n(an honest \"not connected / no activity yet\"), never a synthesized trend.",
Fields:map[string]string{
"metricsQuery.range":"Range is the window to bucket: 24H, 7D or 30D. Anything else reads as 30D.",
"metricsView.range":"echoes the requested window (24H|7D|30D)",
@@ -85,7 +85,7 @@ func init() {
Example:json.RawMessage(`{"range":"7D"}`),
})
zip.Describe("GET /v1/agents/sessions",zip.Doc{
Description:"ListSessions returns the caller org's live sessions, newest first — each with\nits event count, its direct-child count and a one-line preview of its latest\nevent. With no filter it returns ROOT sessions only, so a dashboard shows one\nrow per flow rather than one per subagent; ?root= or ?parent= descends.",
Description:"Returns the caller org's live sessions, newest first — each with\nits event count, its direct-child count and a one-line preview of its latest\nevent. With no filter it returns ROOT sessions only, so a dashboard shows one\nrow per flow rather than one per subagent; ?root= or ?parent= descends.",
Fields:map[string]string{
"sessionList.sessions":"Sessions is the matching sessions, each with its event and child counts and\na one-line preview of its latest event.",
"sessionQuery.limit":"Limit caps the page. Absent, zero or over 500 reads as 100.",
Description:"GetSession returns one session with its direct child sessions and its 50 most\nrecent events, oldest of those first.",
Description:"Returns one session with its direct child sessions and its 50 most\nrecent events, oldest of those first.",
Fields:map[string]string{
"sessionRef.id":"ID is the session to act on, from the path.",
"sessionView.host":"Execution context (mission-control): the machine/repo/cwd a card shows and\nthe run-target a session is dispatched to. Omitted when a surface didn't report it.",
Description:"DrainSessionControl returns the steering commands (pause/resume/stop/message)\nrecorded against the caller's own session that are newer than the cursor,\noldest first, with the cursor to poll from next. It is how a locally started\n`hanzo code` session — which is not task-backed, so nothing forwards its\ncommands to an execution engine — consumes what the dashboard posted. Read-only\nand bounded at 200 per poll, so a steady poll is cheap and an applied command is\nnever redelivered.",
Description:"Returns the steering commands (pause/resume/stop/message)\nrecorded against the caller's own session that are newer than the cursor,\noldest first, with the cursor to poll from next. It is how a locally started\n`hanzo code` session — which is not task-backed, so nothing forwards its\ncommands to an execution engine — consumes what the dashboard posted. Read-only\nand bounded at 200 per poll, so a steady poll is cheap and an applied command is\nnever redelivered.",
Fields:map[string]string{
"controlDrain.commands":"Commands is the session's control commands newer than the cursor, oldest first.",
"controlDrain.cursor":"Cursor is the seq to send as `after` on the next poll — the highest seq in\nthis page, or the cursor sent in when the page is empty.",
Description:"SessionTree returns the subagent-flow graph rooted at this session: the session,\nits children, their children, each node carrying its own event count. One\nindexed read pulls the whole flow (every node of a flow shares a root id), so\nthe shape is assembled in memory rather than by walking the store per node.",
Description:"Returns the subagent-flow graph rooted at this session: the session,\nits children, their children, each node carrying its own event count. One\nindexed read pulls the whole flow (every node of a flow shares a root id), so\nthe shape is assembled in memory rather than by walking the store per node.",
Fields:map[string]string{
"sessionRef.id":"ID is the session to act on, from the path.",
"sessionView.host":"Execution context (mission-control): the machine/repo/cwd a card shows and\nthe run-target a session is dispatched to. Omitted when a surface didn't report it.",
Description:"Binds a Service-scoped handler to a route: it adapts a\n`func(*Service[S], *zip.Ctx) error` to the plain `func(*zip.Ctx) error` the\nrouter takes, capturing s. One adapter, so packages write free-function\nhandlers and register them with `app.Get(\"/path\", cloud.Handle(s, myHandler))`.",
})
zip.Describe("GET /v1/agents/targets",zip.Doc{
Description:"ListTargets returns every machine registered to the caller's org, newest\nfirst, each with its live session load.",
Description:"Returns every machine registered to the caller's org, newest\nfirst, each with its live session load.",
Description:"GetTarget returns one registered machine, with its live session load.",
Description:"Returns one registered machine, with its live session load.",
Fields:map[string]string{
"GPU.memory":"VRAM bytes, 0 = unknown",
"GPU.model":"\"GB10\", \"8060S\", \"RTX 4090\"",
@@ -171,15 +174,16 @@ func init() {
Example:json.RawMessage(`{"id":"tgt_1"}`),
})
zip.Describe("PATCH /v1/agents/:ref",zip.Doc{
Description:"UpdateAgent changes an agent in place. Every field is optional; a field the\nrequest omits keeps its stored value. The resulting mode+schedule are\nre-validated together, so a partial update can never leave a long-running\nagent without the cron the scheduler needs to fire it, and a transition INTO\nlong-running counts against the per-org cap on scheduled agents.",
Description:"Changes an agent in place. Every field is optional; a field the\nrequest omits keeps its stored value. The resulting mode+schedule are\nre-validated together, so a partial update can never leave a long-running\nagent without the cron the scheduler needs to fire it, and a transition INTO\nlong-running counts against the per-org cap on scheduled agents.",
Fields:map[string]string{
"updateAgentIn.ref":"Ref is the agent to update — its public id or org-unique name, from the path.",
},
Example:json.RawMessage(`{"ref":"helper","instructions":"be terse and cite sources"}`),
Description:"PatchSession updates a session's surface-owned truth: its status, its title,\nthe run-target it is dispatched to, and the product it built plus whether that\nbuild's story is public. A FINISHED session stays finished — reopening a\ndone/error run would fabricate liveness — and publishing is refused unless the\nsession names the project it built, because the public build route is keyed on\n(org, project).",
Description:"Updates a session's surface-owned truth: its status, its title,\nthe run-target it is dispatched to, and the product it built plus whether that\nbuild's story is public. A FINISHED session stays finished — reopening a\ndone/error run would fabricate liveness — and publishing is refused unless the\nsession names the project it built, because the public build route is keyed on\n(org, project).",
Fields:map[string]string{
"patchSessionIn.cwd":"Cwd is where the session is working NOW.\n\nIt was write-once — captured at register and never again — which is right\nfor a run that starts in a directory and stays there, and wrong for a linked\nshell, which is a place a person moves around in. The console showed the\ndirectory `hanzo link` happened to be run from and kept showing it after the\nshell had walked away, so the field answered \"which work is this\" with an\nanswer that was true once. A pointer, so an unchanged path is an omitted\nfield rather than a repeated write.",
"patchSessionIn.id":"ID is the session to update, from the path.",
"patchSessionIn.project":"Project tags the product this session built; Published is the author's\ndecision to let anyone read the story (provenance.go). Both are pointers so\n\"absent\" and \"cleared\" are different requests.",
"patchSessionIn.target":"Target re-dispatches a session to a run-target (the #48 association). \"\" detaches.",
Description:"PatchTarget updates one machine in place. Every field is optional; a field the\nrequest omits is left alone. A metrics patch IS a heartbeat — the server stamps\nits own clock, so a client can neither forge nor backdate staleness.",
Description:"Updates one machine in place. Every field is optional; a field the\nrequest omits is left alone. A metrics patch IS a heartbeat — the server stamps\nits own clock, so a client can neither forge nor backdate staleness.",
Description:"CreateAgent defines an agent in the caller's org: a model, a system prompt\n(instructions) and a set of tool names. The name must be unique in the org and\nmatch ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$. An omitted model takes the\ndeployment's configured default; a named one is checked against the gateway's\nserved catalog, so a model this deployment never serves is refused here rather\nthan failing at run time. A long-running agent must carry a 5-field cron\nschedule (the scheduler would otherwise never fire it) and counts against a\nper-org cap on scheduled agents.",
Description:"Defines an agent in the caller's org: a model, a system prompt\n(instructions) and a set of tool names. The name must be unique in the org and\nmatch ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$. An omitted model takes the\ndeployment's configured default; a named one is checked against the gateway's\nserved catalog, so a model this deployment never serves is refused here rather\nthan failing at run time. A long-running agent must carry a 5-field cron\nschedule (the scheduler would otherwise never fire it) and counts against a\nper-org cap on scheduled agents.",
Description:"Binds a Service-scoped handler to a route: it adapts a\n`func(*Service[S], *zip.Ctx) error` to the plain `func(*zip.Ctx) error` the\nrouter takes, capturing s. One adapter, so packages write free-function\nhandlers and register them with `app.Get(\"/path\", cloud.Handle(s, myHandler))`.",
})
zip.Describe("POST /v1/agents/sessions",zip.Doc{
Description:"RegisterSession opens a live agent session in the caller's org — the row every\nsurface (the CLI's outer agent, hanzo.bot, the console, chat) hangs its\nactivity off. A session with a parentSessionId becomes a subagent of that\nsession and inherits its root, so one flow is one tree; without one it is\nitself a root. Registering with a terminal status records a session that has\nalready finished.",
Description:"Opens a live agent session in the caller's org — the row every\nsurface (the CLI's outer agent, hanzo.bot, the console, chat) hangs its\nactivity off. A session with a parentSessionId becomes a subagent of that\nsession and inherits its root, so one flow is one tree; without one it is\nitself a root. Registering with a terminal status records a session that has\nalready finished.",
Fields:map[string]string{
"registerReq.host":"Execution context — where this session runs (all optional).",
"registerReq.project":"The readable build (provenance.go): which product this session builds, and\nwhether its story may be read by the world.",
@@ -230,8 +237,23 @@ func init() {
},
Example:json.RawMessage(`{"agent":"hanzo-dev","title":"ship the landing page","host":"gpu-01"}`),
Description:"Binds a Service-scoped handler to a route: it adapts a\n`func(*Service[S], *zip.Ctx) error` to the plain `func(*zip.Ctx) error` the\nrouter takes, capturing s. One adapter, so packages write free-function\nhandlers and register them with `app.Get(\"/path\", cloud.Handle(s, myHandler))`.",
Description:"Binds a Service-scoped handler to a route: it adapts a\n`func(*Service[S], *zip.Ctx) error` to the plain `func(*zip.Ctx) error` the\nrouter takes, capturing s. One adapter, so packages write free-function\nhandlers and register them with `app.Get(\"/path\", cloud.Handle(s, myHandler))`.",
Description:"Binds a Service-scoped handler to a route: it adapts a\n`func(*Service[S], *zip.Ctx) error` to the plain `func(*zip.Ctx) error` the\nrouter takes, capturing s. One adapter, so packages write free-function\nhandlers and register them with `app.Get(\"/path\", cloud.Handle(s, myHandler))`.",
Description:"Binds a Service-scoped handler to a route: it adapts a\n`func(*Service[S], *zip.Ctx) error` to the plain `func(*zip.Ctx) error` the\nrouter takes, capturing s. One adapter, so packages write free-function\nhandlers and register them with `app.Get(\"/path\", cloud.Handle(s, myHandler))`.",
Description:"Binds a Service-scoped handler to a route: it adapts a\n`func(*Service[S], *zip.Ctx) error` to the plain `func(*zip.Ctx) error` the\nrouter takes, capturing s. One adapter, so packages write free-function\nhandlers and register them with `app.Get(\"/path\", cloud.Handle(s, myHandler))`.",
})
zip.Describe("POST /v1/agents/targets",zip.Doc{
Description:"RegisterTarget registers a machine as an agent target, or re-links one that is\nalready registered. Re-linking is idempotent and keyed on org+host+owner, so a\nmachine that reconnects refreshes its own row rather than piling up duplicates;\nit answers 200, while a first registration answers 201.",
Description:"Registers a machine as an agent target, or re-links one that is\nalready registered. Re-linking is idempotent and keyed on org+host+owner, so a\nmachine that reconnects refreshes its own row rather than piling up duplicates;\nit answers 200, while a first registration answers 201.",
Description:"MintTargetClaimKey mints (or rotates) the claim key a `hanzo code --serve`\ndaemon presents to claim work for this machine, and returns it ONCE: only its\nSHA-256 hash is stored. Rotating supersedes any prior daemon, so only the\nmachine's owner — or an org admin — may call it; every other caller gets the\nsame not-found an unknown id gets, and learns nothing about what exists.",
Description:"Mints (or rotates) the claim key a `hanzo code --serve`\ndaemon presents to claim work for this machine, and returns it ONCE: only its\nSHA-256 hash is stored. Rotating supersedes any prior daemon, so only the\nmachine's owner — or an org admin — may call it; every other caller gets the\nsame not-found an unknown id gets, and learns nothing about what exists.",
Fields:map[string]string{
"claimKeyOut.claimKey":"ClaimKey is the capability itself. It is returned ONCE and never again — only\nits SHA-256 hash is stored — so a daemon that loses it mints a new one.",
"claimKeyOut.targetId":"TargetID is the machine the key authenticates.",
Description:"ReportRoutedRun completes a claimed run: it delivers the terminal result to the\nrun's durable owner, which is what lets that workflow finish. Scoped to (org,\ntarget, run) and claim-key authenticated, so a machine can only ever report a\nrun it legitimately holds. Idempotent — a report for an unknown or\nalready-finished run answers delivered:false rather than failing, because the\nsession's terminal state was already set by the machine's own stream.",
Description:"Completes a claimed run: it delivers the terminal result to the\nrun's durable owner, which is what lets that workflow finish. Scoped to (org,\ntarget, run) and claim-key authenticated, so a machine can only ever report a\nrun it legitimately holds. Idempotent — a report for an unknown or\nalready-finished run answers delivered:false rather than failing, because the\nsession's terminal state was already set by the machine's own stream.",
Fields:map[string]string{
"reportOut.delivered":"Delivered is true when a waiting durable owner received this result. False\nmeans there was none to deliver to — an unknown or already-finished run — which\nis a clean no-op, not an error.",
"reportRunIn.branch":"Branch, CommitSha and Diffstat describe what the run produced; Error is the\nfailure when OK is false. Each is clamped, never rejected.",
Description:"Tools reports what THIS PROCESS's MCP door carries: how many tools its own\nregistry projects, optionally their names, and which subsystems this process\ncomposed. It is the answer to \"is this door up and does it have anything behind\nit\" — a question a status code cannot answer, since an empty door and a full\none are both 200. What the FLEET's door carries is the fleet door's own answer:\nPOST /v1/mcp, tools/list, which asks every subsystem and names the ones that\ndid not reply.",
Fields:map[string]string{
"aiMCPApp.name":"Name is the subsystem, as the manifest names it.",
"aiMCPApp.served":"Served reports that THIS process mounted it, so its tools are on this\nprocess's door rather than behind a sibling this process only knows the name\nof.",
"aiMCPQuery.names":"Names asks for this process's tool NAMES and not only how many there are.\nOff by default: a list of names is a page, and the question this op exists\nto answer (\"is the door up and does it have anything behind it\") is answered\nby the count.",
"aiMCPSurface.apps":"Apps is one row per subsystem this deployment composes, in manifest order.",
"aiMCPSurface.names":"Names are this process's own tool names, present only when the query asked\nfor them.",
"aiMCPSurface.tools":"Tools is how many tools THIS PROCESS's door carries: its own typed-op\nregistry, projected. It is the only number a subsystem can state honestly —\nwhat the FLEET's door carries is a question only the host can ask, and it\nasks it by asking every subsystem (POST /v1/mcp, tools/list).",
t.Fatalf("anonymous /v1/event on a non-site host want 200 (anonymous lane, kind dropped), got %d",code)
}
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
antje
aandhanzo-dev