POST /v1/lsp — one door, because there is one value: a language server
rooted at a workspace, asked about a position. code (static index) and
lsp (live server) are two reads of the ONE checkout.
{repo, rev?, path, line, character, method} → hover | definition |
references | typeDefinition | implementation | documentSymbol |
completion | diagnostics
Positions are the LSP's and pass through untouched: 0-based line, 0-based
UTF-16 character. Re-basing them would corrupt every multi-byte line, and
the callers already speak LSP.
Isolation is structural, not checked. The org comes from the validated
principal and is BOTH the pool key and the OWNER SEGMENT of the git URL;
a caller supplies only a repo slug. There is no input from which one
tenant can name another tenant's repository.
Scripts-off by default. A dependency fetch that runs dependency-authored
code is RCE triggered by whatever the caller asked us to check out, and it
buys nothing — servers resolve from source. So: npm ci --ignore-scripts,
cargo fetch (not build), go mod download; the python fetch builds sdists
and therefore does not run. rust-analyzer is additionally told not to run
build.rs or expand proc macros, because otherwise the server does at load
time exactly what the fetch was chosen to avoid. One predicate, one place.
The deployed worker must still be sandboxed — see fetchable's comment.
Language table ported verbatim from hanzo-tools-lsp (same binaries, same
argv, same root markers); install_cmd deliberately dropped — a worker that
can npm install -g at request time is one an attacker can make write to
its own filesystem.
server.go is the testable core: Content-Length JSON-RPC with a SINGLE
reader goroutine demultiplexing responses, server→client requests and
notifications. The python tool reads inline from each call, which drops
every frame that is not the awaited response — which is why it cannot
report diagnostics, and why an unanswered client/registerCapability
deadlocks it. 24 tests drive a fake server over in-process pipes; no
toolchain, no network.
Two bugs the tests found, both real in production:
- Close() wrote a polite shutdown unconditionally, so a server that had
stopped reading its stdin blocked it forever — holding a pool slot and,
at Shutdown, the whole binary.
- rel() compared a symlink-resolved root against an unresolved target, so
any data dir with a symlink component (/var on a Mac, a mounted volume)
made every location "outside" the checkout and handed the caller the
worker's absolute path instead of a repo-relative one.
Metered, not Free: cost is the COLD start (checkout + fetch + first index),
not the query. Warm point queries are recorded and free, so the pricing
does not teach callers to re-key their workspace. Gate before the work.
apps/code has NO checkout to reuse — it indexes files POSTed to it and says
so. apps/deploy has the only working-tree checkout and is not importable
(it would link k8s into this binary). This is therefore a second one,
following deploy's invocation and hardened env exactly; the fix is hoisting
it into the root cloud package, not made here.
Mount order: apps.Wire() and its integers are gone. manifest.Apps slice
position IS the order, so lsp's row sits after code's, with order_test's
frozen sequence updated in the same commit.
The comment blamed apps/answer for exposing no constructor. That is true and
it is not the reason.
Research carries an explicit 25-cent per-answer fee (apps/answer/mode.go),
charged through Bill.Gate on the request path where a payer has been resolved
and can be refused. A tool call has no payer, so a direct seam to the engine
would be an unbilled 25-cent operation an agent may invoke in a loop — free
inference, reached by the exact route this codebase keeps closing.
That the package makes it awkward is not an accident to route around: Params is
built from request-scoped billing context and Sink's methods are unexported, so
the money gate is structurally hard to bypass. Wiring it properly means an entry
that takes a payer and charges it — a billing decision, not an adapter.
web_search and fetch_url are different in KIND, not merely cheaper: their HTTP
routes gate on AUTHENTICATION, and the agent request reaching the tool was
already authenticated and metered at /v1/responses. In-process use matches how
they are reached over HTTP. deep_research does not.
A unix socket address is capped near a hundred bytes, and t.TempDir embeds
the test's own long name — on darwin the bind failed on a discarded goroutine
error and every dial refused, so the suite reported a phantom (socket never
began listening) instead of the truth. A short anonymous dir keeps the
address inside the cap on every platform; on a Mac without a mounted tmpfs
the tests now fail for the honest reason instead — the pure-Go SQLCipher
codec refuses to decrypt to persistent storage, which is the fail-closed
property it exists for.
The probe was a closure marshalling a map — a route and nothing else, in none
of the five projections. It is now a typed op whose answer is byte-identical
to the map it replaces (the struct's field order mirrors the map's sorted
keys), pinned by test, and whose prose rides the handler's doc comment the
way every typed op's does — so its openapi.Describe entry is gone.
Every other raw registration now states its survival in one sentence. The
webhook intake speaks the provider's protocol (HMAC over the raw payload),
the invoice PDF serves bytes, the fail-closed wildcard must shadow every
method under every prefix, and the rest obey the module-handler rule, stated
once above Mount: a handler living in hanzoai/commerce behind unexported
internals cannot ride a typed op, and a typed twin would be a second
implementation of the same money move — the drift payments.go exists to
prevent. The typed door onto a module rail is that file's exported-core
pattern, which is module work.
The eight operations that said nothing about themselves — wire, the crypto
rail, and the saved-card family — now state their gate, their tenant scope
and what they fail closed on, which is what let the commerce subset
regenerate at all. That regeneration also repaired two stale artifacts the
committed subset carried: GET /v1/billing/methods claimed its portal
sibling's operationId (the duplicate the weave gate died on), and the two
saved-card POSTs were absent outright.
The fleet golden and the floor now agree with the module: the deposit proxy
and the webhook relay were removed upstream (deposits are commerce's own
rails; the real receiver is /v1/billing/webhooks/:provider), so the golden
drops those four operations, the commerce floor is hand-lowered to four, and
the dead prose goes with them.
commerce v1.50.7 added PUT /_/commerce/providers/{name} — the verb behind "a payment
rail can be turned on and off" — and the dep bump landed without prose for it, so
`commerce describe` refused and the app could not project its own document. The
subset is fail-closed, so nothing was written and the drift stayed invisible until
the next regeneration asked.
It gets the sentence, and the same gate its GET twin states: the tenant comes from
the IAM owner claim and nowhere else, so a cross-tenant write is not expressible;
404 for an unknown provider is byte-identical to the cross-tenant probe's answer. The
one thing worth saying that the shape does not: this owns a single bit and never the
credential, which is why a rail can come back with the same stored secret.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
analytics' subset caught up in the regeneration and started publishing GET
/v1/event.js — an address the fleet then routed to ai, because a prefix owns a
SEGMENT subtree and "/v1/event" does not cover "/v1/event.js". It fell through to
ai's "/v1" and the router oracle said so: the fleet published a path it delivers
somewhere else.
The address is analytics' own — its binary projected it from its own router. So it
gets its own entry, which is what "deeper than the sibling that currently wins"
means for a path whose distinguishing character is a dot.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The weave was red on main and nothing downstream could move: openapi/weave_test.go
is the SOLE writer of openapi.yaml, so the served document, the CLI, every SDK, MCP
and the docs were all frozen behind
operationId "get_v1_billing_portal_methods" is claimed by both
"GET /v1/billing/methods" and "GET /v1/billing/portal/methods"
The derivation was never at fault. operationID(method, path) walks every segment and
yields two distinct ids for those two paths, and it has verified uniqueness inside
From since 2026-07-27. The subset that claimed otherwise was written on 2026-08-04 —
so no run of `commerce describe` at any version could have emitted it. It was
hand-edited: d7024e88 moved the three saved-card verbs to this app and wrote the two
new paths into plugin/commerce/openapi.json by copying the /v1/billing/portal/methods
block, operationId and prose together. The copied description then described itself
("the SERVICE-TOKEN face of the same list a customer reads at /v1/billing/methods",
published AT /v1/billing/methods).
It was hand-edited because it could not be generated: `commerce describe` refused,
and still refused here, with eight operations saying nothing about themselves. So the
fix is the prose, and the file follows from it.
- Eight operations get the sentence they owe a caller: the customer saved-card
family (GET/POST/DELETE /v1/billing/methods), its portal POST, and the top-up
rails nobody had described at all — wire instructions, crypto options, the
deposit mint and the deposit poll. Five of them were not in the published
document in any form.
- The two portal twins stop claiming a proxy that no longer exists. Both families
are served in this process; they are two addresses because they admit two
principals, not because either forwards to the other (apps/billing/billing.go
says the same at the spot the hop used to sit).
- mount.go said the opposite of the manifest — that /v1/billing/methods belongs to
billing and "a registration here is unreachable in the fleet", beside a live
registration of it. manifest.Apps gives commerce both prefixes and withholds them
from billing. Left alone, that comment invites deleting a route that works.
- /v1/commerce/{deposits,deposits/:id/confirm,deposits/:id/status,webhooks/:provider}
lose their prose. The broker-dealer proxy behind them was deleted and the manifest
stopped naming them; what was left was prose for routes nothing serves, which
renders nowhere and reads in source as though it were live. openapi/floor.json
drops commerce 8 → 4 in the same commit, which is where the ratchet asks for the
reason to be.
AND THE ASSUMPTION THAT LET IT LAND IS NOW A CHECK. Weave did refuse the document —
but a collision inside ONE part reaches it with no app attached, so it could name the
two addresses and nothing else, and which of 123 subsets shipped them was a search.
openapi.Subsets is holding the app's name when it decodes the bytes, so it asks there
whether the part is injective — the same uniqueOperationIDs, one statement of the
rule, asked once per part and once over the composition, because neither fact implies
the other. TestSubsetsRefusesAnAppWhoseOwnIDsCollide fails without it.
Regenerating also lands 13 subsets that had gone stale behind the red gate, and drops
zipdoc entries that were lifting commercemid.RequestContext's doc comment — a
paragraph about mint-gated context locals — as eight routes' descriptions.
plugin/iam is deliberately NOT regenerated: its current projection introduces a
schema "Role" that means something different from framework's ({role, user}), which
Weave refuses and which needs a rename in one of those two apps. That is a separate
defect this one merely uncovered.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Carries the P0 fix a live-browser pass proved: the assistant's streamed
completion posted with no Authorization header, was refused, and the card
then blamed the user's session. The release routes every self-reading
stream through the client's one authorized door, moves preferences onto
cloud's /v1/prefs, tells the truth on a 401, and mounts exactly one
composer per viewport.
ai v1.832.30 declares web_search / fetch_url / deep_research but holds no
backend for the two this host serves. agent/builtin_tool/web must stay a leaf
package (object imports agent, agent imports the registry), and websearch lives
here in any case — so this is where the seam closes, beside the balance and tier
readers, for the same reason they are here: this package links both sides and
the host does not.
In-process, never over api.hanzo.ai. The edge validates a CUSTOMER credential
and answers 401 to a service; routing our own calls back through it is what once
fail-closed every completion at 503 on a perfectly healthy pod.
deep_research is deliberately NOT installed yet. apps/answer builds its Params
from unexported fields and exposes no constructor, so wiring it means giving
that package an entry point rather than reaching into it from here. Until then
the tool reports that it is UNAVAILABLE IN THIS DEPLOYMENT — which is the honest
answer and specifically not an empty result: an agent told "no results"
concludes the web holds nothing on the subject and answers from memory in a
confident voice.
fetch_url needs nothing here — ai installs its own crawl at bootstrap.
/v1/event.js resolved the anonymous id itself, out of localStorage alone. That is
ORIGIN-scoped, so it never saw the cookie @hanzo/event shares across *.hanzo.ai
and never saw the id hz.js had already left behind under its own key: an origin
carrying only this tag was a separate population, and one visitor was two or
three people depending on which snippet a surface happened to load.
anon.js is that chain, vendored BYTE-FOR-BYTE from @hanzo/event
(hanzoai/ui pkgs/event/src/anon.js), and tag.go now serves it with the tag as one
asset inside one wrapper — so the door holds no second implementation, and
neither half leaves a name on the page it is pasted into. tag.js calls hzAnonId
and no longer names an identity key at all.
Resolution is cookie · localStorage hz_anon_id · localStorage hz_id · in-memory ·
mint: every id already in a browser is adopted, and only a browser holding none
is given a new one.
TestTagBehavior now runs the COMPOSED asset rather than tag.js, because the chain
is half of what ships and the other half no longer runs alone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The bank details were stored and the rail still answered 'Wire transfer not
configured'. Both doors reach the same KMS store keyed by (path, name, env), so
a read at a path nothing writes is indistinguishable from a bank nobody entered
— it would have stayed silent indefinitely.
cloud writes every in-process secret under /orgs/{org} (apps/destinations,
apps/integrations, credz all build that shape, and the REST surface folds writes
under it from the validated org claim). commerce alone spelled it
/tenants/hanzo/wire. v1.50.7 adopts the host's convention and drops the
hardcoded org: the brand serving the page now decides whose bank is read.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The extension renders structured result widgets (comparison, steps, stats,
timeline, entity, definition) but nothing emitted them: /v1/ask's stream is
status|sources|text|follow_ups|done and the system prompts never mentioned the
format. The client was ready and the producer was missing.
widgetRule teaches the one format both modes share, so search and research
cannot drift apart. It says three things beyond the schema:
- Only when the question's SHAPE calls for one, at most two, most answers none.
A widget on every answer is a worse document, not a better one.
- The PROSE MUST STAND ALONE. The client validates every block and drops
anything malformed — a ragged table, an unknown kind, a field of the wrong
type — keeping the prose. An answer that leaned on a widget to be complete
would read as a hole exactly when validation refused one.
- DATA ONLY, NEVER HTML. This model reads the open web, so every page it
fetches is a potential injection source. Markup it authored would be a path
into the extension's origin, which holds the user's session. The renderer
owns the shapes and escapes every field; the model owns only content.
widget_rule_test.go is the closest thing to a shared type across the two repos.
The renderer lives in the extension, so a typo here fails nothing: it produces
answers whose widgets silently never appear, which looks exactly like a model
choosing not to emit one. The test parses every example out of the prompt and
holds it to the shape the client accepts — one example per kind, every required
field present, the comparison example rectangular (the client drops ragged
tables rather than padding them), and both modes carrying the rule.
Negative control run: renaming the steps example's field to "items" fails with
`kind "steps" example omits required field "steps"`.
Pre-existing and unrelated: the root package does not build right now
(middleware_spend.go, another session's in-flight work). apps/answer builds and
tests clean.
No release has pinned since v1.801.454. .455 and .457 built, published, and
were never referenced by anything — the fleet has been serving .454 while
three versions sat in the registry.
The rollout car reads UNIVERSE_PIN_TOKEN from
/v1/kms/orgs/<org>/secrets/deploy/... and the broker has no such route. The
store root is derived from the validated claim, which is exactly what makes
another tenant's secret unnameable rather than merely refused, so there is no
org in a KMS path and never was. Measured against kms.hanzo.ai:
/v1/kms/auth/login 401 (route exists)
/v1/kms/orgs/hanzo/secrets/deploy/UNIVERSE_PIN_TOKEN 404
/v1/kms/secrets/deploy/UNIVERSE_PIN_TOKEN 403 (route exists)
fanout reads FLEET_DISPATCH_TOKEN through the same wrong shape and would have
failed the same way the moment rollout stopped failing first.
WHY IT WAS INVISIBLE, WHICH IS THE HALF WORTH FIXING. The step is
`set -euo pipefail` and the read is `TOKEN=$(curl -fsS ... | jq ...)`. curl
-f exits 22 on the 404, pipefail propagates it, set -e aborts the assignment —
so the authored `::error::UNIVERSE_PIN_TOKEN missing in KMS ...` is
unreachable code. Every one of these releases died with a bare `exitcode
'22': failure` and never printed the sentence written to explain it. Three
guards in this file were dead the same way; the two KMS logins are fixed here
too, and `receipt`'s create-or-update on a tag with no release yet is the
third (left alone — it is a different car and a different bug).
A guard that cannot run is worse than no guard: it reads as diagnosis
already handled.
KMS_ORG goes with the paths. The org is the credential's, so a variable that
named it was a knob that decided nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The corpus has two sources and both were in-process globals the catalog
process cannot see. The index write was one (previous commit). This is the
other, and it failed more quietly still.
projects.LiveSites answers nil when its package is unmounted, on the stated
reasoning that a deployment which hosts no sites is not an error. That is
true of a DEPLOYMENT and false of a PROCESS: catalog, index and projects are
three separate apps and therefore three separate processes, so in the
catalog process nil never meant "nothing is serving" — it meant "you asked
the wrong half of the fleet". nil and empty are the same answer, so the
corpus was published with no sites in it and nothing anywhere said so.
What that costs is the whole `site` kind: every demo URL, the lineage that
files a remix under community instead of leaving it looking like one of our
starters, and the deployed demos the template lane is mostly made of. The
repos alone would have made a catalog of source with nothing live in it.
plane sites_live, and it takes no org — the same shape as
sites_resolve beside it. This is THE cross-org read; the rule
that makes it safe (public, live, not hidden) is applied in the
query by the app that owns the store, so there is no tenant here
for a caller to widen into.
projects Ready(), so a caller can finally tell "nothing is serving" from
"ask the process that owns the store" — the distinction
LiveSites cannot make, because it answers nil for both.
catalog serving(), the third seam with the same two legs as lexical and
write.
Same test file, same reason: the split is what production runs, so the test
reaches a real peer over a real socket.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud.SetRiskScorer has held the seam and the whole fail policy since it was
written and has never had a producer. It could not have one: the model is
in-process mutable state, so exactly one binary may hold it, and the pod forks
one process per app. Every gate in the fleet therefore read a nil scorer, took
the absent exemption, and allowed unscored — fleet-wide, silently. The
observability plane's event door was the same shape and learned it the
expensive way; obsevents.go is gone and apps/o11y/obs_rpc.go replaced it.
So the scorer is REACHED rather than linked. apps/risk publishes risk_decide on
its own socket, commerce installs a plane client as the seam's first producer,
and POST /v1/billing/topup/token — the one self-serve credit door, whose mint
authority is a settled card charge — asks it before the charge.
The tenant is minted from the PLANE CALLER, never from the body. RiskDecideIn
carries no org and cannot: a model is trained on one organisation's own
behaviour, so naming which model answers would be the only cross-tenant read
this plane has to offer. The HTTP mint (tenantOf) reads a principal parked on a
request, which a plane call does not have, so it would fail closed on every
call; planeTenant qualifies cloud.Who(ctx).Org with the deployment's own brand
instead, and qualify() refuses an empty org.
The gate states Privileged explicitly. cloud.Privileged() matches IAM and KMS
paths and not this one, so the default would be the fail-OPEN branch — a scorer
outage waving through the one route that mints spendable balance. With the bit
set, a scorer that is present and cannot answer refuses and the caller retries;
a scorer that is NOT DEPLOYED still allows, which is the exemption cloud's fail
policy already carries and the plane client is careful to preserve: over a
socket "not deployed" would otherwise arrive as a failed call and take the
closed branch, so the socket is probed, ErrNoPeer is read as absence, and the
lazy child is woken off the request path rather than inside a 150ms budget.
A REFUSAL CARRIES NO SCORE. The engine assigns one before it checks whether the
model has warmed, so a declining model returns a populated, meaningless number;
publishing it would turn "no opinion" into "this is fine". And an alert is a
REVIEW, never a block: cloud's own vocabulary says a statistical judgement may
reach review and no further on its own, and this model is exactly that.
IT SHIPS IN SHADOW. No org is armed, no regime is changed, and a model nobody
has reviewed is in shadow — where alert is forced false however high the score.
So every legitimate top-up proceeds today and every decision is on the record
with the shape and policy version that produced it. The subject kinds move to
the call contract for the same reason the signal names are there: two spellings
of "account" would not read as a disagreement, they would namespace one subject
into two.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The published catalog has been empty since catalog and index became two
plugin rows, and not because a sync failed to run. It ran every hour, read
GitHub and the sites table correctly, assembled the whole corpus — and then
handed it to index.Reconcile, which serves out of the index's own
process-level global. In the catalog process that global is nil and always
will be. So the corpus was never written a single time.
The READ leg was given a plane op when the split happened; the write leg was
deliberately left in-process, on the reasoning that the store has one writer
and it lives where the file lives. That property is right and the conclusion
was wrong: the writer is still one and still the index's, whether the corpus
reaches it through a function call or a socket. Handing it over does not add
a second writer — a second process OPENING that SQLite would, and neither
leg does.
Fixing the read alone therefore could not have shown a row. It turned a
503 into 200 {"data":[],"total":0}: from a page that said it was broken to
one that said the fleet had built nothing. hanzo.app's /community and
/templates have rendered that empty ever since.
plane index_reconcile, the mirror of index_query, with the corpus
relayed verbatim for the same reason the read's rows are raw.
index publishes it beside the read; the swap still executes here,
in the process that owns the file. query_rpc.go -> rpc.go, the
name the other multi-op peers already use.
catalog write(), the mirror of lexical(): in-process first, then the
plane. Both legs now go through the GENERATED index client, so
the app name, the op and the In/Out pair are fixed to each
other by the compiler instead of at run time.
The suite could not have caught this: every existing catalog test mounts the
index and the lens on ONE app, which is the topology this deployment stopped
having — index.Ready() is true there, so the plane leg was never executed by
a test at all. split_test.go models the split instead and reaches a real peer
over a real socket. It fails on the old code with "index: not mounted".
A silent write failure outlives a loud read failure, and it is the harder one
to see: an error names itself, an empty page looks like an answer.
Also commits plane/team, generator output for an op added without a
regenerate in c8b7c31.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The provider list that decides which rails pay.hanzo.ai offers was readable and
not writable, so card, Apple Pay, Google Pay, Cash App, ACH, wire and crypto were
whatever happened to be in the tenant row and moving one meant editing the
database. v1.50.5 adds the write verb, one rail per call — a whole-list PUT built
from what the admin read projection exposes would write every provider back with
an empty KMS path and silently disconnect all of them from their credentials.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/commands serves zip.CommandsFromSpec over the rendered document —
the same function the CLI's tree is derived from, on the same bytes every
SDK is generated from. No new registry, no generator, no build step: a
route registered this morning is a command this afternoon.
serve() gains one line, so both document sources light up at once — Mount
for an app binary, MountFleet for the front door that answers api.hanzo.ai.
Total, and unfiltered by caller on purpose. zip has no per-op scope; it has
Authorizer, which decides on the DECODED INPUT of every op. Permission is a
fact about an input, so any list filtered here would be a second static
claim the Authorizer is free to contradict — wrong in the direction that
hides working functionality from people who have access.
Two things the design did not know, both measured here:
The size. The projection is 2,344,651 bytes, 454,881 gzipped — 1.32x
smaller than the document, not the 4.5x/107 KB the design quotes. That
figure was taken from hanzoai/cli's spec/products.json, a different
artifact with every description, summary and operationId stripped. This
one keeps the prose, and the prose is most of it (Description alone is
1,028,067 bytes). Recorded in command.go rather than fixed by forking
zip.Command into a trimmed wire shape.
The order. 41 of the 2,323 commands share zip's (Service, Name) sort key
— `mq streams-delete` is claimed by three — and sortCommands is unstable,
so the tie fell to a map walk. Two replicas weaving one document served
identical content under different ETags, which is a full re-download on
every conditional request that lands on a different pod. order() completes
the key with (Method, Path).
openapi.Door names what serve registers. Three gates needed exactly that
fact and each had written the one literal that was true when it was written
— Complete skipping a description no app can own, cmd/cloud exempting the
doors from a scoped deployment's surface, manifest refusing an app row that
claims one. The second door made all three wrong the same afternoon.
Pins: every served command is a route the document carries; the served
bytes are CommandsFromSpec of openapi.yaml exactly; two mounts of one
document agree on bytes AND ETag; a conditional request gets 304 and no
body; the projection is smaller than what it projects. Plus the door itself
answering on the host's own spec() mount.
TestFleetIsTheWeaveOfItsApps and two cmd/cloud gates are red on this tree
already, over a duplicate operationId between GET /v1/billing/methods and
GET /v1/billing/portal/methods. Not touched, not absorbed: openapi.yaml
therefore does not yet carry /v1/commands, and the full-fleet version of
the door test belongs in the commit that fixes the weave.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The image job reserves a version by creating refs/tags/v<N> AT THE COMMIT on
github.com. A ref can only point at an object that is there, so for a commit
github has never seen the claim answers 404 — 'Object does not exist' — and
refuses to build.
It routinely has not seen it. CI runs on git.hanzo.ai, which is canonical and
where the push lands; github is fed by a push mirror on an EIGHT-HOUR interval,
and the claim runs seconds after the push. The object the tag must name is
normally hours away, so the failure is not a permissions problem that looks like
a race — it is a race that looks like a permissions problem, and it is why four
days of releases stacked up behind one commit.
The commit is now published to github before the claim reads it, on a ref of its
own: main is the mirror's to move and the two lineages do diverge, while this
step's only job is to make the object exist.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
metric v1.10.0 moved the OTel bridge out of its root package, and zip took
the bump, so the framework and everything that inherits it now link no
OpenTelemetry SDK at all. Cloud's own meter pipeline still imports the SDK
directly and deliberately — it feeds the status page until zip's rows are
proven where the gauges read, and it is deleted with that proof, not before.
Moving /v1/billing/methods in-process (it had been forwarded through a base URL
this deployment never sets, so a signed-in customer got 401 listing their own
cards) quietly changed what an ANONYMOUS caller gets: the proxy answered 401 and
the co-resident chain answers 403, because PinBillingSubject refused both of its
cases with ErrForbidden.
Those two cases are different answers and must not share a status. A service
token is a credential: presenting one and omitting X-Org-Id is an authenticated
request that names no scope, and 403 is right. Presenting nothing is not signed
in. The difference is load-bearing on the customer path — a browser
re-authenticates on 401 and merely reports 403 — so an expired session on the
saved-cards screen showed a permission error instead of sending the customer to
sign in. The billing app it moved from answered 401 on purpose; its test said so
in as many words, and that test was deleted with the proxy.
So the tests move with the route, which is how this surfaced at all:
apps/commerce gains the assertion the billing app used to carry — anonymous
GET and POST answer 401, and a 404 fails loudly rather than passing as
"refused", since an unmounted route also declines every request.
apps/billing loses seven tests that pinned a proxy that no longer exists, the
three handlers behind them (paymentMethods, createPaymentMethod,
deletePaymentMethod — defined, registered nowhere, dead since the move), and
an openapi.Describe for DELETE /v1/billing/methods/{id}, an address this app
published and did not serve. That last one is the exact publish/serve
disagreement manifest.Apps exists to catch.
Also merges the GitHub lineage, which had diverged: 4 commits there (console
embed pinned by semver, platform forge sibling, the ai subscriber-scoring fix)
against 19 here. The version claim mints its tag on github.com, so a commit that
only ever reached the forge cannot be claimed — which is what the image job hit
after the containment fix let it run at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every team surface resolved its caller by decoding an HS256 token itself, so
"who is calling" was answered in six places against one signing key. This adds
identity (apps/team/account.go): ONE seam that resolves a caller, and the only
place a credential's algorithm is routed on. account, typed, files, billing,
collab and the transactor hold the seam instead of the secret, and three of them
stop importing the condemned package — account.go is the only reader left inside
apps/team.
Verification, tenancy and authorization are three questions, answered separately:
- VERIFICATION is cloud's own IAM validator, narrowed. A signature from a
trusted issuer says IAM minted the token, never that it was minted FOR this
surface — IAM's signer emits the same claims into an access token and an
id_token but for aud/tokenType/nonce. A session door must say which it
means, so the lane takes access tokens whose audience this deployment NAMES.
The boundary's no-audience-gate posture is right for an API door and wrong
here; the divergence is stated at the pin.
- TENANCY is the home org from the signed membership set, never `owner`.
`owner` carries the application's org, so it is chosen by whichever app the
caller authenticated through, and a lane reading it scopes every store query
to an org the caller selected. No membership set means no home, which is
also every machine credential — a team session is a person's.
- IDENTITY is the `sub` claim, resolved through the store. The canonical user
id falls back to preferred_username, and an account id derives from a UUID
verbatim, so a token with no sub whose username is a colleague's account uuid
resolved to the colleague. Subject-only, confirmed against the rows a login
created.
An IAM credential never leaves the seam: it is an estate-wide bearer held in an
HttpOnly cookie so page JS cannot read it, and the account RPC echoes a caller's
token back to page JS.
Workspace authorization on the IAM lane is the membership rows (admit) — the
server decides, the caller signs nothing. The transactor keeps its path-borne
workspace token and gains no ambient lane: a WebSocket is exempt from CORS, so a
cookie-borne credential would make the Origin list the data plane's only access
control, and that list no longer carries a wildcard either.
The existing credential answers first on every carrier, so a client that has one
behaves exactly as it did and the new lane serves only a browser holding nothing
else. meet decides a room join and does not own the workspace rows, so team
publishes them on the internal plane and meet asks, off the boundary's own
attestation rather than off headers a client can set. analytics is untouched:
its trust order already resolves a validated IAM bearer ahead of the team token.
The HS256 arm is deleted when login mints IAM-only and front/love/
analytics-collector verify IAM; getWorkspaceInfo is the one surface that still
needs a client change first.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Carries the converged console main: the Recent + Suggested strip on Models,
the topbar slimmed to navigation (environment, theme and alerts fold into
the account menu and drawer; production is the default environment), the
cross-app launcher restored, and the assistant defaulting to Enso. The pin
names the image GHCR serves — probed 200 with a negative control — and the
sha is the forge lineage's, which is the lineage that builds.
The framework boundary now exports a server span and a request record per
request to the o11y ears this binary already runs (planesink: 4317 spans,
4318 logs), shaped by zip against the collector's own receivers — no SDK,
nothing any plugin inherits. The address is the switch: O11Y_SPANS_ADDR and
O11Y_LOGS_ADDR turn each signal on, and O11Y_METRICS_ADDR — the name the
manifest already states — is now genuinely read (v1.24.x read a name the
manifest had stopped matching and reached the right ear by loopback
coincidence).
Every cloud release since 62590c3 — nine consecutive commits, four days — was
blocked in the `containment` job, and nothing about those commits caused it:
go: github.com/hanzoai/zen@v1.4.10: invalid version: git ls-remote ...
remote: Repository not found.
The token was valid and non-empty. hanzoai/zen simply has one collaborator where
its sibling modules — ai, commerce, orm, account — have fifteen, so the build
identity could read every private module in the graph except that one. GitHub
answers 404 for "private and not yours" and for "does not exist" alike, so the
error could not distinguish an ACL from a deletion. An ACL drifting beside the
code stopped the fleet, and the earlier green runs only hid it behind a warm
module cache.
So the build no longer asks GitHub for code that is ours. The module PATH stays
github.com/hanzoai/* — that is the package's name, not its address — while git
dials git.hanzo.ai, which is canonical anyway. go.sum is untouched and still
decides: the forge mirrors the same objects, the zip hashes to the committed h1:
line, and a forge serving different bytes would fail the build instead of
shipping them. Proved cold, with no GitHub credential present at all. GH_PAT
remains the fallback for anything the forge has not mirrored, and the four
modules it was missing (goauthorizenet, namespace, sendgrid-go, commerce) are
mirrored now.
Riding along, because it could not ship until the train moved: commerce v1.50.3
carries the wire rail reading the deployment's bank through the host's secret
plane, the payer reference on the wire memo, and the saved-card and tier guards.
Its work had landed on the lineage the history rewrite replaced, invisible to
the module path everything actually resolves; it is replayed onto the published
history, with thirdparty/square taken from the published side because fitKey()
is applied at all three Square boundaries there and carries tests.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The pin named a sha, then a digest. Both say which bytes built the bundle
but not which console RELEASE it is, so "what console is in cloud
v1.801.N" needed a second lookup, and the last six bumps read like
receipts for nothing.
The builder already publishes both: sha-<sha7>-amd64 on every main push,
and the bare semver on a cut v* tag. v8.5.37 is the tag for console
origin/main's tip.
This SUPERSEDES the digest pinned by #383 rather than reverting it: that
pin shipped the anonymous-visitor entry fix, and v8.5.37 contains that
commit (b1a76fff2a, verified ancestor) plus the toast render-loop fix --
the OAuth return stacked twelve identical cards and now raises one. It is
a strict forward move, not a swap.
Both images were pulled in-cluster with the fleet's own credentials before
this was written, because pinning an image the registry cannot serve has no
rollback path: v8.5.37 is 2383695 bytes, the superseded digest 2375128.
The discipline shifts rather than disappears. A digest cannot be re-pushed
to different bytes; a semver tag can, and :v8.4.118 was. So the rule is now
that a cut tag is never re-pointed -- cut the next patch instead.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The recipe handed each app 41000+index*10 and that was accidental complexity.
The question is "does this binary compose". Answering it does not require owning
a port namespace, a stride, or an index.
It also answered WRONG. A second run inside sixty seconds collided with the first
run's sockets in TIME_WAIT — which `ss -lnt` does not show, so the ports looked
free — and reported up to SIXTEEN healthy apps as DIED. A check that invents
failures gets ignored exactly as fast as one that misses them, and this check
exists because fifteen plugins reached production without one.
:0 on all four listeners deletes the bookkeeping, the stride, the TIME_WAIT
window and the concurrency cap in one move. The kernel already allocates ports
correctly; the recipe just had to stop doing it by hand.
Measured on this tree, same command, minutes apart:
before >> compose FAILED: 16 of 120 (15 of them "address already in use")
after >> compose FAILED: 1 of 120
The one is `kafka`, which fail-closes when no broker answers at
nats://127.0.0.1:4222. That is the app being honest about a missing dependency,
it reproduces identically on unmodified main, and it is not a composition fault.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.801.426 was pinned, rolled out and served traffic while the job meant to build
it sat Failed. The image had been built from OLDER source: two fixes reported as
shipped were not in the running binary, and nothing detected it, because there
was no way to ask the process what commit it was. /v1/health returned
{"status":"ok"} and nothing else, /v1/version was a 404, and no revision variable
existed in the Go source. Establishing the truth took exec-ing into the pod,
running the image's own /admin binary and reading its panic.
ONE fact, stamped ONCE, read through ONE function:
-X github.com/hanzoai/cloud.revision=<40-hex>
reported on the health payload every one of these processes already serves —
/v1/health on the product API, and /healthz, /readyz, /health on the ops listener
(CLOUD_HEALTH_LISTEN, the unauthenticated in-cluster read). No new route, no new
port, no new env var. The two surfaces built their bodies separately — one a map,
one a fixed byte string — which is exactly how a field comes to exist on one and
not the other, so both now build from healthBody and a literal can no longer miss
an addition.
MEASURED, on plugin/base linked with the flags the Dockerfile passes:
:18110 /v1/health {"revision":"d25b0f5e70f79bfb04ec60b1f535f20db9a62062",...}
:18112 /healthz {"revision":"d25b0f5e70f79bfb04ec60b1f535f20db9a62062",...}
:18112 /readyz {"revision":"d25b0f5e70f79bfb04ec60b1f535f20db9a62062",...}
and, linked with no -X at all, {"revision":"unknown"} on all three.
UNKNOWN IS A VALUE, NOT A BLANK. Only a full 40-hex lowercase object name is
reported; an unexpanded ${REVISION}, a branch name, "dev", a short sha and the
empty string all read "unknown" — measured on a real binary stamped `main`, which
serves "unknown". A value that is NEARLY a commit is worse than none, because
someone acts on it. That rule is cloud.IsCommit, and it is the rule the BUILDER
already applied before passing build-arg:REVISION, so apps/platform's private
isCommitSHA copy is deleted and calls it: the two ends of that wire cannot drift
into disagreeing about what they hand each other.
THE STAMP NOW REACHES THE STAGE THAT SHIPS. `ARG REVISION` existed already — in
the FINAL stage, feeding the OCI label, invisible to `go build`, because an ARG
is per-stage. The wire was connected at one end. Worse, the one -X that did exist
reaches nothing: cmd/cloud does not link the root package, so
`-X …cloud.Version=` on /cloud has always been dropped — measured, the flag is in
that binary's `go version -m` record and the value is nowhere in its bytes. The
PLUGINS serve /v1/health and they carried no -X whatsoever, so stamping only the
entrypoint would have left the answering process mute. Both facts now go into one
GO_LDFLAGS used by every binary in the image.
`-X` on a symbol that does not resolve is SILENTLY DROPPED, and under the rule
above a dropped stamp reads as the legitimate "unknown" — invisible, exactly like
the image revision LABEL that has read `unknown` in this fleet with nobody
noticing. Two things close it: the image greps its own linked binaries for the sha
(`go version -m` is not a witness — it echoes the flag that was REQUESTED, present
even when the symbol was never set), and version_test.go LINKS a real binary and
asks the process. The first draft of that test read the child's EXIT CODE, and a
deliberately mis-named symbol sailed through it green, because a test that skips
itself exits 0 like one that passes; it reads the child's OUTPUT now, and the
mutation fails it.
The Dockerfile's ARG sits as late as it can, below `COPY . .` — everything from
there down is already re-keyed by any source change, so a per-commit value costs
nothing, while the same value in scope above would re-key `go mod download` and
turn every build into a full one.
`make` builds report "unknown" on a dirty tree, correctly: REVISION comes from
`git describe --always --abbrev=40 --match='' --dirty`, and `-dirty` is not a
40-hex name, so an uncommitted tree cannot name a commit whose source is not what
was built — the same lie, locally.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three defects shipped today because the only thing that could see them was a
running binary and nothing ran one. Each is now the smallest program that
reproduces it.
scope_group_test.go — the group, which broke twice in one day, in opposite
directions:
- TestGroupWithLaterUseComposes. `g := app.Group(p); g.Use(mw)` while the
routes register through ZipApp on the ROOT. Group returning the raw router
hangs middleware on a node whose subtree is necessarily empty; zip refuses
that at boot, which is what crash-looped fifteen plugins. Asserted with
zip.App.Build — Listen minus the sockets, verdict returned rather than
thrown. RED against the raw-router Group with the exact production text.
- TestGroupPrefixesWhatRegistersThroughIt. The opposite kind: a program that
composes perfectly and answers somewhere else. A child Group must prefix
what registers through it down BOTH paths — the route methods and OpScope,
because `zip.Get(app.Group("/v1"), "/bots", h)` is a real idiom here — and
must leave an absolute path at the subsystem root alone. Asserted on the
composed route table, since no compose check can see a route that merely
MOVED. RED separately against each half of the fix.
- TestGroupUseOutsideThePrefixesFailsTheMount / ...IsAllowed. Confinement
through the door Group opened, and its limit: middleware at a prefix the
subsystem does not own installs nothing and fails the mount, while a BARE
group there is ordinary — a prefix is just a path.
plugin_surface_test.go — the two drifts, DERIVED from the specs the mains
declare (go/ast) and the routes the committed projections hold, so a subsystem
added tomorrow is checked tomorrow:
- TestHealthOwnershipMatchesWhatIsRegistered. OwnsHealth is a claim with two
halves. Claimed falsely, one address is declared twice and zip refuses the
program — reverting plugin/authz/main.go reproduces the crash verbatim:
`GET /v1/authz/health: declared by "authz" at serve.go:324 and by "authz" at
serve/mount.go:31`. Claimed while owning nothing, the address silently 404s.
The second half asks the manifest which health address is this app's rather
than assuming /v1/<name>/health — plan answers /v1/plans, storage /v1/s3.
- TestDeclaredPrefixesCoverTheSurface. A grant a main WRITES must cover the
surface it serves. One-directional on purpose: containment, never equality,
or deploy's 14-leaf row makes its own /v1/deploy bridge an escape. It checks
the half a document can answer; middleware has no address, so the other half
is `make compose`, which runs the binaries.
plugin/bot/main.go — bot declared OwnsHealth: true and registers no health
route anywhere, so the field's only effect was to suppress serve.go's generic
route and leave GET /v1/bot/health answering 404. Measured before: 404. After:
{"service":"bot","status":"ok"}. Nothing changes in the fleet, where /v1/bot/health
routes to `runtime`; this is the standalone binary's own liveness answer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The awk class was [a-zA-Z_-]+, which excludes digits, so `make help` silently
omitted e2e and e2e-ui — two targets that have existed all along and that nobody
browsing help could discover. One character.
Found by running `make help` in console and reading the output against the file,
which is the only way this shows up: the recipe succeeds either way.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The timeout is the cost and it is paid per app, so one-at-a-time made this most of
an hour for 120 apps. A check that takes an hour gets skipped, and a check that
gets skipped is how fifteen crash-looping plugins reached production in the first
place — the mechanism has to be cheap enough that it is actually used.
Each app already had its own data dir and its own port block, so nothing contends;
xargs -P just stops them queueing. Failures go to files rather than racing onto
stdout, and the summary names how many of how many failed. Measured: 120 apps in
minutes rather than ~50.
First full run on this tree: `>> compose: 120 apps boot`.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Aliases, not recipes: dev -> run, lint -> vet. Both targets already existed
and already did the right thing; only the names the rest of the fleet uses
were absent, so there is still exactly one recipe behind each.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip refuses to compose a program whose middleware could never run, and it
refuses at BOOT. So on v1.801.425/.426 fifteen plugins compiled, linked, vetted
and passed unit tests, then crash-looped in production. Nothing in this Makefile
could have caught that: the only thing that finds a compose panic is running the
binary.
SURVIVAL is the signal, and it is the only honest one. A compose panic is fatal,
so a process still alive when the timeout kills it (rc 124) composed. Grepping
the log for a success line does NOT work — `"message":"zip new"` is printed
BEFORE composition, and reading it as a pass is exactly how a broken build got
reported as shipped twice in one day.
Each app gets a writable data dir and its OWN four ports, because without them it
dies on `mkdir /var/lib/cloud/orgs` or on binding :8080/:9653/:9090/:8081 long
before it reaches the router — and an early death looks like silence, which reads
as a pass. That mistake is why a "16 binaries, 0 panics" check was worthless: the
binaries had exited before composing.
It reuses `apps`, so there is one way to build an app binary and no second
mechanism. `clean` takes the scratch dir with it.
Proven both directions on this commit:
GREEN make compose APPS="admin prefs" -> >> compose: 2 apps boot
RED inject `ZipApp(app).Group("/v1/prefs/zzz").Use(Bridge())` into prefs
-> PANIC prefs
zip: the group "/v1/prefs/zzz" declares middleware at
prefs/prefs.go:147 and no routes anywhere beneath it
-> >> compose FAILED, make exits 1
(The injection is needed because every real call site is already fixed — deploy's
was corrected by bd2c284e, so it can no longer reproduce the fault.)
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/console d761fbc. Clicking "Sign in" on cloud.hanzo.ai appeared to do
nothing: console.hanzo.ai/ served a SECOND copy of the Hanzo Cloud marketing page
wearing the byte-identical @hanzogui/shell header, so the click landed on a page
indistinguishable from the one it left and read as a re-render. Reaching hanzo.id
took three clicks, two of them through pages that only asked "did you mean it?".
The console is the application; the marketing face is cloud.hanzo.ai. `/` is no
longer a special surface — an anonymous visitor STARTS the authorize hop.
Pinned by digest, not the usual sha-<sha7>-amd64 tag: that build published to
:latest, which is the moving target this file already warns about. The digest
names these bytes and no other.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The billing gate died of a peer call that was an HTTP URL. COMMERCE_URL defaulted
to the public api.hanzo.ai edge — which is THIS binary — so the /v1/billing/*
forwarder re-entered itself, and apps/commerce/transport still carries the scar
tissue: a whole app republished as an http.Handler, every edge middleware re-run
per peer read, and a goroutine-keyed depth counter (maxDepth = 8) to stop the
recursion it cannot otherwise prevent. A call by name cannot express that mistake,
because there is no address to point at the wrong thing.
An app already declares everything a caller needs:
zip.Post[plane.BalanceIn, plane.Balance](cloud.Plane(), "/finance/balance",
planeBalance, zip.WithOperationID(plane.FinanceBalance))
— the app, the op, the request type and the response type, in one expression.
Cloud already projects that registry as OpenAPI, a CLI, an MCP tool list and a
routing declaration. A typed Go client for a peer call is ONE MORE PROJECTION of
it, which is why it is generated here rather than hand-written once per caller.
commerce.FinanceBalance(ctx, &plane.BalanceIn{Currency: "usd"})
plane/gen emits one package per peer (14 apps, 28 ops) holding ONLY request and
response types and call stubs. What it buys is a check no care buys today: a
hand-written peer call is four independent facts that must agree at RUN time, and
nothing stops pairing commerce's op with iam's name, or BalanceIn with Txns. The
wrapper fixes all four to each other where they are declared.
THE CLIENT HALF MOVED TO THE LEAF, and that is what makes any of this possible.
package cloud is itself a caller — the edge rate-limiter reads finance_scope_rules
— so a client that imported cloud could never be imported BY cloud, and the one
call that most needed to stop being a URL is the one the mechanism could not have
expressed. Ask and everything under it now live in package plane; cloud keeps the
server half (Plane, ServePlane) because binding a socket reports itself to o11y.
cloud.Ask stays as a forwarder, so the 44 existing call sites do not move and
there is still exactly one implementation.
It does not drag the peer's tree: plane/commerce is 355 packages against
apps/commerce's 1231, and imports zero apps/ packages — one more than the leaf it
needs. An importable client that linked the implementation would have rebuilt the
problem with extra steps.
Generated FROM SOURCE, judged BY THE RUNNING REGISTRY. zipdoc already reads these
same call sites; reading source buys hermeticity a mount cannot (no store opened,
no boot order, no app that must come up before it can be described). zip's rule —
project from the live router, never the AST — is about a host discovering a plugin
it does not build, and it still binds: plane_registry_test.go mounts commerce and
asserts the generated surface IS the live plane registry, so the generator never
gets to be quietly wrong. Reading the AST's index expression alone had already
been quietly wrong once — treasury spells its registration with inferred type
arguments, so its only op was dropped; types.Info.Instances sees both spellings.
Proven against the real thing, not a fake. plane_client_test.go mounts commerce as
a plugin process does, binds its plane socket as Serve does, and calls the
generated function: commerce answers amount="0" currency="USD" over the socket. A
cold peer with no router answers ErrNoPeer naming the app — a named absence, never
a timeout a caller would have to guess at.
Three root-package call sites converted, including both money ops — the prepaid
gate and the meter now reach commerce as commerce.FinanceAuthorize and
commerce.FinanceRecord. Those are the imports that were structurally impossible
before, so they are the proof the direction is real.
Full suite: 133 failing test names / 26 packages, byte-identical to the same
measurement on origin/main. Regression set EMPTY. go vet ./... exit 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/graph was never a graph database. Its package doc says so: "chain data:
your block indexers and how far each has caught up, plus the on-chain price
feeds." It proxies luxfi/indexer (explorer REST) and luxfi/graph (GraphQL) --
"graph" here meant GraphQL, not a property graph.
Both upstreams already mount under /v1/explorer (client.go: graphd default
prefix), so the name follows the contract the app already speaks rather than
inventing one.
Wire unchanged: the app keeps /v1/indexers and /v1/oracles, keeps its frozen
mount position, and the woven document moves only x-app and the tag prose.
References to luxfi/graph -- GRAPH_URL, graphQLPath, the upstream log key --
stay, because those name the upstream, not this app.
/v1/graph is now free for the embedded per-org graph layer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The self-hosted-git allowance existed, was correct in intent, and could never
match. `selfGitHost` was `deps.Domain` verbatim — "api.hanzo.ai" — while
hostAllowed grants that host or a SUBDOMAIN of it. The forge is "git.hanzo.ai":
a sibling. So every native build was refused with
repo.url host "git.hanzo.ai" is not an allowed git provider
and the estate fell back to building from GitHub, which is exactly what broke
when hanzoai/cloud moved to hanzo-inc/cloud and the build credential lost access.
The code's own comment already said APEX; only the assignment disagreed.
selfGitHost is now the registrable apex ("hanzo.ai"), so every sibling the
deployment owns — git., ci., cd. — is a trusted build source by construction,
for hanzo.ai, lux.network, zoo.network and any white-label domain, with no
per-brand list to maintain. publicsuffix rather than "last two labels" so a
multi-label suffix (co.uk) yields the registrable domain and not the suffix
itself, which would trust every domain under it.
Proven both directions: TestSelfForgeIsAllowedFromTheApex admits
git/ci/cd.hanzo.ai and still REFUSES git.evil.com;
TestOldDomainVerbatimRefusedTheForge pins the old behaviour as the defect, so
this cannot silently regress.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/runtime was never a second product. It was the transport to @hanzo/bot —
the TS service that executes a run — plus a relay of that service's own ops
paths at /v1/bot/*. What separated it from apps/bots was a LANGUAGE boundary,
Go surface and TS executor, wearing the shape of a product boundary. Both
answer for the same thing: a bot doing your work on a real desktop.
So they merge. runtime.go becomes transport.go, ops.go becomes relay.go,
bots.Mount mounts both faces, and the manifest holds one row for the pair.
THE WIRE DOES NOT MOVE. Every path this fleet serves is the path it served
before — the whole diff to openapi.yaml is `x-app: runtime` -> `x-app: bots`
on the seven relayed operations. No CLI, SDK, MCP tool or doc regenerates to
a different address, because none of them has a different address to go to.
Two things the merge had to earn rather than assume:
- apps/coding also dispatches to that executor, so it followed the transport
from apps/runtime to apps/bots. That is not new coupling wearing a new
name: coding runs its tasks ON the bot runtime, which is what the import
now says out loud.
- the typed-or-named gate was TWO gates, one per old package, each blind to
half of what is now one surface. They are one gate over the whole product,
and mountWith mounts the relay so the gate can actually see it. Two gates
stapled together would have kept passing while measuring nothing.
/v1/bot is still shared with apps/bot, whose three deeper prefixes win on it by
specificity. That sharing is the remaining defect and it is not this commit's
to fix: apps/bot's product is connected machines, not a bot, and it is the one
that has to vacate.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip exported TWO things named Mount and they were unrelated:
zip/remote.go func Mount(prefix, addr string, decl ...Declaration) (*App, error)
112 subsystems func Mount(app cloud.Router, deps cloud.Deps) error
The second is this repo's registration contract, now enforced by the compiler
through cloud.MountFunc. The first was a leftover — one of five composition
verbs (Listen/Mount/Add/Graft/Use), the other four dissolved into Use, and this
one survived only by being the one that pointed at another process.
Renamed upstream to Proxy, which is what it BUILDS: every route it registers
runs one handler whose whole body is forward() to the given address. It is also
a noun, which a function returning a value should be. Not Remote — zip.Remote
already exists and is the CALLER's side; this is the SERVER's side, a stand-in
inside this program's own routing table.
Source-compatible here: cloud had no zip.Mount call site, only the one comment
in cmd/cloud, which now names the function that exists.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`type Store` is declared 40 times under apps/ and 35 of those are structurally
IDENTICAL — `struct { db *sql.DB }`. That number is a trap and I am not acting
on it: in Go, git.Store and bot.Store are different types because they are in
different packages, each carries its own app's schema in its own methods, and
"consolidating" a one-field handle wrapper would produce `type Store struct{
shared.DB }` — a rename that moves no code and removes no duplication. 35 of 40
share a shape; ~0 share a MEANING. Optimising that number would be pure damage.
The duplication the number was pointing at is one level down, in the
CONSTRUCTOR, and it is exact. 33 stores open themselves with byte-identical
code modulo their own name:
db, err := cek.Open(namespace.System(), "<app>", dir)
if err != nil { return nil, fmt.Errorf("open <app> store: %w", err) }
sqlpool.Single(db)
Those three lines are a PAIR that nothing paired. sqlpool.Single's own doc says
a two-statement read-modify-write (tracker's per-project issue number, agents'
MAX(seq)+1) is atomic ONLY because no second connection can interleave — so the
cap is a correctness requirement, and it was a separate call every caller had to
remember. 33 remembered.
apps/framework did not. It opens a cek database through an engine OpenDB
callback and never capped it, so its DocType store has been running with an
uncapped pool. That is the defect, and it is the one a 33-way copied prologue
exists to produce: the rule holds until someone writes the 34th store.
sqlpool.Open is the rule moved INSIDE the opener, so there is nothing left to
forget — the same argument as cloud.App being the only way to obtain an app.
36 call sites converted, framework included; the stores that open differently
(git takes a *sql.DB from the OrgStore cache, others carry extra migrations)
still do, because they are different and the point is not uniformity.
Measured, not assumed: shapes compared by AST field set, not by name. Of the
other collision families the brief named, Result (9 decls) and Config (10) have
ZERO structurally identical pairs, and Client has exactly one family of 3
(`base string; http *http.Client; token string`). Those are Go's package
qualifier working, and they were left alone.
Regression set EMPTY by name: all 15 failures in the touched packages
(TestDocTypeAndDocumentRoundTrip, TestForkCreatesProjectFromTemplate, the Redeem
family, …) fail identically on forge/main.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
All five diverged from the canonical Mount signature, and the divergence was
invisible because nothing checked it. Now MountFunc does (see the parent commit),
so these are what the compiler demanded.
agent, ai, commerce took *zip.App, so they had to be wired through the
old App field — which ALSO granted app-wide middleware.
They now take cloud.Router and reach the concrete app
through cloud.ZipApp, the named hole for it.
agent installs no middleware at all (hanzoai/agent
calls Use nowhere) and now mounts SCOPED: it had the
whole binary's gate to buy a typed-op registry.
ai and commerce genuinely do gate everything —
commerce wraps all of /v1 — and now DECLARE it as
Plugin.Global at their composition root, where a
capability belongs, instead of implying it with a
parameter type.
dataroom imported cloud as `hcloud`. The alias is cosmetic to
the compiler and fatal to every grep-based check —
which is how these five stayed invisible. One name.
treasury/anchor not a Mount, but the same disease one level down and
the cleanest Pike case in the repo: status() took
ledger.Backend — ELEVEN methods — to call Root and
nothing else. The parameter said the anchor could
accrue revenue, seed the reserve, debit a program and
rewrite the revenue-share policy. It can do none of
those. It now takes a one-method `rooted`, declared by
the consumer, which both backends satisfy without a
line of change.
RETRACTED: rollingcap's `func Mount(_ cloud.Router, _ cloud.Deps) error` was
reported to me as the fifth violation, possibly dead. It is neither. It installs
the rolling AI-spend cap reader — live, and load-bearing — and `_` is CORRECT Go
for parameters it genuinely does not read. It already IS a MountFunc; the
compiler says so. Naming those parameters to satisfy a text pattern would make
the code worse. Left exactly as it is.
Regression set EMPTY by name (148 failing tests before, the same 148 after).
apps/commerce's TestBalanceCents and TestInProcessClient fail identically on the
parent commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two names in this file each meant two things, and both cost the fleet something
real.
1. Plugin had TWO mount fields with TWO signatures
Mount MountFunc // func(Router, Deps) error
App func(*zip.App, Deps) error // "for a subsystem that gates everything"
App braided a POLICY question (may this subsystem install middleware over the
whole binary) into a TYPE question (what shape is its entry point). Because the
grant carried its own signature, a subsystem could take the grant just to get
the concrete *zip.App — and three did. agent, ai and commerce each held app-wide
middleware authority they never asked for, purely because their Mount named a
concrete type. hanzoai/agent calls Use nowhere; agent's grant bought it nothing
and risked everything after it in the mount order.
Nothing reported this. The field's own doc claimed "apps.TestWireFrozen fails on
a new one"; no such test exists anywhere in this repo — grep it. A contract
stated in a comment and violated five times means nothing is checking it.
Unbraided: App becomes `Global bool`, and every subsystem — scoped or global —
goes through spec.Mount. The grant now decides only WHICH Router arrives (the
bare app, or a scope bound to declared prefixes). One shape, so MountFunc is the
whole enforcement and it is the compiler: at 123 composition roots a divergent
Mount cannot be assigned and cannot link. It found a sixth violator I had missed
by grep on the first build — cloud.MountMetrics, whose doc says "adapts
hanzoai/metrics into a MountFunc" while its signature took *zip.App. It is one
now, and mounts SCOPED (hanzoai/metrics installs no middleware either).
The concrete app stays reachable through ZipApp, the named hole that already
existed for exactly this and reports nil rather than pretending.
2. Router restated zip.Router instead of embedding it
Ten method lines were COPIED here. Copying an interface makes cloud a second
place zip's routing surface is defined, and the two agree only while someone
keeps them agreeing. When zip v1.23 widened one signature (Use took Component,
not Handler), every implementor that had spelled the methods out had to move in
lockstep — which is what stalled v1.19+ adoption across the fleet.
Embedded, a zip routing change costs this file zero edits, and Fiber() +
Plugins() are visibly what cloud ADDS rather than being buried among ten lines
cloud merely echoes. It carries zip.OpTarget along, which is a gain, not a
widening: scope, *zip.App and commerce's mintRouter all already have OpScope,
and a Router that IS an OpTarget is one zip.Get[In, Out] accepts directly.
Regression set EMPTY, compared BY NAME against a pristine worktree at the same
commit: 148 failing tests across 29 packages before, the identical 148 after
(comm -13 on the sorted name sets is empty in both directions).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerceTierToLadder allow-listed only `starter`->trial and `pro`/`enterprise`
->paid, and folded EVERYTHING ELSE to free. The plans commerce actually sells
are go ($9), dev ($19), pro, max, team and business — so a subscriber on
go/dev/max/team/business was scored free, then refused every SKU carrying a
trial or paid floor AND throttled by the free-tier flash cap on top. We took
their money and shut the door.
v1.832.29 inverts the default: a plan is PAID unless it is explicitly free
("", "free", or a `*-free` suffix), with starter/trial the one middle rung.
That direction is safe because A TIER IS NOT A PAYMENT. filter_balance.go has
no exemptions and fails closed — "nothing runs on credit it has not been funded
for" — so an unrecognized plan reaching `paid` still cannot spend a cent it has
not been funded. The OLD default was the dangerous one: it silently cost
revenue every time anyone added a plan slug. Now, add a plan and it works.
Both tests had encoded the defect as intent ("developer"->free,
"mystery"->free) and were corrected to the real plan list. 22 packages ok,
0 FAIL at the module.
Also carries the zen seed fix (v1.832.24) and the object TestMain fix
(v1.832.27), which took that package from ZERO tests executing to 153 PASS —
TestMain called os.Exit(0) before m.Run(). That immediately surfaced a
crawl-storage default still naming the retired MinIO Service, a correction that
had been made twice and could never take effect.
Verified: `go build ./...` clean at this pin.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A signed-in customer got 401 listing their own cards, and the checkout's
prefill failed on every load: cloud's billing app forwarded /v1/billing/methods
to commerce over HTTP, and that proxy is unconfigured here
(CLOUD_COMMERCE_HTTP_URL is unset), so the customer address for saved cards has
never worked on this deployment.
An internal HTTP hop to a service compiled into the same binary is the wrong
shape whatever its config, so the three verbs move to the commerce app and are
served in-process on the same pinned-subject chain as their portal twins — the
gate that keeps a caller inside its own account whatever it sends. The prefix
moves with them, because the router must deliver where the handler lives.
Also drops /v1/commerce/deposits and /v1/commerce/webhooks from the manifest
and the published document: the broker-dealer proxy behind them is deleted, and
a manifest that claims an address nothing serves is how a path silently routes
to the wrong app.
The routing oracle is what found all of it — both halves, in both directions.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Per-org KMS hydration runs only under KMS_ENABLED, unset here, so the rail
answered "not configured" with every field stored correctly. v1.50.1 falls
back to deployment env, which universe now supplies from commerce-secrets.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`p.Status = "error"` on a non-live completion was UNCONDITIONAL, so "this project
is broken" was reachable from any deployment row the org could name — including
one that had already been superseded.
Found by triggering it on hanzo-ai, not by reading it. v5 was live with all 8401
objects correct in the bucket; completing v3 — an older probe deployment left
queued during the Sites-plane migration — as `error` flipped the PROJECT to
"error", and the sites edge stopped serving. Nothing was wrong with the site. The
bytes never moved. A status field on a superseded row took the host down, and
re-completing v5 as live brought it straight back.
The ordinary production shape is the same bug with worse timing: a rebuild that
fails while the PREVIOUS build is still live. The old content is still being
served and the site is up, yet the project gets marked broken — and the "report a
failed build" step that every CI workflow carries (so a dead build cannot leave a
deployment queued forever) is exactly the thing that would send it.
The rule is now named rather than inlined: failureOwnsProject(currentDeploy,
deployID). Error propagates to the project only when the deployment IS the one the
project points at, or when it points at nothing — a first deploy that fails leaves
a project that has genuinely never served, and that one should read as error. The
deployment row is still error in every case and LifecycleDeployFailed still fires,
so the failure stays visible where it belongs: on the deployment, not on the health
of a site that is up.
The `live` branch already recorded p.CurrentDeploy, so the information needed to
make this distinction was there the whole time and simply was not consulted.
TestFailureOwnsProject pins all four cases. Mutation-checked: restoring the old
`return true` fails exactly the two that matter — the superseded deployment and
the failed rebuild — so the test cannot pass against the behaviour it exists to
prevent.
(TestDeleteStays204WithNoBody fails on darwin both with and without this change:
the SQLCipher codec refuses to decrypt without tmpfs, and macOS has no /dev/shm.
Pre-existing and unrelated.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A plan-refused caller was told to request limited-preview access — a
door that cannot open when the plan is the blocker. Each family gate
now speaks its own sentence: upgrade for the subscription floor, paid
capacity for the funding floor, request-access only where a grant is
truly what is missing.
The edge resolved an extension-less path to two candidates — the exact key and
`<rel>/index.html` — and Next.js `output: export` writes neither for a normal
route. Without `trailingSlash` it emits the FLAT file `pricing.html`, so the
homepage served and every other route 404'd.
Measured on the hanzo.ai export (759 pages, 8402 objects) published to
hanzo-ai.hanzo.app:
/ 200 /pricing 404 (/pricing.html 200)
/zen 404 /zen.html 200
/zen/models 404 /zen/models.html 200
That is the shape of a site that reports `status: live`, serves its homepage,
and is unusable — the failure is invisible from the deploy and from the root URL.
`<rel>.html` is added as a middle candidate. Both spellings are legitimate and
both are now tried: Hugo, Jekyll, Vite MPA and `trailingSlash: true` emit the
directory-index form, Next's default emits the flat form. The alternative was
setting `trailingSlash: true` in every repo, which pushes a server limitation
onto each site and rewrites every canonical URL to work around it — this is one
extra HEAD miss on the paths using the other convention, in one place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The embedded family catalog no longer gates any enso SKU, so discovery
stops advertising a waitlist and ai serves every caller without a grant.
The app builder's default model refused everyone outside the launch org;
now it answers.
The envelope's RAM-backed guard now recognises a native macOS tmpfs
(sudo mount_tmpfs <dir>, then HANZO_SQLITE_RAMFS_DIR=<dir>), so the
cek-backed suites can run on a Mac instead of failing on every commit.
Fail-closed is unchanged: anything statfs cannot verify as tmpfs is
still refused.
Record.NeedCommit carried a xorm-style `db:"index"`, and dbx reads that tag
as the column NAME — so the column was called `index` while
ScanNeedCommitRecords queries `need_commit`. Live in this pod: 'no such
column: need_commit' every five minutes, and the record-chain commit task had
never committed a record.
Also carries a guard for the class, placed in routers rather than object
because object's TestMain os.Exit(0)s that package without a seeded database —
a test there reports ok without ever running.
hanzo_service_up is now recorded a second time, through Metrics() — the
registry the framework exports natively — under the exact series name and
label the meter pipeline has always published. Two producers, one probe, one
truth: every rule and reader sees an uninterrupted series whichever road the
sample travelled, which is what lets the old road be retired without a gap
once every reader is confirmed on this one.
The verdict is recorded where it is decided: the probe client's transport,
which already tells each target's reason on change. A nil registry skips
recording — a test that mounts probes without one measures the probing, not
the export.
zip moves to v1.24.6 for Metrics(), which also brings the native span and log
export and the convention that finds a collector with no configuration.
hanzoai/ai is a library this binary links, not a service of its own, so the
crawl4ai leg it dropped stays live in production until this pin moves. v1.832.25
points /v1/crawl at apps/crawl.Fetch — the same guarded dialer the answer
engine's read stage uses, refusing non-public addresses on every hop including
redirects — instead of dialling crawl.hanzo.svc.cluster.local:11235, a name that
does not resolve and returned success:false for every request.
The survey shipped with the loop steerable by the pages it read. Its decision
prompt carries titles and URLs crawled from hosts we do not control, so the move
that comes back is partly authored by them — and nothing checked it. Three
consequences, all reachable by getting one page into the ranked set:
a `read` URL was fetched whether or not the survey had ever gathered it, so a
page could point the cluster's egress at any address and carry the user's
question there in a query string, then have the answer filed in the tenant's
corpus;
a round's `queries` list was unbounded — mode.maxQueries bounded round zero and
nothing else — so one move could issue hundreds of serial meta-searches from a
shared egress that is already bot-challenged;
page text was spliced into the numbered source block unfenced, so a body
containing `[9] Title\nURL\nbody` became a source indistinguishable from a real
one, and nothing validated that the URLs the answer cited were sources at all.
CONTAINMENT, NOT INSTRUCTION. A prompt is advice to a model. These are properties
of the text:
`read` is intersected with the pool the survey itself gathered, and `queries`
and `read` are clipped where the untrusted value crosses into the loop;
ground.go fences each source with a per-request nonce the page cannot know,
because it was written before the request existed;
every markdown link in the answer is checked against the gathered set — on the
stream and in `done`, through the same function — so a citation always points at
a page THIS request fetched. An ungrounded link keeps its text and loses its
target: the sentence still reads.
Two bounds that could not bind now do. tokenCeiling measured the survey's own
subtotal while the plan and the synthesis — the expensive calls — sat outside it,
so research's 400k ceiling was ~50x the reachable total; it now takes the
request's running total. And plan, survey and synthesis shared one 300s clock
with no reservation, so a survey that spent it handed a full corpus to a
completion that could not start and the caller got "the model is unavailable"
after five minutes; the gather now gets 70% and synthesis keeps the rest.
Also fixed, and each one a bug on its own:
unread MARKED every URL a move named while read FETCHED only the first six, so
a move naming ten pages blacklisted four without ever fetching one — and the
round then tripped saturation and ended the survey early, losing evidence on
exactly the runs working hardest. The cap now comes before the marking.
Source.Text carries the fetched page and Source.Snippet stays the search
summary. Reading no longer overwrites what the wire shows, so a research answer
stops re-sending up to a megabyte of duplicate page text across twelve
snapshots, and one frame per round is now the whole story.
A round's queries run concurrently. Serially, at 12s each, a six-query round
spent most of the answer's wall clock waiting.
An unreadable move gets ONE stricter reprompt. A model that opens with "Sure!
Let me look at the JVM next" collapsed research into a single pass, and nothing
downstream could tell that from a model that judged the evidence complete.
A client that hung up ends the run: one research answer costs five minutes,
three dozen fetches and up to eight completions, and a closed tab bought all of
it. The plan's topic headings now reach the client as a planning detail, which
is what makes a three-minute wait legible. Crawl and completion failures are
logged — a degradation nobody can see is a degradation nobody can fix.
The package doc claimed METERED ONCE. It is not true: build.go hands this engine
a metered AI plane, so every internal completion also debits the payer per token
alongside the flat fee. Which layer should price /v1/ask is an open decision,
recorded rather than claimed away.
The SearchEvent union is unchanged — same variants, same keys, same order.
v1.801.445 failed to build, twice, on `go mod download`:
github.com/hanzoai/commerce@v1.49.67: reading go.mod at revision
v1.49.67: unknown revision v1.49.67
v1.49.67 is a PHANTOM. It exists on no remote: github.com/hanzoai/commerce
carries exactly two tag refs (v1.50.0), and git.hanzo.ai/hanzoai/commerce
lists ...v1.49.65, v1.49.66, v1.49.68 — .67 is skipped. It survives only in
warm module caches, which is why .442/.443/.444 built with the same pin: the
BuildKit `cloud-gomod-v4` cache is PER NODE, .444 landed on a node that still
held it (runner-pool-32g-3mn0fk) and .445 landed on one that did not
(runner-pool-32g-3mnls1). The release train was therefore passing by luck of
placement, and any cold node broke it — exactly the failure the Dockerfile
comment above `go mod download` anticipates.
Nothing here can be fixed by retrying. The Dockerfile sets
GOPRIVATE=github.com/hanzoai/* with GOPROXY=...,direct, so hanzoai modules
resolve DIRECT FROM GITHUB, and v1.50.0 is the only commerce version GitHub
has. This is an upgrade, not a workaround: commerce was re-published under
MIT OR Apache-2.0 and cut v1.50.0, and go.mod simply had not caught up.
Verified rather than assumed: `go build ./...` is clean against v1.50.0, and
v1.50.0 carries the same 1875 dirs and 8 billing packages as .67 including the
Cloudflare 502 work (thirdparty/cloudflare, api/billing) — the ~134 fewer .go
files are the tests/enterprise the OSS publish strips.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.832.24 stops seeding an admin/zen provider row that pointed at do-ai's base
https://inference.do-ai.run/v1. familyProvider reads that row as an operator
override of ZEN_URL, and the family paths append their own /v1, so every zen
catalog refresh hit .../v1/v1/models and 404'd — no zen SKU was listed or
served on api.hanzo.ai, once a minute, silently. Sibling enso was never seeded
and never broke.
The seed also self-heals ProviderUrl on every boot and re-creates a deleted
row, so this was not fixable in the database; v1.832.24 drops the seed entry
AND prunes the existing row, scoped to the exact stale shape.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
money.Amount.Minor() rescales, and hanzoai/decimal's Rescale rounds
HALF-AWAY-FROM-ZERO (decimal.go:145). Both balance call sites carried a
comment saying it "truncates toward zero". It never did.
So a balance of 4.995 USD read as 500 cents. hanzoai/ai's transaction check
is `avail < priceCents`, and apps/billing/gpu_charge.go is
`available < req.AmountCents` — both admit a 500-cent charge against a
balance that cannot cover it. The debit that follows is exact, so the
difference lands as a negative balance nobody authorized. Half a cent, every
time, with no error to find.
plane.Money.FloorMinor is that rounding made once, where the wire conversion
already lives, for every caller that COMPARES rather than debits. Minor()
refusing a sub-unit amount is still right for a debit and is untouched.
big.Int.Div is Euclidean, so a negative balance floors away from zero rather
than drifting back toward solvency the way truncation would, and the scale
comes from the currency exactly as Minor() takes it — cents are a property
of USD, not of money.
The guard that existed for this GREPPED ai.go for `a.Minor()` and for the
sentence claiming truncation, so it confirmed the wrong claim was still
written down. A test that reads the source cannot notice the sentence is
false. plane/money_test.go asserts the arithmetic instead — including the
4.995 case that was admitted, the real prod balance, and both signs. What is
left in apps/ai is the one thing only that package can say: that its gate
still asks for the floored figure.
Also corrects apps/billing/balance.go's comment calling this number "a
DISPLAY, nothing is billed from it". It is `available` on
/v1/billing/balance, which is what ai's balance gate reads over the S2S HTTP
path, and what gpu_charge.go compares against a GPU's price.
Verified: money.Amount.Minor()=500 vs FloorMinor()=499 for 4.995.
apps/billing's TestGPUCharge_* fail identically at origin/main -- they refuse
to run without a tmpfs (/dev/shm), which macOS has not; unrelated to this.
Deep research was the one thing the answer engine could not do. `research` planned
sub-queries, searched them ONCE, read six pages, and wrote. A question whose answer
is only visible after the first round of reading — which is what "research" means —
got a search with a longer prompt.
SURVEY is the value that was missing: search and read applied to a plan, and
ITERATED. Each round asks the model for one compact JSON move (`next`, `queries`,
`read`, `done`), runs it, and discards the prose that came with it — gathering is
not writing. No tool plane is needed or used, and a model that answers in prose
ends the survey instead of derailing it.
`rounds == 0` is the single pass, byte for byte. search/news are untouched: same
queries, same one-per-host set, same 90s, same 2c. One code path, parameterized by
a mode value — not a second engine, and not a second route. /v1/ask stays the one
door and `mode` stays a value handed to it.
Bounded four ways, because an unbounded agent loop cannot be gated by Bill.Gate
before it runs and an ungated loop on a per-org ledger is a money bug:
rounds the mode's budget, hard-capped at maxRounds=8
deadline per-mode wall clock (search 90s, research 300s)
tokenCeiling per-mode token spend (120k / 400k)
saturation a round that found no new source and read no new page
Saturation replaces the `len(srcs) >= maxSources` guard the design called for.
That test cannot do the job it was written for: rank() already caps the set at
maxSources, so it fires on the FIRST productive round and collapses research back
into the single pass the survey exists to iterate. "No new evidence arrived" is
the bound that was meant, it cannot be satisfied vacuously, and it also terminates
a model that keeps proposing a query it has already run.
Every exit lands on the same return, and the caller always goes on to synthesize
whatever was gathered — the envelope reaches `done` from every path.
The contract did not move. status | sources | text | follow_ups | done, four
stages, no fifth: the round's next step rides as `status{planning, detail}`, and
per-page reading progress as `status{reading, detail:host}` — which the union has
always declared and nothing used to emit. `sources` stays a CUMULATIVE snapshot
because all three SDK consumers replace their list on it.
Alongside, four things the port made necessary:
rank takes a per-host cap. One page per host is right for a six-source answer,
where breadth is the value, and wrong for research, where three pages from
an authoritative domain are the point. hostCap<=1 reproduces the old set.
rank cleans `[PDF]`/`(Official Site)` furniture off a title so a citation reads
as a document name — keeping the original when a title is entirely
bracketed, which would otherwise cite the page by bare hostname.
read takes the url list and the clip from its caller instead of a mode's top-N,
and reports progress per source. An iterated survey clips to 3000 runes:
many sources x a long page is the one way this loop could overrun a window.
synth streams through a joiner that holds a markdown link until its closing
paren, so a citation never renders as `[Rich Hickey](htt` and rewrite
itself. Delivery only — the finished text is identical.
research is repriced 10c -> 25c. It now gathers across rounds, decides between
them, and reads more than once; the price follows the work.
Not ported, deliberately: code interpreter, X search, chart artifacts, images, and
every external-SaaS leg (Tavily, Exa, Firecrawl, Notte, Supadata, Daytona). Search
is apps/websearch in-process and keyless; reading is apps/crawl, SSRF-guarded, with
its archive->fetch->headless ladder already standing in for Exa->Firecrawl->metadata.
Provider reasoning tokens are never mapped to `text`: every consumer appends delta
into answer and would corrupt both the answer and the persisted training sample.
Round decisions want temperature 0. cloud.ChatRequest carries no temperature field
and inventing one would fork the AI contract for a single caller, so the prompt
buys the determinism instead. Flagged, not faked.
Tests: survey_test.go proves rounds==0 is the old loop, that the gather actually
iterates (a later round searches and reads what round zero did not, and round
zero's page survives the re-rank), that the plan is carried verbatim into every
round, that snapshots are cumulative, that a round's prose never reaches the
answer, and that each of the five exits leaves the frame sequence intact.
stream_test.go adds the wire ordering invariant and proves the server never emits
the union's `error` variant.
v1.832.24 stops seeding an admin/zen provider row that pointed at do-ai's base
https://inference.do-ai.run/v1. familyProvider reads that row as an operator
override of ZEN_URL, and the family paths append their own /v1, so every zen
catalog refresh hit .../v1/v1/models and 404'd — no zen SKU was listed or
served on api.hanzo.ai, once a minute, silently. Sibling enso was never seeded
and never broke.
The seed also self-heals ProviderUrl on every boot and re-creates a deleted
row, so this was not fixable in the database; v1.832.24 drops the seed entry
AND prunes the existing row, scoped to the exact stale shape.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The rail's "could not generate a deposit address" was a 502, and Cloudflare
replaces an origin 502 with its own HTML interstitial — so the customer read
"request failed" while a clear JSON message sat at the origin, unreachable.
v1.49.68 answers 503 (which passes through, as the wire rail's own 503
already proves) with a sentence naming the rail, saying it is temporary, and
pointing at the alternatives.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`g := r.Group(p); g.Use(mw)` is the form that panicked — the group carried
middleware, the routes were registered on the root, and zip refused to compose
a guard that could never run. It is fixed, and it was the only one of the three
Use idioms with no test. Eleven production apps write it (books, captable,
commerce, company, compliance, dataroom, framework, git, legal, risk,
validators); the tests covered the other two.
Reverting scope.Group to hand back the raw router reproduces the original panic
verbatim and turns TestGroupThenUseGuardsOnlyItsSubtree red, which is the point:
asserting only that the panic is gone would also pass on a scope that silently
dropped the middleware, and a dead guard is worse than a panic because the panic
is honest. Each case asserts both halves — the guarded path answers 401, its
unguarded sibling answers 200 — across three subsystems, one with declared
prefixes and one nesting a group inside a group.
Addressing is asked separately from gating, because a gated route answers 401
whether or not it exists, so a test that asked both at once could not tell "the
route is where I said" from "the guard refused a 404".
And the residue is written down: a scope installs at the root, so its middleware
runs for what follows it. Use before the routes guards them; Use after them
guards nothing, and it composes either way because the root always has routes.
apps/company depends on exactly that — its fundraise/deck leaf sits above
g.Use(limitBody) on purpose, since the deck is document bytes and a JSON body
cap on it would be wrong. That deliberate exemption rested on an unwritten rule.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
INERT IN PRODUCTION. CLOUD_WRITER_LEASE is unset everywhere and this change
does not set it. With it unset every path here is a no-op, the lock file is
never created, and boot is byte-identical to what is running now. The env flip
and the strategy change remain a separate, human-gated step. Nothing to watch
on deploy.
The lease states a POD-level fact — "this pod owns this volume" — with a
PROCESS-level primitive, flock(2) on {DataDir}/.writer.lock. Those are the same
sentence only when exactly one process per pod reaches for the lock, and cloud
stopped being one process: cmd/cloud is a router that spawns every subsystem as
its own child, all sharing one DataDir.
The acquire lived in cloud.Listen — the body every PLUGIN binary runs, and the
one thing the router never calls. So turning the lease on aimed a single-holder
lock at the siblings instead of at the other pod. kms won the flock; pubsub and
kafka waited out the 90s fail-closed budget; nothing bound :8000; the liveness
probe killed the pod and its replacement deadlocked identically. api.hanzo.ai
served 503 from 08:52Z to 08:56Z on 2026-08-04.
The repair after that (aa416cc6) made the children skip the lock and stopped
the deadlock. It also left the lock in nobody's hands, because the router does
not run that code at all — an interlock that logs as armed and holds nothing.
That is the worse of the two states: the deadlock announced itself, whereas a
lock held by nobody is quiet until someone believes the log line and switches
to RollingUpdate. With S3_ADMIN_* armed the hydrate path renames over the live
DB, so the quiet failure is the expensive one.
So the duty is decided once, in internal/writerlease, from the only thing that
can distinguish these processes — their position in the tree:
Take the pod root, which takes the lock BEFORE it spawns anything and
releases it AFTER the last child is gone
Inherit a child, which is handed the answer and never contends
Off no lease configured — today, and correct under Recreate
cmd/cloud takes it before its mount loops and stamps CLOUD_WRITER_LEASE_HELD
with its own pid; zip already spawns children with append(os.Environ(), …), so
each is born knowing the volume is claimed. The stamp carries the pid rather
than a bare flag so a child CHECKS it — it counts only when it names that
child's own parent — which is what stops a stray value in a manifest from
talking a fresh pod root out of taking the lock. cloud.Listen makes the SAME
call: in the fleet it inherits, and run standalone (`hanzo kms` on its own
volume) it is a pod root and takes the lease itself. One rule, one reader of
CLOUD_WRITER_LEASE; the Config bool is gone, since a bool parsed per process
is true in all of them alike, which was the original misreading.
Also: Acquire rechecks that the inode it locked is still the file at that path,
because flock locks an inode and an unlink+recreate under it yields two holders
who both believe they are alone.
Tests run real processes, since the defect was a property of the process tree:
TestLegacyRule_SiblingsDeadlock reproduces the incident — 3 subsystems, one
DataDir, nobody above them: 1 serves, 2 spend the whole budget waiting for
a handoff that cannot come
TestFixedRule_SiblingsAllServe same topology, root holds first: all 3 serve,
duty=inherit, zero contention
TestFixed_VolumeStillDefended the one that fails against main as it stands
("a second pod OPENED the volume while this pod holds the lease") — proves
the fix did not simply disconnect the alarm
TestStampCannotBeForgedByConfig a hand-placed stamp cannot disarm a pod root
go build ./... green; go test ./... introduces no new failures (29 packages
fail on origin/main before and after, all unrelated).
Follow-ups NOT in this change, deliberately: flock is a lie on NFS/CIFS, so a
shared-filesystem refusal is still worth having (see blue/writer-ha); and
cto/writer-interlock-honest argues the exclusive-store premise is now stale,
which is a question about whether to keep the mechanism at all, not about
whether it should be correct.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Not crashing was not the fix. The ai router maps ANY non-2xx from
/v1/billing/tier to TierZenFree, so the 400 that replaced the 500 downgrades a
paying customer exactly as the crash did — 60 rpm against 500 for pro, silently,
with no error the customer or we would ever see. The tier has to RESOLVE.
Its caller is a service, not a person, and the two resolve an org by different
doors. IAMTokenRequired admits only a gateway-validated user identity (ownerID
AND X-User-Id AND email) and deliberately falls through on a bare X-Org-Id,
because admitting on that alone once let an off-gateway caller name any victim
org. The router sends precisely that shape — verified in ratelimit.go, a service
bearer plus X-Org-Id and no user — so it arrived with a nil org.
TokenRequired is the door for a caller that IS a credential: it verifies the
service token first and only then calls ensureIAMOrg to resolve the header.
Credential before trust. Same reason the catalog CRUD and the recharge poke each
carry their own TokenRequired rather than riding an IAM chain.
The commerce v1.49.67 handler fix stays and is still right: a tier that genuinely
cannot be read is refused rather than answered Free. This makes it readable.
The saved-card family had no way in. A customer could list cards and remove
one; adding required POST /v1/billing/methods, which the billing app
forwards to commerce's /v1/billing/methods — an address commerce does not
serve co-resident, and which the in-cluster `commerce` Service resolves back
to these pods, so the forward re-enters the forwarder. It never got that
far: CLOUD_COMMERCE_HTTP_URL is unset, so the proxy is unconfigured and
every saved-card call, GET and POST alike, answers 501 "billing is not
configured". Measured live on pay.hanzo.ai. Nothing could bill a monthly
plan to a card on file because no card could be put on file.
The portal face is where its siblings already live, so the POST goes there
too: no HTTP hop, nothing to self-dispatch, same gate. commerce's
CreatePaymentMethod vaults the Square nonce as a reusable card-on-file and
stores the billing address with it — that vault is what a renewal charges.
PinBillingSubject is load-bearing here rather than decorative: this handler
takes its subject from the BODY (customerId), and the pin rewrites the
subject keys there while preserving card/type/sourceId, so a caller can only
attach a card to its own account whatever the body claims.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The route registration landed in v1.801.440 and /v1/billing/tier still answered
500: the chain was never the problem, the handler was. GetTier type-asserted the
organization, and IAMTokenRequired resolves one only from a gateway-validated
user identity — so every S2S caller, which is this route's main caller, arrived
with nil. Fixed upstream where the handler lives, not worked around here.
Outside apps/iam no app file imports what a bearer is hand-rolled from
(crypto/hmac, a JWT library) or joins apps/team/token, except the pinned
entries: foreign auth wires (LiveKit, SigV4, webhook schemes, the mpc ring),
non-identity seals (OAuth state, unsubscribe links, a KDF), and the condemned
team token with its complete reader set, which only shrinks. The root's
deleted second authority failed permissively — key confusion, machine
principals as humans — and apps/ had no guard against growing another.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud/apps/<name> is plugin-mode wiring for github.com/hanzoai/<name>; the
product owns the functionality and ships its own standalone daemon. The
boundary now counts itself: an app either mounts its product or holds a pin
naming which debt it is (mismatched / unwired / unextracted), and the lists
only ratchet down. Measured at pin time: 140 apps — 16 mounted, 1 mismatched
(deploy mounts hanzoai/cd), 27 unwired, 96 unextracted.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The status page published three incidents for services that were up. Not one
bug — one structural defect: a hand-maintained address table drifted from the
fleet, and the prober faithfully published three wrong addresses. iam was
probed on a path it serves on a different port; kms was probed at a deployment
scaled to zero years after the fold into cloud; hanzo-mpc named a Service that
selects nothing.
fleetTargets is now the one registry of where a service answers, and both
availability reads take their address from it. iam is probed at the OIDC
discovery document — what its own readiness reads, and the first thing every
client fetches, so answering it means a customer can sign in. kms is probed at
cloud's embedded /v1/kms/health, which still fails closed without the master
key, so the API can be up while KMS honestly is not. hanzo-mpc is removed
rather than retargeted: the real ring answers 307 to every path including
absent ones — a probe that can only say yes trades a false outage for a false
all-clear.
The second address table is deleted. productmap synthesized addresses by
convention — port 80, try /health then /healthz — wrong for 19 of 27 products
and burning two timeouts per miss. An unwatched workload now has no address
and is probed not at all.
A failure names its target, address and reason, edge-triggered: one line when
it breaks, one when it recovers, and a changed reason reports again because
that is a different chase.
Verified from inside the live pod: the shipped list answers 20/20. A red test
turned green with it — the scoped-status read that blocked a full second
dialing a service that was never there.
/v1/search and /v1/vector belong to provisioning; product owns exactly
four routes inside them. Hanging requireKey on the two parent groups
claimed both subtrees, so the confinement gate refused the boot — and it
was right to: in the unified binary that middleware would have gated
provisioning's routes with product's key.
The credential is a REQUEST fact, so each op now declares it: keyedIn
carries the Authorization header as a typed input field (zip's stated
replacement for exactly this middleware) and requireKey opens every
handler. Same statuses in the same order — unset key 503s, wrong key
401s, search and vector keys never cross — and the four addresses are
byte-identical; the document gains only the header parameter each op
always required but never published.
Verified: apps/product suite green, bin/product boots (was the one
compose failure in 120), openapi weave green with no path added, moved
or removed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every program now emits its request log natively — method, path, status,
duration, trace and span, the caller when the environment parked one — through
its own logger, so the per-app Logger install is deleted along with the three
test copies of it. The framework also propagates trace context and serves
/metrics from the one registry.
Kept deliberately: the OTel meter pipeline (metrics_http.go, installMeter) and
the span path. The datastore pipeline they feed is what /v1/summary and
availability READ, and the framework's export has not yet been proven to land
where those readers look. Two instruments briefly is safe; a blinded status
page is not. The compiler found the callers a text search missed — the span
middleware records the RED metrics — which is exactly why they stay until the
export path is measured.
Also carries luxfi/metric v1.9.1 transitively pinned by zip — the registry
that records by default.
failRedirect is the one place a failed OAuth return lands, and it logged
nothing, so every failure was indistinguishable from a request that never
arrived. It now logs the opaque public reason together with a precise
internal cause; the browser still gets exactly one reason, so no oracle is
offered to a caller probing states.
verify() wraps errBadState with that cause. errors.Is still matches, so
control flow is unchanged. The distinction it makes visible is between a
state signed by a different key -- what happens when the signing key is
absent and each boot invents its own, breaking every flow that spans a
restart -- and one that simply expired.
Folds the two ad-hoc warns into the funnel, carrying org in the cause so
no detail is lost.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The module-side registrations (commerce v1.49.65+) serve nothing in the
fleet by themselves — commerce's api.Route() bundle is never compiled here,
and the host hands an app only the prefixes its manifest row names. So the
four rails get what every self-service billing address before them got:
co-resident registrations on the pinned-subject IAM chain, and their
prefixes named deeper than the bare /v1/billing stem nobody claims.
- GET /v1/billing/wire — the serving brand's receiving bank
details, caller's billing key in the payment reference (attribution is
why it sits on the pinned chain; an unpinned wire is unattributable).
- GET /v1/billing/crypto/options — chains+tokens from the live MPC
processor; the pay SPA's asset picker renders exactly this.
- POST /v1/billing/crypto/deposit — per-payer custody address via the
signer fleet's /keygen; payer is the PINNED subject, never a body value;
open intents are reused so a refresh cannot spray keygens.
- GET /v1/billing/crypto/deposit/:id — caller-scoped intent state.
Nothing mints on any of these: wire settles via the admin wire/credit verb
on bank receipt, crypto via the chain watcher on real confirmations.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
geometry is a closed sum and family derives from the geometry's own type, so
"half-space parameters attached to a transformer" does not compile and this
package carries no check for it. family leads the content address, domain
separated, because every term after it is one family's arithmetic — two
families' numerically identical masses can no longer be named as one value.
The detector seam is the six methods the learner already calls. The half-space
counters are the first implementation and behave as before. legacy() is the one
remaining braid, at the disk boundary, and refuses a family the resume column
cannot carry rather than recording a shape for a model that is not one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.49.65's crypto rail called an MPC API the deployed fleet never served
(vaults routes, all 404). v1.49.66's GenerateAddress speaks the live
luxfi/mpc contract: POST /keygen {"org_id"} → all-chain addresses in one
threshold keygen. Env to arm it: MPC_ENDPOINT + MPC_API_KEY (KMS-synced
into commerce-secrets).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestTheThreeAddressesAreToldApartNotShared has been red on main since
"identity is the composer's" moved cloud.Bridge out of every subsystem and into
cloud.App, which installs it at the root ahead of every typed route. surfaceApp
was not updated, so it mounted o11y onto a bare zip.App with no Bridge — and a
typed op with no Bridge has no validated org on its context, so it answers 403
to everything, including the RED read this test asks for its OWN datastore
refusal.
That reads like a live outage of the o11y surface and is not one: the real
process composes through cloud.App. A harness that stops composing the way the
process does does not measure the process, and this one was reporting a
middleware it never installed as a broken handler.
scopeApp already says exactly this, one file over. surfaceApp is where it was
missed. Package goes green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
event.trace is the summary the read plane resolves a trace id against, and it
is fed by the SPAN WRITER — one partial row per trace per batch, folded by an
AggregatingMergeTree — not by a materialized view. hanzoai/o11y's own driver
does exactly this (pkg/datastoretraces/writer.go, traceSQLTmpl). This sink is
the production span writer and implemented event.span without it, so the
summary stayed empty while 710k spans piled up beside it.
An empty summary is not a missing feature, it is a silent outage.
TraceTimeRangeFinder resolves every trace_id predicate against this table
first, and a lookup that returns no row makes the querier SHORT-CIRCUIT THE
TRACE QUERY TO EMPTY (pkg/querier/builder_query.go, narrowWindowByTraceID) —
a missing summary row means "no spans exist", which is the one thing it did
not mean. The detail read, the waterfall, the flamegraph and every funnel
answered empty over a complete span table.
So: traceRowsOf folds the rows the span writer ALREADY built — one row-building
path, not two, with the four column indices pinned by a test so a reorder goes
red instead of corrupting the summary. end is max(start+duration), not
max(start): the longest span need not start last. The summary write fails SOFT
— a derived rollup must never be able to take down the fact ingest it is
derived from.
And GET /v1/o11y/traces, the org's trace LIST: the one address in the family
the module leaves open. It declares the detail (/traces/{traceId}), the field
catalog and three per-trace projections, every one of which needs an id this
read is where you get — the detail was reachable only by someone who already
knew the answer. Claiming the collection and nothing under it keeps that a
composition rather than a second declaration at an address that has an owner.
Typed, declared at the ROOT at its full path so zipdoc can resolve it, org
pinned as a bound parameter on the table's leading sort key, no admin
widening: a trace list is a tenant's records, not a rollup over them. It
aggregates on read because a merge is asynchronous and never a promise —
skipping that reports a BATCH as a trace.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Brings GET /v1/billing/wire (brand receiving-bank details, payer-attributed
reference), GET /v1/billing/crypto/options and POST /v1/billing/crypto/deposit
(per-payer MPC custody addresses) onto the served billing surface, plus the
$5 pay-as-you-go floor (topupBounds min 100→500 cents). v1.49.65 is the merge
of the forge lineage (zip v1.24.2, these rails) with the GitHub lineage
(dual-license, billing payment/invoice cores) — both remotes now converge.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The gate refused ANY apps/* import from cmd/cloud as a stand-in for "links no
subsystem graph". The stand-in went wrong the moment an app's EDGE had to run in
this process: cmd/cloud/sites.go serves <slug>.hanzo.app, and it must, because
the image runs this binary on the public port — a published site mounted anywhere
else is served nowhere, which is the outage that put it there.
MEASURED on origin/main: cmd/cloud links 392 packages, inside the ~395 the gate's
own prose names, and apps/sites pulls 2 cloud packages. Nothing grew. The gate has
been red for a day over a graph that never changed — and because every later car
declares `needs:` on it, that is the entire release train stopped by its own
approximation. Live is v1.801.431 while tags reach v1.801.436.
So the exception is NAMED with its reason, the discipline go-unit's -skip list
already follows, and the property it approximates is measured directly beside it.
Two checks, because they are two questions: the name catches a subsystem leaking
in, the count catches an ALLOWED leaf that quietly grew a graph. A widened pattern
would have answered only the first and silently given up the second.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud.App(name, cfg, deps, tools) now builds every program — the host and all
125 plugin programs — and installs the canonical chain through identity in one
place. Identify states the order invariant once: the boundary that mints the
validated org runs before the enrichment that parks it, and neither exists
without the other. Middleware order at the host is proven byte-identical
before and after (18 entries), so a refused request is still audited.
Every subsystem self-install of the enrichment is deleted — 74 sites across 67
packages. A subsystem asserting identity for itself repeats a claim it cannot
check; two of those copies sat on nodes owning no routes, which zip refuses to
compose, and that is the outage that took /v1/o11y and /v1/integrations down.
The childless spelling is gone everywhere: a gate passed TO Group is installed
at the root bounded by its prefix, and the two-step form that slipped past
that (a bare Group then Use on it) is collapsed at its last holdouts —
referrals' three gates and the audit fixtures.
Tests stop rebuilding the composer by hand: each package that mounts on a
bare app owns one compose helper, so anonymous cases still refuse and
principal-carrying cases reach the handler exactly as production does.
Also fixed, found by the constructor's own test: the markdown negotiation
replaced the Vary header CORS had written, so every CORS response on every
program advertised Vary: Accept — a shared cache could hand one origin's body
to a different origin. The later writer is additive now.
Census: zero non-test enrichment installs under apps/, zero childless chains.
Build and vet clean over the whole tree in the shipping mode.
The workspaces + members tables were 443 lines of hand-written database/sql:
positional placeholders, a hand-rolled scanWorkspace whose Scan matched wsCols by
POSITION, and a prefixed() that rebuilt the projection as a string. They now go
through hanzoai/orm's relational plane — orm.Select/orm.Typed for the reads, the
dbx builder for the writes.
Not orm.Model. That is the document plane over the JSON _entities table, and
these tables are typed and indexed with composite uniqueness ((owner_org, slug),
(workspace_id, user_id)) that a json_extract store cannot hold; orm's own guidance
names this file as a dbx target for exactly that reason.
The handle still comes from cek and orm is ADAPTED onto it. orm's SQLiteDBConfig
carries no master key, so orm.OpenSQLite would have written this store — every
workspace, membership and display name — as a plaintext `SQLite format 3` file.
That is the regression apps/iam documents removing, and it is not reintroduced
here: encryption at rest is a property of who opens the file.
The positional Scan is gone with it. wsCols and the struct now agree by `db` tag,
so a column added without a field is a mapping that does not resolve rather than a
silent shift of every value one position left.
THREE STATEMENTS STAY VERBATIM, each because the builder cannot say the thing that
makes it correct, and each now running through orm's own NewQuery rather than a
database/sql handle — so the file has ONE data path:
migrate — CreateUniqueIndex emits neither IF NOT EXISTS nor a WHERE, so it can
express neither the idempotence nor the PARTIAL (owner <> '') index; dbx.Sync
writes no indexes at all.
the EnsureWorkspace create — dbx's Upsert only ever emits DO UPDATE SET, and
there is no seam for a conflict target carrying the partial index's WHERE. As a
DO UPDATE the meaning INVERTS: the racing loser would overwrite the winner
instead of yielding to it, and the converge-to-one-workspace property is gone.
AddMember — Upsert fans EVERY inserted column into the SET list, so a re-invite
would overwrite joined_at and is_bot. joined_at is the order GuestRank ranks by
and the guest cap admits by, so that is not a cosmetic overwrite: it silently
reshuffles which guests keep access.
TestAddMemberPreservesJoinOrderAndBotFlag pins that last one at the four columns
the statement treats differently. Swapping in db.Upsert turns it red on three of
them (joined_at, is_bot, display_name) — checked, not assumed.
The test fixtures moved to the builder too, so no `?` placeholder survives in the
package.
apps/team + apps/analytics + apps/meet green, CGO_ENABLED=0 -tags sqlite_fts5,
against the ten pre-existing zip-compose failures already red on this commit's
parent.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zen composing was never the same as zen running, and only the first had been
fixed. It is Coresident: it answers no path and works by wrapping ai's "/v1",
routing zen-SKU requests and Next()ing the rest. A middleware cannot be a
separate process, so the light host deliberately skips Coresident apps
(cmd/cloud mount returns early) and plugin/zen exists only for the gen-app-cmds
bijection. plugin/ai linked only ai. The Claim was therefore mounted in NO binary
the fleet runs — which cmd/cloud had already recorded as "zen's child never saw a
request" — so every zen SKU served through ai's catch-all with zen's gate and its
meter never consulted.
zen mounts FIRST in plugin/ai, ahead of the greedy All("/v1/*") ai registers:
zip scopes middleware to the entries that follow it, so a Claim installed after
that catch-all would sit behind the very route it exists to gate.
MEASURED, on the real zen.Mount through the real MountAll:
zen5 -> 402, reached host catch-all = false (claimed and gated)
gpt-4o -> 200, reached host catch-all = true (falls through untouched)
Both halves matter. The first is the gate doing its job; the second is the proof
zen did not become a wall in front of the whole model API.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.4.4 billed every streamed call from an accumulator no chunk had touched.
c.SendStreamWriter hands its closure to fasthttp, which spawns it on its own
goroutine (stream.go:43) and returns immediately, so the recordCtx that followed
read usage.tally() before a byte arrived: completion -> 0, cached -> 0 (the whole
prompt re-billed as fresh) and ResponseID -> "", which is the join key /v1/feedback
and the learning ledger need to tie streamed traffic to its own feedback.
Reproduced here before bumping, on the exact shape: `go test -race` reports the
write from the writer goroutine against the handler's read, and the value visible
at metering time is 0 where the streamed total is 50. It is not merely racy — the
closure genuinely has not run, so a mutex would silence -race and change no number.
Both same-dialect paths were affected (proxy.go stream, sse.go anthropic-native);
streamTranslated, buffered and ultra meter synchronously and were always correct.
The drift runs BOTH ways, which is why this is not just our lost revenue:
answer-heavy shapes under-collect 62-85%, and cache-heavy short answers OVERCHARGE
a live ledger by up to 4x.
v1.4.9 was cut for this — the fix existed on zen's main and sat in no released
tag, so every consumer was still on the defect. MEASURED at the boundary:
v1.4.8 fails `go test -race` at proxy.go:509; v1.4.9 exits 0 with no races.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The rebase produced a program that does not compile — `undefined: sv1`, so the whole
apps/commerce test binary failed to build, a package that was GREEN on main.
Both sides fixed the same zip v1.24 composition rule and picked different halves of
it. This branch dropped the /v1 group and moved the middleware to Use; main kept the
group and moved the ROUTE onto it, so the chain guards something. The auto-merge took
this branch's deletion of `sv1 := app.Group("/v1")` together with main's
`sv1.Get("/store/current", ...)` that still names it.
Main's shape is the better statement of what the test is for: the case is an envelope
that must apply to commerce's own route and must NOT clobber a sibling's, and putting
commerce's route back on the group is what production does (Mount's storeV1 group).
So this file returns to main's version byte for byte — `git diff a4a1f998 --
apps/commerce/errorscope_test.go` is empty — and the branch keeps no opinion about it.
No commerce money logic is touched, on this branch or in this commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The zipdoc -check gate at the top of `make test` was red on three packages, so the
suite aborted before a single test ran. The lift is regenerated from source; all 99
directories carrying the directive now answer -check clean.
What the regeneration is, measured rather than assumed:
marketing prose only. Operation set identical, field-doc set identical.
account prose, plus doc comments for onboardResp.accessKey/accessSecret.
projects prose, plus a doc comment for projectsProject.key, plus the lift for
POST /projects/resolve-key (an internal cloud.Plane op whose file
landed without a regeneration).
The extra entries are DESCRIPTIONS for fields that already exist: AccessKey and
AccessSecret are account.go:604-605 and Key is projects.go:159, all three already
carrying the comments lifted here. zipdoc supplies prose; a schema property comes
from struct reflection at describe time. So this adds documentation to the surface
and no field to it.
THE PUBLISHED ARTIFACTS ARE DELIBERATELY NOT INCLUDED. Regenerating
plugin/{account,marketing,projects}/openapi.json from this same source does NOT
come out prose-only, and it is not this commit's business to land it quietly:
- plugin/account: onboardResp gains accessKey + accessSecret as published
properties, and THREE operationIds are renamed — post_v1_orgs -> v1.post_orgs,
post_v1_keys -> v1.post_keys, delete_v1_keys -> v1.delete_keys.
- plugin/projects: projectsProject gains a published `key` property.
The rename is zip's scheme change, not ours, and the fleet is mid-migration: the
committed account subset still carries the old form while marketing and projects
already carry the new one. Every SDK is generated from these files, so completing
that migration renames methods for callers and is one deliberate fleet-wide act
with an owner behind it, not a side effect of unblocking a test gate. The same
change is what TestTargetOpsProjectEverywhere has been red about (it asserts
post_v1_agents_targets and zip now emits v1.agents.post_targets).
Verified identical on a pristine a4a1f998 worktree, so none of the drift above is
this branch's: it is the published document catching up to source that already
landed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
main answered the red ratchet by RECORDING apps/risk/policy_wire.go in
allowedRequestUses (f345ac70). This branch answered it by removing the second call
site instead — caller() moved beside ops.gate, which already reads that same
X-User-Id header, so the package reaches for the raw request in one file and the
pin stays one entry.
Both landed, so the rebase left both: the entry describes a call site that is no
longer there. The gate catches that in the direction people forget — it walks
allowedRequestUses and fails on any file that no longer calls cloud.Request, so a
pin cannot outlive the code it justifies. Removing it is what makes the ratchet
green, not an exemption.
Net effect against main: the escape hatch does not grow. apps/risk/typed.go states
both reasons on the one entry it already had.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/networks answered 403 "X-Org-Id required" to a validated caller. Bridge was
installed on the group, twice — ng.Use and mg.Use — and middleware on a group wraps
what is composed beneath it. The two collection ROOTS are declared on the App with
their whole path on purpose: joining "/v1/networks" with an empty leaf yields
"/v1/networks/", a different address from the one they have always served. So the
group's Bridge covered /v1/networks/routers and /:id and missed /v1/networks itself,
the op read no parked org, and the tenant gate refused a request that was fine.
One Use replaces both. On a scope it is bounded by the prefixes the manifest already
declares for zt (/v1/networks and /v1/mesh/services), which is every route here and
nothing else; on a bare app — what this package's tests mount on — it is app-wide,
which is what the tests want. The groups stay as what they are, path prefixes.
TestBridgeIsInstalledOnEveryPrefix already stated this and had been red: it asserts a
validated caller is SERVED on all four routes and an anonymous one still refused.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two more of the shape the previous commit fixed in nineteen places, found by the
gate rather than by reading: apps/deploy panicked the fleet's own surface-check
(`the group "/v1/deploy" declares middleware at deploy.go:316 and no routes
anywhere beneath it`), and apps/agents failed 54 of its own tests.
deploy is the clearer one. deploy.go built app.Group(dashPrefix) to hold
cloud.Bridge + bounce, and dashboard.go builds ANOTHER app.Group(dashPrefix) to
hold the routes. Group returns a new definition per call, so those are two nodes
at one path: the middleware sat on the empty one and NEITHER the bridge NOR the
sign-in bounce ever ran for a single route of that surface.
agents is the same mistake with a subtler tell, because its group is not empty —
/metrics, /activity and the :ref leaves ARE beneath it, so nothing panicked. The
rest of the surface is not: the collection root, mountSessions and mountTargets
all register on the Router by absolute path. So Bridge parked no org for
/v1/agents/targets or /v1/agents/sessions, and every op under them answered 403
"X-Org-Id required" to a request that carried one.
It only ever showed up in tests, and that is the part worth keeping in mind:
Serve installs a Bridge app-wide, so serving was unaffected and only a bare Mount
could see the hole. TestHTTPTargetRejectsOversizeGPUList is the example — it
asserts 400 for an oversize GPU list, got the 403 first, and so had never once
exercised the bound it is named after. It passes now for its stated reason.
No bound widens. Every prefix either subsystem declares is under its own group's
path (manifest/apps.go), so a scope confines this to exactly what the group named
and the plugin binaries serve nothing else.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestRequestEscapeHatchIsPinned was red on main: apps/risk/policy_wire.go called
cloud.Request from a second file in a package that already had one, and the pin knew
nothing about it.
The fact caller() needs is the validated user id (X-User-Id) a policy version is
recorded against, and nothing parks that in the context — principal parks the org and
validated-ness, so principal.OrgFrom and principal.ValidatedFrom cannot answer it, and
an attribution the caller could state in a body is not an attribution. So the request
is genuinely required. What was not required is a second call site for it: ops.gate,
in typed.go, already reads that exact header for the meter's actor.
caller moves there, beside gate. Two functions, one seam, one pin — which is the shape
the map's own entries describe for wallets, tools, deploy, usage and o11y, and the
reason it is a per-FILE list. The pin's existing apps/risk/typed.go entry now states
both reasons; no entry is added, and the escape hatch does not grow to make a test
green. The function itself moves byte for byte.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nineteen sites carried the same defect scope.Use had, at the other door: a group
created with middleware, its routes registered on the App with their whole paths.
Middleware on a group wraps what is composed BENEATH that group, so every one of
those groups was empty — the middleware never ran, and since zip v1.20 the program
does not compose at all, which is a panic at mount for the plugin and an aborted
test binary for the package.
TWO GATES HAD SILENTLY STOPPED RUNNING, and that is the part that matters more than
the panic. apps/referrals put requireOrgOnWrite on a /v1/referrals group and
requireAdmin on each of the two /v1/admin/referrals boards, and registered all three
leaves on the App: the write gate never ran on POST /v1/referrals/claim and the admin
gate never ran on EITHER board. apps/team installed Bridge on one /v1/team group
while every file builds its own group from the same constant — the same routing
subtree, a different definition — so the bots, files, blob and cookie planes ran with
no validated org and answered 403 to valid requests (eight tests, all of which were
unreachable behind the package's own panic).
The identity and error-shape middleware moves to Use, which a scope bounds to the
prefixes the manifest declares for the subsystem and which is app-wide under a bare
Mount — admin, admission, auditlog, catalog, guide, integrations (x2), label,
marketing, prefs, projects, team, templates. referrals keeps three different bounds
and states each as the predicate a group prefix was standing in for (under), the same
shape apps/commerce already uses for its error envelope. o11y installs on its own
App, which its routes are already beneath. In team the install moves ABOVE the group
it used to sit on: fiber runs middleware in registration order, so one added after a
subtree never wraps it.
Five test harnesses mirrored the invalid shape on a bare app and move the same way.
No address changes and no gate widens: every bound here is the one the group prefix
named.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The door description still promised "NO CREDENTIAL IS ALSO ADMITTED ... filed
under the reserved $public tenant" — and it goes into the OpenAPI document
customers read. The code has refused since the key work landed: 401
ingest_key_required with nothing presented, 403 ingest_key_unknown for a
credential that names no project. A contract that promises the opposite of the
code is worse than no contract, because a client writes against it.
The projection survives for the reduced principal it was narrowed for — a
workspace token writing into its own org.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The escape-hatch ratchet has been failing on origin/main, naming
apps/risk/policy_wire.go and then apps/commerce/invoices.go. Both are
legitimate; neither was recorded, so the guard could not tell them from a new
one and main stayed red.
A red main is not a cosmetic problem here. Three of today's near-misses got as
far as they did because "the tests pass" had stopped meaning anything, and the
next reader has to re-derive whether each failure is theirs. So the fix is to
answer the pin, not to loosen it.
apps/risk/policy_wire.go caller() reads c.User() for an attributable policy
record. No ctx helper answers it — OrgFrom gives
the tenant, not WHO changed the appetite bounds.
Off the HTTP path it returns empty and plane.enact
refuses, so a change is never recorded anonymously.
apps/commerce/invoices.go eventsFrom/kmsFrom lift two request-scoped side
channels out of c.Locals(), which no ctx helper
exposes. Both optional by design: a missing
analytics collector must never fail a money move,
and a missing KMS client is the dev/test posture.
The root package is green again.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
event.fact is not a rename of event.event, it is a merge — five signals in one
table. Moving the source without naming the signal counted errors and spans as
device observations, and this is the one cross-tenant reader, so a phantom
identity inflates both k-anonymity floors: 425 identities unfiltered against
423 real ones on the live plane.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
config.go read CLOUD_CORS_ORIGINS and fell back to GATEWAY_CORS_ORIGINS, the
pre-rename gateway spelling, "shared with the gateway so both trust boundaries
agree on one list". Measured across every chart in universe, GATEWAY_CORS_ORIGINS
is set ZERO times — by cloud's chart, by the gateway's, by anyone — so the shared
name agreed on nothing. cloud's own values file sets CLOUD_CORS_ORIGINS.
Two spellings for one security-relevant allowlist is a way to half-configure it:
set the dead name and the browser silently gets no ACAO. No test pinned the
fallback, and the direction of failure on removal is restrictive — an origin that
is not listed is refused, never admitted.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
docs/consolidation.md was the execution queue for merging the standalone Go
services into one binary. Nothing references it, and every mechanism its Method
section instructs the next wave to use has since been deleted:
apps/apps.go no such file
apps.Wire no such symbol
cloud.Register no such symbol (only unrelated seam registrars remain)
"order < 150" there are no orders; manifest/apps.go is the table
clients/<svc>/Mount an app is plugin/<name>/main.go, one binary per app
Its inventory has drifted the same way — "37 native apps/*" against today's 120
manifest rows, three of the apps it names (paassvc, console, prompt) no longer
exist, "Wave 1 — THIS build" shipped long ago, and its CGO_ENABLED=0 "production
parity" rule is the opposite of what the Dockerfile does for plugins
(CGO_ENABLED=1 + libsqlite3 + sqlite_fts5 + sqlite_math_functions). A doc that
tells the next engineer to write code against a registration mechanism that does
not compile is worse than no doc.
The durable half — which tiers stay out of this binary and why — is not lost: the
edge/data-plane split is in README.md, and the isolation reason for identity is
stated where it binds, in apps/iam/iam.go. If that table is wanted as a fleet-wide
statement it belongs in LLM.md, which is the one doc this repo maintains, not in a
migration queue.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
scripts/smoke-runtime.sh said it was "intended for CI smoke gating", and CI
never called it — nothing in this repo, in .hanzo/workflows, in the Makefile or
in universe named it. The role it claimed is already served twice over, by
things that ARE invoked: `make smoke` runs plugin/smoke, the release gate's
functional prober, and `make e2e` runs e2e/run.sh, which builds the host and its
plugins and boots them for the Playwright suite.
It was also the weakest of the three. Five hardcoded probes against
plugin/smoke's one read per subsystem, with none of the contract that makes that
prober worth running — no 402-on-a-read rule, no 5xx rule, no authed/tolerant
distinction. And it reached for `curl -fsS`, the flag that turns a failing probe
into a silent one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
migration/ + plugin/migrate-pg-to-sqlite ported a `hanzo_cloud` PostgreSQL
database into per-org SQLite. That database belonged to the deployed cloud-api
PYTHON service, which this repo replaced; the runbook the tool cites
(CLAUDE_PG_TO_SQLITE_MIGRATION.md) no longer exists, no Makefile target, CI job,
compose file, chart or sibling repo invokes it, and it is absent from
manifest/apps.go so the image never built it. Its only two mentions were the
generator's exemption list and a comment naming it as an example. A one-shot
import for a service nobody runs is not part of v1.
The bijection test derives tool-ness by PARSING for a cloud.Listen call rather
than matching a name list, so it needed nothing; only gen-app-cmds' notApps map
did.
LLM.md's release section described an architecture that was deleted with the
fused binary: a `hanzo.yml binaries:` lane publishing ./cmd/cloud to S3, every
app resolving to "the multi-call binary with a different --enable", pinned by
TestRemote_DedicatedBeatsMultiCall. hanzo.yml now states NO binaries lane and
gives the reason; remote() looks up exactly name/os/arch with no multi-call
fallback; and the cited test was replaced by TestRemote_NoMultiCallFallback,
which asserts the opposite of what the doc claimed. The line numbers it quoted
had come to land on the comment saying the lane was removed. Corrected to what
the code does, and CLOUD_PLUGINS named for what it is: a supported input with no
producer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The catalog shipped 66 entries and 45 of them pointed at repositories that do
not exist. github.com/hanzo-templates/<slug> 404s for all of them, authenticated
or not, and their demo URLs 404 too — a gallery where two thirds of 'fork this'
leads nowhere.
The work was never missing, only misfiled. Those templates live in hanzo-apps
under a template- prefix, which is the org convention for a static site; the
catalog was written against a hanzo-templates layout that only 21 of them ever
moved to. 66 template-* repos in hanzo-apps, 66 entries here — the sets match.
21 kept real in hanzo-templates (expo-*, flutter*, swiftui*, desktop-*)
42 fixed repointed to hanzo-apps/template-<slug>
3 dropped innovise, kalli, unfixed — no repo in any org, and a catalog entry
that cannot be forked is worse than an absent one
All 63 remaining sources verified 200 against the GitHub API before this landed,
not assumed from the naming rule.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The plane's product-event table was renamed event.event -> event.fact. Writers
moved; these four readers did not, and nothing errored because the old table
still exists — it just stopped receiving rows on 2026-08-02. A frozen table
answers every query, so the failure was stale numbers, not an outage: the GTM
funnel was still reporting $13.37 of revenue from July.
event.fact holds every signal in one table (act, clip, error, log, span), so
the rename alone is not the fix. Each read pins signal='act', the predicate
apps/analytics already carries in scope(). Without it the funnel counts 85
visitors where 83 acted, and risk folds a person's error rows into things
they did.
risk binds the signal as a literal rather than a positional arg: all four
rollups share one 5-arg storeExec call, and the SQL already spells 'person'
and 'session' the same way.
Verified against the live warehouse — every query returns growing data where
it returned frozen, and apps/risk tests pass.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/o11y went native and now declares all 367 of its routes by name.
There is no wildcard left for a host route to shadow, so the three
addresses cloud also declared stopped being a silent precedence artefact
and became a refusal to compose — the plugin crash-looped and every
/v1/o11y read 503'd.
o11y.Claimed(...) made it boot, but a claim SUPPRESSES the module's
declaration. It did not resolve the collision, it just picked a winner and
wrote the choice down: each of the three quietly cost the fleet the
module's real read at that address, and one of them was cloud's forward
into a runtime path that no longer exists. What the addresses needed was
to be told apart.
GET /v1/o11y/logs — DELETED. Not a stub (it was a real two-view read
over event.log/event.span) but it had NO caller: the console reaches
logs through the query engine, and nothing in cloud calls it either.
An address nobody calls is not a contract. The module's real
log-record read answers there now.
GET /v1/o11y/metrics — MOVED to /v1/o11y/product/metrics. The module's
is the metric-NAME CATALOG; ours is one product's RED window keyed by
?product=. Two questions, so two names — ours says which one it
answers, and the module keeps the bare name. Live in the console
(App Platform drawer + canvas sparklines); patched there on the same
branch.
POST /v1/o11y/query_range — DELETED, along with POST /v1/o11y/query and
the whole of query.go. Both pinned /api/v3/<resource> to reach the v3
engine. The runtime registers every route at its full public path and
dropped prefix-stripping, so it serves no /api/* route at all, and
queryRangeV3 has no caller left — the forward reached the terminal /*
catch-all, not an engine. A route that forwards to an address nothing
serves is dead. The module's v5 querier answers there now.
Cloud therefore claims nothing: o11y.Mount(a) takes no options and the
boot log reads claimed:0. A host that has to name the addresses it takes
is a host that took addresses it did not own.
STILL OWED, and it is a console change: the trace/log explorers send a v3
composite that v5 refuses with unknown field "queryType". They were
already broken before this (the v3 pin reached the catch-all), so nothing
regressed — an opaque non-answer became an honest 400. The v5 migration
is recorded in apps/o11y/LLM.md.
The route table is unchanged in size: 389 method+path pairs before and
after. Only two lines move — /v1/o11y/product/metrics arrives and the dead
POST /v1/o11y/query leaves — because the module reclaimed exactly the
addresses cloud vacated.
openapi.yaml is the whole-fleet weave and cannot be regenerated per app,
so it also catches up on two other lanes' already-committed subsets
(billing +5 paths, payments +2) that it was stale against.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
IAM's error envelope rode on HTTP 200, so an SDK checking the transport before
the body read a refused signup as a successful one. The envelope is unchanged;
the status now agrees with it.
cloud's own IAM client is unaffected by construction: iamClient.do parses the
envelope whatever the status and decides on status != "ok", so it reached the
same verdict before and after. The bump is what makes the two agree at the wire
as well as in the body.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The unclassified branch rendered err.Error() verbatim as the 500 body, so a
customer's first fault told them about ZapDB migrations, CLOUD_KMS_MASTER_KEY_REF,
http://iam.hanzo.svc and features that are "not yet implemented". That text is
written for an operator; putting it on the wire published our internals to
whoever tripped it.
The rule is provenance, not status. An *HTTPError or a *fiber.Error was
constructed by a handler that chose a status AND a sentence — a decision the seam
does not second-guess, which is what keeps "Billing temporarily unavailable"
readable. An error that chose neither now renders a stable sentence naming the
request id instead.
The detail is not lost, it is relocated: ErrorHandler logs every 5xx whole, with
method, path and the X-Request-Id the response carries, so the log line and the
response a caller holds identify each other. The wrapped chain reaches the
operator, which is more than the client ever saw, since the client only ever got
the outermost sentence.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Federated sign-up files a brand-new user under the sign-up application's own
organization (iam internal/oidc/federation.go: `org := app.Organization`), so
the first request a new customer ever makes already carries an X-Org-Id. onboard
read that as "already has an org": the first org they asked for took the
ADDITIONAL branch, which creates an org and leaves the founder outside it, and
`personal: true` answered 409 "you already have an organization" — true of the
landing org, and useless to someone thirty seconds past signing up.
The orgs a sign-up can land in are exactly the ones this package already refuses
to hand to a customer (onboarding.go's reservedOrgs). One list, one fact, asked
twice: an org no customer may CREATE is one no customer can be said to OWN.
Naming the set rather than a single brand constant keeps a white-labelled
deployment correct, where the landing org is that brand's own.
Standing beats the landing. A SuperAdmin IS a member of the reserved `admin`
org, so treating that as a landing and moving them out would strip the privilege
it defines; an org admin therefore always counts as owning their org, read from
the authoritative IAM row rather than a header, since the answer is the one that
MOVES a user. A caller already in a real tenant is spared the read entirely, so
an invited member creating a second org is never yanked out of the team that
invited them.
The provisioning response now carries the credential it minted. IAM stores the
argon2id digest and blanks the plaintext, so the secret is readable exactly once
— in the answer to the call that mints it. Dropping it left a customer holding
an account whose credential had been issued and could never be obtained; a
replay, which mints nothing, reveals nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Merging the three analytics branches leaves publicTenant named where it no
longer exists: the heatmap tests normalize under it, and five comments describe
a lane the key work deleted. The tests take a real org like the rest of the
file; the comments say what the door does now — the org is the reduced
principal's own, resolved from its credential.
apps/reference read event.event, which stopped receiving rows on 2026-08-02
while the plane moved to event.fact. Columns are identical, so the device
aggregate was grouping a frozen table.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The golden is DERIVED from plugin/*/openapi.json, and the merge of origin/main
into forge/main brought a subset the forge-side golden predates: 1969abfc gave
POST /v1/runner's description its release: true SUPERADMIN paragraph. Weaving
the committed subsets reproduces every other byte, so the whole delta is those
three lines of prose.
No route moves — 1695 paths before and after, none added, none removed, and the
floor ratchet is untouched at 179 products. TestFleetIsTheWeaveOfItsApps and
TestTheServedDocumentIsTheArtifact both read this file and both go green.
Only the weave step ran. `make describe` also regenerates the subsets from
code, and that step is broken on BOTH mains independently of this merge — forge
fails first in zipdoc on apps/o11y/annotation_queues.go (which origin's dc013156
fixes), and past that the authors app is refused for installing middleware at
/v1/admin/authors, outside the /v1/authors prefix it owns. Neither is this
merge's to decide, and neither touches the derivation performed here.
zip removed App.Shadow — the verb that let an op be declared in one scope and
answered in another. Cloud never referenced it, so nothing here changes shape.
o11y v1.5.58 carries the same removal plus one narrowing of under().
Proven route-neutral: the woven fleet document is byte-identical before and
after (1695 paths, 3676621 bytes, cmp clean), so no SDK repo sees a route move.
Suite: 161 packages ok, 143 failing tests, and the failing set is byte-identical
by NAME to main's — regression set empty.
`release: true` on POST /v1/runner publishes releaseImage — ghcr.io/hanzoai/cloud,
the binary every service in every org runs. The gate read
principal.IsSuperAdmin(c) ||
(imageInOrgRegistry(releaseImage, org) && principal.IsOrgAdmin(c))
and the second term is not a narrower platform predicate, it is a self-service one
wearing a registry map. `isAdmin` is org-scoped and an org's OWN admin sets it on a
member of THEIR org, so every `hanzo` admin could enrol any `hanzo` member into the
gate — and `hanzo` is the deployment's brand org in every deployment, and the one
orgRegistryNamespaces gives `hanzoai`. So the far side of the gate chose its own
callers, and what it chose them for was the tag the whole fleet rolls onto at the
next reconcile: iam, kms, gateway and every customer app, in every tenant.
Owning the registry namespace is the right bound for an ordinary PUSH, and it stays
there. A push lands ONE tenant's artifact in the namespace that tenant owns, so the
caller's own org is exactly what should bound it. A release lands OURS on everyone,
so no property of the caller's own org can be what admits it. The two lanes part
company at that one fact, and nowhere else on this endpoint.
SuperAdmin <=> `owner == "admin"` is the ONE platform predicate, so the gate is now
cloud.Super and nothing conjoined to it:
func mayRelease(c *zip.Ctx) error {
if !cloud.Super.Admits(cloud.AuthorityOf(c)) { ... }
}
Validated is part of that scope, which collapses the MACHINE arm into the same
expression rather than a second rule beside it: PLATFORM_BUILD_CALLBACK_TOKEN mints
no principal, so a leaked build token still enqueues an ordinary build and still
cannot cut a release — the property TestRunnerRelease_SharedTokenCannotRelease
already pinned, now held by the scope itself.
READING a release took the same org-scoped gate (mayReadReleases), mirrored by
comment. Both are now the one function, so "the 202 hands back an id its caller may
ask about" is true by construction. Nothing was widened: the published contract for
both GETs already read "SuperAdmin only — this is the platform's own publishing
record, not a tenant surface", and the code did not do that. The /v1/runner prose
said cutting a release was "IAM's decision alone", which was true of the credential
and silent on the scope; it now names SuperAdmin.
The production release path does not go through this gate at all — a merge to
cloud's own main dispatches launchRelease in-process (push.go, isReleasePush), so
what tightens here is only the hand-cut, and the CLI cannot even send the flag
(cli.BuildReq has no Release field).
TestReleaseSurfacesTakeOneAuthority drives the REAL handlers across all three doors
— cut, list, read-one — over the role axis, and asserts they agree principal for
principal. Against the parent the brand-org ADMIN case fails with the escalation
verbatim, on every door:
cutting a release: admitted=true, want false (HTTP 502 — resolve main: ...)
listing releases: admitted=true, want false (HTTP 200 — {"data":[]})
reading one release: admitted=true, want false (HTTP 404 — no such release ...)
The 502 is the tell: the seams are stubbed to 500, so a request only reaches the
pipeline by having been authorized. It holds the SuperAdmin row on all three doors
too, so merely disabling the path would not pass, and TestRunnerRelease_-
OwningOrgAdminRefused pins the cut on its own.
It replaces TestReadingAReleaseIsNotStricterThanCuttingOne, which read runner.go
and release.go as TEXT and asserted both mentioned the same predicate NAMES.
Spelling was all it could ever see: it was green while both surfaces contradicted
the published contract, and it would have stayed green had the two admitted
different callers under the same names.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every release since v1.801.427 has failed. The image build runs
'go generate -run zipdoc ./...' and it exits 1:
zipdoc: apps/o11y/annotation_queues.go:94:10: zip.Get: cannot resolve the path
prefix of the router this op registers on
That is why seven plugins — templates, integrations, admin, guide, marketing,
prefs, o11y — answer 503 'no instance running' in production while the fix for
them sat on main unable to ship.
The cause was under{}, introduced to declare o11y's typed ops at the ROOT so
their ids stay unqualified and their schemas keep the app's origin. The goal is
right; the mechanism was a composite literal, and zipdoc resolves a router's
prefix only from a .Group() call or a variable assigned from one. A custom
OpTarget is invisible to it, and zipdoc treats what it cannot resolve as an error
rather than assuming an empty prefix — correctly, since a wrong prefix files
prose under the wrong identity.
Same intent, statically visible: register on the app with the full path,
o11yPrefix+"/reviews". The router is the *App (root, no prefix) and the path is a
constant expression the type checker folds, so zipdoc reads both. Declaration
stays at the root, so the ids and schema names under{} protected are unchanged.
Addresses are byte-identical — the regenerated zipdoc_gen.go changes by pure
ADDITION: GET /v1/o11y/sessions is documented now, because zipdoc could not reach
it before. under.go is deleted; nothing else used it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A project mints a publishable key at create — the ONE thing that attributes its
site's events. The key resolves to (org, project), so it names the site as well
as the tenant, and a key no project holds resolves to nothing: delete the
project and recording stops, structurally.
Removes the two attribution paths that were not that:
- the keyless lane. A beacon carrying no credential was ACCEPTED into a
reserved `$public` tenant and answered {"accepted":1}. No org could read
that partition, so every such caller lost everything it sent behind a 200.
Three first-party properties shipped keyless without one failed build.
handle now refuses: 401 ingest_key_required with no credential, 403
ingest_key_unknown with one that names no project.
- the site-host carve. A POST to <slug>.hanzo.app routed into the anonymous
lane with a host-derived tenant — a second mechanism, and the one that could
not be checked, since that middleware runs before the identity boundary. A
site's beacon carries its project key to the ingest door instead.
The projection, its bounds and the opt-out gate survive for the reduced lane (a
team guest writes into the org that invited it, at reduced capability).
CLOUD_ANALYTICS_PUBLIC_CAPTURE gated the deleted lane and is gone with it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Group handed back the raw zip router, so `g := app.Group(p); g.Use(mw)` walked
past every gate in this file and hung middleware on a node whose subtree is
necessarily empty — typed ops register through ZipApp, on the ROOT, so a
subsystem's routes never land beneath the group. zip v1.24 refuses to compose
that, correctly, and crash-looped fifteen plugins on v1.801.425/.426.
This file's own header said that idiom "needs no policing: the group already
bounds it". It bounds the MIDDLEWARE and says nothing about where the ROUTES
went, which is the half that mattered.
Group now returns a child scope, so all three idioms are ONE install — at the
root, gated by path — and no node is left that can be empty. The child carries
`at`, its path prefix, because a group PREFIXES what is registered through it:
without that, `zip.Get(app.Group("/v1"), "/bots", h)` (bots, entitlements)
registers /bots — a route silently MOVED, which still composes, so no compose
check could have caught it. OpScope carries the same prefix for the same reason.
Also, three subsystems that were escaping their bounds silently, because a bare
Group used to skip the check entirely:
- team answers /collaborator; the manifest says so and the plugin did not.
- label answers /v1/risk/labels; same.
- zt installed one bridge at /v1/mesh, a level ABOVE the only path it
serves there. One app.Use, gated by scope to what zt declares.
And OwnsHealth on authz/domain/experiments/metrics, which serve their own
health — so serve.go's generic liveness route was a second declaration of it.
Verified three ways, because a weaker check let a broken build reach production
twice today. The binaries were exiting on `mkdir /var/lib/cloud/orgs: permission
denied` BEFORE composing, and I read that silence as a pass:
1. survival — a compose panic is fatal, so rc 124 under timeout is the only
honest signal; "zip new" is logged before composition and proves nothing.
17/17 affected plugins survive, each with private ports and a writable dir.
2. route projection — `describe` diffed before vs after across 34 plugins:
34 identical, 0 changed. deploy/label/referrals now project where they
previously panicked. This is what caught the moved-route defect above.
3. go test -run 'Scope|Mount|Prefix|Route' green.
o11y is NOT fixed here: its three routes (logs, metrics, query_range) are
duplicate declarations against upstream o11y@v1.5.55, and zip dedupes on the
resolved path, so moving the node cannot help. Separate change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The anonymous lane projected the property bag down to the @hanzo/observe
annotation and dropped everything else, so a $click crossed it naming WHICH
element was clicked and never where on the page it sat. Element identity cannot
be drawn as a heat map, and logged-out traffic is the bulk of what one is made
of — so the position survived only on the signed-in lane, for a minority of
clicks.
The position is the second declared family. It does not get the annotation's
free ride: nothing lifts these into a column, so each really does add a key to
the attributes dictionary. What bounds it is that the set is CLOSED and spelled
by this server — five keys, never a caller's own vocabulary, values that are
numbers and a boolean, so the dictionary grows by five and stops.
boundedPosition filters rather than clamps, for the same reason
boundedAnnotation does: a clamped coordinate is a click somewhere the visitor
did not click, and a heat map is a picture of exactly that.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/event.js is the install path for a surface with no bundler:
<script defer src="https://api.hanzo.ai/v1/event.js" data-key="pk-…"></script>
One line, and the same line for a Hanzo property and for a customer's own
page. It autocaptures pageviews (initial and SPA) and uncaught errors onto
the canonical {batch:[…]} wire, carrying the key as a bearer on fetch and as
?ingest_key on the sendBeacon drain that cannot set a header. Identity uses
@hanzo/event's own storage keys and session TTL, so a page carrying both
clients resolves to one person.
WITHOUT A KEY IT SENDS NOTHING. The keyless beacons this replaces named
their site in a body field and carried no credential, so their events were
accepted 200 into $public — a reserved tenant the owning org cannot read.
The tenant comes from the publishable key, never from a body, so an unkeyed
page is silent rather than reporting success into a tenant nobody reads.
It is served beside the door because a tag that drifts from its wire is a
tag that 400s: asset and ingest ship in one binary and version together.
openapi grows the response half it had deferred until a route asked: Bytes
declares a body under the media type the handler sets, mirroring Binary on
the request side, so the document states JavaScript instead of claiming
JSON. Register copied the body half field by field and would have dropped
it; the copy now carries it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip refuses a program whose middleware has no routes beneath it, and the refusal
was right about three surfaces here. Nothing in this repo called Build() from a
test, so the refusal could only ever surface as a startup panic — or, for a
subsystem no test drove, as silence.
auditlog and catalog declared Bridge (and audit's noStore) on a group at their
own prefix while declaring the op on the App with its WHOLE path. The op is
therefore that group's SIBLING, not its child, so the group's subtree was empty
and the middleware could not run. Both packages' entire test suites were
panicking out of app.Test, which builds. Use is the verb that says the true
thing, and it needs no second copy of the prefix: cloud's scope bounds it to the
subsystem's declared subtrees, and a bare *zip.App treats root middleware as
live. The ops keep their exact paths — moving them onto the group with an empty
leaf would publish /v1/audit/ and /v1/catalog/, which neither API has served.
zen was worse and not the same defect. Its Claim gates ai's "/v1", it declares no
prefix of its own (Coresident), and plugin/zen fed manifest.PrefixesFor("zen") —
a ROUTING answer — into cloud.Plugin.Prefixes, which is the MIDDLEWARE grant. One
field was answering two questions, so dropping "/v1" from zen's row (right, for
routing: it duplicated ai's claim) silently revoked the gate, and MountAll refused
the mount outright. manifest.App.Gates states the second fact where the first
cannot, GrantFor reads it, and TestGrantMatchesPrefixesForRoutedApps keeps it from
becoming a second list.
crm was the same failure wearing an ordering convention. Its intake limiter used
to cover "everything registered after this line" — the public form plus the three
staff routes. Under lexical scoping it narrowed to the one route chained onto it
and the staff routes lost cover with no error anywhere. The covered routes are
composed BENEATH the limiter now, which states the coverage instead of implying
it. TestIntakeRateLimitScope was red before this and is green after.
compose_build_test.go is the gate that was missing: Build() through BOTH routers,
because MountAll hands a scope that can HIDE a seam a bare app refuses. It also
pins the rule itself, so a zip that stopped refusing the shape could not let the
seam rot back in green.
Deps: base v1.5.15 (the sqlite_math guard is a probe now, so `go build ./...`
needs no tags) and zip v1.24.2, plus iam v1.34.19 and o11y v1.5.57 restored —
394ba2e8 downgraded all three and left go.sum missing commerce v1.49.64, so
origin/main did not build at all.
guide's blueprint assertion matched a flat tool-naming zip no longer uses, so it
selected nothing; it matches the subject now, not the scheme.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
394ba2e8 said "base v1.5.15" and did not bump base. What it actually did was
roll THREE dependencies BACKWARD:
iam v1.34.19 -> v1.34.18
zip v1.24.2 -> v1.24.1
o11y v1.5.57 -> v1.5.56
base v1.5.11 unchanged <- the only change it advertised
Cause, so it is not repeated: `go get` was run against a working tree that was
BEHIND origin/main. go get pins the versions it is handed and leaves the rest at
whatever the tree already had, so every dependency a concurrent lane had already
advanced got written back to the older pin. The commit message described the
intent; the diff recorded the accident. Nothing verified the two agreed.
This restores all three and lands the bump that was missed:
zip v1.24.2, base v1.5.15, iam v1.34.19, o11y v1.5.57, commerce v1.49.64
Each is the newest tag on its remote, read with `git ls-remote` — GOPRIVATE
means proxy.golang.org cannot serve hanzoai/*, so the proxy is not an authority
for these and `@latest` silently answers from a stale public view.
base v1.5.15 is the one that pays for itself immediately: v1.5.11 carried a
guard whose symbol was defined nowhere, so `go build ./...` failed at default
CGO_ENABLED=1 and every build in this repo needed -tags sqlite_math_functions by
hand. Verified after this change: `go build ./...` with NO tags exits 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestMergedAndChainedAreIndistinguishable... pinned a fact so that a zip bump
which changed it would fail here. A zip bump changed it, so it failed — this is
the test working, not breaking.
The old fact: a duplicate registration and a middleware chain produce the SAME
observable route, so the generator must never read the handler count. From zip
v1.24 the duplicate half is not constructible at all — a second registration of
one pattern is REFUSED at composition time instead of merged, and the refusal
arrives as a panic out of Registry(), which took the whole package down before
any assertion could run.
The conclusion is unchanged and now rests on something stronger: a handler count
above one can ONLY be a chain, because a collision can no longer reach the
registry. Both halves are still pinned — the chain projects to exactly one
operation, and the duplicate is still refused — so if either moves, the
generator's assumption is forced back into review.
The chain's shape is now MEASURED rather than assumed. zip registers a GET as
GET plus an automatic HEAD companion, so one registration is two route entries;
the old test asserted one and would have failed on that alone. The GET is what
carries the chain and the HEAD is fiber's own, dropped from the projection.
openapi.yaml is regenerated because this branch adds six payment/invoice path
keys. Nothing is removed and no other app's operations move.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`tools/list` at api.hanzo.ai answered 570 tools and not one of them could take
money. Searching payment / charge / checkout / invoice returned nothing; the only
"pay" hit was an admin payout. An agent could incorporate a company, issue its
cap table, open a data room and sign a contract, and then had no way to be paid
for any of it.
The rails were never the problem. Square is live in the commerce module compiled
into this binary — official SDK, real charges, per-org KMS credentials, first in
the fiat priority list — and POST /v1/billing/topup/token has been charging cards
through it all along. What was missing was SHAPE: that route is a raw
func(*zip.Ctx) error, so it is a route and nothing else. No registry entry means
no OpenAPI operation, no MCP tool, no SDK method, no CLI command. The same was
true of the whole invoice lifecycle, of which this binary mounted only the list
and the PDF — an org could read invoices it had no way to create.
So commerce v1.49.64 lifted the money logic into cores that take values instead
of requests, and this adds seven typed ops over them:
takePayment POST /v1/payments
getPayment GET /v1/payments/:id
raiseInvoice POST /v1/billing/invoices
getInvoice GET /v1/billing/invoices/:id
issueInvoice POST /v1/billing/invoices/:id/issue
collectInvoice POST /v1/billing/invoices/:id/collect
voidInvoice POST /v1/billing/invoices/:id/void
They are TYPED, which is the whole point: each publishes a real JSON Schema with
per-field prose lifted from the handler's doc comment, down to a $defs for an
invoice line item. A payment tool with no parameters is useless to an agent — it
can be listed and never called — and this estate already has 469 untyped writes.
These add none.
THERE IS STILL ONE WAY TO TAKE A PAYMENT. Every op delegates to the same core
commerce's own HTTP handlers now delegate to, so the server-side amount bounds,
the idempotency derivation, the processor selection and the ledger credit are
shared rather than reimplemented. A second charge path would be a second set of
bounds to drift and a second idempotency key to disagree, which is a double
charge waiting for the right retry.
IDENTITY IS NEVER AN INPUT. The paying org is read from the validated principal
cloud.Bridge parks on the context; there is no org field and no subject field on
any of these ops, so a caller cannot steer money to an account it did not prove.
Mode is not an input either — sandbox versus live follows the org's credentials,
and the answer STATES which bucket it credited, so a sandbox receipt can never be
read as live money. Tests pin both: the schemas must not publish org/subject/test,
and every op must refuse a caller with no validated org rather than defaulting.
Also fixes errorscope_test's own empty group node — it declared middleware on a
/v1 group and registered every route on the app, so from zip v1.24 the group
guarded nothing and Registry() panicked before any assertion ran. The store route
moves onto the group, which is what production does and what makes the test a
valid program; its address and every expectation are unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
base v1.5.11 carried a guard referencing cgoBuildNeedsSQLiteMathFunctions, a
symbol defined NOWHERE — its only mechanism was the compile error it produced.
So the plainest command in Go failed in this repo and in every other importer of
base/core, reporting a missing symbol rather than the actual problem, and the
`-tags sqlite_math_functions` workaround had to be passed by hand on every
build. It was also a false negative in the config the driver's own docs call
production, because the tag was a PROXY for a capability rather than a
measurement of it.
v1.5.14 replaced the guard with a probe that asks the ENGINE — it runs the real
expression once against a throwaway in-memory DB and refuses the connection when
the answer is no, so the check reports a measured absence instead of a failure
to measure. v1.5.15 carries that plus zip v1.24.2.
Verified here: `TMPDIR=... go build ./...` with NO tags exits 0.
zip v1.24.2 and commerce v1.49.63 were already on origin/main when this landed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Fifteen plugins refused to compose and every route they own answered 503,
including /v1/event — the telemetry front door.
zip v1.24 refuses a group that declares middleware with no routes beneath
it, because that middleware would silently never run. Six apps wrote
app.Group("/v1/x").Use(mw) and then registered their routes on the app at
full paths, so the group node was always empty. f28bf6bd fixed exactly
this for scope.Use; these six bypassed it by reaching for Group directly.
app.Use is now the one way: scope.Use already gates a subsystem's
middleware to the prefixes it owns, by path, so the group node is not
needed and cannot go empty. One verb, one rule, no second mechanism.
Verified by running all sixteen affected plugin binaries: 0 compose
panics.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
tools/list proves an agent is TOLD a data room can be opened. It does not prove
one can open it, and the gap between those is a real failure mode with its own
shape: over MCP there is no URL, so zip passes the arguments object as the body
with a NIL path map, and an op whose address reaches it only from the path is
addressable over REST and NOWHERE else. Every existing test here drives HTTP,
where the path always binds, so all of them would stay green while the agent got
not-found — and three of these ops carry an id.
So the demo is driven the way the agent will drive it: open a room, grant a party
access naming the room by argument alone, read it back by id, and list it. The
negative half is what makes the positive half mean anything — the same read with
no id must NOT resolve a room, or 'found' proves only that something answered.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Tranche D listed dataroom 17. It is 10 typed and 7 refused now, and the note
records the two things a later reader needs: that the shared bundle-backed kit
(Scalar, SizedIn, BundleErr, Envelope) moved to apps/goja on its SECOND use
rather than becoming a second copy, and that ScalarList exists because a wrong
type on an access-control list is a SILENT failure — the room discards a
non-array and reports success, so a link meant for one investor would admit
everyone.
The tranche's remaining-count column is left alone: it already disagrees with
its own row (104 declared, 51 remaining after pricing/ml/automations/compliance
were struck without it), so reconciling it here would be a second lane's edit
wearing this one's commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip v1.24.2 stops composition renaming a DECLARED operation id by the prefix it
is included under. That mattered here more than anywhere: this binary is the host
that includes o11y under /v1/o11y, and doing so was renaming 217 of its 353
declared ops — every cached MCP tool name, operationId, CLI command and generated
SDK method moved as a side effect of one wiring line.
With commerce v1.49.63, iam v1.34.19 and o11y v1.5.57, every subsystem this
binary mounts is published on the same framework version. That agreement is the
point: Router is the type a decorator implements, so a host on one version and a
subsystem on another is a decorator that cannot be written.
Measured against a clean tree on this host: 97 failing packages before, 97 after —
zero new. macOS SQLCipher, unrelated and unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/dataroom/* has served fourteen routes and reached no agent. An untyped route
appends no op to zip's registry, and the MCP door renders tools FROM that
registry — so the fleet answered tools/list with 570 tools and not one of them
opened a data room. The hole was never the mount (dataroom mounts, and
/v1/dataroom/health has been answering in production); it was that every route
was an untyped relay.
Ten of them are typed ops now — every JSON route on the admin surface, which is
the whole demo surface: open a room, put documents in it, grant a party access,
and list what exists. Each still relays the bundle's own (status, body); what
changed is that it now carries an In and an Out, so the same declaration yields
the tool, the OpenAPI operation, the SDK method and the CLI command.
The package doc claimed NONE of these could be typed, on two premises the shared
kit answers: that a relayed answer is opaque (it is not — the bundle's shapers
are total and schema.go types them, which is what the models are), and that a
typed error path would overwrite the bundle's envelope (it does not — BundleErr
carries the bundle's status and BYTES). captable had already disproved both;
this makes that the second use rather than the second copy, so Scalar, SizedIn,
BundleErr and Envelope move to apps/goja, beside the bundle seam they serve.
ScalarList is the one piece captable did not need. A bundle substitutes an EMPTY
list for anything that is not an array, so an agent told allowList is a `string`
sends one, the room discards it, and the call SUCCEEDS having ignored the access
control — a link meant for one investor admitting everyone, reported as success.
Declaring the array is what puts that failure out of reach.
Four routes stay relays for reasons in the wire, each named at its registration:
the upload takes the file itself as the body, the two /file routes answer with a
byte stream, and the three public viewer routes have no principal to read.
Proven: the demo flow end to end over the typed routes; the reads byte-identical
to the bundle they replace; the cross-tenant link index still written, so a
granted link still opens for an anonymous visitor; org scoping; the room's own
refusal envelope intact; and the ten tools present with descriptions and schemas.
The regenerated captable subset is operationId-only (40 lines, 0 schema changes)
— pre-existing drift between the committed spelling and what zip v1.24.1 derives,
corrected by regenerating from source rather than by hand. The same drift had
left one captable test asserting a tool name nothing produces; it now asserts the
derived one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two independent ways a host WE run could serve a tenant's site. Both are the same
mistake: a set or a test that answers "is this ours" was allowed to shrink.
FIRST — the self-domain set was a default, not a floor.
self := list("CLOUD_SITES_SELF_DOMAINS")
if len(self) == 0 { self = defaultSelfDomains(apex, domain) }
An explicit list WON OUTRIGHT, so an operator adding a vanity domain silently
subtracted the derived ones. That set is the sole unconditional gate on the
site_hosts table (projects Store.bindHost asks sites.Ours) and the serve gate's
self-host exclusion, so dropping hanzo.ai makes every `<label>.hanzo.ai` a tenant's
to claim — a first-come row on api.hanzo.ai that denies us our own production host
for good, verbatim the defect SetSelfDomains was added to close. Nothing would have
caught it in review either: hanzo.ai reaches the set by DERIVATION from CLOUD_DOMAIN,
so no deployment states it and no deployment diff would show it leaving. The
reserved LABELS have had this floor all along ("trimming the env only ever ADDS ...
never subtracts"); the half guarding the more dangerous decision did not. Config now
adds to selfFloor and can never subtract from it.
In practice the brand domain also arrived a second way, via FirstPartyApex, so
BOTH had to be misconfigured before it actually vanished — which is why the new
test moves the first-party apex to prove the floor holds on its own.
SECOND — requestHost decided "is the parsed host real" by asking "is it a host we
would SERVE": siteSlug, else customCandidate. Those are different questions, and
the gap between them is exactly OUR OWN domains. api.hanzo.ai names no site, and
customCandidate excludes it BY DESIGN (IsSelfHost), so neither arm fired and the
client-supplied X-Forwarded-Host won:
Host: login.hanzo.ai
X-Forwarded-Host: <any bound custom domain>
resolved and served that domain's site — a tenant's content returned for a request
addressed to our auth apex, needing no site_hosts row on hanzo.ai at all. The
file's own comment already stated the property correctly ("a request that HAS a
host ignores the header completely"); the code did not have it. The two tests that
look like they pinned it both send a host that IS a site, so the early return fired
and the header was never read — they proved the property only where it already
held.
Whether we serve a host has nothing to do with which host was asked for, so
requestHost no longer asks: a parsed name with a dot is a hostname and is final.
That is the same set the three-way test actually admitted, minus the coupling that
made our own names the exception — and it leaves the header exactly the job it was
added for, the ingress case where fiber parses no host at all, plus bare internal
names no client addresses us by.
TestForwardedHostNeverOverridesOurOwnHost and TestSelfDomainsAreAFloorNotADefault
both fail on the parent:
login.hanzo.ai: the binding resolver was asked about [attacker.example]
hanzo.ai held only via the first-party apex — it must come from the floor
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Bind, verify and release must agree on what a hostname IS, and they did not. Bind
required fqdn.Valid; verify and release required only fqdn.Clean, and release did
not even require that much beyond non-empty.
site_hosts holds TWO shapes: a project's custom FQDNs, and its BARE SLUG — the
structural row deploy.go binds so `<slug>.<apex>` serves. A bare label is not Valid
(nameRE wants labels, a dot and a TLD), so release accepted a row bind could never
re-create. The domains panel renders that row like any other claim, as a live
`https://<slug>`, with a delete control beside it. A tenant tidying away the
odd-looking entry drops its OWN subdomain, and the domains API cannot put it back:
resolution falls back to ResolveUniqueLiveSlug, which refuses once two live
projects share the slug, so the subdomain 404s for everyone — and the next tenant
holding that slug to deploy takes the freed row, and the subdomain, for good. An
add-only asymmetry in a first-come global namespace is a transfer primitive.
hostOf is now the ONE reading of a hostname off a request — canonical form, then
the syntax this surface deals in — and all three ops ask it. A blank entry inside a
bind LIST is still skipped rather than refused; that is a list semantic, not a
disagreement about names.
TestReleaseOnlyAddressesNamesBindCouldHaveMade drives the real DELETE at its real
path. Against the parent it fails with the takeover verbatim:
release of the bare slug = 204, want 400
EXPLOIT: the site's own subdomain row was DELETED through the domains API
It also holds the positive case — a real custom domain still releases 204 and the
row goes — so refusing everything would not pass.
TestVerifyDomainPromotesOnlyOnProof closes a coverage hole rather than a defect:
Store.VerifyHost was tested and fqdn.Verify was tested, but the HANDLER that joins
them had no test at all, so nothing pinned that this surface requires the proof. It
injects a fake resolver — the projects package had none, so any test that reached
verifyDomain would have hit real DNS — and drives no-record, wrong-token,
right-token, already-verified, unclaimed and un-addressable in one pass. The token
is read from the ROW, never from the request, and that is what the wrong-token case
pins.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both halves of an authorization comparison must be the same value. The
registry-ownership lookup folded one of them:
orgRegistryNamespaces[strings.ToLower(strings.TrimSpace(org))]
while the org it compares came from principal.Org, which returns the validated IAM
owner VERBATIM and never lowercases — deliberately, because "acme" and "ACME" are
DISTINCT tenants in IAM and a fold would let a member of one select the other
(principal.go; TestMembershipMatchIsByteExact pins it).
So a tenant who self-serves an org named `Hanzo` — a different owner from `hanzo`,
and one whose own RoleOwner makes IsOrgAdmin true INSIDE it — folded onto the
`hanzo` key and inherited the `hanzoai` namespace. That is both lanes at once:
push over another brand's production images on the build path (runner.go
imageInOrgRegistry, repoOwnerInOrg), and satisfy the org term of the RELEASE gate
on ghcr.io/hanzoai/cloud, the binary every pod in the fleet runs. A fold applied to
one side of a comparison is not a normalization, it is a collision, and here the
collision IS a cross-tenant privilege grant — the same defect 26f69224 removed from
the custom-domain operator set, in a lane with the fleet downstream of it.
ownedBy is now the ONE lookup, keyed verbatim, and both callers ask it. Trimming
stays: whitespace is not an identity. repoOwnerInOrg keeps EqualFold on the OTHER
side — the forge owner parsed out of a repo URL, which genuinely is
case-insensitive — because that side is not an identity this system issues.
TestRegistryOwnershipIsVerbatim drives both lanes over both directions. Against the
parent it fails with the grant verbatim:
tenant "Hanzo" folded onto a brand's REGISTRY namespace — it could overwrite
that brand's production images and cut a release of the binary the fleet runs
tenant "Hanzo" folded onto a brand's FORGE owner
It holds the positive cases and the cross-brand refusal too, so narrowing the map
to nothing would not pass.
NOT addressed here, and flagged for review rather than changed: the release gate
admits `imageInOrgRegistry(releaseImage, org) && principal.IsOrgAdmin(c)` at all.
runner.go:255-267 argues that deliberately and at length — publishing your own
org's artifact is not cross-tenant — but the artifact is the platform's own binary,
`hanzo` is a customer org like any other, and its admin bit is one that org's own
admins set on their own members. fleet.go:254-261 reaches the opposite conclusion
for the mutation next door and gates it on cloud.Super. Overturning a documented
decision in this app is a call for its owner, not for this change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Binding a custom domain WITHOUT proving control of it — the bind lands VERIFIED
and routes immediately, and the "a host we operate" refusal does not apply — is
platform authority. The gate read
principal.IsSuperAdmin(c) || (operatorOrgs[org] && principal.IsOrgAdmin(c))
and that second term is not a second platform predicate, it is a self-service one
wearing config. `isAdmin` is org-scoped and an org's OWN admin sets it on a member
of THEIR org, so every `hanzo` admin could enrol any `hanzo` member into the gate
— and operatorOrgs defaults to the deployment's brand org in EVERY deployment
(config.go getenv CLOUD_BRAND, brand.Default "hanzo"). Naming the org in config
does not repair that: config grants a capability TO a tenant, but the tenant still
decides who inside it holds the role, so the deployment ends up delegating a proof
bypass to an authority it does not administer. A gate whose far side can enrol its
own callers is not a gate.
SuperAdmin <=> `owner == "admin"` is the ONE platform predicate. A second, weaker
spelling of platform authority IS the escalation, whatever conjunction dresses it
up, so vouches() is now that predicate alone:
func vouches(c *zip.Ctx) bool { return principal.IsSuperAdmin(c) }
It is cross-tenant by construction, so it still vouches in ANY org — the operator
switched into a customer's org to bind the domain it manages DNS for, which is how
a customer domain is onboarded. That path is unchanged and stays pinned.
Everything the org term reached goes with it: state.operatorOrgs,
operatorOrgsFromEnv, and CLOUD_PLATFORM_OPERATOR_ORGS. The variable appears
nowhere in the universe manifests, so no deployment states it and none needs
editing — a deployment must never state a variable nothing reads. The gate now has
no configuration at all, which is also why a test can no longer under-configure
it: there is nothing left to pass, so the tests drive exactly what production runs.
TestVouchIsSuperAdminOnly drives the real handler over the ROLE axis in the
deployment's own brand org. Against the parent, with the state Mount actually
builds — operatorOrgsFromEnv("hanzo"), the default {hanzo} — the brand-org ADMIN
case fails with the exploit verbatim:
EXPLOIT: brand-org ADMIN bound a bank's login host with NO proof:
{Host:login.example-bank.com Status:live Verified:true
URL:https://login.example-bank.com Records:[]}
It holds the SuperAdmin case too, so merely disabling onboarding would not pass.
TestVouchDoesNotTurnOnTheOrg keeps the ORG axis the verbatim pin covered: four org
names spanning the classes that used to matter — the brand org, its case fold, an
unrelated tenant, and the reserved `admin` org's own NAME — each asserted twice,
org-admin never vouches and SuperAdmin always does. The fold collision that pin
existed for is now dead by construction rather than by comparison, and THAT is the
property it now pins.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The table declares every route it owns by name, so there is no wildcard left
for a host's own route to win against, and two declarations at one address is
a refusal to compose rather than a silent shadow. cloud declares three of them
natively — GET /v1/o11y/logs, GET /v1/o11y/metrics, POST /v1/o11y/query_range
— and used to win them only by registering first.
o11y v1.5.56 gives that fact a spelling: Claimed names the addresses the HOST
serves, and the table then declines to declare exactly those. Say it here.
It is worth being explicit about WHY these three are the host's, because the
ordering that used to decide it decided it invisibly. This package's handlers
are tenant-scoped: handleLogs resolves the caller's org and pins the read to
it. The table's relay hands the call to the runtime with no org on it. Both
answered the same address, the first one registered won, and the two could
have drifted apart forever without anyone being told. The claim is that
decision written down where it can be read — and read the same way on both
sides of the seam, since the string that names the conflict in zip's own
diagnostics is the string that resolves it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/integrations and /v1/connectors answered 503 for the same reason o11y did:
the subsystem exited before it listened, because it declared middleware on two
group nodes that owned no routes.
app.Group("/v1/integrations").Use(cloud.Bridge(), zip.H(bridgeFacts))
app.Group("/v1/connectors").Use(cloud.Bridge(), zip.H(bridgeFacts))
Every op below both lines registers on zapp at an ABSOLUTE path, so neither
group ever received a leaf. A group's middleware wraps the routes in its own
subtree, and an empty subtree means the bridge could never run — which zip
refuses to compose rather than serve ungated.
scope.Use is the install that already solves this: once at the root, gated by
`owns`, confined to exactly the prefixes the manifest declares for this
subsystem. The manifest lists /v1/connectors and /v1/integrations both, so one
install covers what two group nodes were reaching for, and connectorRoutes
needs no bridge of its own.
The test fixture had drifted from the thing it reproduces. installV1Flatten
still hung commerce's error envelope on a Group("/v1"), while commerce itself
moved to gating by path (commerceErrorScope) precisely because a shared /v1
node wraps every subsystem mounted after it. Same shape, same empty subtree,
same refusal — it aborted the package on origin/main before this change. It
now installs the way its twin does.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The subsystem's ops were declared through a.Group(o11yPrefix), and zip
qualifies an op's id by the prefix of the occurrence it is declared under.
That rule exists so one definition included twice cannot publish one
operationId for two operations, and it is right — but o11yPrefix is not that
kind of prefix. It is not a composition point a host chose; it is this
subsystem's own address. Read through a Group it looked like one, and every
published id came out "v1.o11y.get_logs": the OpenAPI operationId, the MCP
tool, the CLI command and the generated SDK method, all renamed, with no
opt-out — an explicit WithOperationID is qualified the same way.
The same Group also cost the ops their origin, which is the app an op is
declared in and the thing that qualifies published TYPES. A Group is not that
app, so 23 schemas went out bare — logsResponse where the weave expects
o11y.logsResponse, and a bare name is one another subsystem can collide with.
Declaring at the root gets both, because an occurrence there is unqualified
and carries the app's own origin. OpScope.Prefix then prepends to the op's
path exactly as a Group's does, so every ADDRESS is byte-identical: same
method, same full path, same middleware. Only the two names change, and they
change back to what the declaration wrote.
hanzoai/o11y's own table reaches this conclusion for the same reason in its
relay.go; this is that shape on the cloud-native half. mountAlerts is left
alone — it registers raw routes, which publish no id and no type.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/o11y, /v1/sentry, /v1/event, /v1/errors, /v1/analytics, /v1/insights,
/v1/integrations and /v1/summary all answered 503 with
mount /v1/o11y: no instance running
while the pod stayed Ready with 0 restarts. The subsystem runs as its own
child process, and it was exiting before it listened: zip refuses a program
that does not compose, and this one declared middleware at two addresses it
had no leaf beneath.
a.Group(o11yPrefix).Use(cloud.Bridge()) // o11y.go
a.Group("/v1/summary").Use(cloud.Bridge()) // summary.go
Neither group ever received a route. The o11y leaves register through a
SECOND Group(o11yPrefix) in scope.go and annotation_queues.go, the module's
353 typed ops register on the app at a root prefix, and /v1/summary's own
leaf registers on the app at the group's address rather than beneath it. A
group's middleware wraps the routes in its OWN subtree, so all three sat
outside the thing meant to guard them — which is the defect the walk names,
and it named it correctly.
Install it once, on the app, where serve.go already says it belongs: a
subsystem whose routes are spread across several top-level nouns owns no
single prefix to hang it on. This one owns eight. That also keeps the module's
353 ops wrapped, which a prefix group silently would not have done — the org
a typed op reads off its context has to be parked for every leaf, not for the
fraction that happens to share a prefix.
The tests carried the same shape and are moved with it, so they exercise what
serving installs rather than a composition only the test builds.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The avatar write failed for the whole session with two different errors, and
both were this one door mis-reading a reply it had assumed the shape of:
iam non-envelope response (400) an error body it could not unmarshal
iam: iam status 200 a SUCCESS it rejected
Some IAM routes answer the {status,msg,data} envelope this client was written
for. Others — /v1/iam/users/get among them — answer the RESOURCE DIRECTLY, and
their errors are {"status":404,"error":"…"} where `status` is a NUMBER, not the
string "ok". Measured against the running service:
GET users/get?owner=hanzo&name=z
-> {createdAt,updatedAt,deleted,id,owner,name,…} no envelope at all
So a raw row parsed with Status "" and was rejected as `iam status 200`, and an
error body failed to unmarshal and became `non-envelope response`. Every read
through this client was one of those two.
The HTTP STATUS now decides and the body is only read for what it carries: a
non-2xx yields its `error` or `msg`, a 2xx envelope keeps its old meaning, and a
2xx that is not an envelope IS the resource.
`go get github.com/hanzoai/cloud@latest` fails outright:
module source tree too large (max size is 524288000 bytes)
v1.801.413 is the last consumable version; .420 and everything after cannot be
downloaded by anyone. That is every consumer of this module, not just ours — ai
hit it trying to take the release it had been waiting on.
The cause is build output committed at the repo root. `go build ./apps/gateway`
drops the binary HERE by default, and five landed that way: gateway 53M,
account 33M, authz 30M, smoke 8M, gen-app-cmds 4M. They are ELF x86-64, ELF
aarch64 AND Mach-O arm64 — three different people's machines, over three weeks,
each an accident nobody could see because the repo already had them. `account`
arrived today and is what crossed the line.
Untracked and ignored by exact path. Every one is built from a package whose
source is untouched (apps/gateway, apps/account, plugin/authz, plugin/smoke,
plugin/gen-app-cmds), so nothing is lost and `go build ./...` is unchanged.
Tracked tree: 174MB → 40MB.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerce v1.49.62, iam v1.34.18, o11y v1.5.55, ai v1.832.21 — every subsystem now
published on zip v1.24.1, so cloud and everything it mounts agree on one framework
version rather than four.
That agreement is the point. Router is the type a decorator implements, and the
verbs a decorator must satisfy changed in v1.23 (Use is the one composition verb,
taking a Component); a host on one version and a subsystem on another is a
decorator that cannot be written.
Measured against a clean tree on this host: 97 failing packages before, 97 after —
zero new. That is the macOS SQLCipher limit (no tmpfs for the pure-Go codec),
unrelated and unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The seam's doc said every app is its own process and read that as the reason
SetRiskScorer cannot reach the gateway. The premise is false as stated.
plugin/campaign, plugin/integrations and plugin/guide each link three or four
sibling apps/* into ONE process and wire process-global seams across them
(seams.go in each), under no build tag, and the image builds every plugin
directory. Co-residency is a per-plugin composition choice: one import in one
composition root is the whole distance between the two arrangements. As
composed today plugin/risk links risk alone and plugin/gateway links gateway
alone, which is true and is all that should have been claimed.
The remedy stands for a simpler reason that holds whatever the topology is:
apps/risk exports Mount and Shutdown and nothing else. There is no scoring
function to install, so the seam has no producer because none can be spelled,
not because a boundary forbids one. Giving it one means EXPORTING a scorer and
then deciding what it answers for — this global answers for its own process,
while arming asks whether the risk plane can answer for the fleet, which is a
cross-process ask.
The obs event door stays as the precedent it is, in its own clause rather than
as the justification.
Comments only; no behaviour changes and nothing is armed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two independent holes let a name the platform holds enter the global
site_hosts table.
SHAPE. Store.bindHost asked sites.IsReserved with its argument, but that
argument is a bare slug on the deploy path (deploy.go siteHost) and a full
hostname on the custom-domain path, while the reserved set holds bare LABELS.
So every FQDN compared against a set of labels matched nothing: login.hanzo.ai
sailed past the only guard behind the claim gate and took a first-come row on
our own auth apex. The storage invariant that is supposed to make the
serve-time gate a mere backstop contributed nothing at all for half the table.
sites.Ours splits on shape — a bare label asks the reserved policy, a hostname
asks the self-domain set — so one predicate answers both, and the claim gate
(domains.go ours) and the host table now ask the SAME question. Note what the
split avoids: `www` and `login` are reserved labels AND the two most common
custom domains a customer brings, so a backstop keyed on "the first label of
this FQDN is reserved" would refuse www.example.com. The label policy must
never reach a name a customer owns.
COMPOSITION. The self-domain set was published only by sites.New. Which apps
share a process is a per-plugin choice, and as composed today plugin/projects
links apps/sites for exactly these two predicates and never calls New — the
edge is not in that process. So the set was EMPTY in the very process that
enforces the claim gate and the host table, and full in the one that serves:
IsSelfHost answered false for everything there. Measured on the parent commit,
in a process that constructs no Server:
IsSelfHost("hanzo.ai") = false
IsSelfHost("api.hanzo.ai") = false
IsSelfHost("login.hanzo.ai") = false
IsSelfHost("hanzo.app") = false
which is the exact defect SetSelfDomains was added to close, defeated by
composition rather than by logic. No test could see it: they publish the set
themselves, in one binary.
The policy is now DERIVED from config wherever it is asked. ConfigFromEnv is
the one reader of that config and every process reads the same environment, so
the answers cannot drift the way a hand-off between processes does, and it
stays right however the plugins are later composed. An explicit publication
still wins and is never re-read over. Production needs no deployment change:
hanzo.ai is derived from CLOUD_DOMAIN's registrable domain, not configured.
selfOf folds the first-party apex in itself, so New's published set is complete
by construction rather than by statement order — the ordering hazard the
previous fix had to hold by hand.
Tests: the Server-less derivation, the shape split over both halves including
the customer hostnames that must stay bindable, and a concurrent first touch
under -race.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Binding a custom domain WITHOUT proving control of it — the bind lands
VERIFIED and routes immediately, and the "a host we operate" refusal does not
apply — is platform authority. The gate read
vouched := c.IsAdmin() || s.State.operatorOrgs[org]
and that second term is bare MEMBERSHIP. operatorOrgs defaults to the
deployment's own brand org (config.go getenv CLOUD_BRAND, brand.Default
"hanzo"), so the set is {hanzo} in every deployment, and `org` is the
IAM-validated effective org, gated upstream by isMember alone. Every staff
account whatever its role, plus anyone a brand-org admin ever invited, could
bind login.example-bank.com live with no DNS-01 proof: attacker content served
at any custom-domain customer whose DNS already points at our edge, and the
name denied to its rightful owner for good, since a verified row is first-come
and global.
vouches() now names the two grants, both admin-scoped:
SuperAdmin platform sudo (owner == the reserved admin org).
Cross-tenant by construction, so it vouches in ANY
org — the operator switched into a customer's org to
bind the domain it manages DNS for. Unchanged.
operator-org ADMIN the deployment named this org an operator AND IAM
says the caller administers it.
The second is a conjunction of two independently administered facts: a
capability the deployment grants to an org, and the role IAM grants inside it.
The set names an org; it never names an authority. The org-admin bit is asked
of the EFFECTIVE org — the same value SanitizeIdentity keys X-User-IsOrgAdmin
on — so the pair reads "admin OF this operator org" and never "admin of some
org I switched out of". Both bits are stripped on ingress and re-minted only
from validated claims.
Fail-secure: an issuer that stops signing the org role drops the operator-org
grant to a PENDING claim carrying the DNS challenge, the same self-service path
every other tenant takes. SuperAdmin onboarding never depended on that claim.
TestOperatorVouchNeedsAdminScope drives the real handler over four identities.
On the parent commit it fails with the exploit verbatim — a plain member's bind
of login.example-bank.com comes back {Status: live, Verified: true}. It also
holds the positive cases, so a fix that merely disabled operator onboarding
would not pass. TestOperatorVouchIsVerbatimEndToEnd keeps the verbatim-owner
pin and now has both callers be org admins, leaving the org name as the only
axis; it went from panicking to passing because it no longer builds the whole
surface.
The harness registers the ops it drives rather than calling routes(), which
composes only on the Router production gives it: routes() declares middleware
with Group(prefix, mw) and registers its typed ops on the App with full paths,
so on the bare *zip.App a test holds the prefix node has no routes beneath it
and zip refuses to compose. cloud.Listen mounts on a *scope, whose Use and
Group install at the root and gate by request path, so the same registration
composes there. What is driven is the production chain at the production paths:
cloud.Bridge parking the request, siteOf resolving the tenant, bindDomains
deciding.
The lifted prose and both published specs carry the corrected contract; the old
text told customers that membership of the operator org was the vouch. zipdoc
regenerates; the OpenAPI subsets cannot be projected for this app yet, so those
two files take the identical substitution zipdoc made.
apps/projects: 55 -> 58 tests passing, no test that passed on the parent fails.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The pin sat at sha-9da3984 while console main moved four commits ahead. Since
console.hanzo.ai IS this binary, none of them was reachable by any customer:
a0a4899 onboarding: Continue keeps its place, and no step can strand you
06e4163 retire the ML Pipelines (Kubeflow) product — its backend is gone
da26897 auth: a refusal is not always a failure
2271c29 profile: a photo you can change, instead of one you can only look at
The embed image for sha-a0a4899 is published and was probed before moving the
pin (200, against a known-good positive and a bogus negative control).
855 files, 360.7 MB, 192 of them real .rlib/.so/.a binaries: a complete cargo
build tree committed to git. Every published hanzoai/cloud module version has
carried it, so every consumer downloads 360 MB of another project's build output
to compile Go. The local module cache alone holds ten copies.
It is orphaned, not merely misplaced. native/flags/ contains NOTHING ELSE — no
Cargo.toml, no src/, no crate at all — so there is not even a project here to
rebuild it. No .go file, Makefile, Dockerfile or shell script references the
path. It arrived as collateral in 56c1b003 and nothing has needed it since.
.gitignore has listed `native/flags/target/` since before this commit; an ignore
rule does not untrack what is already tracked, which is exactly how 360 MB stays
in a tree everyone believes is ignored. `git rm --cached` is the part that was
missing.
Files stay on disk; only the index changes. History still carries the blobs, so
this shrinks FUTURE module versions rather than past ones.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nine plugins failed to start and every route they own answered 503 — account,
catalog, analytics, o11y and the rest — on one panic:
panic: zip: the group "/v1/avatar" declares middleware at scope.go
and no routes anywhere beneath it
THE NODE WAS THE BUG, NOT THE PREFIX. scope.Use installed middleware with
s.app.Group(p).Use(...), which creates a node AT p; scope.Get and its siblings
delegate straight to s.app, so the routes land on the ROOT node. Same paths, two
nodes. zip >= 1.23 checks the subtree of the node the middleware is on, finds it
empty, and refuses to compose a program whose middleware could never run. It was
right to.
Declaring the prefixes did not fix it — it moved the panic from /v1/account to
/v1/avatar — and analytics failed identically while already declaring them.
Both gates now install ONCE at the root and test the request path: Use against
the subsystem's prefixes, Group against its own. The root always has routes, so
nothing is empty, and `owns` does on the request what the per-prefix node was
there to do on the tree. `under` is that one meaning of "inside my subtree",
shared by both.
Confinement is unchanged and still proven by the tests that were red:
ScopeConfinesUseToTheSubsystem, ScopeHonoursDeclaredPrefixes and
ScopeAllowsGroupInsideItsPrefixes all pass. The root package now reports ZERO
composition panics; its 43 remaining failures are two macOS-only causes (40 cek
"no RAM-backed scratch", 3 unix-socket path length), identical before and after.
THE SUITE ALREADY REPRODUCED THIS. `go test ./` was red on main throughout the
outage, with the exact production panic, in under a second and with no cluster.
It was read as environmental noise while the fix was hunted in production.
SetRiskScorer has no production caller. The seam's own doc said to follow the
observability plane's event door "(obsevents.go)" — a file deleted in 4a6f3918,
by the commit that established the opposite: "A plugin is a process;
cloud.SetObsEventIngest / SetObsErrorIngest could never have worked across that
boundary." That door read nil in the process that needed it and answered 503 to
every Sentry SDK until it became a plane op (apps/o11y/obs_rpc.go).
SetRiskScorer is the identical shape, and the fleet is one process per app:
cmd/cloud mounts each subsystem as its own binary, plugin/risk lists risk alone
and plugin/gateway lists gateway alone, and no binary links both. So the wire
this seam invites — call SetRiskScorer from the risk app's Mount — would arm the
risk process and nothing else, while apps/gateway's RiskScorerInstalled, running
in the gateway binary, stayed false and PUT /v1/gateway/config went on refusing
every arming request. It would read as wired and change no outcome.
The tests cannot show this: they link cloud and the app into one binary, where
the handoff always works. That is why 43 references pass over a seam with no
producer. Both comments now say so — what the seam reaches, and that the
gateway's refusal is currently unconditional and fail-SAFE, answering "is a
scorer linked here" rather than the question arming asks, which is whether the
risk plane can answer for the fleet.
Comments only; no behaviour changes and nothing is armed. Reaching the fleet
answer is a plane op, and that is a decision to take deliberately, not a wire
to restore.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
New() published the self-domain set and THEN folded the first-party apex into
the slice it had just handed over. SetSelfDomains copies its argument
(reserved.go), so that append could never reach IsSelfHost — the one predicate
that answers "is this host ours", read by both the serve gate and the claim
gate.
Under a config that names the apex only as FirstPartyApex — not also in
CLOUD_SITES_SELF_DOMAINS — every non-allowlisted <label>.<fpApex> was therefore
a custom-domain CANDIDATE on the apex that carries api/login/console: a claim
row a customer could take, and a per-request binding lookup on the hot path the
exclusion exists to keep clear. Production lists hanzo.ai in BOTH, which is why
TestSelfDomainsCoverTheBrandApex passes either way and never saw it.
The publish now runs after the fold, so one set leaves the constructor.
Server.selfDomains went with it: it was written once here and read nowhere,
a second copy of a set whose only reader is the package global in reserved.go.
TestFirstPartyApexReachesThePublishedSet
fix reverted, test kept → RED (IsSelfHost false, customCandidate true
for hanzo.ai / api.hanzo.ai / login.hanzo.ai)
fix restored → GREEN
apps/sites and cmd/cloud green; apps/crm unchanged from its baseline failure
(TestIntakeRateLimitScope, pre-existing on origin/main).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/risk/search ranked model spaces against a tenant's own history and
answered with the one that fit. Nothing could promote it, and nothing could:
adoption refused a shape change, and a winner is a different shape by
definition. The stated replacement for Katib produced advice nobody could take.
A model value now carries the SPACE its masses describe, not only the masses,
and install REPLANTS — it builds that space, restores into it, and swaps it in.
Every gate the refusal used to carry still holds: the tenant comes from the
validated principal, the geometry seed is checked against the model already
running before anything is rebuilt, and the recorded shape is rebuilt and
compared by digest rather than trusted. A value recording no shape is refused
rather than defaulted, and a failed replant leaves the residency untouched.
The winner arrives as one of the organisation's own published values: the run
fits it once more after the grid — a sixty-fifth pass, gated and metered as one
— under that organisation's OWN geometry, because the grid's reference partition
is a constant and a model an organisation runs must partition the space in a way
an outsider cannot predict. Keeping all sixty-four fitted stores instead would
hold 21 MiB for sixty-three shapes nobody adopts.
Also:
- the shape is its own value, and the half of a config that belongs to the
state. The grid's candidate embeds it, the residency records it, a published
value stores it: one spelling, so adopting a shape cannot restate a policy.
- the resume row's shape is what the next process plants. Without it an
adopted shape was silently lost on every rollout and the organisation went
back to the default, warming, deciding nothing.
- the fold watermark travels with an adopted value. It is in the address for a
reason; leaving it behind meant a rollback would never re-read the fold it
skipped.
- snapshot+restore collapse to POST and PUT on /v1/risk/state/model: one
address for one kind of thing, the same collapse GET and PUT on
/v1/risk/policy already made. openapi/floor.json loses one PATH (the risk
product keeps all 31 operations).
- the value bound is sized on the widest shape the grid declares and MEASURED
at full occupancy (2,214,100 bytes), because a fitted sample understates a
busy organisation's tree by an order of magnitude.
- open() no longer reads the state believing the read plants a tenant's trees.
Measured against the pinned engine: it does not.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The two Kubernetes submit paths in ai/cluster no longer name a vendor in a
scheduler contract. The TrainJob's resourcesPerNode and the KServe
InferenceService's predictor both take the device-plugin resource name the
CLUSTER advertises, and a cluster that advertises no accelerator is refused at
submit time rather than accepting a workload that can never be scheduled --
which is what the KServe path did, silently, with a live controller
reconciling the result into a permanently-pending predictor and a model
registered on api.hanzo.ai whose every call fails.
Pairs with agents.Need/Spec.Satisfies landed here: one vocabulary for what a
job requires and what a machine advertises, and no vendor field in either.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Spec already carried what a machine IS -- os/arch/cpus/memory and each
accelerator's vendor/model/memory. Nothing carried what a job WANTS, so the
only way to ask for a GPU was a scheduler contract that named a vendor:
resourcesPerNode.limits."nvidia.com/gpu". That is not a requirement, it is one
vendor's name for a requirement, and it made a job unroutable to an AMD or
Apple machine that could have run it.
Need is the other half of Spec, in the same vocabulary, and Satisfies is the
one place the two meet -- a pure function of two values, so the dispatch gate,
a scheduler and a UI preview cannot disagree. Need has no vendor field: which
vendor clears an accelerator requirement is the machine's business, and
hanzo-kernel lowers one kernel source to CUDA/ROCm/Vulkan/Metal so a job never
has to care.
Unknown memory does not clear a memory floor. Every unified-memory
accelerator advertises 0 today (nvidia-smi answers "[N/A]" on a GB10,
system_profiler emits no VRAM line on Apple Silicon, lspci carries no memory
at all), so a VRAM floor currently refuses all three -- fail-closed, and the
reason the probe should report the memory an accelerator can actually address.
The three boxes are fixtures here, carrying their measured values, so that
change shows up in one place.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
account went down because its plugin declared no Prefixes and MountPrefixes
falls back to /v1/<name> — a path it does not serve — so the scope guarded a
subtree with no routes and zip refused to compose. Same fault, same fix, in
every other plugin that carries it.
Found by predicate rather than by waiting for each outage: undeclared Prefixes
AND a manifest row without /v1/<name> AND an app that calls app.Use(). All three
are needed — 25 plugins match the first two and serve fine, because the panic
only fires when a subsystem actually installs middleware at its scope root.
admission bot dataset do graph knowledge leaderboard treasury
Each now declares what the manifest already says it answers, which is what the
host routes to it either way — so this changes no address, it only stops the
scope guarding one that was never served.
Every account route answered 503 in production — /v1/keys, /v1/csrf, /v1/avatar
— because the plugin panicked at mount:
panic: zip: the group "/v1/account" declares middleware at scope.go:134
and no routes anywhere beneath it (via root -> /v1/account)
The plugin declared no Prefixes, and undeclared is not "no prefixes":
MountPrefixes falls back to the /v1/<name> convention (subsystem.go:78). Account
answers at NONE of /v1/account — its routes are /v1/keys, /v1/csrf, /v1/avatar,
/v1/orgs, /v1/embed and /v1/commerce/topup/*. So scope.Use installed the
subsystem's Bridge on a path with nothing beneath it, and zip refuses to compose
a program whose middleware can never run.
The same fallback bit analytics and entitlements before this; both carry the
same one-line fix, and this is it.
WHY THE TESTS DID NOT CATCH IT, which is the part worth keeping: they mount on a
bare zip.App, where Use attaches at the root and the root HAS routes. Production
mounts through a SCOPE. Same code, opposite outcome — so a green suite proved
nothing about the composition that actually ships.
manifest/clients_test.go's ledger of addresses a first-party client asks for and
the fleet does not answer held three entries. All three are closed; the ledger is
empty.
GET /v1/billing/portal/methods — this is what made "card save is broken" true END
TO END. cloud's billing app serves the saved-card list by proxying here, and
nothing in the fleet served it, so the proxy forwarded a 404 verbatim and the list
rendered as "no cards saved" no matter how many cards were vaulted. Claimed on
commerce's manifest row and registered co-resident.
THE GATE, which is the actual work: PortalPaymentMethods keys tenancy on a
?customerId QUERY PARAM, so the chain has to pin the subject for both principals
that arrive. TokenRequired (not IAMTokenRequired) authenticates and resolves the
ORG from the gateway-pinned X-Org-Id for an IAM member AND for the raw service
token the proxy presents; PinBillingSubject is the IDOR control — it overwrites
every billing-subject key with the validated caller's own account.Payer subject
and drops ?org, passes the query through only for a bearer that constant-time
matches COMMERCE_SERVICE_TOKEN, and fail-closes anyone who is neither. The tenant
is never a caller-supplied field on either path.
DELETE /v1/billing/methods/{id} — 405 at the live edge: a customer could ADD a
card and never REMOVE one. billing registers the sub-resource on the same router
as the collection (the host claims a prefix for ONE app across every method) and
proxies it to commerce's DELETE /v1/billing/portal/methods/{id}, the target that
does not self-dispatch. The org comes from principal.Org — the VALIDATED
principal only, never readerOrg's service-token admission, because this is a
mutation and that is the rule createPaymentMethod and gpuCharge already follow.
The id is escaped into the upstream URL: it names a resource, not a route.
PATCH stays unserved — no client edits a card.
POST /v1/billing/payment — DELETED, not served. No app in either server repo has
ever registered it, in any commit, so the crypto top-up's recording step always
failed and a customer who had already sent USDC to the treasury got a 502; it was
501 besides, since TOPUP_RAILS is configured in no environment. There is nothing
to point it at: money-IN has one door (commerce's mint-gated POST
/v1/billing/deposit) and the fleet routes NO mint address at the edge — the only
two money-in paths manifest.Apps hands to an app are the card ones, both with a
server-authoritative amount. What the browser client sends here is a
client-supplied amount and a client-supplied subject, which is the exact shape the
mint gate exists to refuse. So the surface that existed only to call it goes with
it, and openapi/floor.json records the two-operation reduction next to its reason.
Tenant isolation: TestDeletePaymentMethod_TenantIsolation proves org A cannot aim
a delete at org B through the org, the id, or any subject key; the far side is
proved in commerce (payment_methods_tenant_test.go) for both handlers and both
caller profiles.
Needs hanzoai/commerce fix/billing-methods for the portal detach it proxies to.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The lease is a single-holder flock on one inode under DataDir. Between two POD
GENERATIONS rolling over the same PVC that is exactly what is wanted — it is the
thing that stops a surge writer double-opening the exclusive-lock ZapDB and audit
stores. Between the sibling processes of ONE pod it is a deadlock.
cloud is a plugin host: kms, pubsub and kafka are separate processes in one
container, sharing this DataDir by design. Every one of them called
acquireWriterLease on the same path, so the first to start won and the rest
blocked to the 90s fail-closed deadline. Nothing bound :8000, the liveness probe
killed the pod, and the replacement deadlocked identically — api.hanzo.ai served
503 in a restart loop produced entirely by the safety mechanism.
`CLOUD_WRITER_LEASE` was reverted in the values file to stop the bleeding
(universe 47e195f1, "the binary is not one process"). This is the other half: the
variable can be set again without taking the deployment down.
The guard is underRouter(), which already exists and already means this — ZIP_ADDR
is proof of a parent that owns a plugin table. A child is not a second pod; it is
part of the writer that is already holding the lease.
The test pins all three facts: the host takes it, a second host-shaped process is
still refused, and a child never asks.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The wire was V1-shaped and ours in the places it was not the spec's, so a
compliant @x402/fetch or x402[httpx] client could not pay us at all: it sends
PAYMENT-SIGNATURE and we read X-Payment; it signs `payTo`/`asset`/`amount` and we
quoted `payee`/`token`/`amount`; it speaks CAIP-2 and we published a network label
beside a separate chainId. Every field name, header name and error reason on the
wire is now the one x402-specification-v2.md prints.
header X-PAYMENT / X-PAYMENT-RESPONSE -> PAYMENT-SIGNATURE / PAYMENT-RESPONSE
(plus PAYMENT-REQUIRED for the challenge), all base64 JSON
version "1" (string) -> 2 (number)
scheme "erc3009" -> "exact" (+ extra.assetTransferMethod)
network "hanzo" + chainId 36963 -> "eip155:36963", chain id read OUT of it
challenge bare PaymentRequirements -> PaymentRequired{resource, accepts[]}
payment flat Proof -> PaymentPayload{accepted, payload{
signature, authorization}}
answer Receipt on a header -> SettlementResponse, on success AND failure
The V1 shapes are deleted, not aliased. A header a client may send under either
name is two wires, and the one the server forgot to read is the one where a payer
pays and is never served.
Two things the alignment forced, both real bugs:
* The client's echoed `accepted` is now checked field-by-field against what we
offered (spec 6.1.2 step 5). `extra` IS the EIP-712 domain, so a payload free to
restate it would sign a message of its own devising and verify against itself.
* settleLedger documented a hole and left it: debit lands, credit fails, nothing
served, and the ONLY recovery was the client re-presenting an authorization that
expires in 300s. A client that gave up for five minutes was permanently debited
with nothing delivered.
Fixed by ordering, not by compensation (a reversing entry refunds a payer who
was correctly charged when the failure was a timeout). The settlement row is now
CLAIMED before any money moves and flipped to settled after both halves land, so
an interrupted settlement is a durable row naming the payer, the payee subject
and the amount — keyed on the id both money writes are idempotent on. And the
time window gates ACCEPTING an authorization, not COMPLETING one already
accepted: EIP-3009's validBefore bounds when a transfer may be submitted, not how
long submission takes. Two independent paths now converge — the client is served
whenever it comes back, and Reconcile finishes it from the claim if it never
does.
The payee's ledger SUBJECT is on the claim rather than re-resolved, so completing
a settlement pays the payout wallet the listing named. Re-resolving credited the
seller ORG, which no balance assertion could catch (finance aggregates at the
org) — the test now asks the ledger where the money went.
Also: POST/GET /v1/wallets answered 403 "sign in" to everyone. zip v1.23 scopes a
definition's middleware to its own subtree, and the collection root was declared on
the app, outside the group carrying cloud.Bridge — so it reached its handler with
no validated principal. Declared on the /v1 parent now. This is what blocked all 8
x402 and 4 marketplace tests from running at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A version was allocated by READING — max(registry tags, git tags) + 1 — and a
read reserves nothing. Two lanes publish ghcr.io/hanzoai/cloud (this repo's
.hanzo/workflows/cicd.yml, and apps/platform/release.go behind POST /v1/runner);
both computed the same next number, both built for ~20 minutes, and both pushed.
GHCR TAGS ARE MUTABLE, so the second push silently REPLACED the first's bytes
under a name the first had been told was its own:
v1.801.361 overwritten at 04:40:53
v1.801.410 overwritten at 08:17:12, twelve minutes after the real release,
by an image whose revision label read `unknown`
The loser found out at its tag step — three quarters of the way through, after
it had already corrupted the winner's image, which the winner then smoked,
pinned and shipped. A check twenty minutes downstream of the act it guards is
not a check.
Creating refs/tags/v<N> is the one operation in either lane the server performs
as a COMPARE-AND-SWAP: 201 if absent, 422 if present, decided under its lock.
So the claim IS the allocation, and it moves to the FRONT of both lanes. A
number that cannot be claimed was never ours to build; the loser walks to the
next one in a single HTTP call, before building anything. The tag step that used
to mint now VERIFIES the claim still names this commit before the pin.
Cost: a failed build leaves a hole — a tag with no image. That is the cheap
direction. A hole is inert and visible (pin.sh refuses a tag that does not
resolve); a reused number is invisible and serves the wrong bytes.
Also:
- resume is decided by the REGISTRY, not the tag. With the claim moved before
the build, a tag on HEAD no longer implies an image exists, so the old
`git tag --points-at HEAD` resume would have skipped the build of a release
that had none. The tag says what we own; the image says how far we got.
- the pushed image is verified to BE the commit we built, read back off the
registry rather than trusted from our own build output. This is what makes a
clobber by any lane — including one that ignores the claim — loud instead of
a green pin onto foreign bytes.
- REVISION is passed as a build-arg by both lanes. The Dockerfile declares
`ARG REVISION=unknown` and stamps the label from it; buildFrontendCmd never
passed it, so every platform-built image was untraceable to a commit. That
is why the two v1.801.410 images could not be told apart without diffing
layers. Branch refs are refused (isCommitSHA) — a label that says "main" is
populated and useless.
- .hanzo/scripts/image-revision.sh reads an image's commit, descending an
index to its amd64 child so multi-arch images are not silently unchecked.
- the two 422s are distinguished. "Reference already exists" is the collision
this loop is for; "Object does not exist" means our own commit is not on
github.com — reachable, since these lanes run on git.hanzo.ai and claim
against GitHub — and no amount of walking forward fixes it.
Tests: TestClaimReleaseVersion_IsExclusive reproduces the race (a number another
commit holds is skipped and never overwritten; our own claim is a resume that
mints nothing), and TestTagRelease_VerifiesTheClaim asserts the tag step only
READS. claimFrom is split from computeReleaseVersion so the exclusion property
is testable without the registry.
`make openapi` was renamed to `make describe` in e247e255 — correctly, since
the target now projects every app rather than only the spec — and 27 places
were left naming the old one. Two of them are FAILURE MESSAGES: a developer
whose golden is stale is told "run `make openapi`", which prints "No rule to
make target". A gate that says how to fix it, and names a command that does
not exist, is a gate that reads as broken tooling.
Comment-only apart from those two t.Fatalf strings; no target, no behaviour and
no artifact changes. `go build ./openapi/` and the four apps with the most
edits build with the tags hanzo.yml's own go-unit gate uses.
openapi/fleet.go also gains the projection it was missing. It lists four
projections of this API compared against each other by test; there is a fifth,
downstream and in another repo — hanzoai/openapi's hanzo.yaml, which every
published SDK is generated from — and it refutes itself against the LIVE
endpoint in that list, because that is the only one of the four a repo with no
checkout of this one can read.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Upstream landed the v1.23 verb migration (Graft/Add/Mount folded into Use). This
is the half that was missing, and it is the half that made tests lie.
App.Test used to skip prepare, which installs the deferred projections — /mcp, the
OpenAPI document, the op-call plane, the plugin route. So those four addresses
answered 404 under test and 200 in production, and the papering-over was an
exported Prepare each caller had to remember. zip v1.24.1 makes Test prepare;
apps/ai's MCP door test passes because of that, not because of anything here.
414 call sites move from app.Fiber().Test(...) to app.Test(...) with
zip.TestConfig. That is the point of the escape hatch living on the concrete type:
reaching through it bypasses what App.Test does, so the tests most wanting to
exercise the real program were the ones that did not. Sites whose receiver is a
raw fiber app keep fiber's type — the two are not interchangeable and pretending
otherwise is how the first sweep broke things.
Also: the multi-line `Use(func(c *zip.Ctx) error {…})` literals in tests, which
the verb migration missed because they fail vet rather than build; and the last
`.Prepare()` calls, now that it is implicit.
iam v1.34.11 → v1.34.12.
Measured against upstream on the same host: 103 failing packages before, 97 after
— ZERO new, 6 fixed. The remainder is the macOS SQLCipher limit (no tmpfs for the
pure-Go codec), unrelated and unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
With the read shape fixed, the avatar write moved one step and stopped: the
lookup now reaches IAM and asks for user `hanzo/2d4d67ab-…`, which does not
exist under that spelling. IAM addresses a row by its NAME — the row whose id is
2d4d67ab-… is named `z` — and on the direct-Bearer path the only user handle a
token carries is the UUID `sub`, because X-User-Name is not stamped there.
So a failed direct lookup now resolves the id against the org's roster
(get-users?owner=…, which carries both id and name) and retries once. Measured:
that roster returns 268 rows for hanzo, each with both fields, so the mapping is
available exactly where it was needed.
It runs ONLY after the direct read has already failed — the gateway path, where
username == name, never pays for it.
zip v1.23 deletes graft.go outright. Graft(children ...*App) error was the third
way to attach something to an app, alongside Add and Use; v1.23 keeps exactly
one, and an *App IS a Component, so a child is included by reference through the
same verb as a middleware.
Two call sites, both the "include a whole subsystem's app" case Graft existed
for: apps/iam grafting iamserver.NewApp(db), apps/o11y grafting its assembled
app. Both now Use it.
ONE SEMANTIC CHANGE, recorded because it is not visible at the call site: Graft
refused an address conflict EAGERLY and returned the error to the caller, having
checked every child address against the parent's router and its siblings before
mutating anything. Use appends and defers that verdict to Build, where the whole
program is known. Same refusal, later and with more information — but a mount
that used to fail at its own line now fails at seal, so read Build's error, not
the mount's.
Verified: `go build -tags sqlite_math_functions ./...` returns ZERO across the
whole tree with iam v1.34.11, o11y v1.5.54 and commerce v1.49.61 — the three
dependencies that had to publish first. Graft appears in no .go file in any repo
in the estate.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Cloud pinned zip v1.18.23 while latest was v1.23.0 — five minor versions of
drift on the framework every subsystem composes through. v1.23 removes the
second and third ways to attach something to an app and leaves exactly one:
Use(cs ...Component), where a Component is a Handler or an *App included by
reference.
Cloud's own four adaptations:
- Router.Use widens from ...zip.Handler to ...zip.Component, mirroring
zip.Router exactly. That mirroring is load-bearing: ZipApp's type switch asks
whether a *zip.App satisfies this interface, and a narrower Use made that case
IMPOSSIBLE — the compiler rejected the switch outright rather than silently
taking the wrong branch.
- scope.Use forwards Components unchanged, still once per declared prefix.
- (*zip.App).Add is gone. zip.Load already returns the leaf *App and an *App IS
a Component, so both call sites capture the leaf and Use it. The eager and
lazy rungs of the plugin ladder keep their existing error handling; only the
attach verb changed.
- Five bare closures passed to Use now go through zip.H. Go will not implicitly
convert func(c *zip.Ctx) error to an interface that only the named Handler
type implements. Route methods still take ...Handler, so no route registration
changed — only Use sites.
Verified: with zip at v1.23.0, `go build -tags sqlite_math_functions ./...`
reports ZERO errors from apps/, cmd/, clients/ or internal/. The only remaining
failures are in three dependencies that must publish first — commerce
(mintRouter implements the old Use, and reaches Fiber() through the Router
interface, which no longer exposes it), iam and o11y (both call the now-
unexported app.Prepare; the public replacement is app.Build, which returns an
error Prepare did not). Those are in flight; this commit is the cloud half and
does not build green until they land.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The site edge is mounted in two processes — the light router that owns the
public port (cmd/cloud) and cloud.Listen in every per-app child — and each
resolved sites.Config for itself, from two spellings of the first-party keys
(CLOUD_SITES_FIRSTPARTY_* against CLOUD_SITES_FIRST_PARTY_*) with two sets of
defaults. Neither spelling is set in production, so the two processes booted
different policy off their DEFAULTS alone: the router resolved no first-party
apex, so hanzo.ai never entered its self-domain set, so every hanzo.ai host —
api.hanzo.ai included — was a custom-domain candidate and took the per-request
binding lookup the self-domain exclusion exists to keep off that path.
The reserved denylist was NOT affected. Both spellings read the same
CLOUD_SITES_RESERVED key, the operator value only ever ADDS via
SetReservedExtra, and the labels an attacker wants are baked into reserved.go —
so an empty value cannot un-reserve anything. Measured at the live edge:
www/api/app/admin/stg/login/wallet.hanzo.app all fall through with no
X-Hanzo-Site, against quest.hanzo.app which answers with one.
The env keys and their defaults now live in apps/sites, the package that owns
the type, and both call sites read that one function. Config keeps only Domain,
which feeds the self-domain set.
Also removes exports with no caller in this repo or any that import it:
OKList/OKRaw (envelope.go, whose note about a clients/admin/core delegator was
stale — that package is gone), BrandInfo, DegradedNames/IsDegraded (the release
smoke reads /v1/health over HTTP; the in-process accessors had only tests, whose
coverage of the live Degraded/Degradations pair is kept), and the four inert
Stage-0 control-plane fields with their NODE_ID/PEERS/ROLE/CONTROL_PLANE_QUORUM
reads.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every op on /v1/engine was dialling a refused connection in production. The
upstream default named :1234; svc/engine in namespace hanzo exposes ONE port and
it is 36900 (name http, port 36900, target 36900), and the cloud Deployment sets
no ENGINE_UPSTREAM — so the wrong default WAS the production value. status
answered reachable:false, and models, model and system answered 503. The
subsystem was honest about being unreachable, which is why nothing alarmed: a
truthful report of a broken configuration reads exactly like a runtime that is
down.
1234 is standalone `hanzo serve`'s default. 36900 is the port the engine binds as
the node's engine, and 36900 is what is deployed — so the comment claiming 1234
was "the in-cluster Service of the engine deployment" described a process this
cluster does not run. The 1234-vs-36900 confusion is already on record from the
desktop build, where a frontend discovered models at one port while the engine
that answers ran at the other; this is the same mistake on the cloud side.
MEASURED against the deployed engine over a port-forward to svc/engine:36900,
which answers exactly the three endpoints this plane calls, all 200 —
GET /health ("OK"), GET /v1/models (the model list, with a loaded model carrying
"status":"loaded") and GET /v1/system/info (the SystemInfo document: os, kernel,
cpu, memory). So the upstream is present and correct and only the port was wrong;
this is one token, not a redesign.
ENGINE_UPSTREAM stays the override, and 1234 remains right where it is right — a
dev box running `hanzo serve`. Behaviour only: no route, no schema, no operation
id and no published byte changes, and regenerating the subset produces no diff.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An operation id is the generated SDK METHOD NAME and the CLI COMMAND, and a
summary is what the MCP tool list shows a model choosing between tools. This
plane stated neither, so both were defaults, and both defaults were wrong in a
way only a caller sees.
zip derives an unstated id from the path, so the four ops published
`get_v1_engine_status`, `get_v1_engine_models`, `get_v1_engine_model`,
`get_v1_engine_system` — path mangling where the rest of the fleet publishes the
product and the noun. The risk product's thirty-one operations are `riskScore`,
`riskState`, `riskDatasets`, `riskLabelCoverage`; these are now `engineStatus`,
`engineModels`, `engineModel`, `engineSystem`, which is also the rule this
package already applied to its own SCHEMA names and only to those.
A summary defaults to the first sentence of the Go doc comment, and a Go doc
comment opens with the Go IDENTIFIER — so the published summaries read "Status
reports whether the engine deployment is reachable", "Models lists the models
the engine serves", "Model reads one model's load state". A Go symbol name was
the first word a CLI user, an SDK reader and a model picking a tool saw. Each op
now states a summary written in the imperative for the person calling it, and
the doc comment stays a Go doc comment that zipdoc still lifts as the
description: two audiences, two sentences, one declaration.
FORWARDS-ONLY, and it costs nothing: this plane has no customers on it. No
alias, no redirect, no compat shim.
Regenerating from source changes exactly four operation ids. No path is added or
removed, no (path, method) pair moves, no schema changes, and openapi/floor.json
is byte-identical because the operation count did not move.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
openapi/floor.json is the ratchet: "a published surface may grow and may not
quietly shrink". It carries a global path and operation count AND a per-PRODUCT
count, and the per-product half is the half that binds — floor.go compares
`f.Products[p]` against the regenerated count and refuses a product that lost
operations. The file's own code states the failure mode out loud: "an absent floor
makes every shrink legal".
The merge 94df22e1 resolved this generated file to
`{"paths":1684,"operations":2336}`: the entire `products` map — 180 entries —
GONE, and both global counts rolled backwards. Every commit before it carries the
map. So on main right now every product may shrink without the gate saying
anything, and the global floor sits five paths and five operations below the
surface actually published.
IT ALREADY COST SOMETHING, within hours. Deleting the /v1/train facade removed ten
operations, and a ten-operation shrink is exactly what this ratchet exists to make
an author state deliberately — the dataset move had to lower `ml: 14 -> 7` in the
same commit that moved the routes, and said so. The train deletion needed no such
line, because there was no per-product floor left to lower. The gate was not
merely stale; it was switched off, and a shrink walked through it unremarked.
Restoring it is `make describe` and nothing else: 1689 paths, 2341 operations, 178
products. `train` is simply absent from the restored map rather than floored at
zero, because the map is regenerated from the document that exists — the ratchet
resumes from the current truth, which is the only honest baseline available once
the old one has been discarded.
Committed on its own because it belongs to no surface change: these counts are
identical with or without the engine renaming beside it, since an operation id
does not move a count.
The lesson is the fleet's own about derived artifacts, with a sharper edge: two
derived files can agree with each other while both are wrong, and a MERGE is a
third way for a derived file to become wrong — resolved by hand toward whichever
side the conflict presented. A ratchet resolved toward the weaker side does not
look stale. It looks fine, and it is off.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/label addressed /v1/ml/labels, apps/reference addressed /v1/ml/reference and
apps/dataset addressed /v1/ml/datasets. All three are the RISK product — the
labels a decision is adjudicated with, the lookup data it consults, and the
snapshot its model was fitted on — and all three were corrected to /v1/risk/*.
The reason is the same each time: openapi.Product takes a product from the FIRST
/v1 segment and a per-op tag cannot override it, so an address IS a published
product membership, and /v1/ml is the live serving product. Filing a second
product there makes /v1/ml/models mean two things at once.
apps/dataset's address_test.go says the third time "stops being a recollection
and becomes this gate" — and builds it, for apps/dataset. A per-app gate cannot
catch the fourth time, because the fourth time happens in a fifth app that does
not have one. So the invariant is stated once, on the side every app must pass
through: the ROUTING GRANT. A plane cannot publish under /v1/ml without a
manifest row saying so, which makes the row the one place the mistake is always
visible.
It refuses a second OWNER, not a second app, and that boundary is measured
rather than assumed: fourteen products here are answered by more than one app and
27 apps publish into more than one product, so neither "one app per product" nor
"one product per app" is a fleet invariant, and asserting either would invent a
rule the fleet does not keep. What all three mistakes actually broke is narrower
and true — the serving prefix has one owner.
The second test is the half the ai incident argues for: a gate that only refuses
intruders stays green when the owner itself vanishes. Narrowing ai's row to
/v1/ai once took the entire inference surface off the wire with every probe
green, so the owner's own leaves are asserted individually — a count cannot say
WHICH address stopped routing.
BOTH DIRECTIONS ARE MUTATION-PROVEN, because a gate nobody made fail is a gate
nobody knows works. Re-addressing dataset to /v1/ml/datasets fails with the
ROUTING GRANT message naming the app, the prefix and the product it would join;
narrowing ml's row to /v1/mlops fails BOTH tests — the non-vacuity check (which
also proves `under` is segment-aware, since /v1/mlops is correctly not under
/v1/ml) and each unrouted leaf by name.
/v1/train is deliberately NOT fenced: it was just deleted because the Kubeflow
CRDs behind it are not served. A gate must fence prefixes that exist, and naming
a deleted one would trip the non-vacuity check while saying nothing true.
It also does not judge the NAME. `ml` being the vague word is why this keeps
happening, but renaming a prefix that is published is a wire change with its own
cost; until that is worth paying the ambiguity is fenced, not resolved.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/train/* was a thin proxy onto two Kubeflow CRDs that this cluster does not
serve: trainer.kubeflow.org/trainjobs and kubeflow.org/{experiments,trials}.
GET /v1/train/health answers 503 degraded in production right now
({"crds":{"experiments":false,"trainjobs":false},"status":"degraded"}) with
nothing in cloud changed — the CRDs were retired underneath it. Ten operations
go, and with them the degraded door.
Measured before deleting, cluster-wide: zero TrainJobs, zero Experiments, zero
Trials, and no ClusterTrainingRuntime for a TrainJob to reference. The katib
half could not have worked at all — katib's admission webhook requires the
namespace label katib.kubeflow.org/metrics-collector-injection=enabled,
ensureNamespace writes only {managed-by, hanzo.ai/org}, and no namespace in the
cluster carries it. So POST /v1/train/experiments took the billing gate, created
an Experiment, and katib never admitted a Trial.
There were also TWO doors onto one TrainJob CRD: this one and the hanzoai/ai
broker at /v1/finetune/*, which has the product around it (presets, HF pickers,
status polling, deploy-to-serving). One door survives.
KServe STAYS. /v1/ml/models is the only path in the estate that serves a
classical artifact end to end, and it is proven: POST /v1/ml/models with an
sklearn joblib -> 201, the storage-initializer pulls the model, then POST
/v1/ml/models/{name}/predict -> 200 with correct predictions on the
kserve-mlserver runtime. GET /v1/ml/health is 200 today. Its runtime-capacity
clause and every serving test are untouched.
openapi/floor.json needs NO edit on this base: the shrink guard is a FLOOR, and
main's committed floor (1684 paths / 2336 operations) is already below the
post-deletion document, measured at 1689 / 2341. The guard still bites — raising
the floor above the real count fails TestFleetIsTheWeaveOfItsApps with the same
"THE PUBLISHED SURFACE SHRANK" report, which is how these numbers were read.
(An earlier pass here lowered a floor that ALSO carried a per-product map; main
has since dropped that map, so the rebase takes main's shape unchanged.)
The billing-gate integration tests keep
their coverage by exercising the surviving create (POST /v1/ml/models) — the
gate is the shared create() body, not a per-kind one.
Also repointed every pointer that named the deleted route, so none dangles:
apps/engine's intentRefused reason and LLM.md (now /v1/finetune/jobs), spend.go's
routing-union example, apps/platform/drift.go's analogy, and the mutation in
scripts/mutate.py whose anchor line and target test are both gone (it would have
reported ANCHOR-MISS).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
getUser sent the `<owner>/<name>` composite as `id`, and IAM wants the two as
separate fields. Measured against the running service with the console client:
?id=hanzo/2d4d67ab-… 400 field "owner" is required
?id=hanzo/z 400 field "owner" is required
?owner=hanzo&name=z 200
So NO caller of this ever read a user row. The avatar write is where it became
visible — "photo stored but the profile could not be updated", with
`iam non-envelope response (400)` behind it — but moveUserToOrg carries the same
fault silently, and its failure mode is worse: the whole-row re-submit it feeds
would have nothing to re-submit.
`name` is the USERNAME (the row's own `name`, "z"), never the UUID that `sub`
carries — the row this now reads has id 2d4d67ab-… and name z, which is why
addressing it by the uuid found nothing.
The composite is KEPT for the one caller that has no owner to send: a first-run,
org-less user, where resolving their authoritative (owner, name) is the entire
purpose of the read. Refusing that here would break onboarding to fix the avatar.
The fake IAM now insists on the same shape, so a client that regresses to `id`
for a caller that HAS an owner fails in the suite instead of in production.
A sibling dissolved PUT /v1/risk/state/appetite while this branch was open (the
decision regime has one address now), so typed.go conflicted on the op that used to
sit between `state` and `snapshot`. Resolved by taking MAIN'S typed.go whole and
re-applying this branch's five changes onto it, rather than by editing the conflict
markers: the incoming change deletes an op, and a hunk-level resolution is how a
deletion gets silently un-deleted.
zipdoc_gen.go is generated — resolved by regenerating from the merged source, never
by merging the artifact. Same for openapi.yaml and plugin/risk/openapi.json.
Verified after the merge: 11 operations before and after, floor.json identical to
main (risk 31, paths 1695, operations 2351), and main's own new gates — verb_test
and learn_cost_test — pass beside this branch's.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Regenerating every subset from source is the drift gate, and it turns up drift in two
apps beyond the ones whose prose this branch fixes. Both are real, and both are in the
direction of the published document being AHEAD of the code: it advertises money fields
the runtime stopped returning.
7b4ddd9e removed the mint from the referral read and 3a8be85b removed it from the promo
redemption. Both regenerated their zipdoc_gen.go, so the PROSE moved; neither
regenerated plugin/<app>/openapi.json, so the SCHEMAS did not. The result is 15
referral fields and 4 promo fields published as response properties no handler can
populate -- refereeGrantCents, referrerGrantCents, creditsEarnedCents, refereeBonusCents,
referrerBonusCents, grantedCents, creditsCents, the credited counters and the txn ids on
the referral side; creditCents and creditEntryId, plus the plan and seats request fields,
on the promo side. apps/referrals' assertNoMoneyKeys already fails the runtime response
if any of them reappears, so source and test agreed with each other and only the artifact
disagreed. Two fields the code does return -- Redemption.discountCents and
sweepResult.qualified -- were missing for the same reason.
Every SDK, the MCP tool list and the CLI are projections of this file, so those were dead
money fields in every generated client's types.
The operation-count ratchet did not and could not catch it: openapi/floor.json counts
paths, operations and per-product operations, and none of those move when a response
schema loses a field. It ratchets UP here -- billing 26 -> 27, operations 2351 -> 2352 --
for the newly described GET /v1/billing/tier.
Proved structurally rather than by diff, over the parsed documents with description and
summary elided at every depth: across openapi.yaml the only identity change is that one
operation GAINED, no operation was lost, no operationId moved, and all 22 contract
differences are the marketing and referrals schema fields named above. The commerce prose
entry contributes none of them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The repo's own gates could not pass on main. Both failures are the same shape --
prose that drifted from the routes it describes -- and both are fixed at the one
place the prose is written.
GET /v1/billing/tier published an operationId and nothing else, so openapi.Complete
refused commerce's document outright and `make -f mk/fleet.mk surface-check` died
there. Its handler is commerce's, in another module, and its path reaches the router
through a table rather than a literal, so there is no doc comment here for zipdoc to
lift -- which is what describe.go exists for, and what every sibling on that same
registration loop already uses. The entry states what the caller gets, that the
subject keys are pinned to the validated caller before the handler runs, that tier
is derived from active and trialing subscriptions, and the two rules a reader
otherwise gets wrong: gate on effectiveAvailable rather than prepaidAvailable,
because granted credits spend too and an account funded only by a grant reads zero
prepaid while holding real spendable credit; and a subscription-store error answers
500 rather than downgrading a paid subscriber to free.
POST /finance/starter was the inverse -- prose describing an operation that no
longer exists. The op, its middleware and its tests went with the automatic
money-mint (41b23f12) and the lift was never rerun, so commerce's registry still
explained a route the router does not carry. Regenerating drops it; nothing here
re-adds it.
POST /sites/resolve and /sites/resolve-org are live typed ops in
apps/projects/sites_rpc.go whose doc comments had never been lifted at all.
Verified per package, the way the generator loads: zipdoc -check is clean across all
100 directories that declare it, where commerce and projects were stale.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
THE MODEL WAS A PLACE. The `model` table is keyed on the tenant and written ON
CONFLICT DO UPDATE, so it is ONE CELL per organisation and every write destroys
the state before it. That single fact was three separate open problems: there was
nothing to roll back TO, so rollback needed an op that shipped the masses out to
the caller and an op that took them back in; two fitted models could not both be
named, so champion-and-challenger had nowhere to live; and no decision could say
which model produced it.
A content address ends all three. The value's name is a pure function of what
makes two models answer the same event differently — the shape, the geometry seed,
the position in the window, the threshold, the masses as IEEE-754 bits, and the
FOLD WATERMARK, which is not redundant with the mass count: two models with
identical masses reached by different routes disagree about what is left to fold,
and one will re-teach history the other will not.
THE IN-PROCESS MODEL STAYS MUTABLE, and that is the design. Measured: one encoded
value is 466 KiB at the reachable shape — 25 trees of 511 nodes over two windows,
every mass carrying a full mantissa because folding blends them. The sweep writes
every 500 events or every 30 seconds, so "a value per write" is 56 MiB an hour per
active organisation to record a counter going up. So identity is a SUCCESSION OF
STATES: the `model` row stays a place and its whole job is resuming a killed
process, and `published` is append-only, addressed by content, retained under a
budget stated in BYTES (10 values, following policyBudget's reasoning).
WHAT THIS DELETES. riskSnapshotBody is gone from the wire in BOTH directions, with
its two conversions. Publishing answers with a NAME; adopting takes one. The
masses never leave the organisation's own store, so the caller stops being the
custodian of a tenant's model — the engine's own Restore asks for exactly this
("it belongs where the tenant's own data belongs, and sealed if it travels"). Two
tests changed from refusing a threat to proving it unreachable: a caller can no
longer describe a model instead of naming one, so there is no body to compose and
no seed to choose. Rollback is naming a prior address; what the working state
descends from is DERIVED from its mass count, so rolling backward is right for
free where a stored head pointer is exactly what would fall out of step.
A SCORE NOW NAMES THE MODEL SPACE IT RAN IN. The shape is cached on the residency
and read in the same critical section as the verdict, so it costs nothing and
cannot cite a space the score did not run in. It is deliberately NOT an address:
the masses at the instant of a score are counters between two published values, so
citing one would claim that value produced this score. The shape, the policy
version and the event's own time are what IS true, and the value history's clock
brackets the decision from there.
ISOLATION, MEASURED RATHER THAN ASSERTED. The address deliberately omits the
organisation — a name that must be unguessable is obscurity, not isolation. Two
mutations were run and two drafts of the test comment were wrong before they were:
the per-organisation FILE and the `tenant = ?` row predicate are two layers and
EITHER ONE ALONE HOLDS, so a foreign org handed a real address resolves nothing
through all three doors with either layer removed. The single-layer regression is
caught where the file layer is already absent by design — two brands' identically
named organisations share one file — and that test fails on that one mutation
alone. The finding underneath inverts the obvious reading: the per-org file is not
what makes this isolated; the row predicate holds on its own.
11 operations before, 11 after: no path moved and the floor is untouched. The two
paths SHOULD collapse into one (/v1/risk/state/model, POST to publish and PUT to
adopt) — that is the prefix plane's call, not this commit's.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/risk/learn recorded a batch, trained on it AND answered the model's
verdict on every event. Three things under one name: you could not observe
without training, and could not train without being answered.
An observation is a VALUE the plane records; learning is a TRANSFORMATION over
observations; a verdict is a QUERY against the result. POST /v1/risk/score is
already the query and is already pure. So learn drops the verdict and answers
`learned` — how many events the model actually learned from.
WHAT IT COST TO CARRY. plane.learn called the engine twice per event, Inspect for
the response's verdict and Assess for the counters, and both enter the engine's
judge: two projections of the point at three aggregate reads each, and two walks
of the forest. Above the cut both also ran the counterfactual attribution, a
further walk per dimension over nine dimensions. Assess's own return was
discarded, so the attribution was computed twice and thrown away once.
MEASURED, and stated as measured (learn_cost_test.go, BenchmarkLearn):
before after
batch 8 31.5 µs/event 27.8 µs/event -12%
batch 128 20.0 µs/event 17.5 µs/event -12%
batch 128 6717 allocs 6169 allocs -8%
A tenth, not a half: the durable record and the aggregates are the larger part of
what a caller waits for, and the attribution is reached only by the share of the
stream the appetite admits — one per cent by default.
NOTHING DEPENDED ON THE SYNCHRONOUS VERDICT. No CLI command references risk, no
SDK carries a risk client, and the one in-process consumer of a verdict is
cloud.Decide, whose scorer is never installed outside tests (SetRiskScorer has no
non-test caller), so every question it asks answers {allow, scorer-absent}. The
live plane reports one resident model built once. Observe-and-judge in one round
trip is now a COMPOSITION and the published prose says which order: score first,
then learn, so the verdict is the model's opinion of an event it has not yet
learned from.
A DUPLICATE IS NOW WHOLLY INERT. It moved nothing before and was still judged;
now it costs no model work at all, is not counted in `learned`, and is not
metered — the meter runs on what was DONE, which is this app's own stated rule
for the gate/meter pair. The gate still bounds the batch the caller stated,
because how much of it is new is unknowable until the record is written.
TestPlane_TheVerbsStaySeparate (verb_test.go) walks the package and fails if
learn calls Inspect or score calls Assess. It is structural because the braid is
invisible behaviourally: with Inspect put back, every other test in this package
still passes.
zipdoc regenerated; plugin/risk/openapi.json and openapi.yaml rewoven
mechanically (make -f mk/fleet.mk openapi-weave, exit 0). riskScoreOut, riskCause
and riskValue stay in the document — score still answers them; they are simply no
longer reachable from riskLearnOut. Package green under -race.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/risk/state answered seven kinds of thing at once and PUT
/v1/risk/state/appetite answered all of them back to a call that changes three
numbers. This lands the half whose home already exists: the decision regime is
read and written at ONE address, /v1/risk/policy, and both verbs answer
riskPolicyOut. The published surface loses a path and keeps its operation count.
plane.appetite no longer computes the model value — r.mod.State and r.vel.strain
are gone from the policy write. What the model IS stays for the model-value track;
the telemetry kinds (refusals by reason, blind by feature, the realised share,
saturation, aggregate strain) stay on GET /v1/risk/state until they are emitted on
the fleet's one metric road, because deleting them before that would remove an
organisation's only view of its own refusals.
Per-organisation isolation is proven through the new address by the two policy
tenancy tests. Mutation table in the commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The regime was written at PUT /v1/risk/state/appetite and read at GET
/v1/risk/policy — two addresses for one plane, the writing one named after the
mutable spot the regime happened to sit in. `state` is literally the word for
that spot; a regime is not a spot. It is three numbers an organisation adopted,
at a time, by a named identity, kept under a version forever after.
Worse than the second address is what the write ANSWERED: riskModelState, fifteen
fields describing the whole MODEL — its shape, what it had learned, whether it
was warm, the threshold in force, the realised share beside the stated one, every
refusal by reason, every feature that read blind, the fold coverage of the event
surface, and the aggregate strain. A call that changes three numbers knows none
of that. It was reporting a model it happened to be holding a lock on, which is
the same braid, one level up, as the regime living on the learned state's row —
the defect the versioned policy record was cut to fix.
So the plane has ONE address and both verbs answer riskPolicyOut. plane.appetite
returns the VERSION it left in force and nothing else: the two calls to
r.mod.State and r.vel.strain are gone from the policy write, so a policy write no
longer computes the model value at all. policyOut is the one projection both verbs
render through, and the version in force is a PARAMETER to it — a write knows what
it enacted from inside the lock it enacted under, and re-reading it there would let
two concurrent restatements each report the other's version.
There is no `minted` flag. plane.enact is idempotent on the regime, so a
restatement answers the version already in force and equality of the VALUE is the
signal. A boolean about the operation is a second thing to keep true beside the
value that already says it.
The published surface SHRINKS by one path and holds its operation count: the write
joined the address that already existed instead of keeping a second one. Mechanically:
paths 1696 → 1695 (-/v1/risk/state/appetite)
operations 2351 → 2351 (PUT moved onto /v1/risk/policy)
openapi/floor.json is lowered in this commit because the weave refuses a shrink
otherwise, which is the gate working: a reduction is reviewed next to its reason.
The op's published schema description loses 27 lines of riskModelState /
riskSurface / riskAggregates prose it had no business publishing.
Nothing consumed the old address. Swept every checkout under ~/work: the only
references outside apps/risk's own tests were the derived specs. Live, PUT
/v1/risk/state/appetite is routed and GET /v1/risk/policy is 404 — the policy
plane is merged but not yet deployed — so this is the one moment the consolidation
costs a deployed client nothing.
Per-organisation isolation is untouched and proven through the NEW address:
TestPolicy_HistoryIsPerTenantOverTheWire and
TestPolicy_TwoBrandsShareAFileAndNotAHistory both state their regimes at PUT
/v1/risk/policy and both still see nothing of the other organisation.
Mutation table — each named test RED under the defect, GREEN after revert:
M1 re-register PUT /v1/risk/state/appetite TestPolicy_HasOneAddress RED
M2 put a model field back on the answer TestPolicy_AWriteAnswersThePolicyAndNotTheModel RED
Verified: go build -tags "sqlite_purego sqlite_math_functions sqlite_fts5" ./... =
0; go vet apps/risk = 0; go test apps/risk = ok; -race = ok (23s); zipdoc -check
./apps/risk/ = 0. Fleet-wide zipdoc -check reports the same 16 stale/missing
generated files before and after this change (measured by stashing it), so this
adds no drift: apps/risk is not among them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/iam/* was forwarded to a separate identity origin by an edge in cloud:
a tenant gate, an org pin, a read/write allowlist and a sign-in passthrough,
all re-deciding what IAM already decides for itself.
It mounted on `!cfg.Enabled("iam") && iamHost() != ""`. An empty enable list
mounts everything (Config.Enabled), and no deployment in the fleet sets one,
so Enabled("iam") is true everywhere and the condition never held: the iam
app owns /v1/iam through its graft, on every deployment, and has since the
graft landed. The edge answered no request anywhere.
Deleting it removes the second reader of the identity address and the second
place a tenant scope is decided. The prefixes are unchanged — manifest's iam
row already routes them — and the org pin the edge applied is IAM's own
authorize, in the process that owns the store.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Removing CLOUD_ENABLE/--enable (ecafb31c) deleted the flag from the binary but
left three launchers passing it. Go's default flag.CommandLine is ExitOnError,
so this is not a warning:
$ ./bin/cloud --enable=iam
flag provided but not defined: -enable
`make run` and the documented compose quickstart both died before boot, and the
four README brand examples were copy-paste instructions to do the same. The
removal was deliberate and test-enforced (cmd/cloud/mount_test.go asserts the
flag's absence) — the launchers were simply never updated with it. My miss.
RUN_ENABLE is renamed RUN_PLUGINS because it never mounted anything: it is the
list of plugin BINARIES to build so local dev does not build all 106. The host
mounts what manifest.Apps lists and resolves each plugin as a file beside
itself; a lazy one with no binary never starts, a Required one fails loudly.
Keeping the name would have said the binary takes a mount list, which is the
thing that took devnet down twice.
compose.yml's HANZO_ENABLE goes with it. Its environment: block is inert
anyway — HANZO_BRAND/DOMAIN/DATA_DIR have no Go read sites and work only
because the same file passes the flags, which main.go forward() republishes as
CLOUD_*. That is a separate cleanup, not this one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/risk answered ONE refusal in TWO shapes, with the same status and the same
sentence in both. Measured on this package's priced ops:
this app's own copy of the rule 403 {"status":403,"error":"no validated principal"}
the fleet's money door 403 {"error":{"code":"forbidden","message":"no validated principal"}}
The nested one is canonical — it is what the edge gate and every other Hanzo
surface emit, so a client that reads `error.code` reads it everywhere. The flat one
was zip rendering a returned error, which meant `error.code` was absent from
exactly one product's refusals.
[ops.gate] held its own `if ledger == ""` copy of the rule from before the fleet
door had one. [cloud.ResourceMeter.Gate] now refuses an empty org above both of its
branches as [cloud.ErrNoLedger], and [cloud.denial] renders that as the 403 with
this exact sentence, so the copy decided nothing except the shape. Deleted; the
twenty-four lines of prose explaining a defect that is fixed elsewhere are replaced
by a citation of where.
Deleting it exposed a residual the report did not name, because a status assertion
cannot see a shape: POST /v1/risk/search reaches [ops.admit] BEFORE it prices
anything, so its refusal came from [tenantOf] — still flat. Eight ops nested and
one flat is worse than nine flat, so tenantOf's no-principal branch answers with
the same fleet value. A malformed org is a different fact and keeps its own
sentence.
And the hole the deletion could have opened, closed: ResourceMeter.Gate returns
EARLY at zero cost, before its own empty-org refusal, and the price is an operator
knob where 0 is legal. So "the money door refuses an unidentified caller" holds
only while somebody is charged; the app's deleted copy had covered that by
accident, running before the price was computed. tenantOf covers it on purpose, and
a test sets the price to zero and requires the same 403.
E1 risk holds its own copy again TestPricedOps_RefuseAnUnidentifiedCallerInTheFleetsOwnEnvelope RED
E2 tenantOf answers flat again TestPricedOps_RefuseAnUnidentifiedCallerInTheFleetsOwnEnvelope RED
E3 tenantOf answers flat again TestPricedOps_RefuseAnUnidentifiedCallerEvenWhenTheOperatorPricesThemAtZero RED
E4 the fleet door stops refusing TestPricedOps_ResolveTheTenantBeforeTheyAskForMoney RED
E5 identity takes a money status TestPricedOps_ResolveTheTenantBeforeTheyAskForMoney RED
All GREEN after revert. E4 and E5 mutate cloud's own money door, which is where
gate_order_test.go's mutation note now points — the note said "delete the
empty-ledger guard from ops.gate and every op reports 503", and that had silently
stopped being true.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The label row said a specific prefix MUST be registered before the bare /v1/risk
or "every label op lands on the decision plane", and cited
TestSpecificPrefixesPrecede as the gate that pinned it. Neither half was true.
No test of that name exists anywhere in the tree, and the rule is not the
router's: registering the bare /v1/risk FIRST and all three specific prefixes
after it, the live router still delivers /v1/risk/labels to label,
/v1/risk/reference to reference and /v1/risk/datasets to dataset, and the router
oracle stays green. zip resolves nested static prefixes by specificity; mount
order decides only between EQUAL patterns, which is what the ai-before-zen note
is actually about.
Says that, and names the gate that does hold —
TestEveryServedPathReachesTheAppThatServesIt builds the real router from these
rows and asks it, per published path, which app receives the request. An oracle
over every row at once is why no row needs a rule of its own to remember.
Prose only; no row moves and no behaviour changes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The dataset plane addressed /v1/ml/datasets, beside the KServe model-SERVING
plane's own /v1/ml/health and /v1/ml/models — one prefix, two products.
openapi.Product reads an operation's product off the FIRST /v1 segment and
nothing else, so `ml` published 14 operations that were seven serving ops with
customers on them and seven dataset ops whose rows feed the risk model, which
learns in-process from the org's own events and is never served by KServe.
Moves the five paths to /v1/risk/datasets and renames the seven operation ids
and nine schema names to the risk face, so the SDK method, the CLI command and
the generated type each name the product they belong to. Fourteen of the fifteen
Go types take the bare risk<Noun> name; the dispose pair carries the noun it
disposes of, because apps/label already publishes riskDisposeIn for LABEL
disposal and the fleet's schema namespace is flat.
The tag was never the product: openapi.Fold assigns op.Tags from the router
projection, so zip.WithTags("ml") had been inert. It now reads "risk" too, so a
declaration and a projection do not disagree.
/v1/ml is untouched. Its four serving paths (7300 bytes) and its three schemas
(1533 bytes) are byte-identical, and regenerating the fleet document from source
moves exactly the seven dataset operations and the nine dataset schemas: 2350
operations, 1695 paths and 2060 schemas before and after, with every other
triple unchanged.
No alias and no redirect from the old prefix, because nothing calls it: a sweep
of every git checkout under work/hanzo finds /v1/ml/datasets named only in this
repo's own generated artifacts and prose. No client, SDK, CLI or MCP consumer
names it.
The floor ratchet refuses a shrink, so `ml: 14 -> 7` is lowered here in the same
commit that moves the routes; `risk` rises 23 -> 30 on the same regeneration.
address_test.go makes the address a gate rather than a third recollection after
label and reference: the product, the operation-id prefix and the schema prefix
are each asserted off the published projection, and each assertion fails when
the projection is empty, so none of the three can pass by examining nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
kserve admits an InferenceService whose model format no ClusterServingRuntime
supports and then never schedules it, so /v1/ml/health answered 200 while every
deploy hung. A served CRD is not capacity, and the probe read only the CRD.
health now takes the cluster-scoped coordinate the plane needs at least one of
(the zero GVR for a plane with no such fact — training carries its own images)
and reports the count as its own field. An unreadable list reports the read
error instead, because a missing grant is a broken probe and not an empty
cluster, and the two call for different acts.
scripts/mutate.py carries four rows: dropping the clause, folding the read error
into the count as zero, asking capacity of training, and reading the runtime kind
at the InferenceService's v1beta1 instead of v1alpha1.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The bounds on `review` and `sample` were spelled twice — once at [ops.appetite]
and once in [admitRegime]. Collapsing them to one spelling is right. But the
SURVIVING spelling had no test: admitRegime was made to `return nil`
unconditionally and the WHOLE package stayed green, so the published contract
(`review ∈ (0, 0.5]`, `sample ∈ [0, 1]`) was held by code that could be deleted
without one failure. Same shape as a control switching itself off, one layer down:
the bound is present and nothing measures it.
Both halves are asserted, because a refusal that half-applies is worse than no
bound: six out-of-contract appetites are refused 400 with the field named, AND the
regime in force is untouched afterwards — no version minted, no history row, the
model still deciding under 0.02/0.10. Over the WIRE, because that is where the
contract is published and where the op's deleted copy used to answer.
A1 admitRegime returns nil for every regime named test RED, whole package RED
A2 plane.enact stops calling admitRegime named test RED, whole package RED
Both GREEN after revert.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An organisation that took its model out of shadow BEFORE the model had learned
anything was told live=true and had nothing written down. The regime lived on the
same row as the learned state, and that row's writer declines to write while the
snapshot holds no learned mass — correctly, because there is no state to lose. So
PUT /v1/risk/state/appetite answered 200, reported live, and persisted nothing;
this binary deploys Recreate at one replica, so the next rollout rebuilt from
defaultConfig — shadow — and the model decided nothing. No error, no log, nothing
to alert on: a model silently disarmed, on a routed door.
Two conflicts, both resolved by REGENERATING rather than choosing a side:
openapi.yaml and openapi/floor.json are derived, and the branch was cut when the
fleet had 1684 paths. `make -C apps/risk describe` + the weave produce 1696 paths
and risk 24 operations — main's 1695/23 plus this branch's one op, GET
/v1/risk/policy. apps/risk/zipdoc_gen.go and plugin/risk/openapi.json regenerate
byte-identical to the branch's, so the projection did not drift.
One semantic conflict: `plane.close` gained the shutdown window on main
(cda5a031), so the branch's four `close()` calls in policy_test.go take
context.Background() like the other eighteen in the package.
Mutation table re-run against THIS merge, since main moved under the branch —
each named test RED under the defect, GREEN after revert:
M1 plane.enact writes no durable row TestPolicy_GoingLiveSurvivesTheRollout RED
M2 verdict drops the version citation TestPolicy_EveryScoreCitesTheRegimeItWasDecidedUnder RED
M3 no value-equality short circuit TestPolicy_ARestatementOfTheSameRegimeMintsNoVersion RED
M4 the 24-per-24h bound deleted TestPolicy_TheRateBoundBindsAndIsNamed RED
M5 a pre-existing regime is not adopted TestPolicy_ARegimePredatingTheRecordIsAdopted RED
M6 the history read loses its predicate TestPolicy_TwoBrandsShareAFileAndNotAHistory RED
M7 retention stops disposing TestPolicy_RetentionIsCountedAndNotSilent RED
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Four defects on the LIVE /v1/risk plane, each closed so the wrong thing is
unrepresentable rather than merely unlikely, plus the harness rows that hold
them.
THE ORG STORE. CloseAll emptied its maps, which made "closed" indistinguishable
from "nothing opened yet" — so a request still in flight during a rollout
reached For() after shutdown and opened the file again, re-hydrating it and
re-claiming the fence lease the SUCCESSOR pod was claiming. Closing is now
terminal and every door answers ErrStoreClosed.
THE ROLLOUT. close() was `stop(); wg.Wait()` with every save queued BEHIND it.
The wait was unbounded inside a 30-second window while ONE search's durable
Sync is bounded at 30 seconds on its own, so the process was killed with not
one tenant's model written down — every tenant back to warming, and a warming
model refuses to score, which reads as clean. The saves now run
unconditionally; the drain gets what is left of the caller's own window; a
drain that did not finish is ErrDrainIncomplete and not a silence.
THE SEARCH BOUND. "ONE RUN PER TENANT" checked the slot and set it two
warehouse operations later, so sixteen concurrent callers each rolled the
tenant's source planes and read its whole history before any of them claimed
anything — and the ledger gate was the LAST thing in the sequence, so all of it
was free. The slot is claimed atomically with the check, and a search is priced
in TWO halves, each before the half it prices: the surface on the window the
caller stated, the grid on the measured history.
THE STRAIN REPORT. velocity caps PER SHARD, so the store drops a tenant's
subjects long before the flat census notices: 200 subjects in, the store holds
198, and the report said forgotten=0, saturated=false. Two of that
organisation's own subjects read as "has done nothing" and nothing said so.
reconcile now measures the store/census shortfall as a high-water mark, and the
report describes the bound that actually binds.
THE RESIDENT BOUND had no test at all: evict() could be made to return nil —
the bound fully disarmed, an OOM on a one-replica Recreate deployment — with
the whole suite green. Its operating point is now four properties held
together: SERVED past the bound, BOUNDED at it, LOUD on the probe, LOSSLESS on
the way out.
Mutation-proved: scripts/mutate.py "risk:" — eighteen rows, each reintroducing
one defect and going red on the named test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The ceiling was the wrong way round. maxOverrides was a chosen number (10,000)
and the byte figure was computed beside it from `row` — the SUM OF THE WIRE
TERMS, 1,152 bytes. A row does not cost what it carries: measured on a real
per-org store, the widest row this door admits costs 1,952 bytes once the
implicit index over the primary key, page slack and the encryption are counted.
So the published 120 MiB per-organisation ceiling was understating the truth by
1.69x, on the ONE volume every organisation's store shares.
Inverted, so the bound is in the dimension that binds:
ownBudget = 128 MiB the primary figure — what one org may occupy
rowBytes = 2048 MEASURED (1,952) plus a page of room
maxOverrides() = ownBudget / (sets x rowBytes) = 5,957 per set
The count is now the division rather than a number standing next to one, so
count x rowBytes IS the byte bound. Adding a set lowers the count instead of
quietly multiplying the volume.
`row` is renamed `stated` and keeps its old meaning — the WIRE width — because
the two figures are different quantities and conflating them is what caused
this. TestTheVolumeOneOrgMayOccupyIsStated now checks the quotient (one more
entry per set must not fit) instead of pinning a constant, and
TestOneOrgsOverridesCostWhatTheyArePublishedToCost fills a real store with
worst-case rows and fails if one costs more than the figure the count is divided
from — so rowBytes can never drift from a store again.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The header claimed "nothing held the mechanism at all — the whole suite stayed
green". That was true when it was written and is not true now: hold_test.go
landed the cross-tenant reclaim tests on main, and MEASURED with
scripts/mutate.py, TestEviction_IsCountedOnTheProbe and
TestEviction_WritesTheVictimsStateDownFirst kill all three mutations that disarm
this bound. A test whose stated reason for existing is false is a test the next
reader deletes for the wrong reason, or keeps for one.
What it actually adds is two things the pair cannot see: the bound asserted at
every one of maxResident+8 arrivals rather than once just past it, and a lossless
leg with no t.Skip exit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The surface read — rolling up to four source planes into the organisation's own
feature surface and reading the window back — ran BEFORE ANY GATE AT ALL. A
caller with no balance drove the whole warehouse cost, was refused at the very
end, and paid for none of it, as often as it cared to ask. Two rows: the gate
moved back below the work it prices, and the surface half deleted so only the
grid is paid for. Each goes red on the test that names the property.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Sixteen rows for the org-store lifecycle, the rollout, the search bound, the
strain report and the resident bound — so every assertion this cycle added is
held by a mutation that breaks the thing it guards, in the harness rather than in
a transcript.
AND THE HARNESS COULD DESTROY ITS OWN EVIDENCE. The restore is in a finally,
which does not run when the process is killed — a CI timeout, a ^C — so an
interrupted run leaves the mutant in the tree and the original in .mutbak. The
next run then copied over that backup, destroying the only clean copy, and
reported ANCHOR-MISS on a tree it had silently corrupted. That happened here and
cost a file. A leftover backup is now a hard refusal that says how to recover.
One row is written against the CONDITION rather than as an early return, because
`return nil` after the guard is unreachable code: go vet rejects it, the row
scores NO-COMPILE, and a mutant that never ran proves nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The run slot now bounds one tenant's CONCURRENCY. Nothing bounded its RATE: the
setup ran before any gate at all — the ledger check was the LAST step of begin —
so a caller with no balance drove a roll of up to four source planes and a
full-window read on the one warehouse every tenant shares, was refused at the
end, and paid for none of it, as often as it cared to ask.
Pricing the grid on its upper bound would close the same hole and leave no
viable operating point: the upper bound is maxHistory x the whole grid whatever
that organisation's history holds, so a tenant with two hundred events would be
refused unless it could cover twenty thousand.
So a search is priced TWICE, each half before the half it prices. The surface
from the WINDOW, which the caller states and which is therefore a number before
anything runs — the same unit and the same price ops.features already pays for
the same work, now spelled once in windowScreens. The grid from the MEASURED
history, where it already was. Each meters on what it actually did, so a run
cancelled by a rollout is still billed for the trials that ran.
The plane takes ONE money seam (charge: gate n, get the meter for what was done)
instead of a gate parameter and a book parameter, which is what makes "gate
before, meter after" the same shape at both halves rather than a convention.
And the debit test now waits for BOTH debits. With a synchronous surface debit
landing first, waiting for one was satisfied without the background meter ever
running — the fixture would have made the property it exists for unobservable.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
velocity applies its cap PER SHARD — MaxKeys/shards+1, which is FIVE keys against
a published census ceiling of 320 — and subjects hash unevenly, so a shard fills
long before the total does and the store drops that shard's least-recently-used
key. The census is a flat count and saw none of it. Measured:
subjects=200 storeKeys=198 censusLen=200 forgotten=0 SATURATED=false
subjects=320 storeKeys=290 censusLen=320 forgotten=0 SATURATED=true
At 200 subjects two of that organisation's own subjects are gone and the model
reports itself healthy; each one reads as "has done nothing", scores as
unremarkable, and raises nothing. At 320 the state is right by luck and the COUNT
an operator acts on still says zero while thirty subjects are gone. The published
8 MiB budget buys 320 subjects and the bound that binds starts biting near 200 —
a bound stated in one dimension and enforced in another.
reconcile already read the store's own key count and threw the comparison away.
It now measures the shortfall against the census under ONE lock, so the two
numbers describe one moment, and keeps it as a HIGH-WATER MARK: a dropped subject
is re-admitted the moment it is active again, which closes the shortfall but does
not un-blind the window in which that subject read as inactive. A gauge would
report that a control which switched itself off never did.
ALSO: the resident bound had no guard at all. `evict` could be made never to trip
and the whole suite stayed green — the bound fully disarmed, residents growing
without limit, which is 64 x (8 MiB of rings + its model) against a 9 GiB
GOMEMLIMIT on a ONE-replica Recreate deployment. A bigger constant was never the
fix; the operating point is four properties at once, and each is a way to be
wrong: SERVED past the bound (a refusal is a cliff, not an operating point),
BOUNDED at every step, LOUD on the probe (eviction is lossless, so the count is
the only sign), LOSSLESS (otherwise one tenant's arrival costs another its model,
which is the cross-tenant defect wearing a capacity hat).
Mutation-proved: scripts/mutate.py "strain"/"bound" — seven rows. Dropping the
store's half of Saturated reports "the store holds 142 of this organisation's 144
subjects and strain reports saturated=false". Refusing past the bound reports
"organisation 64 of 72 was refused a residency". Dropping the evicted tenant's
save reports "an evicted organisation came back having learned 0 of 120".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TWO DEFECTS, both "a bound that is not on the dimension that matters".
THE ROLLOUT LOST EVERY TENANT'S MODEL. close() was `p.stop(); p.wg.Wait()` with
every save BEHIND that wait, and the wait had no bound at all — while Shutdown
was handed the shutdown window as a context and threw it away. The process gets
30 seconds (serve.go) inside a 60-second grace period, and ONE search finishing
after cancellation writes its result to a shelf whose durable Sync is bounded at
durableOpTimeout — thirty seconds, the whole window, on its own. So the wait
outlived the window, the pod was killed, and not one resident model had been
written down. Every tenant returned to warming, from one tenant's search, once
per deploy — and a warming model REFUSES to score, which reads as clean to
anything that does not check the refusal.
The saves now run unconditionally and never behind the drain; background work
gets whatever is left of the caller's own window, bounded by it and by
drainBudget; and a drain that did not finish is ErrDrainIncomplete — a named
state joined into the error, with the count of models written down anyway, rather
than a silence.
THE SEARCH BOUND BOUNDED NOTHING EXPENSIVE. begin() checked "is a search already
running for this organisation?" and set the flag that answers it two warehouse
operations later — check-then-act with the entire cost of a search setup in the
window. Measured: 16 concurrent callers for ONE organisation were all accepted,
all rolled that tenant's source planes, and all read its whole history, against
the one warehouse every tenant shares; 16 background grids then raced for one
shelf row. The slot is now claimed atomically with the check that grants it,
before any of that work, and released on every path that does not reach the run.
claim/settle/unclaim are one fact in one place — the background goroutine's
hand-rolled delete is gone with them.
Mutation-proved: scripts/mutate.py "rollout"/"search" — six rows. The original
close() ordering does not merely fail, it HANGS to the test timeout, which is the
production SIGKILL. Making the saves conditional on a clean drain reports "tenant
A came back having learned 0 of 300". Restoring the check-then-act reports "16 of
16 concurrent searches were accepted".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
OrgStore.CloseAll closed every handle and then installed fresh empty maps, which
made "closed" indistinguishable from "nothing opened yet". Every one of the
fifteen callers is a Shutdown path, so a request still in flight during a rollout
reached For() after it and opened the file again — on a durable deployment
re-hydrating it and re-claiming the fence lease the SUCCESSOR pod was claiming at
that moment. Two live writers for one org, through a handle nothing thought was
reachable.
Closing is now terminal: the flag is set before the maps are cleared, so there is
no window in which the store is empty and still openable, and every door that can
open a file answers ErrStoreClosed. An arrival after shutdown is a fact worth
surfacing, so it is an error rather than a silent no-op.
The risk plane reaches this path by construction — its own close() empties the
resident map, so an in-flight score rebuilds a residency and asks for the shelf.
Mutation-proved: scripts/mutate.py "orgstore" — three rows, one per half of the
mechanism (the flag, the For guard, the Each guard). Each goes red on
TestOrgStoreCloseAllIsTerminal at a distinct assertion.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ownVolume() is what one org may occupy on the volume every org's SQLite file
shares: maxOverrides x len(Catalog()) x row, where row = maxKey + maxNote + 128.
The key and the note were bounded at the wire door. The third term was not: the
writer is actor()'s reading of the X-User-Id the request carries, so it was a
caller-sized value stored on up to 10,000 rows in every set the catalog
publishes. A count over caller-sized values is not a byte bound, and the stated
120 MiB ceiling was a figure nothing held to.
maxActor = 128 (three times the UUID IAM mints), enforced at the store door
rather than the wire op — the row is what the budget is about, so bounding it
where a row is written covers every path that reaches the store and not only the
one a reviewer read. row now DERIVES from it, so raising the bound moves the
ceiling instead of silently detaching it.
Refused, never shortened: the writer is who an adverse action is attributed to,
and a truncated identity names someone who does not exist. errActor separates
the two refusals put() can produce — the row cap is a 409 (the org already holds
what it holds), an over-long writer is a 400 (a value that arrived on this call).
TestTheWriterOnARowIsBoundedLikeEveryOtherTerm rides the wire, because the
defect was in what the wire hands the store: a unit test on put() would have
passed against a handler that never bounded the header. call() is now callAs()
with the default principal, so there is one request builder and not two.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/ is where mounted, billable subsystems live: 116 of its 139 directories
declare `func Mount(`. apps/money declares none, serves zero routes, and is 216
non-test lines that already wrap github.com/hanzoai/money (imported as `hz`) to
pin ONE unit — creditUSD, dollars at 18 decimals. That is a library, and filing
it under apps/ said it was a product.
Pure path move: apps/money -> money. No API change, 44 import lines rewritten,
0 residual references. Full tree builds (`go build -tags sqlite_math_functions
./...` rc=0).
Not deleted and not folded upstream: hanzoai/money is the general library
(multi-currency, rates, min/max/sum) and this is the platform's credit-asset
facade over it. creditUSD is Hanzo-specific and does not belong in a public
money library. One library, one facade, each in the right place.
Unrelated, found while verifying: `go build ./...` fails on
hanzoai/base@v1.5.11/core/sqlite_math_required.go (undefined
cgoBuildNeedsSQLiteMathFunctions) on a CLEAN tree too — it needs
-tags sqlite_math_functions. The default build command does not work in this
repo and that is worth fixing separately.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ResourceMeter.Gate is the one pre-create gate all 23 priced surfaces call, and
handed an empty org it asked the money plane to price a spend for a nameless
subject. The money plane then answered a question about IDENTITY in the
vocabulary of MONEY, in two measured shapes:
co-resident ledger — metering refuses an empty org fail-closed, which is not a
4xx, so the wire's fallback renders 503 "Billing temporarily unavailable":
the caller is told the biller is broken.
peer ledger — gatePeer ships AuthorizeIn{Subject:""}, commerce's own
validate:"required" rejects it, and because that refusal IS a 4xx the wire
preserves it verbatim: 400 `field "subject" is required` — a field that
appears in no published request schema on any of these surfaces, so no
caller can ever satisfy it.
apps/risk already carried this guard at its own door; the same disagreement is
reachable wherever a handler's tenant check and principal.Ledger differ.
provisioning is the live instance: create() admits an admin with no org, and an
org over MaxOrgLen, through tenant() — and Ledger answers "" for both. Its seven
routes serve on api.hanzo.ai today.
An empty org is now refused as ErrNoLedger before either branch runs, and denial
— the one decision both renderings read — renders it as the tenant gate's own
403 rather than as a fault of the biller. This changes no allow/deny outcome on
a named caller: an empty org already failed on both branches. It does close a
worse case than the wrong status, proven by the mutation below: under fail-OPEN
with the biller down, an unidentified caller was allowed, so the surface was
free rather than gated.
Mutation-proven via scripts/mutate.py:
gate: ask the money plane to price a spend for a nameless subject KILLED
gate: render an identity refusal as a fault of the biller KILLED
risk: widen the empty-ledger guard until it refuses every caller KILLED
Two rows are deliberately absent and say so in place: "guard below fail-open"
is a semantic no-op (fail-open lives inside Authorize/gatePeer, so no placement
in Gate can follow it), and "delete ops.gate's guard" now SURVIVES — with Gate
refusing, the risk-level guard is no longer load-bearing for the status or the
sentence, only for the envelope.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
plugin/<app>/mcp.json stopped being a generated artifact when the door started
asking each subsystem for its tools at the moment it is asked (package fleet,
mk/plugin.mk describe). The committed copies were the exact hazard that change
removed: plugin/o11y/mcp.json held 12 tools while the o11y binary served 365,
and nothing compared them.
plugin/dataset/mcp.json was the last one left — no go:embed names it, no Go
source opens it, and every remaining mention in the tree is prose recording that
the mechanism was retired. Removing it leaves 0 of 124 app dirs carrying one, so
"the tool catalogue is not an artifact" is a property of the tree rather than a
sentence in a comment.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The last attempt at this served a plane op from deps.AI, which in the `ai`
process is the HTTP gateway to api.hanzo.ai — so it added a hop instead of
removing one, and was reverted. The premise was right and the mechanism was
wrong. zip states the mechanism itself:
"Because it is an ordinary route on the app, it rides EVERY transport the
app Listens on with no extra wiring — ZAP over a unix socket is simply the
address the caller dialed."
`ai` already answers its whole /v1 surface on $ZIP_RUNTIME_DIR/ai.sock. Nothing
in hanzoai/ai has to change and no second inference API has to exist: a sibling
speaks the SAME OpenAI-compatible wire to the SAME routes, over the socket.
before sibling --HTTPS--> Cloudflare --> ingress --> api.hanzo.ai --> ai
after sibling --unix----> ai
AIHTTPOn / AIHTTPM2MOn state the TRANSPORT separately from the address, so
there is still ONE inference client; only the route it travels differs. The
M2M token exchange keeps the default transport — IAM is a different peer and
naming it is a separate question, still open.
aiRoute is the one decision both pickers share, so completions and embeddings
cannot disagree about where the peer is. It is answered by WHAT THE PROCESS IS:
!Enabled(ai) means this process does not carry the app, which is exactly when
`ai` is a sibling. The ai process and the host keep the configured address —
routing inference back through the picker there would be a self-call, and
TestTheAIProcessDoesNotDialItself pins it.
The socket transport WAKES the peer through the same reach() every plane call
uses, so a cold lazy app is "not started yet" rather than "not deployed here".
CLOUD_AI_ZAP_ADDR is deleted with its field: it made a deployment state where
its own code lives, which is the thing being removed. CLOUD_AI_BASE_URL now
only describes inference that is genuinely elsewhere — and on the Hanzo
deployment it is no longer consulted, because that pod carries `ai`.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The app was renamed to /v1/risk/reference — openapi.Product reads the product
off the first /v1 segment, and these six operations are the risk product's —
but the committed projection and the woven document were generated before the
rename and still published /v1/ml/reference. manifest's
TestEveryServedPathReachesTheAppThatServesIt is the gate that caught it: four
paths the fleet published and routed to the /v1 catch-all instead.
Regenerated plugin/reference/openapi.json from the app's own live router and
re-wove openapi.yaml from the subsets. The delta is exactly the four reference
paths; no neighbour moved.
plugin/reference/mcp.json goes with it. The tool catalogue stopped being a
generated artifact when the door started asking each subsystem for its tools at
the moment it is asked (package fleet) — 122 of 124 apps ship no such file, and
a stale one beside a regenerated projection is a second answer to a question
that has one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The coordinates are the part worth proving, and ai's own tests cannot reach them.
ai declares the store structurally (object.SecretStore) precisely so it needs no
KMS dependency — which means its tests can only use a fake, and a fake proves the
logic while proving nothing about where the bytes land.
A bare ref resolves to path "/", which fileOrg treats as the FACADE and lands in
the deployment's system partition. The REST surface folds the caller's org and
lands the same name at /orgs/{org}. Those are different databases. Write through
one door and read through the other and the secret is simply not there: no error,
no warning, just a key the gateway cannot find. This asserts the admin seal and
the completion-path resolve use one door.
Second test pins the migration's direction: env serves until a key moves into KMS,
and KMS wins after. Without it "migrate the key" could be a no-op that looks
complete.
Verified on Linux, where libsqlcipher is linked and the store actually opens.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The commit claimed a sibling would stop reaching `ai` through Cloudflare. It
does not. In the `ai` process cfg.Enabled("ai") is true, so deps.AI is the HTTP
gateway pointed at https://api.hanzo.ai/v1 — the pod's own public address — and
the plane handler I added served inference THROUGH it. The path became
sibling --UDS--> ai process --HTTPS--> api.hanzo.ai --> airouters
where before it was one HTTPS call. Strictly worse, and the opposite of what
the message said.
The premise was right and the wiring was wrong: `ai` exposes its inference as
HTTP ROUTES (airouters), with no in-process ChatCompletion to call, so serving
the plane op from deps.AI just re-entered the transport. Closing this for real
means an in-process entry point in hanzoai/ai that the plane op can call
without a socket — a change in that module, not a rewiring in this one.
Reverted whole rather than patched so main does not carry a half-measure that
reads, from the commit log, as if the round trip were gone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`ai` is a plugin of this same binary running as its own process, and the only
way a sibling could get a completion was its PUBLIC address. So the pod left
through Cloudflare, performed a client_credentials exchange to authenticate to
its own deployment, and came back — to reach code one socket away. Its own
startup line said so:
deps.AI (completions) -> HTTP gateway (IAM M2M)
base_url https://api.hanzo.ai/v1 <- the pod's OWN public address
token_url http://iam.hanzo.svc/... <- a token to call itself
Every part of that was a consequence of addressing a peer by URL. plane.AIChat
and plane.AIEmbed are addressed by APP NAME — zip.SocketPath resolves it and
reach() starts the app if it is not listening — which is how the meter already
debits commerce and how the gate already reads the ledger. There is no address
to configure, no credential to mint, and no second answer to where `ai` is.
Which client a process gets is decided by WHAT IT IS, not by an env:
!cfg.Enabled("ai") means this process does not carry the app, which is exactly
when `ai` is a sibling. The process that IS `ai` (and the host, which carries
everything) falls through to the real transport — asking the plane there would
be this process calling itself, and the test pins that.
CLOUD_AI_ZAP_ADDR is deleted with its field: it was the previous attempt at
this, and it still made a deployment state where its own code lives. The
gateway stays for inference that is genuinely REMOTE — a different fact, not a
second way to reach the same thing.
Billing is deliberately not on the servant side: the caller reserves before it
asks and settles on the reported usage, so pricing it again there would bill
one completion twice. MaxTokens rides the request so the ceiling the caller
reserved against is the ceiling the servant honors.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The upload reached S3 and the profile did not move: 502 "photo stored but the
profile could not be updated", and the log said why — `iam non-envelope
response (400)` for id hanzo/2d4d67ab-….
IAM parses a user id as `<owner>/<name>` through GetOwnerAndNameFromId, and on
the direct-Bearer path X-User-Id is a UUID subject, so `<owner>/<uuid>` names no
user. keyID() is the composite that resolves — the key ops (mint/revoke) already
use it for exactly this reason, and the comment explaining it sits in the file I
edited. On the gateway path username == name, so the two are the same value and
nothing changes there.
Found by uploading a real PNG to production rather than by reading the route
back. The honest failure is what made it findable in one step: the handler
reported that the bytes had landed and the record had not, instead of a 500 or a
success the user could not see.
The branch carried a second one: a root `cloud.Qualify` returning `type Tenant
string`. Two spellings of the key the dataset plane writes, the risk plane reads
and this plane joins on — and the two did not agree.
It did not canonicalise the brand. A deployment started with CLOUD_BRAND=Hanzo
filed `Hanzo/acme` while apps/dataset asked for `hanzo/acme`. Silent,
permanent, no error: one business, two key spaces.
It did not ask the registry. A brand nothing vouches for minted a key the
writers of these surfaces refuse to produce.
`Tenant` is a STRING type, so it can be written as a literal in any package and
decoded straight out of a request body. `tenant.Key` is a struct with one
unexported field: no literal, no JSON, no caller-asserted tenancy. Only
tenant.Mint and tenant.Of produce one.
tenant.Of additionally compares the token's verified issuer brand against the
deployment's — a token minted by lux.id cannot become `hanzo/acme`.
So the root file goes and apps/label takes tenant.Key/tenant.Of throughout. The
door shrinks: tenantOf no longer re-checks the shape it just minted, because the
mint is the check.
Mutation-tested at the ONE place the property now lives: drop the case fold in
tenant.canon and TestTheMintAgreesWithTheWriterOfTheSourceTable reports
`Mint("Hanzo","acme") = "Hanzo/acme"` against a writer that writes `hanzo/acme`,
RED; restored, green. 67 label tests pass on the ported key.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Eight of the ten declared /v1/risk paths answered a 400 that named a field the
caller cannot send. Measured on api.hanzo.ai with each operation's own declared
body: score, learn, features, state, state/appetite, state/snapshot,
state/restore and search/{id} all returned
{"error":{"code":"","message":"field \"subject\" is required"}}
`subject` is plane.AuthorizeIn's field, not any risk operation's. It appears in
no published request schema on this surface, so no caller could ever satisfy it
— a door that answers and cannot be opened.
ops.gate re-derived the caller's ledger with principal.Ledger, which answers ""
for exactly the requests the tenant gate refuses (it composes the same Validated
check), and then handed that empty ledger to the money plane. Asked to price a
spend for a nameless subject, the money plane answered in the vocabulary of
money about a question of identity, in two shapes:
co-resident ledger — metering refuses an empty org fail-closed, which is not a
4xx, so the money wire's fallback renders 503 "Billing temporarily
unavailable": the caller is told the biller is broken.
peer ledger (what deploys) — the gate ships AuthorizeIn{Subject:""} over the
internal plane, commerce's own `validate:"required"` rejects it, and because
that refusal IS a 4xx the money wire preserves it verbatim as the 400 above.
ops.search never had the defect, for the one reason that it reaches ops.admit
before it prices anything. This makes that ordering general: an empty ledger is
an identity refusal, answered with the tenant gate's own sentence from the one
function that owns it, before the money plane is asked anything.
The route itself was never missing. GET /v1/risk/score is a 405 here — the route
is declared, the verb is not — and the report of a 404 traces to the deployed
edge flattening that to plain-text "not found", byte-identical to an unregistered
path. Named in the report; not reachable from this suite.
TestTypedOpsRefuseAnUnvalidatedPrincipal asserted this exact property and passed
throughout, because it mounts deps with no metering client: with no client the
money gate is a no-op and the op fell through to the honest 403 the test wanted.
The new tests mount through mountBilled, the only fixture in which the ordering
is observable. A companion assertion counting calls to the ledger fake was
written, found unfalsifiable — an empty org is refused inside AuthorizeVerdict
before any HTTP request, so the counter reads zero either way — and deleted
rather than shipped; the reason is recorded where it would have gone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`oversize` is on the wire, so it is in the document: the app's own subset, the
prose map and the woven openapi.yaml every SDK repo pulls. Generated by
`make -C apps/dataset describe` + `make -f mk/fleet.mk openapi-weave OUT=openapi.yaml`
— no hand edit. No new operation, so the floor is unmoved.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A COUNT OVER CALLER-SIZED VALUES IS NOT A BOUND. maxRows capped how many rows a
materialisation holds and `page` capped how many an export returns, and neither
bounded a single byte. A row's coordinates are float64s this plane counts, but its
SUBJECT is a string this plane does not write: the rollup lifts it from
`distinct_id`, `session_id` and `user_id`, which arrive on /v1/event from the
caller. distinct_id is capped at 256 bytes on the anonymous lane and REPLACED by
the token's own subject on the signed one — but session_id, which the `session`
rollup files as a subject verbatim, is capped nowhere. So "200k rows of ten
float64s plus a subject key is tens of megabytes" and "eight jobs is a few hundred
megabytes" were arithmetic over an unknown, and one tenant's traffic decided the
real number for the whole process.
maxSubjectBytes bounds the one caller-sized value a row carries, at 256 — what the
identified lane already states for a subject, for the reason that carries over
unchanged: a minted id is a uuid, the value is KEYED, and something longer is not
an id. With it, count times max IS the byte bound, and every byte figure the
package states is now DERIVED from it rather than written down beside it:
maxRowBytes, maxResidentBytes, maxProcessBytes, maxPageBytes. The stale prose
claims are gone rather than corrected — that was the other spelling.
ONE ENFORCEMENT POINT. `representable` is the predicate and the only place it is
written. It is applied on the way OUT of the source, which is the way IN to this
process and to the rows table, so no read path needs a second check: an export
page is bounded because every row it can return already came through it.
AND THE DEGRADATION IS NAMED. A bound that quietly drops rows is worse than no
bound — the dataset that comes back looks complete, and a model fitted on it is
blind to a population nobody can see was missing. The census measures both halves
on ONE pass (conditional aggregates, not a filter), the excluded subject count
rides on the version, the manifest, the lineage and the wire, and it is in the
source fingerprint, so `reproducible` is measured over it too: a window that grew
a subject too large to carry is a window that MOVED, and lineage now says so.
census and facts cannot disagree about the population, because the predicate is
one expression used by both — measured against, then read with. Two spellings
would sample the share from rows the read never returned.
Three gates, each mutation-tested (defect reintroduced, named test RED, reverted,
green):
TestEveryReadOfTheSourceIsBoundedInBytes the package's own AST — a
function that reads the source without the bound fails, the same shape as the
admission gate beside it, because it is the same failure: a new op skipping a
property nobody checks. Carries the same anti-vacuity floor.
TestAnUnrepresentableSubjectIsExcludedAndCounted end to end: the bound binds,
the representable rows all survive, the count is on the version AND the
lineage, and it is falsifiable.
TestTheByteBoundIsDerivedFromTheValueBound the arithmetic, so no byte
figure can be asserted independently again.
The fake store EVALUATES the bound rather than ignoring it — a fixture that
ignored it would let every test above pass with the predicate deleted. It also now
binds `?` positionally across the WHOLE statement, as a driver does, which is what
the census's conditional aggregates require.
TestTheTenantLeadsEveryPredicate was reading args[0] and calling it the tenant.
That was a positional coincidence, true only while no statement carried
placeholders before its WHERE; it now reads the argument bound to the leading
`org = ?` itself, which is the property it always meant.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
(cherry picked from commit b851446a2a)
CLOUD_ENABLE / --enable named the subsystems to mount. The binary already
knows: manifest.Apps is the host's set, and a plugin IS its app (Listen sets
cfg.Enable from what plugin/<app>/main.go was built as — that stays; it is
the binary stating itself, not a deployment restating it).
A second source of truth can only add disagreement, and it did. Both devnet
outages on 2026-08-02 were this list: one named "plans", an app the manifest
does not have, and one omitted "kms", the credential broker every other child
pulls its data-plane key from — so every child failed closed at its first
store open. Production has never set it.
Removing the input removes the failure class and five branches with it: the
broker precondition, the unknown-name guard, and three `on != nil` selections
that only existed to police a list nobody should have been writing. Empty
already meant all, which is what production runs.
Values move in the same change — devnet and testnet drop the list, so no
deployment is left naming a variable the binary no longer reads. Docs and
cloud-probe.sh follow; the probe had been STRIPPING the variable, so it
already agreed.
TestTheAppSetIsNotNamedTwice replaces TestAnAllowlistWithoutTheBrokerIsRefused
— that test policed the list, and the guard is now that no source states the
set again.
Also fixes a pre-existing red on main, unrelated to this: apps/catalog's
cloud.Request call site was never added to allowedRequestUses, so the escape
hatch ratchet failed on clean origin/main. It is a legitimate use (the
published corpus reads as PublicOrg for everyone, so the tenant is re-pointed
while the caller's authority travels whole) and is now recorded with that
reason.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An hourly goroutine in authors deposited credit with nobody in the loop, and a
GET on three surfaces accrued-and-paid on read. Both are gone, along with the
capability that made them one line each.
- apps/authors/scheduler.go, sweepAndPayout, autoPayoutAuthor — the unattended
hourly accrue+pay loop, default ON, no env var, no route, no human.
- the lazy sweep on GET /v1/authors, /v1/affiliates, /v1/affiliates/me and
/me/earnings. Reads read; the admin POST sweep still accrues.
- payout settlement in both programs. A payout RECORDS what is owed, for every
method including credits; a human settles it. Accrual — the product — stays.
- treasury.Reserve/Credit returned backed=true when unmounted, and `mounted` is
a package global, so in one-binary-per-app it was ALWAYS nil in callers: every
"reserve-backed" payout was an unbacked mint that logged itself as reserved.
With settlement gone it has no callers, so it is deleted rather than fixed.
- POST /v1/admin/credits — a second admin mint with no cap and no positivity
check, whose audit did not fail closed. core.ApplyGrant is the one door: it
caps, rejects non-positive amounts, checks the org, and refuses without a
durable audit store. The relay and its wire client are deleted.
- payout.Client.Deposit, the ONE money-in primitive all three programs shared,
and the deposit method on each program's seam. The seams now carry a single
read, matching referrals: reviving a mint has to start by re-declaring the
capability, in front of a test that says no.
- the published POST /finance/starter op, advertising a grant deleted in
41b23f12.
openapi.yaml, plugin/admin/openapi.json and openapi/floor.json drop the deleted
route in this commit, so the reduction is reviewed next to its reason.
Tests assert the guarantee rather than the old behaviour: a GET grants nothing
and the ledger receives zero deposits, proven at the wire against a commerce stub
that fails on any write.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/ws/query_progress is in openapi.yaml and in plugin/o11y/openapi.json, so it is a
method in every generated SDK — and no manifest row named it, so the composed
binary routed it nowhere. Its own prose says the address "was unreachable from the
composed binary until the route table named it, because the old wildcard covered
only the o11y prefix"; the wildcard went and the row never grew the prefix back.
TestEveryServedPathReachesTheAppThatServesIt has been RED on main for this one
path. It reads 1673-of-1684 now against 1672-of-1684 before, and the recorded
ledger is unchanged — nothing else changed hands.
Mutation-tested: prefix removed, the named test reports UNREACHABLE and fails;
restored, green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both avatar routes answered 404 in production. The routes were registered and
the handlers were fine; the HOST routes by manifest prefix, and account's row
did not name /v1/avatar — so `ai`, which owns the /v1 remainder, won it.
manifest's own router test said so in as many words on the tree that shipped:
UNREACHABLE: account /v1/avatar -> ai
UNREACHABLE: account /v1/avatar/{org}/{user}/{digest} -> ai
I did not run it. Targeted package tests passed and I shipped on those, which
is how a route can be complete, tested, deployed and unreachable at once.
With the prefix named, both paths mount to account (zip mounting … addr=account)
and the ledger drops from three unreachable paths to one — o11y
/ws/query_progress, which is not this change's and stays as it was.
iamurl.go opens by saying the IAM-address policy "existed as three inlined
copies ... so there is exactly one now". There were four, and they had grown
three different fallbacks:
iamurl.go IAM_URL, else the public issuer (the policy)
auth_apikey.go IAM_URL, else IAM_INTERNAL_URL, else NOTHING
apps/account/iam.go IAM_URL, else a hardcoded cluster address
apps/platform re-read IAM_URL to answer a different question
A copy of a policy does not disagree until one is edited. These had already
diverged: auth_apikey read a fifth env name no cloud deployment sets
(IAM_INTERNAL_URL is on admin-guard and chat only), so a single-process deploy
that knew only its issuer resolved "" and API-key auth stayed silently
unconfigured; apps/account reached a cluster address that a non-cluster deploy
does not have.
iamurl.go now answers both questions the estate actually asks, over ONE env
read: IAMBase() for the address, IAMExternal() for whether a separate IAM is
named. They are separate because conflating them IS the bug — a deployment
with only a public issuer resolves a real address while naming no external
IAM, and a caller inferring one from the other reaches for a store that is
not there. TestIAMAddressHasOneReader walks the tree and fails on a second
reader, so this cannot drift back.
Also: pickCompletionsClient preferred the PUBLIC gateway over the in-cluster
peer. `ai` is a plugin of this same binary running as its own process, so that
ordering sent the pod out through Cloudflare and back, minting an OAuth token
to authenticate to its own deployment, to reach code one socket away. The peer
is now first. Ordered, not gated: nothing sets CLOUD_AI_ZAP_ADDR today, so
this is inert until an address is named and the gateway keeps answering.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
main was pinned to hanzoai/ai v1.832.18, which is RETRACTED. That version seals
the secret MASK as a provider key: the admin API returns "***" for a stored
secret, so saving a provider form without touching the key field seals the literal
"***" into KMS under the provider's own name — and because ai resolves KMS-first,
the store then answers "***" for every read, outranking the env var that was
serving the real key. The provider stops authenticating while its row still looks
correct. v1.832.19 carries the guard.
.19 also brings: secrets resolved from the EMBEDDED in-process KMS (this binary's
own apps/kms) instead of over HTTP to the standalone deployment, which had never
worked — 404 on the path it used, 401 on the correct one; /v1/provider-flags
renamed to /v1/models/providers and derived from the served catalog, so the
provider set and the model list cannot disagree; and a model family is now
controlled from admin.hanzo.ai rather than only from deployment env.
apps/referrals did not compile on main: payout.Deposit gained a required `ref`
(commerce guards on it so a retried payout credits AT MOST ONCE) and this caller
was not updated with it. A referral pays TWO wallets, so the referral id alone
would name both and commerce would dedupe the second against the first — the
referee's bonus would silently never land. bonusRef(id, side) makes each credit
its own event while staying stable across retries.
The published spec is regenerated: /v1/provider-flags is gone from openapi.yaml
and plugin/ai/openapi.json (the .18 pin never regenerated it, so the drift gate
was red), and the retired provider-flags product is dropped from the floor.
Verified on Linux (macOS has no tmpfs for the pure-Go SQLCipher codec, so the
store-backed suites cannot run there): apps/referrals, apps/ai, apps/kms and
openapi all green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The plane fix shipped and /v1/catalog answered 500 "index: no org on the call"
— the public browse reaching the index as nobody.
cloud.For() states a caller, but zip's forwardIdentity prefers an INBOUND
request over a stated one ("an inbound request always wins over what it said"),
and a typed handler's ctx carries the in-flight request. So For() was silently
overridden and the call went out as whoever asked. For a signed-out visitor
that is nobody, and the index refuses a call with no org — correctly.
cloud.As() is the form for this: it re-points the tenant on a context with no
request behind it, which is the one place zip reads what we stated. The caller's
authority still travels whole; only the tenant moves — which is the point, since
the published corpus is read as PublicOrg by everyone, signed in or not.
Caught by asking production rather than by trusting the deploy: the previous
commit was verified live and answered 500, not rows.
payout.Deposit sent no idempotency key at all, so every retry of an author
royalty or affiliate commission credited the wallet again. Commerce now
requires the key (it is the reference to the money event), and both callers
already hold the right value: the payout row's own id.
- Deposit takes ref and sends it as X-Idempotency-Key, via a post helper
kept beside do because only a WRITE carries a reference — a read has no
event to name.
- ErrNoRef refuses an unnamed deposit HERE rather than at commerce, so the
failure names the caller's missing value instead of arriving as a status,
and a new payout path cannot quietly ship without one.
- authors and affiliates pass "payout:"+payoutID through their commerce
seams.
Tests: an unnamed payout never reaches commerce (the fake server records
that it was not called), and the ref travels as the key commerce guards on.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/referrals ran a "lazy qualify sweep" before listing, and that sweep
reached commerce.Deposit. Loading your own referrals page minted platform
credit — $10 to you, $5 to your referee — on a read. The middleware in front of
the prefix gated writes by asking `method != POST`, so it waved the GET through;
the two safeties behind it did not hold either. treasury.Reserve returns
backed=true when treasury is unmounted, so the "backed by the reserve fund"
claim was a passthrough in any deploy without it, and the at-most-once latch
bounds the mint per referral, not in total.
The precedent is set twice on main: 41b23f12 deleted the $5 starter grant and
45b3b5cf deleted finance.Deposit along with its imports. Same here. The deposit
path is DELETED, not disabled — a flag-disabled money mint is one flag from an
enabled one.
What goes: the two bonus constants, the ledger currency + grant:referral tag,
grant(), the treasury reservation and its import, LatchCredit, SetTxns, the
grant/txn/credited_at columns, the `credited` status, and every cents field on
the wire. A field that can only ever report zero is a lie about what the surface
does, so creditsEarnedCents and the two bonus amounts go rather than freeze at 0.
The commerce seam keeps ONE method, spendCents — it is a question, not an
instruction — and TestCommerceSeamIsReadOnly fails if it grows a write.
What stays, because it is the actual product: who referred whom, the stable
code, the share link, and qualification. Qualification is a WRITE, so it now
happens only on POST /v1/admin/referrals/sweep. A GET reports; it does not
transition.
The gate is fixed at the defect class, not the instance: it waves through GET
and HEAD by name and requires an org for every other verb, including ones this
package does not serve. "Not POST" meaning "harmless" is the reasoning that let
a read reach a deposit.
What a qualified referral is WORTH is not this package's question. That is an
affiliate payable in hanzoai/commerce, settled by wire or to a connected wallet
— never minted as platform credit.
Tests: a GET in the exact state that used to pay leaves the row byte-identical
and touches the money plane zero times; the real payout client against a stub
commerce that fails on any write proves zero deposits at the wire.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The reservation landed with a 32768 constant standing in for "the most a
completion can be". That number is wrong per model by construction: it caps a
1M-context model at whatever was typed, and it is one release out of date the
moment a model ships. This estate has already paid for that shape twice —
ai/model's name-matching table gave deepseek-v4-pro 16384 and 402'd every long
prompt, and glm-5.2 dead-ended /compact on a stale 16K fallback. Both were
fixed by moving the number into models.yaml. This does the same for billing.
- completionCeiling(model) resolves through SetCompletionCeiling, installed
in apps/ai from ModelConfig.MaxOutput (ai v1.832.18), falling back to the
model's context window — still a true bound, since prompt + completion can
never exceed it. The constant survives only as a FLOOR for a model the
catalog does not declare, and is documented as never a per-model answer.
- the seam exists because hanzoai/ai/controllers imports hanzoai/cloud, so
the catalog is a CYCLE from cloud's root, not merely weight. apps/ai links
both, which is where every other cross-module hook is installed.
- atMost no longer writes the ceiling onto the request. What we reserve is a
billing fact; req.MaxTokens is the CALLER's, and forwarding a limit they
never asked for silently truncates their answer. The reservation is sound
regardless — a model cannot exceed its own max output — so the bound holds
whether or not we restate it on the wire. Only a caller-set MaxTokens now
reaches the provider.
TestCeilingComesFromTheModelNotAConstant pins all four: a 1M model reserves
>=1M, an undeclared model takes the floor, a caller's MaxTokens wins, and
atMost never mutates it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzo.app's Community page rendered "ERROR: CATALOG: 503" under an otherwise
fully-drawn page. api.hanzo.ai/v1/catalog answered
{"status":503,"error":"catalog: index not mounted"} on EVERY request, and
nothing was down.
browse() guarded on index.Ready(), and Ready() reports whether the index is
mounted IN THIS BINARY — its own doc says so. That was true and harmless while
cloud was one fused process. It stopped being harmless when the fleet became
one process per app: `index` and `catalog` are two manifest rows, index.Mount
is only ever called by plugin/index, so inside the catalog process that global
is nil and always will be. An in-process dependency survived a process split,
and the only symptom was a status code.
The index is now ASKED, on the internal plane, exactly as visor asks tasks for
its activities and as commerce serves its ledger. Not opened: the index store is
one encrypted SQLite with a single writer (MaxOpenConns(1), keyed through cek),
so a second process opening the same file to read it is the collision, not the
cure.
Both legs are kept and neither is dead: a fused binary that mounted both apps
still reads in-process, because a wire hop to something in the same address
space is pointless; the deployed fleet takes the plane. The write side does not
move — Reconcile stays in the process that owns the file.
apps/search carries the same dependency and is mounted by no plugin at all, so
it is unreachable rather than broken. Left alone here; naming it so the next
reader does not mistake this fix for having covered it.
Moving imageType into internal/magic took the only use of "bytes" in files.go
with it and left the import behind — which does not compile, and it is what
turned the release train red twice (the image job exited 1 at ~6m18s and no
cloud image published, while v1.801.399 had built fine from the commit before).
It was invisible locally for a reason worth writing down: `go build ./apps/team`
on macOS fails first on hanzoai/base's `cgo && !sqlite_math_functions` guard, so
the package was never type-checked and my error hid behind someone else's. The
build CI and the Dockerfile use is CGO_ENABLED=0 — under that, the package
compiles and the mistake is immediate.
Verified the way CI does: CGO_ENABLED=0 go build ./... and go vet ./... both
clean. The apps/team test failures that remain on this machine are all one
environmental cause (75 of 75: "cek: no RAM-backed scratch", macOS has no
/dev/shm) and fail at store setup before any assertion.
The drift gate regenerates every app's spec subset FROM SOURCE and fails on any
diff, so adding /v1/avatar without re-running it turned CI red — the release
train's image job exited 1 after 6m29s and no cloud image was published.
Regenerated, not hand-edited: zipdoc lifts the doc comments into zipdoc_gen.go,
`describe` projects account's own subset, and the weave proves the fleet spec
equals the sum of the subsets. floor.json moves by exactly what was added —
+2 paths, +2 operations, one new `avatar` product with 2.
zip dials every plugin with zaphttp's DEFAULT transport, whose readTimeout is
30 seconds, and nothing overrode it. That deadline is armed ONCE, just before
the response head is read, and for a STREAMED response it is never re-armed:
the head arrives, SetBodyStream returns, and the whole body must then land
within what is left of the original 30 seconds.
So it is not an idle timeout. It is a hard cap on the TOTAL DURATION of a
response — applied to the plane whose responses are model completions, because
`ai` owns the /v1 remainder (/v1/chat/completions and the rest of the
OpenAI-compatible surface) and runs as a plugin behind this wire.
Every completion longer than 30 seconds was cut:
- streamed: the body stops mid-token at ~30.1s with NO finish_reason, which
every layer downstream reads as a complete answer. Measured against the
live gateway, a long-page request to claude-opus-4.8 returned 7056 bytes,
no </html>, total 30.1s — while the identical request straight to the
upstream streamed 389s and finished properly at 15661 bytes. Reproduced
in-cluster too, so it was never the edge.
- unstreamed: 502 `zaphttp: read response: read unix …: i/o timeout`.
This is why hanzo.app's builder could produce a small landing page but not a
real app: an app is just a longer generation, and the truncation was silent. It
also made a slow-to-first-token model look broken rather than slow — enso
spends ~19s thinking, so most of its budget was gone before it emitted a token.
The ZAP scheme is re-registered before any plugin mounts (the transport is
resolved at dial time from a process-global registry, so a later registration
would leave already-dialed plugins on the default). BOTH halves are supplied: a
Dial-only Transport would silently remove the host's ability to LISTEN on its
own default scheme.
15 minutes, not zero: zero is no deadline at all and a wedged plugin would hold
the connection forever. The honest shape is an idle timeout — time BETWEEN
frames — which this transport does not offer, so it stays a total-duration cap
set past anything a real completion reaches.
There was no way to set one. IAM carries an `avatar` on every user row and the
console renders it, but the only writers were federation (a GitHub avatar_url,
an OIDC `picture` claim) and SCIM — so a user who signed up with a password had
a monogram and no way to replace it, and the console's Profile card answered the
attempt with "Edit in IAM", which links to an IAM that cannot do it either.
Production agreed: /v1/avatar was a 404 while /v1/keys was a 403.
POST /v1/avatar stores the image and records its URL on the caller's IAM row, so
every surface that already reads `avatar` picks it up with no further call.
GET /v1/avatar/:org/:user/:digest serves it.
Storage is deps.VFS — the existing S3 seam (SeaweedFS via clients/s3vfs), which
was chosen for exactly this: "an adapter+crypto is needless complexity for small
avatars". No new store.
Three properties make it safe to serve an uploaded file back from an API origin:
- THE FORMAT IS DECIDED BY THE BYTES. png/jpeg/gif/webp by magic number;
anything else is 415 on the way in and 404 on the way out. A filename and a
part Content-Type are the client's to choose, so neither may decide what
this origin later serves — an SVG is a program, not a picture.
- THE ADDRESS IS THE CONTENT. The key ends in the sha256 of the bytes, so a
new photo is a new URL rather than a stale cache of the old face, and the
read caches for a year. A replaced photo is deliberately NOT deleted: the
old URL is already inside issued tokens and rendered pages, and an object
store costs bytes where a broken face costs a person their profile.
- THE READ TAKES NO CREDENTIALS, AND MUST NOT. Its whole job is to be an <img>
from console.hanzo.ai, a different origin that sends no cookies and cannot
set a header. So the 64 hex of sha256 IS the capability — producible only by
someone who already has the image. The org and user segments are REFUSED
unless they are plain identifiers rather than sanitized: apps/team's seg()
folds "a/b" and "a_b" onto one key, and a fold in a tenancy key is two
identities sharing an address.
internal/magic is the one magic-byte allow-list, now shared with apps/team's
files plane instead of copied. (The three `seg` functions are NOT duplicates —
same name, three different concepts — so they stay where they are.)
The oversize test mounts the app at production's 16 MiB edge body limit. Left at
zip's 4 MiB default the framework refuses the request first and the handler's
own 413 is unreachable and untested — the shape of the bug where studio's 4K
sources could not enqueue.
POST /v1/marketing/promos/:code/redeem was a self-service money mint. The only
gate was tenant(ctx) -- ANY validated principal. `plan` and `seats` came off the
REQUEST BODY unvalidated and were multiplied into a finance.Deposit, so
{"plan":"team","seats":10} deposited 10 x $179.10 = $1,791.00 of real spendable
credit into the caller's own org. Nothing ever collected the charge the discount
was supposedly against; the charge was computed, returned to the caller, and
discarded. `instrument` was the anti-farming key, but instrumentUsed("")
returned false, so OMITTING the field skipped the guard entirely. With the
1,000-org cap, open signup and a personal org per account, that is ~$1.79M of
self-serve credit. The seed shipped active=1 in v1.801.398, which is live.
THE CAMPAIGN WAS NEVER AUTHORIZED. It became live because a schema migration
INSERTed it on every boot -- a business decision arriving as a side effect of a
code change. The seed is deleted, and because deleting an INSERT does nothing
for a database that already ran it, migratePromos now DELETEs the row on every
boot. Redemption history is deliberately KEPT: it is the evidence of what
happened while the campaign was live, and destroying it would destroy the audit
trail exactly when it matters.
CREDIT INTO AN ORG IS AN ADMIN DECISION -- deliberate, through the admin
surface, against an auditable ledger. So the deposit is not fixed, it is GONE,
along with the finance/money/types imports that made it reachable: reviving a
mint here would have to start by reviving an import. This follows 41b23f12,
which deleted the automatic $5 starter grant rather than switching it off, for
the same reason -- a money-mint left disabled is one flag away from enabled.
The subsystem now ships OFF (campaignsLive=false, read from no env var, no
platform switch, no column; TestCampaignsShipOff asserts the shipped value).
The guards are hardened anyway, so a REVIVED campaign cannot resurrect the hole:
- Plan is DERIVED from the org's live ACTIVE/TRIALING paid subscription via
cloud.PlanChecker -- the same seam SpendGate resolves -- and RedeemInput no
longer HAS plan/seats fields. A field that does not exist cannot be trusted
by the next reader. No qualifying subscription means no redemption.
- FAIL CLOSED on an unreadable plan authority. SpendGate deliberately fails
OPEN on this same read, because refusing on an outage 402s every paying
customer at once. Here the asymmetry inverts: an outage must not be able to
manufacture a claim that money is later granted against.
- instrumentUsed("") now returns TRUE. An absent instrument is not evidence of
a fresh card, it is the absence of evidence, and the partial unique index
(WHERE instrument <> '') means the database will not catch it either.
- maxClaimCents bounds every recorded claim, checked under the same lock as
every other guard, and REFUSES rather than clamps -- a silent clamp would
record a wrong figure and hide the bug that produced it.
- Seats is the single-seat floor, never a caller-supplied multiplier.
Redemption.CreditCents/CreditEntryID become DiscountCents: the row records a
claim, not a balance, and a field named for credit that credits nothing is how
the next bug gets written.
Tests drive the shipped routes through the real router, because the hole was a
handler that trusted its input -- a store-level test would have proved the store
fine and missed it. The hardening tests force a campaign live (revivedPromoRoutes)
to answer the question that matters if the decision is ever reversed. Proven: the
exploit body cannot move the recorded figure; a live ledger receives ZERO
deposits; a redemption while closed is refused; nothing is seeded; migrate purges
an already-seeded row while preserving its redemptions.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A prepaid gate reads a SETTLED balance and a completion's cost is not known
until it finishes. Two ways an org spent money it did not have sat between
those facts:
- the gate priced only EstTokens(prompt), so the completion was never
covered. A 1c org asking for a million-token completion was allowed and
settled at $2.00, ending at -$1.99.
- N calls in flight each read the balance before any of them debited, so
each was authorized for the whole of it. Twenty simultaneous callers
against 5c spent 20x the balance.
meteredAI now commits a call's worst case before weighing it and releases
that commitment when the debit reaches the ledger:
- types.ChatRequest gains MaxTokens, and atMost() resolves the ceiling ONTO
the request so the transport forwards the very number the gate priced.
Reserving a ceiling nobody enforces leaves the completion just as unfunded,
only less visibly, so clients/aihttp sends it on both the buffered and the
streamed path.
- commitments tracks per-org committed-but-unsettled cents. commit() returns
the RUNNING TOTAL, which is what the balance must cover, so a second
concurrent caller must clear the first one's commitment. Nothing else has
to know reservations exist.
- the release runs inside the recording goroutine (meterUsage's new posted
hook, threaded through the peer path too). Releasing when the call returns
would let the next gate read a balance that still contains money already
being spent — the very window this closes. It runs on every exit: a hold
that leaks is a paying customer locked out of their own balance.
Holds stay per-pod. apps/finance owns the settled truth and says so
("transient holds are the caller's in-pod concern, never persisted here"),
so this never becomes a second ledger.
MeterUsage keeps its signature — one debit verb, twelve callers untouched.
Tests drive the real meteredAI against a wallet whose balance MOVES; the
existing fixtures answer every gate from a fixed body, which is why neither
gap showed up before. A barrier holds the concurrent callers at the balance
read so the TOCTOU is deterministic rather than a race won by luck.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The starter grant minted $5 into a wallet from middleware, on first credential
contact, with no human in the loop. Credit into an org is an ADMIN decision --
made deliberately, through the admin surface, against an auditable ledger --
so an automatic path that creates money is not a feature to fix but a mechanism
to remove.
DELETED RATHER THAN SWITCHED OFF. A disabled money-mint is one flag away from
an enabled one, and the flag is the kind of thing a later reader flips to
"unblock" something. There is no starter code left to re-enable: the middleware,
its mount in serve.go, the cross-process plane op (finance_starter / StarterIn /
Granted) that let a non-ledger binary ask for it, and their tests are gone.
Note this also removes the shared-signup-org exclusion that lived in the gate.
It was sound anti-abuse for a grant that no longer exists, and keeping half a
mechanism to guard the other half is how dead code survives.
The paywall consequence is deliberate and is NOT taken here: SpendGate stays
behind its kill switch. With no automatic funding, enforcing it 402s every new
account from its first request -- an honest paywall, and a product decision that
deserves its own change rather than arriving as a side effect of this one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
sha-846069c predates the fix. The console SPA is baked into this binary by
//go:embed, so nothing about a console release reaches production until this
line moves — which is exactly what the pin is for, and why it is a sha and not
`:latest`.
sha-9da3984 carries three commits: the token store moved off sessionStorage
(a second tab started signed OUT while the first was still signed in), the
landing CTA starts the sign-in instead of routing to a page that asks again,
and the @hanzo/iam bump to 0.21.6 without which the static export dies on
`ReferenceError: sessionStorage is not defined` while prerendering
/auth/callback.
Verified before pinning: the console CI run for 9da3984 is green and pushed
console-embed:sha-9da3984-amd64. The previous run, on b9d31aa, was RED — so the
image for the storage fix alone never existed, and pinning to it would have
failed the CONSOLE-GATE here rather than shipping anything.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two corrections to what shipped in v1.801.397.
TIER. It was registered on the bare public chain plans uses, but GetTier opens
with middleware.GetOrganization, so it panicked on a nil interface conversion and
answered 500 — the route went from unreachable to reachable-and-broken. A tier is
org state; the org has to be resolved first. Moved onto the billingRead loop,
which supplies the IAM leg its six siblings already rely on.
ROUNDING. The balance now reads (502 -> 200, $149,913.08), but it rounds UP, not
down as the comment claimed. hanzoai/decimal's Rescale rounds half-away-from-zero
(decimal.go:145) and Minor() is a Rescale — measured live, …078983985999994361
served 14991308 cents, a tenth of a cent above the true balance. The comment is
corrected rather than the behaviour: this number is a display, nothing is billed
from it, and the spend gate reads the exact decimal itself. A debit that must not
overstate has to round down deliberately instead of reusing this.
apps/ai carries the same "truncated toward zero" claim on the same call and is
wrong the same way. Noted in place; its gate compares > 0, so a sub-cent rounding
cannot change its verdict.
GET /v1/billing/balance answered 502 "billing upstream unreachable" on every
call and the console rendered "Unavailable" while the ledger held the money.
Nothing upstream was involved — that label is wrong and it is why this looked
like a connectivity problem for days. plane.Money.Minor() REFUSES a value finer
than a cent rather than round behind the caller, and the ledger keeps eighteen
decimals because per-token charges are routinely finer than a cent. So a REAL
balance broke the read: live the org held $149,913.078983985999994361 and the
error was "is finer than its minor unit; round explicitly". It got worse as
usage accumulated, since a longer history makes a sub-cent tail likelier.
Minor()'s own doc says a caller that wants a rounded figure — "a display, a
summary" — should round explicitly, where the choice is visible. This view is
exactly that and never did. It now rounds DOWN, the same choice apps/ai
documents for the same value: a displayed balance must never exceed what the
account can actually spend, and truncation understates by under a cent. Nothing
is billed from this number.
Also reverts a wrong fix from earlier in this session: registering balance
co-resident in apps/commerce. The manifest gives /v1/billing/balance to the
`billing` app, not commerce, so that registration was in a subtree it does not
own and could never have run.
Two things this exposed, both left alone deliberately:
- The 502's message names an upstream that is not in the path. Renaming it is
a behaviour change to an error contract callers may match on.
- apps/billing's test binary needs libsqlcipher and cannot compile on a
laptop, so `go build` passing there proves nothing about the test file. That
masked a first version of this patch which called a Minor() that does not
exist on the local money type; caught by type-checking with go vet instead.
GET /v1/billing/balance answered 502 "billing upstream unreachable" on every
call, and the console renders that as "Unavailable" while the ledger holds real
money.
The cause is the one apps/account/billing_coresident.go already documents in its
header: co-resident there is NO standalone commerce — the in-cluster service
selects the cloud pods themselves — so the /v1/billing/* bridge forwards to a
default base that is the public edge, and the read re-enters the same bridge in
an unbounded self-dispatch loop. Six sibling reads (invoices, subscriptions,
alerts, payouts, settings, credits) were already registered co-resident to
shadow that wildcard. Balance was not one of them.
It now registers through the identical chain — RequestContext, IAMTokenRequired,
PinBillingSubject — so the subject pin is byte-for-byte what the bridge applied
and a read scopes to exactly the account the spend gate debits, never wider.
balance/all rides the same prefix, since a prefix owns its whole subtree.
The apps/commerce test binary does not compile on a laptop (hanzoai/base needs
libsqlcipher, which the build image links and macOS does not); verified by
stashing that the identical failure predates this change.
Every AI request logged
tier_cache: Commerce lookup failed for key=...: commerce returned 404
(defaulting to zen-free)
and that default is 60 rpm against 500 for pro, 2000 for team, 50000 for
enterprise. So the failure did not give anything away — it silently served every
PAYING customer the most restrictive tier in the table.
Cause is the exclusive-subtree rule again: account-bridge owns /v1/billing, and
commerce's row named seventeen sibling paths but not tier, so
GET /v1/billing/tier never reached the handler that answers it
(commerce api/billing/handlers.go:43 registers it; live it 404s).
The UNREACHABLE ledger in manifest/router_test.go did not catch this and is not
wrong to have missed it: it fires on paths the fleet PUBLISHES, and cloud does
not publish tier — commerce serves it and only the ai router calls it. A
cross-binary caller is outside what that gate can see from here.
The pinned console-embed was sha-147ecd3, built 2026-07-27. console main is 63
commits ahead of it, so a week of console work — including the onboarding fix
below — has never reached production: the pin does not track main BY DESIGN (a
floating tag made a release silently bake the PREVIOUS console), so it only moves
when someone moves it, and nobody had.
sha-846069c is console main HEAD. Verified published before pinning:
ghcr.io/hanzoai/console-embed:sha-846069c-amd64 resolves to
sha256:f2849a31b082da0d690bc5403a41ad995624fdfd6595089008821c07718a9255, and
console-embed:latest now points at the same digest. Probed with all four Accept
types against a bogus-tag control — a two-header probe returns a FALSE 404 on an
image that is present, which is how a healthy registry can be misread as an
outage.
What this carries to production, beyond 62 other commits: a refused org create no
longer reads as a complaint about the NAME. /v1/iam/onboard answers 409 for two
opposite reasons — the first-run gate (this account already admins an org;
founding a second would orphan it) and a name genuinely held by another tenant —
and the console showed the server's organization-level message immediately after
the customer typed a name. It now reads the account to tell the two apart, and
offers the way into the org the identity is actually in instead of dead-ending on
a form that can never submit.
cloud pinned ai v1.832.16 while five commits sat on ai's main untagged, so a fix
merged hours ago was in no release and no binary. That is the third instance
today of merged-and-unshipped, each with a different cause: an image published
before its own fix landed, a build job silently skipped by a stale generated-doc
gate, and now a module change in no tag at all. The symptom is identical from
outside — the code is right and production is wrong — which is what makes it
expensive to notice.
Carries: the ok-with-no-user key refusal (IAM answering status=ok with a null
user used to fall through to a bare "invalid API key" that named no cause), the
openrouter seed URL carrying a /v1 the family appends itself, a family's kms://
key resolved before it becomes an Authorization header, family control from
admin.hanzo.ai rather than env alone, and /v1/provider-flags becoming
/v1/models/providers derived from the served catalog.
No release cut here. The next cloud build carries it: cloud is one replica with
strategy Recreate, so every release is a total outage of inference, billing and
auth, and a dependency bump does not justify one on its own.
The site edge has never run in production. It was mounted in serve.go — the
FUSED composition root — but the image runs cmd/cloud (the light router) on the
public port with one process per app beside it, and none of those is serve.go.
So every <slug>.hanzo.app fell through to the console SPA, and the whole /v1
surface answered on the customer's own hostname.
Proven at the pod rather than inferred: the process holding :8000 is /cloud (a
separate 23MB binary from /plugins), and `grep -c sites_resolve /cloud` is 0 —
the code was not in the running binary at all, even at the tag that contains it.
The two earlier fixes this session were both real and both invisible for this
reason: the host resolution fix (8b729f8ef) and the plane resolver (5f0b74fe9)
were compiled into a composition root nothing runs.
The router "deliberately links none of" the fleet's package graph, and that
holds: `go list -deps ./cmd/cloud` still contains ZERO of the root package.
apps/sites is a leaf, and the cross-app call is made here with zip.DialApp —
the same door wake.go already uses to publish one op without cloud.Plane().
apps/sites exports the wire types so the caller restates no mapping.
Mounted BEFORE webui, which owns "/" for every unclaimed path and would
otherwise answer first for every site host.
The test reads run()'s own source for the call site, because the defect was
never a logic error — the package was always correct — it was a middleware that
ran nowhere. My first version called mountSites directly and PASSED with the
call site deleted, which is exactly the mistake this file is about. Both failure
modes are now negative-controlled: no call at all, and mounted after the console.
Every published site served the console SPA. The host fix in 8b729f8ef was
necessary and not sufficient: with the host resolving correctly the edge still
found no site, because sites.SetResolver writes a PACKAGE-LEVEL registry inside
`projects` and the edge middleware reads it inside whichever process fronts
:8000. The pod boots ~25 single-app processes ("enabled":["<one>"], 25 distinct,
zero multi-app — measured on the live pod), so those are never the same process
and the registry is always nil where it is consulted.
A nil registry is a clean MISS, not a fault. So every lookup failed silently,
every request fell through to the API pipeline, and no error was logged anywhere
because nothing had failed. Proven at the pod with the ingress bypassed:
wget --header="Host: app.maxpower.hanzo.app" http://127.0.0.1:8000/
-> <title>Hanzo Cloud Console
for a site that is genuinely published and has its own Ingress.
The fix is the seam this repo already uses for exactly this shape:
FinanceScopeRules is on the plane, in its own words, because "the READER is a
cloud EDGE middleware" and the fact belongs to another app. projects now
publishes sites_resolve / sites_resolve_org from the one process that owns the
store, and the edge falls back to them. Co-residence still wins with no hop —
currentResolver prefers the in-process registry and only then asks.
Not-found stays a clean 404; a failure to ASK stays an error, so the edge can
render 503. Collapsing those would serve 404s for live customer sites during any
transient failure of the owning app, which is indistinguishable from deletion.
The comment on SetResolver said "until it is set, every site request is an honest
404 (the projects subsystem is not mounted)". That premise was the bug: projects
IS mounted, just elsewhere, and 404 is not honest when the site exists.
Negative-controlled: removing the fallback fails the new test with the
fall-through this commit is named for. The second test pins that a co-resident
store is still used without the hop.
The relational engine wrapped an identifier in the dialect delimiter but wrote
the body verbatim, so a delimiter inside the identifier closed the quote the
writer opened and the rest executed — Desc("name`,(subquery)--") emitted two
quoted identifiers plus bare SQL rather than one identifier. xorm v1.4.5 doubles
the delimiter, the standard SQL identifier escape, so a name that smuggled one
becomes a single identifier that does not exist and fails closed.
orm is the only place that version is chosen — consumers name hanzoai/orm and
the engine arrives underneath — so this is the hop that makes it live here.
go list -m confirms xorm resolves to v1.4.5 in this build.
Bump only. The four failing packages are the standing pre-existing set
(apps/iam, apps/plan, apps/pricing, manifest); the bump adds none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
plugin/dataset/main.go called cloud.Serve, which does not exist. Every other
plugin calls cloud.Listen. The image build enumerates the manifest and compiles
each plugin in turn, so this failed the whole build at the dataset step:
plugin/dataset/main.go:29:18: undefined: cloud.Serve
That means no cloud image has been buildable since ae3a30994 landed. `go build
./...` at the repo root does not catch it — the plugin mains are only reached by
the Dockerfile's per-plugin loop, so the gap between "compiles locally" and
"produces an image" is exactly one word wide.
Every published site served the CONSOLE, and mounted the whole cloud API under
the customer's own hostname. Measured live 2026-08-03:
quest.hanzo.app/ -> <title>Hanzo Cloud Console
quest.hanzo.app/v1/billing/plans -> 200
fiber parses the request URI once, and behind the ingress the parsed host is
empty — so siteSlug("") failed, customCandidate("") failed, and every request
fell through to c.Continue() into the API pipeline. The site edge was mounted and
configured correctly the whole time; it just never learned which host was asked
for. Same accessor and same failure as commerce's tenant resolver, in a second
codebase, the same night.
The parsed host still WINS whenever it names something this server serves — that
ordering is the security property, not a detail, because the host picks the ORG
here and a client able to override a real host could serve itself another
tenant's site. X-Forwarded-Host is consulted only when the parsed host is not a
host we can serve, which is the ingress case and never a direct request.
TestMiddlewareTenantKeyedByHostNotPath (pre-existing, unchanged) still passes.
Negative-controlled: reverting to Hostname() alone fails the new test with the
fall-through this commit is named for. One correction on the way: the first
version of the new test used req.Host="" to express "no parsed host" — httptest
synthesizes "localhost" for that, so it proved nothing until measured.
studio could not enqueue a single render. `POST /v1/tasks/.../activities` answered
403 "identity required", so thirteen jobs sat `queued` in its worklog for up to
nineteen hours while both GPUs polled an empty namespace every two seconds and
reported themselves healthy. Nothing in the queue, the worker logs, or the studio
UI said why.
The cause is one branch. homeOrg already knows that a machine cannot mis-attribute
its org the way a human choosing an app can — that is why the KMS sync identity
reads `owner`. But KMS is recognised by audience, which works only because its
client id is DERIVED from its org ("<org>-platform-kms"). No other app's id is,
so every other client_credentials principal fell through to the estate rule, found
the empty `orgs` a machine correctly carries, and resolved nothing. SanitizeIdentity
then minted X-User-Id with no X-Org-Id, and each org gate refused it.
So the recognition comes from the token's SHAPE, which a human token cannot wear: in
a client_credentials token the client IS the subject — IAM sets sub to "<org>/<app>"
— and azp equals the sole audience, because the app asked for a token for itself. A
human's subject is the user and azp is whichever app they signed in through, which is
the exact mis-attribution homeOrg exists to prevent. Every field read is IAM-signed;
none is a header a caller sets.
This does not widen who may cross tenants. It resolves an org for a principal that
has exactly one and can no more choose it than an sk- key can: `owner` is the
application's own organization, and getting the token requires that application's
client secret. A human with no `orgs` still resolves nothing and still fails closed —
tested, along with each half of the shape, because a partial match is a human token
that merely resembles a machine.
An organisation that took its model out of shadow BEFORE the model had learned
anything was told live=true and had nothing written down. The regime lived on the
same row as the learned state, and that row's writer declines to write while the
snapshot holds no learned mass — correctly, because there is no state to lose. So
PUT /v1/risk/state/appetite answered 200, reported live, and persisted nothing;
this binary deploys Recreate at one replica, so the next rollout rebuilt from
defaultConfig — shadow — and the model decided nothing. No error, no log, nothing
to alert on. A model silently disarmed, on a routed door.
The two facts are decomplected. The regime is now its own append-only versioned
record on the tenant's own shelf, written BEFORE anything in memory moves, so a
policy that cannot be written down is refused rather than answered from state the
next rollout will undo.
A regime is a VALUE: a version is minted only when the numbers CHANGE, so a
version means "the Nth distinct policy this organisation adopted" rather than "the
Nth time somebody pressed save", and a client that restates its config on every
deploy is free rather than the cheapest way to fill a disk.
Every score now cites the version it was decided under. Cut is derived from the
appetite that version states, so without the citation a restated appetite made
every earlier decision unreconstructible — the threshold it was measured against
no longer existed anywhere. GET /v1/risk/policy reads the history back.
Bounds, both per tenant and both on the tenant's own table:
RATE at most 24 distinct regimes per rolling 24h, refused past it with the
organisation's own bound named and the regime in force untouched.
TOTAL 381 versions, which IS 256 KiB divided by a measured worst-case row.
At the ceiling the oldest is disposed of and the number disposed of is
REPORTED, derived from the lowest surviving version so it cannot drift.
A regime that predates this record is ADOPTED as version 1 on first residency.
Without that, resolving the regime only from the new record would have returned
every already-live organisation to shadow on the first rollout after this ships —
causing the very defect being fixed, to every tenant at once.
The bounds on review and sample had two spellings, one at the op and one in the
plane. The op's copy is deleted; admitRegime is the one door, at the same
strictness the published contract always had.
Two test gates widened, because both could have been passed by looking at
nothing:
- TestOps_EveryOpIsAdmittedAndPriced parsed typed.go alone, so an op declared in
any other file was admitted and priced by nobody's assertion. It now parses
the package, matches on the RECEIVER TYPE (the plane carries score/learn/
state/appetite too), and fails when a registered op is declared nowhere.
- the per-tenant history assertion over the wire cannot observe the query's
tenant predicate: two orgs are two FILES. The predicate is load-bearing where
two brands share one file, and TestPolicy_TwoBrandsShareAFileAndNotAHistory is
the test that fails when it is dropped.
Eleven mutations applied, each named test red under the defect and green after
revert.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
THE ADDRESS IS THE PRODUCT. openapi.Fold takes an operation's product tag from
the first /v1 segment of its path and nothing else (openapi.Product); a per-op
zip.WithTags names a different axis and cannot override it. The seven ops were
addressed /v1/ml/labels, so seven compliance operations — their own writers
(commerce adjudicates the dispute, the compliance face closes the case, an
analyst files the review), their own five-year retention floor, their own
per-tenant file — would have been published as part of the KServe model-SERVING
product, which is four paths and live with customers on it. Nothing in the fleet
would have said so: the floor ratchet reads `ml: 7 -> 14` as growth, because it
refuses a shrink and only a shrink. It is the same mistake apps/risk's own
manifest row already records having made and corrected once, one layer up.
So: /v1/risk/labels, tag risk, every operation id and schema name risk-prefixed
(riskLabelEvent and not riskEvent — apps/risk publishes a riskEvent already, and
it is a scored decision rather than a judged one). floor.json returns ml to 7 and
raises risk to 17. The manifest row precedes risk, whose prefix is the bare
/v1/risk, and TestEveryServedPathReachesTheAppThatServesIt proves all six paths
reach label over the real fleet router rather than a comment claiming they do.
address_test.go walks the live projection — the same openapi.FleetSpec that writes
the committed subset — so a route re-addressed into somebody else's product fails
at the plane.
A BOUND ON COUNT OVER CALLER-SIZED VALUES IS NOT A BOUND. maxResolve capped a
resolve at 500 named events and nothing capped a subject: the rows were bounded
and the bytes were bounded only by the edge's BodyLimit, which is a fact about the
deployment. Each subject is then amplified below the door — a dedupe key, a
grouping key, one bound parameter per event in a statement against a single-writer
file. The write door had the ceiling all along (admit, subjectMax); the read doors,
added after, did not, and nothing compared them.
There is now ONE spelling of each ceiling — admitSubject, admitKind, admitSource,
admitEvidence, and instantMax inside stamp(), which is the one parser every time
field passes — and every door asks it. So `count × ceiling` IS the byte bound of
everything this plane binds, holds and stores. An unknown kind or source is
refused on the READ path too: it can only ever match zero rows, so refusing says
so instead of charging for the scan. bound_test.go proves it twice: reflect walks
every In type and fails on a caller-sized field with no declared ceiling (the
structural half — a new field cannot arrive unbounded), and every declared ceiling
is refused over the wire with a refusal that does not carry the value back.
A LITIGATION HOLD THAT ARRIVES MID-SWEEP KEEPS THE RECORD IN BOTH PLANES OR IN
NEITHER. dispose sweeps the derived copy FIRST so nothing is orphaned in the
warehouse, then deletes from the record re-asserting `hold = 0`. That protected
the record and silently corrupted the copy: a record the delete declines to remove
has already been swept, its seq is behind the delivery cursor, and deliver() asks
the cursor rather than the world — so no retry re-sends it, pending() answers zero,
and the row is present in the compliance record and permanently absent from the
answer key a training join reads. A missing fraud label reads as an honest
customer, and the row is the one somebody is litigating. remove() now reports what
it kept, the sweep writes those back from the record, a repair that fails refuses
the request rather than acknowledging a short copy, and `restored` is a NAMED state
on the response. `disposed` counts what was disposed of rather than what was
identified: a compliance report that says it deleted a record it is still holding
is the wrong answer to the only question the report is asked.
THE PUBLISHED PRECEDENCE RULE NAMES THE FIELD THE RESOLVER READS. The op exists so
a caller holding a contested resolution can reproduce it, and its second term said
`seen` while stronger() compares `knowable`. The two are equal for a live pipeline
and differ for exactly the backfilled history the derivation exists to hold back,
so a caller reproducing the rule got a different winner and no way to see why. The
test counted the terms, which made the only property that matters unobservable; it
now pins each term to the field at its position.
Also: plugin/label/mcp.json is deleted. It is the only mcp.json in the tree, no
app on main has one, the generator that wrote them was retired with its gate
(manifest/mcp_test.go says so), and it declared seven tools under the old
operation ids with nothing left to regenerate or compare it.
25 mutants in scripts/mutate.py, 11 of them new, 25 KILLED: each reintroduces one
of these defects and the named test goes RED.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
plugin/dataset declares Price: cloud.Metered. TestMeteredSurfacesRequireStanding
(added on main after this branch forked) fails on a metered surface missing from
meteredApps: it would spend a provider's money with no standing check. The scan
that materialises a set is priced per source row read.
Also carries the wove openapi.yaml for the merged surface.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This binary links orm/db, whose toJSONFieldName lowercased the first rune of
each dot-segment and returned the rest untouched — so an all-lowercase payload
passed through byte for byte into json_extract(data, '$.%s'), a SQL string
literal a quote ends.
v0.6.19 keeps only identifier characters and drops the rest, rather than
escaping them: there is then no escaping to get wrong and no dialect to be right
about. It fails closed by construction — a name that had illegal characters
becomes a field that does not exist, so a filter on it matches no rows and an
ORDER BY on it sorts every row equally.
Bump only — no cloud source changes. The four failing packages are the standing
pre-existing set (apps/iam, apps/plan, apps/pricing, manifest); the bump adds
none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The resident bound is a count of tenants with an LRU over it, so one
organisation's arrival drops another's residency. Two properties make that a
capacity decision rather than one tenant unlearning another: the victim's state
is saved before the residency goes, and the reclaim is counted on the probe.
Both were prose. Removing either left the whole suite green.
Losing the save also leaves the slot the eviction took in p.opening held, so the
victim is locked out for the life of the process as well as returned to its last
save.
TestRings_SurviveAnEviction names the lossless property but reimplements it —
it deletes the map entry, bumps the counter and calls save itself — so it cannot
fail when the plane stops doing any of the three. The two new eviction tests
drive plane.resident past maxResident and read only what an operator can read.
Two further tests state the per-field byte cap and the forgotten-subject count
from the direction an operator reads them. Those bounds were already covered
(TestField_IsRefusedAtTheDoorAndNotTruncated,
TestRings_TheCeilingIsMeasuredNotAsserted) and no gap is claimed for them.
Test-only. Nothing is armed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two injections landed upstream today and this binary was still pinned behind
both of them, which is the only part that decides whether they are fixed in
production.
commerce v1.49.50 is the one that matters. Its generic REST list handler put
`?sort=` into ORDER BY json_extract(data, '$.%s') — and into data->>'%s' on the
postgres side — with no check. `a') , (select 1) --` closed the path and the
call. Ordering by a conditional turns ROW ORDER into a one-bit oracle, so it
read another kind's rows across the per-kind authorization boundary a column at
a time. The `data` column is a BLOB, which is why it read as harmless: a bare
LIKE against it matches nothing, and only cast(data as text) shows the leak.
ai v1.832.14 is the original sortField fix in GetDbQuery, and v1.832.15 covers
the two siblings that had copied the same unguarded ORDER BY into chat and
provider listing.
Bump only — no cloud source changes. money v0.2.5 rides along transitively. The
four failing packages are the standing pre-existing set (apps/iam, apps/plan,
apps/pricing, manifest); the bump adds none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
surface-check piped every step to /dev/null and then, on failure, said
'see the message above' — but there was no message above, because it had just
been thrown away. The weave's diagnostic names exactly which schema collided in
which two apps; discarding it is what made the o11y collision expensive to
diagnose.
Proven by the first run after the change: the failure had been reported as
'account cannot project its own document — an app that cannot describe itself is
the bug', which points at apps/account. The captured output says '/bin/sh: go:
not found'. The app was never the problem; the gate was blaming it for a missing
compiler.
Three sites, same shape: both describe loops and the weave. Output is captured
and printed only on failure, so success stays quiet.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The v1.5.51 bump covered the v5 statement builders. This covers the v3/v4 ones,
which are the live path for the older query API and were raw in three positions
rather than one: the ORDER BY direction (a bare string, so
"asc,(select count() from system.tables)" was a whole extra ordering term and
needed neither a quote nor a backtick), the column, and the attribute key in the
single quotes of the map accessors — that last one landing in the WHERE clause,
where it is a predicate the caller writes rather than an ordering oracle.
On the lines that build a filter the VALUE already went through
QuoteEscapedString and the KEY beside it went in bare, which is the same
asymmetry the original sortField bug had.
Bump only — no cloud source changes. The four failing packages are the standing
pre-existing set (apps/iam, apps/plan, apps/pricing, manifest); the bump adds
none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The embedded o11y query plane serves /v1/o11y out of this binary, so the
injection fixed in o11y 25bb4acfb0 was live here on v1.5.50: a telemetry field
key name — caller JSON on the query-range door, and replayed out of stored
dashboards and alert rules — reached ClickHouse unquoted in two positions a
bound parameter cannot occupy, the backtick-quoted identifier (SELECT alias,
GROUP BY, ORDER BY) and the single-quoted attribute accessor.
A metrics group-by name of
le`,(select/**/grouparray(name)/**/from/**/system.tables)/**/as/**/`x
emitted that subquery as an additional SELECT column, and its rows came back in
the response body. Metrics is the reachable one because a metric label is
free-form: an unrecognised name falls back to the labels column by design, so
there is no key allowlist for it to fail against.
Bump only — no cloud source changes. The four failing packages are the standing
pre-existing set (apps/iam, apps/plan, apps/pricing, manifest); the bump adds
none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
plugin/ai/openapi.json was last written at 3b50091c, with go.mod pinned to
hanzoai/ai v1.832.10. d5d768f1 took v1.832.12 for an unrelated key fix and
754fb821 took .13; neither regenerated a subset. Since then the published
document has been describing a router the binary no longer has.
v1.832.12 rewrote the ai door's resource table. Three resources moved address and
one was deleted:
/v1/ai/applications -> /v1/ai/deployments 8 operations
/v1/ai/sessions -> /v1/ai/signin-sessions 7
/v1/ai/users -> /v1/ai/usages/user-names 1
/v1/ai/users/table-infos -> /v1/ai/usages/by-user 1
/v1/ai/permissions deleted 6
The six deleted are GET and POST /v1/ai/permissions and GET, PUT, PATCH and
DELETE /v1/ai/permissions/{owner}/{name}. Every handler behind them was an iam.*
call to the IAM server — controllers/permission.go held no rows of its own — so
the address was a second door onto /v1/iam/permissions, which apps/iam serves and
still serves. Upstream deleted the controller with the routes and says so in the
same edit.
So openapi.yaml has been advertising 23 addresses that answer 404 and hiding 18
that are served: the 17 renamed above plus POST /v1/iam/oauth/device/info, whose
prose was written in b5d8f927 and has been waiting for a regeneration to reach the
document. Every SDK, the CLI's command tree and docs.hanzo.ai are projections of
this file, so a client generated from it calls /v1/ai/applications and gets
nothing.
floor.json comes down by six, which is the act it is designed for: the deletion is
upstream, deliberate and already shipped, so the number moves in the commit that
carries it rather than being absorbed by a re-measure. paths and operations follow
(-1 and -5 — the six net against iam's one), and iam goes up by one.
Forty-two other subsets change prose and schema only: a summary now leads with
what the operation does rather than with the Go identifier that implements it, and
visor's cluster list publishes the degraded field its response type already
carried.
Nothing in the router changed here. Regenerating a second time reproduces all 119
generated files byte for byte.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Complete judged an orphan by Product() — the first path segment after /v1/. A
product is not an app, and fourteen of them are answered by more than one,
because the manifest separates nested prefixes exactly as the router does.
/v1/s3 PROVISIONS an s3 resource (provisioning) while /v1/s3/buckets is the DATA
plane (storage), so storage was charged with provisioning's POST /v1/s3 — a
declaration provisioning both serves and describes — and could not project its
own document. surface-check died there, on an app with nothing wrong with it.
Attribute by manifest.OwnerOf instead: the longest-prefix rule the host itself
routes by, so the gate blames the app the request would actually reach. Asked
rather than re-derived, the two cannot drift.
It is passed IN rather than imported. manifest's own tests read this package to
pin the spec door's address, so an openapi that imported manifest back makes
manifest's test binary an import cycle — the compiler says so. nil is refused
rather than defaulted, because judging nothing is a gate that reports success on
every defect it exists to catch.
The host's own door is declined explicitly. owner answers a MANIFEST question,
and for /v1/openapi.json the manifest's answer is precisely the misroute
cmd/cloud's static route exists to correct; it is served by the host, mounted by
no app, and renders in the fleet document. Product() hid that by accident,
returning "" for any segment holding a dot.
The gate is unchanged where it matters. A declaration misfiled inside an app's
own prefix still fails — pinned here, and mutation-proven against apps/storage,
where an added GET /v1/s3/buckets/:bucket/files takes the run red while the
false positive stays gone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The admin read counted "active" and "trialing" alike when summing mrrCents off
commerce's subscriptions wire. commerce's own rollup counted active only, and
the warehouse SQL behind the SaaS board counted everything except trialing --
so past_due and unpaid were revenue there and nowhere else. Three definitions,
three different MRR numbers for the same account depending on which board you
opened, and no test anywhere asserted the two agreed.
commerce now owns the answer: subscription.Status.CountsTowardMRR. A trial is
not revenue, because nobody has been charged and the trial may end in a cancel
-- counting it lifts the board exactly when a promotion drives signups. This
surface asks that function instead of matching status strings.
Revenue and entitlement are separate questions and the loop still answers both.
A trialing subject is subscribed and still names its plan; it just contributes
no money yet.
The metrics SQL cannot call Go, so it spells the same predicate as
status = 'active' and says so where a reader will find it. That also drops
past_due and unpaid from the board's run-rate, which the Go side never counted.
Picks up commerce v1.49.48, which also stops overstating a quarterly plan by 3x
-- MRR now divides by IntervalCount, so the mrrCents this reads is already
right and nothing here re-derives it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The package, its directory, its plugin and its manifest row all said
"agentskills" — a compound naming the AUDIENCE (agents) alongside the thing
(skills). The thing is skills; every reader of this catalogue is an agent, so
the qualifier distinguishes it from nothing.
Renames the token only. The wire is untouched on purpose: the served addresses
stay /.well-known/agent-skills/... and the document schema stays
hanzo.agent-skills/v1, because those are an external discovery convention we
implement rather than a name we own. The manifest row's Prefixes are unchanged,
so nothing moves address — only the app's own name, its import path
(apps/skills), its plugin binary and its file names change.
Also fixes two comments still pointing at clients/agentskills, a path that
stopped existing when the subsystems moved to apps/.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ed4fbe5b corrected the auth line in all three brand catalogues — hk- is not a
key, so the sentence pointed a reader at a credential that resolves to nobody.
The correction is right. But SKILL.md is a GENERATED file whose bytes are
attested by a sha256 in index.json beside it, and the hand-edit changed the
bytes without recomputing the digest. All three brands went stale at once.
That is not cosmetic: the digest IS the integrity contract a client checks
after fetching a skill document, so every reader that verified was told the
catalogue had been tampered with. TestServeSkillDigest and TestCatalogIntegrity
were failing on main for exactly this, and they were right.
Recomputes the three digests and nothing else — the served bytes were already
correct, only the attestation had fallen behind them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`make -f mk/fleet.mk openapi-weave` has refused since 593aa309 regenerated the
o11y subset:
schema "Service" means different things in "ingress" and "o11y"
Six names carried two shapes across the fleet — `Service` and `TLSConfig` against
ingress, `Account` against books, `Channel` against content, `Event` against
analytics, `Host` against plugins — all six from hanzoai/o11y's internal type
packages. The weave is right to refuse: a generated SDK binds whichever shape the
merge reads last. But openapi.yaml is regenerated by ONE command for the whole
fleet, so the refusal did not merely hold o11y's typed ops out of the document —
it meant nobody could add or change ANY api surface and prove it.
Renaming the six upstream only reveals the seventh. They are ordinary words, five
other apps use them, and a type name that has to stay unique against every app in
the fleet is a name nobody can choose safely. The seam is the defect, not the
nouns.
apps/iam already answered this and there is exactly one mechanism: zip.Graft. A
grafted op carries Origin = the child's AppName, and zip qualifies every named
type it reaches as "<origin>.<Type>" — unconditionally, not on collision, so a
published name is never a function of who else is in the room. That is why
identity's 95 schemas are iam.* and have never collided with anything.
MountO11y now builds one zip.App{AppName: "o11y"}, mounts the whole observability
surface on it — cloud's own org-pinned reads and hanzoai/o11y's relay table
together — and grafts that into the host. ONE origin for the product: splitting it
down the module seam would namespace half of o11y's types and leave the other half
bare, which is two conventions for one thing.
make -f mk/fleet.mk openapi-weave ok
…and again, byte-identical (idempotent)
THE PUBLISHED SDK SURFACE MOVES. Schema names are part of the contract:
- 777 o11y schema names go from X to o11y.X. Against the openapi.yaml published
today, 24 of those are renames of names it already carries; the other 753
appear for the first time, because the weave has refused to publish them since
they arrived.
- NO other app's schema names change. Each of the six formerly-colliding bare
names stays with the app the published document already binds it to —
Service/TLSConfig ingress, Account books, Channel content, Event analytics,
Host plugins — with the same shape.
- Nothing outside apps/o11y changes to keep compiling. These are generated
schema keys, not Go identifiers; no test in this repo pins one of o11y's.
- The MCP tool list is untouched: plugin/o11y/mcp.json is byte-identical.
The rename is PURE, and measured rather than eyeballed. Against the subset that
PRISTINE main regenerates from the same source: all 389 (path, method,
operationId) triples identical, and all 389 operation objects byte-identical once
the o11y. prefix is undone — parameters, bodies, responses, tags, prose and x-app
included. The woven document resolves 2987 $refs with zero dangling.
Two facts the graft made local, both load-bearing:
ALL /v1/sentry/* stays on the HOST. zip.App.Declaration drops HEAD and OPTIONS
unconditionally — they are the shadows fiber generates for a GET and for CORS — so
a door opened with All cannot cross a graft intact, and OPTIONS is a method that
proxy genuinely answers and publishes as an operation. It is registered at the
same point in the same order, and costs nothing there: a wildcard proxy declares
no typed op and contributes no schema.
cloud and the module BOTH claim GET /v1/o11y/{logs,metrics} and POST
/v1/o11y/query_range. scope.go is the one owner — it pins the caller's org
server-side — and first-registered is what makes it the half that answers. That
used to depend on the host's global mount order; both halves now register on one
app, two adjacent lines apart. TestHostRoutesStillWinTheThreeSharedAddresses is
the gate, and it is mutation-proven: put o11y.Mount first and all three go red.
openapi/floor.json rises with the document — 1410 → 1679 paths, 1979 → 2333
operations, o11y 28 → 363, sentry 7 → 24 — and no product falls, which is what the
ratchet checked before writing it.
One document defect SURVIVES and is unchanged: at POST /v1/o11y/query_range the
router answers cloud's untyped builderQueryHandler while the document publishes
the module's typed contract, because only the module's half has a registry entry
to publish. It predates this and goes away with the console v3→v5 migration that
deletes cloud's half (apps/o11y/LLM.md).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/iam/oauth/device/info arrived with hanzoai/iam v1.34.0 -> v1.34.5
(85f5697c) and reached the router with no prose anywhere. It is an UNTYPED route
inside a GRAFTED app, so neither seam that normally carries prose reaches it:
zipdoc lifts doc comments off TYPED ops, and the upstream module ships no lift
for this one, so there is nothing in zip's extraction for a host to read.
openapi.Complete then refused the whole document — "1 operation(s) say nothing
about themselves" — so apps/iam could not describe itself at all, and the drift
gate stopped there without reaching any app after it.
openapi.Describe is the seam that refusal names for exactly this case. It is
additive metadata on a route the router already carries, so it cannot add, move
or rename an operation; it renders nothing if the address is not live. The
sentence is the upstream handler's own, restated where a host can reach it, and
it says the three things a caller cannot guess: the fields come off the pending
code's own application, the read is a POST because the user_code is a secret,
and every refusal is the same opaque one so a 40-bit code cannot be hunted.
The sentence belongs upstream, on the operation. When github.com/hanzoai/iam
gives the op its own prose, this goes away.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zap-proto/zip went v1.18.16 -> v1.18.23 (c516e5a3, then 85f5697c) and the doc
extraction changed with it: a lift no longer keeps the leading Go identifier,
so "RevokeKey revokes the caller's own API key" became "Revokes the caller's own
API key". The committed lifts were never regenerated against it, so `zipdoc
-check` — the per-package gate `make test` runs before the suite — was red in 53
of the 95 directories that carry the generate line.
This is `go generate -run zipdoc` and nothing else. Parsing both trees' ASTs
rather than diffing their text, it moves:
377 Description lifts losing their leading identifier
8 Fields lifts catching up to a doc comment their source already changed
86 routes gaining a lift at all — the bump also reads an UNTYPED route's
prose, which is the whole point of c516e5a3
and, in the same comparison:
0 Examples changed
0 routes lost a lift
0 Descriptions changed in any other way
A lift carries Description, Fields and Example and nothing else, so it can only
ever move PROSE — it cannot add, remove, rename or retype an operation. The
router remains the only thing that decides what is published.
51 of the 86 new lifts are the doc comment of cloud.Handle, not of the handler
it wraps: the extraction follows the Service-scoped binder rather than the
function passed to it. Every one of those routes already declares its prose with
openapi.Describe, which outranks a lift, so none of it reaches a published
document — verified: the string "Binds a Service-scoped handler" appears zero
times in plugin/*/openapi.json. It is inert, and it is what the generator emits;
editing a generated file to remove it would only turn the gate red again.
The published documents are deliberately NOT regenerated here. Doing so moves
real API surface that is nothing to do with this bump — plugin/ai alone renames
23 operations and adds 18 behind the ai v1.832.12 bump — and openapi-weave
refuses to run at all while six schema names mean two different things across
apps. Both need their own review.
Regenerating a second time produces zero further change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GetCert qualified the certificate id with the caller's tenant while
GetApplication used the platform partition, so a deployment whose IAM_ORG was not
"admin" resolved its application and then could not find that application's
certificate. With no cert there is no key to validate a bearer against, so the
process authenticates nothing — and the message named a missing certificate
rather than a lookup pointed at the wrong tenant.
It surfaced the moment identity moved in-process on devnet: /v1/models 503 with
`read cert "cert-hanzo" ... does not exist`, while the same cert answered 200 at
id=admin%2Fcert-hanzo. Production never hit it because it reads the standalone
IAM, whose store has been converged for months.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
main moved again while the suite ran. One conflict, modify/delete on
plugin/gateway/mcp.json: main deleted all 116 committed tool files because the
catalogue is a query now — the host asks each child over its ZAP socket rather
than reading an array some earlier build wrote down — and this work had
regenerated the gateway one.
The delete wins, and it takes plugin/risk/mcp.json with it. That file is the
same artifact class main just retired: nothing embeds it any more
(plugin/embed.go), nothing writes it any more (mk/plugin.mk, describe.go), and
leaving it would put back one of the 116 second sources whose whole defect was
that they could only be stale or accidentally correct.
Still nothing armed and nothing routed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
internal/org and apps/mpc each turned the master into a key themselves, so the
estate had three answers to one question and no test compared them.
The snapshot cipher derived HKDF(master; salt="hanzo/org-db/v1"; info=orgID) by
hand while the local file it snapshots was opened by cek. The two were supposed
to agree and nothing checked that they did — a comment's promise, which is the
same shape of bug as base's deleted core/encryption.go, where the claim was
simply false. Worse, keying on the org alone meant every database an org owned
sealed under ONE key: its settings snapshot opened its ledger. cek binds the
subsystem, so Seal/Open now take the name (ns, subsystem) the file was opened
under, and TestCipherKeyIsCEKDerived opens a sealed blob with a key derived
independently through cek — the equality is now a test, not a sentence.
apps/mpc ran HKDF(master; salt=orgSlug; info="cek-aes256gcm"), inlined verbatim
from hanzoai/kms/sdk/go. That repo is archived and read-only, so this is the
only live copy and there is nothing left to stay byte-identical with. Argon2id
stays exactly where it was: stretching a guessable passphrase and deriving a key
for a named thing are different jobs, and cek does the second one. The client
takes a namespace instead of a slug — cloud has one door where a string becomes
a tenant, and TestOnlyOrgnsBuildsANamespace is right that this package is not it.
Also drops the NewReplicator/WithEncryption aliases. Nothing in cloud shipped
through them; they only survived to let one test seal through replica.Cipher,
whose string-shaped Seal a namespace-keyed cipher cannot satisfy. That test is
replaced by TestDurableShipsCiphertext, which asserts the same property against
the path that actually ships: Sync writes a blob the marker does not appear in,
and a successor reads the row back out of it.
Nothing migrates. A database is born under its key or it does not exist.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Prefixes NEST. provisioning is metered and routed /v1/vector and /v1/search;
product is Free and routed the more specific /v1/vector/collections,
/v1/vector/stats, /v1/search/indexes and /v1/search/stats. Billable scanned the
metered trees with a bare HasPrefix, so all four of product's surfaces billed on
provisioning's standing — a Free product gated behind a balance.
The router resolves by longest prefix, so ownership does too: manifest.OwnerOf
answers which app actually serves a path, and if that app is not metered then
nothing is spent and nothing is owed. No new list to drift — the manifest already
owns routing, meteredApps already owns price.
Also deletes an invented history. The comment claimed "/v1/provisioning/, which
provisioning does not answer" had been in this file; `git log -S '"/v1/provisioning/"'
-- spend.go` returns exactly one commit, the one that wrote the claim. The real
defect was assuming provisioning's tree was /v1/provisioning when the manifest
routes it elsewhere, which is what the corrected text now says. A comment
describing a defect that never existed is the same failure the commit was
punishing.
Negative-controlled: removing the ownership check fails the new test on all four
paths. The 19 TestAudit_* failures in this package are pre-existing (verified by
stashing) and untouched.
vmproxy.go was the final VM reader in the repo. It spelled a vendor into the
route table (/v1/o11y/vm/{query,query_range}), admitted 19 exact PromQL
strings, and returned that store's Prometheus envelope byte-for-byte. The store
is gone, so the route named for it and speaking its wire is gone too — keeping
it would be a dependency's vocabulary outliving the dependency, which is the
thing metricsgauge.go already refused when it declined to keep the envelope.
Of the 19 queries, exactly 3 are still MEASURED. `up`, `sum(up)` and `count(up)`
were the platform-health board asking how much of the fleet is up now and
lately, and the fleet prober has been recording hanzo_service_up every 30s the
whole time. So that question is asked plainly instead: GET /v1/o11y/availability,
a typed op, platform-sudo as before, answering the instant inventory and the
trend in ONE read where the board used to make three PromQL round-trips.
The number changes and the endpoint does not hide it. `count(up)` counted every
scrape target VM federated — hundreds. `total` counts fleetTargets, a couple of
dozen services we chose to knock on. Per-replica identity stays gone, for the
reason status.go already recorded: a Service address is not a pod.
The other 16 have no native producer and are DELETED rather than answered
empty — verified, not assumed:
node_memory_* node-exporter, scraped per node
container_memory_working_set kubelet cAdvisor, scraped per pod
kube_deployment_* kube-state-metrics, scraped
kube_statefulset_* kube-state-metrics, scraped
lux_validator_* (4) ghcr.io/luxfi/monitoring /validator-exporter —
lux_network_* (6) binds :9101, renders a Prometheus exposition,
no push/OTLP/egress at all; a lux-side vmagent
scraped it and remote-wrote to the hub. That
write had already POSTed into a 404 for 1369
consecutive retries in July, so the board was
dark before this change.
ALERTS{alertstate="firing"} vmalert's remote-write of its own state, and
nothing else.
availability.go's header is the ledger: each dead producer by name, and what
would have to be MEASURED to get the panel back. Two of them should not come
back as metrics at all — the replica grid is the API server's object status and
should be watched at the source, and the firing set is rows in o11y's ruler
store, so it is a projection of rule state rather than a series to re-collect.
The lux boards come back when that exporter pushes to the ZAP metric receiver
the way this process does; gaugeSeries then works on those names unchanged.
Five of the console's 18 declared queries were already dead weight — the four
freeze-detection signals and the alert rollup were allowlisted here and fetched
by no panel.
Console impact. The client reads data.result[].value and never checks `status`,
so a reshaped 200 would parse to [] and paint a board of zeroes; a deleted route
throws and renders its error card. That is why this is a deletion.
MetricsModule/InfraMetrics (/metrics) KEEPS everything, move to the new op
StatusModule (/status) KEEPS everything but the Endpoint col
LuxNetworkModule (console.lux.cloud) goes fully dark; nothing can serve it
Also retires the prose VM left behind: telemetry.go and metrics_http.go claimed
metrics were collected by scrape because "every metric READER queries
VictoriaMetrics" — both false since the :9464 exposition was deleted. Every
reader now goes through metricsgauge.go. serviceUp is shared with summary.go
rather than copied, since it is the same value.
Gates: build, vet, ./apps/o11y/ + root tests, and cmd/cloud deps — all pass.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both changes were merged and neither was reachable: v1.832.10 and v1.49.46
predate them, so the fused binary kept resolving hk- through the old
dispatch no matter how many times main was rebuilt.
ai v1.832.11 makes isIAMApiKey recognize sk- and asks the provider table
first, so a retired prefix falls through to IAM and comes back key_unknown
with the one sentence a stranded holder can act on. v1.832.12 stops that
sentence's sibling — the pk- refusal — pointing at /api/models, which 404s.
commerce v1.49.47 drops the same dead prefix from its edge comments.
The dispatch order moved, so the hk- refusal has to be re-verified against
the rolled image rather than assumed: it reaches IAM by a different road now.
A valid, funded sk- key could not buy one token of inference. IAM resolved it
correctly — get-user?accessKey returned {"status":"ok","data":{"owner":…}} —
while POST /v1/chat/completions answered 401 "invalid API key" for the same key
in the same second.
Both answers were right. They were answering different questions, because the
inference path never asked IAM.
ai v1.832.10 is what this module pinned, and there isIAMApiKey(token) meant
strings.HasPrefix(token, "hk-"). Once hk- was retired estate-wide, IAM minted
only pk-/sk-, so every key a customer can hold missed that branch and fell to
the dispatch default — which read sk- as an UPSTREAM VENDOR key, looked it up in
the provider table, missed, and returned the bare authError("invalid API key").
The bare string with no ": %s" suffix is the fingerprint: it is the one refusal
on that path that carries no cause, because a provider-table miss has none to
give. IAM was never consulted, which is exactly why querying IAM directly
disagreed with the endpoint.
ai v1.832.11 fixed it by asking the STORE rather than the spelling: the provider
table first (an exact lookup, so it can never claim a key it does not hold), and
anything it misses is put to IAM, whose refusal names the cure where a provider
miss can only say "invalid". That fix has been tagged and unreachable since —
this module still pinned .10, and so did v1.801.375, so rolling that tag would
have shipped the same 401.
Nothing here weakens the door. The two shapes and their guards are unchanged and
still covered: pk- authenticates nothing (KeyWrongDoor, never a principal — it
ships in client JS), sk- keeps its same-tenant pin (KeyForeignUser still fires on
a cross-tenant reference), and an hk--shaped string is simply not a key and takes
the generic key_unknown path that renders "mint a new one at cloud.hanzo.ai/keys"
— no branch of its own, so the third family cannot come back.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
main moved eight commits while this was building, so it is integrated here
rather than pushed past. One conflict, in spend.go, and it is the good kind:
both sides were right about different things.
main replaced the hand-written meteredTrees path list with meteredApps — a set
of app NAMES resolved through manifest.PrefixesFor — because a copied routing
table goes stale, and that one had in six ways. This branch had added
"/v1/risk/" to the list main deleted.
The mechanism is main's and the fact is this branch's. plugin/risk declares
Price: cloud.Metered, and main's new TestMeteredSurfacesRequireStanding fails
in BOTH directions — a Metered surface missing from the list is a surface that
charges and cannot be gated. So risk is carried across as a name, and that test
passes because of it, not in spite of it.
Nothing is armed and nothing is routed. apps/risk is still mounted nowhere;
/v1/risk appears only as an abuse-gate exemption, a manifest entry and now a
standing requirement. The abuse gate stays shadow per org — edge.Store.Mode
returns live only on an exact match and is not inherited. /v1/ml is untouched.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The fleet's agent door answered from plugin/<app>/mcp.json: the tool array each
app's binary projected when it was BUILT, embedded by plugin/embed.go and handed
to zip as Plugin.Tools. 116 files, 49,865 lines, and a second source for a fact
every child already knows.
A second source can only be stale or accidentally correct. This one was stale in
the way no gate in this repository could see: o11y's 353 missing ops live in
github.com/hanzoai/o11y, so a go.mod bump in ANOTHER repo invalidated an artifact
in this one with nothing in the diff to say so. Regenerating it more often is not
the fix — a generator on a hook is still two sources with a race between them,
and the trigger is in a different repository. (593aa309 did regenerate it, which
is why the file reads 365 today. The next cross-repo bump silently un-fixes it.)
So the host asks. POST /v1/mcp is the HOST's own handler now (zip's is Disabled,
so exactly one handler holds the address). A tools/list forwards the CALLER's own
message to every composed subsystem's own /mcp over its private ZAP socket, in
parallel, and unions the replies — zip.App.Start resolves a cold child on the
same single-flighted path a prefix request takes, so the first list pays one
start per app and nothing after it does. A tools/call goes to the app that listed
the name, verbatim; the child's own registry decides whether the tool exists.
A SUBSYSTEM THAT DOES NOT ANSWER IS NAMED, in result._meta["hanzo.ai/unavailable"],
because a silently-short list and a stale file are the same defect: the caller
cannot tell an app that serves nothing from one that did not answer. Measured on
the built binaries — host + real o11y child, kms pointed at a dead address:
tools=364, unavailable=[{kms, connection refused}]
364 and not 365 because o11y projects get_v1_o11y_logs twice; the door serves the
first and logs the collision. THAT DUPLICATE IS WHY cmd/cloud's tests were red on
main — zip refused the Load ("tool is already served by plugin o11y"), a boot
failure. One name still has one owner; it is no longer fatal to the fleet.
Also gone with the mechanism they configured: manifest.App.Open and zip's
one-open-plugin rule. The host forwards the caller's own request to EVERY
subsystem now, so each answers for this caller out of its own rows, and being
asked per caller is no longer a privilege one app holds.
The release gate moved with the door. Car 3 compared the live tool count against
`jq -s length` over the committed files — both sides were the same bytes, so it
proved only that the image carried its own tree, and it passed while o11y's
catalogue held 12 of 365. It asks the better question now: did every subsystem
answer. A broken deployment used to match the files exactly.
plugin/<app>/openapi.json SURVIVES, for the one reason the catalogue could not:
the weave carries each subsystem's prose, and that prose is lifted from the app's
SOURCE at describe time (openapi.Synopsis). A running child has no comment to
read and would answer with its deployment's brand blurb, which the weave would
publish as the description of every product tag. Deleting that half waits on the
synopsis becoming a declared value.
Tests are against RUNNING subsystems (fleet/mcp_test.go): real zip children on
real ZAP sockets, exact sets, bodies never status codes. Mutation-checked three
ways — unmount a child, silence the outage report, ask only the first app — all
three go red.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nothing textually conflicted, so the merge could not see this: the branch was
written when hk- was one of the key shapes, and main dropped it from
APIKeyPrefixes in the meantime. isAPIKey("hk-...") is now false, so a validated
caller presenting one falls through credentialClass to session rather than
secret, and TestCredentialClass_ReadsTheCredentialNotTheClient failed on the
one subtest that asserted the old taxonomy.
Main is right and the branch is stale, so the three sites the branch
reintroduced are re-cast the way main re-cast every other one. The fixture is
kept rather than deleted, restated as the negative it now proves: an hk- string
contributes nothing to the classification, so putting hk- back into
APIKeyPrefixes turns this red.
No behaviour changes. Both lanes involved are attributed lanes — secret maps to
agent, session to human — and neither is the bot lane, so nothing is granted or
refused differently. The authentication path is untouched: hk- reaches no key
door and resolves to no principal, which is main's point.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud pinned cek v0.2.1. The releases after it were a trap: v0.2.2 shipped a
sidecar.go written against sqlitedrv APIs that hanzoai/sqlite v0.5.0 removed,
so anyone bumping cek and sqlite together got a module that does not compile —
and cloud now takes sqlite v0.5.0.
v0.2.3 is the release where the sidecar states its own scheme instead of
reaching into the driver for it, which is why it builds against v0.5.0 at all.
Verified: cek at v0.2.3 builds and its own suite passes on sqlite v0.5.0, and
cloud's suite is unchanged by the bump — the same 7 packages fail before and
after, none of them cek's.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
tools and automations both debited a unit on the way out and never gated on the
way in. An org at $0 ran either one unbounded and the ledger simply went
negative — the rate budget automations already had bounds the SHAPE of the spend,
never the amount. Both now gate the same unit they meter, from the same knob, so
what is authorized and what is charged cannot drift. automations gates in
startRun, the one choke point all four run-start paths pass through (manual, MCP,
trigger, cron), so the gate is not something three entrypoints walk around;
engineErr stops folding a 402 into "engine: …" 500, which told a customer their
automation was broken when they had run out of credit.
meteredTrees restated the fleet's routing table by hand and had gone stale in six
ways, each one a surface that charges and could not be gated: it said
"/v1/provisioning/", which nothing answers — the manifest routes provisioning at
/v1/{datastore,docdb,kv,search,sql,vector}, so every vector/sql/kv/docdb create,
the exact set the non-LLM billing gap was opened for, was not a billable path at
all. projects answers /v1/sites, venue answers /v1/cloud, tools answers
/v1/skills and /v1/plugins, and ten Metered surfaces were missing outright. It is
a set of NAMES now, resolved through manifest.PrefixesFor — which paths an app
answers is the manifest's fact, read from there rather than copied.
spend.go has cited TestMeteredSurfacesRequireStanding since the day the list was
written. It did not exist. That is the third comment in this repo caught
describing a gate nobody built, which is worse than no gate: it reads as
enforced, so nobody looks. It exists now, reads Price out of every
plugin/<name>/main.go the way TestPriceDeclared does, and fails in both
directions — a Metered surface missing from the list, and a listed surface that
is no longer Metered.
Also deletes automations' meterUnit, which had no callers.
On 08-01 api.hanzo.ai/v1/models and /v1/chat/completions answered 503
{"error":"mount /v1: no instance running"} for ~30 minutes while the pod
reported Ready with 0 restarts.
Nothing in that sentence was a bug in mount(). o11y v1.5.41 seized
:4317-:4319 from the `ai` child, the child's listen failed, and the host
degraded it to absent — which is the 2026-07-29 fix working exactly as
designed. The defect is that absence was reported in a FIELD and the probe
reads the STATUS CODE. `ai` owns the greedy "/v1", so every specifically
mounted prefix (/v1/sentry, /v1/o11y, /v1/commerce/tenant, /v1/admin/*)
kept answering from its own subsystem and only a path falling THROUGH to
`ai` showed it.
So split the question the two probes were sharing:
/healthz liveness. 200 while the process routes, unchanged. Restarting
cannot help — the cause is in the image or the config, so the
replacement fails identically — and it would take the console,
the log stream and 111 healthy siblings with it.
/readyz readiness. 503 when draining, or when a VITAL subsystem is
absent. New route; it did not exist on :8000 at all.
App.Vital is the other half of App.Required, kept separate because they
answer different questions. Required asks "may this process run", and its
own doc names "a pod that never goes Ready" as the one thing aborting buys
before rejecting the whole package. Vital buys exactly that and nothing
else: the pod stays up, keeps serving its siblings, and can still say why.
Required's bar is "serving without it is UNSAFE"; Vital's is narrower —
"serving without it is POINTLESS". Exactly one app qualifies, and "/v1" is
why: it is not a subsystem's prefix, it is the product API's remainder.
The timing is what makes a 503 cheap: a rollout whose image cannot start
`ai` never gets a Ready pod, so the Deployment stalls and the OLD pods keep
serving — the bad config stops at the first replica instead of reaching all
of them. Fleet-wide, endpoints empty and the product is down, but it was
already down; now kubectl says so.
A non-vital absence stays READY and is still reported. Pulling a pod
because one minor subsystem died turns a partial failure into a total one,
which is the same mistake as aborting, made later.
Also closes a contract drain.go describes and nothing delivered: it states
that SIGTERM flips /readyz to 503 so K8s marks the pod NotReady, and the
root package implements that on the ops listener at :9090 — but both k8s
probes point at :8000, where /readyz 404'd. The drain signal guarding the
M3 writer-election handoff was never read by anything. The host keeps its
own flag rather than borrowing cloud.SetDraining(): it does not link the
root package, and it is a different process anyway.
Tests reproduce the outage with the real app name and the real child-exec
path (/bin/false), and fail without the fix:
GET /readyz = 200, want 503.
The chart still probes /healthz for readiness and MUST NOT be repointed
until an image serving /readyz is rolled — a 404 fails readiness exactly as
hard as a wrong port, which is its own recorded outage.
All three white-label catalogs told agents an hk- key was accepted. It is
not — it resolves to nobody — so the sentence sent a reader to a credential
that cannot work. Names the secret sk- instead, and says plainly that a
publishable pk- identifies an org and never authenticates.
Found by a binary-safe grep: these files sit beside committed binaries whose
string tables contain the prefix, and the default grep skips those paths, so
a normal sweep reports the tree clean.
cloud already embeds IAM (apps/iam, grafted in-process) and prod already enables it,
but /v1/iam/.well-known/jwks answers {"keys":[]} because the store it opens is empty:
/var/lib/cloud/orgs/_platform/global.db, 4096 bytes. The real identity graph — 391
users, 134 orgs, 286 applications and the 10 signing certs — lives in the standalone
pod's /data/iam/iam.db. That is why every service still points at iam.hanzo.svc, and
why a single pod failing readiness takes identity down for the fleet.
It is not a file copy, because the stores differ three ways at once. The standalone
writes PLAINTEXT — the literal "SQLite format 3" header, which is the exposure
apps/iam/openStore exists to remove. The embed writes through cek, keyed from the
process master. And it opens exactly one namespace-derived path, so a file dropped
beside it is not a file it reads.
So the rows move, not the bytes: read the source with a plain driver (read-only and
immutable, so a live pod's WAL is never touched), write through the same cek.Open the
subsystem itself uses. One transaction — identity half-moved is worse than not moved.
Columns and DDL are read from the source rather than restated here, so a schema that
grows cannot silently lose its newest field. It refuses a source with no certs, and
refuses a non-empty destination unless told, because merging two identity stores is a
decision rather than a default.
Rehearsed against a verified 96MB snapshot of prod on a Linux box (the cek codec needs
RAM-backed scratch, so not on macOS): 5745 rows across 15 kinds written and verified
kind-by-kind, certs=10, and the destination reads back as ciphertext rather than
"SQLite format 3".
The :9464 Prometheus exposition existed for exactly one caller: a scraper that
filled VictoriaMetrics, because VictoriaMetrics was where every reader looked.
The readers moved to the telemetry store and the measurements now push
themselves there, so what is left is a listener whose only client no longer
exists. That is not a way out, it is an open port.
The registry STAYS. It stops being a published surface and becomes the buffer
the meter provider renders into and the push drains — which is why the otelprom
reader is untouched while its HTTP face is deleted. The exporter was never the
scrape; it is the encoder, and it is the one that already speaks the family
model the datastore receiver takes.
cloud.Metrics() goes with the listener it served. MetricGatherer stays and is
now the only door onto the registry: read what was measured, never register.
Note the :9090 ops /metrics is untouched — it is a hardcoded `cloud_up 1` stub
for liveness, not the registry, and it answers a different question.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The lifecycle defense: an edge traffic sensor, a per-credential abuse gate and
the identity boundary's own attestation. Compiled in and mounted, SHADOW per
org — edge.Store.Mode returns live only on an exact "live" match, is absent
from the inherited base, and is never set here, so nothing is armed by landing
it. /v1/risk stays unrouted; /v1/ml is untouched.
Four conflicts, each resolved on its merits rather than by side:
middleware_identity.go — an import collision where both sides were right.
main added namespace.Sanitize (OrgHasUnsafeRune, the cross-org fold refusal),
the branch added principal.Mint. Both symbols are live in the merged body, so
both imports stay; either blanket resolution drops a defense and the build.
middleware_ratelimit.go — the branch's code, main's reasoning. The branch
tests the ROUTER's path (RoutePath + underPrefix) instead of the raw spelling,
which is what stops /v1/billing/../v1/ai/chat from prefix-matching its way
into a rate-limit exemption. Its rationale, though, describes an in-process
HTTP self-dispatch that main has since replaced with a typed ZAP op, so the
comment kept is main's, which is the one that is true here.
plugin/o11y/main.go — complementary, not competing. The branch installs the
identity boundary and the abuse gate in this hand-written main; main added
cloud.ErrorHandler so a propagated refusal renders as its own status instead
of 500. Those two middlewares are precisely what emits refusals, so the
handler matters more after the branch lands, not less. Both kept.
openapi/floor.json — the pointwise maximum, which is what Floor.Raise means:
taking the lower side would silently undo the other's ratchet. Then
regenerated from source, so the regeneration is the truth.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two duplications, both of the same shape: cloud held a second copy of a fact
that another service owns.
PKCE (iam v1.34.5)
------------------
apps/deploy/login.go had pkceChallenge, whose own comment said it was
"byte-identical to IAM's own pkceChallenge", and apps/integrations had
twitterChallenge -- the same two statements again under a third name. iam is
the authorization server; it decides what a code_challenge is. It just moved
that derivation out of internal/oidc into pkg/pkce so a client can import it,
so both copies are deleted and both call sites use pkce.Challenge. The
iam-facing one also sends pkce.Method rather than its own "S256" literal, so
this client cannot ask for a method that server refuses. Twitter's call keeps
its own literal: that is X's protocol requirement, not iam's.
twitterVerifier stays exactly as it was -- it is a different thing (a
per-app-constant verifier, with the reasoning for why that is safe only for a
confidential client), and only the challenge transform was duplicated.
MRR (commerce v1.49.46)
-----------------------
apps/admin/commerce carried monthlyNormalized, a third copy of commerce's
interval normalization. It agreed with commerce's arithmetic and still got the
answer wrong, because it never read quantity at all: a 10-seat plan at
$20/seat reported $20 here and $200 in commerce's own rollup. commerce
invoices Price x quantity, so $20 was never the revenue -- it was the unit
price wearing the name MRR.
commerce now puts mrrCents on each subscription in /v1/billing/subscriptions,
computed by the one definition that also feeds its rollup and its event
stream. This is a display surface, so it reads that number. Price and Interval
are dropped from subscriptionsWire entirely -- not left in place unused, which
would invite the next reader to normalize them again.
Tests
-----
- deploy TestPKCEChallenge: deleted. It pinned the RFC 7636 Appendix B vector
against the local function that is gone; the vector is pinned in iam's
pkg/pkce, which also now checks the encoding is unpadded base64url. The
authorize-redirect test still proves the published challenge derives from
the stored verifier -- that part is this package's own and is kept.
- integrations: the "challenge must be the HASH of the verifier" assertion is
kept verbatim, repointed at pkce.Challenge.
- admin/commerce TestMonthlyNormalizedCents: deleted with the function it
pinned; that arithmetic is pinned in commerce's api/billing. Replaced with
five tests over a stub commerce that prove Plan SUMS mrrCents, ignores
canceled subscriptions, returns an honest pay-as-you-go zero, decodes the
mrrCents tag, and -- the case the old code failed -- reports 20000 for a
subscription whose price says 2000.
- apps/admin cockpit/finance stubs now serve mrrCents, because that is what
commerce serves. Their expected MRR figures are unchanged; the stubs were
simply a wire shape behind.
Failing-test set unchanged: the same 7 packages (apps/ai, apps/iam, apps/plan,
apps/pricing, cmd/cloud, manifest, openapi) fail on origin/main before this
change and after it. cek stays at v0.2.1 deliberately -- v0.2.2 adds a
sidecar.go built against hanzoai/sqlite v0.4.0 APIs and does not compile
against the v0.5.0 this module already pins.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The measurements were never the problem — the prober has recorded
hanzo_service_up for as long as it has existed. The problem was that the only
way to READ it was a PromQL instant query, so /v1/summary (PUBLIC,
unauthenticated) and /v1/o11y/status each held a VictoriaMetrics client, and VM
could not be removed without both going dark.
latestGauge answers one question — the newest value of a gauge, per label set —
against event.metric joined to event.series on the fingerprint, because a sample
carries no labels and a series carries no values. argMax over the window, not an
average: a service down for four of the last five minutes and up now must read
1, and any aggregate answers 0.8, a number true of nothing. Asking it in ONE
place is what stops the two callers growing two dialects of the same SQL.
⚠️ status.go loses per-replica identity, and that is a real loss rather than one
to paper over. It read `up{service=…}` — the SCRAPE's own metric, one series per
target, so instance/pod came free because something had visited each pod to
produce it. Nothing scrapes anything now. hanzo_service_up is recorded per
SERVICE: the prober asks a Service address whether the service answered, and a
Service address is not a replica. So the inventory is one row per service, keyed
by what was actually measured. Synthesising a pod name would invent a fact no
measurement supports. Wanting per-replica health back means MEASURING it — the
prober resolving endpoints and probing each — not deriving it.
Source says "datastore" now because it is. The dead instant-query decoder and
its envelope go with the caller that needed them; vmproxy.go is the last VM
holdout and is SuperAdmin-only.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
APIKeyPrefixes drops hk-, so a string with that prefix no longer reaches
isAPIKey and never travels to IAM's get-user door. It resolves to no
principal, exactly as any other unrecognized bearer does.
pk- is unchanged: it is still recognized, still short-circuited to nil in
validatedPrincipal, and still resolves through resolve-key to an org and
nothing else. Nothing here lets it authenticate.
The analytics free-text scrubber loses its hk alternative for the same
reason — the prefix names no credential, and the bearer/entropy/query-param
branches still cover every real one.
Fixtures that used an hk- value as a working key are re-cast to sk-;
enumerations of the family throughout the comments now read pk-/sk-. Those
enumerations also carried fw_ and hz_, which isAPIKey has not recognized
since they were dropped as never-minted entries, so they go too — a comment
that claims to describe isAPIKey should describe it.
plugin/dns/openapi.json is regenerated from apps/dns; openapi.yaml takes the
identical string. The remaining hk- in openapi.yaml and plugin/{iam,ai} is
projected from hanzoai/iam and hanzoai/ai and has to be ripped there.
Prometheus is gone. Metrics were the last signal still leaving this process by
being PULLED: a exposition on :9464 that something outside had to visit. That
was never a statement about metrics, it was a statement about where they were
KEPT — VictoriaMetrics, because every reader queried VictoriaMetrics. With one
telemetry store left, traces and logs already reach it in-process over ZAP and
metrics had no way there at all. cloud was the only service in the fleet whose
own measurements existed nowhere once the scrape stopped.
So metrics travel like their two siblings: gather the registry on a timer and
hand the batch straight to datastoremetrics.WriteMetrics, in-process. The
receiver on :4319 is for OTHER processes; this process owns the writer, so it
calls it rather than dialing its own socket to reach a function already in
scope — the same reasoning that makes traces in-process by default.
The registry stops being a published surface and becomes an internal buffer.
That is why this gathers Prometheus families instead of writing a second OTel
exporter: zapmetricreceiver.MetricBatch IS the family model — name, help, type,
labels, value, buckets, quantiles — and the meter provider already renders into
exactly that. One encoder between two spellings of one structure; a second one
would drift from it. MetricGatherer hands out a Gatherer, not the *Registry, so
a reader cannot become a registrant.
Tests drive the whole path: install the provider, record through the ordinary
OTel API, push, and assert what arrives — including that a ZERO gauge survives
the trip with its labels, because hanzo_service_up == 0 is a real negative
answer and a translation that dropped it would turn a down service into a
missing one. Every family shape is pinned, a failing store costs one skipped
push, and shutdown is idempotent.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
typed_wire_test.go asserted a flat 200 for an unparseable body. That was the old
contract — the receiver answered 200 to everything — so with the status code now
describing DELIVERY it read 503 (no egress configured under test) and failed CI.
The fact the test actually exists to pin is that a malformed payload is RECORDED
rather than REFUSED: a 4xx would make Alertmanager retry it forever. That is now
asserted directly, with an egress installed so delivery is not the variable
under test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v0.5.0 deleted the key-derivation and DEK-wrapping API (DeriveKey,
DeriveChildKey, NewDEK, WrapDEK, UnwrapDEK, PrincipalType, PrincipalAAD,
WithPrincipalKey). cloud stopped calling it in 4cb56f6b; commerce and
tasks were the two things left in the build graph that still did, and
both have now moved to cek + namespace:
commerce v1.49.45 per-org store keyed by cek.DeriveKey, placed by
namespace.Path; DEK sidecar, rewrap and the
plaintext->encrypted migration deleted
tasks v1.52.9 shards opened by cek.Open under the shared
orgs/<org>[/projects/<p>]/<ns>.db layout; dek.go
and tasksd upgrade deleted
cek goes v0.2.2 -> v0.2.1, which is a REQUIREMENT of v0.5.0, not a
preference: v0.2.2 added sidecar.go to read legacy wrapped-DEK databases,
and it is written against the very symbols v0.5.0 removed, so v0.2.2 does
not compile against v0.5.0 at all. Its exported API is identical to
v0.2.1 — the delta is that legacy read path plus its tests — and reading
databases written under the old scheme is the compatibility this
migration exists to remove.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.5.49 carried the request-target fix in ONE of five relay bodies. The other
four — infra, metrics, rules/alerts, integrations — still handed the embedded
runtime a request with RequestURI empty, so 94 of 367 routes answered 404 in this
binary while the standalone served them 200. That is the configuration cloud runs
in production, so the whole of infra, metrics, rules/alerts, integrations,
cloud_integrations, downtime_schedules, route_policies, metric_reduction_rules
and gateway ingestion_keys was unreachable through the unified door.
v1.5.50 collapses the four copies into the one relay and adds the reachability
census that measures it — registered and reachable are now the same set.
It also stops a restart from locking every principal out of o11y (localauthz now
rehydrates its tuples from the durable role rows), and stops a schedule-less
maintenance window from panicking inside encoding/json — a panic that in this
binary would take iam, commerce and ai down with it, and that only became
reachable now that the downtime routes are.
plugin/o11y is unchanged by the bump, verified by regenerating: the fix is in how
a relayed request is shaped, not in which routes exist.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A control that switches itself off has to be visible from OUTSIDE the process,
and "visible" has to mean a test can drive a tenant into the state and read it
back. The probe published `strained` and nothing held it: a constant zero would
have passed every assertion here.
Now one organisation is pushed past its own aggregate ceiling — where it starts
forgetting its least-recently-active subjects, and every velocity feature for
those subjects reads as inactive, which is exactly what a quiet subject looks
like — and the probe is read back for a non-zero count.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A figure in a comment is a claim. One scope at both its ceilings is 4.0 MiB of
the 128 MiB budget — 3% — not the vague fraction that was written there.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
resolve could spend the whole process on one authenticated request. Two
amplifiers composed. A key had a count bound and no BYTE bound, and the domain
matcher split a host into labels and re-joined every tail, so an L-label host
allocated a copy of each of its L suffixes: one 8 KB dotted key materialised
16.8 MB and 100 of them 1.7 GB, measured over the router. And `sets` had no
bound and no dedupe, so naming one set N times ran N times the answers.
- maxKey bounds one key in bytes at every door it crosses — looked up,
written, removed — and REFUSES rather than truncating, because a shortened
key is a different key.
- a domain suffix is now a slice of the host, not a join of its labels: the
same answers in O(L) headers over one backing array.
- a call may name each published set once, and a set named twice is consulted
once.
The override write had the same hole from the other side: maxOverrides bounded
rows and nothing bounded a row, so 10,000 entries x 11 sets was gigabytes of
attacker-chosen bytes on the one volume every other organisation's store lives
on. The same maxKey closes it; an over-long note is refused rather than trimmed;
and what one organisation may occupy on that volume is now a figure the code
computes and a test pins, so raising rows, a key, a note or the catalog is an act
with its consequence next to it.
An override is a record, so it is now shipped before it is acknowledged, the way
apps/research and apps/books do it — this deployment is one replica with a
recreate rollout, and an unshipped write is a control an operator believes is in
force and is not.
prune spared ONE version: the call site passed the current version for both of
the statement's two placeholders, deleting the rows behind every citation taken
in the window before a refresh. What a take supersedes is now decided by
sweepOld over what the plane held before it, and proved against a warehouse
rather than against the text of the statement.
The publisher's end of the same amplifier is closed with the same door. A take
is refused whole if it carries a member longer than maxKey — one no lookup could
ever reach, so it is only weight in the warehouse, in every hydrate and in the
snapshot every request reads — or more members than a published set holds: swing
measures GROWTH against the version a take replaces, and a first take has nothing
to measure against, which after a cold start is every take. maxBody comes down to
six times the largest source in the catalog (measured: 2.6 MB), because the parse
allocates before any later gate can look at what it made. A publisher's redirect
must keep the two properties its origin already had, TLS and a destination
outside this network: this process runs in the cluster, where "wherever the
publisher says" reaches the pod network and the metadata address.
Also: a take whose size swings past 4x is refused and the previous version
stands (force is the operator lever); the disposable list is refused whole if it
names a mailbox provider, which is the one-row attack the size gate cannot see
on the one unpinned source; an attest receipt with no version or no designations
is recorded as a refusal instead of a current, fresh list; the plane sweeps at
cold start instead of refusing every set for six hours after a deploy; the one
cross-fleet aggregation states its own memory and time budget; every source
states a typed redistribution Basis from a closed vocabulary, on the wire, so
the licence position is an audit rather than a sentence; refresh reads the ONE
SuperAdmin predicate rather than restating it, and an admin of their own org is
refused there, because this route writes the baseline every org reads; and the
bridge sits on this app's own leaf, not on the /v1/ml parent two other products
answer under.
65 tests green under -race (one skipped: it dials the real publishers). Every fix
has a regression test that fails when that fix alone is reverted: 24 mutants, 24
killed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/ml/reference publishes ten sets — disposable email domains, hosting and
Tor address ranges, crawler user-agent patterns, delegated autonomous system
numbers, card-scheme prefixes, browsers the fleet sees everywhere, and the
freshness of the designation lists the screening engine holds. Six typed ops.
The unit of version and freshness is the SOURCE, not the set, so one publisher's
outage neither blocks the others' updates nor silently shrinks the set. A
version IS the content digest of the sorted entries, which makes a re-take of an
unchanged publisher a no-op that says so, and a half-landed version resumable
from its cursor without depending on it — the primary key already deduplicates.
Two planes, two stores. The baseline tables carry no tenant column, so a
cross-tenant write is unrepresentable rather than refused; a tenant's own
allow and deny entries live in that organisation's own store. Resolution is
override then baseline, both through one candidate function.
The baseline carries only published data under terms we hold — every source
states its licence — and aggregates above a k-anonymity floor no single
organisation can reach. Sources we may not redistribute are declared seams that
refuse, because an absent set and an unlicensed one look identical from outside.
A set that never loaded refuses rather than answering "not listed", and a stale
set answers and says so: every answer carries the version, its as-of, its age
and whether it is past the bound.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Mutation testing found three assertions that survive the defect they name, which
means they were not holding anything:
THE FEATURES GATE was proved only by its METER. Pricing the gate at zero screens
left every assertion green — an op that bills the right number while admitting a
caller with no balance, which is the control off with the books looking perfect.
It is now proved as a bracket, the shape the search gate already used: a cent
under the measured cost refuses, exactly the cost admits.
THE SEARCH METER was proved on a run that RAN TO COMPLETION, where "what it was
admitted for" and "what it did" are the same number and no assertion can tell
them apart. It is now proved on a run a rollout cuts short — the case that
matters, since this binary deploys at one replica with the old pod stopped first,
and the case where metering at accept charges in full for a grid that never ran.
The arena check that needs a completed run is its own test now.
THE FOLD MARK's second step was invisible to every behavioural test, because
replayable applies the window a SECOND time in memory where the nanosecond still
exists, so the re-read row is dropped downstream and nothing upstream can see it.
It is asserted at the seam where it is real instead: a mark must still exclude
the bucket it names AFTER a round trip through tsLiteral, which is second-grained.
Also: the snapshot interval had no test at all — only the count watermark did.
sweep takes its schedule as a parameter, so newPlane states it at the call site
and the loop is measurable without a thirty-second wait. A model that learns a
little and then goes quiet never reaches the watermark, and that is the case the
interval exists for.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The scan is under the lock and the sort is not, so the two numbers that matter
are the report at a full table and an observation while it runs. Measured
rather than argued: 844us for the whole report over 4,096 callers, 313ns to
observe one request.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three findings, one root cause: the sensor and the enforcement were keyed on
the raw Authorization value, validated or not — a string the caller picks. A
held verdict was walked out of five times in five by editing that header; the
lane with no tenant could be filled on demand; and a caller could open one
table entry per request.
A CALLER IS A FACT WE STATED. Signal now carries two fingerprints of the same
credential under different trust: Cred, set only when the identity boundary
VALIDATED it, which is the only thing that can key a caller; and Presented,
whatever the request carried, counted only as spread because a wall of invalid
credentials from one address IS the stuffing signature. callerKey cannot see
the second one. Both are built in ONE place (observation), pinned by a test.
The scorer is asked about the same thing the sensor holds, so a refused caller
cannot be re-judged as somebody else by editing a header.
NOTHING IS REMOVED TO MAKE ROOM FOR SOMETHING ELSE. The reclaim pass that
dropped the oldest half is deleted, not guarded: a table reclaims keys that are
DEAD (unseen for a window, under no live verdict) and REFUSES what does not
fit. So a flood cannot erase a neighbour's counts, cannot release a held
verdict, and cannot overrun the ceiling — the previous rule skipped pinned keys
and then admitted anyway, which is how 25,000 held callers lived in a table
that published 20,000.
THE BOUND IS IN BYTES. A cap on the NUMBER of keys is not a bound when the
values behind them are not bounded, so every string that can enter an entry is
clamped at the door, every entry has a published worst-case size, and every
admission charges it against ONE process budget. Count x size IS the byte
bound; a test fills a table with worst-case values, measures it, and fails if
the published numbers understate it. The per-scope ceilings stay as the
FAIRNESS bound, so the anonymous lane cannot take the room the tenants need.
A BOUND THAT BINDS SAYS SO. Strain is graded — clear, full, refuse, blind —
carried on the observation so an unmeasured caller is not screened as a brand
new one, reported on the scope's own view, and announced once per rise rather
than once per request. The lane that has no tenant is readable at last: it is
named by the empty scope (?org=), which cannot collide with any tenant.
WHAT THE SENSOR CANNOT SEE IT DOES NOT INVENT. A request with no validated
credential and no client address has no identity; keying it under the empty
address would file the whole internet in one row, read as the worst stuffing
run ever recorded, and let one verdict refuse everybody. It is counted as
traffic, named blind, and nothing is held against it. This is the live shape
today: the balancer in front of the ingress is TCP with no PROXY protocol, so
no client address reaches this process at all.
THE LANE IS DERIVED FROM THE COUNTS, so it is computed inside the observation
that produced them. Passing it in meant stating it before those counts existed,
and every request ever counted landed in "unknown".
ABSENCE IS NOT SILENCE. The fail policy turns on two facts, the same rule
hanzoai/iam applies at its own gate: a privileged grant waits for a scorer that
is THERE and did not answer, and proceeds when there is no scorer at all — or
when every slot is held and nothing has come back for a stall, which is a
deadlocked scorer, not a queue. Without that second fact one hung goroutine
403s every armed org's key store until the pod restarts.
AN UNANSWERED SCREEN IS NOT A SALE. Only a scored verdict is billed; the count
is split into answered and unanswered so a scorer that has gone dark is a
number on the org's own report. A fact we do not have is ABSENT from the
signals, never empty — an empty string is a value a scorer can group by.
A SCOPE KEY NAMES THE ROUTE. canonicalService and the rate limiter's
exemptions read the router's path, not the spelling: "/V1/AI/chat" reached the
"/v1/ai/chat" handler while producing a different rate bucket and a different
spend-cap axis.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/ml/labels is the answer key the model plane cannot build for itself: what
turned out to be fraud, who said so, and when they could first have said it.
Chargeoff, dispute, case, refund, review and the below-the-line sample all file
into one append-only record per tenant — its own encrypted SQLite file, no org
column, so a cross-tenant read is not forbidden but inexpressible — with a
derived ClickHouse copy for joining at training scale.
Seven typed zip ops, so the REST route, the OpenAPI operation, the MCP tool, the
CLI command and every SDK method are projections of one declaration: mlLabel,
mlLabels, mlResolveLabels, mlLabelCoverage, mlLabelVocabulary, mlDisposeLabels,
mlHoldLabels.
THREE PROPERTIES, EACH THE POINT OF THE PLANE, NONE OF WHICH THE FIRST CUT HAD.
DURABILITY. Nothing in the package called OrgStore.Sync, so the only ship was
CloseAll on a graceful shutdown. cloud deploys strategy Recreate at one replica:
an ungraceful termination lost every acknowledged record since process start,
and the successor hydrated the older durable snapshot OVER the local file — an
acknowledged compliance record was not merely at risk, it was overwritten by an
older copy of the tenant's own history. state.ship is now the ship-before-ack
step every write path calls before it answers, and an unacked ship fails the
request rather than acknowledging a divergent local copy. That covers BOTH
shapes: a replica that never held the lease (ErrNotOwner) and one deposed
between the write and the ship, whose fenced Put is refused at a stale round
with no error at all. The two sibling durable planes hold the same contract
(apps/research shipFor, apps/books shipLedger).
DELIVERY. The cursor was the pair (wrote, id) over a write clock truncated to
the second and a content digest — an order the writer never took. A write that
commits after a concurrent delivery has read, whose digest sorts lower inside
the same second, was already behind the mark: never mirrored, unreachable by any
retry, and pending() answered zero because it asked the same predicate. A hole
in the answer key reads as an honest customer. The cursor is now the store's own
AUTOINCREMENT position, allocated inside the insert on the single connection
every statement for a tenant runs on (sqlpool.Single), so cursor order is commit
order by construction.
LEAKAGE. `seen` is whatever the caller sent, bounded only by At <= Seen <=
now+skew, and nothing tied it to any fact the server observed — a dispute filed
today with seen == at was knowable a year before the record existed, and a
backtest standing two days after the event resolved it. Fact.Knowable is derived
server-side as the later of `seen` and the clock at the write, it is the only
time visible() and stronger() read, and it is a column in the derived copy so
the warehouse applies the same predicate the record plane does. A live pipeline
is unaffected: Wrote is within minutes of Seen and the derivation changes
nothing.
Also: the coverage window now ends where maturity begins, so the gate on
training no longer answers zero on its own defaults; Group returns the whole
matured cohort so `matured` counts what matured and `unlabelled` says why
`judged` is low; a litigation hold is a fact about the record with its own op
that can also release one, instead of a digest-excluded flag silently dropped on
any record that already existed; the warehouse partitions by month like every
other table in the fleet rather than by tenant, whose cardinality is the
customer count; and resolve answers one row per distinct event, so an event
named twice is no longer its own conflict.
Every one of those carries a mutant in scripts/mutate.py that reverts it and a
test that goes RED when it does: 14 rows, 14 killed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An org name is unique within an issuer and not across issuers: acme on
hanzo.id and acme on zoo.ngo are two unrelated businesses. A per-org SQLite
file never confuses them, because DataDir is per deployment and the brand is
the directory the file is in. A COLUMN in the shared columnar warehouse has no
such directory — org = 'acme' there is a predicate over both — so every row a
brand-shared table carries has to be keyed on <brand>/<org> and every read of
it has to bind the same form.
Qualify joins SanitizeOrg and OrgNamespace as the third org-naming door and the
only one for a shared plane. Qualified is derived from Qualify rather than
restated, so a change to the shape cannot leave a validator behind. Tenant
carries no JSON tags and has no constructor but Qualify, so no In struct can
decode one and no caller can assert a tenant for itself.
The brand half comes from Deps.Brand, never a header: a caller that can choose
its brand has chosen which tenant space its org lands in.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/o11y calls o11y.Mount(a) unconditionally, which registers the module's 353
typed operations. plugin/o11y's committed projections were last written the day
BEFORE that call landed, and nothing regenerated them: mcp.json held 12 tools and
openapi.json 20 paths / 32 operations.
So the fleet MCP door at POST /v1/mcp offered agents twelve o11y tools — the
reviews queue and one summary — while the process behind it served three hundred
and sixty-five. The observability surface was absent from /v1/openapi.json and
from every SDK generated off it. Regenerating with the app's own `describe`
target gives 365 tools and 289 paths / 388 operations, and takes the fleet total
from 932 to 1285.
apps/o11y/zipdoc_gen.go was stale in the same window: it still carried
POST /obs/event/claim, a route that no longer exists, and the pre-rewrite prose
for POST /v1/o11y/alerts/:receiver, which now states that delivery is
synchronous. Both come back from source.
This is what `make -f mk/fleet.mk surface-check` exists to catch, and it did not,
because the gate has been failing earlier in its own loop on unrelated apps — a
red gate stops reporting anything new. Recorded rather than papered over: the
regeneration is verified idempotent (a second run reproduces it byte for byte),
and the remaining blockers are in other apps.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The daily recompute wrote hanzo.risk_baseline whether or not anything read it,
and nothing did — a warehouse cost paid every day and collected on never, which
is the same "declared but unwired" defect as a reader with no writer, inverted.
The catalogue gains a NETWORK lens beside its model and surface lenses: the
published bands over the same window, carrying no tenant and the same for every
caller, so an organisation's own surface has something to be read against. One
table, one reader, so the disclosure argument has one place to hold.
Two lenses can fail in one response — an unreachable warehouse fails both — so
the gap accumulates instead of reporting one outage as the other's.
The recording warehouse files baseline rows under the TABLE and not under a
tenant, because that table has no tenant to file them under. Keying them by the
bound org meant keying them by the window's start, which a test had to guess to
the second in order to see its own fixture.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
30497e64 capped the buildkitd volume because it is the one that grew. The
artifact job's /w and the smoke job's /data were left as `emptyDir: {}`, and
that is the same defect with a different volume name — which is why this states
the rule as a table rather than adding two more one-off assertions.
An uncapped emptyDir is charged to the NODE's ephemeral storage. It therefore
does not fail the pod that fills it: it fills the runner's rootfs, trips
DiskPressure, and the kubelet answers by evicting the pod's NEIGHBOURS. A job
can stay inside every resource limit it declares and still take down every other
job on the node, so the bound has to be on the VOLUME and not only on the
container — three of eight runners under DiskPressure, 70 finished build Jobs
still holding their pods, and a release Evicted mid-flight for "node was low on
resource: ephemeral-storage".
/w is not scratch: it is the clone, HOME, GOPATH, the npm cache and every
produced binary for every platform, for every recipe entry in turn, in one
volume — the same order of magnitude as the build cache, so the same half-a-node
bound. /data is genuinely small (one 120s boot test against a fresh SQLite) and
is capped anyway, because the failure it prevents is an image that loops writing
on boot, and that one is the neighbours' problem rather than its own.
The three consts sit together: "how much node disk may a job take" is one
question with three answers, not three unrelated numbers, and a reader who finds
one now finds all of them. TestBuildCacheIsCapped is folded into the table — it
was exactly the table's `build` row, and a rule with two gates is a rule that can
be half-updated. Its incident evidence moves into the table's doc. Proven by
uncapping /w: the artifact row fails with the volume, the wanted bound and what
it actually found.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both were lost the same way — a conflict resolved by taking one side whole,
where the correct answer was on both sides.
THE PLANE SINK STOPPED COUNTING. 30497e64's predecessor put
cloud.ObserveRows at datastoreSink.Insert, after Send, because that is the one
choke point every plane row this process writes passes through, and because
"event.span went to zero" is the sentence nobody could say for four and a half
months. fix/o11y-convergence deleted event_ingest.go — correctly; it was a
writer that had never once reached a table — and carried datastoreSink forward
into planesink.go from a branch point 32 commits behind, so the counter went
with the dead file. The merge was a modify/delete, git kept the deletion, and
nothing failed: metrics_plane.go still seeds event.span at zero, so the series
exists and reads zero, which is indistinguishable from the outage it was added
to detect. Restored at the same place, after Send, on the surviving sink.
THE NEWEST PLANE TYPE WAS OUTSIDE THE GATE. bf48977a dropped ObsClaimIn and
ObsClaimed from TestNoPlaneTypeCarriesAnUnencodableKind because the op was
retired, which is right, but fix/upstream-status-and-spend-alerts-rpc had added
plane.ScopeRules in the same window and it was never added to the walk. That
test exists because the failure is a property of the KIND — the next map added
to any of these types is another 24h door-shut-while-green — so a plane type the
walk does not reach is the one place the next one will appear. ScopeRules is
walked rather than ScopeRule: the walk descends a slice of structs, and the
reply is the only way the row type ever crosses.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The merge brought two implementations of the same rule together. main landed
mcpAlias (34554a19) five hours after feat/mcp-one-door was opened, both answering
the bare /mcp with a 308 onto /v1/mcp. Keeping both is not belt-and-braces: the
route wins over the terminal handler, so webui/mcp.go's FrameworkMCPPath branch
becomes unreachable code that its own tests still exercise in isolation — a rule
that passes its test and never runs.
mcpAlias goes, and the terminal handler stays, because only one of them is
self-scoping. app.All("/mcp") claims the path unconditionally, so the same call
inside cloud.Serve would hijack a PLUGIN's own zip door — a plugin serves MCP at
exactly FrameworkMCPPath, and an alias there redirects the door onto itself.
webui's handler is terminal: it sees only paths no route claimed, so the plugin
matches its real route and never reaches it, and the host, which moved its door
to MCPPath, does. Same behaviour on the host, no footgun on the child.
It is also the superset. mcpAlias answered /mcp only; mcpDoor also answers a
non-POST /v1/mcp with 405 + Allow: POST (Streamable HTTP's "no SSE stream" —
where a 404 says the door is absent) and gives both replies a JSON body, so a
caller reading bytes instead of following the hop still never gets HTML. And the
address is now one value, manifest.MCPPath, where main still had it as a literal
in cmd/cloud and a second literal in manifest/mcp_test.go.
main's test survives intact and gets stronger: app308 no longer calls a helper,
so it now proves the COMPOSED host redirects. Neutering mcpDoor reproduces the
original defect exactly — POST /mcp = 405, GET /mcp = 200 text/html, 3576 bytes,
the same byte count 34554a19 recorded from production.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
(cherry picked from commit c4c3f40986725d1e99b6050c09de2aa8dfb63d48)
The kind-walk gained ObsClaimIn/ObsClaimed when it was written; the
convergence lane then retired obs_event_claim along with the dead
LLM-obs write path it fed, so the walk referenced types that no longer
exist. Dropped from the list and the header note rewritten to say why:
the property this test guards is a property of the KIND, which is
exactly why it walks kinds instead of a list of types, and it outlives
any individual op.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The floor is a monotone history of what the surface published, per product; the
merge left it at main's totals while the woven document carries risk's ten. The
ratchet raises it from the regenerated document rather than from either side's
guess.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The rename to Listen reached every caller but this one, and the release has not
built since: the image builds each plugin main as its own binary, so it stopped at
plugin/tools/main.go:20:18: undefined: cloud.Serve
The call shape was already right — only the verb was left behind.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The learning ops move off /v1/ml, which a live product already owns: the
Kubernetes model-SERVING plane (InferenceServices, /v1/ml/models/{name}/predict)
is on it and has customers. /v1/ml/models would otherwise have meant "models you
serve" and "models that learn" at once. Nine typed ops move to /v1/risk/*, the
serving routes are untouched, and there is no alias — nothing shipped under the
old prefix to be compatible with.
A CAP ON A COUNT OF CALLER-SIZED VALUES IS NOT A BOUND. Every per-tenant ceiling
here was a COUNT multiplied by something a caller chose, so the published figures
understated the real cost by whatever the caller picked. maxField bounds, in
bytes, every string an observation carries; the ceilings are derived from it; and
the published figures are now MEASURED against a real worst case rather than
recomputed from the same formula.
A partly-applied bucket is no longer a state the fold can be in. ctx is consulted
at bucket boundaries only, so an interrupted fold resumes exactly where it
stopped instead of re-teaching what it already applied. The watermark steps by a
whole second because that is the finest value tsLiteral can bind — a nanosecond
mark was truncated back onto the bucket it meant to exclude.
The fold marks only what the surface can HOLD. horizon() is one function: the
rollup stops rollLag behind the present, so the fold reads and marks there too.
It used to mark `now`, which claimed buckets the rollup had not written yet — the
next rollup wrote them and the fold never came back, so every cycle dropped that
organisation's most recent history while the model reported itself warm.
A gap re-arms. Only a fold that SUCCEEDED is a fold that happened, so a warehouse
blip on a tenant's first touch no longer costs it the moat for the life of its
residency.
The recording warehouse now honours the bound window. A fake that ignored it made
every window assertion in this package a stub, and fixtures derive their bucket
stamps from horizon() rather than from `now` — a row stamped `now` is a row that
could not exist.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The one-shot seed of hanzo.usage_rollup_daily could never run. EnsureUsageRollup
creates the incremental view on the FIRST leaderboard or activity read; the view
starts capturing on the next ledger insert; and the handler then asked "does the
rollup have any rows at all?" — which is true forever after. Every non-forced
POST /v1/usage/rollup/backfill answered 409, and the only way past it was
?force=true, which the code itself documents as double-counting. So pre-view
history was never laid down.
Measured on the live datastore: the rollup holds 530 of hanzo.cloud_usage's
19,792 requests (2.7%) and 567 of its 266,088 cost cents (0.2%) — everything
from 2026-07-01 to 2026-07-27 is missing, plus most of 07-28, because that is
when the view was created. Every leaderboard and activity read is served from
the rollup, so all of them under-report by ~97%. Nothing errored: an MV is an
insert trigger, and history it never saw produces no row and no complaint.
The guard was asking the wrong question. It must count only the days the seed
would WRITE — `WHERE day < toDate(?)` — not the whole table. Then a rollup that
the live view is already filling forward does not block a seed of the days
behind it, while re-seeding a day already covered still refuses.
The two ranges also have to be the same set of days to be comparable: the seed
selects the ledger by `timestamp`, the guard counts the rollup by `day`. A
mid-day cutoff seeds a PARTIAL day that the view may also hold, and once both
land in a SummingMergeTree no count can separate them. rollupCutoff snaps the
bound to UTC midnight — the rollup's own grain — in one place, used by the seed
and the guard alike.
The bound stays one-sided: this seeds "everything before the view existed", once.
Widening the cutoff later is a real re-seed of covered days and is refused;
?force=true remains the deliberate override.
Tests pin the regression directly — a rollup in exactly the state the live view
leaves it (rows from its creation day onward, nothing before) must accept a seed
of the earlier days, and the guard must be the range query. Reverting the guard
to the whole-table count fails it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
MCP clients are configured with a HOST and reach for the door at /mcp. Nothing
claimed that path, so it fell through to the console catch-all and answered
twice-wrong on api.hanzo.ai:
POST /mcp -> 405 method not allowed (the SPA route is GET-only)
GET /mcp -> 200 text/html, 3576 B (the console shell)
Both read like a server that is up, which is how this survived: the host
answers, the door never opens. The 404 on mcp.hanzo.ai is the milder half of
the same bug — a 404 at least says no.
308 and not 301/302: only the permanent-redirect pair preserves the method and
the body, and MCP is a JSON-RPC POST. A 302 would arrive at /v1/mcp as a
bodiless GET — the same dead end, one hop further along.
It is an ALIAS, not a second handler. /v1/mcp stays the only place MCP is
served, so the tool list, the auth path and the transport cannot drift between
two doors. Registered before webui.Mount so it wins the path, and extracted as
mcpAlias() alongside spec()/health() so it is reachable from a test.
The test is the defect: neutering mcpAlias reproduces 405 and the 3576-byte
shell — the same byte count TestAnAbsentPrefixBeatsTheConsoleCatchAll already
records for an unclaimed path. Its third case asserts the target is real, which
caught that zip installs no door at all when the registry is empty (installMCP),
so an alias could otherwise have pointed at nothing and still gone green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The buildkitd worker cache and snapshots are the whole of a build's working set
and the only thing in that pod that grows without bound. An emptyDir with no
sizeLimit is charged to the NODE's ephemeral storage, so one overrun fills the
node rootfs and trips DiskPressure — and the kubelet answers that by evicting
every pod on the node, not the build that caused it.
Three of eight runners sat under DiskPressure with 70 finished build Jobs still
holding their pods and emptyDirs, and a release was Evicted mid-flight for
"node was low on resource: ephemeral-storage". A cap turns that into this pod
overrunning and being retried.
The same fix exists in infra/k8s/hanzo-build/image-build-job.yaml, which is a
hand-applied template deliberately kept out of every kustomization — no build
this cluster runs comes from it. buildJobSpec is what launches them, so the cap
has to be here to be real.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Serve and Listen were two names for one act. zip's App already calls it Listen —
`app.Listen(zapAddr, httpAddr)` — and this function's whole job is to build that
app and hand it its addresses, so calling it Serve made the entry point disagree
with the thing it enters. One verb, all the way down: a plugin's main says
cloud.Listen, cloud says app.Listen, and nothing has to be translated in a
reader's head on the way through.
117 composition roots move with it. ServePlane is untouched — it names a
different act (bind one app's own socket for the internal plane), and collapsing
it into this would be the opposite of the point.
Also fixes apps/iam's TestMain, which had gone red on every store test:
credz.Boot's last resort is cek.EnsureDevKey, and that DECLINES on a codec-linked
build by design — a build that can really encrypt must be handed a real key, not
invent one. So the throwaway goes in through the same door a deployment uses, and
only when nothing else supplied one. Six failures back to the one pre-existing
ratchet (iam serves 97 untyped ops against a budget of 88).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cek v0.2.1 derives every key from the master and the namespace, which is the
right steady state and cannot read a single file this fleet already has. Every
one of them was born under the wrapped-DEK scheme: a DEK from crypto/rand,
wrapped beside the database in a .dek sidecar. Random key material is not
reproducible by derivation, so a release on v0.2.1 meets those pages the way
SQLCipher meets a wrong key —
page 1: sqlcipher: wrong key or corrupted page
— across 211 org directories and every subsystem in each: kms, agents, finance,
code, treasury, crm, the audit log. That is not a migration, it is the data
becoming unreadable at the moment of a deploy.
v0.2.2 reads the sidecar when there is one and derives when there is not, so the
file says which scheme it belongs to. It writes nothing, so this changes no byte
on disk and stays reversible; new stores are still born derived and sidecar-free,
and the set the compat path governs only shrinks.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The port landed in three pieces across two commits and this is the piece that
matters at runtime: cmd/reach/main.go was committed and openapi/reach.py was
deleted, but .hanzo/workflows/cicd.yml still said
run: python3 openapi/reach.py openapi.yaml ...
so the reach job would have failed on a file that is no longer in the tree. LLM.md
likewise still named the Python path.
Both were staged and verified staged twice, and twice the commit carried only the
Go file. Whatever dropped them, the lesson is the one already written down: read
the committed value back, do not trust that `git add` stuck.
committed cicd.yml before: python3 openapi/reach.py (script absent -> fail)
committed cicd.yml after: go run ./cmd/reach
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6965c345 removed openapi/reach.py but did not carry cmd/reach/main.go, the
workflow change or the LLM.md line with it: a multi-path `git add` did not stage
the new file, and I committed without reading back what was staged. main was left
with the gate script deleted and nothing calling its replacement.
This is the other three quarters of that change:
cmd/reach/main.go the port
.hanzo/workflows/cicd.yml go run ./cmd/reach, plus setup-go and the
private-module env the containment job uses
LLM.md the gate table names the Go tool now
Verified before and after: go build ./cmd/reach and go vet pass, and the tool
agrees with the deleted Python exactly against production — 872 addresses,
646 literal, 226 parameterised, 0 dark.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Search, Crawl, Vector, Console and Managed Services had their prices stated
nowhere but hanzo.ai's components. That is how the same product came to be
advertised twice at different prices: one tab priced Hanzo Vector at Starter $29 /
Growth $299, another at Free $0 / Pro $25 / Business $99, both live, one saying 1M
vectors cost $29/mo and the other saying 1M vectors was free.
plans v1.4.11 carries the rate cards (every number copied from what those pages
already advertise — nothing offered changes); pricing v1.4.7 serves them. This is
the route.
DISPLAY rate cards: no entitlement or limit fields, so nothing can bill off them.
What a product COSTS and what a plan GRANTS are different questions, and conflating
them is how a catalog row becomes a self-serve mint.
Spec regenerated the same way as datastore — zipdoc first (a route with no
generated doc registers but does NOT project, which is why the first describe
produced nothing), then the subset in a Linux userspace, then the weave. The weave
also picks up three /v1/o11y/* paths that the committed o11y subset already had and
the golden did not: pre-existing drift, corrected rather than carried. floor.json
1397->1399 paths, 1966->1968 operations.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud is a Go service: 2284 .go files, no Python in the shipped image. The ONLY
thing that wanted a Python runtime was this gate, and on Ubuntu 24.04 that turned
into a dependency the runner refuses to install:
error: externally-managed-environment (PEP 668)
- To install Python packages system-wide, try apt install python3-xyz
So a check about ROUTING kept failing over package management — first as a
JSONDecodeError from a json.load "fallback" that could never parse YAML, then as
a refused pip, then as an apt step existing only to feed one script. Each fix was
smaller than the question of why it was Python at all.
gopkg.in/yaml.v3 is ALREADY a direct dependency, so the Go version provisions
nothing: no pip, no apt, no runner-image dependency, and the toolchain is the one
the release already builds with.
Behaviour preserved exactly, including the load-bearing parts:
- wildcard keys dropped, not filled
- the literal/parameterised asymmetry in isDark: ANY 404 condemns a literal
address, while a parameterised one is dark only on the exact router-miss body
(the rule that caught an edge worker whose 404 carried JSON)
- checkInstrument() still runs at measure time, not in a test file nobody runs
- the ratchet may still only SHRINK, and a healed line still fails the release
Verified by running BOTH against production, same arguments, minutes apart:
python3 openapi/reach.py 872 addresses (646 literal, 226 parameterised), 0 dark
go run ./cmd/reach 872 addresses (646 literal, 226 parameterised), 0 dark
reach.py is deleted rather than kept alongside: two implementations of one gate
is the drift this gate exists to catch.
The reach job gains setup-go and the same private-module env the containment job
uses, since github.com/hanzoai/* must resolve direct+authenticated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Records what the door work established, so the next 4xx here is diagnosed by
SHAPE rather than re-guessed:
{"status":"error","msg":…} gate() itself (fall-through route)
{"status":403,"error":…} a typed op — relay re-wrapped it
{"status":404,"error":"404 page not found"} the runtime's web provider, no path
The three probes are the control: mountHealth dispatches them and never calls
relay, so "probes 200, everything else 404" is a lost path, never auth.
Names every exempt op and the argument for it — the rule being this gate's own
purpose read backwards: it exists because the runtime trusts X-Org-Id as
gateway-minted, so an op whose gate reads no tenant from the request has nothing
for a forged tenant to reach. Exemption is not authorization; each still faces
its own credential one layer in.
States the SPA question rather than leaving it open: /v1/o11y is the API and only
the API — the module names all 367 routes so an unconverted one 404s instead of
falling through a wildcard, so /v1/o11y/ is a 404 and every answer on the prefix
is JSON. The console is o11y-site at o11y.hanzo.ai / obs.hanzo.ai behind
admin-guard.
AND FLAGS SOMETHING THE DOOR WORK UNCOVERED BUT DID NOT FIX. The section headed
"the o11y pin is BLOCKED at v1.5.34 — do not bump it alone" listed three
in-handler forwards that a bump would break. The pin is now v1.5.49 and those
forwards were never updated: "/api/sessions" is registered nowhere in the pinned
module (the list is /v1/o11y/llm/sessions) and "/api/v3" survives only in
parser_test.go. sessions.go and query.go therefore name routes the runtime no
longer serves. Both are org-gated, so neither is reachable anonymously and
neither appeared in the door probes. They need their own pass — and the fix is
to forward to the current spellings, not to re-pin.
Also notes why editing r.URL.Path is not a redirect against the embed:
adaptor.FiberApp routes on RequestURI, so a handler that rewrites URL.Path and
leaves RequestURI alone is routed by the original path.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
api.hanzo.ai/v1/o11y/version answered 403 and, once the gate was fixed, 404.
Two independent defects stacked at one seam; the second was invisible while the
first refused the request before it could reach anything.
1. THE GATE (already fixed here by 89ae5449, shipped now). Its exempt list named
/v1/o11y/api/v1/health and three /api/v2 siblings — the namespace hanzoai/o11y
stopped rewriting onto at v1.5.37. Four names, zero routes, so every public op
was refused. The answer is o11y.Anonymous now: the fact lives beside the routes.
2. THE PATH (o11y v1.5.49, picked up here). relay handed the runtime a request
built by http.NewRequest — a CLIENT request, whose RequestURI is empty by
design. The embedded runtime is adaptor.FiberApp, which copies RequestURI into
fasthttp verbatim, so the path was erased and every typed op fell through to
the console web provider's http.NotFound. That is the
404 {"status":404,"error":"404 page not found"} the door answered on /version,
/health, /global/config and /users/me the moment the gate stopped refusing
them. Only livez/healthz/readyz worked, because mountHealth dispatches those
itself and never calls relay.
WHY NEITHER WAS CAUGHT. red_forge_test.go calls gate() directly against a
backend that answers 200 to anything — it proves the predicate and cannot see
the chain, and it passed throughout. door_test.go exercises the REAL MountO11y
route table against a runtime that routes: anonymous /version and /health must
return the RUNTIME's bytes, tenant reads must still be refused with the DOOR's
own reason (not the runtime's 401, which would mean the request got through),
and the four dead /api/v1|v2 names must NOT be exempt. Verified by reverting
each fix in turn: with the dead list restored it fails with the exact production
body, 403 {"status":403,"error":"no validated principal"}.
THE SPA QUESTION, ANSWERED: /v1/o11y is the API, and only the API. There is no
catch-all under it — hanzoai/o11y v1.5.48 named all 367 routes precisely so an
unconverted route 404s instead of silently falling through — so /v1/o11y/ is a
404 and every answer on the prefix is JSON. The console is o11y-site at
o11y.hanzo.ai / obs.hanzo.ai behind admin-guard, which 302s a browser to
hanzo.id PKCE and 401s a machine. One door per concern: api.hanzo.ai serves the
operations, the site serves the app.
Pre-existing and NOT from this change: TestUntypedRoutesKeepTheirWire fails on
pristine origin/main (unparseable alert receipt = 503, want 200) — the alert
lane's, untouched here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
My previous commit added `python3 -m pip install pyyaml`, which the runner
rejects:
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install python3-xyz
The image is Ubuntu 24.04, where a system-wide pip install is blocked by PEP 668.
The error names its own remedy, so this uses it: python3-yaml IS PyYAML, and
apt-get is already how this fleet installs system packages (hanzoai/engine does
the same, unsudoed — the job runs as root).
Also drops `--quiet`, which was a mistake of mine: it hid the reason the step
failed, so the log showed only "exitcode 1" with no message. The step now prints
the installed version, so "PyYAML is present" is evidence rather than assumption.
The rest of the pipeline was unaffected — gate, cicd, containment, image and
rollout all passed on the previous run; reach was the first failing job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reach job failed with
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
which reads as a corrupt spec. It is not. reach.py did
try: import yaml # "the forge runner has it"
except ImportError: doc = json.load(open(spec))
The runner does NOT have it, and that fallback could never work: the script is
always called on openapi.yaml, and json.load on YAML fails exactly like the above.
So a missing package presented as a broken document, and the check it guards has
not actually run.
PyYAML is now installed in the job and REQUIRED by the script, which exits saying
so instead of mis-parsing.
That makes the check run — and it immediately found real drift: 5 published
addresses that api.hanzo.ai does not route. Three were the o11y probes:
/v1/o11y/api/v2/healthz 404
/v1/o11y/healthz 403 <- routed, auth-gated
Our o11y fork moved off /api/v2; openapi.yaml still advertised the old shape, so
the document promised three addresses that answer 404. Renamed to the paths that
exist. (Also what the house rule says: /v1/, never an /api/ prefix.)
reach.py before 5 dark
reach.py after 2 dark
⚠️ Still dark, and NOT fixed here because both need a product decision rather
than a transcription fix — either the route should exist or the document should
stop publishing it:
GET /v1/ai/mcp/tools 404
GET /v1/pricing/datastore 404 (/v1/pricing itself is 200)
The ratchet file openapi/unreachable.txt is empty, i.e. the standing policy is
zero dark addresses, so neither was silently allowlisted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every org store is answering 500 right now:
cloud: OrgDB pragma "PRAGMA busy_timeout=5000": file is not a database
Measured across two INDEPENDENT subsystems — /v1/agents/sessions and
/v1/tracker/projects. Keys are derived per (namespace, subsystem), so two
subsystems failing identically is not two corrupt files; it is one wrong MASTER.
cek is careful about the absent case: DeriveKey returns ErrNoMaster when no
master is installed, so "unset" fails loudly. What it cannot distinguish is a
master that is present and WRONG — and this package can hand it one. Branch 3 of
resolve() mints a random master when nothing is configured, which is exactly
right for a laptop and catastrophic for a deployment that has lost its KMS: every
file was encrypted under the key being replaced, so each opens as "file is not a
database" while the data sits intact and unreadable.
It is the same mistake the branch above already refuses for a launched child —
"Inventing a second key here is worse than not starting, because it succeeds" —
and the reasoning does not depend on there being a token. It depends on whether
anything preceded this process.
So ask that, and ask it of the disk: resolve() already takes dataDir. A random
master is honest over an EMPTY directory and refused over one already holding
databases, with an error naming the remedy rather than 500s thirty frames later.
No notion of "production" is required, which is why this is checkable at all.
An unreadable subtree is reported, never skipped: the point is to refuse when we
cannot be SURE the directory is empty. A missing directory is not an error — it
is the clearest possible "nothing preceded this process".
This is a fail-closed guard, not a diagnosis of the live incident: whether the
running deployment took this branch needs its boot log (credz reports posture and
`bootFrom`), which needs cluster access this session does not have. The guard is
correct either way.
The five failing tests in this package fail identically without this change —
macOS has no socket peer-credential lookup ("peer credentials unavailable on this
platform"). The two added here pass.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
github.com/hanzoai/plans embeds blockchain.json, which /v1/pricing/blockchain
serves. v1.4.9 adds the Indexer API (hanzo.ai advertises it; the catalog did not
publish it). v1.4.10 restores the Wallet API tiers the page actually shows —
Free 0/5K, Growth $49/250K, Scale $249/2.5M. The catalog's row had been
byte-identical to the NFT one, which is what an uncorrected copy-paste looks like.
This lands BEFORE hanzo.ai is wired to the live catalog, deliberately: the live
read replaces the page's fallback, so shipping the site first would have repriced
the Wallet API downward on the public pricing page for as long as cloud lagged.
The same module also carries the Go/Dev/Pro/Max ladder, so /v1/pricing/subscriptions
stops disagreeing with /v1/billing/plans about what is sold.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The embedded IAM is not the identity authority. It mounts, serves routes and
answers OIDC discovery — and its store is EMPTY:
[iam] "seed: read init_data.json: no such file" -> "iam seed skipped"
[iam] "iam embedded in-process"
/var/lib/cloud/iam/global.db is 32KB of schema, and the cloud pod's own JWKS
answers with ZERO keys where iam.hanzo.svc answers with nine. Every org, user,
key and signing cert lives in the standalone IAM's PVC — a different file, in a
different pod.
So resolving a key against that store resolves nothing, in every process: the iam
child reads the empty store directly, and every other child asks the iam child
over the plane and gets the same nothing back. API-key authentication has been
broken since v1.801.359. The HTTP call this replaced went to iam.hanzo.svc, which
is where the keys actually are.
The direction was right and the premise was not. Reading identity in-process
requires the identity to BE in the process, and consolidating it is a DATA
migration — the standalone IAM's store into the embedded one — not a transport
change. Until that happens the resolver talks to the service that holds the data.
What stays, because it was never about the transport: the plane-topology lesson
(each app is its own composition root and calls cloud.Serve, so a package-global
seam is nil in every process but its owner), apps/iam's TestMain booting the
data-plane key the way the binary does, and the cloud.Request gate no longer
walking .claude/worktrees.
What I should have done: I flagged twice that I could not positively test a real
key and shipped anyway. A resolver change is not verified by the absence of 401s
in a low-traffic window.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`campaignWhere` wrote its own predicate — org, time bounds, `utm_campaign` — and
no `signal` at all. On five tables that was harmless, because naming
`event.event` selected product events by construction. On the ONE fact table the
signal is a PREDICATE, and a predicate can be forgotten: this one was, so
`uniqExact(distinct_id)` and `sum(revenue)` ranged over every log line, span and
error in the window as well as the acts.
It is latent only because nothing carries a `utm_campaign` yet. The day a
campaign ships it stops being latent, and the first number anyone sees is wrong.
So the campaign read composes `eventsWhere` instead of restating it. Which rows
and which sort is ONE decision and now has ONE spelling; the campaign and the
optional variant are what this read adds. Arg order follows the composition —
[org, signal, start, end], then campaign, then variant — and everything
user-derived is still bound, never interpolated.
The regression test pins the composition rather than the string: the built
predicate must have `eventsWhere`'s output as a prefix, with its args leading.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzo.ai's Infrastructure tab renders a Hanzo Datastore section that fetches
/v1/pricing/datastore. Cloud never registered that route, so it 404d and the
component fell back to "Live pricing is temporarily unavailable. Contact sales for
current rates." Permanently — which is the worst version of that message: it reads
as a passing outage, so nobody investigates, and it tells a visitor to wait for
something that was never coming.
The data existed the whole time. datastore.json sat in hanzoai/pricing unembedded
and unserved; pricing v1.4.6 embeds it and the goja bundle serves it whole (it is
a rate card, not a plans list, so it is not wrapped like iam/base/paas). This adds
the injection and the route.
Verified the payload against the CONSUMER's own validator (hanzo.ai
DatastorePricing isDatastorePricing): 3 tiers, usage.storage.pricePerGBMonth and
usage.egress.public_internet.pricePerGB numeric, included[] an array.
Spec regenerated, which is the part that is easy to skip: the weave gate does NOT
catch a route added without regenerating its subset — both artifacts are derived,
so they agree with each other while both are wrong, and plugin/ingress lost eight
paths that way with `make test` green. `make describe` cannot run on macOS at all
(hanzoai/sqlite's isRAMBacked is `return false` off Linux, so the encrypted
catalog store refuses to open), so the pricing app was cross-compiled and
described in a Linux userspace, then woven. Golden diff is one added path and none
removed; floor.json ratchets 1396→1397 paths, 1965→1966 operations.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/ai fetched the IAM signing cert in a package init() and panicked when it
could not, so identity was a boot-ordered hard dependency of every route in the
binary. That took api.hanzo.ai down twice: 2026-07-27, when a manifest typo killed
IAM's pods, and 2026-08-02, when an IAM build answered no /healthz and its Service
held zero endpoints for 25 minutes. Neither time could the process recover once
IAM came back — the dial had already happened, before any subsystem mounted.
v1.832.10 resolves the cert lazily, retries a failure at most once per 5s, and
refuses requests with 503 at the door while it is unresolved. Never 401, which
blames the caller for our outage, and never served unauthenticated — the
fail-open that once left the service Running and answering 401 to everything.
decimal and money come along as ai's own transitive patches.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Arrival and delivery were the same word. This endpoint logged PAGE-DELIVERED,
answered 200 `ok`, and then failed to send — because the egress ran detached in
a goroutine the response never waited for, so the answer was written before the
send was attempted and could not possibly have been about it. Alertmanager
recorded Notify success over a silent pager.
Split the two facts. ALERT-RECEIVED is arrival; ALERT-DELIVERED /
ALERT-UNDELIVERED is egress. Delivery is synchronous under an 8s budget and THE
STATUS CODE REPORTS DELIVERY: nothing carried it means 503, so Alertmanager
retries and counts it. No egress configured is a failure, not a no-op — that
silent early return is how a whole deployment could page nobody forever.
Egress is a chain: the org's KMS-custodied Slack bot token first, then a plain
POST to CLOUD_ALERTS_WEBHOOK_URL, which needs no integrations peer, no org and
no connected workspace — it works in exactly the state that silenced the first.
A failure is recorded even when a later egress succeeds, so a broken egress
cannot hide behind a working one.
Then the reason no rule could have caught any of this: the data plane emitted
nothing. 904 metric names in the store and not one described a row moving.
hanzo_http_requests_total existed in code with ZERO callers, so /v1/event's 5xx
rate was unalertable while the Sentry envelope 503'd for a day. Wire it into the
one middleware that already has path, status and validated org in hand, and add
the missing measurements: rows landed per warehouse table, ingest admission
outcomes, plane sockets bound, cron fires against the engine's own promise,
memory against GOMEMLIMIT, and alert egress itself.
Every counter is SEEDED AT ZERO at boot. A counter first touched by its first
event has no series until that event, so an ingest path that never runs looks
exactly like one that was never built — which is how span ingest stayed dead for
four and a half months without a single rule being able to notice. Seeding is
what makes silence measurable.
Tests pin the new contract: undeliverable answers 503, no egress is a failure,
the fallback carries it while the Slack failure is still recorded, the chain
stops at the first success, delivery is synchronous, and PAGE-DELIVERED is gone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
api.hanzo.ai/v1/o11y/version answered 403 {"status":403,"error":"no validated
principal"}. So did /health, the three probes, sign-in and the shared-dashboard
reads. o11y.hanzo.ai answered all of them 200. The unified door refused exactly
what the standalone door served, which is why o11y still needed a door of its own.
The refusal was never o11y's. gate() here kept its own list of the paths that
need no principal, and the list named /v1/o11y/api/v1/health, /api/v2/healthz,
/api/v2/readyz and /api/v2/livez — the INTERNAL namespace hanzoai/o11y rewrote
onto until v1.5.37 deleted the rewrite. Four names, zero routes. The exemption
matched nothing, so the gate refused everything behind it, including every op
whose runtime gate is OpenAccess. LLM.md had listed the drift as a known trap
since v1.5.37; the pin reached v1.5.46 with the list still in place.
Two shapes at the door were the diagnosis, not the status codes. /v1/o11y/livez
returned the gate's own {"status":"error","msg":...} and /v1/o11y/version the
zip envelope {"status":403,"error":...} — the second is a typed op relaying to
the same gate and re-raising its {msg} through relay's refusal(). Same gate, two
hops. /v1/o11y/nonexistent 404s, which proves the wildcard is gone and the typed
table is live.
o11y.Anonymous is the fix: the fact lives beside the routes it describes, and
this file asks. The rule is this gate's own purpose read backwards — the gate
exists because the runtime trusts X-Org-Id as gateway-minted, so an op whose own
gate reads no tenant from the request has nothing for a forged tenant to reach,
and gating it can only remove an answer. The exempt set is the runtime's
OpenAccess routes plus the two public-dashboard reads it gates with
CheckWithoutClaims. Every read of a tenant's telemetry stays gated here AND at
the runtime: one rule enforced twice, not two rules. Exemption is not
authorization — the DSN key, the service-account key, the share's scope and the
session cookie are each still the op's own admission test, one layer in.
Also gone: isHealthPath, isErrorIngestPath and isSentryIngestPath, all three
copies of a route table one repo away from the routes; the five /v1/o11y/*
wildcard descriptions, whose address no longer exists; and the three probe
descriptions' dead /api/v2 spelling. The eleven escape hatches hanzoai/o11y
registers by hand now carry prose here, because Describe lives in this module
and o11y deliberately no longer imports it.
The published document still lacks o11y's 353 typed ops: regenerating
plugin/o11y/openapi.json now succeeds (389 operations, up from 34) but the weave
fails closed on six schema names that mean different things in o11y and in
ingress/books/content/analytics/plugins. Named in LLM.md; it is a rename in
hanzoai/o11y, not a change here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The fleet's agent door is composed, not written — zip projects every typed op
into a tool and the host serves the union — so there was never an aggregation
problem to solve and this adds no registry. What was missing is the ANSWER to
"what is on the door", which no process could give and no test asserted; an
inventory nobody can read is how a door serving nothing passes for a healthy
one, which is this estate's recurring defect and not a hypothetical.
ai owns it, in cloud/apps/ai rather than hanzoai/ai: hanzoai/ai imports
github.com/hanzoai/cloud, so the aggregate cannot live there without a cycle,
and a mounted module cannot see its siblings anyway. This package is already
the sibling that imports both.
One typed op, GET /v1/ai/mcp/tools (operationId aiMCPTools), reporting three
numbers that are three different questions:
published — every tool this BUILD can serve, from the same plugin/<app>/mcp.json
bytes the host hands zip at Load. 930 across 116 subsystems.
served — what THIS PROCESS's door actually composed, read from the live
composition (App.Plugins + App.MCPTools). The number that can be
zero while published is nine hundred.
local — the part of served this process registered itself.
Being a typed op, it is itself a tool on the door it describes — ai's first, and
the reason the fleet's total moves 929 -> 930. Nothing here declares that; zip
projects it, zipdoc lifts its prose, and the weave carries its schema.
The gate is the op's own — principal.ValidatedFrom, the bit cloud.Bridge parks —
so it holds identically over REST and MCP and fails closed off the HTTP path.
mcp_test.go proves it in both directions on the real op: anonymous over HTTP is
403 "sign in to read this deployment's MCP tool surface", anonymous over MCP is
isError with that same string, and validated over each answers the same body.
The tests compose the REAL fleet — every manifest row, its real catalogue, as
remote mounts so nothing is spawned — and read BODIES: 930 tools on one door,
no name with two owners, o11y unmounted leaves exactly its 12 behind by name and
by count, and the inventory equals what the door actually serves at three
different compositions. Both halves were mutation-checked: unmounting iam from
the composition turns the count test red (831 vs 930), and making the inventory
report the build instead of the door turns three assertions red.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The plane was five tables with an identical envelope — event.event, event.error,
event.log, event.span, event.metric. That is CONSISTENT and not UNIFIED: no
cross-signal question could be asked without a five-way UNION ALL, and a new
product meant a new table name, which is how a namespace grows a _v2.
Occurrences now land in ONE table, event.fact, with `signal` saying which sort
of thing happened. `signal` also leads the partition key, so per-signal
retention still drops whole parts, `WHERE signal='log'` still prunes to a fifth
of them (measured: Parts 1/5), and the Replacing collapse stays per-signal so an
id collision across two signals cannot delete a row.
event -> act a namespace cannot also be a member of itself
group -> issue it identifies an issue, and `group` is one letter from `groups`
site -> origin the place of the fault; `site` already names a published Site
level + severity_text + severity_number -> severity UInt8, the OTLP number
alone, with the word a read-time function of it
group0..group4 -> groups Map(type, key)
clip is the worked example of why a signal is a value and not a place: session
replay costs one signal value and two columns (object, bytes), reuses
duration/session_id/url, and adds no name to the namespace. The blob it indexes
stays in object storage — it is wrong for a bus message and wrong for a
warehouse row.
Write path: `table` is where a signal lands and how its row is built, and there
are exactly two because there are two grains — an occurrence happened, a sample
was measured. Five writers share one column list, one args builder and one
INSERT string, so they cannot drift into five shapes of one envelope.
landableSignals still DERIVES from writers, so the door and the store are one
edit. sample keeps no writer until event.sample has its name (o11y migration
0003): a door that accepts what the sink cannot land is a 200 that means
discarded.
Read path: `scope(org, signal)` is the mandatory leading predicate of every read
— the tenant first because it leads the sort key, the signal second because it
leads the partition key, both bound. Two lenses were hand-rolling `org = ?` and
would have silently read every signal as a product event.
The fact struct is flat because the table is flat; four optional bodies were the
right shape when each had its own table. Sparsity is not a cost — measured on
the live event.log (798,375 rows), an unpopulated column costs 515 bytes for the
whole table.
DDL is unchanged here and stays unchanged here: hanzoai/o11y owns it
(deploy/datastore/migrations/0002_event_fact.sql), which the pre-existing scan
over this package's sources enforces.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Six instances of one defect shipped: the billing surface dropped compound words
from its route names (payment-methods → methods, payment-config → settings,
auto-recharge → recharge, spend-alerts → alerts, credit-grants → credits, and
four more), cloud and commerce both converged, every published document agreed
with itself, every server test passed — and not one client followed. The
clients' own tests passed too, because a test that pins the URL its client sends
is a test of nothing: it agrees with the client while the client is wrong.
router_test.go asks whether every path an app PUBLISHES reaches that app, and it
is complete on that side. Nobody was asking the other question. manifest/
clients_test.go asks it: for every address a first-party client requests, is it
ROUTED (some row in manifest.Apps hands it to an app) and is it SERVED (that app
registers it, per the subset that app's own binary projects from its own
router). Neither side is derived from the other — `calls` is hand-authored
source, the router is built through the real zip.Load, and the subsets are
forced back to source by `make test`.
A second gate refuses the retired NAMES outright, in `calls` and in Apps. The
general one can only catch a compound name once somebody writes it down, and the
whole failure was that nobody did.
Proven by reverting each half. Point the clients back at the old names and four
failures name the client, the address and its replacement. Drop /v1/billing/
alerts from the commerce row and four more name the address, the app the fleet
delivers it to instead ("ai", the tail /v1 remainder) and the fact that it does
not register it. Both green with the fix.
Writing the ledger turned up three breaks nothing was watching, all the same
shape:
- GET /v1/billing/portal/methods. cloud's billing app serves the saved-card
list by proxying here, and NOTHING in the fleet serves it — so the list is
empty however many cards were vaulted. It was wrong twice: it also asked for
the retired /v1/billing/portal/payment-methods, which does not exist in the
commerce module this binary links. The name is fixed here. The missing owner
is NOT, and is recorded rather than guessed: PortalPaymentMethods keys
tenancy on a customerId query param, so the console chain cannot serve its
service-token caller and the S2S chain would let any authenticated browser
read another tenant's cards by naming their customerId. Choosing that gate
is an IDOR decision, not a route.
- DELETE/PATCH /v1/billing/methods/{id}. The billing app owns the prefix and
registers only the collection, so the live edge answers 405: a customer can
add a card and never remove one. Reaching the handlers means proxying an
address this app owns, which is the self-dispatch loop that produced the
depth-8 502s on top-up. Recorded.
- POST /v1/billing/payment. Never served by anything, ever. Recorded.
GET /v1/billing/credits WAS fixable and is fixed: the handler sat in the
vendored module with no app registering it, so the Credits tab read a 404 the
client swallowed into an empty list. Registered co-resident beside its siblings,
named on the commerce row, and now in the woven spec — so the SDKs get it too.
Minting is untouched and stays on the mint-gated POST /v1/billing/credit.
Regenerating the commerce subset also surfaced the same defect in the prose
layer, which is why `make -C apps/commerce describe` was already red on main:
POST /v1/store/token had no description because the description that exists was
keyed to /v1/store/storefront-token — an address the router has never
registered. Prose written for a route nobody serves is the identical mistake one
layer up. Re-keyed, so the gate is green again.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two money paths read "I could not reach the authority" as "nothing is priced",
and a priced tool was therefore served for free in both. The tests that caught it
say so plainly: a priced tool was SERVED with no payment rail, and a priced tool
was SERVED while its price was unknowable.
The shared mistake is that cloud.ErrNoPeer does not mean what both call sites
took it to mean. It is documented as "this app is not part of the deployment",
but reach() also returns it when it simply cannot get through and no router is
present to contradict it — and a killed peer leaves a socket file behind that
refuses every connection, so a marketplace that DIED is indistinguishable from a
fleet that never had one. Both layers were entitled by that ambiguity to guess,
and both guessed in the direction that gives the shop away.
x402 held an explicit ErrNoPeer-means-unpriced exception, four lines below the
rule it contradicts ("the PRICE is unknown ⇒ error, never free"). It is gone: an
unreached price is unknown, and unknown is never zero. The exception bought
nothing anyway — manifest/apps.go lists marketplace beside x402, so a rail
deployed without its table is a misconfiguration, and one that refuses loudly
costs less than one that quietly sells everything for nothing.
tools mapped the same error to ErrChargerUnset and then asked the LOCAL row
whether it was priced — but in a split fleet that row never carries a price,
because prices are marketplace listings living in another process. Silence read
as free. It now asks the table's owner instead: for sale means refuse, as the
outage it is; not priced, or no table in this deployment either, keeps the old
answer, so a deployment holding neither rail nor table still lets a tool's own
declaration have the last word.
The refusal rides the existing 424 rather than a 402 — no challenge can be issued
for a rail that is gone, and a client cannot satisfy one. A free tool is refused
too while the table is unreachable: the table is the only thing that can say
"free", and this is the direction that costs nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Components states which value each seam contributes — Register a *Schema, the
typed fold zip's JSON verbatim — and Fold is documented as a merge that must not
replace what Register already named. Project then undid both: nouns.into
marshalled every schema to canonical JSON and unmarshalled it back into a bare
any, so every *Schema that reached a document through Project came out the other
side as map[string]any. It ran unconditionally, with or without relays, which
means it was not a relay behaviour at all — it flattened the whole components
block of any document that merely passed through.
The canonical bytes were never the problem; they are the right way to compare two
claims to one name, since two trees that serialize identically describe the same
type. They stay the comparison. What changes is that the claimant's own value is
kept beside them and is what reaches the document, so the bytes decide conflicts
and the value decides the type. Published JSON is unchanged — the bytes came from
marshalling that value in the first place.
into can no longer fail, so it stops pretending it can.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nine admin tests asserted that money reconciles, and all nine failed with the
same signature: credits correct, spend $0, and every per-org revenue read folded
into "partial: one or more org revenue reads failed". Read as a product bug that
is every money panel in admin showing zero against real balances.
It was the fixtures. The billing route became /usage/rollup, and the rename
reached the client and the four other callers but not the four fake-commerce
stubs here, which still matched on HasSuffix(path, "/usage-rollup"). So Spend
requested a path the stub 404s, and the aggregators reported the outage they were
correctly told about. /balance was untouched, which is exactly why credits read
right and spend read nothing.
The stubs each document themselves as mimicking the live contract — one says
"Verified against live commerce" — while routing on a path that contract does not
serve. commerce registers Get("/usage/rollup") under /v1/billing; that is the
server these fakes stand in for.
No assertion moved. Every expected value is what it was; six strings changed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
basedb existed for exactly one reason: cek.Open chose a path and did not create
its parent, and on the pure-Go codec that does not fail the open — the database
is written back at CLOSE, so a missing directory loses every write of the session
after the caller has been told it had a store. cek v0.2.1 does the MkdirAll
itself, pinned by a round-trip test, so the wrapper is now a synonym. It is
deleted and its 70 call sites name cek directly.
The larger duplication was underneath it. Sixty stores each set
SetMaxOpenConns(1) and re-applied busy_timeout, journal_mode=WAL and
foreign_keys=ON by hand. hanzoai/sqlite already applies those, on EVERY
connection — which the hand-rolled db.Exec did not: a one-shot Exec lands on
whichever connection happens to serve it and is gone the moment that connection
is recycled. So the fifty-two copies were not merely a fact restated fifty-two
times, they were the weaker of the two mechanisms shadowing the stronger one.
They say nothing now, and sqlpool_test.go asserts the driver still delivers each
default, so the deletion goes red in one place instead of rotting in fifty.
What is NOT already universal is the pool cap. sqlite sets it on the envelope
path and not on the live-libsqlcipher path, which is the one the shipped image
builds — so the cap is real, and it is stated once, in sqlpool.Single. The
package imports nothing but database/sql, so every store can reach it.
Two databases were outside all of this and are not any more. team keeps its
pragmas, because a dynamic journal_mode and foreign_keys=OFF are an override
rather than a restatement. git's ssh-key registry was a bare sql.Open on a
hand-joined path — no namespace, no key, and consequently the only store in the
binary written to disk in plaintext. It opens through cek like everything else.
hanzoai/sqlite stays at v0.4.0. v0.5.0 deletes the DEK/principal API
(PrincipalType, NewDEK, WrapDEK, UnwrapDEK, PrincipalAAD, DeriveKey) that
hanzoai/commerce and hanzoai/tasks still compile against, and no published
version of either has migrated, so taking it breaks the build. cek v0.2.1 does
not want it either — it requires v0.4.0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The plan reconcile assigned SKU and Metadata, which the catalog does not publish,
so every reconcile cleared a stored SKU and replaced the Metadata map with nil.
planEqual compared neither, so whether a value survived depended on which other
fields happened to differ — same data, same code, two outcomes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
limitsEqual compared Limits by ADDRESS (every field is a *int), so the reconciling
seed would have rewritten every plan row on every boot — right values, but
'corrected' never reaching zero, which is the signal that says the catalog has
converged rather than being fixed over and over.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two defects, each fixed at its one seam.
402/403 -> 500 (#141). zip renders 500 for any error that is not a
*zip.HTTPError, so a clean refusal reached the console as an internal
fault: commerce answers 402 out of funds, the gate answers 403, a plane
callee answers either — and the UI drew a DEAD CARD, with no code to
branch on. The status always existed (ErrInsufficientBalance MEANS 402;
zip's callFault carries a callee's status across the plane intact); what
was missing was a seam to carry it to the wire, so ~20 handlers
remembered cloud.Denied and every other one 500'd.
errmap.go decides it ONCE — refused() is the single classifier, read by
both the app's renderer and the money wire's denial() — and Serve installs
it. A refusal is 4xx ONLY: a 5xx is the absence of a decision, and reading
an unreachable-commerce 502 as a refusal would answer the money wire with
a transport message where "Billing temporarily unavailable" belongs. The
body stays zip's own {status, code, error}; this fills `code` rather than
inventing a second envelope.
spend-alerts 502 recursion (#146). ScopeRateLimit read its config with a
service-token GET /v1/billing/alerts through the commerce transport, which
dispatches in-process by publishing the WHOLE shared app — so the fetch
re-ran the entire edge chain, including ScopeRateLimit, whose cache is
filled only AFTER the fetch returns and was therefore still cold. It asked
again, and again, to the depth guard: 502, ~135 per half hour on the live
pod, with the ceiling failing open throughout. A split deploy recursed the
same way over the network.
commerce owns the rows, so commerce answers for them: a new plane op
(plane.FinanceScopeRules, apps/commerce/scoperules_rpc.go) over the ZAP
socket, which carries that app's ops and no edge chain — the recursion is
structurally absent, not bounded. metering.Client.ScopeRules and the
/v1/billing/alerts path are deleted, so there is one way. The op runs the
same whole-org query as commerce's cap verdict, so a rate ceiling and a
spend cap can no longer bind on different rows.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestCanonicalDoorFallsThroughWhenObsPeerIsAbsent was written around two worlds —
a peer that claims the body, and a peer that is absent so the product wire takes
it. Only one of them was ever reachable: the claim's sink wrote to tables that do
not exist, so it declined every body it was ever offered, and the claim is now
deleted outright. A test that distinguishes a state from itself pins nothing.
The assertion it carries is the one worth keeping, and the parallel lane already
fixed it — the body lands ZERO facts and must be told so (400 unroutable_events)
rather than receipted 200. That is the mechanism that hid a four-and-a-half-month
span outage: a green check beside a total loss. It now asserts BOTH halves,
because either alone is a lie in one direction — a 400 with facts landed is as
wrong as a 200 with none.
So this keeps the assertion, drops the world, and renames to what it actually
pins: an obs-shaped body is ORDINARY at this door, refused on its own merits, and
the product wire beside it is untouched.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A convergence audit read a deployed vintage and reported seven /v1-plane
drifts. Four were already closed on main; three were real, and the largest was
a write path that had never once reached a table.
apps/o11y/event_ingest.go inserted UNQUALIFIED `traces` / `observations` /
`scores`. The DSN it opened (O11Y_DATASTORE_DSN) names no database — hanzo-ds
sets Auth.Database from the URL path and applies no fallback — so those names
resolved to `default`, which is EMPTY on the live datastore. The file said as
much about itself ("ASSUMED SCHEMA", ungrounded against a producer that ships
dist-only) and the datastore's query log holds no such INSERT in its whole
retained window: not failures, never attempted.
It was also the third claimant on one concept. LLM observability is READ off
gen_ai spans in event.span by the o11y runtime, and the eval product owns the
grounded projections — hanzo.eval_traces / hanzo.eval_scores, with DDL
apps/eval/telemetry.go creates itself. So the sink is deleted rather than
qualified: pointing a duplicate at the right database would have made a path
nothing produces for look canonical.
Deleting it retires the plane op it existed to serve (obs_event_claim), which
means POST /v1/event — the fleet's busiest door — no longer makes a
synchronous cross-process round-trip per event to be told "not mine". The
answer the caller gets is unchanged: the claim could only ever decline.
planesink.go keeps the shared datastoreSink, beside its one remaining caller,
and states every table fully qualified. Two pins hold the shape it depends on:
tables name their database (the mistake above, made unrepeatable), and no
plane writer binds ingested_at — event.span measures retention AND partitioning
from that column and defaults it to now64(3), so a writer that names it hands
the wire control of when its own row expires. apps/analytics states the same
rule for its four warehouse writers; this is that pin for these two.
manifest/apps.go drops /v1/insights/e. A prefix there is a claim that the app
ANSWERS the path, and analytics retired that door — a claim it 404s kept the
retirement invisible in the one table that states what the fleet serves. The
four comments still describing it as live are corrected with it.
Verified against the live plane before touching anything: the JetStream stream
is EVENT (4839 messages, 5 consumers, durable on a retain-class volume) and
EVENTS does not exist, so the constant is right as written and a rename would
have orphaned the data or 503'd the plane.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET https://api.hanzo.ai/mcp returned 200 text/html — the console SPA. A client
that checks a status code reads that as a healthy MCP endpoint. POST there
returned 405 "method not allowed" in the console's own voice, which reads as a
door that exists and was called wrong. Neither was true: nothing was mounted at
that path at all.
Registration order is not what did it. The zap-proto/fiber fork matches by
ServeMux-1.22 specificity, so a static /mcp beats the console's /* catch-all no
matter which registered first — proven by the same app answering POST /mcp with
a real tools/list when the door IS there. The path simply had no route in this
process, and the terminal handler could not tell a machine door from a
client-side console route, so it served the shell.
Why there was no route: the address was written down twice and only once.
cmd/cloud moved zip's door to /v1/mcp with a literal; webui's apiPrefixes — the
list of paths the SPA must never answer — did not know the framework default had
been vacated. Two copies of one fact, and the second was missing.
So the address is now ONE value, manifest.MCPPath, read by the app that serves
the door and by the front door that must refuse to answer it with HTML. The gate
in manifest/mcp_test.go reads it too, instead of restating the literal a third
time — a guard that spells its own address can pass while the door has moved.
/v1/mcp is canonical: /v1/<thing> is the house rule, it is what production
already serves, and being under /v1/ is what puts it in the namespace where an
unmatched sibling is a real 404 instead of HTML. Not /v1/ai/mcp — the door
serves the union of every subsystem's build-time catalogue (929 tools across 116
apps), so filing it under one subsystem would either shrink it or misname it.
The terminal handler now owes a caller two honest answers, and neither serves
MCP:
/mcp -> 308 to /v1/mcp, JSON body. 308 keeps method and body, so a POSTed
initialize or tools/list reaches the real door instead of the
shell. A signpost, not a second surface — nothing here has a tool
list.
/v1/mcp -> 405 + Allow: POST for a non-POST. zip registers only the JSON-RPC
POST; MCP Streamable HTTP says a door with no SSE stream answers
405, and the 404 it used to give says the door is absent — the
same lie pointing the other way.
Both rules are self-scoping because the handler is TERMINAL: it sees only paths
no route claimed. A plugin serving its own door at zip's default — which is
where a host forwards a composed tools/call, and why cloud.Serve must keep it —
matches a real route and never reaches them.
And the API-namespace 404 now runs BEFORE the static-console gate. WHAT the path
is decides the answer, not what verb a browser would have used: a POST to an
unmatched /v1 path was answered 405, telling a client the endpoint exists and it
used the wrong method. 405 is a claim about a door; only a door may make it.
Proven against the built host binary, by body:
POST /v1/mcp -> 200 application/json, 42 tools, and an initialize handshake
returning protocolVersion 2025-06-18
POST -L /mcp -> follows the hop, id echoed, lands on the tool list
GET /mcp -> 308, Location: /v1/mcp, no HTML
GET /v1/mcp -> 405, Allow: POST
/, /settings, /mcp-servers, /auth/callback -> 200 text/html, the console shell
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.49.39 alone would have shipped a HALF-MIGRATED catalog. Every stored plan row
carries Managed=true, which commerce's seed sets on its own writes, and the seed
skipped any Managed row — so a boot would have created `go` and `dev` while
leaving `pro` at $20 and `developer`/`plus` on sale beside them.
v1.49.40 gives the seed a real discriminator (AdminEdited, set only by the admin
CRUD) so it can correct rows it wrote itself, and archives what the catalog
stopped publishing. Rolling this binary is now the whole cutover: no SuperAdmin
bearer, no script run against production, no manual step to forget.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/event/{project}/envelope has answered 503 "error ingest unavailable"
for 24h+ with zero successes, and every signal said the path was healthy: the
o11y process was up, /var/lib/cloud/run/o11y.sock was bound, and obs_error_post
was registered on it. The tell was in the access log, not the error — dur_ms=0.
The request was never leaving analytics.
ObsErrorIn.Headers was a map[string]string. zapenc carries scalars, strings,
byte slices, structs, pointers and slices, and REFUSES anything else at encode
rather than dropping it, precisely so a field can never silently fail to arrive:
zip: encode obs_error_post: ObsErrorIn.Headers:
zapenc: map cannot cross the plane; give it a type that can
So zip.Call failed on the caller's side of the socket, Ask returned (nil, err),
and the handler's `err != nil || out == nil` branch reshaped it into the generic
503 that made this look like an unreachable peer. It was an unencodable argument.
Its sibling op on the SAME socket is why this hid so well. ObsClaimIn is two
scalar fields, so obs_event_claim encoded fine and POST /v1/event stayed 200
throughout — the door looked half-open, which pointed every investigation at
routing and at the socket. Binding that socket (f43e2b10) was necessary and did
not fix this: two independent faults on one path, and the first one masked the
second until it was fixed.
Headers is now []Header, a slice of structs — the shape zapenc already carries,
one complete ZAP message per element. Not a workaround for the map; the map was
never a wire type here.
The tests are the two halves of the rule. TestObsErrorInCrossesThePlane makes a
REAL crossing over a real socket and echoes a header back, so a dropped header
fails loudly rather than passing as an empty map; it fails on the old shape with
the encode error above. TestNoPlaneTypeCarriesAnUnencodableKind walks field
KINDS across the plane's types, because the fault is a property of the kind and
the next map added to any of them is this same outage. Nothing in the suite
could have caught it before: the op's own tests call the handler directly and
never encode.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/event answered 200 {"accepted":0,"dropped":1} for EVERY wire shape it
publishes when nothing could be attributed. The anonymous projection refuses a
kind it cannot name (publicKinds stores pageviews and errors; a log, a span and
an exception envelope are none of those) and said so in a receipt field no
client parses and no probe reads. A caller whose key was absent, revoked or
mistyped lost 100% of what it sent with a green check beside it — the mechanism
that hid an 88% log loss, a four-and-a-half-month span outage and a day of
missing Sentry traffic.
The rule is now exactly "did anything land", decided ONCE in `answer` (event.go)
— the tail every lane already reached:
accepted > 0 -> 200, unchanged, INCLUDING the partial batch. A batch is
never failed whole for its worst element.
nothing sent -> 200. Dropping nothing is not losing anything.
nothing landed -> 4xx naming the one thing the caller can fix, in
HTTPError.Code: 401 ingest_key_required (no credential),
403 insufficient_capability (a guest token that RESOLVED —
it has a key, so telling it to get one is a false
instruction), 400 unroutable_events (full capability, and
the body still named nothing storable).
DNT stays 200: an honored opt-out is the one total drop that is not a failure.
The o11y claim goes through the same receipt — it is where the missing logs and
spans were headed, so exempting it would leave the defect in the lane it cost
the most. obs_door_test.go asserted `want 200 via the product wire` for a body
that landed ZERO facts; it now asserts the truth.
A nonzero drop is visible: hanzo_ingest_dropped_total{org,source,reason} (what
an ingest-drop alert reads) plus a warn line naming tenant and door.
Shape dispatch is untouched — isInsightsWire still runs BEFORE the canonical
object branch, pinned directly at the decoder.
25 existing tests encoded the 200-while-storing-nothing as the contract. Each
keeps its capability assertion and swaps only the status observable.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The ladder lived in two places: this script's inline JSON and @hanzo/plans. Two
statements of one thing, kept in step by hand, which is the same defect that put
Pro on the pricing page at $49 while billing charged $20.
It is now one. @hanzo/plans is the catalog; commerce embeds it and reconciles the
live rows to it on boot (v1.49.41), archiving what the catalog stopped
publishing. So changing prices is: edit the package, publish it, bump commerce,
bump cloud, deploy — every step reviewed, versioned and revertible.
This script was the alternative: a human pointing curl at production while
holding a SuperAdmin bearer, with the ladder retyped in bash. Deleting it removes
the second way, the token that had to exist, and the copy that had to be kept
honest. The retire step it performed happens on the next boot instead.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.49.39 alone would have shipped a HALF-MIGRATED catalog. Every stored plan row
carries Managed=true, which commerce's seed sets on its own writes, and the seed
skipped any Managed row — so a boot would have created `go` and `dev` while
leaving `pro` at $20 and `developer`/`plus` on sale beside them.
v1.49.40 gives the seed a real discriminator (AdminEdited, set only by the admin
CRUD) so it can correct rows it wrote itself, and archives what the catalog
stopped publishing. Rolling this binary is now the whole cutover: no SuperAdmin
bearer, no script run against production, no manual step to forget.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
These two edits are ONE change: either alone does not compile. The pin bump
alone breaks the call site (Mount lost its second parameter); the call-site edit
alone breaks against v1.5.41 (Mount still wants two). So they land together.
WHAT THE PIN CARRIES. hanzoai/o11y v1.5.46 is the decomplected seam:
func Mount(app *zip.App) error // was Mount(app *zip.App, deps cloud.Deps) error
Deps was read for exactly ONE field, Logger, on exactly one line, and the router
already carries that same logger — app.Logger() IS the value Deps.Logger was,
reached through the argument that was already there. The second argument carried
nothing the first did not have, and it cost a MODULE CYCLE: o11y -> cloud ->
o11y. github.com/hanzoai/cloud is now absent from o11y's go.mod entirely.
WHY THE CYCLE MATTERED HERE. It is the reason o11y's own community image could
not link its own declarations: with the cycle, `go mod download` was unresolvable
in an image with no cloud checkout, so cmd/community built a package that did not
import the typed table, and 353 typed ops shipped in code the running process
never linked. Proven on the tag, not asserted:
go list -deps ./cmd/community | grep hanzoai/cloud -> 0 lines
go list -deps ./cmd/community | grep -x github.com/hanzoai/o11y -> 1 line
The second check is what makes the first non-vacuous — before the decomplect
BOTH were empty, i.e. the cycle proof passed precisely because the table was
absent.
ALSO CARRIED, and load-bearing for the chart: v1.5.44's zapingest fix. The
`enabled` boolean is deleted and every address defaults to EMPTY, so linking the
module binds nothing. v1.5.41 defaulted enabled with 0.0.0.0:4317-4319 baked in,
which is why cloud needed O11Y_ZAPINGEST_ENABLED=false — a library must not take
a port because someone linked it. That tourniquet comes out in the same chart
bump that ships this image, and not before.
Verified: go build ./cmd/cloud exit 0. Full suite unchanged against pristine
main — 151 ok / 59 FAIL both before and after, byte-identical failure sets (56
are the pre-existing hanzoai/base@v1.5.11 cgo sqlite break; 3 are the known
admin/code/iam set). Zero new failures.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
"cloud holds no crypto: one derived key, no wrapping" (4cb56f6b) deleted the
cloud/cek package outright — cek.go, frozen_test.go, and the
cek/testdata/frozen fixture with it — and moved encryption at rest to
github.com/hanzoai/cek v0.2.0. It edited this Dockerfile to drop the
cek-rewrap binary, but left the frozen-format RUN behind, still testing
./cek.
So every image build since has died at Dockerfile:195 with
FAIL ./cek [setup failed]
stat /src/cek: directory not found
which is the whole of why the release lane is stuck at v1.801.360: gate,
cicd and containment all go green, `image` fails here, and rollout, reach,
fanout and receipt are skipped. Runs 16387 and 16451 are the same failure
twice.
Nothing is unguarded by removing it. The concern it names — a sqlcipher-dev
pin or base bump silently changing the on-disk format and bricking existing
stores — is held by the RUN immediately above, which passes: TestEncryptionProof
proves real ciphertext at rest under SQLITE_REQUIRE_CODEC=1, and
TestUnwrapGoldenFixture opens a frozen golden fixture under the shipped codec.
Both run INSIDE this image under the same pinned Alpine libsqlcipher, which was
the point of gating here rather than only in Go CI. The frozen-fixture guard did
not survive the move because the thing it froze — the per-file DEK sidecar — is
what 4cb56f6b deleted; hanzoai/cek derives its key instead, so there is no
sidecar left to freeze.
The freeze note above loses its last true reference to that test for the same
reason, and now names the golden fixture that does run.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The retire sweep issued DELETE, which was only defensible because nothing had
launched — it takes the row's history with it and orphans any subscription that
recorded the slug. commerce v1.49.39 makes hiding and destroying different
operations, so this uses the one that is not destructive: PUT {"status":"archived"}.
Also: the ladder rows now carry "status":"active" explicitly, so re-running the
script un-archives anything previously retired rather than silently leaving it
hidden; `team` is called out as deliberately NOT retired, since the pricing page
reads its seat price and minimum live from this catalog; and the final check is
scoped to the ladder's own category so `team` does not read as a mismatch.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud registers commerce's own handlers in-process for both halves of the plan
surface (apps/commerce/mount.go: GET /v1/billing/plans -> commercebilling.ListPlans,
and /v1/plans/entries -> planapi.AdminRoute), so the module version IS what
api.hanzo.ai serves. Nothing else delivers it: there is no standalone commerce
deployment, and the commerce container image does not front this endpoint.
What v1.49.39 brings:
- the Go / Dev / Pro / Max ladder with annual pricing on every rung, from
@hanzo/plans 1.4.8
- plan.Status (active|draft|archived) — retiring a plan stops it being listed and
bought without destroying the row, so an invoice or renewal that recorded the
slug still resolves
- the admin CRUD can finally re-describe a plan, not only reprice it: features
and limits were being silently dropped on PUT
- paidTier counts a contact-sales plan as paid, closing a self-serve mint that a
null-priced enterprise tier would otherwise have opened
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`cwd` was write-once: captured at register, absent from both the patch input and
UpdateSession. Right for a run that starts in a directory and ends there; wrong
for a linked shell, which is a place a person moves around in. The console showed
the directory `hanzo link` happened to start in and kept showing it after the
shell had walked away — so the one field that answers "which work is this" gave
an answer that was true once.
`cwd` joins the patch as a pointer, so an unchanged path is an omitted field
rather than a repeated write, and it is bounded by the SAME `maxCwd` register
applies — one rule for one field, whichever door the value arrives through.
The column also joins the UPDATE statement, which is the half that is easy to
forget and impossible to see: a field the patch accepts but the statement omits
returns 200 with the new value echoed in the body and persists nothing. That
exact bug already happened once here with `terminal`, so the note now lives above
UpdateSession and the test sits beside the one that caught it.
The test cannot run on macOS — the pure-Go SQLCipher codec refuses to decrypt to
persistent storage without a RAM-backed scratch dir, and there is no tmpfs. Its
pre-existing sibling TestTerminalSurvivesUpdate fails identically here, so this
is the repo's environment, not the change. Verified by inspection meanwhile: 9
SET + 2 WHERE placeholders against 11 arguments, x.Cwd ninth.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud/cek is deleted and github.com/hanzoai/cek v0.2.0 is the whole of
encryption at rest. A database's key is DERIVED from the deployment's master
and the namespace that owns it — HKDF(master, "hanzo/cek/v1/" + ns + "/" +
subsystem) — so it is not generated, not wrapped, not stored and not rotated
in place.
What that deletes, and why each one was a hazard rather than a feature:
- the per-file DEK and its .dek sidecar. Key material beside a file is
key material that can go missing, and it did: the sidecar had to be
framed into every durable snapshot so a successor could open what it
restored, and a successor that received the database without it had an
unreadable store.
- rewrap, and the self-heal retry in OrgDB that called it. A derivation
that must be migrated is a derivation that can be half-migrated; that is
what took the git plane and the mirror engine down in production, and
with them every deploy.
- cek.Global / cek.Org / cek.Principal. There was cloud's name for an
entity (namespace) and cek's name for the same entity (Principal), with
nsPrincipal translating between them. Now the namespace IS what the key
is derived from, so a file and its key cannot name different things.
- cek.Exists and its sidecar probe. A store is a file; asking the
filesystem is os.Stat, at the one call site that asks.
- replication.go, 84 lines of unwired design commentary. Replication is
hanzoai/replicate over hanzoai/vfs.
Callers pass a DIRECTORY and a SUBSYSTEM NAME, never a path — cek renders
the path from the namespace itself. Three hand-rolled org→slug encoders go
with that: finance's orgPattern, treasury's tenantSlug and team's seg were
each a second answer to "which file holds this tenant's data", and the
treasury one needed a reserved slug to keep a tenant out of the house fund.
The system namespace is a different KIND, so no tenant string can render to
it however it is spelled.
New, and small:
basedb.Open is the ONE opener: cek plus the directory the file lives in.
cek does not create it, and on the pure-Go codec the database is written
back at CLOSE — so a missing parent does not fail the open, it loses the
data at the end. Stated once, beside the open, instead of in ~50 stores.
internal/devmaster keys a test binary. cek reads no environment, so a
process with no KMS mints its own master; one blank import per test
package says so, replacing seventeen near-identical TestMains that set
CLOUD_KMS_MASTER_KEY_REF for a reader that no longer exists.
Two consequences worth naming. A store that is OPEN has no file yet on the
pure-Go codec, so OrgStore.Has is the union of the open set and the disk,
and Each and Stored both go through it. And apps/iam never closed its
*sql.DB at all (orm's AdaptSQLDB borrows the handle; its Close is a
documented no-op), which on that codec means the identity store was never
written back — it now has a Shutdown, wired like every other subsystem's.
Databases written under the old wrapping will not open under this
derivation. That is expected: there is no migration, no fallback and no
version probe, because a second derivation tried on failure is exactly what
made the old binding unenforceable.
Also fixes six test-only KMS fakes that never gained DeleteSecret and two
missing imports in apps/kms — pre-existing at origin/main, and the reason
eight packages could not be test-verified at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The catalog lives in commerce's store, not in code, so it changes through the
SuperAdmin CRUD that admin.hanzo.ai's editor drives. That direction is the point:
hanzo.ai reads GET /v1/billing/plans live, so the site follows the catalog on its
own. Editing a marketing page instead is exactly how Pro came to be published at
$49 on four account surfaces while billing took $20.
Seeds $9 Go / $19 Dev / $49 Pro / $99 Max with the feature copy each tier sells
on, then retires the 16 rows that are not on the ladder — developer, plus,
team-max, both enterprise rows, and the whole world, social and dns categories.
Those categories were charging rows the site either mispriced or never rendered
at all: social had five rows and zero surface.
Idempotent: PUT then POST per slug, so a re-run updates rather than duplicates.
DRY_RUN=1 prints the payloads without sending. It re-reads the catalog afterwards
and fails loudly on any slug that is not the ladder.
The destructive retire step is only acceptable because nothing has launched and
there are no subscribers. The script says so where someone will read it before
running: with live subscriptions, repricing pro 20 -> 49 in place charges people
2.45x on their next renewal with no notice, and deleting a slug strands whoever
is on it. Grandfather first, reprice second, if that ever changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
plugin/o11y/main.go is hand-written (it carries edge middleware and its own
teardown), so it never went through cloud.Serve — and cloud.Serve is what
calls ServePlane for every generated app. o11y therefore bound only zip's
own listener at /tmp/zip-o11y-<random>/o11y.sock; /var/lib/cloud/run/o11y.sock
never existed, and zip.DialApp("o11y") found nothing.
Nothing reported it. o11y served /v1/o11y and /v1/sentry perfectly, so it
looked up — but every cross-plugin call to it failed silently: POST
/v1/event/{project}/envelope answered 503 to every Sentry SDK, and the
LLM-obs claim declined so those batches walked the product wire. It is the
reason the plane ops added for both could not work; they were correct and
unreachable.
Fail-soft on purpose: a plane that will not bind must not take down the HTTP
surface, because this process is what answers /v1/o11y and /v1/sentry.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Picks up the seed that writes enso 4/20, enso-flash 2/4 and enso-ultra
5/25 into the commerce catalog, which is what the overlay added in the
previous commit reads. Without the bump the overlay resolves an empty
first-party section and logs the warning it was built to log.
Adds the test over BOTH halves. They ship as separate modules and each
was green on its own while the number a customer is quoted depends on
the pair composing — seed writes the price, overlay publishes it. It
asserts the three public SKUs land at what production bills and that the
two internal vision engines stay off the list despite being priced.
/v1/pricing has been serving github.com/hanzoai/pricing's embedded
data/pricing.json, which is stamped 2026-03-14 and contains no Enso rows
at all — so the public price list has been silent about a family we bill
for, while other copies of the same number advertised the pre-reprice
20/60 and marketing advertised 3/12, below the charge.
The number now comes from the catalog in commerce, which is where an
admin edits it. The snapshot keeps what it is actually good at: the
document's shape, the editorial copy, and the resold third-party section
whose price is upstream cost x markup and was never the thing that
drifted.
Scope is first-party only (enso, zen) — the models whose price we SET,
which is exactly where the divergence was. Repointing the 403 resold rows
as well would mean betting a public endpoint on production catalog state
this change cannot verify, and buys nothing: their price already derives
from cost. Retiring the snapshot entirely is the follow-up.
The overlay is applied wherever the served document is assembled — Mount
AND RunSync — because RunSync re-reads the embedded snapshot and would
otherwise quietly restore the stale first-party prices on the next sync.
It never fails the read path: this is a public price list, and refusing to
boot over a catalog query would trade a stale price for no price. But it
does not degrade quietly either, which is the whole lesson here — an
unreachable store logs an error naming which prices are then suspect, and
a store that prices nothing logs a warning saying so.
Billing is untouched. enso meters from its own catalog and must keep doing
so: a charge that fails when commerce is unreachable is strictly worse
than one reading a local copy CI proves equal.
/v1/models is the catch-all mount, and on v1.832.8 it holds a gate that
authenticates nobody: it refuses an ABSENT credential and a MALFORMED one,
then admits any string SHAPED like a key. Measured on production this hour:
no Authorization header -> 401
Bearer totally-bogus -> 401
Bearer hk-<36 zeroes> -> 200 (never minted)
a 3.8-day-expired JWT -> 200
The cost is diagnostic. /v1/models is the natural "is my auth working?"
probe, so answering 200 to a dead credential sends a holder to debug the
wrong system. The catalogue is public by design and five things already say
so — the authz filter lists it public, filter_balance refuses to gate it,
the rate limiter excludes it, spend.Reachable carries /v1/models/, and
docs.hanzo.ai fetches it from a browser. listAvailableModels takes no
principal: the catalogue is identical for everyone.
The header is still read for ONE thing, annotating gated SKUs with the
caller's own access standing, which degrades to nothing without a VERIFIED
principal. Nothing new is disclosed.
Pairs with 9caf0eb8 on this side: that commit records IAM's refusal `code`
beside the nil principal instead of dropping it, and v1.832.9 is the half
that SPENDS it — each code maps to one cure ("revoked — mint a new one"
rather than "the entity does not exist"), with the key named by prefix only
and never echoed.
Verified before pushing:
CGO_ENABLED=0 go build -tags sqlite_fts5 ./... exit 0
go test -tags sqlite_fts5 ./apps/ai/... ok
root-package failures: 18 on v1.832.8, 18 on v1.832.9 — unchanged, and
pre-existing (audit/orgstore/tenantdb, none on the key path). The zipdoc
-check gate is likewise already red on main for ~50 packages, before this.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nothing about turning a name into a place was cloud's to own. The slugger, the
door an org and a project walk through, and the key and path they render to are
one primitive, and half of it lived here while the value it produced lived in
hanzoai/namespace. That is one fact with two homes, and these particular strings
are directory names on live volumes and keys in live buckets: a disagreement
between the two homes does not fail, it opens an empty database beside a real
one.
So the primitive moved (namespace v1.2.0: Sanitize, OrgProject, MustOrgProject,
Key, Path) and cloud calls it.
Deleted here: SanitizeOrg with isDNSLabel, looksSuffixed and its hash
disambiguation; nsKey and nsPath; the bodies of OrgNamespace and
MustOrgNamespace; provisioning's SanitizeOrg/sanitizeOrg, which were a two-hop
delegation to the same function and not a second implementation of it. Every
call site now says namespace.Sanitize, namespace.Key or namespace.Path, so a
reader looking for the rule finds one place to look.
OrgHasUnsafeRune stays at the identity boundary — that is where a request loses
its org-scoping — but it no longer re-decides the rune class. It is
`s != "" && namespace.Sanitize(s) == ""`, which is the same predicate by
construction rather than by two loops agreeing. Verified equal to the old loop
on every rune below U+3000 and the empty string.
OrgNamespace / MustOrgNamespace / PlatformNamespace keep their names. They are
cloud's DOOR, not a second implementation: which of cloud's values may name a
database is a question about cloud's principals, and orgns.go is the only file
allowed to ask it. TestOnlyOrgnsBuildsANamespace now also refuses
namespace.OrgProject and MustOrgProject outside that file, so the new
constructors cannot be reached from a handler holding a query parameter.
Byte-identity, which is the whole risk: 45 orgs x 7 projects x 4 subsystems plus
the system partition, the zero namespace, the empty dir and the non-org kinds
were rendered before and after through this exact code path. 2055 lines, same
SHA-256. No live store moves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The register retry added earlier exists so a rolling control plane cannot kill a
worker mid-render: api.hanzo.ai answers 503 for a few seconds while its pod is
replaced, and riding that out saves whatever was sampling.
401/403 is the opposite kind of failure. It says this node's token is not accepted,
and waiting never changes that. Retried, it cost 30s per boot inside a systemd restart
loop — found at restart counter 10 on a node whose credential had expired — and buried
the one line naming the cause under five that said "retrying".
It now fails on the first refusal and says what to do: run `hanzo login` on that node.
Same lesson as the held spool, one layer over: a permanent condition wearing a
retryable shape is worse than an error, because it looks like progress.
Two defects the engine-on-plane pass left behind, both of which make a migrated
read return honest-looking nothing forever — the exact failure that migration was
built to end.
SERVICE — planeService resolved service.name, then the wire app name, then the
`app` label, then gave up. The fleet's LARGEST log producer states none of the
three: the otel-agent's filelog receiver stamps k8s.* on every tailed container
line and nothing else. Measured on the live plane: 143,475 of 157,652 event.log
rows in a 30-minute window (91%) carried service='', and NOT ONE row existed for
any product in the console catalog — cloud, gateway, iam, o11y, kms, studio. The
infra log lens filters `service = <workload>` (apps/o11y/logs.go), so it was dark
for every product, and event.log's (org, service, time) sort key was degenerate
besides.
So resolve the WORKLOAD, in one place, for spans and logs alike: OTel's own
service.name recommendation — k8s.deployment.name → replicaset → statefulset →
daemonset → cronjob → job → container. k8s.pod.name sits in that list in the spec
and is deliberately omitted: it is per-replica (ingress-b7854888d-gb8xw), so it
would make a LowCardinality column unbounded and could never match a
workload-keyed read. On today's rows this yields ingress, hanzod-mv, mpc-node,
csi-do-node. sdkSpanRowsOf now shares the resolver instead of reading
attrs["service.name"] itself — one function decides what a service IS.
WIRE — event.span has been empty since 2026-08-01 00:30 while FIVE read paths
query it (admin fleet board, admin subsystems RED, per-org request logs,
per-org RED, eval GenAI latency). The sink was live and the spans were real; they
had nowhere to go. Cloud runs its ~20 subsystems as sibling plugin PROCESSES in
one pod, the plane sink registers in exactly one of them, and the wire fallback
the design already describes for "o11y as a plugin" had NO endpoint to fall back
to: OTEL_EXPORTER_ZAP_ENDPOINT and OTEL_EXPORTER_OTLP_ENDPOINT are both empty in
prod, so every sibling's Send returned ErrNoRoute and the batch was dropped.
wireEndpointFor makes the destination table total: explicit endpoint, else the
fleet collector when a legacy OTLP endpoint declares remote intent, else the pod's
own loopback (127.0.0.1:4317 — planeSpanListen) whenever O11Y_TRACES_ZAP_INPROCESS
says a sink exists in this DEPLOYMENT. Same pod, same lifecycle, no Service. The
process HOLDING the sink is routed and never reaches the fallback, so it cannot
self-dial.
TESTS — the k8s precedence table (including that a pod name is NOT a service), the
filelog resource end-to-end on logRowsOf, the SDK-span twin, and the four legs of
wireEndpointFor. LLM.md's write-plane sections are rewritten off the code that
actually exists: they still described ingest.go's otelcol pipeline writing
o11y_traces/o11y_logs, tracesink.go's pdata conversion, and metrics as DEFERRED
on a driver fork — all three retired.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Its door was already deleted, but the canonical decode never learned the
shape, so every PostHog SDK beacon landed on /v1/event and decoded with an
EMPTY person and an unnamed kind — which admitPublic drops whole. The SDK
saw 200 and stored nothing. That is the exact failure the team wire had,
and insights.hanzo.ai's /e, /batch and /capture all rewrite onto this door,
so it was the live path for every PostHog client.
decodeIngest now dispatches by shape: PostHog spells the person
'distinct_id' where the canonical wire spells 'distinctId', and the team
wire is a bare ARRAY, so the three are disjoint and the probe is
positive-signal-only — a miss falls through unchanged.
The probe runs FIRST, ahead of the object branch, because a PostHog batch
envelope spells the same 'batch' key as the canonical one: routing on the
key alone hands it to a decoder that cannot see distinct_id. Shape wins
over key — which is what the /v1/insights/e deletion assumed and did not
implement. Pinned by a test covering the bare event, the batch envelope,
and the canonical wire staying canonical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Red held this plane on three ship-blockers, all of which were the same shape:
a property the package claimed and did not have.
EVERY READ OF THE SOURCE IS NOW ADMITTED. `lineage` re-ran the exact census a
materialisation is charged 10c for — an exact distinct-count over up to 400 days
of one tenant's feature surface — with no gate, no meter and no bound of any
kind, on a GET any authenticated caller could loop in parallel against the one
stateful store every product on api.hanzo.ai shares. It is not fixed by adding
three checks to that op: `census` and `facts` now take a `scan`, a value with no
exported field that exists only where `admit` returned one, so an unpriced
unbounded warehouse read is not a thing this package can spell. admit is the one
door — it prices at the meter, takes the tenant's single slot and one of the
plane's eight, and lineage additionally runs under the plane's own deadline
rather than the caller's patience. A test reads the package's own AST and fails
if any function that reads the source takes no admission.
A JOB CAN RECORD THAT IT RAN OUT OF TIME. The write that ended a job shared the
context of the work it was reporting on, so the one case a refusal exists for —
the fifteen-minute wall expiring — could never be written: the driver refused
before sending anything and the version sat in `materializing` with an EMPTY
refusal forever. The work's context and the record's are now two, and `record`
opens its own, so they cannot be fused again.
ONE MINT. hanzo.risk_feature.org is written by the rollup's `qualify`, which
lower-cases the brand and requires a registered one, and was read here by a mint
that only trimmed and accepted anything. CLOUD_BRAND=Hanzo therefore had the
writer filing rows under `hanzo/acme` while this plane asked for `Hanzo/acme`
and got nothing, forever, with no error anywhere. The mint canonicalises and
requires a brand the registry vouches for; a test pins it byte-for-byte against
the writer's algorithm over every brand and spelling, and Mount refuses at boot
rather than 403-ing every request.
Also closed, from the same review:
the biller's own words A gate that cannot be ASKED fails with the peer's
transport detail, which zip renders as the body of a
500. Both priced ops now refuse through cloud.Denied
— the fleet's one 402/503 contract — and the reason
goes to the log.
version numbers Disposal dropped the register, so the counter reset:
a second `orders` reached v3 again with different
bytes and nothing recorded that the first v3 existed.
Disposal now drops the BYTES and marks the register,
`disposed` outranking `ready` in the engine's own
version column, so numbering is monotone across a
disposal and "prove you deleted it" is answered by a
record instead of by silence.
a falsifiable claim lineage certified a window re-derivable whenever the
source had not SHRUNK — but the source is fed by a
rollup running behind the events, so holding MORE is
the ordinary case and re-running the spec would not
reproduce the digest. Reproducible now means exact
agreement on all four recorded measurements.
one brand's org cloud trusts every white-label brand's issuer, so a
lux.id token reached a hanzo deployment and minted
`hanzo/<org>` for it. The identity boundary now
stamps the brand it VERIFIED from `iss`, and the
mint refuses when the two facts disagree.
what is per process The package claimed two processes over one store
answer identically. Reads, declarations and disposals
do; ADMISSION does not, and that is now stated with
the deployment contract it implies rather than
claimed.
And one the review missed: the branch already failed the fleet's own
TestRequestEscapeHatchIsPinned, because `who` reads the request for the BILLING
identity — the ledger, the validated project, the attribution — which is
deliberately a different value from the tenant. The reason is now on the record
in allowedRequestUses, where the next reader will find it.
Every fix carries a regression test, and each was mutation-tested: with the fix
reverted, its test fails.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A dataset is a VALUE, not a query. Storing a spec and re-running it is the
design that guarantees irreproducibility: the source is a SummingMergeTree
whose parts merge, its retention drops the tail, and the rollup behind it can
be re-run — so the same question asked twice is two different answers, and a
model that cites "the query" has cited nothing. Here a dataset is declared as
a version, materialised once, fingerprinted over the spec AND the rows, and
never rewritten. That is the only form in which an audit can be answered.
Seven typed zip ops under /v1/ml/datasets — declare, list, describe,
materialise, lineage, export, dispose. One declaration each, so the route, the
OpenAPI operation, the MCP tool, the CLI verb and every generated SDK method
come from the same place and cannot drift.
The four properties, and where each is enforced:
tenancy every statement opens `org = ?` with a tenant.Key, which has no
exported field, cannot be written as a literal outside its own
package and cannot be decoded from a request body. The key also
leads both tables' sort keys and both partition expressions, so
a per-tenant read is a prefix scan and a disposal cannot be
spelled across a tenant. A row that comes back belonging to
someone else is a REFUSAL, not a filter.
immutability `ready` is the greatest rank of the ReplacingMergeTree version
column, so no later write of any other stage displaces a
published version; the door refuses any transition out of a
terminal state. Two layers, engine and door.
determinism splits are temporal cuts and then grouped by SUBJECT — a random
split puts one device on both sides of the line and the model
memorises the entity. Membership under the row cap is a seeded
sample of the subject, recorded on the manifest, so a capped
dataset reproduces instead of being whichever rows came back
first. Coordinates are hashed as IEEE-754 bits, not as text.
expiry NO table TTL, deliberately: a table TTL is a fleet-wide clock no
tenant can hold longer. Disposal is the tenant's own DROP
PARTITION on (org, name).
The maturity horizon is what keeps a training set from knowing the future: a
chargeback lands 30 to 120 days after the transaction it condemns, so a row is
admitted only once it has aged past the horizon its version declares. Lineage
is MEASURED rather than recalled — the plane re-asks the source and says
plainly when the window can no longer be re-derived, because an admitted gap is
actionable and an unfalsifiable claim is not.
Bounded so a build cannot become a DoS: the window by the source's retention,
the horizon at a year, rows at 200k, names and versions per org, ONE
materialisation per tenant and eight in the process, a fifteen-minute job wall,
a bounded export page, and the priced act metered at the same gate the fleet
uses. Materialising answers 202 as soon as the attempt is on record; a client's
timeout is never a data plane's timeout.
Its own app rather than a leaf of another: it shares no state with a scorer, so
it restarts empty and answers identically — which a process pinned to one
replica for its in-memory forests cannot promise, and which is exactly what a
plane holding the record of what a model trained on must do.
apps/tenant is its own package because the key is not any one subsystem's: the
dataset plane, the scoring plane and the compliance plane must agree on it byte
for byte, or a dataset is keyed differently from the model fitted on it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Production was answering 503 'error ingest not initialized' to every Sentry
SDK on POST /v1/event/{project}/envelope, and silently sending every LLM-obs
batch down the product wire — both because analytics (pid 79) read package
globals that o11y (pid 46) had set in ITS OWN process. A plugin is a process;
cloud.SetObsEventIngest / SetObsErrorIngest could never have worked across
that boundary, exactly like the Slack egress before them.
o11y now publishes both claims as plane ops (obs_rpc.go): obs_event_claim
offers one authenticated body to the LLM-obs sink, obs_error_post relays a
Sentry envelope/store request. analytics asks over the socket.
Two properties the relays keep, because getting either wrong is worse than
the bug: the Sentry answer is returned VERBATIM, so a 401 'invalid ingest
key' still tells the SDK to stop retrying rather than being reshaped into a
5xx it will hammer; and the claim is FAIL-SOFT AND UNCLAIMED — any plane
error (o11y absent, asleep past the wake budget, mid-restart) falls through
to the product wire, because a dropped claim costs one event in the obs
store while a failed door costs every event. Pinned by a test that drives
the door with no peer reachable and asserts the warehouse still receives.
The dead seams and their file are deleted; comments across both packages now
name the ops rather than the globals.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The previous commit read the identity store in process and was right about the
wrong topology. Each app is its own composition root — plugin/<app>/main.go calls
cloud.Serve, and Serve installs the identity boundary — so EVERY process resolves
API keys, while the store has one writer and it is the iam app. Reading it in
process therefore answers in exactly one of the fleet's processes and returns nil
in all the others: every hk-/sk- key anonymous on /v1/chat/completions,
/v1/agents, /v1/gpus, which is the .244 break again by a different route.
So the rule is stated once — READ THE STORE WHERE IT LIVES — and the transport
follows from where that is. The iam app publishes the store to this package when
it mounts, so the process that HAS it reads it with no hop; every other process
asks iam over the internal plane, a unix socket in the pod's own runtime dir.
Both paths run the same two functions (PrincipalFromStore, OrgFromStore), so the
transports cannot answer differently — only the distance changes.
Two ops, not one: iam_resolve_key answers WHO for a secret key, iam_resolve_org
answers WHICH ORG for a publishable one. A single op with a mode flag would make
"resolve this pk- to a user" expressible, and that is the browser-key catastrophe
the two doors exist to prevent. It is also the one op on the plane that carries a
credential in its argument rather than taking its subject from the caller — it
must, because it runs BEFORE a principal exists; it IS the authentication. The
key is already in the calling process and the socket does not leave the pod.
What this replaces is a request that left the pod, crossed the cluster network
and came back to a sibling container, carrying a confidential client credential
so cloud could authenticate to its own deployment.
Failure modes, deliberately different:
* the HANDLER fails closed on a store this process was supposed to own — a nil
handle there is a boot-order fault, and answering "unresolved" would turn it
into a silent fleet-wide de-authentication;
* the CALLER resolves nothing when iam is unreachable — an unresolvable key is
anonymous, which is what an unconfigured resolver has always meant, and a bad
key has never granted trust.
The tests now drive the wire. apps/iam mounts the store, takes it away, and calls
cloud.OrgForKey — so what runs is the dial, the op and the decode, not a branch
production never takes. Deleting either plane branch fails exactly those three
tests, which is the regression above.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
IAM is grafted into this binary — apps/iam mounts iamserver.NewApp on cloud's own
router — so the code that resolves an API key and the code that asks for one
resolved were already the same process. They could not be the same CALL: apps/iam
imports this package, so this package cannot import apps/iam, and the resolver
reached its own identity store over HTTP at iam.hanzo.svc. A network round trip a
process made to itself, forced by the direction of an import.
Everything around that hop was scaffolding for a call that should never have left
the process: a confidential client credential to authenticate to ourselves, a
5-second timeout on the hot auth path, JSON envelopes to re-decode rows we already
have, and an init() that panicked when iam.hanzo.svc was unreachable — taking
api.hanzo.ai down over a dependency this binary contains.
So invert the dependency rather than work around the cycle: IAM hands its opened
store to this package when it mounts (SetIAMStore, retracted on a fail-closed
mount), and both resolvers read it directly — UserByAccessKey for a secret key,
PublishableKeyByAccessKey for the org-only door. The import direction is unchanged;
only the direction of the call is. Requires iam v1.34.0, where the store and the
identity model are importable.
The tests move with it. They used to stand up an httptest server and assert on URL
paths and JSON envelopes — a fair test of an HTTP client and no test at all of key
resolution, because the stub answered whatever the test told it to. They now seed a
real store and run the real queries, which is what lets them assert the invariants
that matter: a pk- resolves to an org and never to a principal even when its row
names a real user in its own tenant; an sk- whose row names a foreign user is
refused as a forgery, not as a typo; a cached answer survives the store being taken
away. All three hold under mutation.
Two gates that were not measuring what they claimed:
* the cloud.Request ratchet walked "." and skipped only ".git", so it also walked
.claude/worktrees — whole checkouts of this same repo — and reported every call
site two or three times under paths that exist for nobody else. A dot-directory
is never this module's source; one rule, so nothing has to be added there again.
* apps/iam had no TestMain, so cek refused to open a store with no master key and
every test in the package failed before it asserted anything. Nine tests read an
empty router and reported a lost registry, a missing schema set, a 503 where a
401 belonged — one sentence, no key, told nine ways. credz.Boot is the path the
binary itself uses, so boot it first and the package measures the real surface.
That last one makes a real backlog visible: iam serves 97 untyped operations against
a ratchet of 88. It is not a regression from this change (97 at v1.33.43 too) — it
grew while the gate was dark. The cure is the verb-alias surface, /v1/iam/get-user
beside the typed /v1/iam/users, which still has callers in cloud, console, universe
and the Python SDK and so migrates on its own terms.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Inlining has a ceiling and photographs do not. The tasks API takes a 4MB body, so a
4K source (19MB, 25MB base64) could never ride inside a job: the dispatch was refused,
the refusal was reported as "GPU render worker is momentarily unavailable", and the
render waited on a fleet that was healthy the whole time.
An input may now arrive as a URL. The worker GETs it with its own credentials — the
same ones uploadOutputs already uses, and the coordinator resolves the org from the
token rather than from anything the job claims — and stages the bytes into the local
studio exactly as an inlined input is staged. One code path from there on.
Inlined inputs still work; this only adds the other way. Studio starts emitting URLs
once every worker can read them, which is what this commit is for.
The engine-on-plane pass (#379) moved every write and read cloud owns in ITS OWN Go
onto event.span / event.log, and production proves it: o11y_logs.distributed_logs_v2
took its last insert at 22:55 UTC and event.log has carried the sink since. What it
could not move is the write cloud does not own — the /v1/sentry face delegates to the
EMBEDDED hanzoai/o11y runtime (prod logs: "o11y runtime handler installed (in-process
runtime)"), so the table that face writes is whichever one the PINNED module names.
v1.5.34 named o11y_sentry.o11y_sentry_events. So did the tag-suggest read
(o11y_traces.tag_attributes_v2). Both were still open, and a database that is still
open cannot be dropped.
hanzoai/o11y v1.5.41 is the version that closes them:
- implsentry/eventstore.go — defaultEventsDB/Table are event/error, the same
15-column envelope event.event / event.log / event.span already share.
- telemetrytraces/tables.go — SpanKeyTableName/SpanAttributeTableName are the
plane's span_key / span_attribute, so the ~15-min tag-suggest loop reads the
plane instead of o11y_traces.
The bump is the whole fix; there is no second way to point an embedded runtime at a
table. go mod tidy then demotes the otelcol require block to indirect — #379 deleted
the collector pipeline that made those direct, and this is the receipt for it.
typed_wire_test.go moves with the wire: v1.5.41 retired the terminal /v1/o11y/*
catch-all, so the eleven routes it used to hide are now named and visible to the gate
— three probes, three streams, three sign-in callbacks and the four Sentry ingest
paths. Each is listed with the wire fact that keeps it un-typed (a stream has no one
complete JSON value; a 303 has no body; an x-sentry-envelope frame is a shape we
receive, not one we publish), and the stale {wildcard1} entries are gone. That the
test went red on the bump — naming both the new routes AND the dead reasons — is the
gate working.
GATE: build ./... clean; apps/o11y green; apps/eval green with a dev master key.
apps/admin's nine money failures reproduce byte-identically at origin/main.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
IAM is grafted into this binary — apps/iam mounts iamserver.NewApp on cloud's own
router — so the code that resolves an API key and the code that asks for one
resolved were already the same process. They could not be the same CALL: apps/iam
imports this package, so this package cannot import apps/iam, and the resolver
reached its own identity store over HTTP at iam.hanzo.svc. A network round trip a
process made to itself, forced by the direction of an import.
Everything around that hop was scaffolding for a call that should never have left
the process: a confidential client credential to authenticate to ourselves, a
5-second timeout on the hot auth path, JSON envelopes to re-decode rows we already
have, and an init() that panicked when iam.hanzo.svc was unreachable — taking
api.hanzo.ai down over a dependency this binary contains.
So invert the dependency rather than work around the cycle: IAM hands its opened
store to this package when it mounts (SetIAMStore, retracted on a fail-closed
mount), and both resolvers read it directly — UserByAccessKey for a secret key,
PublishableKeyByAccessKey for the org-only door. The import direction is unchanged;
only the direction of the call is. Requires iam v1.34.0, where the store and the
identity model are importable.
The tests move with it. They used to stand up an httptest server and assert on URL
paths and JSON envelopes — a fair test of an HTTP client and no test at all of key
resolution, because the stub answered whatever the test told it to. They now seed a
real store and run the real queries, which is what lets them assert the invariants
that matter: a pk- resolves to an org and never to a principal even when its row
names a real user in its own tenant; an sk- whose row names a foreign user is
refused as a forgery, not as a typo; a cached answer survives the store being taken
away. All three hold under mutation.
Two gates that were not measuring what they claimed:
* the cloud.Request ratchet walked "." and skipped only ".git", so it also walked
.claude/worktrees — whole checkouts of this same repo — and reported every call
site two or three times under paths that exist for nobody else. A dot-directory
is never this module's source; one rule, so nothing has to be added there again.
* apps/iam had no TestMain, so cek refused to open a store with no master key and
every test in the package failed before it asserted anything. Nine tests read an
empty router and reported a lost registry, a missing schema set, a 503 where a
401 belonged — one sentence, no key, told nine ways. credz.Boot is the path the
binary itself uses, so boot it first and the package measures the real surface.
That last one makes a real backlog visible: iam serves 97 untyped operations against
a ratchet of 88. It is not a regression from this change (97 at v1.33.43 too) — it
grew while the gate was dark. The cure is the verb-alias surface, /v1/iam/get-user
beside the typed /v1/iam/users, which still has callers in cloud, console, universe
and the Python SDK and so migrates on its own terms.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
FOUR MORE COPIES OF THE ADDRESS RULE. apps/security, apps/legal and
apps/compliance each carried their own clientIP — left-most X-Forwarded-For,
falling back to X-Real-Ip — and apps/plugin put the raw header straight into an
audit record's SourceIP. Every one of them writes the value into a durable audit
row, and both headers are written by the party being audited: an address the
subject chose is not evidence. They read cloud.ClientIP now, which is the one
rule, so there is one place to be right and one place to review.
THE DOCUMENT. plugin/gateway/{openapi,mcp}.json re-describe the config plane from
the typed ops — mode's prose says what it now does, and the traffic view carries
the tenant's own saturation count — and openapi.yaml is the weave of the subsets,
which this branch had left behind: it added GET /v1/gateway/traffic without
regenerating the golden, so the published document was one operation short of the
routes and openapi/floor.json still said gateway had two. The floor RAISES
(2 → 3, paths 1396 → 1397), which is the direction a new route is meant to move
it. `make -f mk/fleet.mk openapi-weave` is green and idempotent.
The o11y plugin's own chain test pins what the new boundary buys: a forged
X-User-Id and X-User-IsAdmin do not survive it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Six defects, and the first two are the same defect twice: state a caller can
grow, held in one place, bounded once for everybody.
THE AGGREGATES ARE PER TENANT (ring.go). One process-wide velocity.Store with a
~9,225-key cap over the whole fleet backed eight of the nine model dimensions.
A key is <org, axis, subject> and the subject comes off the wire, so the busiest
organisation evicted the quietest one's keys — and a tenant reading zero for
eight dimensions scores as unremarkable and raises nothing. No error, no log, no
alert: a cross-tenant denial of service costing the attacker ordinary use of
their own account. The fix is the shape, not a bigger number: a ring set belongs
to one resident, newRings is the only constructor in the package, and the bound
is a per-tenant byte budget, so a tenant that outgrows it evicts its OWN
least-recently-active subject. The process ceiling is the product of two stated
numbers (64 x 8 MiB) rather than a hope.
THE RINGS ARE DURABLE, as a projection rather than as state. This binary rolls
one replica at a time with the old pod stopped first, so in-memory aggregates
were lost on every rollout — the same silent quiet, arriving on a schedule. They
are now rebuilt deterministically from the tenant's own record of what it
taught, bounded by age AND by count, because every tenant's shelf shares one
volume and a record bounded only by age is one tenant filling another's disk.
TIME IS BOUNDED IN BOTH DIRECTIONS. `at` was parsed and believed. The rings
track a leading edge, so one event stamped in the future moved it there and
every real event for that subject was then older than every window: one request,
and that subject's velocity features read as nothing for as long as the state
lived. Per-subject detector evasion, free, for any authenticated caller. Refused
at the wire door with a 400 and refused again in the rings, so no path can
poison the edge.
SCORE AND LEARN ARE GATED AND METERED. The only Gate/Meter call site was the
search. The two ops an abuser would actually call in a loop were free, unbounded
compute against per-tenant model state and a per-tenant disk write — a denial of
service and lost revenue at the same time, and which one it is depends only on
who finds it first. The billable unit is a SCREEN: one event judged against an
organisation's own model. Scoring is one, a batch is one per event, a search is
one per candidate per event, priced from the measured size of the run rather
than as a flat fee. Each tenant's in-flight calls are bounded too, per tenant,
so at the bound that organisation is told to slow down and nobody else notices.
THE MOAT IS WIRED. rollup() was the only writer of hanzo.risk_feature and had no
production caller, so the per-org feature surface was a table nothing wrote, the
warm read nothing, and the whole thing was prose. plane.roll is that caller: one
window per source plane per tenant under a durable per-tenant watermark, because
the surface is a SummingMergeTree and rolling a window twice does not replace
it, it doubles that organisation's history. Windows are aligned to the surface's
own five-minute grain — a window that ends mid-bucket counts every uniqExact
column once per partial insert — and every surface read rolls first, so a search
cannot answer "your history is empty" while the fold that would have filled it
is still running.
ONE ORGANISATION, ONE VOTE. The k-anonymity floor counted CONTRIBUTORS: twenty
five organisations satisfied it while one supplied a million of the million-and
-twenty-four values, and the published network median was then that
organisation's own median, republished under a name that says it is everyone's.
Each organisation is now reduced to one number — its own median over its own
subjects — before any quantile is taken, so every contributor's weight is
exactly 1/orgs, which the floor bounds at 4%.
Three more found while proving the above:
A FOLD IS ADMITTED, NOT ASSUMED. A fold rolls four source planes and reads a
window, and one goroutine per residency is unbounded fan-out at the warehouse.
Tickets bound the concurrency; a tenant that finds none is not marked folded,
so its next touch retries — deferred and reported, never dropped and silent.
A HISTORY IS FOLDED ONCE. The fold watermark travels in the snapshot's own
row, so a model restored from its own state does not read the same thirty days
again. Without it, a resident bound a busy fleet hits routinely meant the
masses counted how often we evicted a tenant rather than what its traffic did.
Both watermark-driven folds serialise per tenant, because idempotent-under-a
-watermark is only true one at a time.
A METER OWNS THE STRINGS IT RETAINS (metering.Usage.Clone). A Usage built in a
handler carries zero-copy views into the reused request arena, and MeterUsage
records on a background goroutine — so the debit could marshal the NEXT
request's bytes onto this caller's row, on a connection two tenants took turns
on. Not a crash: a wrong record. Fixed once where the retention happens rather
than in each of eleven callers.
Nine new tests, each named for the defect it refutes and each mutation-proven:
remove the fix and the test that names it fails. zipdoc -check green, the woven
document regenerates from source, and the suite is green under -race.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three ways a caller was deciding its own case, and one shared table every
tenant could evict from.
THE ADDRESS. ClientIP took the LEFT-MOST X-Forwarded-For entry — the one the
client writes — so one host could present a million clients: it defeated the
per-IP edge limit keyed on it, put a chosen address into an audit row, and fed
the sensor's address table without bound from an unauthenticated request. The
rule is now the peer first (a direct caller cannot lie about its socket) and,
for a trusted peer, the right-most chain entry that is not one of our own hops.
Our hops are a CIDR set (CLOUD_TRUSTED_PROXIES, private space by default), not
a hop count: a count is a promise about topology that nothing enforces. All
X-Forwarded-For header lines are read, since a second line hides the first.
THE LANE. principalValidated read c.Org() || c.User(), and both are headers:
X-Org-Id survives the boundary on the anonymous path by design, and a
hand-written plugin process may have no boundary at all. Two headers plus an
sk--shaped string that never validated moved a bad bot into the AGENT lane. The
boundary now states what it minted (principal.Mint, a request-local slot no
client can write) and the classifier reads that. No attestation ⇒ anonymous.
The gate's tenant comes from the same place, so a forged org cannot write into
another tenant's sensor state or read its posture.
THE PATH. Privileged compared strings.HasPrefix against the raw c.Path() while
fiber routes case-insensitively and ignores a trailing slash, so /V1/KMS/...
reached the key store and matched no grant prefix — the scorer's silence
ALLOWED what fail-closed exists to refuse. cloud.RoutePath is fiber's own
detection path, and every security comparison runs against it on segment
boundaries (which also stops /v1/iam/signupfoo matching /v1/iam/signup).
THE TABLE. edge.Traffic held one map for the whole fleet under a global cap,
so the org that filled it evicted whoever was quietest and that victim's abuse
controls went quiet with no error. State is now per tenant with a per-tenant
bound, reached only by indexing tenants[org]: a cross-tenant read or eviction
is unwritable, not merely refused, and a tenant at its ceiling degrades itself
and reports it (TrafficView.Saturated). One reclaim policy in table[V] —
unexported map, cap as a constructor argument, a live verdict never dropped.
ARMING. Mode was seeded from the platform row for every org, so the one PUT
that arms the anonymous lane armed every tenant silently. It reads the org's
own row now; the platform row governs the lane with no tenant and nothing else.
And it is no longer self-service: writing it takes SuperAdmin whichever row it
lands on, because the subject of an abuse control must not be able to switch
the control off — an org-admin credential is what a stolen key buys.
Also: asking is bounded (MaxScorerCalls in flight, RefusalBusy past it, so a
stuck scorer cannot become an out-of-memory), and the o11y plugin installs the
identity boundary it was reading headers without.
Every fix carries a regression test, and each was mutation-proven: revert the
fix, exactly that test goes red.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
When a held verdict lapsed, the caller went back to being judged only by the
local pattern — so an attacker refused on evidence the SENSOR cannot see got a
minute of enforcement and then walked. The scorer reads the org's own history:
prior accounts on a device, a spend curve, a chargeback. None of that leaves a
trace in a rolling minute of request counts, so waiting for the pattern to
re-trip waits forever.
A lapsed hold now forces the question again on the next request. Once the scorer
allows the caller, the record is released so it stops forcing a screen — the
hold record is exactly the interval between 'a verdict ended' and 'the scorer
said it is fine now'.
The test backdates the deadline rather than sleeping a millisecond, because a
millisecond hold is a race and a test that can flake is a test that will.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
credentialClass read the Authorization header directly, which is a SECOND answer
to a question the identity boundary already answers. callerToken is the one
resolution SanitizeIdentity and CallerBearer share — Authorization bearer, then
X-Authorization, then Basic, then a session cookie — so a client authenticating
with any of the others validated upstream and was then counted here as anonymous,
pooling its traffic under its ADDRESS instead of under itself. That is precisely
the caller the sensor exists to tell apart from its neighbours, and it was the
one it could not see.
Now it reads callerToken, with X-Api-Key after it: that header is not in the
boundary's precedence but several SDKs send it, and a caller the boundary could
not identify is still a caller the sensor must distinguish from the next one.
Three spellings of one key now fingerprint to one caller, which is asserted.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
edge.Store.Mode resolved an empty org to shadow, which meant the anonymous lane
could never be armed — there was no org to arm, so the one lane the gate exists
for was permanently a sensor. It now resolves to the PLATFORM row, the same shape
PerIPRPM already has and for the same reason: a request with no tenant at
evaluation time is governed by the platform scope. A SuperAdmin arms it by
targeting the reserved admin org, whose row IS that scope, so this is one
mechanism rather than a second one for the case that has no tenant.
Also: a liveness probe is never a grant and is never screened, through ONE
predicate both the exemption and Privileged read, so the two cannot disagree
about what a probe is. Without it a fail-closed deployment answered 403 to its
own kubelet on GET /v1/kms/health. Read-only methods only, so a route cannot be
named into the exemption to dodge the gate.
And the exempt list is now only what it has to be: the appeal surface (/v1/risk,
because a refusal nobody can read the decision for is unappealable) and the
internal money plane. /v1/ml and /v1/aml come off it — they are ordinary
surfaces served by ordinary apps, and the gate calls the scorer as a Go function
rather than as a route, so watching them cannot re-enter anything.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two enforcement points now defend the lifecycle, and neither of them scores.
cloud.Decide (risk.go) is the ONE door to /v1/risk: the app that owns that
prefix hands its scoring function to the core with SetRiskScorer, the same
inversion SetObsEventIngest already uses because package cloud cannot import an
app. The fail policy lives in one function and nowhere else — an absent,
erroring, silent, out-of-vocabulary, panicking or over-budget scorer ALLOWS an
ordinary request and BLOCKS a privileged grant, and every answer names why it
was not scored, so an allow that happened because nobody was listening is never
recorded as clean.
AbuseGate sits between the rate limiter and the funding gates and keys on the
CREDENTIAL, which is what neither existing limiter can see: EdgeRateLimit keys
on client IP before identity, ScopeRateLimit on (org, project, service), so a
key lifted out of a CI log and used inside its org's normal ceiling is invisible
to both. It counts, classifies, asks and enforces; a non-allow verdict is held
for a minute so an attack costs one screen rather than one per request, and the
refusal is a 401/403 that AuditTrail already records — one event, one record.
Shadow per org by default, and arming is refused while no scorer is installed.
That is the line between a defense and an outage: failing closed on a component
that was never wired would answer 403 to a deployment's own key store.
Agency reads our own issuance and never the client's self-description. An
attributable machine credential is the agent lane whatever its user-agent
claims; a browser session is human; an unattributable caller is unknown until it
shows an abuse shape, and only then bot. Anonymous is not malicious.
The sensor lives in the leaf edge package so the middleware and /v1/gateway
share one object. Bounded by construction, org leading every key, credentials
present only as a keyed per-process fingerprint that cannot be tested against a
candidate key off-box.
GET /v1/gateway/traffic reports the caller's own lane split, denials, screens
and busiest credentials — the differentiator has to be visible to be worth
anything.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A new app, apps/risk, owning the native leaves of /v1/ml: the per-organisation
feature surface over the event surface every org already writes to, and the
per-organisation models trained on it.
The leaves are on this row and not on ml's because the model is IN-PROCESS
MUTABLE STATE. A manifest row is a binary; if one process learned and another
scored, the two would hold different mass counters and answer one question two
ways, with no error and no log. One owner of the state, one row. ml keeps the
kserve/Kubeflow bridge at /v1/ml/models and /v1/train/*, and longest-prefix
match separates them.
Training is in-process on half-space trees (luxfi/aml pkg/anomaly): no training
pass, no retained sample, no job — the geometry is built before data arrives and
the model IS a set of mass counters, per tenant, ~336 KB, snapshot-restorable on
the per-org encrypted store the rest of the fleet already uses. Attribution is a
counterfactual on the model that raised the alert, so there is no second
explainer to disagree with the scorer. Only four base-free packages are linked;
the image's sqlite gate is satisfied.
The boundary is three arguments, not one rule:
TYPE — the warehouse is reachable from one file, and every function in it that
builds a feature statement takes a tenant, which has no exported constructor
and is minted in one place from the validated principal.
STATEMENT — the tenant is the leading BOUND predicate of every statement; the
window is bound; every column and aggregation resolves through a fixed
allowlist. Nothing user-derived is ever an identifier.
BEHAVIOUR — an organisation naming another's subject reads zero rows, not a
refusal, because a refusal is a probe oracle.
The key is <brand>/<org>, so two brands' identically named organisations are two
tenants, two models and two sets of rows. The source planes are read with the
bare slug they carry and the surface is written with the qualified key, and the
direction is asserted rather than commented.
Cross-organisation learning is aggregate-only and it is ONE table with no tenant
column at all: four quantiles of one dimension over one day, published only when
at least twenty-five organisations and a thousand buckets contributed, enforced
in the statement and again on read. No model reads it. The anonymous lane is
refused at the mint, so there is no filter to forget.
Shadow is the default, per organisation, with stated appetite reported beside
realised. Shutdown snapshots every resident model: this binary rolls one replica
at a time, and a model not written down is a tenant returned to warming — and a
warming model refuses to score, which reads as clean to anything not checking
the refusal.
Nine typed zip ops and one route untyped by design (the real probe, whose 503
carries the report a typed error envelope would drop), held to a closed list by
a test. zipdoc -check green; the app's own subset and the woven openapi.yaml
regenerate from source; 39 tests green, and green under -race.
Eleven mutations proven: drop the org predicate, add an org column to the
baseline, key the model on the bare org, carry the bare org on the transaction,
fold two subject kinds onto one key, write the bare org into the surface, drop
the snapshot tenant check, lower the k-anonymity floor, admit the anonymous
lane, let a second file reach the warehouse. Each fails the test that names it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Inlining has a ceiling and photographs do not. The tasks API takes a 4MB body, so a
4K source (19MB, 25MB base64) could never ride inside a job: the dispatch was refused,
the refusal was reported as "GPU render worker is momentarily unavailable", and the
render waited on a fleet that was healthy the whole time.
An input may now arrive as a URL. The worker GETs it with its own credentials — the
same ones uploadOutputs already uses, and the coordinator resolves the org from the
token rather than from anything the job claims — and stages the bytes into the local
studio exactly as an inlined input is staged. One code path from there on.
Inlined inputs still work; this only adds the other way. Studio starts emitting URLs
once every worker can read them, which is what this commit is for.
Carries the collapse further through the callers: an org is named by a Namespace
value rather than passed around as a bare slug that each site validates its own
way. Landed as found — the agent doing this reached the session limit mid-run, and
the tree it left builds with apps/agents, manifest and the root package all green,
so the work is committed rather than discarded.
It is NOT finished. orm/db still declares its own Namespace string type and
accepts names hanzoai/namespace would reject, so the single spelling holds by
convention until orm takes the value type.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud embeds IAM in the same binary (zip.Graft composes iamserver.NewApp), and
then reaches its own identity code over HTTP: auth_apikey.go dials
http://iam.hanzo.svc/v1/iam/resolve-key to resolve an access key it is sitting in
the same process as. That was never a design choice — PublishableKeyByAccessKey
and UserByAccessKey lived in the iam module's internal/store, so no importer
outside the module could call them, and HTTP was the only door left.
v1.33.43 collapses iam's two store packages into one at pkg/store. Both resolvers
are now importable here, which is the precondition for deleting the hop and
everything hung off it: the Cloudflare 403 on server-side POSTs to the public
issuer, the CLOUD_KMS_IAM_TOKEN_URL → IAM_URL → public-issuer fallback chain in
the KMS broker, and the init() that panicked when iam.hanzo.svc was unreachable
and took api.hanzo.ai with it.
Verified rather than assumed: at v1.33.43 the module's internal/store is empty,
pkg/store carries all 13 files, and a probe compiled inside this module resolves
both symbols. The call sites still use HTTP — this commit only makes the
in-process path reachable.
Note on versions: v1.33.39 carries the same change but numbers BELOW v1.33.42,
so module selection never picks it. It was cut before v1.33.40..42 were noticed.
v1.33.43 is from main, which contains those releases and this change.
The build failure in hanzoai/base@v1.5.11 core (undefined
cgoBuildNeedsSQLiteMathFunctions, a missing sqlite_math_functions build tag) is
present identically before and after this bump. It is not from this change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.0.16 got the run constructed and then lost it in step one. The reusable
fetches its own bin/imgver at the ref it was imported at, and it read that ref
from GITHUB_WORKFLOW_REF — a variable GitHub sets and the forge runner does not.
Under `set -u` the miss is an abort, so the step that had a GitHub fallback
written into it never reached the fallback, and build, test, image, smoke and
deploy all reported skipped behind one line of shell.
v1.0.17 is v1.0.16 with that variable defaulted. It is a new immutable tag, so
the forge has it by the same rule that refuses to move an existing one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two sessions closed the same lane's holes off the same parent, and both halves
belong: the name (an error is `error`, never the caller's class), the family (an
exception is carried on an error and nowhere else), the two value bounds, and
the identity (an unattested subject is namespaced, never a name the org's own
rows join on).
Where they overlapped, the union is the smaller thing. resolveName keeps the
named-default constants, so the one spelling of `error` serves both the route
and the kind table. admitPublic reads canonicalType ONCE and hands that kind to
publicException while publicSubject files the ids — one projection, one pass, no
second place a decision could drift. The header's rule list gains the family and
identity bullets beside the name rule they belong with.
The duplicate name test is dropped for the stronger one: a table of five classes
(the honest TypeError, 3 KiB, the demonstrated 60 KiB, a commerce name, and an
admitted autocapture name arriving through the wrong family) beats a single
string, and it is paired with a fingerprint test proving the class still groups
the issue.
openapi.yaml and the analytics subset are regenerated from the merged source.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The anonymous lane claimed one rule — the SERVER names the row — and the
document shipped that claim to customers. Two holes made it false.
resolveName's error branch fell back to the caller's exception class. It is
shared with the credentialed lane, where naming a row after its class is
right, so `{"type":"error","error":{"type":"…"}}` with no credential put
chosen bytes straight into `name`. That is not a cardinality nuisance:
`name` IS the whole predicate the money lenses count on —
countIf(name = 'order_completed') orders
countIf(name = 'order_completed' OR 'signup_completed' OR 'conversion') conversions
countIf(name = 'click' OR 'ad_click') clicks
— with no second column narrowing them to a vouched-for writer, so a row an
anonymous caller named is a row that counts. On a published-site host it
lands in a real customer's org. An error is now named `error`, and the class
keeps the two places it was always the fact for: the fault's own `class`
column and the fingerprint that groups the issue. Naming the row after it was
a third copy under a third spelling.
The second hole was the projection carrying Error onto every admitted kind.
The projection runs BEFORE the fold, and foldException stamps
attributes['$exception'] onto whatever ingestDecoded hands it, so a `$click`
shipped the caller's message and stack — measured at 32 KiB — into an
interaction row's attributes dictionary. Closed by asking which family the
row is rather than by bounding two more fields: the fields were never the
problem, an interaction is not a fault, and a bounded stack trace on a click
is still a field with no meaning on it.
Two values also carry their own bounds, because a request cap does not bound
a stored value: inside 64 KiB a caller can spend nearly all of it on one $el
or a very wide $path, and this package is forbidden from declaring the DDL
that would narrow the column. The bounds are read off @hanzo/observe's own
output (maxDepth 12, MAX_NAME 80) so no honest annotation can reach them, and
they DROP rather than clip — a clipped label is a different element identity
than the one the visitor touched. A non-string is refused outright instead of
being stringified into an annotation it never was.
The door prose said errors were named by the route. It is corrected rather
than quietly left, because it was the guarantee a live probe disproved.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An anonymous write is admitted on one argument — the SERVER decides what it
says. Two fields made that argument false.
resolveName's error branch fell back to the caller's exception class, and it is
the function this lane leans on for its whole name rule: a credential-less
{"type":"error","error":{"type":"…"}} chose the stored `name`, fifty distinct
values per request, and on a published-site host in a REAL org's partition —
unbounded cardinality in a column every signal here treats as low-cardinality.
An error is named `error`. The class is not lost and was never this column's
fact to hold: it is the fault's own `class`, the first thing fingerprint()
hashes into `group`, and what the error lens reads back from
attributes['$exception'].
distinct_id was carried verbatim. attribute() already refuses to let a
workspace token pin events on a colleague; this lane has no signed subject to
substitute, so it kept the caller's — `victim@corp.com` off the wire landed as
the SAME join key that org's identified rows use, and fabricated interactions
pinned on a named user in every person-level lens. It is filed under the
reserved `$anon:` prefix instead, which holds by publicTenant's own argument: an
IAM subject and an app's person id are never spelled with it. The bytes survive
the prefix, so uniqExact(distinct_id) still counts one browser as one visitor;
the collision does not survive it. An empty id stays empty — a bare prefix would
be a subject naming nothing — and 256 bytes is past any minted id.
Dropping the id instead would have been the other way to break the join, and it
breaks the count with it. Special-casing the site carve would have left the lane
with a rule and an exception; both doors run one projection, so both get the
rule.
The file's own prose asserted both properties while neither held. It now says
what the code does, and openapi.yaml plus the analytics subset are regenerated
from source — which also lands the two insights summaries that were already
drifted from their comments.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The forge and github tips diverged a second time after e60c1f8b: the forge
carried the per-org store naming and the KMS open-handle registry, github
carried the IAM refusal reason, the README correction and the openapi compat
note. Neither side was a superset, so neither could be released — and CD reads
the forge while the author writes to github, so leaving them split means the
thing that ships is not the thing that was reviewed.
The merge is clean; both sides are kept whole. This also unblocks the /v1/summary
CORS fix (fc168a98), which has been merged since 10:43 and carried by no tag:
o11y is the only http-serving plugin main that does not call cloud.Serve, so
EdgeCORS never ran for its public prefixes, and every tag since predates it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The KMS secret store kept its own map of open databases, its own lock and its own
first-touch-opens-once dance, keyed on a slug it computed itself. That is
cloud.OrgStore, written a second time. It is also where a real defect lived: the
key used to be the RAW org, so a tenant path spelling "/orgs/_platform/…" shared
a cache slot with the deployment's own facade store and whichever opened first
served the other. That was fixed by keying on the slug instead and arguing that
SanitizeOrg never emits an underscore — an argument that is true today and is a
property of a slugger somebody could widen tomorrow.
Now the key is the namespace. The facade is the system namespace and a tenant is
an org namespace, so they are different kinds and the collision is not
expressible — there is nothing left to argue about. namespaceFor is the one door
a secret path walks through to become a name, and it decides facade-or-tenant by
the boolean fileOrg returns, never by an org string, so no org a caller can spell
reaches the deployment's partition.
The reader's boot check moves with it. hasRestoredStore walked {DataDir}/orgs
looking for kms.db files, which is a second place that knew where a store lives;
it is now OrgStore.Stored, which asks the registry and shares the registry's own
rendering of that layout. Stored counts the system namespace too, because a
volume holding only the deployment's partition does have something to serve and a
boot check that said otherwise would refuse to start a replica that was in fact
hydrated.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
OrgStore.For took an org and a project and turned them into a name itself, which
meant every one of the forty-seven places that called it was a place where a
name could be built. Answering "could this database have been named by something
the caller sent" required reading all forty-seven, and the answer was only as
good as the last handler somebody wrote.
Now For, Has and Sync take a namespace.Namespace, and OrgDB takes one too. A
namespace has three constructors in this repository and they all live in
orgns.go, so the question is answered by reading the calls to OrgNamespace
instead — one per package, sitting next to the principal it reads. The
subsystems that had no single door now have one: flags, tracker, translate,
experiments, guide, research, books and the webhooks dispatcher each grew a
storeFor whose whole body is "name it, then ask for it", and several of them
shed a repeated nil-check in the process. tracker's agent-PR and GitHub-sink
seams had been reaching past its storeFor into the registry directly; they go
through it now, which is the bug that shape invites not being available anymore.
PlatformDB is gone, because OrgDB(dir, PlatformNamespace(), sub) is the same
call and having both meant the deployment's own partition had two spellings.
Each hands its callback the namespace rather than a directory name, so a
cross-org sweep holds the same value a request does; a directory that no
namespace can name is now reported to the fold instead of silently swept up.
Two things the survey said would collapse do not, and both stay with the reason
written down. agents' mountedStore keeps its MaxOrgLen bound: SanitizeOrg maps an
over-long owner to a forty-nine-byte slug rather than refusing it, so the
namespace constructor would accept a ten-kilobyte org id and the bound is not
subsumed. It is the same bound principal.OrgOf applies at the HTTP boundary — one
rule, two readers — so an in-process seam cannot be granted an org key the HTTP
path would refuse. storeForPublic keeps its Has-then-For: an unauthenticated
caller naming an org must not be able to mint a directory and an open handle for
every name it invents, and For creates on first touch. It now names the database
once and asks both questions about that one value, where before it resolved the
name twice.
flags and admission keep a real org namespace literally named "platform" for
their deployment-wide switches. The system namespace is the right name for that
and would make it unsquattable, but it renders to a different file and moving a
live store is a migration, not a rename. Said so at both constants so the next
reader does not have to rediscover it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The key resolver read IAM's envelope for a principal and threw the rest
away, so a refused key produced a bare nil and every surface downstream
could only repeat IAM's generic "the entity does not exist" — which is
what told a holder with a REVOKED key to go looking for a deleted org.
lookup now records the `code` beside the (nil) principal, and
RefusalForKey is the door a user-facing surface asks why. It shares the
resolver's existing cache, so asking costs no extra IAM call: the reason
is a by-product of the resolution that already happened.
Resolution is unchanged. A refused key is still nil, still anonymous,
still fails closed — only the diagnosis is added.
KeyHint is the ONE way a key is named in a log or an error: prefix only,
never the credential.
Also makes the zero cache usable — put() creates its map on first write.
A partially-constructed iamKeys (any caller naming only the caches it
cares about, which every test does) hit a nil-map panic otherwise.
Architecture. The headline said "one Go binary ... every subsystem mounted into a
single multi-org process". That binary is gone — it linked every subsystem's
graph into a ~3105-package build and went with `apps.Wire()`. What ships is
`cmd/cloud`, a light host that links zip + manifest + webui and NOTHING else,
with each of 116 subsystems as its own `plugin/<name>` process, started lazily
on the first request that reaches its prefix; 4 (pubsub, kafka, o11y,
catalogsync) own a listener and start with the host. Counted in
manifest/apps.go, which is also the answer to the next defect:
`apps/apps.go:Wire()` was named as "the one ordered list of everything mounted"
and that file does not exist in this repo. `manifest/apps.go` is the list.
The CLI section. Every verb in it is real for `cmd/hanzo` — I read the cobra
tree — but the section sat under a Quick start whose install line is
`curl hanzo.sh | sh`, which installs the RUST CLI, where `login`, `whoami`,
`apps`, `deploy`, `build` and `completion` do not exist and are read as a task
for the coding agent. Two programs answer to `hanzo`; the README now says which
one it is documenting and what the other one does with those words. It also had
the delegation backwards ("the Rust CLI installed alongside as hanzo-node"):
the Rust CLI is the primary `hanzo` on a developer's machine and writes
`hanzo-node` as a symlink to itself; the Go binary is the one that delegates.
Listed the eight verbs the section had omitted (auth, logout, run, agent, bot,
engine, runner, link/unlink, security, version).
Links. Every URL fetched. The HIP-0106 link 404'd and named a document that
does not exist — the real file is hip-0106-hanzo-plugin-contract.md, so the
line "Per HIP-0106" was pointing at a title someone invented. Same for the
Specs list: HIP-0037 is not in hanzoai/HIPs at all, HIP-0129 is "Eval — the
Judgment Plane" not "Open Cloud Planes", HIP-0302 is "Encrypted SQLite
Replication". Every entry now links its real filename and all 8 resolve 200.
`hanzoai/zip` 404s AND is the wrong repo — go.mod imports
github.com/zap-proto/zip v1.18.22, which is public.
Image tag: `:v1.801.206` was 149 builds stale. Any number written here rots, so
the try-it line is `:latest` (manifest 200) with a sentence saying to pin for
anything real.
docs/bring-your-gpu.md printed `hanzo login` with no hint which binary, and
shipped two `—` escapes as literal text.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The org-DB layer knew where a store lived twice. filepath.Join built the local
path and org.DBPath built the object key, from the same slug and the same
subsystem, in two functions that had to agree and nothing made them. A fact with
two homes is a fact that can disagree with itself, and the day the two disagree
a store hydrates from one place and ships to another.
So the fact gets a value. github.com/hanzoai/namespace names the one database an
entity's data lives in, and nsKey renders that name into the place its subsystem
file goes — under DataDir on disk and byte-identically as the object key. One
rendering, two consumers, so the file and its remote slot cannot drift.
The names are built in exactly one place, orgns.go, and OrgNamespace is the only
door a principal-supplied org can walk through. It still folds through
SanitizeOrg, which is what makes the map injective; namespace.Org then admits
only [a-z0-9][a-z0-9_-]* and folds case, so the two ids SanitizeOrg would have
turned into a directory called "-<hash>" are refused at the door instead of
written down. nsOnDisk is the one construction that does not start at a
principal — it reads back the segment OrgNamespace wrote — and it is a named
function rather than an inline call so that asymmetry stays greppable.
OrgStore now caches by that namespace rather than by the resolved path. The
namespace is the fact and the path is a rendering of it, so keying on the
rendering allowed two entries that meant one file, which is two open handles on
one SQLite and something the at-rest cek layer does not support. Keyed by the
value, that state cannot be constructed.
The platform partition stops being an argument about runes. It was disjoint from
every tenant's file because its slug carried a "_" and SanitizeOrg never emits
one — true, but a property of a slugger somebody could widen. It is now the
system namespace, a different kind entirely, and no edit to a slugger can undo
that. On-disk it still renders as orgs/_platform, so every file that exists keeps
its path.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The declaration now rides all the way out: re-weaving with Fold's carry-through
puts `compat` on the 23 legacy IAM addresses in openapi.yaml, which is what
hanzoai/openapi's merge reads to keep `get-users`, `add-application`,
`set-preferred-mfa` and the pre-plural `application` out of the published
document. They stay SERVED; they stop being taught.
Three places read tags to answer "which products does this document publish" —
Measure's floor ratchet, the weave's ownership claims, and the document's own tag
list — and all three iterated Tags directly. That worked while every tag WAS a
product. With a second, orthogonal tag it does not: the first re-weave invented a
23-operation product called `compat`, ratcheted the floor onto it, and offered a
doc site a heading nothing answers to. The floor gate caught it on the next run,
which is the gate working.
So the distinction has one name, openapi.Products, and the three callers ask it
instead of iterating. Measured after: 23 compat tags on operations, 0 in the
document's tag list, floor.json unchanged.
Only apps/iam's subset is regenerated here, because it is the only one this change
affects. `make describe` across all 116 is red on main for an unrelated reason —
apps/storage carries an openapi.Describe keyed to POST /v1/s3, a route it does not
serve — so a full regeneration would have folded that lane's work into this commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every cloud release since the consolidation died before it existed. The gate
resolves hanzoai/ci at `v1`, and on the forge `v1` still names a commit whose
tree is `.github` and README alone — no `.hanzo/` at all. Dispatch answered
`PrepareRun: InsertRun: read hanzoai/ci@v1:.hanzo/workflows/build.yml: object
does not exist` and wrote no run row, which is the absent-not-red case this
job already documents, reached through the alias rather than the path.
The alias cannot heal itself: sync-from-github fast-forwards branches but
refuses to move a tag that exists, so a rolling alias is frozen at whatever
it pointed to when the forge first saw it.
v1.0.16 is the commit github `v1` resolves to today — same tree, byte-identical
build.yml — and the forge already carries it, so the gate is pinned there
instead. An immutable patch tag also states which contract is being called:
`tests` is an input from v1.0.12 onward, and v1.0.11 and the current v2 do not
take it, so the alias was hiding a signature the caller depends on.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The forge and github had diverged two ways: three commits existed only on
the forge, ten only on github. CD builds the forge, so a release cut in that
state would have shipped the per-org agents isolation and silently omitted
the authz fix that makes platform authority a membership rather than a
position. Neither side was disposable and neither could fast-forward onto
the other, so this merge descends from both and pushes as a fast-forward to
each.
The split is manufactured by an asymmetric remote: in the shared checkout
origin FETCHES github and PUSHES the forge, so pull-commit-push lands work
on the forge alone while github moves on. Reconciling the tips clears the
release; making the remotes symmetric is what stops it recurring.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The golden is derived, so a rebase conflict in it is not a conflict — it is two
regenerations of one thing. Re-woven from the subsets on the merged base.
1396 paths / 1938 operations, ZERO with no summary. The 50 that carry a summary
and no long description are all hanzoai/ai (48) and its router (2), whose emitter
is a separate lane's; nothing here invents a sentence for them.
Floor ratchets 1384 -> 1396 paths, iam 179 -> 192 operations.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The producer gate has been refusing apps/iam since it landed — 88 operations with
nothing to say — and the fix was never in this repo. IAM's untyped routes are
IAM's: the OIDC surface, SCIM, MFA, service accounts, memberships, the legacy read
aliases. openapi.Describe is this repo's seam and cannot reach them, and a table
of strings here for routes cloud does not own is the duplication this package
exists to remove.
zip v1.18.22 lets those routes carry the doc comment on their handler, and exports
it. From now asks there — LAST, after this app's own declarations, so a host that
has something to add about a route it mounts still wins and the owning service's
sentence is what fills the silence.
plugin/iam 155p/182o, 88 with no summary, 182 with no description
-> 167p/195o, 0 and 0. MCP tools 94 -> 98, all described.
openapi.yaml 1220 paths / 1628 operations, 0 without prose.
`make -C apps/iam describe` goes green for the first time since the gate existed.
Every sentence traces to a Go doc comment in hanzoai/iam — none was written here.
Requires hanzoai/iam v1.33.42 and zap-proto/zip v1.18.22.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
AppendEvent allocates a session's next sequence number by reading MAX(seq)+1 and
then inserting, and that read-then-write is atomic only because the org's file is
served on one connection. This package used to set that itself, two lines above
the code that depended on it. It no longer does — cloud.OrgDB opens the file now
— so the guarantee travelled into another package while the code resting on it
stayed here, which is exactly the arrangement where someone eventually raises a
connection limit for a good reason and silently breaks a subscriber's cursor.
Seq is a resume cursor: ListEvents and ListControlAfter both read seq > since. A
duplicate makes a watcher skip an event and a gap makes it stall, and neither
shows up as an error anywhere — the writes all succeed. So the assertion has to
be about the sequence a concurrent writer actually observes, which is what the
first test does: eight writers, two hundred appends, and the result must be
exactly one through two hundred with nothing repeated and nothing missing.
The second test states the underlying property in one line against cloud.OrgDB
directly, so when the first one starts failing the reason is already written down
rather than something the next person has to rediscover.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`/v1/iam/get-users` and `/v1/iam/users` are two live routes. Nothing in the route
table relates them — only the code that registers them knows one replaced the
other. hanzoai/iam says so, tagging its fifty-one inherited entity verbs
(`get-users`, `add-application`, `set-preferred-mfa`) and the singular
`application` address it had before that kind was pluralized. Fold threw the
declaration away: `op.Tags = shape.Tags`, with a doc comment asserting "zip's
per-op tags are a different axis and cloud registers none". That was true when it
was written and is not now.
The cost of dropping it is downstream and customer-facing. hanzoai/openapi's
merge reads `compat` to keep a legacy spelling out of the published document;
without it every one of those operations reaches a customer twice — two SDK
methods, two docs entries, two `hanzo iam` commands, with nothing saying which is
the one to use.
So Fold keeps the product tag FIRST, because that is the axis every generator
files an operation under, and appends `compat` when the typed op declared it. The
name is exported (openapi.Compat) because it is a term two repos have to agree
on, and a test pins both halves: the canonical address carries the product tag
alone, the legacy one carries the product tag and then the declaration.
Nothing about what cloud SERVES changes. `make describe` is red on main today for
an unrelated reason — apps/iam projects 88 undescribed operations, the seam
tracked separately — so the woven openapi.yaml is regenerated by that lane, not
this commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Records the three findings behind the authorization fix, in the repo's own doc
rather than a summary file: what the two admin scopes mean and where they are
asked, that no isAdmin CLAIM exists in either token, the positional-read defect
that made the reserved org unreachable in practice, why honoring a signed
membership widens nothing, and the two hollow surfaces (a Visor version skew that
read as an empty fleet, and a documented list projecting the CR kind with zero
instances).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/deploy/applications projects `hanzo.ai/v1` App CRs. Production holds ZERO
of them — the CRD is served and nothing has ever created an instance, which this
package already discovered twice from the other direction (pin.go and release.go
both record "The CRD kind exists, but there has never been a `cloud` CR for it to
patch"). The 328 applications actually deployed are `apps.hanzo.ai/v1alpha1`
Applications reconciled by Hanzo CD, and they were reachable only through
/v1/deploy/gitops. So the endpoint an operator reaches for answered `items: []`:
complete, correct, and useless.
Two kinds, one word, and the documented one was the empty one.
The fix is NOT to pick a kind. They are not interchangeable: `hanzo.ai/v1 App` is
the TENANT plane — per-tenant namespace, org-labelled, what a customer deploys —
and the CD Application is the PLATFORM plane, what CD reconciles for the estate.
Collapsing them would either leak the fleet to tenants or lose the tenant model.
The answer is to answer for the CALLER, which this handler already knows how to
do, because scope.namespaces() branches on exactly that distinction.
So: the tenant path is untouched and gains nothing, and a platform SuperAdmin
additionally gets the CD plane folded in — the same source and the same gate as
/v1/deploy/gitops. This widens the ENDPOINT, never the audience; no CR becomes
visible to anyone who could not already read it. TestApplicationsDoesNotWidenForA
Tenant is the half that keeps it that way.
projectCDApp goes through observeGitOpsApp, the one reading of a CD Application in
this package, so the list and /v1/deploy/gitops can never disagree about an
application's sync state, health, or applied revision. An absent CD CRD contributes
nothing rather than erroring, matching how gitops already treats that cluster.
The projection is also more honest than what it replaced: this endpoint already
presents itself as an argoproj.io/v1alpha1 ApplicationList, so emitting real CD
Applications in that envelope beats relabelling App CRs into it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/k8s/clusters answered 200 {"clusters":[]} to an operator running eight
clusters. Production Visor sits four tags behind the commit that introduced
/v1/k8s/clusters (deployed v1.108.12; the route arrives in v1.108.13), so every
call 404s; listK8sClusters folded that to a Warn and returned the BYO half, which
is empty. On the wire "the provider is down" and "you own nothing" were the same
three bytes, and the only trace was a log line nobody reads during a fleet check
— which is exactly when you are least able to tell the difference.
The fold itself was right and stays: a page that 502s on an optional provider is
worse than one that shows what it can. Only the silence was the bug. Every folding
surface now reports the source it could not reach, so an empty list means "you
have none" if and only if `degraded` is absent.
The field is ADDITIVE and omitempty, so a healthy response is byte-identical to
what it was and no consumer has to change to keep working.
terse() exists because the raw error is not fit to return. The client formats a
non-2xx as "visor: upstream %d: %s" with a snippet of the response BODY, and Visor
answers an unknown path with an HTML error page — so the unabridged string is a
DOCTYPE and a stylesheet, which was already riding into every log line carrying
it. First line, markup dropped, hard-capped; the exact production string is a test
case.
Applied at both cluster surfaces (/v1/k8s/clusters and /v1/clusters), which share
the response type. The machines/GPU fold (managedMachines) has the same shape and
is not covered here — it feeds three surfaces through a different type, and a
partial fix that leaves the same trap in a sibling is worth doing deliberately
rather than in passing.
This does not by itself make the k8s surfaces return data: that needs a Visor
release. No image exists past v1.108.12 — ghcr.io/hanzoai/visor:v1.108.13 is
not found — so the routes exist in git and in no running binary. What this
changes is that the gap now announces itself instead of reading as an empty fleet.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`app.All("/v1/*")` is one entry in this process's route table and 190 patterns in
the registry mounted behind it. Reading the router alone published
`/v1/{wildcard1}` and seven operations for the whole model API, so no generated
SDK and no MCP tool list carried chat completions, and hanzoai/cli grew a
`{wildcard1}` command because that is what the document named. Document's own note
has always stated the limit — "this document can name the prefix and nothing under
it". This is that limit, closed.
The door's owner is the one that mounted the thing behind it, so it is the one
that can say what is there: same process, same objects, same instant.
openapi.Front declares that; Mounted projects a sub-app through the same Spec the
host's own document is, Table projects a foreign registry — its route table AND
the sentence each route owes a reader. hanzoai/ai's routers.App.Patterns and
routers.Prose are read out of the pinned module at describe time: no network, no
vendored copy, no second list, reproducible from a checkout and a go.mod.
/v1/{wildcard1} x 1 path / 7 ops -> 177 paths / 323 ops, 0 undescribed
fleet 1208 -> 1384 paths, 1636 -> 1952 operations, 149 -> 179 products
(chat, models, embeddings, messages, rerank, images, audio,
memory, rag, router, videos, finetune, traffic, ...)
wildcard path keys 25 -> 24; the only path this commit REMOVES is /v1/{wildcard1}
Structure comes from From — the same builder the router projection uses — so a
relayed operation earns the same operationId, path parameters and product tag a
native one does. Only the prose is the registry's, because only the registry has
it. It earns no schemas: App.Router names a controller method and the body types
are read off the beego context inside the handler, never declared. That is
per-operation typing work in hanzoai/ai and a different job from this one.
Same laws as every other seam here. Register declares bodies and renders only on a
live route; Describe declares prose and renders only on a live route; a relay
declares the routes behind a route and renders only on a live door. What it adds
is four refusals, each naming the source so a wrong placement is traceable to the
repo that registered it: a registry that published nothing (the shrink), a
registry that could not describe itself (the outage), an operation outside its own
door (the routing bug), and a registry that says nothing about a route it serves —
the same law openapi.Complete already holds an app's own operations to, applied
where another repo's surface enters this document. A name collision goes through
the SAME noun gate the weave uses, extracted to `nouns` and now called by both,
because one schema name meaning two things is one law whether the claimants are
two apps or an app and the registry behind its door.
A door yields where the router says it does, in both directions. Inside a binary,
a specific route the host registered wins the ADDRESS — but not the SENTENCE. A
host route that says nothing is one the registry behind the door still answers:
hanzoai/ai promotes /v1/models and the enso access pair onto this router at their
real patterns, pointing at the same relay the glob uses, so the words belong to the
handler either way and taking them from the registry is what keeps them from being
written a second time here. The three hand-written Describe blocks 03091a0 added
for exactly those addresses are deleted with the other five. A host route that DOES
say something has its own handler and its own prose (apps/o11y's /v1/o11y/scope)
and is left exactly as it is.
Across the fleet, Weave now resolves exactly one overlap — an operation that
arrived through a door loses to one that did not — and it reads that off the data
rather than preferring an app, because a relayed operation names its own registry
in x-app and a direct one does not. Two specific claims, or two doors, are still a
refusal. And /v1 is a REMAINDER, not a namespace: ai's row is last in
manifest.Apps, so manifest.Elsewhere answers which of its registrations the fleet
delivers to a sibling. Those are not published, because a path no request reaches
is a phantom and a router-derived document exists to make phantoms impossible.
x-app is PROVENANCE, on all 1952 operations: the registry that registered this
one. For an app's own route it is the app, so the code is apps/<name> here; for a
relayed route it is the module behind the door, so the code is that repo. Written
once by whichever producer knows, never overwritten.
openapi/floor.json is THE RATCHET. Every gate here compared the document to
something that moved with it — the weave compares two derived artifacts,
surface-check regenerates them both from source — so a surface could lose products
with everything green, and has: 46 products in one bad reading, eight ingress paths
in a stale subset. The floor is the counts, per product, and a regeneration that
comes in under any of them fails, names the deltas, and writes nothing. Proven on
the shape it exists to catch: deleting three paths from one subset reports
`chat 2 -> 1`, `embeddings 1 -> 0 <- the whole product`, `models 3 -> 2` and
refuses. A deliberate deletion lowers it by hand in the same commit, where a
reviewer sees the number go down next to the reason — and the floor is born
without `edge`, because 34c13d3 deleted /v1/edge/nodes on the stated grounds that
"edge" named a position rather than a route.
All eight hand-written Describe blocks in apps/ai are deleted, and that deletion is
the point: hanzoai/ai can hand over the API itself, so nothing here has to describe
the door and nothing here can be wrong about it.
Requires hanzoai/ai v1.832.7 (cloud pins v1.832.8), which lifts each route's sentence from the doc
comment on the handler its registration names and holds the bijection — every
route has a sentence, every sentence has a route — in its own tests.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`hanzo status` could not see the cloud because the token it carries could not
pass the gate. The reason was not the token.
SanitizeIdentity granted platform sudo on `claims.homeOrg() == adminOrg`, which
is `Claims.Orgs[0].Org` — a POSITIONAL read. IAM's MemberOrgRefs always writes
the user's own org at index 0 and appends every granted membership after it, so
that test could only ever be true for someone whose USER ROW lives in the admin
org. An operator anchored in a brand org and GRANTED admin-org membership — the
deliberate, signed, revocable way operators are actually made — was structurally
unreachable by it. z@hanzo.ai carries orgs:[{hanzo,admin},{admin,admin},
{lux,admin},{pars,admin},{zoo,admin}] and was refused every superAdminOf surface
because `admin` sits at index 1.
This widens nothing. The authority was already signed by IAM and already guarded
on the write side: memberships.mayGrant refuses to create a membership into a
reserved org unless the caller is already a SuperAdmin, on the stated grounds
that it "seeds admin-org (SuperAdmin) tenancy". IAM protects the grant as
platform authority; this gate now honors it as platform authority. The two
agreeing is the fix.
Both admin scopes now ask the predicates hanzoai/authz publishes — the issuer's
own statement of what its claims mean — narrowed by the one denial only cloud can
make (the per-org KMS-sync machine, named by its owner-bound audience; authz
decides machine-ness from an empty membership set, which a machine carrying
memberships would defeat). cloud's private re-derivations are deleted rather than
kept beside them, because two readings of one claim is the condition that package
exists to end.
The org-admin bit was the same defect one scope down: `claims.IsAdmin ||
isOrgAdmin(...)` is an UNSCOPED disjunct, so a token carrying the bit would have
been org-admin in whatever org it switched INTO. authz.Claims.OrgAdmin scopes it
to the home org. The term is inert against IAM today, which is exactly why it
could sit there reading wrong — a dead term cannot fail a test. Closes the
standing "scope the legacy isAdmin bit to home org" item by adopting the
published predicate rather than patching the local one.
There is no isAdmin CLAIM in any of this, in either direction. IAM mints one into
NEITHER token: internal/oidc/jwt.go's Claims struct has no such field, and
(*Signer).claims is the single place an Identity becomes a claim set, so the
access token and the id_token differ only in aud/tokenType/nonce. The bit exists
only as a user-row column that userinfo and whoami report in a response body.
Copying it into the access token would have changed nothing, because no gate
reads it.
The adminOrg parameter is gone. The reserved org is the ISSUER's constant — IAM
hardcodes `owner == "admin"` — so a consumer-side knob could only ever let cloud
disagree with the contract it is reading. This file's own test doc asserted
"Hanzo pins it to hanzo", which, had anyone set IAM_ADMIN_ORG that way, would
have handed platform sudo to every member of the hanzo org while IAM considered
none of them a SuperAdmin. Production never set it, so the default carried the
truth by luck.
TestPlatformSudoIsMembershipNotPosition pins the fact end to end through real
JWKS-validated tokens, including z's live membership set verbatim. It fails
against the positional predicate and passes against the set one; the negative
cases (admin of every brand org but no reserved membership, a look-alike "Admin",
an empty set) pass under both, which is how the change is shown to widen nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every org's agents, runs, sessions, events, targets and claim keys lived in one
{DataDir}/agents.db opened on a single connection, isolated by an org column and
a WHERE clause on every statement. That was correct — no handler skipped the
predicate — but correct by repetition: forty-eight store methods each had to
remember, and the only thing standing between a tenant and its neighbour was
that nobody had yet written the one query that forgot. It also meant every event
append in the fleet queued behind the same connection.
An org's records now live in the org's own file, opened through cloud.OrgStore
exactly like the fifteen other per-org subsystems, under that org's own cek key.
A query cannot reach past the database it runs in, so the boundary is structural
rather than remembered. tenancy.go is the whole of the argument and is the only
file allowed to resolve a store; a test asserts that over the package source, so
a handler added next year cannot quietly name a database from a path segment or a
request body.
The org predicate stays on every statement. It is no longer what provides the
isolation, it is what makes a mis-resolved store answer with nothing instead of
answering with somebody else's rows.
One read genuinely comes from an unauthenticated caller: the public build page is
addressed as {org}/{project}. Resolving a store the normal way would let a
stranger mint a directory and an open handle for every name they invented, so
that route goes through storeForPublic, which refuses to materialise anything and
can only reach a database a real tenant already wrote to. cloud.OrgStore gained
Has for it — the question "is there anything here" without the side effect of
creating it.
The scheduler's long-running sweep and the published-builds list are the only
reads that legitimately span tenants, and they now say so out loud in crossorg.go
by folding over every org's file rather than by omitting a predicate.
A deployment that already has the single agents.db is fanned out into per-org
files once, on mount, before any route exists to read them. It fails the mount if
it cannot finish, because an empty registry served over live rows is worse than
not booting, and it leaves the legacy file exactly where it is so the upgrade
stays reversible. The tests seed a pre-split file and read every kind of row back
through the new one — including the event sequence numbers, which are a live
subscriber's resume cursor, and the claim keys, which are live capabilities held
by machines that must keep authenticating after the split.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The column was added to the insert and the scan but not to UpdateSession, so a
PATCH that published a terminal was accepted, echoed back in the response, and
then dropped. The session read as if it had never published one, and the console
had nothing to frame — a silent write, which is the worst kind.
The test covers the update path specifically; the existing one only ever inserted.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The header described the probe it used to run — "every literal GET", 490
addresses. reach.py probes 748 now, 194 of them parameterised, and a ledger that
misdescribes its own instrument is the same class of thing it exists to catch.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three operations were live, authenticated and answering in production while the
document showed /v1/{wildcard1} where they are. No generated SDK offered the model
catalogue, no MCP tool listed it, no CLI command reached it, and the enso access
flow — how a caller asks for a limited-preview model — could not be found by
anyone reading the API.
They were dark for a second reason too, now gone: a Cloudflare worker claimed
/v1/models* and /v1/pricing* by naive prefix match and answered from a ten-key KV,
so GET /v1/models/{model}/access 404ed at the edge and POST was refused outright
by its `access-control-allow-methods: GET, HEAD, OPTIONS`. The origin never saw
either. Verified live before this commit: /v1/models, /v1/models/enso/access and
every /v1/pricing/* now answer `server: hanzo`, `x-api-version: v1.801.350`, with
no cf-ray at all. One path, one owner, and the owner is the one that can describe
itself.
Naming is the last step. hanzoai/ai v1.832.8 promotes these addresses onto the
host router at their real patterns, pointing at the SAME relay the /v1/* glob
uses, so the request takes the identical path through the identical handler and
only the description changes — not a second implementation of /v1/models, a
description of the one that exists. Verbs come from its live route table and the
promotion list carries patterns only.
Document: 1208 -> 1210 paths, 1615 -> 1618 operations, nothing removed. The
wildcard floor is unchanged at 25 — this does not spend it, the remaining ~190 ai
addresses still need the host to refuse a duplicate claim at compose time before
they can be promoted. reach.py probes both new addresses and both answer.
The Describe keys use the ROUTER spelling (:model), not the document\x27s ({model}),
because that is what the registry is keyed on — the same reason /v1/* is written
there rather than /v1/{wildcard1}.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TWO holes in the emitter, both of them "the document publishes an address the
router does not serve" and neither covered by anything.
FIRST: reach.py probed 553 of 1208 path keys. Every parameterised path — 432 keys,
36% of everything published — was checked by NOTHING, and the reason it was
skipped was real: a parameterised path probed with a made-up value returns a
correct resource-404 from a working handler, and mistaking those for dead routes
had already produced five "defects" that were all fine.
But a resource-404 and a router miss are DISTINGUISHABLE. zip/fiber\x27s router-miss
body is exactly `404 page not found`; anything else proves a HANDLER RAN, which
proves the route exists and only the id did not. So they are probed now, and the
verdict has two branches for one question:
literal ANY 404 is dark. No made-up value to blame, and the strict rule
is the one that caught the edge worker, whose 404 carried JSON.
parameterised Only the router-miss body is dark.
Wildcards are dropped, not filled: a catch-all\x27s sentinel fill asks about a path
nothing was meant to serve, and measured, they are the ONLY two "misses" a naive
sweep reports. Coverage goes 553 -> 748 addresses. check_instrument() proves the
oracle on every run, including the case that matters most — a JSON-bodied 404
that a body-only rule would call healthy — because an instrument checked
somewhere other than at the moment of measuring is an instrument nobody checks.
SECOND: SpecConfig hands a describe run zero values so "a published spec is a
function of the code alone". *Config is not the only door: 372 os.Getenv /
os.LookupEnv calls in apps/ and clients/ read straight past it. A route
registered under one of those is published or omitted on a difference the
document cannot express, and the dangerous direction is the quiet one — a route
registered when the var is EMPTY is published by the describe run and never
registered in production, a 404 on a published address in eight SDKs, the MCP
list and the CLI at once. reach.py cannot see it until after a release ships.
The count is ZERO today, which is exactly why it is worth writing down: the
document is honest about this by accident, and nothing stopped the 373rd env read
from being the one inside an `if` around a route. The test walks the AST, catches
the direct shape, and says plainly what it cannot see.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The fanout counted HTTP status codes. 204 means GitHub recorded the event; it
says nothing about a runner ever picking it up — and on this fleet that gap is
where every projection has been disappearing.
Measured:
github.com self-hosted runners: 0 at the hanzoai org, 0 on hanzoai/cloud,
0 on hanzoai/cli. Every client caller declares
`runs-on: [hanzo-build-linux-amd64]`, a pool registered to git.hanzo.ai.
hanzoai/python-sdk CI/CD queued since 2026-08-01T18:19, never started.
hanzoai/java-sdk CI/CD queued since 2026-07-31T23:36, never started.
go/rust/kotlin/cpp startup_failure, every run.
git.hanzo.ai, which HAS the runners, has no repository_dispatch at all:
POST /v1/repos/hanzoai/cli/dispatches -> 404
POST /v1/repos/hanzoai/cli/actions/workflows/cicd.yml/dispatches -> 401
so `workflow_dispatch` is the forge's dispatch verb and this event cannot
reach it. Two of the nine — hanzo-kotlin/sdk and hanzo-cpp/sdk — have no
forge repository under either name at all.
So the car now waits, up to eight minutes, for a run of OUR event created at or
after the dispatch to leave `queued`, and names every projection that accepted
the event and never ran it. It will be RED today, on most of the nine, and that
is the correct reading: a release whose clients cannot be regenerated is
incomplete. A receipt reading 9/9 for nine jobs that will queue until they are
garbage-collected is the same class of lie as a green gate that runs no tests.
The remedy is one decision, not a workaround: the projections build where the
runners are. Either the nine repos become forge-native (a pull-mirror is
read-only and the client lane commits its regenerated projection, so a mirror
cannot host it — hanzoai/mirrors' NATIVE table is where that is declared, and
only python-sdk is in it today), and this car POSTs the forge's
workflow_dispatch endpoint with a forge token; or github.com gets runners for
these repos. Until then the train tells the truth about it every release.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A logged-out visitor's interactions were discarded behind a 200. @hanzo/observe
emits every autocapture through capture(), so a $click is type:"event" — and the
`event` KIND is the whole custom commerce/billing/metering surface, refused here
since the beginning. Widening the kind would have handed that surface to anyone
on the internet, so it stays refused.
The lane admits a closed set of NAMES under that kind instead. Both families it
now carries are one rule — THE SERVER NAMES THE ROW — and publicName is where it
is decided: pageview and error keep taking their name from the route, and an
autocapture takes its name from a table declared in the file. The lookup folds
case and space and stores the TABLE'S VALUE, so an unattested caller can
introduce neither a new name nor a second spelling of an admitted one, which is
the cardinality property the narrow door was protecting.
The property bag is projected by the same argument: the @hanzo/observe annotation
and nothing else. That family is the one that widens nothing — annotationOf lifts
those keys into the `el` tuple and attributesOf skips the same set — so a click
carries the element identity a heatmap is drawn from while the attributes
dictionary still holds only what the server put there. A click with a url and no
element is a count, not a heatmap.
No per-value bound comes with it: maxPublicBytes is the one bound on anonymous
caller bytes and it already governs url, path, referrer and the rest.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The fanout car dispatched to hanzoai/<repo> for all nine projections. Five of
them are not there anymore. Measured against api.github.com — POST to
/dispatches with an event type nothing listens for, so the probe is inert:
hanzoai/go-sdk 307 hanzo-go/sdk 204
hanzoai/rust-sdk 307 hanzo-rs/sdk 204
hanzoai/kotlin-sdk 307 hanzo-kotlin/sdk 204
hanzoai/cpp-sdk 307 hanzo-cpp/sdk 204
hanzoai/docs 307 hanzo-docs/docs 204
hanzoai/python-sdk 204 hanzoai/js-sdk 204 hanzoai/java-sdk 204 hanzoai/cli 204
A redirect is not a delivery. GitHub follows a rename for GET, but a dispatch
POST answers 307 and drops the body — so the first real release would have gone
out with four clients regenerated and five still describing the previous one,
and the receipt would have recorded 4/9 without saying why. Curl still gets no
-L on purpose: a projection is addressed where it lives, and a 3xx now prints
"this repo was renamed or transferred; put its new address in this list" rather
than being papered over.
The credential changes shape for the same reason, so it is named correctly
before anyone tries to mint it. Nine repos across SIX owners — hanzoai,
hanzo-go, hanzo-rs, hanzo-kotlin, hanzo-cpp, hanzo-docs — and a fine-grained
PAT is scoped to one owner. FLEET_DISPATCH_TOKEN is a classic PAT with `repo`,
or a GitHub App installed on all six. The old wording would have produced a
token that works for four repos and 404s on five.
FLEET_DISPATCH_TOKEN still does not exist in KMS, and the car still fails loudly
naming the exact path rather than skipping.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`hanzo status` was implemented twice and the two answers disagreed. This
binary's version called exactly one endpoint, GET /v1/fleet/workers, whose
server side returns only BYO machines that dialled in. So the CTO ran it
against a fleet of eight clusters, forty-nine nodes and three hundred and
thirty-nine deployed applications and was shown two laptops — with nothing on
screen to suggest anything else existed.
The fabric CLI's `status` composes the cluster list, the application list and
the same worker list, leads with whatever is actually broken, and renders the
identical per-machine compute block. It is a strict superset, so the fix is not
to reimplement it here — it is to stop having a second one. `newStatusCmd` and
`runFleetStatus` are deleted; `status` now routes through Passthrough like every
other verb this binary does not own.
The router derives itself from the command tree, so deleting the command IS the
deregistration. `status` moves to delegatedVerbs in cli_test.go, which pins the
delegation: registering a second status here fails the test rather than quietly
re-forking the fleet view. statusDot went with it; fleetWorker stayed, it is the
shape registration round-trips.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Fourteen /v1/pricing/* addresses were dark when the ratchet was written — a
Cloudflare worker intercepted /v1/pricing* and /v1/models* by naive prefix match
and 404ed anything its ten-key KV lacked, while the origin implemented every one
of them twice and never saw the request. Re-probed against the deployed
release's own document and host: all fourteen answer 200 from server: hanzo. The
origin took them back, and one path has one owner again — the one that can
describe itself.
The other three (/v1/billing/{gpu-eligibility,payment-methods,payment-config})
were a stale subset publishing addresses one binary had already renamed;
regenerating the document retired the path keys, so there is nothing left to
probe.
The file stays, empty, with the rule and the history in it. A ratchet that is
deleted when it empties is a ratchet nobody re-reads when the next address goes
dark.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`hanzo --version` answered `hanzo dev` on every build ever made, the installed
one included. cmd/hanzo declared `var version = "dev"` documented as "overridden
at build time via -ldflags", and `-X main.version=` appeared NOWHERE — not the
Makefile, not mk/*.mk, not the workflow. cmd/hanzo had no make target at all, so
the flag had no place to live even in principle.
The fix does not depend on anyone remembering a flag. resolveVersion is a ladder,
most authoritative first: the stamped tag; info.Main.Version from ReadBuildInfo
(the toolchain derives a pseudo-version from the checkout by itself, so a bare
`go build ./cmd/hanzo` already knows its commit); a 0.0.0-dev+<12-char-sha>
[-dirty] synthesised from the vcs.* settings, which is the rung that carries an
older toolchain where Main.Version is "(devel)"; and only then "dev". Resolved
ONCE in main and assigned to cli.Version — cli holds the answer, it never
re-derives it. readBuildInfo is a var so every rung is reachable from a test:
which rung fires depends on the toolchain, and a rung nobody can reach is a rung
nobody has checked.
The stamp is wired too, so a release carries its exact tag. `make hanzo` — the
target that was missing — appends `-X main.version=$(VERSION)` to LDFLAGS, with
VERSION from `git describe --tags --always --dirty`. Appended rather than folded
into the `LDFLAGS ?= -s -w` default, so `make LDFLAGS=...` overrides exactly what
it always did and still cannot produce an unstamped binary. `make cloud` stamps
cloud.Version the same way, matching what the Dockerfile already passed; plugin
builds have no version symbol and are untouched.
And the version command splits its streams. STDOUT is EXACTLY one line, `hanzo
<version>`, always — it is the answer to the question asked, and it is also what
a parent hanzo parses back out of a delegate (delegateVersion reads the first
line's last token), so one line keeps that honest in both directions. The
delegate used to print a second `delegate: <path> <version>` line right there on
stdout, unconditionally, competing with the answer. It is a stderr WARNING now,
and only when actionable: version differs (naming both paths, both versions, and
that verbs handed to hanzo-node run THAT build), or present-but-unreadable.
Absent or in agreement says nothing — silence is the healthy case. Ours is
compared with its leading `v` trimmed, the same normalisation delegateVersion
applies to the delegate's, so v1.2.3 against 1.2.3 is agreement, not a false
alarm.
Tests are red against the old code and green against the new: the ladder at every
rung, the stream contract in all four delegate states, and an end-to-end build
that requires the binary to name the commit it was built from.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/edge/nodes -> /v1/networks/routers.
Four unrelated things wore the word, which is exactly why /v1/edge read like a
missing product: hanzoai/edge (the on-device inference runtime — a binary the
customer runs on their OWN machine), the public catalogue cache, the gateway
policy role, and these — ZT fabric edge-routers. A prefix belongs to a product a
customer calls, so a position word gets none. hanzoai/edge keeps the repo name;
it is the one honest use, because it genuinely runs at the edge of the network,
the users device. /v1/edge now 404s at every depth and that is the right answer.
The routers move UNDER the network because an edge-router IS a node of the
overlay, so the resource lives where its parent does — zt already owned
/v1/networks. The envelope moved with the address ({routers:[...]}, not
{nodes:[...]}): an address and its payload naming one thing two ways is the same
defect one level down. edgeNodeView/edgeNodeList/toEdgeNodeView follow.
"routers" is a literal beside "/:id". MEASURED on this router, not assumed: the
static segment wins over its param sibling in EITHER registration order, so
unlike the /v1/s3 order-118-vs-120 case there is no ordering for a test to
freeze. It still registers first, per routes() own stated rule.
The edge tag is gone from the woven document (149 -> 148 tags) and no longer
inherits zt package sentence.
gateway: the tag sentence customers read said "live control of your API edge",
which spends the word on a position in the same document where Hanzo Edge is a
product. It now states the property. The doc also says plainly what gateway is:
PLUMBING — the trust boundary compiled into cloud as gateway.Mount, not a network
hop — whose one product door is the config plane at /v1/gateway/config. That is
what earns the prefix; the plumbing earns none.
plugin/admin/openapi.json is PRE-EXISTING drift, not part of this change:
apps/admin/o11y.go already said event.log / event.span while its committed subset
still said distributed_logs_v2 / o11y_index_v3, so app-contract was red on main
before this. Regenerated from source, which is what the gate demands.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/iam hung github.com/hanzoai/iam on five `app.All` wildcards through
zip.AdaptNetHTTP, which takes an http.Handler and returns a closure — the App
went in and a bare function came out, taking the child's op registry with it.
Cloud published FIVE path keys and 35 placeholder operations for an identity
provider holding 94 typed ops. Not one had a schema, an MCP tool, a CLI
command or an SDK method. apps/iam/typed_wire_test.go gated that as permanent;
the refusal was right about the SEAM and wrong about iam.
zip v1.18.16 adds Graft, which composes the App: cloud's router learns iam's
route patterns AND its registry while iam's router keeps iam's behaviour.
published paths 5 -> 155
published operations 35 -> 182 (94 with a real schema, from 0)
component schemas 0 -> 94 (published iam.<Type>, so iam's Application
and the fleet's — 83 props vs 16 — coexist)
MCP tools 0 -> 94
CLI commands 0 -> 94
wildcard path keys 5 -> 0
fleet: 1058 -> 1207 paths, 1490 -> 1634 operations, 1140 -> 1234 schemas,
833 -> 924 MCP tools, wildcard ratchet 28 -> 25.
Serving is unchanged and strictly cheaper: no net/http round trip, so the
adapter's ~5% and its dropped fasthttp user-context both go. iam's own Guard
still answers {"status":401,"error":"authentication required"}, its discovery
document is still minted by iam, its oauth endpoints still form-decode their
own bodies. TestServingIsUnchanged pins all of it. The surface also NARROWED:
a wildcard swallowed every unknown path under the prefix, a graft registers
only what the child declares, so an undeclared path falls through to cloud.
A graft refuses a duplicate address at compose time instead of letting
registration order decide. It found three, all pre-existing:
GET /v1/iam/keys, POST /v1/iam/onboard — apps/account registered deprecated
key aliases and an onboard handler inside iam's prefix. Measured live,
neither has ever answered in production: api.hanzo.ai routes /v1/iam/* to
IAM, so both return IAM's own Guard envelope from server: zip, with no
Deprecation header and no x-api-version. The aliases are deleted (canonical
/v1/keys unchanged) and onboard moved to /v1/orgs — named for the resource,
the same rule that moved the key surface off /v1/iam/keys, and reachable now
for the first time. deprecatedFor had no other caller and went with them.
GET /healthz — iam declared it on its public router; cmd/cloud registers it
as the HOST's because it must answer while every subsystem is cold. zip's
ops.go states the rule (a second listener the deployment names, never the
public one). hanzoai/iam dropped it. This is the shadowing incident
apps/iam's own doc comment records as an outage, and it is published by
nobody — which is why Graft reads the ROUTER, never the document.
It also closed a reachability gap. manifest/router_test.go's `unreachable`
ledger carried "iam /.well-known/{wildcard1} -> nothing" — a relying party's
FIRST call reaching no app. Relayed, the only prefix that could route it was
/.well-known, which owns the subtree and would have taken agentskills' with
it. Grafted, iam declares the three exact documents its router holds, so
manifest.Apps routes exactly those three.
TestIAMKeysBeatsWildcard proved the shadowing was safe and is replaced by
TestAccountClaimsNothingUnderIAM, which proves account claims nothing under a
prefix another app owns. typed_wire_test.go's permanent refusal becomes a
ratchet: 94 typed may only rise, 88 untyped may only fall, zero wildcards is
an assertion.
zip v1.18.15 -> v1.18.16, hanzoai/iam v1.33.26 -> v1.33.37.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The front-door commit inserted the absolute-address rationale between the
"Bridge FIRST" comment and app.Use(cloud.Bridge()), leaving the reason for
middleware ordering four lines and one error return away from the statement it
explains — and reading, at a glance, as one wall of text about two unrelated
things. Bridge keeps its own comment; the op-declaration rationale sits with the
ops. No behaviour change: ZipApp is a type assertion with no side effects, so
acquiring it after Use is identical, and the regenerated subset is byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/world 404'd. The prefix has one cloud owner (apps/world) and five
operations, but nothing at the prefix itself, so the one address a caller
reaches for first answered "not found".
It answers now, as a typed op: one registry entry, so the schema, the prose,
the OpenAPI operation, the MCP tool, the CLI command and every generated SDK
method all follow from it. Fleet document 1058 -> 1059 paths; the world subset
4 paths/5 ops -> 5/6.
What it carries that nothing else could. /v1/world/mcp and /v1/world/zap are
real, live and PUBLIC — the ingress (routes.yaml api-hanzo-ai-world-gw,
priority 100) carves them off the cloud catch-all to world-gw:9999. Measured:
MCP initialize answers 200 unauthenticated (serverInfo hanzo-world), tools/list
is fail-closed JSON-RPC -32001, /v1/world/zap is 401 missing_token. Cloud does
not route either one, and openapi.Describe renders prose ONLY for a route the
router actually serves — the property that stops the document claiming an
operation nothing answers. So the document cannot declare them and must not.
Until now the only way to learn they existed was to read the ingress config.
The front door names them, and TestIndexNamesEveryWireCompletely keeps them
named. It does NOT restate the REST operation list: GET /v1/openapi.json stays
the one enumeration of those.
Declared absolute on the app, not on a group: an op that IS the prefix has no
leaf, and zip.Get(g, "") composes to "/v1/world/", a different route. Same
reason and same form as apps/pricing and apps/plan; a group for the leaves plus
an app-level exception for the one bare op would be two idioms for one job.
manifest/mcp_test.go: apps/world/index.go joins foreignDoors. The gate reads
source for /mcp literals, and this one is an address in a data value, not a
second JSON-RPC envelope — world-gw owns that engine's tools. Reason recorded
there, including the distinction from apps/tasks: cloud names this door, it
never serves it.
NOT done, deliberately, and recorded in LLM.md: hanzoai/world is a Go module
(150 Go files) whose routes.go registers 117 routes, including the AI-plane
read surface handlers_worldgw.go names (events, conflicts, infra, vessel, news,
markets, feeds). Those are deployed NOWHERE — world.hanzo.ai runs v2.4.37 while
that surface landed after it (main is 2.4.60), and world.hanzo.ai/v1/world/events
returns that server's own catch-all, {"error":"Not found: /v1/world/events"}.
Declaring cloud ops that forward there would publish operations into
openapi.yaml, every SDK and every MCP tool list that 404 in production — the
same dark hole api-hanzo-ai-catalog opened under /v1/models and /v1/pricing,
which cost 17 documented-but-uncallable operations. Re-home that surface after
the upstream ships it, not before.
Gates: apps/world ok (3 new + the projection gate), manifest ok, openapi ok
(weave proves the subsets compose to openapi.yaml), go vet ok, gen-app-cmds
bijection intact. Proven on the real binary: GET /v1/world -> 200 with the
wire list, and the binary's own /v1/openapi.json carries the path and both
schemas.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
updateSession's UPDATE listed every mutable column except terminal, so a
session's terminal URL survived the insert and was silently dropped by the
next update — the watch link goes blank while the row still says the session
is live.
- SET status=?, title=?, ended_at=?, updated_at=?, target=?, project=?, published=?
+ SET status=?, title=?, ended_at=?, updated_at=?, target=?, terminal=?, project=?, published=?
apps/agents/sessions_terminal_test.go is already here and already covers this
(TestTerminalRoundTripsThroughTheStore), so this repo has been carrying the
test for a fix it did not have.
Verified as far as this host allows: the four pure terminal tests pass, and
the package builds. TestTerminalRoundTripsThroughTheStore cannot run on macOS
— the SQLCipher codec refuses a scratch dir that is not genuinely RAM-backed
("need tmpfs at /dev/shm or HANZO_SQLITE_RAMFS_DIR"), and a plain directory is
rejected on inspection of the mount type. Linux CI has /dev/shm and will run it.
Porting this closes the last content gap between the three cloud copies:
git.hanzo.ai/hanzoai/cloud carried it as 084ae0ba6 and
git.hanzo.ai/hanzo/cloud as 91b364e35 — the same change committed twice, and
absent here. github is otherwise ahead of both (34 and 53 commits) and already
carries the smoke/tmpfs and GHCR-pagination fixes, folded into the new
single-workflow cicd.yml.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Sharing a product is the ordinary case, not the broken one. /v1/plans is the plan
catalog with two of its rows kept by commerce; /v1/s3 is a provisioned add-on
whose bucket data plane is storage; /v1/search and /v1/vector are the same shape.
Reading "two claimants" as ambiguity silenced all four, though none was ambiguous
— depth already says which app the product IS and which merely has routes inside
it.
So the owner is the app answering the shallowest path under /v1/<product>, and
where nobody is alone at the root the answer is still silence: /v1/finance is
billing at /v1/finance/balance and treasury at /v1/finance/accounts, neither
above the other, and picking one would publish a coin flip as a fact. This is one
rule replacing one rule, not a second one beside it: a sole claimant is also the
sole shallowest, so every product already described keeps the sentence it had —
verified, 139 unchanged, 4 gained, 0 lost.
The second rule is untouched and also resolves to silence: the owner's own
sentence, or nothing. Six tags stay blank and each for a stated reason — finance
has no root owner, and authz/licensing/metrics/logs/traces have one that mounts a
subsystem in another module, so there is no package here to read. Those modules
do carry package docs, but maintainer-voiced ones; publishing "the native,
prometheus-free time-series store" as what a customer buys would be worse than
silence. The remedy is a customer-facing package doc in those services, not a
string invented here.
149 tags, 143 described.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every operation says what it does, and everything said is said about an
operation. Both halves fail the same way — a consumer holds an address and no
sentence — so both are refused at the one producer, and the artifact is simply
not written.
Nothing downstream can repair this. hanzoai/cli's generated tree used to print an
operation's own HTTP route when it had no sentence (`hanzo platform health` →
"GET /v1/platform/health") and nobody filed a bug, because a mechanical line
reads exactly like a deliberate one: the fallback did not report the gap, it
disguised it. A placeholder would be the same fallback with better manners — it
would travel into eight SDKs, the MCP tool list and docs.hanzo.ai, every surface
that cannot fix it, and be no more visible in the one that can. A gap belongs in
a build failure that names a file.
The second half is what let the first hide. Describe renders nothing when its key
is not a live route — the property that keeps the registry unable to invent an
operation — so a mis-keyed declaration is prose that was written, reviewed and
then silently dropped. POST /v1/store/storefront-token published an operationId
and nothing else for as long as its description sat under the store's old
/v1/store/token address; that key is fixed here, and the check that would have
caught it lands with it, so the gate is never introduced red. An orphan is judged
only inside the products that app publishes, because every app binary links
cloud's core and therefore carries other subsystems' declarations.
Refused at the artifact, not in Spec: a deployment's own /v1/openapi.json must
answer with what it serves even if a subsystem is behind on its prose; a
committed artifact is the fleet's contract and has no such excuse.
1491 of 1491 operations now carry prose; 0 orphans.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
THE GATE DID NOT GATE. cicd.yml gated and deploy.yml released, and they were two
files with the SAME TRIGGER. Actions cannot express `needs:` across workflow
files, so deploy built, smoked, tagged and pinned while the gate was still
running — or after it had gone red. Measured, not hypothetical: `make -f
mk/fleet.mk surface-check` was RED on main while 87 commits and 6 releases went
out in 24 hours, and what shipped was ONE binary serving
/v1/billing/gpu/eligibility and publishing /v1/billing/gpu-eligibility. The gate
was always correct. It simply had no edge to the thing it was meant to stop.
deploy.yml is DELETED. Its jobs are here, behind `needs:`:
gate ──┐
├─→ image ─→ rollout ─→ reach ─→ fanout ─→ receipt
containment ─┘
AND THE TRAIN HAD ONE CAR. It ended at "pin pushed" — nothing regenerated the
document's projections, so an SDK, the MCP tool list, the CLI's captured command
surface and docs.hanzo.ai each moved only when a human remembered. npm `hanzoai`
had two versions in its whole history; hanzo-client was not on crates.io at all;
four of seven SDK repos had no regeneration path; the three that did listen for
`repository_dispatch: spec-update` had never once received it, because the
assumed sender (hanzoai/openapi) has zero workflows. This is the sender.
THREE CARS ARE NEW AND EACH CLOSES A HOLE NOTHING WAS WATCHING.
rollout now PROVES THE RELEASE IS LIVE — polls x-api-version until it is ours.
A pin is not a release; the running version is. Every car after this one
describes the system to the outside world, and would otherwise describe the
previous version during cd.hanzo.ai's reconcile window.
reach asks whether the document tells the truth ABOUT PRODUCTION.
surface-check proves the document equals the code; nothing proved the address
is reachable, and three things break that independently — a stale subset, a
prefix missing from manifest/apps.go, and an edge worker intercepting before
the origin. 490 literal addresses in 5 seconds, no credential (401 and 403
are proof a route exists). It also checks the MCP tool count against the
committed plugin/*/mcp.json, so "the MCP list was refreshed" is a number
rather than a claim: 833 = 833.
receipt writes release.json onto the tag's GitHub Release, ALWAYS, and exits
non-zero on a hole. "cloud shipped but the CLI is two days stale" becomes a
state you can point at instead of one nobody notices for a week. It lives on
the tag rather than in the repo because a commit would trigger the next
release, and because state belongs with the thing it describes.
THE COUPLER IS THE DOCUMENT, PASSED BY VALUE AT A PINNED SHA. Every car carries
(version, sha, sha256(openapi.yaml)). No car reads api.hanzo.ai to GENERATE
anything: at generation time the deploy has already happened, so reading the host
names whatever it is serving rather than the release that sent it. The host is
read for exactly one purpose — to prove the release is live.
IT DOES NOT ROLL BACK. IT BLOCKS AND RESUMES. Cars are ordered by
irreversibility, and every one is idempotent on (version, digest): a v* tag
already pointing at HEAD IS this release, so a re-run reuses its number instead
of minting a second for one commit; a 422 whose ref names our sha is idempotent
success rather than the hard error that made a resume impossible. Releases queue
rather than cancel — ten numbers between .335 and .350 are images published under
a version that was never tagged, smoked or pinned.
openapi/unreachable.txt is a RATCHET, not an allowlist: 14 /v1/pricing/* lines,
every one owned by a Cloudflare worker that exists in no repo in this fleet. The
file may only shrink — a 404 not listed fails the release, a listed line that
starts answering must be deleted in the same commit. A number could not name the
defect; a list that can only grow smaller can.
CREDENTIAL NAMED, NOT FAKED: FLEET_DISPATCH_TOKEN, a fine-grained PAT with
contents:write + metadata:read on the nine projection repos, in KMS at
orgs/hanzo/secrets/deploy/FLEET_DISPATCH_TOKEN beside UNIVERSE_PIN_TOKEN. It does
not exist yet, so the fanout car fails and prints exactly that. Deliberate: a
release that silently skips its projections is the failure this train was built
to end.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`info.description` claimed "Generated from the live router — every operation
below is a route the unified cloud binary actually serves." No producer can make
that claim.
There is no unified cloud binary. There is a light host that mounts no subsystem
and 116 app binaries that each project their own router when they are BUILT;
what api.hanzo.ai serves is the weave of those projections (openapi.MountFleet),
so nothing in production reads a live router and the artifact is only as fresh
as the last `make -f mk/fleet.mk describe-apps`. It shipped stale: one binary at
v1.801.350 answers /v1/billing/gpu/eligibility while publishing
/v1/billing/gpu-eligibility, because the rename commit did not regenerate the
subset.
A false provenance is worse than a missing one, because it is READ. hanzoai/cli's
genspec quotes this exact sentence as the correctness argument for dropping
operations from its capture — "a route that is not mounted cannot appear in it".
Every projection downstream inherits whatever it claims: eight SDKs, the MCP tool
list, the CLI, the docs.
It now claims what is true and no more — each operation is a route the subsystem
that publishes it registered — and the package doc names the two gaps no reading
of any router closes: a subset older than its code (surface-check refuses that),
and a front door that hands the path to somebody else (only a probe of the
deployed host sees that; it was giving all 23 pricing paths to an edge worker).
plugin/{kafka,zen}/openapi.json carry the same one-line change applied by hand:
both are exempt from describe-apps (kafka fails closed without a live broker, zen
is coresident and has no standalone mount), so no generator can currently produce
them. Both publish zero paths.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Cloud was the last binary holding the retired o11y_* databases open. Every read
and write it OWNS moves onto the one event plane — event.span / event.log — the
same 15-column envelope event.event / event.error / event.metric already share.
The o11y_* names are gone from cloud's Go.
WRITE — the log/trace sink (planesink.go replaces ingest.go + zapingest.go +
spanconv.go + tracesink.go):
- The embedded otelcol pipeline that fed the SigNoz-schema datastorelogsexporter
is DELETED, not ported. A ZAP span receiver (:4317) and log receiver (:4318)
decode the same wire the collector bound and a native datastoreSink appends
prepared batches to event.span / event.log — the exact shape metrics.go already
proved with datastoremetrics.Writer (event.series / event.metric). One write
path for every signal; the whole pdata translation layer goes away.
- The opt-in in-process trace sink keeps its contract (cloud.RegisterTraceSink,
O11Y_TRACES_ZAP_INPROCESS) — cloud's own spans land in event.span with no socket.
- Bound by capability (a datastore DSN), fail-soft at every branch: a bad
telemetry config can never take cloud down. Row ids are derived (a span's is its
span_id; a log line's a content hash salted by batch position) so the plane's
ReplacingMergeTree idempotency is structural.
READ — every cloud query repoints from the o11y indexes to the plane, and the
column names move with it (o11y v3's resource_string_service$$name / duration_nano /
has_error / attributes_string[] -> the plane's plain service / duration / status /
attributes[]; the tenant is org, the plane's first sort-key column, not a map lookup):
- apps/admin/o11y.go fleet board: traces o11y_traces.distributed_o11y_index_v3
-> event.span, logs o11y_logs.distributed_logs_v2 -> event.log.
- apps/admin/subsystems.go per-subsystem RED + last-error over event.span.
- apps/o11y/logs.go infra logs -> event.log; per-org request logs -> event.span.
- apps/o11y/metricsread.go per-org RED -> event.span.
- apps/eval/metrics.go GenAI-span latency -> event.span.
TESTS — planesink_test.go is new: the pure row builders the file promises are
"unit-tested without a store or a socket" (span/log/SDK -> row), with a column-count
guard that pins each row to its column list so a builder and its schema cannot drift
into the silent-zeros failure this migration exists to end. The admin regression
guards are re-pinned: TestSubsystemSQL_UsesPlaneColumns fails on ANY retired o11y-index
spelling (v2's durationNano/serviceName, v3's duration_nano/resource_string_service$$name/
has_error), and the SQL-shape pins name the plane's columns.
NOT here, and why: the Sentry ingest sink and the tag-suggest loop are the embedded
hanzoai/o11y runtime's, not cloud's — cloud only forwards /v1/sentry to it. They move
to event.error / event.span_key+span_attribute inside o11y (implsentry already does),
and cloud picks them up on the o11y v1.5.38 bump. That bump stays BLOCKED: v1.5.37
deleted the v3 /query_range builder route and v5 rejects the v3 composite the console
sends, so the pin holds at v1.5.34 until the console migrates — proven and recorded in
1feda577 / 1c122971, unchanged here.
Gate: go build/vet ./apps/{o11y,admin,eval}/ green; go test ./apps/o11y ./apps/eval
green (KMS dev key injected for the encrypted annotation store, as the root TestMain
does). The apps/admin money/commerce reconciliation suites fail identically with and
without this diff — pre-existing, unrelated, confirmed by stashing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/summary returned 200 with no Access-Control-Allow-Origin, so the browser
could not read a status page built to be read during an outage.
The allowlist was never the problem — cloud.yaml already lists *.hanzo.ai, which
matches insights.hanzo.ai. The middleware never ran. The host in front of every
plugin is a pure router that claims prefixes and proxies them with no middleware
of its own, and EdgeCORS lives inside cloud.Serve. Of 120 plugin mains, o11y is
the only http-serving one that builds its own bare router instead of calling
cloud.Serve — and it serves three public prefixes.
Measured on production: /v1/flags answers with CORS and server: hanzo even on a
404, while /v1/summary, /v1/o11y/health and /v1/sentry/issues all answer
server: zip with no CORS and no x-api-version. The whole edge chain was absent,
not just CORS.
Same middleware, same policy store, same allowlist — installed before the mount,
because fiber runs middleware in registration order. Tests cover the reflected
origin, the 204 preflight, an unknown origin getting nothing, and an empty
allowlist staying a no-op so a deployment where the ingress owns CORS never
emits a duplicate header.
A Job copies only its POD TEMPLATE's labels onto the pod. The four labels set
on the Job stay on the Job, where countActiveBuilds reads them, so the build
pod carried no label of ours at all.
hanzo-build's Cilium policy selects pods: build-egress-deny-internal denies the
namespace every internal CIDR, and artifact-publish-egress reopens s3:9000 for
hanzo.ai/publish=artifact. Unlabelled, the pod could not reach the object store
on either endpoint — proven with a probe: without the label it times out, with
it it connects.
That is where the layer cache belongs. Without it the cache is a registry blob
pulled WHOLE onto the node's 105GB disk beside the images, the snapshots and
the build's own working set, which filled the runner pool (79GB retained
against 26GB free) and evicted builds mid-run.
The test asserts both halves: the pod template carries the label, and the Job's
own labels stay where the build quota counts them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`make describe` — zipdoc lifts the doc comments, each app projects its own
router, the weave composes them. Path count is unchanged at 1058: this
moves prose, not surface.
It also picks up drift nobody had regenerated: apps/git's zipdoc_gen.go was
missing the import/inbound/mirror ops' prose entirely, and apps/admin's had
gone stale. Both were invisible because the weave only proves the subsets
compose with each other — the drift gate is what regenerates from source.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This binary owns a handful of verbs and hands every other one to the fabric CLI,
which is a SEPARATE artifact with its own version. When that delegate is stale
nothing said so — the user types `hanzo`, gets delegated, and runs an old build
whose command surface is not the documented one. A v1.7.2 delegate behind a
current control binary served ~150 commands that no longer existed, silently.
`hanzo version` now names it:
hanzo 1.801.348
delegate: /usr/local/bin/hanzo-node 1.7.2
A version it will not report reads as unreadable rather than being guessed at.
The reason that reporting would have been dead code on arrival: main.go
short-circuited `version` with its own Printf before cobra ever ran, so there
were two implementations of `version` and the one that ran was the one without
the logic. The same shape as the routing map this package just lost. `version`,
`--version` and `-v` now normalise onto the one command.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every published description that named another company's product as the
standard now states the shape instead. The reader of these strings is a
customer of the Hanzo Cloud reading `hanzo <product> --help`, an SDK
docstring or the API reference — not a maintainer, and never someone who
should be told what our surface is a clone of.
flags the verdict is the FLAG verdict, not a vendor's shape
engine a model list is the standard list envelope
benchmark you benchmark against your own chat-completions endpoint
usage a plan is the subscription plan, as the provider names it
iam legacy verb aliases, not a dead vendor's
company incorporation and fundraising, stated outright
analytics/pricing example model ids come from our own Zen family
Naming the third party whose ACCOUNT a customer links stays: `provider:
openai` is a fact about their credential, not a claim about our design.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`hanzo auth login` died on `invalid_client: client authentication failed`
straight out of the device-authorization endpoint. The CLI was borrowing a
different first-party client per flow — hanzo-app for the device grant,
hanzo-console for the password grant and for refresh — and both are registrations
that hold a client secret. A CLI ships to users' machines and can never present
one, so IAM refused the device request before a human ever saw a code.
hanzo-cli is this binary's own client (<org>-<app>, HIP-0111): PUBLIC, no stored
secret, PKCE on the code flow and a human's approval on the device flow. It is
now the single default, and the per-flow override that picked a different id is
gone — one id, defined once in defaultClientID.
Sharing one id across flows is not cosmetic. A refresh token minted under the
device client was later presented under the password client, and the token
endpoint rejects a device_code redeemed by a client it was not issued to, so the
split guaranteed the renewal path could not work even once login did.
The Rust CLI (hanzoai/cli src/iam/oauth.rs) already authenticates as hanzo-cli,
so both binaries now name the same client and one IAM registration serves both.
Login help no longer advertises the password grant as the automation path: it
needs a client secret this public client does not hold, so --token is the honest
answer and ROPC only works against a confidential --client-id.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cmd/hanzo decides per verb whether to run it here or hand it to the Rust fabric
CLI, and it asked a hand-kept map. The map drifted from the commands actually
registered, in both directions:
- it still claimed `code` and `k8s` after those commands were deleted, so the
verbs were routed here, found nothing, and died with `unknown command
"code" for "hanzo"` — while the fabric CLI that implements `code` sat right
there on PATH, never consulted. This is what a user hit typing `hanzo code`.
- it never listed `completion`, so a command registered HERE was handed away
and shell completion could not work.
Two sources of truth for one fact. Delete the map and ask cobra: IsControlVerb
resolves the name against newRootCmd, covering aliases for free, plus cobra's
own completion request verbs. The router and the command set can no longer
disagree, because they are now the same fact.
TestRouterMatchesCommandTree locks the bijection in both directions — every
served verb resolves, every delegated verb does not, and the tree and the list
are each checked against the other, so a command added or removed without
updating the other side fails here rather than in someone's terminal.
Also drop the dead ControlCommands accessor, and correct the docs that still
said this module ships no CLI while it ships one — README, Makefile and LLM.md,
plus the `hanzo k8s target` example for a verb deleted weeks ago.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The cache has to scale independently of a build node and a registry cache does
not: buildkit pulls the WHOLE cache onto the node before it can read any of it and
pushes it back afterwards, so its size lands on the same 105GB disk that holds the
images, the snapshots and the build's own working set. That is what filled the
runner pool — 79GB of retained layers with 26GB left — and evicted builds fifteen
minutes in, repeatedly. S3 is read ranged and per-blob: the node holds the working
set and nothing else, and the cache grows without touching a disk anyone sizes.
Setting BUILD_CACHE_S3_ENDPOINT moves it, using the storage the fabric already
has — artifact.go publishes binaries through the same endpoint under the same
credential, so this adds a bucket and not a dependency. One bucket keyed per
repository, so a new repo needs no provisioning and two never share a cache. The
credential rides the environment, never argv, and is optional: absent, buildkit
reports a miss and the build runs uncached rather than the Job being
unschedulable.
It is not yet the default because it is not yet REACHABLE. The build namespace has
no network path to the object store on either endpoint — both time out, which also
means artifact publishing from a build cannot be working today. Pointing every
build at a cache it cannot open would trade a disk problem for a broken build, so
which backend to use stays a deployment fact: grant hanzo-build ingress to the s3
service, set the endpoint, and the cache moves with no code change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two outside vendor names as the standard, then my own replacement jargon.
Neither is the product's name. It is the Hanzo Cloud, and the help text says
so in the words a customer already uses.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three strings named two outside vendors as the standard for what this CLI is.
Our own product does not need a comparison to say what it does: it is one
command tree over the live estate — identities, apps, deploys, clusters,
builds. State the property, not the analogy.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The cache has to scale independently of the node, and a registry cache does not:
buildkit pulls the WHOLE cache onto the node before it can read any of it and
pushes it back afterwards, so the cache's size lands on the same 105GB disk that
holds the images, the snapshots and the build's own working set. That is what
filled the runner pool — 79GB of retained layers with 26GB left — and evicted
builds fifteen minutes in, repeatedly.
S3 is read ranged and per-blob: buildkit fetches the manifest, then only the blobs
this build actually misses, and writes back only what changed. The node holds the
working set and nothing else, and the cache grows without touching a disk anyone
has to size. It is the storage the fabric already uses — artifact.go publishes
binaries the same way, through the same in-cluster endpoint under the same
credential — so this adds a bucket, not a dependency.
One bucket keyed per repository, so a new repo needs no provisioning and two repos
never share a cache. The credential rides the ENVIRONMENT, never argv, because a
build command is logged. It is optional by the same rule artifact publishing
uses: absent, buildkit reports a cache miss and the build runs uncached rather
than the Job being unschedulable — a cache accelerates, it never gates.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The GitHub→forge mirror had stalled with "import: git import: org required".
Two reasons, both the same shape as the KMS assertion:
InboundGitSync — the per-push half of the mirror — still assumed co-residence.
The app that receives a webhook, or runs the scheduled reconcile, is not the app
that holds the repos, so the seam was nil where it was read. apps/git now
publishes /git/inbound and the request travels. A divergence crosses as a VALUE,
not an error: native is canonical and was left untouched, and the caller needs to
know that rather than retry into an overwrite.
And the reconcile runs on a schedule, so there is no request to forward and both
calls reached the git app anonymous, where a tenant that cannot be seen is
refused. cloud.For states the org the sync acts for — the sync's own org, never a
field of the event — so a job supplies an identity where there is none and can
never launder one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/git/webhook handed every verified push to cloud.OnGitPush, whose only
registrant lives in apps/platform. cloud runs each app as its own OS process, so
in the git process that builder is nil forever — and the handler answered 204
either way. Delivered, signature valid, green on the forge's hook page, nothing
built. main once drifted eight commits past what was live behind that 204, and
the forge's hook_task rows were the only place the truth showed.
The route stays and answers 410 naming platform.hanzo.ai/v1/git-webhook. A
deleted route 404s, and a 404 from this estate is the signal already misread
twice as "the API is switched off" (Hanzo Git serves /v1, so /api/v1 404s and
looks identical to a dead server). A retired door has to say so and name its
replacement; the unexplained 204 is exactly what made this expensive.
With nothing to authenticate, the HMAC, the 8 MiB bound, the pushEvent decode
and GIT_WEBHOOK_SECRET all go with it: -171 lines.
The old tests passed for as long as the door was dead, because they registered a
builder in-process and asserted the call they had just made possible. They are
replaced by the two properties that keep it honest: no input gets anything but
410, and the refusal names where the delivery belongs.
Ran the bump instead of reasoning about it: plugin/o11y built at v1.5.38 with all
three forwards updated, served against an upstream carrying v1.5.38's route table
and its REAL v5 request decoder, so the verdicts are the module's own.
POST /v1/o11y/query_range -> /v1/o11y/query_range 400 unknown field "queryType"
GET /v1/o11y/sessions -> /v1/o11y/llm/sessions 200
GET /v1/o11y/llm/observations 200
GET /v1/o11y/zzz-not-a-route 404
Two forwards fix cleanly; the third has no target, because v1.5.37 deleted the v3
builder route outright (exhaustive dump of every query route literal at the tag:
/v1/o11y/query_range is v5's alone, plus preview, format and the dashboards
widget). They ship together or not at all, so the pin stays at v1.5.34 and the
console keeps its explorers. Unblocking is still the console migrating to the v5
composite, after which builderQueryHandler is deleted rather than repointed.
Settles whether rewriteExternalPath should survive the rename: it is o11y's, not
cloud's, so nothing here deletes it — but the rehearsal shows it was doing real
work at v1.5.34 (an unknown path reached the runtime as /api/zzz-not-a-route,
rewritten) and none at v1.5.38 (the same request arrives verbatim). Internal and
external spellings converged, which is what turned it into an identity rewrite
and why the rename deleted it in the same commit. The only surviving /api/ is the
Sentry SDK's own wire form — its protocol, not our route.
Also corrects a prediction I had recorded as a risk: the failure mode is NOT a
200-shaped envelope wrapping a runtime 404. cloud propagates the runtime's
status, so this drift fails loudly at the client as a real 404/400.
Gate: go build ./apps/o11y/... green; go vet ./apps/o11y/... green;
go test ./apps/o11y/... green 3/3. go build ./... fails identically with and
without this diff (hanzoai/base@v1.5.11 cgo sqlite build tag) — pre-existing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Each connected account is an independent installation with its own token and its
own pagination, so reading them in series made the wall clock their SUM: 816
repos across three accounts is ten sequential pages and about seven seconds. Under
load the largest account exceeded the request deadline and dropped out of the
union — the whole hanzoai listing vanished while the two small accounts answered.
Degrading to the accounts that answered is the right behaviour and stays.
Starving an account because it was listed last is not, so they are read
concurrently and the wall clock is now the slowest account rather than their sum.
The union is assembled in CONNECTION order, never completion order: results land
in a slice indexed by connection and are merged after every account has answered,
so the same set of accounts always yields the same list and a caller paging it
cannot have rows reshuffle because one account happened to be quicker.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GHCR ignores n>1000 and pages via `Link: rel="next"`. cloud has ~1700 tags,
so a single-page read topped out at v1.799.5:
single page : 248 semver tags, highest 1.799.5
paginated : 610 semver tags, highest 1.801.351
REG — the half this file calls "the authority on what has been USED" — was
missing every recent release, so max(REG, GIT) leaned entirely on git tags.
It deadlocks: NEXT is computed one past a stale LAST, and the probe then
refuses a tag that IS published, forever.
Parity with forge commit 9bd7c2a4b. CI builds the forge copy; these two
histories have diverged (12 commits one way, 4 the other, merge-base
0e027aa28) and that divergence is now what blocks `Tag the release` —
the workflow tags a forge-built SHA through the GitHub API, which answers
"Object does not exist". Reconciling them needs an owner decision.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
63ad9918d put the comment BETWEEN backslash-continued arguments. The backslash
escapes the newline, so the shell joins the lines and the `#` starts a comment
mid-command, discarding the image argument:
docker run --rm --entrypoint /bin/sh # mode=1777 is load-bearing...
docker: 'docker run' requires at least 1 argument
The mount fix itself is unchanged and correct; only its comment moved. Mirrors
forge commit 718e9d802 — CI builds the forge copy, which has diverged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud's release has been blocked for six consecutive runs. The smoke gate
sits before `Tag the release` and the universe pin, so nothing has shipped:
listen unix /data/credz.sock: bind: permission denied
open /data/gateway.db.cek.lock: permission denied
open sqlite "/data/audit-kms.db": ... permission denied
zip: Load(kms): exited before listening: exit status 1
zip: Load(pubsub): exited before listening: exit status 1
zip: Load(kafka): exited before listening: exit status 1
SMOKE FAIL: never reached listening
The image runs as USER 65532:65532 (Dockerfile). A bare `--tmpfs /data:rw`
lands root-owned 0755 on the runner's dind daemon, so every /data write is
refused. The credz broker dies first and every service behind it follows, so
the log reads like three独 service failures when it is one mount.
This header already promised "a writable /data" for the smoke env; it just
was not one.
Reproduced and fixed INSIDE a git-runner, which matters — Docker Desktop's
tmpfs defaults differ and report the broken form as already writable, so a
laptop test says the fix is a no-op:
docker run --rm --tmpfs /data:rw --user 65532:65532 alpine:3 \
sh -c 'touch /data/x && echo WRITABLE || echo DENIED'
git-runner: DENIED mode=1777: WRITABLE
macOS: WRITABLE (false negative)
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Declaring 20Gi was right in kind and short in size: the kubelet evicted a build
that had exceeded its request off a node already at its threshold, ten minutes
in. 50Gi is what a run of this shape actually costs — a clone, the module cache,
112 plugin binaries, and the exported layer cache — so the scheduler now places
it somewhere it fits.
The other half is that a finished build keeps holding a node's disk. The
buildkitd state is an emptyDir, freed only when the POD is deleted, and the TTL
was an hour: with a build every few minutes, six finished pods sat on tens of
gigabytes each and the next build was evicted off a node they had filled. Ten
minutes is far past the terminal-state read (waitForJob polls every 5s) and long
enough to pull logs from a failure.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The o11y bump to v1.5.38 does NOT land, because it cannot: o11y v1.5.37 stopped
mounting the v3 builder POST route (RegisterQueryRangeV3Routes no longer
registers /query_range; QueryRangeV3 lost its last reference, and
queryRangeV3/queryRangeV4 survive as unreferenced methods). /v1/o11y/query_range
is served by the v5 querier alone there, and v5 refuses the composite the
console sends — proven against the pinned module, not asserted:
console listQueryPayload() -> v5 QueryRangeRequest
REJECTED unknown field "queryType" in composite query
v1.5.38's own v5 example -> v5 QueryRangeRequest
ACCEPTED requestType="time_series" queries=1
So repointing query.go at the flat path trades a working explorer for a 400.
The engine the console speaks to no longer has a route, and no cloud-only edit
conjures one back; unblocking is a console migration to the v5 composite, after
which builderQueryHandler is an identity rewrite of the path it is already
registered on and should be DELETED rather than repointed. Written down in
LLM.md with the other two forwards the bump moves (sessions.go's /api/sessions
-> /v1/o11y/llm/sessions, o11y.go's health allowlist) so the next attempt reads
the trap instead of re-deriving it.
The trap is reachable because nothing caught it. builderQueryHandler forwards by
rewriting r.URL.Path in-handler, so typed_wire_test.go cannot see where it
points — it catches the health-probe rename and goes green once that is fixed,
with the 400 still in place. query_test.go now pins both halves: the console's
payload is valid v3 and refused by v5 (so "resolve to the highest version" is
not available here), and the forward never names its own public path. The path
is split into builderInternalPath solely to make that assertable.
The refusal is asserted on "unknown field", never on which one: the v3 composite
carries several keys v5 does not know and v5 names one, so the specific field is
map order. Caught as a 1-in-6 flake before landing; 10/10 green after.
Gate: go build ./apps/o11y/... green; go vet ./apps/o11y/... green;
go test ./apps/o11y/... green 10/10. go build ./... fails identically with and
without this diff (hanzoai/base@v1.5.11 cgo sqlite build tag) — pre-existing,
unrelated, proven by stashing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The import crosses the plane now, and arrived anonymous: it runs detached, so
there is no request to forward, and the callee reads the tenant from the caller
identity and refuses one it cannot see — "git import: org required" after the
request had already travelled correctly.
cloud.For states the org the job acts for. An inbound request always wins over a
stated one, so a job supplies an identity where there is none and can never
launder one; the org is the one the authenticated request named, never a field
of the work item.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Third and final hop of one const, each toward the rows that exist:
o11y_ai.observations (a database that never existed — the panel read zero) ->
console.observations (real rows, but a surface name on a store already folded
into the plane) -> event.span, where the SAME rows live as kind='client' gen_ai
spans (identical count 8,867 and identical summed cost, verified against console
today). Model/cost/latency are span ATTRIBUTES (gen_ai.*/_o11y.*), so the
projection consts sit beside the table name — stated ONCE in o11y.go; aimetrics
reads them there. Its former twin const is deleted: the same fact stated twice
is exactly how both drifted into pointing at nothing without either being
noticed.
Gate: the Aim/O11y suites green; TestCommerce_Reconciles*/TestOrgs_RealAggregation
failures are PRE-EXISTING at origin/main (verified detached), unrelated money-
reconcile fixtures.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Deploying cloud destroyed every render in flight on the BYO fleet. api.hanzo.ai
answers "503 no available server" for the seconds its pod is replaced; the presence
write is the call whose error ends the worker; systemd restarts it; the restart kills
the studio that worker supervises — and with it a compose that had been sampling for
minutes. Three restarts landed in one deploy window.
The namespace-ensure was already made best-effort for this same crash-loop, but the
presence write kept the fatal path, and it is the one that was hit. It now retries
with backoff for about thirty seconds — longer than a pod replacement, far shorter
than a real outage — so a blip is ridden out and a genuine rejection still surfaces.
Deploying the control plane is a routine act. It should not be able to reach into
someone's unfinished work and delete it.
A worker that claimed a render it could not serve reported `fail` on it. fail is
TERMINAL — it hands the job to nobody. So the branch whose own message read
"declined so a render-capable worker takes it" was the thing that killed the render:
two workers that had both just restarted each claimed one and each failed it, and the
person who asked for that image got nothing back.
The engine's verbs are claim/complete/fail/heartbeat/cancel; there is no release. The
claim's LEASE is the release — a claim that is neither completed nor failed nor
heartbeated expires and the job returns to pending. So a decline now reports nothing
at all and lets the lease lapse, which is what "declined" always meant here.
The cost is that the job waits out the lease before another worker sees it. The
alternative was the one we had: an immediate, permanent loss.
Killing the mega build deleted cmd/hanzo along with apps.Wire and the fused
cmd/cloud (22f4fc64e). Deleting it was right for the half that mattered: that main
could mount any subsystem or the whole surface, which is what made it a 3102-package
link, and per-app plugin processes are a better answer.
But it was also the ONLY main importing cli, and cli is where `hanzo gpu connect`
lives — the worker process that claims render jobs on every BYO GPU. Nothing in this
repo or any other built it afterward, so the binary running the fleet could not be
rebuilt from source, and a fix to it had nowhere to ship. That is how it was found:
a readiness fix in cli/gpu.go with no path to the machines it was for.
This main links cli and nothing else. apps.Wire is not in its graph — verified, not
asserted — so it cannot mount a subsystem and the fleet stays unlinked: 1558
packages against the old 3102, and the mega build stays dead. Serving is cmd/cloud's
job; asking is this one's.
evo claimed render jobs from the first instant of its process and failed each one.
Readiness was `token != "" && (launchesStudio || reachable)`: passing --studio-dir
counted AS readiness, so the probe never ran on exactly the nodes that need it.
Launching a studio does not make it serve. It makes it serve eventually — after it
binds the port and loads models, minutes on a cold box. Worse, the verdict is taken
BEFORE superviseStudio is started, so the node advertised studio.render and won
work at a moment when nothing had even been launched yet, then failed every job on
arrival. A cold node that claims is worse than an absent one: it takes work from
nodes that could have run it.
Now every node probes, including one that launches its own. Nothing deadlocks: the
launch does not depend on the verdict, and the heartbeat loop already re-evaluates
it every 30s and flips claiming on the way up — the same path that recovers a
studio that died under a node that had been ready. The block reason says which
case it is, because "not serving yet" and "no studio here" call for different
things from whoever reads it.
spark heartbeats every 30s, claims render jobs and runs them. /v1/machines,
/v1/gpus, /v1/fleet/workers, the fleet board and studio's node badges all said
there were no GPUs. Both were true at once.
The durable engine is per-PROCESS — each app embeds its own over its own data dir,
because the engine's SQLite has one writer and a shared store would be the
collision a shared port already was (durable.go). A worker registers its presence
through the TASKS surface, so the `fleet` and `gpu-jobs` namespaces live in that
app's engine. visor read the engine in ITS address space, which nothing had ever
written, and got back an empty page and no error. On disk the split is plain:
tasks/hanzo/_/_/fleet.db takes the writes while visor/hanzo/_/_/fleet.db only ever
sees reads.
So the engine is asked over ZAP on the tasks app's unix socket, exactly as the
prepaid ledger is asked of the one process that may open it. The org rides the
caller, never an argument, so a page can only ever be the caller's own shard; the
rows cross as the engine's own JSON rather than a copy of StandaloneActivity
declared on the plane, where it would be free to drift from what produced it.
An absent engine now FAULTS instead of answering an empty page. That distinction
is the whole bug: the reader above is deliberately fail-soft, so "cannot read" and
"nothing there" rendered identically, and a fleet that was up read as a fleet that
was empty with nothing anywhere reporting a fault.
Every import answered "git importer not registered" while both apps were
healthy. Each subsystem runs as its own process, so cloud.RegisterGitImporter —
an in-process seam — is nil everywhere except the one process that registered it.
The app that DECIDES to import is integrations, because it holds the provider
connection and can mint the credential; the app that owns the repos is git. They
are never the same process, so that seam could never be non-nil where it was
read. The same shape as the KMS assertion this week.
The request travels the internal plane instead: apps/git publishes /git/import,
and ImportGitRepo uses it whenever the local seam is absent. The payload carries
no org — the callee reads the tenant from the plane identity, so an app acting
for one org cannot create a repo in another's namespace, and an anonymous call is
refused rather than defaulted.
The import also carries the GitHub account as the project, because native stores
a repo at <org>/<project>/<name>.git: without it hanzoai/ai, hanzo-apps/ai and
hanzo-docs/ai share one path and each overwrites the last.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
With the ledger finally opening, the real balance surfaced: 149918.078983985999994361
USD. plane.Money.Minor() refuses an amount finer than a cent rather than round behind
the caller's back (the platform books credits at eighteen decimals, and per-token
charges are routinely sub-cent), so the fail-closed gate refused every completion on an
account holding $149,918.
Its doc says a caller wanting a coarse figure should round explicitly, where the choice
is visible. This is that choice, and it is DOWN: rounding up would admit a request the
balance cannot cover, and the debit that follows is EXACT — so the difference lands as
a negative balance nobody authorized. Rounding down can only refuse slightly early,
which is the safe direction for a gate. Nothing is billed from this number; it decides
admission only. An amount too large for int64 errors rather than wrapping a funded
account into an overdrawn one.
The compound-name sweep renamed the ROUTES but the published spec still carried
search-docs, load-balancers and embed-status, so the drift gate failed: openapi.yaml
must equal the weave of its apps' subsets, and the SDK repos pull that file. Three
old names would have shipped to every SDK — the exact thing the rename set out to
remove.
Regenerated from the LIVE routers (make describe per app, then the weave), not
hand-edited: hand-editing a generated artifact is what put the tree in this state,
and the gate exists precisely because a human cannot be trusted to reproduce a
generator byte for byte.
Local regeneration needed the CGO+libsqlcipher backend — the pure-Go codec refuses to
decrypt without RAM-backed scratch, which macOS cannot provide (isRAMBacked is
hard-false off Linux, so a real RAM disk does not help either). Recorded here because
the next person will hit it: CGO_ENABLED=1 GOFLAGS='-tags=libsqlite3,sqlite_math_functions'.
Gate now green: 'ok github.com/hanzoai/cloud/openapi', 0 stale names.
cek-rewrap moved every per-org sidecar from the legacy Global derivation to its owner
on 2026-07-31 01:25 ('a store's key names its owner'). The DATA migrated correctly —
220 stores, verified: 0 need migration, 0 unopenable. sqlstore.Open was never updated
and still asked for cek.Global, so every per-org ledger failed to unwrap with 'wrong
key, wrong principal, or corrupt blob'.
That error reads like data loss. It was not: all 86 finance ledgers are intact and
correctly keyed, and were the whole time. But the AI balance gate is fail-CLOSED, so a
balance it could not read refused EVERY completion fleet-wide — chat, copilot,
documents — and studio renders held behind the same 503.
The principal is now a PARAMETER, because these stores genuinely do not share one: a
per-org ledger is its org's, the platform treasury and the house book are Global. Naming
it at each call site is what stops an opener and a migration from disagreeing silently
again.
Failures: 23 before, 10 after — thirteen tests that could not open a store now can.
commerce renamed POST /v1/store/storefront-token to POST /v1/store/token: the
route is already under /v1/store and mints that store's key, so the qualifier
was the group repeated inside the member, beside literal siblings /current and
/access that do not repeat it.
cloud does not SERVE this route — it DESCRIBES it, in apps/commerce/describe.go,
because the fleet document is one document and a commerce-served address still
has to appear in it. So the two repos move together: commerce owns the route,
cloud owns the sentence about it, and if only one moved the published spec would
name an address nobody answers.
Regenerated: plugin/commerce/openapi.json and openapi.yaml woven from the
subsets.
a49aec756 renamed the subsystem zero-trust -> zt, but its `git add` named
plugin/zero-trust, a path the same commit's `git mv` had already removed. git
add fails on a missing pathspec and stages NOTHING, so that commit captured
only the directory move already sitting in the index. The content edits stayed
in the working tree, and manifest/apps.go was swept into 3e904dc1f by a later
`git add manifest/apps.go`.
That left main BROKEN, not merely half-renamed: manifest.Apps said "zt" while
plugin/zt/main.go still called manifest.PrefixesFor("zero-trust"), which now
matches no row and returns nil — and zip.Load refuses a plugin that answers no
prefix, so the zt child could not have started. apps/zt/zt.go likewise still
mounted as "zero-trust", so the subsystem's scoped middleware and its price
attribution were keyed to a name the manifest no longer carried.
This is the rest of that commit, unchanged from what it should have contained:
plugin/zt/main.go (Name, PrefixesFor, Serve's enable list, and the comment
explaining why this app serves nothing under its own name), apps/zt/zt.go's
Mount, apps/zt/Makefile's APPS, the three manifest tests, and the three places
that documented the name→package exception gen-app-cmds no longer needs
(openapi/synopsis.go, mk/plugin.mk, manifest/gate_coverage_test.go).
The socket path is resolved by BOTH halves: a callee to LISTEN, a caller to DIAL. They
computed it in different places, so they missed each other in a way nothing reported.
Plugins bound private temp paths (/tmp/zip-commerce-*/commerce.sock) while callers
dialed /var/lib/cloud/run/commerce.sock — and a stale socket file at the shared path
turned the miss into 'connection refused', which reads like the callee is DOWN rather
than somewhere else. That is why this looked like a commerce outage for hours.
The router had it worse: cmd/cloud links the plane leaf and NOT the fleet, so it could
not reach cloud's binder at all. Its start door resolved to a temp path, so it existed
nowhere any child looked — waking a lazy app failed with 'this process runs under a
router whose start door is not there', and every call to a not-yet-started app was
unreachable. Money ops are fail-closed, so every AI completion answered 503 on a
perfectly healthy fleet.
Put the rule in the leaf both halves already import (plane.BindRuntimeDir), delegate
cloud's copy to it, and bind in the router before it computes its own socket. Two
halves, one rule, so they cannot drift again. Idempotent; an operator-set
ZIP_RUNTIME_DIR still wins.
Failures unchanged: 18 before, 18 after.
/v1/o11y/annotation-queues[/:id[/items[/:itemId]]] -> /v1/o11y/reviews[…]
o11y's own doc comment already called these "human-review queues", and
annotation_store.go describes a queue as what "holds items — traces,
observations, sessions — for a human to grade". So the thing a caller addresses
is a REVIEW; a queue is how the work is handed out, which is a detail of the
implementation and not a fact about the resource. `reviews` also drops the
compound without inventing anything: the word was already in the code, one line
above the route.
This is the largest address in the sweep — four paths, eight operations — and
all four move together because they are one resource and its sub-collection.
The STORE does not move. AnnotationQueue, AnnotationQueueItem and the
annotation_queues / annotation_queue_items tables keep their names: renaming a
persisted table is a migration, not a rename, and it buys nothing a caller can
see. The seam is the route, and the route is now honest.
console follows for the three READS it makes (lib/api/o11y.ts). Its own
`annotation-queues` PAGE — the route id, the "Annotation Queues" label and the
${DOCS}/annotation-queues link — deliberately stays: that is product copy plus
an external docs page, a coordinated rename with hanzo-docs rather than part of
an api sweep. Same call as the score-configs page in the previous commit.
Regenerated: apps/o11y/zipdoc_gen.go, plugin/o11y/{openapi,mcp}.json, and
openapi.yaml woven from the subsets.
POST|GET /v1/evals/dataset-items -> /v1/evals/datasets/:name/items
POST|GET /v1/evals/score-configs -> /v1/evals/rubrics
`dataset-items` is the only one of these renames that changes more than an
address, and it should. The dataset was ALREADY mandatory on both operations —
POST 400'd without `datasetName` in the body, GET 400'd without it in the query
— so the collection never existed except inside one set. The compound was the
containment, written as a word because it was not written as a path. Now it is
a path: the set comes from the URL, `datasetName` is gone from the body and
from the query, and the 400s that policed a missing field are gone with them
because the route cannot be addressed without a set. The 404 that refuses an
unknown or another tenant's dataset is untouched — that is the ownership check,
not the containment.
`score-configs` is a rename only. A score config declares what a score named X
is allowed to BE — NUMERIC with bounds, CATEGORICAL with a closed set of
labels, or BOOLEAN — and is checked against every score written under that
name. That is a rubric: one academic noun for a grading scheme, replacing a
two-word implementation label ("config" describes how we store it, not what it
is).
The operation ids follow the address, in the form this router's UNTYPED routes
already use: get|post_v1_evals_datasets_by_name_items, beside the existing
delete_v1_evals_datasets_by_name. The nested operations also gained the `name`
path parameter in the document, which the flat address had nothing to declare.
console moves in the same change: EvalsApi.createDatasetItem takes the dataset
as its first argument instead of a body field, listDatasetItems builds the
nested URL, and the rubric reads follow the rename. Its o11y notes claiming
these live at /v1/o11y/score-configs were wrong before this commit — scores and
their definitions have always been served by /v1/evals — and now say so.
Regenerated: apps/eval/zipdoc_gen.go, plugin/evals/{openapi,mcp}.json, and
openapi.yaml woven from the subsets.
GET|POST /v1/admin/spend-caps, PATCH|DELETE .../spend-caps/:id
-> /v1/admin/caps[/:id]
GET /v1/admin/block-storage -> GET /v1/admin/volumes
Both compounds were qualifiers the address already supplied.
`spend-` : under /v1/admin there is one kind of cap — the per-org usage
ceiling — and the sibling it twins is /v1/admin/promos, not
/v1/admin/plan-promos. The DOMAIN keeps the full term where it is load-bearing:
metering.ErrSpendCapExceeded, commerce's AuthorizeSpendCap and the spend alerts
this forwards to are untouched, because "spend cap" is what the concept is
called wherever it is not already inside /v1/admin.
`block-` : the old comment argued the compound was needed to avoid colliding
with the operator's S3 view at /v1/admin/storage. That is solved better by
naming what the endpoint RETURNS — every row of the answer is a volume — than
by qualifying "storage" twice. /v1/admin/volumes and /v1/admin/storage are now
told apart by their nouns rather than by an adjective.
The operation ids are hand-set on this router (zip.WithOperationID), so they do
not follow a path automatically: adminSpendCaps/adminCreateSpendCap/
adminUpdateSpendCap/adminDeleteSpendCap -> adminCaps/adminCreateCap/
adminUpdateCap/adminDeleteCap, and adminBlockStorage -> adminVolumes. Handlers,
the blockStorageOut envelope and block_storage.go follow to listCaps/createCap/
updateCap/deleteCap, volumesOut and volumes.go.
DigitalOcean's "block storage" survives in PROSE wherever it names the vendor's
product — it is the accurate English for what these volumes are — but no longer
anywhere that reads as an address.
Regenerated: apps/admin/zipdoc_gen.go, plugin/admin/{openapi,mcp}.json, and
openapi.yaml woven from the subsets.
GET|POST /v1/load-balancers, GET|DELETE /v1/load-balancers/:id
-> /v1/balancers[/:id]
Nesting was the first choice and it is not available: the natural home would be
/v1/networks/balancers, but apps/zt owns /v1/networks (manifest/apps.go), and a
prefix owns its whole subtree — so a deeper claim there would put two
subsystems on one address for a reason that is presentational. The flat single
noun is what is left, and it is enough: this subsystem serves /v1/vpcs and
/v1/balancers, both bare plurals, both DigitalOcean house-account resources.
DigitalOcean's own vocabulary stays where it belongs. godo.LoadBalancer,
godo.LoadBalancersService and the response's `loadBalancers` field are the
vendor's names for the vendor's resource — the body is unchanged, so no client
has to re-read a payload for a rename of an address.
console moves with it: LoadBalancerModule.tsx calls all three verbs, and
proxy-allow.ts admits by FIRST SEGMENT — `load-balancers` was a real entry
there, not a comment, so the allowlist itself changes or every call 403s at the
proxy before it reaches cloud. The admin Infra tab slug goes with it for the
same one-name reason; its visible label is still "Load balancers", which is
what the e2e clicks.
Regenerated: apps/do/zipdoc_gen.go, plugin/do/{openapi,mcp}.json, and
openapi.yaml woven from the subsets.
GET|POST /v1/captable/share-classes, PATCH .../share-classes/:id
-> /v1/captable/classes[/:id]
GET|POST /v1/captable/equity-plans -> /v1/captable/plans
A capitalization table has one kind of class and one kind of plan. The prefix
already supplies both qualifiers, so `share-` and `equity-` were the group name
repeated inside each member — and they were the only two members that did it:
the siblings are stakeholders, shares, options, safes, convertibles, rounds,
investments, summary, every one a bare plural.
The bundle's own vocabulary does NOT move. shareClasses.create,
shareClasses.update and equityPlans.create are tRPC procedure names in the
upstream captable bundle, and the Go types are still ShareClass and EquityPlan
— those name the DOMAIN objects, which really are share classes and equity
plans. Only the addresses stop repeating the group.
console moves in the same change: lib/api/captable.ts builds both URLs,
CapTableModule.tsx prints one in a failure hint (a hint naming a dead address
is worse than no hint), and proxy-allow.ts enumerates the sub-paths its
`captable` head admits. The allowlist itself is by first segment, so it needs
no new entry.
Regenerated: apps/captable/zipdoc_gen.go, plugin/captable/{openapi,mcp}.json,
and openapi.yaml woven from the subsets.
POST /v1/webhooks/:id/rotate-secret -> POST /v1/webhooks/:id/secret.
The endpoint has one signing secret. POST to it is what mints a new one — the
same rule wallets already follow at POST /v1/wallets/:id/keys, and the same one
this very surface follows at /:id/test and /:id/deliveries. `rotate-` put the
verb in the noun and made the only compound on a router that is otherwise
{:id, :id/deliveries, :id/test}.
The handler stays ops.rotateSecret: rotation is what it DOES, and a Go method
is allowed to be a verb. Only the address had to stop being one.
console moves in the same change (WebhooksModule.tsx calls it directly, and
proxy-allow.ts documents the sub-paths the `webhooks` head admits) — the
allowlist is by first segment, so the route change needs no allowlist change,
only the comment that would otherwise describe an address nobody serves.
Regenerated: apps/webhooks/zipdoc_gen.go, plugin/webhooks/{openapi,mcp}.json,
and openapi.yaml woven from the subsets.
visor.go:230-245 already decomplected this surface: one resource that was spelled
three ways (POST .../bind-agent, GET|DELETE .../agent-binding, GET
/v1/agent-bindings) is now GET /v1/machines/agents and GET|PUT
/v1/machines/:id/agent. The manifest row kept the old collection prefix, so the
light host has been reserving a subtree no app serves.
A stale prefix is not inert. cloud.Declare builds the subsystem-attribution and
price index from this table, and a prefix owns its whole SUBTREE — so the entry
is a standing claim on /v1/agent-bindings/* against any app that might later
want it, for routes that were deleted.
The three remaining agent-bindings hits in the tree are OUTBOUND: bots.go:143
and :549 call the upstream Visor service's own GET /v1/agent-bindings. That is
somebody else's api and it stays exactly as it is.
POST /v1/admin/credit-grants -> POST /v1/admin/credits.
A grant is what a POST to a credit collection IS. Spelling it in the address
made the verb appear twice and turned a plain resource into a compound, and it
is the only compound under /v1/admin/{orgs,users,roles,products,compute,money,
bases,flags,…} — every one of which is the bare plural of the thing it manages.
The upstream contract does NOT move: commerce owns the ledger and still serves
POST /v1/billing/credits, the body is still commerce's CreateCreditGrant
forwarded whole, and the type that holds it is still creditGrantIn — that name
is a statement about whose contract it is, which is exactly why it should not
be renamed to match our address.
The operationId is hand-set here (zip.WithOperationID), so it does not follow
the path automatically and had to move deliberately: adminCreateCreditGrant ->
adminCreateCredit, matching the adminCreate<Resource> shape the rest of this
router uses. createCreditGrant -> createCredit and creditgrant.go -> credits.go
so the route, the handler and the file agree.
Also repaired here: the committed subset's prose still said this route forwards
to "POST /v1/billing/credit-grants". commerce renamed that to /v1/billing/credits
and the projection was never regenerated, so the published document has been
naming an address commerce does not serve. Regenerating from the source fixes
it in the same pass.
Regenerated: apps/admin/zipdoc_gen.go, plugin/admin/{openapi,mcp}.json, and
openapi.yaml woven from the subsets.
manifest.Apps "zero-trust" -> "zt", and plugin/zero-trust -> plugin/zt.
This is the only hyphenated subsystem name in the fleet, and a subsystem name
is not a label: serve.go:440 mints GET /v1/<name>/health for every subsystem,
so `zero-trust` was publishing a live compound address, /v1/zero-trust/health.
subsystem.go:78 derives the default prefix from the same string. The name is
api surface whether or not anyone meant it to be.
`zt` is not an abbreviation invented here — it is what the code has called this
thing all along. The package is apps/zt, the Mount is zt.Mount, the upstream
service is hanzoai/zt. The manifest row was the only place still spelling it
out, and it forced a translation table to exist: gen-app-cmds carried a pkgOf
entry "zero-trust"→"zt" purely to get from the app name back to the package it
lives in, and openapi/synopsis.go and mk/plugin.mk each documented the same
exception. All three lose it — the name-derived guess is now simply right, and
one fewer name means one fewer mapping to keep true.
OPERATOR-VISIBLE, and deliberate: the per-app env overrides are derived from
the app name, so CLOUD_ZERO_TRUST_ADDR / CLOUD_ZERO_TRUST_BIN become
CLOUD_ZT_ADDR / CLOUD_ZT_BIN. Anything setting the old names must move.
TestPluginResolution asserts the new one.
The four served routes do not move — /v1/networks[/:id], /v1/mesh/services and
/v1/edge/nodes were never under the subsystem's name, which is what
plugin/zt/main.go's comment exists to explain. Nothing in openapi.yaml changes:
the health routes this name mints are not in the projection.
GET /v1/search-docs/indexes -> GET /v1/search/indexes
GET /v1/search-docs/stats -> GET /v1/search/stats
`-docs` said "this is the document search, as opposed to the vector search",
which the sibling /v1/vector/{collections,stats} already says by being a
different address. What is left is `search`, and its two reads nest under it
the way vector's already do — so the subsystem now reads
/v1/{search,vector}/… with one shape instead of one compound and one nest.
The reason it was a compound is that provisioning already owns the shallower
/v1/search (it PROVISIONS a search resource and answers /v1/search/:name).
That is not a blocker, it is a solved problem: manifest/apps.go:62-70 records
the identical pair for /v1/s3 — storage names the deeper /v1/s3/buckets and
/v1/s3/health, provisioning keeps /v1/s3, and longest-prefix match separates
them, with /v1/vector (provisioning) against /v1/vector/collections (product)
as the second instance. This is the third, written the same way: product names
/v1/search/indexes and /v1/search/stats, provisioning keeps /v1/search.
Proven, not asserted: manifest's TestEveryServedPathReachesTheAppThatServesIt
routes all 1057 published paths through the real prefix table and reports the
same 18 known-unreachable entries as before this change, with neither new path
among them — i.e. /v1/search/indexes reaches `product`, not `provisioning`.
apps/product's own suite is green.
`indexes` and `stats` are now names a provisioned search instance cannot take.
That is the cost of the nest and it is the same cost /v1/s3/buckets already
pays.
Regenerated: apps/product/zipdoc_gen.go, plugin/product/{openapi,mcp}.json, and
openapi.yaml woven from the subsets.
GET /v1/pricing-policy is removed. GET /v1/pricing/policy stays.
This one is not a rename. The two routes ran the SAME expression —
sectionOf[pricingBlob](ctx, o, "policy") — and the alias's own doc comment said
so: "the same document GET /v1/pricing/policy returns, byte for byte, at the
shorter address the marketing surface links to". One document, two addresses,
two operation ids, two SDK methods, two MCP tools. There is one way to do
everything, so the compound spelling is the one that goes.
Nothing in the fleet called it: no console, no studio, no commerce reference —
the marketing surface it was cut for can link to /v1/pricing/policy, which is
shorter to say than to explain.
It also pays a structural debt. apps/pricing declared FIVE prefixes because
this alias sat at the top level instead of under /v1/pricing; it now declares
four, and manifest/apps.go's literal copy loses an entry with it. cloud.Declare
builds the price index and subsystem attribution from that list, so a prefix
that exists only to carry an alias is a real cost, not bookkeeping.
The openapi encoding rule that this pair motivated is untouched and still
tested: '-' must not fold to '_', or two addresses collapse onto one id. Its
prose said "the live router serves both" — no longer true — and now says what
is: the pair is gone, the rule still governs the hyphenated addresses we do not
own (git-upload-pack, delete-batch) and whatever pair comes next.
Regenerated: apps/pricing/zipdoc_gen.go, plugin/pricing/{openapi,mcp}.json, and
openapi.yaml woven from the subsets.
ZAP over UDS is how our processes talk. But listenOn asked zip.Addr("") for this
process's socket BEFORE anything bound ZIP_RUNTIME_DIR, so every plugin bound a PRIVATE
temp path — observed in prod: commerce listening on
/tmp/zip-commerce-235006849/commerce.sock while callers dialed
/var/lib/cloud/run/commerce.sock.
A stale socket file sat at the shared path from an earlier pod, so the dial failed as
'connection refused' — which reads like the callee is DOWN — instead of 'no such file',
which would have read like it was never there. That is why this looked like a commerce
outage for hours.
Every cross-process plane call was unreachable. For the money ops that is fail-CLOSED:
the AI balance gate could not verify a balance, so EVERY completion answered 503
balance_unavailable — chat, copilot and documents — on pods whose ledger was healthy.
bindRuntimeDir first. It is idempotent and an externally-set ZIP_RUNTIME_DIR still
wins, so this only fills in the default the plane already assumes. Both halves now
agree on one path by construction.
Failures: 35 before, 34 after (one previously-failing test now passes).
A per-org dedicated ClickHouse is capped at 1Gi. Both tenant instances on
hanzo-k8s sit at ~80% of that (808Mi and 820Mi), and one has already been
OOMKilled. Both read 1/1 Running, so nothing pod-shaped shows it.
The cap itself is a product decision and this commit does not change it. What it
changes is that the cap was unreachable: `tag` was already
env("CLOUD_DEDICATED_DATASTORE_TAG", …) while memReq/memLim were compile-time
constants, so an operator could pin a suffocating instance's image during an
incident but could not give it headroom without shipping a release. The one
lever you actually need was the one that was missing.
All four engines now take CLOUD_DEDICATED_<ENGINE>_MEM_REQUEST / _MEM_LIMIT
through the same env() helper, with their existing values as defaults — sql, kv,
datastore and docdb, because a table where one row is operable and three are not
is the kind of asymmetry that gets discovered during the next incident rather
than this one.
Behaviour is unchanged unless an operator sets a variable.
⚠ Not based on local main: that branch is 6 ahead / 337 behind origin (six
unpushed commits belonging to someone else's work), so this is cut from
origin/main and leaves it alone.
Hanzo Dev <dev@hanzo.ai>
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/embed-status -> GET /v1/embed.
`-status` was the method spelled twice. There is one embed resource per brand
app, GET is how you read it, and what a GET returns IS its status — so the
suffix added a word without adding an address. Every other read on this surface
already works that way (/v1/csrf, /v1/keys); this one had carried the shape of
the console route it replaced.
manifest/apps.go carries a literal copy of account's prefixes for the light
host to route on, so it moves in the same change. The comment at the top of
embed.go stopped citing console's app/embed-status/route.ts: that file is gone
— this is now the only implementation — so the citation pointed at nothing.
Regenerated: apps/account/zipdoc_gen.go, plugin/account/{openapi,mcp}.json, and
openapi.yaml woven from the subsets.
POST /v1/wallets/:id/safe-tx -> POST /v1/wallets/:id/transactions.
Two faults in one name. `tx` is an abbreviation of the noun, and `safe` was
already said: custody is a property of the WALLET, the route reads it off the
wallet (`safeCustody` answers, any other custody is a 400), and the caller
never chooses it. What the route creates is a transaction on that wallet, so
the collection it posts into is `transactions` — beside the /:id/keys and
/:id/sign this surface already spells out.
The Safe protocol's own vocabulary does NOT move. SafeTx, SafeTxResult and the
`safeTxHash` field are Gnosis Safe's EIP-712 names — the hash a Safe contract
verifies on-chain is called safeTxHash by the contract, not by us — so renaming
them would be renaming somebody else's contract. Only our address changes;
ops.proposeSafeTx follows it to proposeTransaction so the handler and the route
read the same, and the private safeclient.proposeSafeTx (which speaks to the
ring, not to our callers) keeps its name.
Regenerated: apps/wallets/zipdoc_gen.go, plugin/wallets/{openapi,mcp}.json, and
openapi.yaml woven from the subsets.
THE OUTAGE. Every paid completion in the fleet answered 503 balance_unavailable
on v1.801.340 while every subsystem was healthy.
The cloud pod keeps its run directory on a volume, so /var/lib/cloud/run outlives
the pod that wrote it. commerce is a LAZY app — nothing starts it until someone
asks the router to. A previous pod left commerce.sock behind (mtime 23:10, pod
start 23:41, no listener for it in /proc/net/unix) and reach() decided the peer
was up by STAT'ing that file. So commerce was never woken, the dial hit a kernel
with nothing to hand it, and apps/billing read the failure as "this fleet runs no
commerce" and fell through to an HTTP proxy that is unset in this deployment.
ai's balance gate is fail-CLOSED, so an unreadable balance denied everything. The
whole finance surface went with it: /v1/billing/{balance,usage} and
/v1/finance/{balance,usage,ledger,credits} answered 501 to signed-in customers.
It was invisible for three days because the error was dropped on the floor: one
line turned EVERY cloud.Ask failure into "split deploy", so nothing logged the
reason — after that line nothing had it.
1. plane.go — reach() proves a LISTENER; listening() connects, exists() is gone.
A file that outlives its process is not evidence of a peer, and for a lazy app
that is the whole answer: the file suppresses the wake that would have put a
listener behind it, so "here and broken" (learnable from the call) is really
"here and never coming up" (not). ENOENT and ECONNREFUSED are one fact — no
listener — and both are the wake path's business; any other dial error is an
outage and is returned as one. It removes nothing: the listener already unlinks
a stale path when it binds, so the run dir keeps ONE writer.
2. apps/billing — only cloud.ErrNoPeer may be read as "fall back". Ask's own
contract says so; availableCents, coResidentUsage and peerTxns each collapsed
it with every other error. Now a failed read is (ok=true, err) → the existing
s.Log.Warn + 502 "billing upstream unreachable", which NAMES it. peerTxns
grew the error return its caller needed to say that.
3. middleware_identity.go — the isTrustedServiceToken restore of X-Org-Id was
dead code and is deleted, with the predicate that served only it. The
no-verified-principal tail already restores cliOrg unconditionally, and a
service token is not a JWT, so the early copy could never change an outcome.
It also wrote the header as a string literal, which is what
TestEveryHeaderWrittenIsAName had been failing on; that test is green again.
Its reasoning moved to the line that does the work.
TESTS, each red before its fix and green after:
TestWakeStartsALazyAppBehindALeftoverSocket — real router, real lazy child,
real leftover socket: before, "connection refused"; after, the app comes up.
TestStaleSocketIsNotAPeer — a leftover file reaches the SAME answer as no file
(ErrNoPeer), plus TestALiveSocketIsAPeer so the fix is not "always absent".
TestBalance_APlaneOutageIsNotASplitDeploy — a present, broken commerce peer is
502, never a plausible 200 from a proxy nobody asked; and
TestBalance_NoCommerceInTheFleetStillFallsBack keeps the real split deploy.
middleware_identity_s2sorg_test.go now asserts the BOUNDARY. It re-implemented
the token compare in the test file and asserted the copy, so it passed
whatever the middleware did — it could not have caught the org being dropped,
which is the one thing it was written for.
Test scaffolding: the plane's run dir in tests is now a SHORT mkdtemp. A unix
socket path is capped near 104 bytes and t.TempDir() spends most of it on the
test's own name, so 4 tests failed on darwin with "invalid argument" about
something they were not about. Pre-existing and unrelated; fixed because the new
router test cannot run otherwise.
Baseline compared before and after across every plane consumer (apps/admin, ai,
deploy, marketing, marketplace, o11y, platform, projects, tools, x402, credz):
no new failures, none of the pre-existing ones claimed.
GET /v1/books/trial-balance -> /v1/books/trial
GET /v1/books/balance-sheet -> /v1/books/position
POST /v1/books/bank/link-token -> /v1/books/bank/token
The group already reads {accounts, gl, pnl, export, questions} — single nouns,
with `pnl` establishing that this surface uses accounting's own short form
rather than spelling the statement out. Two members broke that rule for no
reason: `trial-balance` and `balance-sheet` are the colloquial long forms of
reports whose neighbours are already short. So: `trial`, and `position` — the
statement of financial position, which is what a balance sheet IS under IFRS
and the name the standard actually gives it. The group is now
{accounts, gl, trial, pnl, position, export, questions}.
`link-token` was Teller's and Plaid's word, not ours. The bank group is
{exchange, import, sync, transactions, unreconciled} — verbs and nouns we own —
and this route mints a token, so it is `token`. bankLinkTokenHandler follows it
to bankTokenHandler.
The report names are also VALUES, not only addresses: AskResponse.Sources names
the report each figure came from, and it was emitting the old spellings. Those
move too, or the answer would cite a report at an address that no longer
exists. Prose that means the accounting CONCEPT rather than our route keeps the
concept and loses the hyphen (metrics.go's "balance sheet convention"), so no
comment can be mistaken for an address.
manifest/apps.go carries a literal copy of the served prefixes for the light
host to route on; it is updated in the same change, and re-sorted, because the
two are kept equal by hand.
Also here, because without it none of the above could be run: apps/books had no
compiling test binary at all. types.KMSClient grew DeleteSecret and neither
in-memory fake implemented it, so `go test ./apps/books` was a build failure
rather than a result. Adding the two methods brings 35 tests back, including
TestEveryRouteRefusesWithoutAValidatedPrincipal — which walks the route table
this commit renames.
Regenerated: apps/books/zipdoc_gen.go, plugin/books/{openapi,mcp}.json, and
openapi.yaml woven from the subsets.
POST /v1/agents/targets/:id/claim-key -> POST /v1/agents/targets/:id/key.
The compound spelled the verb twice. The sibling POST /targets/:id/claim is
what a claim IS, so "claim-" on the key route was the reader's second guess at
which of the two addresses does the claiming, not information. What the route
returns is the target's key; POST is what mints or rotates it. That is exactly
the rule wallets already follow at POST /v1/wallets/:id/keys, and the header
the key travels in was already X-Target-Key, not X-Claim-Key — so the address
now agrees with the wire.
The CREDENTIAL keeps its name. It is a claim key: the thing a `hanzo code
--serve` daemon presents to claim work, and the doc comments, the store's
UpsertClaimKeyHash and the claimKey field still call it that, because that is
what it is. Only its ADDRESS was a compound.
Regenerated: apps/agents/zipdoc_gen.go, plugin/agents/{openapi,mcp}.json, and
openapi.yaml woven from the subsets.
POST /v1/admin/treasury/bind-anchor -> PUT /v1/admin/treasury/anchor/signer.
`bind-anchor` hyphenated a verb onto a noun that was already there: the
sibling POST /v1/admin/treasury/anchor names the anchor, and what this route
sets is that anchor's SIGNER. Nesting says the same thing with the address
instead of a compound word, and the method carries the verb — so the name is
one noun per segment, the way /v1/admin/treasury/{policy,sweep,seed} already
read.
PUT, not POST, because the operation is idempotent by construction: it
provisions-or-resolves the caller org's treasury wallet on the MPC ring, and a
repeat resolves the same wallet. That was already true and already documented;
the address now agrees with it.
adminBindAnchor -> adminSetAnchorSigner (matching adminSetPolicy next door),
and bindOut/bindData -> signerOut/signerData, so the Go names, the wire schema
names and the route cannot disagree. The doc comment's leading identifier was
`BindTreasuryAnchorSigner`, which named no symbol in the file — zipdoc lifts
prose verbatim, so the spec had been carrying that dead name as the summary of
this operation. It is now the real one.
Regenerated: apps/treasury/zipdoc_gen.go, plugin/treasury/{openapi,mcp}.json,
and openapi.yaml woven from the subsets.
GET /v1/billing/balance answered 401 "sign in to view billing" to ai's prepaid
gate on cloud v1.801.340, and that gate is fail-CLOSED, so every paid completion
fleet-wide — chat, copilot, documents — returned 503 balance_unavailable.
The S2S rule was right and it ran. The request log carries the proof: the same
line holds org=hanzo AND the refusal. balance() admitted the caller, resolved
the org from the gateway-pinned header, then delegated to proxy() for the
split-deploy leg — and proxy() opened by asking principal.Org a SECOND time,
without the rule, and denied what balance() had just admitted.
The split leg is not exotic; it is what prod runs. A plugin process starts as
cloud.Serve(…, []string{"billing"}), so cfg.Enabled("commerce") is false,
wireFinance returns before finance.Publish, and finance.Current() is nil in that
process forever. Every existing test published a ledger, so balance() always took
the co-resident return and only the first resolution was ever exercised.
So the rule moves out of balance() into readerOrg — one tenant resolution the
billing READ surface shares. Scope is unchanged: the token is still compared
constant-time against COMMERCE_SERVICE_TOKEN by the predicate apps/account owns,
the org still comes from X-Org-Id (which the gateway strips from every client
request), and it confers no user, no admin, no roles. The money WRITES
(gpuCharge, createPaymentMethod) and the user-scoped breakdown (usageAccounts,
which needs c.User()) keep asking principal.Org and refuse it.
Tested on the shape prod runs: no ledger published, so balance() reaches the
proxy leg. RED without this change with the exact prod body, and the
scope-widening guard is repeated on the second resolution.
THE OUTAGE: every AI completion answered 503 balance_unavailable — chat, copilot and
documents — on pods whose ledger was perfectly healthy.
ai is its OWN process (prod pod: /cloud pid 7, /billing pid 111, /ai pid 83), so
cloud.BalanceReader() — a package-level var wireFinance sets in the CLOUD process — is
ALWAYS nil there. The ai module then used its HTTP fallback: a self-call to
/v1/billing/balance bearing COMMERCE_SERVICE_TOKEN and no user. That is not a validated
principal at the edge, so it answered 401, and the balance gate is fail-CLOSED. Routing
a money read back through our own PUBLIC edge was the mistake: the edge is for
customers, the plane is for us.
commerce already publishes both money ops (apps/commerce/balance_rpc.go) precisely
because the ledger has ONE writer and must be ASKED, not opened. Ask it the way every
other in-tree caller does (apps/admin/finance, apps/marketplace): typed, ZAP over the
canonical unix socket, org-scoped by the caller — no HTTP hop, no token to mint, no
edge to satisfy. Both directions cross the same way, and the debit keys on the SAME
wallet the gate read, or spend could outrun the balance that admitted it.
Co-resident stays direct: when the process DOES own the ledger, an in-process read
beats a socket round trip. Tested: both plane ops present and org-scoped, and no code
line in this package may name /v1/billing/* (comment lines excluded — a test that
cannot tell an explanation from a call would forbid documenting the bug).
Failures unchanged: 36 before, 36 after.
zipdoc_gen.go is compiled INTO each binary and registers the doc comments
with zip.Describe at init, so a stale one is prose the document never sees.
apps/analytics and apps/kms were both behind their source: analytics' error
and event read views carried full descriptions in the handler and none in
the generated file, so every SDK, the CLI and the MCP door published those
operations bare. Regenerated from source; the wire is untouched.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A plugin is a process, and only the integrations process holds an org's
Slack bot token (TokenFor reads THAT process's in-memory connection
store). My earlier package-global seam (cloud.SlackSend) was as isolated
as the integrations.mounted var it replaced — o11y (a different process)
read a nil copy either way, which is why the forward hit 'integrations:
not mounted' whether it reached into integrations' global or cloud's.
Fixed the native way, the same one x402 uses to move money across the
plane: integrations publishes an internal op (plane.IntegrationsSlackSend,
served on cloud.Plane()'s unix socket, never on the edge), and o11y pages
by cloud.Ask over that socket. The op RUNS in the integrations process, so
SendSlack sees the real token store; cloud.Ask wakes integrations if it is
asleep, so it needs no eager mount (the manifest Eager flag is reverted —
every other plane caller relies on the same wake). The org is the
caller's (cloud.Who(ctx).Org, stamped by cloud.For), never an argument, so
a caller cannot post as another tenant — pinned by test.
Reverts the shared-package seam in obsevents.go; keeps the summary.go
request-gate allowlist from the prior commit (still correct).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Under /v1/billing the qualifier was already said by the prefix, so each name said it
twice and hyphenated to do so. Where the qualifier names a real resource it becomes
one — nesting, not a hyphen — so nothing is invented and nothing abbreviated:
auto-recharge -> recharge (auto is what it IS)
credit-grants -> credits
gpu-charge -> gpu/charge (resource gpu, action charge)
gpu-eligibility -> gpu/eligibility
payment-config -> settings
payment-methods -> methods
test-mode -> mode
usage-rollup -> usage/rollup (a rollup OF usage)
Renamed at every reference — routes, manifest prefixes, the metering gate's
in-process dispatch, the admin surface, and the prose documenting them — so there is
ONE name, never a route and a comment that disagree. No aliases.
Left alone deliberately: /v1/auto/* and the Stripe-shaped resources
(balance-transactions, credit-notes) are external contracts — someone else's names.
Test failures unchanged at 25 before and after.
Under /v1/billing the whole surface is about spend, so 'spend-alerts' said it twice
and hyphenated it to do so. The resource is alerts. Renamed at every reference —
routes, the manifest prefix list, the metering gate's in-process dispatch, the admin
surface and the prose that documents them — so there is ONE name, not a route and a
comment that disagree. commerce (the handler side) and console (the caller) move in
the same breath; no alias is left behind.
SNAPSHOT of the implement pass (authored against ebd0851d): the write core lands
facts on event.event/event.error via the envelope writers instead of the wide
hanzo.events INSERT; every read lens moves to the plane (eventsTable =
signalEvent.table(), /v1/errors -> event.error); pins moved with the write path.
Rebase onto current main follows as its own commit so conflicts with the
event-plane contract commits (b8eb12b2, 269da06a, 719e044e) resolve visibly.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The connection is keyed (org, provider, owner), and nothing set the owner — so
every GitHub account would still have collided on the empty string while the
schema looked correct. The key only helps if the write fills it.
Which providers get one is now declared rather than assumed: MultiAccount marks a
provider that can be connected once per provider-side account, which is what a
GitHub App is (installed per account, one org may hold hanzoai, hanzo-apps and
hanzo-docs). Everything else keeps the empty owner its callers resolve, so no
single-account provider changes shape.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every credential operation failed closed for months while the error blamed
CLOUD_KMS_MASTER_KEY_REF, which was correctly configured the whole time. The
cause was `deps.KMS.(*kms.Client)`: exactly one process owns the store, and every
other app is handed a peer that reaches it over the internal plane, so that
assertion could never succeed in integrations. It now uses deps.KMS as the
interface it already is, addressing secrets by the ref grammar (path/name@env)
that carries the same coordinates the store holds.
That needed one op the plane did not have. DeleteSecret joins the interface and
is exposed as kms.delete, because an app custodying a credential on a customer's
behalf must be able to remove it when that customer disconnects — otherwise the
connection row goes and the material stays. It widens no boundary: the surface is
narrow because material LEAVING is the risk, delete moves nothing outward, and a
caller that can put can already overwrite a secret into uselessness.
Readiness now asks whether a store is reachable at all. A remote store's health
cannot be known without a round trip per check, and a store that is wired but
unhealthy says so on the operation, where the caller can act on it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Four tests against a REAL embedded JetStream, because the defect lives in
JetStream's own rule — reconcile by NAME, bind by OWNERSHIP — and a stub with a
canned overlap error would only prove the stub. Each was run against the
un-fixed code and fails there:
retires an earlier generation the outage itself: EVENTS holds event.>,
EVENT cannot bind, ingest is 503 forever
refuses to destroy undrained data a stream with messages is named in the
error, never deleted
never retires a tenant stream unreachable today; the check is what keeps
a regressed door from costing a customer
their stream
the probe follows the stream delete it under a live connection and the
probe must go red, not ride its cache
The published contract said health reports warehouse connectivity, which is what
it did while ingest was down. It now states both halves, that either one degrades
it, and that the lens block is absent only when the warehouse is unreachable —
not when the plane is.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
JetStream binds subjects by OWNERSHIP but CreateOrUpdateStream reconciles by NAME,
so renaming the plane strands the old stream on event.> and the new name can never
bind it. The store is durable, so the stale stream outlives the code that made it:
every publish 503s and no restart, redeploy or rollback clears it.
That is what took ingest down. EVENTS (webhooks' own declaration, retired in code
when analytics became the plane's one owner) still held event.>, so EVENT could not
bind and 100% of POST /v1/event answered 'subjects overlap with an existing stream'.
EnsureEventStream now retires the earlier generation and retries. It refuses to
destroy data — a stream carrying messages is named in the error with its depth
instead of removed — and it never touches a tenant stream, which the tenant door
already makes unreachable and this now double-checks before deleting anything.
/v1/analytics/health reported ok on warehouse connectivity alone while every write
failed, so the outage was invisible. It now probes the plane by walking the ingest
path itself — same connect, same stream, same names, so the probe cannot disagree
with production — and degrades to 503 when a write would fail.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
mk/fleet.mk derives its app set from `wildcard apps/*/Makefile`, and apps/zen had
no Makefile. Not skipped — never returned by the glob. So surface-check, the only
gate that regenerates a subset FROM SOURCE and fails on any diff, has never once
re-derived plugin/zen, and a set that never contains a name cannot report the name
missing.
Every other protection here is a bijection with plugin/<app> — one_source_test.go
both directions, gen-app-cmds' orphan check, the Dockerfile's per-app existence
check — and zen passes all of them. It has a row, a main, a subset and a
catalogue. The only thing it lacked was anything that would ever regenerate them.
Giving it the Makefile its 112 siblings have turns the invisible hole into a real
failure, which is the point: zen is CORESIDENT, so mounting it alone is refused by
construction ("zen installed middleware at /v1, outside the prefixes it owns") and
it genuinely cannot describe itself. That is a property of coresidency, not a
defect like kafka's live broker, so it is exempt the way kafka is — declared, and
printed when skipped rather than silently passed over.
The exemption is DERIVED from manifest.App.Coresident, not named. The one place an
app declares it is coresident has to be the place this loop learns it, or the next
one goes missing exactly the way zen did.
The test is the part that lasts: it asserts the set the GATE iterates equals the
set the FLEET is — by location for an app with source here, by name for one whose
source is another module — in both directions. Skipping stays a separate question
and a separate list, because a skip is a decision that prints itself and this is
about the names no decision was ever made about.
Mutation-checked, both directions: removing apps/zen/Makefile fails it by name,
and an EXTERNAL entry with no manifest row fails it too.
make -f mk/fleet.mk surface-check
>> skip kafka - needs a live broker to mount (OPENAPI_NEEDS_BROKER)
>> skip zen - coresident: middleware on a sibling's router, no standalone mount
>> openapi.yaml regenerated from source and unchanged - 1059 paths
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
//go:embed all:catalog in apps/agentskills reads apps/agentskills/catalog. The
overlay wrote clients/agentskills/catalog — where every subsystem lived before
f873d1a1 moved them to apps/ — and COPY CREATES a destination that is not there
rather than failing, so the full catalog landed in a directory no package embeds
and nothing anywhere disagreed.
What shipped instead is the fallback the repo tracks so that a bare `go build`
works with zero build deps: ONE skill per brand (ai_models). Measured against the
generator that is 1 of 357 skills x 3 brands, and it has been the whole of
/.well-known/agent-skills/index.json in every image since the move. Correcting the
path costs the agentskills binary 2,662,400 bytes (42,537,122 -> 45,199,522) and is
the only size increase in this branch.
Both build-time overlays now assert they landed. A tracked fallback plus a COPY
that cannot fail is a silent-smaller-binary machine, and the only evidence of the
failure was a number inside a served document — so the claim belongs at the COPY,
next to the path it is about. Each is asserted in the terms its own fallback is
defined by rather than a file count that drifts: one skill per brand, and a static
SPA export carrying no JavaScript at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
mayReadReleases exists and mirrors the release gate — platform sudo, or admin of
the org that owns the image — but both handlers still inlined a SuperAdmin-only
check, so the helper was dead and the read stayed stricter than the cut. A merge
kept the definition and dropped the call sites.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/billing already trusts COMMERCE_SERVICE_TOKEN (account.IsServiceToken) but reads
the org from X-Org-Id, and SanitizeIdentity deletes that header from EVERY ingress. So
the org was empty, billing answered 401 'sign in to view billing', the fail-closed
balance gate could not verify a balance, and every paid completion 503'd fleet-wide —
chat, copilot and documents dead on a pod whose commerce subsystem was healthy. The
handler fix (a5c5808c5) could never take effect because the value it needs was already
gone.
Restore the org, and ONLY the org, at the one place it is removed. No user, no admin,
no roles: a request bearing this token still cannot be an admin. The safety argument is
the one the token already rests on — the gateway 401s a public bearer that is not an
IAM JWT or an hk-/pk-/sk- key, and the 64-hex token is a JWT candidate that fails to
parse, so no external client can present it; only in-proc dispatch reaches a handler
holding it. Constant-time compare, never logged.
(5 pre-existing TestGPUCharge failures in apps/billing are unrelated — identical with
this change stashed.)
Every route apps/projects serves is now a typed zip.Get/Post/Patch/Delete
with a real In and Out and a doc comment, except the two the WIRE refuses.
A typed op is the only registration that reaches OpenAPI, the SDKs, the CLI
and the MCP door with a SHAPE; openapi.Describe gave those 39 operations
prose and no schema, which is an SDK method that takes nothing and returns
nothing. The 37 Describe calls for the routes that could be typed are gone —
the prose is the doc comment now, and the schemas are the Go types.
Measured on the regenerated subset: 39 operations, 14 with a requestBody
(was 0), 33 with a typed 2xx content schema (was 0). The six without are
four DELETEs that answer 204 No Content — declaring content on a 204 would
be a documented lie — and the two refusals.
REFUSED, and it is the wire, not effort:
POST /v1/projects/:slug/deploy (projects.go:452)
POST /v1/platform/sites/:slug/deploy (projects.go:479)
Both take a zip or tar(.gz) of the built site — raw in the body or as a
multipart part — OR a JSON git descriptor chosen by Content-Type, and answer
200 for an artifact published or 202 for a build queued. A typed In is one
JSON shape and zip.WithStatus declares one success status, so typing either
would refuse the archive upload that is its main path and mislabel half its
answers. Both keep their openapi.Describe prose.
THE WIRE DOES NOT MOVE. Same 28 paths, same 39 methods, same statuses, same
json tags, same middleware order, same body caps. Four things were needed to
keep it that way and each is pinned by a test in typed_wire_test.go:
- requireBody replays c.Bind's 400 on a bodyless write. zip's typed decode
skips an empty body, so a bodyless PATCH would have become "200, nothing
changed" instead of the 400 it has always sent.
- the bind and list domain answers are TWO Out types. They were two shapes
already (bound vs claims); folding them into one with omitempty would
drop an empty list from the wire — [] becoming absent.
- the two 204 DELETEs return a nil Out under zip.WithStatus(204), so they
stay 204-with-no-bytes.
- the two answers that were map[string]any declare their fields in
alphabetical json-tag order, because encoding/json sorts map keys.
THE MONEY WIRE, decomplected. Six ops gate on balance, and a refusal is the
fleet's nested {"error":{"code","message"}} 402/503 that zip's error envelope
does not speak. resource_billing.go now states that denial ONCE (denial) and
offers it two ways: DenyResource writes it, for an untyped handler; Denied
returns it and DenyEnvelope writes it back verbatim, for a typed op. This is
the blocker apps/storage, apps/ml and apps/company each filed as their reason
for staying untyped; it is now solved in the one place the denial lives.
TENANCY unchanged: the org is org() — principal.Org plus the SuperAdmin
"admin" bucket OrgFrom cannot express — resolved ONCE in apps/projects/typed.go
and never an In field. That is one new entry in allowedRequestUses, with the
three facts (tenant, platform sudo, payer) that earn it.
NOUN GATE: every schema this app publishes is package-qualified —
projectsProject, not projectView, which apps/platform already claims with a
different shape, and projectsDeployment, not deployment, which apps/o11y
claims. The one unqualified name is Record, the shared internal/fqdn type,
which any app using it emits identically.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The alert forward reached integrations.SendSlack directly and hit
'integrations: not mounted' even though /v1/integrations is live — because
each subsystem is a separate plugin with ISOLATED package globals, so
o11y read integrations' own `mounted` var as a nil peer copy. Same reason
the codebase's other cross-subsystem calls (SetObsEventIngest) live in the
shared cloud package, not in a subsystem global.
Fix mirrors that pattern exactly: cloud.SetSlackSender / cloud.SlackSend
in the shared package; integrations.Mount registers a closure that carries
ITS OWN linkage (so calling it runs against integrations' real token
store); o11y pages through cloud.SlackSend. integrations is made Eager so
the sender is registered at boot rather than on the first /v1/integrations
hit — an alert must page whether or not anyone has touched the connectors
UI. Nil seam errors cleanly, never panics (test).
Also allowlists apps/o11y/summary.go in the typed-request gate: it reads
the request Host for white-label brand selection — a host-borne value that
is neither the org nor nameable on an In field. It was another lane's
addition left un-allowlisted, reddening go test ./ for the package.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud's release lived in apps/platform/release.go, reached at POST /v1/runner
{release:true}, and ran build -> smoke -> tag -> rollout in a goroutine detached
with context.WithoutCancel. It ran inside a service at replicas: 1 with strategy:
Recreate, and its final step moved the pin that rolls that replica. Any sync of
charts/app/values/hanzo/cloud.yaml therefore terminated the only pod and took the
pipeline with it — after the image was pushed, before it was smoked, tagged or
pinned.
That is not a race to tune. It is in the tag list: v1.801.326, 327, 329 and 335
exist; 328, 330-334 and 336+ do not. Every gap is a published image nobody can
name. 336 and 337 orphaned that way, and 338 died the same afternoon when an
unrelated CLOUD_ALERTS_SLACK_CHANNEL restore synced and rolled the pod out from
under it.
So the release runs where the rest of the fleet's releases already run. Cloud was
the only service doing it differently, and being different is what broke it.
The two reasons it was written in Go no longer hold. "The runtime image is alpine
with git and no bash, curl or jq" was only ever true because the release was put
inside the runtime; the forge runner is catthehacker/ubuntu:act-24.04 and has all
three, which is why insights, analytics, studio and bot-hub already call pin.sh
from it. And "pin.sh strips the leading v" was fixed in universe c58c1a808 —
pin.sh now reads the prefix off the current pin and carries it through, so a
v-prefixed service stays v-prefixed.
What carries over, because it was right:
The version maxes over BOTH published image tags AND git tags, never one alone
(release.go nextVersion, floor 1.786.0 included in the same sort). An orphaned
build leaves an image with no tag, so git alone would re-emit a live tag; a tag
can outlive its image, so git alone is not a superset either.
The smoke gate, byte-identical and still BEFORE the tag and BEFORE the pin. It
boots the pushed image and requires "zip listening" with no crash signature, so
a tag can never name an image that did not start. It runs under docker rather
than as an in-cluster Job because launchSmokeJob was itself "the in-container
mirror of release.yml's docker-run smoke" — docker was the original home, and
going back needs no kubeconfig and no RBAC in CI.
The build also passes VERSION so the binary stamps the tag it was published as,
instead of reporting the `dev` ldflag default.
release.go's in-pod path is not removed in this commit: it is retired once this
one is shown to release cleanly, so the fleet is never left with neither.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ai runs as its OWN process (prod pod: /cloud pid 7, /ai pid 83) and these hooks are
package-level vars, so a reader wireFinance sets in the CLOUD process is invisible in
the ai one — permanently, not just at mount time. The ai module treats a NIL hook as
'use my own path' (HTTP to /v1/billing/balance, which cloud now accepts with the S2S
token). My previous commit installed a hook unconditionally that merely reported the
host had none, which SHADOWED that fallback; the balance gate is fail-closed, so every
completion answered 503 balance_unavailable — chat, copilot and documents all dead on
a pod whose commerce subsystem was healthy.
Install a money hook ONLY when this process has one. In the cloud process the snapshot
is safe by construction: wireFinance runs in BuildDeps, which completes before
MountAll. RollingCapReader stays a real trampoline — it IS installed after this
package, and a missing cap is benign.
Three defects on the one ingest door, each of which returned success while
losing or misfiling the caller's events.
The canonical Event carries its KIND. /v1/event publishes three shapes
(openapi.OneOf{Event, []Event, CaptureBatch}) and they have to mean the same
thing. Event had no `type`, so toCapture left it empty, canonicalType mapped
empty to "event", and "event" is not in publicKinds — so a bare object or bare
array was dropped on the anonymous lane every time, with a 200 receipt, while
the identical event inside {batch:[…]} was admitted. An SDK generated from that
document picking the simplest of the three shapes lost 100% of logged-out
traffic and reported success. Carrying the kind is a WIRE fix, not a capability
one: the allowlist is still the whole anonymous surface.
An unresolvable platform key on the bearer carrier REFUSES. presented() names
the carriers eventTenant consults so the two cannot disagree about what
"presented" means, and they did: ingestKey matches the bearer only for pk- (an
hk-/sk- bearer is IAM's to validate, and widening ingestKey would shadow the
identity path), and projectKey never reads Authorization. So an hk-/sk- bearer
that failed to resolve fell through both onto the anonymous lane — 200, rows
filed under $public, a partition the caller's org cannot read. The same key on
x-api-key already refused. A revoked or mistyped key on the carrier every caller
reaches for first is the likeliest misconfiguration there is.
admission reads the ONE key-prefix authority. Its local copy of
{pk-,sk-,hk-} existed "so admission stays self-contained (no cloud-internal
import)", which was never true — waitlist.go in the same package already imports
cloud. The copy bought nothing and cost a second place to edit.
openapi.yaml and plugin/analytics/openapi.json regenerated from the router.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
DELETE /v1/git/repos/{name}, /keys/{id}, /repos/{name}/subscriptions/{id}
and /repos/{name}/mirrors/{id} each return a nil *Out, which zip writes as
204 with no body — git's own tests have always asserted it (git_test.go:188,
ssh_test.go:118, lifecycle_test.go:95,170,434). The document said 200 with a
`noContent` JSON body, so every SDK generated from openapi.yaml expected a
status and a payload the service has never sent.
The cause is one character. zip keys a 204 response on the Out type having NO
NAME (openapi.go: a named OutType publishes "200 with a body"), and git was
the only app in the fleet that DEFINED `noContent` instead of aliasing it —
and therefore the only publisher of a `noContent` schema. `type noContent =
struct{}` restores the fleet's one form; the wire is untouched.
TestVoidOpsPublishTheStatusTheySend makes it a gate rather than a fix: over
every typed op, in both directions — a void op must publish 204 and no
content, and every other typed op must publish a 2xx that carries a schema.
It fails on the old shape, all four ops, with the reason.
Also true-up the prose the gate exists to replace: untypedByDesign holds 30
refusals, not 24 (the project-scoped :org/:project/:repo trio doubled the
smart-HTTP family to 12 and the declared pack bodies to 8), the host-gated
root paths are 12 and not 9, and both closed classes are recorded — the
bodyless POST on /gc and the raw newline in 215 fleet summaries, now 0.
The ZAP five keep their refusal and gain the fact behind it:
zapface/dispatch.go:89-91 forwards env.Msg to the ZAP client as its error
text, so a typed op's {..., error} body would decode to an empty Msg and
every ZAP failure would arrive with no message at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud.hanzo.ai and console.hanzo.ai both answered production traffic with
`x-api-version: dev` — the untagged default in version.go — so the header
carried no support-correlation signal at all.
buildFrontendCmd already hands the published tag to the build as
`--opt build-arg:VERSION=<tag>`, but nothing consumed it: the builder stage
declared no `ARG VERSION` and the link used a bare `-ldflags="-s -w"`. Declare
it and link it, and the binary reports the ref it was pushed under, derived from
the image tag so it cannot drift from what was published.
Verified the flag form and symbol path against an isolated module of the same
shape (package `cloud` at the module root, main under cmd/cloud): default build
prints "dev", `-X github.com/hanzoai/cloud.Version=v1.801.218` prints
v1.801.218. Not verified end-to-end in-cluster — that needs a BuildKit release
build, which does not run locally.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
63 of 65 tenants' commerce databases would not open: encryptTenantFile aborted on the
tables hanzoai/replicate creates in every tenant file, left the file half-migrated, and
printed nothing. Every commerce op on those tenants then errored, which is why
GET /v1/store 500'd, the AI balance gate could not verify a balance and fail-closed
503'd EVERY completion, and studio renders sat held behind 'no available server'.
Also carries the fresh-store fix: Listings was nil, so the first listing upsert — the
one write that makes a storefront non-empty — 500'd on a nil map.
Two customer-money defects from #376, both proven by a test that failed first.
gpu-charge was not idempotent. `requestId` rode the body into commerce's
transaction metadata, where it deduplicated nothing — two identical posts were
two debits, and the endpoint's own description said so instead of fixing it. It
now keys the debit on that same field through the ledger's own (kind, ref)
idempotency, which is what every other debit in this fleet already does: the
edge meter and x402 both hand finance a RequestID and it dedups inside the same
transaction as the insert. Nothing was invented and no field was added; the key
was already on the wire and simply did nothing.
The debit also moves the wallet it is supposed to move. Co-resident, a
customer's prepaid money is cloud's own finance ledger — the file balance reads,
the ai gate admits against, the meter debits — and commerce's transaction store
is left empty in this binary, so a GPU charge written only there moved nothing a
customer or a gate could see. Both gates survive and both still fail closed: a
chargeable card on file, and prepaid alone covering the charge, now read from
the same wallet the debit posts to. A gate that cannot be READ refuses.
RecordUsageOnce answers the ledger's idempotency instead of merely applying it,
from inside the insert's transaction, so a replay is told apart from a first
charge without a second store to ask; RecordUsage delegates to it, so there is
one body. Split deploy keeps the proxy — nothing there to key on — and forwards
the caller's key to the process that does the write.
The ledger's entry kinds are now ONE vocabulary, finance.Kind, held by the
package that writes them and imported by the package that reads them. The reader
had its own string literals: it matched commerce's `deposit`/`withdraw` against
entries this ledger has always written as `finance.deposit`/`finance.usage`, so
whenever the peer path served, a customer's credits page rendered EMPTY, usage
totalled 0, and their own grant signed NEGATIVE. Nothing failed; the strings just
never met. Comparing typed constants makes that a compile error. Each wire is
translated at its own boundary — the plane by ParseKind, commerce's HTTP by
commerceKind — so the three projections classify one typed value and a third
spelling cannot be introduced quietly.
No stored row is affected: apps/finance is the only writer of an org's ledger
file and has written these two kinds since the ledger's first commit. The defect
was entirely on the read side, so there is nothing to migrate.
The blindness is the third fix. finance_test.go stayed green through all of the
above because it only ever answers from the S2S mock, which speaks the
vocabulary the reader expected. apps/commerce/ledger_wire_test.go now drives the
real peer path end to end — a real per-org ledger, the real op on a real socket,
the real reader — with the S2S server wired to fail the test if it is ever
reached. It is the test that was red first.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The event plane was complete and never connected. normalize had no production
caller, publish had no production caller, and the drain was never constructed —
so every signal reached the legacy wide table and stopped there. event.error is
the one table with a shape for a message, a class and a stack, and nothing has
ever written to it: a browser error was stored as a generic product row and was
not queryable as an error at all. The materialized view standing in for the
missing writer could only ever carry product events and excluded errors by
construction, which is why the orphans are all and only errors.
So ingestEvents publishes the fact and analytics starts the sink. ONE admission
decision — the plane's own normalizer, the only one that can see a signal —
feeding two projections: the fact that makes an error queryable as an error, and
the wide row the read lenses still select from. Publish is the FIRST commit
because it is the idempotent one: every event.* table is a ReplacingMergeTree
keyed on the fact id and absorbs a re-send, while hanzo.events is a plain
MergeTree that keeps the duplicate forever, so the failure that asks a client to
retry has to happen before the write that cannot absorb one.
The fold no longer erases the exception it copies into the property bag. Erasing
was right while the bag was the only place an error could go; faultOf reads that
field to build the fault, so a nilled one produced event.error rows with no
message, no class and an empty group — and group leads that table's ORDER BY, so
the row was not merely thin, it could not be assembled into an issue.
A signal with no writer is REFUSED at the door rather than accepted and discarded
four hops later on a subject nothing drains. The accepted set is derived from the
writer list so the two cannot drift, and metric is outside it for a tenancy
reason rather than an unfinished one: event.metric has no org column at all, and
its (env, temporality, metric_name, fingerprint) identity hashes two orgs
reporting the same metric name and labels into ONE series, so landing a sample
is a cross-tenant write. Fold org into the fingerprint and the writer list grows
by one row and the door starts accepting metrics the same day.
The plane also stops losing acknowledged data quietly. It RECONCILES its own
configuration instead of returning early whenever the stream exists, which made
every constant here decorative on a live deployment, and it discards NEW rather
than OLD: a full stream now refuses the publish and the door answers 503, where
before it deleted the oldest undrained facts — precisely the ones no consumer had
reached — to make room for facts nobody had answered for yet, reporting an error
at neither end. What can still be dropped is counted and carried on
/v1/analytics/health, where any non-zero value is an alarm: a message that will
not decode, and a fact the bus abandons after maxDeliver, which it announces on
an advisory nothing was listening to. A counter is the floor and not the ceiling
— neither case recovers the fact, and the dead-letter stream that would is its
own change.
Two vocabularies share this plane and their subjects overlap: a product event
named "$error" folds onto event.error, which is also the error signal's subject,
and "$error" is what a browser error with no name of its own is called. Each
consumer now takes the one it speaks. The warehouse lands facts and leaves
envelopes to their own consumer without counting them as lost, which is what
keeps the loss counter worth alarming on; webhooks delivers envelopes and does
not begin sending subscribers a second body in a shape their endpoints were
never promised.
event.event's materialized-view bridge is now a second writer for a table that
has a real one and must be dropped out of band. The event.* DDL is not cloud's
to create or to remove.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
mountProbes has been knocking on 21 services every 30s and recording
hanzo_service_up through the process meter. The meter's only reader
pushed over the ZAP wire, whose endpoint is empty in every deployment we
run — which luxfi/metric resolves to 127.0.0.1:4317, the OTLP trace
receiver rather than the metric one. So the gauge was written, and went
nowhere, and /v1/summary answered 503 because the series it reads did
not exist. Nothing was broken enough to fail; the path simply had no end.
Metrics now leave by collection. The meter provider's one reader collects
into a registry, and apps/o11y publishes it on its own listener for a
scraper to take. That is where metrics are actually kept here: every
metric READER in this codebase — vmquery.go, the SuperAdmin VM proxy,
status.go's up-inventory, summary.go — queries VictoriaMetrics, and VM is
filled by scraping. Unifying the three signals on one transport read well
but sent metrics to a store nothing reads; signals travel to where they
are kept, and metrics are kept somewhere else than traces and logs.
The listener is o11y's rather than the bootstrap's because binding a port
is a decision about ONE process, and every plugin child shares the host's
environment — a listener opened in InstallTelemetry would have all of them
fighting for one address. The app that owns the prober owns the listener
that publishes it.
Metrics also stop depending on a tracing setting. The meter used to be
installed behind the span-destination check, so turning off
O11Y_TRACES_ZAP_INPROCESS silently stopped hanzo_service_up from existing
at all. A meter has no endpoint to configure and so has nothing to gate on.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A release built its image, proved it boots, minted its tag — and then reported
`failed` at the last step, so nothing shipped. rolloutRelease patched an operator
hanzo.ai/v1 App CR. That kind exists, but there is no `cloud` CR for it to patch:
cloud is reconciled by cd.hanzo.ai, whose `hanzo-cloud` Application is
single-source on git.hanzo.ai/hanzo/universe and renders charts/app against
values/hanzo/cloud.yaml. The image.tag scalar in that file IS what runs. Until it
moves, a release is a published image nobody pulls, which is why v1.801.335 had
to be pinned by hand.
rolloutRelease now moves that scalar: clone the branch CD reads, apply the rule
set, prove the tag pullable, rewrite the one scalar in place, commit, push.
THE RULES ARE charts/app/pin.sh's RULES, refusal for refusal: non-semver, an
image the registry cannot serve, a version older than the current pin, a
repository the caller chose rather than the one the values file declares, and a
service with no values file. They are expressed in Go rather than shelled out to,
because pin.sh cannot run here and could not pin this service if it did. The
runtime image is alpine with git and no bash, curl or jq — pin.sh needs all three
— and the container runs as nonroot, so it cannot install them; carrying three
more packages in the production image to reach logic this package already has
(the semver gate is splitReleaseImage, stricter than pin.sh's; the registry probe
reuses registryPullToken) is the worse trade. And pin.sh strips the leading v to
pin the bare form, while this registry holds the v: ghcr.io/hanzoai/cloud:
v1.801.335 resolves and 1.801.335 is a 404, so pin.sh would fail its own
pullability probe on every cloud release. It fails CLOSED, so a human running it
against cloud is safe; it simply cannot be the mechanism. What matters is not
which language the rules are in but whether the two can disagree about what gets
written, and they cannot: the tag pinned here is the exact string proved
pullable.
NO ROLLBACK LEVER. computeReleaseVersion mints a version strictly greater than
every tag in git AND in the registry, so a backward pin on this path is always a
bug, never an intention — there is no env knob to misread. PIN_ROLLBACK stays
where a deliberate act belongs, with the human running pin.sh.
The push credential is read from KMS (orgs/hanzo/deploy/UNIVERSE_PIN_TOKEN@prod)
and reaches git as an http.extraHeader in an environment built from scratch:
never in argv, never as URL userinfo, never over plaintext, and with no other
secret in cloud's environment inherited by the child. An absent or empty secret
stops the release rather than attempting an anonymous push that would fail deep
in the git seam.
universe is the repository the whole fleet deploys through, so another service
can land its own pin between our read and our push. The push is a plain
fast-forward, never forced, so that case is rejected rather than silently
overwriting someone else's deploy; it is then retried, bounded, by re-reading the
new tip and re-applying. Nothing is merged — there is one line, and the newest
read of it wins. A concurrent pin that already moved cloud PAST this version is
refused by the monotonic rule instead of being clobbered.
The pipeline's last step is renamed notify -> pin, because it no longer notifies
anyone; `reached` now reports "pinned".
deploy/values.yaml is DELETED. It was written for a multi-source Application that
was never wired — the live `hanzo-cloud` Application has a single source, and
nothing in this repo or any chart has ever read that file. A second pin that
nothing reads is a second answer to "what is live" that can silently disagree
with the real one, which is exactly the dual mechanism this change exists to
remove.
The failure stays honest. If the pin cannot move, the release fails and says the
image is tagged but NOT live — a release that claims otherwise is worse than one
that fails.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A release died ten minutes into its build: "The node was low on resource:
ephemeral-storage … Container buildkit was using 256Ki, request is 0". The
request being zero is the whole failure — the pod was best-effort for disk, so
the scheduler could place it on a node with no room and the kubelet evicted it
before anything with a declared need. One of five runner nodes was already at
DiskPressure=True while the other four were clear.
A build's working set IS disk: the clone, the module cache, 112 plugin binaries
and the exported layer cache. It now requests 20Gi so the scheduler avoids a full
node, and limits 60Gi so a runaway build cannot take the node down for everything
else sharing it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The console's status panel fetched a domain left over from the upstream project:
one Hanzo does not own, that resolves nowhere, and that anyone could have
registered to answer inside a logged-in admin session. The fetch ran on every
page load.
GET /v1/summary is that answer from api.hanzo.ai instead — the outward
projection of the fleet health probes o11y already runs, so the status page is
derived from the same gauge that pages us rather than a second thing to keep
true. Unauthenticated by construction: a status endpoint that requires a login
is useless during the outage it exists to report, and there is no tenant data in
it. When the availability source cannot be read it answers 503, because 'we
cannot tell' and 'everything is fine' are different answers.
The read required platform sudo while the cut takes admin of the org that owns
the published image, so the caller who started a release could be refused its
status — a 202 handing back an id it cannot ask about, which is the gap these
routes exist to close. Proven live: the identity that cut v1.801.335 got 403 on
its own release id.
One predicate now, mirroring the gate in runner.go, plus a test that reads both
files so the two cannot drift apart again.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Connecting GitHub answered 503 for a store it never touches. Its provider
declares `Secrets: nil` — installation tokens are minted on demand and sealed
nowhere — so sealTokens runs over an empty map and writes nothing, yet the
connect handler required a ready credential store before reaching any of that.
The gate now asks whether THIS provider custodies anything. A provider with
secrets still requires the store; one with none no longer waits on it.
Why the store was never ready is separate and worth stating, because the error
said otherwise for months: integrations asserts deps.KMS to the embedded
*kms.Client, and it does not get one. The boot log now says so directly —
`deps.KMS → the kms app over the internal plane` and `type=cloud.KMSPeer`. That
is the architecture working as intended (one owner per store, everyone else asks
over the plane), not a misconfiguration, and the plane deliberately exposes
kms.get/put/sign and nothing more. A provider that seals secrets therefore still
cannot connect in this process; moving those onto the interface needs Delete on
the plane, which is a boundary decision rather than a repair.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
1465 published operations, 797 described. The other 668 offered an operationId and
nothing else — a generated SDK method with no docstring, a spec-derived CLI command
with no help text, an MCP tool an agent cannot choose between. Now 1491 of 1491.
The gap was structural, not neglect. Almost every one of them was an UNTYPED route:
a proxy to a vendored module, an SSE stream, a WebSocket upgrade, a byte upload, an
All() wildcard, or a surface owned by another repo entirely. None has a handler doc
comment in this tree for zipdoc to lift, which is exactly why 47 apps carried no
zipdoc directive — adding one would have produced an empty file. The seam they
needed existed and had one caller; it now has 523.
Three surfaces had no seam at all and would have been left behind:
- metrics and licensing are vendored modules that deliberately do not import
cloud, so their prose lands at cloud's OWN wire fact in build.go;
- authz is a leaf forbidden from importing cloud, and its handlers are untyped
closures in another module — both seams shut — so its prose lands in
plugin/authz/main.go, the file whose own doc says it is where "cloud's plugin
contract bends to the leaf."
Every sentence was read off the handler, and reading 668 handlers is most of what
this cost. It found ten defects, filed as #376 — two of them money: gpu-charge is
not idempotent, and the finance ledger's peer path emits a vocabulary its reader
does not classify, so credits render empty and deposits sign negative, with the test
green on both paths because it only exercises the S2S mock. None is fixed here.
Describing is not repairing, and a description that flattered the code would have
been worth less than the silence it replaced — so where a route is broken, the prose
says what it actually does.
Three tests used "has prose" as a proxy for "is a typed op". That equivalence held
while prose could only arrive by lifting a typed op's comment, and Describe breaks
it by design — so each of those tests forbade precisely what the seam exists to do.
They now read zip's own registry and assert something stronger: every operation is
either a typed op with lifted prose or a recorded raw address with declared prose,
and either way it carries prose. apps/exec's is a gate over all 56 of its ops, which
matters most on a pure-proxy surface, where the description IS the product surface.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerce, iam and kms each failed `zipdoc -check` at HEAD, and the cause was the
same in all three: their internal-plane ops are registered as inline closures, so
zipdoc had nothing to lift and the generated file was empty or absent. The gate had
been red long enough to look like the normal state.
Generating an all-empty file would have made the gate green and the document no
better, so each closure is extracted to a named function or a bound method on an ops
receiver, with a doc comment that says what it does — the shape apps/wallets/rpc.go
and apps/git/community.go already use, and the reason they say "a named handler, not
a closure, so zipdoc can lift this prose."
Pure extraction: same bodies, same registrations, same behavior. What changes is
that the ledger's gate, its debit, its credit, its balance and statement reads, the
IAM roster and the KMS secret ops now describe themselves — to the internal plane's
own document, its MCP tool list and its CLI, exactly as the product surface does.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A typed op carries its prose in a doc comment zipdoc lifts. Everything that cannot
BE a typed op — an SSE stream, a raw proxy, a redirect, a byte upload, an All()
wildcard, a route owned by a vendored module — had nowhere to put prose at all.
Describe was already the answer and had exactly ONE caller in the fleet, which is
why 47 apps carried no zipdoc directive: adding one would have generated an empty
file, so the tooling looked complete while 45% of the published surface said
nothing. Two additions make the seam usable at scale.
Methods() publishes the generator's own method set. A subsystem whose whole surface
is an All() registration has no per-method registration site to hang prose on, so it
declares prose in a loop — and that loop has to cover exactly what the document
renders. Reading the projection's own set is what makes it exact in both directions
rather than a hand-copied list that drifts.
DescribeRest fills what nothing has described yet, and exists because six different
addresses proved the point. Each described the methods it cared about — GET reads,
POST acts, PUT is not routed — and each stopped at the same five, so OPTIONS and
TRACE were published bare from all six: the same omission written six times, which
is what a hand-copied list of a thing the generator owns always becomes. Asking for
the leftovers means a method added to the generator is covered the day it appears.
And the document endpoint describes itself. /v1/openapi.json has no owning
subsystem, so nothing would ever have declared it, and it was the last route in the
fleet publishing an operationId and nothing else — offered by every generated SDK as
a method with no docstring. It goes in an init rather than in serve, because serve
runs once per document source and Describe refuses a duplicate: a self-description
that panicked the second time a process mounted would be worse than none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The tag prose was keyed by APP NAME — `said[tagName]` against a map built from
each part's own info.description. A tag is not an app name, it is the first path
segment after /v1/, and the two are only sometimes one word. So every product
whose prefix differs from the app that serves it published no description at all,
while the owning app already carried a reviewed sentence saying exactly what it
is: /v1/kb is knowledge's, /v1/scrape is websearch's, /v1/machines is visor's.
59 of 150 tags were empty for that reason. 50 of them are served by exactly one
app that had already said what they are, and now inherit it.
The other 9 stay empty, deliberately, because every alternative is a guess a
consumer cannot tell from a fact:
finance, plans, s3, vector two apps serve the prefix (billing+treasury,
commerce+plan, provisioning+storage,
product+provisioning). An ambiguous owner must
produce silence — picking one publishes a coin
flip.
authz, licensing, logs, the sole owner has no package doc. The cure is a
metrics, traces doc comment in that app, not a line synthesized
from the product name here.
Nothing is ever derived from a product's own name: an empty description is
honest, an invented one is not. A tag NAMED for an app still keeps that app's own
sentence — /v1/admin is served by seven apps and would go silent under ownership
alone, so all 91 previously-described tags are unchanged.
One lookup, one call site: prose() still answers said[name] and Weave is
untouched. The per-app subsets carry bare tag names (From emits no description),
so this moves openapi.yaml alone — 34 hunks, all inside the tags block.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The release could not pass smoke, ever. The script grepped the boot log for
`"message":"listening"`; the zip transport logs `a.logger.Info("zip listening",
…)`, which renders as `"message":"zip listening"`. The needle excluded the prefix,
so it matched nothing and every release died at "never reached listening" with a
perfectly healthy image.
Proven both directions on the real artifact: ghcr.io/hanzoai/cloud:v1.801.332
booted, logged `{"module":"zip","transport":"http","addr":":8080","message":"zip
listening"}` at ~10s, and failed the old needle even given 300 seconds. With the
needle corrected the same image answers SMOKE PASS.
Two more in the same Job, each of which would have masked the fix:
- it pulled with `kaniko-ghcr`, a Secret the build namespace does not hold, so
every smoke fell back to an anonymous pull. The namespace holds ghcr-pull.
- the boot window was 60 iterations. cloud mounts ~28 subsystems before the
listener opens, so the bound sat close enough to a healthy boot to fire on
one — the same defect the build deadline had. 180.
The existing test asserted the BROKEN needle and the missing secret by name, so
it passed while the gate could not work — the shape that let this survive. It now
holds the needle against a line the binary really emits, and a new test reads the
zip module in the build cache, so renaming that log line fails here rather than
silently at release time.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
These are the remainder of the rule 668c63ac, 55ef5731 and 4cfc10d0 established:
openapi/synopsis.go lifts a package doc's FIRST SENTENCE verbatim into that app's
own subset as info.description, the weave lands it on the product tag, and from
there it is the MCP tool prose an agent reads and the line `hanzo <product>
--help` prints. A sentence that opens "is the ... plane", names a /v1 path, or
points at apps/pubsub answers a question no buyer asked.
Six of the seven are products revived AFTER those passes ran — 9640ea2e mq,
6db519e9 pubsub, a31e0ec9 flow, eafc8a82 engine, 48004de8 registry, 34663c49
auto — so their docs were written to the register the rule replaced. share was
simply missed. (analytics was the eighth and is not here: 5a781b8a landed it
from another lane while this was building, and its opener already holds the
rule, so this takes that one rather than restating it.)
auto "…executed as durable runs on the hanzo tasks plane"
→ Hanzo Auto: build a flow from triggers and actions, publish it,
and watch every run.
engine "…what it serves and what it runs on, read through /v1/engine"
→ Hanzo Engine: which models the serving runtime has loaded, and
the GPUs under it.
flow "…run agent workflows on the unified /v1 plane"
→ Hanzo Flow: build an agent workflow on a visual canvas, run it,
and read every run.
mq "…served at /v1/mq over the broker apps/pubsub embeds"
→ queue and stream admin for your org: create them, watch them
drain, ack what you pulled.
pubsub "…served to tenants at /v1/pubsub over the embedded Hanzo PubSub
(NATS + JetStream) node this same package runs"
→ your message bus: publish, subscribe, and durable streams your
apps read at their own pace.
registry "the management plane over the platform's artifact registries…on
the unified /v1 plane"
→ your container and package registry: push images, pull them back,
see what you store.
share "…folded into the ONE cloud binary"
→ a public URL for a service on your own machine, and a list of
what you have open.
NOTHING IS DELETED. Every clause the old openers carried — the tasks plane, the
/v1 mounts, the broker apps/pubsub embeds, the JetStream node this package runs,
the management-plane framing, the ONE binary — is still in the doc, one sentence
lower, where the engineer it was written for still finds it. Only the ORDER
changed: what the product is, then how it is built.
Each opener now clears the whole rule: ≤90 characters after the "Package <name>
is" the projections strip (81–91); no mounts/surface/subsystem/plane/binary; no
/v1 path, file name, app reference or HIP number; no shouting. And each ends
sentence one with a period on its own paragraph, so go/doc's Synopsis publishes
the SENTENCE and not the paragraph — the failure mode o11y hit.
That is the last of them: all 91 described tags now lead with the product.
apps/share/client.go was already gofmt-dirty on main (struct tag alignment in the
zrok overview response) and is now clean, since it is a file this touches.
Regenerated FROM SOURCE, not hand-edited. The artifact diff is seven description
lines — 1058 paths, byte-identical to main; no path, operation, schema or tool
name moves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
engine's pin entry (74121595) states the gap exactly and then works around it:
"principal.OrgFrom cannot express what the gate needs: it answers with an org or
refuses, and what this wants is the one bit beside it, principal.Validated."
True — and the fix is to park that bit, not to hand the op its raw request to
recompute it. cloud.Bridge already resolves the org by calling principal.Org and
parking the answer; it now resolves validated-ness the same way, in the same
expression, so both facts a gate turns on cross the typed seam as VALUES and
neither costs an escape hatch.
WHY TWO SLOTS AND NOT ONE. They are different questions. OrgFrom composes
validated-ness AND an org, so it refuses a signed-in caller whose token names no
home org — a machine token, or one minted before IAM's `orgs` claim, for which
SanitizeIdentity mints X-User-Id and deliberately no X-Org-Id (it logs that, with
the subject and audience). A plane with per-org rows MUST refuse that caller: a
tenant-less request cannot address a row. A plane whose reads are
deployment-global — engine's ONE shared runtime, o11y's infra health, neither
tenant-partitioned — must SERVE it, because that org-less operator is who a
serving-runtime inventory exists for. Reading the weaker gate through OrgFrom
would 403 them; that is why the bit is its own fact and not a fold of the org.
THE HATCH SHRINKS BY TWO, and one of them is not even this week's. engine's
entry is deleted: caller is principal.ValidatedFrom(ctx) and reaches for
nothing. o11y's callerValidated was the SAME three-line adapter over the same
hatch — the second copy of a fact with no reader — and becomes the one call;
o11y keeps its pin because callerIsAdmin (X-User-IsAdmin) and callerProject
(X-Project-Id) genuinely need headers neither reader carries, and its
justification now says so instead of naming validated-ness. So the seam ends
with ONE way to ask each question, which is what stops the next author from
opening the hatch to ask "am I signed in": the pin's header and its failure
message now name ValidatedFrom beside OrgFrom, since an unnamed alternative is
how this call site was born.
Behaviour is identical everywhere, on every path. WithValidated IS Validated,
carried; both slots are set by Bridge's single expression, so they cannot
diverge; an unvalidated request parks nothing and every gate answers exactly as
before. Off the HTTP path a CLI LocalInvoke parks neither, so ValidatedFrom is
false and OrgFrom finds nothing — the same fail-closed refusal both adapters
already returned.
Tests: apps/principal TestParked_TwoFactsNeverDisagree pins the ONE state where
the two readers differ (the org-less validated caller) and that the forge is
refused by BOTH, TestParked_NothingOffTheHTTPPath pins the CLI direction;
apps/engine TestOrgLessButValidatedIsServed is that caller reaching all four ops
and TestNoPrincipalIs403AndNoUpstreamByte still proves a forged X-Org-Id spends
no upstream byte, so the gate cannot be narrowed or widened silently. apps/engine
10/10 + 1 live-gated skip, apps/principal 15/15, apps/o11y ok, the gate green,
full suite 182 ok / 0 fail, surface-check clean (openapi.yaml unchanged, 1058
paths).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
48004de8 ("registry: revive the product") added a cloud.Request call site, and
74121595 answered the red gate by PINNING it. The pin is the wrong half of the
fix, for the third time this week: mq (233cdca4) and flow (9fddf792) were the
same call with the same verdict, and registry is flow's shape exactly.
caller took the escape hatch to reach three facts that are one fact.
principal.Org COMPOSES the validated-principal check — OrgOf returns false on an
empty X-User-Id, which is exactly what principal.Validated tests — so the
Validated branch could never be the branch that refused: any request it would
have caught, Org refuses one line later. And Org is the value cloud.Bridge
already parked, by calling principal.Org itself, one line before the routes it
serves (g.Use(cloud.Bridge()) at the top of routes). It was the same answer by
the longer way, through a hatch it did not need. Nothing on this plane turns on
admin-ness, a project or a forwarded credential — the reasons the pin list
documents — so there was no justification to write, which is the signal there
was no entry to add.
The pin's stated reason was the one thing OrgFrom folds away: two DIFFERENT 403
strings, "sign in to use Registry" for an anonymous caller and "no validated
org" for a signed-in one whose token names no home org. That is not a reason to
hold the request. Only the STATUS is contract — all three refusals were 403 —
and the state the second string named is already diagnosed where it is KNOWN:
the identity boundary logs it with the subject AND the audience that minted the
token (SanitizeIdentity, "token names no home org"), which names the offending
client, not just the symptom. Re-deriving a weaker copy of that at every leaf
costs the plane its typing and tells the operator less.
Behaviour is unchanged where it is contract. TestNoPrincipalIs403AndNoUpstreamByte
still proves a forged X-Org-Id with no credential is refused BEFORE an upstream
byte — it parks no org, so OrgFrom finds none — and the new
TestValidatedWithNoOrgIsRefused pins the other half: a signed-in caller with no
tenant gets the same 403 and the same zero upstream bytes, because every
repository name here is `<org>/<image>` and a tenant-less request cannot address
one. Fails closed off the HTTP path exactly as before: a CLI LocalInvoke parks
nothing and every op refuses rather than scoping by an org it cannot attest.
So the hatch SHRINKS instead of growing: registry's entry is deleted rather than
kept, and this plane resolves tenancy the ONE way the typed-op rule (LLM.md
rule 4) already told mq and flow to.
Tests: apps/registry 11/11 + 2 live-gated skips (TestImagesAreOrgScoped,
TestTagsAreOrgScoped, TestPackagesAreOrgScoped, TestTokenIsPullScopedAndOrgPinned
included), the gate green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
reach() read a 404 from the start door as "this fleet does not run that app", which
is the one answer a caller is allowed to treat as free. THREE different things
answer 404 on that wire and it could not tell them apart: the router's own "no such
app", zip's "unknown op" when the router predates the op, and any framework 404 for
the path. All three rebuild into the same *HTTPError with an empty Code, so only the
message text differed — and isUnknownApp's own doc rejects matching on text.
The middle one is live on every rolling deploy. A host pod on an older build serves
its plane socket with no host_start on it, answers "unknown op: host_start", and the
rail reads version skew as "nothing is priced here" — every priced tool free,
fleet-wide, for as long as one old pod is still routing. Adding the wake door is what
created that 404 to misread, so this ships with it rather than after it.
So the fact travels in the REPLY. Started grows Known, an unknown app is a 200 saying
Known=false, and every error is an outage — a router that cannot answer this op
cannot claim anything about the fleet. TestWakeAgainstAnOlderRouterIsAnOutage stands
up a previous-generation router (a plane socket, some other op, no host_start); with
the status-matching form restored it fails with "not deployed here".
TWO MORE, from the same pass.
The one hop a CLIENT holds open had no budget. A fiber Ctx.Context() carries no
deadline, so both contexts chargePeer builds were deadline-free and the only limits
left were a transport read timeout stacked on the 90s wake ceiling — two minutes of a
tool request and its goroutine held open. It is 60s now, derived rather than picked:
the rail's own hops are 10s each, so every one of them can still answer and refuse
first, which is what keeps "payee_unavailable" from arriving as an opaque timeout on
the hop that was only waiting.
A settlement kept a string that aliases the request buffer. cloud.Ask detaches every
REPLY; the request side has no such seam, and a handler's decoded input aliases the
server's body buffer that fasthttp recycles. planeSettle writes in.Resource into the
settlement row and the audit record, which holds today only because both happen
before it returns — the first async audit sink makes that timing a bug. Cloned, so it
is a property instead.
And the stranded debit is named rather than left implicit. A credit that fails after
its debit landed is completed by the client's retry, but the authorization carries
validBefore (300s), so a client that gives up can never complete it and the buyer is
permanently down the money with nothing served. The sweep is written down exactly —
every usage row with provider "x402" whose RequestID has no settlements row, older
than the validity window — because it reads two stores in two processes and is its
own piece of work, not a line at the failure site.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The one package the earlier sweep had to leave alone: the event plane was being
reworked at the time (70698bc2), so its doc was not mine to touch. That work has
landed and the window is long clear, so it joins the other forty.
before Package analytics is the product-event plane: it owns the ingest door
every Hanzo client posts to, lands each event in the `hanzo` warehouse,
and serves the per-org read lenses — KPIs, time series, rankings,
captured errors — over what it wrote. (245 chars, published)
after Package analytics is product analytics: send an event, read back who
did what. (78 chars)
245 characters is not a description, it is the whole paragraph: doc.Synopsis ends
a sentence at the first period, and there wasn't one until the end. So the CLI
help column, the product tag and the MCP prose all carried four clauses of
internal vocabulary — "plane", "ingest door", "read lenses" — where a customer
was looking for what the product does.
Every clause survives, one line down, unchanged. Regenerated from source: 1058
paths, 929 MCP tools; the artifact diff is this one description in the two places
it projects.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The public tenant JSON advertised the sandbox Square application permanently:
OrgResolver used a nil loader, so every host resolved to a synthetic org, and a
zero Organization is never Live. Flipping the org changed the record and nothing
else, leaving the card iframe tokenizing against sandbox while the charge path
used the live org — a nonce the production account cannot charge.
v1.49.35 reads the org row behind a 60s cache and a 2s deadline; every failure
path degrades to sandbox, so an outage can never promote a tenant to production.
Five defects an adversarial pass found in the rail this branch just built. Two of
them were mine and one of those made the whole change inert in production.
THE START DOOR WAS NEVER OPEN. `defer func() { _ = serveWake(app)() }()` evaluates
serveWake at defer-RUN time, so the router opened its start door during shutdown,
for an instant, and never while it served. Every test passed because every test
called serveWake directly. So it no longer returns anything: it registers its
teardown on the app's own shutdown hooks, and an API with no handle cannot be
deferred into never happening. It also waits for its socket to ACCEPT before it
logs "listening", watching the listener while it waits — a stale socket fails
Listen instantly, and a poll that only watched the path would spend its whole
budget on a listener that had already given up.
A CAPTURED X-Payment WAS A CROSS-TENANT BEARER TOKEN. The settlement id is
keccak(payer address | nonce) and the proof rides a request header, so a second
org that gets hold of one — a log, a proxy, a shared client — could replay it
verbatim: the id matched, every compared field matched because they describe the
same purchase, and sameTerms returned "idempotent retry", handing back the first
org's receipt and serving the tool for nothing. The payer is part of what a
settlement IS, so it is part of its identity now. A different payer is a replay
and can only ever be one, because the signature commits to the first org's
address. TestSplitFleetProofIsNotABearerToken; without the binding it serves
{"ran":true} to the thief.
A MISSING ROUTER WAS READ AS "NOT DEPLOYED". ErrNoPeer is the one error a caller
may read as free, and an absent host socket produced it unconditionally — so the
router being down looked exactly like a fleet that runs no marketplace, which is
the fail-open this branch exists to close. A process spawned by the router
carries ZIP_ADDR; that is proof a router is expected, and its door being gone is
now an outage. A binary run directly still gets the honest answer.
THE REVERSAL COULD MINT. A credit that failed after its debit landed was
compensated with a refund — but the failure that makes compensation necessary is
usually a TIMEOUT, and a timeout does not mean the credit did not commit. The
refund would pay back a payer who was correctly charged while the seller kept
the money. Both writes are idempotent on the settlement id and no receipt is
issued unless both land, so the retry completes the settlement; an unretried
debit is a number the log names with that id. Minted money is a number nobody
can find.
Two more are named rather than fixed, because both are decisions that belong to
their owners and neither is the rail's: a public listing claims a tool NAME and
names are fleet-wide, so a listing can undercut a capability it does not own
(apps/tools/LLM.md — it is a listing-KEY decision); and finance_credit is the
first op on the plane that CREATES money, on the same socket boundary that
already guards a secret read and a debit (credit_rpc.go — narrowing it is
SO_PEERCRED on the plane, a fleet-wide seam).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
payments_test.go composes the four subsystems in one process and proves the
payment seam end to end. The fleet has never shipped that way, and everything
that broke, broke in the gap between those two sentences.
So this test IS the shipped topology. The test binary re-execs itself once per
app; each child mounts ONE subsystem and serves its plane socket; nothing is
shared but the socket directory. The parent is the TOOLS process — the tool
plane alone, no charger, no price table, no wallet store, no ledger, exactly
what plugin/tools/main.go links. A settlement therefore crosses four unix
sockets between five operating-system processes, and lands in a ledger three
processes away.
What it pins:
- a priced call is challenged with terms a client can act on, then paid, and
both ledger sides tie EXACTLY at $0.0025. A quarter of a cent is zero in
every cents-typed field, so "0.0025 charged and 0.0025 credited" is the
assertion that a sub-cent price survives four hops as a real price.
- a REPLAY of the same authorization is served and moves nothing. This is the
one that found the zero-copy reply bug: the payee address recorded in the
settlement had become the bytes of a later message's amount, so the row
could not match itself and a paid retry read as a replayed nonce.
- with the x402 process KILLED, a priced tool is refused and nothing moves.
- with the MARKETPLACE process killed, a priced tool is refused rather than
given away: an unknown price is not a price of zero. A killed listener
unlinks nothing, so both outage tests run against a socket that is still on
disk and still refuses every connection — the harder half, and the one the
rail must read as an outage rather than as an absence.
- a free tool still dispatches, over the same four hops, with no challenge and
no ledger movement.
It seeds the listing ROW rather than posting it, and says why at the line:
publish asks tools.Default().Exists and install calls tools.Default().Activate,
and in the marketplace binary that registry has no providers and no activation
store — so both answer 422 and 500 respectively. That is the SAME bug class on a
different seam, it needs its own ops, and faking a provider here would have
hidden it. apps/tools/LLM.md names it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A listed tool was not merely unbuyable in the shipped fleet. It was FREE.
The three seams that make a price payable are process-globals — x402.reg,
tools.std's charger, wallets' mounted singleton — and the fleet runs one binary
per app, so all three bind nothing. The earlier reading of that was too kind: the
tools process does refuse a dispatch whose REGISTRY ROW declares a price, but a
marketplace price lives in the LISTING store, so a listing on a tool that
declares none was dispatched for nothing. Remove the first op below and
TestSplitFleetSettlesAPricedTool answers 200 {"ran":true} to an unpaid call for a
$0.0025 tool. Prices were in the shop window and revenue was not.
resource_billing_peer.go documents the identical bug on the identical seam —
"splitting apps into their own binaries turned every priced create free without
changing a line of billing code" — and the answer was to ASK the owning process.
Four ops, each served by the process that owns the answer:
tools → x402 x402_settle settle this tool call
x402 → marketplace market_price what it costs, and who is paid
x402 → wallets wallets_payee resolve the payee wallet
x402 → commerce finance_credit credit the payee
The payer debit reuses finance_record, which already existed.
ONE POLICY, TWO TRANSPORTS. The in-process seam stays the fast path where the
owner happens to be co-resident; the plane answers where it is not; there is no
third way and no second policy. x402.run now takes the payer and the proof as
VALUES rather than a *zip.Ctx, so the middleware, the in-process Settle and the
plane op are three callers of one flow instead of two flows that agree today.
FAIL CLOSED AT EVERY HOP, and the one exception is a decided fact. An unknown
price is never zero, an unresolvable payee is a 503, an unreachable ledger is no
settlement and no receipt. cloud.ErrNoPeer — the router saying it does not run
that app — is the only thing read as "not deployed here", which is what lets a
fleet with no marketplace keep its free tools without letting an OUTAGE give
away a priced one.
BOTH LEDGER SIDES OR NEITHER. Both writes are idempotent on the settlement id
(keccak(payer|nonce)), and a credit that fails after its debit landed is
REVERSED under the same key — a compensating entry, because "both sides or
neither" has to stay true once the first write has already gone through.
Three details that are load-bearing and easy to get wrong:
- the PAYER is resolved once, by principal.Ledger, in the process that holds
the request, and delegated with cloud.As. cloud.For would silently ship the
inbound org: inside a typed handler zip prefers the gateway's assertion and
drops a stated caller, so a masquerading admin would spend the inspected
tenant's ledger. An UNBILLABLE request delegates nothing at all, rather than
letting cloud.As(c, "") turn a raw X-Org-Id into a payer.
- the PAYEE org rides the caller too, stated from the listing ROW, so wallets
scopes the lookup exactly as it does in memory and a buyer cannot redirect a
credit over a socket any more than it could in a function call.
- the resource id "tool:<name>" moves to plane, because it is now a contract
between two binaries. A spelling each end owns half of does not fail — it
quietly agrees the tool is free.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two facts made the internal plane quieter than it looked. Both are about the
transport, so both are fixed once, at the seam every caller already goes
through, rather than in the callers that happen to trip over them.
A LAZY APP IS NEVER WOKEN BY A PLANE CALL. 106 of 112 apps start on a request
reaching their prefix; cloud.Peer dials the app's canonical socket and never
touches the router. So an app addressed only over the plane was never started
and never bound a socket, and the caller dialled a path that does not exist —
correct by design, inert in practice, and it is why every internal call this
fleet has added is one deployment away from answering nothing. zip added the
second door for exactly this (App.Start: "Hanzo's fleet reaches an app over its
own unix socket, which never touches the router"). Nothing called it. The router
now publishes it as host_start on its own socket, and Ask asks before it dials.
The router is what makes the answer DECIDABLE, which matters more than the wake.
zip dials lazily, so a missing socket produced no error until the first Call and
then arrived as a 502 indistinguishable from a peer that answered badly. Every
caller had to pick one meaning for both, and this fleet has shipped that mistake
in both directions — a 403 read as "split deploy", an outage read as "nothing is
configured". Now the router, which owns the manifest, settles it: "no plugin
named x" is ErrNoPeer and a caller may fall back on it; anything else is an
outage and must not be read as absence.
The caller's deadline governs the wake. wakeTimeout is a ceiling, never a floor:
a gate that gave itself ten seconds must not block for ninety inside a call it
thought it had bounded. ServePlane now waits for its own socket to ACCEPT before
returning, which is what makes "the router started it" mean "it is reachable" —
Serve binds the plane before the app's listener and the router waits on that
listener, so the ordering is a guarantee instead of a coincidence.
A REPLY STRING ALIASES THE TRANSPORT BUFFER. ZAP decodes text zero-copy —
unsafe.String over the frame — so every string in a reply is a VIEW of a buffer
the next call on that connection reuses. A reply that outlives its call mutates
under its owner, silently and much later. That is not a theory: settling a
payment recorded a payee address that had already become the bytes of a later
message's amount, "0.0025USD" written through the middle of an 0x… address, so
the row could never match itself and every retry of a paid authorization read as
a replayed nonce. The decoder copies byte slices; strings were the hole. Ask
detaches them, because a rule every caller must remember is a rule some caller
forgets — and this one costs money three hops from the line that forgot it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/engine and apps/registry landed calling cloud.Request without an entry on
the pin, so `go test .` has been red on main since. The pin is not a formality:
it exists so the next reader learns WHY an op holds its raw request instead of
its tenant, and an unexplained call site is exactly what it is built to stop.
Both are legitimate and neither is the same reason.
engine reads deployment-global platform facts — the host's accelerators, the
build's capabilities — so there is no org to scope by and principal.OrgFrom
cannot express what the gate needs: it answers with an org or refuses, and what
this wants is the one bit beside it, principal.Validated.
registry does resolve an org, and still separates two refusals OrgFrom folds
into one. "Not signed in" and "signed in with no org" have different remedies,
and on a plane where the org is a path SEGMENT of every repository name that is
the difference between "log in" and "you are in no org". The org still comes
from principal.Org, which is what makes a foreign repository inexpressible.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The product repo (hanzoai/auto, native Go v2: base storage + tasks durable
execution + embedded canvas) stays the one implementation; cloud mounts it
the way apps/flow mounts its product, over an HTTP seam: eleven typed ops
proxying what the product's server genuinely answers today — flows CRUD +
publish, the compiled-in piece catalog, durable runs, a reachability lens —
each proven against a live auto backend wired to a live tasksd, end to end:
a webhook->set graph completes with the engine's real output and a
webhook->http graph performs a real request whose response lands in the run
record (live_test.go re-proves the loop on demand).
The run loop is real because the product was fixed first: hanzoai/auto@21e3ec65bf
closes the completion seam — before it, every run stayed 'running' forever
(routes.CompleteRun had no caller; the tasks v1 client wire cannot fetch
workflow results). The engine's terminal Complete activity now writes output;
failures are written by the dispatch watcher.
Tenancy is the product's own gateway-header contract doing org duty: every
upstream call carries X-Org-Id minted from the VALIDATED principal — never an
In field — and the product scopes every row by it (one projects row per org),
so foreign ids 404 without leaking existence. The auto Service stays
cluster-private; cloud is its only door. No credential rides the seam, so
there is nothing for KMS to hold on it.
The rest of the 50-path authored intent (deleted unserved in openapi d86248f)
stays refused: connection custody is base64-at-rest until the product's KMS
DEK seam lands, triggers have no create route, identity families are IAM's.
The ledger is a measured gate (typed_wire_test.go intentRefused), not a
comment.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three consecutive builds of cloud's own image ran 15m, 17m and 20m42s against a
20m bound, so the release was a coin flip — and losing cost more than the wait.
The build had already pushed its image by the time the wait gave up, so failing
it discarded a good image AND burned that version number permanently, because
nextVersion folds published image tags in (by design, to prevent phantom tags).
That is exactly what happened to v1.801.330: built, pushed, missed by 42 seconds,
never tagged, number gone.
30m, because a deadline exists to catch a Job that is STUCK and should not sit
close enough to a healthy build to fire on one. The expected direction is down —
the registry layer cache landed separately, so the Go compiles that ran cold
every time should now mostly hit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
spec() took manifest.Names() — the whole fleet, whatever the deployment actually
runs. That quietly dropped a property the fused binary had for free and
openapi.Mount still documents: enablement scopes the spec, so a deployment that
does not mount a subsystem does not advertise it. A white-label started with
--enable would have published routes it 404s, which is the same lie as a route on
the wire that is not in the document, and cheaper to ship.
run() now hands spec() the app set it composed — the allowlist applied, in
manifest order, coresident apps included because middleware on a sibling's router
still serves its routes. Taken before the mount loops, which consume `on` as they
go (delete, so a leftover name is a typo) and split the broker out first, which
would put the fleet's document in an order that is not the fleet's.
Production names no allowlist, so there the two lists are equal and the artifact
comparison stays the whole fleet.
TestTheDocumentIsScopedToWhatTheDeploymentRuns pins the other end: 12 paths for a
two-app deployment, and nothing outside those two products.
(This branch also carried platform's two unpublished release reads. cdeeb1b2
landed the same regeneration on main first, so the rebase kept one copy — the
document was stale for the same reason either way.)
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET https://api.hanzo.ai/v1/openapi.json returned 3.7 KB and EIGHT paths while
the fleet serves 1039. Not a 404, not a 500: 200 OK with a document of
/.well-known/zip/plugin.json, /v1/ai/health, /v1/health, /v1/iam/{wildcard1},
/v1/openapi.json, /v1/{wildcard1}, /zap, /{wildcard1}.
Nothing on the light host claimed the path, so it fell to the only prefix that
covers it — ai's bare "/v1" — and was proxied to the ai child, which answered
with openapi.Mount reading ITS OWN router, whose entire AI surface is one greedy
All("/v1/*"). Every SDK generator, spec-derived CLI and third party reading the
published spec read that instead. No gate could see it, because openapi.yaml was
correct the whole time and openapi.yaml is not what the deployment served: the
four projections were all compared to each other and none to the wire.
The host claims it now (cmd/cloud spec → openapi.MountFleet), and answers with
the WEAVE, not a live router:
- It costs no subsystem. The document is woven from the subsets each plugin
projected when it was BUILT — plugin.Spec, bytes already in the binary, the
same leaf embed and the same reason the MCP catalogues are there. Answering
starts nothing and opens no socket. Making the host mount 113 subsystems to
describe them would have handed back exactly what laziness buys.
- It is not a second source of truth. openapi.Fleet is the composition that
WRITES openapi.yaml — weave_test.go calls the same function over the same
committed files — so the served bytes and the committed artifact are one
document by construction, and surface-check regenerating those files from
source is what makes a drifted spec go red instead of shipping.
Rendered once rather than per request: the route table is fixed after boot, so
re-encoding a megabyte document on a public unauthenticated door is work whose
answer cannot change, and an amplifier anyone can pull. encoding/json, the bytes
the golden is rendered from.
Pinned three ways, none of them a document compared to another document:
cmd/cloud/openapi_test.go the REAL host surface — mount(), spec(),
webui.Mount() in run()'s own order, every plugin
an oracle answering with its own name. The door
answers from the host and never from a plugin;
/v1/chat/completions still reaches ai; and the
served bytes rendered through the golden's own
JSONToYAML are byte-identical to openapi.yaml
(1039 paths). Delete spec(app) and it names "ai".
manifest/openapi_test.go no app row may claim the path byte-identically —
fiber merges identical patterns and the host's
handler would sit behind the proxy, the silent
shadowing mcp_test.go pins for the MCP door. Plus
the misroute itself, recorded at the layer that
causes it.
openapi/fleet_test.go a missing subset is refused by name; the endpoint
5xxs rather than serving a document quietly
smaller than the API, which is the defect itself.
Ships when cloud next deploys — the live endpoint is unchanged until then.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The package doc's opening line is no longer read only by us. openapi/synopsis.go
lifts it verbatim into the app's OpenAPI info.description, the weave lands it on
the product tag, and from there it is the MCP tool prose an agent reads and the
line `hanzo <product> --help` prints. It is the first thing a paying customer
sees about a product they are deciding whether to use.
Forty of them opened by describing the implementation instead: "mounts the Hanzo
Cloud /v1/code/* surface", "folds hanzoai/esign (the Documenso fork) into the
unified hanzoai/cloud binary as an in-process subsystem (HIP-0106, task #100,
epic #96)". A customer reading the tag list learned our file layout, our epic
numbers and our mount order — and not one thing about what they get. Several ran
past 200 characters, so the CLI help column truncated mid-clause.
So sentence one now answers the only question a buyer is asking:
code → search and symbols across your repos, for you and your agents.
esign → a document out for signature, signed and filed with an audit trail.
world → a live news feed filtered to what your project cares about.
NOTHING IS DELETED. Every clause the old opener carried — the mount path, the
fork it wraps, the HIP, the tenancy argument — moves down to sentence two, where
it is read by the person it was written for. Only the ORDER changed: what the
product is, then how it is built. Both were always in the same comment; only one
of them projects.
The rule the forty now hold, and the reason each is a rule: ≤90 characters (the
CLI help column); no "mounts"/"surface"/"subsystem"/"plane"/"binary" (words for
where the code lives, not what it does); no /v1 path, file name or HIP number (an
identifier no customer can look up); no shouting (the tag list is prose, not a
changelog). Sentence two keeps all of it.
Two files were already gofmt-dirty on main and are now clean, since they are
files this touches: apps/content/doctypes.go (comment alignment) and
apps/websearch/websearch.go (list-continuation indent).
Regenerated FROM SOURCE, not hand-edited: make describe → 1039 paths, 906 MCP
tools. The artifact diff is descriptions only — no path, operation, schema or
tool name moves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The registries are running products, not code here: oci.hanzo.ai (hanzoai/
registry — CNCF distribution, S3-backed, IAM token auth) and pkg.hanzo.ai
(hanzoai/pkg verdaccio, with the hanzoai/git forge's ecosystems beside it).
cloud mounts the typed MANAGEMENT surface over both: status, projects, images,
tags, packages, and a pull-token mint — each proven against the LIVE hosts end
to end (the /v2/ challenge parse, a catalog walk, a tags read and a real token
minted through iam.hanzo.ai's realm with the hanzo-registry service credential;
live_test.go re-proves the loop on demand). Control-plane only: the OCI wire
stays on oci.hanzo.ai, never proxied through cloud.
Tenancy is the registries' own namespace conventions doing org duty: images
are the catalog entries under <org>/…, packages are <org> and @<org>/…, both
filtered server-side from the validated principal — no In field can widen the
scope, and a minted token can only ever name repository:<org>/<image>:pull.
The platform credential (REGISTRY_CLIENT_ID/SECRET, KMS-synced) rides only
Basic auth to the realm the registry's own challenge advertises; an upstream
that refuses it surfaces 503, never a caller-auth bug.
The rest of the 13-path authored intent (deleted unserved in openapi d86248f)
stays refused: project CRUD, webhooks, quotas, scans, digest-level artifact
reads, manifest deletion, push grants and forge ecosystem listing are not what
the running backends serve. The ledger is a measured gate (typed_wire_test.go
intentRefused), not a comment.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A package's first sentence is not prose that stays in the file. openapi.Synopsis
reads it at describe time, describe.go stamps it into that app's own subset as
info.description, the weave lifts it onto the product's OpenAPI tag, and the same
string is the CLI group help line and the MCP door prose. So it is the ONE
sentence a paying customer reads before they know anything else about us — and
across this slice it was answering a question they never asked.
"Package functions mounts the Hanzo Cloud /v1/functions surface" tells a buyer
where our routes live. It does not tell them they can publish code and call it
over HTTP. "Package meet is the CONTROL plane for the virtual office" names our
half of a split they cannot see. "Package o11y is the ONE owner of the cloud
binary's observability plane — ... every part of the concept:" published a colon:
the paragraph had no sentence break, so the whole registry-internals paragraph
WAS the description. Thirty-seven packages led with a mount point, a route
prefix, an implementation noun or an internal ordering argument.
Each now opens with what the customer gets, in their words, and every fact that
was in the old opener is kept verbatim one sentence down — the route prefix, the
tenancy boundary, the fail-closed behaviour, the topology. Nothing is deleted;
it is reordered so the first sentence answers "what is this" and the rest answers
"how does it work". The four packages here with no customer (k8s, s3admin,
datastore, controlplane) keep an internal first sentence, minus the ALL-CAPS and
the word "binary" that made them read like release notes.
openapi.yaml and plugin/*/openapi.json are regenerated from source in the same
commit, because a doc change IS a published-surface change and the drift gate is
right to say so.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The product repo (hanzoai/engine, Rust — `hanzo serve`, the OpenAI- and
Anthropic-compatible inference server) stays the one implementation; cloud
mounts its MANAGEMENT plane over an HTTP seam, apps/flow's posture: four
typed ops proxying what the server genuinely answers today — the model table
with load state, one model's state (a GET here over the product's POST-read
seam, since its ids carry slashes), the host/GPU inventory with build
capabilities, and a reachability lens carrying the build revision — each
proven against a live hanzo-server end to end (live_test.go re-proves the
loop on demand against a real serving process; the fake upstream pins the
measured wire, plain-text /health included).
Inference is deliberately NOT here: the fleet's ONE metered inference door is
the OpenAI-compatible /v1 surface (apps/ai + the zen claim); a second
completion door under /v1/engine would split billing, and the ledger pins it.
The deployment is ONE shared runtime with no per-org primitive, so every read
is a platform fact behind the IAM gate (validated principal or 403, before
any upstream byte) and every mutation the server exposes (models/unload,
reload, tune, re_isq, system/doctor) is REFUSED: an org-scoped route onto a
shared runtime hands each tenant every other tenant's availability.
The rest of the 22-path authored intent (deleted unserved in openapi d86248f)
stays refused: clusters/jobs/ray/pipelines/gpus/serve-endpoints described a
GPU cluster manager this product never was — those families live on the
cluster plane (/v1/clusters, /v1/train/jobs, /v1/ml/models) where they are
real. The ledger is a measured gate (typed_wire_test.go intentRefused), not
a comment.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The first sentence of an app's package doc is not internal prose. It projects
verbatim into three places a paying customer reads — the CLI group help line,
the OpenAPI tag description, and the MCP tool prose — so a sentence that opens
"mounts the ... surface", "is the ... plane", or names a /v1 path describes the
implementation to someone who asked what they bought.
Rewrites the opener of 32 app packages across billing/money and
identity/security to state what the customer gets, and reflows the displaced
detail into sentence two. Nothing is deleted: every path, mount note, store
shape and tenancy invariant that was in sentence one is still in the doc, one
sentence lower, where an engineer reading the package still finds it.
billing money door -> your org's balance, what it has spent, the cards it pays with
books "at /v1/books" -> chart of accounts, ledger, bank reconciliation, the reports
o11y "ONE owner of the observability plane" -> your logs, metrics and traces
usage "the usage plane at /v1/usage" -> what your org ran and what it cost
principal "ONE place the data plane turns a request into an org" -> the guarantee
that one org never reads another's data
iam is left alone: "Hanzo's identity provider: users, organizations,
applications, and the OIDC/OAuth2 endpoints every Hanzo service authenticates
against" already leads with the product.
Six of the 32 (finance, payout, metering, money, idv, principal) back no
plugin and so project nowhere; they are rewritten anyway, because the reason
the rule exists does not depend on which reader arrives.
Regenerated openapi.yaml + the 23 plugin subsets from source. The diff is the
description line and nothing else — no route moved.
apps/metering also drops a stale claim to live in the commerce repo; it lives
here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
surface-check is red on origin/main, and not for a code reason: e08670d5
("a 202 that launches nothing is a lie") mounted GET /v1/runner/releases and
GET /v1/runner/releases/:id in platform.go but did not regenerate the
document, so two routes that SERVE were published nowhere.
That is the exact failure the gate names: "routes that exist and are
undocumented ... the SDK repos pull this file, so a route missing here is a
route no generated client can reach." The 202-answerability those handlers
restore is only half the fix — an id you can ask about through a client that
was never generated the method to ask with is still unanswerable.
This is the gate's own prescribed remedy (make describe, then commit
openapi.yaml and plugin/*/{openapi,mcp}.json), and nothing else: regenerated
FROM SOURCE, the diff is +42 lines and 0 deletions, exactly the two routes
and their :id path parameter. mcp.json is untouched because these are raw
cloud.Handle routes rather than typed ops, so they mint no MCP tool.
Found by running the drift gate on a rebase, not by reading the diff — which
is the point of a gate that regenerates rather than compares.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestRequestEscapeHatchIsPinned went red again the moment a31e0ec9 ("flow:
revive the product") landed, for the same reason mq did one commit earlier:
a new cloud.Request call site with no pin entry. Same verdict, same fix.
caller took the escape hatch to reach three facts that are one fact.
principal.Org COMPOSES the validated-principal check — OrgOf returns false
on an empty X-User-Id, which is exactly what principal.Validated tests — so
the Validated branch could never be the branch that refused: any request it
would have caught, Org refuses one line later. The three ErrForbidden
returns were one decision written three times, differing only in the string.
principal.OrgFrom is that decision once, over the org cloud.Bridge already
parked by calling principal.Org itself. Nothing on this plane turns on
admin-ness, a project or a forwarded credential — the reasons the pin list
documents — so there was no justification to write, which is the signal
there was no entry to add.
The refusal that survives is the product-facing one ("sign in to use Flow"),
since all three were 403 and only the status is contract:
TestNoPrincipalIs403AndNoUpstreamByte still proves a forged X-Org-Id with no
credential is refused BEFORE an upstream byte — it parks no org, so OrgFrom
finds none. Fails closed off the HTTP path exactly as before: a CLI
LocalInvoke parks nothing and every op refuses rather than scoping by an org
it cannot attest.
So the hatch stays at 48 pinned call sites — flow and mq now resolve tenancy
the ONE way the typed-op rule (LLM.md rule 4) already told both to.
Tests: apps/flow 9/9 (TestNoPrincipalIs403AndNoUpstreamByte and
TestWorkflowsAreOrgScoped included), the gate green, full suite
179 ok / 0 fail, surface-check clean.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestRequestEscapeHatchIsPinned went red on 9640ea2e ("mq: revive the
product"), which added a cloud.Request call site without a pin entry. The
fix is not the entry.
cloud.Request is the escape hatch that hands a typed op its raw request,
and the pin exists because every use gives back some of what typing bought.
The test's own message names the test for whether an entry is earned: "if
the op needs only its tenant, use principal.OrgFrom(ctx) and delete the
call." mq needs only its tenant. callerOf read the request to compute
principal.Org(c) — which is exactly the value cloud.Bridge already parked,
one line earlier, by calling principal.Org(c) itself. It was the same
answer by the longer way, through a hatch it did not need. Nothing in this
surface turns on admin-ness, a project, a forwarded credential or a
response header — the reasons the list documents — so there was no
justification to write, which is the signal that there was no entry to add.
It is also what LLM.md already tells the next author to do (typed-op rule
4): principal.OrgFrom for the tenant, cloud.Request only for more.
Behaviour is identical on every path. principal.WithOrg has exactly ONE
caller — Bridge, in the same expression that parks the request — so the two
context slots are always set together and can never diverge: a request that
parked no org 403s either way, and off the HTTP path (LocalInvoke) neither
is present and every op still refuses rather than serving an untenanted
one. The two old refusal branches were one decision written twice.
So the escape hatch SHRINKS back instead of growing: 48 pinned call sites,
the same 48 as before mq landed, and the gate goes green because the code
stopped needing the pin rather than because the list learned to excuse it.
Tests: apps/mq 9/9 against the real embedded broker (TestTenancy — the
cross-tenant blindness proof — included), the gate green, full suite
178 ok / 0 fail, surface-check clean.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two failures with one shape: the caller was told something the code could not
know.
A RELEASE ANSWERED 202 AND STARTED NOTHING. `{"repo":"cloud"}` — a bare name
where a clone URL belongs — parses without error but carries no scheme and no
host, so it failed inside the detached pipeline long after the 202 went out with
an image tag. startRelease refuses an unparseable repo before answering, and the
id it returns is now answerable: a release records its outcome (status, the step
it reached, the error) and GET /v1/runner/releases[/:id] serves it, SuperAdmin
only, bounded to the last 20. In memory, which honestly disappears on restart
rather than pretending to be a history the pipeline does not keep. Named
listSelfReleases, because /v1/releases is a tenant's deployments and this is the
platform publishing itself.
"MASTER KEY NOT CONFIGURED" WAS FALSE. Every integrations connect answered 503
naming CLOUD_KMS_MASTER_KEY_REF while that variable was correctly set and the KMS
surface in the SAME process decrypted secrets fine. The real cause is a discarded
type assertion — deps.KMS not being the embedded *kms.Client leaves a nil the
readiness check cannot distinguish from a missing key. The discard now logs what
it actually got, and the error names the symptom instead of a cause the handler
cannot know.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The product repo (hanzoai/flow, Python/FastAPI) stays the one implementation;
cloud mounts it the way apps/iam mounts its product, over an HTTP seam: eight
typed ops proxying what the product's server genuinely answers today —
workflows CRUD, synchronous runs, run records, a reachability lens — each
proven against a live flow v1.8.2 backend end to end (a real graph execution
echoes through /v1/flow/runs; live_test.go re-proves the loop on demand).
Tenancy is the product's own project primitive doing org duty: each org's
workflows live in a flow project named by the validated principal's org,
pinned server-side — no In field can address another org, and foreign ids 404
without leaking existence. The platform credential (FLOW_API_KEY, KMS-synced)
rides only x-api-key upstream; an upstream that refuses it surfaces 503, never
a caller-auth bug.
The rest of the 87-path authored intent (deleted unserved in openapi d86248f)
stays refused: pieces/app-connections/triggers/store-entries are not this
product's primitives, and unproven product families ship no route. The ledger
is a measured gate (typed_wire_test.go intentRefused), not a comment.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The pull op's at-most-once hand-off acks on delivery. An ack the plane refuses
because there is nothing to ack (ack:none, already acked) is nothing; an ack
that FAILS on an explicit consumer means the batch this response already
carried will redeliver, which the operator should be able to see. No wire
change, no projection change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The pubsub PRODUCT was authored (hanzoai/openapi, deleted unserved in d86248f)
but the app mounted no HTTP routes: the embedded NATS+JetStream node served the
cluster and nothing served tenants. Finish the job: one bus, two doors.
- typed.go: 18 typed ops driving the REAL plane through the one bus knob —
publish (durable when captured, Nats-Msg-Id dedup, core fallback),
request/reply, stream CRUD, consumer CRUD, pull fetch (at-most-once hand-off,
acked on delivery), and the KV store (buckets, versioned puts, history,
tombstones). No store of its own: every answer is JetStream's, every refusal
the 4xx it is.
- tenancy is a namespace, not an account: subjects rooted at pub.<org>., stream
and bucket names physically t-<org>-<name> (caller names carry no dash, so
the decode is exactly one (org,name)), org from the validated principal only,
the caller's view always logical — the physical namespace never leaks.
- refused, with pinned reasons (typed.go doc + TestRefusedPubsubOpsStayRefused
pins each intent address to a route-level 404): SSE subscribe (a stream is
not a typed op; the pull op and the NATS port consume), objects (one door per
noun: /v1/storage), and the 7 server monitors (operator plane, cross-tenant;
apps/o11y owns telemetry).
- typed_wire_test.go rides Mount's own embedded node on an ephemeral port:
produce-store-consume loop, request/reply against a live responder on the
NATS port, KV revisions, cross-org 404s and no-principal 403s, the route
oracle (every served op typed and described), and the closed refusal list.
- projections regenerated: plugin/pubsub/{openapi,mcp}.json (18 tools),
openapi.yaml (+10 paths woven).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/account mounted a 112th app whose entire manifest row was two BARE stems,
/v1/billing and /v1/commerce. Behind them sat GET|POST /v1/billing/* and a
five-method /v1/commerce/*, each re-serving a family other apps already own by
re-dialing commerce over an HTTP hop with the admin COMMERCE_SERVICE_TOKEN.
That token satisfies commerce's MayMintMoney = IsServiceToken || IsSuperAdmin.
So every path that reached commerce through this app executed with PLATFORM
authority rather than the caller's: commerce 403s an org admin who calls POST
/v1/billing/deposit directly, and this handed that same person the platform's
own credential and — via the subject-pinning, which aims the call rather than
bounding it — minted into their own account. The only thing in the way was
billingForwardable, a hand-maintained per-method allowlist that had to stay
ahead of every mint route commerce would ever add. Default-refuse is the right
shape for that job. Not having the job is better.
It was already unreachable. Every one of the seventeen forwardable endpoints is
served NATIVELY — six by apps/billing, eleven by the co-resident commerce embed
— at a manifest prefix named DEEPER than the bare stem, and the fiber fork sorts
endpoint routes most-specific-first (zap-proto/fiber router_precedence.go,
ServeMux semantics), so each already won its address. Probed through the real
zip.Load on the parent commit, all fifteen distinct paths answer `billing` or
`commerce`; only the stems themselves and paths nobody serves (/v1/billing/
deposit, /v1/commerce/product) ever reached the forwarder. The published
document says the same: it declared no named path, only /v1/billing/{wildcard1}
and /v1/commerce/{wildcard1}, and removing it deletes exactly those 72 lines
from openapi.yaml with nothing added, moved or re-tagged. No published route
moves; no SDK method changes.
Nothing claims the bare stems now. They are not inherited, because a leaf no row
names is surface no app serves, and a 404 from the /v1 remainder is louder than
a catch-all answering "sign in to view billing" to a pricing page — which is
what these stems did to the entire self-service paid path (plans, invoices,
subscriptions, subscribe/card, topup/token, spend-alerts, payouts,
payment-config, payment-methods) until commerce's and billing's rows named each
leaf. The router oracle is what keeps that honest: a billing leaf added without
a prefix fails manifest/router_test.go instead of being silently swallowed.
The ten merchant store heads under /v1/commerce/* had no native handler — and no
working route either. The forwarder re-dialed COMMERCE_URL, which is configured
nowhere and defaults to the public edge, i.e. this same binary, where /v1/product
matches nothing deeper than ai's /v1. It was a hop to a 402, and it burned an
admin credential to get there. The split-deploy case belongs to
apps/commerce/transport, whose RoundTripper dispatches in-process when commerce
is co-resident and falls back to plain HTTP when it is not — carrying the native
handler's own subject-pinning, with no admin token in any browser path.
What survives is the part that was never the proxy: scopedBillingSearch /
scopedBillingBody (the subject pin PinBillingSubject applies in front of each
co-resident commerce read) and IsServiceToken (the S2S check apps/billing and
apps/commerce gate on). COMMERCE_SERVICE_TOKEN keeps a dozen consumers —
topup.go's HUSD credit, admin, books, payout, referrals, authors, affiliates,
metering, usage, content — so nothing changes in KMS.
apps/account is now typed end to end. Its untypedByDesign list held exactly the
bridge's seven wildcard methods, and all three reasons they could not be typed —
an upstream status and bytes passed through, a body never JSON-validated, an
address bounded by an allowlist instead of a type — are properties of forwarding.
The list is empty and the gate still refuses the next untyped route.
Fleet: 112 apps -> 111.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The authored MQ spec (openapi d86248f^:mq/openapi.yaml) states 41 operations;
nothing served any of them. This serves the 15 the broker genuinely answers —
streams (create/list/get/update/delete/purge), stored-message access
(read by seq / last / walk, delete), durable pull consumers
(create/list/get/delete/next), health and info — every one a TYPED op over a
real NATS/JetStream client dialled through pubsub.URL(), the one bus knob.
The other 26 are REFUSED with pinned reasons (typed_wire_test.go): the
subject side (publish/subscribe/request/subjects) is the pubsub product's
surface — one broker, two ORTHOGONAL products, no op on both; kv and objects
already have cloud doors (/v1/kv, /v1/datastore, /v1/s3); accounts are
broker monitor data no client connection can read.
Tenancy is enforced here, from the validated principal, never an In field:
stream names live under MQ_<org>_ and every subject is confined to
mq.<org>.> (injective org encoding), so tenants cannot see each other or the
platform's own streams (analytics event plane, kafka topics) on the shared
plane. Mount never needs a live broker (retry-connect): describe works
anywhere, ops answer 503 and health says degraded until the plane is
reachable.
Tests run against a REAL embedded broker (github.com/hanzoai/pubsub/embed):
lifecycle, pull-acks-on-delivery, cross-tenant blindness, degraded honesty,
plus the served/refused ledger pinned over the authored 41.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The co-residency caveat was written in the conditional — "in a fleet that runs
marketplace, tools, x402 and wallets as separate binaries" — as though it
described a deployment somebody might choose. It is the only deployment there
is, so the caveat read as a footnote about a hypothetical when it was in fact
the operative fact about production: the seam these two files close binds three
process-globals, and in the shipped fleet each lives in a different process.
The docs now state it as fact with the evidence attached — manifest/apps.go
rows, Coresident set only by zen, the Dockerfile's binary-per-row loop,
cmd/cloud loading each as a child, the deleted monolith — and give the per-
process consequence in a table: the tools process has no charger, marketplace
publishes into its own copy of a registry whose x402 was never mounted, and the
x402 process has neither a price table nor wallets nor a ledger. All three fail
CLOSED, so the safety property holds; what does not hold is that a priced tool
can be bought at all.
And it names the close rather than gesturing at one. The fleet already solved
this exact bug once, for resource billing, with the internal plane — ASK the
process that owns the thing (cloud.Ask / zip.Post on cloud.Plane(); see
apps/commerce/meter_rpc.go). Four ops do it: tools→x402 settle, x402→marketplace
price, x402→wallets resolve-payee, x402→commerce credit-payee. Written down so
the next lane starts from the answer instead of rediscovering the problem.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Enforce is applied to a route group to DECLARE it for sale. It answered two
different "I cannot enforce payment here" conditions — x402 not mounted in this
process, and no Registry published in it — by calling c.Next(). That renders
"I don't know what this costs" as "it costs nothing", permanently and silently,
on exactly the routes somebody marked as revenue.
It is not a hypothetical condition, it is the shipped one. The published
Registry is a process-global installed by marketplace.Mount, and the fleet runs
one process per app: manifest/apps.go declares marketplace, tools, x402 and
wallets as ordinary prefix-routed rows (only zen is Coresident), the Dockerfile
builds a binary per row, cmd/cloud loads each as a child process, and the fused
monolith is deleted. So the table is nil in every process that mounts x402.
This fleet has shipped this bug once before, and said so out loud —
resource_billing_peer.go: "Splitting apps into their own binaries turned every
priced create free without changing a line of billing code."
Both conditions are now a 503 x402_unenforceable carrying no challenge, because
there are no terms to offer: a client cannot pay past a rail that is not there.
A table that IS published and simply prices nothing still passes through — that
is an answer, not a silence, and a priced group may hold free routes.
The tool seam keeps the opposite default and now says why: Settle is offered
EVERY dispatch, so an absent table there means nothing is priced and free tools
must keep working. Two different questions, so two different safe answers,
each stated rather than inherited from one shared default.
No route group applies Enforce today, so no live traffic changes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The end-to-end payment proof mounted its OWN /v1/tools/call — a hand-rolled
route that dispatched, mapped a 402 and let the challenge header out. That
proves the seam and not the product: the door a buyer knocks on is tools' own
callTool, and a payment path is exactly as real as the route that carries it.
newMarket mounts tools.Mount now; the stand-in is gone.
Mount order is the composition root's, and load-bearing: marketplace installs
cloud.Bridge app-wide, fiber runs middleware in REGISTRATION order, so it has to
be registered before the tool plane's leaves or a dispatch reaches no parked
request — no attested payer, and every priced tool 424s instead of settling.
Running through the real door surfaced a confound the stand-in did not have: the
tool plane bills its OWN orchestration unit on every successful call, defaulting
to cloud.DefaultResourceFeeCents ($1.00), and that debit is fire-and-forget on a
background context into the same ledger these tests read. So "the payer was
debited exactly the price" was measuring two charges and racing one of them — it
passed on the priced test only because the balance read won the race, and failed
outright on the free one, where the buyer's whole dollar had already left.
CLOUD_TOOLS_FEE_CENTS=0 prices that unit at zero for the harness, so the only
thing that can move money in a payment test is x402.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
migrateCents updates `price` and drops `price_cents`, but the column it updates
is only ever created by the CREATE TABLE above it — and `CREATE TABLE IF NOT
EXISTS` is a no-op against a table that already exists. So the one store the
migration is FOR is the one store where `price` is absent: the UPDATE failed on
a column that did not exist, Open returned the error, marketplace.Mount returned
it, and the composition root stopped. Not a degraded marketplace — a binary that
will not start, on every deployment that ever published a listing.
The migration adds the column itself now, as its first statement, in the same
transaction as the conversion and the drop.
TestLegacyCentsStoreOpens builds a pre-exact schema by hand and opens it: 750
cents arrives as 7.5, a free row stays 0, the price table still reads the
migrated row, and re-opening a migrated store is a no-op rather than a second
migration.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A GitHub App is installed per account, so an org that owns hanzoai, hanzo-apps
and hanzo-docs holds three installations. The connection was keyed
(org, provider), which holds one, so connecting the second account silently
replaced the first and every token minted for the survivor granted nothing on the
others. Swapping to an App with wider reach made this visible: the stored id
belonged to the old App, and every mint answered 404.
The key is now (org, provider, owner). Existing rows are migrated rather than
reset — account_label already held the GitHub org login, so the owner is
RECOVERED from it and a live connection keeps working. A provider with one
account per org carries owner='', which is not a special case, just one owner.
Reads that meant "this org's account" now say which: the repo list and the import
span every connected account and union the result, so one revoked installation
costs its own repos rather than all of them; Pages searches the accounts and
answers with the token of the one that grants the repo; the mirror and the sync
engine take the owner from the remote they are pushing to or fetching from. The
webhook mints from the installation id in the HMAC-verified body, which is
better than the name lookup it replaces — the id already resolved the org, and
it cannot disagree with itself.
A named owner matching no row still resolves when the org has exactly ONE
connection: a row predating this key, or an account renamed on GitHub, holds the
right installation, and refusing it would break a working mirror over a label.
With several accounts there is no fallback, because the name is then the only
thing telling them apart.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The projections were generated while main carried the de-Stripe sweep; main has
since restored Stripe as a customer connector, so the source comment reads
"com.stripe/mcp" and the committed prose no longer matched it. Regenerated, not
hand-merged — the generator is the only writer of these four files.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
x402.Publish had zero callers and tools.SetCharger had zero outside tests, so
every monetized listing resolved to ErrChargerUnset: a permanent 402 carrying no
terms, which no client could ever satisfy. The catalog advertised prices and the
rail underneath was complete and unreachable.
payments.go closes it from the one store that already holds the price and the
payee. registry is the x402 price table (resource -> Terms); charger is the tool
plane's settlement, which is x402.Settle on that same resource. One table, two
doors onto it, wired at Mount and detached at Shutdown — leaving them installed
past Close would leave a price table answering from a closed database, and a
price lookup that errors fails every dispatch in the process closed.
Tool resources are namespaced `tool:`. No request path begins that way, so
publishing this table can never put a price on a route by accident, and a tool
price can never be bought by hitting a URL.
The payee org is the LISTING ROW's publisher, never anything on the wire, and
wallets resolves an id only within the org it is asked for. Cross-org credit is
therefore unconstructible rather than merely checked: a listing naming another
org's wallet resolves to nothing and the call is refused — the money does not go
to the wrong place, and it does not go anywhere.
Prices are exact. PriceCents could not express $0.0025, which is the shape of a
per-call price on a tool plane; it is money.Amount now, migrated in place (one
cent is 10^16 atto, exact both ways) with the old column dropped in the same
transaction so two columns never both claim to be the price. The cheapest-listing
scan compares in Go, not SQL: $10 is 10^19 atto, past int64, so CAST(price AS
INTEGER) would silently mis-order the expensive half of the shop.
Proven end to end over HTTP with real wallets, a real ledger and a real signature
— challenge, sign, settle, tie — including that a sub-cent charge arrives as
0.0025 and not 0, that free tools are untouched, and that a free tool survives
an absent rail (Settle asks whether a resource is free BEFORE it asks anything of
the world; written the other way it 424s every free tool in the process).
NOT closed, and not faked: all three seams are process-globals, so this binds
within ONE process. In the split fleet a dispatch in the tools process finds no
charger and fails closed. apps/tools/LLM.md states what that needs.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Charge carried a payer, a recipient wallet, a currency and cents — the commerce
graph smuggled into the tool plane, and four chances to disagree with whoever
actually settles. The Charger now takes the TOOL and the context. Who pays is
the attested principal already on the context; what it costs and who is paid
belong to the payment layer's own table.
Deleting the payer deleted a bug: the plane billed p.Owner, the HOME org, which
is exactly the rule principal.BillingOrg repudiates — the SELECTED org pays, or
a member of `acme` acts in `acme` all day while every cent comes out of somewhere
else. There was no reason for a second answer to that question to exist here.
Pricer goes with it. It was a second reader of the same listing table, consulted
to decide whether to consult the settler, on the same row the settler resolves;
one table asked twice is how a gate and a settlement drift apart. Every dispatch
is offered to the Charger now, free ones included — the same single lookup,
one caller. A tool that DECLARES a price with no charger still fails closed.
Price.Amount is money.Amount. AmountCents could not hold $0.0025 — it held 0,
which is free — and a per-token price is the normal case on a tool plane, so
cents were never the right type for this field.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Enforce keyed the flow on the request PATH, which is the only thing a middleware
knows. A tool plane prices a TOOL, and every tool call arrives on the same route
with the name in the body, so nothing that prices a tool could ever reach this
subsystem. Factor the decision (run) from its rendering: Enforce writes a
response, Settle returns an error, and the policy is written once between them.
priceOf is the free/priced question on its own, answerable with nothing mounted
and no request — otherwise a caller that offers EVERY call to the seam cannot
exist, and offering every call is the only way a gate and a settlement cannot
disagree.
Every 402 now carries the requirements, not just the first, so a client whose
authorization expired or replayed can re-sign from the refusal alone.
Sign is the mirror of Verify. The EIP-712 encoding is ONE encoding; the test had
written out a second copy of it, which is free to drift and would fail only in
production, where a real payer's signature stops recovering.
settleLedger skipped either write when its backend was absent. That is the one
thing a settlement may never do: with no meter the buyer was served and never
charged while the seller was credited — money minted out of a missing dependency
— and with no ledger the buyer paid into nothing. Both sides are mandatory now;
a settlement that cannot move both does not happen, and the caller answers 503.
The tests assert BALANCES rather than counting HTTP POSTs. The ledger is the
source of truth about a payment; commerce traffic is an implementation detail of
one deployment, and the old assertion passed while the seller went unpaid.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Restores the payments-catalog Stripe entry (a merchant connecting THEIR
Stripe account, alongside PayPal/Square/Shopify) and the Stripe Identity
option in the IDV adapter. Neither is Hanzo transacting: they are things
a customer connects, and removing them took a capability from merchants
to make a point about our own rail.
Our own rail is unchanged and unchangeable by these: the platform tenant
holds no Stripe credential (env and KMS scope dropped in universe
11c93be8), so credential-driven selection can never route a Hanzo charge
to Stripe. Commerce pin picks up the same correction plus wallet-based
affiliate/partner payouts.
apps/tools kept main's newer prose — another lane had already moved that
example off the vendor name, and theirs says more.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
analytics owns the platform event plane. webhooks consumes it.
Two subsystems each declared a JetStream stream over event.>: analytics'
EVENT (bus.go, envelope key `org`) and webhooks' EVENTS (bridge.go, envelope
key `organization_id`). JetStream refuses the second — "subjects overlap with
an existing stream", err_code=10065 — so whichever lost the race never bound:
either /v1/event ingest 503s on every publish, or the webhooks dispatcher
reconnect-loops and delivers NOTHING, commerce included. And an
analytics-published event resolved org "" through webhooks' orgOf, so it was
delivered to nobody even when the stream did bind.
What died: bridge.go, whole. The EVENTS constant, its subject binding, its
organization_id envelope, subjectFor, publishEvents, and the analytics.AddSink
registration in Mount. A consumer does not publish, and does not name a plane
someone else owns.
Where it went: apps/analytics. It exports the plane's identity (EventStream,
EventSubjects, EventOrgKey), its ONE stream constructor (EnsureEventStream,
which alone knows the retention), and the publish (PublishEvents). forward.go
puts every accepted batch on the plane directly — always, detached, fail-soft
— rather than routing it through a sink another app registers. The subject
grammar (event.<name>) and the subscriber payload are unchanged; only the
tenant key moves, to the `org` every consumer on this plane already reads.
How webhooks consumes: the existing durable-consumer machinery, unchanged.
streamSource now carries the two things a plane's OWNER decides — the tenant
key on its envelope, and its constructor — so consume() calls
analytics.EnsureEventStream rather than a second copy of the config, and
orgOf reads the key the stream declares instead of guessing. commerce.> keeps
organization_id and its own row; nothing about that path moved.
One bus knob: apps/pubsub now exports URL() (CLOUD_PUBSUB_URL, else the
loopback of the server this binary just bound, sharing CLOUD_PUBSUB_PORT with
it), documented in the package doc. analytics, webhooks, kafka and catalogsync
all read it. That retires CLOUD_EVENT_NATS_URL, CLOUD_WEBHOOKS_NATS_URL,
CLOUD_COMMERCE_NATS_URL and CLOUD_KAFKA_PUBSUB_URL — four names for one
loopback NATS, two of which gated their subsystem OFF when unset. No manifest
set either one, so webhook delivery and the reverse storefront edge were
inert in production against a bus that always serves and fails boot closed.
catalogsync now retries instead of degrading to inert, since it no longer has
an operator opting it in.
Tests, red on origin/main and green here:
- TestOneStreamBindsEventSubjects — exactly one stream binds event.>
- TestAnalyticsEventReachesWebhook — analytics publish -> bus -> signed POST,
org resolved from the envelope
- TestCommerceDispatchUnaffected — commerce.> still resolves, matches, queues
- analytics/publish_test.go — subject grammar, subject-space containment,
and one tenant key shared by both envelopes on the plane
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
8c28b24c rewrote the examples in this package's doc comments and stopped there.
Those comments are not comments — zipdoc lifts them into zipdoc_gen.go, which is
the only path they take into the OpenAPI document, the MCP tool descriptions and
every generated SDK. So the source said one thing and the whole published surface
said another, and the drift gate would have found it as a fleet-wide red on
someone else's next push.
Regenerated from source: zipdoc_gen.go, plugin/tools/{openapi,mcp}.json, and the
woven openapi.yaml.
One of the rewritten lines had also gone stale against the code under it: the
handle example still read acme_… after brand started naming the whole domain. It
now reads acme-com_… and says why — acme-com and acme-sh are two publishers, and
that prefix is read on the screen where a credential is pasted.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The document has always known a product's NAME mechanically — the first path
segment after /v1/ — and never what the product IS. A caller reading the tag
list, an agent reading the MCP door, a CLI printing `hanzo <product> --help`
got 144 bare nouns.
There is exactly one place that sentence is already written and already
reviewed: the package doc of the package that implements the app. So this reads
it rather than asking anyone to write it twice.
openapi/synopsis.go Synopsis(plugin/<app>) -> the owning package's synopsis.
describe.go stamps it into that app's own subset as info.description.
openapi/weave.go lifts the tag prose off the subsets it already reads.
ONE computation, at the one moment an app describes itself. The weave does not
look the mapping up a second time in a second process — it reads the value the
app that knows it already wrote down, which is why Weave stays a pure function
of its parts.
The owner comes from the app's own composition root: plugin/<app>/main.go
imports exactly the package it mounts. Nothing else could be the source — four
apps are not named after their package (audit->auditlog, evals->eval,
plugins->plugin, zero-trust->zt) and one package backs two apps (account,
account-bridge), so a name-derived guess is right 107 times and silently wrong
5. An app whose subsystem is another MODULE imports no package here and gets
nothing, which is the honest answer.
And the comment taken is the one that OPENS "Package …", not go/doc's
first-file-in-filename-order fallback. Packages that open their
alphabetically-first file with a note about that FILE and state the real package
doc in <name>.go would otherwise publish "actions.go — the two GitOps write
actions" as the deploy product's description. A misfiled sentence reads exactly
like a real one; an absent one does not.
109 of 112 apps have a package doc; 85 of the 144 product tags gain a
description. The three without are metrics, authz and licensing, whose subsystem
is another module — there is no package here to read. The tag NAME is never
conditional on a description: the list stays a function of the document's
operations, so nothing enumerating products loses a product because nobody wrote
a sentence. The fleet identity remains the fallback for a subset whose package
has no doc, and the weave treats a part carrying it as having said nothing.
THE LIFTED PROSE LOSES THE HANDLER'S OWN NAME, which is the other half of the
same problem. A Go doc comment must open with the identifier it documents, and
that identifier is Go's, not the document's: "GetSQL returns one database"
reached the OpenAPI description, its summary, the MCP tool description an agent
reads, and the CLI help line — naming a function no caller can see. zip drops an
exact leading match of the handler's own name from v1.18.13 (main is on v1.18.14,
whose lift is byte-identical), and nothing had regenerated against it: 35
packages carried prose the pinned zip can no longer produce. They regenerate
here. Three test assertions quoted the leaked identifier and now quote the
projection.
Every generated artifact is regenerated FROM SOURCE (make -f mk/fleet.mk
surface-check, green: 1017 paths). Nothing this commit does moves the wire: of
openapi.yaml's 16,439 non-prose leaf facts, 0 changed. The 4 lost and 94 gained
are all one thing — surface main already decided and never republished:
/v1/insights/e removed and /v1/event given its declared body (6fc2d88c), the six
project-scoped git smart-HTTP paths (811ff080), and the sessions' `terminal`
property (afdda829). The three bare-root git paths reach no app, so they join
router_test.go's unreachable ledger, recorded on the first regeneration that
published them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
manifest.Apps names "/v1/billing/payment-methods" on the BILLING row and
withholds it from commerce's. billing serves both methods there: its GET is the
same-origin proxy to commerce's /v1/billing/portal/payment-methods, and the host
claims a prefix for every method at once, so the POST has to sit on the same
router as the read or it misses on METHOD — which is exactly what killed the
console's save-card call and, with it, auto-recharge (a31a282b).
So commerce's registration was unreachable in the fleet: the request reaches the
billing process and never this one. router_test.go's ledger already recorded it
("commerce /v1/billing/payment-methods -> billing", the last entry of its
ADDRESS-ANOTHER-APP-OWNS class) and says the list may only shrink. It shrinks.
It was also a SECOND claim on one address, which openapi.Weave refuses rather
than pick a winner between — so the fleet document could not be woven AT ALL
once billing's subset was regenerated with its POST. That is why the drift gate
has been red: not a stale artifact, an unroutable fleet.
Nothing moves in the published document. The operation is identical from either
app — same operationId, same tag, no declared body — so only the ownership
changes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`selectImports` was inserted between GithubImport's doc comment and the method
it documents, and the two comment blocks are contiguous `//` lines — so Go read
them as ONE comment group belonging to `selectImports`, and `o.githubImport` was
left with none.
The consequence is not stylistic. zipdoc lifts a typed op's prose from its
handler's doc comment, so POST /v1/integrations/github/repos/import regenerates
with NO description, NO request example and NO 202 response example: the MCP
tool an agent reads to decide whether to call it serves the empty string. The
committed zipdoc_gen.go still carried the old prose, so nothing said this out
loud — it was a fossil the current source can no longer produce, and the first
regeneration after it publishes a documented endpoint as an undocumented one.
The comment moves back onto the function it describes. No behaviour, no wire.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`go build ./...` and `go vet ./...` have both failed on main since 01b38ef6 ("wip:
preserve in-flight work"), which added apps/deploy/chart.go — a Helm chart
renderer importing helm.sh/helm/v3 — without adding the module. Twelve commits
have landed on top of a tree that does not compile, so the CI gate (hanzo.yml
go-vet, go-unit) has been red for every one of them, and the drift gate cannot
run at all: it builds one binary per app, and deploy is an app.
The repair is the repo's own `make tidy`, and the version is MVS's, not a
choice: helm.sh/helm/v3 v3.21.3. It raises the k8s libraries 0.36.1 -> 0.36.2
and a handful of otel exporters with them, because that is what helm's own
requirements make the minimum. Nothing is removed, and no import moves — the
source is unchanged; only the record of what it already needs.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The per-server window I added to stop a dispatch fanning out over the network kept
every entry forever. Each one holds a whole tool set — a fifty-tool server with
real schemas is tens of kilobytes — so a fleet that deregisters and re-enables
servers accumulates them with nothing ever dropping one.
An hour without being asked about is a server whose org deregistered it, renamed
it, or is not coming back. Those go. The pass runs off the listing path and at most
once an hour, rather than from a goroutine: a provider with no traffic has nothing
to forget, and a timer outliving the process's interest in the data is one more
thing to shut down correctly.
The key carries the org, and now a test says so: two tenants holding the same
server id at the same URL each ask for themselves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip refuses a second open plugin at Load, and a host discovers that by failing to
boot — after the manifest already said it. The manifest is where the decision
lives, so it is where the check belongs: exactly one app is open, it is the tool
plane, and it is not co-resident (a co-resident app is never Load'ed, so it could
never be asked and the flag would be a lie).
The second test walks the resolution ladder. Open is a property of the APP and not
of where its binary came from, and it is stamped once in Plugin() for exactly that
reason — so a remotely-mounted tools app must keep it too.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Resolve reads and Write writes, and a request can arrive between them — which is
not exotic, it is a double-clicked Enable button. Both passes then found no
existing row, both resolved the same id, and the second INSERT hit the unique
constraint and answered 500. Nothing was corrupted and nothing was lost; the
caller was simply told their button was broken.
The row the first one created IS the row the second was about to create, so the
second becomes the revise it would have been had it arrived a moment later. Both
constraints route here — the (org, id) key and the (org, listing) index — so the
convergence does not depend on which one fires.
Eight concurrent enables, one server.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An adversarial pass over the catalog found three things worth the word BLOCKING and
a row of smaller ones. Every fix below has a test that fails without it.
A PROCESS KILLER, in zip and mine: the per-caller dedup wrote the caller's names
into the FLEET's shared name set. One tenant's tool name became "already claimed"
for every tenant after it — the second org silently lost a capability and could
infer from the absence that someone else had it — and two lists in flight were two
writes to one map, a runtime fatal no recover() catches, on the method an MCP
client calls constantly. Fixed in zip v1.18.15 (the fleet set is read-only, the
request's dedup is its own); pinned here.
A LOOKALIKE, and it aimed at the one screen where an org pastes a credential.
`brand` took the memorable label out of a namespace, so com.stripe and sh.stripe
both became "stripe" — and anyone can hold sh.stripe by owning stripe.sh. Their
tools would have rendered as stripe_create_payment_link on the enable form. The
handle now names the DOMAIN: stripe-com, stripe-sh, alice-github-io. Longer, and
true. `isOfficial` had the matching flaw: it accepted a match on the site or the
repository, so a listing could carry the badge while its ENDPOINT — the only field
enablement consumes — pointed elsewhere. It now judges the endpoint, and falls
back to site/repo only for a listing that has no endpoint and therefore cannot be
enabled. Live: official 4,839 → 4,747; the 92 that left are exactly that class.
AN ORDERING THAT NEEDED AN UNDO, which is the smell. Registration wrote the row
then sealed the credential, so a KMS failure had to be reversed — and the reversal
was wrong for a re-enable in one direction and absent in the other, leaving a row
asserting a credential nobody stored. That does not fail loudly: the listing
errors, the provider skips the server, and its tools vanish from the org's plane
in silence. Resolve → seal → write has no undo in it at all, because nothing is
written until the credential is held.
The rest:
- an unbounded GET /v1/tools/catalog over 19,323 rows is now paged (50, max 200)
with the whole match as `total`.
- upstream logo/repo/site URLs are dropped unless a browser may follow them.
`javascript:` in an icon src is what arrives, and all three are RENDERED.
- a name outside the registry's own grammar is skipped. "com.foo_bar/mcp" and
"com.foo/bar_mcp" derive ONE id, which is the primary key, and put preserves
curation — so the second would inherit the first's featured, vouched standing
and replace its endpoint.
- deregistering destroys the credential. types.KMSClient has no removal at all,
so this overwrites; the real fix is a fleet-wide seam change and is written up
in LLM.md rather than smuggled in here.
- no redirects on an MCP hop: Go strips only Authorization and Cookie, so an org
using X-Api-Key had it replayed to wherever the server pointed — and the
endpoint may now come from a third party's catalog entry.
- authHeader must be an HTTP field name and the secret has a size. Both failed
later, inside a background listing, where the symptom is a server whose tools
quietly never appear.
- one tools/list per server per minute, not per dispatch. Every dispatch resolves
through the registry, which listed every provider, which asked every one of the
org's servers over the network: thirty enabled listings meant thirty outbound
requests per tool call, each with a 20s timeout.
- the sync commits a page at a time instead of a row at a time — 19k fsyncs
inside a request someone is waiting on.
- a search term's % and _ are literal characters; a third party's cursor has a
length; Dispatch guards its store like List already did; and the door LOGS when
it cannot resolve a caller, which is otherwise indistinguishable from a tenant
with no tools.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Picks up the deletion: no Stripe provider, no processor.Stripe type, no
per-org Stripe registration or KMS hydration, no Stripe branch in
checkout. Square via commerce is the one rail, and the cloud pod no
longer mounts STRIPE_* (universe 11c93be8).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A connector implies we support it. Removed the customer-key Stripe entry
from the payments catalog (PayPal/Square/Shopify remain), with a note
against re-adding it, and its catalog-wiring test. The identity-
verification adapter drops Stripe Identity as an option — Persona and
Onfido are the vendors — and the doc examples that reached for Stripe as
the canonical third party (MCP publisher names, the webhook signature
header shape) now say what they mean without borrowing the brand.
The sk_/pk_ patterns in the security scanners stay: those are about
OTHER people's leaked credentials, which is a defensive concern that has
nothing to do with which rail we charge on.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every reference in the run-in-cloud section had already drifted by one from a doc
commit landing above it. A file:line that is wrong is worse than no reference —
it sends the next reader to the wrong function and quietly teaches them the note
is stale. The symbol is what does not move, so it is cited beside the number and
the numbers are pinned to a commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The pins named o11y_ai.observations. There is no o11y_ai DATABASE — checked
against system.tables — so the pin was pinning a fiction, and every AI number on
the fleet board read zero while 8,867 observations sat in `console`. A pin is
only worth having if it names a table that exists.
`console` is a surface name on a store and is wrong too; it moves to o11y.spans
with #102 and these pins move with it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Registering a server is two writes — the row, then the credential into KMS — so a
failed seal has to be undone, and the undo was written when there was only one
kind of write. Making a re-enable REVISE the existing row gave that undo a second
meaning nobody re-read it against: an org re-enabling a working server with a new
credential, against a KMS having a bad second, lost the server. The row was
deleted because the code that deleted it had only ever seen rows it created.
Create now reports whether the row is FRESH, and the rollback deletes only that.
For a revise, the row the org already had stands, still pointing at the credential
it was already using — which is the state it was in before the request, which is
what "undo" means.
The fresh case is unchanged and still deletes: a row claiming a credential KMS
does not hold is a server that would dispatch unauthenticated.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The first live sync read 19,321 servers. The page cap was 200 pages of 100 —
20,000 — chosen as "comfortably past the whole registry", and it was, by 3%. One
good quarter from now it would have started SILENTLY TRUNCATING the catalog: the
walk would stop, Sync would return nil, and the shelf would be short with nothing
anywhere saying so. That is the failure the cap was written to prevent, arrived at
quietly.
The cap was never the loop guard anyway. A cursor that does not end is a cursor
that REPEATS, and that is now caught directly and immediately — one page, not five
thousand — with what was already read kept, because a partial catalog is not a
reason to discard the part that was fine. The count becomes a backstop two orders
of magnitude out, and hitting it is an ERROR: a short catalog that says so beats a
short catalog that does not.
Evidence, against the real registry rather than a fixture: 19,321 listings in 33s
over 194 requests, 4,839 official, 9,216 with a streamable-http endpoint an org
could enable today; a second pass reports added=0 updated=0. That run is
TestLiveRegistry, skipped unless CLOUD_TOOLS_LIVE=1 — the shape being parsed is a
third party's, so it has to be checked against theirs, but a suite that needs the
internet is a suite that goes red when they have a bad afternoon.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
go/doc concatenates every comment glued to a package clause, in file order, and
that concatenation IS the OpenAPI tag description, the MCP door prose and the
CLI group help. 18 apps glued 52 surplus file notes to their package clause; in
11 of them a file note sorted first and WON, so the published product
description was going to read "GET /v1/usage/activity — the per-day
contribution series" (leaderboard), "browse.go — Hanzo Git's JSON read/browse
surface" (git), "O11Y LLM-OBSERVABILITY EVENT INGEST" (o11y), "grant.go — how
CI gets bytes into a site's S3 prefix" (projects), "client.go is the ONE HTTP
path" (graph, zt), "The node registry" (bot), "Pure core of the SBOM lens"
(sbom), "connectors.go is the per-USER connector plane" (integrations), "World
plan enforcement contract." (world) and "activeplan.go answers the subscription
paywall's ONE question" (commerce).
Every surplus note keeps its text and gets a blank line before the package
clause, which is all Go needs to stop reading it as package documentation.
apps/commerce had no package doc at all — three file comments concatenated —
so it gets the one true sentence it was missing.
All 131 app packages now render a synopsis that names the product.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An org could reach an outside MCP server only by knowing its URL and typing it
in. Meanwhile the public registry publishes thousands of them, and the fleet's
one agent door could not show a tenant a single one of its own tools — the door's
list is a build artifact, and a tenant's capabilities are rows.
THE SHELF. apps/tools now holds a canonical copy of registry.modelcontextprotocol.io:
name, vendor, description, repo, version, transports, packages, remotes. Canonical
and not a cache, because a storefront that goes blank when a third party does is
not a storefront, and because the copy carries what only we know. A row has two
halves that never mix — the upstream fields, replaced wholesale on every sync, and
hidden/featured/official/logo, which the sync statement does not NAME and therefore
cannot touch. Idempotent by construction: the id IS the publisher's reverse-DNS
name, so a second pass reports added=0 updated=0 over an unchanged registry.
OFFICIAL IS MECHANICAL, not editorial. A registry attests the PUBLISHER, not the
product: "com.stripe" is issued against proof of stripe.com, but "io.github.alice"
attests an account on a forge, and a re-hoster publishes hundreds of other
people's servers under its own perfectly-verified namespace. So: the namespace
must name a domain, and that domain must serve the listing's own endpoint, site or
repository. stripe.com serving mcp.stripe.com is official; a proxy on someone's
workers.dev is not. What the data cannot settle is what the admin override is for,
and setting it makes the answer final.
ENABLING IS REGISTERING. Picking a listing off the shelf writes the SAME record
typing a URL writes — one thing an org can have, one place it lives, one place the
credential is in KMS — with `source` derived from whether a listing is recorded. A
second enablement path would have been a second kind of server for everything
downstream to learn about. Enabling twice revises one row. The server id, which
PREFIXES every tool the server contributes, is minted from the vendor's brand, so
an agent reads stripe_charge and not m4f21c8_charge.
THE DOOR GETS ITS OTHER HALF. zip v1.18.14 adds the seam and cloud fills it:
tools.Door() is a zip.Source whose Tools is the registry's activated dispatchable
set for the caller and whose Call IS callTool — so activation, precedence, the
x402 gate, the metered unit and the audit record are the ones POST /v1/tools/call
already enforces, and the door adds no policy. The tools app is declared Open in
the manifest; the host asks it only on a tools/list that NAMES a caller, so an
anonymous list is still a memcpy that starts no child.
NOT BUILT, and said so rather than stubbed: running a stdio package (npx/uvx/docker)
in our cloud. The App CR carries no command/args (apps/platform/k8s.go:523), its
CRD is another repo, there is no bridge image, our client speaks one-shot JSON-RPC
and not streamable HTTP, and the SSRF guard exists to refuse exactly the in-cluster
address such a sandbox would have. apps/tools/LLM.md records each seam with its
file:line and the shape the route must take when they land — which is to produce
the same registration record an enable does, or the plane has grown a second kind
of server.
Schema names are a FLEET-WIDE claim: Listing is marketplace's and catalogList is
prompts', so these are MCPListing/MCPPackage/MCPRemote, the shape MCPServer
already set here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
pricing and usage published the WRONG doc entirely. A file comment glued to the
package clause IS package doc, and go/doc concatenates them in filename order —
so pricing led with admin.go's "Admin surface for the catalog enablement overlay
(SuperAdmin only)" and usage with entitlement.go's "Analytics entitlement
contract". Those become the OpenAPI tag description, the MCP door prose and the
CLI group help. Seven such comments (pricing admin/catalog/enablement, usage
entitlement/query, billing finance, wallets wallets) move below the clause, which
is the convention books/store.go and x402/store.go already follow, and each
package now leads with its own sentence.
pricing also claimed /v1/models, /v1/gpu and /v1/tools aliases. It binds none of
them; the manifest and its published subset agree. It omitted the enablement
registry it does own.
billing named five routes. It serves seven under /v1/billing plus the six
/v1/finance projections in finance.go, and its PASSTHROUGH paragraph said it
proxies commerce verbatim — balance and usage read the co-resident finance ledger
first and only fall back to that proxy.
treasury claimed the /v1/finance/* surface. It owns two of that prefix's eight
routes; billing owns the other six. Its ledger sat "above the per-org commerce
credit ledger" — commerce's credit route is injected with apps/finance, which
posts to the same apps/treasury/ledger engine. bind-anchor was missing from the
table.
finance called itself the ZAP-native money subsystem. It registers no ops and no
routes; it is the in-process FinanceClient.
wallets said three custody backends over four Kinds — safeclient.go is the fourth.
books said commerce transactions are the SOLE posting source. bank.go and scan.go
post through the same choke point.
entitlements described only its enablement store; GET /v1/entitlements is the
plan-entitlement projection and the package holds both authorities.
plan and rollingcap named clients/* packages that are apps/*, and rollingcap
named an apps.Wire() the host no longer has.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
captable led with a migration ("the PILOT of epic #96"), team with a Phase-1
READ PLANE and a hand-copied route table that had drifted past every write it
now serves, and product with a hostname (api.cloud.hanzo.ai) its own next
paragraph contradicts. Each now opens with the product: the cap table's
instruments, the workspace's planes, the two backends the four reads inventory.
team's route list is deleted rather than corrected — plugin/team/openapi.json is
the published surface, and a hand copy beside it is the second source that rots.
plugin claimed the second "what is a plugin here" reader was gone. It is not:
GET /v1/plugins lives in apps/tools and answers from cloud.Subsystems(), the
BOOT snapshot — so it cannot see the enable/disable/reload this package
performs. Say that, and say which one is true right now. Its manifest.Apps is
hand-authored, not generated; manifest/apps.go says so in its first line.
erp declares the ERPNext model and no binary imports it, so its init never runs
and POST /v1/framework/modules/erp/install answers "unknown module: erp".
Plus the clients/<pkg> paths in these docs, which name a directory the repo no
longer has: clients/{index,tools,content,settings,goja,plan,pricing,framework,
cms,o11y} -> apps/*.
Comments only. Verified: gofmt clean against the same baseline, the ten
packages build with -tags sqlite_fts5, and go generate -run zipdoc reproduces
every zipdoc_gen.go byte for byte.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
go/doc concatenates EVERY comment attached to a `package` clause, in file
order — so a file header on an alphabetically-earlier file becomes the
package doc. `go doc ./apps/visor` opened "board.go — GET /v1/fleet"; platform
opened with applylive.go, deploy with actions.go. Detach the 43 file headers
with a blank line so the one product paragraph is the whole package doc — the
text that becomes the OpenAPI tag, the MCP door prose and the CLI group help.
Four docs named the wrong thing or claimed something untrue:
deploy said "Package gitops" — it is Hanzo CD at /v1/deploy
goja said "Package gojahost"
do said DigitalOcean is Hanzo's EXCLUSIVE cloud venue; venue links
DigitalOcean, AWS, GCP and Azure. do is the HOUSE account
membership cited cloud.SetLiveSource and apps/ installing it; both are gone
(22f4fc64) and nothing calls K8s — say so, the outage it fixes is
not fixed
And platform pointed at a Goa design module that emits 15 ops against the 30
the router registers — a second source that has already drifted; say which one
is the contract. clients/<app> paths in these docs are now apps/<app>.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
webhooks carried TWO package comments — bridge.go's abutted its package
clause, so go/doc led with "bridge.go — the ingest→bus half…" and the real
product doc never surfaced. A blank line makes it a file comment again.
notify and pubsub opened on the migration ("folds…", "embeds…as an
in-process subsystem") rather than the product; kafka named a module that
does not exist (github.com/hanzoai/stream — it is github.com/hanzoai/kafka).
The package doc is the OpenAPI tag, the MCP door prose and the CLI group
help, so each now states what the product does.
Every clients/<app> path in these six packages is stale — the subsystems
moved to apps/ in f873d1a1 — and apps.Wire() no longer exists, so the three
mount-order notes point at manifest/apps.go, which is the source of truth.
gofmt fixes the pre-existing drift in webhooks/api.go and kafka/interop_test.go.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The package doc is the OpenAPI tag, the MCP door prose and the CLI group help,
so it has to say what the product does — not how it is mounted. Thirteen docs
across the growth domain led with "mounts the Hanzo Cloud /v1/X/* surface: a
native-Go, per-org … on Base/SQLite" plus fold history; each now opens with the
capability.
Four said something the code does not:
- campaign claimed it fans out to paid -> /v1/ads, organic -> /v1/publish and
email -> /v1/marketing. /v1/publish exists nowhere in the repo, and only the
paid executor is registered (plugin/campaign/seams.go). The doc now names
apps/social as the organic surface and states that organic and email have no
executor, so a campaign carrying them records "unavailable". channel.go's
"three registrations" and its copy of /v1/publish go with it.
- analytics called itself a "read API" and its surface "read-only". It owns
the ingest doors every Hanzo client posts to and the write core behind them;
the doc now states both halves and lists the four doors. It also still
counted "six ingest doors" — a69e7549 retired three name-aliases and left
two, and routes()'s "four of the six" was stale the same way.
- ads described a campaign store; provider.go launches, pauses and reads spend
on six ad networks, and the launch route was missing from the surface block.
- destinations, flags, leaderboard, marketing, social, crm, affiliates,
referrals and authors led with storage or fold history rather than what they
do.
Where a capability has two implementations the doc now says so at both ends
rather than in neither: marketing's calendar and apps/social are two stores for
one scheduled social post, marketing's campaign record is a third campaign
beside campaign and ads, /v1/admin/referrals/* is split across referrals and
affiliates, and /v1/usage/* is co-owned by leaderboard and usage.
clients/<app> paths inside these doc blocks are stale — the packages are
apps/<app>; fixed where the doc block carries them. gofmt fixes the pre-existing
import order in affiliates.go.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Eleven packages in the agents cluster. Four docs said something false: benchmark
and research cite an apps.go composition root deleted with the mega build,
agentskills cites a go:generate path that moved, and eval/exec/bots/runtime/
connectorruntime/coding/automations cite clients/<app> packages that are now
apps/<app>.
Two packages carried TWO package docs — automations (automations.go + types.go)
and runtime (runtime.go + ops.go). Go concatenates them in file order, so the
product description an OpenAPI tag reads was 'types.go ports the ActivePieces
shared contract' and 'ops.go mounts /v1/bot/*'. The second comment in each is now
a file comment, separated from the package clause.
exec's doc did not say it owns four top-level /v1 segments (/v1/exec, /v1/upload,
/v1/download, /v1/files) and therefore publishes under four product tags; eval's
led with the console fork it replaced instead of with the product; coding's led
with 'the keystone that turns @hanzo from a chatbot into an engineer' and did not
say it is a library with no route, no plugin and no manifest row.
eval was missing GET /v1/evals/metrics, prompts GET /v1/prompts/catalog, and
research its three /v1/research/artifacts ops.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The package doc is the OpenAPI tag, the MCP door prose and the CLI group
help, so a package with FOUR of them publishes whichever one sorts first,
and a package whose doc describes a route it does not serve publishes a
lie.
knowledge had four package docs attached (connectors.go, kb.go,
subsystem.go, sync.go); go/doc takes the first file alphabetically, so the
product was described as "connectors.go is the per-org app-connector
control plane" — one file's notes, standing in for the knowledge base.
kb.go is now the one package doc and says what the product is; it also
opened "Package kb" for a package named knowledge. The other three keep
their prose as file comments.
websearch had two (search.go, websearch.go) and the same rule picked
search.go's "Native Go meta-search — the SEARCH half of /v1/websearch",
which describes one half of one file. websearch.go's product doc now
stands alone.
search claimed to be "THE search entry point: ONE surface, POST /v1/search".
It is not mounted: no manifest row, no plugin/search binary, and /v1/search
is apps/provisioning's (list/create a provisioned search index). Its only
live caller is apps/team's fulltext RPC via ForOrg — and in the team binary
index.Mount never runs, so the lexical leg reports disabled on every query.
The doc now states that.
Comments only; no route, type or behaviour moves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
world named a mount order (142/150) that no longer exists — the host routes by
manifest.Apps sequence — and its surface block omitted GET /v1/world/limits, which
the code binds and the published subset documents.
content promised phase 2 would widen the cms module's status field onto the shared
lifecycle. No binary imports apps/cms, so its init never runs, the module is never
registered, and no org can install it; the marketing status IS the one lifecycle in
service.
templates called apps/guide's Template a notification template. It is a Guide
playbook prompt/snippet.
blueprint credited cloud.HealthOwner for its own health route. The plugin declares
OwnsHealth.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
storage's doc opened "Package s3" on a package named storage and called
itself Fiber-facing; s3admin claimed there is no second S3 client
construction in the binary while apps/provisioning builds its own from the
same S3_ADMIN_* variables; base opened with a migration note and counted
two lanes where three prefixes are served, the third from a different
store. graph, sync, framework and datastore opened vague, aspirational, or
ambiguous against apps/provisioning's own `datastore` kind.
The package doc is the OpenAPI tag, the MCP door prose and the CLI group
help, so each now states what the product is and what it answers.
Comments only — build output is byte-identical before and after.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A wire is a SHAPE, and a shape has never earned a path. /v1/insights/e existed
only because the PostHog wire spells fields differently; decodeIngest already
sniffs object-vs-array and bare-vs-envelope on one route, so sniffing one more
encoding is the mechanism that is already there, not a new one.
The wire does NOT go away — decodeEvent picks the decoder by sniffing keys, and
the ingress rewrite that fed the old door (insights-cloud-ingest-rewrite:
insights.hanzo.ai /e,/batch,/capture) now replacePaths onto /v1/event, so every
PostHog-wire caller keeps working. Six doors become five.
The sniff is on KEYS, not on 'did the first decoder return anything'. I wrote the
count-based fallback first and TestMount_HostCarve_IngestsForSiteOrg refuted it:
decodeIngest ACCEPTS a PostHog body as a bare canonical Event and returns ONE
event, which is then dropped whole downstream (canonicalType("") is "event",
not in publicKinds). The caller gets 200 and the event vanishes. A count of 1 is
not evidence the body was understood.
The wires are distinguishable exactly: canonical spells `distinctId` (camel) and
carries `type`; PostHog spells `distinct_id` (snake) and carries `api_key`.
Neither key appears in the other wire, so presence is proof, not a heuristic.
Batches are probed on elements because `batch` is shared.
Security properties re-proven on the one door, not assumed: key extraction from
body/query/x-api-key, presented-but-unresolvable => 403 fail-closed, keyless =>
the anonymous lane. Those tests now drive /v1/event and pass. Full package green.
$source says nothing real used the old door: 3584 rows via 'event', 15 via
'capture', 1 via 'posthog' — and that 1 is my own probe from this session.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
git.hanzo.ai was in CrashLoopBackOff on a corrupted queue LevelDB while work
continued on GitHub, so the two heads drifted: 2 commits reached the forge, 135
reached GitHub. Merged rather than force-pushed — a force-push would have silently
dropped the forge's two.
Nothing is lost, and that is checked rather than assumed. Both sides had
independently added apps/tools/skillstore.go (hence AA) with a BYTE-FOR-BYTE
identical exported API — OpenSkillStore, Close, Put, List, Delete, and the
orgSkillProvider Source/List/Dispatch trio. The forge's five registries.go
functions (putSkill, deleteSkill, listAuthoredSkills, listBySource, listPlugins)
all exist on main as typed toolOps methods instead of raw handlers. Diffing the
symbol sets both ways leaves nothing on the forge side that main does not already
serve, so "the org can add a skill without a redeploy" shipped twice and main
carries the better shape.
Conflicts resolved to main throughout: three source files where main is the
superset, and openapi.yaml + plugin/tools/openapi.json, which are GENERATED and
must be regenerated rather than hand-merged.
Why this mattered: a push to git.hanzo.ai is what fires GitPushEvent ->
isReleasePush -> launchRelease (build -> smoke -> tag). The forge IS the release
trigger, so while it was down nothing built — which is why no cloud image exists
past v1.801.327 despite 135 commits.
go build ./apps/tools/... = 0, gofmt clean.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
security had no package doc and bot's began with a filename; the package
doc is the OpenAPI tag, the MCP door prose, and the CLI group help, so
each now states what the product does. gofmt fixes the pre-existing
import order in security.go.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
o11yAIObs and aimO11yAIObs both named `o11y_ai.observations`. There is no
`o11y_ai` database — verified against system.tables 2026-07-31. Every AI number
on the fleet board has therefore always been zero, and the surrounding
`if err == nil` swallowed the error so it rendered as honest-empty rather than
as a failure. It was not empty: 8,867 observations (and 8,819 traces) sit in
`console`, which nothing reads.
Columns match the queries exactly — project_id, type, start_time, Nullable
end_time, provided_model_name, internal_model_id, total_cost — so the fix is the
name. Proven against the live store: 8,867 generations, $0.0017, 75.8ms avg
latency, and a real per-model leaderboard (qwen3-8b, deepseek-r1-0528,
qwen3-235b-a22b, glm-5) where the panel showed nothing.
The rows span 2026-02-11..2026-03-14, so a recent window still totals zero —
correctly, because no gen_ai observation has landed since. sinceTS already
filters on start_time, so the time selector tells the truth at both ends.
`console` is a SURFACE name on a store and is wrong too; HIP-0132 folds this
signal into o11y.spans (a gen_ai span IS the observation of record). Both consts
move there together under #102 — they are one fact stated twice, which is how
they drifted into pointing at nothing without either being noticed. Pointing at
the rows that exist is the step that does not require o11y to exist first.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The project sub-scope rode X-Project-Id alone, and a git client sends no
headers, so a repo outside the org's default scope had no remote a client could
reach: cloneURL emitted /v1/git/<org>/<name>.git for every repo, and
resolvePackRepo dropped the scope entirely for anonymous reads.
Smart-HTTP and SSH both take the scope as an optional middle segment —
/v1/git/:org/:project/:repo and git@host:org/project/repo.git — beside the
existing two-segment routes, which keep their exact meaning. cloneURL and sshURL
advertise whichever form matches the repo, so a caller is never told a URL that
does not work.
The path wins over the header when both are present, because the path is what a
client can express. An anonymous caller may use it: naming a project addresses a
repo rather than asserting a scope, and the repo's Public flag still decides the
read, whereas an unauthenticated X-Project-Id stays unvalidated input and is
ignored as before. The segment is checked against projectRE, since it becomes a
storage path segment.
This is what lets one Hanzo org hold repos from several GitHub owners:
hanzo/hanzo-apps/ai and hanzo/hanzo-docs/ai are distinct repos rather than two
upstreams fighting over hanzo/_/ai.git.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The subsets and the fleet spec are generated from each app's real router, so
adding subscribe/card and test-mode to commerce means regenerating both or the
drift gate fails: a route missing from openapi.yaml is a route no generated SDK
client can reach, and these two are the paid path's front door and its live
switch.
The weave also removes /v1/sessions and its tag. That is not collateral — it is
the gate working. afdda829c folded the sessions roster into apps/agents and
deleted apps/sessions and plugin/sessions/ outright, but left the fleet spec
naming three routes nothing serves. Documented and gone is the same defect as
served and undocumented, in the other direction.
organization.TestMode() is !o.Live, and that org record is the single authority
for both the Square environment and the ledger bucket. An org nobody has flipped
therefore transacts in SANDBOX — fail-closed by design, and the reason a
deployment holding production Square credentials can still hand a buyer a
sandbox card form.
The switch had no route in this binary. It lives on commerce's mint group,
which the co-resident embed never compiles, so the flip could not be performed
at all. Registered here on auto-recharge/run-all's chain, because it is the same
class: a money-MINT control gated to the service token or a platform global
admin, never the org-level Admin bit — an org admin must not be able to move
their own org between sandbox and production.
Routing it does not flip anything. hanzo stays test-mode until an owner calls it.
The doc asserted "Org routing header is X-Hanzo-Org" and cited
commerce/middleware/accesstoken.go GetHeader("X-Hanzo-Org") for it. That
citation is false: commerce v1.49.32 reads X-Org-Id (accesstoken.go:110,167)
and contains no X-Hanzo-Org read anywhere. The code was always right —
metering.go:77 sends X-Org-Id — so only the doc pointed the wrong way.
It pointed the wrong way at the one header whose failure is silent. Commerce's
selector falls back stashed-org -> X-Org-Id -> COMMERCE_SERVICE_ORG -> "hanzo"
and never refuses, so following the doc would not have errored; it would have
billed the house org for every tenant. The doc even stated that consequence
("Wrong header -> debits the default hanzo ns") while naming the wrong header.
X-Hanzo-Org is real but is a different header going the other way: cloud stamps
it on served /v1/cloudflare responses as the acting org (cloudflare.go:461) and
platform asserts it to prove no comingling (cloudflare-pages.ts:84). Both
directions are now written down so the names stop being interchangeable.
Every money endpoint the commerce plugin registers co-resident — the public
plan catalog, invoices, subscriptions, top-up, spend caps, payouts — was
published by an app the router never handed the request to. account-bridge
owns the /v1/billing REMAINDER, and a prefix is an exclusive subtree, so each
leaf the manifest did not name deeper landed on the bridge and answered
"sign in to view billing": to a signed-in buyer, and to the anonymous
pricing page that reads GET /v1/billing/plans. The registrations were correct
and unreachable; hanzo.ai/pricing could never leave its static fallback.
subscribe/card had no route anywhere. commerce publishes it on the api.Route()
'user' group the co-resident embed never compiles, and mount never registered
it — so the one endpoint that turns a visitor into a subscriber was not
addressable in this binary. It is registered here on topup/token's chain,
which it matches exactly: both are browser money-writes charging a single-use
Square nonce, both server-authoritative on price, PAN never touching us.
manifest/apps.go names each leaf deeper than the sibling that was swallowing
it, and the unreachable ledger shrinks by twelve. Proven by reverting the
prefix and watching the oracle name the exact defect.
- Route to commerce: plans, invoices, payment-config, payouts, spend-alerts,
subscribe/card, subscriptions, topup/token
- billingForwardable gains subscribe/card, so the split-deploy fallback
matches every sibling money-write
A repository name is unique within one GitHub owner, and a Hanzo org may hold
several installations — hanzoai/ai is the Go backend, hanzo-apps/ai is the
hanzo.ai site, hanzo-docs/ai is docs.hanzo.ai. Selection matched on the bare
name, so {"repos":["ai"]} imported whichever the listing reached first.
repos[] now takes "owner/name" as well as "name", and a bare name reaching more
than one granted repository is refused with all of them listed, so the caller
names the one it meant. Selection moves to selectImports, separate from
resolving the token and listing the installation, which is what makes it
testable.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
sentry.hanzo.ai answered "Access required — this account isn't authorized for
this, it's an admin-only surface" to a signed-in customer whose own 75 errors were
sitting in event.error. Errors, logs, traces and metrics are a tenant's own
telemetry, so membership of an org is the whole admission test. Gating the PRODUCT
on platform sudo meant no customer could ever use it, and that the only way to see
it was to become sudo — which then shows every org's errors instead of your own.
Sudo belongs one level in, on the cross-tenant view.
The first cut of this also added a scopeToTenant that deleted ?org=/?orgId=/
?tenant=/?allOrgs= from a member's query. Adversarial review killed it, correctly:
- INERT. hanzoai/o11y has no query-parameter org selector. Every read takes its
tenant from orgFromContext -> ClaimsFromContext, set only from the X-Org-Id
this gate validated. The eight keys were read by nothing.
- BYPASSABLE. url.ParseQuery SKIPS a pair containing ';', so ?org=victim;x=1
left q.Has("org") false and forwarded the raw query verbatim, org=victim
included. It was case-sensitive besides (Org=, ORG= passed) and named none of
organization=, owner=, orgs=, workspace=.
- LOSSY. On a match it re-encoded the whole query, dropping pairs Go rejects and
rewriting %20 to + inside a caller's own ?query=.
A denylist over a lossy parser is not a filter, and one that guards nothing while
reading as a control is worse than none. Deleted, with the reasoning kept where it
was so it is not re-added.
Isolation is the org pin: SanitizeIdentity deletes every client X-Org-*/X-User-*
header at ingress and re-mints X-Org-Id from the validated principal's own claim.
The tests now assert THAT — a request naming another tenant in the query is served
scoped to the caller's own org — across every product path, including the
spellings and the semicolon form the denylist missed.
Verified: gofmt clean, go build ./apps/o11y = 0, suite ok. Negative control:
restoring the admin-only term fails TestProductServesAnOrdinaryOrgMember and
TestMemberCannotReadAnotherOrg; restoring passes. DSN ingest and health probes
still short-circuit ahead of the decision.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
admin.hanzo.ai showed "Could not load — iam: iam status 200" on Organizations and
on most other admin views, because they all read through one client.
The string is the tell. apps/admin/iam decodes IAM's {status,msg,data} envelope and
errors when status != "ok". IAM's NATIVE surface returns typed output with no status
field at all, so a healthy 200 carrying real data decoded as Status:"" and Msg:"",
and the empty-msg fallback printed the HTTP code of a response that had in fact
succeeded. The transport was never broken; the contract was.
git log -S found how: 85dd3a65 ("name: a thing the host loads is a Plugin") was a
mechanical MountSpec->Plugin rename across 132 files, and it collaterally rewrote
seven IAM paths while leaving their params and the decoder untouched. It left the
comment "; was get-user?id=" behind while changing the selector above it.
That surface is not a legacy prop — it is what the rest of cloud already speaks.
apps/account/iam.go calls /v1/iam/keys, add-organization and update-user through the
same decoder and the same status check. apps/admin was the only caller that had
drifted, and update-user escaped the rename, so this client was split across two
surfaces; it is now one.
The fakes are why CI never caught it. They matched HasSuffix(path, "/users"), which
accepts the native path while returning the compat envelope — a pairing IAM never
produces. Green tests, broken production. They now match the FULL path exactly, so
the next rename fails loudly instead of being quietly absorbed, and a new
apps/admin/iam/iam_test.go (the package had none) reproduces the production string.
Verified on this commit: go build ./apps/admin/... = 0, go test ./apps/admin
./apps/admin/iam = ok, gofmt clean on every file touched.
Credentials and scoping are untouched: still the caller's own Cookie plus
Authorization, no service credential, and IAM pins a non-super principal to its own
org on every one of these paths.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cd.hanzo.ai renders the shared chart from hanzoai/universe and pins cloud's image
in that repo's values/hanzo/cloud.yaml. The Application has been Synced/Healthy
throughout — nothing was broken — but the pin moved by hand, so a release stopped
one manual step short of production. That is why v1.801.327 outlived every commit
after it.
This file is the overlay the Application layers last, read from THIS repo at the
tag CD resolves. The 848 lines of real configuration stay in universe; the only
key here is the only key it overrides.
Seeded at the version already live so the first multi-source sync is a no-op that
proves the wiring without moving production.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A live session already records the machine, repo and cwd it runs on. What was
missing to WATCH one is its address: the URL that machine published for its
terminal (zrok gives one without opening a port). One column, carried through
register, patch and the view, alongside the execution context it belongs to.
It is a URL rather than a stream because the bytes belong to the machine running
the shell — cloud holds the address, never the connection, so a session that ends
stops answering in its own frame instead of leaving a console holding a half-open
stream. https only: the console frames this value, and any other scheme is a way
to get a javascript: or file: URL rendered on a signed-in page. A pointer on
patch, so a session that stops sharing can withdraw it.
This REPLACES the separate /v1/sessions plane added earlier in this branch, which
was a second answer to a question /v1/agents/sessions already answers.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/event and /v1/insights/e are the two WIRES this package accepts. The
other three doors — /v1/analytics, /v1/analytics/batch, /v1/tracker — were
more spellings of the canonical wire /v1/event already serves, kept on the
stated grounds that callers still named them. Neither ground survives:
- @hanzo/event 0.3.x posts /v1/event and is what every Hanzo surface now
ships. Its predecessor @hanzo/capture 0.1.1 POSTed /v1/analytics and
beaconed /v1/tracker; the fleet's last importer moved this cycle.
- /v1/tracker was never reachable here. apps/tracker owns the prefix in the
manifest and registers only /v1/tracker/projects/…, so the bare path has
answered 405 in the fleet while passing analytics' single-app tests —
manifest/router_test.go carried it as a known two-claimant name. Dropping
the squatter resolves the collision and the ledger line goes with it.
- the batch alias was held open by "openapi analytics_batch, the generated
python SDK, and `hanzo analytics batch`". Those name analytics.hanzo.ai:
its batch takes an array of SendPayload and answers
{size,processed,errors,details}, this one answers CaptureResult, and cloud
serves none of that collector's routes. They were never a contract here.
Batch stays a BODY, not a path — decodeIngest takes {batch:[…]} and a bare
array at the one door, the same reason there is no /v1/event/batch.
The READ lenses are untouched: /v1/analytics/{overview,timeseries,top,health},
/v1/errors, /v1/insights/{events,health} all keep their routes, and every
manifest prefix stays (a missing prefix is an outage). What ends is this
package's claim on those paths as WRITE doors.
$source loses 'capture' with the doors that stamped it; rows already carrying
it keep their value.
The subset and the fleet spec are regenerated from the router, so
plugin/analytics/openapi.json and openapi.yaml drop exactly the three POSTs.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The public tenant read (which the pay SPA hits on every boot to initialize its
Square card iframe) returned 404 {"error":"unknown tenant"} for pay.hanzo.ai and
api.hanzo.ai alike, so the card and top-up path could not load at all.
Cause was host resolution, not tenant data and not the pinned version: fiber
parses the request URI once, so commerce's forwardedHostMiddleware — which lifts
X-Forwarded-Host into the Host HEADER — never changed what Host() returns.
Behind the ingress the parsed host is empty, so every host normalized to "" and
hit Resolve's single error path.
v1.49.32 resolves the host at the point of use (parsed host, then
X-Forwarded-Host left-most, then the raw Host header) across all three
tenant-resolution call sites, with the parsed host always taking precedence.
main did not compile. "event: the fact, the stream, and the writer" added 1,529
lines across three new files and never touched capture.go, so the consumer landed
without the producer: fact.go read e.Kind, e.Span, e.Metric, e.Site, e.Level and
e.Release off a CaptureEvent that had none of them, and warehouseReady/
warehouseExec were declared twice. Every build in the repo was red, which is every
test, every CI run and every image.
SITE IS NOW ON EVERY EVENT, and that is the substantive half.
It lived on the exception alone. sentry.hanzo.ai could therefore group faults by
site while analytics.hanzo.ai, reading the event stream, had no site column at all
— "all sites" was a question the data could not answer, and the two surfaces
described different worlds. One property per row is what makes them read the same
one. Level, Release, Environment and Service move for the same reason: they
qualify a signal whatever its kind, and none of them is error-only.
The four OTel signals now share ONE envelope with three bodies — Log, Span,
Metric, each a pointer so absent stays distinct from empty. They differ in their
BODY, not in who sent them or when, and an envelope per signal would duplicate
org, time, session and identity four ways and let them drift.
Exception gains structured Frames beside the raw Stack text. Neither derives from
the other and a client may send either: raw text cannot say whether a frame is
ours, and cannot be scrubbed field by field.
warehouseReady/warehouseExec keep ONE declaration, in warehouse.go beside the
writer that uses them, so a test substituting the store cannot substitute half.
The four new nested types join the proseless ledger, which that file documents
exactly: they reach the document through openapi.Register, which derives schemas
by REFLECTION, and Go drops comments — so zipdoc cannot lift their prose however
well they are commented, and they are commented.
Subset and fleet golden regenerated (make -C apps/analytics describe; openapi-weave).
177 packages green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The tool required CLOUD_KMS_MASTER_KEY. Every deployment sets
CLOUD_KMS_MASTER_KEY_REF — cloud's own env, the direct Secret cloud-kms-master-key.
So the tool exited 2 in the only environment it was ever meant to run in.
That is part of why its migration never ran: not merely that nothing invoked it,
but that invoking it did not work. A tool that cannot read the deployment's own
configuration was never going to be run, and the stores it was written to migrate
stayed unopenable until an outage surfaced them.
It now reads the same variable cloud reads, keeping the old name as a fallback so
a one-off already invoked with it keeps working.
Verified against production: with the variable mapped by hand, -dry-run reported
220 stores needing migration and 0 unopenable, and the run migrated 221 carrying
each store's existing DEK. /v1/git/repos and /v1/sync returned to 200, the mirror
drained, and git.hanzo.ai caught up to GitHub at 3bf3d2da6 — which is the revision
cd.hanzo.ai now reports synced.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A coding session runs on a developer own machine, not in the cluster, so nothing
in the cluster can enumerate them — the machine has to say so. A host agent runs
a terminal, publishes it through zrok for a public URL without opening a port,
and beats here. This surface holds the roster; it never proxies the terminal.
Liveness is a TTL, not a state machine. A session is live if it beat within
SessionTTL, so a laptop that sleeps mid-session drops off the roster and returns
when it wakes, with nothing having to observe that it died.
Isolation keys on the ORG rather than the person — watching a teammate build is
the point — and is a mandatory predicate on every statement, never a query param.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
PRODUCTION IS DOWN ON TWO STORES AND THIS IS WHY.
"a store's key names its owner" changed the sidecar derivation from Global to
owner-bound. The code that WRITES the new form shipped. The migration for stores
already written in the old form did not — it exists only as cmd/cek-rewrap, and
nothing invokes it: not boot, not open, not a Job. Its own commit message says
"needed and did not ship".
So two long-lived stores stopped opening:
OrgDB open "/var/lib/cloud/orgs/hanzo/git.db" … message authentication failed
OrgDB open "/var/lib/cloud/orgs/hanzo/sync.db" … message authentication failed
That is the native git plane and the mirror engine, both 500. With the mirror
engine down, pushes to GitHub never reach git.hanzo.ai, so cd.hanzo.ai has been
reconciling a universe that is two commits stale — and every deploy stopped,
including the deploy that would have carried the migration. Newer stores were
unaffected, which is why the outage reads as two odd endpoints rather than an
incident.
The need is discovered at open, so it is answered at open: Rewrap carries the SAME
DEK to the owner-bound wrapping and the open is retried, once. No tool to
remember, no boot walk, no ordering to get right — one way, at the only place that
knows it is needed.
Conservative by construction. Rewrap reports Already when the owner-bound key
already opens the store, and refuses when NEITHER identity does, so a genuinely
wrong master key or a corrupt sidecar stays an error rather than being papered
over by rewriting a sidecar we could not read. Both are pinned:
TestOrgStoreMigratesItselfOnOpen creates a store under the legacy identity WITH
DATA and proves the rows survive the migration — a changed DEK would leave every
page unreadable — and TestUnreadableStoreStaysAnError corrupts a sidecar and
requires the error.
Removing the retry reproduces the production failure byte for byte, down to the
message.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The unified telemetry plane had a schema and no writer. event.log and event.span
have sat at 0 rows since the databases were created, because nothing in any
shipped binary knew how to fill them — these three files existed only as
untracked working-tree state and had never been committed, built, or run.
fact.go the fact. ONE envelope (org, time, id, name, kind, product,
session/distinct/anonymous/person, url, path, attributes, el)
plus what each signal adds. kind is a COLUMN, not a table:
event.event carries track|page|identify|group, so a caller verb
never becomes a schema decision.
bus.go publish to the EVENT stream, subject per signal (event.<kind>).
LimitsPolicy, not WorkQueue — warehouse, alerts and replay are
independent consumers and each needs its own copy.
warehouse.go durable pull consumer, one per table, ACK only AFTER the insert
commits, idempotent on event.id, so a redelivery cannot double
count and a crash cannot silently drop.
Org rides in the signed envelope, never in the subject, so wildcard
subscriptions stay stable and tenancy stays an authorization concern.
Committed alone, out of a tree carrying 551 files of other in-flight work.
Builds and vets clean on its own package.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This caller already lives under .hanzo/workflows; its `uses:` did not. Reusables
are resolved through services/actions.ResolveUses, which enforces the same
directory allowlist on the REFERENCED path:
"uses:" path %q must be under a configured workflow directory
so `hanzoai/ci/.github/workflows/build.yml@v1` stops resolving the moment
WORKFLOW_DIRS narrows to .hanzo/workflows alone. The failure mode is the one
this file already documents: the forge refuses at InsertRun, before any run row
is written, so there is no failed run to look at — pushes and workflow_dispatch
alike silently do nothing.
Safe to point at .hanzo now because the v1 channel tag moved. hanzoai/ci
66d4f21 publishes build.yml at BOTH paths from every tag, byte-identical apart
from its header, since GitHub resolves only .github/workflows and git.hanzo.ai
only .hanzo/workflows. Verified against the live remote: v1 and v2 each serve
.hanzo/workflows/build.yml as blob 1100f49a672b. Same pipeline, same @v1 — only
the path changed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
PlatformSudo read the HOME org — orgs[0] — so every real operator was denied.
An operator is anchored in a brand org, where they bill and do ordinary work, and
holds the reserved org as a FURTHER membership. The anchor and the authority are
different questions, and reading one for the other made the reserved org
unreachable in practice while the predicate looked correct.
This is what stopped anyone cutting a release or reaching an admin surface.
The anchor still decides the LEDGER, so an operator inspecting a customer spends
their own org's money and never the customer's — cloud's homeOrg is unchanged and
still resolves from the membership set, the API key it authenticated, or the
owner-bound KMS audience.
Full suite: 176 packages.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Cutting the release demanded platform SUDO, and that was a category error.
SuperAdmin is the CROSS-TENANT scope — the authority to act in an org you do not
belong to. Publishing your own org's artifact is not cross-tenant. Requiring the
broadest scope in the system for it did not make the operation safer; it made
releasing impossible for the engineers who own the artifact, while the only
identities that could were the ones already trusted with every other tenant's
data. Conflating "privileged" with "cross-tenant" is the same error that let an
org-role bit be read as platform authority.
Release now takes the SAME authority an ordinary build takes, through the SAME
function: admin of an org that OWNS the registry namespace being published to
(imageInOrgRegistry, which already confines a lux admin from pushing
ghcr.io/hanzoai/*). No new mechanism — the rule that already bounded builds now
bounds the release.
The namespace comes from the CONSTANT releaseImage, never from the request:
launchRelease publishes releaseImage whatever req.Image says, so binding on the
request would check a value the caller chooses.
Four cases pinned: the owning org's admin is authorized and reaches the pipeline;
a lux admin is refused hanzo's release; a plain member of the owning org is
refused (owning the namespace is necessary, not sufficient); and the shared build
token is refused a release while keeping the ordinary build path git-push-to-deploy
runs on.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
PLATFORM_BUILD_CALLBACK_TOKEN could cut a release. It is a bearer secret with no
identity behind it — no membership, no expiry, nothing to revoke but a rotation
that restarts every holder, and nothing in an audit log but "the token". A second
auth system standing beside IAM, deciding the most privileged operation this
surface has: publishing the image the whole fleet runs.
Release now takes principal.IsSuperAdmin and nothing else.
The token STAYS for the ordinary build path. git-push-to-deploy runs on it
(apps/git/build_on_push.go), and removing a credential before its replacement
exists breaks that — the same rule that keeps a service's own identity boundary
running until the edge is genuinely in front of it. Narrowed, not deleted.
TestRunnerRelease_SharedTokenCannotRelease posts the SAME credential to the SAME
endpoint twice, differing only in the `release` flag, so the flag is provably what
the gate turns on: the build is admitted, the release is refused.
THIS IS WHY CI CANNOT SELF-RELEASE, and the fix is not to hand the secret back. A
robot should be able to cut a release — through an IAM identity, not a shared
secret: an application principal holding a release capability, which is IAM saying
THIS named app may release, revocable by name and attributable to it. Provisioning
that app is the follow-on. Until then a release is a SuperAdmin's to cut, and a
SuperAdmin is a human PROVISIONED in the reserved org — never a brand-org user
given a membership row there, which grants nothing and only looks like it should.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/billing/payment-methods answered 405 in production. The handler,
the bridge allowlist and the co-resident commerce mount all shipped — but
a specific route shadows the console pkg's /v1/billing/* wildcard for its
whole PATH, so a GET-only registration made every POST miss on method
before anything else could serve it. The console's save-card call died
there, and with it auto-recharge, which charges the vaulted card.
Registered beside the GET with the same discipline as gpu-charge: the
subject is pinned server-side to the caller's own org so a forged body
can never attach a card to another tenant, and commerce's status is
forwarded VERBATIM so a 402 decline keeps its reason.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An independent review caught both. The previous change renamed the symbol INSIDE a
sentence and left the verb, so two comments read "X-Project-Id is MINTED from the
validated `project` claim (claims.renderProject)" — the citation says render, the
prose says mint, and a reader has to guess which is current.
Six citations still named `iamauth`, a package deleted two changes ago
(hanzoai/gateway/v2/iamauth, iamauth.Claims.MintedProject, iamauth.CookieToken,
iamauth.DefaultProject, iamauth.StripIdentityHeaders). A pointer to a package that
does not exist is worse than no pointer: it sends the next reader looking for a
contract they cannot find. They now name where the contract lives —
hanzoai/authz/edge, edge.Strip, edge.Cookie.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
MINTING IS ISSUING AUTHORITY, and only IAM does it — it signs the token. An edge
verifies one and RESTATES it as headers. The two were one word here, so
mintedProject/mintedBillingAccount are renamed renderProject/renderBillingAccount
and the prose follows. The word stays where it is correct: IAM minting a token, a
sign-in minting a session cookie, mint-user-keys issuing a credential.
Separately, TestRewrapCarriesTheSameDEK was red on main and is not mine — it passed
ALONE and failed in the package. The master key resolves through a sync.Once, so the
test's plain SetMasterKey was a no-op once a sibling had already resolved it: it
minted its sidecar under its own key while Rewrap read whichever key won the race,
and reported "sidecar opens under neither owner nor global" — which reads like the
migration is broken rather than like a test that chose its key too late.
resetMaster clears the Once first, which is what every other test in the package
already does.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Which org a request acts in is ONE rule — the client's selection when the signed
membership set admits it, the home org otherwise, any org for a platform operator —
and it is stated twice: authz.Claims.EffectiveOrg, and cloud's own switch in
SanitizeIdentity.
The second statement is not redundant and should not be collapsed. Cloud resolves
the HOME org from facts the claims do not carry — an API key it authenticated, an
owner-bound KMS audience — so it applies the rule with more information than the
leaf has. Forcing one implementation would mean passing cloud's home AND a sudo
predicate into a parameterized shell, which reads worse at both call sites and buys
nothing: what must never differ is the RULE, not the code.
So the agreement is pinned instead of assumed. Wherever cloud's home org equals the
leaf's, both must resolve the same effective org — across no selection, the home
org, a granted org, an ungranted one, a non-injective one, an org admin, and a
platform operator with and without a selection. Where they legitimately differ the
case skips itself rather than asserting a comparison that has no meaning.
Falsified through the real middleware with a local replace: dropping the membership
loop from the leaf turns it red on exactly the two granted-org cases.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/world/stream is the one operation here that cannot be a typed op (the
pin now cites the wire fact at stream.go:151), and it reached the document
bare. stream.go's init declares its summary and description through
openapi.Describe — SSE contract, heartbeat, best-effort delivery, the GET it
re-fetches truth from, the 403 — so the subset, the fleet golden, the SDKs
and the spec-derived CLI all carry it. TestEveryRefusalCarriesProseInTheDocument
closes the loop: a pinned refusal with neither summary nor description goes
red, so the world surface measures ZERO bare operations and stays there.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Register declares the bodies the router cannot derive; nothing declared the
PROSE of an operation the wire refuses to let become a typed op, so a pinned
refusal published an operationId and nothing else — every SDK generated off
the document offered a call it could not explain. Describe is the prose half
of the same registration, with the same drift-proof property: a declaration
whose route is not in the router never renders, and each half guards its own
duplicate, so one package Registering the bodies and Describing the prose of
one operation is one declaration in two statements, not a clash.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nothing has ever reached a human. Alertmanager's slack_configs read a
secret holding the receipt sink's own URL, so 439 'slack' notifications
— including criticals firing right now — were delivered into a log, and
the config's own comment admits PAGE-DELIVERED is a log line, not a page.
The fix is NOT an incoming webhook: that is a second Slack credential
living outside KMS and a second egress beside the one the product
already uses. This receiver now forwards each notification through
integrations.SendSlack — the ONE Slack egress, posting with the org's
KMS-custodied bot token from the installed Hanzo app, shared with
channels and automations. One credential, one egress, one receipt.
Detached and fail-soft by construction: Alertmanager is waiting on this
request, and an alert path that can block or fail on a third party goes
quiet exactly when the third party is having the outage. The receipt
lands first and unconditionally, so a paging failure is itself logged
against an alert we can still prove arrived. Resolved notifications page
too — 'it recovered' is the half people wait for — and a storm is
bounded to 20 lines with the overflow COUNTED, never silently dropped.
Config: CLOUD_ALERTS_SLACK_CHANNEL (no channel, no paging) and
CLOUD_ALERTS_SLACK_ORG (defaults to the platform tenant).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
That change moved KEK derivation from fileID alone to (principal, fileID). Every
sidecar written before it was wrapped under Global, so EVERY pre-existing org
store stopped opening the moment it landed — reported as 'wrong master key or
corrupt sidecar', which is true and misleading: the key is right, the sidecar is
intact, only the identity moved. Measured on prod: the hanzo org's kms.db.dek
unwraps cleanly under Global and not at all under Org(hanzo), so every KMS read
and write for that org has been 502ing.
Rewrap unwraps under the legacy principal and re-wraps the SAME DEK under the
owner, atomically. wrapExisting is why the DEK survives: mintSidecar always
generates a fresh one, which is right when a store is born and destroys the data
when a store's key is re-homed. The test asserts the DEK is byte-identical after
migration, because that equality IS the safety argument.
Not a compatibility shim — one derivation, and this walks the old world into it
once. A sidecar that opens under neither identity is reported and skipped, never
'repaired' with a new DEK, which would answer every future read with plausible
garbage instead of an error.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
DESIGN ONLY — nothing wired. This marks the seam so the next change lands here
instead of adding a sixth object to every stateful pod.
Each replicated service currently carries four objects and a key: a replicate
container, a generated ConfigMap, a restore initContainer, and its own age
keypair. All of it exists because replicate is a separate binary watching a file
it does not own, so the file has to be described to it.
That arrangement produced four independent outages in one day (2026-07-29): a
misindented age stanza replicate refused, an age/plaintext mismatch between
config and bucket, a service whose data dir was not mounted, and a restore path
that had never once run. The last is the instructive one — restore only runs
-if-db-not-exists, so while the local file happened to exist it was never
exercised. The backups were configured, not current, and not restorable, and
nothing said so until a volume was lost.
Open is already 'the single way a cloud store opens its file' and Exists already
answers 'is there a store here' — which is the entire question the initContainer
shelled out to ask. Native, the lifecycle collapses into Open: hydrate if
absent, follow after. Restore stops being a lifecycle stage and becomes what
Open does; the ConfigMap, initContainer, second container and the ordering
between them all disappear.
ONE KEY. cek already holds CLOUD_KMS_MASTER_KEY_REF and refuses to open a store
unkeyed. The age identity is a SECOND key system encrypting the SAME data, with
no rotation story at all — an age identity cannot be rotated after the fact, so
losing it makes every replica under it unreadable. So the age keypair should not
be migrated to KMS; it should stop existing, and the replica should be encrypted
under the key the process already holds. That makes the KMSSecret file added to
universe today unnecessary rather than merely unarmed.
Three verbs, all about bytes at a path: Has, Hydrate, Follow. Follow returns
when following STARTS, not when caught up — a store that refuses to open until
its backup is current will not open during an S3 incident, trading a durability
risk for an availability one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
That change moved KEK derivation from fileID alone to (principal, fileID). Every
sidecar written before it was wrapped under Global, so EVERY pre-existing org
store stopped opening the moment it landed — reported as 'wrong master key or
corrupt sidecar', which is true and misleading: the key is right, the sidecar is
intact, only the identity moved. Measured on prod: the hanzo org's kms.db.dek
unwraps cleanly under Global and not at all under Org(hanzo), so every KMS read
and write for that org has been 502ing.
Rewrap unwraps under the legacy principal and re-wraps the SAME DEK under the
owner, atomically. wrapExisting is why the DEK survives: mintSidecar always
generates a fresh one, which is right when a store is born and destroys the data
when a store's key is re-homed. The test asserts the DEK is byte-identical after
migration, because that equality IS the safety argument.
Not a compatibility shim — one derivation, and this walks the old world into it
once. A sidecar that opens under neither identity is reported and skipped, never
'repaired' with a new DEK, which would answer every future read with plausible
garbage instead of an error.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
git.hanzo.ai is canonical and GitHub is its mirror, but the two had drifted the
wrong way round: GitHub carried 150 commits the forge had never seen, and the forge
carried three the mirror never carried back —
ae978f2b6 ci: v* tags never triggered a build — a duplicate YAML key ate the filter
5693f87e8 ci: the reusable build was pinned to a path that does not exist — CI was dead
7376db9fb docker: cgo plugin builds need -tags sqlite_math_functions
Each of those is a real repair to the thing that BUILDS us, made where the build
runs and stranded there. Meanwhile every dev commit — including the AI
balance-reader repair hanzo.app is currently 503ing without — sat on GitHub where
nothing builds it.
Merged clean, no conflicts. This puts the CI repairs and the product history on one
line so a build can exist at all.
plane.Money is a decimal string so an amount survives the process boundary
unrounded, and the receiver already honors it (meter_rpc parses the decimal and
debits it verbatim). Every sender defeated it: each plane.Amount call site was
plane.Amount(money.FromUSD(x.Cents())) — the exact value, flattened, re-wrapped
as "exact". After the Usage.Money guard fix let sub-cent debits SURVIVE to the
peer path, meterPeer posted them as $0.00. The hole had moved, not closed.
resource_billing_peer.go meterPeer sends u.Money() whole; the failure log
prints the amount, not a cents field that reads 0
for exactly the debit that was lost
apps/commerce/balance_rpc balance and usage rows carry the ledger's own value;
rebuilding an "exact" amount FROM r.Cents was the
sharpest form — the wire type promised precision the
value had already lost
apps/finance UsageRow carries Amount (exact) beside Cents (its
rounding), the split TxnRow beside it has had since
it was written
apps/billing the usage envelope gains `decimal` — amount stays
cents for the wire the console parses today; a row
built without the exact value omits the field rather
than asserting a zero the cents deny
money.Unwrap is the one bridge to the shared type the plane speaks; when the
wrapper collapses into hanzoai/money, call sites lose the call and nothing else.
Cents-only senders left alone, stated here so nobody re-files them: gatePeer
(the gate reserves; its input is priced in whole cents), starter and treasury
reserve (stored as cents; conversion is exact).
Three tests, each shown red without its fix. The peer one captures the RecordIn
on a real plane socket: without the sender fix the debit crosses as wire "0"
"USD"; with it, 0.0025 arrives whole.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud's boundary wrote its identity headers as STRING LITERALS while authz owned
the names, so the two lists were free to drift — and one already had. X-App-Id was
written here and named nowhere in the estate, so an edge stripping authz.Headers
would have left a client copy standing for whoever read it next. It is a caller
label rather than an isolation boundary, which makes forging it cheap, not a reason
to leave it forgeable. authz names it now, and every write here goes through the
constants.
TestEveryHeaderWrittenIsAName holds the property mechanically. It greps the source
rather than exercising a request, because the hazard is a write no test happens to
reach: a literal is what it catches. Reverting one constant to its literal turns it
red.
homeOrg composes instead of restating: the two branches above it are the facts only
cloud has — it authenticated the API key, it knows the owner-bound KMS audience —
and everything else defers to Claims.Home, which is now the same rule cloud
discovered and the leaf had not learned.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`export TMPDIR ?= $(HOME)/.cache/go-tmp` pointed the linker at disk but never made
the directory. go does not create it; it stats it and fails:
go: creating work dir: stat /root/.cache/go-tmp: no such file or directory
account.go:160: running "go": exit status 1
!! account cannot project its own document — an app that cannot describe itself is the bug
That message reads like a contract defect in the account app. It is not. The app
never got to describe itself, because `go` could not start.
Only CI could hit it. A dev box has ~/.cache already, and macOS always exports
TMPDIR so `?=` never even takes this branch — which is why it passed locally and
failed in the container, where TMPDIR is unset and HOME=/root.
The cost: cloud has not built since 2026-07-29 01:10. Every image since failed on
`test app-contract`, so v1.801.322-324 do not exist and the fleet still runs .321.
The AI balance-reader repair sat un-shippable behind it while hanzo.app answered
503 balance_unavailable on every completion.
Verified under the CI condition, not the local one:
env -u TMPDIR HOME=/tmp/fh make -f mk/go.mk -> creates /tmp/fh/.cache/go-tmp
A `$(shell)` on its own line does not run; the simply-expanded assignment forces it.
The ai/destinations/dns/licensing/runtime/templates tranche of the typed
migration converts nothing, and that is the verified result, not a punt:
all 29 undescribed operations across these six subsets are one of five
registrations whose typing would move the wire.
- ai: app.All("/v1/*") beego adapter (hanzoai/ai v1.832.5 mount.go:128)
- dns: Group("/v1/dns").All("/*") (apps/dns/dns.go:87)
- runtime: app.All("/v1/bot/*") (apps/runtime/ops.go:72)
- licensing: app.All("/v1/licensing/*") (hanzoai/licensing v0.1.5 mount.go:71)
- destinations: g.Post("/:platform", connect) (apps/destinations/destinations.go:170)
The four wildcards relay a foreign handler's own status and Content-Type
verbatim; connect binds a body whose property NAMES the addressed
platform's Spec chooses at request time, with string|number|bool values —
and a typed op would also 400 a malformed body before the handler's
403/404/503 gates, an error-precedence move. Each ground re-verified
against zip v1.18.12 (no All[In,Out]; typed ops answer only c.JSON(out)
under their declared status, typed.go:302-311; bindURL sets scalars, never
a greedy sub-path) — the dns and runtime ledgers now cite the version
go.mod actually pins.
LLM.md's opaque-subset taxonomy had licensing in the wrong class: its
Mount hangs hanzoai/licensing's OWN net/http mux on the wildcard through
zip.AdaptNetHTTP, in THIS process — iam's shape (an absence of
composition), not bot/sentry's cross-process proxy. The fix direction
differs by class, so the misfile mattered: licensing's ops become typeable
by typing them in hanzoai/licensing, not by teaching cloud a foreign
route table.
All six subsets regenerate byte-identical (describe; zipdoc drift: none),
so the published documents are current: 38 ops, 9 described — destinations
4/5, templates 5/5, the four relays 0/7 by design, each refusal pinned in
a typed_wire ledger or named here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Commit 16576f81 typed 19 of this tranche's 27 silent operations and refused 8,
saying all eight were gated by untypedByDesign + TestEveryRouteIsTypedOrNamed.
Seven were. The four /v1/agent operations were not: apps/agent had no test at
all, so the refusal lived in a comment that cannot go red — a route added
untyped there would publish nothing and nobody would be told, and the day
hanzoai/agent ships these ops typed the stale reasons would outlive them.
apps/agent/typed_wire_test.go now measures what the comment asserts, against
the REAL Mount: the served surface is exactly the four relayed operations,
none carries a typed registry entry, and the two ledgers must partition the
surface — so it breaks loudly in both directions. Each entry names the op's
own binding fact beside the shared one (registered by hanzoai/agent v0.1.3
agent.go:166-169, not by cloud): the verbatim upstream-4xx relay on
POST /v1/agent (round.go:110-113), the Deps.Principal func(*zip.Ctx) caller
seam the reads share, and the live-Ctx tool dispatch.
Re-verified with the subsets regenerated from the live routers: all six
(usage, venue, world, agent, bot, help) byte-identical to what is committed —
19/27 described, 0 bare properties, no phantom bodies, no embedded-struct
tells, no schema collisions, no newline summaries.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The nine refusals across this six-plugin tranche (websearch's eight, content's
one) were re-checked against the zip actually pinned today, not the v1.18.11
the note recorded: still no typed All (typed.go registrars are the five named
verbs), op.invoke still 400s on any unparseable non-empty body, WithStatus
still admits one 2xx, and the error handler still renders only *HTTPError —
so neither the firecrawl 200-on-malformed contract nor the resource-deny
envelope is expressible. The stamp moves so the next sweep re-litigates
against the right baseline or not at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud held the third independent reading of what an IAM token means: its own claim
struct, its own algorithm allowlist, its own JWKS cache, its own key selection, its
own issuer comparison. The file said so at the top, and explained why — the gateway
is heavyweight and already imports cloud, so importing its validator back would
braid a module cycle. Both halves were true. The conclusion, write our own, is what
produced the copy.
The premise is gone: hanzoai/authz is 148 packages with one non-stdlib dependency
and imports nothing from cloud or the gateway. So idClaims now EMBEDS authz.Claims
rather than restating it, and validate() is the shared edge.Verifier.
The copy was not free, and both costs were real:
it declared a `type` claim IAM emits NOWHERE and read it as the machine
discriminator, so every machine principal arrived as a human (fixed last change);
its key selection FELL BACK — after the kid-matched key failed it tried every RSA
signing key in the JWKS and accepted the first that verified. A token naming
cert-hanzo was accepted on a signature from cert-lux, and a token naming NO key
was accepted on any of them. Not exploitable by a tenant, because IAM publishes
only certs owned by a reserved platform org, but the INVARIANT was gone: any
future widening of what reaches the JWKS becomes an impersonation path silently.
TestTokenIsVerifiedByTheKeyItNamed pins it through cloud's own boundary, with TWO
platform keys published — the real shape, since rotation is additive and each brand
has its own cert.
A FALSIFICATION THAT DIDN'T BITE, which is the finding worth recording. Reverting
isHuman to the pre-fix reading left every SuperAdmin probe GREEN: that arm is
separately blocked because a machine resolves no home org at all. What actually
depended on isHuman was the ORG-admin bit, and nothing tested it. So the predicate
was load-bearing in exactly one place and pinned in none.
TestMachineIsNeverMintedTheOrgAdminBit now covers it, and reverting isHuman turns
it red. `captured` gained the org-admin bit, because no test was reading it.
WHAT STAYED, deliberately: the trusted-issuer SET (the brands this binary fronts —
authz gained an issuer allowlist for exactly this), the API-key resolver and the
subjectOrg it yields, the memo that keeps a replayed ZAP credential from
re-verifying per frame, and username()'s legacy `name` fallback, which now EXTENDS
authz.Username rather than restating it. VerifiedIdentity.Orgs stays []model.OrgRef
— it is cloud's published contract, copied verbatim into a session by clients/team —
so there is ONE conversion, at that surface.
auth_identity.go: 717 → 520 lines. Full suite 174 packages, CGO_ENABLED=0
-tags sqlite_fts5 with the dev KMS key.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The door's own assertion was stricter than the truth and would have failed a
correct fleet: `{}` is what an op whose body is an ARBITRARY document honestly
publishes — apps/flags' put_v1_flags_defs_key takes a PostHog-shaped definition
the evaluator consumes verbatim, so zip's rootSchemaOf constrains nothing and is
right not to. Absent is the failure; empty is an answer.
The Go gate beside it (manifest/mcp_test.go) already read it correctly — it tests
the raw bytes, where `{}` is length 2 — so the two now agree.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Each build job is a fresh pod with an empty local cache and the fabric passed no
cache flags, so buildkit re-resolved and re-downloaded everything from the network
every time: for studio that is the entire torch stack plus requirements ON EVERY
PUSH, which is why a build takes tens of minutes rather than a couple.
Import and export a per-repo cache at <image-repo>:buildcache (the standard
convention — no extra credentials, GC'd with the package). mode=max exports the
intermediate stages, so a one-line code change reuses the dependency layers instead
of rebuilding them. Both flags are advisory in buildkit: a missing or unreadable
cache ref is a cache MISS, never a failure, so a first build behaves exactly as it
does today. A digest-pinned ref names no tag to hang a cache off and is skipped.
(apps/platform tests don't build on this machine — a pre-existing CGO/sqlite issue
in github.com/hanzoai/base, reproduced on a clean stash — so the added test is
verified by inspection here and will run in CI.)
Typing a route bought OpenAPI prose, an SDK method and a CLI command, and NOTHING
on the public MCP surface. zip has projected every typed op into an MCP tool since
v1.18.6 and cloud never called it: manifest routed /v1/mcp to apps/tools, which
hand-rolled its own tools/list + tools/call over a route-table scrape, and
apps/automations hand-rolled a THIRD catalogue. Three registries for one concept,
and the one the public reached exposed none of the 549 typed ops.
THE DOOR IS THE HOST'S. cmd/cloud sets zip.MCPConfig{Path:"/v1/mcp"} and hands
each plugin its own catalogue at Load. The host is the only process that CAN own
it: MCPTools() is in-process, so a plugin cannot enumerate a lazy sibling, and a
plugin-hosted door costs its own wake on the first list. Measured: POST /v1/mcp
beats ai's "/v1" remainder by specificity, not registration order.
THE LIST IS A BUILD ARTIFACT, so tools/list costs ZERO wakes. It has to be: 112
plugins mount LAZILY, and an MCP client calls tools/list constantly — a door that
fanned out over ZAP to ask would destroy the one invariant that makes 112 services
affordable. The answer is already fixed at build time, by the same typed-op
registry that emits openapi.json, so `<app> describe <dir>` now writes BOTH
projections from ONE mount at ONE instant: openapi.json and mcp.json. They cannot
be generated apart, so a tool cannot exist without its op or carry a stale schema.
The leaf plugin/embed.go go:embeds them (cmd/cloud goes 344 → 345 packages, still
zero from apps/). Measured live with the WHOLE fleet mounted: 549 tools listed,
child count 4 → 4 (the four eager apps, untouched).
tools/call is the ONLY trigger and starts exactly one child — p.target(), the same
single-flighted lazy path a prefix request takes — then forwards the SAME message
to that plugin's own /mcp over ZAP on its 0700 unix socket. Never HTTP. The child's
registry answers, so the host can only NAME a tool, never invoke one the child did
not declare. Measured live: get_v1_pricing woke 1 child and returned the pricing
catalog; get_v1_company answered its own handler's "X-Org-Id required" through the
plugin's full cloud.Serve identity chain.
DELETED, not left dark:
apps/tools/builtin.go (223 lines) — the "full-cloud-control" route→tool scrape.
Structurally dead since the monolith died: in the tools CHILD, GetRoutes() sees
only tools' own ~13 routes, and its schemas were opaque {query,body} objects a
model cannot fill. The new door is what it meant to be, with real schemas.
apps/tools/http.go's mcp/mcpToolList/mcpToolCall/rpcResult/rpcError + the route.
apps/automations/mcp.go's mcp/mcpTools/mcpResultObj/mcpErrorObj + its route.
GET /v1/mcp — a Source view that is GET /v1/tools?source=mcp by its own comment.
Principal.credential + credentialHeaders — replay state only builtin.go read.
KEPT, because it is a different capability: apps/tools' EXTERNAL MCP server
registry (records, KMS-sealed secrets, SSRF-validated dialer, tools/list fan-out),
now owning /v1/mcp/servers alone. Its tools, org skills, agents, functions and
connector actions are ROWS, not code, so no build-time catalogue can hold them —
they are reached through the typed POST /v1/tools/call, which is itself a tool on
the door. Nothing lost: connectorToolProvider already published every connector
action into that one registry.
THE GATE. mk/fleet.mk surface-check (which .hanzo/workflows/cicd.yml → hanzo.yml
app-contract actually invokes) regenerates every app FROM SOURCE and fails on
`git status --porcelain -- openapi.yaml plugin/` — mcp.json is under plugin/, so it
was covered the moment it landed there. PROVEN TO FIRE: adding one typed op to
apps/guide without regenerating turned it red on BOTH plugin/guide/mcp.json and
plugin/guide/openapi.json; reverted, green. Four more, all cheap: no App row may
claim /v1/mcp (fiber MERGES byte-identical patterns, so a Load there would shadow
the door silently); no served path may END in /mcp; no Go source outside cmd/cloud
may name an /mcp path unless it is a named foreign engine (apps/tasks' own
surface, which is not a projection of our ops); every catalogue tool must be an
operationId of its own app, unique fleet-wide, with a NON-EMPTY description —
the last one because a nameless tool is a silent failure a model pays context for.
549 tools across 36 apps, 349KB on the wire. zip v1.18.11 → v1.18.12.
Capability check, precisely: the 17 executable connector actions the deleted
automations door listed are NOT tool names on the fleet door, because they are
per-tenant rows — connectorToolProvider publishes every one of them into the ONE
registry from the same `registry` map that door read, so they are reached through
tools_call with the same activation, price, meter and audit. Nothing is lost; one
hop is added. Same for org skills, agents, functions and external MCP servers.
One door this gate structurally cannot claim: /v1/tasks/mcp is hanzoai/tasks' own
engine surface behind cloud's identity gate, mounted on a raw net/http mux so it
is in no subset at all. It is a foreign engine's tools, not a projection of ours,
so it is NAMED in foreignDoors with the reason rather than deleted.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The work list was the ARTIFACT, not a grep: every operation in
plugin/{wallets,webhooks,account-bridge,ads,channels,code}/openapi.json carrying
neither a description nor a summary — 44 of them, six plugins at 100% undescribed,
which is exactly the set that projects to NOTHING: no prose, no MCP tool, no CLI
command, no typed SDK method.
35 are now typed ops. Per plugin: wallets 8/8, webhooks 8/8, code 7/7, ads 6 of 7,
channels 6 of 7, account-bridge 0 of 7. Each converted package carries
untypedByDesign + TestEveryRouteIsTypedOrNamed + TestEveryTypedOpIsDescribed
reading the LIVE router, whose two ledgers must SUM to the served surface — so a
route added untyped here goes red and a stale reason goes red too.
THE NINE REFUSALS, each wire-bound and each MEASURED:
* account-bridge's 7 are TWO registrations, and they were already a closed
refusal one package over: verbatim per-tenant forwards on a greedy wildcard
(apps/account/account.go routesBridge), held by apps/account/typed_wire_test.go.
Nothing to convert.
* POST /v1/ads/campaigns/{id}/launch is deliberately BODY-TOLERANT: it discards
the Bind error, so a malformed body launches on the stored account at 200,
where op.invoke's unconditional decode 400s. TestLaunchStillIgnoresAMalformedBody.
* POST /v1/channels/{channel}/send has a package-local 1 MiB body cap a typed op
never sees, AND DisallowUnknownFields, which refuses a spoofed identity field
loudly where jsonenc.Unmarshal drops it silently.
TestSendKeepsItsCapAndItsStrictness.
WIRE PRESERVED, and the three places that took care:
* POST /v1/code/ask reads ?q= first and lets a non-empty body `query` WIN —
the opposite of zip's body/query/path order. One field per source
(json:"-" url:"q" beside json:"query" url:"-") reproduces it rather than
inverting it; all four combinations asserted.
* ?since= on /v1/channels/inbox 400s on a non-integer, and setScalar silently
zeroes one — so it stays a STRING. Where the handler DEFAULTS instead
(?limit= on ads, webhooks, code) an int is wire-identical.
* every body-only field carries url:"-": zip's binder fills an In from the query
too, and ?custody=, ?prune=1, ?url= and ?dmPolicy= would each have redirected a
write the body never asked for. Four TestTheQueryStringCannotRedirectAWrite.
LATENT DEFECTS, all fixed:
1. five packages had NO //go:generate zipdoc directive — which is why 44
operations published nothing: the prose had nowhere to be lifted to.
2. GET|POST /v1/webhooks/ published a TRAILING SLASH (failure mode #9) for a
collection every caller addresses without one. Artifact fixed, wire unmoved —
both spellings still reach the handler.
3. apps/code's test harness RECONSTRUCTED its seven routes by hand, so nothing it
asserted was evidence about the served surface. routes() is a function now.
4. none of the five installed a cloud.Bridge of its own, relying on Serve's
app-wide install that no package test harness runs — the org path was untested.
5. two one-name/two-shape collisions the weave would have refused on entry:
wallets' Account (books') and ads' Campaign (marketing's). Both unpublished
names yielded — WalletAccount, AdCampaign, no wire movement.
ONE DELTA, recorded not glossed: webhooks' 401 had two messages; principal.OrgFrom
folds both halves into one answer, so the typed ops answer one 401 naming both.
Status, shape and ordering unchanged.
apps/{wallets,code,channels}/... enter allowedRequestUses with their reasons: the
audit actor and the server-minted project scope (wallets), the billing payer and
project (code), the org-admin mutation gate (channels). None is a tenant key.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestRequestEscapeHatchIsPinned has been failing on main since fc46e916:
apps/do/do.go, apps/flags/routes.go, apps/research/research.go and
apps/treasury/treasury.go each added a cloud.Request call site without the
allowedRequestUses entry that is the whole point of the pin. Not caught earlier
because the escape hatch's own gate is in the ROOT package, and a pass that
touches only apps/<x> never runs it.
Each entry below was written by READING the call site, not by pattern: do's org()
turns on validated-ness and the SuperAdmin "admin" namespace OrgFrom cannot
express; flags' callerOf needs the project scope and the audited actor; research's
project() is a sub-scope column and answers DefaultProject off the HTTP path;
treasury's admin/myAccounts are the platform-sudo gate and the ?org= that is the
only way across the ledger's tenant boundary.
Verified: `go test -tags sqlite_fts5 .` is green again.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Five plugin subsets published 56 operations with ZERO descriptions: no schema, no
prose, no MCP tool, no CLI command, no SDK method. 39 of them are now typed ops,
which is ONE registry entry with N projections — the REST route, the OpenAPI
operation, the /mcp tool, the CLI command and every generated client all follow
from the same declaration.
authors 0/11 -> 11/11 described
legal 0/11 -> 10/11
campaign 0/11 -> 9/11
tracker 0/10 -> 8/10
base 0/13 -> 1/13
fleet golden: 517 -> 556 described, 1407 operations unchanged.
THE WIRE IS UNCHANGED, and that is the point of the exercise. Six details had to
be carried over deliberately rather than inherited, each pinned by a test:
* the 1 MiB request-body cap in apps/legal, as ONE middleware in FRONT of the
typed ops. A typed op receives its DECODED In, so a size check inside one
runs after the parse it exists to precede — zip answers 400 about unparseable
bytes where this package has always answered 413. 403 still outranks 413, and
the signature completion (which discards its decode error) is deliberately
not capped.
* the body REQUIREMENT on every write that bound one with c.Bind. zip's typed
decode is TOLERANT by construction, so a naive conversion turns a bodyless
PATCH from 400 into 200-with-nothing-changed — measured against the untyped
handlers, not assumed.
* 201 where it is unconditional (legal's two creates, campaign's create),
DECLARED with zip.WithStatus so the document keys its response on the code
the route sends. Where it is CONDITIONAL (authors connect/verify/record answer
201-on-create and 200-on-found from one address) cloud.Created sets the code
the route has always sent and the prose states it — zip cannot declare two
success codes for one op.
* Cache-Control: no-store on legal's two document reads, which carry contract
text.
* the JSON ARRAY the tracker listings answer: a NAMED slice Out, because an
unnamed one publishes no response content at all.
* the conditional keys of legal's document view, as two Out types rather than
an omitempty that would also drop an empty body from the single read.
SEVENTEEN operations stay untyped, each for a MEASURED wire fact, each held as a
closed list a new route cannot join by accident:
* 12 x /v1/collections[/*] (base) — a VERBATIM reverse proxy to the managed
Base orchestrator. Status, headers and bytes are the upstream's; no Out can
carry that.
* POST /v1/tracker/projects and .../issues — the pre-create balance gate renders
its denial with cloud.DenyResource, the fleet's NESTED {"error":{...}} at
402/503, which a typed op's returned error cannot carry.
* POST /v1/campaign/{id}/launch and /pause — neither has ever read a request
body, and zip's invoke refuses one it cannot parse BEFORE the handler runs.
Measured: 200 untyped, 400 typed.
* POST /v1/legal/documents/{id}/sign/complete — DISCARDS its decode error, so
an unparseable body still drives the provider-reported completion.
The last two are the same zip gap: an op cannot declare that it takes no body, or
tolerates one it cannot parse. hasRequestBody already computes "binds nothing the
URL does not carry"; invoke reading the body only when that is false closes both.
Nine published schema names collided with already-published ones (Campaign with
marketing, Filing/Signer with company, Template with guide, Metrics with agents,
repoView with git, projectView with platform, healthReport twice). openapi.Weave
refuses one name meaning two things, and it refused this. The colliding types are
renamed on MY side only, as DEFINED types over the same structs — the json tags,
and therefore the wire, are byte-identical.
Also: registers the four new cloud.Request call sites in the escape-hatch pin with
the reason each one needs the request (the project sub-scope that picks a physical
store, the SuperAdmin claim, the audit ACTOR, the body gate).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The weave went red the moment both landed: apps/treasury's accountView is a
ledger account ({address, balanceCents}) and apps/venue's is a linked cloud
provider account ({provider, label, externalId, clusters, …}). openapi.Weave
refuses one name with two shapes, because every generated SDK would bind
whichever it read last — and neither collision existed while either route was
untyped, since an untyped route contributes no schema at all. Failure mode #5,
landed by two concurrent passes rather than by one.
venue yields, and qualifying it is the better name anyway: the product is
"connect a cloud account", so cloudAccountView / cloudAccountsView say which
kind of account this is. No wire movement — the json tags and every response
byte are unchanged, only the Go type name and therefore the published schema
name.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The work list was the ARTIFACT, not a grep: every operation in
plugin/{usage,venue,world,agent,bot,help}/openapi.json carrying neither a
description nor a summary — exactly the set that projects to nothing at all. No
prose, no MCP tool, no CLI command, no typed SDK method. It was 27 of 27.
19 are typed ops now: usage 5/5, venue 5/5, help 4/4, world 4 of 5, bot 1 of 4.
Every one carries a doc comment that is TRUE of its handler, and every published
schema property carries its own — 0 bare properties across all five subsets.
Eight stay untyped and each names its wire fact AT its registration, gated by
untypedByDesign + TestEveryRouteIsTypedOrNamed so the two ledgers must sum to
what the live router serves:
GET /v1/world/stream Server-Sent Events; no Out expresses a stream.
GET /v1/bot/connect a WebSocket upgrade; 101 then duplex frames.
POST /v1/bot/nodes/{id}/invoke a 403 carrying a DOMAIN body a client switches
on, plus the caller's X-Device-Id, which no In
field may carry.
POST /v1/bot/peer/invoke a net/http machine hop with text/plain refusals
and a MaxBytesReader cap.
the four /v1/agent ops registered by github.com/hanzoai/agent v0.1.3
(agent.go:166-169), not by cloud. They become
typeable upstream, which additionally needs a
per-request bridge there; POST /v1/agent also
relays an upstream 4xx's status AND body.
Two latent defects found by typing and fixed:
plugin/venue/main.go declared no Prefixes, so the standalone binary's scope
owned only the /v1/<name> default — and venue is named "venue" and serves
/v1/cloud, so it owned NOTHING it registers. cloud.Declare attributed every
route to no subsystem and scope.Use installed the app's middleware where no
route lives, which is load-bearing now that cloud.Bridge parks the org a typed
op reads. The apps/plan defect, one app over.
apps/bot had no Makefile, so `make -C apps/bot openapi` could not run and that
subset could never be regenerated by the per-app chain — despite mk/plugin.mk
claiming an app cannot have a main and no Makefile. catalog, crawl, meet and
zen are still missing theirs.
One delta, MEASURED rather than glossed and not fixable in cloud: encoding/json
validates the whole document before invoking any custom Unmarshaler, so the
record-it-and-judge-it-later input that preserves gate order for an oversized or
wrong-shaped body cannot preserve it for bytes that are not JSON at all — zip
refuses those first. TestSyntacticallyInvalidJSONIs400Early (help) and
TestSyntaxErrorIs400BeforeTheAdminGate (venue) pin exactly what moved.
Wire preserved otherwise, and pinned: the 413-after-404-after-503 intake order,
the body-tolerant sync, the ?project cross-check that a body cannot become, the
no-store header on the two money reads, the one-or-many usage report, and the
tenant that comes from the validated principal in every case.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The lifecycle is signup → qualified → credited (was signed_up). Store
migration rewrites existing rows idempotently; the API counts field is
signup (was signedUp); zipdoc regenerated; the last signed_up example in
the bridge header goes with it. Console alignment ships beside this.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
knowledge, do, flags, research, treasury and storage published 49 operations
between them and ZERO descriptions: no prose, no MCP tool, no CLI command, no
typed SDK method. 39 are now typed ops — one registry entry that is at once the
REST route, the OpenAPI operation with its schemas, the tool and the command.
do 0 -> 8 of 8 treasury 0 -> 8 of 8
flags 0 -> 8 of 8 knowledge 0 -> 8 of 9
research 0 -> 7 of 8 storage 0 -> 0 of 8
The ten refusals are wires this stack cannot yet describe, each recorded at its
own registration so the next engineer re-checks the blocker instead of
re-deriving it:
- storage's seven data-plane ops + /health. A refused balance answers through
cloud.DenyResource, which writes the fleet's NESTED
{"error":{"code","message"}} 402/503 IN BAND; a typed op's only refusal
channel is a returned error, which zip renders flat. The same refusal
apps/ml and apps/company already file. /health answers ONE object under TWO
statuses, which a single WithStatus cannot say. Two of the seven are refused
twice over: fiber's `*` has no typed-op spelling — zip leaves it in the op
path while openapi.translate renders the route as {wildcard1}, so Fold would
fail with "typed op has no live route".
- POST /v1/kb/import takes an UPLOAD (a vault zip, an .enex, a JSON export).
zip decodes a typed body as JSON before the handler runs.
- GET /v1/research/artifacts/:sha256 streams raw bytes under the artifact's
own Content-Type. A typed op serialises a Go value as JSON.
Defects the typing surfaced, all fixed here:
- flags.Store.Upsert PANICKED on a body of `null`: it unmarshals into a NIL
map without error and the next line assigned into it. Any caller could send
it. Now refused like every other non-object. Pinned.
- apps/treasury registered a typed op with NO //go:generate zipdoc directive,
and its handler was a closure — a function literal has no doc comment, so
/treasury/reserve published an empty description on every projection. Named
and documented.
- SIX schema-name collisions the weave caught, because the published schema
namespace is FLAT (zip keys on the bare Go type name): research's `Totals`
against admin's, knowledge's authorizeOut/connectorsOut/syncOut against
integrations' and admin's, and the treasury ledger's `Entry`, `Report` and
`Policy` against catalog, admin and gateway. Renamed to JournalEntry,
TreasuryReport, SharePolicy, ResearchTotals and kb*-prefixed. Go identifiers
only — every json tag is untouched, so no wire moved.
- the six apps had no cloud.Bridge of their own, so their tests mount on a
bare app where Serve never runs. Installed per DECLARED prefix.
PUT /v1/flags/defs/:key is the interesting conversion: its body IS an open
PostHog document, stored verbatim. An ordinary struct In would drop every field
it does not name — silent data loss behind a green 200 — so the In states its
own wire form and zip publishes "any JSON" instead of a fabricated object. The
path still wins over the body's own key. Four tests pin it.
Every wire is preserved: 201 and 204 declared on the op (a DEFINED empty Out
publishes "200 with a body" about a 204 — apps/git and apps/tools still do),
query-vs-body binding unchanged, the connector rows' *string keeps
present-but-empty distinct from absent.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The six subsets — notify, product, referrals, validators, zero-trust,
blueprint — were at described=0. Every operation carried neither a
description nor a summary, which is exactly the set that projects to
NOTHING: no prose, no MCP tool, no CLI command, no typed SDK method.
Nineteen are typed ops now (notify 1/4, product 4/4, referrals 4/4,
validators 4/4, zero-trust 4/4, blueprint 2/3), each with a doc comment
that says what the route does, per-field prose on every published
property, and a gate in the package's own typed_wire_test.go whose two
ledgers must sum to what the live router serves.
The four refusals are ONE class: two 200 shapes at one address.
notify's three /send routes answer with a bare SendResponse for a single
recipient and {items:[…]} for several; blueprint's /sbom answers a bare
Estimate for ?template= and {data:[…]} for none. An op declares one Out,
so either shape would publish the other as a lie — worse than none,
because every generated SDK binds it. TestSendAnswersTwoShapes measures
that pair rather than asserting it, so the conversion is a test away the
day zip can declare a polymorphic response.
The wire did not move. A refusal that used to precede c.Bind still
precedes the decode, because a typed op runs after it: requireOrgOnWrite
/ requireAdmin are method-scoped gates on the subsystem's own group, so
an anonymous caller with a malformed body is still 403 and the sibling
/health probes stay open. Every body-only field on a converted POST
carries url:"-", because zip binds query OVER the body and would
otherwise mint a higher-authority ?field= twin no route ever read. A
query scalar whose existing parse trims stays a STRING, measured rather
than assumed: fiber percent-decodes a query value but not a path
segment, so an int field would have narrowed ?limit= and ?tokenId=.
Every body that used to marshal a map[string]any is a struct whose
fields are declared in the map's sorted key order, and the byte order is
pinned.
Four latent defects, all surfaced BY typing:
- apps/zt had no cloud.Bridge on any of its three prefixes. Its own
harness mounts on a bare app with no Serve, so every typed op there
would have seen no org and refused a valid request.
- plugin/{product,zero-trust,referrals}/main.go declared no Prefixes,
so MountPrefixes' /v1/<Name> default covered nothing product or
zero-trust serves and half of what referrals does — the apps/plan
defect, three more times. All three take manifest.PrefixesFor now.
- 38 published properties (26 referrals, 12 zero-trust) were about to
reach openapi.yaml, every SDK and every MCP inputSchema bare.
- zipdoc cannot resolve zip.App.With as a router, so the obvious fix
for the decode-order problem fails generation. Recorded in LLM.md.
Regenerated: six subsets and the woven openapi.yaml (1013 paths,
unchanged — the diff is prose plus the two _by_id -> _id operationId
renames a route always takes when it goes typed).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
One resource, three spellings, and a create that did not live at its own read:
POST /v1/machines/{id}/bind-agent a VERB in the path
GET /v1/machines/{id}/agent-binding SINGULAR
DELETE /v1/machines/{id}/agent-binding
GET /v1/agent-bindings PLURAL, and top-level with no service
HIP-0128 §1 forbids each one: the resource is a plural noun, never a verb, and the
method carries the verb. A caller had to learn that binding an agent happens at a
different address than reading the binding it just made, and every projection —
OpenAPI, the MCP tool list, the CLI, four SDKs — published three names for one
thing.
A machine hosts at most one agent, so it is a to-one sub-resource:
GET /v1/machines/agents every machine's agent in the org
PUT /v1/machines/{id}/agent bind
GET /v1/machines/{id}/agent read
DELETE /v1/machines/{id}/agent unbind
PUT rather than POST because binding is idempotent — re-binding the same agent to
the same machine is the state the caller asked for, not a second binding. Singular
at the member and plural at the collection is not the old inconsistency: it is the
ordinary rule, applied to one resource instead of two.
Both literals register ahead of /v1/machines/:id so no machine id captures them —
the ordering /v1/machines/launch already relies on.
WHAT DID NOT MOVE, and the file now says so where it matters: the cl.call paths in
bots.go are VM'S wire, and the botVM test stub keeps vm's spelling because it
impersonates vm. Renaming those would call a route that does not exist. Two wires,
one translation, stated once so nobody fixes the wrong side.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
bots, entitlements, sbom, translate, agentskills and gateway published 16
operations between them and 16 of 16 carried neither a description nor a
summary — the exact set that reaches no prose, no MCP tool, no CLI command and
no typed SDK method. Eleven are typed ops now; the five that are not each name
the wire fact that keeps them raw, in a CLOSED list a test reads.
entitlements 3/3 gateway 2/2 bots 2/3
sbom 2/3 translate 2/3 agentskills 0/2 (structural)
The five refusals, each re-measured against the PINNED zip v1.18.11 rather
than inherited as prose:
POST /v1/bots/run answers 501 unconditionally. A typed op publishes
a SUCCESS response it can never send, and mints an
MCP tool and CLI command for an operation that
cannot succeed. It is also body-tolerant.
GET /v1/sbom/{wildcard1} a greedy wildcard: fiber binds it as `*1`, the
document publishes `{wildcard1}` as a PATH param,
and a typed op publishes op.Path verbatim — so the
address, the parameter name and the parameter
LOCATION would all move for a wire that did not.
POST /v1/translate a bulk-tier spend denial answers 402/503 carrying
the fleet's NESTED {"error":{code,message}} domain
body; a typed op's only refusal is a returned
error, which zip renders flat.
GET /.well-known/agent-skills/index.json embedded bytes served verbatim
(its sha256 digests are computed over what is
served), a Cache-Control zip cannot set, and an
{"error":…} 404 body errorHandler does not produce.
GET /.well-known/agent-skills/{skill}/SKILL.md text/markdown. A typed op
answers application/json — no In/Out serves it.
WIRE PRESERVED. No path moved, no status moved, no field name moved; the only
published deltas are the three operationId renames typing always makes
(_by_id -> _id) and the prose that did not exist before. Two map[string]any
responses became structs whose fields are spelled in the order encoding/json
emits a map's sorted keys, so the BYTES did not move either —
TestHealthAndIngestKeepTheirBytes asserts that.
The org is never an input field. `url:"-"` (new in zip v1.18.11) lets the :org
segment bind from the URL while staying out of the published body and
invisible to the decoder, and the gate still compares it to the VALIDATED
principal — so it is an address the gate re-checks, never an assertion the gate
believes. Five identity seams needed more of the principal than the org
(SuperAdmin-ness, the user id a write is attributed to, the ?org= a SuperAdmin
targets a tenant with); each is one function, pinned in allowedRequestUses with
its reason, and each fails closed off the HTTP path.
Latent defects the typing surfaced, beyond the two prefix ones fixed above:
- NONE of the six installed cloud.Bridge. Serve installs it binary-wide so
nothing was live-broken, but none of these packages' own test harnesses
runs Serve, so a typed op added here would have 403'd in its own tests with
no hint why. All six install it through the SUBSYSTEM router now.
- apps/entitlements' tests RECONSTRUCTED the router by hand rather than
driving the mount, so they could have gone green against a surface the
binary does not serve. Mount's registration is one routes() function now,
and the tests call it.
- edge.Policy grouped three fields under one section comment, and zipdoc
lifts the comment directly above a field — so "Platform-scope (admin-org
row)" shipped as the published description of cors_origins ALONE. Every
published field in all six subsets carries its own prose now; the check is
in the pass notes in LLM.md.
- Two generic names were qualified before they could collide: bots.botView ->
BotRun (apps/visor already publishes botView for a bot MACHINE) and
translate.Entry -> MemoryEntry (six packages under apps/ declare an Entry).
Neither was published yet, so both were free.
Each package carries untypedByDesign + TestEveryRouteIsTypedOrNamed reading the
REAL mount, whose two ledgers must SUM to the served surface — so a route added
untyped here goes red, and a reason naming a route that is gone goes red too.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
MountPrefixes falls back to /v1/<name> when a plugin declares no Prefixes, and
for these two that default is wrong — the apps/plan defect, found twice more:
agentskills serves the ROOT discovery convention (/.well-known/agent-skills
/…), so /v1/agentskills covered NOTHING it registers. Measured:
SubsystemOf resolved every one of its requests to "" and PriceOf
to Undeclared, and any middleware it installed through its own
router landed on a prefix nothing was ever routed to.
entitlements owns TWO top-level nouns — /v1/entitlements (the commerce
projection) and /v1/orgs/:org/entitlements (the enablement
store). The default covered the FIRST, so half the surface was
unattributed and ungated. That partial shape is the harder one
to see: the app looks covered.
Both now pass manifest.PrefixesFor(<name>), so the list cannot drift from the
prefixes the host actually routes there.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The work list was the ARTIFACT, not a grep: every operation in
plugin/{graph,meet,prefs,settings,share,admission}/openapi.json carrying neither
description nor summary — exactly the set that projects to nothing, no prose, no
MCP tool, no CLI command, no typed SDK method. It was 11 operations, all of them.
8 are now typed ops with true doc comments; 3 are refused, each with its reason
recorded AT its registration and MEASURED by a test rather than asserted.
Typed (described-count 0 -> 8 across the five subsets):
GET /v1/indexers graph GET /v1/settings/{product} settings
GET /v1/oracles graph PUT /v1/settings/{product} settings
GET /v1/prefs prefs GET /v1/share share
GET /v1/flags/waitlist admission POST /v1/share/enable share
Refused, and each is "zip cannot state this":
POST /v1/meet/getToken answers the raw join token as text/plain (the office
client reads it with res.text()); a typed op always marshals JSON.
GET /v1/meet/health answers 200 or 503 with the SAME body, `ready` being
the whole dashboard fact at both — the multi-status gap (#78). A typed op's
only refusal is a returned error, rendered as zip's flat {status,code,error},
so ready:false would vanish from the degraded answer.
PATCH /v1/prefs three facts at once: a 16 KiB REQUEST-BYTE cap that
answers 413 and a typed op cannot see; an empty body and a literal `null`
body each answering 400 where op.invoke skips the decode and null decodes to
a nil map; and an OPEN key space whose only carrier is map[string]any, whose
typeName is "" so hasRequestBody publishes no request body at all.
WIRE PRESERVED. Statuses, JSON shapes, field names and error precedence are
unchanged; apps/{graph,share,prefs,admission,meet} carry tests that drive the
REAL router (routes(), which Mount calls) rather than a reconstruction.
Latent defects found by typing:
1. Five apps have plugin/<app>/main.go and NO apps/<app>/Makefile, and
mk/fleet.mk reads APPDIRS := $(wildcard apps/*/Makefile) — so openapi-check,
the gate that regenerates the document from source, has NEVER regenerated
plugin/{meet,bot,catalog,crawl,zen}/openapi.json. Five published subsets sit
outside the only gate that catches the ingress-class loss. apps/meet/Makefile
is added (its subset regenerated clean); the other four are reported.
plugin/gen-app-cmds writes no Makefile at all, so each Makefile's own header
claim that an app "cannot have a main and no Makefile" is false.
2. Failure mode #9 (the empty leaf) was live twice more: prefs and share each
declared their root as g.Get("", ...), publishing /v1/prefs/ and /v1/share/ —
paths this API has never served. Declared on the app at the whole path now;
the operationIds are unchanged (get_v1_share either way), the untyped PATCH
sibling moved with it so one resource stays one document key, and fiber's
non-strict routing keeps both URL forms answering (pinned by test).
3. Neither graph's Authorization forwarding nor admission's ?host= default had
ANY test. Both degrade silently — a 200 with an anonymous upstream read; a 200
with known:false for every guard that omits the query. Both are pinned now.
4. apps/share had no route-level test at all (its fakeController was unused);
apps/prefs had none either. Both have one now.
cloud.Request gains three entries, each a request FACT and not a tenant: graph
forwards the caller's Authorization to the indexer/graph when no service token is
configured, prefs' isolation key is the qualified <owner>/<name> rather than the
org, admission's ?host= default is the request's own Host. All fail closed off
the HTTP path.
Known, reported, not worked around: settingsReq.config is map[string]any and
publishes additionalProperties:{"type":"object"} — LLM.md failure mode #8. Every
alternative MOVES THE WIRE (json.RawMessage turns {"config":null} from "store {}"
into "store null"; map[string]json.RawMessage changes number formatting and
large-int precision), so the wire wins and the fix belongs in zip's schemaOf.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zipdoc lifts each typed handler's doc comment and its In/Out field comments
into zipdoc_gen.go, which is compiled INTO the binary — Go drops comments at
compile time, so this file is the only path from the code to the published
description and the MCP tool. plugin/deploy/openapi.json is the app's own
subset, projected from the app's own live router by its own binary; openapi.yaml
is the weave of every app's subset.
Measured on the subset: 21 operations, described 0 -> 11.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Eleven of /v1/deploy's twenty-one published operations projected to NOTHING:
no schema, no prose, no MCP tool, no CLI command, no SDK method. They are
typed ops now, each one In/Out with a doc comment that IS the published
description.
The thing that made it possible is a decomplection, not a rewrite. refuse(c)
braided two facts into one function only a handler holding the request could
call: the DECISION (this caller may not have this) and the PRESENTATION (a
browser navigation goes to sign-in, an API call keeps its 403). A typed op
holds no request, so braided they forced a choice between losing the bounce
for every browser and gating in middleware — which the MCP, CLI and call
projections never run, i.e. a hole in three of the four. Split, the decision
is forbidden() wherever it is made and the shape is one middleware every
/v1/deploy route passes through. The wire is exactly what it was, both arms
pinned by TestRefusalIsA403AndANavigationIsBounced.
The scope itself stays where it was: resolveScope, reached through
cloud.Request in ONE file (typed.go), because a SuperAdmin's scope has no org
at all and a tenant's is the SanitizeOrg slug — neither is what
principal.OrgFrom carries. Never an In field.
TEN operations are deliberately NOT typed, each for a measured wire fact
recorded at its registration and asserted in typed_wire_test.go:
- account/can-i/* a fiber wildcard; zip's closeColonParams leaves '*' alone
while cloud's translate emits {wildcard1}, and Fold then
refuses the whole document.
- health answers 503 carrying the same domain body as its 200.
- login, callback succeed with a 302 + Set-Cookie.
- the two streams answer an unbounded text/event-stream.
- the four POSTs zip decodes the body before the handler and 400s an
unparseable one; these read no body, so typing them turns
today's 200/403/503 into a 400 — and for the three gated
ones puts that 400 ahead of the 403.
Bytes are preserved, not merely equal JSON: every model that replaced a
map[string]any declares its fields in the order encoding/json sorted the map,
and sessionUser carries *[]string so `groups: []` can be present for a
signed-in caller and ABSENT for an anonymous one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three spellings of one concept had drifted apart, and one of them was a
live bug: campaign conversions counted event = 'signup', which NOTHING
emits — the @hanzo/event grammar is <object>_<verb-past> and the signup
funnel's terminal event is signup_completed — so signup conversions
always read zero. The query now counts the event that exists.
The destinations-internal normalized token drops its underscore
(StandardEvent 'signup'; it is a map key that never leaves the process),
while adapters keep rendering each platform's own required name — GA4
'sign_up', Meta 'CompleteRegistration'. The bus bridge's examples speak
the real vocabulary instead of an invented 'Signed Up'.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every operation in these five plugin subsets carried neither description nor
summary, which is exactly the set that projects to NOTHING: no prose, no MCP
tool, no CLI command, no typed SDK method.
TYPED (3):
GET /v1/audit org-scoped audit trail
GET /v1/catalog the cross-org discovery lens
GET /v1/x402/settlements/{id} one payment receipt
REFUSED (2), each named at its registration and MEASURED by a test:
POST /v1/ask one route, two success shapes; an SSE branch; a
cloud.DenyResource 402/503 carrying the fleet's NESTED error
POST /v1/crawl body-TOLERANT with a domain refusal body; a 1 MiB
io.LimitReader bound a typed op cannot see
Neither refusal publishes nothing now: both declare their request through
openapi.Register (crawl its response too; ask's is polymorphic, so a single
declared shape would be false and the silence is pinned).
WIRE PRESERVED. Paging and tri-state filters stay STRINGS because zip's bindURL
zeroes an unparseable value and reads a bare ?flag as true — audit matches its
admin twin, which already publishes pageSize/p as strings. audit's envelope
fields are declared ALPHABETICALLY because the map they replace was sorted by
encoding/json, and the tail bytes are asserted.
LATENT DEFECTS FOUND BY TYPING:
- apps/crawl registered the same route twice (Group + empty leaf and "/") and
published /v1/crawl/ — a trailing slash in the document, the operation id, the
MCP tool and every SDK's URL. One registration at /v1/crawl now; both spellings
still answer.
- none of the five installed cloud.Bridge of its own; each converted app does now.
- apps/ask's AskRequest/AskResponse are the two names apps/books ALREADY
publishes with different shapes — ask yielded to askRequest/askAnswer.
- five apps have a plugin main and no apps/<app>/Makefile, so openapi-check
cannot see their subsets. catalog and crawl are fixed here; bot, meet and zen
are still uncovered and the class fix is in plugin/gen-app-cmds.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The measured gap for this unit was 13 operations; six became typed reads one
commit ago, and what is left is refusals, not backlog. Main moved underneath
this pass — /v1/event/collect folded away, the two Sentry doors arrived — so
the ledger reads 6 typed / 8 refused, and every one of the eight was re-derived
against zip v1.18.11's own typed.go rather than inherited as prose:
GET /v1/analytics/health answers 503 CARRYING the degraded report; zip
stamps a non-nil Out 200 and WithStatus refuses
a non-2xx.
POST /v1/event polymorphic wire (Event | [Event] | {batch:[…]});
POST /v1/analytics op.invoke jsonenc.Unmarshals every non-empty body
POST /v1/analytics/batch into the In, so an array body 400s where it
POST /v1/tracker answers 200 today.
POST /v1/insights/e an object — but admission still isn't reachable.
POST /v1/event/{project}/envelope a raw Sentry envelope stream, and a DSN
POST /v1/event/{project}/store key the o11y consumer verifies itself.
Each also gained a blocker the first pass had not written down: ORDER. invoke
decodes the body BEFORE the handler is entered, while the anonymous lane
refuses 403 (capture disabled), then 429 (rate), then 413 (64 KiB) with the raw
bytes in hand and nothing parsed. Typing any door would answer 400 to a beacon
that is answered 413 or 429 today, and error precedence is wire.
What DID change is the other half of the cost. All eight published an
operationId and nothing else — indistinguishable, to every SDK generator
reading the document, from a route that takes no body and returns none. So
`post_v1_event`, the door every Hanzo product beacons to, shipped in every
generated SDK as a call with nowhere to put the event.
They declare their bodies now, through openapi.Register, driven off the doors
table itself so a door cannot be routed with one wire and documented with
another — the drift that once put /v1/tracker in the router and not in the
site-host carve. The gate quantifies over untypedByDesign, not over doors, so a
refusal added tomorrow owes its bodies by construction rather than by someone
remembering.
openapi gains `OneOf`, for the same reason it gained `Binary`: the canonical
wire is genuinely three shapes, and naming one of them would publish an ingest
API that cannot batch. The Sentry pair takes Binary for its request and
declares NO response, named in `relayed` — it copies back whatever
cloud.ObsErrorIngest installed, and publishing a shape there would be inventing
one. And the health probe's map[string]any became healthReport, because a map's
shape cannot be declared without hand-writing a schema beside it, which is the
drift Register exists to prevent.
The subset: 0 -> 7 requestBody, 6 -> 12 responses, 19 -> 30 schemas. The
described count did NOT move — still 6, exactly the typed ops — because prose,
an MCP tool and a CLI command are the three things only zip's registry
supplies. Reported as unchanged rather than counted as progress.
The cost is named and gated, not absorbed: Register cannot lift field prose (Go
drops comments; zipdoc walks typed registrations only), so eleven components
publish 63 bare properties. `proseless` is that ledger and it may only SHRINK —
it caught its own first staleness during this rebase, when main's removal of
the Team door left teamEvent named and unpublished. The fleet ships 1,627 such
properties for the same reason; that measurement is in LLM.md now.
Wire preserved exactly. No status, body, field name or error precedence moves;
the health report's serialization ORDER changes (struct order, not sorted map
keys), which the JSON data model does not carry, and
TestHealthReportKeepsTheMapItReplaced pins both bodies field-for-field.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
content, leaderboard, marketplace, prompts and sync published 30 operations
between them and not one carried a description or a summary. An operation that
publishes neither projects to NOTHING — no prose in the document, no MCP tool,
no CLI command, no typed SDK method — so the whole of five products was
addressable and unexplained. 29 are typed ops now; the described count in the
regenerated subsets goes 0/38 to 29/38.
The wire is unchanged, and that was the constraint rather than an afterthought:
- every status is preserved where zip's default is not it — marketplace
listings and prompts create keep 201 via zip.WithStatus, sync run keeps 202,
every delete keeps its 204 by returning a nil Out through an ALIAS for the
unnamed empty struct (a defined type there would publish "200 with a body");
- the query string still binds. `?limit=abc` reaches the board and the
leaderboard as the default page, not as a 400: zip's binder leaves an int at
zero for a value it cannot parse, and both handlers still read non-positive
as "take the default". TestBoardLimitTolerance pins both halves;
- backfill's `force` stays a STRING and is still compared literally. A bool
field would have bound through ParseBool and quietly widened the guard to
"1", "t", "T", "TRUE" and a bare `?force` — on a rollup that ACCUMULATES,
so a run that should have been refused doubles every day it re-reads.
TestBackfillForceIsLiteralTrue walks every near-miss spelling to the 409;
- sync's PATCH keeps pointer fields, and TestPatchSync_NullIsAbsent pins WHY
that is safe here: encoding/json leaves a pointer nil for an explicit null
as well as for an absent key, which silently turns a "clear" into a no-op on
a route that distinguishes them. None of these fields is clearable, so the
two have always meant the same thing — and the test is where a future
clearable field shows up.
The tenant is never an In field. It comes from principal.OrgFrom(ctx), which
cloud.Bridge parks on the context; an In field is caller-supplied, so a tenant
key read from one is a cross-tenant read the caller asserted for itself. Each
subsystem installs Bridge on its own mount as well, so it still resolves its
tenant when a test or a non-Serve composition root mounts it on a bare app
instead of refusing every caller.
Two subsystems reach the request through cloud.Request, and both are registered
in allowedRequestUses with the reason: a public leaderboard is a CONSENT surface
gated on the validated username and on org/platform admin-ness, and a
marketplace install is scoped to (org, PROJECT) and attributed to a user — none
of which principal.OrgFrom carries.
NINE operations are deliberately NOT typed, each for a reason that is a wire
fact rather than a preference:
- websearch's eight. /v1/websearch/search is registered with All and answers
seven methods including OPTIONS and TRACE; zip has no typed All, so
declaring the named verbs instead would DROP two of them — a routing change.
Its POST/PUT/PATCH arms also read the query and IGNORE the body, while a
typed op 400s on any unparseable non-empty body. /v1/scrape deliberately
answers 200 {"success":false,...} to a malformed or oversized body because
firecrawl clients read data.success rather than the status line, and it caps
the read at 1 MiB with an io.LimitReader rather than refusing; a typed op
can express neither. The package says all of this where the next sweep will
read it.
- POST /v1/content/generate. A studio-render billing denial answers the
platform resource-deny envelope, {"error":{"code","message"}} at 402/503
(cloud.DenyResource, resource_billing.go:224). A typed op can answer its Out
schema or zip's {status,code,error} and nothing else, so typing it would
rewrite that body for every client reading error.code.
Full suite green with the gate env (174 ok, 0 fail); each package regenerated
with `go generate -run zipdoc`, each subset reprojected from its own binary, and
the fleet golden rewoven.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The platform already had both halves and no middle: the ONE door
(POST /v1/event) commits to the warehouse and fans out through the
analytics sink seam; the ONE bus (embedded Hanzo PubSub) carries
commerce.> events to the webhooks delivery engine. This joins them:
- forward.go's seam is now ONE seam, N consumers: AddSink (with remover)
replaces the single SetSink slot; destinations and the new bridge both
register, each dispatched detached and panic-guarded.
- apps/webhooks/bridge.go publishes every accepted batch onto stream
EVENTS as event.<kind> (canonical names fold to NATS-safe tokens:
$pageview -> event.pageview, 'Signed Up' -> event.signed_up; bounded,
wildcard-proof) carrying THE standard Envelope — organization_id
first, because the delivery engine resolves the tenant from the
envelope and an org-less event is delivered to nobody.
- The SAME dispatcher consumes EVENTS beside COMMERCE, so one Endpoint
row subscribes an org to anything on the platform: commerce.order.*,
event.error, event.signed_up — one bus, one envelope grammar, one
delivery engine, one signature scheme.
Fail-soft by construction: bus down = warehouse still durable, nothing
blocks ingest. TestSpineEndToEnd drives seam -> bus -> signed webhook
delivery wholly in-process over the embedded pubsub.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ff188fab ("event: no /api/, no /collect — the door is the whole surface")
regenerated plugin/analytics/openapi.json and did not re-weave openapi.yaml, so
the fleet golden on main is stale against the subsets it is woven from. This is
failure mode #1 in LLM.md, and the reason that mode is written down is that a
gate comparing two DERIVED artifacts agrees with itself while both are wrong.
Wrong in BOTH directions, which is what makes it worth a commit of its own:
- it PUBLISHED 8 operations nobody serves — /v1/event/api/{wildcard1} across
all seven methods, plus POST /v1/event/collect — in openapi.yaml, and
therefore in every generated SDK and in the MCP tool list.
- it OMITTED 2 operations that ARE served: POST /v1/event/{project}/envelope
and POST /v1/event/{project}/store, registered at
apps/analytics/analytics.go:237-238 and present in the committed subset. The
Sentry-compat ingest pair was unreachable from any generated client.
Not hand-merged. Regenerated: the weave is deterministic from the committed
subsets, so this is `make -f mk/fleet.mk openapi-weave OUT=openapi.yaml` and
nothing else. The only content in the diff is those ten operations.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both subsystems are one All("/…/*") registration that the untyped projection
explodes into seven undescribed operations — so each publishes no prose, no MCP
tool and no CLI command for its whole surface. That refusal is correct and it
was written at the registration, which is a promise: nobody can re-check it, so
a route added later inherits the exemption silently and a reason that stops
being true keeps being believed.
untypedByDesign + TestEveryRouteIsTypedOrNamed read the LIVE router of the real
Mount and close both halves: a route here is typed BY DEFAULT, and a reason
naming something the package no longer serves goes red. Verified by mutation —
drop one entry and the gate names the operation.
The reason is re-derived from zip v1.18.11 source rather than inherited: a typed
op's only response path is c.JSON(out) under its DECLARED status (typed.go:
302-311), which a verbatim relay of the upstream's status and Content-Type
cannot survive; there is no All[In, Out]; and a greedy wildcard's value is a
whole sub-path, not a scalar bindURL/setScalar can set on an In field. Each
independently sufficient.
ai and licensing stay ungated on purpose — their registrations live in
github.com/hanzoai/{ai,licensing}, so there is no cloud-side route for a
cloud-side gate to hold.
LLM.md records the re-derivation and the thing the op-level count cannot see:
the same field check run over the committed subsets puts the class at 1,601
bare published properties across 17 packages, several of them packages this
migration already marks done.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Typing a route documents its ADDRESS and its SHAPE. It does not document the
shape's FIELDS — those come from doc comments on the In/Out structs, which
zipdoc lifts per field — so the six-plugin pass left 26 published properties
bare: every property of DestinationStatus (the card all five destinations
routes answer with) and of DestinationField, eleven of StarterKit, and
Variant.source. They reached openapi.yaml, every generated SDK and every MCP
inputSchema with no description.
The split says where the hole comes from: publishKitIn and replaceKitIn, both
written AT the conversion, describe every field; StarterKit, which predates it,
described four of fourteen. The request side got documented and the response
side did not.
The two that mattered most:
- connected / enabled / live are three DIFFERENT facts — configured here once,
forwarding now, and a credential still resolves (KMS secret, else the
integrations fallback token, else none needed). Connected && !Live is
exactly the reconnect-me state, and nothing published said so.
- tier and rating are public-catalog curation. No request can set them:
neither write body has the field and neither kit() carries one, so they are
absent on every customer-published kit. A caller could see two numbers and
nowhere that they are server-owned.
Comment-only in the Go source — no field renamed, retyped, reordered or
retagged — and the regenerated artifacts are 79 lines added, 0 removed.
TestEveryPublishedFieldIsDescribed gates both packages the way
TestEveryTypedOpIsDescribed gates the op side, mutation-checked: blank one
description and it names it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
{status, msg, data, data2} is Casdoor's response type. Its second slot was
untyped, so the row count went there, and the name came along whole — through the
Casdoor Go SDK (hanzoai/iam auth.go still declares Status/Msg/Data/Data2) and
through Casdoor's console, whose getList<T> reads data?.data2 ?? rows.length.
cloud's own envelope.go then called it "the canonical /v1 envelope", which it was
only in the sense that everything had inherited the same shape.
Nothing specifies it. HIP-0111 names it directly as the shape a list MUST NOT
return ("Lists return the SCIM ListResponse envelope (totalResults/Resources),
not a {status,data,data2} one"). Casdoor is dead by standing rule. So the field is
`total`, which is what it holds — same int, same position, a name a reader can act
on. 142 sites, 30 files, plus the doc-comment examples zipdoc lifts into the
published spec.
ONE EXCEPTION, and it is the point of the change rather than an escape from it:
apps/admin/iam/iam.go DECODES hanzoai/iam's answer, and IAM still writes data2.
That struct now reads `Total json.RawMessage \`json:"data2"\`` — the Go side speaks
cloud's language, the tag records the foreign wire, and a comment says when it
converges. A blind rename there would have read nothing: the total silently
becomes zero and every paginated admin list reports its own page size. The same
trap was live in the tests, whose fake IAM server writes data2 on purpose — four
tests caught it, which is why they exist.
The console (hanzoai/admin b030f80) ships the reader in the same window and
accepts total → data2 → rows.length, so either side can deploy first.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/ is the only prefix this platform speaks. The /v1/event/api leaf from
the previous commit is gone; the Sentry wire is POST
/v1/event/{project}/envelope|store, carried by the door's owner (the
project segment is variable, so no static prefix could route it) and
forwarded to the obs plane's installed consumer (cloud.SetObsErrorIngest,
the second seam beside the batch claim), which maps onto the clean
/v1/sentry runtime ingest routes before the principal gate looks — the
existing ingest exemption stays the only one.
/v1/event/collect is DELETED, not sunset: the team wire rides the
canonical door by shape (isTeamArray), so the path said nothing, and
sourceTeam goes with it — a team-shaped body on the door stamps the
door's own source. The POST ledger and the typed/named ledger both name
the envelope wire explicitly; openapi artifacts regenerated; zero
occurrences of the old spellings remain.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The tool plane's two untyped routes were correctly refused (zip cannot express a
body-tolerant op, and cannot answer a non-2xx carrying a domain body), but the
refusal was leaking a second, separable fact into the document: both rendered as
an operationId and a tag and NOTHING ELSE — no requestBody, no response — which
is EXACTLY what a route taking no input and returning none publishes. No consumer
can tell those apart, so every SDK generated off openapi.yaml offered an MCP call
with nowhere to put the JSON-RPC envelope and a plugin build with nowhere to put
the source.
Staying out of zip's registry costs prose, an MCP tool, a CLI command and a typed
SDK method. It must not also cost the SHAPE. openapi.Register (tools.go init) now
states the halves that ARE statable — mcpRequest→mcpResponse and
buildRequest→buildOut — the same seam apps/books and apps/company use, attached
to routes the router already carries so it can never contradict the router.
Two map literals became named structs to make that possible, with ALPHABETICAL
fields because encoding/json writes a map in sorted key order. The pins assert the
marshalled BYTES, not a status code: TestMCPEnvelopeIsByteIdentical over all four
envelope shapes (result, result with a null id, error, parse-error with no id to
echo) and TestBuildReceiptIsByteIdentical. Naming a shape did not move it.
The refusals themselves were re-derived from zip v1.18.11 SOURCE rather than
inherited as prose, and three of their file:line citations did not land:
op.invoke's unconditional ErrBadRequest on an unparseable body is typed.go:242
(cited 243), and the cmp.Or(op.Status, 204) a nil Out stamps is typed.go:305
(cited 308). A citation nobody can land on is how a refusal stops being
re-checkable — which is the whole reason the reason is written down. Both
refusals stand, unchanged, now verifiable.
Described ops stay 14 of 16, deliberately: Register buys the shape, never the
prose, and a description invented for a route nobody typed is worse than none.
17 of the subset's 69 published properties are now bare for the same reason —
zipdoc lifts field comments off TYPED ops only. Recorded, not glossed.
openapi.yaml also picks up a drift that was ALREADY on main and is not mine:
02827405 regenerated plugin/o11y/openapi.json (/v1/event/error → /v1/event/api)
without regenerating the fleet golden, so the published document advertised five
operations under /v1/event/error/{wildcard1} that the router no longer serves and
omitted the five /v1/event/api/{wildcard1} ones it does. The drift gate was red on
main. This golden is regenerated from source, never hand-merged, so the repair is
the generator's output rather than an edit to a neighbour's app.
Gates: apps/tools green under the Makefile env (baseline green, still green),
zipdoc -check clean, openapi-weave clean, plugin/tools/openapi.json and
openapi.yaml regenerated from source.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The Tasks data surface admitted any request carrying X-User-Id. cloud's
identity boundary mints exactly that with NO X-Org-Id whenever a validated
token's homeOrg() is empty (auth_identity.go — no `orgs` claim: a
pre-IAM-v1.33.0 human JWT, a non-KMS machine token), on purpose, so that every
org() gate refuses it rather than guessing a tenant.
apps/tasks was the one that did not. An empty org is the ZERO Principal to the
engine, i.e. the shared UNSCOPED store (hanzoai/tasks store/principal.go) — so
an org-less caller registered a namespace and a DIFFERENT org-less caller
listed it back. One store, every principal cloud could not resolve an org for.
The fix is not a better local check, it is the removal of a local check:
principal.OrgOf is the ONE decision the cloud data plane makes about a request
and an org, and it takes plain strings precisely so a reader holding headers
rather than a *zip.Ctx asks the same function. Refusal status, content type and
body are byte-identical — only the admitted set narrows to what every sibling
subsystem already admits.
No route moved: plugin/tasks/openapi.json regenerates identical (28 ops,
0 described — the wire refusals recorded at the mount still stand, and
typed_wire_test.go still measures all four).
Also: two file:line citations in the refusal record pointed at lines that had
already moved. Replaced with the symbol, which does not drift.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The team SPA's wire is identified by its own keys (snake_case distinct_id,
numeric epoch-millis timestamp — the canonical wire spells distinctId/time),
so the ONE canonical decode now dispatches it by shape (isTeamArray,
positive-signal only: a canonical array can never be mis-read as team).
/v1/event/collect stops being a second wire and joins the sunsetting
caller-owned aliases — the published Team SPA appends /collect to its
collector URL, so the PATH lingers on the $source='team' sunset metric,
but it binds the same decode; a rebuilt SPA pointed at /v1/event needs
nothing else. The old TestCanonicalWireSilentlyDropsTeamBatch pinned the
exact failure this fold fixes — inverted into the proof the fold works.
The Sentry wire drops its /error segment: /v1/event/api/<project>/… is
what a real SDK produces from a DSN of …/v1/event/<project> (SDKs insert
the api segment), mapped onto the /v1/o11y/api ingest routes. plugin/o11y
openapi regenerated; manifest o11y leaf /v1/event/error -> /v1/event/api.
The whole surface is now: POST /v1/event (product | team | LLM-obs, by
shape) + /v1/event/api/… (Sentry DSN wire) + the sunsetting /collect.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The leaf lasted one commit. /ingestion said on the path what the payload
already says in its bytes, so the door now decides by shape: handle offers
every authenticated POST /v1/event body to the observability plane FIRST
(cloud.ObsEventIngest, installed by o11y's mount), which claims LLM-obs
ingestion batches — a batch whose EVERY element carries a recognised type;
a product CaptureBatch spells {"batch":[…]} too and has none, so ambiguity
always loses to the door's own wire — and declines everything else
untouched. One receipt shape (CaptureResult) either way.
Only the canonical door's FULL lane offers: obs events are tenant data,
so the anonymous and reduced projections never reach that plane, and the
other doors never consult the claim (all pinned by test). The o11y typed
op is gone with its route — the door's identity lives with the door.
manifest keeps /v1/event/error (the Sentry wire) as o11y's one deeper
leaf; analytics owns the root.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The work list was the ARTIFACT, not a grep: every operation in
plugin/{ai,destinations,dns,licensing,runtime,templates}/openapi.json
publishing neither description nor summary — 38 of 38, the set that projects
to nothing at all. No prose, no MCP tool, no CLI command, no typed SDK method.
Nine are now typed ops. templates goes 5 of 5, destinations 4 of 5, and both
are GATED rather than promised: untypedByDesign + TestEveryRouteIsTypedOrNamed
+ TestEveryTypedOpIsDescribed read the LIVE router of the real Mount, so the
next route added here is typed by default and a stale reason goes red.
The other 29 are FOUR registrations. ai, dns, licensing and runtime each
publish seven operations that are one All("/…/*") apiece, exploded across the
methods by the untyped projection: greedy wildcard (*1 to fiber, {wildcard1}
to the document — a bound field and a published parameter that cannot agree),
verbatim upstream status through c.Bytes(res.StatusCode, …), verbatim
Content-Type, and no All[In, Out] to hang seven ops on. Two are not cloud's to
type at all — ai is hanzoai/ai's beego tree behind zip.AdaptNetHTTP, licensing
is hanzoai/licensing's http.Handler behind the same. Each refusal is written
at its registration now, not only in LLM.md.
WIRE PRESERVED, and the two places it could have moved silently:
- url:"-" on every body-only field of both write ops. zip's binder fills an In
field from the QUERY as well as the body, so a converted POST starts taking
?slug= — which on publish redirects the write to a name the body never
asked for. c.Bind read the body and nothing else.
TestTheQueryStringCannotRedirectAWrite is the measurement.
- destinationTest carries POINTERS in alphabetical order. The route reports a
platform rejection as DATA at 200, so it answers {ok,error} and
{ok,sent,message}; a non-pointer with omitempty drops a real "sent": 0, one
without adds "sent": 0 to every failure, and the map it replaced marshalled
its keys sorted.
POST /v1/destinations/{platform} stays untyped and the reason is a wire fact,
not a preference: its body's property NAMES are chosen at request time by the
addressed platform's Spec, and toStr accepts each value as a string, a number
OR a bool precisely so a console may send a numeric pixel id — a typed string
field turns today's accepted {"pixel_id": 123} into a 400. It declares both
bodies through openapi.Register instead, so the document stops saying it takes
no body, which was the one thing it cannot work without.
Latent defects this surfaced:
- Neither package installed cloud.Bridge on its own prefix. Both worked only
because Serve installs one app-wide, and both packages' own tests mount on a
bare zip.App — so the moment an op became typed it read no org. Installed on
each subtree, ahead of the leaves, and pinned by
TestTheBridgeIsInstalledAheadOfTheLeaves.
- main was RED before this change: apps/agents/routing_http.go calls
cloud.Request and was never added to allowedRequestUses, so
TestRequestEscapeHatchIsPinned failed on a clean checkout. Pinned with its
real reason (the X-Target-Key claim header).
- destinations' collection root was declared as the group's EMPTY leaf, so the
document published /v1/destinations/ — a path this API has never served.
Declared on the parent with a non-empty leaf; the trailing slash is gone from
openapi.yaml.
- Typing is what makes a package enter the FLAT schema namespace, and both
collided on entry: Status is apps/plugins', Template is apps/guide's. The
published name keeps it, the unpublished one qualifies — DestinationStatus,
DestinationField, StarterKit. Go-level renames, no wire movement.
- plugin/ai publishes seven {wildcard1} operations and none of the ~200 real
AI routes, so chat completions reach no SDK and no MCP tool list. Named at
the mount; the fix belongs in hanzoai/ai.
zip v1.18.11 also retires failure mode #7 for us: hasRequestBody skips an input
whose every field is a path param, so POST …/test types with no phantom body.
described: destinations 0 -> 4 of 5, templates 0 -> 5 of 5.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Thirteen operations published NOTHING — no prose, no schema, no MCP tool, no CLI
command, no SDK method. Six of them were reads that only ever needed their tenant
and their query string, and they are typed ops now: /v1/analytics/{overview,
timeseries,top}, /v1/errors, /v1/insights/{events,health}. Their In/Out are real
structs with per-field prose, so `errorRate` is documented as a ratio and `pct` as
a share of the WINDOW rather than of the rows returned — 19 schemas, every property
described, gated by TestEveryPublishedFieldIsDescribed.
The wire did not move. bindURL fills the same three window fields and the same
limit off the same query string, and an unparseable value still leaves the field at
its zero — which is what strconv.Atoi's discarded error already did, so `?limit=abc`
is still one caller's typo about one field and not a 400. The 403/400/503
precedence is unchanged and now pinned for /v1/errors and /v1/insights/events,
which nothing read back before.
The other seven stay untyped and each says why AT ITS REGISTRATION, measured rather
than asserted. GET /v1/analytics/health answers 503 CARRYING the degraded report as
its body; zip stamps a non-nil Out with cmp.Or(op.Status, 200), WithStatus refuses a
non-2xx, and a nil Out is stamped 204 over anything written from inside — so the
status and the body are one answer a typed op cannot give. The six ingest doors
share ONE admission decision resolved from facts that never reach a typed op: the
presented credential, the client IP and socket peer the anonymous rate caps key on,
DNT/Sec-GPC, and the RAW body length that is the anonymous lane's 64 KiB -> 413
bound — invisible to a typed op and far below the fleet's global BodyLimit. Four of
them add a second blocker: the canonical wire is polymorphic (object | array |
{batch:[…]}) and the team SPA's is a bare array, bodies zip's op.invoke would 400
where they answer 200 today. TestArrayBodiedDoorsStillAnswer200 measures exactly
that, so the day zip can declare a polymorphic body this is a test away.
Two latent defects fell out of the conversion and are fixed here:
- plugin/analytics/main.go declared no Prefixes, so MountPrefixes fell back to
the /v1/<name> convention and five of this app's six prefixes were outside
anything it could gate — cloud.Declare attributed /v1/errors, /v1/insights/*
and /v1/event to NO subsystem for tracing and price lookup, and scope.Use could
install nothing on them. That is the pricing defect one app over.
- apps/analytics installed no cloud.Bridge of its own, relying entirely on
Serve's app-wide one, which the package's own test harness never runs. A typed
op reads its tenant from principal.OrgFrom, so that reliance was the difference
between an org and a permanent 403.
And one collision refused before it shipped: SeriesPoint is already apps/admin's
{t,value} launch-board point, and the schema namespace is FLAT across the fleet —
one name with two shapes is what openapi.Weave rejects. analytics yielded, since
its schema was not yet published: UsagePoint.
TestRoutedPostSetIsExactlyTheDoors now reads GetRoutes(true). fiber keeps
middleware in the same stack as routes and reports it under every method at its
prefix, so the unfiltered read was never "the POST surface" in the real binary
either — Serve's own app-wide middleware has always been in it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every one of provisioning's 28 operations was route-only: no description, no
summary, no request shape, no response shape, and therefore no MCP tool, no CLI
command, no SDK method and no schema. The cause is one line, and it generalises:
for _, kind := range kinds { app.Post("/v1/"+kind, create(s, kind)) ... }
A computed path is not a constant, so cmd/zipdoc refuses it outright ("route path
is not a constant string, so the operation has no identity to document"), and a
handler returned by a FACTORY is a call expression with no doc comment to lift.
A loop that registers N routes publishes prose for NONE of them however well the
handler is commented. So the paths are spelled out per kind — one declaration per
published operation, which is what every projection keys on anyway.
21 typed: list, get and delete for each of sql, kv, datastore, docdb, vector,
search and s3. The wire is byte-identical. The listing's Out is a NAMED SLICE
(provisionedList []provisionedSummary), not an envelope struct, because the wire
is a bare JSON array — the envelope was the natural typed shape and a silent
break; TestTypedReadsKeepTheirWire asserts the empty listing's BYTES are `[]`.
The delete keeps its 204-with-no-body (a nil Out on an unnamed Out type).
7 refused, all the same fact once per kind: POST /v1/<kind> runs the pre-provision
balance gate and renders a denial through cloud.DenyResource, whose body is the
fleet's NESTED {"error":{code,message}} at 402/503. A typed op can only refuse by
RETURNING an error, which zip renders as its flat HTTPError, and writing the
nested body inside the op does not escape it either — a nil Out makes zip stamp
cmp.Or(op.Status, 204) over the 402. Gating in middleware would move today's
400-on-a-bad-name to a 402. The refusal is GATED, not prose: untypedByDesign +
TestEveryRouteIsTypedOrNamed read the router of the REAL routes() and require the
two ledgers to SUM to the served surface. The seven still DECLARE their bodies
through openapi.Register, so provisionRequest/provisionResult reach the document
and an SDK caller has somewhere to put the name.
Tenancy could not go through principal.OrgFrom, and that is a wire fact: tenant()
folds the org through sanitizeOrg — the slug every physical name, S3 bucket and
tenant-<org> namespace is keyed on, so a read that skipped the fold would look in
a different bucket than the create wrote — and buckets an ORG-LESS SuperAdmin
under the literal "admin" org, which OrgFrom refuses outright. tenantOf reaches
the request (cloud.Request, pinned with that reason) and asks the SAME tenant()
the untyped create beside it uses, so one surface cannot key its tenancy two ways.
Two more found by typing:
- dedicated.go's header still called sql and kv shared kinds; they moved to the
dedicated strategy in the same file.
- the escape-hatch pin was red on main — apps/agents/routing_http.go's
claimKeyOf called cloud.Request with no entry in allowedRequestUses, so the
gate that exists to stop that hatch growing quietly had itself grown. Closed
concurrently by the tools pass (30adfdce); this rebase takes that entry.
Mount now fails when its Router does not expose the op registry, rather than
serving 21 reads no projection knows about. The 14 :name operationIds take the
_by_name -> _name rename typing brings.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/plans published fifteen operationIds and NOTHING else: no schema, no prose,
no MCP tool, no CLI command, no SDK method. An untyped route is invisible to all
five, and this surface was 15 of 15.
The premise that held it back — "a typed op cannot proxy the bundle's bytes
verbatim" — is false here for the reason it was false one subsystem over in
apps/pricing: apps/goja already re-marshals the bundle's answer with Go's
encoding/json (Host.DispatchWith, json.Marshal(m["body"])) before any handler
sees it, so what the raw pass-through wrote was never the JS engine's bytes, it
was Go's, keys sorted. Each Out declares its keys in that sorted order or carries
the value as json.RawMessage, and apps/plan/wire_test.go PROVES byte-equality
against the live router rather than asserting it: all 12 sections × 3 identity
shapes (anonymous, forged X-Org-Id, validated member), plus both parameterised
addresses over EVERY plan id the shipped catalog holds.
Opaque catalog values are json.RawMessage, not map[string]any. @hanzo/plans owns
the shape of a plan, a tier and a schema document; a Go struct restating one is a
staler second source that silently drops what the catalog adds. zip v1.18.9+ asks
whether a type marshals itself before asking what it is made of, so a RawMessage
publishes {} — "any JSON", the only true thing to say — while map[string]any
publishes additionalProperties:{"type":"object"}, which the first
"priceMonthly": 20 in the catalog refutes. A false schema is worse than a thin
one.
TWO RESIDUAL DELTAS, recorded and pinned rather than glossed:
- Content-Type gains "; charset=utf-8" — what the health probe and every zip
error on this surface already sent. JSON is UTF-8 by definition (RFC 8259)
and charset is not a registered parameter of application/json.
- A non-200 body gains zip's `status` field beside the bundle's own message.
The status is the bundle's and the message is the bundle's, byte for byte,
under the same key (dispatchErr); returning an error is the ONLY path a typed
op's refusal can take. Same trade main already took for
GET /v1/pricing/model/{name}, whose 404 is equally first-class.
TestResolutionIsByteIdenticalForEveryPlanInTheCatalog asserts the error body is
EXACTLY {status,error} and nothing more, so a third difference cannot appear
quietly.
LATENT DEFECT, found by typing and fixed: the subsystem is named "plan" and
serves "/v1/plans", so MountPrefixes' /v1/<Name> default covered nothing it
registers. Measured: SubsystemOf("/v1/plans") was "" and PriceOf undeclared — the
tracing and price index cloud.Declare builds attributed every request on this
surface to NOBODY — and any middleware the subsystem installed, including the
typed-op Bridge that carries the validated org, landed on /v1/plan and never ran.
plugin/plan/main.go now passes manifest.PrefixesFor("plan"), the same one source
the host routes by.
The tenant is never an In field: catalogTenant reads principal.OrgFrom, so an
anonymous caller and a forged X-Org-Id both read the public "hanzo" catalog and a
validated reseller reads its own. TestTheTenantIsNeverAnInputField gates it off
the published document, where such a field would become visible to every SDK.
Measured after: plugin/plan/openapi.json 15/15 described (was 0/15), 9 published
schemas, 15 MCP tools all described on the running binary, 15 ops on the call
plane. openapi.yaml rewoven. The two parameterised ops take the documented
operationId rename (_by_id -> _id); no path, status or field name moved.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The tool plane's whole subset was prose-blind: every one of its 16 operations
reached openapi.yaml, every generated SDK and every MCP tool list with no
summary and no description, so a model choosing a tool and a developer reading
the SDK both got a bare operationId. Fourteen are now typed ops — one registry
entry each, which is what makes an operation an OpenAPI description AND an MCP
tool AND a CLI command AND a typed SDK method rather than only a route.
The two that stayed raw stayed for the wire, and both are now GATED rather than
claimed. POST /v1/tools/mcp is deliberately body-tolerant: a malformed body is
HTTP 200 carrying JSON-RPC -32700, which op.invoke's unconditional 400 cannot
express, and its request/response are envelopes whose shape depends on `method`.
POST /v1/plugins/build answers 422 carrying the build diagnostics as a domain
body, and a typed op's only refusal is a returned error, which zip renders flat
with nowhere to put the source that failed. untypedByDesign + the two ledgers
summing to the served surface make a third refusal a deliberate edit.
The three ?activated / ?all filters stayed STRINGS. These routes compare the raw
query value to the literal "true", and zip's setScalar reads a bare ?activated
and ?activated=1 as true — a bool In would answer with a different set of tools
for the same URL. That is the whole conversion's rule: describing a route is not
licence to move it.
Also closed the response half the op gate cannot see. Tool, Skill, MCPServer,
AuthoredPlugin and Price are store rows, so their properties reached every SDK
and every MCP inputSchema bare — `hasSecret` a flag with nothing saying the
VALUE is never returned. All 52 published properties carry prose now, gated.
Two latent defects fell out. The tools test harness mounted the plane with no
cloud.Bridge, which was invisible while every route was untyped and would have
made all fourteen typed ops answer 403 to a request carrying a valid org.
And TestRequestEscapeHatchIsPinned was already RED on main: apps/agents/
routing_http.go:61 reads the machine claim-key header through cloud.Request and
was never added to allowedRequestUses — named now, with the reason.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
metering.Usage carries three amount sources with a documented precedence — the
typed money.Amount wins, then micro-USD, then whole cents — and one function
resolves them. Client.Record has always guarded on that resolved value.
ResourceMeter.MeterUsage asked its own version of the question:
if u.AmountCents <= 0 && u.AmountMicros <= 0 { return }
Two of the three. So a Usage whose cost is carried ONLY as a typed Amount — which
is exactly what a per-token 18-decimal caller sends, zen among them — returned
early and never reached Record. No error, no log, no row, no debit. The one place
in the fleet that disagreed with itself about what money is, and it disagreed by
dropping it.
Usage.Money is now exported and both callers read it, because "is there anything
to bill here?" is the same question wherever it is asked and asking it any other
way gets a different answer.
TestResourceMeter_MeterUsageBillsAnExactAmount meters $0.0025 with no cents and no
micros set. It records 0 usages against the old guard and 1 against this one. The
amount is deliberately sub-cent: it survives only because it is exact, which is
the entire reason the typed field exists.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The tasks product publishes four addresses and describes none of them: /v1/tasks,
/v1/tasks/*, /tasks and /tasks/* — 28 operations in plugin/tasks/openapi.json, 0
with a description, so no schema, no MCP tool, no CLI command and no SDK method
for any of them.
None can become a typed op inside cloud, and the reason is the wire in every
case. /v1/tasks answers 307 with a Location, which a typed op has no vocabulary
for. /v1/tasks/* is one route over 64 engine operations this router never sees:
hanzoai/tasks matches them by path SEGMENT inside its own ServeMux, their inputs
are anonymous structs local to that module's handlers, and the engine hands cloud
its surface only as http.Handler — its programmatic seam (View + three *ForOrg
helpers) reaches 13 of the 64. That one route also carries four content types at
once, 12 of its verbs deliberately run on a malformed body a typed op would 400,
and its error envelope carries `code` as a number where zip's carries `status`.
/tasks/* is the SPA: bytes under their own content types, not JSON.
So the record at the mount states each refusal and typed_wire_test.go MEASURES
each one, rather than leaving the claim as prose nobody re-runs. The place these
operations become typed is hanzoai/tasks, which owns the surface; a second copy
of its route table living here would be free to drift from the one that answers
the requests.
Also: LLM.md's opaque-product count re-measured over the committed subsets (3
wholly opaque — dns, licensing, sentry; bot has left that list, 20 mixed), and
three doc pointers to the deleted clients/tasks package corrected.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The count in the commit before this one was wrong, from the same mistake this
file keeps warning about: it was measured with a hand-written method filter
(get/post/put/patch/delete/head/options) instead of read off the document, and
`trace` fell out of it. `app.All` registers nine fiber methods and openapi.From
publishes seven — the five body-bearing ones plus OPTIONS and TRACE — so five
wildcards yield 35 operations, not the 30 I wrote or the 25 a REST reader assumes.
The sum check in typed_wire_test.go was already derived from the live document and
so was green either way; only the prose beside it was false, which is the failure
this repo cares about, since that prose is what the next reader measures against.
Reading the methods instead of assuming them also surfaced a fleet-wide fact worth
more than the correction: 27 operations across 10 packages publish `trace` — exec
8, iam 5, tasks 4, o11y 3, base 2, and one each in ai, dns, licensing, runtime,
websearch — every one from an `app.All` catch-all, because All means all. So
openapi.yaml, every generated SDK and the MCP tool list offer HTTP TRACE on ten
products including the identity plane. Recorded with the command that re-finds it
and deliberately not fixed here: the repair is one decision in openapi.From's
method filter, it moves the document for ten packages at once, and regenerating
nine other subsets inside an iam change is exactly how a concurrent agent's work
gets clobbered.
plugin/iam/openapi.json still regenerates byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The whole product is five `app.All` wildcards relaying iamserver.Handler(db) —
github.com/hanzoai/iam's entire standalone zip app, 94 typed ops of its own,
adapted to net/http. So zero of the 25 undescribed operations convert, and the
reason is structural rather than a backlog item: one registration serves an OPEN
set of sub-paths; the bytes, status and Content-Type are the nested app's own
(its Guard's flat {"status":401,…}, /login/oauth's 302 + Location); and the oauth
token/introspect/revoke endpoints take application/x-www-form-urlencoded, which
zip's op.invoke jsonenc.Unmarshals into a 400.
That refusal is now a GATE, not a paragraph. untypedByDesign is keyed by PATH
rather than METHOD /path because one app.All refuses for every method at once —
keying by method would state one fact six times and let five copies rot — and
TestEveryRouteIsTypedOrNamed expands it over the methods the document publishes
and checks the sum (0 typed + 30 named = 30), so a sixth wildcard, a narrowed
wildcard, or a route added here as a raw handler all go red.
TestTheRelayIsWhyNothingIsTyped drives the live mount and asserts the two bodies
cloud never composes: the nested Guard's envelope and the minted OIDC discovery
document.
Reading it that closely found a live defect. mountFailClosed iterated Prefixes
alone while safeMount also registered the root /.well-known/*, so the degraded
surface was strictly SMALLER than the mounted one — and the terminal handler in
every plugin binary is webui.Mount's `/*` console catch-all, with /.well-known
outside the console's apiPrefixes. With IAM broken, GET
/.well-known/openid-configuration — the FIRST call every relying party makes —
answered 200 with the SPA's HTML instead of the honest 503, so an OIDC client
parsed a web page as its discovery document. Both halves derive from one
patterns() list now, and the test checks the derivation as well as the statuses.
Three doc comments were saying untrue things and are corrected, since prose is
the product surface here: the package header claimed iamserver.Route registers
"ZIP-NATIVELY … no net/http adaptor round-trip" (safeMount's own comment says
the opposite, and is right), and Prefixes claimed to be "the ONE list" handed to
MountAll by apps.Wire(), a composition root that no longer exists — the host's
list is manifest.Apps' iam row, deliberately separate so cmd/cloud stays light.
plugin/iam/openapi.json regenerates byte-identical: the wire and the document are
untouched. LLM.md records iam as the fourth wholly-opaque product and why closing
it is COMPOSITION (merging the nested app's own document at its prefix) and not
typing — including the two facts that gate it, measured: hanzoai/iam v1.33.26
ships no zipdoc_gen.go and no zip.Describe anywhere, so those 94 ops carry
summaries and no descriptions; and cloud's own /.well-known/openapi.json plus
agentskills' /.well-known/agent-skills/* win under iam's root wildcard only
because zip matches the most specific pattern regardless of registration order.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The code-interpreter surface publishes 56 operations and not one carries a
description, a schema, an MCP tool, a CLI command or an SDK method. It is not a
backlog item: Mount hands all 8 paths to httputil.NewSingleHostReverseProxy and
the sandboxed executor supplies every byte, every Content-Type and every status,
so there is nothing here to describe and four wire facts a typed op would move —
verbatim upstream status (zip answers its own declared one, typed.go:305-311),
response fields this repo never named (an Out drops them), a multipart
/v1/upload body (typed.go:242 jsonenc.Unmarshals every non-empty body) and a
byte-bodied /v1/download/{id} (typed.go:311 always c.JSONs). 16 of the 56 are
OPTIONS/TRACE, which zip has no typed registrar for at all. openapi.Register is
refused too rather than reached for: it could only publish a guess at
@librechat/agents' contract, and Binary's application/octet-stream is not a
multipart envelope.
So the refusal becomes a gate. typed_wire_test.go crosses two closed lists
(untypedPaths x servedMethods) into the same 56 addresses the document uses and
fails three ways: a route neither typed nor named, a reason naming an address
this mount no longer serves, and the day one of them becomes typable. Eight
sub-tests measure the wire facts through the REAL Mount rather than asserting
them. Both directions were mutation-checked: a fifth prefix in exec.go and a
stale ledger entry each turn it red.
Two findings the pass surfaced, recorded in LLM.md rather than fixed here:
- The migration's own inventory cannot see this app. Every documented grep
anchors on `("` or `("/`, so `app.All(p, h)` in a range over prefixes matches
nothing — apps/exec reads ZERO under both commands while serving 56
operations, which is why it has never appeared in a tranche. Same shape in
apps/knowledge (subsystem.go:61-69), apps/iam (258-259, 282) and
apps/commerce (mount.go:551). Count operations, not lines.
- plugin/exec/main.go told the next reader to "edit the spec below directly".
There is no spec in main.go, and the one it means is generated by
`make -C apps/exec openapi` and must never be hand-edited. Corrected here;
110 sibling mains still carry the sentence.
exec.go gains only a comment; the regenerated subset is byte-identical, which is
the honest measurement: 56 operations, 0 described, before and after.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
28edc6e0..3489bc0b moved o11y's ingest onto the one /v1/event family and did not
regenerate plugin/o11y/openapi.json, so five served operations were published
nowhere: /v1/event/error/{wildcard1} on all five methods (mountEventFamily
registers All(), and the handler's own method gate 404s what it does not accept —
which is why the document lists five, the same convention every other All() route
here follows).
Caught by the drift gate on the next regeneration, which is the whole point of it
regenerating FROM SOURCE rather than comparing two derived artifacts: nothing about
the committed subset and the woven golden disagreed, because both were built before
the routes existed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TWO THINGS, ONE CAUSE — zip v1.18.11.
The route. PATCH /v1/admin/catalog/providers/{name} was refused because `overrides`
is an RFC 7386 merge patch stored and echoed verbatim, which pins its Go type to
json.RawMessage, which zip published as an ARRAY OF INTEGERS. typed_wire_test.go
had pinned that lie deliberately, with instructions: "If zip now describes a raw
JSON value as one, the providers/{name} reason is stale — type the route." v1.18.9
describes it as one. The test went red, and this is the route it asked for.
The overlay upsert is now applyPatch(ctx, kind, id, patch) with no request in
sight, called by the typed op AND by the *zip.Ctx door the models/* wildcard route
still needs — one implementation, two doors, so they cannot drift. The patch
fields stay POINTERS: absent must differ from a zero the caller meant, and
encoding/json leaves a pointer nil for an explicit null too, so {"enabled":null}
and {} arrive identically — which is what this route already did.
One residual delta, recorded in ops.go rather than hidden: zip decodes before the
handler runs, so a non-admin sending malformed JSON now sees 400 where the raw
handler answered 403. zip's authorizer is deliberately post-decode (it authorizes
the decoded value, so the decision cannot diverge from execution), so this is not
avoidable while the route is an op. It reveals only that the body was unparseable.
The four apps. Declaring that op exposed a defect in v1.18.9 itself: its published
requestBody was ABSENT. The input is a path param plus an embedded unexported
patch body, and every projection asked "what fields does this type carry" with its
own loop over NumField — so it skipped the embedded type on IsExported and saw
only `name`, which IS a path param. wireFields (v1.18.11) is the one function that
knows encoding/json's promotion rule. Regenerating with it corrected four apps:
pricing gains the request body it always accepted;
admin publishes SaaSMetrics' fields flattened, where it had a property
literally named "SaaSMetrics" that the wire never sends;
agents gains 117 lines of response fields that were published nowhere;
visor gains 81 lines, same cause.
Every one of those was a document describing a shape the service does not have,
in openapi.yaml and in every SDK generated from it. Found by READING the artifact
after the bump instead of trusting that the bump was an improvement.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The bump alone changes 23 published artifacts, because zip v1.18.9 fixes what the
projection SAYS rather than what any route does. No wire moves.
phantom request bodies 41 -> 9. A POST binding its whole input from the path
published a required body whose only property was the path param, so every
generated SDK gained an argument the caller must build to repeat a value it
already passes in the URL. The 9 left are the raw-body family (git-upload-pack,
bank-statement import, a deck upload) — they eat bytes, not JSON, and owe a
binary content type via openapi.Binary rather than an empty object.
time.Time stopped publishing as a $ref to a schema with no properties and now
says format: date-time. Its fields are unexported, so reflection over them
described nothing: every timestamp in every generated SDK was untyped.
summaries lost their embedded line breaks — one sentence on one line, which is
what the spec, the CLI's one-line help and an SDK's first docstring line all
want.
imported types' FIELD docs reach the document at all. zipdoc matched the parsed
and type-checked views of a struct by byte offset, which only agrees for a
package loaded from source; an imported type's position comes from export data
with a synthetic offset. So every op whose In and Out live in the call plane
published a description and zero field descriptions.
AND ONE STALE PUBLISHED SURFACE, which the regeneration exposed rather than
caused. plugin/authz/openapi.json documented GET, POST and DELETE
/v1/authz/policies. hanzoai/authz v1.10.15 does not serve them, and says why in
serve/mount.go: the grant set belongs to IAM, "a second writable copy behind this
surface would be a second source of truth for who may do what". So cloud was
advertising a writable authorization-policy API that had been deliberately
removed, and three methods in every generated client answered 404. Verified as
pre-existing by regenerating the subset at origin/main on the OLD zip: the same
three paths vanish.
That drift means the gate was red on main and stayed red. It is invoked
(hanzo.yml:134), which leaves the two ways it could have been red and unnoticed —
worth a look, not a guess.
Also: openapi-apps now honours OPENAPI_NEEDS_BROKER, which only openapi-check
did. The gate's own failure text says "fix: make openapi", and that fix routed
through openapi-apps, which mounts kafka, which fails closed with no broker — so
the single command told to repair a red gate could not run. One exemption list,
read everywhere it applies.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every git subprocess shares cloud's cgroup, and pack generation scales
with REPO size, not request size: an upload-pack of a multi-GiB repo is
a multi-GiB allocation the Go runtime cannot see or govern (GOMEMLIMIT
bounds only the Go heap), landing as a kernel OOM kill of the entire
API — the exact profile of the ~hourly exit-137 kills (multi-GiB spikes
faster than the metric step, fatal request never logged). GIT_CONFIG_COUNT
in the ONE subprocess constructor caps delta search (64m x 2 threads),
pack mmap (256m in 32m windows) and delta cache (64m): a few-hundred-MB
worst case per operation, traded against clone speed on the biggest repos.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/event was already the canonical product-event door (the analytics
doors table's target; its sunsetting aliases name it). The o11y ingest
kinds now live under the SAME family instead of their own path roots:
- POST /v1/event/ingestion — the LLM-obs batch (traces/observations/
scores), moved from /v1/o11y/ingestion (nothing called it yet).
- POST /v1/event/error/… — the Sentry wire. A DSN of
https://<key>@api.hanzo.ai/v1/event/error/<project> works as-is:
SDK-expanded api/<project>/envelope|store forms map onto the
/v1/o11y/api ingest routes, the bare form onto /v1/sentry, both
rewritten before the principal gate so its two existing ingest
exemptions stay the only exemptions. The family carries ingest ONLY —
no READ API is reachable through it (pinned by test).
manifest: deeper /v1/event/{ingestion,error} prefixes route to o11y
while analytics keeps the root and /collect — deeper prefix wins, so
neither app shadows the other (the reachability test proves it).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Picks up the hanzoai/kafka rework: offsets stamped in batch headers only
(sparse e18 sequences and holes behave like dense), poison messages
skipped instead of served (the insights outage class), commits accepted
for any non-negative offset (the 2^50 'plausibility' guard rejected every
real one), OFFSET_OUT_OF_RANGE instead of dead-position stalls, LeaveGroup
handled, response encoder grows to fit, request frames capped. Proven by
the repo's new in-process franz-go e2e and this package's interop test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/agents had 26 routes and 5 typed ops, so 21 of them were a route and
nothing else: no schema, no prose, no MCP tool, no CLI command, no SDK
method. 19 are now typed ops. The route table is unchanged — 31 operations
before, 31 after, none added, none removed — and the wire is byte-identical:
a differential harness drove 83 request/response pairs (every status, every
error body, every ?limit=/?after= edge, the 204s, the 201s, a real routed-run
claim) against origin/main and diffed them, and after canonicalising JSON
object key order the two dumps are the same bytes.
What the typed registry gained: descriptions 5 -> 24 on the subset, 419 -> 438
on the fleet golden, 32 new schemas, and the query parameters of every read
(?root, ?parent, ?status, ?project, ?limit, ?after, ?range) documented for the
first time — they were invisible, because a raw handler's c.Query() is not a
contract.
THREE LATENT DEFECTS, surfaced by typing:
1. cloud.Bridge was installed in mountTargets, THREE calls into Mount. fiber
runs middleware in registration order, so it never ran for any leaf
registered above it — this file's, mountSessions'. It was harmless while
only the target ops were typed, and would have 403'd every read here the
moment they were not. It is now installed once, at the top, before any leaf.
2. PATCH /v1/agents/targets/{id} shipped a request body schema holding `id`
ALONE. patchTargetIn embedded its mutable fields from a body struct, and
zip's schema walk takes only EXPORTED fields — an embedded field of an
unexported type is not one — so label/kind/status/capacity/host/spec/metrics
were absent from openapi.yaml and no generated client could send any of
them. Flattened: eight properties now, same wire (json promotes either way).
The same zip gap still shortens agentDetail and sessionDetail, where the
embedded view is genuinely shared and a second copy would be a field that
silently stops being sent; both carry a note pointing at the one fix.
3. Two fleet schema-name collisions the weave refused: `runView` also means a
platform run (apps/platform), `createReq` also means a git repo create
(apps/git). One name, two shapes would bind whichever an SDK read last.
Renamed here (agentRunView, createAgentIn) — a schema name is document-only,
never on the wire.
POST /v1/agents also declared 200 while always answering 201; it declares 201
now (zip.WithStatus), and DELETE declares the 204 it sends.
SEVEN routes stay untyped, each because typing would move the wire:
GET /v1/agents/sessions/stream an open SSE feed written by a loop that
outlives the handler; no In/Out is a feed
POST /v1/agents/:ref/run 502 carries the RECORDED RUN as its body,
and a balance denial answers the fleet
402/503 contract (cloud.DenyResource)
POST /v1/agents/sessions/:id/events \ the guard gate refuses a credential in
POST /v1/agents/sessions/:id/pause \ a transcript with 422 IN BAND, naming
POST /v1/agents/sessions/:id/resume / the rule, line and fingerprint of each
POST /v1/agents/sessions/:id/stop / finding; zip's error type carries
POST /v1/agents/sessions/:id/message {status,code,error} and no findings[]
All six of those wait on the same zip capability: a response body per status.
Verified with the gate env (CLOUD_KMS_MASTER_KEY_REF, -tags sqlite_fts5), not
bare go test: apps/agents green (baseline was green), zipdoc -check clean,
weave green, and every in-process consumer of this package — coding, link,
team, visor, cli, plugin/agents, cmd/cloud — still builds and tests green.
cmd/cloud gained no apps/* import.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/ml served seventeen routes and published seventeen paths with no schema, no
prose, no MCP tool, no CLI command and no SDK method — the whole cost of an
untyped route, paid seventeen times. Ten of them are now typed ops, so one
registration is the whole contract: the reads of all three Kubeflow-family
resources (models/jobs/experiments), their deregistrations, and the katib trials
leaf.
The seven that stay raw are wire-bound, and they are a GATE now rather than a
paragraph: untypedByDesign + TestEveryRouteIsTypedOrNamed hold the closed list
and its two ledgers must SUM to the surface the live router serves, so a route
added untyped goes red without anyone remembering to name it.
- the three creates answer 402/503 IN BAND through cloud.DenyResource, with the
fleet's nested {"error":{"code","message"}} contract. A typed op can only
refuse by RETURNING an error, which zip renders as the flat
{"status","code","error"} HTTPError — a NEW refusal class, distinct from the
multi-status gap: a non-2xx carrying a DOMAIN body. Moving the gate into
middleware does not rescue them, because it would run before the body decode
and turn today's 400-on-a-bad-name into a 402.
- PATCH /v1/ml/models/{name} relays an opaque RFC 7386 merge patch VERBATIM to
the Kubernetes API. Through map[string]any every number becomes a float64, so
{"replicas":1000000} re-marshals as 1e+06 and patches a float over an int.
- POST .../predict returns the predictor's own status, bytes and Content-Type.
- both /health probes answer 503 carrying the degraded REPORT as their body,
which is the point of a real probe.
Identity crosses the typed seam on the context and never as an In field: one
tenantFrom, pinned in allowedRequestUses with the reason it needs the REQUEST
rather than principal.OrgFrom — ml's boundary is a per-org(+project) KUBERNETES
NAMESPACE, and OrgFrom refuses an empty org outright, which would turn the live
org-less-admin "ml-admin" bucket into a 403.
Wire preserved, and measured rather than asserted. ONE view() now returns the
published mlResource for both the typed reads and the untyped create/patch, so
the shape cannot depend on which route served it; its fields are in ALPHABETICAL
order because it replaced a map[string]any and encoding/json sorts a map's keys,
and TestView asserts the marshalled BYTES. Spec and Status are pointers because
the wire distinguishes an absent key from a present-but-empty object, which a map
with omitempty does not. The two wire tests pin their own state (a fake client, a
deliberately nil one) instead of depending on whether the box has a kubeconfig.
Seven operationIds rename _by_name -> _name, which is what a route going typed
does to the generated SDK's method name. No path added, removed or moved: 1013
before and after.
Two latent defects surfaced and REPORTED rather than quietly fixed:
- dynForOrg documents itself as "the ONE federation seam" and no handler calls
it, so an org with a registered BYO cluster silently gets its models and jobs
on the home cluster. Wiring it changes which cluster a resource lands on and
owns a migration question, so it needs its own change.
- openapi-check is red on main in two apps nobody typed: plugin/authz loses
/v1/authz/policies on regeneration (authz is an EXTERNAL module, so that may
be this box failing to mount it — committing it would delete three documented
endpoints from every SDK) and plugin/tools retags 35 lines. Both reverted,
both written up in LLM.md.
Also dropped two parameters neither function ever read (tenant's service,
k8sErr's request) and split mount() out of Mount() so the routes can be exercised
over a state a test pins.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cd4de37e (automations) established the rule this pass had to be measured against:
a typed op must be addressable through its In ALONE. Over MCP and the ZAP call
plane there is no URL — zip passes the arguments object as the BODY with a nil
path map — so an op whose address reaches it only from the path works over REST
and nowhere else.
The two :id writes typed here are exactly the shape that risks it: a struct In
carrying an id field beside a body, decoded by an UnmarshalJSON of its own. They
pass, and for a reason worth writing down rather than trusting: ID is a field of
the type UnmarshalJSON decodes, so `"id"` in the arguments binds it, and over
REST bindURL then overwrites it from the path, which is the authority there.
Measured both ways rather than asserted. With `v.ID = ""` spliced into the
stakeholder patch — an In that does not receive its id from the body — this test
goes red naming what was lost ("cannot address a stakeholder over MCP ... the
REST wire survived and this projection did not"), while every REST pin in this
file, the byte-identity comparisons against the relay, TestHTTPEndToEnd and
TestFullLifecycle all stay GREEN. That gap is the whole reason the test exists.
newAppMCP is the harness that made it reachable, and it records the same fact
automations found: zip's projections of the registry (/mcp,
/.well-known/zip/op/) are routes on the APP, outside every subsystem group, so
the /v1/captable group's own Bridge never runs for them — cloud.Serve's root
Bridge is what gives them a validated org.
Tests only. No route changes, no wire changes: zipdoc -check is green and the
weave reproduces openapi.yaml byte for byte.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/captable was 17 typed of 31, and the fourteen relays were all held to be
untyped for one reason: the goja bundle validates with COERCING helpers, so a Go
struct would accept less than the route does.
Re-checked against goja/src/validate.ts, that reason is true of eleven of them
and not of three. The blocker is `num`/`intNum`/`optNum` — a number OR a numeric
string — and a float64 field cannot accept `"1.5"` NOR carry the token it
rejects onward, so it moves both what the route takes and which envelope refuses
it. Those eleven stay relays.
PUT /company, PATCH /stakeholders/:id and POST /rounds/:id/close carry no number.
Every field they take goes to reqString, optString or optDateString, and a
verbatim `scalar` carrier hands each token to the bundle unchanged — so the
bundle stays the ONLY validator, of what is accepted and of how it says no.
The carrier is a string KIND, so every projection describes these fields as
`string`, which is what they are; and it is NOT a pointer, because encoding/json
nils a pointer for an explicit `null` without calling UnmarshalJSON, which would
collapse the absent-vs-null distinction stakeholders.update writes columns on.
The relay's 413 survives: a typed op never sees the request, so the body size is
recorded in the input's own UnmarshalJSON and read back AFTER the tenant — which
keeps a 403 ahead of a 413 for the caller that has both problems.
writes_test.go proves it rather than asserting it: every case is sent through the
typed route AND dispatched on the bundle the way the relay did, and compared on
status, Content-Type and bytes — including a number where the bundle reads an
optString (a 200 storing "5", which a *string would have 400'd), a number where
it reads a reqString (the bundle's 400 with its `errors` list, which zip's
envelope has nowhere to put), null vs absent vs "" on the partial update, and a
body that is not an object at all.
Two orderings do move, both for a request with no valid tenant: malformed JSON
or an oversized body now answers before the 403, because zip decodes ahead of the
handler. Named in writes.go rather than left to be found.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The ONE command this file tells agents to trust filters untyped registrations with
`grep -v 'zip\.'` — a LINE filter standing in for a syntactic one. The
discriminator wanted is the RECEIVER of the call (`zip.Get(` package-qualified
generic vs `<router>.Get(` method on a router), so dropping every line that merely
CONTAINS `zip.` also drops an untyped registration whose handler argument names the
package on the same line: an inline `func(c *zip.Ctx) error`, or
`zip.AdaptNetHTTP(`.
Measured across apps/ at this merge: 660 by the documented command, 675 anchored —
15 hidden routes in 9 packages. Two of them read ZERO and are not zero:
apps/plan 3 /health, /resolve/:id, /entitlements/:id (plan.go:77-109)
apps/product 4 /v1/search-docs/{indexes,stats},
/v1/vector/{collections,stats} (product.go:80-141)
Every registration in both is an inline `func(c *zip.Ctx) error`, so the filter
eats all of them and the partition table has never dispatched either package. This
file already documents two errors in this same command, in the opposite direction —
a phantom sends an agent at nothing; a hidden route means nobody is ever sent, which
costs more. Both commands (the per-app loop and the fleet one) now anchor on the
call, and the third bullet under "Partitioning the remaining work" carries the
measurement.
Found because o11y read 7 against the 8 its own conversion recorded: the missing
one is `a.All("/v1/sentry/*", zip.AdaptNetHTTP(…` at apps/o11y/o11y.go:231. The
corrected command reproduces 8.
Also records two live openapi-check failures on main, under failure mode 1 where
that class already lives: plugin/authz/openapi.json publishes GET|POST|DELETE
/v1/authz/policies, which hanzoai/authz v1.10.15's serve.Mount does not register at
all (it serves only health/readyz/check) — three operations in openapi.yaml, in
every generated SDK and in the MCP list that no binary answers, with
manifest/apps.go:38 still routing the prefix at them; and plugin/tools/openapi.json
is a golden emitted by an older generator (four tags collapsed to one, an escaped
em dash). Neither is regenerated here: doing another package's subset inside an
unrelated change is how a concurrent agent's work gets clobbered.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Follow-on to the CLI-collision commit, using the seam apps/books added an hour
earlier. git's 24 refusals were correct as refusals and stay raw — the webhook's
HMAC covers the raw bytes it verifies before json.Unmarshal, smart-HTTP streams
x-git-* pack media, the UI renders html/template, and the ZAP adapters' error
envelope ({status:"error", msg}) is a shape zip's errorHandler cannot produce —
but "cannot be a typed op" had been read as "must be undocumented". All 24
published an operationId, tags and NOTHING else, which no SDK generator can tell
apart from a route that takes no body. The published forge webhook therefore
offered a delivery with nowhere to put it, and three ZAP procedures a repo name
with nowhere to put it.
Eight now declare the request they actually read, through openapi.Register in an
init (Register panics on a duplicate; routes() runs once per Mount):
POST /v1/git/webhook pushEvent
POST /v1/git/zap/{createRepo,getRepo,deleteRepo} zapProcReq
POST {/v1/git,}/{org}/{repo}/git-{upload,receive}-pack openapi.Binary
Pure description: no route, status, field or byte moves, and openapi.yaml gains
81 lines with ZERO deletions. The two ZAP procedures that read NO body
(listRepos, usage) are deliberately left silent — declaring one for them would
swap an honest silence for a fresh falsehood — as are the ref advertisement and
the twelve HTML pages.
No RESPONSE is declared, and the reason is worth keeping: the ZAP envelope's
data is repoView / []repoView / usageView, names zip's typed fold already
publishes as components off the typed /v1 ops, so reflecting them here through
openapi.schemaOf would put two derivations behind one schema name — precisely
what openapi.Weave exists to refuse. The pack responses and the HTML pages have
no seam at all: openapi.Binary is request-only by design.
declaredBodies + TestRefusedRoutesDeclareTheBodyTheyRead pin it in both
directions — a declared body that vanishes fails, and a body declared for one of
the sixteen that read none fails too — and both directions were verified red
before this was committed green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/git was already 24 typed / 24 refused, and every refusal re-verified here
against the handlers and zip v1.18.6 itself: the webhook's HMAC covers the raw
bytes it verifies before json.Unmarshal, smart-HTTP answers c.SendStream over
x-git-* pack media, the UI renders html/template through render(), and the ZAP
adapters' failure envelope ({status:"error", msg}) is a shape zip's errorHandler
({status:<int>, code, error}) cannot produce. Nothing there can be typed without
moving a wire, so nothing was.
What typing DID surface is a defect one projection down. A typed op is one value
with four projections; three key on the op's own identity, but the CLI keys on a
name zip SPELLS from the route, and commandName (zip/cli.go:253-311) keeps the
segments before the first path parameter and after the last one while dropping
everything between. So `mirrors` and `subscriptions` — the words that say WHICH
thing a DELETE removes — never reach the name, and all three of
DELETE /v1/git/repos/{name}
DELETE /v1/git/repos/{name}/mirrors/{id}
DELETE /v1/git/repos/{name}/subscriptions/{id}
derive `git repos-delete`. Two of git's 24 typed ops therefore have no command a
caller can reach, while the wire, the document, the MCP tool and the SDK method
are all correct and distinct. An untyped route has no command to collide, which
is why only typing finds this.
Measured with zip's own derivation over the committed subsets rather than a
reimplementation of it: 20 colliding names hiding 23 ops across 11 packages, in
four shapes — interior segments dropped (git, cloudflare x2, o11y), a collection
colliding with its own item (compliance, marketing, framework), PATCH and PUT
both spelling `update` (base, exec x4, iam x2, websearch), and one op at two
addresses differing only by the version segment isVersion strips (tasks x5).
git holds the worst single instance, three ops on one name.
The fix is commandName carrying the interior segments — not WithOperationID
here, which would make git's ids a special case of a general bug. Until it
lands, cliNameCollisions + TestCLINamesCollideExactlyWhereKnown pin the damage
in BOTH directions: a new collision fails, and a collision that has GONE fails
too, so the zip fix retires the list instead of outliving it. Verified red both
ways before being committed green.
Also recorded, both measured from the golden and neither fixable in this
package: nine host-gated root paths (the six root UI pages and three root
smart-HTTP routes, all falling through with c.Next() off the git host) are
published under the document's single `servers` entry https://api.hanzo.ai, so
every generated SDK gains nine methods that 404 against the host the document
names — GET / worst of all, whose operationId is the bare word `get`. And git is
the only app that parks its own principal instead of installing cloud.Bridge,
because Bridge carries the org alone while git's ops need the project sub-scope
and the acting SSH-key owner; widening the shared bridge is what makes that one
way again.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The four exclusion pins landed in fce62ae1 close the retype a reader reaches
for first (a non-struct In: takes any body, receives no path param). They do
not close the one that looks like a repair: a STRUCT In carrying an id field
plus an UnmarshalJSON that swallows the whole body. Nothing can fail it, and
bindURL still binds :id from the path, so the REST wire is preserved exactly.
Measured, not argued. With resume retyped that way the ENTIRE package suite
stays green — all four pins included. What it gives up is what typing is for:
- a tools/call and a ZAP by-name call carry every argument in ONE JSON object
and bind no path from it (zip mcp.go / call.go pass `arguments` as the body
with a nil path map), so an In that discards its own keys never receives the
address: the tool answered "run not found" for a run that exists.
- the projection then advertises a body of {"id": string} that this route has
never accepted — the MCP inputSchema and the OpenAPI requestBody both.
So the rule the exclusions rest on is sharper than "an In cannot bind both the
body and the URL": a typed op must be addressable through its In ALONE, because
for two of its four transports the In is the only channel there is.
TestOpsAddressThroughArgumentsAlone pins that. It exercises the channel on an op
that IS typed (a tools/call naming a run id must answer that run, and one
without it must not), so it is a live assertion rather than a dormant one, and it
would go red for all fourteen ops at once if zip stopped binding an address from
the arguments object. The two URL-addressed exclusions register no op, so no tool
is derived for them today — the moment one is, this test calls it and fails on
the fact the REST pins cannot see. Verified both ways: red under the retype with
the message naming what was lost, green without it.
newAppMCP is the harness that made it measurable, and it records a fact worth
knowing: zip's own projections of the registry (/mcp, /.well-known/zip/op/) are
routes on the APP, outside every subsystem group, so a group's Bridge never runs
for them — cloud.Serve's root Bridge is what gives them a validated org.
No route changes, no wire changes, no regenerated artifacts: zipdoc -check is
green and the registry is untouched.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
openapi.Binary landed minutes ago (books, cfd7b30b) and it changes what company's
two refusals cost. It does NOT make either one typeable — re-checked, not assumed:
zip's typed path still jsonenc.Unmarshals every non-empty body unconditionally
(v1.18.6 typed.go:232), so a typed In on the deck still turns 201 into 400, and
zip's HTTPError is still flat so /payment's nested 402/503 denial still cannot
survive a typed op. Both stay out of the registry, and typed_wire_test.go still
says why.
What changes is the DOCUMENT. Both routes published an operationId and a tag and
nothing else, which to every consumer is indistinguishable from a route that takes
no body and returns none. So every SDK generated off openapi.yaml offered a deck
upload with nowhere to put the deck, and neither call had a return type. That is
not an absent feature, it is a WRONG description, and it is worse than the
registry gap it stood in for.
POST /v1/company/fundraise/deck — request application/octet-stream string/binary
(OpenAPI's spelling for an opaque body, what a generator turns into a file
parameter), response deckOut.
POST /v1/company/payment — no request declaration, because the handler
genuinely reads no body; declaring one it ignores would be invention. Response
is the shared formationView every other action here answers with, so it $refs
the same component rather than minting a second shape for one wire.
deckOut replaces the handler's map[string]any literal so the declared type and the
emitted value are the SAME value — a map here and a struct in the document is
exactly how the two drift. Byte-identical on the wire ({"documentId":"..."}), and
TestDeckTakesRawBytes proves it unchanged.
Three things remain undeclarable, and each is named at the registration instead of
being glossed over: the deck's ?name= query (the untyped projection derives path
params from the router and has no vocabulary for query), /payment's 402/503 denial
bodies (apply states 2XX only), and field prose on deckOut (zipdoc lifts comments
off typed ops only, so Register publishes shapes without descriptions).
WIRE UNCHANGED. The spec diff adds requestBody/responses to two operations that
had neither; no path, operationId or status code moves.
Gate: apps/company ok, openapi ok, vet clean, go build ./... clean.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/company was already 20 typed ops of 22 operations. What it did not have was
a reason to STAY that way: "2 left, both permanent" was a paragraph, and a
paragraph cannot fail. A route added tomorrow as a raw func(*zip.Ctx) error
would have left the claim standing and the route invisible to OpenAPI, MCP, the
CLI and every generated SDK.
typed_wire_test.go makes it a gate — untypedByDesign + TestEveryRouteIsTypedOrNamed,
the same shape crm/compliance/ingress carry. It is not vacuous: dropping either
entry turns it red naming the route.
Both refusals were re-verified against zip v1.18.6 source rather than inherited:
- POST /v1/company/fundraise/deck — op.invoke unconditionally jsonenc.Unmarshals
every non-empty body (typed.go:232). That decode does not depend on the In
binding any field, so even an empty In cannot escape it: a PDF becomes 400
where the route has always answered 201. v1.18.6 has no octet-stream/binary
request declaration to decline the decode with.
- POST /v1/company/payment — a billing denial answers the fleet-wide NESTED
{"error":{code,message}} at 402/503 (cloud.DenyResource). zip's HTTPError is a
flat {status,code,error} and errorHandler is the only path a typed op's error
takes. Writing the nested body from inside the op does not help either: a nil
Out makes zip stamp cmp.Or(op.Status, 204) over the 402, and an error makes
errorHandler replace the body. Needs zip errors that can carry a body.
The op-level gate cannot see the other half of the surface, and that half was
broken: 41 properties of Formation, Founder, Filing, Genesis, Registration,
Signer and RoundInput reached openapi.yaml, every generated SDK and every MCP
inputSchema with NO description at all — they are store row types nobody had
written field prose on. A reader could see equityBps was an integer and nowhere
that it is BASIS POINTS of 10000. Every field now says what it is, and
TestEveryPublishedFieldIsDescribed keeps it that way.
Money units are the one thing NOT asserted: RoundInput's three float amounts pass
verbatim into the cap table's rounds.create contract, which does not document a
minor/major unit either, so the prose says what the field is and does not invent
one.
WIRE UNCHANGED. The regenerated openapi.yaml diff is description text only — no
path, operationId, requestBody, response or status-code line moves.
Gate: apps/company ok, openapi ok, vet clean, go build ./... clean
(CLOUD_KMS_MASTER_KEY_REF + -tags sqlite_fts5 + CGO_ENABLED=0, matching the
Makefile). Baseline before the change was the same green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/o11y is already fully typed — 12 ops on the /v1/o11y group, all 7 remaining
registrations re-read against their handlers and re-refused, every one wire-bound:
two verbatim-status VictoriaMetrics passthroughs (c.Bytes(status, body) — a typed
op answers its ONE declared status), three reverse proxies into the runtime
(query, query_range, sessions — there is no Go type for "whatever the runtime
answered"), and two text/plain Alertmanager receipts of which the POST
deliberately accepts an unparseable body, because a receipt that 400s makes
Alertmanager retry forever. Nothing here is convertible without moving the wire,
so nothing here was converted.
What WAS missing is that all of that lived in prose, and prose cannot go red.
typed_wire_test.go makes it a gate:
- untypedByDesign is the CLOSED list, keyed the way the DOCUMENT writes each
address, and TestEveryRouteIsTypedOrNamed reads the live router of the REAL
MountO11y — not a reconstruction of it — so a route added anywhere inside that
mount (or in the upstream module it ends with) is typed by default, and
dropping one out of the registry takes a deliberate edit with a reason. The
stale direction is gated too: a name for an operation o11y no longer serves.
Proven red by adding one route and watching it fail by name.
- TestEveryTypedOpIsDescribed holds the prose to the schema's bar, because that
prose IS the product surface — the OpenAPI description AND the MCP tool
description a model reads to pick the tool.
- TestUntypedRoutesKeepTheirWire measures three of the claimed wire facts on the
real router (the text/plain receipt, its 200 over a body that is not JSON, the
text/plain replay), so the refusals are evidence rather than assertion.
- TestIngestOpIsTypedButUnreachableWithoutADSN gates the one real gap here.
POST /v1/o11y/ingestion IS a typed op with prose — the test registers it and
reads it out of zip's registry, so that half is measured — but mountEventIngest
returns before registering it with no Datastore DSN, and `bin/o11y openapi`
runs with GIT_SSH_ADDR and nothing else, so the LLM-obs write path reaches no
SDK, no MCP tool, no CLI command and no published schema. Closing it is a
behaviour decision (zip.Post registers route and registry entry inseparably,
so publishing the op stops the path falling through to the order-70 wildcard);
this does not decide it, it fails the moment somebody does.
Zero document movement: `make -C apps/o11y openapi` and `go generate -run zipdoc
./apps/o11y/...` both regenerate byte-identically, which is the point — typing is
a description task and this change describes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/pricing is 32 operations, 30 of them typed ops. The remaining two are the
admin overlay PATCHes, and their reasons for staying raw were prose — a claim
about zip and openapi.translate, recorded once and then trusted. A claim about a
dependency rots, so both are now executable against the toolchain in go.mod: the
tests register the shape at issue on a throwaway app and assert the toolchain
still behaves as the reason says. A blocker fixed upstream turns the suite RED
and names the route that just became convertible.
Checking them also found the models/* reason materially understated. It read as
a preference about an unreadable parameter name; the measured fact is that typing
that route REFUSES THE WHOLE DOCUMENT. zip keys a typed op by the fiber pattern
(".../models/*") while the document keys the same route by its URI template
(".../models/{wildcard1}", because `*1` is not a legal template name), so
openapi.Fold cannot find the op's route and errors — and Spec builds ONE
document, so every other pricing operation goes down with it. An engineer reading
the old reason could reasonably have decided to accept the ugly name and turned
the whole spec red. The parameter-name half is still true and still blocking, and
is now stated as the SECOND half.
The providers/{name} reason gains the general form of its blocker: verbatim echo
pins the field to json.RawMessage, json.RawMessage publishes an integer array,
and neither map[string]any nor any is an escape because both re-marshal and sort
the patch's keys. The escape is a schemaOf that can describe an arbitrary JSON
value, which it has no arm for.
No route changes: the golden and plugin/pricing/openapi.json regenerate
byte-identical, zipdoc -check is clean, and the app's suite is green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/books finished its typed migration at 20 ops and 5 refusals. Four of the five
refusals were right and stay: three routes eat RAW BYTES (a receipt PDF/image on
POST /v1/books/scan and /v1/books/inbox, an OFX/QFX/CSV statement on
/v1/books/bank/import) and zip's typed path decodes every body with
jsonenc.Unmarshal, so declaring any In would turn a working upload into a 400 —
the wire would MOVE, which a description task may not do. Two answer 501
unconditionally, so they have no success body to state.
But "cannot be a typed op" was quietly being read as "must be undocumented", and
those three published operationId and tags and NOTHING ELSE. No consumer of the
document can distinguish that from a route that takes no body and returns none,
so every SDK generated off openapi.yaml offered a receipt scan with nowhere to put
the receipt and no return type for what came back. That is not an absent feature,
it is a WRONG description, and it is worse than the typed-op gap it stands in for.
So declare the halves that are true. openapi.Register already carries request and
response types off the handler's own structs; what it could not express was a body
that is not JSON, because no Go struct describes a file. openapi.Binary is that
one value: passed as req it renders OpenAPI's own spelling for an opaque body,
application/octet-stream with {type: string, format: binary} — the shape an SDK
generator turns into a file parameter. Schema gains Format for it, carrying exactly
one value, because "string" and "string/binary" are the only distinction here a
consumer acts on.
POST /v1/books/scan Binary -> ScanDraft
POST /v1/books/inbox Binary -> InboxItem
POST /v1/books/bank/import Binary -> BankTally
Binary is REQUEST-only. A byte response is a second fact no route needs yet, and
adding it before one asks is how one seam becomes two.
The two 501 stubs are declared NOWHERE, and a test now asserts that silence, so a
later "document these too" argues with a gate instead of inventing a contract for
a call that has never once succeeded.
Pure description: not one route, status, field or byte moves. Every books wire
test passes unchanged, and the value is measured rather than asserted —
TestTheRawBodyRoutesDeclareBytesInAndAShapeOut reads the document through JSON,
the way a generator does, and pins bytes-in, the $ref out, and the stubs' silence.
What the three still lack, and only a typed op can give them, is prose, an MCP
tool and a CLI command. Typing them needs a zip capability that does not exist:
v1.18.6 has no octet-stream request declaration and its whole OpOption set is
WithSummary/WithTags/WithOperationID/WithStatus.
Gates: apps/books, openapi, apps/cloudflare, apps/platform, manifest and the root
package all green under the Makefile env; zipdoc -check clean; plugin/books/
openapi.json and openapi.yaml regenerated from source through the weave.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/crm was already 19 of 20 typed with its single refusal gated
(untypedByDesign + TestEveryRouteIsTypedOrNamed). Re-measured from the LIVE
router rather than the prose: 20 served, 19 typed, 15 schemas with 0 bare
properties, and POST /v1/crm/applications the one operation with no registry
entry — which openapi.yaml shows as the cost, an operationId and a tag and
nothing else.
The refusal was anchored to zip v1.18.6, the version cloud pins. All five of its
wire facts are now re-read in v1.18.8, the newest published: op.invoke is still
what a tools/call and LocalInvoke dispatch into (mcp.go:152, cli.go:427), it
still unmarshals before the handler (typed.go:234 ahead of :259), the OpOption
set is still WithSummary/WithTags/WithOperationID/WithStatus (:80/83/86/110), and
MCP.Disabled is still app-wide (zip.go:131). v1.18.8 adds ask/declare/ops/peer/
tenant and moves none of them, so 19 is still this package's honest floor and the
line numbers are recorded so the next agent does not redo the reading.
Re-verifying the refusal surfaced a live defect in the meter it leans on. The
intake limiter is keyed on c.Fiber().IP(), and that is the TCP peer: zip's
fiber.Config sets no ProxyHeader and no trusted proxy, and fiber only reads a
forwarding header when both are set. For proxied public traffic — the only
traffic the limiter exists to bound, as EdgeRateLimit's own scope rule attests by
treating a request with no X-Forwarded-For as an in-cluster caller — every
submission therefore shares ONE 20/min bucket, along with the three staff
application routes registered after it. One host can spend the whole budget.
Stated once, at intakeRateLimit, including why the one-line fix is wrong:
middleware.RateLimit's bucket map is only ever reset, never evicted, so keying it
on real client IPs grows without bound, which is why EdgeRateLimit carries its
own eviction rather than reusing that primitive. Closing it moves when callers
see 429 — a metering decision, not a typing one — so typing left the wire alone
and the defect is reported, not silently changed.
No route changed. Tests, vet and per-package zipdoc -check all green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/team was already at its typable floor — 9 typed ops, 10 refusals gated by
untypedByDesign. Re-verified that floor against zip v1.18.8 (the newest tag,
newer than the v1.18.6/7 the note was written against), because "the floor is 9"
is a claim about a DEPENDENCY and expires when the dependency moves. All three
blocking capabilities are still absent: WithStatus still panics outside 2xx
(typed.go:110), the REST arm still ends in c.JSON(out) with no bytes or upgrade
path, and op.invoke still unmarshals any non-empty body into In before the
handler and answers 400 on a parse failure. v1.18.8's delta is app.ops ->
app.registry plus new ask/declare/ops/peer/tenant files; none of it touches the
three. No team route can be typed without moving its wire, so none was.
What the op-level count did NOT measure is the field surface. team published
three bare properties — ProviderInfo.name, ProviderInfo.displayName and
botMember.active — each reaching openapi.yaml, all four generated SDKs and the
MCP inputSchema with no description, for the crm reason exactly: op prose and
field prose live in different places and only the op one was counted.
botMember.active is the one that cost a reader something real. It is not the
agent's own flag but a DERIVED projection (botActive: empty/"active"/"ready" are
live, archived and retired are not), so a caller could see a boolean and nowhere
that a retired agent stays in the roster as an inactive member with its
authorship intact.
TestEveryPublishedFieldIsDescribed now gates team the way it gates crm, and it
was proven to bite by stripping the active prose and watching it name
botMember.active.
Wire preserved exactly: the change is doc comments only. plugin/team/openapi.json
gains 3 lines, all of them "description" keys, and openapi.yaml gains the same 3
descriptions — no status, path, operationId, field name or schema shape moves.
Subset regenerates byte-identical on a second run.
Gates: make -C apps/team test green (baseline was green), vet clean, zipdoc
-check clean, openapi weave and manifest router oracle green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/framework was already 17 of 19 typed. The two document writes (POST
/v1/framework/:doctype, PUT /v1/framework/:doctype/:name) stay raw, and the
reason re-verifies: their body IS the document's own field data, an open object
the DocType defines at run time, and typing it needs THREE zip properties none
of which shipped in v1.18.6 (the pin) or v1.18.8 (the newest tag) — read out of
the source, not taken from the note:
1. DECLARE an open object — schemaOf has no reflect.Interface case, so
map[string]any's element falls to the default (openapi.go:486).
2. BIND the URL onto one — bindURL returns early on a non-struct In
(typed.go:152).
3. Carry the params OUTSIDE the body namespace — op.invoke gets no path map
off the REST path, and for a prompt-named DocType the create body's `name`
IS the document's name (framework ops.go:321 stringField(in,"name") →
doctype naming.go:168 ResolveName).
Leg 3 was the one nothing read: it was cited as prose, so the day the engine
stopped naming a document from its body the refusal would have outlived its
cause silently. It now asserts the fact over the live wire — define a
prompt-named DocType, create with a body `name`, and the document must carry
that name. Verified to BITE: flipping the fixture's autoname to "hash" turns it
red.
Also records two things that were assumed rather than measured:
- What the refusal COSTS. The two writes DO reach openapi.yaml — as route-only
entries with path parameters and no requestBody, no responses, no prose. So
the choice is not "typed and broken vs. absent", it is "a schema that lies
about the body vs. no schema at all", and only the second stops being wrong
by itself once the capability lands.
- Leg 1 is a LIVE spec defect, not only a blocker. docView is map[string]any
and is already the Out of four typed ops (get/submit/cancel a document, and
the list's items), so openapi.yaml currently tells every SDK and every agent
that each field of a returned document is a JSON object. It is not. The fix
is one `case reflect.Interface` in zip's schemaOf returning `{}`; it needs a
zip release, so it is not made here.
No route, handler, status or body changed — the wire is untouched, and
plugin/framework/openapi.json regenerates byte-identical.
Gate: go test -tags sqlite_fts5 ./apps/framework/... ok · go vet clean ·
zipdoc -check clean · make openapi (framework subset) no diff.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/account is already fully typed — 11 typed ops, 7 raw. Re-verifying the
partition (not asserting it) turned up one thing that was actually wrong, and
it was in the prose that IS the product surface for the raw seven.
`verbatimForward` is the reason string recorded against all seven untyped
operations, and those seven publish with no description and no schema, so this
constant is the only explanation a future engineer or a route-typing agent
gets for why they stay raw. It claimed "the body is forwarded as received at
any content type". Two other places in the same tree state that claim was
REMOVED: TestUntypedByDesignForwardsVerbatim's own comment says
"verbatimForward no longer claims 'at any content type'", and LLM.md says the
audit "refuted 'forwarded as received, at any content type'". It never was
removed — the literal still said it, three lines above an assertion that
proves it false (commerceDo rewrites the request Content-Type to
application/json; the test pins exactly that).
So the reason now says what the test proves: the BYTES reach commerce as
received whatever their content type, only their DECLARED type is rewritten.
The refusal itself is untouched and still decisive — status passthrough alone
(`c.Bytes(status, raw)`, a 402 spend cap, a PDF at invoices/{}/pdf) is
something a typed dispatch cannot express, since it ends in c.JSON under the
one status the op declared and WithStatus panics on a non-2xx.
No wire change: a test-file constant nothing asserts on, plus a doc row.
LLM.md's tranche row said "account 19" while its body says "11 of 18"; the row
now carries the correction the ingress row already does, and names the closed
list that keeps an eighth raw route red.
Verified: zipdoc -check green (no lifted prose moved, zipdoc_gen.go
unchanged), go vet clean, apps/account green before and after,
TestEveryRouteIsTypedOrNamed passing — which is the machine-checked proof
there is no untyped account operation beyond the seven named.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Five of the six routes apps/guide leaves untyped already carry a test that
pins the wire fact behind the refusal — TestDocumentPutsAcceptYAML for the
two YAML-or-JSON document PUTs, TestHTTPTransitionsAndGating for the
structured 409 on the gated pair, TestDoStreamsSSE for the /do stream. The
sixth did not: PATCH /v1/guide/blueprint/:collection/:id claims its body is
a JSON merge-patch whose explicit nulls clear a key, and nothing held that
claim.
An unpinned refusal is a claim that can rot into a stale one, which is worse
than no claim: the next reader takes it on faith. So verify it and hold it.
The behaviour is exactly as documented — mergeItemPatch overlays the patch
keys onto the marshalled item and decodes into a fresh value, so an ABSENT
key changes nothing and an EXPLICIT null leaves the field at its zero, and a
nil Enabled reads as ENABLED (absence == on). `{}` keeps a disable in place;
`{"enabled": null}` lifts it.
That pair is precisely why the route cannot be typed: encoding/json decodes
both `{}` and `{"enabled": null}` into a nil *bool, so a typed In collapses
"re-enable" and "change nothing" into one request. Confirmed against zip
v1.18.6, whose typed decode is jsonenc.Unmarshal and whose only OpOptions
are WithSummary/WithTags/WithOperationID/WithStatus — no raw-document In, no
multi-status.
Test only. No route, no doc comment, no generated artifact moves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
THE THIRD MIRROR, and the same dead check. isMachinePrincipal read
`type == "application"`, and the IAM line this cloud runs against stamps that
value NOWHERE — `tokenType` takes exactly "access-token" and "id-token"
(internal/oidc/jwt.go), and the object/token_oauth.go the comment cited is not in
it. So the check could not fire, every machine fell through to the KMS-audience
clause, and that clause matches ONE identity in the estate.
A generic admin-org client_credentials token therefore read as a HUMAN and took
the SuperAdmin arm: cross-tenant reads, plus the org-switch that decides which
ledger pays. The repo's OWN red-team probe reproduces it —
TestRedIso_C_AdminCrossOrg returned 200 with s3kr3t-of-maxpower — and it passed
all along because the harness minted its machine fixture with `type`, the claim
IAM does not emit, while defaulting nil `orgs` to [owner]. The fixture described a
token that cannot exist; production had the other one.
THE ACTUAL DEFECT was one predicate answering two questions whose fail-closed
directions are OPPOSITE:
"may this hold an admin scope?" GRANTS the only cross-tenant scope, so an
unidentifiable principal must be REFUSED →
needs a positive HUMAN test.
"which org is this in?" GRANTS an org out of the app-selected `owner`
claim, so it needs a positive MACHINE test.
Splitting them is the fix, not just changing the body. isHuman = a membership set
is present (IAM signs one for every user token — store.MemberOrgRefs opens with
the home org — and none for a machine, because "a machine token has no user and
therefore no membership set"). The org question keys on isKMSMachinePrincipal, the
owner-bound audience that actually vouches for a machine.
ONE RULE FALLS OUT, with no third case to get wrong: a token carrying no
membership set never has an org read out of `owner` unless that audience vouches
for it. A generic machine and a human token minted before the `orgs` claim are
indistinguishable, and now they are treated identically — refused. That is the
rule TestLegacyOrgsClaimFailsClosed already asserted for one of them; keying the
first version of this fix on absence alone let the machine branch swallow it and
hand a legacy human the app's org, which those tests caught.
FAIL-CLOSED, and it costs something: a human token with no `orgs` loses the two
admin scopes and its org. That is an availability cost bounded by the token TTL,
taken over admitting an unidentifiable principal to a cross-tenant scope.
Falsified: restoring `claims.Type == "application"` turns TestSanitizeIdentity and
TestRedIso_C_AdminCrossOrg red. Full suite green — 174 packages, CGO_ENABLED=0
-tags sqlite_fts5 with the dev KMS key, and the failing-package set is byte-identical
to the clean tree.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/automations is typed 14/18 (8fa5ff0e), with four routes excluded and each
fact pinned (1e50cb87). This re-derives all four from zip's source and PROVES the
pins, and two of them did not hold: the retyping a reader would actually reach
for leaves them GREEN while destroying the wire.
The escape hatch neither pin covered is the non-struct In. bindURL (zip typed.go)
returns early unless the In's kind is Struct, and unmatched names are silently
ignored — so an In CAN describe an open or arbitrary body (map[string]any, any),
and the moment it does it stops receiving path params. An In can describe the
body or be addressed by the URL. Not both. hooks and resume need both.
resume In = any takes 42, "hi", [1,2], true, null — and never sees :id, so
GetRun is asked for "" and EVERY resume 404s. The pin compared the
payloads to EACH OTHER, so a uniform 404 passed. Verified: retyped,
pin green, addressing gone. Now pins that the seeded run and an
unknown one answer differently.
hooks TWO ways to break it, and the loud one was the one pinned. A struct
In 400s {"source":42} — but a payload key it has no field for is not
an error, it is DISCARDED: 200, matched:1, and {{trigger.msg}}
arrives EMPTY. Every webhook keeps "working" while every payload is
blank. Verified by retyping: the whole suite stayed green except the
raw-byte dedupe test; the payload loss was invisible. A map In takes
the open body and loses :source/:event, so nothing matches — and the
pin read the body without asserting the count, so that passed too.
Now pins DELIVERY: matched == 1, and the flow receives the payload
verbatim including the colliding key.
The other two exclusions verified sound, one for a deeper reason than recorded:
mcp closed BELOW zip. The decoder is stdlib encoding/json, which
validates the whole input before dispatching to any UnmarshalJSON,
so an In of json.RawMessage and an In whose UnmarshalJSON never
fails both still answer 400. A syntax error is unreachable from Go,
so -32700-at-200 cannot be recovered by any In type.
operations one Out, two shapes. The union reason checks out — Flow's
externalId/folderId/publishedVersionId carry no omitempty and are
emitted unconditionally, so the omitempty a union needs would delete
them from the flow branch. `Out any` is not a conversion either: it
publishes {"type":"object"}, which is what the untyped route already
offers, and spends the route's one chance to be described. Waits on
a response-per-outcome declaration in zip.
Also: the package's surface table enumerated 18 routes while the prefix serves
NINETEEN — connectorruntime is sub-mounted at automations.Mount and contributes
POST /v1/automations/connectors/{id}/run, which the published subset has and the
table did not. It is typed there; the table now says so.
Wire, prose and published document unchanged: zipdoc_gen.go and
plugin/automations/openapi.json regenerate byte-identical (rerun, not assumed) —
every comment edited belongs to an untyped handler, which zipdoc does not lift.
Converted zero, because zero were convertible without moving the wire.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/captable had 11 typed ops (the collection reads) and 20 raw relays. All 20
refusals rested on one claim: a /v1/captable write relays the goja bundle's own
(status, body), including four envelopes cloud has no vocabulary for — 400
{success,message,errors}, 404/409 {success,message}, and the top-level catch's 500
— so a typed op whose error path renders zip's {status,code,error} cannot express
them without moving the wire.
That half is true and now solved rather than avoided. bundleErr carries the
bundle's status and its BYTES as a Go error; bundleEnvelope, a group middleware
registered beside cloud.Bridge, writes them back untouched. A reachable non-2xx is
no longer a reason for a captable route to stay raw. bundleErr.Unwrap yields a
*zip.HTTPError, so OFF the HTTP path — an MCP tools/call, an in-process CLI invoke,
neither of which passes the middleware — the answer is still the bundle's status and
message rather than a blanket 500.
The REAL blocker is on the request, and it is why 14 routes are still raw. The
bundle validates with COERCING helpers (goja/src/validate.ts): `num` accepts a
number OR a numeric string (z.coerce.number), `optString` accepts any scalar and
calls String(v), and addStakeholders accepts a single object OR an array. zip
decodes a typed In with encoding/json, which answers 400 "invalid body" to every
one of those — so typing would make the route accept LESS. Each of the 14 now names
the field that does it instead of citing the response.
That splits the surface on a line that is checkable, not a matter of taste: a route
with NO REQUEST BODY has nothing to coerce, so its In is faithful by construction.
There are exactly six such routes left, and they are the six typed here:
GET /v1/captable/rounds/:id round + its cheques
DELETE /v1/captable/stakeholders/:id
DELETE /v1/captable/shares/:id
DELETE /v1/captable/options/:id
DELETE /v1/captable/safes/:id
DELETE /v1/captable/convertibles/:id
17 of 31 operations typed and described, up from 11. plugin/captable/openapi.json
and openapi.yaml gain 160 and 148 lines of schema and prose and lose NOTHING: 21
captable paths and 31 captable operations before, 21 and 31 after, zero removals,
zero additions.
WIRE, PROVEN BY TEST — bodyless_test.go re-derives the pre-typing answer on every
run (a direct Dispatch of the same bundle route with the same params IS what the
raw relay wrote) instead of trusting a recorded golden, and compares status,
Content-Type and body bytes on BOTH arms:
- 2xx, through zip's typed JSON writer: the round detail is byte-identical on the
closed PRICED round (closeDate/pricePerShare/preMoneyValuation/shareClassId all
set, one cheque with a comment) and on the OPEN SAFE round (all four null, no
cheques); each delete's {"success":true} is compared against a direct-dispatch
delete of the matching row in a second identically-seeded tenant.
- non-2xx, through bundleErr: every 404, plus the 400 that MOTIVATES the whole
mechanism — refusing to orphan issued equity carries an `errors` LIST, and
zip's envelope has nowhere to put it. The list survives, byte for byte, under
the same bare `application/json` the relay sent.
Both proofs were watched go RED: stubbing errors.As out of bundleEnvelope fails
three tests, and swapping two fields of captableRoundDetail fails the byte
comparison.
DELETE takes its input from the URL and carries no body — zip's hasBody rule, which
the document reads too — so the five deletes publish a path parameter and no
requestBody, which is what the raw routes already did (readBody=false).
ONE LATENT DEFECT, FIXED. The typed reads' non-2xx arm was NOT unreachable. Their
read() turned any non-2xx into zip's own 500 "captable dispatch failed", and the
bundle's top-level catch answers 500 {success,message} on a SQL error — so a read
that hit one had silently moved from the bundle's envelope (with its message) to
cloud's (without it) when the reads were typed. read() is gone; every op now goes
through one call(), so that arm relays the bundle's bytes again, as it did before
typing. Unreachable-by-construction arms (getCompany/capTable's defensive
notFound) are unaffected.
Registration order is unchanged in effect: no two /v1/captable routes overlap on
method + pattern, so moving the six into the typed block cannot shadow anything.
Gate: make -C apps/captable {test,vet} green (baseline was green), 12 tests, all
pass; zipdoc regenerated and idempotent; ./openapi (the weave) and ./manifest
green; apps/{esign,goja,company} — the other goja-bundle leaves — still green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/compliance/health is the only route on this surface that does not read a
tenant, and it had no test at all. It is also the route the group's newly
installed cloud.Bridge could most easily have broken: Bridge is what parks the
validated org for every other typed op, and had it REFUSED a request carrying no
org, installing it in front of the leaves would have turned liveness into a 403 —
the failure mode where a subsystem reports itself down to every prober that
correctly sends no tenant header.
Bridge is fail-open by construction (it parks what it has and continues), so the
route was in fact fine; this is the assertion that keeps it that way, and it
measures the answer rather than asserting the middleware's shape.
Verified: GET /v1/compliance/health with no X-Org-Id and no X-User-Id answers 200
{"status":"ok","provider":"manual"}.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Went to type apps/git's untyped routes and found the partition already COMPLETE
and already a gate (3c9dd462): 24 typed ops, 24 refusals in untypedByDesign, and
TestEveryRouteIsTypedOrNamed/TestEveryTypedOpIsDescribed pinning both halves.
Re-verified rather than trusted — every refusal read against its handler, not its
prose — and all four families hold: HMAC-over-raw-bytes (webhook), pack streams
(smart-HTTP, both hosts), text/html (the browser UI, both hosts), and the ZAP
envelope whose failure body is {status:"error", msg} where a typed op's returned
error renders zip's {status:<int>, code, error}. Zero routes were convertible
without moving a wire, so this commit converts none. 24/24 confirmed from the
committed subset: plugin/git/openapi.json is 48 operations, exactly 24 described.
What the verification DID surface is upstream of every typing task in this repo:
`hanzo.yml`'s two raw-go gates cannot build the tree they gate. hanzoai/base
v1.5.11 — pinned TODAY, da53ab30 — declares a deliberate compile error under
`cgo && !sqlite_math_functions` (base/core/sqlite_math_required.go), because its
search layer emits SQL calling acos/cos/sin/radians/sqrt that the cgo sqlite has
only behind that tag. 47 of cloud's 306 packages reach base/core (apps/git,
apps/agents, apps/billing, apps/base, … and their plugin/<app> mains). go-unit
sets CGO_ENABLED=1 explicitly; go-vet sets nothing and so inherits the toolchain
default, which is 1 wherever a C toolchain exists — and hanzoai/ci provisions one.
Measured, both directions, from this worktree:
go vet ./... -> EXIT=1, base/core
go vet -tags "sqlite_fts5 sqlite_math_functions" ./... -> EXIT=0, all 306
So the gate reported "[build failed]" where a test run was expected, and a gate
that cannot build is a gate that never ran a test — the same failure mode
hanzoai/ci's own Test step exists to refuse. Both steps now pass the tags the
Dockerfile already passes; sqlite_fts5 rides along because it is the tag the image
and `make test` carry and without it an FTS5-backed store cannot open. The root
cause is that these two steps restate the Makefile's posture instead of using it,
so LLM.md now names the floor and the count with the command to re-derive them.
Also corrected there: apps/git no longer belongs on the "fails under make test by
design" list — measured green under exactly that posture (dev key, -tags
sqlite_fts5, CGO_ENABLED=0) in 18s.
NOT patched, because it is a policy call and copying it would re-create the same
drift: go-unit still has no CLOUD_KMS_MASTER_KEY_REF, which `make test` injects
once for the whole suite, so cek-backed packages still fail there (measured:
apps/code, 6 tests). Either CI gets the dev key or the step routes through the
Makefile — one declaration, not two.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
crm was already 19 typed of 20 served, and the route census here proves it is
20/19/1 exactly (typed_wire_test.go). What the count did NOT measure is the half
of the published surface that does not come from typing a route: a typed op
documents its ADDRESS and its SHAPE, never the shape's FIELDS. Those come from
doc comments on the In/Out struct fields, and crm's REQUEST types carried them
while its RESPONSE types — Company, Contact, Opportunity, Application,
ScreenResult, StageEvent — are store ROW types nobody had written field prose on.
All 65 of their properties reached openapi.yaml, all four generated SDKs and the
MCP inputSchemas with no description at all. A reader could see that `arr` was an
integer and nowhere that it was CENTS.
Every field now carries prose derived from the code that writes it (cents,
server-owned unix seconds, the upper-cased stage vocabulary, the 422 on a
cross-org ref, the cleared-on-delete relations, the snapped credit ladder), and
the whole change is proven to be DESCRIPTION only: strip description/summary/
example from openapi.yaml before and after and the documents are byte-identical.
Zero wire movement.
The partition becomes a GATE, the way team's and git's did — prose cannot fail:
- TestEveryRouteIsTypedOrNamed — untypedByDesign is the closed list; fails on
an operation that is neither typed nor named, on a name crm no longer serves,
and on a name that IS a typed op. All three modes verified by perturbation.
- TestEveryTypedOpIsDescribed — the lifted prose reached the binary.
- TestEveryPublishedFieldIsDescribed — the new one, gating the half above.
Verified: deleting one field comment turns it red (Company.arr).
The one refusal is re-verified against zip v1.18.6's own source, and it is a
DIFFERENT gap from multi-status (#78): per-op projection SCOPE. POST
/v1/crm/applications is guarded by fiber middleware (IP rate limit) and a
pre-parse 64 KiB raw-body cap. zip's MCP arm dispatches tools/call straight into
op.invoke (mcp.go:152) and the CLI's LocalInvoke does the same (cli.go:427) —
neither runs the route's middleware — and zip has no per-op way to decline a
projection (OpOptions are WithSummary/WithTags/WithOperationID/WithStatus;
MCP.Disabled is app-wide). So typing it publishes an unmetered, uncapped alias of
the one deliberately metered public write in the surface, and apply() never calls
tenant(): it writes into intakeOrg(s), the deployment BRAND's pipeline. The alias
would let any caller reaching /mcp inject unbounded rows into the brand's own
CRM. Its 200-vs-201 split would shim and the honeypot's third body needs only
omitempty — neither is what blocks it.
LLM.md records the refusal as its own zip-gap family and the measured scale of
the field-prose class, which is fleet-wide and not a crm quirk: 1,424 of 2,716
published properties (52%) carry no description and 195 schemas are 100% bare
(appView 26, Wire 21, Node 20, Volume 20, Totals 18). crm is now 0 of 65.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The apps/account (11 of 18) paragraph carried the claim the audit refuted. It said
each of the seven forwards a body "as received (any content type)". commerceDo
(topup.go) SETS the request Content-Type to application/json whenever there is a
body, so the bytes forward and the type does not. It returns no response header at
all, which is why billing.go pins application/json over commerce's own type and
drops Content-Disposition, and it truncates the response at 1 MiB under the
upstream's own 200.
The live consequence is one route: GET /v1/billing/invoices/{id}/pdf, the only
non-JSON entry in billingForwardable, delivers PDF bytes labelled JSON with no
filename, against a commerce that sets application/pdf + attachment. Left unfixed
and recorded at the lines that cause it — the repair is commerceDo returning
response headers across three call sites, one of them the top-up money path.
The two facts that actually carry the refusal are unchanged and now stronger:
status passthrough and an unvalidated request body, both proven against the live
bridge by TestUntypedByDesignForwardsVerbatim instead of asserted in prose. The
"11 typed, 7 refused" ledger row is confirmed by the corrected per-app measure —
7 untyped registrations, 11 zip.X ops, matching what the gate reads off the router.
Docs only.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/account was already fully typed — 11 typed ops, 7 catch-all forwarders held
as a closed list — so this carries no conversion. It closes the gap that made the
refusal unfalsifiable and corrects what the refusal claimed.
The reason the seven stay raw lived only in prose: a paragraph citing zip's source
for facts about THIS package's handlers. Nothing went red if a handler stopped
behaving that way, which is the same failure the closed list was invented to fix
one level up. TestUntypedByDesignForwardsVerbatim now proves the two decisive
facts against the live bridge — commerce's 402 arrives as 402 with its bytes
intact (a typed dispatch answers the one 2xx it declared, via c.JSON), and a
text/csv request body reaches commerce byte for byte (a typed op answers
ErrBadRequest before the handler runs). Each assertion was mutation-checked: flip
the upstream status, the upstream bytes, or the forwarded body and the test fails.
Auditing that claim found it over-stated in one direction and the code wrong in
three. commerceDo is a JSON transport, not a transparent proxy: it SETS the
request Content-Type to application/json whenever there is a body, it returns no
response headers at all, and it truncates the response at 1 MiB while reporting
the upstream's own 200. So billing.go pins Content-Type: application/json over
whatever commerce sent, and drops Content-Disposition — which means
GET /v1/billing/invoices/{id}/pdf, the one non-JSON entry in billingForwardable,
delivers PDF bytes labelled JSON with no filename (verified against a stub
upstream: status=200, content-type="application/json", content-disposition="",
body="%PDF-1.4…"). Commerce sets application/pdf + attachment there
(api/billing/invoice_pdf.go).
None of the three is repaired here. Fixing them means teaching commerceDo to
return response headers — three call sites, one of them the top-up money path —
and deciding which headers are safe to relay while keeping Cache-Control:
no-store, which is a tenancy property and not a content one. That is its own
change with its own test, not a side effect of a doc pass. Each is recorded at
the line that causes it so the next reader lands on the fact, not on a summary.
Wire unchanged: billing.go and topup.go are comment-only, and the sole new code
is a test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/framework was already 17 of 19 typed; the two document writes
(POST /v1/framework/:doctype, PUT /v1/framework/:doctype/:name) stay raw. I
re-derived that refusal from scratch and reached the same three blockers the
record names, so nothing converts here. What DID need fixing is that the
reason was neither true at the registration site nor checkable anywhere.
1. The registration site named ONE blocker. framework.go's Mount still said
"zip needs an open-object input (additionalProperties: true) before these
convert" — precisely the framing 4876a56c was written to retire, because a
reader who fixes only that comes back and converts, publishing a request
schema that names the two path segments and nothing else. That commit
updated the test and LLM.md and left Mount behind, so the one place an
engineer reads before acting carried the misleading half. Mount now names
all three: DECLARE an open object, BIND the URL onto one, and carry the
bound params OUTSIDE the body namespace.
2. Nothing read the citation, so the refusal could not expire. rawRoutes cited
two properties of zip and no test observed either — the day zip ships them,
nothing goes red and two routes stay untyped forever behind a stale reason.
TestOpenObjectRefusalStillHolds now asserts both, through exported API only:
that GET one document publishes response schema
{"type":"object","additionalProperties":{"type":"object"}} (shipped and
FALSE — the package's own round-trip test reads a document back holding a
string and a number, neither of which that schema admits), and that an
open-object In receives no :doctype. Both expectations are wrong on purpose;
either going red IS the signal to convert the two writes and delete the
test. The second assertion is not a tautology: a struct In binds
:doctype=Task on the identical harness, so the map failing to is zip's
behaviour, not the probe's.
3. A rotted fact. The record claimed re-verification against "v1.18.6, the
current pin and the newest published version"; v1.18.8 has since published.
Re-verified against it: schemaOf still has no reflect.Interface case,
bindURL still returns early on a non-struct, mcp.go still invokes with a nil
path map. None of the three shipped, now stated with the version that is
actually newest.
No route, In/Out type or artifact changes: the wire is untouched, zipdoc -check
is clean, and zipdoc_gen.go / plugin/framework/openapi.json / openapi.yaml
regenerate byte-identical. go vet clean; apps/framework green (21 tests).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The comment I just landed said "14 more places", derived by subtracting one from
LLM.md's recorded 15. That is exactly the tally-from-prose mistake the playbook
warns about, and it is wrong: MEASURED over the golden with team's instance
already removed, 15 remain. So LLM.md's 15 was a count taken before more apps got
typed, and the class had already grown past it.
That is the interesting fact, not the arithmetic: an UNTYPED route contributes no
schema, so it cannot state anything false yet. Every app this migration types
converts its any-valued maps from silent to loudly wrong, which means the class
GROWS as the migration progresses and any figure written in prose is stale on
arrival. The comment now names the owners it measured — guide (JourneyStep.args,
stepView.args), pricing (seven list envelopes), admin (adminCatalogOut),
framework (documentList.data), Application.metadata, StepSettings.input,
runIn.props — and says to count, never tally.
Source-only: the rationale lives on the TYPE, which zipdoc does not lift, so
zipdoc_gen.go, plugin/team/openapi.json and openapi.yaml are byte-identical. That
asymmetry is the useful seam — a field comment IS product surface an SDK user and
an MCP client read, a type comment is for the next reader of the code.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/books was already converted (20 typed, 5 refused) and wire_test.go proves the
answers did not move. That is the safety half of the migration and on its own it
justifies nothing: leaving every handler untyped also moves no answer. The VALUE half
— that a typed op lights up OpenAPI, MCP, the CLI and the op-call plane at once — was
asserted in comments and measured by nothing.
projection_test.go measures it, over the whole surface rather than a sample:
- each of the 20 ops must reach the document WITH its doc-comment prose and a
declared response, reach MCP as a tool carrying that same prose, and reach the CLI
as a `hanzo books …` command — all under ONE operation id;
- each of the 5 exempt routes must still answer 401 (a live, fail-closed route, so
the exemption list cannot rot into naming paths that no longer exist) and must
appear in NONE of the three derived surfaces;
- the two ledgers must sum to 25, so a route added to the surface is measured by
something rather than by nothing.
The exemption ledger fails in the GOOD direction too: the day one of these becomes
typeable it names the route to move. Debt nobody is forced to look at is how an
exemption becomes permanent.
Verified non-vacuous by mutation: dropping an op from the ledger, renaming a published
CLI command, and claiming an exempt route is typed each go red.
The refusals hold, and the blocker is named precisely rather than restated:
zip v1.18.7 decodes EVERY typed body with jsonenc.Unmarshal and answers ErrBadRequest
on failure, and its whole OpOption set is WithSummary/WithTags/WithOperationID/
WithStatus — there is no octet-stream/binary request declaration. So an In on
POST /v1/books/{scan,inbox} or /v1/books/bank/import turns a working PDF/OFX upload
into a 400. Faking it with a custom UnmarshalJSON would be worse: the document would
then tell every generated SDK to send JSON to a route that eats bytes. Typing those
three is a zip change, not a books change. The other two (bank/link-token, bank/exchange)
answer 501 unconditionally — a typed op must declare a success it has never sent.
No wire change: the only source edit is a test file, so zipdoc_gen.go, openapi.yaml and
plugin/books/openapi.json all regenerate byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/team is fully typed already — 9 of its 19 operations carry a registry entry
and the other 10 are structurally untypeable at zip v1.18.6, each named in
untypedByDesign with the reason. This pass re-derived all ten against zip's own
source rather than trusting the prose, and found nothing convertible:
- op.invoke decodes a typed In before the handler runs and answers 400 on a
malformed body (typed.go:233). That is the whole reason POST /v1/team/account
(a JSON-RPC envelope whose refusal is HTTP 200 carrying {error: Status}) and
PUT /v1/team/account/cookie (which IGNORES an unparseable body and falls back
to the bearer) must stay untyped — typing either would refuse a request they
have always served.
- the typed handler ends at c.JSON(out) and the OpOption set carries no
content-type, multipart or raw-body option at all. So the two WebSocket
upgrades, the two OAuth 302s, the three byte-serving routes (wallet page,
blob download) and the multipart upload cannot be ops, not merely have not
been made into ops.
What the re-derivation DID surface is a live instance of the any-valued-map
class: statsOut.metrics was a map[string]any, and zip's schemaOf has no
reflect.Interface case, so the element type fell to the default and the document
asserted `additionalProperties: {"type": "object"}` — that every value in the map
is a JSON object — for a map that has never held one. This is worse than the
under-describing gaps (a bodyless POST, an undeclarable second success status):
those state less than the truth, this one states something false, and an SDK
regenerated from the golden typed the field Dict[str, Dict] when the only value
it ever carries is {}.
The field is now an ANONYMOUS empty struct, which is what the wire IS —
`"metrics":{}` on every response, pinned byte-for-byte by
TestTypedStatisticsServesBothPaths, so the change is provably description-only.
Anonymous because there is no value to name, keeping the honest shape out of the
fleet's flat schema namespace.
The class is zip-side and wider than this field: openapi.yaml carries the same
false claim in 14 more places. The one-line fix is a reflect.Interface case in
schemaOf projecting the OPEN schema `true` — an unconstrained element is open,
not an object. Recorded at the field so the next reader has the whole fact.
Gates: apps/team green (baseline was green, identical after), openapi weave green
against the regenerated golden, manifest + root green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/ingress was dispatched for typing and had nothing left to type: 18 routes
before the conversion (7be3160d), 18 typed ops after, 0 dropped, all 18 published
with prose in plugin/ingress/openapi.json AND openapi.yaml, zipdoc current, the
subset regenerating byte-identical from source, and TestSurfaceIsRegistered
gating the surface as an exact set.
What sent an agent there is the playbook's own re-measure command, which had no
path anchor and so counted r.Header.Get("X-Forwarded-Proto") in middleware.go as
an untyped route. A route is a VERB PLUS A PATH; the measure has to say both.
Two half-right commands lived in this file, disagreeing by 83 routes:
- no path anchor -> 83 phantoms across apps/ (integrations read 45 for 19,
platform 32 for 30, tools 18 for 16, ingress 1 for 0) — the same miscount
already documented for team, rediscovered because the command was never
fixed;
- anchored on ("/ alone -> drops the 7 real EMPTY-leaf registrations at a
collection root (prefs 2, webhooks 2, share, crawl, destinations).
So: one command, anchored on a path — ("/ or ("" — with the // filter that the
slash-only form already carried (11 of the empty-leaf hits are comments quoting
the form). Corrected, apps/ holds 666 untyped route registrations. The
independent check on the number is integrations: the corrected measure
reproduces its 19 exactly, which is the count its own conversion recorded as
refusals.
No source, no wire and no generated artifact changes here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/authz v1.10.15 deleted the Casbin enforcer and split the decision from
the edge, so Mount moved to authz/serve. This repo held the estate's only
consumer, and it used nothing but Mount.
The adapter lives HERE. authz/serve takes a logger because it is a leaf that must
never import cloud; cloud's Plugin contract wants Deps. Bending the plugin to the
leaf keeps the dependency pointing one way — a leaf that learned about Deps would
stop being importable by anything that cannot link cloud, which is the whole
property being bought.
Full suite: 173 packages, exit 0. The security probes and the money gate pass —
TestRedIso, TestAudit_AnonRequest, TestGate_*, and
TestResourceMeter_UnconfiguredIsNoop, which together hold that a forged identity
reads nothing and an unreachable biller refuses rather than allows.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/pricing had 15 typed ops and 17 raw routes. Fifteen of those refusals rested
on one claim: the /v1/pricing/{compute,cloud/*,subscriptions,blockchain,iam,base,
paas,policy,tools,gpu} routes and /v1/pricing-policy are byte-for-byte proxies of
the @hanzo/pricing goja bundle, so a typed op cannot express them without a Go
re-marshal that reorders keys.
The claim was wrong, and checkable. apps/goja re-marshals the bundle's answer with
Go's encoding/json before any handler sees it — Host.DispatchWith ends in
`json.Marshal(m["body"])` over the exported map. What the raw pass-through wrote
was never the JS engine's bytes; it was Go's, keys already sorted. Decoding those
bytes into an Out and re-marshalling them is therefore the identity function.
So they are ops now: 30 of 32 operations typed, 30 described, up from 15/15.
Each is ONE registry entry the OpenAPI operation, the MCP tool, the CLI command
and the generated SDK method all follow from. plugin/pricing/openapi.json and
openapi.yaml gain 276 lines of schema and prose and lose nothing — 1011 paths
before, 1011 after, zero removals.
WIRE, VERIFIED. A probe drove all 15 addresses under four identity shapes
(anonymous, member, SuperAdmin, and the forged X-Org-Id with no principal the
enablement attack tests pin), before and after, comparing status + Content-Type +
body sha256. All 60 body hashes identical. All 60 statuses identical. The bodies
do not vary with the caller either, which is the property that makes these
sections and not catalog reads.
The whole delta is one header, and it is a normalisation:
Content-Type: application/json -> application/json; charset=utf-8
fiber's typed JSON writer sets the charset form; the raw pass-through set the bare
one. Every OTHER answer on this surface — every typed op, every zip error, the 403s
in admin.go and enablement.go — already sent the charset form, so this ends a split
inside one subsystem rather than starting one.
The 503 arm is preserved as a status and pinned as a test. A section the catalog
does not hold still answers 503 with the bundle's own message, because the status
comes from the bundle (dispatchErr), not from a declaration. Its BODY moves from
the bundle's {"error":…} to zip's {"status":503,"error":…} — same status, same
message, one added field, the same shape this surface's every other error already
had. The shipped catalog cannot reach that arm, so
TestSectionsDegradeWithTheBundlesStatus strips the sections and drives it.
The proof is a test, not this message. sections_wire_test.go re-derives the
pre-typing answer on every run (rawDispatch is exactly what the raw route wrote)
instead of trusting a recorded golden, and a companion test fails if mountSections
declares an address the proof does not cover.
TWO RAW LEFT, both halves of the admin overlay PATCH, both wire-bound:
- PATCH /v1/admin/catalog/models/* addresses a model id that may contain '/',
so it routes through a greedy wildcard. fiber's runtime name for it is `*1`;
the document's is `{wildcard1}` (openapi.translate, because `*1` is not a legal
URI-template name). An In field can bind one or publish the other, never both.
- PATCH /v1/admin/catalog/providers/:name carries `overrides`, an RFC 7386 merge
patch STORED AND ECHOED VERBATIM. json.RawMessage publishes as an array of
integers (schemaOf takes the Slice arm — it is []byte); map[string]any
re-marshals and sorts the keys, so the overlay this route echoes and the one
GET /v1/admin/catalog echoes under "_overlay" would come back reordered.
Its old stated reason was also wrong and is corrected: it claimed retyping to
map[string]any would move {"overrides":null} from "clear the override" to "leave it
alone". encoding/json already sets a *json.RawMessage field to nil on a JSON null
(indirect breaks on the first settable pointer when decodingNull), so null has
ALWAYS meant "leave it alone" here — normalizeOverride's "null" branch is dead for
the literal. A refusal justified by a mechanism that does not exist is a refusal
nobody can re-check.
dispatch() and passthrough() are gone with their last caller: the read path no
longer touches *zip.Ctx at all.
Gate: make -C apps/pricing test green (baseline was green), vet clean, zipdoc
-check clean, openapi weave green, ./openapi and ./manifest green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/automations is typed 14/18 (8fa5ff0e). The remaining four each name a wire
fact a typed op cannot carry, but the reasons lived only in prose — so a later
reader could retype any of them, watch the suite stay green, and ship a silent
wire change. Three of the four break on inputs no existing test sends.
untyped_wire_test.go pins the facts, so the exclusion is enforced rather than
asserted. Each test fails the moment its route becomes a typed op and names the
fact that was lost:
mcp an unparseable body is a JSON-RPC result, not a transport failure:
HTTP 200 carrying -32700. A typed op answers 400 with no envelope.
resume the payload is an arbitrary JSON value; 42, "hi", [1,2] and true
are legal today and are 400s under any struct In.
hooks the body is open-keyed while :source/:event bind by NAME, so a
typed In must own fields source and event — and {"source": 42} is
a legal event today, `invalid body:` 400 when typed.
operations TWO body shapes on one route and one status: the Flow on
CHANGE_STATUS, the FlowVersion otherwise. Verified disjoint on
their discriminators, so one Out cannot be both.
Each pin was proved to bite: temporarily retyping resume and hooks turns the
corresponding test red, while every pre-existing test in the package stays
green.
That exercise also corrected the hooks reason, which was misleading in a way
that invited the break. It cited the raw-byte dedupe hash, the size gate and two
headers — all four of which ARE reachable from a typed op via cloud.Request(ctx),
so a reader who recovered them would believe the route was now typeable. The
decisive fact is the path-param/body key collision, and zip returns its 400
before the handler, so nothing inside the handler can recover it. The comment now
separates the blocking fact from the recoverable ones and says why.
Wire unchanged: comments and one new test file. No route registration moved, so
zipdoc_gen.go, openapi.yaml and plugin/automations/openapi.json regenerate
byte-identical (verified by rerunning the generators, not assumed).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/compliance was already 16 of 17 typed. The 17th — the provider webhook —
was refused in a comment, which is the weakest form a decision can take: prose
does not fail, so nothing stops the 18th route from landing untyped, and nothing
notices when the reason stops being true.
Make it a GATE. untypedByDesign is the closed list of compliance operations
that are not typed ops, and TestEveryRouteIsTypedOrNamed fails on any route
that is neither typed nor named there — so the next route added here is typed by
default, and dropping one out of the registry takes a deliberate edit with a
reason. It reads the LIVE router (openapi.Spec for what is served, openapi.Typed
for what carries a registry entry), never the source, so a route added anywhere
in routes() surfaces whether or not anyone remembers the list. Its second arm
fails on an entry naming a route the app no longer serves, so the refusal list
cannot rot into stale prose. TestEveryTypedOpIsDescribed holds the lifted
prose to the same bar as the schema: that prose IS the OpenAPI description AND
the MCP tool description a model reads to pick the tool, so an op added without
regenerating zipdoc shows up as a nameless tool rather than shipping as one.
Both arms were proven to bite before being trusted — emptied, the list named
POST /v1/compliance/verifications/webhook; given a route that does not exist,
it named the staleness. A gate nobody has watched fail is not known to run.
The refusal itself is re-verified against zip v1.18.6's own source rather than
against the comment claiming it, because "cannot be typed" is a claim about a
dependency and a dependency moves. Two independent wire facts: the HMAC covers
the EXACT received bytes (apps/idv/webhook.go Verify: mac.Write(body)) which zip
has already unmarshaled into In before the handler runs (typed.go:236), so a
re-encoded In is not the signed value; and the route answers TWO 200 shapes (the
reconciled check, or {"ignored": …} for a reference it does not know) where an op
declares exactly one Out — unioning them would add zero-valued fields to the
no-op body, which is a wire change, not a description. Re-check when zip gains
raw-body binding or multi-status responses.
routePrefix hoists /v1/compliance out of the four places that spelled it — the
group and bodyCap's three exact paths — so the router and the body gate cannot
disagree about where a route lives. The package is gofmt-clean again (the
import order drifted in the clients/ -> apps/ move).
No route, method, In, Out or lifted comment changed, so the published document
is byte-identical: this commit describes the surface, it does not move it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/captable relays a goja bundle's own (status, body). The eleven READS end in
okRes on every reachable path, so 200 is the only answer and the body is a shape
Go can state: they become zip typed ops and reach the document, the MCP tool
list, the CLI and the generated SDKs — 11 operations that carried a path and a
method and nothing else now carry a schema and prose.
The twenty WRITES stay untyped, each with the reason written at its registration.
The shared one is not effort, it is the wire: the bundle authors its own error
envelope — 400 {success,message,errors}, 404/409 {success,message} — and a typed
op's failure path can only render zip's {status,code,error}. Typing one would
change what every existing client parses on every validation failure, so it does
not get typed. Two also read the body in ways a Go struct cannot state (a single
object OR an array; `quantity` OMITTED meaning "the whole certificate", which a
zero value cannot say). That is the same class as multi-status: a contract detail
zip has no vocabulary for yet.
FIELD ORDER IS LOAD-BEARING, once. The bundle's rows cross goja as
map[string]any and are serialised by encoding/json, which sorts object keys, so
every model declares its fields in alphabetical json-tag order and the typed
response is BYTE-identical to the relay it replaces — not merely equal as JSON.
Nullability is the DDL's: a nullable column is a pointer, so null stays null.
TestTypedReadsAreByteIdenticalToTheBundle pins that against the bundle's own
bytes on an empty tenant AND on one written through the untyped relays with every
nullable column exercised both ways.
cloud.Bridge goes on the group: a typed op receives only a context, so the
validated org reaches it by being parked there, never as an In field — an In field
is caller-supplied, so a tenant key read from one is a cross-tenant read the
caller asserted for itself. TestTypedReadsAreOrgScoped proves the typed plane
refuses byte-for-byte as the untyped one does and that one tenant never sees
another's rows.
Verified beyond the suite: the concrete route table is unchanged (same 31
method+path pairs, nothing added, nothing lost), the bare prefix still 404s, and
a 110-line dump of every route's answer — success and every error branch, anon
refusals, cross-tenant reads — is byte-identical between this tree and main
across three runs each.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Almost every failure tonight had the same shape, not the same cause: one fact
written down twice, with nothing forcing the copies to agree, and each copy
locally correct so the disagreement was silent.
billing_account on Claims, but every spend site holds a *User
balanceReader an in-process hook OR an HTTP fallback, chosen by a process
topology neither half knows about
audit_log.seq one chain, N in-memory counters
image.tag and the probe port: which port serves health, said twice
72 App CRs two owners
GIT_CUSTOM per-container, and the main one was missed
an app's paths manifest/apps.go AND plugin/<app>/main.go
The last one is what took inference down on v1.801.318/.319: ai's row read
"/v1/ai" while its router served /v1/chat/completions and /v1/models at top
level. Both halves were reasonable; only the pair was wrong, and nothing examined
the pair. 405/404 fleet-wide, every pod Ready, UI serving 200.
Four plugins restated their prefixes. All four AGREED when I checked — and that
is the defect, not the reassurance: the copies live in different files, move in
different changes, and the next disagreement is as invisible as the last. So the
host's list is THE list and an app reads it (manifest.PrefixesFor). iam's and
pricing's package-level Prefixes vars now have zero consumers; zen's and tools'
literals are gone.
Two guards, and I had to fix each after it lied to me:
- TestNoPluginRestatesItsPrefixes — verified by re-planting a literal in
plugin/zen and watching it fail.
- TestEveryPluginNameIsInTheManifest — PrefixesFor returns nil for an unknown
name, which zip reads as "claims nothing", so an absent row must be loud.
Its tool-exemption is DERIVED ("does it call cloud.Serve") rather than a name
list that would need editing forever and would eventually hide a real app.
Deriving it by SUBSTRING read the scaffold generator as an app, because
gen-app-cmds contains "cloud.Serve" inside the template it emits — so it
parses for a call expression instead. Matching text finds the mention; only
parsing finds the call.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Restores work that was lost: the original commit swept another session's
in-flight manifest changes in with `git add -A` in this shared worktree, and
went away when that session reset. Same change, staged by explicit path.
The brand's skills are generated from the OpenAPI source of truth and embedded,
so changing them is a rebuild and a redeploy. That is right for the catalogue a
deployment ships and wrong for the one an org writes. POST /v1/skills stores an
org's own skill and it is listed immediately.
Same split clients/templates already keeps: a PUBLIC embedded catalogue with no
write route, and a PRIVATE per-org store whose every read binds org. A private
skill has no path onto the public discovery surface by CONSTRUCTION — different
containers — not by a filter someone has to remember to write.
Two providers share SourceSkill deliberately. The registry dedups by NAME with
equal-rank ties going to whoever registered first, and clients/agentskills
mounts at order 8 against this at 123, so a brand skill always wins a collision
with an org's. The shipped catalogue is the one that cannot be shadowed.
The write surface lives here rather than in clients/agentskills because that
subsystem mounts BEFORE iam to win the /.well-known discovery routes, so it has
no validated principal to scope a store by.
Co-Authored-By: Hanzo Dev <dev@hanzo.ai>
Almost every failure tonight had the same shape, not the same cause: one fact
written down twice, with nothing forcing the copies to agree, and each copy
locally correct so the disagreement was silent.
billing_account on Claims, but every spend site holds a *User
balanceReader an in-process hook OR an HTTP fallback, chosen by a process
topology neither half knows about
audit_log.seq one chain, N in-memory counters
image.tag and the probe port: which port serves health, said twice
72 App CRs two owners
GIT_CUSTOM per-container, and the main one was missed
an app's paths manifest/apps.go AND plugin/<app>/main.go
The last one is what took inference down on v1.801.318/.319: ai's row read
"/v1/ai" while its router served /v1/chat/completions and /v1/models at top
level. Both halves were reasonable; only the pair was wrong, and nothing examined
the pair. 405/404 fleet-wide, every pod Ready, UI serving 200.
Four plugins restated their prefixes. All four AGREED when I checked — and that
is the defect, not the reassurance: the copies live in different files, move in
different changes, and the next disagreement is as invisible as the last. So the
host's list is THE list and an app reads it (manifest.PrefixesFor). iam's and
pricing's package-level Prefixes vars now have zero consumers; zen's and tools'
literals are gone.
Two guards, and I had to fix each after it lied to me:
- TestNoPluginRestatesItsPrefixes — verified by re-planting a literal in
plugin/zen and watching it fail.
- TestEveryPluginNameIsInTheManifest — PrefixesFor returns nil for an unknown
name, which zip reads as "claims nothing", so an absent row must be loud.
Its tool-exemption is DERIVED ("does it call cloud.Serve") rather than a name
list that would need editing forever and would eventually hide a real app.
Deriving it by SUBSTRING read the scaffold generator as an app, because
gen-app-cmds contains "cloud.Serve" inside the template it emits — so it
parses for a call expression instead. Matching text finds the mention; only
parsing finds the call.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Restores work that was lost: the original commit swept another session's
in-flight manifest changes in with `git add -A` in this shared worktree, and
went away when that session reset. Same change, staged by explicit path.
The brand's skills are generated from the OpenAPI source of truth and embedded,
so changing them is a rebuild and a redeploy. That is right for the catalogue a
deployment ships and wrong for the one an org writes. POST /v1/skills stores an
org's own skill and it is listed immediately.
Same split clients/templates already keeps: a PUBLIC embedded catalogue with no
write route, and a PRIVATE per-org store whose every read binds org. A private
skill has no path onto the public discovery surface by CONSTRUCTION — different
containers — not by a filter someone has to remember to write.
Two providers share SourceSkill deliberately. The registry dedups by NAME with
equal-rank ties going to whoever registered first, and clients/agentskills
mounts at order 8 against this at 123, so a brand skill always wins a collision
with an org's. The shipped catalogue is the one that cannot be shadowed.
The write surface lives here rather than in clients/agentskills because that
subsystem mounts BEFORE iam to win the /.well-known discovery routes, so it has
no validated principal to scope a store by.
Co-Authored-By: Hanzo Dev <dev@hanzo.ai>
The playbook's nine failure modes each cost somebody a discovery, and this one
is larger than any of them: 56% of the fleet's described operations publish a
`summary` with a line break in it, across 18 packages, and nothing had counted
it because nothing reads the summary looking for one.
firstSentence (zip/openapi.go:606) returns the text up to the first ". "
verbatim — no whitespace collapse — so a doc comment whose opening sentence wraps
in the Go source ships that wrap into the OpenAPI summary, the CLI command
summary and the first docstring line of every generated SDK method. The summary
is a one-line field by construction, so this is a false rendering of a true
value, on the surface SDK users and models actually read.
Counted from the committed subsets with the command to re-count, per the house
rule that enumerated instances are always the ones somebody happened to look at.
The named fix is one whitespace collapse in zip, and the note says explicitly NOT
to reflow 215 doc comments: that keeps the class alive for the next op and makes
cloud a special case of a general bug. It also retires step 6's "keep sentence
one on one line" as the workaround it now is — zip v1.18.6 no longer cuts a
wrapped summary mid-sentence, it just carries the newline.
Also records that apps/git's 24/24 partition is now a GATE rather than prose,
next to team's, so the form is discoverable from the playbook and not only from
the app.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/team is at its typed floor already (9 ops typed, 10 refusals gated by
untypedByDesign) and this pass converted none — every one of the ten was
re-verified against zip v1.18.6's OWN source rather than against the comment
that claimed it, because "cannot be typed" is a claim about a dependency and a
dependency moves:
- op.invoke json.Unmarshals any non-empty body BEFORE the handler runs
(typed.go:227), which is exactly what turns the account JSON-RPC's and the
cookie PUT's deliberately TOLERATED garbage into a 400;
- the REST arm ends in c.JSON(out) with no raw-bytes and no upgrade path
(typed.go:303) — the wallet page's bytes, the blob download, and the two
WebSockets;
- WithStatus panics on a non-2xx — the two OAuth 302 redirects;
- hasBody("POST") is unconditional, and the multipart upload's part filename
IS the blob id, which no JSON In can carry.
v1.18.7 is byte-identical to v1.18.6, so the floor is 9 until zip gains
raw-body binding, a bytes Out, or a non-2xx status.
What the pass DID surface is one route away from the ops. team's second plane is
app-level — the Team front derives BOTH the Y.js WebSocket (GET /collaborator)
and the markup-snapshot RPC (POST /collaborator/rpc/{documentId}) from
COLLABORATOR_URL, not from the /v1/team base — and manifest.Apps named only
/v1/team. cmd/cloud builds the fleet router from that list, so both fell past
every prefix to the console the host serves at "/": the collaborative editor got
the HTML shell, and the TYPED collaborator RPC — published in openapi.yaml and
therefore in every generated SDK and in the MCP tool list — reached no app at
all. It was recorded in the router oracle's `unreachable` ledger, whose own
contract is that a fix is one line in Apps and one line out of the ledger.
team's row names /collaborator now; the two entries are gone from the ledger and
TestEveryServedPathReachesTheAppThatServesIt proves the router delivers both to
team. No route, status, body or field name moves — the document already said
team serves these; the router now agrees.
The general lesson, recorded in LLM.md: a route's typed-ness is invisible to the
manifest, so an app whose surface is not wholly under one /v1/<name> prefix can
publish a perfect op the fleet never delivers, and only manifest/router_test.go
asks the router.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/company is 21 typed ops of 23 routes. The other two carry a comment
explaining why they cannot be typed, and nothing else — no test asserted the
wire either reason rests on. A comment is not a gate: the next typing pass could
convert both, move the wire, and stay green.
POST /v1/company/payment answers the fleet-wide billing-denial contract —
402 insufficient_balance, 402 spend_cap_exceeded, 503 balance_unavailable, each
a {"error":{"code","message"}} body, the same shape the edge gate returns. zip's
HTTPError renders a FLAT {status,code,error}, so a typed op cannot express it.
That was the entire justification and the denial path had zero coverage:
fakeCharge has carried an `err` field that no test ever set. TestPaymentDenialWire
sets it, for all three outcomes, and reads the code and message out of the NESTED
object — so the flat shape fails. It also asserts a refused charge leaves the
formation unpaid, which keeps the machine's payment guard shut.
TestPaymentChargesLast pins the other half: the gate runs LAST, after the stage
check and the paid short-circuit. Arm the charger to deny and a wrong-stage call
still answers 409, a paid formation still answers 200 — which is only true if the
charge was never attempted. That ordering is why this gate cannot lift into
middleware, where it would charge a caller the machine is about to refuse.
TestDeckTakesRawBytes pins POST /v1/company/fundraise/deck: a PDF body is
ingested (a JSON decoder would refuse it with 400), ?name= names the document,
an absent name takes the handler's default, an empty body is the one 400. The
existing body-cap test happened to prove bytes are accepted; nothing stated the
contract, so raw() now returns the body and the deck's shape is asserted.
Both mutations were run: rendering the denial as zip.Errorf flattens the body and
TestPaymentDenialWire goes red; moving the charge ahead of the stage check turns
the 409 into a 402 and TestPaymentChargesLast goes red.
No wire moved. Source changes are comments pointing at the guards.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/account was already 11 typed ops of 18 operations; its seven refusals — the
GET|POST /v1/billing/* and five-method /v1/commerce/* bridges — were recorded in
prose only, which cannot stay true. The next route added here would have been
untyped and nothing would have gone red, so the migration could silently regress
to the state it was moved out of.
typed_wire_test.go closes the list. It reads BOTH projections of the live router
at their one shared address form — what the document says is served
(openapi.Spec) and which of those carry a typed registry entry (openapi.Typed) —
and fails on any operation that is neither. It mounts through mountBoth, so it
covers BOTH of this package's subsystem registrations (account @48 and
account-bridge @122); all seven refusals live in the second one, and a gate over
the self-service half alone would have declared the partition complete while
covering none of it. Three directions are checked: a served operation that is
neither typed nor named, a named reason for an operation no longer served (stale
prose), and a typed op the document does not serve (a published address that
404s). Verified to bite: a probe route added to the mount fails it by name, and
a fabricated reason fails it as stale.
The seven reasons are ONE wire fact re-verified against zip v1.18.6's own source
rather than taken from the comment that claimed it. The answer is commerce's own
bytes AND status (c.Bytes(status, raw), including a PDF at invoices/{}/pdf) where
a typed dispatch ends in c.JSON(out) under one declared status (typed.go:270-303)
and WithStatus panics on anything but a 2xx (typed.go:110); the body is forwarded
as received at any content type where op.invoke json.Unmarshals it BEFORE the
handler and 400s on failure (typed.go:225-231); and the path and query are OPEN
sets bounded by an allowlist (billingForwardable, commerceStoreHeads) where a
typed op publishes a closed parameter list. Opaque by construction, not by
omission — convert when zip ships raw-body binding and passthrough responses.
No route, no handler and no In/Out type changed, so the wire is untouched: both
regenerated subsets (plugin/account, plugin/account-bridge) come back
byte-identical, and zipdoc_gen.go is unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/git was already COMPLETE at 24 typed / 24 refused, but its partition lived
in LLM.md prose — and prose cannot fail. It decays two ways: a new raw route
lands and the count is silently wrong, or a refusal is retired and the reason
outlives the route it described. Either way the file still reads "COMPLETE".
untypedByDesign (typed_wire_test.go) is the same partition as a VALUE, carrying
the wire fact behind each of the 24, and TestEveryRouteIsTypedOrNamed checks it
in three directions: a served operation that is neither typed nor named, a name
git no longer serves, and a name that IS a typed op. Each direction was made to
fail before being trusted. TestEveryTypedOpIsDescribed pins the other half —
every typed op carries lifted prose, because an op committed without
regenerating zipdoc_gen.go is a nameless MCP tool. Same shape as
apps/team/typed_wire_test.go; one form of this gate, not a second.
All 24 refusals re-verified against the handlers and against zip v1.18.6, not
against the prose. The ZAP family's reason is now precise about WHY no shim
exists: a typed op's error renders zip's HTTPError {status:<int>, code, error}
(zip/ctx.go:200), so typing renames msg->error AND retypes status string->int,
and cloud.Bridge applies a handler-set status only when err == nil (typed.go:78)
with Created/Accepted the only exported setters.
Surfaced while verifying: firstSentence (zip/openapi.go:606) returns the text up
to the first ". " verbatim, so a first sentence that wraps in the Go source ships
its line break into the OpenAPI summary, the CLI summary and every generated
SDK's first docstring line. 20 of git's 24 ops, and 215 of the fleet's 387
described operations across 18 packages. Counted from the committed subsets, with
the command to re-count. The fix is one whitespace collapse in zip, NOT 215
reflowed doc comments — reflowing keeps the class alive and makes cloud a special
case of a general bug.
No wire change: zero routes converted (none remain that can be), zipdoc
regenerates identically, and plugin/git/openapi.json is byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/guide is 13 typed ops of 19 routes; the 6 that stay untyped each carry a
wire reason at the registration site. Five of those reasons were pinned by a
test — the structured 409 ({error, step, blockedBy}) by TestHTTPTransitionsAndGating
and the blocked-/do case, the YAML-or-JSON document bodies by
TestDocumentPutsAcceptYAML. The SIXTH was not: nothing exercised the SSE branch
of POST /v1/guide/steps/{id}/do, so the second half of that route's refusal was
an unverified claim. Delete the stream and every test still passed.
TestDoStreamsSSE pins it: asked for a stream by either trigger wantsSSE accepts
(Accept: text/event-stream, or ?stream=1), the route answers
Content-Type: text/event-stream and writes the agent's actions as frames —
`event: plan` through `event: end` carrying the terminal state. A typed op
answers exactly one JSON value, so the pin goes red on the exact change typing
this route would make (verified: stubbing wantsSSE to false fails it with
"got application/json; charset=utf-8").
No route, type or handler moves — the wire, the lifted prose (zipdoc_gen.go),
openapi.yaml and plugin/guide/openapi.json are byte-identical. The route comment
now names its evidence, so a reader of the refusal can check it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/o11y needed no conversion: re-audited all 20 routes from source before
reading the two prior commits, and reached their conclusion independently. 12 are
typed ops; the other 8 cannot be without moving the wire, and every reason
already recorded in LLM.md holds against the code (the two VM proxies forward
VictoriaMetrics' own status and envelope through c.Bytes(status, body); the two
builder queries and the sessions list are reverse proxies with no Go type for
"whatever the runtime answered"; the two alert routes are text/plain and the
receiver deliberately accepts an unparseable body; /v1/sentry/* is a wildcard).
Verified current, not assumed: zipdoc regenerates apps/o11y byte-identical, the
suite passes under the gate env, and openapi.yaml carries 11 described o11y
operations with their query/body schemas.
What the audit adds is the one fact that record was missing, and it is the fact
the open decision turns on. POST /v1/o11y/ingestion is a typed op that reaches no
consumer, because mountEventIngest only registers it behind a reachable
Datastore and the process that writes the document has none. LLM.md said closing
it means the path "stops falling through to the order-70 wildcard", and whoever
took it owned that wire change. Measured against the pinned runtime, that
fallthrough serves NOTHING: hanzoai/o11y v1.5.34 registers no /ingestion route at
all -- its only ingest-named paths are the unrelated
/api/v2/gateway/ingestion_keys* -- and a no-DSN process cannot init the embed
either, so mountRuntime installs the reverse-proxy fallback and the request lands
on the same server build, which also has no such route.
So the change on the table is 404 -> an honest 503, with no working write path at
risk. "Stops falling through" reads like remote ingest breaking, which is why
this looked more expensive than it is. Recorded with the command to re-measure
it, because a version-pinned claim goes stale.
No code and no wire touched: this is the description task finishing its own
record.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.801.320 denied EVERY paid inference call. The cause is one 401:
[billing] GET /v1/billing/balance err="sign in to view billing"
[ai] balance_gate: balance unverifiable for cold subject=hanzo:
commerce returned 401 (fail-CLOSED, retryable)
ai's prepaid gate reads this endpoint to admit or refuse a paid request.
build.go's wireFinance installs an in-process balanceReader so that read is a
direct typed call — but a Go func var cannot cross a PROCESS boundary, and once
ai became its own plugin process it stopped seeing the hook and fell back to the
HTTP path balance.go already documents as the split-deploy fallback. That request
carries COMMERCE_SERVICE_TOKEN rather than a user session, so principal.Org was
empty and the handler refused it. The gate is fail-closed on purpose — a balance
it cannot verify must never degrade to free inference — so a single unauthorized
read took down all paid traffic while every pod stayed Ready.
The fallback was supposed to work; now it does. Before refusing, the handler
accepts a caller bearing the verified service token, using the SAME predicate
apps/account already trusts (account.IsServiceToken — constant-time compare
against the configured COMMERCE_SERVICE_TOKEN), and takes the org from the
gateway-pinned X-Org-Id, which the gateway strips from every client request.
Scope is not widened. balance_s2s_test.go covers both directions: the trusted
read is served AND a wrong token, an absent token, a prefix near-miss, and a
valid token with no org are each still 401 with the ledger never touched. The
pre-existing TestBalance_RequiresSignIn — anonymous and forged-X-Org-Id both 401 —
still passes unchanged, which is the evidence this adds a trusted path rather
than opening one.
(The test needed one correction: mountApp sets COMMERCE_SERVICE_TOKEN itself, so a
t.Setenv before it is silently overwritten with "" — the token has to be passed
through mountApp. The first version of the test failed for that reason, not the
handler's.)
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The wip-preserved go.mod carried helm v3.21 with k8s.io/api floating to v0.36,
while the gitops-engine's replace block pins kubectl and the kubernetes staging
tree to v0.35.3. Two k8s minor versions in one binary fails twice over:
kubectl's scheme imports alpha groups (scheduling/v1alpha1) that v0.36 removed,
and component-base 0.36 registers feature gates kubernetes 1.35 also registers
— 'feature gate CRDObservedGenerationTracking with different spec already
exists', a panic before a single test runs.
api, apimachinery, client-go, apiextensions-apiserver, component-base and
streaming now join the same v0.35.3 replace set as every pin already there. One
k8s version, stated once.
apps/deploy (incl. TestRenderRealChartMatchesHelm — byte-identical to the helm
binary), apps/git and cek all green; whole module builds.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ai's router registers the OpenAI-compatible endpoints at TOP LEVEL, and apps/zen
holds the other half: zen is "a MIDDLEWARE, not a route owner" that claims zen
SKUs and calls c.Next() "so ai's /v1/* catch-all serves non-zen models". So ai
must claim a prefix those paths fall under — and when its row read "/v1/ai" it did
not: POST /v1/chat/completions answered 405 and GET /v1/models 404 on v1.801.318
and .319, with the pod Ready, probes green and chat.hanzo.ai serving 200. Every
SDK caller got nothing and nothing looked wrong.
The test asserts ai CLAIMS those paths, not that it claims them FIRST, and that is
the whole point. I wrote the first-match-wins version first and it passed with the
bug reintroduced — zen holds /v1 and precedes ai, so every path "matched"
something — then a second version blamed commerce for owning everything. Apps
legitimately hold overlapping prefixes and chain through c.Next(); presence in the
chain is the invariant, position is not. Both wrong versions were run against the
actual bug before either was discarded.
Also NOT asserted, on purpose: the zen-before-ai ordering. apps/zen documents it,
but that comment predates this manifest (it names Wire(), which is gone) and the
live order is the reverse. A test encoding it failed against intentional upstream
state, so it is a documented open question for whoever owns the split rather than
a build break — the orders differ for billing, since zen's Gate/Meter only run if
zen's Claim sees the request first.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TWO defects, one root: an app's declared prefix is the ONLY thing the light host
routes on, and a wrong one fails silently.
1. INFERENCE WAS UNREACHABLE. ai's row said Prefixes{"/v1/ai"}, but ai's router
registers the OpenAI-compatible surface at TOP LEVEL — /v1/chat/completions,
/v1/models, /v1/messages, /v1/completions, /v1/responses, /v1/embeddings — and
apps/zen states the other half of the contract: zen is "a MIDDLEWARE, not a
route owner" that claims zen SKUs and calls c.Next() "so ai's /v1/* catch-all
serves non-zen models". Scoped to /v1/ai, ai dropped out of the /v1 chain, so
zen's c.Next() fell through to the console catch-all: POST
/v1/chat/completions answered 405 and GET /v1/models 404 on v1.801.318 and
.319 — pod Ready, probes green, chat.hanzo.ai serving 200, and every SDK
caller getting nothing. Nothing looked wrong from outside.
2. commerce CLAIMED A BARE "/v1". That silently overlapped billing, catalog,
projects, agent, agents and kms, and it made a future /v2/commerce impossible:
a bare version root swallows every sibling under that version, including ones
that do not exist yet. Replaced with the nine prefixes apps/commerce actually
registers — behaviour-preserving, only narrower.
The guard is what keeps this from returning. TestNoBareVersionPrefix rejects any
/vN root (verified: it fires on a planted bare /v2), with a CLOSED exemption for
exactly zen and ai — zen because it dispatches on a body field and cannot
enumerate paths, ai because it is the catch-all zen falls through to. A third
entry has to be argued for in a test diff.
TestInferenceSurfaceIsRoutable asserts ai claims the paths customers' SDKs call.
I got that test wrong twice before it was worth having. A first-match-wins model
PASSED with the bug reintroduced (zen matches /v1 first, so the path looked
routed), then blamed commerce for owning everything. Several apps legitimately
hold overlapping prefixes and chain through c.Next(), so the real invariant is
"ai CLAIMS these paths", not "ai claims them first". Both versions were checked
against the actual bug before being kept.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cek refuses to open a store without a master key on any build that can
encrypt, so the four store-backed tests here failed while agents, authors,
automations and catalog all passed — which reads as a defect in the tools plane
rather than a missing two-line TestMain. Same harness, same throwaway key, and
it never overrides a key the environment already provided.
Verifiable: the failure moves from "CLOUD_KMS_MASTER_KEY_REF is required" to
the platform's RAM-backed-scratch refusal, i.e. the key now lands and the
remaining wall is isRAMBacked being false off Linux by design.
Co-Authored-By: Hanzo Dev <dev@hanzo.ai>
POST /v1/plugins/build takes TypeScript, or an API document to generate it
from, and returns a plugin the runtime has already loaded once.
The build is the SAME pipeline the committed connectors go through — esbuild
to one CommonJS program (connectorruntime.Bundle), then compiled in goja
(Runtime.Compile). Compiling IS the gate: source that bundles but will not
load is rejected and never stored, so "it built" means the artifact this
deployment will execute compiled, not that a model produced plausible text.
A failed build returns the generated source with the error, because the source
is the thing worth reading when generation goes wrong.
CREDENTIALS ARE NOT PART OF A PLUGIN. A plugin names the connectors provider
it needs and reads ctx.auth at run time, where the secret is already under KMS
custody. Source that carries something shaped like a key is REFUSED, not
scrubbed — a silently-stripped key looks like it worked, and the caller never
learns the secret went somewhere it does not belong. That also means rotating
a key never means rebuilding a plugin.
/v1/plugins lists what this DEPLOYMENT mounted; /v1/plugins/authored lists what
an ORG built. Different sets, different lifecycles, so a subpath rather than one
mixed collection.
audrecord grows an action parameter instead of a copy: the builder records
plugin.build, which is a different act on a different resource than tools.call.
Tests cover the gate itself — a connector bundles, a syntax error and an empty
name are refused, and the credential shapes are caught without false-positiving
ctx.auth or propsValue.token. One of them found a real bug: stripFences trimmed
before removing the closing fence, leaving the newline behind.
Co-Authored-By: Hanzo Dev <dev@hanzo.ai>
/v1/tools answered "what can this org call" only if the caller knew to pass
?source=. Give the three sets people actually ask for their own path.
skills and mcp are VIEWS over the same registry — one implementation
(listBySource) mounted twice — so a tool is still registered in exactly one
place and activation still lives in exactly one place. A source view filters
the per-principal list, so it can never widen what a caller may see.
plugins is deliberately NOT a tool source. A plugin here is a mounted
subsystem (cloud.Plugin: Name, Mount, Price, Prefixes) — code that extends the
deployment's surface — while a tool is something called through that surface.
Its inventory is cloud.Subsystems(), the boot snapshot Declare installs, so it
reports what the binary actually mounted rather than a second list free to
disagree. An earlier draft added a SourcePlugin with no producer; that was
speculative and is gone.
The external MCP server registry MOVES /v1/tools/servers to /v1/mcp/servers.
A server is a record an org creates, not a tool the registry enumerates, and
splitting one MCP integration across two prefixes had no defender.
Declare the new prefixes in BOTH manifest/apps.go and plugin/tools/main.go: an
undeclared prefix resolves to no subsystem, which would leave every request to
these paths priced Undeclared and unlabelled in tracing.
Co-Authored-By: Hanzo Dev <dev@hanzo.ai>
Sixteen /v1/compliance routes become typed ops (health, status, records, audit,
subjects CRUD-reads, verifications start/list/get/refresh/decision,
accreditation create/list/get/decision), so each now projects to OpenAPI, MCP,
the CLI and the SDK from the one registration. cloud.Bridge is installed on the
group — the subsystem had none, so no typed op could have resolved its org here
— and the package gains its //go:generate zipdoc directive, also missing.
One route stays untyped, naming its wire fact at the registration and the
handler: the provider webhook authenticates by HMAC over the RAW body bytes,
verified before any parse — a typed op decodes its In first — and an unknown
reference answers a second 200 shape (a benign no-op, not a check).
Wire preserved exactly: 201 on the three creates via zip.WithStatus, the
map-built views become structs whose omitempty matches the maps' conditional
keys, bodyCap keeps the 1 MiB / 413 gate in front of the parse it has always
preceded, and noStore keeps Cache-Control: no-store on the PII-bearing reads.
TestTypedOpsPreserveTheWire pins the envelope (no-store, 413, ?limit binding,
empty-body tolerance); the behavior suite passes unchanged against baseline.
The reviewer role gates, emitAudit's attribution and noStore are the pinned
cloud.Request uses (allowlisted); the tenant itself is principal.OrgFrom, never
the request. Typing surfaced two false doc claims before they shipped as prose:
the audit read returns rows NEWEST first (ORDER BY seq DESC), and a check's
status vocabulary includes expired — plus result filters are success|deny|error,
not "denied".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/automations/connectors/{id}/run becomes a typed op, so the last
schema-less stub under /v1/automations now projects to OpenAPI, MCP, the CLI
and the SDK from its one registration: runIn ({action, auth, props} + the path
id) in, runResp ({ok, output, error}) out, with the infra-vs-piece split kept
exactly — an action that ran and failed answers HTTP 200 ok:false, never a
5xx; unknown connector 404, missing action 422.
Typing surfaced the same two defects it surfaced in automations proper: the
package had NO cloud.Bridge (registered bare on the app root, it worked only
because automations' group Bridge happened to cover the prefix — mounted alone,
a typed op could never have resolved its org) and NO //go:generate zipdoc
directive, so no prose could have reached the document. It also had no wire
test at all; http_run_test.go now mounts the subsystem ALONE and pins the
403/404/422 gates and the ok:false 200 — the standalone mount is what proves
the new group Bridge makes it self-contained.
Regenerated: apps/connectorruntime zipdoc, plugin/automations/openapi.json,
openapi.yaml (weave green). automations + connectorruntime + openapi tests
green under the Makefile gate env.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The two document writes (POST /v1/framework/:doctype, PUT
/v1/framework/:doctype/:name) stay raw, re-verified against zip v1.18.6:
typing them takes THREE halves of one zip capability, not the two the
rawRoutes pin named. Beyond declaring an open object and binding the URL
onto one, the bound params must ride OUTSIDE the body namespace — off the
REST path op.invoke receives no path map (MCP tools/call and the call
plane pass nil), so URL params could only travel as body keys, and a
create body's `name` IS the requested document name (engine ops.go,
stringField(in, "name")). Folding :name into the body collides with a
field the document owns, so a map-In workaround would ship ambiguous
MCP/CLI projections. No wire change; no generated artifact moves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The typed migration for apps/guide landed in 7e50740e (13 of 19 routes; the six
refusals each named at their registration), but two of its refusal facts were
prose only: PUT /v1/guide/curriculum and PUT /v1/guide/blueprint accept a raw
YAML-or-JSON document (Parse, sigs.k8s.io/yaml), and no HTTP test sent YAML —
the acceptance was pinned at the Parse unit level, one layer below the wire the
refusal is about. TestDocumentPutsAcceptYAML PUTs raw YAML through both routes:
typing either one (a typed In is decoded as JSON before the handler sees it)
now turns CI red instead of silently 400ing every YAML caller.
The 409 family (steps/:id/start|done|do answer a structured {error, step,
blockedBy} body a zip.HTTPError cannot carry) was already wire-pinned by
TestHTTPTransitionsAndGating; those routes wait on zip multi-status.
LLM.md's tranche table still listed guide as open — the table is the
migration's coordination surface, and an uncrossed row invites a duplicate
effort. Crossed out with the counts.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every route on this surface that CAN be a typed op already is (72a87f87 typed
15 of 32); the other 17 are refused for wire facts. Those facts lived only in
ops.go prose, which nothing enforced — the next route added here would default
to raw and nobody would notice, and a stale reason would outlive the route it
described. apps/team/typed_wire_test.go is the worked example of doing better:
the refusals are a CLOSED list a test holds against the live router.
pricing now carries the same gate. untypedByDesign names the 17 raw operations
at their document-form addresses, each with the wire fact that keeps it raw;
TestEveryRouteIsTypedOrNamed fails on any operation that is neither a typed op
nor on that list (so the next pricing route is typed by default) AND on any
listed operation the surface no longer serves (so the list cannot go stale);
TestEveryTypedOpIsDescribed fails on a typed op whose prose did not reach the
registry (an op added without regenerating zipdoc_gen.go is a nameless MCP
tool). The filter is the package's own Prefixes — the same five cloud.Declare
scopes by — so a route mounted outside what the subsystem declares surfaces as
uncovered, which is exactly the undeclared-prefix defect 72a87f87 fixed.
Each refusal was re-verified against zip v1.18.6's own source rather than
carried forward from the comment that claimed it, because "cannot be typed"
is a claim about a dependency and a dependency moves:
- the 15 verbatim proxies: a typed dispatch ends in c.JSON(out) under the one
status the op declared (typed.go registerTyped), so the bundle's own status
(200/503) + unmodified bytes are still inexpressible;
- PATCH models/{wildcard1}: bindURL matches path params by json field name,
so binding fiber's *1 still takes an In field tagged json:"*1" that every
projection would publish;
- PATCH providers/{name}: schemaOf still reflects json.RawMessage ([]byte) as
an array of integers (openapi.go, reflect.Slice arm) — a false schema — and
map[string]any still cannot hold RFC 7386's explicit-null-deletes.
No route, wire byte or artifact moves: zipdoc_gen.go, plugin/pricing/
openapi.json and the golden regenerate identical. These convert when zip ships
raw passthrough / multi-status (#78's family), and the gate is where that
conversion will be forced into view.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ai is a LAZY plugin (mounts on the first /v1/chat/completions), so it can mount
before the app that installs the balance reader. The hooks were wire-time
SNAPSHOTS, so nil latched for the process lifetime; the ai module then fell back
to an HTTP self-call to /v1/billing/*, which the edge 401s (the toothless-gate
bug build.go already names), and the balance gate is fail-CLOSED — so every
completion answered 503 balance_unavailable.
Observed in prod on v1.801.320: 'balance_gate: balance unverifiable for cold
subject=hanzo: commerce returned 401 (fail-CLOSED, retryable)' on a pod whose
commerce plugin mounted ten minutes later. Chat, studio's assistant and every
SDK caller were down on a healthy pod.
Resolve cloud.TierReader/BalanceReader/UsageRecorder per call — the pattern the
rolling-cap hook in this same function already uses and documents. The balance
trampoline fails LEGIBLY when nothing is wired rather than reporting 0, which
would read as a real zero balance and deny a paying caller.
The o11y typing pass landed in d0bb72e7 (+6e22e075): 12 of 20 routes are
typed ops; the other 8 cannot be typed without moving the wire, each named
with its reason in apps/o11y/LLM.md — 2 verbatim-status VM proxies, 3
reverse proxies (query/query_range/sessions), 2 text/plain Alertmanager
receipts, 1 sentry wildcard. Re-verified against source at ac98f6ea:
zipdoc regenerate + make -C apps/o11y openapi produce zero drift, package
tests green with the gate env. The table's "o11y 11" overcounted: the
re-measure grep picks up 3 comment lines quoting app.All("/v1/o11y/*")
(event_ingest.go:14,221 + scope.go:17); the real untyped count was 8.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The partition table still listed crm 20 as pending while the inventory
forty lines down records it done: 19 typed ops, one refusal (the public
Startup Program intake POST, whose IP rate limit and pre-parse 64 KiB
body cap are wire). Re-verified on this tree: re-measure finds the one
raw registration only; zipdoc and plugin/crm/openapi.json regenerate
byte-identical; the gated tests pass; the subset audits clean against
the bodyless-POST, embedded-struct and empty-leaf classes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/crawl is served by this package -- manifest/apps.go is mount order AND
routing order, crawl at 78 against ai at 111 -- and Fetch is one http.Get. For
the client-rendered part of the web that returns a near-empty shell, extract
finds almost no text, and the caller gets 200 with nothing in it. A crawl that
returns nothing and reports success is worse than one that fails, because
nothing upstream can tell.
So: static first, and if what came back is under 512 bytes of markdown, ask
Hanzo Crawl (headless Chromium, ghcr.io/hanzoai/crawl:sha-7b8dc59, CR landed in
universe e0942b664) for the rendered version.
Length is the whole heuristic on purpose. "Does this have a <div id=root>"
recognises today's frameworks and misses tomorrow's; "the extractor found
almost nothing" is the symptom itself and does not date.
Three things keep escalation from being a downgrade:
- Longer text wins, not "the browser answered". A render can come back thinner
-- a consent wall, a bot check, a page that needed no JS -- and taking it on
faith would make this a regression on exactly the pages it should not touch.
- An absent, slow or unhappy browser leaves the static Page standing. That is
the state this ships in, since nothing has deployed the CR yet.
- A sufficient page never pays for a render at all.
TWO FINDINGS FROM THE TESTS, both of which had already shipped in my first pass:
1. It could never have worked. browse() used `client`, whose dialer refuses
non-public addresses -- and the browser lives at crawl.hanzo.svc, private by
design. Every escalation would have been refused with ErrBlocked. There are
now two clients for two trust classes: `client` dials wherever a CALLER
asked and stays guarded, `service` dials the one address WE configured.
2. It was an SSRF bypass. Read escalates when Fetch FAILS, and one reason Fetch
fails is the guard refusing an internal address -- so "crawl http://10.0.0.1/"
would have been refused here and then forwarded to a Chromium that fetches it
happily, reachable by anyone who can call /v1/crawl. reachable() now applies
the same check before the browser sees a URL, including refusing a host that
answers with ANY internal address, since one public IP listed beside the
target is the documented way around a first-answer check.
The resolver is a var so that boundary is testable with a hostile answer; a
check you cannot test with the attack is one you are only assuming holds.
11 tests, all passing, covering both halves of the escalation contract, both
markdown wire shapes, and the guard.
Six independent reds, five of which were the test lying about the code rather
than the code being wrong. Each is named here because the distinction is the
finding.
kmsreseal read nothing from the standalone. One client built one URL for two
faces that spell the tenant differently: luxfi/kms serves
/v1/kms/orgs/{org}/secrets and matches the path against the token's orgs, while
cloud's apps/kms serves /v1/kms/secrets with the org from the principal. The
client now carries the `route` of its face — the only per-face difference there
is — named at each construction. Second bug found the same way: listFolder
decoded only cloud's `secrets[].name`, so every folder-sync CR would have
reported "folder EMPTY at source"; both faces emit `names`, so one decode reads
either. The isolation probe asserted a cross-org 403 on a URL that no longer
exists; it now asserts the stronger property that replaced it — a tenant-naming
URL reaches no route at all, carrying a VALID credential so the 404 means "no
such URL", not "no such caller". runbook.go claimed an isolation matrix verify
does not run, and that a base-URL repoint is all a consumer needs; cloud serves
no /v1/kms/orgs/… route, so it is not.
apps/storage's bucket tests never reached the assertions they made: guard gates
money first, the $1.00 default fee met an unconfigured commerce, and all three
died 503 before any name or key was validated. The subject of that file is the
tenant and validation gates, so the fee is 0 there — the documented un-gated
posture — and the priced posture stays billing_test.go's subject. The 13s per
app was construction, not the request: BuildDeps probes the store against an
address nothing listens on, exhausting the retry budget three times. It now
points at an in-process endpoint that refuses, which reaches the same posture in
microseconds and makes the file self-contained. That let doOrTimeout go, and
exposed TestBucketsRouteReachesS3NotProvisioning as vacuous — its with-org leg
503'd at the money gate and never reached the s3 handler it claimed to prove
owned the route. It now asserts the exact 502 only that handler can produce.
TestRunnerBuild_IAMReleaseRejected sent a SuperAdmin and expected a refusal, so
it was reading a correct admission as a failure — and made a live call to
api.github.com from a unit test to get there. It now sends an org admin, which
is the property it is named for. The permit side shipped with no coverage at
all; it is pinned now, hermetically, on stubbed seams.
apps/graph's flake was a real DNS round-trip per request against a 1s deadline,
reproducible 6 runs in 10 under resolver load. Port 0 is not listenable, so the
kernel refuses locally and immediately: unreachable becomes a fact of the test
rather than a name the resolver has to fail to resolve.
books gained two cloud.Request sites. Both earn their place — narrateAsk reads
principal.Ledger, which a SuperAdmin masquerade moves off the effective org, so
principal.OrgFrom would bill the org being inspected. The general untyped
`query(ctx, name)` did not: it justified one fact and permitted any. It is now
sandboxFrom, which can express only the ledger selector and composes the same
sandboxQuery the untyped handlers use, so both planes resolve it by one rule.
And the reason seven subsystems each spelled their own gate: there was nowhere
to put it. gate.go is that place — Authority (what a caller has, the three
HIP-0519 predicates and nothing else), Scope (what a route requires), Admits
(the whole rule, one expression, with `Super || OrgAdmin` written out so the
superset is visible at the gate), Guard (its standard fail-closed application).
kms, platform's fleet, and storage now read it; deploy and admin/core keep their
own refusal shape but move onto the canonical predicate. Everything that is not
authorization — store readiness, org syntax, per-op billing, tenant confinement
after the door opens — composes around it and stays in the app, so no real
difference was flattened. platform's two transports, HTTP and the internal
plane, now read the same rule instead of two copies of it.
The ingress boundary is untouched: app.Use(IdentityMiddleware) stays until the
network policy makes the gateway the only ingress, because the red-team probe
reads another tenant's secret without it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Strike it from tranche D and record the split where the next agent looks first:
the two latent defects typing surfaced (missing cloud.Bridge, missing zipdoc
directive) and the wire fact behind each refusal (two success bodies, arbitrary
JSON value, raw-byte dedupe + headers, JSON-RPC 200-on-parse-error).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestRequestEscapeHatchIsPinned has been red on main since apps/books went typed:
ask.go (narrateAsk reads the BILLING ledger off the request — a header fact,
never an In field) and typed.go (query reads a URL-borne value for a
body-carrying op, where an In field would move it off the URL). Both are the
exact uses the escape hatch exists for; the entries state why, from the call
sites' own prose.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Fourteen /v1/automations routes become typed ops (connectors + pieces alias,
flow CRUD, versions, run/enable/disable, run history), so each now projects to
OpenAPI, MCP, the CLI and the SDK from the one registration. cloud.Bridge is
installed on the group — it was MISSING, so no typed op could have resolved its
org here — and the package gains its //go:generate zipdoc directive, also
missing. populatedFlow spells out the Flow fields it embedded, so the published
schema matches the wire instead of documenting a nested object the route never
sent. setEnabled returns the flow instead of writing the response, one seam for
its three callers; auditHTTP is the pinned cloud.Request use (allowlisted) that
keeps the enable/disable audit record attributed.
Four routes stay untyped, each naming its wire fact at the registration and the
handler: operations (TWO success body shapes — Flow on CHANGE_STATUS, else
FlowVersion), resume (arbitrary JSON value body, raw-byte size gate), hooks
(raw-byte content-hash dedupe + two contract headers), mcp (JSON-RPC answers an
unparseable body 200 with a -32700 object; zip would 400 it).
Statuses preserved exactly: 201 create/version/run via zip.WithStatus, 204
delete via nil Out, 200 elsewhere; the package's own HTTP tests pin them and
stay green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerce's row held bare "/v1", which made it the fleet's route of last
resort: /v1/chat/completions, /v1/models, /v1/embeddings, /v1/responses,
/v1/audio/* — the whole OpenAI-compatible surface — landed on commerce and
answered its 404, for every caller. apps/commerce/mount.go had warned in
prose that reading its "/v1" group as a claim would hand commerce every
request in the fleet; the manifest did exactly that.
Proved on the real router (router_test.go's oracle, built from manifest.Apps
through the same zip.Load the host calls): before this change all nine
OpenAI paths -> commerce; after, all nine -> ai, and every path commerce
publishes still reaches commerce or its recorded owner.
- commerce now owns its published FAMILIES, each named DEEPER than the
sibling that shares the stem, so catalog keeps bare /v1/catalog, plan
keeps the rest of /v1/plans/*, and account-bridge keeps the console's
/v1/commerce/* + /v1/billing/* data bridges. Not commerce.Prefixes
imported — the app states its fail-closed set once; the row states what
the router may hand it; the oracle keeps the two honest.
- ai's row is "/v1": the remainder, behind every deeper prefix. Its own
/v1/* catch-all is what serves the OpenAI surface.
- ai precedes zen (frozen order edit — a decision): equal "/v1" claims
resolve by mount order, and zen's Claim-middleware contract cannot be
expressed as a per-process prefix, so its row is deliberately shadowed.
- metrics gains /v1/logs + /v1/traces, its own published ingestion doors,
which the bare-/v1 row was swallowing (405, silently).
Ledger: 26 unreachable entries routed (ai's catch-all, commerce's webhook/
auto-recharge/commerce/catalog/plans families, metrics' seven doors);
27 remain recorded. Oracle: 1005 published paths, 978 reach their app.
TestOpenAISurfaceLandsOnAI pins the nine concrete OpenAI endpoints so the
single-path catch-all can never again hide the whole product behind one
ledger line.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip's structSchema publishes an embedded EXPORTED struct as one NESTED
property named after its type, while encoding/json PROMOTES its fields —
so every step object in GET /v1/guide and the skip/reset ops documented
{JourneyStep: {...}} for a wire that has always been flat {id, title,
deps, ..., state}, in openapi.yaml, in every generated SDK and in the MCP
tools' schemas. Fourth live instance of the embedded-struct class (recipe
rule 7: patchTargetIn, botView, clusterDetailView), and the published-
subset check that now rides in LLM.md finds two more in plugin/admin
(MetricsData -> SaaSMetrics, ServiceView -> ServiceRow).
The wire is untouched: the fields are inlined in promotion order with the
same json tags, TestStepViewCarriesJourneyStep pins the spelled-out copy
against JourneyStep so a field added later cannot silently drop out of
the view, and the step fields now carry their prose in the document
instead of riding on a property that never existed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The one convertible refusal of the six: zip's schema walk publishes an embedded
struct as a NESTED property while encoding/json flattens it, so the Out spells
Metrics' fields out flat (the sanctioned fix for the embed class) and
metricsResponseOf is the one constructor. The drift the old refusal feared is
pinned red by TestMetricsResponseCarriesEveryMetricsField (reflection-filled,
so a new Metrics field cannot silently drop off the wire), and
TestMetricsSchemaMatchesItsWire keeps the published schema equal to the wire
keys from here on. Wire unchanged: same 401/500 order and messages, same flat
JSON, same no-store, same operationId (get_v1_books_metrics), sandbox/from/to
still query parameters.
The other five stay untyped, each re-verified against zip v1.18.6's source:
scan, inbox and bank/import take raw document bytes that op.invoke would
json-decode into a 400 before the handler runs (typed.go:231); link-token and
exchange answer 501 unconditionally, and WithStatus panics on a non-2xx
(typed.go:110) — a typed op would publish a success contract neither has ever
sent.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/team converted in 7889d98c + ea0804f9 (9 typed ops); the other 10 routes
cannot be typed without moving the wire, and each is named with its reason in
untypedByDesign (typed_wire_test.go), which TestEveryRouteIsTypedOrNamed
enforces as a closed list. Re-verified this pass against zip v1.18.6: the typed
success path is unconditionally c.JSON (typed.go:304) and a typed POST/PUT
unconditionally JSON-decodes its body (hasBody, typed.go:276), so the upgrade,
redirect, multipart, raw-bytes and tolerant-bind routes are all wire-refused.
zipdoc regen is a no-op, plugin/team/openapi.json regenerates byte-identical,
apps/team tests green under the gate env.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The two cloud.Plane() ops (/git/files, /git/publish) were registered as
closures, and the commit that added them never regenerated zipdoc_gen.go, so
zipdoc -check was red on main for apps/git and the registry carried no prose
for either op. Named handlers (planeFiles, planePublish) with true doc
comments; regenerated. plugin/git/openapi.json is byte-identical — plane ops
never reach the public document.
The package is otherwise COMPLETE: 24 typed / 24 refused. apps/git/LLM.md now
names each refusal's wire fact with line cites (raw-byte HMAC webhook,
smart-HTTP pack protocol x6, server-rendered HTML x12, ZAP envelope adapters
x5) and counts the one bodyless-POST instance (/repos/{name}/gc). Root LLM.md
tranche B marks git done.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/account went typed on main in 0a134476 + cb226978 (11 of 18 ops; the
Bridge it never had installed in the first). The tranche-C table still said
'account 19', which is exactly the stale-count failure the company section
warns about — it sent another agent to redo the package. Verified at
8087c757: zipdoc + both openapi subsets regenerate byte-identical, tests
green under the gate env. The seven refusals (the /v1/billing/* and
/v1/commerce/* verbatim wildcard bridges) are now recorded with their wire
facts so they are never re-derived.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/crm converted in a15f5ca3 (19 typed ops, one raw route: the public Startup
Program intake, whose IP rate limit and pre-parse body cap are wire). The LLM.md
inventory never picked it up; re-measured this pass — zipdoc regen is a no-op,
apps/crm tests green under the gate env, the intake is the only raw registration.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
721a3039 dropped app.Use(IdentityMiddleware) on the reasoning that identity is
the gateway's and is verified once. That reasoning is right and is now written
down as HIP-0519. Its correctness rests on one assumption the HIP states
plainly: the gateway is the only ingress.
That assumption does not hold here yet. With the middleware gone, the estate's
own red-team probe reads another tenant's secret VALUE off the in-cluster KMS
listener:
PROBE (b) forged org + forged X-User-Id + IsAdmin → 200 {"value":"…"}
TestAudit_AnonRequestNotAttributedToForgedOrg fails the same way: a forged
X-User-IsAdmin survives into the audit record and an anonymous request is
stamped with a claimed org.
So the middleware stays. Reaching HIP-0519 is a network-policy change FIRST —
service listeners unreachable except through the gateway — and a code deletion
second. Doing the code half alone is a cross-tenant secret read, and those two
tests are the gate on the real work rather than obstacles to it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
721a3039 removed app.Use(IdentityMiddleware) — cloud's second verification of a
token the gateway had already verified — and said nothing about it. A commit
about the money gate carried a change to the identity boundary in its diff and
not in its message. Recording it here rather than leaving it to be discovered.
The change itself stands and is now specified: HIP-0519 defines identity as
verified exactly once, at the edge, against IAM, with everything behind it
reading the assertion and forwarding it unchanged. Cloud reads; it no longer
re-derives.
The stale comments that still described the middleware as a step in this chain
are corrected, because a comment describing a middleware that is not installed
is worse than no comment.
NOT DONE, and named so it is not mistaken for done: middleware_identity.go,
auth_identity.go's validator half, and identity_cache.go still COMPILE — nothing
installs them, but ~1,300 lines of a second identity implementation remain in
the tree, and HIP-0519 conformance means deleting them. The cut is mechanical
but not trivial: token-shape predicates, OrgHasUnsafeRune, cookieTokenNames and
OrgForKey live in those files and serve audit, analytics and tenancy rather than
authentication, so they must be lifted out first. NewTokenValidator is a real
second question — apps/team and apps/deploy verify a token at LOGIN MINT time,
which is a different moment from an inbound request and needs its own answer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Revert of 06e0a0e8, plus the fix it should have been.
I made an unreachable biller ALLOW, to turn three red tests green. Those tests
were the stale ones: they assert the pre-split rule, where no commerce URL meant
nothing billed. resource_billing_test.go states the rule that replaced it, and
states why — once apps are their own binaries the ledger has ONE writer and it
lives with commerce, so a meter without a local URL asks it, and a biller it
cannot reach is UNKNOWN, never allowed. Allowing turns every priced act free the
moment an app is split out, silently.
Four tests now agree on that instead of three contradicting one, and each says
what it is protecting: a priced invoke, create and op are refused with no biller
reachable, and the sandbox, provisioner and handler run ZERO times — because
doing the work first and discovering later that nobody could bill it is a
resource somebody has to find.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Gate's own comment named the distinction — "nobody bills in this deployment"
versus "the biller is one socket away" — and the code never made it. gatePeer
treated a missing socket as an error, so a deployment with no commerce at all
503'd every priced act, which is the behaviour "billing not configured" was
never supposed to have.
The socket is what answers it. No socket means this deployment does not run
commerce, so the gate is inert, exactly as before the split. A socket that
exists and does not answer is a real fault and still fails closed — which is
the direction that matters, because allowing there is what once turned every
priced act free, silently.
Both directions are pinned: TestGate_NoMoneyPlaneIsInert and
TestGate_UnreachableBillerFailsClosed, the second against a real socket that
accepts and hangs up. PeerPresent requires a socket, not merely a name on disk.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The op-call plane marshalled its body with encoding/json, so two of our own
processes on one host — holding the same struct in memory — serialized it to
text and parsed it back to cross a socket. That is a boundary encoding doing an
internal job.
zip v1.18.6 carries the plane in ZAP: the layout is derived from the In/Out type
itself, a field is its offset, and no name travels. The bytes on the socket are
the bytes in memory, which is what the hand-written codecs achieved and what
replacing them was never supposed to give up. Refusals cross as ZAP too, status
intact.
Nothing in cloud changed to get it — the ops were already typed, so the encoding
moved underneath them. TestPlaneWireCarriesNoFieldNames pins the result against
the real socket: the values cross, the field names and the braces do not.
The `json` tags on plane types name fields in the OpenAPI schema this plane also
projects. They are the document's vocabulary, never the wire's — and because the
layout is the type, fields may only be APPENDED.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/base v1.5.11 (pulled in by da53ab30) turns a long-standing silent
mismatch into a compile error on purpose:
base@v1.5.11/core/sqlite_math_required.go:30:6:
undefined: cgoBuildNeedsSQLiteMathFunctions
base's search layer generates SQL calling acos/cos/sin/radians/sqrt (the
geoDistance token in tools/search). SQLite only has those with
SQLITE_ENABLE_MATH_FUNCTIONS, and the cgo backend gets them ONLY behind
csqlite's sqlite_math_functions tag — so a cgo build shipped a SMALLER SQL
surface than the code above it writes against, and the failure surfaced as a
customer's search returning 'no such function: acos' from an endpoint that
works in production. The file is //go:build cgo && !sqlite_math_functions and
references an undefined symbol so the two build modes cannot disagree in
silence.
Added to all four CGO_ENABLED=1 tag sites (the modernc gate, the two codec
tests, and the per-plugin build) plus the tag string quoted in the gate's own
error message. The CGO_ENABLED=0 builds are untouched: the pure-Go backend
always has the functions.
Verified against v1.5.11 in a scratch module: without the tag the exact CI
error reproduces (exit 1); with it, exit 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/base v1.5.11 (pulled in by da53ab30) turns a long-standing silent
mismatch into a compile error on purpose:
base@v1.5.11/core/sqlite_math_required.go:30:6:
undefined: cgoBuildNeedsSQLiteMathFunctions
base's search layer generates SQL calling acos/cos/sin/radians/sqrt (the
geoDistance token in tools/search). SQLite only has those with
SQLITE_ENABLE_MATH_FUNCTIONS, and the cgo backend gets them ONLY behind
csqlite's sqlite_math_functions tag — so a cgo build shipped a SMALLER SQL
surface than the code above it writes against, and the failure surfaced as a
customer's search returning 'no such function: acos' from an endpoint that
works in production. The file is //go:build cgo && !sqlite_math_functions and
references an undefined symbol so the two build modes cannot disagree in
silence.
Added to all four CGO_ENABLED=1 tag sites (the modernc gate, the two codec
tests, and the per-plugin build) plus the tag string quoted in the gate's own
error message. The CGO_ENABLED=0 builds are untouched: the pure-Go backend
always has the functions.
Verified against v1.5.11 in a scratch module: without the tag the exact CI
error reproduces (exit 1); with it, exit 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
rpc.go, dial.go and payloads.go were a second implementation of the op-call
plane zip already has: an fnv-hashed method registry, payloads packed by hand
against literal byte offsets, and a capability the callee parsed with nothing
verifying it. Sixteen methods lived there — the prepaid gate, the meter, the
balance, the welcome grant, the statement, the secret reads, the mailable
roster, the fleet — and not one of them appeared in the OpenAPI document, the
MCP tool list, the CLI or any SDK, because none of them was an op.
They are ops now. One registry, and the internal surface is as described as the
product surface. 1,300 lines of transport deleted; the contracts that remain are
types in plane/, a leaf both ends import so neither drags the other's graph.
The ops are declared on a SECOND app. A typed op rides every transport its app
listens on, and the host proxies edge traffic to its children over a private
socket, so neither "is this HTTP" nor "did this arrive on a socket" separates an
internal call from a public one. The plane app listens on exactly one address
and is never mounted on the edge router, so there is no path from the internet
to a plane op — the same way there is no path to a route nobody registered.
Identity rides the caller. zip.SocketPath is now the ONE socket path, used by
both halves. A background job with no request to forward states its tenant once
with cloud.For; cloud.As delegates a live request's principal, optionally
re-pointed at another tenant for the operator reading someone else's books. An
inbound request always wins, so a job can supply an identity and never launder
one. plane_test.go attacks the boundary over a real socket, and
TestNoPlaneInputCanNameAnOrg pins it structurally: no input may carry an Org.
The gate weighs the exact amount. metering.AuthInput gains a typed Amount beside
the int64 it had, and the comparison happens in the exact domain — a cents-
rounded charge let a sub-cent price round to zero and fall through to the bare
"any positive balance" gate. money.CentsUp rounds away from zero for the cap
surface, which still speaks cents, so a fractional spend is never weighed as
nothing.
The embedded tasks engine listens on a socket (tasks v1.52.4 EmbedConfig.Address),
which removes the free-port draw that seven of eight children used to lose. The
cluster-reachable gated listener stays TCP, because consumers in other pods dial
it and a unix socket does not leave the host.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The analytics row listed the READ endpoints and none of the six INGESTION doors
apps/analytics/event.go serves, so every beacon the products emit fell through to
commerce's bare "/v1" catch-all, which does not serve them. HTTP 405, silently,
for every event in the fleet.
The row was harmless while each app called its own routes(); killing the mega-build
made manifest.Apps THE router, so a missing prefix became an outage. It shows in the
warehouse: the last row landed 2026-07-29 04:15:29, eighteen seconds before the
ReplicaSet running v1.801.318 — the first image where this row is load-bearing.
Before that, 300-800 events per 15 minutes, continuously.
Bare /v1/analytics covers the batch door and the four read lenses; /v1/event covers
the Team SPA's /v1/event/collect suffix. /v1/tracker is deliberately NOT routed here:
apps/tracker owns that name for the issue tracker, so the capture alias is retired at
the caller instead. One name, one owner.
The router oracle caught the fix as a ratchet (FIXED, STILL LISTED) — ledger updated.
Nothing else moved: host stays 401 packages with zero apps/* imports.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two of team's twelve untyped routes are now typed ops — the ONE registry entry
the REST route, the OpenAPI operation's detail, the MCP tool, the CLI command
and the SDK method all read. team's published subset goes from 7 described
operations of 19 to 9, and POST /collaborator/rpc/{documentId} gains a request
schema and path parameter it never had: it was method + path and nothing else.
The wire is unchanged, and that is tested rather than asserted.
typed_wire_test.go pins every arm of both routes — `{"content":{}}` for a
getContent with no snapshot, a bare `{}` for updateContent,
`{"error":"unknown method x"}` for an unknown verb, `{"result":true}` plus the
expiring HttpOnly Set-Cookie for the DELETE, and the 400/401/404/503 gates —
and the SAME test file passes against the pre-conversion source. That is the
proof: the assertions were measured on the untyped handlers first.
collabResult.Content is a POINTER to its map on purpose. The three verbs answer
three different bodies on one 200 and the difference is load-bearing to the
client: createContent and getContent always carry `content`, possibly EMPTY —
the honest answer for a getContent with no source, which is a first-class case
since `source` is optional — while updateContent carries nothing and must stay
`{}`. A plain map with omitempty renders both as `{}`.
TEN routes stay untyped, each because typing it would move the wire, each named
at its registration with the reason, and the list is CLOSED by
TestEveryRouteIsTypedOrNamed (a new team route is typed by default, or it takes
a deliberate edit with a written reason):
GET /collaborator a WebSocket upgrade, not a value
GET /v1/team/transactor/{token} a WebSocket upgrade, not a value
POST /v1/team/account a JSON-RPC envelope: the verb is a
body field, `result` is a
different shape per verb, a
refusal is HTTP 200 carrying
{error: Status} INCLUDING for an
unparseable body, and the
entitlement arm answers 402
PUT /v1/team/account/cookie a body this route cannot parse is
IGNORED (the token falls back to
the bearer and the request
SUCCEEDS) where a typed In
answers 400
GET /v1/team/account/auth/{provider} a browser redirect: 302 +
GET .../auth/{provider}/callback Location + Set-Cookie, no body
GET /v1/team/billing/ui{,/*} the embedded wallet page's BYTES
under a per-asset Content-Type
POST /v1/team/files/{workspace} a MULTIPART form whose part
filename IS the blob id
GET /v1/team/files/{workspace}/{filename} the blob's raw BYTES under a
byte-derived Content-Type
LATENT DEFECT, surfaced by typing and fixed here: the collaborator plane is
app-level (/collaborator, because the front derives it from COLLABORATOR_URL),
and team's cloud.Bridge was scoped to the /v1/team group — so it never covered
that plane. A typed op receives only a context and reaches its caller ONLY
through the request Bridge parks, so the first typed op there would have 401'd
every call under a bare Mount (the app's own tests, and any embedder that mounts
without Serve). Bridge is now installed on the plane, AFTER the WebSocket
registration so the upgrade path is untouched, and
TestCollabRPCBridgedUnderBareMount is the regression bar.
Also surfaced: collabService had no `degraded` field. Mount's guard wrapper
carried the fail-closed 503 for both its routes, and a typed op cannot be
wrapped by a zip.Handler — so the field now exists and the op asks for itself,
exactly like billingService and filesService already do.
Identity: never an In field. tokenOf() joins sessionOf/admin/noStore/cookie in
typed.go — ONE file for the whole subsystem, which is why the cloud.Request pin
has one team entry instead of one per plane. The collaborator plane needs the
TOKEN and not just the tenant because it gates on the workspace claim too, and
the cookie writer is the other end of that same identity: the account-token
cookie is set on the RESPONSE, which only the request reaches. All of them fail
closed off the HTTP path. The pin's justification is updated to describe the
code that now exists.
Baseline: apps/team green before and after; the repo-wide suite fails the SAME
13 tests in the SAME 5 untouched packages (functions, platform, provisioning,
storage, kmsreseal) before and after, verified on a clean origin/main worktree.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Eleven /v1/books routes were untyped: a route and nothing else — no schema, no
prose, no MCP tool, no CLI command, no SDK method. Five of them are now typed
ops, and the document says so: 351 described operations, up from 346, with the
route table itself unmoved (1421 operations, 1005 paths, before and after).
Typing is a DESCRIPTION task; nothing here changes what a caller sends or gets.
POST /v1/books/ask AskRequest -> AskResponse
POST /v1/books/bank/sync syncIn -> BankTally
POST /v1/books/scan/book BookRequest -> BookResponse
POST /v1/books/vendors VendorRow -> VendorRow
POST /v1/books/rules Rule -> Rule
THE REASON THEY WERE STUCK WAS WRONG, and it is worth naming because it will
recur. Each carried the note "a typed POST documents its input as a body, which
would move the ?sandbox selector". That conflates the DOCUMENT with the BINDER.
zip binds a typed op's input from three sources in increasing authority — body,
then query, then path (typed.go:241) — for every method, so the selector was
never at risk on the wire. Only its DECLARED home was: zip's OpenAPI projection
emits query parameters just where there is no requestBody, so a Sandbox field on
a POST's In would publish a URL value as a body field. So the ops read it off
the request instead, through cloud.Request — the seam cloud/typed.go exists to
provide, and the one apps/agents/targets.go already uses for the facts a typed
signature drops. One helper, `query(ctx, name)`, in books/typed.go.
Guarded, not asserted: TestTheLedgerSelectorStaysOnTheURLForBodyWrites posts a
body that ASKS for the sandbox and proves it selects nothing, then proves the URL
still does. Naming `sandbox` on one of these Ins turns it red — which is exactly
when the wire would have moved.
SIX STAY UNTYPED, each measured rather than assumed.
metrics MetricsResponse EMBEDS Metrics. encoding/json flattens
an embedded struct; zip's schema walk nests it. Typing
it would publish a response no answer of the route
matches. TestMetricsCannotBeTypedYet measures both
shapes and goes RED the day zip learns to flatten —
read that failure as the go-ahead.
scan, inbox, bank/import RAW document bytes (PDF, image, OFX/QFX/CSV) as the
body. zip's decoder unmarshals a body as JSON, so
typing these would answer 400 to every upload.
bank/link-token, both answer 501 unconditionally. A typed op must state
bank/exchange what it answers on SUCCESS, and neither ever succeeds.
That last pair is a LATENT DEFECT typing surfaced, and it is not small: the
connectors behind those two routes are fully written — plaidConn.LinkToken mints
the Link session token, plaidConn.Exchange trades Link's public_token for the
durable access_token and seals it into KMS, tellerConn.exchange/linkConfig are
the Teller half — and NOTHING on the HTTP path calls any of them. Only their
tests do. The bank link flow is implemented end to end and unreachable: no org
can connect a bank through the API. teller.go even claimed "the route handler
(bankExchangeHandler) calls this"; it does not, and that comment is now true.
Wiring the handlers is a wire change (a route that has only ever answered 501
would start answering 200 with a body nothing has specified), so it is a
deliberate follow-up, not a side effect of describing the surface.
Eight schemas reach the published document for the first time — AskRequest,
AskResponse, BankTally, BookRequest, BookResponse, Figure, Leg, Voucher — and
the weave accepted all eight, so none collides with another app's meaning of a
name. Their fields carry prose now too, because a schema whose fields say
nothing is half a description.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
27 of this plane's 33 routes are typed ops (01e21366). The remaining six were
left with a written reason and nothing else, so the document said method, path
and path params about them and stopped. That is not the same fact for all six:
three of them still take ORDINARY JSON, and a route that cannot be a typed op
is not a route that must be undocumented.
All six refusals re-verified against zip v1.18.3's own source, not against the
comment claiming them:
PUT /workers/scripts/:script typed.go bindURL binds path AFTER body and
matches on the json tag, so path `script`
(the NAME) overwrites body `script` (the
module SOURCE). There is no per-field
opt-out; renaming either side moves the wire.
POST /pages/.../deployments typed.go op.invoke unmarshals whenever the
body is non-empty and 400s on failure. This
route IGNORES an unparseable body and falls
back to the production branch.
POST /d1/databases/:database/query the body is forwarded to D1 verbatim; a
typed In re-encodes from its own fields and
drops the rest, starting with params.
POST /ai/run/* wildcard model, model-defined body, and a
response that is frequently image or audio
bytes under Cloudflare's content type.
GET/PUT /kv/.../values/:key a KV value is opaque bytes under the
caller's own content type, both ways.
The first three are JSON on the wire and now declare their body through
openapi.Register — the seam that exists for exactly this case, adjacent to the
route table, reflecting the schema off the very struct the handler binds.
Pure description: no route, status, field, header or byte moves. The last three
have nothing to declare because they are not JSON.
PagesDeploy {branch} the handler's own struct, hoisted
out of the function so one value
is bound and published
WorkerScriptPut {script, mainModule, ...} already the struct the handler
binds; unchanged
D1Query {sql, params} the one declaration the handler
does not bind, and it says so on
itself: verbatim forwarding means
no struct it binds could state
the shape. OpenAPI objects are
open, so a field D1 takes that is
not named still reaches D1.
Response side is cfResult for all three, which register.go renders honestly
UNCONSTRAINED rather than as a shape this plane does not model.
LATENT DEFECT, found by declaring WorkerScriptPut and fixed here:
openapi/register.go published every json.RawMessage field as {"type":"string"}.
The []byte rule ("marshals as base64") fired before the custom-marshaler rule,
which lived inside the Struct case only — but the rule is about the MARSHALER,
not about being a struct, and json.RawMessage is a []byte that emits raw JSON.
So `bindings` would have told every generated SDK to send a base64 string for
the one thing it can never be. The check is hoisted above the kind switch; a
plain []byte still reflects as a base64 string, because a plain []byte has no
marshaler. Blast radius verified, not assumed: apps/platform is the only other
caller of openapi.Register, and regenerating its subset produces a byte-identical
file.
Also removes WorkerRouteCreate, dead since routeCreateIn replaced it in the
typed pass — an exported request struct nothing sends, describing a body
nothing reads.
TestUntypedJSONRoutesDeclareTheirBody reads the declaration back out of the
document the way a consumer does — resolving the $ref, through JSON — so the
three cannot quietly lose their shape again. openapi.yaml is +60 lines and
-0: three schemas and three requestBody/responses blocks, nothing removed.
Baseline: apps/cloudflare green before and after; apps/platform's
TestRunnerBuild_IAMReleaseRejected fails identically on origin/main
(want 403, got 502) and is untouched by this.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
openapi/weave_test.go looked for "an app serves a path the fleet routes
nowhere" and could not find one. It skipped any path already present in
openapi.yaml — the artifact it exists to protect — so the golden exempted
every defect it already contained, permanently; whatever survived that was
reported with t.Logf, which never fails a build. Both sides of the comparison
were DERIVED, and two derived artifacts agree with each other while both are
wrong. That is how plugin/ingress lost eight paths from every published SDK.
`go test ./openapi` printed zero UNROUTED lines and passed. The fleet routes
58 published paths somewhere other than the app that serves them.
So the question moves to the only thing that can answer it. manifest/
router_test.go builds the host's router from manifest.Apps — hand-authored
source — through the same zip.Load cmd/cloud calls, mounts every app on a
transport that answers with its own name, and asks it where each published
path goes. No routing is reimplemented: Load, Mount, the patterns and the
match are the fleet's own; only the wire is replaced.
The 58 are recorded, one line each, with the app that receives them instead.
They are a ledger of defects, not exemptions: a 59th fails the gate, and so
does an entry that stops being true, so the list can only shrink and a fix
nobody records is a fix nobody can see. The failure prints the current list
ready to paste.
The weave keeps composition and loses its routing opinion — the silent
delete it made was already dead (every subset path is in the golden), so the
document is byte-identical: 1005 paths, 505 schemas, 142 tags.
Proof: making dns claim /v1/dnsX turns the new gate red and leaves
TestFleetIsTheWeaveOfItsApps green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
DELETE /v1/keys and its deprecated alias DELETE /v1/iam/keys were the last two
addressable routes in this package that were a route and nothing else: method,
path and a tag in the published document, and past that no request schema, no
query parameter, no prose, no MCP tool, no CLI command, no SDK method. Both are
now one typed op — the single registry entry all five projections read. Nine of
this package's eighteen routes were typed; eleven are now, which is all of them
that have a shape to describe.
The reason they were left behind was real, and it is answered rather than waived.
A DELETE addresses what it deletes with its URL and carries no body (zip's
hasBody), while this route resolves the key class from `?type=` and FALLS BACK TO
THE JSON BODY. Binding the input and stopping there would have made a
body-selected revoke resolve to the empty string, default to secret, and destroy
the caller's session-equivalent credential in place of the publishable one they
named — a silent wire break on a credential-revocation route.
So the input declares the half the method has (`?type=`, which is what the
document now describes and a generated client fills in) and revokeClass reads the
other half off the request, where it always lived. The order is unchanged: query
first, body only when the query is absent. TestKeys_RevokeReadsTheClassFrom-
TheBodyWhenTheQueryOmitsIt pins both halves and the precedence between them;
deleting the fallback turns it red with the exact failure it exists to prevent
("the class in the body must reach IAM, got [secret]").
Everything else on the wire is byte-identical: the same 200, the same {ok,type}
body (a struct in the field order the map already serialised), the same 400 on an
unknown class, 403/501/502, and the same gates — `limit(csrf(…))` became the
`write` group and `deprecated(limit(csrf(…)))` the `aliasWrite` group, which
compose in that same order (zip's Chain).
Two coverage gaps the conversion surfaced, both on the gate it moved:
- No test held the revoke's CSRF gate. It is a money write that destroys a
credential, and the gate is a property of the GROUP an op is registered on —
invisible at the handler, and therefore droppable without anything looking
wrong. TestCSRF_AmbientWriteWithoutTokenIsRefused now covers all four key
writes instead of one.
- TestIAMKeysBeatsWildcard proved GET and POST beat clients/iam's /v1/iam/*
wildcard but never DELETE — the method where losing the race is worst, since
the request reaches IAM's own Guard, 401s, and tells the caller their key
still works when nothing tried to revoke it.
SEVEN routes stay untyped, and they are the seven catch-alls: GET|POST
/v1/billing/* and the five verbs of /v1/commerce/*. The path is a wildcard
remainder, the body is forwarded verbatim to commerce and the answer is
commerce's own bytes and status. There is no In and no Out to name — they are
opaque by construction, not by omission, and what they may reach is bounded by an
allowlist rather than by a type (billing.go).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An EMPTY leaf on a group composes to the group's prefix plus "/", so
`zip.Get(b, "", …)` on `b := g.Group("/blueprint")` declared the plane's root
at /v1/guide/blueprint/. op.Path IS the identity every projection reads, so
that trailing slash reached all of them: the document keyed the resource on
/v1/guide/blueprint/, the operationId (and therefore the MCP tool an agent
picks and the method a generated SDK exposes) was get_v1_guide_blueprint_,
and every generated client called the slashed URL. Fifteen sibling guide
paths carry no slash; the tests, the FE and the untyped PUT beside it have
always used the slashless form.
Nothing was red, because the router is non-strict: both spellings answer,
before and after. That is also what makes the correction wire-preserving —
TestBlueprintPathIsSlashless pins BOTH spellings at 200 and asserts the op
registry publishes no trailing slash, and it fails on the old registration.
The root is declared on g with a /blueprint leaf now, the same shape overview
already used to avoid naming /v1/guide/ ("declaring it on g would name
/v1/guide/, which this API never served") — the file had reasoned past this
exact trap one group up and walked into it one group down, which is the tell
that it is mechanical. The untyped PUT moves with it, or the document splits
one resource across two keys. Only the sub-paths hang off the group now,
where the leaf is non-empty and the composition is exact.
Repo-wide there is no second instance; LLM.md carries it as failure mode 8
with the grep, since every remaining typing tranche can hit it.
No route converted here: guide's remaining 6 untyped routes each name a wire
fact the declaration still cannot carry (verified, not assumed) — a YAML-or-
JSON document body that a JSON In would 400 (PUT /curriculum, PUT
/blueprint), a merge-patch whose explicit null DELETES a key that a pointer
field cannot tell from absent (PATCH /blueprint/{collection}/{id}), and the
structured 409 {error, step, blockedBy} that zip's {status, code, error}
envelope cannot express (POST /steps/{id}/start|done, plus SSE on /do).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`vouched` is the flag that skips DNS-01 domain-ownership proof entirely: a
vouched org's custom domain is BOUND live immediately (BindHost), an
unvouched one is CLAIMED pending behind a TXT challenge (ClaimHost), and the
"that is a host we operate" refusal only applies to the unvouched. So the
platform-operator set is an authorization boundary, not a label.
It was built and read with two different values. operatorOrgsFromEnv folded
every CLOUD_PLATFORM_OPERATOR_ORGS entry through sanitizeOrg — lowercase,
non-alnum→'-', truncate-32, no hash suffix, no refusal — while setDomains
looked the caller up VERBATIM, off principal.Org, which trims and clones and
deliberately never folds (projects.go org(): verbatim is what keeps two
distinct IAM owners off one S3 prefix).
Configuring "Acme" therefore wrote the key "acme", and a DIFFERENT tenant —
whoever's real IAM owner is literally "acme" — was vouched as PLATFORM
OPERATOR: it could bind any hostname live with no proof of ownership, and
claim hosts we run. Meanwhile the genuine operator "Acme" silently lost its
own vouch. "team.a" → "team-a" is the same collision, as is any pair of
owners differing only past 32 characters.
A fold applied to one side of a comparison is not a normalization, it is a
collision — and here the collision IS a cross-tenant privilege grant. Fix is
to delete the fold, not to move it: sanitizeOrg is gone (its only callers
were these two lines), and both halves are now the same verbatim owner,
TrimSpace and nothing else. There is one spelling of an org in this package.
Not renamed to orgLabel/displayLabel: that would assert a display-only
contract the function never had, and bury the defect behind a truthful-looking
name. Not keyed through cloud.SanitizeOrg either — an injective slugger on ONE
side of a verbatim lookup is the same defect with a better hash.
Tests: TestOperatorOrgsFromEnv is INVERTED in this commit — it asserted
got["acme"] for the input "Acme" and so locked the bug in; it now asserts the
verbatim "Acme"/"team.a" are present and the folded "acme"/"team-a" are NOT.
TestOperatorVouchIsVerbatimEndToEnd is new and drives the real route: with
CLOUD_PLATFORM_OPERATOR_ORGS="Acme", tenant "acme" gets a PENDING claim with
DNS records and a 403 on a host we operate, while operator "Acme" stays live.
Both fail on the parent commit — the e2e one reporting the lookalike bound
live, verified, with no challenge.
Deployment: no live behavior change. CLOUD_PLATFORM_OPERATOR_ORGS is unset in
every deployment (98 CLOUD_* keys in the operator CR, not one of them this),
CLOUD_BRAND is unset so brand = "hanzo", and sanitizeOrg("hanzo") ==
TrimSpace("hanzo"). IAM_ORG is "hanzo" verbatim, so the real operator matches.
From here both vars must carry the EXACT IAM owner: case and dots are kept.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/company was already 20 of 22 typed (93513f3f). This re-verifies the two that
are left against zip v1.18.3's own source rather than the comment that claimed
them, and writes down the floor so the next agent is not dispatched to redo it.
Both refusals hold, and each names ONE missing zip capability:
POST /fundraise/deck takes the deck as the raw request BODY (any content type,
named by ?name=). hasBody("POST") is unconditional (openapi.go:262) and
op.invoke json.Unmarshals whatever it is handed BEFORE the handler runs
(typed.go:227), so a typed In answers a PDF with
ErrBadRequest("invalid json body") — typing does not mis-document this route,
it BREAKS it. Waits on a raw-body binding.
POST /payment reads no body and its success path is already op-shaped (200 +
formationView); only the DENIAL blocks. cloud.DenyResource
(resource_billing.go:217) renders the fleet-wide {"error":{"code","message"}}
(402 insufficient_balance / spend_cap_exceeded, 503 balance_unavailable) and
zip's HTTPError (ctx.go:184) renders a flat {"status","code","error"}. Bridge
carries a STATUS back out, never a body, so the shim does not reach it — typing
would reshape that error for every metered client.
The audit did surface a real defect, via the check ec519f8c added one commit
earlier: company ships SIX instances of the bodyless-POST gap (#7). documents,
esign, genesis, kyc, kyc/refresh and skip each take noInput and therefore publish
requestBody:{required:true} over an object with no properties, for a body they
never read. That is the largest single share in the fleet — and running that
check over the committed subsets puts the class at 27 across 10 packages, not the
ten its prose had tallied. Counted, not enumerated: the instances anybody lists
by hand are the ones they happened to look at.
Everything else is clean: no embedded struct in any In/Out (the #7 field-dropping
class), no schema-name collision across the 30 names company publishes into the
flat namespace, zipdoc_gen.go and plugin/company/openapi.json both regenerate
byte-identical, and the fleet golden carries exactly the 21 paths the subset does.
Measured the MCP plane live: 20/20 tools described, and the 8 with no input
properties are exactly the noInput ops. deck and payment project nothing at all —
the cost of the refusal, paid knowingly.
LLM.md's tranche-B row was stale enough to misdirect work (company 22, actually
2; git 24 not 28, books 11 not 25, o11y 11 not 23) — re-measured with the
document's own command.
One doc-truth fix: the surface table stated POST /v1/company returns 201 flatly.
It is conditional (201 create, 200 idempotent) and the spec publishes 200; the
handler's own comment already said so.
No route, status, body or field name moves: description only. apps/company tests
green, zipdoc and the subset regenerate byte-identical, weave gate green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/framework is 17 of 19 typed and the remaining two — POST /v1/framework/:doctype
and PUT /v1/framework/:doctype/:name — stay raw. The recorded reason named only
half of what they need, which is the half that converts nothing.
Their body IS the document's own field data: an open object the DocType defines at
run time. Typing them takes two things, not one.
1. zip must be able to DECLARE an open object. It cannot. schemaOf has no
reflect.Interface case, so map[string]any falls to the default and projects
additionalProperties: {"type":"object"} — every VALUE is a JSON object. This
is already SHIPPED and already FALSE on the four ops that return a document:
http_test.go reads back {"subject":"Ship framework","docstatus":0}, a string
and a number, past a schema that admits neither. openapi.yaml carries the
claim in 15 places fleet-wide. Unlike multi-status and the bodyless POST,
which under-describe a true wire, this describes a false one — an SDK
regenerated from the golden types a document Dict[str, Dict], a shape that
cannot hold one. hanzoai/openapi's authored master has it right
(framework_Document: additionalProperties: true), so the two documents
genspec joins disagree about the same value.
2. bindURL must be able to BIND the URL onto one. It cannot: it returns early
unless the In is a struct, so an open-object In carries no :doctype/:name
while a struct In carries no document.
So the two convert together or not at all, and typing them today would replace a
correct authored request shape with a reflected one naming the path segments and
nothing else — an SDK method that cannot send a document. A schema that lies is
worse than the route-only entry they carry.
The refusal now lives with both halves named, in the test that already checks it
(rawRoutes) rather than only in prose. LLM.md gains the false-schema failure mode
as #8 and framework as the second worked SPLIT tranche.
No route, type or artifact changes: the wire, zipdoc_gen.go, plugin/framework/
openapi.json and openapi.yaml are all untouched and regenerate byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/o11y finished its typed migration: 12 of 20 routes are ops, and the other
8 cannot be without moving the wire (VM's verbatim status+envelope, three
reverse proxies, two text/plain alert receipts, one wildcard). Re-verified each
against the source; nothing left to convert.
What the audit did surface is that being typed is necessary and not sufficient.
POST /v1/o11y/ingestion is a typed op with In/Out and lifted prose, and it is in
neither plugin/o11y/openapi.json nor the woven openapi.yaml — so the LLM-obs
write path has no SDK method, no MCP tool, no CLI command and no schema.
mountEventIngest registers it only behind a reachable Hanzo Datastore, and the
process that writes the document has none; the generator says so in its own log.
Recorded in LLM.md rather than fixed: zip.Post registers a fiber route and a
registry entry inseparably, so making the op visible necessarily stops the path
falling through to the order-70 wildcard. That is a behaviour decision, and
typing is a description task.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/integrations measures 45 untyped routes and serves 19. The other 26 hits are
hdr.Get("Retry-After") and the `// app.Post(…)` MOUNT HANDOFF blocks each adapter
file carries, so the tranche table's "integrations 47" would have sent the next
agent on a pass with nothing to convert.
All 19 are refusals, and checking each against zip v1.18.3 rather than against the
comment found the comment wrong. It grouped Teams and Telegram under "auth is a
signature over the RAW request bytes". Neither is: Teams verifies a Bot Framework
JWT header, Telegram a shared secret header. Their real blocker is a different
wire fact — both answer an EMPTY 200 to a body they cannot parse so the platform
does not retry-storm, and zip's invoke unmarshals BEFORE the handler
(typed.go:227), which turns that 200 into a 400. For telegram it also inverts the
auth order, leaking a parse result to a caller that today gets 401 first.
That prose is load-bearing: it is what the next agent reads to decide whether a
route converts. A refusal filed under the wrong reason is a refusal nobody can
re-check, so the taxonomy now cites the line numbers it rests on.
Three families, not two, and the HTML one was missing entirely:
- 8 legs answer 302; zip.WithStatus PANICS on a non-2xx (typed.go:104).
- 5 answer text/html and set __Host- cookies; a typed dispatch ends in
c.JSON(out) and an op holds no response to set a cookie on.
- 6 inbound webhooks, splitting on WHY: four are signed over the raw received
bytes (Slack/GitHub HMAC, Discord Ed25519), two are the 200-on-unparseable
pair above.
Also records six new instances of the bodyless-POST gap (#7) that this package
already ships — /connectors/{id}/refresh, /device/{flow}/poll, /pages/builds,
/{provider}/disconnect, /{provider}/verify each publish a required body whose only
properties ARE their path params, and /telegram/connect publishes one over noArgs,
an object with no properties at all. The class is ten now and grows with every
tranche, so LLM.md carries a check that reads the published subsets rather than
the source.
No route, status, body or field name moves: description only. Verified against the
baseline — apps/integrations tests green, zipdoc -check clean, the regenerated
subset byte-identical, weave gate green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three claims in "Generated and frozen artifacts" were false, and each one
pointed the next reader at work already done:
- "no gate in this repo uses -check yet" — `make test` has run
`zipdoc -check` per package, on every package carrying the directive,
for some time. Read as written it invites a SECOND gate.
- "mk/plugin.mk:45-46 still says they are not committed" — that comment
now says the opposite ("The files ARE committed today, deliberately"),
so the fix it asks for is a regression.
- "15 zipdoc_gen.go files" — 27, 1:1 with the directives.
Line-number citations (Makefile:214, Dockerfile:178) are dropped rather
than corrected: both had already drifted, and a file+target names the value
where a line number names a place that moves under it.
Measured on apps/git while auditing its untyped routes: 48 operations, 24
typed, 24 route-only — and all 24 are structurally untypable (HTML pages,
binary pack streams, an HMAC-over-raw-bytes webhook, a ZAP envelope whose
error body is non-2xx). Nothing to convert there; this was the one real
defect the audit surfaced.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every route apps/ingress serves is already a typed op (18 of them), and the
projection test pinned three of the four surfaces that one registration feeds:
the live router, the registry the CLI reads, and the prose in the OpenAPI
document. The fourth was unpinned, and it is the one that fails SILENTLY.
zip's tool list once read op.Summary — a field cloud sets nowhere, because the
handler's doc comment is the source — so all 164 MCP tools served an empty
description over a nameless schema while the spec looked perfect. It was fixed in
zip v1.17.6 by reading the same docFor extraction its siblings read, and an older
zip reverts it invisibly. TestSpecCarriesProse cannot see that: it reads a
DIFFERENT field of the same registry entry.
So the pin asserts what an agent actually receives: 18 tools, every one carrying
its handler's prose, and every op that takes input NAMING its fields (id from the
URL, the object's own fields from the body). The five no-input ops — status, tls
and the three lists — take nothing off the wire, so an empty schema is the truth
for them and the test says so out loud.
It is not vacuous: a phantom tool name and a phantom schema field each fail it,
checked before committing.
Test-only. No source, no route, no wire, no regenerated artifact changes. Verified
on this surface at the same time: 0 untyped routes left in apps/ingress, zipdoc
-check clean, plugin/ingress/openapi.json regenerates byte-identical, and the
fleet golden carries all 18 ingress operations described, over 10 schemas whose
every field is described.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/crm finished its typed migration in a15f5ca3: 19 of 20 routes are typed ops
and the public intake stays a raw handler, because its 20/min-per-IP limiter and
its 64 KiB pre-parse body cap are HTTP-plane facts an MCP or CLI projection would
not run — typing it would publish an unauthenticated, unmetered alias of a
deliberately metered endpoint, and move the size cap after the parse it exists to
prevent. Verified end to end: 19/19 ops carry a description and an input schema in
both the OpenAPI golden and the MCP tool list, and the intake appears in neither.
What that migration left unguarded is registration ORDER. The limiter is a second
app.Group("/v1/crm", …), so it is a prefix-scoped Use covering every /v1/crm route
registered after it and none registered before. crm.go says so in prose —
"REGISTRATION ORDER IS WIRE HERE" — and nothing enforced it. Measured, the current
order is exactly what the prose claims:
* POST /v1/crm/applications and the three staff /applications routes are metered
* every companies/contacts/opportunities/summary op is not
Both halves matter, and each fails in a different direction. The natural next step
of this very migration — typing the intake — moves that registration across the
line and silently un-meters a public form. The mirror slip throttles the CRM's own
CRUD to 20 requests a minute per IP, which no console could use. The type system
says nothing about either, so the test does; it goes red on the mutation that
moves the limiter above the CRUD ops.
Also gofmt: applications.go and applications_test.go were not gofmt-clean on main
(composite-literal key alignment). No wire, no golden, no zipdoc change — the
lifted prose regenerates byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cicd.yml called hanzoai/ci/.hanzo/workflows/build.yml@v1. At tag v1 that
directory is EMPTY; build.yml lives under .github/workflows/, which is also the
import its own header documents. So the forge could not construct a run:
PrepareRun: InsertRun: read hanzoai/ci@v1:.hanzo/workflows/build.yml:
object does not exist
InsertRun fails before any run row is written, so there is no failed run to
look at. Pushes and workflow_dispatch alike silently did nothing — dead CI that
is absent rather than red. hanzoai/cloud's last run of ANY kind was
2026-07-26T07:59:31Z; every commit since landed with no build at all, which is
why the live image sat at v1.801.318.
Points at the path that exists. Moving the file in hanzoai/ci instead would
mean moving the v1 tag, which we do not do.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cicd.yml called hanzoai/ci/.hanzo/workflows/build.yml@v1. At tag v1 that
directory is EMPTY; build.yml lives under .github/workflows/, which is also the
import its own header documents. So the forge could not construct a run:
PrepareRun: InsertRun: read hanzoai/ci@v1:.hanzo/workflows/build.yml:
object does not exist
InsertRun fails before any run row is written, so there is no failed run to
look at. Pushes and workflow_dispatch alike silently did nothing — dead CI that
is absent rather than red. hanzoai/cloud's last run of ANY kind was
2026-07-26T07:59:31Z; every commit since landed with no build at all, which is
why the live image sat at v1.801.318.
Points at the path that exists. Moving the file in hanzoai/ci instead would
mean moving the v1 tag, which we do not do.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`tags: ["v*"]` sat under `workflow_dispatch:` rather than `push:`, where it
means nothing, and a SECOND `workflow_dispatch:` key below then overwrote that
whole mapping — so the filter was dropped twice over. YAML takes the last
duplicate key and reports nothing, so the file parsed, the workflow ran on
main pushes, and the tag trigger was simply absent. Effective `on:` was
{push: {branches: [main]}, workflow_dispatch: None, pull_request: None}
Two edits landing on the same block at different times is all it takes, and
nothing in a normal parse tells you. Both comments are kept; the dispatch
trigger now carries the sync note as well as the on-demand-rebuild one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`tags: ["v*"]` sat under `workflow_dispatch:` rather than `push:`, where it
means nothing, and a SECOND `workflow_dispatch:` key below then overwrote that
whole mapping — so the filter was dropped twice over. YAML takes the last
duplicate key and reports nothing, so the file parsed, the workflow ran on
main pushes, and the tag trigger was simply absent. Effective `on:` was
{push: {branches: [main]}, workflow_dispatch: None, pull_request: None}
Two edits landing on the same block at different times is all it takes, and
nothing in a normal parse tells you. Both comments are kept; the dispatch
trigger now carries the sync note as well as the on-demand-rebuild one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every /v1/cloudflare route was a raw fiber handler, so the published document
carried method + path + path params and nothing else: no request schema, no
query parameters, no prose, no MCP tool, no CLI command, no SDK method. 27 are
now typed ops declared on the group, which is the ONE registry entry all five
projections read.
The wire is unchanged. Every handler keeps the exact gate it had (verified
handler-by-handler: 33/33), the same statuses and messages, the same upstream
Cloudflare paths and bodies, and the same acting-org stamp. The response is
still Cloudflare's own payload relayed verbatim — cfResult marshals the raw
upstream bytes, so field order, unmodeled fields and integers past float64
survive untouched (pinned in TestRelayIsVerbatim).
SIX routes stay untyped, each because typing it would move the wire, and each
named at its registration with the reason on the handler:
POST /ai/run/* the response is often not JSON at all
(image/audio bytes under CF's own
content type) and the body is the
model's, forwarded verbatim
GET/PUT /kv/.../values/:key a KV value is opaque bytes under the
caller's own content type
POST /d1/databases/:database/query the body is forwarded to D1 VERBATIM; a
typed In drops params and batch fields
PUT /workers/scripts/:script path param `script` (the NAME) collides
with body field `script` (the SOURCE),
and zip's URL binder gives the path the
last word
POST /pages/.../deployments an unparseable body is IGNORED here (the
deploy falls back to the production
branch); a typed In answers 400
TestEveryRouteIsTypedOrNamed closes that list: a new route here is typed by
default, or it takes a deliberate edit with a written reason.
Identity: cloud.Bridge() on the group, principal.OrgFrom(ctx) for the tenant —
never an In field, which is caller-supplied. The org-admin bit lives in a
header principal.OrgFrom does not carry, the acting-org stamp is a response
header, and ?account= must NOT become an In field (zip binds an In field from
the body too, and this route has never accepted an account there), so the
plane is pinned in allowedRequestUses with those three reasons. authWrite
fails closed off the HTTP path.
Known imprecision, reported not hidden: cfResult renders as
{"type":"object"} because zip's schemaOf has no vocabulary for "any JSON" —
json.RawMessage reflects as an array of integers, which is why cfResult is a
struct at all. For the list endpoints the document therefore says object where
Cloudflare answers an array. A zip patch teaching schemaOf that
json.RawMessage means `{}` upgrades all 27 with no change here.
Baseline check: apps/cloudflare green before and after; the repo-wide suite
fails the same 12 tests in the same 5 untouched packages (functions, platform,
provisioning, storage, kmsreseal) before and after.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The embedded server took NATS's own 1 MiB max_payload because nothing set it —
production advertised max_payload=1048576. That is the hard bound on everything
riding the bus, and the Kafka-wire adaptor rides it: insights-plugin's ingestion
loop produces downstream, a >1 MiB record failed with 'Broker: Message size too
large', the plugin treats that as an unhandled rejection and exits(1), and it
crash-looped — 127 restarts.
The failure is on the PRODUCER, which is why an earlier fix aimed at the
consumer could not clear it: universe's insights deployment already raises
STREAM_CONSUMER_MAX_PARTITION_FETCH_BYTES to 10 MiB, correctly, for the fetch
path. The bus itself was always the ceiling.
pubsub v1.0.0 -> v1.4.5 adds embed.Options.MaxPayload (default 8 MiB, tested
against what a client is actually advertised). CLOUD_PUBSUB_MAX_PAYLOAD
overrides it; a non-positive value is refused at Mount rather than silently
falling back.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The pricing surface had no typed op at all: 32 routes, no schema, no prose, no
MCP tool, no CLI command, no SDK method. 15 of them are now ops, so each is ONE
registry entry the document, the tool list, the CLI and the generated clients all
follow from. plugin/pricing/openapi.json: 32 operations, 15 described, up from 0.
WIRE, VERIFIED BYTE FOR BYTE. A probe drove every route on this surface under
four identity shapes (anonymous, member, SuperAdmin, and the forged X-Org-Id with
no principal the enablement attack tests pin), before and after, with an empty
overlay and a populated one, and diffed status + Content-Type + body. Every 2xx
body is byte-identical — same keys, same order, same values, same sha. That is
not luck: each Out struct replaces a map[string]any, Go emits a map's keys
sorted, so every one of them declares its fields in that same order.
The whole delta is on error paths, and it is one fact:
404 {"error":"Model not found: zen4"} -> {"status":404,"error":"Model not found: zen4"}
401 {"error":"admin required"} -> {"status":401,"error":"admin required"}
A typed op states an error as an error, and zip's HTTPError carries the status in
the body. Same HTTP status, same message, one added field — and it is the shape
this surface already emitted for its 403s (zip.ErrForbidden was already in
admin.go and enablement.go), so the change is toward one error shape, not away
from it. The unknown-model 404 additionally moves Content-Type from
"application/json" to "application/json; charset=utf-8", which every other JSON
answer on this surface, including that same 403, already sent. POST
/v1/pricing/sync's 500 loses its second field ("message"); zip's error carries
one message, so the field both shapes have keeps exactly the text it had and the
cause is logged instead of returned.
RAW ROUTES LEFT: 17, each for a reason that is a property of the wire.
- The fourteen /v1/pricing/{compute,cloud/*,subscriptions,blockchain,iam,base,
paas,policy,tools,gpu} routes and /v1/pricing-policy are a VERBATIM PROXY of
the @hanzo/pricing goja bundle: the bundle picks the status (200, or 503 when
a section is absent — six of them have that branch) and its bytes are written
unmodified. A typed op answers the one status it declared, over a Go
re-marshal. Two wire changes, so they stay raw. The gated routes typed above
do NOT have this property: they already decoded and re-marshalled through Go,
so typing them is pure description.
- PATCH /v1/admin/catalog/models/* addresses model ids containing '/', so it
routes through a greedy wildcard. fiber names that parameter `*1` and the
document renders it {wildcard1}; binding it needs an input field tagged
json:"*1", which every projection would then publish. A schema nobody can read
is worse than none.
- PATCH /v1/admin/catalog/providers/:name carries `overrides`, a raw JSON merge
patch (RFC 7386). zip reflects json.RawMessage as an ARRAY OF INTEGERS — it is
[]byte — so typing it publishes a false schema; and retyping the field to
map[string]any moves {"overrides":null} from "clear the override" to "leave it
alone", which is a wire change.
LATENT DEFECT, and the reason for the second half of this commit: pricing serves
FIVE prefixes and declared ONE. Its plugin spec named no Prefixes, so
MountPrefixes fell back to the /v1/<name> convention and cloud.Declare built its
tracing/price table with /v1/pricing alone — /v1/pricing-policy, /v1/enablement,
/v1/admin/catalog and /v1/admin/enablement resolved to another subsystem's prefix
or to none, mislabelling their spans and leaving their price unanswered. It also
meant the subsystem could not install middleware on four fifths of itself:
scope.Use installs once per DECLARED prefix. manifest/apps.go, which the light
host routes by, had the correct five all along — the app's own composition root
disagreed with it. pricing.Prefixes now states them once, in the app, and
plugin/pricing/main.go reads it.
Bridge: Serve installs one app-wide, so production always had it, but this
subsystem's own tests mount it on a bare zip app and would have exercised a
different identity path than production. It installs its own now (nesting is
harmless — the inner one is what the handler sees), which is what makes the
existing admin/enablement HTTP tests, including the two cross-tenant attack
tests, prove the typed path and not a weaker one.
cloud.Request grows a fifth call site, pinned with its justification:
callerIsAdmin. Every read op here branches on admin-ness (an admin sees disabled
models, flagged, where a customer sees them hidden) and that bit lives in a
header principal.OrgFrom does not carry. The tenant itself is read with
principal.OrgFrom, never through the request and never from an In field.
Found while typing, NOT fixed here because it is another app's file:
apps/agents/targets.go's patchTargetIn embeds the unexported patchTargetReq, and
zip's structSchema skips unexported fields — including an embedded unexported
struct. encoding/json still promotes them, so PATCH /v1/agents/targets/:id works;
but its published schema has exactly one property, `id`, and openapi.yaml has
said so since it landed. Every SDK, the MCP tool and the CLI command for that op
offer only `id` — label, kind, status, capacity, host, spec and metrics are
invisible. The ops here flatten their fields rather than embed, so none of them
reproduces it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/guide had a route table and nothing else: no schema, no prose, no MCP tool,
no CLI command, no SDK method. Thirteen of its nineteen routes are typed ops now
and carry all five. The other six each name a wire fact the declaration cannot
yet make, and say so at the registration rather than being quietly skipped.
Typed: GET /v1/guide, /analytics, /profile, /strategies, /suggest, /curriculum,
/actions, /blueprint, /blueprint/versions; POST /chat, /steps/:id/skip,
/steps/:id/reset; DELETE /curriculum.
Left untyped, with the reason:
- PUT /curriculum and PUT /blueprint take a YAML-**or**-JSON document
(sigs.k8s.io/yaml). A typed In is decoded as JSON before the handler sees it,
so typing them would answer 400 to every YAML body they accept today.
- PATCH /blueprint/:collection/:id takes an opaque JSON merge-patch whose keys
are the patched item's own — not a declarable In.
- POST /steps/:id/start|done answer a blocked step with a STRUCTURED 409
({error, step, blockedBy}) written in-band. A typed op's only non-2xx is the
error it returns, whose envelope is a different shape. They convert with zip
multi-status (#78).
- POST /steps/:id/do also STREAMS SSE, and an op answers exactly one JSON value.
The gated/ungated split is the discriminator worth copying: skip and reset pass
gate=false, so the 409 branch is unreachable for them and their whole answer set
is expressible. One shared body (applyStep) serves both halves; the gate is a
parameter and the blocked case is a VALUE the untyped pair renders.
Wire preserved exactly, and checked rather than assumed:
- every response is the same JSON (maps became structs with the same keys);
- GET /strategies still binds category/stage/workload from the query, and now
DECLARES them;
- DELETE /curriculum takes no body, before and after;
- the SuperAdmin 403 on the blueprint plane is now one shared errNotSuperAdmin,
so the untyped wrapper and the typed ops cannot drift into two refusals;
- the URL stays the addressing authority on skip/reset — a body naming another
step cannot redirect the write (TestTypedStepOpsFailClosed pins it).
Two things typing surfaced that nothing else would have:
1. openapi.Weave REFUSED the whole fleet document: guide's Step and marketing's
Step are one schema name with two shapes, and a generated SDK would bind
whichever it read last. guide's type is JourneyStep now — marketing's is
already published and guide's was not, so guide yields. No wire change: the
JSON keys live on the fields. The comment on the type says why, so nobody
"simplifies" it back into the collision. LLM.md's failure mode 5 gains this
second instance and a shape-aware scan, since a name-only grep passes when two
apps legitimately agree.
2. zip cannot declare a bodyless POST (hasBody is unconditional), so skip and
reset publish a requestBody they never read. The wire is unharmed — bindURL
binds the path LAST, so the URL still names the target — but the document
asserts something false. apps/admin already ships two of these. Logged as
failure mode 7; same shape of gap as multi-status.
Three operationIds move (get_v1_guide_blueprint_, post_v1_guide_steps_id_skip,
..._reset). That is failure mode 6 and the house rule is explicit: TAKE the
rename, never pin it back with WithOperationID, which would make one app's ids a
special case. It is not the wire — no status, body or field name moves. The
registration says so where the next reader will look.
apps/guide gets its //go:generate zipdoc directive (it had none, so its prose
could never have reached the spec) and cloud.Bridge on its own subtree, so the
validated org reaches an op that receives only a context — never an In field,
which is caller-supplied.
TestRequestEscapeHatchIsPinned fired on this change, which is the gate working:
superAdminOK and ledgerOf are new cloud.Request sites, justified in the allowlist
rather than waved through — admin-ness lives in X-User-IsAdmin and the payer is
the SELECTED billing org, neither of which principal.OrgFrom carries, and both
fail closed off the HTTP path.
Green: apps/guide, openapi (the weave), the root package, manifest, cmd/cloud,
apps/marketing; go build ./...; zipdoc -check; openapi.yaml + plugin/guide
regenerated from source and clean under the drift gate.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Typing apps/crm surfaced a trap the playbook did not name, and it is already live
in the committed golden three times over: zip's structSchema skips every field
reflect says is unexported, which an EMBEDDED unexported type is — while
encoding/json promotes those same fields onto the wire. The route works, the test
passes, and the published schema is missing most of the payload. That is the one
direction nothing catches.
patchTargetIn (apps/agents/targets.go) -> publishes {id} alone, so
PATCH /v1/agents/targets/{id} documents none of label, kind,
status, capacity, host, spec, metrics — not in openapi.yaml,
not in a generated SDK, not in the MCP tool's inputSchema.
botView (apps/visor/bots.go) -> {agent,binding}, dropping the 14
machine fields it embeds.
clusterDetailView (apps/visor/k8s.go) -> {nodes} alone.
Exporting the embedded type is not the fix — zip then publishes a NESTED object
the wire does not have. Spell the fields at the top level, or teach structSchema
to flatten embedded structs the way the decoder does, which fixes the class.
Also records how zip picks the OpenAPI `summary` (first ". ", else the first
LINE), because a first sentence wrapped across two lines gets cut mid-sentence.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/crm was 20 raw fiber handlers: a route and nothing else, so /v1/crm/* had
no schema, no prose, no MCP tool, no CLI command and no SDK method. 19 of the 20
are now typed ops, which is ONE registry entry with N projections — the REST
route, the OpenAPI operation, the /mcp tool, the CLI command and every generated
client all follow from the same declaration.
The wire is unchanged, and that is the point of the exercise:
* 201 on the three creates, declared with zip.WithStatus(201) rather than set
per request, so the document keys its response on the code the route sends.
* 204 with no body on the three deletes — a typed op says that by returning a
nil Out.
* {"data": [...]} list envelopes, the {companies,contacts,opportunities}
summary, and every field name kept verbatim.
* ?limit=, ?companyId= and ?stage= now bind off the In instead of c.Query, and
still trim, case-fold and bound exactly as before.
* ids still bind from the PATH, which is the addressing authority — a body
cannot smuggle a different target past the org gate.
cloud.Bridge() is installed on the /v1/crm group, ahead of its leaves: a typed
op is handed only a context, so the VALIDATED org has to be parked there. It is
never an In field — an In field is caller-supplied, so a tenant key read from one
is a cross-tenant read the caller asserted for itself. Every op resolves its org
through principal.OrgFrom and fails closed off the HTTP path, so an MCP tools/call
or a CLI invocation with no principal gets the same 403 an anonymous REST call
gets. TestRed_NoPrincipalForgedOrgRefused still passes unchanged.
POST /v1/crm/applications stays a raw handler, deliberately: the public intake's
protections are an IP rate limit (HTTP middleware, which the MCP and CLI
projections do not run) and a 64 KiB body cap (which can only refuse BEFORE the
decode a typed op is handed). Typing it would publish an unmetered alias of a
deliberately metered public endpoint. It converts when zip can carry both.
One doc-truth trap the conversion surfaced, avoided here and live elsewhere:
zip's schema builder skips unexported struct fields, so an In that EMBEDS an
unexported request struct publishes a schema carrying only its own fields. Every
In here spells its fields at the top level, so all 15 published schemas are
complete.
Verified: apps/crm tests green (incl. a new TestTypedWire pinning 204-no-body and
the query bindings), the intake rate limiter's reach probed byte-identical on
HEAD and on this tree, plugin/crm/openapi.json + openapi.yaml regenerated from
source, and go test ./openapi green — 1006 paths, unchanged; 15 schemas and 19
descriptions added where there were none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/o11y/{logs,metrics,status}, the eight /v1/o11y/annotation-queues* routes and
POST /v1/o11y/ingestion are typed ops now, declared on the /v1/o11y group. Each
is ONE registry entry, so the OpenAPI schema, the MCP tool, the CLI command and
the SDK method all follow from it where there was a URL and nothing else. The
document gains 11 described operations and, for the first time, the request and
query SHAPE of this surface: `?product`, `?sinceNs`, `?window`, `?limit`,
`?range`, `?stepSec`, `?status`, `?page`, `?limit` are declared parameters with
types and examples, and the annotation-queue bodies have schemas.
WIRE PRESERVED. 1005 paths before, 1005 after — none added, none removed. The
clamps now take the decoded value instead of parsing the raw string, which lands
on the same branch (`?limit=abc` binds as 0, which IS "no limit given"). The two
creates keep their 201 via zip.WithStatus; the DELETE takes its id from the URL
and no body, as it always did. The app's own tests — 403 forged, 400 malformed
product, honest-empty, the full queue lifecycle with 201/200/404/409/400, org and
project isolation — pass unchanged.
Three latent defects surfaced, all fixed here:
1. NO cloud.Bridge IN THE o11y PROCESS. o11y runs as its own binary
(plugin/o11y/main.go builds a bare zip.App), and a context value does not
cross the host→plugin socket. cloud.Serve's app-wide Bridge parks the org in
the HOST, so every typed op in the child would have answered 403 to a caller
the host had already validated — a total outage of the surface, not a
degradation. MountO11y installs its own on the /v1/o11y group, first.
Pinned by TestTypedOpsResolveTheirOrgThroughTheBridge, which fails without it.
2. ONE NAME, TWO SHAPES: `usagePoint`. o11y's per-bucket {t,calls,tokens,
costCents} collided with admin's daily {date,requests,spendCents,tokens} in
the fleet document's single schema namespace — a generated SDK would bind
whichever it read last. o11y's renames to `usageBucket` (its name had never
been published; admin's has). Caught by the weave gate the moment the type
entered the document.
3. AN EMBEDDED STRUCT PUBLISHES A SCHEMA WITH HOLES. encoding/json flattens an
embedded struct; zip's schema builder skips it, so annQueueDetailView would
have published 3 of its 9 fields and the PATCH bodies would have published
none of theirs. The typed shapes are written FLAT — same bytes, and now the
schema says so. (apps/agents/targets.go has the same pattern: patchTargetIn
publishes only `id` today, so the whole patch body is missing from the
published spec and from every SDK. Not touched here — it is that app's diff.)
Eight routes stay untyped BECAUSE typing them would move the wire, and each is
named with its reason in apps/o11y/LLM.md: the two VM proxies return
VictoriaMetrics' own status and envelope verbatim; the two builder queries and
the sessions list are reverse proxies with no Go type for "whatever the runtime
answered"; the two alert routes are text/plain and the receiver deliberately
ACCEPTS an unparseable body (a body that will not parse still proves delivery,
and a 400 would make Alertmanager retry forever); /v1/sentry/* is a wildcard.
Also noted, not fixed: POST /v1/o11y/ingestion is registered only when a
datastore DSN is present, so it is absent from the published document — the
route exists in production and no generated client can reach it.
The three cloud.Request uses this needs (platform-sudo, validated-ness, project
scope — none of which principal.OrgFrom carries) are concentrated in one seam
file, apps/o11y/typed.go, added to allowedRequestUses with its justification.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
There were two `hanzo code`s. The real one is hanzoai/cli: it runs the agent
HEADLESS, parses its JSONL event stream, persists a resume handle and a
transcript pointer, relaunches with `--resume`, and answers pause/resume/stop/
message from the cloud control plane. This one only exec'd a binary and handed
over the terminal, so by construction it could never read a single event.
Its four unshared capabilities — the zen⇒carrier model map, a config home
separate from the user's `~/.claude`, that home's first-run seeding, and the
identity a carried model needs — now live in hanzoai/cli (v1.9.8), verified
against a live model. Nothing is lost, so this is a deletion and not a
deprecation: no shim, no alias.
Also gone, because they configured only this command: bare `hanzo` dropping
into an agent, the `code_tool` and `code_model` config keys, and their two
Config fields. `code_model` was already dead — it was settable and never once
read, even here.
The REST of this package stays. `cli/gpu.go` is a 2,364-line GPU link daemon
(heartbeat/claim/execute, nvidia-smi / rocm-smi / kfd-topology parsing, ComfyUI
supervision, systemd install) that has no Rust counterpart — `hanzo node join`
is a 90-line one-shot registration and says so itself. Until that is ported,
this code is the only specification of the work it does, so deleting it would
destroy the spec along with the duplicate.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A tenant's team page called /v1/iam/get-users. get-users is org-scoped but not
org-KEYED, so the edge's pin never reached it and the request was forwarded BARE
under cloud's ONE service credential. A bare read is not unscoped — IAM scopes it
to whatever org that credential resolves to — so every tenant was served the
credential's org: measured live as 262 rows, all owner=hanzo.
The same three-segment allowlist left add-user/update-user/delete-user without a
body check, so a tenant org-admin could name a foreign owner in a write body.
Both are now one rule: every gated segment that is not org METADATA (guarded by
NAME, its object owned by "admin") carries the caller's own org explicitly — in
the query, in the id, and in the write body. A super admin is still unpinned.
This is also the gate on IAM v1.33.31. Once Scope honours-or-refuses, a bare read
still answers from the credential's org, and the naive fix — giving the edge a
credential that can cross orgs — is strictly WORSE: IAM's listHandler calls
Scope(ctx, ""), which returns "" for a super principal, and then applies no
Owner filter at all. Unpinned + super = every tenant in one response. The pin is
what makes either credential safe, so it lands FIRST.
Found while collapsing the two IAM lineages: the edge is the boundary between
them, and it was trusting the far side to decide scope.
Tests: the leak and the write hole both reproduce red on the parent commit.
TestEdgePinRidesInTheId pins a break the first fix introduced — IAM's ReadTarget
only falls back to ?id= while ?owner= is empty, so a blanket owner pin would have
turned every id-addressed read into "id (owner/name) or name is required".
TestEdgeRefusesBareId records a real semantic divergence across the hop (cloud
reads a bare id as an OWNER, IAM as a NAME) and keeps the stricter side.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/account had eighteen routes and no typed op, so every one of them was a
route and nothing else: no schema, no prose, no MCP tool, no CLI command, no SDK
method. Nine are now typed ops — ONE registry entry each, which the REST route,
the OpenAPI operation, the MCP tool, the CLI command and every generated SDK
method are projections of:
GET /v1/csrf POST /v1/keys
GET /v1/keys POST /v1/iam/keys
GET /v1/iam/keys POST /v1/iam/onboard
GET /v1/embed-status POST /v1/commerce/topup/wallet
GET /v1/commerce/topup/rails
Measured: the account subset went from 0 described operations and 0 schemas to
9 and 12; the fleet golden gained 55 descriptions with 1006 paths / 1422
operations UNCHANGED and not one deleted line — the wire is the same, only the
document knows more about it.
LATENT DEFECT, fixed here: the subsystem installed no cloud.Bridge(). Serve
installs one app-wide so production was never broken, but the subsystem was not
self-sufficient — and its own tests mount it on a bare zip.New, where the first
typed op would have failed closed. Proven by removing the line: 20 tests go red.
It goes through Router.Use, which fans it over the prefixes the composition root
declared for account; account owns six top-level nouns and so has no single
group to hang it on.
WIRE PRESERVED, and nine routes left untyped BECAUSE of it:
- DELETE /v1/keys and DELETE /v1/iam/keys select the key class from `?type=`
and FALL BACK TO THE JSON BODY. A typed DELETE carries no body (zip's
hasBody), so typing them would revoke a caller's SECRET key when they named
the publishable one in the body. That is a wire change on a
credential-revocation route; they keep their raw handlers.
- the seven /v1/billing/* and /v1/commerce/* bridge routes are catch-alls: the
path is a wildcard remainder, the body is forwarded verbatim to another
service, and the answer is that service's bytes and status. There is no In
and no Out to name — opaque by construction, not by omission.
The gates that had to be satisfied, and did their job: the weave REFUSED the
first attempt because `keyList` already means git's SSH deploy keys — one name,
two shapes, which would have bound every generated SDK to whichever it read
last. Account's is now `apiKeyList`/`apiKey`, named for what it is.
Middleware is now zip.Middleware in one form each (requireCSRF, rateLimit,
deprecatedFor), composed through With so a typed op is gated exactly as the
untyped route beside it — a decorator that dropped the gate there would register
the op ungated. rateLimit also loses a parameter it never read.
cloud.Request is pinned; apps/account/account.go is added to allowedRequestUses
with its reason: account IS the caller's own account, and resolving them needs
the user id, the IAM username and validated-ness, none of which
principal.OrgFrom carries. ONE function (requestCaller) that every op asks, and
it fails closed off the HTTP path.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every route apps/ingress serves is now a typed op, so ONE registration feeds the
OpenAPI document, the MCP tool list, the CLI and the SDK. Before this the whole
edge control plane was 18 bare routes: no schema, no prose, no tool, no command,
no SDK method. It now contributes 18 described operations and 10 schemas.
Typing is a DESCRIPTION task, so the wire is preserved exactly, and that is
PROVEN rather than asserted: control_test.go drives the real router and pins the
status of every op — 200 on POST (not 201), 204 with an empty body on DELETE,
409 on a contested host, 403 for a validated non-admin, 404 across orgs, the
three list envelope keys, host normalisation, and hot-apply. The identical file
passes against the untyped handlers it replaces; that equality is the evidence.
- One receiver, `ops`, and every op a method value — the only bound form
cmd/zipdoc can lift prose from. The twelve CRUD ops share four generic
helpers (listOf/getOf/putOf/deleteOf) so a kind stays a parameter.
- Declared on the group, so each op's path is the prefix composed with its
leaf, which is the identity every projection keys on.
- cloud.Bridge() on the group, before the leaves: a typed op receives only a
context, so the request its SuperAdmin gate reads has to be parked there.
admin() now fails closed off the HTTP path — no request, no attested admin —
with no second gate to keep in sync. Pinned in typed_request_gate_test.go.
- DELETE takes its id from the URL and reads no body (zip v1.18+), and a body
naming another object can neither redirect a PUT nor smuggle a second delete.
Two things typing surfaced that were invisible while these routes were untyped:
- The fleet's OpenAPI schema namespace is FLAT. A typed op's Go type name IS
its schema name, and openapi.Weave refused this package outright because
`serviceList` is already apps/admin's launch board — one name, two shapes,
which every generated SDK would bind whichever it read last. An untyped route
contributes no schema, so the collision did not exist until now. The list
envelopes carry the product the namespace cannot (ingressRoutes, ...).
- plugin/websearch/openapi.json was stale on main: e83d7e90 moved the scrape to
/v1/scrape without re-emitting the subset, so the published spec named two
paths nobody serves and omitted the one that is. Regenerated here — the same
failure openapi-check exists to catch, second instance, found by running it.
Also: the status view claimed tlsHosts was a subset of liveHosts. It is not — an
extraHost owns no route, and a TLS route naming a missing service is skipped
while its host still wants a cert. Corrected, because that prose ships to the
document and the MCP tool description.
Gates: apps/ingress green (13 new tests), openapi green, root green, vet clean,
zipdoc -check clean, openapi-check regenerated from source. The 13 failures in
apps/{functions,platform,provisioning,storage} + plugin/kmsreseal are identical
at HEAD, before this change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Each converted route is now ONE registry entry: the REST route, the OpenAPI
operation with its schema and prose, the MCP tool, the CLI command and the
generated SDK method all follow from it. The 12 that stay untyped could not be
typed without changing what they accept or return, and each says why in the
code beside it.
Typed (7):
GET /v1/team/bots -> botRoster
POST /v1/team/bots/sync -> botSync
GET /v1/team/account/providers -> providerList
GET /v1/team/billing/plan -> planInfo
DELETE /v1/team/files/:workspace/:filename -> 204, URL-addressed
GET /v1/team/transactor/statistics -> statsOut
GET /v1/team/transactor/api/v1/statistics -> statsOut (the front's alias)
Left untyped, with the reason: the account JSON-RPC multiplexer (one POST, 20
verbs, a different result shape per verb); two OAuth redirects and two
cookie writers (302 + Set-Cookie is not an Out); the wallet page and the blob
download (asset BYTES under a per-response Content-Type); the multipart upload
(a form, not JSON, whose part filename IS the blob id); the collaborator RPC
(a second multiplexer); and the two WebSockets.
Latent defects this surfaced:
- team had NO cloud.Bridge. Serve installs one for the whole binary, so the
fused and plugin binaries were fine, but a bare Mount — the app's own test
harnesses, and any embedder that does not go through Serve — parked no
validated org at all. Installed on the /v1/team group, before the leaves.
- two test harnesses (billingApp, gateApp and its two siblings) built a
/v1/team group by hand with no Bridge, so they were exercising a wiring no
deployment has. They now build what Mount builds.
- the weave REFUSED the first attempt: team's botList/botView collided with
visor's — one schema name, two shapes (a workspace roster entry vs a bot
MACHINE), which would have bound every generated SDK to whichever it read
last. Renamed to botRoster/botMember.
- team had no //go:generate zipdoc directive, so no team prose could ever have
reached the document or the MCP tool list. Added in typed.go.
- GET /v1/team/billing/plan sets Cache-Control: no-store on per-tenant data and
nothing tested it; DELETE .../files/... answers an empty 204 and nothing
tested that either. Both are now pinned (typed_test.go), along with the
canonical /transactor/statistics path, which had no test at all — only its
/api/v1/ alias did.
The wire is byte-identical: same 19 routes, same statuses, same bodies, same
headers. The degraded (no SERVER_SECRET) 503 is preserved — a TypedHandler is
not a zip.Handler and cannot be wrapped by Mount's guard, so each typed op
returns the same refusal from its own first line, out of the one function that
states it. cloud.Request is confined to apps/team/typed.go (team authenticates
its billing/files planes with its OWN HS256 session token, which rides in a
header or an HttpOnly cookie that principal.OrgFrom cannot carry) and is pinned
with that reason.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The books surface was 25 routes and nothing else — no schema, no MCP tool, no CLI
command, no SDK method, and no `//go:generate zipdoc` directive in the package at
all, so its prose could not have reached the document even if a route had been
typed. Fourteen of them are now typed ops: one registry entry, four projections.
Typing is a DESCRIPTION task, so the wire is unchanged and that is now PROVEN
rather than asserted. wire_test.go pins all 25 routes — status, Cache-Control,
and the exact JSON envelope — and it was written against the untyped handlers,
run green there FIRST, and runs green against the typed ops with the same
literals. The package had no route test at all before this; every assertion in it
was over the store and the report engine.
Three things the typed signature drops, one home each:
- the VALIDATED org: principal.OrgFrom(ctx), parked by cloud.Bridge on the
/v1/books group. Never an In field — an In field is caller-supplied, and this
surface is a LEDGER, where that is another org's money.
- the ledger selector: `?sandbox` stays a STRING, because only the literal
"true" has ever selected the sandbox. Bool binding would additionally accept
"1" and a bare "?sandbox", handing a caller who asked for their live books the
sandbox's empty ones.
- Cache-Control: no-store. A typed op returns its Out and has no response to set
a header on, so it moves to the one place every books answer passes through —
noStore, on the group, on success only, exactly as booksJSON did.
A list route answers a bare JSON array, so its Out is a NAMED slice (accountList,
glList, bankTxnList): zip documents an anonymous type as no content at all, so the
name is what makes the array describable without wrapping it and moving the wire.
WHAT STAYED UNTYPED, and why — none of these is a wire change waiting to happen,
each is a wire change REFUSED:
- scan, inbox upload, bank/import take RAW document bytes (a PDF, an OFX/CSV
statement) as their body. There is no JSON input to name.
- ask, scan/book, vendors, rules, bank/sync read ?sandbox from the QUERY, and a
typed POST documents its whole input as a body. Typing them would move a
live/sandbox selector off the URL it lives on.
- metrics returns MetricsResponse, which EMBEDS Metrics. Go flattens an embedded
struct onto the wire; zip v1.18.3's schema walk does not. The published schema
would not match any answer this route has ever sent.
- bank/link-token and bank/exchange always answer 501. A typed op would publish
a success schema for a response neither has ever sent.
TWO LATENT DEFECTS SURFACED:
- `Vendor` meant two different things in books and admin — a vendor BOOK row
here, a vendor COST LINE there. The weave gate refuses it, correctly: every
generated SDK binds whichever it read last. books' side moved (VendorRow,
matching GLRow/BankTxnRow) because books' schema had never been published, so
the rename costs no caller anything; admin's would move a live SDK model.
admin's is the misnamed one — that is for the fleet dedup pass, not this one.
- zip drops or nests EMBEDDED struct fields when it builds a schema, and it has
already shipped: PATCH /v1/agents/targets/{id} publishes a body of `{id}` and
nothing else, because patchTargetIn embeds an unexported patchTargetReq. Every
generated SDK can send the id and none of the seven fields the patch exists
for. Same generator gap, already live, in the worked example.
The golden gained 818 lines and lost none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/company was 22 routes and nothing else: no schema, no prose, no MCP tool, no
CLI command, no SDK method. Twenty of them are typed ops now, so the whole
Stripe-Atlas-class incorporation flow — begin, structure, founders, KYC, docs,
esign, genesis, advance, skip, import, fundraise — carries a request/response
schema and a description into the document, the tool list and every generated
client. The app subset goes from 20 documented lines to 1149; openapi.yaml grows
907 lines and DELETES NONE, which is the document-level proof the wire did not
move.
Wire preserved exactly.
- begin answers 200 on the idempotent repeat and 201 on the first call. That is
correct REST and zip.WithStatus cannot express it (one declaration, one
status), so it is typed-but-shimmed: cloud.Created on the create branch only.
The conditional-status class, same as registerTarget.
- fundraise/round and fundraise/safe always answer 201, so they DECLARE it —
zip.WithStatus(201) — and the document now says 201 about routes that have
always sent 201.
- The 1 MiB JSON body cap was a line inside every handler's decode(). decode()
is gone (a typed op never sees the request), so the cap is now the ONE group
middleware limitBody, registered after the deck leaf and before every JSON
leaf. Same 413, same routes, one place instead of five.
TWO ROUTES ARE DELIBERATELY LEFT UNTYPED, and both are named at their
registration:
- POST /v1/company/payment. A billing denial answers the fleet-wide contract
cloud.DenyResource renders — 402 insufficient_balance / spend_cap_exceeded,
503 balance_unavailable, each {"error":{"code","message"}}. zip's HTTPError
renders {"status","code","error"}. Typing the route would silently reshape
that error for every metered client. This is not a company problem: it blocks
every metered create route in the fleet (~35 call sites) from typing until a
zip error can carry a body.
- POST /v1/company/fundraise/deck. The deck is the raw request BODY of any
content type, named by ?name=. A typed In would declare a JSON request the
route does not take.
Four things typing surfaced that nothing else would have:
- cloud.Bridge was not installed for this subsystem. Serve installs it app-wide,
so the monolith was fine and every app test — which mounts on a bare zip.App —
would have 403'd the moment an op became typed. Installed on the group, before
the leaves.
- openapi.Weave REFUSED the composition: schema "Summary" means a formation
register row here and campaign counters in apps/marketing. One name, two
shapes; every generated SDK would bind whichever it read last. marketing
already publishes its Summary, so company's — package-internal, no external
referent — becomes Registration. The untyped route hid this; the typed one
could not.
- apps/company had no //go:generate zipdoc directive, so its prose could never
have reached the spec or the tool list regardless.
- The surface root (POST/GET /v1/company) cannot be declared on the group: a
leaf of "" joins to "/v1/company/", a different path. It is declared on the
App with its whole path, and TestBodyCapCoversJSONNotTheDeck pins that the
group's middleware still reaches it — the fact the 201 and the 413 both
depend on.
TestRequestEscapeHatchIsPinned fired on my own change, which is the gate working.
apps/company/register.go is justified in the allowlist rather than waved through:
Hanzo forms the entity and carries the KYC/AML obligation, so the register and a
founder KYC decision are SuperAdmin operations that need X-User-IsAdmin and
X-User-Id for attribution, neither of which principal.OrgFrom carries. One
reviewer() in one file, failing closed off the HTTP path.
Gates: apps/company green (baseline was green), zipdoc -check clean, vet clean,
openapi weave green. plugin/kmsreseal's 6 failures are pre-existing — verified
identical with the change stashed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every /v1/framework route but the two document WRITES is now a typed op
(zip.Get[In, Out] and friends), declared on the group so each op's path is the
prefix composed with its leaf — the identity every projection keys on. One
registry entry, and the REST route, the OpenAPI operation, the MCP tool, the CLI
command and the generated SDK method all follow from it. Before this the whole
surface was route-only: 19 operations with no schema, no prose, no tool and no
client method.
The wire is unchanged — same paths, same statuses (201 on define/assign, 204 on
the three deletes, 200 elsewhere), same JSON, same percent-decoding of path
segments. The full pre-existing suite is green untouched, and the new
ops_projection_test.go pins the surface, the registry, the empty collection ([],
never null) and the summary body.
IDENTITY. A typed op receives only a context, so the engine Caller is assembled
from two carriers parked ahead of the leaves by g.Use(cloud.Bridge(),
bridgeFacts): the validated org from principal.OrgFrom, and the user id +
platform-admin bit from this package's own bridge. It is the same decision
caller() makes on the request — never an In field, which is caller-supplied and
would be a tenant key the caller asserted for itself. Off the HTTP path (an MCP
tools/call, a CLI LocalInvoke) neither bridge runs, both reads come back empty,
and the engine refuses 403 — the handler's own gate, no second gate to sync.
TWO ROUTES STAY RAW, with the reason recorded in Mount and checked by the test:
POST /v1/framework/:doctype and PUT /v1/framework/:doctype/:name take the
document's own field data as their body — an open object the DocType defines at
run time. A typed op's request schema is REFLECTED off its In type, and no Go
struct both accepts that body verbatim and describes it, so typing them would
publish a schema naming the two path segments and nothing else: an SDK method
that cannot send a document. They convert when zip can declare an open-object
input; a schema that lies is worse than the route-only entry they carry today.
WHAT TYPING SURFACED. The weave gate refused the first cut: engine.Summary as an
Out claims the fleet-global schema name "Summary", which apps/marketing already
owns — one name, two shapes, and every generated SDK binds whichever it read
last. Restated as a local summaryView. Re-exporting an external module's generic
type names into the fleet schema namespace is the general form of that hazard
(Role, Install and Document are the next candidates).
Also newly published: the ?filters / ?fields / ?order_by / ?limit parameters of
the document list, which existed on the wire and in no document.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The four routes that made a repo, an ssh key, a subscription and a mirror
target were the last /v1/git routes with a real JSON shape still registered as
raw handlers. They stayed raw for ONE reason, written in each of their doc
comments: they answer 201, and zip's typed registrar wrote 200 for a value and
204 for none with no seam to say otherwise. zip v1.18.2 closed that gap —
zip.WithStatus(201) declares the status ON the op, so the document's response
object is keyed on 201 and a generated client expects what the service sends.
That reason is now stale, so the routes convert:
POST /v1/git/repos createRepo -> repoView
POST /v1/git/keys registerKey -> keyView
POST /v1/git/repos/:name/subscriptions subscribe -> subscriptionView
POST /v1/git/repos/:name/mirrors addMirror -> mirrorTargetView
Each was a route and nothing else — no schema, no prose, no MCP tool, no CLI
command, no SDK method. Each is now one registry entry with all of them. The
wire is unchanged and the suite proves it rather than asserting it: 201, 400,
403, 404 and 409 on these four paths are pinned by existing tests (git_test,
ssh_test, lifecycle_test, hardening_test, public_repo_test, tenant_isolation_test)
and they pass untouched.
Registration moves onto the group. Every zip.<Verb> in routes() now takes `g`
rather than the *zip.App with an absolute path, so the /v1/git prefix lives in
exactly one place and each op's path is the prefix composed with its leaf — the
same composition the router does, and the identity every projection keys on.
cmd/zipdoc resolves it the same way since zip v1.18.3, so the prose reaches
both the document and the tool list.
The repo scope resolver collapses to one function. repoScope() existed only so
the raw creators could share the typed ops' preamble; with no raw creators left,
scoped() absorbs it and there is one way to turn a :name into a validated repo.
The principal keeps carrying the user. tenantFrom already read the validated
org and project off the request; it now reads c.User() there too, so registerKey
gets its owner from the bridge like everything else. An In field is
caller-supplied — a key written under a user read from one would let a caller
register a key in someone else's name.
Prose is product surface now, so it had to be TRUE, and one line was not:
addMirror's comment named git.hanzo.ai as an allowed mirror TARGET. It is
precisely the host mirrorOutHostAllowed excludes on purpose (Red MED-1 — an
internal SSRF that would make the server force-push with the shared token at an
arbitrary internal path). Typing lifts that sentence into the published
description and the MCP tool description, where it would have told SDK users and
agents to do the one thing the code refuses. Corrected to the real allowlist,
{github.com, gitlab.com}.
24 typed ops now, 24 raw. What is left raw has no JSON shape to type and says so
per route: the smart-HTTP pack protocol streams binary, the browser UI serves
HTML, the ZAP adapters answer a {status:"error", msg} envelope at a non-2xx that
a typed op cannot produce, and the webhook HMACs the RAW bytes and verifies
before it parses — typing it would invert that order and decode an
unauthenticated body.
Regenerated: apps/git/zipdoc_gen.go, plugin/git/openapi.json, openapi.yaml.
Described operations across the fleet: 170 -> 174.
hanzoai/cd renamed its module, so the engine is now published as
github.com/hanzoai/cd/gitops-engine. The old path stops at v0.7.2 — a tag can
never move — so an import pinned there sees no release ever again. This follows
it to v0.7.3, the first tag cut at the new path.
The version is the easy half. The engine at v0.7.3 builds against k8s 0.36.1 /
kubernetes 1.36.1, and replace directives in a dependency's go.mod are ignored,
so the pins that make it work inside hanzoai/cd do nothing here — this module
has to state them itself. It was pinning the staging tree to 0.35.3 while its
own requires already said 0.36.1, and k8s.io/api had no pin at all, so it
floated to 0.36.3 and lost the staging packages kubernetes 1.36.1 imports.
The whole tree now says 0.36.1, k8s.io/api included, and kubernetes 1.36.1.
controller-runtime follows to v0.24.1. v0.23.3 cannot compile against client-go
0.36: ResourceEventHandlerRegistration gained HasSyncedChecker, and its
handlerRegistration does not implement it. v0.24.1 targets client-go 0.36.
go vet ./... clean across the module, apps/deploy green, cmd/cloud links.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/integrations/github/repos/import and
/v1/integrations/github/repos/:repo/pages/builds were the only routes on this
surface with a real request and response shape still registered raw. The reason
was honest at the time: both answer 202 Accepted, zip wrote 200 (204 for a nil
Out) and cloud.Created only covered 201, so declaring 202 was not expressible —
and staying raw meant no schema, no prose, no MCP tool, no CLI command and no
SDK method for either one.
zip v1.18.3's WithStatus states that fact where every projection reads it, so
both are typed ops and the document says 202 because the op does. The wire is
byte-identical: same paths, same 202, same JSON keys (queued/repos,
repo/status/url), same 403/400 messages from the same two-step org gate. zip
binds body then query then PATH, so :repo still wins over anything a body
claims — the addressing authority is unchanged.
Latent defect this surfaced: the input type was named repoRef, and apps/git
already publishes a repoRef keyed by `name` (a repo hosted BY us) while this one
is keyed by `repo` (a repo GitHub grants our App). The OpenAPI schema namespace
is flat across the fleet and openapi.Weave refuses two apps that mean different
things by one name. The collision was invisible while every op taking this input
was bodyless — a GET/DELETE emits path params and no request schema — and the
first one with a body made the weave refuse. Renamed to githubRepoRef, matching
githubReposOut and githubPagesView, which already say what they are about.
Also: TestSpecCarriesProse told a failing reader to run
`go generate ./clients/integrations`. That path does not exist; the directive
lives in apps/integrations/ops.go. An error message that sends you nowhere is
worse than none, and only a reader who tried it would ever find out.
Raw routes left, both reasons properties of the wire rather than of effort:
thirteen 302 link/callback legs (WithStatus refuses a non-2xx at declaration, a
redirect is a Location header and not a JSON body, and the legs set signed
__Host- cookies), and six inbound webhooks whose auth reads bytes or headers a
typed op is never handed — Slack/GitHub HMAC over the raw body, Slack's
form-encoded /commands, Discord Ed25519, Teams' Bot Framework JWT and Telegram's
secret-token header. The webhooks also answer 200 with no body, which a nil Out
would turn into 204.
22 typed ops, 19 raw, each raw one carrying its reason in ops_projection_test.go
so the reason is checked and not merely written down.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
DESIGN ONLY — nothing wired. This marks the seam so the next change lands here
instead of adding a sixth object to every stateful pod.
Each replicated service currently carries four objects and a key: a replicate
container, a generated ConfigMap, a restore initContainer, and its own age
keypair. All of it exists because replicate is a separate binary watching a file
it does not own, so the file has to be described to it.
That arrangement produced four independent outages in one day (2026-07-29): a
misindented age stanza replicate refused, an age/plaintext mismatch between
config and bucket, a service whose data dir was not mounted, and a restore path
that had never once run. The last is the instructive one — restore only runs
-if-db-not-exists, so while the local file happened to exist it was never
exercised. The backups were configured, not current, and not restorable, and
nothing said so until a volume was lost.
Open is already 'the single way a cloud store opens its file' and Exists already
answers 'is there a store here' — which is the entire question the initContainer
shelled out to ask. Native, the lifecycle collapses into Open: hydrate if
absent, follow after. Restore stops being a lifecycle stage and becomes what
Open does; the ConfigMap, initContainer, second container and the ordering
between them all disappear.
ONE KEY. cek already holds CLOUD_KMS_MASTER_KEY_REF and refuses to open a store
unkeyed. The age identity is a SECOND key system encrypting the SAME data, with
no rotation story at all — an age identity cannot be rotated after the fact, so
losing it makes every replica under it unreadable. So the age keypair should not
be migrated to KMS; it should stop existing, and the replica should be encrypted
under the key the process already holds. That makes the KMSSecret file added to
universe today unnecessary rather than merely unarmed.
Three verbs, all about bytes at a path: Has, Hydrate, Follow. Follow returns
when following STARTS, not when caught up — a store that refuses to open until
its backup is current will not open during an S3 incident, trading a durability
risk for an availability one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/deploy/projects asked for appprojects at group argoproj.io. Nothing has
ever served that group here, so the request returns "the server doesn't have a
resource type" — and listAppProjects treats any error as "the CRD is absent"
and synthesizes a project set instead.
The cluster serves appprojects.apps.hanzo.ai and has real ones (default,
hanzo). The fallback was not covering for a missing CRD; it was covering for
asking the wrong question, so the operator's actual policy envelopes never
reached the dashboard.
The comment above the GVR said "this plane does not run argocd, so the CRD is
normally absent." True once. It stopped being true when Hanzo CD was installed,
and the code kept believing it.
CDApplications already named apps.hanzo.ai correctly one file over. The project
GVR was declared privately in projection.go and missed, so it moves to apps/k8s
beside its sibling — where GVRs live so the next one cannot drift alone.
Unchanged deliberately: the argoproj.io/v1alpha1 apiVersion strings the
projection EMITS are response shape the cd-ui SPA consumes, not a query. Those
need the UI checked before they move.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
research and deep were never two things. Same system prompt, same models, same
plan gate; only the dials differed (4 queries/12 sources/4 reads at 5c vs
6/16/6 at 10c). Two names made the product look like it offered a choice, and
hid what that choice cost behind an adjective.
research now always does the deeper pass and carries the price that pass actually
costs. 'deep' stays a retired NAME that folds into research in both IsMode and
resolveMode -- not a mode. Dropping it outright would have been worse than a
rename: an unrecognised mode falls through to search, so a client that had not
shipped the collapse would answer a deep request with ONE query. A wrong answer
is worse than an error.
Note search and news are now the closer pair -- identical but for newsBias.
Left alone: that one is a real editorial difference, not a duplicate.
Tests updated to the new contract and negative-controlled: removing the fold
turns IsModeAndResolve red.
It was mounted at /v1/websearch/v1/scrape -- a /v1 inside a /v1. That was not
a choice, it was fallout: the firecrawl client always builds
{apiUrl}/{version}/scrape, and firecrawlApiUrl pointed at the group.
Point firecrawlApiUrl at the API ROOT instead and the same client lands on a
clean top-level /v1/scrape. The path was freed by deleting ai's crawl-and-index
route of that name, which was a second door onto object.ScrapeAndIndex.
Requires FIRECRAWL_API_URL=https://api.hanzo.ai (was .../v1/websearch) in the
chat config. FIRECRAWL_VERSION stays v1.
Also drops the bare /v1/websearch/scrape duplicate -- two paths for one handler
is the thing this commit is removing.
zip v1.18.3 teaches cmd/zipdoc to resolve a group's prefix the way the router
composes it, so the ergonomic form is finally the correct one. The five target
ops move from spelling their whole path on the App to
`zip.Post(g, "/targets", …)` on the group the subsystem already has.
The published document is BYTE-IDENTICAL across the change — the drift gate
regenerated all 1006 paths and found nothing to write — which is the point: the
op's identity was always the composed path, and only the source ergonomics were
awkward.
Verified rather than inferred, because the whole lesson of this bug is that
extraction can silently disagree with what you believe you registered:
zipdoc_gen.go now keys all five under /v1/agents/..., and
TestTargetOpsProjectEverywhere asserts the handler's doc comment appears in BOTH
the OpenAPI description and the MCP tool description. It passes on the group
form.
The playbook drops the "spell the WHOLE path" workaround and tells the next
agent to declare on the group, with a note that an unresolvable router now fails
loudly naming the call rather than filing prose under a path that does not exist.
A doc that instructs people toward the awkward form is the same defect as a
comment promising a guarantee the code does not provide — it stops them looking.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestTargetOpsProjectEverywhere is the demonstration that makes the rest
mechanical: ONE typed op, asserted in all four places it now exists — the
OpenAPI operation (operationId, the handler's doc comment as description, a
$ref'd schema, the doc comment's own Example), the MCP tool (same description,
same fields in inputSchema), the CLI command (`agents targets-create`, flags off
the In type), and the same operation id addressing it through all of them. Plus
TestTargetDeleteIsURLOnly, which pins the v1.18 wire: no requestBody, one
required path parameter.
An untyped route has exactly ONE of those. That is the whole argument for
converting 986 of them, stated as a test that fails if any surface stops being
derived.
The playbook in LLM.md is the recipe as actually executed, with the parts that
are not obvious: a receiver rather than a closure (the only bound form zipdoc can
lift prose from), spell the WHOLE path because zipdoc keys on the path literal
and a group-declared op files under its leaf, Bridge on the group before the
leaves, identity never an In field, and doc comments written true because they
ship to both the reference and the tool list.
It carries the four failure modes found the hard way, because whoever takes a
partition will hit them and will not know to look: a gate comparing two derived
artifacts agrees with itself while both are wrong (how ingress lost 8 paths from
every SDK); the stale-tree pin walk-back, which is mechanical and will recur;
verify what CI actually invokes before trusting a gate you add to a make target;
and porcelain-not-diff, because a new app's subset is untracked and a diff cannot
see it. registerTarget is written up as the worked conditional-status example —
typed-but-shimmed until zip's multi-status responses land, so nobody invents a
third mechanism.
The partition list is six tranches over disjoint apps/<app>/ trees with the
re-measure command, since the counts move every few merges. Source does not
collide; two artifacts do — the regenerated golden and go.sum — and both resolve
by rebasing and regenerating rather than by hand, because the generator is
deterministic.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The first tranche of the typing migration, and the proof the chain works end to
end. /v1/agents/targets register|list|get|patch|delete are typed ops now, so the
machine-target API has a schema, prose, an MCP tool, a CLI command and a
by-name call target where it had a URL and nothing else.
Wire preserved EXACTLY. registerTarget still answers 200 on the idempotent
re-link and 201 on first registration — a distinction zip.WithStatus cannot
express, because it declares ONE status and this route legitimately has two. It
therefore keeps cloud.Created on the create branch only, and is the worked
example of the conditional-status class: typed-but-shimmed until zip grows
multi-status responses. Every other route answers what it always answered; the
app's own tests, which assert 201/200/400/403/404 across all five, pass
unchanged.
Two things typing surfaced that nothing else would have:
- cloud.Bridge was not installed for this subsystem. The untyped handlers read
identity straight off the request; a typed op receives only a context, so
the validated org has to be parked there. Installed on the group, the shape
apps/search and apps/integrations already use, before the leaves — fiber
runs middleware in registration order, so one installed after them never
runs.
- cmd/zipdoc keys prose on the path LITERAL in the registration call, so an op
declared as ("/targets") on a group files under "POST /targets" while its
real identity is "POST /v1/agents/targets". docFor never matches and every
doc comment is dropped from the document AND the tool list — silently, which
is the failure mode this whole effort exists to kill. zip gained group
registration in v1.18.0 and zipdoc did not catch up; that is mine to fix.
Until it does, a typed op spells its whole path (the apps/git shape) and the
group carries only the Bridge, which is prefix-matched and applies either
way. The comment at the registration says so, so the next person does not
rediscover it.
apps/agents gets its //go:generate zipdoc directive — it had none, so its prose
could never have reached the spec regardless.
TestRequestEscapeHatchIsPinned fired on my own change, which is the gate working:
targetOwns and targetCaller are new cloud.Request sites. They are justified in
the allowlist rather than waved through — ownership needs X-User-Id and
org-admin-ness, neither of which principal.OrgFrom carries, and both fail closed
off the HTTP path where there is no attested caller.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A plugin that will not start returned an error from zip.Load, and this host
escalated it to os.Exit(1). pubsub is Apps[0] and Eager, so when its child could
not open /var/lib/cloud/audit.db the process died with every other subsystem in
it — the API, IAM validation, billing, the team backend — and api.hanzo.ai and
cloud.hanzo.ai were 502/503 for 25 minutes:
cloud: zip: Add service 0: zip: Load(pubsub): exited before listening: exit status 1
zip returning an error is a library reporting the truth; turning that into a dead
process was policy, and the policy lived twice as `return err` in two loops. It is
now one function that both loops call:
required -> abort, naming the app. Nothing sets it; the argument is pinned.
otherwise -> ABSENT. The mount stands with no process, so the prefix answers
zip's own 503 instead of falling through to the console at "/".
Keeping the mount is load-bearing, not cosmetic. webui refuses only its
apiPrefixes list, so seven prefixes across five apps answer 200 text/html when
unregistered — including iam's /login/oauth, where an OAuth client would receive
the console shell instead of a redirect.
Absence is loud in three places, because a silently missing subsystem is the
failure this fleet keeps paying for: an error log, the reason on the host's
/healthz, and Running=false in zip's plugin table. /healthz stays 200 and
"status":"ok" — failing liveness for an optional plugin would recreate the outage
one layer up.
Required is a property of the app rather than of start order. Being first in a
list is not a claim on everyone else's availability.
Second half: why the child had no key. Every child this host spawns carries a
CREDZ_TOKEN, and credz refuses to fall back to a dev key once a token is present,
so a deployment with no broker is one where EVERY child resolves Unkeyed and dies
at its first store open. Two silences fixed: an --enable list that omits the
broker mounted zero brokers and said nothing (now refused, like a name the
manifest does not list), and a process holding the root key that declines to
broker now says which half it is missing rather than returning nil.
Reproduced end to end with the real binaries, then proven: host alive, /v1/flags
200 application/json, /v1/pubsub 503 JSON, "/" still the console, /healthz naming
both absent subsystems with their reasons. Four rows added to scripts/mutate.py;
4/4 KILLED.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
I had written that credz was a second protocol doing what the internal plane does,
and that its transport should collapse into kms methods. That is wrong, and the
correction matters: answer() calls parseIdent(call.Cap) and nothing checks it, so a
capability on the plane is whatever the caller wrote. credz proves the app name with
a launcher-minted token the caller cannot forge, which is precisely how a child gets
its own scoped bundle and not a sibling's.
Collapsing credz into the plane would have traded a proof for a claim. Recording what
each actually establishes, so the next person does not make the trade I nearly did —
and so nobody reads a plane capability as an authorization decision.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Five subsystems broke the same way in one day — a process without the store found
nothing and said nothing useful — and each was diagnosed from scratch because the
rule was not written down anywhere. It is now: a store has one owner, everyone else
asks, and the peer-absent/peer-answered-badly distinction is the part that decides
whether a fallback is correct or a 502 on a healthy deployment.
Also records what make build does NOT build, which cost an afternoon of 503s that
looked like a broken product.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/runner {release:true} demanded the machine build token ALONE. That
put a second auth system beside IAM: a SuperAdmin identity — which by
definition may do anything, and is trusted with KMS and every tenant's data —
was refused a release.
Every other privileged surface in cloud reads principal.IsSuperAdmin. Release
now reads the same predicate, or accepts the machine token CI runs under. An
org admin can still build and still cannot release, so the separation that
mattered is kept; what goes is the parallel authority.
IAM decides permission. A token is a transport, not an authority.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The only trigger was push-to-main, so recovering from a bad image meant landing
another commit and waiting — and POST .../workflows/cicd.yml/dispatches answered
500, which reads like a broken forge rather than a workflow that never opted in
to workflow_dispatch. Today that cost real time while production sat on a rolled-
back image.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.801.313 refused EVERY POST across the fleet with
"audit: persist: UNIQUE constraint failed: audit_log.seq" — ~94 a minute, not
self-healing, hitting tasks, integrations, visor and chat completions alike.
Because the audit gate fails CLOSED (correctly, AU-5), the entire write surface
was down while reads looked fine.
The cause is not the audit code, which is right: audit_log.seq is a gapless chain
position and every row's prev_hash seals the one before it, assigned under a
mutex, recovered from MAX(seq) at boot. That is correct for ONE writer. Every
process opened {DataDir}/audit.db, which was harmless while cloud was a single
binary and became a total outage the moment subsystems became plugin CHILD
PROCESSES: each child recovers its own in-memory nextSeq from the shared file and
then they all race for the same PRIMARY KEY.
Retrying the insert would not fix it. Two writers cannot share a hash chain, they
can only fork it — a retry would seal the new row against a prev_hash that
another process has already superseded, trading a loud constraint error for a
silently broken chain. So each process gets its OWN chain, which is exactly what
procName already exists for ("per-process resources ... instead of contending for
one global name"). The host keeps audit.db, so its history and the
/v1/admin/audit surface are untouched; children get audit-<app>.db.
Fleet-wide completeness is preserved by the OLAP mirror, which every Recorder
already writes to a SHARED datastore table — per-process files are the
tamper-evident authority, the mirror is the aggregate query surface.
audit_serve_chain_test.go pins the property that matters: distinct processes
never resolve to the same file. Collapse them back onto one name and the test
fails before the fleet does.
(./audit/... is unrunnable on macOS — it needs a RAM-backed scratch dir; it fails
identically on clean upstream, so it is untouched by this change and runs in CI.)
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
credits, usage and the ledger page are three projections of ONE list — the org's
entries — and all three answered 501 from a process that does not hold the ledger,
which is every process but commerce. financeTxns asks the owner now and falls back
to the configured commerce URL only when no peer serves it, the same split-deploy
shape balance already honours.
finance gained the read it was missing. ListUsage keeps only the usage debits by
design, but a credit and a welcome grant are transactions too when a customer is
looking at their account, so ListEntries returns the entries unfiltered and each
page decides which kinds it shows.
The amount crosses as its 18-DECIMAL INTEGER — money.AttoString, the storage and
on-chain form, parsed back by money.ParseInt. That pair is exact by construction:
no decimal point to misplace and no scale to agree on. It is flattened to cents at
the boundary where commerceTxn is already a cents-shaped view, so the day that view
stops being cents-shaped the precision is already on the wire waiting.
Two money packages exist and both are right: hanzoai/money is the general exact
value, and apps/money is cloud's USD carried at 18 decimals so an off-chain ledger
amount and an on-chain uint256 are THE SAME INTEGER. The ledger speaks the latter,
so the wire does too.
A missing peer stays INERT for the starter grant rather than becoming an error. A
deployment with no money plane at all is a real shape, and erroring would put a line
in the log on the first request of every wallet in it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The welcome grant never fired. StarterGrant is middleware on EVERY app's chain and
it bailed the moment it found no local ledger — which, once apps became their own
binaries, is every process but commerce. A new org reached tracker or billing, the
grant looked for a ledger one socket away, and returned silently. The org opened
broke, and the paywall then refused it correctly for a reason nobody had chosen.
Asking the owner is safe to do from anywhere: the grant's idempotency key is the
ACCOUNT and nothing else, and finance dedups on it inside the same transaction as
the insert, so two processes racing the same new wallet still grant once.
Usage had the same hole and answered 501 on a customer's own usage page. It carries
ROWS, not a rendered envelope: the ledger's owner knows what was debited, the HTTP
surface knows what its page looks like. Sending the envelope would have put one
app's response shape inside another app's process and required the renderer to live
with the ledger — which is the import cycle that shape implies, made visible
(billing already imports commerce; the reverse would close the loop).
This is what spec 136's exactly-once debit was waiting on. It has SKIPPED all day —
acme could never cover the fee, because acme was never funded — so the property the
whole prepaid suite exists to prove was the one thing it did not prove. It now runs:
17 passed, nothing skipped.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
CI did not run `make test`. It never has — there is no .github/workflows, and
hanzo.yml runs six discrete steps that name `go test` and `make -f mk/fleet.mk`
directly. So the gate added to `make test` protected local runs and nothing else,
and the constraint "CI runs make test, never test-fast" could not be satisfied by
splitting the make target alone. hanzo.yml had to change.
app-contract now calls mk/fleet.mk's openapi-check instead of restating half of
it inline. That is the fix, not just a refactor: the step used to regenerate the
per-app SUBSETS and check those, so openapi.yaml — the file the SDK repos
actually pull — was never checked against source by anything. openapi-composed
compares it to the subsets, and both are derived. Two gates, neither of them
looking at the routes, which is precisely how plugin/ingress lost eight paths
with everything green.
Calling the target also means one gate definition rather than two that drift, and
CI inherits the kafka exemption instead of dying on an app whose Mount is
fail-closed on a live broker.
The gate now checks `git status --porcelain`, not `git diff`. A NEW app produces
a NEW subset, which is untracked and therefore invisible to a diff — the failure
that matters most is the one a diff would miss. That reasoning was already in the
step it replaced; it belongs in the gate.
test-fast is the inner loop: everything `test` runs except the drift gate, which
rebuilds one binary per app and dominates the wall clock. It ANNOUNCES the skip
on every run, names what will fail in CI, and prints the command that checks
properly — the same reason the gate names its kafka exemption out loud. A skip
nobody sees is how a gate becomes decorative. It is documented nowhere as the
default, and CI does not reference it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`make test` claimed the weave caught a route added without regenerating. It did
not, and the comment saying so was the bug in one sentence.
openapi-weave compares two COMMITTED artifacts — the per-app subsets and the
golden they weave into. Both are derived, and nothing in that comparison forces
either back to the routes, so they agree with each other while both are wrong.
plugin/ingress proved it: eight paths (/v1/ingress/routes, /services,
/middlewares, /tls, /status and their :id forms) were added, the subset was never
regenerated, the golden was woven from that same stale subset, the gate stayed
green — and the entire ingress API was absent from openapi.yaml, and therefore
from every generated SDK. No Python, Go or TS caller could reach it at all.
openapi-check regenerates every subset and the fleet spec FROM SOURCE and fails
on any diff, printing what to run. It is in `make test`, expensive half and all,
because the cheap half is exactly the check that passed while the published
document was missing an API.
Both drift classes are PROVEN caught, not hoped for:
- a route added in source without regenerating: added one to apps/ingress, ran
the gate, watched it name the stale subset, reverted.
- a dependency walked back so the document can no longer be reproduced: reset
the pins to what bb10586e committed (commerce v1.49.30 -> v1.49.29, zip
v1.18.1 -> v1.17.6) and the gate went red with a 6045-line diff in
openapi.yaml plus four subsets. Nothing else on main detects that.
On that second one, since it will recur: bb10586e is not a bad merge. It is a
single-parent commit directly on top of 49f8eeec whose go.mod hunk downgrades
both pins outright — the signature of `go get`/`go mod tidy` run in a tree that
predated the bump and then committed wholesale. Any agent working from a stale
tree reproduces it, which is why the answer is a gate rather than a note asking
people to be careful.
kafka is exempt BY NAME and says so when it skips: its Mount is fail-closed on a
live pubsub broker, and a document is a projection of routes, not a reason to
need a running message bus. That is a defect in the app, not the gate. The
exemption is provably free — kafka's subset declares zero paths — and it goes
away when the adaptor moves to hanzoai/stream.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip v1.18.2 adds WithStatus, so the status a successful op answers with is
declared on the op and keyed into the document's responses object. cloud.Created
and cloud.Accepted are the reason it exists: they set 201/202 per request from
inside the handler, which works on the wire and nowhere else. The status is a
CONTRACT detail, and setting it there writes it into a side channel no
projection can read — the document says 200, every SDK generated from it says
200, and the route has always sent 201. Same failure as a query parameter's
required-ness being invisible: a contract detail that exists only at run time is
not a contract.
Both are now marked Deprecated, pointing at zip.WithStatus, and both still work.
They are NOT ripped out: 13 call sites depend on them today and 85 untyped
routes still return 201/202 and have not been converted. Converting those on top
of a workaround would have been knowingly writing debt, which is why zip got the
fix first; new ops declare WithStatus and the shims retire as the migration
reaches them.
cloud.Request is pinned. It is the escape hatch that hands a typed op its raw
request, so every use gives back some of what typing bought, and nothing in the
signature stops the next one. TestRequestEscapeHatchIsPinned asserts exactly the
four that exist and carries the reason for each — three identity gates that need
more of the validated principal than the org (admin-ness lives in a header
principal.OrgFrom does not carry) and one tenant-scoped proxy that FORWARDS the
caller's identity upstream. A fifth now has to edit the gate and write its
justification, which is a decision rather than a drift. Verified by adding a
fifth call site and watching it go red.
Also restores the pins bb10586e walked back — commerce v1.49.29 -> v1.49.30 and
zip v1.17.6 -> v1.18.2. Main was green either way, but it had my regenerated
documents committed against a zip that could not produce them, so the next
`make openapi` would have silently dropped the parameter examples, the derived
required-ness and the $ref sharing.
Regenerating also caught real staleness nobody had noticed: plugin/ingress was
missing 8 paths (/v1/ingress/routes, /services, /middlewares, /tls, /status and
their :id forms). Routes had been added without regenerating the subset, and
because openapi.yaml was woven from that same stale subset the two agreed with
each other while both omitted the surface — so the published spec the SDK repos
pull has been missing the whole ingress API.
Pre-existing failures unchanged (functions/provisioning/storage/kmsreseal need
billing config, S3 and a real KMS; graph is a DNS flake that passes on retry).
Host stays 401 packages with zero apps/* imports.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
isOrgAdmin matched only "admin", and IAM's coarse membership vocabulary is three
values: owner, admin, member (iam internal/store/membership.go). `owner` is the
one IAM assigns to whoever CREATES an org — self-service provisioning calls
EnsureMembership(..., RoleOwner) exactly so "a self-service org is born with
nobody on it" cannot happen (iam internal/oidc/provision.go:247).
So every org created through self-serve signup had a founder its own org-scoped
admin surface refused, across 25 files that consume principal.IsOrgAdmin. It is
the strictly worse version of the bug this function was WRITTEN to fix — the
comment above it describes the org-scoped admin surface "refusing its own owner
with 'admin required'" — because an admin could be granted by someone else, while
an owner has nobody above them to escalate from. It landed with self-serve org
creation, which is why it was not visible before.
This only restates a membership IAM already signed: the role is read from the
verified `orgs` claim, and a caller who is not in that set is admitted by nothing.
The org stays a VERBATIM compare (a fold would let a member of "acme" claim
"ACME"); the ROLE is folded, since it is a closed vocabulary IAM controls. IAM's
money path already treats the two as one (billingAccountFor admits
{RoleOwner, RoleAdmin}); this is the authz half of that same fact.
Test proven to FAIL on the old code before it passed on the new: owner, cased,
and whitespace-padded variants all returned false. Full package diffed against
baseline — identical failure set (21 pre-existing env failures needing tmpfs at
/dev/shm), zero introduced.
Closes the loop between the two moneys. cloud is what can see the vendor, so it
publishes; ai is where spend happens, so it enforces. The state is pushed from
the SAME numbers this board renders — credit remaining and average daily burn —
which is what stops the guard and the dashboard from disagreeing about whether we
are spending real money. OnCash is credit <= 0: the promo grant is gone.
The ceiling is CLOUD_DAILY_CASH_CEILING_CENTS and defaults to 0, which DISARMS
the breaker, so this is observational until an operator states a number. Every
ambiguous input — unset, blank, garbage, negative, and notably "200.00" (cents
written as dollars, the likely typo) — resolves to 0 rather than to some small
accidental ceiling. The failure direction is always "allow": this value gates all
paid inference, so a bad ConfigMap must cost a day of unguarded spend, never a
fleet-wide outage. cash_ceiling_test.go pins each of those.
Context: $250,573 of outstanding platform credit stood against $0.00 of DO promo
credit and ~$65/day of real burn, with nothing between them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip v1.18.x closes four gaps between what a typed op DOES and what its document
SAYS. Taking it regenerates the published spec — the file the SDK repos pull —
and unblocks the migration these routes were waiting on.
* A typed DELETE no longer reads a request body; its input is the URL. All 20
of cloud's DELETE ops keep working: 15 take only path params, and the other
5 take scalars that bind from the query exactly as the document already said
they would. DELETE /v1/marketing/suppressions is the one with no path param
at all, so its whole input is now ?channel=&address= — which is what a
client generated from openapi.yaml has always sent.
* A URL-borne field's `validate:"required"` reaches its parameter, so an
argument the handler refuses to run without stops being described as
optional.
* A bodyless op's example survives. openapi.Parameter had no name for it, so
the round-trip through Typed() dropped it — exactly the "dropped honestly,
if nothing here has a name for it" its own doc comment warned about. Every
GET and DELETE reached the published reference with no example at all;
adminDeleteSpendCap now carries example: cap_1 on the path and acme on the
query, from the one Example its doc comment already had.
* A path parameter is typed from the field it binds to, so ?sizeGiB= is an
integer in the document because it is an int in Go.
openapi.yaml is 245 lines SHORTER despite 128 new example/required/query lines:
a named struct is now one definition every op $refs instead of being inlined at
each use. 148 schemas across the admin subset alone.
probe/ is inverted, which is what it told whoever came next to do. It pinned two
zip limitations — a typed op cannot see its own URL, and a templated path is
emitted with no parameter object — and both are gone, so it now asserts the
capabilities: the whole URL binds, and /v1/agents/sessions/{id} declares its
parameter, typed. The 16 of 25 clients/agents routes that carry a path param
are unblocked. What stays pinned is the DEFAULT, not a framework limit: an op
with no Authorize installed answers an anonymous MCP caller.
commerce moves to v1.49.30 for the same reason — Mint decorates a zip.Router,
so it now answers for where a typed op lands, gate included.
Pre-existing failures unchanged (functions/provisioning/storage/kmsreseal need
billing config, S3 and a real KMS); the two probe failures are fixed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Enabled() had three answers where it needed two. A "staged" set was held back from
the mount-everything default and reachable only by being named — in CLOUD_ENABLE or
in the parallel CLOUD_ENABLE_STAGED — so "what runs here" was decided by two env
vars and a table, and a subsystem could be on by one lever while off by the other.
The membership rule was stated plainly: a subsystem is staged while its Mount can
ABORT STARTUP. Apps are their own processes now. A Mount that fails takes its own
child down, the host stays up, and its prefix answers 502 — which is what happened
when kms crashed and the rest of the fleet kept serving. The risk the exception
existed for is gone, so the exception goes: empty list mounts everything, a
non-empty list mounts exactly what it names, and there is no third case. The knob
appears in no manifest.
Also fixing main, which was red before any of this: TestOrgForKey asserted cloud
should read a key through /v1/iam/users/get. That route takes (owner, name) and
returns a bare user — it cannot answer "who owns this access key", so the
assertion named a door that could not open. The caller was right all along;
get-user?accessKey is what resolves a secret key, and IAM's own doc calls
resolve-key "the dual of get-user?accessKey". The test's four comments already
said so; only its want list disagreed.
go vet ./... clean; go test . ok; e2e 16 passed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Found by settling a question rather than assuming it: cmd/cloud's own doc says
it "serves the whole API by mounting every subsystem as its own process". So a
Register* seam — which resolves a package var inside ONE process — cannot
connect two apps at all. projects calls, git registers, and they are never the
same binary.
That means the visibility publisher I wired earlier tonight has never worked in
production. Public projects have not been getting their canonical repos, and
nothing said so: share() logged a warning only when the publisher RETURNED an
error, and a nil publisher returns nil. Silence is what the seam produced, and
silence is exactly what this plane exists to stop.
Now: projects packs cloud.Visibility and calls git.publish over git's socket;
git exposes it beside git.files. A missing git is an error naming the app, not
a no-op. The rest of the contract is unchanged and still lives on git's side —
the repo is created either way, visibility is applied on both hosts, and a
retraction is a flag flip rather than a delete.
community.go is deleted: RegisterPublisher/PublisherRegistered/Publish had no
callers left. Visibility itself moves to payloads.go, which is where it belongs
once it is a wire contract — the type and its codec in one file, so the two
halves cannot drift.
The projects tests now stand a real git peer on a real socket instead of
registering an in-process func. That matters more than it sounds: the old test
passed precisely BECAUSE it registered something, which is the one thing
production never did.
EIGHT more seams are broken the same way and are NOT fixed here, all failing
loud (ErrGitImporterUnavailable and friends) rather than silently, so they are
visibly-unavailable features rather than wrong data:
ImportGitRepo, InboundGitSync, GitRepoStatuses, EnsureGitMirror git ← integrations, sync
UpsertIssue tracker ← integrations
Sync sync ← integrations
OnGitPush, OnServiceRelease platform ← git, integrations, deploy
Four seams are genuinely in-process and stay: the commerce and KMS client
factories, the org-scope resolver, the lifecycle fanout and the trace sink.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Four stages, checked separately so a failure names the broken link.
Staged rather than one assertion because of what it currently reports: stage 1
passes (POST /v1/event -> 200, with a 404 control proving it is routed) and
stage 4 passes (insights, analytics, sentry all 200), while stage 3 fails —
traces 24h stale, logs 32h. The door accepts and the surfaces load, so a
shallower test reports green on a pipeline dead for over a day.
Stage 3 is load-bearing: accepted is not stored.
Stage 2 names the cause — neither 4317 nor 4318 is bound, because the traces
and logs receivers both bind 4317 and the collision kills the whole pipeline.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Picks up the two fixes for the class of outage that hid a broken backup
plane for eleven days while every dashboard read green.
v1.52.2 — the scheduler stops discarding errors. sweepSchedules' return
was `_ =` and per-schedule StartWorkflow failures hit a bare `continue`,
so a schedule that could never fire said nothing, forever.
v1.52.3 — the layer that ACTUALLY failed here becomes observable. Our
incident was one step below the scheduler: apps/cron fires a JobWorkflow
whose RunJobActivity re-reads the entry's ConfigMap at fire time, and the
ServiceAccount could not read ConfigMaps. StartWorkflow SUCCEEDED every
time — actionCount reached 4489 — so the scheduler was healthy and
correctly silent while the activity failed ~4500 times with no log, no
counter and no durable record. Activity/workflow failures now write a
durable per-(workflowType, activityType, taskQueue, scheduleId) streak
row, readable via View.FailureStreaks(ns), plus throttled WARN/ERROR/INFO
lines that distinguish "failed once, will retry" from "failing
persistently".
Keying on the recurring shape of the work rather than the run is the
load-bearing detail: every cron fire mints a fresh runId, so a run-keyed
counter would have reported "attempt 1 of 10" forty thousand times and
never once "dead for eleven days". Persistence is measured by AGE, not
count — a nightly backup reaches a count of 2 in two days and is plainly
broken, while a single run burning its 10 default attempts in ~5.5
minutes must not page anyone.
Retry and isolation semantics are unchanged; this is observability only.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The file credited an upstream identity server as the origin of Hanzo IAM.
That is wrong. github.com/hanzoai/iam is clean-room original work and its
own LICENSE says so; the derived identity server was iam-v1, which is
retired and ships in no current product. A NOTICE entry is a legal claim
about this codebase, so a wrong one gets removed, not reworded.
The Casibase attribution stays untouched: the Hanzo AI module genuinely
derives from it, and Apache-2.0 section 4 requires that notice.
The backend repo dropped its -go suffix once the TypeScript codebase holding the
name moved to hanzoai/team-v1, so the module path is github.com/hanzoai/team.
apps/team builds clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CLOUD_KMS_ZAP_ADDR selected clients.KMSRPCAt — a stub whose every method returned
"not yet wired (zapc-gen pending)". Setting it produced a KMS client that failed
every call while looking configured, which is worse than having none: a deployment
that sets the address gets silent, total secret failure and a config file that says
secrets are wired.
There is one way to reach the store now: the kms app over the internal plane. The
knob, its config field and the stub go with it, because a second path that has
never worked is not a fallback, it is a trap with a name.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
8a1b35d4 pointed the resolver at the right endpoints and left one expectation
behind, so main's own suite failed on a contract the code had already got
correct.
IAM has three doors and only one answers this question. get-user?accessKey
resolves a SECRET key (hk-/sk-) to its owning user behind CapKeyResolve, and
refuses a pk- by design; resolve-key is the publishable door, org-only;
users/get is the typed (owner, name) read, which carries no accessKey and
cannot answer at all. The test wanted users/get for a key lookup.
Fixed the test, not the code — the resolver was right. The comment now names
what each door is for, so the next reader does not have to re-derive it from
the IAM module to know which one belongs here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
"not a member of this room's workspace" describes a membership determination
this code never makes. meet has no members table, no store, and makes no call to
IAM — measured: zero lookups in the package. Membership was decided upstream at
the IAM login that minted the session, and is already signed into the token as
`workspace`.
What actually happens is narrower and worth naming: the room asked for must
belong to the workspace the token already names. It refuses to WIDEN an existing
decision; it does not make one.
The old wording reads as a second authorization system living in meet, which is
exactly how it was reported. The check is right; the sentence was wrong.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-28 17:04:17 -07:00
2891 changed files with 390910 additions and 58944 deletions
# TWO DIFFERENT FAILURES SHARE THIS STATUS, and treating them alike
# would turn one of them into a ten-attempt loop that ends in the
# wrong diagnosis. "Reference already exists" is the collision this
# loop is for. "Object does not exist" means OUR OWN COMMIT is not
# on github.com — which is a live possibility, because this workflow
# runs on git.hanzo.ai and claims against GitHub, so a commit that
# reached the forge and not the mirror lands exactly here. It is not
# a name to skip past; it is a repo that has not been published, and
# the next number would fail identically.
WHY=$(jq -r '.message // empty' /tmp/claim.json)
if [ "$WHY" != "Reference already exists" ]; then
echo "::error::claiming v${CAND} was refused with: ${WHY:-unknown}. If this is 'Object does not exist', commit ${SHA} is on the forge but not on github.com — the release lane claims versions against GitHub, so the mirror must carry the commit first."
exit 1
fi
# Taken. By us, or by somebody else? The distinction is the whole
# difference between a resume and a collision, and it is one GET.
echo "v$NEXT is already published from our sha — skipping the build"
RESUMED=1
else
echo "::error::v$NEXT is a version this run OWNS (tag at ${SHA}) but the registry already serves bytes built from '${REV:-an unlabelled commit}'. Another lane pushed onto a name it did not hold. Nothing here may overwrite it — publish the intended bytes under a new number and delete the foreign image."
exit 1
fi
elif [ "$CODE" != "404" ]; then
echo "::error::manifest probe for v$NEXT returned $CODE — cannot tell whether the name is free"
exit 1
fi
echo "version=$NEXT" >> "$GITHUB_OUTPUT"
echo "resumed=$RESUMED" >> "$GITHUB_OUTPUT"
echo "highest seen v$LAST (registry + git tags) -> v$NEXT, claimed at ${SHA} (document sha256:$SPEC_SHA)"
echo "::error::${img} resolves, but the bytes behind that tag were built from '${REV:-an unlabelled commit}', not ${{ github.sha }}. Another lane pushed over the tag this run owns. Nothing downstream may pin it."
exit 1
fi
echo "$img is built from ${{ github.sha }} — tag, commit and image agree"
# THE SMOKE GATE. Boot the image that was actually pushed and require it to
# reach "zip listening" without a crash signature, on release.go's boot env
# — a writable /data, CLOUD_ENV=smoke and a throwaway 32-byte master key so
# the KMS plane mounts on its normal ready path rather than a degraded one.
#
# The script is handed over base64 because it contains both single and
# double quotes (the `"message":"zip listening"` needle), and re-quoting it
# for `sh -c` is how a gate quietly stops matching what it is looking for.
| tr -d '\r' | sed -n 's/^[Xx]-[Aa]pi-[Vv]ersion: *//p' | head -1 || true)
if [ "$GOT" = "$WANT" ]; then echo "api.hanzo.ai is serving ${WANT} after $((i*10))s"; exit 0; fi
sleep 10
done
echo "::error::api.hanzo.ai never reported ${WANT} (last saw '${GOT:-none}'). The image is published, tagged and pinned; cd.hanzo.ai has not reconciled it. Nothing downstream may describe a version that is not running — re-run this workflow at the same sha to resume once it has."
exit 1
# ══ CAR 3 ══ THE DOCUMENT TELLS THE TRUTH ABOUT PRODUCTION.
#
# `surface-check` proves the document equals the code. Nothing proved the
# address is REACHABLE, and three things break that independently: a stale
# subset, a prefix missing from manifest/apps.go, and an edge worker that
# intercepts before the origin. Every address here is a method in eight SDKs, a
# tool in the MCP list and a command in the CLI, so a dark address is a call
# every client offers and no caller can make.
reach:
needs:[image, rollout]
runs-on:[hanzo-build-linux-amd64]
timeout-minutes:20
steps:
- uses:actions/checkout@v4
# Go, not Python. cloud is a Go service with no Python in the shipped
# image, and this gate was the only thing that needed a Python runtime —
# which on Ubuntu 24.04 means a dependency the runner refuses to install
# (PEP 668: pip answers "externally-managed-environment"). So a check about
# ROUTING kept failing over package management. gopkg.in/yaml.v3 is already
# a direct dependency, so nothing extra is provisioned.
- uses:actions/setup-go@v5
with:
go-version-file:go.mod
# Same private-module contract as the containment job, including the forge
# substitution: our own modules resolve from git.hanzo.ai (the canonical
# address) with GitHub as the fallback, so one drifted GitHub ACL cannot
# stop a release. go.sum still decides whether the bytes were right.
echo "::error::FLEET_DISPATCH_TOKEN missing in KMS at deploy/ (env ${KMS_SECRET_ENV}, org from the KMS credential). Create it with contents:write + metadata:read on hanzoai/{python-sdk,js-sdk,java-sdk,cli} hanzo-go/sdk hanzo-rs/sdk hanzo-kotlin/sdk hanzo-cpp/sdk hanzo-docs/docs — six owners, so a fine-grained PAT cannot carry it; use a classic PAT with repo scope or a GitHub App installed on all six. Until it exists, every cloud release ships a document no client is regenerated from — which is the failure this car was built to end, so it fails rather than skipping."
exit 1
fi
echo "::add-mask::${TOKEN}"
PAYLOAD=$(jq -nc \
--arg v "v${{ needs.image.outputs.version }}" \
--arg s "${{ github.sha }}" \
--arg d "${{ needs.image.outputs.spec_sha256 }}" \
'{version:$v, sha:$s, spec_sha256:$d}')
# ONE EVENT TYPE for every projection — SDKs, the CLI and the docs
# alike. A second name for "the document moved" would be a second
# thing to keep in step.
#
# EACH PROJECTION BY ITS OWN ADDRESS. Five of the nine no longer live
# under hanzoai/: go, rust, kotlin and cpp each moved to a per-language
# org, and docs to hanzo-docs. Measured against api.github.com rather
# than assumed — POST .../dispatches with an event nothing listens for:
#
# hanzoai/go-sdk 307 hanzo-go/sdk 204
# hanzoai/rust-sdk 307 hanzo-rs/sdk 204
# hanzoai/kotlin-sdk 307 hanzo-kotlin/sdk 204
# hanzoai/cpp-sdk 307 hanzo-cpp/sdk 204
# hanzoai/docs 307 hanzo-docs/docs 204
#
# A redirect is not a delivery. GitHub follows a repo rename for GET,
# but a dispatch POST answers 307 and drops the body, so this loop
# would have failed five of nine on its first real run — five clients
# silently describing the previous release. Curl is deliberately NOT
# given -L: a projection is addressed where it lives, and a redirect
# that has to be followed means this list is stale and should say so.
# Taken BEFORE the first dispatch, so "a run created at or after T0" can
# never be satisfied by a run that was already there.
T0=$(date -u +%Y-%m-%dT%H:%M:%SZ)
ok=0; bad=""
for repo in hanzoai/python-sdk hanzoai/js-sdk hanzo-go/sdk hanzo-rs/sdk \
| jq -r --arg t "$T0" '[.workflow_runs[]? | select(.created_at >= $t) | .status] | first // ""')
case "$S" in
in_progress|completed) started="${started} ${repo}"; echo " started ${repo}" ;;
*) pending="${pending} ${repo}" ;;
esac
done
[ -n "$pending" ] && sleep 20
done
echo "::endgroup::"
if [ -n "$pending" ]; then
echo "::error::accepted but never ran:${pending}. No runner picked the dispatch up within 8 minutes. github.com has zero self-hosted runners and these callers ask for the forge's pool (hanzo-build-linux-amd64); the forge cannot receive a repository_dispatch. Until a runner answers on the side the event lands on, these projections do not move and this release is incomplete."
exit 1
fi
# ══ CAR 5 ══ THE RECEIPT. This artifact IS the definition of "shipped".
#
# Always runs, so a hole is RECORDED rather than merely absent from a green
# run. It is the release's own GitHub Release — state belongs with the thing it
# describes, and a tag's receipt belongs on that tag. Writing it back into the
# repo would trigger the next release; an asset on the tag triggers nothing and
# is what the next run reads to know whether the last one finished.
-d "$(jq -nc --arg t "$TAG" --arg b "$BODY" '{tag_name:$t, name:$t, body:$b}')" >/dev/null
fi
echo "receipt written to the ${TAG} release"
# A HOLE IS A RED RELEASE. Not a warning, not a green run with a
# footnote — the receipt exists precisely so "cloud shipped but the CLI
# is two days stale" is a state you can point at rather than one that
# goes unnoticed for a week.
if jq -e '[.cars[]] | any(. != "success")' /tmp/release.json >/dev/null; then
echo "::error::this release is INCOMPLETE — see the car table on the ${TAG} release. Fix the failing car and re-run this workflow at ${{ github.sha }}: image and rollout will recognise their own receipts and no-op, and only the failed car runs again."
exit 1
fi
echo "${TAG} is complete: document, image, production, reachability and all nine projections."
["${n:-0}" -gt 1]||{echo"SKILLS-GATE FAIL: apps/skills/catalog holds the ${n:-0}-skill fallback — the overlay missed the //go:embed path";exit 1;};\
j="$(find /src/webui/dist -type f -name '*.js'| wc -l)";\
["$j" -gt 0]||{echo"CONSOLE-GATE FAIL: webui/dist carries no JavaScript — the overlay missed the //go:embed path and the image would ship the fallback shell";exit 1;};\
MODERNC="$(CGO_ENABLED=1 go list -tags "libsqlite3 sqlite_fts5" -deps ./cmd/... ./plugin/... 2>/dev/null | grep -c 'modernc.org/sqlite'||true)";\
["$MODERNC"="0"]||{echo"SQLITE-GATE FAIL: a per-app binary links modernc.org/sqlite ($MODERNC pkgs) under CGO=1 — double-registers \"sqlite\" with hanzoai/sqlite(mattn) and panics at init. Find it: CGO_ENABLED=1 go list -tags 'libsqlite3 sqlite_fts5' -deps ./plugin/<app> | grep modernc";exit 1;}
MODERNC="$(CGO_ENABLED=1 go list -tags "libsqlite3 sqlite_fts5 sqlite_math_functions" -deps ./cmd/... ./plugin/... 2>/dev/null | grep -c 'modernc.org/sqlite'||true)";\
["$MODERNC"="0"]||{echo"SQLITE-GATE FAIL: a per-app binary links modernc.org/sqlite ($MODERNC pkgs) under CGO=1 — double-registers \"sqlite\" with hanzoai/sqlite(mattn) and panics at init. Find it: CGO_ENABLED=1 go list -tags 'libsqlite3 sqlite_fts5 sqlite_math_functions' -deps ./plugin/<app> | grep modernc";exit 1;}
# RED gate — ENCRYPTION PROOF + the cek.go GOLDEN-VECTOR KAT, under the SAME CGO +
# libsqlcipher build this image ships. TestEncryptionProof asserts real
# ciphertext-at-rest (SQLITE_REQUIRE_CODEC=1 makes a plaintext link FAIL → NO
@@ -157,17 +215,9 @@ RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
# encrypted stores stay readable, or NO image.
RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
CGO_ENABLED=0 go build -ldflags="-s -w" -o /smoke ./plugin/smoke
CGO_ENABLED=0 go build -ldflags="$GO_LDFLAGS" -o /smoke ./plugin/smoke
# EVERY subsystem, each as its OWN binary in /plugins beside the host. The host
# fork/execs a sibling <dir>/<name> (manifest.App.Plugin) on the first request that
# reaches its prefix, so the binary must be in the image or the mount aborts:
@@ -211,7 +292,22 @@ RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
# fleet union the fused binary was. 112 lean links, sequential, none of them mega —
# which is the whole point of this change.
#
# CGO_ENABLED=1 + libsqlite3 + sqlite_fts5, exactly as the fused binary was built:
# CGO_ENABLED=1 + libsqlite3 + sqlite_fts5 + sqlite_math_functions, exactly as the
# fused binary was built:
#
# sqlite_math_functions is not optional under cgo. hanzoai/base's search layer
# generates SQL calling acos/cos/sin/radians/sqrt (the geoDistance token in
# tools/search); SQLite only has those with SQLITE_ENABLE_MATH_FUNCTIONS, which
# the cgo backend gets ONLY behind that tag. base v1.5.11 turned the mismatch
# into a compile error on purpose (core/sqlite_math_required.go, //go:build cgo
# && !sqlite_math_functions) rather than let a cgo build ship a smaller SQL
# surface than the code above it writes against — the failure otherwise is a
# customer's search returning "no such function: acos" from an endpoint that
# works in production. Without the tag every plugin build dies with
# base@v1.5.11/core/sqlite_math_required.go:30:6:
# undefined: cgoBuildNeedsSQLiteMathFunctions
# The CGO_ENABLED=0 builds below do not need it: the pure-Go backend always has
# the functions.
# every app that opens a store needs the SQLCipher codec (a plaintext link silently
# no-ops PRAGMA key), so they are built uniformly — one contract for all, the
# non-sqlite apps merely carrying a libc dep they do not use. The modernc gate above
@@ -224,8 +320,34 @@ RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
for p in $names;do\
[ -d "./plugin/$p"]||{echo"FATAL: manifest app '$p' has no plugin/$p — run 'make generate' and commit";exit 1;};\
echo"building plugin $p";\
CGO_ENABLED=1 go build -tags "libsqlite3 sqlite_fts5" -ldflags="-s -w" -o "/plugins/$p""./plugin/$p";\
CGO_ENABLED=1 go build -tags "libsqlite3 sqlite_fts5 sqlite_math_functions" -ldflags="$GO_LDFLAGS" -o "/plugins/$p""./plugin/$p";\
done
# THE STAMP LANDED — asked of the ARTIFACT, not of the flag string.
#
# `-X` naming a path or symbol the linker cannot resolve is not an error: it is
# dropped, the build succeeds, and every binary then reports the entirely
# legitimate-looking "unknown" forever. A renamed package or variable would fail
# in exactly the one way nobody looks at, which is how this started.
#
# `go version -m` is NOT a witness — it echoes the -ldflags string that was
# REQUESTED, and that string is present even when the symbol was never set
# (measured on /cloud, whose Version stamp has been dropped all along). Only the
# linked bytes answer.
#
# strings|grep rather than a bare grep: grep treats binary input as non-text and
# its exit status there is not portable across implementations, so a plain
# `grep -qF` can report no match on a binary that demonstrably contains the sha.
# strings normalises to text lines first; binutils is already installed above.
#
# An image built with no REVISION is not a failure — it is a build that cannot
# name its commit, and it says so here and on every health response it serves.
RUNset -eu;\
if["$REVISION"="unknown"];then\
echo">> no REVISION build-arg: this image cannot name its commit, and every health response it serves will report revision=unknown";\
else\
strings -a /plugins/base | grep -qF "$REVISION"||{echo"FATAL: -X did not reach /plugins/base — github.com/hanzoai/cloud.revision was not resolved, so it was dropped and every health response would report 'unknown'";exit 1;};\
echo">> revision $REVISION linked into the plugins";\
fi
# Prove a SHIPPED sqlite-backed plugin binds sqlite3_* to libsqlcipher, not a
# plaintext libsqlite3. /plugins/base opens per-org stores under the SAME CGO=1 +
# libsqlite3 build every plugin above got, so it is a real witness for the set.
agentskills:## Regenerate the FULL agent-skills catalog into apps/agentskills/catalog (go:embed source) from the openapi SOT. OPENAPI_DIR=<path to openapi>.
skills:## Regenerate the FULL agent-skills catalog into apps/skills/catalog (go:embed source) from the openapi SOT. OPENAPI_DIR=<path to openapi>.
@test -f "$(OPENAPI_DIR)/skills.py"||{echo"openapi checkout not found at $(OPENAPI_DIR) — set OPENAPI_DIR=<path> or clone hanzoai/openapi";exit 1;}
# skills.py rewrites the whole catalog dir; the .gitignore keeps only the tiny
# `ai` fallback tracked, so the full set is embedded at build but never committed.
# Builds the host plus EXACTLY the plugins it is told to mount — not all 106.
# The host resolves a plugin as a file beside itself (manifest.App.Plugin), so a
# name in RUN_ENABLE with no binary in ./bin is the one way this fails; building
# that same list here is what keeps the two in step.
RUN_ENABLE?= iam,base,kms,gateway,o11y
# Builds the host plus the plugins you want to exercise locally — not all 106.
# The host mounts what manifest.Apps lists and resolves each plugin as a file
# beside itself (manifest.App.Plugin); a lazy one with no binary simply never
# starts, and a Required one fails loudly. So this list is a BUILD list, not a
# mount list — the binary has never taken one, and stating the app set a second
# time is what took devnet down twice.
RUN_PLUGINS?= iam,base,kms,gateway,o11y
run:cloud## Run the host with iam,base,kms,gateway,o11y (matches README quickstart); builds just those plugins.
@for a in $$(echo$(RUN_ENABLE)| tr ','' ');do$(MAKE) --no-print-directory plugin APP=$$a;done
./bin/cloud --enable=$(RUN_ENABLE)
run:cloud## Run the host, building the plugins in RUN_PLUGINS (iam,base,kms,gateway,o11y).
@for a in $$(echo$(RUN_PLUGINS)| tr ','' ');do$(MAKE) --no-print-directory plugin APP=$$a;done
./bin/cloud
# dev and lint are the names every repo in the fleet answers to. They are ALIASES
# of the two targets that already do the work, never copies of them, so each of
# those two things still has exactly one recipe.
dev:run## Alias for run.
smoke:## Build and run the smoke prober (mount-time integration check).
$(GO) run ./plugin/smoke
@@ -216,6 +260,26 @@ test: ## Run unit + integration tests (pure-Go, with the FTS5 tag the image ship
# never disagree with the generator it polices.
@set -e;for d in $$(grep -rl '^//go:generate go run github.com/zap-proto/zip/cmd/zipdoc' --include='*.go' clients cmd . 2>/dev/null | xargs -n1 dirname | sort -u);do(cd$$d &&$(GO) run github.com/zap-proto/zip/cmd/zipdoc -check)||{echo"$$d/zipdoc_gen.go is stale — run: go generate -run zipdoc ./$$d/...";exit 1;};done
@echo ">> openapi.yaml — $$(grep -c '^ /' openapi.yaml) paths. The MCP tool list is NOT an artifact: POST /v1/mcp asks every subsystem."
test-cgo:## Prove the cgo build works too — forces the fork's pure-Go backend via -tags sqlite_purego so the embedded modernc importers don't double-register "sqlite".
**The Open AI Cloud as one Go binary.** Identity, secrets, data, AI, gateway, observability, and the console — every Hanzo-native subsystem mounted into a single multi-org process. Per [HIP-0106](https://github.com/hanzoai/HIPs/blob/main/HIPs/hip-0106-unified-hanzo-cloud-binary.md).
**The Open AI Cloud as one deployment.** Identity, secrets, data, AI, gateway, observability, and the console — 116 Hanzo-native subsystems behind one origin and one `/v1`, each its own binary, composed by a light host router through the plugin contract in [HIP-0106](https://github.com/hanzoai/HIPs/blob/main/HIPs/hip-0106-hanzo-plugin-contract.md).
@@ -12,39 +12,77 @@ The same artifact serves `api.hanzo.ai`, `api.lux.cloud`, `api.zoo.cloud`, `api.
## Quick start
```bash
# Run the unified binary (pin a released version)
docker run -p 8080:8080 ghcr.io/hanzoai/cloud:v1.801.206
# The `hanzo` CLI is a separate Rust binary (hanzoai/cli) — this module ships no CLI
curl hanzo.sh | sh
brew install hanzoai/tap/hanzo
# Run the unified binary. `:latest` to try it; pin a v1.x.y tag for anything real
# — the tags are cut per build, so any number written here is stale by tomorrow.
docker run -p 8080:8080 ghcr.io/hanzoai/cloud:latest
```
Open <http://localhost:8080> for the embedded console; the API is served under `/v1` on the same origin.
Build this repo's own client binary with `go build ./cmd/hanzo` — see below for what it
serves and what it delegates. It is NOT what `curl -fsSL https://hanzo.sh | sh` installs;
that gets the Rust CLI (`hanzoai/cli`), which is the primary `hanzo` on a developer's
machine and whose verbs are different.
## What this is
`hanzoai/cloud`is one Go binary that mounts every Hanzo subsystem (iam, kms, base, gateway, ai, commerce, vfs, mq, dns, amqp, mcp, o11y, tasks, …) into a single multi-org process. The same artifact serves `api.hanzo.ai`, `api.osage.cloud`, `api.lux.cloud`, `api.zoo.cloud`, and every white-label reseller. Brand, enabled subsystems, and org scope are deployment configuration.
`hanzoai/cloud`serves the whole API from one origin. `cmd/cloud` is the front door: it
links `zip`, the app manifest and the console embed — and nothing else. It knows only
where each app lives and what path it answers, never what the app does. Each subsystem
(iam, kms, base, gateway, ai, commerce, vfs, mq, dns, amqp, mcp, o11y, tasks, …) is its
own `plugin/<name>` binary serving its own prefixes through the same `cloud.Listen`
middleware it would serve standalone.
Apps start **lazily**, on the first request that reaches their prefix; the four that own
a listener or a background loop (`pubsub`, `kafka`, `o11y`, `catalogsync`) say so and
start with the host. That is what makes 116 subsystems affordable — an app nobody calls
costs a route entry and a struct, not a process and a resident set.
This was one fused process once, and that binary is gone: it linked every subsystem's
graph into a ~3105-package build, and `apps.Wire()` went with it.
The same deployment serves `api.hanzo.ai`, `api.osage.cloud`, `api.lux.cloud`,
`api.zoo.cloud`, and every white-label reseller. Brand, enabled subsystems, and org
scope are deployment configuration.
## `hanzo` — cloud control CLI
The same binary is also a gcloud/doctl-class CLI. The first token selects the mode:
`cmd/hanzo` is the **client-only** control binary: a thin client over Hanzo IAM
(`hanzo.id`), the platform control plane (`platform.hanzo.ai/v1`) and the cloud
`/v1` API, inventing no parallel API. It cannot serve a subsystem — that is
Every subsystem mounts through the same `Mount(app, deps)` seam. Cross-subsystem calls ride a narrow in-process interface; no subsystem reaches into another's store.
Every app is loaded through the same `Mount` seam and answers on its own prefix; the
host takes the first prefix that matches and starts the app if it is not up yet. The
console is registered LAST so no app prefix can be shadowed. Cross-subsystem calls ride
ZAP; no subsystem reaches into another's store.
The host owns three things no app can: it serves the white-labelled console at `/`, it
threads the deployment's operator flags to the children as `CLOUD_*` env, and it SCOPES
CREDENTIALS — it scrubs the KMS root key from its own environment so no child inherits
it, and hands it to the kms broker child alone.
## White-label fork pattern
Customers fork `hanzoai/cloud` to launch their own ecosystem in one binary. Brand
detection, enabled subsystems, and ZAP endpoints (payments / vault backends) are all
deployment configuration.
Customers fork `hanzoai/cloud` to launch their own ecosystem. Brand detection, enabled
subsystems, and ZAP endpoints (payments / vault backends) are all deployment
configuration.
## Web framework
[hanzoai/zip](https://github.com/hanzoai/zip) — Sinatra-style Go web framework
built on Fiber v3. The ONE Go web framework. No `.Fast` escape hatch.
[zap-proto/zip](https://github.com/zap-proto/zip) — Sinatra-style Go web framework built
on Fiber v3. The ONE Go web framework. No `.Fast` escape hatch. That is the module path
this repo imports (`github.com/zap-proto/zip`, currently v1.18.22); `hanzoai/zip` is the
old home and is not what `go.mod` resolves.
## Console UI — embedded in the ONE binary
## Console UI — embedded in the host
The same `hanzoai/cloud` binary serves the [console](https://github.com/hanzoai/console)
(`@hanzo/gui`) UI at the web root AND the`/v1`API from one process — one
artifact, one origin, no separate console Service. The UI is compiled in via
`//go:embed` (see `webui.go`).
The host binary serves the console (`@hanzo/gui`, `hanzoai/console` — private) UI at the
web root AND routes`/v1`— one origin, no separate console Service. The UI is compiled
in via `//go:embed` (see `webui.go`).
Pipeline (in the `Dockerfile`, before `go build`):
@@ -150,30 +204,31 @@ calls same-origin), wraps the client catch-all pages for `output: 'export'`,
neutralizes the root layout's request-time `headers()` read, and emits a real
static export at `out/` (a ~360 KB `index.html` + `_next/` chunks). The image
build (and `make webui`) run it and overlay `webui/dist`, so `//go:embed` bakes
the FULL `@hanzo/gui` console into the ONE binary. The Dockerfile console stage
the FULL `@hanzo/gui` console into the host binary. The Dockerfile console stage
FAILS HARD if that bundle is missing or degenerate — the placeholder shell can
never silently ship to prod (escape hatch: `--build-arg ALLOW_PLACEHOLDER=1` for a
pure-Go dev image).
## Specs
Implements:
- HIP-0014 Application Deployment
-HIP-0026 IAM
-HIP-0027 KMS
-HIP-0037 AI Cloud Platform
-HIP-0105 In-Process Extension Runtime
-HIP-0106 Unified Cloud Binary
-HIP-0129 Open Cloud Planes
-HIP-0302 Encrypted SQLite + ZapDB Durability
Implements, by the filenames in [hanzoai/HIPs](https://github.com/hanzoai/HIPs/tree/main/HIPs):
Description:"Revokes the caller's own API key of the requested class. The class is\nthe same field mint takes — `?type=publishable`, defaulting to secret — so\nrevoking the key that ships in a browser bundle does not sign its holder out of\ntheir own API: the other key keeps working.\n\nRevoking is how a key is replaced when it does not need replacing; minting the\nsame class again rotates it in one step. IAM drops the credential immediately,\nbut the gateway caches keys for a few minutes, so a request that beat the cache\nexpiry may still be served.\n\nFor callers written against the older shape, the class is also accepted in a JSON\nrequest body, read only when `?type=` is absent.",
Fields:map[string]string{
"keyTypeIn.type":"Type is the key class to act on: \"secret\" (sk-, session-equivalent, belongs\non a server) or \"publishable\" (pk-, org-identifying, safe in a browser\nbundle). Omitted means secret, which is what every existing caller means.",
"revokedKey.ok":"OK is true when the key was revoked. A failure is an error status, never a\nfalse here.",
"revokedKey.type":"Type is the key class that was revoked, resolved — so a caller that named\nnothing can see it revoked the secret key.",
Description:"Streams a stored photo. No credentials — see the file header.",
})
zip.Describe("GET /v1/csrf",zip.Doc{
Description:"IssueCSRFToken mints the anti-CSRF token a browser echoes as X-CSRF-Token on\nevery money write (mint/revoke a key, top up, onboard, and the billing/commerce\nwrite verbs). The token is bound to the caller's validated identity and expires,\nso one minted for one identity cannot authorize a write as another.\n\nIt is answered no-store, so it is never cached by a shared proxy. This is the\nsame-origin endpoint the embedded console reads — the Same-Origin Policy is what\nstops a cross-site page from reading the response and forging a write.",
Fields:map[string]string{
"csrfResp.csrfToken":"Token is the value to send back in the X-CSRF-Token header. It is bound to the\ncaller's identity, so it authorizes writes as them and as nobody else.",
"csrfResp.expiresIn":"ExpiresIn is the token's lifetime in seconds. Fetch a new one when it lapses;\na write with an expired token is refused.",
},
})
zip.Describe("GET /v1/embed",zip.Doc{
Description:"Reports whether one of this brand's shared embedded apps (cms, erp,\nhelp) may be framed by the caller and is actually running, so a console module\ncan choose between the embed and the provision panel.\n\nIt answers two questions the browser cannot answer for itself. ENTITLEMENT is\nserver-authoritative: each app is a single shared per-BRAND instance, so only a\nmember of the owning brand org — or a SuperAdmin — is given the embed URL; every\nother caller gets phase \"not-entitled\" and no URL. REACHABILITY is a probe of\nthat origin, which a cross-origin page cannot read for itself.\n\nThe probed host is always <app>.<this deployment's own brand domain>: no part of\nit comes from the request, so this can never be steered into probing an\narbitrary origin.",
Fields:map[string]string{
"embedStatusReq.app":"App is the embedded app to report on: cms (Content Studio), erp or help.",
"embedStatusResp.app":"App is the app this verdict is about.",
"embedStatusResp.embedUrl":"EmbedURL is the in-app landing URL to frame. Empty when the caller is not\nentitled — a non-entitled caller never receives it.",
"embedStatusResp.entitled":"Entitled is whether the caller's org may frame this brand-owned app.",
"embedStatusResp.origin":"Origin is the app's origin on this deployment's own brand domain.",
"embedStatusResp.phase":"Phase is the verdict in one word: not-entitled, not-provisioned or ready.",
"embedStatusResp.reachable":"Reachable is whether the app answered the liveness probe.",
},
Example:json.RawMessage(`{"app":"cms"}`),
})
zip.Describe("GET /v1/keys",zip.Doc{
Description:"Returns the caller's own API keys — every type they hold, read\nAUTHORITATIVELY from IAM rather than from the session claim, which lags a key\nminted moments ago. No secret material comes back: a secret key is represented\nby its prefix, and only a publishable key (public by construction) carries its\nfull value.\n\nA transient IAM read failure reports an empty set rather than a 5xx, so the\npage shows the honest empty state and never a fabricated key.",
Fields:map[string]string{
"apiKey.createdAt":"CreatedAt is when the key last changed, as IAM records it.",
"apiKey.key":"Key is the FULL value, and is present for a publishable key only: it is\npublic by construction and useless to its holder if it cannot be read back.",
"apiKey.prefix":"Prefix is the recognizable, non-secret head of the key — enough to tell two\nkeys apart, never enough to use one.",
"apiKey.type":"Type is the key class: secret (sk-) or publishable (pk-).",
"apiKeyList.keys":"Keys is every key the caller holds, at most one per type.",
},
})
zip.Describe("POST /v1/keys",zip.Doc{
Description:"Creates — or rotates — the caller's API key of the requested type and\nreturns it ONCE. A real IAM failure surfaces as 502, never a fabricated key.\n\nRotating is what creating means here: a user holds one key per type, so the\nendpoint is idempotent by (caller, type) and the superseded credential stops\nworking. Two live secrets for one user would make \"revoke my key\" a lie.",
Fields:map[string]string{
"keyTypeIn.type":"Type is the key class to act on: \"secret\" (sk-, session-equivalent, belongs\non a server) or \"publishable\" (pk-, org-identifying, safe in a browser\nbundle). Omitted means secret, which is what every existing caller means.",
"mintedKey.accessKey":"AccessKey is the same value under its predecessor name, carried so callers\nwritten against the older field keep working. One value, two names.",
"mintedKey.key":"Key is the credential, returned ONCE — a secret key is unreadable afterwards.",
"mintedKey.type":"Type is the class of key that was minted.",
Description:"Onboard creates the caller's organization. Two flows, keyed on whether the caller\nalready has a home org (mirrors app/onboard/route.ts):\n\n - FIRST-RUN (no home org): create + MOVE the user in as admin, so their next\n JWT carries the new owner and the cloud scopes everything to it. This is the\n path a fresh OAuth sign-up takes, from the sign-up application's org.\n - ADDITIONAL (owner set): create the org but do NOT move the user — a move\n changes their IAM owner (stripping a SuperAdmin's status + orphaning their\n current org). They reach the new org via the OrgSwitcher, which re-scopes\n X-Org-Id without touching IAM membership. A personal-org request from someone\n who already has an org is meaningless → 409.",
Fields:map[string]string{
"onboardReq.name":"Name is the organization's display name. Ignored when personal is true, which\nderives the name from the caller's own username instead.",
"onboardReq.personal":"Personal asks for the caller's own workspace: the name is derived from their\nusername and the slug auto-suffixes to stay unique. Meaningless — and refused\n— for a caller who already has an organization.",
"onboardResp.accessKey":"AccessKey is the identifier of the org-scoped credential provisioning minted\nwith the organization. Present on a first run that actually minted one.",
"onboardResp.accessSecret":"AccessSecret is that credential's confidential half, returned ONCE — on the\nresponse that mints it and never again. IAM keeps only its argon2id digest\nand blanks the plaintext, so this is the single moment it exists in a form\nits owner can read; a replay of the same provision re-reveals nothing.",
"onboardResp.additional":"Additional is true when the caller already had an organization and this one\nwas created WITHOUT moving them into it — they reach it via the org switcher.",
"onboardResp.displayName":"DisplayName is the organization's human name.",
"onboardResp.org":"Org is the created organization's slug, which is what X-Org-Id carries.",
Description:"Records reads cloud's tamper-evident audit trail, newest first, with the chain's live\nintegrity attached so a listing can be badged as verified.\n\nWhen cloud has no local store configured it falls back to forwarding IAM's own\nget-records trail verbatim — a DIFFERENT trail, federated so the endpoint never\nregresses to an empty list. Those rows carry no integrity of ours, so the field is\nnull there.",
Description:"Reads cloud's tamper-evident audit trail, newest first, with the chain's live\nintegrity attached so a listing can be badged as verified.\n\nWhen cloud has no local store configured it falls back to forwarding IAM's own\nget-records trail verbatim — a DIFFERENT trail, federated so the endpoint never\nregresses to an empty list. Those rows carry no integrity of ours, so the field is\nnull there.",
Fields:map[string]string{
"Integrity.brokenAt":"BrokenAt is the seq of the FIRST record that failed verification, or -1 when\nOK. Reason describes the break (recomputed-hash mismatch, prev-hash\ndiscontinuity, or a seq gap).",
"Integrity.count":"Count is the number of records walked.",
"Integrity.headHash":"HeadHash is the hash of the last record (or the genesis anchor for an empty\nchain). Pin this externally over time to detect tail-truncation.",
"Integrity.ok":"OK is true iff every record's stored hash equals the recomputed hash AND the\nchain links are continuous (each PrevHash == the prior record's Hash, seqs\ngapless from 0).",
"RecordsIn.action":"Action restricts it to one action name, e.g. \"admin.waitlist.grant\".",
"RecordsIn.org":"Org restricts the trail to one tenant.",
"RecordsIn.p":"Page is the 1-based page number, driving the offset.",
@@ -24,10 +28,16 @@ func init() {
"RecordsIn.until":"Until is the upper time bound, RFC3339, with the same tolerance.",
Description:"Verify walks the WHOLE hash chain and reports whether it is intact: how many records\nwere checked, the head hash to pin externally against tail-truncation, and — when the\nchain is broken — the seq of the first bad record and why.\n\nbrokenAt is -1 exactly when ok is true. An unconfigured store is an honest failure\nhere rather than a fabricated pass.",
Description:"Walks the WHOLE hash chain and reports whether it is intact: how many records\nwere checked, the head hash to pin externally against tail-truncation, and — when the\nchain is broken — the seq of the first bad record and why.\n\nbrokenAt is -1 exactly when ok is true. An unconfigured store is an honest failure\nhere rather than a fabricated pass.",
Fields:map[string]string{
"Integrity.brokenAt":"BrokenAt is the seq of the FIRST record that failed verification, or -1 when\nOK. Reason describes the break (recomputed-hash mismatch, prev-hash\ndiscontinuity, or a seq gap).",
"Integrity.count":"Count is the number of records walked.",
"Integrity.headHash":"HeadHash is the hash of the last record (or the genesis anchor for an empty\nchain). Pin this externally over time to detect tail-truncation.",
"Integrity.ok":"OK is true iff every record's stored hash equals the recomputed hash AND the\nchain links are continuous (each PrevHash == the prior record's Hash, seqs\ngapless from 0).",
Description:"Customers lists every customer org at a glance, sorted by slug: owner email, plan,\nsuspend status, member count, balance, month-to-date spend and MRR.\n\nEach row costs one IAM read plus the org's money reads, fanned out under a fixed\nconcurrency ceiling so a large fleet cannot stampede the upstreams. Every read is\nbest-effort per row: an upstream miss degrades THAT field to its honest zero rather\nthan failing the fleet.",
Description:"Lists every customer org at a glance, sorted by slug: owner email, plan,\nsuspend status, member count, balance, month-to-date spend and MRR.\n\nEach row costs one IAM read plus the org's money reads, fanned out under a fixed\nconcurrency ceiling so a large fleet cannot stampede the upstreams. Every read is\nbest-effort per row: an upstream miss degrades THAT field to its honest zero rather\nthan failing the fleet.",
"OrgIn.org":"Org is the tenant slug from the path.",
},
})
zip.Describe("GET /v1/admin/grants",zip.Doc{
Description:"Grants reads the credit-grant ledger across ALL orgs, newest first — who granted what\nto whom, when, and from which money bucket.\n\nIt is a PROJECTION of the tamper-evident audit trail, not a second store: every grant\nis written there as action \"admin.customer.credit\", so this view cannot drift from\nwhat actually happened, and FAILED grants appear too.\n\nA deployment with no local audit store has no history to project, and says so with an\nempty list and a msg rather than an error.",
Description:"Reads the credit-grant ledger across ALL orgs, newest first — who granted what\nto whom, when, and from which money bucket.\n\nIt is a PROJECTION of the tamper-evident audit trail, not a second store: every grant\nis written there as action \"admin.customer.credit\", so this view cannot drift from\nwhat actually happened, and FAILED grants appear too.\n\nA deployment with no local audit store has no history to project, and says so with an\nempty list and a msg rather than an error.",
Fields:map[string]string{
"GrantRow.actor":"staff email (or sub) who issued it",
"GrantRow.result":"success | error",
@@ -34,23 +34,31 @@ func init() {
"GrantsIn.result":"Result filters by outcome: \"success\" or \"error\". Empty returns both, which is\nthe point of this view — a refused grant is as interesting as a granted one.",
Description:"GrantCredit issues a staff credit grant to the org named in the path — a comp, refund\nor promo — through the ONE credit-write path core.ApplyGrant, which validates the\namount against the per-grant cap, checks the org exists, moves the money and records\nthe tamper-evident audit row.\n\nThe credit lands on the account account.Payer resolves, NOT necessarily the org: name\na member of a pooled org and the pool is credited. The receipt echoes the subject so\nthe caller can see which.",
Description:"Issues a staff credit grant to the org named in the path — a comp, refund\nor promo — through the ONE credit-write path core.ApplyGrant, which validates the\namount against the per-grant cap, checks the org exists, moves the money and records\nthe tamper-evident audit row.\n\nThe credit lands on the account account.Payer resolves, NOT necessarily the org: name\na member of a pooled org and the pool is credited. The receipt echoes the subject so\nthe caller can see which.",
Fields:map[string]string{
"GrantIn.amountCents":"AmountCents is the credit, in whole cents. Must be positive and within the\nper-grant cap.",
"GrantIn.currency":"Currency is the ISO code, lower-cased. Empty means usd.",
"GrantIn.org":"Org is the tenant to credit. Required.",
"GrantIn.reason":"Reason is the operator's justification, recorded on the audit row.",
"GrantIn.source":"Source is the money bucket: \"trial\" (default) for a non-cash comp that is never\nrefundable, or \"prepaid\" for real money. Anything unknown falls back to trial.",
"GrantIn.user":"User optionally names a MEMBER to credit, by bare IAM username. Empty credits\nthe org. Which of the two the money actually lands on is decided by\naccount.Payer, not here: a pooled org keeps one balance whatever is named.",
"GrantIn.amountCents":"AmountCents is the credit, in whole cents. Must be positive and within the\nper-grant cap.",
"GrantIn.currency":"Currency is the ISO code, lower-cased. Empty means usd.",
"GrantIn.org":"Org is the tenant to credit. Required.",
"GrantIn.reason":"Reason is the operator's justification, recorded on the audit row.",
"GrantIn.source":"Source is the money bucket: \"trial\" (default) for a non-cash comp that is never\nrefundable, or \"prepaid\" for real money. Anything unknown falls back to trial.",
"GrantIn.user":"User optionally names a MEMBER to credit, by bare IAM username. Empty credits\nthe org. Which of the two the money actually lands on is decided by\naccount.Payer, not here: a pooled org keeps one balance whatever is named.",
"GrantResult.balanceCents":"BalanceCents is the account balance AFTER the grant, in whole cents.",
"GrantResult.balanceExact":"BalanceExact is that same balance at full 18-decimal precision, so a sub-cent\ndebit is visible rather than rounded away.",
"GrantResult.currency":"Currency is the lower-cased ISO code the grant was denominated in.",
"GrantResult.grantedCents":"GrantedCents is the amount actually credited.",
"GrantResult.org":"Org is the tenant whose ledger was credited.",
"GrantResult.source":"Source is the money bucket: \"trial\" (non-cash comp) or \"prepaid\" (real money).",
"GrantResult.subject":"Subject is the ACCOUNT the credit landed on inside that ledger: the org slug for\na pooled org, \"<org>/<name>\" for a member of a per-member one. It is echoed\nbecause the operator does not choose it — account.Payer does — so naming a\nmember of a pooled org credits the pool and the receipt has to say so.",
"GrantResult.transactionId":"TransactionID is the ledger entry id, for reconciliation against commerce.",
Description:"ReactivateCustomer restores access for every member of the org, undoing a suspend. It\nreports the same per-user breakdown.",
Description:"Restores access for every member of the org, undoing a suspend. It\nreports the same per-user breakdown.",
Fields:map[string]string{
"AccessChange.affected":"Affected lists the usernames that were updated.",
"AccessChange.failed":"Failed lists the usernames that were NOT updated. Non-empty means the org is in\na mixed state and the action should be retried.",
Description:"SuspendCustomer cuts off every member of the org: IAM refuses a forbidden user at\nlogin AND at token issuance, so a suspended customer can neither sign in nor mint a\nfresh token. Fully reversible with ReactivateCustomer.\n\nThe result names every user updated and every user that was NOT — a partial failure\nleaves the org in a mixed state and says so instead of reporting a clean success.",
Description:"Cuts off every member of the org: IAM refuses a forbidden user at\nlogin AND at token issuance, so a suspended customer can neither sign in nor mint a\nfresh token. Fully reversible with ReactivateCustomer.\n\nThe result names every user updated and every user that was NOT — a partial failure\nleaves the org in a mixed state and says so instead of reporting a clean success.",
Fields:map[string]string{
"AccessChange.affected":"Affected lists the usernames that were updated.",
"AccessChange.failed":"Failed lists the usernames that were NOT updated. Non-empty means the org is in\na mixed state and the action should be retried.",
Description:"IssueGrant issues a credit grant to any org from the operator Grants view, with the\ntarget named in the body. It funnels through the SAME core.ApplyGrant that\nPOST /v1/admin/customers/:org/credit uses, so there is exactly ONE credit-write path\nand one audit trail behind both.",
Description:"Issues a credit grant to any org from the operator Grants view, with the\ntarget named in the body. It funnels through the SAME core.ApplyGrant that\nPOST /v1/admin/customers/:org/credit uses, so there is exactly ONE credit-write path\nand one audit trail behind both.",
Fields:map[string]string{
"GrantIn.amountCents":"AmountCents is the credit, in whole cents. Must be positive and within the\nper-grant cap.",
"GrantIn.currency":"Currency is the ISO code, lower-cased. Empty means usd.",
"GrantIn.org":"Org is the tenant to credit. Required.",
"GrantIn.reason":"Reason is the operator's justification, recorded on the audit row.",
"GrantIn.source":"Source is the money bucket: \"trial\" (default) for a non-cash comp that is never\nrefundable, or \"prepaid\" for real money. Anything unknown falls back to trial.",
"GrantIn.user":"User optionally names a MEMBER to credit, by bare IAM username. Empty credits\nthe org. Which of the two the money actually lands on is decided by\naccount.Payer, not here: a pooled org keeps one balance whatever is named.",
"GrantIn.amountCents":"AmountCents is the credit, in whole cents. Must be positive and within the\nper-grant cap.",
"GrantIn.currency":"Currency is the ISO code, lower-cased. Empty means usd.",
"GrantIn.org":"Org is the tenant to credit. Required.",
"GrantIn.reason":"Reason is the operator's justification, recorded on the audit row.",
"GrantIn.source":"Source is the money bucket: \"trial\" (default) for a non-cash comp that is never\nrefundable, or \"prepaid\" for real money. Anything unknown falls back to trial.",
"GrantIn.user":"User optionally names a MEMBER to credit, by bare IAM username. Empty credits\nthe org. Which of the two the money actually lands on is decided by\naccount.Payer, not here: a pooled org keeps one balance whatever is named.",
"GrantResult.balanceCents":"BalanceCents is the account balance AFTER the grant, in whole cents.",
"GrantResult.balanceExact":"BalanceExact is that same balance at full 18-decimal precision, so a sub-cent\ndebit is visible rather than rounded away.",
"GrantResult.currency":"Currency is the lower-cased ISO code the grant was denominated in.",
"GrantResult.grantedCents":"GrantedCents is the amount actually credited.",
"GrantResult.org":"Org is the tenant whose ledger was credited.",
"GrantResult.source":"Source is the money bucket: \"trial\" (non-cash comp) or \"prepaid\" (real money).",
"GrantResult.subject":"Subject is the ACCOUNT the credit landed on inside that ledger: the org slug for\na pooled org, \"<org>/<name>\" for a member of a per-member one. It is echoed\nbecause the operator does not choose it — account.Payer does — so naming a\nmember of a pooled org credits the pool and the receipt has to say so.",
"GrantResult.transactionId":"TransactionID is the ledger entry id, for reconciliation against commerce.",
Description:"Finance answers GET /v1/admin/finance. It reads the multi-vendor COGS from commerce\n/v1/costs, the DO promo-credit/burn-down treasury view, and the fleet commerce revenue,\nthen hands them to ComputeFinance. SuperAdmin only.",
Description:"Answers GET /v1/admin/finance. It reads the multi-vendor COGS from commerce\n/v1/costs, the DO promo-credit/burn-down treasury view, and the fleet commerce revenue,\nthen hands them to ComputeFinance. SuperAdmin only.",
Description:"UsageFunding splits our upstream AI usage by how it was FUNDED: one row per (provider,\nmodel) over the window, tagged credit (provider grant still remaining), paid (grant\nexhausted) or paid_only (no grant at all).\n\nThe class is resolved at the PROVIDER level from the credit ledger, not per call — the\nper-call split, and the `byo` class, arrive when the metering write stamps a funding\ncolumn on cloud_usage and this can GROUP BY it directly. Until then a provider with\nremaining grant reports all of its usage as credit, which is right in aggregate and\napproximate at the boundary where a grant runs out mid-window.\n\nAn unparseable window falls back to the last 30 days rather than refusing: this is a\ndashboard read, and a typo in a date must not blank the board.",
Description:"Splits our upstream AI usage by how it was FUNDED: one row per (provider,\nmodel) over the window, tagged credit (provider grant still remaining), paid (grant\nexhausted) or paid_only (no grant at all).\n\nThe class is resolved at the PROVIDER level from the credit ledger, not per call — the\nper-call split, and the `byo` class, arrive when the metering write stamps a funding\ncolumn on cloud_usage and this can GROUP BY it directly. Until then a provider with\nremaining grant reports all of its usage as credit, which is right in aggregate and\napproximate at the boundary where a grant runs out mid-window.\n\nAn unparseable window falls back to the last 30 days rather than refusing: this is a\ndashboard read, and a typo in a date must not blank the board.",
Fields:map[string]string{
"UsageFundingIn.from":"From is the inclusive start of the window. Unparseable or absent, together with\nTo, falls back to the last 30 days.",
"UsageFundingIn.to":"To is the exclusive end of the window.",
Description:"Backfill carries ONE org's current commerce prepaid balance into the native finance\nwallet — the one-time cutover between the two ledgers.\n\nIt is IDEMPOTENT: the deposit uses the fixed ref \"backfill:<org>\", so re-running it\ncredits the wallet at most once. Safe to retry.\n\nThe pre-migration balance is read from the CO-RESIDENT commerce ledger, not over HTTP:\nthe admin HTTP client dials an unroutable in-process address and would read $0, and a\nphantom zero would silently carry nothing while reporting success. When commerce is\nnot co-resident this fails rather than migrating nothing.",
Description:"Carries ONE org's current commerce prepaid balance into the native finance\nwallet — the one-time cutover between the two ledgers.\n\nIt is IDEMPOTENT: the deposit uses the fixed ref \"backfill:<org>\", so re-running it\ncredits the wallet at most once. Safe to retry.\n\nThe pre-migration balance is read from the CO-RESIDENT commerce ledger, not over HTTP:\nthe admin HTTP client dials an unroutable in-process address and would read $0, and a\nphantom zero would silently carry nothing while reporting success. When commerce is\nnot co-resident this fails rather than migrating nothing.",
Fields:map[string]string{
"BackfillIn.org":"Org is the tenant to migrate. Required — there is no fleet-wide form of this\ncutover, because each org must be reconciled on its own.",
"Backfilled.entryId":"EntryID is the finance ledger entry created, or \"\" when the balance was\nnon-positive and there was nothing to carry.",
Description:"deleteDroplet destroys a droplet the board has just proven is NOT a DOKS node. There\nis no snapshot-first undo for a droplet the way there is for a volume: the local disk\ngoes with it.",
Description:"Destroys a droplet the board has just proven is NOT a DOKS node. There\nis no snapshot-first undo for a droplet the way there is for a volume: the local disk\ngoes with it.",
Fields:map[string]string{
"DropletIn.disk":"Disk requests a PERMANENT resize that grows the disk. DO can never resize such a\ndroplet down again, so it defaults false — a CPU/RAM-only change, reversible.",
"DropletIn.id":"ID is the DO droplet id, from the path. Numeric.",
Description:"deleteVolume destroys a volume the board has just proven no PersistentVolume in any\ncluster references. Irreversible, so it snapshots first unless explicitly waived —\nthe snapshot IS the undo.",
Description:"Destroys a volume the board has just proven no PersistentVolume in any\ncluster references. Irreversible, so it snapshots first unless explicitly waived —\nthe snapshot IS the undo.",
Fields:map[string]string{
"VolumeIn.id":"ID is the DO volume id, from the path.",
"VolumeIn.name":"Name is the snapshot name on the snapshot action. Blank gets a deterministic\n\"<volume>-predelete-<unix>\" so the undo is findable in the DO console.",
Description:"read serves the whole DigitalOcean infrastructure board: droplets, volumes, DOKS\nclusters and load balancers, each cross-referenced against every cluster's live\nKubernetes state so the board can say what is safe to destroy and what is not.\n\nIt is cached for up to a minute because one read is a fan-out over the DO API plus a\nfull pod/PV listing per cluster. Staleness is never load-bearing: every MUTATION\nre-scans from scratch and ignores this cache.\n\nOnly an unusable DO account is a hard failure. A partial read still produces a board,\nwith the failing source named in sources[] — except for clusters and volumes, which\nthe safety verdict depends on; without those the analysis degrades rather than\nclassifying anything it cannot prove.",
Description:"Serves the whole DigitalOcean infrastructure board: droplets, volumes, DOKS\nclusters and load balancers, each cross-referenced against every cluster's live\nKubernetes state so the board can say what is safe to destroy and what is not.\n\nIt is cached for up to a minute because one read is a fan-out over the DO API plus a\nfull pod/PV listing per cluster. Staleness is never load-bearing: every MUTATION\nre-scans from scratch and ignores this cache.\n\nOnly an unusable DO account is a hard failure. A partial read still produces a board,\nwith the failing source named in sources[] — except for clusters and volumes, which\nthe safety verdict depends on; without those the analysis degrades rather than\nclassifying anything it cannot prove.",
Fields:map[string]string{
"Cost.wastedMonthly":"WastedMonthly is what the fleet pays every month for provisioned-but-empty space on\nthe volumes a kubelet actually measured.\n\nIt is NOT ReclaimableMonthly and must never be added to it. Reclaimable is money a\nbutton on this board collects, by deleting volumes proven to belong to no one.\nWasted is money locked inside volumes that are IN USE and holding live data:\nDigitalOcean can only ever grow a volume, so collecting it means copying a database\nonto a smaller one. See shrinkRecipe.\n\nIt is also a LOWER BOUND — unmeasured volumes contribute nothing.",
"LoadBalancer.service":"Service is the `namespace/name` of the live type=LoadBalancer Service that claims\nthis load balancer, proven from the cluster scan. Non-empty means IN USE.",
Description:"scaleNodePool sets a node pool's node count — the ONE correct way to change how many\nnodes a DOKS cluster has.\n\nThe response states what the board could NOT prove: DOKS picks which nodes a shrink\nremoves, so no particular pod is shown to survive one. See NodePool.ScaleTo.",
Description:"Sets a node pool's node count — the ONE correct way to change how many\nnodes a DOKS cluster has.\n\nThe response states what the board could NOT prove: DOKS picks which nodes a shrink\nremoves, so no particular pod is shown to survive one. See NodePool.ScaleTo.",
Fields:map[string]string{
"ScaleIn.count":"Count is the node count to set.",
"ScaleIn.id":"ID is the DOKS cluster id, from the path.",
Description:"resizeDroplet changes a droplet's plan. Same refusal as delete and for the same\nreason: a DOKS node's size is the node pool's to declare.\n\ndisk=true is a PERMANENT resize — the disk grows and DO can never resize the droplet\nDOWN again. disk=false (the default) changes CPU/RAM only and is reversible. DO\nrequires the droplet to be powered off and applies the change asynchronously, so the\nresponse carries the action to poll, not a completed change.",
Description:"Changes a droplet's plan. Same refusal as delete and for the same\nreason: a DOKS node's size is the node pool's to declare.\n\ndisk=true is a PERMANENT resize — the disk grows and DO can never resize the droplet\nDOWN again. disk=false (the default) changes CPU/RAM only and is reversible. DO\nrequires the droplet to be powered off and applies the change asynchronously, so the\nresponse carries the action to poll, not a completed change.",
Fields:map[string]string{
"DropletIn.disk":"Disk requests a PERMANENT resize that grows the disk. DO can never resize such a\ndroplet down again, so it defaults false — a CPU/RAM-only change, reversible.",
"DropletIn.id":"ID is the DO droplet id, from the path. Numeric.",
Description:"cordonNode marks one cluster node unschedulable — or schedulable again — and can drain\nthe pods already on it.\n\nIt is the ONE infra change that does not go through the run discipline, because there\nis no destructive verdict to check: cordoning is reversible and evicting respects the\ncluster's own PodDisruptionBudgets. It reads the cached board for the same reason.\nThe outcome is audited either way, and the result reports how many pods were evicted.",
Description:"Marks one cluster node unschedulable — or schedulable again — and can drain\nthe pods already on it.\n\nIt is the ONE infra change that does not go through the run discipline, because there\nis no destructive verdict to check: cordoning is reversible and evicting respects the\ncluster's own PodDisruptionBudgets. It reads the cached board for the same reason.\nThe outcome is audited either way, and the result reports how many pods were evicted.",
Fields:map[string]string{
"CordonIn.cordon":"Cordon true marks the node unschedulable; false restores it.",
"CordonIn.drain":"Drain additionally evicts the pods already running there.",
Description:"expandVolume grows a volume. GROW ONLY — see Volume.ExpandTo for why the other\ndirection is a data migration this board deliberately refuses to run.\n\nThe MECHANISM follows the volume's owner, because there is exactly one way to grow each\nkind completely. A volume a PVC claims is grown by patching the claim: the CSI driver\nthen resizes the DigitalOcean device AND grows the filesystem on it, leaving claim, PV,\ndevice and filesystem all agreeing. Calling DigitalOcean directly for that volume would\ngrow the device while the PV kept declaring the old capacity and the filesystem never\ngrew at all. One operation, one correct mechanism per owner — not two ways to do it.",
Description:"Grows a volume. GROW ONLY — see Volume.ExpandTo for why the other\ndirection is a data migration this board deliberately refuses to run.\n\nThe MECHANISM follows the volume's owner, because there is exactly one way to grow each\nkind completely. A volume a PVC claims is grown by patching the claim: the CSI driver\nthen resizes the DigitalOcean device AND grows the filesystem on it, leaving claim, PV,\ndevice and filesystem all agreeing. Calling DigitalOcean directly for that volume would\ngrow the device while the PV kept declaring the old capacity and the filesystem never\ngrew at all. One operation, one correct mechanism per owner — not two ways to do it.",
Fields:map[string]string{
"VolumeIn.id":"ID is the DO volume id, from the path.",
"VolumeIn.name":"Name is the snapshot name on the snapshot action. Blank gets a deterministic\n\"<volume>-predelete-<unix>\" so the undo is findable in the DO console.",
Description:"snapshotVolume takes a point-in-time snapshot of one volume — the undo a delete relies\non, available on its own so an operator can take one before any risky change.\n\nIt re-scans the board first (never the cache) so the volume it snapshots is one that\nexists right now, and audits the outcome either way.",
Description:"Takes a point-in-time snapshot of one volume — the undo a delete relies\non, available on its own so an operator can take one before any risky change.\n\nIt re-scans the board first (never the cache) so the volume it snapshots is one that\nexists right now, and audits the outcome either way.",
Fields:map[string]string{
"VolumeIn.id":"ID is the DO volume id, from the path.",
"VolumeIn.name":"Name is the snapshot name on the snapshot action. Blank gets a deterministic\n\"<volume>-predelete-<unix>\" so the undo is findable in the DO console.",
Description:"Metrics answers GET /v1/admin/metrics by aggregating commerce.events directly\n(fleet-wide, no per-org fan-out). SuperAdmin only.\n\n\tGET /v1/admin/metrics?window=30d&limit=20",
Description:"Answers GET /v1/admin/metrics by aggregating commerce.events directly\n(fleet-wide, no per-org fan-out). SuperAdmin only.\n\n\tGET /v1/admin/metrics?window=30d&limit=20",
Fields:map[string]string{
"MetricsIn.limit":"Limit caps the top-customers table.",
"MetricsIn.window":"Window is the movement window the new/churned MRR and the recent feed are\nmeasured over. Anything unrecognised falls back to the board default.",
Description:"Revenue is the fleet money board: total prepaid balances held, total realized spend,\nMRR, ARPU, a per-customer table sorted highest-revenue first, and a real 30-day spend\ntrend from the usage ledger.\n\nORTHOGONAL to /v1/admin/finance, which is the COGS/margin view of what WE pay vendors.\nThis is the customer side: what each customer holds, spends and subscribes to.\n\narpu divides realized spend by PAYING customers, not by all of them — a fleet of free\nsignups must not deflate the number. A customer counts as paying when it has spend or\nMRR.\n\nAn org whose money did not read degrades to honest zeros and marks the commerce source\ndegraded in sources[], so a partial fleet read is visible instead of quietly low.",
Description:"Is the fleet money board: total prepaid balances held, total realized spend,\nMRR, ARPU, a per-customer table sorted highest-revenue first, and a real 30-day spend\ntrend from the usage ledger.\n\nORTHOGONAL to /v1/admin/finance, which is the COGS/margin view of what WE pay vendors.\nThis is the customer side: what each customer holds, spends and subscribes to.\n\narpu divides realized spend by PAYING customers, not by all of them — a fleet of free\nsignups must not deflate the number. A customer counts as paying when it has spend or\nMRR.\n\nAn org whose money did not read degrades to honest zeros and marks the commerce source\ndegraded in sources[], so a partial fleet read is visible instead of quietly low.",
Description:"deleteSpendCap removes one cap by id, lifting the ceiling entirely.",
zip.Describe("DELETE /v1/admin/caps/:id",zip.Doc{
Description:"Removes one cap by id, lifting the ceiling entirely.",
Fields:map[string]string{
"capIn.id":"ID is the cap to edit or remove, from the path. Unused by the list and create ops.",
"capIn.org":"Org is the tenant to act on. Required for a SuperAdmin — they must name their\ntarget; ignored for a white-label admin, who always acts on their own org.",
Description:"aimetrics is the fleet AI board: O11yAI generations (count, cost,avg/p95 latency,\nper-model), per-model usage from the live cloud_usage ledger, andthe eval plane\n(traces, scores, score names, runs, and the average-score trend).\n\nEvery signal degrades INDEPENDENTLY — a table that is absent or errors contributes its\nzero value and the read still succeeds. O11yAI latency is a SEPARATE query from\ngenerations and cost on purpose: a Nullable end_time or a column mismatch there must\nnot zerothe two numbers that did read.",
Description:"Is the fleet AI board: LLM generations over gen_ai spans (count, cost,\navg/p95 latency,per-model), per-model usage from the live cloud_usage ledger, and\nthe eval plane(traces, scores, score names, runs, and the average-score trend).\n\nEvery signal degrades INDEPENDENTLY — a table that is absent or errors contributes its\nzero value and the read still succeeds. Generation latency is a SEPARATE query from\ngenerations and cost on purpose: a duration/attribute mismatch there mustnot zero\nthe two numbers that did read.",
Description:"analytics is the SaaS product-analytics board over the caller's tenant window: active\ncustomers, new and churned, retention, MRR, ARPU, the usage trend and the top\ncustomers by spend — every number folded from the commerce ledger, not sampled.\n\nThe window is the caller's, not the fleet's: a SuperAdmin gets every org, a\nwhite-label admin only their own subtree (core.ScopedOrgs, the one scope predicate).\n\nsources[] carries each upstream's freshness so a partial read is VISIBLE rather than\nsilently low: a ledger that answered for only some orgs marks commerce-ledger degraded\ninstead of publishing an undercount as healthy.",
Description:"Is the SaaS product-analytics board over the caller's tenant window: active\ncustomers, new and churned, retention, MRR, ARPU, the usage trend and the top\ncustomers by spend — every number folded from the commerce ledger, not sampled.\n\nThe window is the caller's, not the fleet's: a SuperAdmin gets every org, a\nwhite-label admin only their own subtree (core.ScopedOrgs, the one scope predicate).\n\nsources[] carries each upstream's freshness so a partial read is VISIBLE rather than\nsilently low: a ledger that answered for only some orgs marks commerce-ledger degraded\ninstead of publishing an undercount as healthy.",
Fields:map[string]string{
"analyticsData.activeCustomers":"Active customers — from the usage ledger.",
"analyticsData.churn":"Churn — logo churn (count) + rate.",
Description:"applications lists IAM applications for one owner org, forwarded VERBATIM from IAM's\nget-applications. These are the platform's OIDC clients — the console reads clientId\noff each row.",
Description:"Lists IAM applications for one owner org, forwarded VERBATIM from IAM's\nget-applications. These are the platform's OIDC clients — the console reads clientId\noff each row.",
Fields:map[string]string{
"iamPageIn.owner":"Owner is the org whose rows to read. Defaults to the admin org, which owns the\nplatform's roles and applications.",
"iamPageIn.p":"Page is the 1-based page number. Forwarded only when set — IAM applies its own\ndefault otherwise.",
"iamPageIn.pageSize":"PageSize is rows per page. Forwarded only when set.",
Description:"bases lists the tenant Base instances in the caller's window — a SuperAdmin sees every\ntenant's, anyone else only their own subtree's.\n\nThe scope is enforced TWICE: the upstream is asked for the caller's org, AND every row\nit returns is re-checked against the resolved scope. An upstream that ignored the\nfilter therefore degrades to empty, never to a cross-tenant leak.\n\nThe Base engine is being embedded into cloud; until it lands this proxies\nBASE_ADMIN_URL and, when that is unset, answers 200 with an empty list and msg saying\nso — the honest not-yet state, never fabricated instances.",
Description:"Lists the tenant Base instances in the caller's window — a SuperAdmin sees every\ntenant's, anyone else only their own subtree's.\n\nThe scope is enforced TWICE: the upstream is asked for the caller's org, AND every row\nit returns is re-checked against the resolved scope. An upstream that ignored the\nfilter therefore degrades to empty, never to a cross-tenant leak.\n\nThe Base engine is being embedded into cloud; until it lands this proxies\nBASE_ADMIN_URL and, when that is unset, answers 200 with an empty list and msg saying\nso — the honest not-yet state, never fabricated instances.",
Description:"blockStorage is the realtime block-storage board: the DigitalOcean volume fleet\n(count, capacity, monthly list cost, per-volume region and attachment) plus the\nanalytics datastore's OWN fill, read from its system.disks.\n\nA volume's usedGiB and pct are null, always: DO exposes capacity and attachment but no\nfill, so the console renders \"—\" rather than a number nobody measured. The datastore\ncard is the one real fill here, and it is the number to scale on.\n\nThe two sources degrade independently — a DO outage still returns the datastore fill,\nand a disconnected datastore still returns the DO fleet.",
Description:"Reads one org's usage caps: its spend alerts plus the derived period\nspend, over/warn state and reset time.\n\nThese are the SAME rows the customer edits in their own console — a platform override\nand a customer budget are one model, not two.",
Fields:map[string]string{
"capIn.id":"ID is the cap to edit or remove, from the path. Unused by the list and create ops.",
"capIn.org":"Org is the tenant to act on. Required for a SuperAdmin — they must name their\ntarget; ignored for a white-label admin, who always acts on their own org.",
Description:"compute rolls the fleet's compute usage up to one row per (org, app, project, kind):\nhow many distinct machines ran in the window, how many are still active, what they\nbilled, and when each group last emitted an event. The console folds these into its\norg → app → project tree.\n\nA machine counts as ACTIVE when its LATEST lifecycle event is not a terminal one\n(stop/destroy/terminate/delete/off/shutdown/expire and their past tenses) — the same\nfold the console applies, done in the warehouse so the count is over every machine and\nnot just the page.\n\nHonest-empty when the warehouse is not connected or hanzo.compute_usage is not\nprovisioned yet: an empty list, never a fabricated fleet.",
Description:"Rolls the fleet's compute usage up to one row per (org, app, project, kind):\nhow many distinct machines ran in the window, how many are still active, what they\nbilled, and when each group last emitted an event. The console folds these into its\norg → app → project tree.\n\nA machine counts as ACTIVE when its LATEST lifecycle event is not a terminal one\n(stop/destroy/terminate/delete/off/shutdown/expire and their past tenses) — the same\nfold the console applies, done in the warehouse so the count is over every machine and\nnot just the page.\n\nHonest-empty when the warehouse is not connected or hanzo.compute_usage is not\nprovisioned yet: an empty list, never a fabricated fleet.",
Fields:map[string]string{
"computeIn.kind":"Kind narrows to one workload class (bot | machine | cluster | nodepool |\ncontainer | function | …). An OPEN spectrum matched as a plain string, lowercased\nto the warehouse's convention; empty means every kind.",
"computeIn.org":"Org narrows to one tenant. Empty means every tenant — this board is\ncross-tenant by nature.",
"computeIn.range":"Range is the lower time bound: 24h, 7d or 30d. Anything else reads as 30d.",
Description:"flagsBoard reads the platform control-plane board: every runtime launch/release\nswitch (waitlist, public signup, subsystem activation, gateway limits, network ids)\nwith its LIVE value and where that value came from — a stored definition or the\ncompiled-in default.",
Description:"Reads the platform control-plane board: every runtime launch/release\nswitch (waitlist, public signup, subsystem activation, gateway limits, network ids)\nwith its LIVE value and where that value came from — a stored definition or the\ncompiled-in default.",
Response:json.RawMessage(`{"status":"ok","msg":"","data":{"switches":[{"key":"waitlist.chat","category":"launch","label":"Chat waitlist","description":"Gate chat behind the waitlist","value":true,"source":"default"}]}}`),
})
zip.Describe("GET /v1/admin/me",zip.Doc{
Description:"me answers with the validated operator identity — who the console is signed in as,\nwhich tier they are, and how wide their tenant window is. The fields come from the\nsanitized identity headers the gate just read, so they are authoritative and never\nclient-forgeable; nothing is looked up.",
Description:"Answers with the validated operator identity — who the console is signed in as,\nwhich tier they are, and how wide their tenant window is. The fields come from the\nsanitized identity headers the gate just read, so they are authoritative and never\nclient-forgeable; nothing is looked up.",
Fields:map[string]string{
"adminMe.isWhiteLabel":"IsWhiteLabel marks the admitted NON-super tier: an admin of an enabled\nwhite-label tenant org. Mutually exclusive with IsSuperAdmin (the gate lets\nexactly one tier through). The operator SPA reads it to render the SUBTREE\ncockpit — the fleet god-view nav (finance/revenue/metrics/o11y/providers) is\nhidden — while a super sees the whole fleet.",
"adminMe.scopeOrgs":"ScopeOrgs is the caller's visible tenant window: empty for a SuperAdmin (means\nALL orgs), or the WL tenant's own subtree (today the singleton {org}). The SPA\nthreads it through the faceting/drill-down layer so a WL tenant can never widen\na filter past their subtree.",
@@ -92,18 +97,19 @@ func init() {
zip.Describe("GET /v1/admin/money",zip.Doc{
Description:"moneyBoardHandler answers GET /v1/admin/money.",
Description:"o11y is the fleet-wide observability board: LLM usage (requests, tokens, cost,\nerrors, top orgs, top models), trace RED metrics (count, p50/p95/p99 latency in ms,\nerror rate, top services), fleet log volume, and the O11yAI generation rollup — all\naggregated across EVERY tenant, with no org filter applied.\n\nEvery signal degrades INDEPENDENTLY. A table that is absent or errors contributes its\nzero value and the read still succeeds, so the board renders exactly what the\nwarehouse holds rather than failing whole because one of four sources is missing.\nSame when the warehouse is not connected at all: the zero board, never a fabricated\nfleet.",
Description:"Is the fleet-wide observability board: LLM usage (requests, tokens, cost,\nerrors, top orgs, top models), trace RED metrics (count, p50/p95/p99 latency in ms,\nerror rate, top services), fleet log volume, and the O11yAI generation rollup — all\naggregated across EVERY tenant, with no org filter applied.\n\nEvery signal degrades INDEPENDENTLY. A table that is absent or errors contributes its\nzero value and the read still succeeds, so the board renders exactly what the\nwarehouse holds rather than failing whole because one of four sources is missing.\nSame when the warehouse is not connected at all: the zero board, never a fabricated\nfleet.",
Fields:map[string]string{
"o11ySvcStat.errorRate":"percent (0..100)",
"o11yTotals.logVolume":"Logs (distributed_logs_v2), fleet volume over the window.",
"o11yTotals.logVolume":"Logs (event.log), fleet volume over the window.",
"o11yTotals.requests":"LLM usage (hanzo.cloud_usage), all orgs.",
"o11yTotals.traceCount":"Traces (o11y_index_v3), all services.",
"o11yTotals.traceCount":"Traces (event.span), all services.",
"o11yTotals.traceErrorRate":"percent (0..100)",
"rangeIn.range":"Range is the lower time bound: 24h, 7d or 30d. Anything else reads as the\nboard's own default.",
Description:"orgs lists the tenant directory one row per org, sorted by slug: member count and the\norg's month-to-date spend and credit balance, read live from IAM and commerce.\n\nThe rows are the caller's tenant window, not the fleet: a SuperAdmin gets every org, a\nwhite-label admin only their own subtree. A per-org read that fails degrades THAT row\nto an honest zero — this panel carries no sources[] channel to report freshness on, so\nthe alternative would be a fleet total that silently reads healthy.",
Description:"Lists the tenant directory one row per org, sorted by slug: member count and the\norg's month-to-date spend and credit balance, read live from IAM and commerce.\n\nThe rows are the caller's tenant window, not the fleet: a SuperAdmin gets every org, a\nwhite-label admin only their own subtree. A per-org read that fails degrades THAT row\nto an honest zero — this panel carries no sources[] channel to report freshness on, so\nthe alternative would be a fleet total that silently reads healthy.",
Description:"overview is the Platform Overview tiles: how many orgs and users are in the caller's\ntenant window, the fleet workload counts, and month-to-date spend and credits.\n\nIt ALWAYS answers 200 — a tile board that fails as a whole because one upstream is\ndown is useless. Instead every upstream reports itself in sources[]: ok, degraded, or\nnot-configured. A commerce read that failed for ANY org marks that source degraded,\nbecause the spend/credits totals are then an undercount and must not read healthy.\n\ntokens30d is 0 for the same reason /usage has no series: there is no fleet token\ncounter to read yet.",
Description:"Is the Platform Overview tiles: how many orgs and users are in the caller's\ntenant window, the fleet workload counts, and month-to-date spend and credits.\n\nIt ALWAYS answers 200 — a tile board that fails as a whole because one upstream is\ndown is useless. Instead every upstream reports itself in sources[]: ok, degraded, or\nnot-configured. A commerce read that failed for ANY org marks that source degraded,\nbecause the spend/credits totals are then an undercount and must not read healthy.\n\ntokens30d is 0 for the same reason /usage has no series: there is no fleet token\ncounter to read yet.",
Description:"products lists the fleet workload registry: every operator App CR across the platform\nnamespaces with its declared vs running image tag, reconciled health/phase and drift\nverdict. Optionally narrowed by kind, tier or env, each an exact match.\n\nThe rows are the SAME observation /v1/platform/fleet renders — read through the in-process\nplatform seam, not a second k8s client — so the two boards can never disagree about what\nthe fleet is. A PaaS plane that is not co-resident yields an honestly empty registry,\nnever a fabricated row.",
Description:"Lists the fleet workload registry: every operator App CR across the platform\nnamespaces with its declared vs running image tag, reconciled health/phase and drift\nverdict. Optionally narrowed by kind, tier or env, each an exact match.\n\nThe rows are the SAME observation /v1/platform/fleet renders — read through the in-process\nplatform seam, not a second k8s client — so the two boards can never disagree about what\nthe fleet is. A PaaS plane that is not co-resident yields an honestly empty registry,\nnever a fabricated row.",
Fields:map[string]string{
"productRow.cluster":"hanzo-k8s",
"productRow.declaredTag":"spec.image.tag on the App CR (declared truth)",
@@ -140,38 +146,29 @@ func init() {
"productsIn.tier":"Tier matches the derived infra grouping (cloud|data|edge|daemon|paas|app).",
Description:"getPromo reads the current platform plan promo — the singleton discount offer, e.g.\nthe 50%-off launch promo. Commerce stores it in the reserved platform namespace, so\nthe org sent with the read is the admin org and the service token is what passes\ncommerce's own platform-admin gate.",
Description:"Reads the current platform plan promo — the singleton discount offer, e.g.\nthe 50%-off launch promo. Commerce stores it in the reserved platform namespace, so\nthe org sent with the read is the admin org and the service token is what passes\ncommerce's own platform-admin gate.",
Description:"services reads the launch board: every hosted service in the registry with its LIVE\nwaitlist mode, evaluated through the flag engine. This is the \"remove the waitlist one\nservice at a time\" view.",
Description:"Reads the launch board: every hosted service in the registry with its LIVE\nwaitlist mode, evaluated through the flag engine. This is the \"remove the waitlist one\nservice at a time\" view.",
Fields:map[string]string{
"serviceList.services":"Services is every registered service with its live waitlist mode.",
Description:"listSpendCaps reads one org's usage caps: its spend alerts plus the derived period\nspend, over/warn state and reset time.\n\nThese are the SAME rows the customer edits in their own console — a platform override\nand a customer budget are one model, not two.",
Fields:map[string]string{
"capIn.id":"ID is the cap to edit or remove, from the path. Unused by the list and create ops.",
"capIn.org":"Org is the tenant to act on. Required for a SuperAdmin — they must name their\ntarget; ignored for a white-label admin, who always acts on their own org.",
Description:"subsystems answers GET /v1/admin/subsystems. ?range=24h|7d|30d bounds the telemetry\nwindow (default 30d) — the same enum, and the same helpers, as the o11y board.",
Fields:map[string]string{
@@ -182,7 +179,7 @@ func init() {
},
})
zip.Describe("GET /v1/admin/usage",zip.Doc{
Description:"usage returns the month-to-date money totals: one org's when org names one, else the\nfleet sum across every org a SuperAdmin can see.\n\nseries and byProduct are ALWAYS empty. A daily trend and a per-product split are not\nderivable from the commerce billing API — they live in insights/datastore — so this\nanswers with the honest empty arrays rather than fabricating a shape the console would\nthen chart. Same reason tokens and requests are 0: there is no fleet counter to read.",
Description:"Returns the month-to-date money totals: one org's when org names one, else the\nfleet sum across every org a SuperAdmin can see.\n\nseries and byProduct are ALWAYS empty. A daily trend and a per-product split are not\nderivable from the commerce billing API — they live in insights/datastore — so this\nanswers with the honest empty arrays rather than fabricating a shape the console would\nthen chart. Same reason tokens and requests are 0: there is no fleet counter to read.",
Fields:map[string]string{
"usageIn.org":"Org reads ONE tenant's month-to-date total instead of the fleet sum. Honoured\nfor a SuperAdmin only — a white-label admin always reads their own org.",
Description:"users lists the user directory across the caller's tenant window, one page at a time.\ndata2 is IAM's REAL total, so the console can page through it.\n\nA SuperAdmin may aim the read at one tenant with org; a white-label admin cannot — for\nthem the owner is hard-pinned to their own org and org is ignored, which is what keeps\nthe directory from becoming a cross-tenant read.",
Description:"Lists the user directory across the caller's tenant window, one page at a time.\ntotal is IAM's REAL total, so the console can page through it.\n\nA SuperAdmin may aim the read at one tenant with org; a white-label admin cannot — for\nthem the owner is hard-pinned to their own org and org is ignored, which is what keeps\nthe directory from becoming a cross-tenant read.",
Fields:map[string]string{
"usersIn.org":"Org narrows the directory to ONE tenant. Honoured for a SuperAdmin only — a\nwhite-label admin is pinned to their own org and this is ignored.",
"usersIn.p":"Page is the 1-based page number. Defaults to \"1\"; IAM returns zero rows AND a\nzero total when it is unset, so this layer never leaves it empty.",
@@ -198,10 +195,14 @@ func init() {
"usersIn.q":"Query is a free-text filter, matched by IAM as a \"contains\" over the user name.",
Description:"Returns the realtime block-storage board: the DigitalOcean volume fleet\n(count, capacity, monthly list cost, per-volume region and attachment) plus the\nanalytics datastore's OWN fill, read from its system.disks.\n\nA volume's usedGiB and pct are null, always: DO exposes capacity and attachment but no\nfill, so the console renders \"—\" rather than a number nobody measured. The datastore\ncard is the one real fill here, and it is the number to scale on.\n\nThe two sources degrade independently — a DO outage still returns the datastore fill,\nand a disconnected datastore still returns the DO fleet.",
Description:"waitlist reads one waitlist's leaderboard from the Hanzo waitlist engine — position,\npoints and referral standing per entry — proxied server-authed with the engine secret,\nnever a client credential.\n\nThe engine's payload is forwarded VERBATIM as data; the console normalizes it. When\nthe engine is not configured on this deployment the read still succeeds, with an empty\nobject and a msg saying so, so the panel shows an honest not-wired state instead of an\nerror the operator would chase.",
Description:"Reads one waitlist's leaderboard from the Hanzo waitlist engine — position,\npoints and referral standing per entry — proxied server-authed with the engine secret,\nnever a client credential.\n\nThe engine's payload is forwarded VERBATIM as data; the console normalizes it. When\nthe engine is not configured on this deployment the read still succeeds, with an empty\nobject and a msg saying so, so the panel shows an honest not-wired state instead of an\nerror the operator would chase.",
Fields:map[string]string{
"waitlistIn.page":"Page is the 1-based page number.",
"waitlistIn.pageSize":"PageSize is entries per page.",
Description:"updateSpendCap edits one cap by id — raise or lower the ceiling, flip enforcement. The\nbody is commerce's spend-alert patch contract, forwarded byte-for-byte.",
zip.Describe("PATCH /v1/admin/caps/:id",zip.Doc{
Description:"Edits one cap by id — raise or lower the ceiling, flip enforcement. The\nbody is commerce's spend-alert patch contract, forwarded byte-for-byte.",
Fields:map[string]string{
"capIn.id":"ID is the cap to edit or remove, from the path. Unused by the list and create ops.",
"capIn.org":"Org is the tenant to act on. Required for a SuperAdmin — they must name their\ntarget; ignored for a white-label admin, who always acts on their own org.",
Description:"createCreditGrant mints credit for one org. It is the ONE admin mint surface, and it\ndoes NOT mint in-process: it forwards the request to commerce's already-mint-gated\nPOST /v1/billing/credit-grants, authenticated by the service token and scoped to the\ntarget org, then writes one tamper-evident compliance record. Commerce stays the sole\ncredit ledger; this is a thin, audited relay so there is exactly one place credit is\ncreated.\n\nThe body is commerce's OWN CreateCreditGrant contract, forwarded whole — every field\nit carries reaches commerce. The only two this layer reads are the target org (`org`,\nor `user` as the org-pool alias), which selects the namespace commerce's EdgeAuth\ntrusts, and `idempotencyKey`, which makes a double-clicked grant credit once.\n\nA FAILED grant is audited too, with the request body attached: an attempted mint is\nexactly as interesting to a compliance auditor as a successful one.",
Description:"Sets a usage cap on one org — a platform override of a customer budget,\nwritten to the customer's own spend-alert rows. The body is commerce's spend-alert\ncontract, forwarded byte-for-byte.",
Fields:map[string]string{
"capIn.id":"ID is the cap to edit or remove, from the path. Unused by the list and create ops.",
"capIn.org":"Org is the tenant to act on. Required for a SuperAdmin — they must name their\ntarget; ignored for a white-label admin, who always acts on their own org.",
Description:"upsertService onboards a hosted service, or edits one, so a new host comes under the\nlaunch gate WITHOUT a redeploy. Re-registering an existing service PRESERVES its live\nswitch — editing the hosts of a service that is already open must not silently close\nit again.",
Description:"Onboards a hosted service, or edits one, so a new host comes under the\nlaunch gate WITHOUT a redeploy. Re-registering an existing service PRESERVES its live\nswitch — editing the hosts of a service that is already open must not silently close\nit again.",
Fields:map[string]string{
"serviceOne.service":"Service is the row as it stands after the write, live mode included.",
Description:"setServiceMode flips ONE service's waitlist switch — the launch lever. Hot: it takes\neffect on this pod immediately and on peers within one evaluation TTL, with no\nredeploy. An unknown service is a 404, not a silent create; onboarding goes through\nupsertService.",
Description:"Flips ONE service's waitlist switch — the launch lever. Hot: it takes\neffect on this pod immediately and on peers within one evaluation TTL, with no\nredeploy. An unknown service is a 404, not a silent create; onboarding goes through\nupsertService.",
Fields:map[string]string{
"serviceModeIn.service":"Service is the slug to flip, taken from the path.",
"serviceModeIn.waitlistMode":"WaitlistMode is the new mode: true gates the service behind the waitlist, false\nopens it. This is the launch lever.",
Description:"createSpendCap sets a usage cap on one org — a platform override of a customer budget,\nwritten to the customer's own spend-alert rows. The body is commerce's spend-alert\ncontract, forwarded byte-for-byte.",
Fields:map[string]string{
"capIn.id":"ID is the cap to edit or remove, from the path. Unused by the list and create ops.",
"capIn.org":"Org is the tenant to act on. Required for a SuperAdmin — they must name their\ntarget; ignored for a white-label admin, who always acts on their own org.",
Description:"syncNow answers the operator's \"Sync now\" button. There is nothing to kick: admin\naggregates LIVE on every read, so the button is just a re-read. It acknowledges\nhonestly with started:true rather than pretending a batch job was queued.",
Description:"Answers the operator's \"Sync now\" button. There is nothing to kick: admin\naggregates LIVE on every read, so the button is just a re-read. It acknowledges\nhonestly with started:true rather than pretending a batch job was queued.",
Description:"waitlistBoost grants a user waitlist points, moving them up toward the access cutoff.\nThis is the access lever: the cutoff itself does not move, the person does.\n\nIt funnels through the engine's verified grant seam (POST /v1/waitlist/award with\nsource=\"grant\" — the ONE path that honours an explicit points amount) and writes a\ntamper-evident audit row either way, so a FAILED grant is recorded too. The reason\nfield goes only to that row.",
Description:"Grants a user waitlist points, moving them up toward the access cutoff.\nThis is the access lever: the cutoff itself does not move, the person does.\n\nIt funnels through the engine's verified grant seam (POST /v1/waitlist/award with\nsource=\"grant\" — the ONE path that honours an explicit points amount) and writes a\ntamper-evident audit row either way, so a FAILED grant is recorded too. The reason\nfield goes only to that row.",
Fields:map[string]string{
"waitlistBoostRequest.email":"Email identifies the entry to boost. Either this or RefCode is required.",
"waitlistBoostRequest.points":"Points is how many points to award. Must be positive — this seam exists to move\nsomeone UP toward the cutoff.",
Description:"setFlag stores or overwrites ONE platform switch's definition and answers with the\nwhole board as it now stands. The flip is hot: this pod applies it immediately and\npeers converge within one evaluation TTL (15s by default), with no redeploy.\n\nThe body reaches the flag engine BYTE-FOR-BYTE — it is the engine's definition\nformat, not this layer's, so a field the engine understands and admin does not must\nstill arrive intact. setFlagIn names the two fields that matter for documentation; it\nis not a filter.\n\nThe write is recorded in the store's activity log against the caller's email.",
Description:"Stores or overwrites ONE platform switch's definition and answers with the\nwhole board as it now stands. The flip is hot: this pod applies it immediately and\npeers converge within one evaluation TTL (15s by default), with no redeploy.\n\nThe body reaches the flag engine BYTE-FOR-BYTE — it is the engine's definition\nformat, not this layer's, so a field the engine understands and admin does not must\nstill arrive intact. setFlagIn names the two fields that matter for documentation; it\nis not a filter.\n\nThe write is recorded in the store's activity log against the caller's email.",
Fields:map[string]string{
"setFlagIn.active":"Active is the switch itself: true enables the flag for every evaluation.",
"setFlagIn.filters":"Filters is the optional rollout/payload block of a VALUED switch, e.g.\n{\"groups\":[{\"properties\":[],\"rollout_percentage\":100}],\"payloads\":{\"true\":250}}.",
@@ -278,7 +274,7 @@ func init() {
Response:json.RawMessage(`{"status":"ok","msg":"","data":{"switches":[{"key":"waitlist.chat","category":"launch","label":"Chat waitlist","description":"Gate chat behind the waitlist","value":true,"source":"stored"}]}}`),
})
zip.Describe("PUT /v1/admin/promos",zip.Doc{
Description:"putPromo upserts the platform plan promo — the ONE place the offer is configured.\n\nThe body is commerce's own promo contract and is forwarded BYTE-FOR-BYTE, so no field\ncommerce accepts is dropped in transit. promoIn names its documented fields.",
Description:"Upserts the platform plan promo — the ONE place the offer is configured.\n\nThe body is commerce's own promo contract and is forwarded BYTE-FOR-BYTE, so no field\ncommerce accepts is dropped in transit. promoIn names its documented fields.",
Fields:map[string]string{
"promoIn.active":"Active is the master switch: false parks the offer without deleting it.",
"promoIn.end":"End is when the offer closes (RFC3339).",
@@ -287,6 +283,6 @@ func init() {
"promoIn.start":"Start is when the offer opens (RFC3339).",
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
antje
aandhanzo-dev
zandhanzo-dev