Restores the payments-catalog Stripe entry (a merchant connecting THEIR
Stripe account, alongside PayPal/Square/Shopify) and the Stripe Identity
option in the IDV adapter. Neither is Hanzo transacting: they are things
a customer connects, and removing them took a capability from merchants
to make a point about our own rail.
Our own rail is unchanged and unchangeable by these: the platform tenant
holds no Stripe credential (env and KMS scope dropped in universe
11c93be8), so credential-driven selection can never route a Hanzo charge
to Stripe. Commerce pin picks up the same correction plus wallet-based
affiliate/partner payouts.
apps/tools kept main's newer prose — another lane had already moved that
example off the vendor name, and theirs says more.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
analytics owns the platform event plane. webhooks consumes it.
Two subsystems each declared a JetStream stream over event.>: analytics'
EVENT (bus.go, envelope key `org`) and webhooks' EVENTS (bridge.go, envelope
key `organization_id`). JetStream refuses the second — "subjects overlap with
an existing stream", err_code=10065 — so whichever lost the race never bound:
either /v1/event ingest 503s on every publish, or the webhooks dispatcher
reconnect-loops and delivers NOTHING, commerce included. And an
analytics-published event resolved org "" through webhooks' orgOf, so it was
delivered to nobody even when the stream did bind.
What died: bridge.go, whole. The EVENTS constant, its subject binding, its
organization_id envelope, subjectFor, publishEvents, and the analytics.AddSink
registration in Mount. A consumer does not publish, and does not name a plane
someone else owns.
Where it went: apps/analytics. It exports the plane's identity (EventStream,
EventSubjects, EventOrgKey), its ONE stream constructor (EnsureEventStream,
which alone knows the retention), and the publish (PublishEvents). forward.go
puts every accepted batch on the plane directly — always, detached, fail-soft
— rather than routing it through a sink another app registers. The subject
grammar (event.<name>) and the subscriber payload are unchanged; only the
tenant key moves, to the `org` every consumer on this plane already reads.
How webhooks consumes: the existing durable-consumer machinery, unchanged.
streamSource now carries the two things a plane's OWNER decides — the tenant
key on its envelope, and its constructor — so consume() calls
analytics.EnsureEventStream rather than a second copy of the config, and
orgOf reads the key the stream declares instead of guessing. commerce.> keeps
organization_id and its own row; nothing about that path moved.
One bus knob: apps/pubsub now exports URL() (CLOUD_PUBSUB_URL, else the
loopback of the server this binary just bound, sharing CLOUD_PUBSUB_PORT with
it), documented in the package doc. analytics, webhooks, kafka and catalogsync
all read it. That retires CLOUD_EVENT_NATS_URL, CLOUD_WEBHOOKS_NATS_URL,
CLOUD_COMMERCE_NATS_URL and CLOUD_KAFKA_PUBSUB_URL — four names for one
loopback NATS, two of which gated their subsystem OFF when unset. No manifest
set either one, so webhook delivery and the reverse storefront edge were
inert in production against a bus that always serves and fails boot closed.
catalogsync now retries instead of degrading to inert, since it no longer has
an operator opting it in.
Tests, red on origin/main and green here:
- TestOneStreamBindsEventSubjects — exactly one stream binds event.>
- TestAnalyticsEventReachesWebhook — analytics publish -> bus -> signed POST,
org resolved from the envelope
- TestCommerceDispatchUnaffected — commerce.> still resolves, matches, queues
- analytics/publish_test.go — subject grammar, subject-space containment,
and one tenant key shared by both envelopes on the plane
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
8c28b24c rewrote the examples in this package's doc comments and stopped there.
Those comments are not comments — zipdoc lifts them into zipdoc_gen.go, which is
the only path they take into the OpenAPI document, the MCP tool descriptions and
every generated SDK. So the source said one thing and the whole published surface
said another, and the drift gate would have found it as a fleet-wide red on
someone else's next push.
Regenerated from source: zipdoc_gen.go, plugin/tools/{openapi,mcp}.json, and the
woven openapi.yaml.
One of the rewritten lines had also gone stale against the code under it: the
handle example still read acme_… after brand started naming the whole domain. It
now reads acme-com_… and says why — acme-com and acme-sh are two publishers, and
that prefix is read on the screen where a credential is pasted.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The document has always known a product's NAME mechanically — the first path
segment after /v1/ — and never what the product IS. A caller reading the tag
list, an agent reading the MCP door, a CLI printing `hanzo <product> --help`
got 144 bare nouns.
There is exactly one place that sentence is already written and already
reviewed: the package doc of the package that implements the app. So this reads
it rather than asking anyone to write it twice.
openapi/synopsis.go Synopsis(plugin/<app>) -> the owning package's synopsis.
describe.go stamps it into that app's own subset as info.description.
openapi/weave.go lifts the tag prose off the subsets it already reads.
ONE computation, at the one moment an app describes itself. The weave does not
look the mapping up a second time in a second process — it reads the value the
app that knows it already wrote down, which is why Weave stays a pure function
of its parts.
The owner comes from the app's own composition root: plugin/<app>/main.go
imports exactly the package it mounts. Nothing else could be the source — four
apps are not named after their package (audit->auditlog, evals->eval,
plugins->plugin, zero-trust->zt) and one package backs two apps (account,
account-bridge), so a name-derived guess is right 107 times and silently wrong
5. An app whose subsystem is another MODULE imports no package here and gets
nothing, which is the honest answer.
And the comment taken is the one that OPENS "Package …", not go/doc's
first-file-in-filename-order fallback. Packages that open their
alphabetically-first file with a note about that FILE and state the real package
doc in <name>.go would otherwise publish "actions.go — the two GitOps write
actions" as the deploy product's description. A misfiled sentence reads exactly
like a real one; an absent one does not.
109 of 112 apps have a package doc; 85 of the 144 product tags gain a
description. The three without are metrics, authz and licensing, whose subsystem
is another module — there is no package here to read. The tag NAME is never
conditional on a description: the list stays a function of the document's
operations, so nothing enumerating products loses a product because nobody wrote
a sentence. The fleet identity remains the fallback for a subset whose package
has no doc, and the weave treats a part carrying it as having said nothing.
THE LIFTED PROSE LOSES THE HANDLER'S OWN NAME, which is the other half of the
same problem. A Go doc comment must open with the identifier it documents, and
that identifier is Go's, not the document's: "GetSQL returns one database"
reached the OpenAPI description, its summary, the MCP tool description an agent
reads, and the CLI help line — naming a function no caller can see. zip drops an
exact leading match of the handler's own name from v1.18.13 (main is on v1.18.14,
whose lift is byte-identical), and nothing had regenerated against it: 35
packages carried prose the pinned zip can no longer produce. They regenerate
here. Three test assertions quoted the leaked identifier and now quote the
projection.
Every generated artifact is regenerated FROM SOURCE (make -f mk/fleet.mk
surface-check, green: 1017 paths). Nothing this commit does moves the wire: of
openapi.yaml's 16,439 non-prose leaf facts, 0 changed. The 4 lost and 94 gained
are all one thing — surface main already decided and never republished:
/v1/insights/e removed and /v1/event given its declared body (6fc2d88c), the six
project-scoped git smart-HTTP paths (811ff080), and the sessions' `terminal`
property (afdda829). The three bare-root git paths reach no app, so they join
router_test.go's unreachable ledger, recorded on the first regeneration that
published them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
manifest.Apps names "/v1/billing/payment-methods" on the BILLING row and
withholds it from commerce's. billing serves both methods there: its GET is the
same-origin proxy to commerce's /v1/billing/portal/payment-methods, and the host
claims a prefix for every method at once, so the POST has to sit on the same
router as the read or it misses on METHOD — which is exactly what killed the
console's save-card call and, with it, auto-recharge (a31a282b).
So commerce's registration was unreachable in the fleet: the request reaches the
billing process and never this one. router_test.go's ledger already recorded it
("commerce /v1/billing/payment-methods -> billing", the last entry of its
ADDRESS-ANOTHER-APP-OWNS class) and says the list may only shrink. It shrinks.
It was also a SECOND claim on one address, which openapi.Weave refuses rather
than pick a winner between — so the fleet document could not be woven AT ALL
once billing's subset was regenerated with its POST. That is why the drift gate
has been red: not a stale artifact, an unroutable fleet.
Nothing moves in the published document. The operation is identical from either
app — same operationId, same tag, no declared body — so only the ownership
changes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`selectImports` was inserted between GithubImport's doc comment and the method
it documents, and the two comment blocks are contiguous `//` lines — so Go read
them as ONE comment group belonging to `selectImports`, and `o.githubImport` was
left with none.
The consequence is not stylistic. zipdoc lifts a typed op's prose from its
handler's doc comment, so POST /v1/integrations/github/repos/import regenerates
with NO description, NO request example and NO 202 response example: the MCP
tool an agent reads to decide whether to call it serves the empty string. The
committed zipdoc_gen.go still carried the old prose, so nothing said this out
loud — it was a fossil the current source can no longer produce, and the first
regeneration after it publishes a documented endpoint as an undocumented one.
The comment moves back onto the function it describes. No behaviour, no wire.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`go build ./...` and `go vet ./...` have both failed on main since 01b38ef6 ("wip:
preserve in-flight work"), which added apps/deploy/chart.go — a Helm chart
renderer importing helm.sh/helm/v3 — without adding the module. Twelve commits
have landed on top of a tree that does not compile, so the CI gate (hanzo.yml
go-vet, go-unit) has been red for every one of them, and the drift gate cannot
run at all: it builds one binary per app, and deploy is an app.
The repair is the repo's own `make tidy`, and the version is MVS's, not a
choice: helm.sh/helm/v3 v3.21.3. It raises the k8s libraries 0.36.1 -> 0.36.2
and a handful of otel exporters with them, because that is what helm's own
requirements make the minimum. Nothing is removed, and no import moves — the
source is unchanged; only the record of what it already needs.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The per-server window I added to stop a dispatch fanning out over the network kept
every entry forever. Each one holds a whole tool set — a fifty-tool server with
real schemas is tens of kilobytes — so a fleet that deregisters and re-enables
servers accumulates them with nothing ever dropping one.
An hour without being asked about is a server whose org deregistered it, renamed
it, or is not coming back. Those go. The pass runs off the listing path and at most
once an hour, rather than from a goroutine: a provider with no traffic has nothing
to forget, and a timer outliving the process's interest in the data is one more
thing to shut down correctly.
The key carries the org, and now a test says so: two tenants holding the same
server id at the same URL each ask for themselves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip refuses a second open plugin at Load, and a host discovers that by failing to
boot — after the manifest already said it. The manifest is where the decision
lives, so it is where the check belongs: exactly one app is open, it is the tool
plane, and it is not co-resident (a co-resident app is never Load'ed, so it could
never be asked and the flag would be a lie).
The second test walks the resolution ladder. Open is a property of the APP and not
of where its binary came from, and it is stamped once in Plugin() for exactly that
reason — so a remotely-mounted tools app must keep it too.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Resolve reads and Write writes, and a request can arrive between them — which is
not exotic, it is a double-clicked Enable button. Both passes then found no
existing row, both resolved the same id, and the second INSERT hit the unique
constraint and answered 500. Nothing was corrupted and nothing was lost; the
caller was simply told their button was broken.
The row the first one created IS the row the second was about to create, so the
second becomes the revise it would have been had it arrived a moment later. Both
constraints route here — the (org, id) key and the (org, listing) index — so the
convergence does not depend on which one fires.
Eight concurrent enables, one server.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An adversarial pass over the catalog found three things worth the word BLOCKING and
a row of smaller ones. Every fix below has a test that fails without it.
A PROCESS KILLER, in zip and mine: the per-caller dedup wrote the caller's names
into the FLEET's shared name set. One tenant's tool name became "already claimed"
for every tenant after it — the second org silently lost a capability and could
infer from the absence that someone else had it — and two lists in flight were two
writes to one map, a runtime fatal no recover() catches, on the method an MCP
client calls constantly. Fixed in zip v1.18.15 (the fleet set is read-only, the
request's dedup is its own); pinned here.
A LOOKALIKE, and it aimed at the one screen where an org pastes a credential.
`brand` took the memorable label out of a namespace, so com.stripe and sh.stripe
both became "stripe" — and anyone can hold sh.stripe by owning stripe.sh. Their
tools would have rendered as stripe_create_payment_link on the enable form. The
handle now names the DOMAIN: stripe-com, stripe-sh, alice-github-io. Longer, and
true. `isOfficial` had the matching flaw: it accepted a match on the site or the
repository, so a listing could carry the badge while its ENDPOINT — the only field
enablement consumes — pointed elsewhere. It now judges the endpoint, and falls
back to site/repo only for a listing that has no endpoint and therefore cannot be
enabled. Live: official 4,839 → 4,747; the 92 that left are exactly that class.
AN ORDERING THAT NEEDED AN UNDO, which is the smell. Registration wrote the row
then sealed the credential, so a KMS failure had to be reversed — and the reversal
was wrong for a re-enable in one direction and absent in the other, leaving a row
asserting a credential nobody stored. That does not fail loudly: the listing
errors, the provider skips the server, and its tools vanish from the org's plane
in silence. Resolve → seal → write has no undo in it at all, because nothing is
written until the credential is held.
The rest:
- an unbounded GET /v1/tools/catalog over 19,323 rows is now paged (50, max 200)
with the whole match as `total`.
- upstream logo/repo/site URLs are dropped unless a browser may follow them.
`javascript:` in an icon src is what arrives, and all three are RENDERED.
- a name outside the registry's own grammar is skipped. "com.foo_bar/mcp" and
"com.foo/bar_mcp" derive ONE id, which is the primary key, and put preserves
curation — so the second would inherit the first's featured, vouched standing
and replace its endpoint.
- deregistering destroys the credential. types.KMSClient has no removal at all,
so this overwrites; the real fix is a fleet-wide seam change and is written up
in LLM.md rather than smuggled in here.
- no redirects on an MCP hop: Go strips only Authorization and Cookie, so an org
using X-Api-Key had it replayed to wherever the server pointed — and the
endpoint may now come from a third party's catalog entry.
- authHeader must be an HTTP field name and the secret has a size. Both failed
later, inside a background listing, where the symptom is a server whose tools
quietly never appear.
- one tools/list per server per minute, not per dispatch. Every dispatch resolves
through the registry, which listed every provider, which asked every one of the
org's servers over the network: thirty enabled listings meant thirty outbound
requests per tool call, each with a 20s timeout.
- the sync commits a page at a time instead of a row at a time — 19k fsyncs
inside a request someone is waiting on.
- a search term's % and _ are literal characters; a third party's cursor has a
length; Dispatch guards its store like List already did; and the door LOGS when
it cannot resolve a caller, which is otherwise indistinguishable from a tenant
with no tools.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Picks up the deletion: no Stripe provider, no processor.Stripe type, no
per-org Stripe registration or KMS hydration, no Stripe branch in
checkout. Square via commerce is the one rail, and the cloud pod no
longer mounts STRIPE_* (universe 11c93be8).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A connector implies we support it. Removed the customer-key Stripe entry
from the payments catalog (PayPal/Square/Shopify remain), with a note
against re-adding it, and its catalog-wiring test. The identity-
verification adapter drops Stripe Identity as an option — Persona and
Onfido are the vendors — and the doc examples that reached for Stripe as
the canonical third party (MCP publisher names, the webhook signature
header shape) now say what they mean without borrowing the brand.
The sk_/pk_ patterns in the security scanners stay: those are about
OTHER people's leaked credentials, which is a defensive concern that has
nothing to do with which rail we charge on.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every reference in the run-in-cloud section had already drifted by one from a doc
commit landing above it. A file:line that is wrong is worse than no reference —
it sends the next reader to the wrong function and quietly teaches them the note
is stale. The symbol is what does not move, so it is cited beside the number and
the numbers are pinned to a commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The pins named o11y_ai.observations. There is no o11y_ai DATABASE — checked
against system.tables — so the pin was pinning a fiction, and every AI number on
the fleet board read zero while 8,867 observations sat in `console`. A pin is
only worth having if it names a table that exists.
`console` is a surface name on a store and is wrong too; it moves to o11y.spans
with #102 and these pins move with it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Registering a server is two writes — the row, then the credential into KMS — so a
failed seal has to be undone, and the undo was written when there was only one
kind of write. Making a re-enable REVISE the existing row gave that undo a second
meaning nobody re-read it against: an org re-enabling a working server with a new
credential, against a KMS having a bad second, lost the server. The row was
deleted because the code that deleted it had only ever seen rows it created.
Create now reports whether the row is FRESH, and the rollback deletes only that.
For a revise, the row the org already had stands, still pointing at the credential
it was already using — which is the state it was in before the request, which is
what "undo" means.
The fresh case is unchanged and still deletes: a row claiming a credential KMS
does not hold is a server that would dispatch unauthenticated.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The first live sync read 19,321 servers. The page cap was 200 pages of 100 —
20,000 — chosen as "comfortably past the whole registry", and it was, by 3%. One
good quarter from now it would have started SILENTLY TRUNCATING the catalog: the
walk would stop, Sync would return nil, and the shelf would be short with nothing
anywhere saying so. That is the failure the cap was written to prevent, arrived at
quietly.
The cap was never the loop guard anyway. A cursor that does not end is a cursor
that REPEATS, and that is now caught directly and immediately — one page, not five
thousand — with what was already read kept, because a partial catalog is not a
reason to discard the part that was fine. The count becomes a backstop two orders
of magnitude out, and hitting it is an ERROR: a short catalog that says so beats a
short catalog that does not.
Evidence, against the real registry rather than a fixture: 19,321 listings in 33s
over 194 requests, 4,839 official, 9,216 with a streamable-http endpoint an org
could enable today; a second pass reports added=0 updated=0. That run is
TestLiveRegistry, skipped unless CLOUD_TOOLS_LIVE=1 — the shape being parsed is a
third party's, so it has to be checked against theirs, but a suite that needs the
internet is a suite that goes red when they have a bad afternoon.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
go/doc concatenates every comment glued to a package clause, in file order, and
that concatenation IS the OpenAPI tag description, the MCP door prose and the
CLI group help. 18 apps glued 52 surplus file notes to their package clause; in
11 of them a file note sorted first and WON, so the published product
description was going to read "GET /v1/usage/activity — the per-day
contribution series" (leaderboard), "browse.go — Hanzo Git's JSON read/browse
surface" (git), "O11Y LLM-OBSERVABILITY EVENT INGEST" (o11y), "grant.go — how
CI gets bytes into a site's S3 prefix" (projects), "client.go is the ONE HTTP
path" (graph, zt), "The node registry" (bot), "Pure core of the SBOM lens"
(sbom), "connectors.go is the per-USER connector plane" (integrations), "World
plan enforcement contract." (world) and "activeplan.go answers the subscription
paywall's ONE question" (commerce).
Every surplus note keeps its text and gets a blank line before the package
clause, which is all Go needs to stop reading it as package documentation.
apps/commerce had no package doc at all — three file comments concatenated —
so it gets the one true sentence it was missing.
All 131 app packages now render a synopsis that names the product.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An org could reach an outside MCP server only by knowing its URL and typing it
in. Meanwhile the public registry publishes thousands of them, and the fleet's
one agent door could not show a tenant a single one of its own tools — the door's
list is a build artifact, and a tenant's capabilities are rows.
THE SHELF. apps/tools now holds a canonical copy of registry.modelcontextprotocol.io:
name, vendor, description, repo, version, transports, packages, remotes. Canonical
and not a cache, because a storefront that goes blank when a third party does is
not a storefront, and because the copy carries what only we know. A row has two
halves that never mix — the upstream fields, replaced wholesale on every sync, and
hidden/featured/official/logo, which the sync statement does not NAME and therefore
cannot touch. Idempotent by construction: the id IS the publisher's reverse-DNS
name, so a second pass reports added=0 updated=0 over an unchanged registry.
OFFICIAL IS MECHANICAL, not editorial. A registry attests the PUBLISHER, not the
product: "com.stripe" is issued against proof of stripe.com, but "io.github.alice"
attests an account on a forge, and a re-hoster publishes hundreds of other
people's servers under its own perfectly-verified namespace. So: the namespace
must name a domain, and that domain must serve the listing's own endpoint, site or
repository. stripe.com serving mcp.stripe.com is official; a proxy on someone's
workers.dev is not. What the data cannot settle is what the admin override is for,
and setting it makes the answer final.
ENABLING IS REGISTERING. Picking a listing off the shelf writes the SAME record
typing a URL writes — one thing an org can have, one place it lives, one place the
credential is in KMS — with `source` derived from whether a listing is recorded. A
second enablement path would have been a second kind of server for everything
downstream to learn about. Enabling twice revises one row. The server id, which
PREFIXES every tool the server contributes, is minted from the vendor's brand, so
an agent reads stripe_charge and not m4f21c8_charge.
THE DOOR GETS ITS OTHER HALF. zip v1.18.14 adds the seam and cloud fills it:
tools.Door() is a zip.Source whose Tools is the registry's activated dispatchable
set for the caller and whose Call IS callTool — so activation, precedence, the
x402 gate, the metered unit and the audit record are the ones POST /v1/tools/call
already enforces, and the door adds no policy. The tools app is declared Open in
the manifest; the host asks it only on a tools/list that NAMES a caller, so an
anonymous list is still a memcpy that starts no child.
NOT BUILT, and said so rather than stubbed: running a stdio package (npx/uvx/docker)
in our cloud. The App CR carries no command/args (apps/platform/k8s.go:523), its
CRD is another repo, there is no bridge image, our client speaks one-shot JSON-RPC
and not streamable HTTP, and the SSRF guard exists to refuse exactly the in-cluster
address such a sandbox would have. apps/tools/LLM.md records each seam with its
file:line and the shape the route must take when they land — which is to produce
the same registration record an enable does, or the plane has grown a second kind
of server.
Schema names are a FLEET-WIDE claim: Listing is marketplace's and catalogList is
prompts', so these are MCPListing/MCPPackage/MCPRemote, the shape MCPServer
already set here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
pricing and usage published the WRONG doc entirely. A file comment glued to the
package clause IS package doc, and go/doc concatenates them in filename order —
so pricing led with admin.go's "Admin surface for the catalog enablement overlay
(SuperAdmin only)" and usage with entitlement.go's "Analytics entitlement
contract". Those become the OpenAPI tag description, the MCP door prose and the
CLI group help. Seven such comments (pricing admin/catalog/enablement, usage
entitlement/query, billing finance, wallets wallets) move below the clause, which
is the convention books/store.go and x402/store.go already follow, and each
package now leads with its own sentence.
pricing also claimed /v1/models, /v1/gpu and /v1/tools aliases. It binds none of
them; the manifest and its published subset agree. It omitted the enablement
registry it does own.
billing named five routes. It serves seven under /v1/billing plus the six
/v1/finance projections in finance.go, and its PASSTHROUGH paragraph said it
proxies commerce verbatim — balance and usage read the co-resident finance ledger
first and only fall back to that proxy.
treasury claimed the /v1/finance/* surface. It owns two of that prefix's eight
routes; billing owns the other six. Its ledger sat "above the per-org commerce
credit ledger" — commerce's credit route is injected with apps/finance, which
posts to the same apps/treasury/ledger engine. bind-anchor was missing from the
table.
finance called itself the ZAP-native money subsystem. It registers no ops and no
routes; it is the in-process FinanceClient.
wallets said three custody backends over four Kinds — safeclient.go is the fourth.
books said commerce transactions are the SOLE posting source. bank.go and scan.go
post through the same choke point.
entitlements described only its enablement store; GET /v1/entitlements is the
plan-entitlement projection and the package holds both authorities.
plan and rollingcap named clients/* packages that are apps/*, and rollingcap
named an apps.Wire() the host no longer has.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
captable led with a migration ("the PILOT of epic #96"), team with a Phase-1
READ PLANE and a hand-copied route table that had drifted past every write it
now serves, and product with a hostname (api.cloud.hanzo.ai) its own next
paragraph contradicts. Each now opens with the product: the cap table's
instruments, the workspace's planes, the two backends the four reads inventory.
team's route list is deleted rather than corrected — plugin/team/openapi.json is
the published surface, and a hand copy beside it is the second source that rots.
plugin claimed the second "what is a plugin here" reader was gone. It is not:
GET /v1/plugins lives in apps/tools and answers from cloud.Subsystems(), the
BOOT snapshot — so it cannot see the enable/disable/reload this package
performs. Say that, and say which one is true right now. Its manifest.Apps is
hand-authored, not generated; manifest/apps.go says so in its first line.
erp declares the ERPNext model and no binary imports it, so its init never runs
and POST /v1/framework/modules/erp/install answers "unknown module: erp".
Plus the clients/<pkg> paths in these docs, which name a directory the repo no
longer has: clients/{index,tools,content,settings,goja,plan,pricing,framework,
cms,o11y} -> apps/*.
Comments only. Verified: gofmt clean against the same baseline, the ten
packages build with -tags sqlite_fts5, and go generate -run zipdoc reproduces
every zipdoc_gen.go byte for byte.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
go/doc concatenates EVERY comment attached to a `package` clause, in file
order — so a file header on an alphabetically-earlier file becomes the
package doc. `go doc ./apps/visor` opened "board.go — GET /v1/fleet"; platform
opened with applylive.go, deploy with actions.go. Detach the 43 file headers
with a blank line so the one product paragraph is the whole package doc — the
text that becomes the OpenAPI tag, the MCP door prose and the CLI group help.
Four docs named the wrong thing or claimed something untrue:
deploy said "Package gitops" — it is Hanzo CD at /v1/deploy
goja said "Package gojahost"
do said DigitalOcean is Hanzo's EXCLUSIVE cloud venue; venue links
DigitalOcean, AWS, GCP and Azure. do is the HOUSE account
membership cited cloud.SetLiveSource and apps/ installing it; both are gone
(22f4fc64) and nothing calls K8s — say so, the outage it fixes is
not fixed
And platform pointed at a Goa design module that emits 15 ops against the 30
the router registers — a second source that has already drifted; say which one
is the contract. clients/<app> paths in these docs are now apps/<app>.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
webhooks carried TWO package comments — bridge.go's abutted its package
clause, so go/doc led with "bridge.go — the ingest→bus half…" and the real
product doc never surfaced. A blank line makes it a file comment again.
notify and pubsub opened on the migration ("folds…", "embeds…as an
in-process subsystem") rather than the product; kafka named a module that
does not exist (github.com/hanzoai/stream — it is github.com/hanzoai/kafka).
The package doc is the OpenAPI tag, the MCP door prose and the CLI group
help, so each now states what the product does.
Every clients/<app> path in these six packages is stale — the subsystems
moved to apps/ in f873d1a1 — and apps.Wire() no longer exists, so the three
mount-order notes point at manifest/apps.go, which is the source of truth.
gofmt fixes the pre-existing drift in webhooks/api.go and kafka/interop_test.go.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The package doc is the OpenAPI tag, the MCP door prose and the CLI group help,
so it has to say what the product does — not how it is mounted. Thirteen docs
across the growth domain led with "mounts the Hanzo Cloud /v1/X/* surface: a
native-Go, per-org … on Base/SQLite" plus fold history; each now opens with the
capability.
Four said something the code does not:
- campaign claimed it fans out to paid -> /v1/ads, organic -> /v1/publish and
email -> /v1/marketing. /v1/publish exists nowhere in the repo, and only the
paid executor is registered (plugin/campaign/seams.go). The doc now names
apps/social as the organic surface and states that organic and email have no
executor, so a campaign carrying them records "unavailable". channel.go's
"three registrations" and its copy of /v1/publish go with it.
- analytics called itself a "read API" and its surface "read-only". It owns
the ingest doors every Hanzo client posts to and the write core behind them;
the doc now states both halves and lists the four doors. It also still
counted "six ingest doors" — a69e7549 retired three name-aliases and left
two, and routes()'s "four of the six" was stale the same way.
- ads described a campaign store; provider.go launches, pauses and reads spend
on six ad networks, and the launch route was missing from the surface block.
- destinations, flags, leaderboard, marketing, social, crm, affiliates,
referrals and authors led with storage or fold history rather than what they
do.
Where a capability has two implementations the doc now says so at both ends
rather than in neither: marketing's calendar and apps/social are two stores for
one scheduled social post, marketing's campaign record is a third campaign
beside campaign and ads, /v1/admin/referrals/* is split across referrals and
affiliates, and /v1/usage/* is co-owned by leaderboard and usage.
clients/<app> paths inside these doc blocks are stale — the packages are
apps/<app>; fixed where the doc block carries them. gofmt fixes the pre-existing
import order in affiliates.go.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Eleven packages in the agents cluster. Four docs said something false: benchmark
and research cite an apps.go composition root deleted with the mega build,
agentskills cites a go:generate path that moved, and eval/exec/bots/runtime/
connectorruntime/coding/automations cite clients/<app> packages that are now
apps/<app>.
Two packages carried TWO package docs — automations (automations.go + types.go)
and runtime (runtime.go + ops.go). Go concatenates them in file order, so the
product description an OpenAPI tag reads was 'types.go ports the ActivePieces
shared contract' and 'ops.go mounts /v1/bot/*'. The second comment in each is now
a file comment, separated from the package clause.
exec's doc did not say it owns four top-level /v1 segments (/v1/exec, /v1/upload,
/v1/download, /v1/files) and therefore publishes under four product tags; eval's
led with the console fork it replaced instead of with the product; coding's led
with 'the keystone that turns @hanzo from a chatbot into an engineer' and did not
say it is a library with no route, no plugin and no manifest row.
eval was missing GET /v1/evals/metrics, prompts GET /v1/prompts/catalog, and
research its three /v1/research/artifacts ops.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The package doc is the OpenAPI tag, the MCP door prose and the CLI group
help, so a package with FOUR of them publishes whichever one sorts first,
and a package whose doc describes a route it does not serve publishes a
lie.
knowledge had four package docs attached (connectors.go, kb.go,
subsystem.go, sync.go); go/doc takes the first file alphabetically, so the
product was described as "connectors.go is the per-org app-connector
control plane" — one file's notes, standing in for the knowledge base.
kb.go is now the one package doc and says what the product is; it also
opened "Package kb" for a package named knowledge. The other three keep
their prose as file comments.
websearch had two (search.go, websearch.go) and the same rule picked
search.go's "Native Go meta-search — the SEARCH half of /v1/websearch",
which describes one half of one file. websearch.go's product doc now
stands alone.
search claimed to be "THE search entry point: ONE surface, POST /v1/search".
It is not mounted: no manifest row, no plugin/search binary, and /v1/search
is apps/provisioning's (list/create a provisioned search index). Its only
live caller is apps/team's fulltext RPC via ForOrg — and in the team binary
index.Mount never runs, so the lexical leg reports disabled on every query.
The doc now states that.
Comments only; no route, type or behaviour moves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
world named a mount order (142/150) that no longer exists — the host routes by
manifest.Apps sequence — and its surface block omitted GET /v1/world/limits, which
the code binds and the published subset documents.
content promised phase 2 would widen the cms module's status field onto the shared
lifecycle. No binary imports apps/cms, so its init never runs, the module is never
registered, and no org can install it; the marketing status IS the one lifecycle in
service.
templates called apps/guide's Template a notification template. It is a Guide
playbook prompt/snippet.
blueprint credited cloud.HealthOwner for its own health route. The plugin declares
OwnsHealth.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
storage's doc opened "Package s3" on a package named storage and called
itself Fiber-facing; s3admin claimed there is no second S3 client
construction in the binary while apps/provisioning builds its own from the
same S3_ADMIN_* variables; base opened with a migration note and counted
two lanes where three prefixes are served, the third from a different
store. graph, sync, framework and datastore opened vague, aspirational, or
ambiguous against apps/provisioning's own `datastore` kind.
The package doc is the OpenAPI tag, the MCP door prose and the CLI group
help, so each now states what the product is and what it answers.
Comments only — build output is byte-identical before and after.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A wire is a SHAPE, and a shape has never earned a path. /v1/insights/e existed
only because the PostHog wire spells fields differently; decodeIngest already
sniffs object-vs-array and bare-vs-envelope on one route, so sniffing one more
encoding is the mechanism that is already there, not a new one.
The wire does NOT go away — decodeEvent picks the decoder by sniffing keys, and
the ingress rewrite that fed the old door (insights-cloud-ingest-rewrite:
insights.hanzo.ai /e,/batch,/capture) now replacePaths onto /v1/event, so every
PostHog-wire caller keeps working. Six doors become five.
The sniff is on KEYS, not on 'did the first decoder return anything'. I wrote the
count-based fallback first and TestMount_HostCarve_IngestsForSiteOrg refuted it:
decodeIngest ACCEPTS a PostHog body as a bare canonical Event and returns ONE
event, which is then dropped whole downstream (canonicalType("") is "event",
not in publicKinds). The caller gets 200 and the event vanishes. A count of 1 is
not evidence the body was understood.
The wires are distinguishable exactly: canonical spells `distinctId` (camel) and
carries `type`; PostHog spells `distinct_id` (snake) and carries `api_key`.
Neither key appears in the other wire, so presence is proof, not a heuristic.
Batches are probed on elements because `batch` is shared.
Security properties re-proven on the one door, not assumed: key extraction from
body/query/x-api-key, presented-but-unresolvable => 403 fail-closed, keyless =>
the anonymous lane. Those tests now drive /v1/event and pass. Full package green.
$source says nothing real used the old door: 3584 rows via 'event', 15 via
'capture', 1 via 'posthog' — and that 1 is my own probe from this session.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
git.hanzo.ai was in CrashLoopBackOff on a corrupted queue LevelDB while work
continued on GitHub, so the two heads drifted: 2 commits reached the forge, 135
reached GitHub. Merged rather than force-pushed — a force-push would have silently
dropped the forge's two.
Nothing is lost, and that is checked rather than assumed. Both sides had
independently added apps/tools/skillstore.go (hence AA) with a BYTE-FOR-BYTE
identical exported API — OpenSkillStore, Close, Put, List, Delete, and the
orgSkillProvider Source/List/Dispatch trio. The forge's five registries.go
functions (putSkill, deleteSkill, listAuthoredSkills, listBySource, listPlugins)
all exist on main as typed toolOps methods instead of raw handlers. Diffing the
symbol sets both ways leaves nothing on the forge side that main does not already
serve, so "the org can add a skill without a redeploy" shipped twice and main
carries the better shape.
Conflicts resolved to main throughout: three source files where main is the
superset, and openapi.yaml + plugin/tools/openapi.json, which are GENERATED and
must be regenerated rather than hand-merged.
Why this mattered: a push to git.hanzo.ai is what fires GitPushEvent ->
isReleasePush -> launchRelease (build -> smoke -> tag). The forge IS the release
trigger, so while it was down nothing built — which is why no cloud image exists
past v1.801.327 despite 135 commits.
go build ./apps/tools/... = 0, gofmt clean.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
security had no package doc and bot's began with a filename; the package
doc is the OpenAPI tag, the MCP door prose, and the CLI group help, so
each now states what the product does. gofmt fixes the pre-existing
import order in security.go.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
o11yAIObs and aimO11yAIObs both named `o11y_ai.observations`. There is no
`o11y_ai` database — verified against system.tables 2026-07-31. Every AI number
on the fleet board has therefore always been zero, and the surrounding
`if err == nil` swallowed the error so it rendered as honest-empty rather than
as a failure. It was not empty: 8,867 observations (and 8,819 traces) sit in
`console`, which nothing reads.
Columns match the queries exactly — project_id, type, start_time, Nullable
end_time, provided_model_name, internal_model_id, total_cost — so the fix is the
name. Proven against the live store: 8,867 generations, $0.0017, 75.8ms avg
latency, and a real per-model leaderboard (qwen3-8b, deepseek-r1-0528,
qwen3-235b-a22b, glm-5) where the panel showed nothing.
The rows span 2026-02-11..2026-03-14, so a recent window still totals zero —
correctly, because no gen_ai observation has landed since. sinceTS already
filters on start_time, so the time selector tells the truth at both ends.
`console` is a SURFACE name on a store and is wrong too; HIP-0132 folds this
signal into o11y.spans (a gen_ai span IS the observation of record). Both consts
move there together under #102 — they are one fact stated twice, which is how
they drifted into pointing at nothing without either being noticed. Pointing at
the rows that exist is the step that does not require o11y to exist first.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The project sub-scope rode X-Project-Id alone, and a git client sends no
headers, so a repo outside the org's default scope had no remote a client could
reach: cloneURL emitted /v1/git/<org>/<name>.git for every repo, and
resolvePackRepo dropped the scope entirely for anonymous reads.
Smart-HTTP and SSH both take the scope as an optional middle segment —
/v1/git/:org/:project/:repo and git@host:org/project/repo.git — beside the
existing two-segment routes, which keep their exact meaning. cloneURL and sshURL
advertise whichever form matches the repo, so a caller is never told a URL that
does not work.
The path wins over the header when both are present, because the path is what a
client can express. An anonymous caller may use it: naming a project addresses a
repo rather than asserting a scope, and the repo's Public flag still decides the
read, whereas an unauthenticated X-Project-Id stays unvalidated input and is
ignored as before. The segment is checked against projectRE, since it becomes a
storage path segment.
This is what lets one Hanzo org hold repos from several GitHub owners:
hanzo/hanzo-apps/ai and hanzo/hanzo-docs/ai are distinct repos rather than two
upstreams fighting over hanzo/_/ai.git.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The subsets and the fleet spec are generated from each app's real router, so
adding subscribe/card and test-mode to commerce means regenerating both or the
drift gate fails: a route missing from openapi.yaml is a route no generated SDK
client can reach, and these two are the paid path's front door and its live
switch.
The weave also removes /v1/sessions and its tag. That is not collateral — it is
the gate working. afdda829c folded the sessions roster into apps/agents and
deleted apps/sessions and plugin/sessions/ outright, but left the fleet spec
naming three routes nothing serves. Documented and gone is the same defect as
served and undocumented, in the other direction.
organization.TestMode() is !o.Live, and that org record is the single authority
for both the Square environment and the ledger bucket. An org nobody has flipped
therefore transacts in SANDBOX — fail-closed by design, and the reason a
deployment holding production Square credentials can still hand a buyer a
sandbox card form.
The switch had no route in this binary. It lives on commerce's mint group,
which the co-resident embed never compiles, so the flip could not be performed
at all. Registered here on auto-recharge/run-all's chain, because it is the same
class: a money-MINT control gated to the service token or a platform global
admin, never the org-level Admin bit — an org admin must not be able to move
their own org between sandbox and production.
Routing it does not flip anything. hanzo stays test-mode until an owner calls it.
The doc asserted "Org routing header is X-Hanzo-Org" and cited
commerce/middleware/accesstoken.go GetHeader("X-Hanzo-Org") for it. That
citation is false: commerce v1.49.32 reads X-Org-Id (accesstoken.go:110,167)
and contains no X-Hanzo-Org read anywhere. The code was always right —
metering.go:77 sends X-Org-Id — so only the doc pointed the wrong way.
It pointed the wrong way at the one header whose failure is silent. Commerce's
selector falls back stashed-org -> X-Org-Id -> COMMERCE_SERVICE_ORG -> "hanzo"
and never refuses, so following the doc would not have errored; it would have
billed the house org for every tenant. The doc even stated that consequence
("Wrong header -> debits the default hanzo ns") while naming the wrong header.
X-Hanzo-Org is real but is a different header going the other way: cloud stamps
it on served /v1/cloudflare responses as the acting org (cloudflare.go:461) and
platform asserts it to prove no comingling (cloudflare-pages.ts:84). Both
directions are now written down so the names stop being interchangeable.
Every money endpoint the commerce plugin registers co-resident — the public
plan catalog, invoices, subscriptions, top-up, spend caps, payouts — was
published by an app the router never handed the request to. account-bridge
owns the /v1/billing REMAINDER, and a prefix is an exclusive subtree, so each
leaf the manifest did not name deeper landed on the bridge and answered
"sign in to view billing": to a signed-in buyer, and to the anonymous
pricing page that reads GET /v1/billing/plans. The registrations were correct
and unreachable; hanzo.ai/pricing could never leave its static fallback.
subscribe/card had no route anywhere. commerce publishes it on the api.Route()
'user' group the co-resident embed never compiles, and mount never registered
it — so the one endpoint that turns a visitor into a subscriber was not
addressable in this binary. It is registered here on topup/token's chain,
which it matches exactly: both are browser money-writes charging a single-use
Square nonce, both server-authoritative on price, PAN never touching us.
manifest/apps.go names each leaf deeper than the sibling that was swallowing
it, and the unreachable ledger shrinks by twelve. Proven by reverting the
prefix and watching the oracle name the exact defect.
- Route to commerce: plans, invoices, payment-config, payouts, spend-alerts,
subscribe/card, subscriptions, topup/token
- billingForwardable gains subscribe/card, so the split-deploy fallback
matches every sibling money-write
A repository name is unique within one GitHub owner, and a Hanzo org may hold
several installations — hanzoai/ai is the Go backend, hanzo-apps/ai is the
hanzo.ai site, hanzo-docs/ai is docs.hanzo.ai. Selection matched on the bare
name, so {"repos":["ai"]} imported whichever the listing reached first.
repos[] now takes "owner/name" as well as "name", and a bare name reaching more
than one granted repository is refused with all of them listed, so the caller
names the one it meant. Selection moves to selectImports, separate from
resolving the token and listing the installation, which is what makes it
testable.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
sentry.hanzo.ai answered "Access required — this account isn't authorized for
this, it's an admin-only surface" to a signed-in customer whose own 75 errors were
sitting in event.error. Errors, logs, traces and metrics are a tenant's own
telemetry, so membership of an org is the whole admission test. Gating the PRODUCT
on platform sudo meant no customer could ever use it, and that the only way to see
it was to become sudo — which then shows every org's errors instead of your own.
Sudo belongs one level in, on the cross-tenant view.
The first cut of this also added a scopeToTenant that deleted ?org=/?orgId=/
?tenant=/?allOrgs= from a member's query. Adversarial review killed it, correctly:
- INERT. hanzoai/o11y has no query-parameter org selector. Every read takes its
tenant from orgFromContext -> ClaimsFromContext, set only from the X-Org-Id
this gate validated. The eight keys were read by nothing.
- BYPASSABLE. url.ParseQuery SKIPS a pair containing ';', so ?org=victim;x=1
left q.Has("org") false and forwarded the raw query verbatim, org=victim
included. It was case-sensitive besides (Org=, ORG= passed) and named none of
organization=, owner=, orgs=, workspace=.
- LOSSY. On a match it re-encoded the whole query, dropping pairs Go rejects and
rewriting %20 to + inside a caller's own ?query=.
A denylist over a lossy parser is not a filter, and one that guards nothing while
reading as a control is worse than none. Deleted, with the reasoning kept where it
was so it is not re-added.
Isolation is the org pin: SanitizeIdentity deletes every client X-Org-*/X-User-*
header at ingress and re-mints X-Org-Id from the validated principal's own claim.
The tests now assert THAT — a request naming another tenant in the query is served
scoped to the caller's own org — across every product path, including the
spellings and the semicolon form the denylist missed.
Verified: gofmt clean, go build ./apps/o11y = 0, suite ok. Negative control:
restoring the admin-only term fails TestProductServesAnOrdinaryOrgMember and
TestMemberCannotReadAnotherOrg; restoring passes. DSN ingest and health probes
still short-circuit ahead of the decision.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
admin.hanzo.ai showed "Could not load — iam: iam status 200" on Organizations and
on most other admin views, because they all read through one client.
The string is the tell. apps/admin/iam decodes IAM's {status,msg,data} envelope and
errors when status != "ok". IAM's NATIVE surface returns typed output with no status
field at all, so a healthy 200 carrying real data decoded as Status:"" and Msg:"",
and the empty-msg fallback printed the HTTP code of a response that had in fact
succeeded. The transport was never broken; the contract was.
git log -S found how: 85dd3a65 ("name: a thing the host loads is a Plugin") was a
mechanical MountSpec->Plugin rename across 132 files, and it collaterally rewrote
seven IAM paths while leaving their params and the decoder untouched. It left the
comment "; was get-user?id=" behind while changing the selector above it.
That surface is not a legacy prop — it is what the rest of cloud already speaks.
apps/account/iam.go calls /v1/iam/keys, add-organization and update-user through the
same decoder and the same status check. apps/admin was the only caller that had
drifted, and update-user escaped the rename, so this client was split across two
surfaces; it is now one.
The fakes are why CI never caught it. They matched HasSuffix(path, "/users"), which
accepts the native path while returning the compat envelope — a pairing IAM never
produces. Green tests, broken production. They now match the FULL path exactly, so
the next rename fails loudly instead of being quietly absorbed, and a new
apps/admin/iam/iam_test.go (the package had none) reproduces the production string.
Verified on this commit: go build ./apps/admin/... = 0, go test ./apps/admin
./apps/admin/iam = ok, gofmt clean on every file touched.
Credentials and scoping are untouched: still the caller's own Cookie plus
Authorization, no service credential, and IAM pins a non-super principal to its own
org on every one of these paths.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cd.hanzo.ai renders the shared chart from hanzoai/universe and pins cloud's image
in that repo's values/hanzo/cloud.yaml. The Application has been Synced/Healthy
throughout — nothing was broken — but the pin moved by hand, so a release stopped
one manual step short of production. That is why v1.801.327 outlived every commit
after it.
This file is the overlay the Application layers last, read from THIS repo at the
tag CD resolves. The 848 lines of real configuration stay in universe; the only
key here is the only key it overrides.
Seeded at the version already live so the first multi-source sync is a no-op that
proves the wiring without moving production.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A live session already records the machine, repo and cwd it runs on. What was
missing to WATCH one is its address: the URL that machine published for its
terminal (zrok gives one without opening a port). One column, carried through
register, patch and the view, alongside the execution context it belongs to.
It is a URL rather than a stream because the bytes belong to the machine running
the shell — cloud holds the address, never the connection, so a session that ends
stops answering in its own frame instead of leaving a console holding a half-open
stream. https only: the console frames this value, and any other scheme is a way
to get a javascript: or file: URL rendered on a signed-in page. A pointer on
patch, so a session that stops sharing can withdraw it.
This REPLACES the separate /v1/sessions plane added earlier in this branch, which
was a second answer to a question /v1/agents/sessions already answers.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/event and /v1/insights/e are the two WIRES this package accepts. The
other three doors — /v1/analytics, /v1/analytics/batch, /v1/tracker — were
more spellings of the canonical wire /v1/event already serves, kept on the
stated grounds that callers still named them. Neither ground survives:
- @hanzo/event 0.3.x posts /v1/event and is what every Hanzo surface now
ships. Its predecessor @hanzo/capture 0.1.1 POSTed /v1/analytics and
beaconed /v1/tracker; the fleet's last importer moved this cycle.
- /v1/tracker was never reachable here. apps/tracker owns the prefix in the
manifest and registers only /v1/tracker/projects/…, so the bare path has
answered 405 in the fleet while passing analytics' single-app tests —
manifest/router_test.go carried it as a known two-claimant name. Dropping
the squatter resolves the collision and the ledger line goes with it.
- the batch alias was held open by "openapi analytics_batch, the generated
python SDK, and `hanzo analytics batch`". Those name analytics.hanzo.ai:
its batch takes an array of SendPayload and answers
{size,processed,errors,details}, this one answers CaptureResult, and cloud
serves none of that collector's routes. They were never a contract here.
Batch stays a BODY, not a path — decodeIngest takes {batch:[…]} and a bare
array at the one door, the same reason there is no /v1/event/batch.
The READ lenses are untouched: /v1/analytics/{overview,timeseries,top,health},
/v1/errors, /v1/insights/{events,health} all keep their routes, and every
manifest prefix stays (a missing prefix is an outage). What ends is this
package's claim on those paths as WRITE doors.
$source loses 'capture' with the doors that stamped it; rows already carrying
it keep their value.
The subset and the fleet spec are regenerated from the router, so
plugin/analytics/openapi.json and openapi.yaml drop exactly the three POSTs.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The public tenant read (which the pay SPA hits on every boot to initialize its
Square card iframe) returned 404 {"error":"unknown tenant"} for pay.hanzo.ai and
api.hanzo.ai alike, so the card and top-up path could not load at all.
Cause was host resolution, not tenant data and not the pinned version: fiber
parses the request URI once, so commerce's forwardedHostMiddleware — which lifts
X-Forwarded-Host into the Host HEADER — never changed what Host() returns.
Behind the ingress the parsed host is empty, so every host normalized to "" and
hit Resolve's single error path.
v1.49.32 resolves the host at the point of use (parsed host, then
X-Forwarded-Host left-most, then the raw Host header) across all three
tenant-resolution call sites, with the parsed host always taking precedence.
main did not compile. "event: the fact, the stream, and the writer" added 1,529
lines across three new files and never touched capture.go, so the consumer landed
without the producer: fact.go read e.Kind, e.Span, e.Metric, e.Site, e.Level and
e.Release off a CaptureEvent that had none of them, and warehouseReady/
warehouseExec were declared twice. Every build in the repo was red, which is every
test, every CI run and every image.
SITE IS NOW ON EVERY EVENT, and that is the substantive half.
It lived on the exception alone. sentry.hanzo.ai could therefore group faults by
site while analytics.hanzo.ai, reading the event stream, had no site column at all
— "all sites" was a question the data could not answer, and the two surfaces
described different worlds. One property per row is what makes them read the same
one. Level, Release, Environment and Service move for the same reason: they
qualify a signal whatever its kind, and none of them is error-only.
The four OTel signals now share ONE envelope with three bodies — Log, Span,
Metric, each a pointer so absent stays distinct from empty. They differ in their
BODY, not in who sent them or when, and an envelope per signal would duplicate
org, time, session and identity four ways and let them drift.
Exception gains structured Frames beside the raw Stack text. Neither derives from
the other and a client may send either: raw text cannot say whether a frame is
ours, and cannot be scrubbed field by field.
warehouseReady/warehouseExec keep ONE declaration, in warehouse.go beside the
writer that uses them, so a test substituting the store cannot substitute half.
The four new nested types join the proseless ledger, which that file documents
exactly: they reach the document through openapi.Register, which derives schemas
by REFLECTION, and Go drops comments — so zipdoc cannot lift their prose however
well they are commented, and they are commented.
Subset and fleet golden regenerated (make -C apps/analytics describe; openapi-weave).
177 packages green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The tool required CLOUD_KMS_MASTER_KEY. Every deployment sets
CLOUD_KMS_MASTER_KEY_REF — cloud's own env, the direct Secret cloud-kms-master-key.
So the tool exited 2 in the only environment it was ever meant to run in.
That is part of why its migration never ran: not merely that nothing invoked it,
but that invoking it did not work. A tool that cannot read the deployment's own
configuration was never going to be run, and the stores it was written to migrate
stayed unopenable until an outage surfaced them.
It now reads the same variable cloud reads, keeping the old name as a fallback so
a one-off already invoked with it keeps working.
Verified against production: with the variable mapped by hand, -dry-run reported
220 stores needing migration and 0 unopenable, and the run migrated 221 carrying
each store's existing DEK. /v1/git/repos and /v1/sync returned to 200, the mirror
drained, and git.hanzo.ai caught up to GitHub at 3bf3d2da6 — which is the revision
cd.hanzo.ai now reports synced.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A coding session runs on a developer own machine, not in the cluster, so nothing
in the cluster can enumerate them — the machine has to say so. A host agent runs
a terminal, publishes it through zrok for a public URL without opening a port,
and beats here. This surface holds the roster; it never proxies the terminal.
Liveness is a TTL, not a state machine. A session is live if it beat within
SessionTTL, so a laptop that sleeps mid-session drops off the roster and returns
when it wakes, with nothing having to observe that it died.
Isolation keys on the ORG rather than the person — watching a teammate build is
the point — and is a mandatory predicate on every statement, never a query param.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
PRODUCTION IS DOWN ON TWO STORES AND THIS IS WHY.
"a store's key names its owner" changed the sidecar derivation from Global to
owner-bound. The code that WRITES the new form shipped. The migration for stores
already written in the old form did not — it exists only as cmd/cek-rewrap, and
nothing invokes it: not boot, not open, not a Job. Its own commit message says
"needed and did not ship".
So two long-lived stores stopped opening:
OrgDB open "/var/lib/cloud/orgs/hanzo/git.db" … message authentication failed
OrgDB open "/var/lib/cloud/orgs/hanzo/sync.db" … message authentication failed
That is the native git plane and the mirror engine, both 500. With the mirror
engine down, pushes to GitHub never reach git.hanzo.ai, so cd.hanzo.ai has been
reconciling a universe that is two commits stale — and every deploy stopped,
including the deploy that would have carried the migration. Newer stores were
unaffected, which is why the outage reads as two odd endpoints rather than an
incident.
The need is discovered at open, so it is answered at open: Rewrap carries the SAME
DEK to the owner-bound wrapping and the open is retried, once. No tool to
remember, no boot walk, no ordering to get right — one way, at the only place that
knows it is needed.
Conservative by construction. Rewrap reports Already when the owner-bound key
already opens the store, and refuses when NEITHER identity does, so a genuinely
wrong master key or a corrupt sidecar stays an error rather than being papered
over by rewriting a sidecar we could not read. Both are pinned:
TestOrgStoreMigratesItselfOnOpen creates a store under the legacy identity WITH
DATA and proves the rows survive the migration — a changed DEK would leave every
page unreadable — and TestUnreadableStoreStaysAnError corrupts a sidecar and
requires the error.
Removing the retry reproduces the production failure byte for byte, down to the
message.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The unified telemetry plane had a schema and no writer. event.log and event.span
have sat at 0 rows since the databases were created, because nothing in any
shipped binary knew how to fill them — these three files existed only as
untracked working-tree state and had never been committed, built, or run.
fact.go the fact. ONE envelope (org, time, id, name, kind, product,
session/distinct/anonymous/person, url, path, attributes, el)
plus what each signal adds. kind is a COLUMN, not a table:
event.event carries track|page|identify|group, so a caller verb
never becomes a schema decision.
bus.go publish to the EVENT stream, subject per signal (event.<kind>).
LimitsPolicy, not WorkQueue — warehouse, alerts and replay are
independent consumers and each needs its own copy.
warehouse.go durable pull consumer, one per table, ACK only AFTER the insert
commits, idempotent on event.id, so a redelivery cannot double
count and a crash cannot silently drop.
Org rides in the signed envelope, never in the subject, so wildcard
subscriptions stay stable and tenancy stays an authorization concern.
Committed alone, out of a tree carrying 551 files of other in-flight work.
Builds and vets clean on its own package.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This caller already lives under .hanzo/workflows; its `uses:` did not. Reusables
are resolved through services/actions.ResolveUses, which enforces the same
directory allowlist on the REFERENCED path:
"uses:" path %q must be under a configured workflow directory
so `hanzoai/ci/.github/workflows/build.yml@v1` stops resolving the moment
WORKFLOW_DIRS narrows to .hanzo/workflows alone. The failure mode is the one
this file already documents: the forge refuses at InsertRun, before any run row
is written, so there is no failed run to look at — pushes and workflow_dispatch
alike silently do nothing.
Safe to point at .hanzo now because the v1 channel tag moved. hanzoai/ci
66d4f21 publishes build.yml at BOTH paths from every tag, byte-identical apart
from its header, since GitHub resolves only .github/workflows and git.hanzo.ai
only .hanzo/workflows. Verified against the live remote: v1 and v2 each serve
.hanzo/workflows/build.yml as blob 1100f49a672b. Same pipeline, same @v1 — only
the path changed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
PlatformSudo read the HOME org — orgs[0] — so every real operator was denied.
An operator is anchored in a brand org, where they bill and do ordinary work, and
holds the reserved org as a FURTHER membership. The anchor and the authority are
different questions, and reading one for the other made the reserved org
unreachable in practice while the predicate looked correct.
This is what stopped anyone cutting a release or reaching an admin surface.
The anchor still decides the LEDGER, so an operator inspecting a customer spends
their own org's money and never the customer's — cloud's homeOrg is unchanged and
still resolves from the membership set, the API key it authenticated, or the
owner-bound KMS audience.
Full suite: 176 packages.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Cutting the release demanded platform SUDO, and that was a category error.
SuperAdmin is the CROSS-TENANT scope — the authority to act in an org you do not
belong to. Publishing your own org's artifact is not cross-tenant. Requiring the
broadest scope in the system for it did not make the operation safer; it made
releasing impossible for the engineers who own the artifact, while the only
identities that could were the ones already trusted with every other tenant's
data. Conflating "privileged" with "cross-tenant" is the same error that let an
org-role bit be read as platform authority.
Release now takes the SAME authority an ordinary build takes, through the SAME
function: admin of an org that OWNS the registry namespace being published to
(imageInOrgRegistry, which already confines a lux admin from pushing
ghcr.io/hanzoai/*). No new mechanism — the rule that already bounded builds now
bounds the release.
The namespace comes from the CONSTANT releaseImage, never from the request:
launchRelease publishes releaseImage whatever req.Image says, so binding on the
request would check a value the caller chooses.
Four cases pinned: the owning org's admin is authorized and reaches the pipeline;
a lux admin is refused hanzo's release; a plain member of the owning org is
refused (owning the namespace is necessary, not sufficient); and the shared build
token is refused a release while keeping the ordinary build path git-push-to-deploy
runs on.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
PLATFORM_BUILD_CALLBACK_TOKEN could cut a release. It is a bearer secret with no
identity behind it — no membership, no expiry, nothing to revoke but a rotation
that restarts every holder, and nothing in an audit log but "the token". A second
auth system standing beside IAM, deciding the most privileged operation this
surface has: publishing the image the whole fleet runs.
Release now takes principal.IsSuperAdmin and nothing else.
The token STAYS for the ordinary build path. git-push-to-deploy runs on it
(apps/git/build_on_push.go), and removing a credential before its replacement
exists breaks that — the same rule that keeps a service's own identity boundary
running until the edge is genuinely in front of it. Narrowed, not deleted.
TestRunnerRelease_SharedTokenCannotRelease posts the SAME credential to the SAME
endpoint twice, differing only in the `release` flag, so the flag is provably what
the gate turns on: the build is admitted, the release is refused.
THIS IS WHY CI CANNOT SELF-RELEASE, and the fix is not to hand the secret back. A
robot should be able to cut a release — through an IAM identity, not a shared
secret: an application principal holding a release capability, which is IAM saying
THIS named app may release, revocable by name and attributable to it. Provisioning
that app is the follow-on. Until then a release is a SuperAdmin's to cut, and a
SuperAdmin is a human PROVISIONED in the reserved org — never a brand-org user
given a membership row there, which grants nothing and only looks like it should.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/billing/payment-methods answered 405 in production. The handler,
the bridge allowlist and the co-resident commerce mount all shipped — but
a specific route shadows the console pkg's /v1/billing/* wildcard for its
whole PATH, so a GET-only registration made every POST miss on method
before anything else could serve it. The console's save-card call died
there, and with it auto-recharge, which charges the vaulted card.
Registered beside the GET with the same discipline as gpu-charge: the
subject is pinned server-side to the caller's own org so a forged body
can never attach a card to another tenant, and commerce's status is
forwarded VERBATIM so a 402 decline keeps its reason.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An independent review caught both. The previous change renamed the symbol INSIDE a
sentence and left the verb, so two comments read "X-Project-Id is MINTED from the
validated `project` claim (claims.renderProject)" — the citation says render, the
prose says mint, and a reader has to guess which is current.
Six citations still named `iamauth`, a package deleted two changes ago
(hanzoai/gateway/v2/iamauth, iamauth.Claims.MintedProject, iamauth.CookieToken,
iamauth.DefaultProject, iamauth.StripIdentityHeaders). A pointer to a package that
does not exist is worse than no pointer: it sends the next reader looking for a
contract they cannot find. They now name where the contract lives —
hanzoai/authz/edge, edge.Strip, edge.Cookie.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
MINTING IS ISSUING AUTHORITY, and only IAM does it — it signs the token. An edge
verifies one and RESTATES it as headers. The two were one word here, so
mintedProject/mintedBillingAccount are renamed renderProject/renderBillingAccount
and the prose follows. The word stays where it is correct: IAM minting a token, a
sign-in minting a session cookie, mint-user-keys issuing a credential.
Separately, TestRewrapCarriesTheSameDEK was red on main and is not mine — it passed
ALONE and failed in the package. The master key resolves through a sync.Once, so the
test's plain SetMasterKey was a no-op once a sibling had already resolved it: it
minted its sidecar under its own key while Rewrap read whichever key won the race,
and reported "sidecar opens under neither owner nor global" — which reads like the
migration is broken rather than like a test that chose its key too late.
resetMaster clears the Once first, which is what every other test in the package
already does.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Which org a request acts in is ONE rule — the client's selection when the signed
membership set admits it, the home org otherwise, any org for a platform operator —
and it is stated twice: authz.Claims.EffectiveOrg, and cloud's own switch in
SanitizeIdentity.
The second statement is not redundant and should not be collapsed. Cloud resolves
the HOME org from facts the claims do not carry — an API key it authenticated, an
owner-bound KMS audience — so it applies the rule with more information than the
leaf has. Forcing one implementation would mean passing cloud's home AND a sudo
predicate into a parameterized shell, which reads worse at both call sites and buys
nothing: what must never differ is the RULE, not the code.
So the agreement is pinned instead of assumed. Wherever cloud's home org equals the
leaf's, both must resolve the same effective org — across no selection, the home
org, a granted org, an ungranted one, a non-injective one, an org admin, and a
platform operator with and without a selection. Where they legitimately differ the
case skips itself rather than asserting a comparison that has no meaning.
Falsified through the real middleware with a local replace: dropping the membership
loop from the leaf turns it red on exactly the two granted-org cases.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/world/stream is the one operation here that cannot be a typed op (the
pin now cites the wire fact at stream.go:151), and it reached the document
bare. stream.go's init declares its summary and description through
openapi.Describe — SSE contract, heartbeat, best-effort delivery, the GET it
re-fetches truth from, the 403 — so the subset, the fleet golden, the SDKs
and the spec-derived CLI all carry it. TestEveryRefusalCarriesProseInTheDocument
closes the loop: a pinned refusal with neither summary nor description goes
red, so the world surface measures ZERO bare operations and stays there.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Register declares the bodies the router cannot derive; nothing declared the
PROSE of an operation the wire refuses to let become a typed op, so a pinned
refusal published an operationId and nothing else — every SDK generated off
the document offered a call it could not explain. Describe is the prose half
of the same registration, with the same drift-proof property: a declaration
whose route is not in the router never renders, and each half guards its own
duplicate, so one package Registering the bodies and Describing the prose of
one operation is one declaration in two statements, not a clash.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nothing has ever reached a human. Alertmanager's slack_configs read a
secret holding the receipt sink's own URL, so 439 'slack' notifications
— including criticals firing right now — were delivered into a log, and
the config's own comment admits PAGE-DELIVERED is a log line, not a page.
The fix is NOT an incoming webhook: that is a second Slack credential
living outside KMS and a second egress beside the one the product
already uses. This receiver now forwards each notification through
integrations.SendSlack — the ONE Slack egress, posting with the org's
KMS-custodied bot token from the installed Hanzo app, shared with
channels and automations. One credential, one egress, one receipt.
Detached and fail-soft by construction: Alertmanager is waiting on this
request, and an alert path that can block or fail on a third party goes
quiet exactly when the third party is having the outage. The receipt
lands first and unconditionally, so a paging failure is itself logged
against an alert we can still prove arrived. Resolved notifications page
too — 'it recovered' is the half people wait for — and a storm is
bounded to 20 lines with the overflow COUNTED, never silently dropped.
Config: CLOUD_ALERTS_SLACK_CHANNEL (no channel, no paging) and
CLOUD_ALERTS_SLACK_ORG (defaults to the platform tenant).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
That change moved KEK derivation from fileID alone to (principal, fileID). Every
sidecar written before it was wrapped under Global, so EVERY pre-existing org
store stopped opening the moment it landed — reported as 'wrong master key or
corrupt sidecar', which is true and misleading: the key is right, the sidecar is
intact, only the identity moved. Measured on prod: the hanzo org's kms.db.dek
unwraps cleanly under Global and not at all under Org(hanzo), so every KMS read
and write for that org has been 502ing.
Rewrap unwraps under the legacy principal and re-wraps the SAME DEK under the
owner, atomically. wrapExisting is why the DEK survives: mintSidecar always
generates a fresh one, which is right when a store is born and destroys the data
when a store's key is re-homed. The test asserts the DEK is byte-identical after
migration, because that equality IS the safety argument.
Not a compatibility shim — one derivation, and this walks the old world into it
once. A sidecar that opens under neither identity is reported and skipped, never
'repaired' with a new DEK, which would answer every future read with plausible
garbage instead of an error.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
DESIGN ONLY — nothing wired. This marks the seam so the next change lands here
instead of adding a sixth object to every stateful pod.
Each replicated service currently carries four objects and a key: a replicate
container, a generated ConfigMap, a restore initContainer, and its own age
keypair. All of it exists because replicate is a separate binary watching a file
it does not own, so the file has to be described to it.
That arrangement produced four independent outages in one day (2026-07-29): a
misindented age stanza replicate refused, an age/plaintext mismatch between
config and bucket, a service whose data dir was not mounted, and a restore path
that had never once run. The last is the instructive one — restore only runs
-if-db-not-exists, so while the local file happened to exist it was never
exercised. The backups were configured, not current, and not restorable, and
nothing said so until a volume was lost.
Open is already 'the single way a cloud store opens its file' and Exists already
answers 'is there a store here' — which is the entire question the initContainer
shelled out to ask. Native, the lifecycle collapses into Open: hydrate if
absent, follow after. Restore stops being a lifecycle stage and becomes what
Open does; the ConfigMap, initContainer, second container and the ordering
between them all disappear.
ONE KEY. cek already holds CLOUD_KMS_MASTER_KEY_REF and refuses to open a store
unkeyed. The age identity is a SECOND key system encrypting the SAME data, with
no rotation story at all — an age identity cannot be rotated after the fact, so
losing it makes every replica under it unreadable. So the age keypair should not
be migrated to KMS; it should stop existing, and the replica should be encrypted
under the key the process already holds. That makes the KMSSecret file added to
universe today unnecessary rather than merely unarmed.
Three verbs, all about bytes at a path: Has, Hydrate, Follow. Follow returns
when following STARTS, not when caught up — a store that refuses to open until
its backup is current will not open during an S3 incident, trading a durability
risk for an availability one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
That change moved KEK derivation from fileID alone to (principal, fileID). Every
sidecar written before it was wrapped under Global, so EVERY pre-existing org
store stopped opening the moment it landed — reported as 'wrong master key or
corrupt sidecar', which is true and misleading: the key is right, the sidecar is
intact, only the identity moved. Measured on prod: the hanzo org's kms.db.dek
unwraps cleanly under Global and not at all under Org(hanzo), so every KMS read
and write for that org has been 502ing.
Rewrap unwraps under the legacy principal and re-wraps the SAME DEK under the
owner, atomically. wrapExisting is why the DEK survives: mintSidecar always
generates a fresh one, which is right when a store is born and destroys the data
when a store's key is re-homed. The test asserts the DEK is byte-identical after
migration, because that equality IS the safety argument.
Not a compatibility shim — one derivation, and this walks the old world into it
once. A sidecar that opens under neither identity is reported and skipped, never
'repaired' with a new DEK, which would answer every future read with plausible
garbage instead of an error.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
git.hanzo.ai is canonical and GitHub is its mirror, but the two had drifted the
wrong way round: GitHub carried 150 commits the forge had never seen, and the forge
carried three the mirror never carried back —
ae978f2b6 ci: v* tags never triggered a build — a duplicate YAML key ate the filter
5693f87e8 ci: the reusable build was pinned to a path that does not exist — CI was dead
7376db9fb docker: cgo plugin builds need -tags sqlite_math_functions
Each of those is a real repair to the thing that BUILDS us, made where the build
runs and stranded there. Meanwhile every dev commit — including the AI
balance-reader repair hanzo.app is currently 503ing without — sat on GitHub where
nothing builds it.
Merged clean, no conflicts. This puts the CI repairs and the product history on one
line so a build can exist at all.
plane.Money is a decimal string so an amount survives the process boundary
unrounded, and the receiver already honors it (meter_rpc parses the decimal and
debits it verbatim). Every sender defeated it: each plane.Amount call site was
plane.Amount(money.FromUSD(x.Cents())) — the exact value, flattened, re-wrapped
as "exact". After the Usage.Money guard fix let sub-cent debits SURVIVE to the
peer path, meterPeer posted them as $0.00. The hole had moved, not closed.
resource_billing_peer.go meterPeer sends u.Money() whole; the failure log
prints the amount, not a cents field that reads 0
for exactly the debit that was lost
apps/commerce/balance_rpc balance and usage rows carry the ledger's own value;
rebuilding an "exact" amount FROM r.Cents was the
sharpest form — the wire type promised precision the
value had already lost
apps/finance UsageRow carries Amount (exact) beside Cents (its
rounding), the split TxnRow beside it has had since
it was written
apps/billing the usage envelope gains `decimal` — amount stays
cents for the wire the console parses today; a row
built without the exact value omits the field rather
than asserting a zero the cents deny
money.Unwrap is the one bridge to the shared type the plane speaks; when the
wrapper collapses into hanzoai/money, call sites lose the call and nothing else.
Cents-only senders left alone, stated here so nobody re-files them: gatePeer
(the gate reserves; its input is priced in whole cents), starter and treasury
reserve (stored as cents; conversion is exact).
Three tests, each shown red without its fix. The peer one captures the RecordIn
on a real plane socket: without the sender fix the debit crosses as wire "0"
"USD"; with it, 0.0025 arrives whole.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud's boundary wrote its identity headers as STRING LITERALS while authz owned
the names, so the two lists were free to drift — and one already had. X-App-Id was
written here and named nowhere in the estate, so an edge stripping authz.Headers
would have left a client copy standing for whoever read it next. It is a caller
label rather than an isolation boundary, which makes forging it cheap, not a reason
to leave it forgeable. authz names it now, and every write here goes through the
constants.
TestEveryHeaderWrittenIsAName holds the property mechanically. It greps the source
rather than exercising a request, because the hazard is a write no test happens to
reach: a literal is what it catches. Reverting one constant to its literal turns it
red.
homeOrg composes instead of restating: the two branches above it are the facts only
cloud has — it authenticated the API key, it knows the owner-bound KMS audience —
and everything else defers to Claims.Home, which is now the same rule cloud
discovered and the leaf had not learned.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`export TMPDIR ?= $(HOME)/.cache/go-tmp` pointed the linker at disk but never made
the directory. go does not create it; it stats it and fails:
go: creating work dir: stat /root/.cache/go-tmp: no such file or directory
account.go:160: running "go": exit status 1
!! account cannot project its own document — an app that cannot describe itself is the bug
That message reads like a contract defect in the account app. It is not. The app
never got to describe itself, because `go` could not start.
Only CI could hit it. A dev box has ~/.cache already, and macOS always exports
TMPDIR so `?=` never even takes this branch — which is why it passed locally and
failed in the container, where TMPDIR is unset and HOME=/root.
The cost: cloud has not built since 2026-07-29 01:10. Every image since failed on
`test app-contract`, so v1.801.322-324 do not exist and the fleet still runs .321.
The AI balance-reader repair sat un-shippable behind it while hanzo.app answered
503 balance_unavailable on every completion.
Verified under the CI condition, not the local one:
env -u TMPDIR HOME=/tmp/fh make -f mk/go.mk -> creates /tmp/fh/.cache/go-tmp
A `$(shell)` on its own line does not run; the simply-expanded assignment forces it.
The ai/destinations/dns/licensing/runtime/templates tranche of the typed
migration converts nothing, and that is the verified result, not a punt:
all 29 undescribed operations across these six subsets are one of five
registrations whose typing would move the wire.
- ai: app.All("/v1/*") beego adapter (hanzoai/ai v1.832.5 mount.go:128)
- dns: Group("/v1/dns").All("/*") (apps/dns/dns.go:87)
- runtime: app.All("/v1/bot/*") (apps/runtime/ops.go:72)
- licensing: app.All("/v1/licensing/*") (hanzoai/licensing v0.1.5 mount.go:71)
- destinations: g.Post("/:platform", connect) (apps/destinations/destinations.go:170)
The four wildcards relay a foreign handler's own status and Content-Type
verbatim; connect binds a body whose property NAMES the addressed
platform's Spec chooses at request time, with string|number|bool values —
and a typed op would also 400 a malformed body before the handler's
403/404/503 gates, an error-precedence move. Each ground re-verified
against zip v1.18.12 (no All[In,Out]; typed ops answer only c.JSON(out)
under their declared status, typed.go:302-311; bindURL sets scalars, never
a greedy sub-path) — the dns and runtime ledgers now cite the version
go.mod actually pins.
LLM.md's opaque-subset taxonomy had licensing in the wrong class: its
Mount hangs hanzoai/licensing's OWN net/http mux on the wildcard through
zip.AdaptNetHTTP, in THIS process — iam's shape (an absence of
composition), not bot/sentry's cross-process proxy. The fix direction
differs by class, so the misfile mattered: licensing's ops become typeable
by typing them in hanzoai/licensing, not by teaching cloud a foreign
route table.
All six subsets regenerate byte-identical (describe; zipdoc drift: none),
so the published documents are current: 38 ops, 9 described — destinations
4/5, templates 5/5, the four relays 0/7 by design, each refusal pinned in
a typed_wire ledger or named here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Commit 16576f81 typed 19 of this tranche's 27 silent operations and refused 8,
saying all eight were gated by untypedByDesign + TestEveryRouteIsTypedOrNamed.
Seven were. The four /v1/agent operations were not: apps/agent had no test at
all, so the refusal lived in a comment that cannot go red — a route added
untyped there would publish nothing and nobody would be told, and the day
hanzoai/agent ships these ops typed the stale reasons would outlive them.
apps/agent/typed_wire_test.go now measures what the comment asserts, against
the REAL Mount: the served surface is exactly the four relayed operations,
none carries a typed registry entry, and the two ledgers must partition the
surface — so it breaks loudly in both directions. Each entry names the op's
own binding fact beside the shared one (registered by hanzoai/agent v0.1.3
agent.go:166-169, not by cloud): the verbatim upstream-4xx relay on
POST /v1/agent (round.go:110-113), the Deps.Principal func(*zip.Ctx) caller
seam the reads share, and the live-Ctx tool dispatch.
Re-verified with the subsets regenerated from the live routers: all six
(usage, venue, world, agent, bot, help) byte-identical to what is committed —
19/27 described, 0 bare properties, no phantom bodies, no embedded-struct
tells, no schema collisions, no newline summaries.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The nine refusals across this six-plugin tranche (websearch's eight, content's
one) were re-checked against the zip actually pinned today, not the v1.18.11
the note recorded: still no typed All (typed.go registrars are the five named
verbs), op.invoke still 400s on any unparseable non-empty body, WithStatus
still admits one 2xx, and the error handler still renders only *HTTPError —
so neither the firecrawl 200-on-malformed contract nor the resource-deny
envelope is expressible. The stamp moves so the next sweep re-litigates
against the right baseline or not at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud held the third independent reading of what an IAM token means: its own claim
struct, its own algorithm allowlist, its own JWKS cache, its own key selection, its
own issuer comparison. The file said so at the top, and explained why — the gateway
is heavyweight and already imports cloud, so importing its validator back would
braid a module cycle. Both halves were true. The conclusion, write our own, is what
produced the copy.
The premise is gone: hanzoai/authz is 148 packages with one non-stdlib dependency
and imports nothing from cloud or the gateway. So idClaims now EMBEDS authz.Claims
rather than restating it, and validate() is the shared edge.Verifier.
The copy was not free, and both costs were real:
it declared a `type` claim IAM emits NOWHERE and read it as the machine
discriminator, so every machine principal arrived as a human (fixed last change);
its key selection FELL BACK — after the kid-matched key failed it tried every RSA
signing key in the JWKS and accepted the first that verified. A token naming
cert-hanzo was accepted on a signature from cert-lux, and a token naming NO key
was accepted on any of them. Not exploitable by a tenant, because IAM publishes
only certs owned by a reserved platform org, but the INVARIANT was gone: any
future widening of what reaches the JWKS becomes an impersonation path silently.
TestTokenIsVerifiedByTheKeyItNamed pins it through cloud's own boundary, with TWO
platform keys published — the real shape, since rotation is additive and each brand
has its own cert.
A FALSIFICATION THAT DIDN'T BITE, which is the finding worth recording. Reverting
isHuman to the pre-fix reading left every SuperAdmin probe GREEN: that arm is
separately blocked because a machine resolves no home org at all. What actually
depended on isHuman was the ORG-admin bit, and nothing tested it. So the predicate
was load-bearing in exactly one place and pinned in none.
TestMachineIsNeverMintedTheOrgAdminBit now covers it, and reverting isHuman turns
it red. `captured` gained the org-admin bit, because no test was reading it.
WHAT STAYED, deliberately: the trusted-issuer SET (the brands this binary fronts —
authz gained an issuer allowlist for exactly this), the API-key resolver and the
subjectOrg it yields, the memo that keeps a replayed ZAP credential from
re-verifying per frame, and username()'s legacy `name` fallback, which now EXTENDS
authz.Username rather than restating it. VerifiedIdentity.Orgs stays []model.OrgRef
— it is cloud's published contract, copied verbatim into a session by clients/team —
so there is ONE conversion, at that surface.
auth_identity.go: 717 → 520 lines. Full suite 174 packages, CGO_ENABLED=0
-tags sqlite_fts5 with the dev KMS key.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The door's own assertion was stricter than the truth and would have failed a
correct fleet: `{}` is what an op whose body is an ARBITRARY document honestly
publishes — apps/flags' put_v1_flags_defs_key takes a PostHog-shaped definition
the evaluator consumes verbatim, so zip's rootSchemaOf constrains nothing and is
right not to. Absent is the failure; empty is an answer.
The Go gate beside it (manifest/mcp_test.go) already read it correctly — it tests
the raw bytes, where `{}` is length 2 — so the two now agree.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Each build job is a fresh pod with an empty local cache and the fabric passed no
cache flags, so buildkit re-resolved and re-downloaded everything from the network
every time: for studio that is the entire torch stack plus requirements ON EVERY
PUSH, which is why a build takes tens of minutes rather than a couple.
Import and export a per-repo cache at <image-repo>:buildcache (the standard
convention — no extra credentials, GC'd with the package). mode=max exports the
intermediate stages, so a one-line code change reuses the dependency layers instead
of rebuilding them. Both flags are advisory in buildkit: a missing or unreadable
cache ref is a cache MISS, never a failure, so a first build behaves exactly as it
does today. A digest-pinned ref names no tag to hang a cache off and is skipped.
(apps/platform tests don't build on this machine — a pre-existing CGO/sqlite issue
in github.com/hanzoai/base, reproduced on a clean stash — so the added test is
verified by inspection here and will run in CI.)
Typing a route bought OpenAPI prose, an SDK method and a CLI command, and NOTHING
on the public MCP surface. zip has projected every typed op into an MCP tool since
v1.18.6 and cloud never called it: manifest routed /v1/mcp to apps/tools, which
hand-rolled its own tools/list + tools/call over a route-table scrape, and
apps/automations hand-rolled a THIRD catalogue. Three registries for one concept,
and the one the public reached exposed none of the 549 typed ops.
THE DOOR IS THE HOST'S. cmd/cloud sets zip.MCPConfig{Path:"/v1/mcp"} and hands
each plugin its own catalogue at Load. The host is the only process that CAN own
it: MCPTools() is in-process, so a plugin cannot enumerate a lazy sibling, and a
plugin-hosted door costs its own wake on the first list. Measured: POST /v1/mcp
beats ai's "/v1" remainder by specificity, not registration order.
THE LIST IS A BUILD ARTIFACT, so tools/list costs ZERO wakes. It has to be: 112
plugins mount LAZILY, and an MCP client calls tools/list constantly — a door that
fanned out over ZAP to ask would destroy the one invariant that makes 112 services
affordable. The answer is already fixed at build time, by the same typed-op
registry that emits openapi.json, so `<app> describe <dir>` now writes BOTH
projections from ONE mount at ONE instant: openapi.json and mcp.json. They cannot
be generated apart, so a tool cannot exist without its op or carry a stale schema.
The leaf plugin/embed.go go:embeds them (cmd/cloud goes 344 → 345 packages, still
zero from apps/). Measured live with the WHOLE fleet mounted: 549 tools listed,
child count 4 → 4 (the four eager apps, untouched).
tools/call is the ONLY trigger and starts exactly one child — p.target(), the same
single-flighted lazy path a prefix request takes — then forwards the SAME message
to that plugin's own /mcp over ZAP on its 0700 unix socket. Never HTTP. The child's
registry answers, so the host can only NAME a tool, never invoke one the child did
not declare. Measured live: get_v1_pricing woke 1 child and returned the pricing
catalog; get_v1_company answered its own handler's "X-Org-Id required" through the
plugin's full cloud.Serve identity chain.
DELETED, not left dark:
apps/tools/builtin.go (223 lines) — the "full-cloud-control" route→tool scrape.
Structurally dead since the monolith died: in the tools CHILD, GetRoutes() sees
only tools' own ~13 routes, and its schemas were opaque {query,body} objects a
model cannot fill. The new door is what it meant to be, with real schemas.
apps/tools/http.go's mcp/mcpToolList/mcpToolCall/rpcResult/rpcError + the route.
apps/automations/mcp.go's mcp/mcpTools/mcpResultObj/mcpErrorObj + its route.
GET /v1/mcp — a Source view that is GET /v1/tools?source=mcp by its own comment.
Principal.credential + credentialHeaders — replay state only builtin.go read.
KEPT, because it is a different capability: apps/tools' EXTERNAL MCP server
registry (records, KMS-sealed secrets, SSRF-validated dialer, tools/list fan-out),
now owning /v1/mcp/servers alone. Its tools, org skills, agents, functions and
connector actions are ROWS, not code, so no build-time catalogue can hold them —
they are reached through the typed POST /v1/tools/call, which is itself a tool on
the door. Nothing lost: connectorToolProvider already published every connector
action into that one registry.
THE GATE. mk/fleet.mk surface-check (which .hanzo/workflows/cicd.yml → hanzo.yml
app-contract actually invokes) regenerates every app FROM SOURCE and fails on
`git status --porcelain -- openapi.yaml plugin/` — mcp.json is under plugin/, so it
was covered the moment it landed there. PROVEN TO FIRE: adding one typed op to
apps/guide without regenerating turned it red on BOTH plugin/guide/mcp.json and
plugin/guide/openapi.json; reverted, green. Four more, all cheap: no App row may
claim /v1/mcp (fiber MERGES byte-identical patterns, so a Load there would shadow
the door silently); no served path may END in /mcp; no Go source outside cmd/cloud
may name an /mcp path unless it is a named foreign engine (apps/tasks' own
surface, which is not a projection of our ops); every catalogue tool must be an
operationId of its own app, unique fleet-wide, with a NON-EMPTY description —
the last one because a nameless tool is a silent failure a model pays context for.
549 tools across 36 apps, 349KB on the wire. zip v1.18.11 → v1.18.12.
Capability check, precisely: the 17 executable connector actions the deleted
automations door listed are NOT tool names on the fleet door, because they are
per-tenant rows — connectorToolProvider publishes every one of them into the ONE
registry from the same `registry` map that door read, so they are reached through
tools_call with the same activation, price, meter and audit. Nothing is lost; one
hop is added. Same for org skills, agents, functions and external MCP servers.
One door this gate structurally cannot claim: /v1/tasks/mcp is hanzoai/tasks' own
engine surface behind cloud's identity gate, mounted on a raw net/http mux so it
is in no subset at all. It is a foreign engine's tools, not a projection of ours,
so it is NAMED in foreignDoors with the reason rather than deleted.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The work list was the ARTIFACT, not a grep: every operation in
plugin/{wallets,webhooks,account-bridge,ads,channels,code}/openapi.json carrying
neither a description nor a summary — 44 of them, six plugins at 100% undescribed,
which is exactly the set that projects to NOTHING: no prose, no MCP tool, no CLI
command, no typed SDK method.
35 are now typed ops. Per plugin: wallets 8/8, webhooks 8/8, code 7/7, ads 6 of 7,
channels 6 of 7, account-bridge 0 of 7. Each converted package carries
untypedByDesign + TestEveryRouteIsTypedOrNamed + TestEveryTypedOpIsDescribed
reading the LIVE router, whose two ledgers must SUM to the served surface — so a
route added untyped here goes red and a stale reason goes red too.
THE NINE REFUSALS, each wire-bound and each MEASURED:
* account-bridge's 7 are TWO registrations, and they were already a closed
refusal one package over: verbatim per-tenant forwards on a greedy wildcard
(apps/account/account.go routesBridge), held by apps/account/typed_wire_test.go.
Nothing to convert.
* POST /v1/ads/campaigns/{id}/launch is deliberately BODY-TOLERANT: it discards
the Bind error, so a malformed body launches on the stored account at 200,
where op.invoke's unconditional decode 400s. TestLaunchStillIgnoresAMalformedBody.
* POST /v1/channels/{channel}/send has a package-local 1 MiB body cap a typed op
never sees, AND DisallowUnknownFields, which refuses a spoofed identity field
loudly where jsonenc.Unmarshal drops it silently.
TestSendKeepsItsCapAndItsStrictness.
WIRE PRESERVED, and the three places that took care:
* POST /v1/code/ask reads ?q= first and lets a non-empty body `query` WIN —
the opposite of zip's body/query/path order. One field per source
(json:"-" url:"q" beside json:"query" url:"-") reproduces it rather than
inverting it; all four combinations asserted.
* ?since= on /v1/channels/inbox 400s on a non-integer, and setScalar silently
zeroes one — so it stays a STRING. Where the handler DEFAULTS instead
(?limit= on ads, webhooks, code) an int is wire-identical.
* every body-only field carries url:"-": zip's binder fills an In from the query
too, and ?custody=, ?prune=1, ?url= and ?dmPolicy= would each have redirected a
write the body never asked for. Four TestTheQueryStringCannotRedirectAWrite.
LATENT DEFECTS, all fixed:
1. five packages had NO //go:generate zipdoc directive — which is why 44
operations published nothing: the prose had nowhere to be lifted to.
2. GET|POST /v1/webhooks/ published a TRAILING SLASH (failure mode #9) for a
collection every caller addresses without one. Artifact fixed, wire unmoved —
both spellings still reach the handler.
3. apps/code's test harness RECONSTRUCTED its seven routes by hand, so nothing it
asserted was evidence about the served surface. routes() is a function now.
4. none of the five installed a cloud.Bridge of its own, relying on Serve's
app-wide install that no package test harness runs — the org path was untested.
5. two one-name/two-shape collisions the weave would have refused on entry:
wallets' Account (books') and ads' Campaign (marketing's). Both unpublished
names yielded — WalletAccount, AdCampaign, no wire movement.
ONE DELTA, recorded not glossed: webhooks' 401 had two messages; principal.OrgFrom
folds both halves into one answer, so the typed ops answer one 401 naming both.
Status, shape and ordering unchanged.
apps/{wallets,code,channels}/... enter allowedRequestUses with their reasons: the
audit actor and the server-minted project scope (wallets), the billing payer and
project (code), the org-admin mutation gate (channels). None is a tenant key.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestRequestEscapeHatchIsPinned has been failing on main since fc46e916:
apps/do/do.go, apps/flags/routes.go, apps/research/research.go and
apps/treasury/treasury.go each added a cloud.Request call site without the
allowedRequestUses entry that is the whole point of the pin. Not caught earlier
because the escape hatch's own gate is in the ROOT package, and a pass that
touches only apps/<x> never runs it.
Each entry below was written by READING the call site, not by pattern: do's org()
turns on validated-ness and the SuperAdmin "admin" namespace OrgFrom cannot
express; flags' callerOf needs the project scope and the audited actor; research's
project() is a sub-scope column and answers DefaultProject off the HTTP path;
treasury's admin/myAccounts are the platform-sudo gate and the ?org= that is the
only way across the ledger's tenant boundary.
Verified: `go test -tags sqlite_fts5 .` is green again.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Five plugin subsets published 56 operations with ZERO descriptions: no schema, no
prose, no MCP tool, no CLI command, no SDK method. 39 of them are now typed ops,
which is ONE registry entry with N projections — the REST route, the OpenAPI
operation, the /mcp tool, the CLI command and every generated client all follow
from the same declaration.
authors 0/11 -> 11/11 described
legal 0/11 -> 10/11
campaign 0/11 -> 9/11
tracker 0/10 -> 8/10
base 0/13 -> 1/13
fleet golden: 517 -> 556 described, 1407 operations unchanged.
THE WIRE IS UNCHANGED, and that is the point of the exercise. Six details had to
be carried over deliberately rather than inherited, each pinned by a test:
* the 1 MiB request-body cap in apps/legal, as ONE middleware in FRONT of the
typed ops. A typed op receives its DECODED In, so a size check inside one
runs after the parse it exists to precede — zip answers 400 about unparseable
bytes where this package has always answered 413. 403 still outranks 413, and
the signature completion (which discards its decode error) is deliberately
not capped.
* the body REQUIREMENT on every write that bound one with c.Bind. zip's typed
decode is TOLERANT by construction, so a naive conversion turns a bodyless
PATCH from 400 into 200-with-nothing-changed — measured against the untyped
handlers, not assumed.
* 201 where it is unconditional (legal's two creates, campaign's create),
DECLARED with zip.WithStatus so the document keys its response on the code
the route sends. Where it is CONDITIONAL (authors connect/verify/record answer
201-on-create and 200-on-found from one address) cloud.Created sets the code
the route has always sent and the prose states it — zip cannot declare two
success codes for one op.
* Cache-Control: no-store on legal's two document reads, which carry contract
text.
* the JSON ARRAY the tracker listings answer: a NAMED slice Out, because an
unnamed one publishes no response content at all.
* the conditional keys of legal's document view, as two Out types rather than
an omitempty that would also drop an empty body from the single read.
SEVENTEEN operations stay untyped, each for a MEASURED wire fact, each held as a
closed list a new route cannot join by accident:
* 12 x /v1/collections[/*] (base) — a VERBATIM reverse proxy to the managed
Base orchestrator. Status, headers and bytes are the upstream's; no Out can
carry that.
* POST /v1/tracker/projects and .../issues — the pre-create balance gate renders
its denial with cloud.DenyResource, the fleet's NESTED {"error":{...}} at
402/503, which a typed op's returned error cannot carry.
* POST /v1/campaign/{id}/launch and /pause — neither has ever read a request
body, and zip's invoke refuses one it cannot parse BEFORE the handler runs.
Measured: 200 untyped, 400 typed.
* POST /v1/legal/documents/{id}/sign/complete — DISCARDS its decode error, so
an unparseable body still drives the provider-reported completion.
The last two are the same zip gap: an op cannot declare that it takes no body, or
tolerates one it cannot parse. hasRequestBody already computes "binds nothing the
URL does not carry"; invoke reading the body only when that is false closes both.
Nine published schema names collided with already-published ones (Campaign with
marketing, Filing/Signer with company, Template with guide, Metrics with agents,
repoView with git, projectView with platform, healthReport twice). openapi.Weave
refuses one name meaning two things, and it refused this. The colliding types are
renamed on MY side only, as DEFINED types over the same structs — the json tags,
and therefore the wire, are byte-identical.
Also: registers the four new cloud.Request call sites in the escape-hatch pin with
the reason each one needs the request (the project sub-scope that picks a physical
store, the SuperAdmin claim, the audit ACTOR, the body gate).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The weave went red the moment both landed: apps/treasury's accountView is a
ledger account ({address, balanceCents}) and apps/venue's is a linked cloud
provider account ({provider, label, externalId, clusters, …}). openapi.Weave
refuses one name with two shapes, because every generated SDK would bind
whichever it read last — and neither collision existed while either route was
untyped, since an untyped route contributes no schema at all. Failure mode #5,
landed by two concurrent passes rather than by one.
venue yields, and qualifying it is the better name anyway: the product is
"connect a cloud account", so cloudAccountView / cloudAccountsView say which
kind of account this is. No wire movement — the json tags and every response
byte are unchanged, only the Go type name and therefore the published schema
name.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The work list was the ARTIFACT, not a grep: every operation in
plugin/{usage,venue,world,agent,bot,help}/openapi.json carrying neither a
description nor a summary — exactly the set that projects to nothing at all. No
prose, no MCP tool, no CLI command, no typed SDK method. It was 27 of 27.
19 are typed ops now: usage 5/5, venue 5/5, help 4/4, world 4 of 5, bot 1 of 4.
Every one carries a doc comment that is TRUE of its handler, and every published
schema property carries its own — 0 bare properties across all five subsets.
Eight stay untyped and each names its wire fact AT its registration, gated by
untypedByDesign + TestEveryRouteIsTypedOrNamed so the two ledgers must sum to
what the live router serves:
GET /v1/world/stream Server-Sent Events; no Out expresses a stream.
GET /v1/bot/connect a WebSocket upgrade; 101 then duplex frames.
POST /v1/bot/nodes/{id}/invoke a 403 carrying a DOMAIN body a client switches
on, plus the caller's X-Device-Id, which no In
field may carry.
POST /v1/bot/peer/invoke a net/http machine hop with text/plain refusals
and a MaxBytesReader cap.
the four /v1/agent ops registered by github.com/hanzoai/agent v0.1.3
(agent.go:166-169), not by cloud. They become
typeable upstream, which additionally needs a
per-request bridge there; POST /v1/agent also
relays an upstream 4xx's status AND body.
Two latent defects found by typing and fixed:
plugin/venue/main.go declared no Prefixes, so the standalone binary's scope
owned only the /v1/<name> default — and venue is named "venue" and serves
/v1/cloud, so it owned NOTHING it registers. cloud.Declare attributed every
route to no subsystem and scope.Use installed the app's middleware where no
route lives, which is load-bearing now that cloud.Bridge parks the org a typed
op reads. The apps/plan defect, one app over.
apps/bot had no Makefile, so `make -C apps/bot openapi` could not run and that
subset could never be regenerated by the per-app chain — despite mk/plugin.mk
claiming an app cannot have a main and no Makefile. catalog, crawl, meet and
zen are still missing theirs.
One delta, MEASURED rather than glossed and not fixable in cloud: encoding/json
validates the whole document before invoking any custom Unmarshaler, so the
record-it-and-judge-it-later input that preserves gate order for an oversized or
wrong-shaped body cannot preserve it for bytes that are not JSON at all — zip
refuses those first. TestSyntacticallyInvalidJSONIs400Early (help) and
TestSyntaxErrorIs400BeforeTheAdminGate (venue) pin exactly what moved.
Wire preserved otherwise, and pinned: the 413-after-404-after-503 intake order,
the body-tolerant sync, the ?project cross-check that a body cannot become, the
no-store header on the two money reads, the one-or-many usage report, and the
tenant that comes from the validated principal in every case.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The lifecycle is signup → qualified → credited (was signed_up). Store
migration rewrites existing rows idempotently; the API counts field is
signup (was signedUp); zipdoc regenerated; the last signed_up example in
the bridge header goes with it. Console alignment ships beside this.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
knowledge, do, flags, research, treasury and storage published 49 operations
between them and ZERO descriptions: no prose, no MCP tool, no CLI command, no
typed SDK method. 39 are now typed ops — one registry entry that is at once the
REST route, the OpenAPI operation with its schemas, the tool and the command.
do 0 -> 8 of 8 treasury 0 -> 8 of 8
flags 0 -> 8 of 8 knowledge 0 -> 8 of 9
research 0 -> 7 of 8 storage 0 -> 0 of 8
The ten refusals are wires this stack cannot yet describe, each recorded at its
own registration so the next engineer re-checks the blocker instead of
re-deriving it:
- storage's seven data-plane ops + /health. A refused balance answers through
cloud.DenyResource, which writes the fleet's NESTED
{"error":{"code","message"}} 402/503 IN BAND; a typed op's only refusal
channel is a returned error, which zip renders flat. The same refusal
apps/ml and apps/company already file. /health answers ONE object under TWO
statuses, which a single WithStatus cannot say. Two of the seven are refused
twice over: fiber's `*` has no typed-op spelling — zip leaves it in the op
path while openapi.translate renders the route as {wildcard1}, so Fold would
fail with "typed op has no live route".
- POST /v1/kb/import takes an UPLOAD (a vault zip, an .enex, a JSON export).
zip decodes a typed body as JSON before the handler runs.
- GET /v1/research/artifacts/:sha256 streams raw bytes under the artifact's
own Content-Type. A typed op serialises a Go value as JSON.
Defects the typing surfaced, all fixed here:
- flags.Store.Upsert PANICKED on a body of `null`: it unmarshals into a NIL
map without error and the next line assigned into it. Any caller could send
it. Now refused like every other non-object. Pinned.
- apps/treasury registered a typed op with NO //go:generate zipdoc directive,
and its handler was a closure — a function literal has no doc comment, so
/treasury/reserve published an empty description on every projection. Named
and documented.
- SIX schema-name collisions the weave caught, because the published schema
namespace is FLAT (zip keys on the bare Go type name): research's `Totals`
against admin's, knowledge's authorizeOut/connectorsOut/syncOut against
integrations' and admin's, and the treasury ledger's `Entry`, `Report` and
`Policy` against catalog, admin and gateway. Renamed to JournalEntry,
TreasuryReport, SharePolicy, ResearchTotals and kb*-prefixed. Go identifiers
only — every json tag is untouched, so no wire moved.
- the six apps had no cloud.Bridge of their own, so their tests mount on a
bare app where Serve never runs. Installed per DECLARED prefix.
PUT /v1/flags/defs/:key is the interesting conversion: its body IS an open
PostHog document, stored verbatim. An ordinary struct In would drop every field
it does not name — silent data loss behind a green 200 — so the In states its
own wire form and zip publishes "any JSON" instead of a fabricated object. The
path still wins over the body's own key. Four tests pin it.
Every wire is preserved: 201 and 204 declared on the op (a DEFINED empty Out
publishes "200 with a body" about a 204 — apps/git and apps/tools still do),
query-vs-body binding unchanged, the connector rows' *string keeps
present-but-empty distinct from absent.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The six subsets — notify, product, referrals, validators, zero-trust,
blueprint — were at described=0. Every operation carried neither a
description nor a summary, which is exactly the set that projects to
NOTHING: no prose, no MCP tool, no CLI command, no typed SDK method.
Nineteen are typed ops now (notify 1/4, product 4/4, referrals 4/4,
validators 4/4, zero-trust 4/4, blueprint 2/3), each with a doc comment
that says what the route does, per-field prose on every published
property, and a gate in the package's own typed_wire_test.go whose two
ledgers must sum to what the live router serves.
The four refusals are ONE class: two 200 shapes at one address.
notify's three /send routes answer with a bare SendResponse for a single
recipient and {items:[…]} for several; blueprint's /sbom answers a bare
Estimate for ?template= and {data:[…]} for none. An op declares one Out,
so either shape would publish the other as a lie — worse than none,
because every generated SDK binds it. TestSendAnswersTwoShapes measures
that pair rather than asserting it, so the conversion is a test away the
day zip can declare a polymorphic response.
The wire did not move. A refusal that used to precede c.Bind still
precedes the decode, because a typed op runs after it: requireOrgOnWrite
/ requireAdmin are method-scoped gates on the subsystem's own group, so
an anonymous caller with a malformed body is still 403 and the sibling
/health probes stay open. Every body-only field on a converted POST
carries url:"-", because zip binds query OVER the body and would
otherwise mint a higher-authority ?field= twin no route ever read. A
query scalar whose existing parse trims stays a STRING, measured rather
than assumed: fiber percent-decodes a query value but not a path
segment, so an int field would have narrowed ?limit= and ?tokenId=.
Every body that used to marshal a map[string]any is a struct whose
fields are declared in the map's sorted key order, and the byte order is
pinned.
Four latent defects, all surfaced BY typing:
- apps/zt had no cloud.Bridge on any of its three prefixes. Its own
harness mounts on a bare app with no Serve, so every typed op there
would have seen no org and refused a valid request.
- plugin/{product,zero-trust,referrals}/main.go declared no Prefixes,
so MountPrefixes' /v1/<Name> default covered nothing product or
zero-trust serves and half of what referrals does — the apps/plan
defect, three more times. All three take manifest.PrefixesFor now.
- 38 published properties (26 referrals, 12 zero-trust) were about to
reach openapi.yaml, every SDK and every MCP inputSchema bare.
- zipdoc cannot resolve zip.App.With as a router, so the obvious fix
for the decode-order problem fails generation. Recorded in LLM.md.
Regenerated: six subsets and the woven openapi.yaml (1013 paths,
unchanged — the diff is prose plus the two _by_id -> _id operationId
renames a route always takes when it goes typed).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
One resource, three spellings, and a create that did not live at its own read:
POST /v1/machines/{id}/bind-agent a VERB in the path
GET /v1/machines/{id}/agent-binding SINGULAR
DELETE /v1/machines/{id}/agent-binding
GET /v1/agent-bindings PLURAL, and top-level with no service
HIP-0128 §1 forbids each one: the resource is a plural noun, never a verb, and the
method carries the verb. A caller had to learn that binding an agent happens at a
different address than reading the binding it just made, and every projection —
OpenAPI, the MCP tool list, the CLI, four SDKs — published three names for one
thing.
A machine hosts at most one agent, so it is a to-one sub-resource:
GET /v1/machines/agents every machine's agent in the org
PUT /v1/machines/{id}/agent bind
GET /v1/machines/{id}/agent read
DELETE /v1/machines/{id}/agent unbind
PUT rather than POST because binding is idempotent — re-binding the same agent to
the same machine is the state the caller asked for, not a second binding. Singular
at the member and plural at the collection is not the old inconsistency: it is the
ordinary rule, applied to one resource instead of two.
Both literals register ahead of /v1/machines/:id so no machine id captures them —
the ordering /v1/machines/launch already relies on.
WHAT DID NOT MOVE, and the file now says so where it matters: the cl.call paths in
bots.go are VM'S wire, and the botVM test stub keeps vm's spelling because it
impersonates vm. Renaming those would call a route that does not exist. Two wires,
one translation, stated once so nobody fixes the wrong side.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
bots, entitlements, sbom, translate, agentskills and gateway published 16
operations between them and 16 of 16 carried neither a description nor a
summary — the exact set that reaches no prose, no MCP tool, no CLI command and
no typed SDK method. Eleven are typed ops now; the five that are not each name
the wire fact that keeps them raw, in a CLOSED list a test reads.
entitlements 3/3 gateway 2/2 bots 2/3
sbom 2/3 translate 2/3 agentskills 0/2 (structural)
The five refusals, each re-measured against the PINNED zip v1.18.11 rather
than inherited as prose:
POST /v1/bots/run answers 501 unconditionally. A typed op publishes
a SUCCESS response it can never send, and mints an
MCP tool and CLI command for an operation that
cannot succeed. It is also body-tolerant.
GET /v1/sbom/{wildcard1} a greedy wildcard: fiber binds it as `*1`, the
document publishes `{wildcard1}` as a PATH param,
and a typed op publishes op.Path verbatim — so the
address, the parameter name and the parameter
LOCATION would all move for a wire that did not.
POST /v1/translate a bulk-tier spend denial answers 402/503 carrying
the fleet's NESTED {"error":{code,message}} domain
body; a typed op's only refusal is a returned
error, which zip renders flat.
GET /.well-known/agent-skills/index.json embedded bytes served verbatim
(its sha256 digests are computed over what is
served), a Cache-Control zip cannot set, and an
{"error":…} 404 body errorHandler does not produce.
GET /.well-known/agent-skills/{skill}/SKILL.md text/markdown. A typed op
answers application/json — no In/Out serves it.
WIRE PRESERVED. No path moved, no status moved, no field name moved; the only
published deltas are the three operationId renames typing always makes
(_by_id -> _id) and the prose that did not exist before. Two map[string]any
responses became structs whose fields are spelled in the order encoding/json
emits a map's sorted keys, so the BYTES did not move either —
TestHealthAndIngestKeepTheirBytes asserts that.
The org is never an input field. `url:"-"` (new in zip v1.18.11) lets the :org
segment bind from the URL while staying out of the published body and
invisible to the decoder, and the gate still compares it to the VALIDATED
principal — so it is an address the gate re-checks, never an assertion the gate
believes. Five identity seams needed more of the principal than the org
(SuperAdmin-ness, the user id a write is attributed to, the ?org= a SuperAdmin
targets a tenant with); each is one function, pinned in allowedRequestUses with
its reason, and each fails closed off the HTTP path.
Latent defects the typing surfaced, beyond the two prefix ones fixed above:
- NONE of the six installed cloud.Bridge. Serve installs it binary-wide so
nothing was live-broken, but none of these packages' own test harnesses
runs Serve, so a typed op added here would have 403'd in its own tests with
no hint why. All six install it through the SUBSYSTEM router now.
- apps/entitlements' tests RECONSTRUCTED the router by hand rather than
driving the mount, so they could have gone green against a surface the
binary does not serve. Mount's registration is one routes() function now,
and the tests call it.
- edge.Policy grouped three fields under one section comment, and zipdoc
lifts the comment directly above a field — so "Platform-scope (admin-org
row)" shipped as the published description of cors_origins ALONE. Every
published field in all six subsets carries its own prose now; the check is
in the pass notes in LLM.md.
- Two generic names were qualified before they could collide: bots.botView ->
BotRun (apps/visor already publishes botView for a bot MACHINE) and
translate.Entry -> MemoryEntry (six packages under apps/ declare an Entry).
Neither was published yet, so both were free.
Each package carries untypedByDesign + TestEveryRouteIsTypedOrNamed reading the
REAL mount, whose two ledgers must SUM to the served surface — so a route added
untyped here goes red, and a reason naming a route that is gone goes red too.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
MountPrefixes falls back to /v1/<name> when a plugin declares no Prefixes, and
for these two that default is wrong — the apps/plan defect, found twice more:
agentskills serves the ROOT discovery convention (/.well-known/agent-skills
/…), so /v1/agentskills covered NOTHING it registers. Measured:
SubsystemOf resolved every one of its requests to "" and PriceOf
to Undeclared, and any middleware it installed through its own
router landed on a prefix nothing was ever routed to.
entitlements owns TWO top-level nouns — /v1/entitlements (the commerce
projection) and /v1/orgs/:org/entitlements (the enablement
store). The default covered the FIRST, so half the surface was
unattributed and ungated. That partial shape is the harder one
to see: the app looks covered.
Both now pass manifest.PrefixesFor(<name>), so the list cannot drift from the
prefixes the host actually routes there.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The work list was the ARTIFACT, not a grep: every operation in
plugin/{graph,meet,prefs,settings,share,admission}/openapi.json carrying neither
description nor summary — exactly the set that projects to nothing, no prose, no
MCP tool, no CLI command, no typed SDK method. It was 11 operations, all of them.
8 are now typed ops with true doc comments; 3 are refused, each with its reason
recorded AT its registration and MEASURED by a test rather than asserted.
Typed (described-count 0 -> 8 across the five subsets):
GET /v1/indexers graph GET /v1/settings/{product} settings
GET /v1/oracles graph PUT /v1/settings/{product} settings
GET /v1/prefs prefs GET /v1/share share
GET /v1/flags/waitlist admission POST /v1/share/enable share
Refused, and each is "zip cannot state this":
POST /v1/meet/getToken answers the raw join token as text/plain (the office
client reads it with res.text()); a typed op always marshals JSON.
GET /v1/meet/health answers 200 or 503 with the SAME body, `ready` being
the whole dashboard fact at both — the multi-status gap (#78). A typed op's
only refusal is a returned error, rendered as zip's flat {status,code,error},
so ready:false would vanish from the degraded answer.
PATCH /v1/prefs three facts at once: a 16 KiB REQUEST-BYTE cap that
answers 413 and a typed op cannot see; an empty body and a literal `null`
body each answering 400 where op.invoke skips the decode and null decodes to
a nil map; and an OPEN key space whose only carrier is map[string]any, whose
typeName is "" so hasRequestBody publishes no request body at all.
WIRE PRESERVED. Statuses, JSON shapes, field names and error precedence are
unchanged; apps/{graph,share,prefs,admission,meet} carry tests that drive the
REAL router (routes(), which Mount calls) rather than a reconstruction.
Latent defects found by typing:
1. Five apps have plugin/<app>/main.go and NO apps/<app>/Makefile, and
mk/fleet.mk reads APPDIRS := $(wildcard apps/*/Makefile) — so openapi-check,
the gate that regenerates the document from source, has NEVER regenerated
plugin/{meet,bot,catalog,crawl,zen}/openapi.json. Five published subsets sit
outside the only gate that catches the ingress-class loss. apps/meet/Makefile
is added (its subset regenerated clean); the other four are reported.
plugin/gen-app-cmds writes no Makefile at all, so each Makefile's own header
claim that an app "cannot have a main and no Makefile" is false.
2. Failure mode #9 (the empty leaf) was live twice more: prefs and share each
declared their root as g.Get("", ...), publishing /v1/prefs/ and /v1/share/ —
paths this API has never served. Declared on the app at the whole path now;
the operationIds are unchanged (get_v1_share either way), the untyped PATCH
sibling moved with it so one resource stays one document key, and fiber's
non-strict routing keeps both URL forms answering (pinned by test).
3. Neither graph's Authorization forwarding nor admission's ?host= default had
ANY test. Both degrade silently — a 200 with an anonymous upstream read; a 200
with known:false for every guard that omits the query. Both are pinned now.
4. apps/share had no route-level test at all (its fakeController was unused);
apps/prefs had none either. Both have one now.
cloud.Request gains three entries, each a request FACT and not a tenant: graph
forwards the caller's Authorization to the indexer/graph when no service token is
configured, prefs' isolation key is the qualified <owner>/<name> rather than the
org, admission's ?host= default is the request's own Host. All fail closed off
the HTTP path.
Known, reported, not worked around: settingsReq.config is map[string]any and
publishes additionalProperties:{"type":"object"} — LLM.md failure mode #8. Every
alternative MOVES THE WIRE (json.RawMessage turns {"config":null} from "store {}"
into "store null"; map[string]json.RawMessage changes number formatting and
large-int precision), so the wire wins and the fix belongs in zip's schemaOf.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zipdoc lifts each typed handler's doc comment and its In/Out field comments
into zipdoc_gen.go, which is compiled INTO the binary — Go drops comments at
compile time, so this file is the only path from the code to the published
description and the MCP tool. plugin/deploy/openapi.json is the app's own
subset, projected from the app's own live router by its own binary; openapi.yaml
is the weave of every app's subset.
Measured on the subset: 21 operations, described 0 -> 11.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Eleven of /v1/deploy's twenty-one published operations projected to NOTHING:
no schema, no prose, no MCP tool, no CLI command, no SDK method. They are
typed ops now, each one In/Out with a doc comment that IS the published
description.
The thing that made it possible is a decomplection, not a rewrite. refuse(c)
braided two facts into one function only a handler holding the request could
call: the DECISION (this caller may not have this) and the PRESENTATION (a
browser navigation goes to sign-in, an API call keeps its 403). A typed op
holds no request, so braided they forced a choice between losing the bounce
for every browser and gating in middleware — which the MCP, CLI and call
projections never run, i.e. a hole in three of the four. Split, the decision
is forbidden() wherever it is made and the shape is one middleware every
/v1/deploy route passes through. The wire is exactly what it was, both arms
pinned by TestRefusalIsA403AndANavigationIsBounced.
The scope itself stays where it was: resolveScope, reached through
cloud.Request in ONE file (typed.go), because a SuperAdmin's scope has no org
at all and a tenant's is the SanitizeOrg slug — neither is what
principal.OrgFrom carries. Never an In field.
TEN operations are deliberately NOT typed, each for a measured wire fact
recorded at its registration and asserted in typed_wire_test.go:
- account/can-i/* a fiber wildcard; zip's closeColonParams leaves '*' alone
while cloud's translate emits {wildcard1}, and Fold then
refuses the whole document.
- health answers 503 carrying the same domain body as its 200.
- login, callback succeed with a 302 + Set-Cookie.
- the two streams answer an unbounded text/event-stream.
- the four POSTs zip decodes the body before the handler and 400s an
unparseable one; these read no body, so typing them turns
today's 200/403/503 into a 400 — and for the three gated
ones puts that 400 ahead of the 403.
Bytes are preserved, not merely equal JSON: every model that replaced a
map[string]any declares its fields in the order encoding/json sorted the map,
and sessionUser carries *[]string so `groups: []` can be present for a
signed-in caller and ABSENT for an anonymous one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three spellings of one concept had drifted apart, and one of them was a
live bug: campaign conversions counted event = 'signup', which NOTHING
emits — the @hanzo/event grammar is <object>_<verb-past> and the signup
funnel's terminal event is signup_completed — so signup conversions
always read zero. The query now counts the event that exists.
The destinations-internal normalized token drops its underscore
(StandardEvent 'signup'; it is a map key that never leaves the process),
while adapters keep rendering each platform's own required name — GA4
'sign_up', Meta 'CompleteRegistration'. The bus bridge's examples speak
the real vocabulary instead of an invented 'Signed Up'.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every operation in these five plugin subsets carried neither description nor
summary, which is exactly the set that projects to NOTHING: no prose, no MCP
tool, no CLI command, no typed SDK method.
TYPED (3):
GET /v1/audit org-scoped audit trail
GET /v1/catalog the cross-org discovery lens
GET /v1/x402/settlements/{id} one payment receipt
REFUSED (2), each named at its registration and MEASURED by a test:
POST /v1/ask one route, two success shapes; an SSE branch; a
cloud.DenyResource 402/503 carrying the fleet's NESTED error
POST /v1/crawl body-TOLERANT with a domain refusal body; a 1 MiB
io.LimitReader bound a typed op cannot see
Neither refusal publishes nothing now: both declare their request through
openapi.Register (crawl its response too; ask's is polymorphic, so a single
declared shape would be false and the silence is pinned).
WIRE PRESERVED. Paging and tri-state filters stay STRINGS because zip's bindURL
zeroes an unparseable value and reads a bare ?flag as true — audit matches its
admin twin, which already publishes pageSize/p as strings. audit's envelope
fields are declared ALPHABETICALLY because the map they replace was sorted by
encoding/json, and the tail bytes are asserted.
LATENT DEFECTS FOUND BY TYPING:
- apps/crawl registered the same route twice (Group + empty leaf and "/") and
published /v1/crawl/ — a trailing slash in the document, the operation id, the
MCP tool and every SDK's URL. One registration at /v1/crawl now; both spellings
still answer.
- none of the five installed cloud.Bridge of its own; each converted app does now.
- apps/ask's AskRequest/AskResponse are the two names apps/books ALREADY
publishes with different shapes — ask yielded to askRequest/askAnswer.
- five apps have a plugin main and no apps/<app>/Makefile, so openapi-check
cannot see their subsets. catalog and crawl are fixed here; bot, meet and zen
are still uncovered and the class fix is in plugin/gen-app-cmds.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The measured gap for this unit was 13 operations; six became typed reads one
commit ago, and what is left is refusals, not backlog. Main moved underneath
this pass — /v1/event/collect folded away, the two Sentry doors arrived — so
the ledger reads 6 typed / 8 refused, and every one of the eight was re-derived
against zip v1.18.11's own typed.go rather than inherited as prose:
GET /v1/analytics/health answers 503 CARRYING the degraded report; zip
stamps a non-nil Out 200 and WithStatus refuses
a non-2xx.
POST /v1/event polymorphic wire (Event | [Event] | {batch:[…]});
POST /v1/analytics op.invoke jsonenc.Unmarshals every non-empty body
POST /v1/analytics/batch into the In, so an array body 400s where it
POST /v1/tracker answers 200 today.
POST /v1/insights/e an object — but admission still isn't reachable.
POST /v1/event/{project}/envelope a raw Sentry envelope stream, and a DSN
POST /v1/event/{project}/store key the o11y consumer verifies itself.
Each also gained a blocker the first pass had not written down: ORDER. invoke
decodes the body BEFORE the handler is entered, while the anonymous lane
refuses 403 (capture disabled), then 429 (rate), then 413 (64 KiB) with the raw
bytes in hand and nothing parsed. Typing any door would answer 400 to a beacon
that is answered 413 or 429 today, and error precedence is wire.
What DID change is the other half of the cost. All eight published an
operationId and nothing else — indistinguishable, to every SDK generator
reading the document, from a route that takes no body and returns none. So
`post_v1_event`, the door every Hanzo product beacons to, shipped in every
generated SDK as a call with nowhere to put the event.
They declare their bodies now, through openapi.Register, driven off the doors
table itself so a door cannot be routed with one wire and documented with
another — the drift that once put /v1/tracker in the router and not in the
site-host carve. The gate quantifies over untypedByDesign, not over doors, so a
refusal added tomorrow owes its bodies by construction rather than by someone
remembering.
openapi gains `OneOf`, for the same reason it gained `Binary`: the canonical
wire is genuinely three shapes, and naming one of them would publish an ingest
API that cannot batch. The Sentry pair takes Binary for its request and
declares NO response, named in `relayed` — it copies back whatever
cloud.ObsErrorIngest installed, and publishing a shape there would be inventing
one. And the health probe's map[string]any became healthReport, because a map's
shape cannot be declared without hand-writing a schema beside it, which is the
drift Register exists to prevent.
The subset: 0 -> 7 requestBody, 6 -> 12 responses, 19 -> 30 schemas. The
described count did NOT move — still 6, exactly the typed ops — because prose,
an MCP tool and a CLI command are the three things only zip's registry
supplies. Reported as unchanged rather than counted as progress.
The cost is named and gated, not absorbed: Register cannot lift field prose (Go
drops comments; zipdoc walks typed registrations only), so eleven components
publish 63 bare properties. `proseless` is that ledger and it may only SHRINK —
it caught its own first staleness during this rebase, when main's removal of
the Team door left teamEvent named and unpublished. The fleet ships 1,627 such
properties for the same reason; that measurement is in LLM.md now.
Wire preserved exactly. No status, body, field name or error precedence moves;
the health report's serialization ORDER changes (struct order, not sorted map
keys), which the JSON data model does not carry, and
TestHealthReportKeepsTheMapItReplaced pins both bodies field-for-field.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
content, leaderboard, marketplace, prompts and sync published 30 operations
between them and not one carried a description or a summary. An operation that
publishes neither projects to NOTHING — no prose in the document, no MCP tool,
no CLI command, no typed SDK method — so the whole of five products was
addressable and unexplained. 29 are typed ops now; the described count in the
regenerated subsets goes 0/38 to 29/38.
The wire is unchanged, and that was the constraint rather than an afterthought:
- every status is preserved where zip's default is not it — marketplace
listings and prompts create keep 201 via zip.WithStatus, sync run keeps 202,
every delete keeps its 204 by returning a nil Out through an ALIAS for the
unnamed empty struct (a defined type there would publish "200 with a body");
- the query string still binds. `?limit=abc` reaches the board and the
leaderboard as the default page, not as a 400: zip's binder leaves an int at
zero for a value it cannot parse, and both handlers still read non-positive
as "take the default". TestBoardLimitTolerance pins both halves;
- backfill's `force` stays a STRING and is still compared literally. A bool
field would have bound through ParseBool and quietly widened the guard to
"1", "t", "T", "TRUE" and a bare `?force` — on a rollup that ACCUMULATES,
so a run that should have been refused doubles every day it re-reads.
TestBackfillForceIsLiteralTrue walks every near-miss spelling to the 409;
- sync's PATCH keeps pointer fields, and TestPatchSync_NullIsAbsent pins WHY
that is safe here: encoding/json leaves a pointer nil for an explicit null
as well as for an absent key, which silently turns a "clear" into a no-op on
a route that distinguishes them. None of these fields is clearable, so the
two have always meant the same thing — and the test is where a future
clearable field shows up.
The tenant is never an In field. It comes from principal.OrgFrom(ctx), which
cloud.Bridge parks on the context; an In field is caller-supplied, so a tenant
key read from one is a cross-tenant read the caller asserted for itself. Each
subsystem installs Bridge on its own mount as well, so it still resolves its
tenant when a test or a non-Serve composition root mounts it on a bare app
instead of refusing every caller.
Two subsystems reach the request through cloud.Request, and both are registered
in allowedRequestUses with the reason: a public leaderboard is a CONSENT surface
gated on the validated username and on org/platform admin-ness, and a
marketplace install is scoped to (org, PROJECT) and attributed to a user — none
of which principal.OrgFrom carries.
NINE operations are deliberately NOT typed, each for a reason that is a wire
fact rather than a preference:
- websearch's eight. /v1/websearch/search is registered with All and answers
seven methods including OPTIONS and TRACE; zip has no typed All, so
declaring the named verbs instead would DROP two of them — a routing change.
Its POST/PUT/PATCH arms also read the query and IGNORE the body, while a
typed op 400s on any unparseable non-empty body. /v1/scrape deliberately
answers 200 {"success":false,...} to a malformed or oversized body because
firecrawl clients read data.success rather than the status line, and it caps
the read at 1 MiB with an io.LimitReader rather than refusing; a typed op
can express neither. The package says all of this where the next sweep will
read it.
- POST /v1/content/generate. A studio-render billing denial answers the
platform resource-deny envelope, {"error":{"code","message"}} at 402/503
(cloud.DenyResource, resource_billing.go:224). A typed op can answer its Out
schema or zip's {status,code,error} and nothing else, so typing it would
rewrite that body for every client reading error.code.
Full suite green with the gate env (174 ok, 0 fail); each package regenerated
with `go generate -run zipdoc`, each subset reprojected from its own binary, and
the fleet golden rewoven.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The platform already had both halves and no middle: the ONE door
(POST /v1/event) commits to the warehouse and fans out through the
analytics sink seam; the ONE bus (embedded Hanzo PubSub) carries
commerce.> events to the webhooks delivery engine. This joins them:
- forward.go's seam is now ONE seam, N consumers: AddSink (with remover)
replaces the single SetSink slot; destinations and the new bridge both
register, each dispatched detached and panic-guarded.
- apps/webhooks/bridge.go publishes every accepted batch onto stream
EVENTS as event.<kind> (canonical names fold to NATS-safe tokens:
$pageview -> event.pageview, 'Signed Up' -> event.signed_up; bounded,
wildcard-proof) carrying THE standard Envelope — organization_id
first, because the delivery engine resolves the tenant from the
envelope and an org-less event is delivered to nobody.
- The SAME dispatcher consumes EVENTS beside COMMERCE, so one Endpoint
row subscribes an org to anything on the platform: commerce.order.*,
event.error, event.signed_up — one bus, one envelope grammar, one
delivery engine, one signature scheme.
Fail-soft by construction: bus down = warehouse still durable, nothing
blocks ingest. TestSpineEndToEnd drives seam -> bus -> signed webhook
delivery wholly in-process over the embedded pubsub.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ff188fab ("event: no /api/, no /collect — the door is the whole surface")
regenerated plugin/analytics/openapi.json and did not re-weave openapi.yaml, so
the fleet golden on main is stale against the subsets it is woven from. This is
failure mode #1 in LLM.md, and the reason that mode is written down is that a
gate comparing two DERIVED artifacts agrees with itself while both are wrong.
Wrong in BOTH directions, which is what makes it worth a commit of its own:
- it PUBLISHED 8 operations nobody serves — /v1/event/api/{wildcard1} across
all seven methods, plus POST /v1/event/collect — in openapi.yaml, and
therefore in every generated SDK and in the MCP tool list.
- it OMITTED 2 operations that ARE served: POST /v1/event/{project}/envelope
and POST /v1/event/{project}/store, registered at
apps/analytics/analytics.go:237-238 and present in the committed subset. The
Sentry-compat ingest pair was unreachable from any generated client.
Not hand-merged. Regenerated: the weave is deterministic from the committed
subsets, so this is `make -f mk/fleet.mk openapi-weave OUT=openapi.yaml` and
nothing else. The only content in the diff is those ten operations.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both subsystems are one All("/…/*") registration that the untyped projection
explodes into seven undescribed operations — so each publishes no prose, no MCP
tool and no CLI command for its whole surface. That refusal is correct and it
was written at the registration, which is a promise: nobody can re-check it, so
a route added later inherits the exemption silently and a reason that stops
being true keeps being believed.
untypedByDesign + TestEveryRouteIsTypedOrNamed read the LIVE router of the real
Mount and close both halves: a route here is typed BY DEFAULT, and a reason
naming something the package no longer serves goes red. Verified by mutation —
drop one entry and the gate names the operation.
The reason is re-derived from zip v1.18.11 source rather than inherited: a typed
op's only response path is c.JSON(out) under its DECLARED status (typed.go:
302-311), which a verbatim relay of the upstream's status and Content-Type
cannot survive; there is no All[In, Out]; and a greedy wildcard's value is a
whole sub-path, not a scalar bindURL/setScalar can set on an In field. Each
independently sufficient.
ai and licensing stay ungated on purpose — their registrations live in
github.com/hanzoai/{ai,licensing}, so there is no cloud-side route for a
cloud-side gate to hold.
LLM.md records the re-derivation and the thing the op-level count cannot see:
the same field check run over the committed subsets puts the class at 1,601
bare published properties across 17 packages, several of them packages this
migration already marks done.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Typing a route documents its ADDRESS and its SHAPE. It does not document the
shape's FIELDS — those come from doc comments on the In/Out structs, which
zipdoc lifts per field — so the six-plugin pass left 26 published properties
bare: every property of DestinationStatus (the card all five destinations
routes answer with) and of DestinationField, eleven of StarterKit, and
Variant.source. They reached openapi.yaml, every generated SDK and every MCP
inputSchema with no description.
The split says where the hole comes from: publishKitIn and replaceKitIn, both
written AT the conversion, describe every field; StarterKit, which predates it,
described four of fourteen. The request side got documented and the response
side did not.
The two that mattered most:
- connected / enabled / live are three DIFFERENT facts — configured here once,
forwarding now, and a credential still resolves (KMS secret, else the
integrations fallback token, else none needed). Connected && !Live is
exactly the reconnect-me state, and nothing published said so.
- tier and rating are public-catalog curation. No request can set them:
neither write body has the field and neither kit() carries one, so they are
absent on every customer-published kit. A caller could see two numbers and
nowhere that they are server-owned.
Comment-only in the Go source — no field renamed, retyped, reordered or
retagged — and the regenerated artifacts are 79 lines added, 0 removed.
TestEveryPublishedFieldIsDescribed gates both packages the way
TestEveryTypedOpIsDescribed gates the op side, mutation-checked: blank one
description and it names it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
{status, msg, data, data2} is Casdoor's response type. Its second slot was
untyped, so the row count went there, and the name came along whole — through the
Casdoor Go SDK (hanzoai/iam auth.go still declares Status/Msg/Data/Data2) and
through Casdoor's console, whose getList<T> reads data?.data2 ?? rows.length.
cloud's own envelope.go then called it "the canonical /v1 envelope", which it was
only in the sense that everything had inherited the same shape.
Nothing specifies it. HIP-0111 names it directly as the shape a list MUST NOT
return ("Lists return the SCIM ListResponse envelope (totalResults/Resources),
not a {status,data,data2} one"). Casdoor is dead by standing rule. So the field is
`total`, which is what it holds — same int, same position, a name a reader can act
on. 142 sites, 30 files, plus the doc-comment examples zipdoc lifts into the
published spec.
ONE EXCEPTION, and it is the point of the change rather than an escape from it:
apps/admin/iam/iam.go DECODES hanzoai/iam's answer, and IAM still writes data2.
That struct now reads `Total json.RawMessage \`json:"data2"\`` — the Go side speaks
cloud's language, the tag records the foreign wire, and a comment says when it
converges. A blind rename there would have read nothing: the total silently
becomes zero and every paginated admin list reports its own page size. The same
trap was live in the tests, whose fake IAM server writes data2 on purpose — four
tests caught it, which is why they exist.
The console (hanzoai/admin b030f80) ships the reader in the same window and
accepts total → data2 → rows.length, so either side can deploy first.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/ is the only prefix this platform speaks. The /v1/event/api leaf from
the previous commit is gone; the Sentry wire is POST
/v1/event/{project}/envelope|store, carried by the door's owner (the
project segment is variable, so no static prefix could route it) and
forwarded to the obs plane's installed consumer (cloud.SetObsErrorIngest,
the second seam beside the batch claim), which maps onto the clean
/v1/sentry runtime ingest routes before the principal gate looks — the
existing ingest exemption stays the only one.
/v1/event/collect is DELETED, not sunset: the team wire rides the
canonical door by shape (isTeamArray), so the path said nothing, and
sourceTeam goes with it — a team-shaped body on the door stamps the
door's own source. The POST ledger and the typed/named ledger both name
the envelope wire explicitly; openapi artifacts regenerated; zero
occurrences of the old spellings remain.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The tool plane's two untyped routes were correctly refused (zip cannot express a
body-tolerant op, and cannot answer a non-2xx carrying a domain body), but the
refusal was leaking a second, separable fact into the document: both rendered as
an operationId and a tag and NOTHING ELSE — no requestBody, no response — which
is EXACTLY what a route taking no input and returning none publishes. No consumer
can tell those apart, so every SDK generated off openapi.yaml offered an MCP call
with nowhere to put the JSON-RPC envelope and a plugin build with nowhere to put
the source.
Staying out of zip's registry costs prose, an MCP tool, a CLI command and a typed
SDK method. It must not also cost the SHAPE. openapi.Register (tools.go init) now
states the halves that ARE statable — mcpRequest→mcpResponse and
buildRequest→buildOut — the same seam apps/books and apps/company use, attached
to routes the router already carries so it can never contradict the router.
Two map literals became named structs to make that possible, with ALPHABETICAL
fields because encoding/json writes a map in sorted key order. The pins assert the
marshalled BYTES, not a status code: TestMCPEnvelopeIsByteIdentical over all four
envelope shapes (result, result with a null id, error, parse-error with no id to
echo) and TestBuildReceiptIsByteIdentical. Naming a shape did not move it.
The refusals themselves were re-derived from zip v1.18.11 SOURCE rather than
inherited as prose, and three of their file:line citations did not land:
op.invoke's unconditional ErrBadRequest on an unparseable body is typed.go:242
(cited 243), and the cmp.Or(op.Status, 204) a nil Out stamps is typed.go:305
(cited 308). A citation nobody can land on is how a refusal stops being
re-checkable — which is the whole reason the reason is written down. Both
refusals stand, unchanged, now verifiable.
Described ops stay 14 of 16, deliberately: Register buys the shape, never the
prose, and a description invented for a route nobody typed is worse than none.
17 of the subset's 69 published properties are now bare for the same reason —
zipdoc lifts field comments off TYPED ops only. Recorded, not glossed.
openapi.yaml also picks up a drift that was ALREADY on main and is not mine:
02827405 regenerated plugin/o11y/openapi.json (/v1/event/error → /v1/event/api)
without regenerating the fleet golden, so the published document advertised five
operations under /v1/event/error/{wildcard1} that the router no longer serves and
omitted the five /v1/event/api/{wildcard1} ones it does. The drift gate was red on
main. This golden is regenerated from source, never hand-merged, so the repair is
the generator's output rather than an edit to a neighbour's app.
Gates: apps/tools green under the Makefile env (baseline green, still green),
zipdoc -check clean, openapi-weave clean, plugin/tools/openapi.json and
openapi.yaml regenerated from source.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The Tasks data surface admitted any request carrying X-User-Id. cloud's
identity boundary mints exactly that with NO X-Org-Id whenever a validated
token's homeOrg() is empty (auth_identity.go — no `orgs` claim: a
pre-IAM-v1.33.0 human JWT, a non-KMS machine token), on purpose, so that every
org() gate refuses it rather than guessing a tenant.
apps/tasks was the one that did not. An empty org is the ZERO Principal to the
engine, i.e. the shared UNSCOPED store (hanzoai/tasks store/principal.go) — so
an org-less caller registered a namespace and a DIFFERENT org-less caller
listed it back. One store, every principal cloud could not resolve an org for.
The fix is not a better local check, it is the removal of a local check:
principal.OrgOf is the ONE decision the cloud data plane makes about a request
and an org, and it takes plain strings precisely so a reader holding headers
rather than a *zip.Ctx asks the same function. Refusal status, content type and
body are byte-identical — only the admitted set narrows to what every sibling
subsystem already admits.
No route moved: plugin/tasks/openapi.json regenerates identical (28 ops,
0 described — the wire refusals recorded at the mount still stand, and
typed_wire_test.go still measures all four).
Also: two file:line citations in the refusal record pointed at lines that had
already moved. Replaced with the symbol, which does not drift.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The team SPA's wire is identified by its own keys (snake_case distinct_id,
numeric epoch-millis timestamp — the canonical wire spells distinctId/time),
so the ONE canonical decode now dispatches it by shape (isTeamArray,
positive-signal only: a canonical array can never be mis-read as team).
/v1/event/collect stops being a second wire and joins the sunsetting
caller-owned aliases — the published Team SPA appends /collect to its
collector URL, so the PATH lingers on the $source='team' sunset metric,
but it binds the same decode; a rebuilt SPA pointed at /v1/event needs
nothing else. The old TestCanonicalWireSilentlyDropsTeamBatch pinned the
exact failure this fold fixes — inverted into the proof the fold works.
The Sentry wire drops its /error segment: /v1/event/api/<project>/… is
what a real SDK produces from a DSN of …/v1/event/<project> (SDKs insert
the api segment), mapped onto the /v1/o11y/api ingest routes. plugin/o11y
openapi regenerated; manifest o11y leaf /v1/event/error -> /v1/event/api.
The whole surface is now: POST /v1/event (product | team | LLM-obs, by
shape) + /v1/event/api/… (Sentry DSN wire) + the sunsetting /collect.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The leaf lasted one commit. /ingestion said on the path what the payload
already says in its bytes, so the door now decides by shape: handle offers
every authenticated POST /v1/event body to the observability plane FIRST
(cloud.ObsEventIngest, installed by o11y's mount), which claims LLM-obs
ingestion batches — a batch whose EVERY element carries a recognised type;
a product CaptureBatch spells {"batch":[…]} too and has none, so ambiguity
always loses to the door's own wire — and declines everything else
untouched. One receipt shape (CaptureResult) either way.
Only the canonical door's FULL lane offers: obs events are tenant data,
so the anonymous and reduced projections never reach that plane, and the
other doors never consult the claim (all pinned by test). The o11y typed
op is gone with its route — the door's identity lives with the door.
manifest keeps /v1/event/error (the Sentry wire) as o11y's one deeper
leaf; analytics owns the root.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The work list was the ARTIFACT, not a grep: every operation in
plugin/{ai,destinations,dns,licensing,runtime,templates}/openapi.json
publishing neither description nor summary — 38 of 38, the set that projects
to nothing at all. No prose, no MCP tool, no CLI command, no typed SDK method.
Nine are now typed ops. templates goes 5 of 5, destinations 4 of 5, and both
are GATED rather than promised: untypedByDesign + TestEveryRouteIsTypedOrNamed
+ TestEveryTypedOpIsDescribed read the LIVE router of the real Mount, so the
next route added here is typed by default and a stale reason goes red.
The other 29 are FOUR registrations. ai, dns, licensing and runtime each
publish seven operations that are one All("/…/*") apiece, exploded across the
methods by the untyped projection: greedy wildcard (*1 to fiber, {wildcard1}
to the document — a bound field and a published parameter that cannot agree),
verbatim upstream status through c.Bytes(res.StatusCode, …), verbatim
Content-Type, and no All[In, Out] to hang seven ops on. Two are not cloud's to
type at all — ai is hanzoai/ai's beego tree behind zip.AdaptNetHTTP, licensing
is hanzoai/licensing's http.Handler behind the same. Each refusal is written
at its registration now, not only in LLM.md.
WIRE PRESERVED, and the two places it could have moved silently:
- url:"-" on every body-only field of both write ops. zip's binder fills an In
field from the QUERY as well as the body, so a converted POST starts taking
?slug= — which on publish redirects the write to a name the body never
asked for. c.Bind read the body and nothing else.
TestTheQueryStringCannotRedirectAWrite is the measurement.
- destinationTest carries POINTERS in alphabetical order. The route reports a
platform rejection as DATA at 200, so it answers {ok,error} and
{ok,sent,message}; a non-pointer with omitempty drops a real "sent": 0, one
without adds "sent": 0 to every failure, and the map it replaced marshalled
its keys sorted.
POST /v1/destinations/{platform} stays untyped and the reason is a wire fact,
not a preference: its body's property NAMES are chosen at request time by the
addressed platform's Spec, and toStr accepts each value as a string, a number
OR a bool precisely so a console may send a numeric pixel id — a typed string
field turns today's accepted {"pixel_id": 123} into a 400. It declares both
bodies through openapi.Register instead, so the document stops saying it takes
no body, which was the one thing it cannot work without.
Latent defects this surfaced:
- Neither package installed cloud.Bridge on its own prefix. Both worked only
because Serve installs one app-wide, and both packages' own tests mount on a
bare zip.App — so the moment an op became typed it read no org. Installed on
each subtree, ahead of the leaves, and pinned by
TestTheBridgeIsInstalledAheadOfTheLeaves.
- main was RED before this change: apps/agents/routing_http.go calls
cloud.Request and was never added to allowedRequestUses, so
TestRequestEscapeHatchIsPinned failed on a clean checkout. Pinned with its
real reason (the X-Target-Key claim header).
- destinations' collection root was declared as the group's EMPTY leaf, so the
document published /v1/destinations/ — a path this API has never served.
Declared on the parent with a non-empty leaf; the trailing slash is gone from
openapi.yaml.
- Typing is what makes a package enter the FLAT schema namespace, and both
collided on entry: Status is apps/plugins', Template is apps/guide's. The
published name keeps it, the unpublished one qualifies — DestinationStatus,
DestinationField, StarterKit. Go-level renames, no wire movement.
- plugin/ai publishes seven {wildcard1} operations and none of the ~200 real
AI routes, so chat completions reach no SDK and no MCP tool list. Named at
the mount; the fix belongs in hanzoai/ai.
zip v1.18.11 also retires failure mode #7 for us: hasRequestBody skips an input
whose every field is a path param, so POST …/test types with no phantom body.
described: destinations 0 -> 4 of 5, templates 0 -> 5 of 5.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Thirteen operations published NOTHING — no prose, no schema, no MCP tool, no CLI
command, no SDK method. Six of them were reads that only ever needed their tenant
and their query string, and they are typed ops now: /v1/analytics/{overview,
timeseries,top}, /v1/errors, /v1/insights/{events,health}. Their In/Out are real
structs with per-field prose, so `errorRate` is documented as a ratio and `pct` as
a share of the WINDOW rather than of the rows returned — 19 schemas, every property
described, gated by TestEveryPublishedFieldIsDescribed.
The wire did not move. bindURL fills the same three window fields and the same
limit off the same query string, and an unparseable value still leaves the field at
its zero — which is what strconv.Atoi's discarded error already did, so `?limit=abc`
is still one caller's typo about one field and not a 400. The 403/400/503
precedence is unchanged and now pinned for /v1/errors and /v1/insights/events,
which nothing read back before.
The other seven stay untyped and each says why AT ITS REGISTRATION, measured rather
than asserted. GET /v1/analytics/health answers 503 CARRYING the degraded report as
its body; zip stamps a non-nil Out with cmp.Or(op.Status, 200), WithStatus refuses a
non-2xx, and a nil Out is stamped 204 over anything written from inside — so the
status and the body are one answer a typed op cannot give. The six ingest doors
share ONE admission decision resolved from facts that never reach a typed op: the
presented credential, the client IP and socket peer the anonymous rate caps key on,
DNT/Sec-GPC, and the RAW body length that is the anonymous lane's 64 KiB -> 413
bound — invisible to a typed op and far below the fleet's global BodyLimit. Four of
them add a second blocker: the canonical wire is polymorphic (object | array |
{batch:[…]}) and the team SPA's is a bare array, bodies zip's op.invoke would 400
where they answer 200 today. TestArrayBodiedDoorsStillAnswer200 measures exactly
that, so the day zip can declare a polymorphic body this is a test away.
Two latent defects fell out of the conversion and are fixed here:
- plugin/analytics/main.go declared no Prefixes, so MountPrefixes fell back to
the /v1/<name> convention and five of this app's six prefixes were outside
anything it could gate — cloud.Declare attributed /v1/errors, /v1/insights/*
and /v1/event to NO subsystem for tracing and price lookup, and scope.Use could
install nothing on them. That is the pricing defect one app over.
- apps/analytics installed no cloud.Bridge of its own, relying entirely on
Serve's app-wide one, which the package's own test harness never runs. A typed
op reads its tenant from principal.OrgFrom, so that reliance was the difference
between an org and a permanent 403.
And one collision refused before it shipped: SeriesPoint is already apps/admin's
{t,value} launch-board point, and the schema namespace is FLAT across the fleet —
one name with two shapes is what openapi.Weave rejects. analytics yielded, since
its schema was not yet published: UsagePoint.
TestRoutedPostSetIsExactlyTheDoors now reads GetRoutes(true). fiber keeps
middleware in the same stack as routes and reports it under every method at its
prefix, so the unfiltered read was never "the POST surface" in the real binary
either — Serve's own app-wide middleware has always been in it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every one of provisioning's 28 operations was route-only: no description, no
summary, no request shape, no response shape, and therefore no MCP tool, no CLI
command, no SDK method and no schema. The cause is one line, and it generalises:
for _, kind := range kinds { app.Post("/v1/"+kind, create(s, kind)) ... }
A computed path is not a constant, so cmd/zipdoc refuses it outright ("route path
is not a constant string, so the operation has no identity to document"), and a
handler returned by a FACTORY is a call expression with no doc comment to lift.
A loop that registers N routes publishes prose for NONE of them however well the
handler is commented. So the paths are spelled out per kind — one declaration per
published operation, which is what every projection keys on anyway.
21 typed: list, get and delete for each of sql, kv, datastore, docdb, vector,
search and s3. The wire is byte-identical. The listing's Out is a NAMED SLICE
(provisionedList []provisionedSummary), not an envelope struct, because the wire
is a bare JSON array — the envelope was the natural typed shape and a silent
break; TestTypedReadsKeepTheirWire asserts the empty listing's BYTES are `[]`.
The delete keeps its 204-with-no-body (a nil Out on an unnamed Out type).
7 refused, all the same fact once per kind: POST /v1/<kind> runs the pre-provision
balance gate and renders a denial through cloud.DenyResource, whose body is the
fleet's NESTED {"error":{code,message}} at 402/503. A typed op can only refuse by
RETURNING an error, which zip renders as its flat HTTPError, and writing the
nested body inside the op does not escape it either — a nil Out makes zip stamp
cmp.Or(op.Status, 204) over the 402. Gating in middleware would move today's
400-on-a-bad-name to a 402. The refusal is GATED, not prose: untypedByDesign +
TestEveryRouteIsTypedOrNamed read the router of the REAL routes() and require the
two ledgers to SUM to the served surface. The seven still DECLARE their bodies
through openapi.Register, so provisionRequest/provisionResult reach the document
and an SDK caller has somewhere to put the name.
Tenancy could not go through principal.OrgFrom, and that is a wire fact: tenant()
folds the org through sanitizeOrg — the slug every physical name, S3 bucket and
tenant-<org> namespace is keyed on, so a read that skipped the fold would look in
a different bucket than the create wrote — and buckets an ORG-LESS SuperAdmin
under the literal "admin" org, which OrgFrom refuses outright. tenantOf reaches
the request (cloud.Request, pinned with that reason) and asks the SAME tenant()
the untyped create beside it uses, so one surface cannot key its tenancy two ways.
Two more found by typing:
- dedicated.go's header still called sql and kv shared kinds; they moved to the
dedicated strategy in the same file.
- the escape-hatch pin was red on main — apps/agents/routing_http.go's
claimKeyOf called cloud.Request with no entry in allowedRequestUses, so the
gate that exists to stop that hatch growing quietly had itself grown. Closed
concurrently by the tools pass (30adfdce); this rebase takes that entry.
Mount now fails when its Router does not expose the op registry, rather than
serving 21 reads no projection knows about. The 14 :name operationIds take the
_by_name -> _name rename typing brings.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/plans published fifteen operationIds and NOTHING else: no schema, no prose,
no MCP tool, no CLI command, no SDK method. An untyped route is invisible to all
five, and this surface was 15 of 15.
The premise that held it back — "a typed op cannot proxy the bundle's bytes
verbatim" — is false here for the reason it was false one subsystem over in
apps/pricing: apps/goja already re-marshals the bundle's answer with Go's
encoding/json (Host.DispatchWith, json.Marshal(m["body"])) before any handler
sees it, so what the raw pass-through wrote was never the JS engine's bytes, it
was Go's, keys sorted. Each Out declares its keys in that sorted order or carries
the value as json.RawMessage, and apps/plan/wire_test.go PROVES byte-equality
against the live router rather than asserting it: all 12 sections × 3 identity
shapes (anonymous, forged X-Org-Id, validated member), plus both parameterised
addresses over EVERY plan id the shipped catalog holds.
Opaque catalog values are json.RawMessage, not map[string]any. @hanzo/plans owns
the shape of a plan, a tier and a schema document; a Go struct restating one is a
staler second source that silently drops what the catalog adds. zip v1.18.9+ asks
whether a type marshals itself before asking what it is made of, so a RawMessage
publishes {} — "any JSON", the only true thing to say — while map[string]any
publishes additionalProperties:{"type":"object"}, which the first
"priceMonthly": 20 in the catalog refutes. A false schema is worse than a thin
one.
TWO RESIDUAL DELTAS, recorded and pinned rather than glossed:
- Content-Type gains "; charset=utf-8" — what the health probe and every zip
error on this surface already sent. JSON is UTF-8 by definition (RFC 8259)
and charset is not a registered parameter of application/json.
- A non-200 body gains zip's `status` field beside the bundle's own message.
The status is the bundle's and the message is the bundle's, byte for byte,
under the same key (dispatchErr); returning an error is the ONLY path a typed
op's refusal can take. Same trade main already took for
GET /v1/pricing/model/{name}, whose 404 is equally first-class.
TestResolutionIsByteIdenticalForEveryPlanInTheCatalog asserts the error body is
EXACTLY {status,error} and nothing more, so a third difference cannot appear
quietly.
LATENT DEFECT, found by typing and fixed: the subsystem is named "plan" and
serves "/v1/plans", so MountPrefixes' /v1/<Name> default covered nothing it
registers. Measured: SubsystemOf("/v1/plans") was "" and PriceOf undeclared — the
tracing and price index cloud.Declare builds attributed every request on this
surface to NOBODY — and any middleware the subsystem installed, including the
typed-op Bridge that carries the validated org, landed on /v1/plan and never ran.
plugin/plan/main.go now passes manifest.PrefixesFor("plan"), the same one source
the host routes by.
The tenant is never an In field: catalogTenant reads principal.OrgFrom, so an
anonymous caller and a forged X-Org-Id both read the public "hanzo" catalog and a
validated reseller reads its own. TestTheTenantIsNeverAnInputField gates it off
the published document, where such a field would become visible to every SDK.
Measured after: plugin/plan/openapi.json 15/15 described (was 0/15), 9 published
schemas, 15 MCP tools all described on the running binary, 15 ops on the call
plane. openapi.yaml rewoven. The two parameterised ops take the documented
operationId rename (_by_id -> _id); no path, status or field name moved.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The tool plane's whole subset was prose-blind: every one of its 16 operations
reached openapi.yaml, every generated SDK and every MCP tool list with no
summary and no description, so a model choosing a tool and a developer reading
the SDK both got a bare operationId. Fourteen are now typed ops — one registry
entry each, which is what makes an operation an OpenAPI description AND an MCP
tool AND a CLI command AND a typed SDK method rather than only a route.
The two that stayed raw stayed for the wire, and both are now GATED rather than
claimed. POST /v1/tools/mcp is deliberately body-tolerant: a malformed body is
HTTP 200 carrying JSON-RPC -32700, which op.invoke's unconditional 400 cannot
express, and its request/response are envelopes whose shape depends on `method`.
POST /v1/plugins/build answers 422 carrying the build diagnostics as a domain
body, and a typed op's only refusal is a returned error, which zip renders flat
with nowhere to put the source that failed. untypedByDesign + the two ledgers
summing to the served surface make a third refusal a deliberate edit.
The three ?activated / ?all filters stayed STRINGS. These routes compare the raw
query value to the literal "true", and zip's setScalar reads a bare ?activated
and ?activated=1 as true — a bool In would answer with a different set of tools
for the same URL. That is the whole conversion's rule: describing a route is not
licence to move it.
Also closed the response half the op gate cannot see. Tool, Skill, MCPServer,
AuthoredPlugin and Price are store rows, so their properties reached every SDK
and every MCP inputSchema bare — `hasSecret` a flag with nothing saying the
VALUE is never returned. All 52 published properties carry prose now, gated.
Two latent defects fell out. The tools test harness mounted the plane with no
cloud.Bridge, which was invisible while every route was untyped and would have
made all fourteen typed ops answer 403 to a request carrying a valid org.
And TestRequestEscapeHatchIsPinned was already RED on main: apps/agents/
routing_http.go:61 reads the machine claim-key header through cloud.Request and
was never added to allowedRequestUses — named now, with the reason.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
metering.Usage carries three amount sources with a documented precedence — the
typed money.Amount wins, then micro-USD, then whole cents — and one function
resolves them. Client.Record has always guarded on that resolved value.
ResourceMeter.MeterUsage asked its own version of the question:
if u.AmountCents <= 0 && u.AmountMicros <= 0 { return }
Two of the three. So a Usage whose cost is carried ONLY as a typed Amount — which
is exactly what a per-token 18-decimal caller sends, zen among them — returned
early and never reached Record. No error, no log, no row, no debit. The one place
in the fleet that disagreed with itself about what money is, and it disagreed by
dropping it.
Usage.Money is now exported and both callers read it, because "is there anything
to bill here?" is the same question wherever it is asked and asking it any other
way gets a different answer.
TestResourceMeter_MeterUsageBillsAnExactAmount meters $0.0025 with no cents and no
micros set. It records 0 usages against the old guard and 1 against this one. The
amount is deliberately sub-cent: it survives only because it is exact, which is
the entire reason the typed field exists.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The tasks product publishes four addresses and describes none of them: /v1/tasks,
/v1/tasks/*, /tasks and /tasks/* — 28 operations in plugin/tasks/openapi.json, 0
with a description, so no schema, no MCP tool, no CLI command and no SDK method
for any of them.
None can become a typed op inside cloud, and the reason is the wire in every
case. /v1/tasks answers 307 with a Location, which a typed op has no vocabulary
for. /v1/tasks/* is one route over 64 engine operations this router never sees:
hanzoai/tasks matches them by path SEGMENT inside its own ServeMux, their inputs
are anonymous structs local to that module's handlers, and the engine hands cloud
its surface only as http.Handler — its programmatic seam (View + three *ForOrg
helpers) reaches 13 of the 64. That one route also carries four content types at
once, 12 of its verbs deliberately run on a malformed body a typed op would 400,
and its error envelope carries `code` as a number where zip's carries `status`.
/tasks/* is the SPA: bytes under their own content types, not JSON.
So the record at the mount states each refusal and typed_wire_test.go MEASURES
each one, rather than leaving the claim as prose nobody re-runs. The place these
operations become typed is hanzoai/tasks, which owns the surface; a second copy
of its route table living here would be free to drift from the one that answers
the requests.
Also: LLM.md's opaque-product count re-measured over the committed subsets (3
wholly opaque — dns, licensing, sentry; bot has left that list, 20 mixed), and
three doc pointers to the deleted clients/tasks package corrected.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The count in the commit before this one was wrong, from the same mistake this
file keeps warning about: it was measured with a hand-written method filter
(get/post/put/patch/delete/head/options) instead of read off the document, and
`trace` fell out of it. `app.All` registers nine fiber methods and openapi.From
publishes seven — the five body-bearing ones plus OPTIONS and TRACE — so five
wildcards yield 35 operations, not the 30 I wrote or the 25 a REST reader assumes.
The sum check in typed_wire_test.go was already derived from the live document and
so was green either way; only the prose beside it was false, which is the failure
this repo cares about, since that prose is what the next reader measures against.
Reading the methods instead of assuming them also surfaced a fleet-wide fact worth
more than the correction: 27 operations across 10 packages publish `trace` — exec
8, iam 5, tasks 4, o11y 3, base 2, and one each in ai, dns, licensing, runtime,
websearch — every one from an `app.All` catch-all, because All means all. So
openapi.yaml, every generated SDK and the MCP tool list offer HTTP TRACE on ten
products including the identity plane. Recorded with the command that re-finds it
and deliberately not fixed here: the repair is one decision in openapi.From's
method filter, it moves the document for ten packages at once, and regenerating
nine other subsets inside an iam change is exactly how a concurrent agent's work
gets clobbered.
plugin/iam/openapi.json still regenerates byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The whole product is five `app.All` wildcards relaying iamserver.Handler(db) —
github.com/hanzoai/iam's entire standalone zip app, 94 typed ops of its own,
adapted to net/http. So zero of the 25 undescribed operations convert, and the
reason is structural rather than a backlog item: one registration serves an OPEN
set of sub-paths; the bytes, status and Content-Type are the nested app's own
(its Guard's flat {"status":401,…}, /login/oauth's 302 + Location); and the oauth
token/introspect/revoke endpoints take application/x-www-form-urlencoded, which
zip's op.invoke jsonenc.Unmarshals into a 400.
That refusal is now a GATE, not a paragraph. untypedByDesign is keyed by PATH
rather than METHOD /path because one app.All refuses for every method at once —
keying by method would state one fact six times and let five copies rot — and
TestEveryRouteIsTypedOrNamed expands it over the methods the document publishes
and checks the sum (0 typed + 30 named = 30), so a sixth wildcard, a narrowed
wildcard, or a route added here as a raw handler all go red.
TestTheRelayIsWhyNothingIsTyped drives the live mount and asserts the two bodies
cloud never composes: the nested Guard's envelope and the minted OIDC discovery
document.
Reading it that closely found a live defect. mountFailClosed iterated Prefixes
alone while safeMount also registered the root /.well-known/*, so the degraded
surface was strictly SMALLER than the mounted one — and the terminal handler in
every plugin binary is webui.Mount's `/*` console catch-all, with /.well-known
outside the console's apiPrefixes. With IAM broken, GET
/.well-known/openid-configuration — the FIRST call every relying party makes —
answered 200 with the SPA's HTML instead of the honest 503, so an OIDC client
parsed a web page as its discovery document. Both halves derive from one
patterns() list now, and the test checks the derivation as well as the statuses.
Three doc comments were saying untrue things and are corrected, since prose is
the product surface here: the package header claimed iamserver.Route registers
"ZIP-NATIVELY … no net/http adaptor round-trip" (safeMount's own comment says
the opposite, and is right), and Prefixes claimed to be "the ONE list" handed to
MountAll by apps.Wire(), a composition root that no longer exists — the host's
list is manifest.Apps' iam row, deliberately separate so cmd/cloud stays light.
plugin/iam/openapi.json regenerates byte-identical: the wire and the document are
untouched. LLM.md records iam as the fourth wholly-opaque product and why closing
it is COMPOSITION (merging the nested app's own document at its prefix) and not
typing — including the two facts that gate it, measured: hanzoai/iam v1.33.26
ships no zipdoc_gen.go and no zip.Describe anywhere, so those 94 ops carry
summaries and no descriptions; and cloud's own /.well-known/openapi.json plus
agentskills' /.well-known/agent-skills/* win under iam's root wildcard only
because zip matches the most specific pattern regardless of registration order.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The code-interpreter surface publishes 56 operations and not one carries a
description, a schema, an MCP tool, a CLI command or an SDK method. It is not a
backlog item: Mount hands all 8 paths to httputil.NewSingleHostReverseProxy and
the sandboxed executor supplies every byte, every Content-Type and every status,
so there is nothing here to describe and four wire facts a typed op would move —
verbatim upstream status (zip answers its own declared one, typed.go:305-311),
response fields this repo never named (an Out drops them), a multipart
/v1/upload body (typed.go:242 jsonenc.Unmarshals every non-empty body) and a
byte-bodied /v1/download/{id} (typed.go:311 always c.JSONs). 16 of the 56 are
OPTIONS/TRACE, which zip has no typed registrar for at all. openapi.Register is
refused too rather than reached for: it could only publish a guess at
@librechat/agents' contract, and Binary's application/octet-stream is not a
multipart envelope.
So the refusal becomes a gate. typed_wire_test.go crosses two closed lists
(untypedPaths x servedMethods) into the same 56 addresses the document uses and
fails three ways: a route neither typed nor named, a reason naming an address
this mount no longer serves, and the day one of them becomes typable. Eight
sub-tests measure the wire facts through the REAL Mount rather than asserting
them. Both directions were mutation-checked: a fifth prefix in exec.go and a
stale ledger entry each turn it red.
Two findings the pass surfaced, recorded in LLM.md rather than fixed here:
- The migration's own inventory cannot see this app. Every documented grep
anchors on `("` or `("/`, so `app.All(p, h)` in a range over prefixes matches
nothing — apps/exec reads ZERO under both commands while serving 56
operations, which is why it has never appeared in a tranche. Same shape in
apps/knowledge (subsystem.go:61-69), apps/iam (258-259, 282) and
apps/commerce (mount.go:551). Count operations, not lines.
- plugin/exec/main.go told the next reader to "edit the spec below directly".
There is no spec in main.go, and the one it means is generated by
`make -C apps/exec openapi` and must never be hand-edited. Corrected here;
110 sibling mains still carry the sentence.
exec.go gains only a comment; the regenerated subset is byte-identical, which is
the honest measurement: 56 operations, 0 described, before and after.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
28edc6e0..3489bc0b moved o11y's ingest onto the one /v1/event family and did not
regenerate plugin/o11y/openapi.json, so five served operations were published
nowhere: /v1/event/error/{wildcard1} on all five methods (mountEventFamily
registers All(), and the handler's own method gate 404s what it does not accept —
which is why the document lists five, the same convention every other All() route
here follows).
Caught by the drift gate on the next regeneration, which is the whole point of it
regenerating FROM SOURCE rather than comparing two derived artifacts: nothing about
the committed subset and the woven golden disagreed, because both were built before
the routes existed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TWO THINGS, ONE CAUSE — zip v1.18.11.
The route. PATCH /v1/admin/catalog/providers/{name} was refused because `overrides`
is an RFC 7386 merge patch stored and echoed verbatim, which pins its Go type to
json.RawMessage, which zip published as an ARRAY OF INTEGERS. typed_wire_test.go
had pinned that lie deliberately, with instructions: "If zip now describes a raw
JSON value as one, the providers/{name} reason is stale — type the route." v1.18.9
describes it as one. The test went red, and this is the route it asked for.
The overlay upsert is now applyPatch(ctx, kind, id, patch) with no request in
sight, called by the typed op AND by the *zip.Ctx door the models/* wildcard route
still needs — one implementation, two doors, so they cannot drift. The patch
fields stay POINTERS: absent must differ from a zero the caller meant, and
encoding/json leaves a pointer nil for an explicit null too, so {"enabled":null}
and {} arrive identically — which is what this route already did.
One residual delta, recorded in ops.go rather than hidden: zip decodes before the
handler runs, so a non-admin sending malformed JSON now sees 400 where the raw
handler answered 403. zip's authorizer is deliberately post-decode (it authorizes
the decoded value, so the decision cannot diverge from execution), so this is not
avoidable while the route is an op. It reveals only that the body was unparseable.
The four apps. Declaring that op exposed a defect in v1.18.9 itself: its published
requestBody was ABSENT. The input is a path param plus an embedded unexported
patch body, and every projection asked "what fields does this type carry" with its
own loop over NumField — so it skipped the embedded type on IsExported and saw
only `name`, which IS a path param. wireFields (v1.18.11) is the one function that
knows encoding/json's promotion rule. Regenerating with it corrected four apps:
pricing gains the request body it always accepted;
admin publishes SaaSMetrics' fields flattened, where it had a property
literally named "SaaSMetrics" that the wire never sends;
agents gains 117 lines of response fields that were published nowhere;
visor gains 81 lines, same cause.
Every one of those was a document describing a shape the service does not have,
in openapi.yaml and in every SDK generated from it. Found by READING the artifact
after the bump instead of trusting that the bump was an improvement.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The bump alone changes 23 published artifacts, because zip v1.18.9 fixes what the
projection SAYS rather than what any route does. No wire moves.
phantom request bodies 41 -> 9. A POST binding its whole input from the path
published a required body whose only property was the path param, so every
generated SDK gained an argument the caller must build to repeat a value it
already passes in the URL. The 9 left are the raw-body family (git-upload-pack,
bank-statement import, a deck upload) — they eat bytes, not JSON, and owe a
binary content type via openapi.Binary rather than an empty object.
time.Time stopped publishing as a $ref to a schema with no properties and now
says format: date-time. Its fields are unexported, so reflection over them
described nothing: every timestamp in every generated SDK was untyped.
summaries lost their embedded line breaks — one sentence on one line, which is
what the spec, the CLI's one-line help and an SDK's first docstring line all
want.
imported types' FIELD docs reach the document at all. zipdoc matched the parsed
and type-checked views of a struct by byte offset, which only agrees for a
package loaded from source; an imported type's position comes from export data
with a synthetic offset. So every op whose In and Out live in the call plane
published a description and zero field descriptions.
AND ONE STALE PUBLISHED SURFACE, which the regeneration exposed rather than
caused. plugin/authz/openapi.json documented GET, POST and DELETE
/v1/authz/policies. hanzoai/authz v1.10.15 does not serve them, and says why in
serve/mount.go: the grant set belongs to IAM, "a second writable copy behind this
surface would be a second source of truth for who may do what". So cloud was
advertising a writable authorization-policy API that had been deliberately
removed, and three methods in every generated client answered 404. Verified as
pre-existing by regenerating the subset at origin/main on the OLD zip: the same
three paths vanish.
That drift means the gate was red on main and stayed red. It is invoked
(hanzo.yml:134), which leaves the two ways it could have been red and unnoticed —
worth a look, not a guess.
Also: openapi-apps now honours OPENAPI_NEEDS_BROKER, which only openapi-check
did. The gate's own failure text says "fix: make openapi", and that fix routed
through openapi-apps, which mounts kafka, which fails closed with no broker — so
the single command told to repair a red gate could not run. One exemption list,
read everywhere it applies.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every git subprocess shares cloud's cgroup, and pack generation scales
with REPO size, not request size: an upload-pack of a multi-GiB repo is
a multi-GiB allocation the Go runtime cannot see or govern (GOMEMLIMIT
bounds only the Go heap), landing as a kernel OOM kill of the entire
API — the exact profile of the ~hourly exit-137 kills (multi-GiB spikes
faster than the metric step, fatal request never logged). GIT_CONFIG_COUNT
in the ONE subprocess constructor caps delta search (64m x 2 threads),
pack mmap (256m in 32m windows) and delta cache (64m): a few-hundred-MB
worst case per operation, traded against clone speed on the biggest repos.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/event was already the canonical product-event door (the analytics
doors table's target; its sunsetting aliases name it). The o11y ingest
kinds now live under the SAME family instead of their own path roots:
- POST /v1/event/ingestion — the LLM-obs batch (traces/observations/
scores), moved from /v1/o11y/ingestion (nothing called it yet).
- POST /v1/event/error/… — the Sentry wire. A DSN of
https://<key>@api.hanzo.ai/v1/event/error/<project> works as-is:
SDK-expanded api/<project>/envelope|store forms map onto the
/v1/o11y/api ingest routes, the bare form onto /v1/sentry, both
rewritten before the principal gate so its two existing ingest
exemptions stay the only exemptions. The family carries ingest ONLY —
no READ API is reachable through it (pinned by test).
manifest: deeper /v1/event/{ingestion,error} prefixes route to o11y
while analytics keeps the root and /collect — deeper prefix wins, so
neither app shadows the other (the reachability test proves it).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Picks up the hanzoai/kafka rework: offsets stamped in batch headers only
(sparse e18 sequences and holes behave like dense), poison messages
skipped instead of served (the insights outage class), commits accepted
for any non-negative offset (the 2^50 'plausibility' guard rejected every
real one), OFFSET_OUT_OF_RANGE instead of dead-position stalls, LeaveGroup
handled, response encoder grows to fit, request frames capped. Proven by
the repo's new in-process franz-go e2e and this package's interop test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/agents had 26 routes and 5 typed ops, so 21 of them were a route and
nothing else: no schema, no prose, no MCP tool, no CLI command, no SDK
method. 19 are now typed ops. The route table is unchanged — 31 operations
before, 31 after, none added, none removed — and the wire is byte-identical:
a differential harness drove 83 request/response pairs (every status, every
error body, every ?limit=/?after= edge, the 204s, the 201s, a real routed-run
claim) against origin/main and diffed them, and after canonicalising JSON
object key order the two dumps are the same bytes.
What the typed registry gained: descriptions 5 -> 24 on the subset, 419 -> 438
on the fleet golden, 32 new schemas, and the query parameters of every read
(?root, ?parent, ?status, ?project, ?limit, ?after, ?range) documented for the
first time — they were invisible, because a raw handler's c.Query() is not a
contract.
THREE LATENT DEFECTS, surfaced by typing:
1. cloud.Bridge was installed in mountTargets, THREE calls into Mount. fiber
runs middleware in registration order, so it never ran for any leaf
registered above it — this file's, mountSessions'. It was harmless while
only the target ops were typed, and would have 403'd every read here the
moment they were not. It is now installed once, at the top, before any leaf.
2. PATCH /v1/agents/targets/{id} shipped a request body schema holding `id`
ALONE. patchTargetIn embedded its mutable fields from a body struct, and
zip's schema walk takes only EXPORTED fields — an embedded field of an
unexported type is not one — so label/kind/status/capacity/host/spec/metrics
were absent from openapi.yaml and no generated client could send any of
them. Flattened: eight properties now, same wire (json promotes either way).
The same zip gap still shortens agentDetail and sessionDetail, where the
embedded view is genuinely shared and a second copy would be a field that
silently stops being sent; both carry a note pointing at the one fix.
3. Two fleet schema-name collisions the weave refused: `runView` also means a
platform run (apps/platform), `createReq` also means a git repo create
(apps/git). One name, two shapes would bind whichever an SDK read last.
Renamed here (agentRunView, createAgentIn) — a schema name is document-only,
never on the wire.
POST /v1/agents also declared 200 while always answering 201; it declares 201
now (zip.WithStatus), and DELETE declares the 204 it sends.
SEVEN routes stay untyped, each because typing would move the wire:
GET /v1/agents/sessions/stream an open SSE feed written by a loop that
outlives the handler; no In/Out is a feed
POST /v1/agents/:ref/run 502 carries the RECORDED RUN as its body,
and a balance denial answers the fleet
402/503 contract (cloud.DenyResource)
POST /v1/agents/sessions/:id/events \ the guard gate refuses a credential in
POST /v1/agents/sessions/:id/pause \ a transcript with 422 IN BAND, naming
POST /v1/agents/sessions/:id/resume / the rule, line and fingerprint of each
POST /v1/agents/sessions/:id/stop / finding; zip's error type carries
POST /v1/agents/sessions/:id/message {status,code,error} and no findings[]
All six of those wait on the same zip capability: a response body per status.
Verified with the gate env (CLOUD_KMS_MASTER_KEY_REF, -tags sqlite_fts5), not
bare go test: apps/agents green (baseline was green), zipdoc -check clean,
weave green, and every in-process consumer of this package — coding, link,
team, visor, cli, plugin/agents, cmd/cloud — still builds and tests green.
cmd/cloud gained no apps/* import.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/ml served seventeen routes and published seventeen paths with no schema, no
prose, no MCP tool, no CLI command and no SDK method — the whole cost of an
untyped route, paid seventeen times. Ten of them are now typed ops, so one
registration is the whole contract: the reads of all three Kubeflow-family
resources (models/jobs/experiments), their deregistrations, and the katib trials
leaf.
The seven that stay raw are wire-bound, and they are a GATE now rather than a
paragraph: untypedByDesign + TestEveryRouteIsTypedOrNamed hold the closed list
and its two ledgers must SUM to the surface the live router serves, so a route
added untyped goes red without anyone remembering to name it.
- the three creates answer 402/503 IN BAND through cloud.DenyResource, with the
fleet's nested {"error":{"code","message"}} contract. A typed op can only
refuse by RETURNING an error, which zip renders as the flat
{"status","code","error"} HTTPError — a NEW refusal class, distinct from the
multi-status gap: a non-2xx carrying a DOMAIN body. Moving the gate into
middleware does not rescue them, because it would run before the body decode
and turn today's 400-on-a-bad-name into a 402.
- PATCH /v1/ml/models/{name} relays an opaque RFC 7386 merge patch VERBATIM to
the Kubernetes API. Through map[string]any every number becomes a float64, so
{"replicas":1000000} re-marshals as 1e+06 and patches a float over an int.
- POST .../predict returns the predictor's own status, bytes and Content-Type.
- both /health probes answer 503 carrying the degraded REPORT as their body,
which is the point of a real probe.
Identity crosses the typed seam on the context and never as an In field: one
tenantFrom, pinned in allowedRequestUses with the reason it needs the REQUEST
rather than principal.OrgFrom — ml's boundary is a per-org(+project) KUBERNETES
NAMESPACE, and OrgFrom refuses an empty org outright, which would turn the live
org-less-admin "ml-admin" bucket into a 403.
Wire preserved, and measured rather than asserted. ONE view() now returns the
published mlResource for both the typed reads and the untyped create/patch, so
the shape cannot depend on which route served it; its fields are in ALPHABETICAL
order because it replaced a map[string]any and encoding/json sorts a map's keys,
and TestView asserts the marshalled BYTES. Spec and Status are pointers because
the wire distinguishes an absent key from a present-but-empty object, which a map
with omitempty does not. The two wire tests pin their own state (a fake client, a
deliberately nil one) instead of depending on whether the box has a kubeconfig.
Seven operationIds rename _by_name -> _name, which is what a route going typed
does to the generated SDK's method name. No path added, removed or moved: 1013
before and after.
Two latent defects surfaced and REPORTED rather than quietly fixed:
- dynForOrg documents itself as "the ONE federation seam" and no handler calls
it, so an org with a registered BYO cluster silently gets its models and jobs
on the home cluster. Wiring it changes which cluster a resource lands on and
owns a migration question, so it needs its own change.
- openapi-check is red on main in two apps nobody typed: plugin/authz loses
/v1/authz/policies on regeneration (authz is an EXTERNAL module, so that may
be this box failing to mount it — committing it would delete three documented
endpoints from every SDK) and plugin/tools retags 35 lines. Both reverted,
both written up in LLM.md.
Also dropped two parameters neither function ever read (tenant's service,
k8sErr's request) and split mount() out of Mount() so the routes can be exercised
over a state a test pins.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cd4de37e (automations) established the rule this pass had to be measured against:
a typed op must be addressable through its In ALONE. Over MCP and the ZAP call
plane there is no URL — zip passes the arguments object as the BODY with a nil
path map — so an op whose address reaches it only from the path works over REST
and nowhere else.
The two :id writes typed here are exactly the shape that risks it: a struct In
carrying an id field beside a body, decoded by an UnmarshalJSON of its own. They
pass, and for a reason worth writing down rather than trusting: ID is a field of
the type UnmarshalJSON decodes, so `"id"` in the arguments binds it, and over
REST bindURL then overwrites it from the path, which is the authority there.
Measured both ways rather than asserted. With `v.ID = ""` spliced into the
stakeholder patch — an In that does not receive its id from the body — this test
goes red naming what was lost ("cannot address a stakeholder over MCP ... the
REST wire survived and this projection did not"), while every REST pin in this
file, the byte-identity comparisons against the relay, TestHTTPEndToEnd and
TestFullLifecycle all stay GREEN. That gap is the whole reason the test exists.
newAppMCP is the harness that made it reachable, and it records the same fact
automations found: zip's projections of the registry (/mcp,
/.well-known/zip/op/) are routes on the APP, outside every subsystem group, so
the /v1/captable group's own Bridge never runs for them — cloud.Serve's root
Bridge is what gives them a validated org.
Tests only. No route changes, no wire changes: zipdoc -check is green and the
weave reproduces openapi.yaml byte for byte.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/captable was 17 typed of 31, and the fourteen relays were all held to be
untyped for one reason: the goja bundle validates with COERCING helpers, so a Go
struct would accept less than the route does.
Re-checked against goja/src/validate.ts, that reason is true of eleven of them
and not of three. The blocker is `num`/`intNum`/`optNum` — a number OR a numeric
string — and a float64 field cannot accept `"1.5"` NOR carry the token it
rejects onward, so it moves both what the route takes and which envelope refuses
it. Those eleven stay relays.
PUT /company, PATCH /stakeholders/:id and POST /rounds/:id/close carry no number.
Every field they take goes to reqString, optString or optDateString, and a
verbatim `scalar` carrier hands each token to the bundle unchanged — so the
bundle stays the ONLY validator, of what is accepted and of how it says no.
The carrier is a string KIND, so every projection describes these fields as
`string`, which is what they are; and it is NOT a pointer, because encoding/json
nils a pointer for an explicit `null` without calling UnmarshalJSON, which would
collapse the absent-vs-null distinction stakeholders.update writes columns on.
The relay's 413 survives: a typed op never sees the request, so the body size is
recorded in the input's own UnmarshalJSON and read back AFTER the tenant — which
keeps a 403 ahead of a 413 for the caller that has both problems.
writes_test.go proves it rather than asserting it: every case is sent through the
typed route AND dispatched on the bundle the way the relay did, and compared on
status, Content-Type and bytes — including a number where the bundle reads an
optString (a 200 storing "5", which a *string would have 400'd), a number where
it reads a reqString (the bundle's 400 with its `errors` list, which zip's
envelope has nowhere to put), null vs absent vs "" on the partial update, and a
body that is not an object at all.
Two orderings do move, both for a request with no valid tenant: malformed JSON
or an oversized body now answers before the 403, because zip decodes ahead of the
handler. Named in writes.go rather than left to be found.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The ONE command this file tells agents to trust filters untyped registrations with
`grep -v 'zip\.'` — a LINE filter standing in for a syntactic one. The
discriminator wanted is the RECEIVER of the call (`zip.Get(` package-qualified
generic vs `<router>.Get(` method on a router), so dropping every line that merely
CONTAINS `zip.` also drops an untyped registration whose handler argument names the
package on the same line: an inline `func(c *zip.Ctx) error`, or
`zip.AdaptNetHTTP(`.
Measured across apps/ at this merge: 660 by the documented command, 675 anchored —
15 hidden routes in 9 packages. Two of them read ZERO and are not zero:
apps/plan 3 /health, /resolve/:id, /entitlements/:id (plan.go:77-109)
apps/product 4 /v1/search-docs/{indexes,stats},
/v1/vector/{collections,stats} (product.go:80-141)
Every registration in both is an inline `func(c *zip.Ctx) error`, so the filter
eats all of them and the partition table has never dispatched either package. This
file already documents two errors in this same command, in the opposite direction —
a phantom sends an agent at nothing; a hidden route means nobody is ever sent, which
costs more. Both commands (the per-app loop and the fleet one) now anchor on the
call, and the third bullet under "Partitioning the remaining work" carries the
measurement.
Found because o11y read 7 against the 8 its own conversion recorded: the missing
one is `a.All("/v1/sentry/*", zip.AdaptNetHTTP(…` at apps/o11y/o11y.go:231. The
corrected command reproduces 8.
Also records two live openapi-check failures on main, under failure mode 1 where
that class already lives: plugin/authz/openapi.json publishes GET|POST|DELETE
/v1/authz/policies, which hanzoai/authz v1.10.15's serve.Mount does not register at
all (it serves only health/readyz/check) — three operations in openapi.yaml, in
every generated SDK and in the MCP list that no binary answers, with
manifest/apps.go:38 still routing the prefix at them; and plugin/tools/openapi.json
is a golden emitted by an older generator (four tags collapsed to one, an escaped
em dash). Neither is regenerated here: doing another package's subset inside an
unrelated change is how a concurrent agent's work gets clobbered.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Follow-on to the CLI-collision commit, using the seam apps/books added an hour
earlier. git's 24 refusals were correct as refusals and stay raw — the webhook's
HMAC covers the raw bytes it verifies before json.Unmarshal, smart-HTTP streams
x-git-* pack media, the UI renders html/template, and the ZAP adapters' error
envelope ({status:"error", msg}) is a shape zip's errorHandler cannot produce —
but "cannot be a typed op" had been read as "must be undocumented". All 24
published an operationId, tags and NOTHING else, which no SDK generator can tell
apart from a route that takes no body. The published forge webhook therefore
offered a delivery with nowhere to put it, and three ZAP procedures a repo name
with nowhere to put it.
Eight now declare the request they actually read, through openapi.Register in an
init (Register panics on a duplicate; routes() runs once per Mount):
POST /v1/git/webhook pushEvent
POST /v1/git/zap/{createRepo,getRepo,deleteRepo} zapProcReq
POST {/v1/git,}/{org}/{repo}/git-{upload,receive}-pack openapi.Binary
Pure description: no route, status, field or byte moves, and openapi.yaml gains
81 lines with ZERO deletions. The two ZAP procedures that read NO body
(listRepos, usage) are deliberately left silent — declaring one for them would
swap an honest silence for a fresh falsehood — as are the ref advertisement and
the twelve HTML pages.
No RESPONSE is declared, and the reason is worth keeping: the ZAP envelope's
data is repoView / []repoView / usageView, names zip's typed fold already
publishes as components off the typed /v1 ops, so reflecting them here through
openapi.schemaOf would put two derivations behind one schema name — precisely
what openapi.Weave exists to refuse. The pack responses and the HTML pages have
no seam at all: openapi.Binary is request-only by design.
declaredBodies + TestRefusedRoutesDeclareTheBodyTheyRead pin it in both
directions — a declared body that vanishes fails, and a body declared for one of
the sixteen that read none fails too — and both directions were verified red
before this was committed green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/git was already 24 typed / 24 refused, and every refusal re-verified here
against the handlers and zip v1.18.6 itself: the webhook's HMAC covers the raw
bytes it verifies before json.Unmarshal, smart-HTTP answers c.SendStream over
x-git-* pack media, the UI renders html/template through render(), and the ZAP
adapters' failure envelope ({status:"error", msg}) is a shape zip's errorHandler
({status:<int>, code, error}) cannot produce. Nothing there can be typed without
moving a wire, so nothing was.
What typing DID surface is a defect one projection down. A typed op is one value
with four projections; three key on the op's own identity, but the CLI keys on a
name zip SPELLS from the route, and commandName (zip/cli.go:253-311) keeps the
segments before the first path parameter and after the last one while dropping
everything between. So `mirrors` and `subscriptions` — the words that say WHICH
thing a DELETE removes — never reach the name, and all three of
DELETE /v1/git/repos/{name}
DELETE /v1/git/repos/{name}/mirrors/{id}
DELETE /v1/git/repos/{name}/subscriptions/{id}
derive `git repos-delete`. Two of git's 24 typed ops therefore have no command a
caller can reach, while the wire, the document, the MCP tool and the SDK method
are all correct and distinct. An untyped route has no command to collide, which
is why only typing finds this.
Measured with zip's own derivation over the committed subsets rather than a
reimplementation of it: 20 colliding names hiding 23 ops across 11 packages, in
four shapes — interior segments dropped (git, cloudflare x2, o11y), a collection
colliding with its own item (compliance, marketing, framework), PATCH and PUT
both spelling `update` (base, exec x4, iam x2, websearch), and one op at two
addresses differing only by the version segment isVersion strips (tasks x5).
git holds the worst single instance, three ops on one name.
The fix is commandName carrying the interior segments — not WithOperationID
here, which would make git's ids a special case of a general bug. Until it
lands, cliNameCollisions + TestCLINamesCollideExactlyWhereKnown pin the damage
in BOTH directions: a new collision fails, and a collision that has GONE fails
too, so the zip fix retires the list instead of outliving it. Verified red both
ways before being committed green.
Also recorded, both measured from the golden and neither fixable in this
package: nine host-gated root paths (the six root UI pages and three root
smart-HTTP routes, all falling through with c.Next() off the git host) are
published under the document's single `servers` entry https://api.hanzo.ai, so
every generated SDK gains nine methods that 404 against the host the document
names — GET / worst of all, whose operationId is the bare word `get`. And git is
the only app that parks its own principal instead of installing cloud.Bridge,
because Bridge carries the org alone while git's ops need the project sub-scope
and the acting SSH-key owner; widening the shared bridge is what makes that one
way again.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The four exclusion pins landed in fce62ae1 close the retype a reader reaches
for first (a non-struct In: takes any body, receives no path param). They do
not close the one that looks like a repair: a STRUCT In carrying an id field
plus an UnmarshalJSON that swallows the whole body. Nothing can fail it, and
bindURL still binds :id from the path, so the REST wire is preserved exactly.
Measured, not argued. With resume retyped that way the ENTIRE package suite
stays green — all four pins included. What it gives up is what typing is for:
- a tools/call and a ZAP by-name call carry every argument in ONE JSON object
and bind no path from it (zip mcp.go / call.go pass `arguments` as the body
with a nil path map), so an In that discards its own keys never receives the
address: the tool answered "run not found" for a run that exists.
- the projection then advertises a body of {"id": string} that this route has
never accepted — the MCP inputSchema and the OpenAPI requestBody both.
So the rule the exclusions rest on is sharper than "an In cannot bind both the
body and the URL": a typed op must be addressable through its In ALONE, because
for two of its four transports the In is the only channel there is.
TestOpsAddressThroughArgumentsAlone pins that. It exercises the channel on an op
that IS typed (a tools/call naming a run id must answer that run, and one
without it must not), so it is a live assertion rather than a dormant one, and it
would go red for all fourteen ops at once if zip stopped binding an address from
the arguments object. The two URL-addressed exclusions register no op, so no tool
is derived for them today — the moment one is, this test calls it and fails on
the fact the REST pins cannot see. Verified both ways: red under the retype with
the message naming what was lost, green without it.
newAppMCP is the harness that made it measurable, and it records a fact worth
knowing: zip's own projections of the registry (/mcp, /.well-known/zip/op/) are
routes on the APP, outside every subsystem group, so a group's Bridge never runs
for them — cloud.Serve's root Bridge is what gives them a validated org.
No route changes, no wire changes, no regenerated artifacts: zipdoc -check is
green and the registry is untouched.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
openapi.Binary landed minutes ago (books, cfd7b30b) and it changes what company's
two refusals cost. It does NOT make either one typeable — re-checked, not assumed:
zip's typed path still jsonenc.Unmarshals every non-empty body unconditionally
(v1.18.6 typed.go:232), so a typed In on the deck still turns 201 into 400, and
zip's HTTPError is still flat so /payment's nested 402/503 denial still cannot
survive a typed op. Both stay out of the registry, and typed_wire_test.go still
says why.
What changes is the DOCUMENT. Both routes published an operationId and a tag and
nothing else, which to every consumer is indistinguishable from a route that takes
no body and returns none. So every SDK generated off openapi.yaml offered a deck
upload with nowhere to put the deck, and neither call had a return type. That is
not an absent feature, it is a WRONG description, and it is worse than the
registry gap it stood in for.
POST /v1/company/fundraise/deck — request application/octet-stream string/binary
(OpenAPI's spelling for an opaque body, what a generator turns into a file
parameter), response deckOut.
POST /v1/company/payment — no request declaration, because the handler
genuinely reads no body; declaring one it ignores would be invention. Response
is the shared formationView every other action here answers with, so it $refs
the same component rather than minting a second shape for one wire.
deckOut replaces the handler's map[string]any literal so the declared type and the
emitted value are the SAME value — a map here and a struct in the document is
exactly how the two drift. Byte-identical on the wire ({"documentId":"..."}), and
TestDeckTakesRawBytes proves it unchanged.
Three things remain undeclarable, and each is named at the registration instead of
being glossed over: the deck's ?name= query (the untyped projection derives path
params from the router and has no vocabulary for query), /payment's 402/503 denial
bodies (apply states 2XX only), and field prose on deckOut (zipdoc lifts comments
off typed ops only, so Register publishes shapes without descriptions).
WIRE UNCHANGED. The spec diff adds requestBody/responses to two operations that
had neither; no path, operationId or status code moves.
Gate: apps/company ok, openapi ok, vet clean, go build ./... clean.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/company was already 20 typed ops of 22 operations. What it did not have was
a reason to STAY that way: "2 left, both permanent" was a paragraph, and a
paragraph cannot fail. A route added tomorrow as a raw func(*zip.Ctx) error
would have left the claim standing and the route invisible to OpenAPI, MCP, the
CLI and every generated SDK.
typed_wire_test.go makes it a gate — untypedByDesign + TestEveryRouteIsTypedOrNamed,
the same shape crm/compliance/ingress carry. It is not vacuous: dropping either
entry turns it red naming the route.
Both refusals were re-verified against zip v1.18.6 source rather than inherited:
- POST /v1/company/fundraise/deck — op.invoke unconditionally jsonenc.Unmarshals
every non-empty body (typed.go:232). That decode does not depend on the In
binding any field, so even an empty In cannot escape it: a PDF becomes 400
where the route has always answered 201. v1.18.6 has no octet-stream/binary
request declaration to decline the decode with.
- POST /v1/company/payment — a billing denial answers the fleet-wide NESTED
{"error":{code,message}} at 402/503 (cloud.DenyResource). zip's HTTPError is a
flat {status,code,error} and errorHandler is the only path a typed op's error
takes. Writing the nested body from inside the op does not help either: a nil
Out makes zip stamp cmp.Or(op.Status, 204) over the 402, and an error makes
errorHandler replace the body. Needs zip errors that can carry a body.
The op-level gate cannot see the other half of the surface, and that half was
broken: 41 properties of Formation, Founder, Filing, Genesis, Registration,
Signer and RoundInput reached openapi.yaml, every generated SDK and every MCP
inputSchema with NO description at all — they are store row types nobody had
written field prose on. A reader could see equityBps was an integer and nowhere
that it is BASIS POINTS of 10000. Every field now says what it is, and
TestEveryPublishedFieldIsDescribed keeps it that way.
Money units are the one thing NOT asserted: RoundInput's three float amounts pass
verbatim into the cap table's rounds.create contract, which does not document a
minor/major unit either, so the prose says what the field is and does not invent
one.
WIRE UNCHANGED. The regenerated openapi.yaml diff is description text only — no
path, operationId, requestBody, response or status-code line moves.
Gate: apps/company ok, openapi ok, vet clean, go build ./... clean
(CLOUD_KMS_MASTER_KEY_REF + -tags sqlite_fts5 + CGO_ENABLED=0, matching the
Makefile). Baseline before the change was the same green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/o11y is already fully typed — 12 ops on the /v1/o11y group, all 7 remaining
registrations re-read against their handlers and re-refused, every one wire-bound:
two verbatim-status VictoriaMetrics passthroughs (c.Bytes(status, body) — a typed
op answers its ONE declared status), three reverse proxies into the runtime
(query, query_range, sessions — there is no Go type for "whatever the runtime
answered"), and two text/plain Alertmanager receipts of which the POST
deliberately accepts an unparseable body, because a receipt that 400s makes
Alertmanager retry forever. Nothing here is convertible without moving the wire,
so nothing here was converted.
What WAS missing is that all of that lived in prose, and prose cannot go red.
typed_wire_test.go makes it a gate:
- untypedByDesign is the CLOSED list, keyed the way the DOCUMENT writes each
address, and TestEveryRouteIsTypedOrNamed reads the live router of the REAL
MountO11y — not a reconstruction of it — so a route added anywhere inside that
mount (or in the upstream module it ends with) is typed by default, and
dropping one out of the registry takes a deliberate edit with a reason. The
stale direction is gated too: a name for an operation o11y no longer serves.
Proven red by adding one route and watching it fail by name.
- TestEveryTypedOpIsDescribed holds the prose to the schema's bar, because that
prose IS the product surface — the OpenAPI description AND the MCP tool
description a model reads to pick the tool.
- TestUntypedRoutesKeepTheirWire measures three of the claimed wire facts on the
real router (the text/plain receipt, its 200 over a body that is not JSON, the
text/plain replay), so the refusals are evidence rather than assertion.
- TestIngestOpIsTypedButUnreachableWithoutADSN gates the one real gap here.
POST /v1/o11y/ingestion IS a typed op with prose — the test registers it and
reads it out of zip's registry, so that half is measured — but mountEventIngest
returns before registering it with no Datastore DSN, and `bin/o11y openapi`
runs with GIT_SSH_ADDR and nothing else, so the LLM-obs write path reaches no
SDK, no MCP tool, no CLI command and no published schema. Closing it is a
behaviour decision (zip.Post registers route and registry entry inseparably,
so publishing the op stops the path falling through to the order-70 wildcard);
this does not decide it, it fails the moment somebody does.
Zero document movement: `make -C apps/o11y openapi` and `go generate -run zipdoc
./apps/o11y/...` both regenerate byte-identically, which is the point — typing is
a description task and this change describes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/pricing is 32 operations, 30 of them typed ops. The remaining two are the
admin overlay PATCHes, and their reasons for staying raw were prose — a claim
about zip and openapi.translate, recorded once and then trusted. A claim about a
dependency rots, so both are now executable against the toolchain in go.mod: the
tests register the shape at issue on a throwaway app and assert the toolchain
still behaves as the reason says. A blocker fixed upstream turns the suite RED
and names the route that just became convertible.
Checking them also found the models/* reason materially understated. It read as
a preference about an unreadable parameter name; the measured fact is that typing
that route REFUSES THE WHOLE DOCUMENT. zip keys a typed op by the fiber pattern
(".../models/*") while the document keys the same route by its URI template
(".../models/{wildcard1}", because `*1` is not a legal template name), so
openapi.Fold cannot find the op's route and errors — and Spec builds ONE
document, so every other pricing operation goes down with it. An engineer reading
the old reason could reasonably have decided to accept the ugly name and turned
the whole spec red. The parameter-name half is still true and still blocking, and
is now stated as the SECOND half.
The providers/{name} reason gains the general form of its blocker: verbatim echo
pins the field to json.RawMessage, json.RawMessage publishes an integer array,
and neither map[string]any nor any is an escape because both re-marshal and sort
the patch's keys. The escape is a schemaOf that can describe an arbitrary JSON
value, which it has no arm for.
No route changes: the golden and plugin/pricing/openapi.json regenerate
byte-identical, zipdoc -check is clean, and the app's suite is green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/books finished its typed migration at 20 ops and 5 refusals. Four of the five
refusals were right and stay: three routes eat RAW BYTES (a receipt PDF/image on
POST /v1/books/scan and /v1/books/inbox, an OFX/QFX/CSV statement on
/v1/books/bank/import) and zip's typed path decodes every body with
jsonenc.Unmarshal, so declaring any In would turn a working upload into a 400 —
the wire would MOVE, which a description task may not do. Two answer 501
unconditionally, so they have no success body to state.
But "cannot be a typed op" was quietly being read as "must be undocumented", and
those three published operationId and tags and NOTHING ELSE. No consumer of the
document can distinguish that from a route that takes no body and returns none,
so every SDK generated off openapi.yaml offered a receipt scan with nowhere to put
the receipt and no return type for what came back. That is not an absent feature,
it is a WRONG description, and it is worse than the typed-op gap it stands in for.
So declare the halves that are true. openapi.Register already carries request and
response types off the handler's own structs; what it could not express was a body
that is not JSON, because no Go struct describes a file. openapi.Binary is that
one value: passed as req it renders OpenAPI's own spelling for an opaque body,
application/octet-stream with {type: string, format: binary} — the shape an SDK
generator turns into a file parameter. Schema gains Format for it, carrying exactly
one value, because "string" and "string/binary" are the only distinction here a
consumer acts on.
POST /v1/books/scan Binary -> ScanDraft
POST /v1/books/inbox Binary -> InboxItem
POST /v1/books/bank/import Binary -> BankTally
Binary is REQUEST-only. A byte response is a second fact no route needs yet, and
adding it before one asks is how one seam becomes two.
The two 501 stubs are declared NOWHERE, and a test now asserts that silence, so a
later "document these too" argues with a gate instead of inventing a contract for
a call that has never once succeeded.
Pure description: not one route, status, field or byte moves. Every books wire
test passes unchanged, and the value is measured rather than asserted —
TestTheRawBodyRoutesDeclareBytesInAndAShapeOut reads the document through JSON,
the way a generator does, and pins bytes-in, the $ref out, and the stubs' silence.
What the three still lack, and only a typed op can give them, is prose, an MCP
tool and a CLI command. Typing them needs a zip capability that does not exist:
v1.18.6 has no octet-stream request declaration and its whole OpOption set is
WithSummary/WithTags/WithOperationID/WithStatus.
Gates: apps/books, openapi, apps/cloudflare, apps/platform, manifest and the root
package all green under the Makefile env; zipdoc -check clean; plugin/books/
openapi.json and openapi.yaml regenerated from source through the weave.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/crm was already 19 of 20 typed with its single refusal gated
(untypedByDesign + TestEveryRouteIsTypedOrNamed). Re-measured from the LIVE
router rather than the prose: 20 served, 19 typed, 15 schemas with 0 bare
properties, and POST /v1/crm/applications the one operation with no registry
entry — which openapi.yaml shows as the cost, an operationId and a tag and
nothing else.
The refusal was anchored to zip v1.18.6, the version cloud pins. All five of its
wire facts are now re-read in v1.18.8, the newest published: op.invoke is still
what a tools/call and LocalInvoke dispatch into (mcp.go:152, cli.go:427), it
still unmarshals before the handler (typed.go:234 ahead of :259), the OpOption
set is still WithSummary/WithTags/WithOperationID/WithStatus (:80/83/86/110), and
MCP.Disabled is still app-wide (zip.go:131). v1.18.8 adds ask/declare/ops/peer/
tenant and moves none of them, so 19 is still this package's honest floor and the
line numbers are recorded so the next agent does not redo the reading.
Re-verifying the refusal surfaced a live defect in the meter it leans on. The
intake limiter is keyed on c.Fiber().IP(), and that is the TCP peer: zip's
fiber.Config sets no ProxyHeader and no trusted proxy, and fiber only reads a
forwarding header when both are set. For proxied public traffic — the only
traffic the limiter exists to bound, as EdgeRateLimit's own scope rule attests by
treating a request with no X-Forwarded-For as an in-cluster caller — every
submission therefore shares ONE 20/min bucket, along with the three staff
application routes registered after it. One host can spend the whole budget.
Stated once, at intakeRateLimit, including why the one-line fix is wrong:
middleware.RateLimit's bucket map is only ever reset, never evicted, so keying it
on real client IPs grows without bound, which is why EdgeRateLimit carries its
own eviction rather than reusing that primitive. Closing it moves when callers
see 429 — a metering decision, not a typing one — so typing left the wire alone
and the defect is reported, not silently changed.
No route changed. Tests, vet and per-package zipdoc -check all green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/team was already at its typable floor — 9 typed ops, 10 refusals gated by
untypedByDesign. Re-verified that floor against zip v1.18.8 (the newest tag,
newer than the v1.18.6/7 the note was written against), because "the floor is 9"
is a claim about a DEPENDENCY and expires when the dependency moves. All three
blocking capabilities are still absent: WithStatus still panics outside 2xx
(typed.go:110), the REST arm still ends in c.JSON(out) with no bytes or upgrade
path, and op.invoke still unmarshals any non-empty body into In before the
handler and answers 400 on a parse failure. v1.18.8's delta is app.ops ->
app.registry plus new ask/declare/ops/peer/tenant files; none of it touches the
three. No team route can be typed without moving its wire, so none was.
What the op-level count did NOT measure is the field surface. team published
three bare properties — ProviderInfo.name, ProviderInfo.displayName and
botMember.active — each reaching openapi.yaml, all four generated SDKs and the
MCP inputSchema with no description, for the crm reason exactly: op prose and
field prose live in different places and only the op one was counted.
botMember.active is the one that cost a reader something real. It is not the
agent's own flag but a DERIVED projection (botActive: empty/"active"/"ready" are
live, archived and retired are not), so a caller could see a boolean and nowhere
that a retired agent stays in the roster as an inactive member with its
authorship intact.
TestEveryPublishedFieldIsDescribed now gates team the way it gates crm, and it
was proven to bite by stripping the active prose and watching it name
botMember.active.
Wire preserved exactly: the change is doc comments only. plugin/team/openapi.json
gains 3 lines, all of them "description" keys, and openapi.yaml gains the same 3
descriptions — no status, path, operationId, field name or schema shape moves.
Subset regenerates byte-identical on a second run.
Gates: make -C apps/team test green (baseline was green), vet clean, zipdoc
-check clean, openapi weave and manifest router oracle green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/framework was already 17 of 19 typed. The two document writes (POST
/v1/framework/:doctype, PUT /v1/framework/:doctype/:name) stay raw, and the
reason re-verifies: their body IS the document's own field data, an open object
the DocType defines at run time, and typing it needs THREE zip properties none
of which shipped in v1.18.6 (the pin) or v1.18.8 (the newest tag) — read out of
the source, not taken from the note:
1. DECLARE an open object — schemaOf has no reflect.Interface case, so
map[string]any's element falls to the default (openapi.go:486).
2. BIND the URL onto one — bindURL returns early on a non-struct In
(typed.go:152).
3. Carry the params OUTSIDE the body namespace — op.invoke gets no path map
off the REST path, and for a prompt-named DocType the create body's `name`
IS the document's name (framework ops.go:321 stringField(in,"name") →
doctype naming.go:168 ResolveName).
Leg 3 was the one nothing read: it was cited as prose, so the day the engine
stopped naming a document from its body the refusal would have outlived its
cause silently. It now asserts the fact over the live wire — define a
prompt-named DocType, create with a body `name`, and the document must carry
that name. Verified to BITE: flipping the fixture's autoname to "hash" turns it
red.
Also records two things that were assumed rather than measured:
- What the refusal COSTS. The two writes DO reach openapi.yaml — as route-only
entries with path parameters and no requestBody, no responses, no prose. So
the choice is not "typed and broken vs. absent", it is "a schema that lies
about the body vs. no schema at all", and only the second stops being wrong
by itself once the capability lands.
- Leg 1 is a LIVE spec defect, not only a blocker. docView is map[string]any
and is already the Out of four typed ops (get/submit/cancel a document, and
the list's items), so openapi.yaml currently tells every SDK and every agent
that each field of a returned document is a JSON object. It is not. The fix
is one `case reflect.Interface` in zip's schemaOf returning `{}`; it needs a
zip release, so it is not made here.
No route, handler, status or body changed — the wire is untouched, and
plugin/framework/openapi.json regenerates byte-identical.
Gate: go test -tags sqlite_fts5 ./apps/framework/... ok · go vet clean ·
zipdoc -check clean · make openapi (framework subset) no diff.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/account is already fully typed — 11 typed ops, 7 raw. Re-verifying the
partition (not asserting it) turned up one thing that was actually wrong, and
it was in the prose that IS the product surface for the raw seven.
`verbatimForward` is the reason string recorded against all seven untyped
operations, and those seven publish with no description and no schema, so this
constant is the only explanation a future engineer or a route-typing agent
gets for why they stay raw. It claimed "the body is forwarded as received at
any content type". Two other places in the same tree state that claim was
REMOVED: TestUntypedByDesignForwardsVerbatim's own comment says
"verbatimForward no longer claims 'at any content type'", and LLM.md says the
audit "refuted 'forwarded as received, at any content type'". It never was
removed — the literal still said it, three lines above an assertion that
proves it false (commerceDo rewrites the request Content-Type to
application/json; the test pins exactly that).
So the reason now says what the test proves: the BYTES reach commerce as
received whatever their content type, only their DECLARED type is rewritten.
The refusal itself is untouched and still decisive — status passthrough alone
(`c.Bytes(status, raw)`, a 402 spend cap, a PDF at invoices/{}/pdf) is
something a typed dispatch cannot express, since it ends in c.JSON under the
one status the op declared and WithStatus panics on a non-2xx.
No wire change: a test-file constant nothing asserts on, plus a doc row.
LLM.md's tranche row said "account 19" while its body says "11 of 18"; the row
now carries the correction the ingress row already does, and names the closed
list that keeps an eighth raw route red.
Verified: zipdoc -check green (no lifted prose moved, zipdoc_gen.go
unchanged), go vet clean, apps/account green before and after,
TestEveryRouteIsTypedOrNamed passing — which is the machine-checked proof
there is no untyped account operation beyond the seven named.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Five of the six routes apps/guide leaves untyped already carry a test that
pins the wire fact behind the refusal — TestDocumentPutsAcceptYAML for the
two YAML-or-JSON document PUTs, TestHTTPTransitionsAndGating for the
structured 409 on the gated pair, TestDoStreamsSSE for the /do stream. The
sixth did not: PATCH /v1/guide/blueprint/:collection/:id claims its body is
a JSON merge-patch whose explicit nulls clear a key, and nothing held that
claim.
An unpinned refusal is a claim that can rot into a stale one, which is worse
than no claim: the next reader takes it on faith. So verify it and hold it.
The behaviour is exactly as documented — mergeItemPatch overlays the patch
keys onto the marshalled item and decodes into a fresh value, so an ABSENT
key changes nothing and an EXPLICIT null leaves the field at its zero, and a
nil Enabled reads as ENABLED (absence == on). `{}` keeps a disable in place;
`{"enabled": null}` lifts it.
That pair is precisely why the route cannot be typed: encoding/json decodes
both `{}` and `{"enabled": null}` into a nil *bool, so a typed In collapses
"re-enable" and "change nothing" into one request. Confirmed against zip
v1.18.6, whose typed decode is jsonenc.Unmarshal and whose only OpOptions
are WithSummary/WithTags/WithOperationID/WithStatus — no raw-document In, no
multi-status.
Test only. No route, no doc comment, no generated artifact moves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
THE THIRD MIRROR, and the same dead check. isMachinePrincipal read
`type == "application"`, and the IAM line this cloud runs against stamps that
value NOWHERE — `tokenType` takes exactly "access-token" and "id-token"
(internal/oidc/jwt.go), and the object/token_oauth.go the comment cited is not in
it. So the check could not fire, every machine fell through to the KMS-audience
clause, and that clause matches ONE identity in the estate.
A generic admin-org client_credentials token therefore read as a HUMAN and took
the SuperAdmin arm: cross-tenant reads, plus the org-switch that decides which
ledger pays. The repo's OWN red-team probe reproduces it —
TestRedIso_C_AdminCrossOrg returned 200 with s3kr3t-of-maxpower — and it passed
all along because the harness minted its machine fixture with `type`, the claim
IAM does not emit, while defaulting nil `orgs` to [owner]. The fixture described a
token that cannot exist; production had the other one.
THE ACTUAL DEFECT was one predicate answering two questions whose fail-closed
directions are OPPOSITE:
"may this hold an admin scope?" GRANTS the only cross-tenant scope, so an
unidentifiable principal must be REFUSED →
needs a positive HUMAN test.
"which org is this in?" GRANTS an org out of the app-selected `owner`
claim, so it needs a positive MACHINE test.
Splitting them is the fix, not just changing the body. isHuman = a membership set
is present (IAM signs one for every user token — store.MemberOrgRefs opens with
the home org — and none for a machine, because "a machine token has no user and
therefore no membership set"). The org question keys on isKMSMachinePrincipal, the
owner-bound audience that actually vouches for a machine.
ONE RULE FALLS OUT, with no third case to get wrong: a token carrying no
membership set never has an org read out of `owner` unless that audience vouches
for it. A generic machine and a human token minted before the `orgs` claim are
indistinguishable, and now they are treated identically — refused. That is the
rule TestLegacyOrgsClaimFailsClosed already asserted for one of them; keying the
first version of this fix on absence alone let the machine branch swallow it and
hand a legacy human the app's org, which those tests caught.
FAIL-CLOSED, and it costs something: a human token with no `orgs` loses the two
admin scopes and its org. That is an availability cost bounded by the token TTL,
taken over admitting an unidentifiable principal to a cross-tenant scope.
Falsified: restoring `claims.Type == "application"` turns TestSanitizeIdentity and
TestRedIso_C_AdminCrossOrg red. Full suite green — 174 packages, CGO_ENABLED=0
-tags sqlite_fts5 with the dev KMS key, and the failing-package set is byte-identical
to the clean tree.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/automations is typed 14/18 (8fa5ff0e), with four routes excluded and each
fact pinned (1e50cb87). This re-derives all four from zip's source and PROVES the
pins, and two of them did not hold: the retyping a reader would actually reach
for leaves them GREEN while destroying the wire.
The escape hatch neither pin covered is the non-struct In. bindURL (zip typed.go)
returns early unless the In's kind is Struct, and unmatched names are silently
ignored — so an In CAN describe an open or arbitrary body (map[string]any, any),
and the moment it does it stops receiving path params. An In can describe the
body or be addressed by the URL. Not both. hooks and resume need both.
resume In = any takes 42, "hi", [1,2], true, null — and never sees :id, so
GetRun is asked for "" and EVERY resume 404s. The pin compared the
payloads to EACH OTHER, so a uniform 404 passed. Verified: retyped,
pin green, addressing gone. Now pins that the seeded run and an
unknown one answer differently.
hooks TWO ways to break it, and the loud one was the one pinned. A struct
In 400s {"source":42} — but a payload key it has no field for is not
an error, it is DISCARDED: 200, matched:1, and {{trigger.msg}}
arrives EMPTY. Every webhook keeps "working" while every payload is
blank. Verified by retyping: the whole suite stayed green except the
raw-byte dedupe test; the payload loss was invisible. A map In takes
the open body and loses :source/:event, so nothing matches — and the
pin read the body without asserting the count, so that passed too.
Now pins DELIVERY: matched == 1, and the flow receives the payload
verbatim including the colliding key.
The other two exclusions verified sound, one for a deeper reason than recorded:
mcp closed BELOW zip. The decoder is stdlib encoding/json, which
validates the whole input before dispatching to any UnmarshalJSON,
so an In of json.RawMessage and an In whose UnmarshalJSON never
fails both still answer 400. A syntax error is unreachable from Go,
so -32700-at-200 cannot be recovered by any In type.
operations one Out, two shapes. The union reason checks out — Flow's
externalId/folderId/publishedVersionId carry no omitempty and are
emitted unconditionally, so the omitempty a union needs would delete
them from the flow branch. `Out any` is not a conversion either: it
publishes {"type":"object"}, which is what the untyped route already
offers, and spends the route's one chance to be described. Waits on
a response-per-outcome declaration in zip.
Also: the package's surface table enumerated 18 routes while the prefix serves
NINETEEN — connectorruntime is sub-mounted at automations.Mount and contributes
POST /v1/automations/connectors/{id}/run, which the published subset has and the
table did not. It is typed there; the table now says so.
Wire, prose and published document unchanged: zipdoc_gen.go and
plugin/automations/openapi.json regenerate byte-identical (rerun, not assumed) —
every comment edited belongs to an untyped handler, which zipdoc does not lift.
Converted zero, because zero were convertible without moving the wire.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/captable had 11 typed ops (the collection reads) and 20 raw relays. All 20
refusals rested on one claim: a /v1/captable write relays the goja bundle's own
(status, body), including four envelopes cloud has no vocabulary for — 400
{success,message,errors}, 404/409 {success,message}, and the top-level catch's 500
— so a typed op whose error path renders zip's {status,code,error} cannot express
them without moving the wire.
That half is true and now solved rather than avoided. bundleErr carries the
bundle's status and its BYTES as a Go error; bundleEnvelope, a group middleware
registered beside cloud.Bridge, writes them back untouched. A reachable non-2xx is
no longer a reason for a captable route to stay raw. bundleErr.Unwrap yields a
*zip.HTTPError, so OFF the HTTP path — an MCP tools/call, an in-process CLI invoke,
neither of which passes the middleware — the answer is still the bundle's status and
message rather than a blanket 500.
The REAL blocker is on the request, and it is why 14 routes are still raw. The
bundle validates with COERCING helpers (goja/src/validate.ts): `num` accepts a
number OR a numeric string (z.coerce.number), `optString` accepts any scalar and
calls String(v), and addStakeholders accepts a single object OR an array. zip
decodes a typed In with encoding/json, which answers 400 "invalid body" to every
one of those — so typing would make the route accept LESS. Each of the 14 now names
the field that does it instead of citing the response.
That splits the surface on a line that is checkable, not a matter of taste: a route
with NO REQUEST BODY has nothing to coerce, so its In is faithful by construction.
There are exactly six such routes left, and they are the six typed here:
GET /v1/captable/rounds/:id round + its cheques
DELETE /v1/captable/stakeholders/:id
DELETE /v1/captable/shares/:id
DELETE /v1/captable/options/:id
DELETE /v1/captable/safes/:id
DELETE /v1/captable/convertibles/:id
17 of 31 operations typed and described, up from 11. plugin/captable/openapi.json
and openapi.yaml gain 160 and 148 lines of schema and prose and lose NOTHING: 21
captable paths and 31 captable operations before, 21 and 31 after, zero removals,
zero additions.
WIRE, PROVEN BY TEST — bodyless_test.go re-derives the pre-typing answer on every
run (a direct Dispatch of the same bundle route with the same params IS what the
raw relay wrote) instead of trusting a recorded golden, and compares status,
Content-Type and body bytes on BOTH arms:
- 2xx, through zip's typed JSON writer: the round detail is byte-identical on the
closed PRICED round (closeDate/pricePerShare/preMoneyValuation/shareClassId all
set, one cheque with a comment) and on the OPEN SAFE round (all four null, no
cheques); each delete's {"success":true} is compared against a direct-dispatch
delete of the matching row in a second identically-seeded tenant.
- non-2xx, through bundleErr: every 404, plus the 400 that MOTIVATES the whole
mechanism — refusing to orphan issued equity carries an `errors` LIST, and
zip's envelope has nowhere to put it. The list survives, byte for byte, under
the same bare `application/json` the relay sent.
Both proofs were watched go RED: stubbing errors.As out of bundleEnvelope fails
three tests, and swapping two fields of captableRoundDetail fails the byte
comparison.
DELETE takes its input from the URL and carries no body — zip's hasBody rule, which
the document reads too — so the five deletes publish a path parameter and no
requestBody, which is what the raw routes already did (readBody=false).
ONE LATENT DEFECT, FIXED. The typed reads' non-2xx arm was NOT unreachable. Their
read() turned any non-2xx into zip's own 500 "captable dispatch failed", and the
bundle's top-level catch answers 500 {success,message} on a SQL error — so a read
that hit one had silently moved from the bundle's envelope (with its message) to
cloud's (without it) when the reads were typed. read() is gone; every op now goes
through one call(), so that arm relays the bundle's bytes again, as it did before
typing. Unreachable-by-construction arms (getCompany/capTable's defensive
notFound) are unaffected.
Registration order is unchanged in effect: no two /v1/captable routes overlap on
method + pattern, so moving the six into the typed block cannot shadow anything.
Gate: make -C apps/captable {test,vet} green (baseline was green), 12 tests, all
pass; zipdoc regenerated and idempotent; ./openapi (the weave) and ./manifest
green; apps/{esign,goja,company} — the other goja-bundle leaves — still green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/compliance/health is the only route on this surface that does not read a
tenant, and it had no test at all. It is also the route the group's newly
installed cloud.Bridge could most easily have broken: Bridge is what parks the
validated org for every other typed op, and had it REFUSED a request carrying no
org, installing it in front of the leaves would have turned liveness into a 403 —
the failure mode where a subsystem reports itself down to every prober that
correctly sends no tenant header.
Bridge is fail-open by construction (it parks what it has and continues), so the
route was in fact fine; this is the assertion that keeps it that way, and it
measures the answer rather than asserting the middleware's shape.
Verified: GET /v1/compliance/health with no X-Org-Id and no X-User-Id answers 200
{"status":"ok","provider":"manual"}.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Went to type apps/git's untyped routes and found the partition already COMPLETE
and already a gate (3c9dd462): 24 typed ops, 24 refusals in untypedByDesign, and
TestEveryRouteIsTypedOrNamed/TestEveryTypedOpIsDescribed pinning both halves.
Re-verified rather than trusted — every refusal read against its handler, not its
prose — and all four families hold: HMAC-over-raw-bytes (webhook), pack streams
(smart-HTTP, both hosts), text/html (the browser UI, both hosts), and the ZAP
envelope whose failure body is {status:"error", msg} where a typed op's returned
error renders zip's {status:<int>, code, error}. Zero routes were convertible
without moving a wire, so this commit converts none. 24/24 confirmed from the
committed subset: plugin/git/openapi.json is 48 operations, exactly 24 described.
What the verification DID surface is upstream of every typing task in this repo:
`hanzo.yml`'s two raw-go gates cannot build the tree they gate. hanzoai/base
v1.5.11 — pinned TODAY, da53ab30 — declares a deliberate compile error under
`cgo && !sqlite_math_functions` (base/core/sqlite_math_required.go), because its
search layer emits SQL calling acos/cos/sin/radians/sqrt that the cgo sqlite has
only behind that tag. 47 of cloud's 306 packages reach base/core (apps/git,
apps/agents, apps/billing, apps/base, … and their plugin/<app> mains). go-unit
sets CGO_ENABLED=1 explicitly; go-vet sets nothing and so inherits the toolchain
default, which is 1 wherever a C toolchain exists — and hanzoai/ci provisions one.
Measured, both directions, from this worktree:
go vet ./... -> EXIT=1, base/core
go vet -tags "sqlite_fts5 sqlite_math_functions" ./... -> EXIT=0, all 306
So the gate reported "[build failed]" where a test run was expected, and a gate
that cannot build is a gate that never ran a test — the same failure mode
hanzoai/ci's own Test step exists to refuse. Both steps now pass the tags the
Dockerfile already passes; sqlite_fts5 rides along because it is the tag the image
and `make test` carry and without it an FTS5-backed store cannot open. The root
cause is that these two steps restate the Makefile's posture instead of using it,
so LLM.md now names the floor and the count with the command to re-derive them.
Also corrected there: apps/git no longer belongs on the "fails under make test by
design" list — measured green under exactly that posture (dev key, -tags
sqlite_fts5, CGO_ENABLED=0) in 18s.
NOT patched, because it is a policy call and copying it would re-create the same
drift: go-unit still has no CLOUD_KMS_MASTER_KEY_REF, which `make test` injects
once for the whole suite, so cek-backed packages still fail there (measured:
apps/code, 6 tests). Either CI gets the dev key or the step routes through the
Makefile — one declaration, not two.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
crm was already 19 typed of 20 served, and the route census here proves it is
20/19/1 exactly (typed_wire_test.go). What the count did NOT measure is the half
of the published surface that does not come from typing a route: a typed op
documents its ADDRESS and its SHAPE, never the shape's FIELDS. Those come from
doc comments on the In/Out struct fields, and crm's REQUEST types carried them
while its RESPONSE types — Company, Contact, Opportunity, Application,
ScreenResult, StageEvent — are store ROW types nobody had written field prose on.
All 65 of their properties reached openapi.yaml, all four generated SDKs and the
MCP inputSchemas with no description at all. A reader could see that `arr` was an
integer and nowhere that it was CENTS.
Every field now carries prose derived from the code that writes it (cents,
server-owned unix seconds, the upper-cased stage vocabulary, the 422 on a
cross-org ref, the cleared-on-delete relations, the snapped credit ladder), and
the whole change is proven to be DESCRIPTION only: strip description/summary/
example from openapi.yaml before and after and the documents are byte-identical.
Zero wire movement.
The partition becomes a GATE, the way team's and git's did — prose cannot fail:
- TestEveryRouteIsTypedOrNamed — untypedByDesign is the closed list; fails on
an operation that is neither typed nor named, on a name crm no longer serves,
and on a name that IS a typed op. All three modes verified by perturbation.
- TestEveryTypedOpIsDescribed — the lifted prose reached the binary.
- TestEveryPublishedFieldIsDescribed — the new one, gating the half above.
Verified: deleting one field comment turns it red (Company.arr).
The one refusal is re-verified against zip v1.18.6's own source, and it is a
DIFFERENT gap from multi-status (#78): per-op projection SCOPE. POST
/v1/crm/applications is guarded by fiber middleware (IP rate limit) and a
pre-parse 64 KiB raw-body cap. zip's MCP arm dispatches tools/call straight into
op.invoke (mcp.go:152) and the CLI's LocalInvoke does the same (cli.go:427) —
neither runs the route's middleware — and zip has no per-op way to decline a
projection (OpOptions are WithSummary/WithTags/WithOperationID/WithStatus;
MCP.Disabled is app-wide). So typing it publishes an unmetered, uncapped alias of
the one deliberately metered public write in the surface, and apply() never calls
tenant(): it writes into intakeOrg(s), the deployment BRAND's pipeline. The alias
would let any caller reaching /mcp inject unbounded rows into the brand's own
CRM. Its 200-vs-201 split would shim and the honeypot's third body needs only
omitempty — neither is what blocks it.
LLM.md records the refusal as its own zip-gap family and the measured scale of
the field-prose class, which is fleet-wide and not a crm quirk: 1,424 of 2,716
published properties (52%) carry no description and 195 schemas are 100% bare
(appView 26, Wire 21, Node 20, Volume 20, Totals 18). crm is now 0 of 65.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The apps/account (11 of 18) paragraph carried the claim the audit refuted. It said
each of the seven forwards a body "as received (any content type)". commerceDo
(topup.go) SETS the request Content-Type to application/json whenever there is a
body, so the bytes forward and the type does not. It returns no response header at
all, which is why billing.go pins application/json over commerce's own type and
drops Content-Disposition, and it truncates the response at 1 MiB under the
upstream's own 200.
The live consequence is one route: GET /v1/billing/invoices/{id}/pdf, the only
non-JSON entry in billingForwardable, delivers PDF bytes labelled JSON with no
filename, against a commerce that sets application/pdf + attachment. Left unfixed
and recorded at the lines that cause it — the repair is commerceDo returning
response headers across three call sites, one of them the top-up money path.
The two facts that actually carry the refusal are unchanged and now stronger:
status passthrough and an unvalidated request body, both proven against the live
bridge by TestUntypedByDesignForwardsVerbatim instead of asserted in prose. The
"11 typed, 7 refused" ledger row is confirmed by the corrected per-app measure —
7 untyped registrations, 11 zip.X ops, matching what the gate reads off the router.
Docs only.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/account was already fully typed — 11 typed ops, 7 catch-all forwarders held
as a closed list — so this carries no conversion. It closes the gap that made the
refusal unfalsifiable and corrects what the refusal claimed.
The reason the seven stay raw lived only in prose: a paragraph citing zip's source
for facts about THIS package's handlers. Nothing went red if a handler stopped
behaving that way, which is the same failure the closed list was invented to fix
one level up. TestUntypedByDesignForwardsVerbatim now proves the two decisive
facts against the live bridge — commerce's 402 arrives as 402 with its bytes
intact (a typed dispatch answers the one 2xx it declared, via c.JSON), and a
text/csv request body reaches commerce byte for byte (a typed op answers
ErrBadRequest before the handler runs). Each assertion was mutation-checked: flip
the upstream status, the upstream bytes, or the forwarded body and the test fails.
Auditing that claim found it over-stated in one direction and the code wrong in
three. commerceDo is a JSON transport, not a transparent proxy: it SETS the
request Content-Type to application/json whenever there is a body, it returns no
response headers at all, and it truncates the response at 1 MiB while reporting
the upstream's own 200. So billing.go pins Content-Type: application/json over
whatever commerce sent, and drops Content-Disposition — which means
GET /v1/billing/invoices/{id}/pdf, the one non-JSON entry in billingForwardable,
delivers PDF bytes labelled JSON with no filename (verified against a stub
upstream: status=200, content-type="application/json", content-disposition="",
body="%PDF-1.4…"). Commerce sets application/pdf + attachment there
(api/billing/invoice_pdf.go).
None of the three is repaired here. Fixing them means teaching commerceDo to
return response headers — three call sites, one of them the top-up money path —
and deciding which headers are safe to relay while keeping Cache-Control:
no-store, which is a tenancy property and not a content one. That is its own
change with its own test, not a side effect of a doc pass. Each is recorded at
the line that causes it so the next reader lands on the fact, not on a summary.
Wire unchanged: billing.go and topup.go are comment-only, and the sole new code
is a test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/framework was already 17 of 19 typed; the two document writes
(POST /v1/framework/:doctype, PUT /v1/framework/:doctype/:name) stay raw. I
re-derived that refusal from scratch and reached the same three blockers the
record names, so nothing converts here. What DID need fixing is that the
reason was neither true at the registration site nor checkable anywhere.
1. The registration site named ONE blocker. framework.go's Mount still said
"zip needs an open-object input (additionalProperties: true) before these
convert" — precisely the framing 4876a56c was written to retire, because a
reader who fixes only that comes back and converts, publishing a request
schema that names the two path segments and nothing else. That commit
updated the test and LLM.md and left Mount behind, so the one place an
engineer reads before acting carried the misleading half. Mount now names
all three: DECLARE an open object, BIND the URL onto one, and carry the
bound params OUTSIDE the body namespace.
2. Nothing read the citation, so the refusal could not expire. rawRoutes cited
two properties of zip and no test observed either — the day zip ships them,
nothing goes red and two routes stay untyped forever behind a stale reason.
TestOpenObjectRefusalStillHolds now asserts both, through exported API only:
that GET one document publishes response schema
{"type":"object","additionalProperties":{"type":"object"}} (shipped and
FALSE — the package's own round-trip test reads a document back holding a
string and a number, neither of which that schema admits), and that an
open-object In receives no :doctype. Both expectations are wrong on purpose;
either going red IS the signal to convert the two writes and delete the
test. The second assertion is not a tautology: a struct In binds
:doctype=Task on the identical harness, so the map failing to is zip's
behaviour, not the probe's.
3. A rotted fact. The record claimed re-verification against "v1.18.6, the
current pin and the newest published version"; v1.18.8 has since published.
Re-verified against it: schemaOf still has no reflect.Interface case,
bindURL still returns early on a non-struct, mcp.go still invokes with a nil
path map. None of the three shipped, now stated with the version that is
actually newest.
No route, In/Out type or artifact changes: the wire is untouched, zipdoc -check
is clean, and zipdoc_gen.go / plugin/framework/openapi.json / openapi.yaml
regenerate byte-identical. go vet clean; apps/framework green (21 tests).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The comment I just landed said "14 more places", derived by subtracting one from
LLM.md's recorded 15. That is exactly the tally-from-prose mistake the playbook
warns about, and it is wrong: MEASURED over the golden with team's instance
already removed, 15 remain. So LLM.md's 15 was a count taken before more apps got
typed, and the class had already grown past it.
That is the interesting fact, not the arithmetic: an UNTYPED route contributes no
schema, so it cannot state anything false yet. Every app this migration types
converts its any-valued maps from silent to loudly wrong, which means the class
GROWS as the migration progresses and any figure written in prose is stale on
arrival. The comment now names the owners it measured — guide (JourneyStep.args,
stepView.args), pricing (seven list envelopes), admin (adminCatalogOut),
framework (documentList.data), Application.metadata, StepSettings.input,
runIn.props — and says to count, never tally.
Source-only: the rationale lives on the TYPE, which zipdoc does not lift, so
zipdoc_gen.go, plugin/team/openapi.json and openapi.yaml are byte-identical. That
asymmetry is the useful seam — a field comment IS product surface an SDK user and
an MCP client read, a type comment is for the next reader of the code.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/books was already converted (20 typed, 5 refused) and wire_test.go proves the
answers did not move. That is the safety half of the migration and on its own it
justifies nothing: leaving every handler untyped also moves no answer. The VALUE half
— that a typed op lights up OpenAPI, MCP, the CLI and the op-call plane at once — was
asserted in comments and measured by nothing.
projection_test.go measures it, over the whole surface rather than a sample:
- each of the 20 ops must reach the document WITH its doc-comment prose and a
declared response, reach MCP as a tool carrying that same prose, and reach the CLI
as a `hanzo books …` command — all under ONE operation id;
- each of the 5 exempt routes must still answer 401 (a live, fail-closed route, so
the exemption list cannot rot into naming paths that no longer exist) and must
appear in NONE of the three derived surfaces;
- the two ledgers must sum to 25, so a route added to the surface is measured by
something rather than by nothing.
The exemption ledger fails in the GOOD direction too: the day one of these becomes
typeable it names the route to move. Debt nobody is forced to look at is how an
exemption becomes permanent.
Verified non-vacuous by mutation: dropping an op from the ledger, renaming a published
CLI command, and claiming an exempt route is typed each go red.
The refusals hold, and the blocker is named precisely rather than restated:
zip v1.18.7 decodes EVERY typed body with jsonenc.Unmarshal and answers ErrBadRequest
on failure, and its whole OpOption set is WithSummary/WithTags/WithOperationID/
WithStatus — there is no octet-stream/binary request declaration. So an In on
POST /v1/books/{scan,inbox} or /v1/books/bank/import turns a working PDF/OFX upload
into a 400. Faking it with a custom UnmarshalJSON would be worse: the document would
then tell every generated SDK to send JSON to a route that eats bytes. Typing those
three is a zip change, not a books change. The other two (bank/link-token, bank/exchange)
answer 501 unconditionally — a typed op must declare a success it has never sent.
No wire change: the only source edit is a test file, so zipdoc_gen.go, openapi.yaml and
plugin/books/openapi.json all regenerate byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/team is fully typed already — 9 of its 19 operations carry a registry entry
and the other 10 are structurally untypeable at zip v1.18.6, each named in
untypedByDesign with the reason. This pass re-derived all ten against zip's own
source rather than trusting the prose, and found nothing convertible:
- op.invoke decodes a typed In before the handler runs and answers 400 on a
malformed body (typed.go:233). That is the whole reason POST /v1/team/account
(a JSON-RPC envelope whose refusal is HTTP 200 carrying {error: Status}) and
PUT /v1/team/account/cookie (which IGNORES an unparseable body and falls back
to the bearer) must stay untyped — typing either would refuse a request they
have always served.
- the typed handler ends at c.JSON(out) and the OpOption set carries no
content-type, multipart or raw-body option at all. So the two WebSocket
upgrades, the two OAuth 302s, the three byte-serving routes (wallet page,
blob download) and the multipart upload cannot be ops, not merely have not
been made into ops.
What the re-derivation DID surface is a live instance of the any-valued-map
class: statsOut.metrics was a map[string]any, and zip's schemaOf has no
reflect.Interface case, so the element type fell to the default and the document
asserted `additionalProperties: {"type": "object"}` — that every value in the map
is a JSON object — for a map that has never held one. This is worse than the
under-describing gaps (a bodyless POST, an undeclarable second success status):
those state less than the truth, this one states something false, and an SDK
regenerated from the golden typed the field Dict[str, Dict] when the only value
it ever carries is {}.
The field is now an ANONYMOUS empty struct, which is what the wire IS —
`"metrics":{}` on every response, pinned byte-for-byte by
TestTypedStatisticsServesBothPaths, so the change is provably description-only.
Anonymous because there is no value to name, keeping the honest shape out of the
fleet's flat schema namespace.
The class is zip-side and wider than this field: openapi.yaml carries the same
false claim in 14 more places. The one-line fix is a reflect.Interface case in
schemaOf projecting the OPEN schema `true` — an unconstrained element is open,
not an object. Recorded at the field so the next reader has the whole fact.
Gates: apps/team green (baseline was green, identical after), openapi weave green
against the regenerated golden, manifest + root green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/ingress was dispatched for typing and had nothing left to type: 18 routes
before the conversion (7be3160d), 18 typed ops after, 0 dropped, all 18 published
with prose in plugin/ingress/openapi.json AND openapi.yaml, zipdoc current, the
subset regenerating byte-identical from source, and TestSurfaceIsRegistered
gating the surface as an exact set.
What sent an agent there is the playbook's own re-measure command, which had no
path anchor and so counted r.Header.Get("X-Forwarded-Proto") in middleware.go as
an untyped route. A route is a VERB PLUS A PATH; the measure has to say both.
Two half-right commands lived in this file, disagreeing by 83 routes:
- no path anchor -> 83 phantoms across apps/ (integrations read 45 for 19,
platform 32 for 30, tools 18 for 16, ingress 1 for 0) — the same miscount
already documented for team, rediscovered because the command was never
fixed;
- anchored on ("/ alone -> drops the 7 real EMPTY-leaf registrations at a
collection root (prefs 2, webhooks 2, share, crawl, destinations).
So: one command, anchored on a path — ("/ or ("" — with the // filter that the
slash-only form already carried (11 of the empty-leaf hits are comments quoting
the form). Corrected, apps/ holds 666 untyped route registrations. The
independent check on the number is integrations: the corrected measure
reproduces its 19 exactly, which is the count its own conversion recorded as
refusals.
No source, no wire and no generated artifact changes here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/authz v1.10.15 deleted the Casbin enforcer and split the decision from
the edge, so Mount moved to authz/serve. This repo held the estate's only
consumer, and it used nothing but Mount.
The adapter lives HERE. authz/serve takes a logger because it is a leaf that must
never import cloud; cloud's Plugin contract wants Deps. Bending the plugin to the
leaf keeps the dependency pointing one way — a leaf that learned about Deps would
stop being importable by anything that cannot link cloud, which is the whole
property being bought.
Full suite: 173 packages, exit 0. The security probes and the money gate pass —
TestRedIso, TestAudit_AnonRequest, TestGate_*, and
TestResourceMeter_UnconfiguredIsNoop, which together hold that a forged identity
reads nothing and an unreachable biller refuses rather than allows.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/pricing had 15 typed ops and 17 raw routes. Fifteen of those refusals rested
on one claim: the /v1/pricing/{compute,cloud/*,subscriptions,blockchain,iam,base,
paas,policy,tools,gpu} routes and /v1/pricing-policy are byte-for-byte proxies of
the @hanzo/pricing goja bundle, so a typed op cannot express them without a Go
re-marshal that reorders keys.
The claim was wrong, and checkable. apps/goja re-marshals the bundle's answer with
Go's encoding/json before any handler sees it — Host.DispatchWith ends in
`json.Marshal(m["body"])` over the exported map. What the raw pass-through wrote
was never the JS engine's bytes; it was Go's, keys already sorted. Decoding those
bytes into an Out and re-marshalling them is therefore the identity function.
So they are ops now: 30 of 32 operations typed, 30 described, up from 15/15.
Each is ONE registry entry the OpenAPI operation, the MCP tool, the CLI command
and the generated SDK method all follow from. plugin/pricing/openapi.json and
openapi.yaml gain 276 lines of schema and prose and lose nothing — 1011 paths
before, 1011 after, zero removals.
WIRE, VERIFIED. A probe drove all 15 addresses under four identity shapes
(anonymous, member, SuperAdmin, and the forged X-Org-Id with no principal the
enablement attack tests pin), before and after, comparing status + Content-Type +
body sha256. All 60 body hashes identical. All 60 statuses identical. The bodies
do not vary with the caller either, which is the property that makes these
sections and not catalog reads.
The whole delta is one header, and it is a normalisation:
Content-Type: application/json -> application/json; charset=utf-8
fiber's typed JSON writer sets the charset form; the raw pass-through set the bare
one. Every OTHER answer on this surface — every typed op, every zip error, the 403s
in admin.go and enablement.go — already sent the charset form, so this ends a split
inside one subsystem rather than starting one.
The 503 arm is preserved as a status and pinned as a test. A section the catalog
does not hold still answers 503 with the bundle's own message, because the status
comes from the bundle (dispatchErr), not from a declaration. Its BODY moves from
the bundle's {"error":…} to zip's {"status":503,"error":…} — same status, same
message, one added field, the same shape this surface's every other error already
had. The shipped catalog cannot reach that arm, so
TestSectionsDegradeWithTheBundlesStatus strips the sections and drives it.
The proof is a test, not this message. sections_wire_test.go re-derives the
pre-typing answer on every run (rawDispatch is exactly what the raw route wrote)
instead of trusting a recorded golden, and a companion test fails if mountSections
declares an address the proof does not cover.
TWO RAW LEFT, both halves of the admin overlay PATCH, both wire-bound:
- PATCH /v1/admin/catalog/models/* addresses a model id that may contain '/',
so it routes through a greedy wildcard. fiber's runtime name for it is `*1`;
the document's is `{wildcard1}` (openapi.translate, because `*1` is not a legal
URI-template name). An In field can bind one or publish the other, never both.
- PATCH /v1/admin/catalog/providers/:name carries `overrides`, an RFC 7386 merge
patch STORED AND ECHOED VERBATIM. json.RawMessage publishes as an array of
integers (schemaOf takes the Slice arm — it is []byte); map[string]any
re-marshals and sorts the keys, so the overlay this route echoes and the one
GET /v1/admin/catalog echoes under "_overlay" would come back reordered.
Its old stated reason was also wrong and is corrected: it claimed retyping to
map[string]any would move {"overrides":null} from "clear the override" to "leave it
alone". encoding/json already sets a *json.RawMessage field to nil on a JSON null
(indirect breaks on the first settable pointer when decodingNull), so null has
ALWAYS meant "leave it alone" here — normalizeOverride's "null" branch is dead for
the literal. A refusal justified by a mechanism that does not exist is a refusal
nobody can re-check.
dispatch() and passthrough() are gone with their last caller: the read path no
longer touches *zip.Ctx at all.
Gate: make -C apps/pricing test green (baseline was green), vet clean, zipdoc
-check clean, openapi weave green, ./openapi and ./manifest green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/automations is typed 14/18 (8fa5ff0e). The remaining four each name a wire
fact a typed op cannot carry, but the reasons lived only in prose — so a later
reader could retype any of them, watch the suite stay green, and ship a silent
wire change. Three of the four break on inputs no existing test sends.
untyped_wire_test.go pins the facts, so the exclusion is enforced rather than
asserted. Each test fails the moment its route becomes a typed op and names the
fact that was lost:
mcp an unparseable body is a JSON-RPC result, not a transport failure:
HTTP 200 carrying -32700. A typed op answers 400 with no envelope.
resume the payload is an arbitrary JSON value; 42, "hi", [1,2] and true
are legal today and are 400s under any struct In.
hooks the body is open-keyed while :source/:event bind by NAME, so a
typed In must own fields source and event — and {"source": 42} is
a legal event today, `invalid body:` 400 when typed.
operations TWO body shapes on one route and one status: the Flow on
CHANGE_STATUS, the FlowVersion otherwise. Verified disjoint on
their discriminators, so one Out cannot be both.
Each pin was proved to bite: temporarily retyping resume and hooks turns the
corresponding test red, while every pre-existing test in the package stays
green.
That exercise also corrected the hooks reason, which was misleading in a way
that invited the break. It cited the raw-byte dedupe hash, the size gate and two
headers — all four of which ARE reachable from a typed op via cloud.Request(ctx),
so a reader who recovered them would believe the route was now typeable. The
decisive fact is the path-param/body key collision, and zip returns its 400
before the handler, so nothing inside the handler can recover it. The comment now
separates the blocking fact from the recoverable ones and says why.
Wire unchanged: comments and one new test file. No route registration moved, so
zipdoc_gen.go, openapi.yaml and plugin/automations/openapi.json regenerate
byte-identical (verified by rerunning the generators, not assumed).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/compliance was already 16 of 17 typed. The 17th — the provider webhook —
was refused in a comment, which is the weakest form a decision can take: prose
does not fail, so nothing stops the 18th route from landing untyped, and nothing
notices when the reason stops being true.
Make it a GATE. untypedByDesign is the closed list of compliance operations
that are not typed ops, and TestEveryRouteIsTypedOrNamed fails on any route
that is neither typed nor named there — so the next route added here is typed by
default, and dropping one out of the registry takes a deliberate edit with a
reason. It reads the LIVE router (openapi.Spec for what is served, openapi.Typed
for what carries a registry entry), never the source, so a route added anywhere
in routes() surfaces whether or not anyone remembers the list. Its second arm
fails on an entry naming a route the app no longer serves, so the refusal list
cannot rot into stale prose. TestEveryTypedOpIsDescribed holds the lifted
prose to the same bar as the schema: that prose IS the OpenAPI description AND
the MCP tool description a model reads to pick the tool, so an op added without
regenerating zipdoc shows up as a nameless tool rather than shipping as one.
Both arms were proven to bite before being trusted — emptied, the list named
POST /v1/compliance/verifications/webhook; given a route that does not exist,
it named the staleness. A gate nobody has watched fail is not known to run.
The refusal itself is re-verified against zip v1.18.6's own source rather than
against the comment claiming it, because "cannot be typed" is a claim about a
dependency and a dependency moves. Two independent wire facts: the HMAC covers
the EXACT received bytes (apps/idv/webhook.go Verify: mac.Write(body)) which zip
has already unmarshaled into In before the handler runs (typed.go:236), so a
re-encoded In is not the signed value; and the route answers TWO 200 shapes (the
reconciled check, or {"ignored": …} for a reference it does not know) where an op
declares exactly one Out — unioning them would add zero-valued fields to the
no-op body, which is a wire change, not a description. Re-check when zip gains
raw-body binding or multi-status responses.
routePrefix hoists /v1/compliance out of the four places that spelled it — the
group and bodyCap's three exact paths — so the router and the body gate cannot
disagree about where a route lives. The package is gofmt-clean again (the
import order drifted in the clients/ -> apps/ move).
No route, method, In, Out or lifted comment changed, so the published document
is byte-identical: this commit describes the surface, it does not move it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/captable relays a goja bundle's own (status, body). The eleven READS end in
okRes on every reachable path, so 200 is the only answer and the body is a shape
Go can state: they become zip typed ops and reach the document, the MCP tool
list, the CLI and the generated SDKs — 11 operations that carried a path and a
method and nothing else now carry a schema and prose.
The twenty WRITES stay untyped, each with the reason written at its registration.
The shared one is not effort, it is the wire: the bundle authors its own error
envelope — 400 {success,message,errors}, 404/409 {success,message} — and a typed
op's failure path can only render zip's {status,code,error}. Typing one would
change what every existing client parses on every validation failure, so it does
not get typed. Two also read the body in ways a Go struct cannot state (a single
object OR an array; `quantity` OMITTED meaning "the whole certificate", which a
zero value cannot say). That is the same class as multi-status: a contract detail
zip has no vocabulary for yet.
FIELD ORDER IS LOAD-BEARING, once. The bundle's rows cross goja as
map[string]any and are serialised by encoding/json, which sorts object keys, so
every model declares its fields in alphabetical json-tag order and the typed
response is BYTE-identical to the relay it replaces — not merely equal as JSON.
Nullability is the DDL's: a nullable column is a pointer, so null stays null.
TestTypedReadsAreByteIdenticalToTheBundle pins that against the bundle's own
bytes on an empty tenant AND on one written through the untyped relays with every
nullable column exercised both ways.
cloud.Bridge goes on the group: a typed op receives only a context, so the
validated org reaches it by being parked there, never as an In field — an In field
is caller-supplied, so a tenant key read from one is a cross-tenant read the
caller asserted for itself. TestTypedReadsAreOrgScoped proves the typed plane
refuses byte-for-byte as the untyped one does and that one tenant never sees
another's rows.
Verified beyond the suite: the concrete route table is unchanged (same 31
method+path pairs, nothing added, nothing lost), the bare prefix still 404s, and
a 110-line dump of every route's answer — success and every error branch, anon
refusals, cross-tenant reads — is byte-identical between this tree and main
across three runs each.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Almost every failure tonight had the same shape, not the same cause: one fact
written down twice, with nothing forcing the copies to agree, and each copy
locally correct so the disagreement was silent.
billing_account on Claims, but every spend site holds a *User
balanceReader an in-process hook OR an HTTP fallback, chosen by a process
topology neither half knows about
audit_log.seq one chain, N in-memory counters
image.tag and the probe port: which port serves health, said twice
72 App CRs two owners
GIT_CUSTOM per-container, and the main one was missed
an app's paths manifest/apps.go AND plugin/<app>/main.go
The last one is what took inference down on v1.801.318/.319: ai's row read
"/v1/ai" while its router served /v1/chat/completions and /v1/models at top
level. Both halves were reasonable; only the pair was wrong, and nothing examined
the pair. 405/404 fleet-wide, every pod Ready, UI serving 200.
Four plugins restated their prefixes. All four AGREED when I checked — and that
is the defect, not the reassurance: the copies live in different files, move in
different changes, and the next disagreement is as invisible as the last. So the
host's list is THE list and an app reads it (manifest.PrefixesFor). iam's and
pricing's package-level Prefixes vars now have zero consumers; zen's and tools'
literals are gone.
Two guards, and I had to fix each after it lied to me:
- TestNoPluginRestatesItsPrefixes — verified by re-planting a literal in
plugin/zen and watching it fail.
- TestEveryPluginNameIsInTheManifest — PrefixesFor returns nil for an unknown
name, which zip reads as "claims nothing", so an absent row must be loud.
Its tool-exemption is DERIVED ("does it call cloud.Serve") rather than a name
list that would need editing forever and would eventually hide a real app.
Deriving it by SUBSTRING read the scaffold generator as an app, because
gen-app-cmds contains "cloud.Serve" inside the template it emits — so it
parses for a call expression instead. Matching text finds the mention; only
parsing finds the call.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Restores work that was lost: the original commit swept another session's
in-flight manifest changes in with `git add -A` in this shared worktree, and
went away when that session reset. Same change, staged by explicit path.
The brand's skills are generated from the OpenAPI source of truth and embedded,
so changing them is a rebuild and a redeploy. That is right for the catalogue a
deployment ships and wrong for the one an org writes. POST /v1/skills stores an
org's own skill and it is listed immediately.
Same split clients/templates already keeps: a PUBLIC embedded catalogue with no
write route, and a PRIVATE per-org store whose every read binds org. A private
skill has no path onto the public discovery surface by CONSTRUCTION — different
containers — not by a filter someone has to remember to write.
Two providers share SourceSkill deliberately. The registry dedups by NAME with
equal-rank ties going to whoever registered first, and clients/agentskills
mounts at order 8 against this at 123, so a brand skill always wins a collision
with an org's. The shipped catalogue is the one that cannot be shadowed.
The write surface lives here rather than in clients/agentskills because that
subsystem mounts BEFORE iam to win the /.well-known discovery routes, so it has
no validated principal to scope a store by.
Co-Authored-By: Hanzo Dev <dev@hanzo.ai>
Almost every failure tonight had the same shape, not the same cause: one fact
written down twice, with nothing forcing the copies to agree, and each copy
locally correct so the disagreement was silent.
billing_account on Claims, but every spend site holds a *User
balanceReader an in-process hook OR an HTTP fallback, chosen by a process
topology neither half knows about
audit_log.seq one chain, N in-memory counters
image.tag and the probe port: which port serves health, said twice
72 App CRs two owners
GIT_CUSTOM per-container, and the main one was missed
an app's paths manifest/apps.go AND plugin/<app>/main.go
The last one is what took inference down on v1.801.318/.319: ai's row read
"/v1/ai" while its router served /v1/chat/completions and /v1/models at top
level. Both halves were reasonable; only the pair was wrong, and nothing examined
the pair. 405/404 fleet-wide, every pod Ready, UI serving 200.
Four plugins restated their prefixes. All four AGREED when I checked — and that
is the defect, not the reassurance: the copies live in different files, move in
different changes, and the next disagreement is as invisible as the last. So the
host's list is THE list and an app reads it (manifest.PrefixesFor). iam's and
pricing's package-level Prefixes vars now have zero consumers; zen's and tools'
literals are gone.
Two guards, and I had to fix each after it lied to me:
- TestNoPluginRestatesItsPrefixes — verified by re-planting a literal in
plugin/zen and watching it fail.
- TestEveryPluginNameIsInTheManifest — PrefixesFor returns nil for an unknown
name, which zip reads as "claims nothing", so an absent row must be loud.
Its tool-exemption is DERIVED ("does it call cloud.Serve") rather than a name
list that would need editing forever and would eventually hide a real app.
Deriving it by SUBSTRING read the scaffold generator as an app, because
gen-app-cmds contains "cloud.Serve" inside the template it emits — so it
parses for a call expression instead. Matching text finds the mention; only
parsing finds the call.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Restores work that was lost: the original commit swept another session's
in-flight manifest changes in with `git add -A` in this shared worktree, and
went away when that session reset. Same change, staged by explicit path.
The brand's skills are generated from the OpenAPI source of truth and embedded,
so changing them is a rebuild and a redeploy. That is right for the catalogue a
deployment ships and wrong for the one an org writes. POST /v1/skills stores an
org's own skill and it is listed immediately.
Same split clients/templates already keeps: a PUBLIC embedded catalogue with no
write route, and a PRIVATE per-org store whose every read binds org. A private
skill has no path onto the public discovery surface by CONSTRUCTION — different
containers — not by a filter someone has to remember to write.
Two providers share SourceSkill deliberately. The registry dedups by NAME with
equal-rank ties going to whoever registered first, and clients/agentskills
mounts at order 8 against this at 123, so a brand skill always wins a collision
with an org's. The shipped catalogue is the one that cannot be shadowed.
The write surface lives here rather than in clients/agentskills because that
subsystem mounts BEFORE iam to win the /.well-known discovery routes, so it has
no validated principal to scope a store by.
Co-Authored-By: Hanzo Dev <dev@hanzo.ai>
The playbook's nine failure modes each cost somebody a discovery, and this one
is larger than any of them: 56% of the fleet's described operations publish a
`summary` with a line break in it, across 18 packages, and nothing had counted
it because nothing reads the summary looking for one.
firstSentence (zip/openapi.go:606) returns the text up to the first ". "
verbatim — no whitespace collapse — so a doc comment whose opening sentence wraps
in the Go source ships that wrap into the OpenAPI summary, the CLI command
summary and the first docstring line of every generated SDK method. The summary
is a one-line field by construction, so this is a false rendering of a true
value, on the surface SDK users and models actually read.
Counted from the committed subsets with the command to re-count, per the house
rule that enumerated instances are always the ones somebody happened to look at.
The named fix is one whitespace collapse in zip, and the note says explicitly NOT
to reflow 215 doc comments: that keeps the class alive for the next op and makes
cloud a special case of a general bug. It also retires step 6's "keep sentence
one on one line" as the workaround it now is — zip v1.18.6 no longer cuts a
wrapped summary mid-sentence, it just carries the newline.
Also records that apps/git's 24/24 partition is now a GATE rather than prose,
next to team's, so the form is discoverable from the playbook and not only from
the app.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/team is at its typed floor already (9 ops typed, 10 refusals gated by
untypedByDesign) and this pass converted none — every one of the ten was
re-verified against zip v1.18.6's OWN source rather than against the comment
that claimed it, because "cannot be typed" is a claim about a dependency and a
dependency moves:
- op.invoke json.Unmarshals any non-empty body BEFORE the handler runs
(typed.go:227), which is exactly what turns the account JSON-RPC's and the
cookie PUT's deliberately TOLERATED garbage into a 400;
- the REST arm ends in c.JSON(out) with no raw-bytes and no upgrade path
(typed.go:303) — the wallet page's bytes, the blob download, and the two
WebSockets;
- WithStatus panics on a non-2xx — the two OAuth 302 redirects;
- hasBody("POST") is unconditional, and the multipart upload's part filename
IS the blob id, which no JSON In can carry.
v1.18.7 is byte-identical to v1.18.6, so the floor is 9 until zip gains
raw-body binding, a bytes Out, or a non-2xx status.
What the pass DID surface is one route away from the ops. team's second plane is
app-level — the Team front derives BOTH the Y.js WebSocket (GET /collaborator)
and the markup-snapshot RPC (POST /collaborator/rpc/{documentId}) from
COLLABORATOR_URL, not from the /v1/team base — and manifest.Apps named only
/v1/team. cmd/cloud builds the fleet router from that list, so both fell past
every prefix to the console the host serves at "/": the collaborative editor got
the HTML shell, and the TYPED collaborator RPC — published in openapi.yaml and
therefore in every generated SDK and in the MCP tool list — reached no app at
all. It was recorded in the router oracle's `unreachable` ledger, whose own
contract is that a fix is one line in Apps and one line out of the ledger.
team's row names /collaborator now; the two entries are gone from the ledger and
TestEveryServedPathReachesTheAppThatServesIt proves the router delivers both to
team. No route, status, body or field name moves — the document already said
team serves these; the router now agrees.
The general lesson, recorded in LLM.md: a route's typed-ness is invisible to the
manifest, so an app whose surface is not wholly under one /v1/<name> prefix can
publish a perfect op the fleet never delivers, and only manifest/router_test.go
asks the router.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/company is 21 typed ops of 23 routes. The other two carry a comment
explaining why they cannot be typed, and nothing else — no test asserted the
wire either reason rests on. A comment is not a gate: the next typing pass could
convert both, move the wire, and stay green.
POST /v1/company/payment answers the fleet-wide billing-denial contract —
402 insufficient_balance, 402 spend_cap_exceeded, 503 balance_unavailable, each
a {"error":{"code","message"}} body, the same shape the edge gate returns. zip's
HTTPError renders a FLAT {status,code,error}, so a typed op cannot express it.
That was the entire justification and the denial path had zero coverage:
fakeCharge has carried an `err` field that no test ever set. TestPaymentDenialWire
sets it, for all three outcomes, and reads the code and message out of the NESTED
object — so the flat shape fails. It also asserts a refused charge leaves the
formation unpaid, which keeps the machine's payment guard shut.
TestPaymentChargesLast pins the other half: the gate runs LAST, after the stage
check and the paid short-circuit. Arm the charger to deny and a wrong-stage call
still answers 409, a paid formation still answers 200 — which is only true if the
charge was never attempted. That ordering is why this gate cannot lift into
middleware, where it would charge a caller the machine is about to refuse.
TestDeckTakesRawBytes pins POST /v1/company/fundraise/deck: a PDF body is
ingested (a JSON decoder would refuse it with 400), ?name= names the document,
an absent name takes the handler's default, an empty body is the one 400. The
existing body-cap test happened to prove bytes are accepted; nothing stated the
contract, so raw() now returns the body and the deck's shape is asserted.
Both mutations were run: rendering the denial as zip.Errorf flattens the body and
TestPaymentDenialWire goes red; moving the charge ahead of the stage check turns
the 409 into a 402 and TestPaymentChargesLast goes red.
No wire moved. Source changes are comments pointing at the guards.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/account was already 11 typed ops of 18 operations; its seven refusals — the
GET|POST /v1/billing/* and five-method /v1/commerce/* bridges — were recorded in
prose only, which cannot stay true. The next route added here would have been
untyped and nothing would have gone red, so the migration could silently regress
to the state it was moved out of.
typed_wire_test.go closes the list. It reads BOTH projections of the live router
at their one shared address form — what the document says is served
(openapi.Spec) and which of those carry a typed registry entry (openapi.Typed) —
and fails on any operation that is neither. It mounts through mountBoth, so it
covers BOTH of this package's subsystem registrations (account @48 and
account-bridge @122); all seven refusals live in the second one, and a gate over
the self-service half alone would have declared the partition complete while
covering none of it. Three directions are checked: a served operation that is
neither typed nor named, a named reason for an operation no longer served (stale
prose), and a typed op the document does not serve (a published address that
404s). Verified to bite: a probe route added to the mount fails it by name, and
a fabricated reason fails it as stale.
The seven reasons are ONE wire fact re-verified against zip v1.18.6's own source
rather than taken from the comment that claimed it. The answer is commerce's own
bytes AND status (c.Bytes(status, raw), including a PDF at invoices/{}/pdf) where
a typed dispatch ends in c.JSON(out) under one declared status (typed.go:270-303)
and WithStatus panics on anything but a 2xx (typed.go:110); the body is forwarded
as received at any content type where op.invoke json.Unmarshals it BEFORE the
handler and 400s on failure (typed.go:225-231); and the path and query are OPEN
sets bounded by an allowlist (billingForwardable, commerceStoreHeads) where a
typed op publishes a closed parameter list. Opaque by construction, not by
omission — convert when zip ships raw-body binding and passthrough responses.
No route, no handler and no In/Out type changed, so the wire is untouched: both
regenerated subsets (plugin/account, plugin/account-bridge) come back
byte-identical, and zipdoc_gen.go is unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/git was already COMPLETE at 24 typed / 24 refused, but its partition lived
in LLM.md prose — and prose cannot fail. It decays two ways: a new raw route
lands and the count is silently wrong, or a refusal is retired and the reason
outlives the route it described. Either way the file still reads "COMPLETE".
untypedByDesign (typed_wire_test.go) is the same partition as a VALUE, carrying
the wire fact behind each of the 24, and TestEveryRouteIsTypedOrNamed checks it
in three directions: a served operation that is neither typed nor named, a name
git no longer serves, and a name that IS a typed op. Each direction was made to
fail before being trusted. TestEveryTypedOpIsDescribed pins the other half —
every typed op carries lifted prose, because an op committed without
regenerating zipdoc_gen.go is a nameless MCP tool. Same shape as
apps/team/typed_wire_test.go; one form of this gate, not a second.
All 24 refusals re-verified against the handlers and against zip v1.18.6, not
against the prose. The ZAP family's reason is now precise about WHY no shim
exists: a typed op's error renders zip's HTTPError {status:<int>, code, error}
(zip/ctx.go:200), so typing renames msg->error AND retypes status string->int,
and cloud.Bridge applies a handler-set status only when err == nil (typed.go:78)
with Created/Accepted the only exported setters.
Surfaced while verifying: firstSentence (zip/openapi.go:606) returns the text up
to the first ". " verbatim, so a first sentence that wraps in the Go source ships
its line break into the OpenAPI summary, the CLI summary and every generated
SDK's first docstring line. 20 of git's 24 ops, and 215 of the fleet's 387
described operations across 18 packages. Counted from the committed subsets, with
the command to re-count. The fix is one whitespace collapse in zip, NOT 215
reflowed doc comments — reflowing keeps the class alive and makes cloud a special
case of a general bug.
No wire change: zero routes converted (none remain that can be), zipdoc
regenerates identically, and plugin/git/openapi.json is byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/guide is 13 typed ops of 19 routes; the 6 that stay untyped each carry a
wire reason at the registration site. Five of those reasons were pinned by a
test — the structured 409 ({error, step, blockedBy}) by TestHTTPTransitionsAndGating
and the blocked-/do case, the YAML-or-JSON document bodies by
TestDocumentPutsAcceptYAML. The SIXTH was not: nothing exercised the SSE branch
of POST /v1/guide/steps/{id}/do, so the second half of that route's refusal was
an unverified claim. Delete the stream and every test still passed.
TestDoStreamsSSE pins it: asked for a stream by either trigger wantsSSE accepts
(Accept: text/event-stream, or ?stream=1), the route answers
Content-Type: text/event-stream and writes the agent's actions as frames —
`event: plan` through `event: end` carrying the terminal state. A typed op
answers exactly one JSON value, so the pin goes red on the exact change typing
this route would make (verified: stubbing wantsSSE to false fails it with
"got application/json; charset=utf-8").
No route, type or handler moves — the wire, the lifted prose (zipdoc_gen.go),
openapi.yaml and plugin/guide/openapi.json are byte-identical. The route comment
now names its evidence, so a reader of the refusal can check it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/o11y needed no conversion: re-audited all 20 routes from source before
reading the two prior commits, and reached their conclusion independently. 12 are
typed ops; the other 8 cannot be without moving the wire, and every reason
already recorded in LLM.md holds against the code (the two VM proxies forward
VictoriaMetrics' own status and envelope through c.Bytes(status, body); the two
builder queries and the sessions list are reverse proxies with no Go type for
"whatever the runtime answered"; the two alert routes are text/plain and the
receiver deliberately accepts an unparseable body; /v1/sentry/* is a wildcard).
Verified current, not assumed: zipdoc regenerates apps/o11y byte-identical, the
suite passes under the gate env, and openapi.yaml carries 11 described o11y
operations with their query/body schemas.
What the audit adds is the one fact that record was missing, and it is the fact
the open decision turns on. POST /v1/o11y/ingestion is a typed op that reaches no
consumer, because mountEventIngest only registers it behind a reachable
Datastore and the process that writes the document has none. LLM.md said closing
it means the path "stops falling through to the order-70 wildcard", and whoever
took it owned that wire change. Measured against the pinned runtime, that
fallthrough serves NOTHING: hanzoai/o11y v1.5.34 registers no /ingestion route at
all -- its only ingest-named paths are the unrelated
/api/v2/gateway/ingestion_keys* -- and a no-DSN process cannot init the embed
either, so mountRuntime installs the reverse-proxy fallback and the request lands
on the same server build, which also has no such route.
So the change on the table is 404 -> an honest 503, with no working write path at
risk. "Stops falling through" reads like remote ingest breaking, which is why
this looked more expensive than it is. Recorded with the command to re-measure
it, because a version-pinned claim goes stale.
No code and no wire touched: this is the description task finishing its own
record.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.801.320 denied EVERY paid inference call. The cause is one 401:
[billing] GET /v1/billing/balance err="sign in to view billing"
[ai] balance_gate: balance unverifiable for cold subject=hanzo:
commerce returned 401 (fail-CLOSED, retryable)
ai's prepaid gate reads this endpoint to admit or refuse a paid request.
build.go's wireFinance installs an in-process balanceReader so that read is a
direct typed call — but a Go func var cannot cross a PROCESS boundary, and once
ai became its own plugin process it stopped seeing the hook and fell back to the
HTTP path balance.go already documents as the split-deploy fallback. That request
carries COMMERCE_SERVICE_TOKEN rather than a user session, so principal.Org was
empty and the handler refused it. The gate is fail-closed on purpose — a balance
it cannot verify must never degrade to free inference — so a single unauthorized
read took down all paid traffic while every pod stayed Ready.
The fallback was supposed to work; now it does. Before refusing, the handler
accepts a caller bearing the verified service token, using the SAME predicate
apps/account already trusts (account.IsServiceToken — constant-time compare
against the configured COMMERCE_SERVICE_TOKEN), and takes the org from the
gateway-pinned X-Org-Id, which the gateway strips from every client request.
Scope is not widened. balance_s2s_test.go covers both directions: the trusted
read is served AND a wrong token, an absent token, a prefix near-miss, and a
valid token with no org are each still 401 with the ledger never touched. The
pre-existing TestBalance_RequiresSignIn — anonymous and forged-X-Org-Id both 401 —
still passes unchanged, which is the evidence this adds a trusted path rather
than opening one.
(The test needed one correction: mountApp sets COMMERCE_SERVICE_TOKEN itself, so a
t.Setenv before it is silently overwritten with "" — the token has to be passed
through mountApp. The first version of the test failed for that reason, not the
handler's.)
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The wip-preserved go.mod carried helm v3.21 with k8s.io/api floating to v0.36,
while the gitops-engine's replace block pins kubectl and the kubernetes staging
tree to v0.35.3. Two k8s minor versions in one binary fails twice over:
kubectl's scheme imports alpha groups (scheduling/v1alpha1) that v0.36 removed,
and component-base 0.36 registers feature gates kubernetes 1.35 also registers
— 'feature gate CRDObservedGenerationTracking with different spec already
exists', a panic before a single test runs.
api, apimachinery, client-go, apiextensions-apiserver, component-base and
streaming now join the same v0.35.3 replace set as every pin already there. One
k8s version, stated once.
apps/deploy (incl. TestRenderRealChartMatchesHelm — byte-identical to the helm
binary), apps/git and cek all green; whole module builds.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ai's router registers the OpenAI-compatible endpoints at TOP LEVEL, and apps/zen
holds the other half: zen is "a MIDDLEWARE, not a route owner" that claims zen
SKUs and calls c.Next() "so ai's /v1/* catch-all serves non-zen models". So ai
must claim a prefix those paths fall under — and when its row read "/v1/ai" it did
not: POST /v1/chat/completions answered 405 and GET /v1/models 404 on v1.801.318
and .319, with the pod Ready, probes green and chat.hanzo.ai serving 200. Every
SDK caller got nothing and nothing looked wrong.
The test asserts ai CLAIMS those paths, not that it claims them FIRST, and that is
the whole point. I wrote the first-match-wins version first and it passed with the
bug reintroduced — zen holds /v1 and precedes ai, so every path "matched"
something — then a second version blamed commerce for owning everything. Apps
legitimately hold overlapping prefixes and chain through c.Next(); presence in the
chain is the invariant, position is not. Both wrong versions were run against the
actual bug before either was discarded.
Also NOT asserted, on purpose: the zen-before-ai ordering. apps/zen documents it,
but that comment predates this manifest (it names Wire(), which is gone) and the
live order is the reverse. A test encoding it failed against intentional upstream
state, so it is a documented open question for whoever owns the split rather than
a build break — the orders differ for billing, since zen's Gate/Meter only run if
zen's Claim sees the request first.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TWO defects, one root: an app's declared prefix is the ONLY thing the light host
routes on, and a wrong one fails silently.
1. INFERENCE WAS UNREACHABLE. ai's row said Prefixes{"/v1/ai"}, but ai's router
registers the OpenAI-compatible surface at TOP LEVEL — /v1/chat/completions,
/v1/models, /v1/messages, /v1/completions, /v1/responses, /v1/embeddings — and
apps/zen states the other half of the contract: zen is "a MIDDLEWARE, not a
route owner" that claims zen SKUs and calls c.Next() "so ai's /v1/* catch-all
serves non-zen models". Scoped to /v1/ai, ai dropped out of the /v1 chain, so
zen's c.Next() fell through to the console catch-all: POST
/v1/chat/completions answered 405 and GET /v1/models 404 on v1.801.318 and
.319 — pod Ready, probes green, chat.hanzo.ai serving 200, and every SDK
caller getting nothing. Nothing looked wrong from outside.
2. commerce CLAIMED A BARE "/v1". That silently overlapped billing, catalog,
projects, agent, agents and kms, and it made a future /v2/commerce impossible:
a bare version root swallows every sibling under that version, including ones
that do not exist yet. Replaced with the nine prefixes apps/commerce actually
registers — behaviour-preserving, only narrower.
The guard is what keeps this from returning. TestNoBareVersionPrefix rejects any
/vN root (verified: it fires on a planted bare /v2), with a CLOSED exemption for
exactly zen and ai — zen because it dispatches on a body field and cannot
enumerate paths, ai because it is the catch-all zen falls through to. A third
entry has to be argued for in a test diff.
TestInferenceSurfaceIsRoutable asserts ai claims the paths customers' SDKs call.
I got that test wrong twice before it was worth having. A first-match-wins model
PASSED with the bug reintroduced (zen matches /v1 first, so the path looked
routed), then blamed commerce for owning everything. Several apps legitimately
hold overlapping prefixes and chain through c.Next(), so the real invariant is
"ai CLAIMS these paths", not "ai claims them first". Both versions were checked
against the actual bug before being kept.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cek refuses to open a store without a master key on any build that can
encrypt, so the four store-backed tests here failed while agents, authors,
automations and catalog all passed — which reads as a defect in the tools plane
rather than a missing two-line TestMain. Same harness, same throwaway key, and
it never overrides a key the environment already provided.
Verifiable: the failure moves from "CLOUD_KMS_MASTER_KEY_REF is required" to
the platform's RAM-backed-scratch refusal, i.e. the key now lands and the
remaining wall is isRAMBacked being false off Linux by design.
Co-Authored-By: Hanzo Dev <dev@hanzo.ai>
POST /v1/plugins/build takes TypeScript, or an API document to generate it
from, and returns a plugin the runtime has already loaded once.
The build is the SAME pipeline the committed connectors go through — esbuild
to one CommonJS program (connectorruntime.Bundle), then compiled in goja
(Runtime.Compile). Compiling IS the gate: source that bundles but will not
load is rejected and never stored, so "it built" means the artifact this
deployment will execute compiled, not that a model produced plausible text.
A failed build returns the generated source with the error, because the source
is the thing worth reading when generation goes wrong.
CREDENTIALS ARE NOT PART OF A PLUGIN. A plugin names the connectors provider
it needs and reads ctx.auth at run time, where the secret is already under KMS
custody. Source that carries something shaped like a key is REFUSED, not
scrubbed — a silently-stripped key looks like it worked, and the caller never
learns the secret went somewhere it does not belong. That also means rotating
a key never means rebuilding a plugin.
/v1/plugins lists what this DEPLOYMENT mounted; /v1/plugins/authored lists what
an ORG built. Different sets, different lifecycles, so a subpath rather than one
mixed collection.
audrecord grows an action parameter instead of a copy: the builder records
plugin.build, which is a different act on a different resource than tools.call.
Tests cover the gate itself — a connector bundles, a syntax error and an empty
name are refused, and the credential shapes are caught without false-positiving
ctx.auth or propsValue.token. One of them found a real bug: stripFences trimmed
before removing the closing fence, leaving the newline behind.
Co-Authored-By: Hanzo Dev <dev@hanzo.ai>
/v1/tools answered "what can this org call" only if the caller knew to pass
?source=. Give the three sets people actually ask for their own path.
skills and mcp are VIEWS over the same registry — one implementation
(listBySource) mounted twice — so a tool is still registered in exactly one
place and activation still lives in exactly one place. A source view filters
the per-principal list, so it can never widen what a caller may see.
plugins is deliberately NOT a tool source. A plugin here is a mounted
subsystem (cloud.Plugin: Name, Mount, Price, Prefixes) — code that extends the
deployment's surface — while a tool is something called through that surface.
Its inventory is cloud.Subsystems(), the boot snapshot Declare installs, so it
reports what the binary actually mounted rather than a second list free to
disagree. An earlier draft added a SourcePlugin with no producer; that was
speculative and is gone.
The external MCP server registry MOVES /v1/tools/servers to /v1/mcp/servers.
A server is a record an org creates, not a tool the registry enumerates, and
splitting one MCP integration across two prefixes had no defender.
Declare the new prefixes in BOTH manifest/apps.go and plugin/tools/main.go: an
undeclared prefix resolves to no subsystem, which would leave every request to
these paths priced Undeclared and unlabelled in tracing.
Co-Authored-By: Hanzo Dev <dev@hanzo.ai>
Sixteen /v1/compliance routes become typed ops (health, status, records, audit,
subjects CRUD-reads, verifications start/list/get/refresh/decision,
accreditation create/list/get/decision), so each now projects to OpenAPI, MCP,
the CLI and the SDK from the one registration. cloud.Bridge is installed on the
group — the subsystem had none, so no typed op could have resolved its org here
— and the package gains its //go:generate zipdoc directive, also missing.
One route stays untyped, naming its wire fact at the registration and the
handler: the provider webhook authenticates by HMAC over the RAW body bytes,
verified before any parse — a typed op decodes its In first — and an unknown
reference answers a second 200 shape (a benign no-op, not a check).
Wire preserved exactly: 201 on the three creates via zip.WithStatus, the
map-built views become structs whose omitempty matches the maps' conditional
keys, bodyCap keeps the 1 MiB / 413 gate in front of the parse it has always
preceded, and noStore keeps Cache-Control: no-store on the PII-bearing reads.
TestTypedOpsPreserveTheWire pins the envelope (no-store, 413, ?limit binding,
empty-body tolerance); the behavior suite passes unchanged against baseline.
The reviewer role gates, emitAudit's attribution and noStore are the pinned
cloud.Request uses (allowlisted); the tenant itself is principal.OrgFrom, never
the request. Typing surfaced two false doc claims before they shipped as prose:
the audit read returns rows NEWEST first (ORDER BY seq DESC), and a check's
status vocabulary includes expired — plus result filters are success|deny|error,
not "denied".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/automations/connectors/{id}/run becomes a typed op, so the last
schema-less stub under /v1/automations now projects to OpenAPI, MCP, the CLI
and the SDK from its one registration: runIn ({action, auth, props} + the path
id) in, runResp ({ok, output, error}) out, with the infra-vs-piece split kept
exactly — an action that ran and failed answers HTTP 200 ok:false, never a
5xx; unknown connector 404, missing action 422.
Typing surfaced the same two defects it surfaced in automations proper: the
package had NO cloud.Bridge (registered bare on the app root, it worked only
because automations' group Bridge happened to cover the prefix — mounted alone,
a typed op could never have resolved its org) and NO //go:generate zipdoc
directive, so no prose could have reached the document. It also had no wire
test at all; http_run_test.go now mounts the subsystem ALONE and pins the
403/404/422 gates and the ok:false 200 — the standalone mount is what proves
the new group Bridge makes it self-contained.
Regenerated: apps/connectorruntime zipdoc, plugin/automations/openapi.json,
openapi.yaml (weave green). automations + connectorruntime + openapi tests
green under the Makefile gate env.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The two document writes (POST /v1/framework/:doctype, PUT
/v1/framework/:doctype/:name) stay raw, re-verified against zip v1.18.6:
typing them takes THREE halves of one zip capability, not the two the
rawRoutes pin named. Beyond declaring an open object and binding the URL
onto one, the bound params must ride OUTSIDE the body namespace — off the
REST path op.invoke receives no path map (MCP tools/call and the call
plane pass nil), so URL params could only travel as body keys, and a
create body's `name` IS the requested document name (engine ops.go,
stringField(in, "name")). Folding :name into the body collides with a
field the document owns, so a map-In workaround would ship ambiguous
MCP/CLI projections. No wire change; no generated artifact moves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The typed migration for apps/guide landed in 7e50740e (13 of 19 routes; the six
refusals each named at their registration), but two of its refusal facts were
prose only: PUT /v1/guide/curriculum and PUT /v1/guide/blueprint accept a raw
YAML-or-JSON document (Parse, sigs.k8s.io/yaml), and no HTTP test sent YAML —
the acceptance was pinned at the Parse unit level, one layer below the wire the
refusal is about. TestDocumentPutsAcceptYAML PUTs raw YAML through both routes:
typing either one (a typed In is decoded as JSON before the handler sees it)
now turns CI red instead of silently 400ing every YAML caller.
The 409 family (steps/:id/start|done|do answer a structured {error, step,
blockedBy} body a zip.HTTPError cannot carry) was already wire-pinned by
TestHTTPTransitionsAndGating; those routes wait on zip multi-status.
LLM.md's tranche table still listed guide as open — the table is the
migration's coordination surface, and an uncrossed row invites a duplicate
effort. Crossed out with the counts.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every route on this surface that CAN be a typed op already is (72a87f87 typed
15 of 32); the other 17 are refused for wire facts. Those facts lived only in
ops.go prose, which nothing enforced — the next route added here would default
to raw and nobody would notice, and a stale reason would outlive the route it
described. apps/team/typed_wire_test.go is the worked example of doing better:
the refusals are a CLOSED list a test holds against the live router.
pricing now carries the same gate. untypedByDesign names the 17 raw operations
at their document-form addresses, each with the wire fact that keeps it raw;
TestEveryRouteIsTypedOrNamed fails on any operation that is neither a typed op
nor on that list (so the next pricing route is typed by default) AND on any
listed operation the surface no longer serves (so the list cannot go stale);
TestEveryTypedOpIsDescribed fails on a typed op whose prose did not reach the
registry (an op added without regenerating zipdoc_gen.go is a nameless MCP
tool). The filter is the package's own Prefixes — the same five cloud.Declare
scopes by — so a route mounted outside what the subsystem declares surfaces as
uncovered, which is exactly the undeclared-prefix defect 72a87f87 fixed.
Each refusal was re-verified against zip v1.18.6's own source rather than
carried forward from the comment that claimed it, because "cannot be typed"
is a claim about a dependency and a dependency moves:
- the 15 verbatim proxies: a typed dispatch ends in c.JSON(out) under the one
status the op declared (typed.go registerTyped), so the bundle's own status
(200/503) + unmodified bytes are still inexpressible;
- PATCH models/{wildcard1}: bindURL matches path params by json field name,
so binding fiber's *1 still takes an In field tagged json:"*1" that every
projection would publish;
- PATCH providers/{name}: schemaOf still reflects json.RawMessage ([]byte) as
an array of integers (openapi.go, reflect.Slice arm) — a false schema — and
map[string]any still cannot hold RFC 7386's explicit-null-deletes.
No route, wire byte or artifact moves: zipdoc_gen.go, plugin/pricing/
openapi.json and the golden regenerate identical. These convert when zip ships
raw passthrough / multi-status (#78's family), and the gate is where that
conversion will be forced into view.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ai is a LAZY plugin (mounts on the first /v1/chat/completions), so it can mount
before the app that installs the balance reader. The hooks were wire-time
SNAPSHOTS, so nil latched for the process lifetime; the ai module then fell back
to an HTTP self-call to /v1/billing/*, which the edge 401s (the toothless-gate
bug build.go already names), and the balance gate is fail-CLOSED — so every
completion answered 503 balance_unavailable.
Observed in prod on v1.801.320: 'balance_gate: balance unverifiable for cold
subject=hanzo: commerce returned 401 (fail-CLOSED, retryable)' on a pod whose
commerce plugin mounted ten minutes later. Chat, studio's assistant and every
SDK caller were down on a healthy pod.
Resolve cloud.TierReader/BalanceReader/UsageRecorder per call — the pattern the
rolling-cap hook in this same function already uses and documents. The balance
trampoline fails LEGIBLY when nothing is wired rather than reporting 0, which
would read as a real zero balance and deny a paying caller.
The o11y typing pass landed in d0bb72e7 (+6e22e075): 12 of 20 routes are
typed ops; the other 8 cannot be typed without moving the wire, each named
with its reason in apps/o11y/LLM.md — 2 verbatim-status VM proxies, 3
reverse proxies (query/query_range/sessions), 2 text/plain Alertmanager
receipts, 1 sentry wildcard. Re-verified against source at ac98f6ea:
zipdoc regenerate + make -C apps/o11y openapi produce zero drift, package
tests green with the gate env. The table's "o11y 11" overcounted: the
re-measure grep picks up 3 comment lines quoting app.All("/v1/o11y/*")
(event_ingest.go:14,221 + scope.go:17); the real untyped count was 8.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The partition table still listed crm 20 as pending while the inventory
forty lines down records it done: 19 typed ops, one refusal (the public
Startup Program intake POST, whose IP rate limit and pre-parse 64 KiB
body cap are wire). Re-verified on this tree: re-measure finds the one
raw registration only; zipdoc and plugin/crm/openapi.json regenerate
byte-identical; the gated tests pass; the subset audits clean against
the bodyless-POST, embedded-struct and empty-leaf classes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/crawl is served by this package -- manifest/apps.go is mount order AND
routing order, crawl at 78 against ai at 111 -- and Fetch is one http.Get. For
the client-rendered part of the web that returns a near-empty shell, extract
finds almost no text, and the caller gets 200 with nothing in it. A crawl that
returns nothing and reports success is worse than one that fails, because
nothing upstream can tell.
So: static first, and if what came back is under 512 bytes of markdown, ask
Hanzo Crawl (headless Chromium, ghcr.io/hanzoai/crawl:sha-7b8dc59, CR landed in
universe e0942b664) for the rendered version.
Length is the whole heuristic on purpose. "Does this have a <div id=root>"
recognises today's frameworks and misses tomorrow's; "the extractor found
almost nothing" is the symptom itself and does not date.
Three things keep escalation from being a downgrade:
- Longer text wins, not "the browser answered". A render can come back thinner
-- a consent wall, a bot check, a page that needed no JS -- and taking it on
faith would make this a regression on exactly the pages it should not touch.
- An absent, slow or unhappy browser leaves the static Page standing. That is
the state this ships in, since nothing has deployed the CR yet.
- A sufficient page never pays for a render at all.
TWO FINDINGS FROM THE TESTS, both of which had already shipped in my first pass:
1. It could never have worked. browse() used `client`, whose dialer refuses
non-public addresses -- and the browser lives at crawl.hanzo.svc, private by
design. Every escalation would have been refused with ErrBlocked. There are
now two clients for two trust classes: `client` dials wherever a CALLER
asked and stays guarded, `service` dials the one address WE configured.
2. It was an SSRF bypass. Read escalates when Fetch FAILS, and one reason Fetch
fails is the guard refusing an internal address -- so "crawl http://10.0.0.1/"
would have been refused here and then forwarded to a Chromium that fetches it
happily, reachable by anyone who can call /v1/crawl. reachable() now applies
the same check before the browser sees a URL, including refusing a host that
answers with ANY internal address, since one public IP listed beside the
target is the documented way around a first-answer check.
The resolver is a var so that boundary is testable with a hostile answer; a
check you cannot test with the attack is one you are only assuming holds.
11 tests, all passing, covering both halves of the escalation contract, both
markdown wire shapes, and the guard.
Six independent reds, five of which were the test lying about the code rather
than the code being wrong. Each is named here because the distinction is the
finding.
kmsreseal read nothing from the standalone. One client built one URL for two
faces that spell the tenant differently: luxfi/kms serves
/v1/kms/orgs/{org}/secrets and matches the path against the token's orgs, while
cloud's apps/kms serves /v1/kms/secrets with the org from the principal. The
client now carries the `route` of its face — the only per-face difference there
is — named at each construction. Second bug found the same way: listFolder
decoded only cloud's `secrets[].name`, so every folder-sync CR would have
reported "folder EMPTY at source"; both faces emit `names`, so one decode reads
either. The isolation probe asserted a cross-org 403 on a URL that no longer
exists; it now asserts the stronger property that replaced it — a tenant-naming
URL reaches no route at all, carrying a VALID credential so the 404 means "no
such URL", not "no such caller". runbook.go claimed an isolation matrix verify
does not run, and that a base-URL repoint is all a consumer needs; cloud serves
no /v1/kms/orgs/… route, so it is not.
apps/storage's bucket tests never reached the assertions they made: guard gates
money first, the $1.00 default fee met an unconfigured commerce, and all three
died 503 before any name or key was validated. The subject of that file is the
tenant and validation gates, so the fee is 0 there — the documented un-gated
posture — and the priced posture stays billing_test.go's subject. The 13s per
app was construction, not the request: BuildDeps probes the store against an
address nothing listens on, exhausting the retry budget three times. It now
points at an in-process endpoint that refuses, which reaches the same posture in
microseconds and makes the file self-contained. That let doOrTimeout go, and
exposed TestBucketsRouteReachesS3NotProvisioning as vacuous — its with-org leg
503'd at the money gate and never reached the s3 handler it claimed to prove
owned the route. It now asserts the exact 502 only that handler can produce.
TestRunnerBuild_IAMReleaseRejected sent a SuperAdmin and expected a refusal, so
it was reading a correct admission as a failure — and made a live call to
api.github.com from a unit test to get there. It now sends an org admin, which
is the property it is named for. The permit side shipped with no coverage at
all; it is pinned now, hermetically, on stubbed seams.
apps/graph's flake was a real DNS round-trip per request against a 1s deadline,
reproducible 6 runs in 10 under resolver load. Port 0 is not listenable, so the
kernel refuses locally and immediately: unreachable becomes a fact of the test
rather than a name the resolver has to fail to resolve.
books gained two cloud.Request sites. Both earn their place — narrateAsk reads
principal.Ledger, which a SuperAdmin masquerade moves off the effective org, so
principal.OrgFrom would bill the org being inspected. The general untyped
`query(ctx, name)` did not: it justified one fact and permitted any. It is now
sandboxFrom, which can express only the ledger selector and composes the same
sandboxQuery the untyped handlers use, so both planes resolve it by one rule.
And the reason seven subsystems each spelled their own gate: there was nowhere
to put it. gate.go is that place — Authority (what a caller has, the three
HIP-0519 predicates and nothing else), Scope (what a route requires), Admits
(the whole rule, one expression, with `Super || OrgAdmin` written out so the
superset is visible at the gate), Guard (its standard fail-closed application).
kms, platform's fleet, and storage now read it; deploy and admin/core keep their
own refusal shape but move onto the canonical predicate. Everything that is not
authorization — store readiness, org syntax, per-op billing, tenant confinement
after the door opens — composes around it and stays in the app, so no real
difference was flattened. platform's two transports, HTTP and the internal
plane, now read the same rule instead of two copies of it.
The ingress boundary is untouched: app.Use(IdentityMiddleware) stays until the
network policy makes the gateway the only ingress, because the red-team probe
reads another tenant's secret without it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Strike it from tranche D and record the split where the next agent looks first:
the two latent defects typing surfaced (missing cloud.Bridge, missing zipdoc
directive) and the wire fact behind each refusal (two success bodies, arbitrary
JSON value, raw-byte dedupe + headers, JSON-RPC 200-on-parse-error).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestRequestEscapeHatchIsPinned has been red on main since apps/books went typed:
ask.go (narrateAsk reads the BILLING ledger off the request — a header fact,
never an In field) and typed.go (query reads a URL-borne value for a
body-carrying op, where an In field would move it off the URL). Both are the
exact uses the escape hatch exists for; the entries state why, from the call
sites' own prose.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Fourteen /v1/automations routes become typed ops (connectors + pieces alias,
flow CRUD, versions, run/enable/disable, run history), so each now projects to
OpenAPI, MCP, the CLI and the SDK from the one registration. cloud.Bridge is
installed on the group — it was MISSING, so no typed op could have resolved its
org here — and the package gains its //go:generate zipdoc directive, also
missing. populatedFlow spells out the Flow fields it embedded, so the published
schema matches the wire instead of documenting a nested object the route never
sent. setEnabled returns the flow instead of writing the response, one seam for
its three callers; auditHTTP is the pinned cloud.Request use (allowlisted) that
keeps the enable/disable audit record attributed.
Four routes stay untyped, each naming its wire fact at the registration and the
handler: operations (TWO success body shapes — Flow on CHANGE_STATUS, else
FlowVersion), resume (arbitrary JSON value body, raw-byte size gate), hooks
(raw-byte content-hash dedupe + two contract headers), mcp (JSON-RPC answers an
unparseable body 200 with a -32700 object; zip would 400 it).
Statuses preserved exactly: 201 create/version/run via zip.WithStatus, 204
delete via nil Out, 200 elsewhere; the package's own HTTP tests pin them and
stay green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerce's row held bare "/v1", which made it the fleet's route of last
resort: /v1/chat/completions, /v1/models, /v1/embeddings, /v1/responses,
/v1/audio/* — the whole OpenAI-compatible surface — landed on commerce and
answered its 404, for every caller. apps/commerce/mount.go had warned in
prose that reading its "/v1" group as a claim would hand commerce every
request in the fleet; the manifest did exactly that.
Proved on the real router (router_test.go's oracle, built from manifest.Apps
through the same zip.Load the host calls): before this change all nine
OpenAI paths -> commerce; after, all nine -> ai, and every path commerce
publishes still reaches commerce or its recorded owner.
- commerce now owns its published FAMILIES, each named DEEPER than the
sibling that shares the stem, so catalog keeps bare /v1/catalog, plan
keeps the rest of /v1/plans/*, and account-bridge keeps the console's
/v1/commerce/* + /v1/billing/* data bridges. Not commerce.Prefixes
imported — the app states its fail-closed set once; the row states what
the router may hand it; the oracle keeps the two honest.
- ai's row is "/v1": the remainder, behind every deeper prefix. Its own
/v1/* catch-all is what serves the OpenAI surface.
- ai precedes zen (frozen order edit — a decision): equal "/v1" claims
resolve by mount order, and zen's Claim-middleware contract cannot be
expressed as a per-process prefix, so its row is deliberately shadowed.
- metrics gains /v1/logs + /v1/traces, its own published ingestion doors,
which the bare-/v1 row was swallowing (405, silently).
Ledger: 26 unreachable entries routed (ai's catch-all, commerce's webhook/
auto-recharge/commerce/catalog/plans families, metrics' seven doors);
27 remain recorded. Oracle: 1005 published paths, 978 reach their app.
TestOpenAISurfaceLandsOnAI pins the nine concrete OpenAI endpoints so the
single-path catch-all can never again hide the whole product behind one
ledger line.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip's structSchema publishes an embedded EXPORTED struct as one NESTED
property named after its type, while encoding/json PROMOTES its fields —
so every step object in GET /v1/guide and the skip/reset ops documented
{JourneyStep: {...}} for a wire that has always been flat {id, title,
deps, ..., state}, in openapi.yaml, in every generated SDK and in the MCP
tools' schemas. Fourth live instance of the embedded-struct class (recipe
rule 7: patchTargetIn, botView, clusterDetailView), and the published-
subset check that now rides in LLM.md finds two more in plugin/admin
(MetricsData -> SaaSMetrics, ServiceView -> ServiceRow).
The wire is untouched: the fields are inlined in promotion order with the
same json tags, TestStepViewCarriesJourneyStep pins the spelled-out copy
against JourneyStep so a field added later cannot silently drop out of
the view, and the step fields now carry their prose in the document
instead of riding on a property that never existed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The one convertible refusal of the six: zip's schema walk publishes an embedded
struct as a NESTED property while encoding/json flattens it, so the Out spells
Metrics' fields out flat (the sanctioned fix for the embed class) and
metricsResponseOf is the one constructor. The drift the old refusal feared is
pinned red by TestMetricsResponseCarriesEveryMetricsField (reflection-filled,
so a new Metrics field cannot silently drop off the wire), and
TestMetricsSchemaMatchesItsWire keeps the published schema equal to the wire
keys from here on. Wire unchanged: same 401/500 order and messages, same flat
JSON, same no-store, same operationId (get_v1_books_metrics), sandbox/from/to
still query parameters.
The other five stay untyped, each re-verified against zip v1.18.6's source:
scan, inbox and bank/import take raw document bytes that op.invoke would
json-decode into a 400 before the handler runs (typed.go:231); link-token and
exchange answer 501 unconditionally, and WithStatus panics on a non-2xx
(typed.go:110) — a typed op would publish a success contract neither has ever
sent.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/team converted in 7889d98c + ea0804f9 (9 typed ops); the other 10 routes
cannot be typed without moving the wire, and each is named with its reason in
untypedByDesign (typed_wire_test.go), which TestEveryRouteIsTypedOrNamed
enforces as a closed list. Re-verified this pass against zip v1.18.6: the typed
success path is unconditionally c.JSON (typed.go:304) and a typed POST/PUT
unconditionally JSON-decodes its body (hasBody, typed.go:276), so the upgrade,
redirect, multipart, raw-bytes and tolerant-bind routes are all wire-refused.
zipdoc regen is a no-op, plugin/team/openapi.json regenerates byte-identical,
apps/team tests green under the gate env.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The two cloud.Plane() ops (/git/files, /git/publish) were registered as
closures, and the commit that added them never regenerated zipdoc_gen.go, so
zipdoc -check was red on main for apps/git and the registry carried no prose
for either op. Named handlers (planeFiles, planePublish) with true doc
comments; regenerated. plugin/git/openapi.json is byte-identical — plane ops
never reach the public document.
The package is otherwise COMPLETE: 24 typed / 24 refused. apps/git/LLM.md now
names each refusal's wire fact with line cites (raw-byte HMAC webhook,
smart-HTTP pack protocol x6, server-rendered HTML x12, ZAP envelope adapters
x5) and counts the one bodyless-POST instance (/repos/{name}/gc). Root LLM.md
tranche B marks git done.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/account went typed on main in 0a134476 + cb226978 (11 of 18 ops; the
Bridge it never had installed in the first). The tranche-C table still said
'account 19', which is exactly the stale-count failure the company section
warns about — it sent another agent to redo the package. Verified at
8087c757: zipdoc + both openapi subsets regenerate byte-identical, tests
green under the gate env. The seven refusals (the /v1/billing/* and
/v1/commerce/* verbatim wildcard bridges) are now recorded with their wire
facts so they are never re-derived.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/crm converted in a15f5ca3 (19 typed ops, one raw route: the public Startup
Program intake, whose IP rate limit and pre-parse body cap are wire). The LLM.md
inventory never picked it up; re-measured this pass — zipdoc regen is a no-op,
apps/crm tests green under the gate env, the intake is the only raw registration.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
721a3039 dropped app.Use(IdentityMiddleware) on the reasoning that identity is
the gateway's and is verified once. That reasoning is right and is now written
down as HIP-0519. Its correctness rests on one assumption the HIP states
plainly: the gateway is the only ingress.
That assumption does not hold here yet. With the middleware gone, the estate's
own red-team probe reads another tenant's secret VALUE off the in-cluster KMS
listener:
PROBE (b) forged org + forged X-User-Id + IsAdmin → 200 {"value":"…"}
TestAudit_AnonRequestNotAttributedToForgedOrg fails the same way: a forged
X-User-IsAdmin survives into the audit record and an anonymous request is
stamped with a claimed org.
So the middleware stays. Reaching HIP-0519 is a network-policy change FIRST —
service listeners unreachable except through the gateway — and a code deletion
second. Doing the code half alone is a cross-tenant secret read, and those two
tests are the gate on the real work rather than obstacles to it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
721a3039 removed app.Use(IdentityMiddleware) — cloud's second verification of a
token the gateway had already verified — and said nothing about it. A commit
about the money gate carried a change to the identity boundary in its diff and
not in its message. Recording it here rather than leaving it to be discovered.
The change itself stands and is now specified: HIP-0519 defines identity as
verified exactly once, at the edge, against IAM, with everything behind it
reading the assertion and forwarding it unchanged. Cloud reads; it no longer
re-derives.
The stale comments that still described the middleware as a step in this chain
are corrected, because a comment describing a middleware that is not installed
is worse than no comment.
NOT DONE, and named so it is not mistaken for done: middleware_identity.go,
auth_identity.go's validator half, and identity_cache.go still COMPILE — nothing
installs them, but ~1,300 lines of a second identity implementation remain in
the tree, and HIP-0519 conformance means deleting them. The cut is mechanical
but not trivial: token-shape predicates, OrgHasUnsafeRune, cookieTokenNames and
OrgForKey live in those files and serve audit, analytics and tenancy rather than
authentication, so they must be lifted out first. NewTokenValidator is a real
second question — apps/team and apps/deploy verify a token at LOGIN MINT time,
which is a different moment from an inbound request and needs its own answer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Revert of 06e0a0e8, plus the fix it should have been.
I made an unreachable biller ALLOW, to turn three red tests green. Those tests
were the stale ones: they assert the pre-split rule, where no commerce URL meant
nothing billed. resource_billing_test.go states the rule that replaced it, and
states why — once apps are their own binaries the ledger has ONE writer and it
lives with commerce, so a meter without a local URL asks it, and a biller it
cannot reach is UNKNOWN, never allowed. Allowing turns every priced act free the
moment an app is split out, silently.
Four tests now agree on that instead of three contradicting one, and each says
what it is protecting: a priced invoke, create and op are refused with no biller
reachable, and the sandbox, provisioner and handler run ZERO times — because
doing the work first and discovering later that nobody could bill it is a
resource somebody has to find.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Gate's own comment named the distinction — "nobody bills in this deployment"
versus "the biller is one socket away" — and the code never made it. gatePeer
treated a missing socket as an error, so a deployment with no commerce at all
503'd every priced act, which is the behaviour "billing not configured" was
never supposed to have.
The socket is what answers it. No socket means this deployment does not run
commerce, so the gate is inert, exactly as before the split. A socket that
exists and does not answer is a real fault and still fails closed — which is
the direction that matters, because allowing there is what once turned every
priced act free, silently.
Both directions are pinned: TestGate_NoMoneyPlaneIsInert and
TestGate_UnreachableBillerFailsClosed, the second against a real socket that
accepts and hangs up. PeerPresent requires a socket, not merely a name on disk.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The op-call plane marshalled its body with encoding/json, so two of our own
processes on one host — holding the same struct in memory — serialized it to
text and parsed it back to cross a socket. That is a boundary encoding doing an
internal job.
zip v1.18.6 carries the plane in ZAP: the layout is derived from the In/Out type
itself, a field is its offset, and no name travels. The bytes on the socket are
the bytes in memory, which is what the hand-written codecs achieved and what
replacing them was never supposed to give up. Refusals cross as ZAP too, status
intact.
Nothing in cloud changed to get it — the ops were already typed, so the encoding
moved underneath them. TestPlaneWireCarriesNoFieldNames pins the result against
the real socket: the values cross, the field names and the braces do not.
The `json` tags on plane types name fields in the OpenAPI schema this plane also
projects. They are the document's vocabulary, never the wire's — and because the
layout is the type, fields may only be APPENDED.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/base v1.5.11 (pulled in by da53ab30) turns a long-standing silent
mismatch into a compile error on purpose:
base@v1.5.11/core/sqlite_math_required.go:30:6:
undefined: cgoBuildNeedsSQLiteMathFunctions
base's search layer generates SQL calling acos/cos/sin/radians/sqrt (the
geoDistance token in tools/search). SQLite only has those with
SQLITE_ENABLE_MATH_FUNCTIONS, and the cgo backend gets them ONLY behind
csqlite's sqlite_math_functions tag — so a cgo build shipped a SMALLER SQL
surface than the code above it writes against, and the failure surfaced as a
customer's search returning 'no such function: acos' from an endpoint that
works in production. The file is //go:build cgo && !sqlite_math_functions and
references an undefined symbol so the two build modes cannot disagree in
silence.
Added to all four CGO_ENABLED=1 tag sites (the modernc gate, the two codec
tests, and the per-plugin build) plus the tag string quoted in the gate's own
error message. The CGO_ENABLED=0 builds are untouched: the pure-Go backend
always has the functions.
Verified against v1.5.11 in a scratch module: without the tag the exact CI
error reproduces (exit 1); with it, exit 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/base v1.5.11 (pulled in by da53ab30) turns a long-standing silent
mismatch into a compile error on purpose:
base@v1.5.11/core/sqlite_math_required.go:30:6:
undefined: cgoBuildNeedsSQLiteMathFunctions
base's search layer generates SQL calling acos/cos/sin/radians/sqrt (the
geoDistance token in tools/search). SQLite only has those with
SQLITE_ENABLE_MATH_FUNCTIONS, and the cgo backend gets them ONLY behind
csqlite's sqlite_math_functions tag — so a cgo build shipped a SMALLER SQL
surface than the code above it writes against, and the failure surfaced as a
customer's search returning 'no such function: acos' from an endpoint that
works in production. The file is //go:build cgo && !sqlite_math_functions and
references an undefined symbol so the two build modes cannot disagree in
silence.
Added to all four CGO_ENABLED=1 tag sites (the modernc gate, the two codec
tests, and the per-plugin build) plus the tag string quoted in the gate's own
error message. The CGO_ENABLED=0 builds are untouched: the pure-Go backend
always has the functions.
Verified against v1.5.11 in a scratch module: without the tag the exact CI
error reproduces (exit 1); with it, exit 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
rpc.go, dial.go and payloads.go were a second implementation of the op-call
plane zip already has: an fnv-hashed method registry, payloads packed by hand
against literal byte offsets, and a capability the callee parsed with nothing
verifying it. Sixteen methods lived there — the prepaid gate, the meter, the
balance, the welcome grant, the statement, the secret reads, the mailable
roster, the fleet — and not one of them appeared in the OpenAPI document, the
MCP tool list, the CLI or any SDK, because none of them was an op.
They are ops now. One registry, and the internal surface is as described as the
product surface. 1,300 lines of transport deleted; the contracts that remain are
types in plane/, a leaf both ends import so neither drags the other's graph.
The ops are declared on a SECOND app. A typed op rides every transport its app
listens on, and the host proxies edge traffic to its children over a private
socket, so neither "is this HTTP" nor "did this arrive on a socket" separates an
internal call from a public one. The plane app listens on exactly one address
and is never mounted on the edge router, so there is no path from the internet
to a plane op — the same way there is no path to a route nobody registered.
Identity rides the caller. zip.SocketPath is now the ONE socket path, used by
both halves. A background job with no request to forward states its tenant once
with cloud.For; cloud.As delegates a live request's principal, optionally
re-pointed at another tenant for the operator reading someone else's books. An
inbound request always wins, so a job can supply an identity and never launder
one. plane_test.go attacks the boundary over a real socket, and
TestNoPlaneInputCanNameAnOrg pins it structurally: no input may carry an Org.
The gate weighs the exact amount. metering.AuthInput gains a typed Amount beside
the int64 it had, and the comparison happens in the exact domain — a cents-
rounded charge let a sub-cent price round to zero and fall through to the bare
"any positive balance" gate. money.CentsUp rounds away from zero for the cap
surface, which still speaks cents, so a fractional spend is never weighed as
nothing.
The embedded tasks engine listens on a socket (tasks v1.52.4 EmbedConfig.Address),
which removes the free-port draw that seven of eight children used to lose. The
cluster-reachable gated listener stays TCP, because consumers in other pods dial
it and a unix socket does not leave the host.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The analytics row listed the READ endpoints and none of the six INGESTION doors
apps/analytics/event.go serves, so every beacon the products emit fell through to
commerce's bare "/v1" catch-all, which does not serve them. HTTP 405, silently,
for every event in the fleet.
The row was harmless while each app called its own routes(); killing the mega-build
made manifest.Apps THE router, so a missing prefix became an outage. It shows in the
warehouse: the last row landed 2026-07-29 04:15:29, eighteen seconds before the
ReplicaSet running v1.801.318 — the first image where this row is load-bearing.
Before that, 300-800 events per 15 minutes, continuously.
Bare /v1/analytics covers the batch door and the four read lenses; /v1/event covers
the Team SPA's /v1/event/collect suffix. /v1/tracker is deliberately NOT routed here:
apps/tracker owns that name for the issue tracker, so the capture alias is retired at
the caller instead. One name, one owner.
The router oracle caught the fix as a ratchet (FIXED, STILL LISTED) — ledger updated.
Nothing else moved: host stays 401 packages with zero apps/* imports.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two of team's twelve untyped routes are now typed ops — the ONE registry entry
the REST route, the OpenAPI operation's detail, the MCP tool, the CLI command
and the SDK method all read. team's published subset goes from 7 described
operations of 19 to 9, and POST /collaborator/rpc/{documentId} gains a request
schema and path parameter it never had: it was method + path and nothing else.
The wire is unchanged, and that is tested rather than asserted.
typed_wire_test.go pins every arm of both routes — `{"content":{}}` for a
getContent with no snapshot, a bare `{}` for updateContent,
`{"error":"unknown method x"}` for an unknown verb, `{"result":true}` plus the
expiring HttpOnly Set-Cookie for the DELETE, and the 400/401/404/503 gates —
and the SAME test file passes against the pre-conversion source. That is the
proof: the assertions were measured on the untyped handlers first.
collabResult.Content is a POINTER to its map on purpose. The three verbs answer
three different bodies on one 200 and the difference is load-bearing to the
client: createContent and getContent always carry `content`, possibly EMPTY —
the honest answer for a getContent with no source, which is a first-class case
since `source` is optional — while updateContent carries nothing and must stay
`{}`. A plain map with omitempty renders both as `{}`.
TEN routes stay untyped, each because typing it would move the wire, each named
at its registration with the reason, and the list is CLOSED by
TestEveryRouteIsTypedOrNamed (a new team route is typed by default, or it takes
a deliberate edit with a written reason):
GET /collaborator a WebSocket upgrade, not a value
GET /v1/team/transactor/{token} a WebSocket upgrade, not a value
POST /v1/team/account a JSON-RPC envelope: the verb is a
body field, `result` is a
different shape per verb, a
refusal is HTTP 200 carrying
{error: Status} INCLUDING for an
unparseable body, and the
entitlement arm answers 402
PUT /v1/team/account/cookie a body this route cannot parse is
IGNORED (the token falls back to
the bearer and the request
SUCCEEDS) where a typed In
answers 400
GET /v1/team/account/auth/{provider} a browser redirect: 302 +
GET .../auth/{provider}/callback Location + Set-Cookie, no body
GET /v1/team/billing/ui{,/*} the embedded wallet page's BYTES
under a per-asset Content-Type
POST /v1/team/files/{workspace} a MULTIPART form whose part
filename IS the blob id
GET /v1/team/files/{workspace}/{filename} the blob's raw BYTES under a
byte-derived Content-Type
LATENT DEFECT, surfaced by typing and fixed here: the collaborator plane is
app-level (/collaborator, because the front derives it from COLLABORATOR_URL),
and team's cloud.Bridge was scoped to the /v1/team group — so it never covered
that plane. A typed op receives only a context and reaches its caller ONLY
through the request Bridge parks, so the first typed op there would have 401'd
every call under a bare Mount (the app's own tests, and any embedder that mounts
without Serve). Bridge is now installed on the plane, AFTER the WebSocket
registration so the upgrade path is untouched, and
TestCollabRPCBridgedUnderBareMount is the regression bar.
Also surfaced: collabService had no `degraded` field. Mount's guard wrapper
carried the fail-closed 503 for both its routes, and a typed op cannot be
wrapped by a zip.Handler — so the field now exists and the op asks for itself,
exactly like billingService and filesService already do.
Identity: never an In field. tokenOf() joins sessionOf/admin/noStore/cookie in
typed.go — ONE file for the whole subsystem, which is why the cloud.Request pin
has one team entry instead of one per plane. The collaborator plane needs the
TOKEN and not just the tenant because it gates on the workspace claim too, and
the cookie writer is the other end of that same identity: the account-token
cookie is set on the RESPONSE, which only the request reaches. All of them fail
closed off the HTTP path. The pin's justification is updated to describe the
code that now exists.
Baseline: apps/team green before and after; the repo-wide suite fails the SAME
13 tests in the SAME 5 untouched packages (functions, platform, provisioning,
storage, kmsreseal) before and after, verified on a clean origin/main worktree.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Eleven /v1/books routes were untyped: a route and nothing else — no schema, no
prose, no MCP tool, no CLI command, no SDK method. Five of them are now typed
ops, and the document says so: 351 described operations, up from 346, with the
route table itself unmoved (1421 operations, 1005 paths, before and after).
Typing is a DESCRIPTION task; nothing here changes what a caller sends or gets.
POST /v1/books/ask AskRequest -> AskResponse
POST /v1/books/bank/sync syncIn -> BankTally
POST /v1/books/scan/book BookRequest -> BookResponse
POST /v1/books/vendors VendorRow -> VendorRow
POST /v1/books/rules Rule -> Rule
THE REASON THEY WERE STUCK WAS WRONG, and it is worth naming because it will
recur. Each carried the note "a typed POST documents its input as a body, which
would move the ?sandbox selector". That conflates the DOCUMENT with the BINDER.
zip binds a typed op's input from three sources in increasing authority — body,
then query, then path (typed.go:241) — for every method, so the selector was
never at risk on the wire. Only its DECLARED home was: zip's OpenAPI projection
emits query parameters just where there is no requestBody, so a Sandbox field on
a POST's In would publish a URL value as a body field. So the ops read it off
the request instead, through cloud.Request — the seam cloud/typed.go exists to
provide, and the one apps/agents/targets.go already uses for the facts a typed
signature drops. One helper, `query(ctx, name)`, in books/typed.go.
Guarded, not asserted: TestTheLedgerSelectorStaysOnTheURLForBodyWrites posts a
body that ASKS for the sandbox and proves it selects nothing, then proves the URL
still does. Naming `sandbox` on one of these Ins turns it red — which is exactly
when the wire would have moved.
SIX STAY UNTYPED, each measured rather than assumed.
metrics MetricsResponse EMBEDS Metrics. encoding/json flattens
an embedded struct; zip's schema walk nests it. Typing
it would publish a response no answer of the route
matches. TestMetricsCannotBeTypedYet measures both
shapes and goes RED the day zip learns to flatten —
read that failure as the go-ahead.
scan, inbox, bank/import RAW document bytes (PDF, image, OFX/QFX/CSV) as the
body. zip's decoder unmarshals a body as JSON, so
typing these would answer 400 to every upload.
bank/link-token, both answer 501 unconditionally. A typed op must state
bank/exchange what it answers on SUCCESS, and neither ever succeeds.
That last pair is a LATENT DEFECT typing surfaced, and it is not small: the
connectors behind those two routes are fully written — plaidConn.LinkToken mints
the Link session token, plaidConn.Exchange trades Link's public_token for the
durable access_token and seals it into KMS, tellerConn.exchange/linkConfig are
the Teller half — and NOTHING on the HTTP path calls any of them. Only their
tests do. The bank link flow is implemented end to end and unreachable: no org
can connect a bank through the API. teller.go even claimed "the route handler
(bankExchangeHandler) calls this"; it does not, and that comment is now true.
Wiring the handlers is a wire change (a route that has only ever answered 501
would start answering 200 with a body nothing has specified), so it is a
deliberate follow-up, not a side effect of describing the surface.
Eight schemas reach the published document for the first time — AskRequest,
AskResponse, BankTally, BookRequest, BookResponse, Figure, Leg, Voucher — and
the weave accepted all eight, so none collides with another app's meaning of a
name. Their fields carry prose now too, because a schema whose fields say
nothing is half a description.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
27 of this plane's 33 routes are typed ops (01e21366). The remaining six were
left with a written reason and nothing else, so the document said method, path
and path params about them and stopped. That is not the same fact for all six:
three of them still take ORDINARY JSON, and a route that cannot be a typed op
is not a route that must be undocumented.
All six refusals re-verified against zip v1.18.3's own source, not against the
comment claiming them:
PUT /workers/scripts/:script typed.go bindURL binds path AFTER body and
matches on the json tag, so path `script`
(the NAME) overwrites body `script` (the
module SOURCE). There is no per-field
opt-out; renaming either side moves the wire.
POST /pages/.../deployments typed.go op.invoke unmarshals whenever the
body is non-empty and 400s on failure. This
route IGNORES an unparseable body and falls
back to the production branch.
POST /d1/databases/:database/query the body is forwarded to D1 verbatim; a
typed In re-encodes from its own fields and
drops the rest, starting with params.
POST /ai/run/* wildcard model, model-defined body, and a
response that is frequently image or audio
bytes under Cloudflare's content type.
GET/PUT /kv/.../values/:key a KV value is opaque bytes under the
caller's own content type, both ways.
The first three are JSON on the wire and now declare their body through
openapi.Register — the seam that exists for exactly this case, adjacent to the
route table, reflecting the schema off the very struct the handler binds.
Pure description: no route, status, field, header or byte moves. The last three
have nothing to declare because they are not JSON.
PagesDeploy {branch} the handler's own struct, hoisted
out of the function so one value
is bound and published
WorkerScriptPut {script, mainModule, ...} already the struct the handler
binds; unchanged
D1Query {sql, params} the one declaration the handler
does not bind, and it says so on
itself: verbatim forwarding means
no struct it binds could state
the shape. OpenAPI objects are
open, so a field D1 takes that is
not named still reaches D1.
Response side is cfResult for all three, which register.go renders honestly
UNCONSTRAINED rather than as a shape this plane does not model.
LATENT DEFECT, found by declaring WorkerScriptPut and fixed here:
openapi/register.go published every json.RawMessage field as {"type":"string"}.
The []byte rule ("marshals as base64") fired before the custom-marshaler rule,
which lived inside the Struct case only — but the rule is about the MARSHALER,
not about being a struct, and json.RawMessage is a []byte that emits raw JSON.
So `bindings` would have told every generated SDK to send a base64 string for
the one thing it can never be. The check is hoisted above the kind switch; a
plain []byte still reflects as a base64 string, because a plain []byte has no
marshaler. Blast radius verified, not assumed: apps/platform is the only other
caller of openapi.Register, and regenerating its subset produces a byte-identical
file.
Also removes WorkerRouteCreate, dead since routeCreateIn replaced it in the
typed pass — an exported request struct nothing sends, describing a body
nothing reads.
TestUntypedJSONRoutesDeclareTheirBody reads the declaration back out of the
document the way a consumer does — resolving the $ref, through JSON — so the
three cannot quietly lose their shape again. openapi.yaml is +60 lines and
-0: three schemas and three requestBody/responses blocks, nothing removed.
Baseline: apps/cloudflare green before and after; apps/platform's
TestRunnerBuild_IAMReleaseRejected fails identically on origin/main
(want 403, got 502) and is untouched by this.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
openapi/weave_test.go looked for "an app serves a path the fleet routes
nowhere" and could not find one. It skipped any path already present in
openapi.yaml — the artifact it exists to protect — so the golden exempted
every defect it already contained, permanently; whatever survived that was
reported with t.Logf, which never fails a build. Both sides of the comparison
were DERIVED, and two derived artifacts agree with each other while both are
wrong. That is how plugin/ingress lost eight paths from every published SDK.
`go test ./openapi` printed zero UNROUTED lines and passed. The fleet routes
58 published paths somewhere other than the app that serves them.
So the question moves to the only thing that can answer it. manifest/
router_test.go builds the host's router from manifest.Apps — hand-authored
source — through the same zip.Load cmd/cloud calls, mounts every app on a
transport that answers with its own name, and asks it where each published
path goes. No routing is reimplemented: Load, Mount, the patterns and the
match are the fleet's own; only the wire is replaced.
The 58 are recorded, one line each, with the app that receives them instead.
They are a ledger of defects, not exemptions: a 59th fails the gate, and so
does an entry that stops being true, so the list can only shrink and a fix
nobody records is a fix nobody can see. The failure prints the current list
ready to paste.
The weave keeps composition and loses its routing opinion — the silent
delete it made was already dead (every subset path is in the golden), so the
document is byte-identical: 1005 paths, 505 schemas, 142 tags.
Proof: making dns claim /v1/dnsX turns the new gate red and leaves
TestFleetIsTheWeaveOfItsApps green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
DELETE /v1/keys and its deprecated alias DELETE /v1/iam/keys were the last two
addressable routes in this package that were a route and nothing else: method,
path and a tag in the published document, and past that no request schema, no
query parameter, no prose, no MCP tool, no CLI command, no SDK method. Both are
now one typed op — the single registry entry all five projections read. Nine of
this package's eighteen routes were typed; eleven are now, which is all of them
that have a shape to describe.
The reason they were left behind was real, and it is answered rather than waived.
A DELETE addresses what it deletes with its URL and carries no body (zip's
hasBody), while this route resolves the key class from `?type=` and FALLS BACK TO
THE JSON BODY. Binding the input and stopping there would have made a
body-selected revoke resolve to the empty string, default to secret, and destroy
the caller's session-equivalent credential in place of the publishable one they
named — a silent wire break on a credential-revocation route.
So the input declares the half the method has (`?type=`, which is what the
document now describes and a generated client fills in) and revokeClass reads the
other half off the request, where it always lived. The order is unchanged: query
first, body only when the query is absent. TestKeys_RevokeReadsTheClassFrom-
TheBodyWhenTheQueryOmitsIt pins both halves and the precedence between them;
deleting the fallback turns it red with the exact failure it exists to prevent
("the class in the body must reach IAM, got [secret]").
Everything else on the wire is byte-identical: the same 200, the same {ok,type}
body (a struct in the field order the map already serialised), the same 400 on an
unknown class, 403/501/502, and the same gates — `limit(csrf(…))` became the
`write` group and `deprecated(limit(csrf(…)))` the `aliasWrite` group, which
compose in that same order (zip's Chain).
Two coverage gaps the conversion surfaced, both on the gate it moved:
- No test held the revoke's CSRF gate. It is a money write that destroys a
credential, and the gate is a property of the GROUP an op is registered on —
invisible at the handler, and therefore droppable without anything looking
wrong. TestCSRF_AmbientWriteWithoutTokenIsRefused now covers all four key
writes instead of one.
- TestIAMKeysBeatsWildcard proved GET and POST beat clients/iam's /v1/iam/*
wildcard but never DELETE — the method where losing the race is worst, since
the request reaches IAM's own Guard, 401s, and tells the caller their key
still works when nothing tried to revoke it.
SEVEN routes stay untyped, and they are the seven catch-alls: GET|POST
/v1/billing/* and the five verbs of /v1/commerce/*. The path is a wildcard
remainder, the body is forwarded verbatim to commerce and the answer is
commerce's own bytes and status. There is no In and no Out to name — they are
opaque by construction, not by omission, and what they may reach is bounded by an
allowlist rather than by a type (billing.go).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An EMPTY leaf on a group composes to the group's prefix plus "/", so
`zip.Get(b, "", …)` on `b := g.Group("/blueprint")` declared the plane's root
at /v1/guide/blueprint/. op.Path IS the identity every projection reads, so
that trailing slash reached all of them: the document keyed the resource on
/v1/guide/blueprint/, the operationId (and therefore the MCP tool an agent
picks and the method a generated SDK exposes) was get_v1_guide_blueprint_,
and every generated client called the slashed URL. Fifteen sibling guide
paths carry no slash; the tests, the FE and the untyped PUT beside it have
always used the slashless form.
Nothing was red, because the router is non-strict: both spellings answer,
before and after. That is also what makes the correction wire-preserving —
TestBlueprintPathIsSlashless pins BOTH spellings at 200 and asserts the op
registry publishes no trailing slash, and it fails on the old registration.
The root is declared on g with a /blueprint leaf now, the same shape overview
already used to avoid naming /v1/guide/ ("declaring it on g would name
/v1/guide/, which this API never served") — the file had reasoned past this
exact trap one group up and walked into it one group down, which is the tell
that it is mechanical. The untyped PUT moves with it, or the document splits
one resource across two keys. Only the sub-paths hang off the group now,
where the leaf is non-empty and the composition is exact.
Repo-wide there is no second instance; LLM.md carries it as failure mode 8
with the grep, since every remaining typing tranche can hit it.
No route converted here: guide's remaining 6 untyped routes each name a wire
fact the declaration still cannot carry (verified, not assumed) — a YAML-or-
JSON document body that a JSON In would 400 (PUT /curriculum, PUT
/blueprint), a merge-patch whose explicit null DELETES a key that a pointer
field cannot tell from absent (PATCH /blueprint/{collection}/{id}), and the
structured 409 {error, step, blockedBy} that zip's {status, code, error}
envelope cannot express (POST /steps/{id}/start|done, plus SSE on /do).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`vouched` is the flag that skips DNS-01 domain-ownership proof entirely: a
vouched org's custom domain is BOUND live immediately (BindHost), an
unvouched one is CLAIMED pending behind a TXT challenge (ClaimHost), and the
"that is a host we operate" refusal only applies to the unvouched. So the
platform-operator set is an authorization boundary, not a label.
It was built and read with two different values. operatorOrgsFromEnv folded
every CLOUD_PLATFORM_OPERATOR_ORGS entry through sanitizeOrg — lowercase,
non-alnum→'-', truncate-32, no hash suffix, no refusal — while setDomains
looked the caller up VERBATIM, off principal.Org, which trims and clones and
deliberately never folds (projects.go org(): verbatim is what keeps two
distinct IAM owners off one S3 prefix).
Configuring "Acme" therefore wrote the key "acme", and a DIFFERENT tenant —
whoever's real IAM owner is literally "acme" — was vouched as PLATFORM
OPERATOR: it could bind any hostname live with no proof of ownership, and
claim hosts we run. Meanwhile the genuine operator "Acme" silently lost its
own vouch. "team.a" → "team-a" is the same collision, as is any pair of
owners differing only past 32 characters.
A fold applied to one side of a comparison is not a normalization, it is a
collision — and here the collision IS a cross-tenant privilege grant. Fix is
to delete the fold, not to move it: sanitizeOrg is gone (its only callers
were these two lines), and both halves are now the same verbatim owner,
TrimSpace and nothing else. There is one spelling of an org in this package.
Not renamed to orgLabel/displayLabel: that would assert a display-only
contract the function never had, and bury the defect behind a truthful-looking
name. Not keyed through cloud.SanitizeOrg either — an injective slugger on ONE
side of a verbatim lookup is the same defect with a better hash.
Tests: TestOperatorOrgsFromEnv is INVERTED in this commit — it asserted
got["acme"] for the input "Acme" and so locked the bug in; it now asserts the
verbatim "Acme"/"team.a" are present and the folded "acme"/"team-a" are NOT.
TestOperatorVouchIsVerbatimEndToEnd is new and drives the real route: with
CLOUD_PLATFORM_OPERATOR_ORGS="Acme", tenant "acme" gets a PENDING claim with
DNS records and a 403 on a host we operate, while operator "Acme" stays live.
Both fail on the parent commit — the e2e one reporting the lookalike bound
live, verified, with no challenge.
Deployment: no live behavior change. CLOUD_PLATFORM_OPERATOR_ORGS is unset in
every deployment (98 CLOUD_* keys in the operator CR, not one of them this),
CLOUD_BRAND is unset so brand = "hanzo", and sanitizeOrg("hanzo") ==
TrimSpace("hanzo"). IAM_ORG is "hanzo" verbatim, so the real operator matches.
From here both vars must carry the EXACT IAM owner: case and dots are kept.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/company was already 20 of 22 typed (93513f3f). This re-verifies the two that
are left against zip v1.18.3's own source rather than the comment that claimed
them, and writes down the floor so the next agent is not dispatched to redo it.
Both refusals hold, and each names ONE missing zip capability:
POST /fundraise/deck takes the deck as the raw request BODY (any content type,
named by ?name=). hasBody("POST") is unconditional (openapi.go:262) and
op.invoke json.Unmarshals whatever it is handed BEFORE the handler runs
(typed.go:227), so a typed In answers a PDF with
ErrBadRequest("invalid json body") — typing does not mis-document this route,
it BREAKS it. Waits on a raw-body binding.
POST /payment reads no body and its success path is already op-shaped (200 +
formationView); only the DENIAL blocks. cloud.DenyResource
(resource_billing.go:217) renders the fleet-wide {"error":{"code","message"}}
(402 insufficient_balance / spend_cap_exceeded, 503 balance_unavailable) and
zip's HTTPError (ctx.go:184) renders a flat {"status","code","error"}. Bridge
carries a STATUS back out, never a body, so the shim does not reach it — typing
would reshape that error for every metered client.
The audit did surface a real defect, via the check ec519f8c added one commit
earlier: company ships SIX instances of the bodyless-POST gap (#7). documents,
esign, genesis, kyc, kyc/refresh and skip each take noInput and therefore publish
requestBody:{required:true} over an object with no properties, for a body they
never read. That is the largest single share in the fleet — and running that
check over the committed subsets puts the class at 27 across 10 packages, not the
ten its prose had tallied. Counted, not enumerated: the instances anybody lists
by hand are the ones they happened to look at.
Everything else is clean: no embedded struct in any In/Out (the #7 field-dropping
class), no schema-name collision across the 30 names company publishes into the
flat namespace, zipdoc_gen.go and plugin/company/openapi.json both regenerate
byte-identical, and the fleet golden carries exactly the 21 paths the subset does.
Measured the MCP plane live: 20/20 tools described, and the 8 with no input
properties are exactly the noInput ops. deck and payment project nothing at all —
the cost of the refusal, paid knowingly.
LLM.md's tranche-B row was stale enough to misdirect work (company 22, actually
2; git 24 not 28, books 11 not 25, o11y 11 not 23) — re-measured with the
document's own command.
One doc-truth fix: the surface table stated POST /v1/company returns 201 flatly.
It is conditional (201 create, 200 idempotent) and the spec publishes 200; the
handler's own comment already said so.
No route, status, body or field name moves: description only. apps/company tests
green, zipdoc and the subset regenerate byte-identical, weave gate green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/framework is 17 of 19 typed and the remaining two — POST /v1/framework/:doctype
and PUT /v1/framework/:doctype/:name — stay raw. The recorded reason named only
half of what they need, which is the half that converts nothing.
Their body IS the document's own field data: an open object the DocType defines at
run time. Typing them takes two things, not one.
1. zip must be able to DECLARE an open object. It cannot. schemaOf has no
reflect.Interface case, so map[string]any falls to the default and projects
additionalProperties: {"type":"object"} — every VALUE is a JSON object. This
is already SHIPPED and already FALSE on the four ops that return a document:
http_test.go reads back {"subject":"Ship framework","docstatus":0}, a string
and a number, past a schema that admits neither. openapi.yaml carries the
claim in 15 places fleet-wide. Unlike multi-status and the bodyless POST,
which under-describe a true wire, this describes a false one — an SDK
regenerated from the golden types a document Dict[str, Dict], a shape that
cannot hold one. hanzoai/openapi's authored master has it right
(framework_Document: additionalProperties: true), so the two documents
genspec joins disagree about the same value.
2. bindURL must be able to BIND the URL onto one. It cannot: it returns early
unless the In is a struct, so an open-object In carries no :doctype/:name
while a struct In carries no document.
So the two convert together or not at all, and typing them today would replace a
correct authored request shape with a reflected one naming the path segments and
nothing else — an SDK method that cannot send a document. A schema that lies is
worse than the route-only entry they carry.
The refusal now lives with both halves named, in the test that already checks it
(rawRoutes) rather than only in prose. LLM.md gains the false-schema failure mode
as #8 and framework as the second worked SPLIT tranche.
No route, type or artifact changes: the wire, zipdoc_gen.go, plugin/framework/
openapi.json and openapi.yaml are all untouched and regenerate byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/o11y finished its typed migration: 12 of 20 routes are ops, and the other
8 cannot be without moving the wire (VM's verbatim status+envelope, three
reverse proxies, two text/plain alert receipts, one wildcard). Re-verified each
against the source; nothing left to convert.
What the audit did surface is that being typed is necessary and not sufficient.
POST /v1/o11y/ingestion is a typed op with In/Out and lifted prose, and it is in
neither plugin/o11y/openapi.json nor the woven openapi.yaml — so the LLM-obs
write path has no SDK method, no MCP tool, no CLI command and no schema.
mountEventIngest registers it only behind a reachable Hanzo Datastore, and the
process that writes the document has none; the generator says so in its own log.
Recorded in LLM.md rather than fixed: zip.Post registers a fiber route and a
registry entry inseparably, so making the op visible necessarily stops the path
falling through to the order-70 wildcard. That is a behaviour decision, and
typing is a description task.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/integrations measures 45 untyped routes and serves 19. The other 26 hits are
hdr.Get("Retry-After") and the `// app.Post(…)` MOUNT HANDOFF blocks each adapter
file carries, so the tranche table's "integrations 47" would have sent the next
agent on a pass with nothing to convert.
All 19 are refusals, and checking each against zip v1.18.3 rather than against the
comment found the comment wrong. It grouped Teams and Telegram under "auth is a
signature over the RAW request bytes". Neither is: Teams verifies a Bot Framework
JWT header, Telegram a shared secret header. Their real blocker is a different
wire fact — both answer an EMPTY 200 to a body they cannot parse so the platform
does not retry-storm, and zip's invoke unmarshals BEFORE the handler
(typed.go:227), which turns that 200 into a 400. For telegram it also inverts the
auth order, leaking a parse result to a caller that today gets 401 first.
That prose is load-bearing: it is what the next agent reads to decide whether a
route converts. A refusal filed under the wrong reason is a refusal nobody can
re-check, so the taxonomy now cites the line numbers it rests on.
Three families, not two, and the HTML one was missing entirely:
- 8 legs answer 302; zip.WithStatus PANICS on a non-2xx (typed.go:104).
- 5 answer text/html and set __Host- cookies; a typed dispatch ends in
c.JSON(out) and an op holds no response to set a cookie on.
- 6 inbound webhooks, splitting on WHY: four are signed over the raw received
bytes (Slack/GitHub HMAC, Discord Ed25519), two are the 200-on-unparseable
pair above.
Also records six new instances of the bodyless-POST gap (#7) that this package
already ships — /connectors/{id}/refresh, /device/{flow}/poll, /pages/builds,
/{provider}/disconnect, /{provider}/verify each publish a required body whose only
properties ARE their path params, and /telegram/connect publishes one over noArgs,
an object with no properties at all. The class is ten now and grows with every
tranche, so LLM.md carries a check that reads the published subsets rather than
the source.
No route, status, body or field name moves: description only. Verified against the
baseline — apps/integrations tests green, zipdoc -check clean, the regenerated
subset byte-identical, weave gate green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three claims in "Generated and frozen artifacts" were false, and each one
pointed the next reader at work already done:
- "no gate in this repo uses -check yet" — `make test` has run
`zipdoc -check` per package, on every package carrying the directive,
for some time. Read as written it invites a SECOND gate.
- "mk/plugin.mk:45-46 still says they are not committed" — that comment
now says the opposite ("The files ARE committed today, deliberately"),
so the fix it asks for is a regression.
- "15 zipdoc_gen.go files" — 27, 1:1 with the directives.
Line-number citations (Makefile:214, Dockerfile:178) are dropped rather
than corrected: both had already drifted, and a file+target names the value
where a line number names a place that moves under it.
Measured on apps/git while auditing its untyped routes: 48 operations, 24
typed, 24 route-only — and all 24 are structurally untypable (HTML pages,
binary pack streams, an HMAC-over-raw-bytes webhook, a ZAP envelope whose
error body is non-2xx). Nothing to convert there; this was the one real
defect the audit surfaced.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every route apps/ingress serves is already a typed op (18 of them), and the
projection test pinned three of the four surfaces that one registration feeds:
the live router, the registry the CLI reads, and the prose in the OpenAPI
document. The fourth was unpinned, and it is the one that fails SILENTLY.
zip's tool list once read op.Summary — a field cloud sets nowhere, because the
handler's doc comment is the source — so all 164 MCP tools served an empty
description over a nameless schema while the spec looked perfect. It was fixed in
zip v1.17.6 by reading the same docFor extraction its siblings read, and an older
zip reverts it invisibly. TestSpecCarriesProse cannot see that: it reads a
DIFFERENT field of the same registry entry.
So the pin asserts what an agent actually receives: 18 tools, every one carrying
its handler's prose, and every op that takes input NAMING its fields (id from the
URL, the object's own fields from the body). The five no-input ops — status, tls
and the three lists — take nothing off the wire, so an empty schema is the truth
for them and the test says so out loud.
It is not vacuous: a phantom tool name and a phantom schema field each fail it,
checked before committing.
Test-only. No source, no route, no wire, no regenerated artifact changes. Verified
on this surface at the same time: 0 untyped routes left in apps/ingress, zipdoc
-check clean, plugin/ingress/openapi.json regenerates byte-identical, and the
fleet golden carries all 18 ingress operations described, over 10 schemas whose
every field is described.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/crm finished its typed migration in a15f5ca3: 19 of 20 routes are typed ops
and the public intake stays a raw handler, because its 20/min-per-IP limiter and
its 64 KiB pre-parse body cap are HTTP-plane facts an MCP or CLI projection would
not run — typing it would publish an unauthenticated, unmetered alias of a
deliberately metered endpoint, and move the size cap after the parse it exists to
prevent. Verified end to end: 19/19 ops carry a description and an input schema in
both the OpenAPI golden and the MCP tool list, and the intake appears in neither.
What that migration left unguarded is registration ORDER. The limiter is a second
app.Group("/v1/crm", …), so it is a prefix-scoped Use covering every /v1/crm route
registered after it and none registered before. crm.go says so in prose —
"REGISTRATION ORDER IS WIRE HERE" — and nothing enforced it. Measured, the current
order is exactly what the prose claims:
* POST /v1/crm/applications and the three staff /applications routes are metered
* every companies/contacts/opportunities/summary op is not
Both halves matter, and each fails in a different direction. The natural next step
of this very migration — typing the intake — moves that registration across the
line and silently un-meters a public form. The mirror slip throttles the CRM's own
CRUD to 20 requests a minute per IP, which no console could use. The type system
says nothing about either, so the test does; it goes red on the mutation that
moves the limiter above the CRUD ops.
Also gofmt: applications.go and applications_test.go were not gofmt-clean on main
(composite-literal key alignment). No wire, no golden, no zipdoc change — the
lifted prose regenerates byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cicd.yml called hanzoai/ci/.hanzo/workflows/build.yml@v1. At tag v1 that
directory is EMPTY; build.yml lives under .github/workflows/, which is also the
import its own header documents. So the forge could not construct a run:
PrepareRun: InsertRun: read hanzoai/ci@v1:.hanzo/workflows/build.yml:
object does not exist
InsertRun fails before any run row is written, so there is no failed run to
look at. Pushes and workflow_dispatch alike silently did nothing — dead CI that
is absent rather than red. hanzoai/cloud's last run of ANY kind was
2026-07-26T07:59:31Z; every commit since landed with no build at all, which is
why the live image sat at v1.801.318.
Points at the path that exists. Moving the file in hanzoai/ci instead would
mean moving the v1 tag, which we do not do.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cicd.yml called hanzoai/ci/.hanzo/workflows/build.yml@v1. At tag v1 that
directory is EMPTY; build.yml lives under .github/workflows/, which is also the
import its own header documents. So the forge could not construct a run:
PrepareRun: InsertRun: read hanzoai/ci@v1:.hanzo/workflows/build.yml:
object does not exist
InsertRun fails before any run row is written, so there is no failed run to
look at. Pushes and workflow_dispatch alike silently did nothing — dead CI that
is absent rather than red. hanzoai/cloud's last run of ANY kind was
2026-07-26T07:59:31Z; every commit since landed with no build at all, which is
why the live image sat at v1.801.318.
Points at the path that exists. Moving the file in hanzoai/ci instead would
mean moving the v1 tag, which we do not do.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`tags: ["v*"]` sat under `workflow_dispatch:` rather than `push:`, where it
means nothing, and a SECOND `workflow_dispatch:` key below then overwrote that
whole mapping — so the filter was dropped twice over. YAML takes the last
duplicate key and reports nothing, so the file parsed, the workflow ran on
main pushes, and the tag trigger was simply absent. Effective `on:` was
{push: {branches: [main]}, workflow_dispatch: None, pull_request: None}
Two edits landing on the same block at different times is all it takes, and
nothing in a normal parse tells you. Both comments are kept; the dispatch
trigger now carries the sync note as well as the on-demand-rebuild one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`tags: ["v*"]` sat under `workflow_dispatch:` rather than `push:`, where it
means nothing, and a SECOND `workflow_dispatch:` key below then overwrote that
whole mapping — so the filter was dropped twice over. YAML takes the last
duplicate key and reports nothing, so the file parsed, the workflow ran on
main pushes, and the tag trigger was simply absent. Effective `on:` was
{push: {branches: [main]}, workflow_dispatch: None, pull_request: None}
Two edits landing on the same block at different times is all it takes, and
nothing in a normal parse tells you. Both comments are kept; the dispatch
trigger now carries the sync note as well as the on-demand-rebuild one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every /v1/cloudflare route was a raw fiber handler, so the published document
carried method + path + path params and nothing else: no request schema, no
query parameters, no prose, no MCP tool, no CLI command, no SDK method. 27 are
now typed ops declared on the group, which is the ONE registry entry all five
projections read.
The wire is unchanged. Every handler keeps the exact gate it had (verified
handler-by-handler: 33/33), the same statuses and messages, the same upstream
Cloudflare paths and bodies, and the same acting-org stamp. The response is
still Cloudflare's own payload relayed verbatim — cfResult marshals the raw
upstream bytes, so field order, unmodeled fields and integers past float64
survive untouched (pinned in TestRelayIsVerbatim).
SIX routes stay untyped, each because typing it would move the wire, and each
named at its registration with the reason on the handler:
POST /ai/run/* the response is often not JSON at all
(image/audio bytes under CF's own
content type) and the body is the
model's, forwarded verbatim
GET/PUT /kv/.../values/:key a KV value is opaque bytes under the
caller's own content type
POST /d1/databases/:database/query the body is forwarded to D1 VERBATIM; a
typed In drops params and batch fields
PUT /workers/scripts/:script path param `script` (the NAME) collides
with body field `script` (the SOURCE),
and zip's URL binder gives the path the
last word
POST /pages/.../deployments an unparseable body is IGNORED here (the
deploy falls back to the production
branch); a typed In answers 400
TestEveryRouteIsTypedOrNamed closes that list: a new route here is typed by
default, or it takes a deliberate edit with a written reason.
Identity: cloud.Bridge() on the group, principal.OrgFrom(ctx) for the tenant —
never an In field, which is caller-supplied. The org-admin bit lives in a
header principal.OrgFrom does not carry, the acting-org stamp is a response
header, and ?account= must NOT become an In field (zip binds an In field from
the body too, and this route has never accepted an account there), so the
plane is pinned in allowedRequestUses with those three reasons. authWrite
fails closed off the HTTP path.
Known imprecision, reported not hidden: cfResult renders as
{"type":"object"} because zip's schemaOf has no vocabulary for "any JSON" —
json.RawMessage reflects as an array of integers, which is why cfResult is a
struct at all. For the list endpoints the document therefore says object where
Cloudflare answers an array. A zip patch teaching schemaOf that
json.RawMessage means `{}` upgrades all 27 with no change here.
Baseline check: apps/cloudflare green before and after; the repo-wide suite
fails the same 12 tests in the same 5 untouched packages (functions, platform,
provisioning, storage, kmsreseal) before and after.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The embedded server took NATS's own 1 MiB max_payload because nothing set it —
production advertised max_payload=1048576. That is the hard bound on everything
riding the bus, and the Kafka-wire adaptor rides it: insights-plugin's ingestion
loop produces downstream, a >1 MiB record failed with 'Broker: Message size too
large', the plugin treats that as an unhandled rejection and exits(1), and it
crash-looped — 127 restarts.
The failure is on the PRODUCER, which is why an earlier fix aimed at the
consumer could not clear it: universe's insights deployment already raises
STREAM_CONSUMER_MAX_PARTITION_FETCH_BYTES to 10 MiB, correctly, for the fetch
path. The bus itself was always the ceiling.
pubsub v1.0.0 -> v1.4.5 adds embed.Options.MaxPayload (default 8 MiB, tested
against what a client is actually advertised). CLOUD_PUBSUB_MAX_PAYLOAD
overrides it; a non-positive value is refused at Mount rather than silently
falling back.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The pricing surface had no typed op at all: 32 routes, no schema, no prose, no
MCP tool, no CLI command, no SDK method. 15 of them are now ops, so each is ONE
registry entry the document, the tool list, the CLI and the generated clients all
follow from. plugin/pricing/openapi.json: 32 operations, 15 described, up from 0.
WIRE, VERIFIED BYTE FOR BYTE. A probe drove every route on this surface under
four identity shapes (anonymous, member, SuperAdmin, and the forged X-Org-Id with
no principal the enablement attack tests pin), before and after, with an empty
overlay and a populated one, and diffed status + Content-Type + body. Every 2xx
body is byte-identical — same keys, same order, same values, same sha. That is
not luck: each Out struct replaces a map[string]any, Go emits a map's keys
sorted, so every one of them declares its fields in that same order.
The whole delta is on error paths, and it is one fact:
404 {"error":"Model not found: zen4"} -> {"status":404,"error":"Model not found: zen4"}
401 {"error":"admin required"} -> {"status":401,"error":"admin required"}
A typed op states an error as an error, and zip's HTTPError carries the status in
the body. Same HTTP status, same message, one added field — and it is the shape
this surface already emitted for its 403s (zip.ErrForbidden was already in
admin.go and enablement.go), so the change is toward one error shape, not away
from it. The unknown-model 404 additionally moves Content-Type from
"application/json" to "application/json; charset=utf-8", which every other JSON
answer on this surface, including that same 403, already sent. POST
/v1/pricing/sync's 500 loses its second field ("message"); zip's error carries
one message, so the field both shapes have keeps exactly the text it had and the
cause is logged instead of returned.
RAW ROUTES LEFT: 17, each for a reason that is a property of the wire.
- The fourteen /v1/pricing/{compute,cloud/*,subscriptions,blockchain,iam,base,
paas,policy,tools,gpu} routes and /v1/pricing-policy are a VERBATIM PROXY of
the @hanzo/pricing goja bundle: the bundle picks the status (200, or 503 when
a section is absent — six of them have that branch) and its bytes are written
unmodified. A typed op answers the one status it declared, over a Go
re-marshal. Two wire changes, so they stay raw. The gated routes typed above
do NOT have this property: they already decoded and re-marshalled through Go,
so typing them is pure description.
- PATCH /v1/admin/catalog/models/* addresses model ids containing '/', so it
routes through a greedy wildcard. fiber names that parameter `*1` and the
document renders it {wildcard1}; binding it needs an input field tagged
json:"*1", which every projection would then publish. A schema nobody can read
is worse than none.
- PATCH /v1/admin/catalog/providers/:name carries `overrides`, a raw JSON merge
patch (RFC 7386). zip reflects json.RawMessage as an ARRAY OF INTEGERS — it is
[]byte — so typing it publishes a false schema; and retyping the field to
map[string]any moves {"overrides":null} from "clear the override" to "leave it
alone", which is a wire change.
LATENT DEFECT, and the reason for the second half of this commit: pricing serves
FIVE prefixes and declared ONE. Its plugin spec named no Prefixes, so
MountPrefixes fell back to the /v1/<name> convention and cloud.Declare built its
tracing/price table with /v1/pricing alone — /v1/pricing-policy, /v1/enablement,
/v1/admin/catalog and /v1/admin/enablement resolved to another subsystem's prefix
or to none, mislabelling their spans and leaving their price unanswered. It also
meant the subsystem could not install middleware on four fifths of itself:
scope.Use installs once per DECLARED prefix. manifest/apps.go, which the light
host routes by, had the correct five all along — the app's own composition root
disagreed with it. pricing.Prefixes now states them once, in the app, and
plugin/pricing/main.go reads it.
Bridge: Serve installs one app-wide, so production always had it, but this
subsystem's own tests mount it on a bare zip app and would have exercised a
different identity path than production. It installs its own now (nesting is
harmless — the inner one is what the handler sees), which is what makes the
existing admin/enablement HTTP tests, including the two cross-tenant attack
tests, prove the typed path and not a weaker one.
cloud.Request grows a fifth call site, pinned with its justification:
callerIsAdmin. Every read op here branches on admin-ness (an admin sees disabled
models, flagged, where a customer sees them hidden) and that bit lives in a
header principal.OrgFrom does not carry. The tenant itself is read with
principal.OrgFrom, never through the request and never from an In field.
Found while typing, NOT fixed here because it is another app's file:
apps/agents/targets.go's patchTargetIn embeds the unexported patchTargetReq, and
zip's structSchema skips unexported fields — including an embedded unexported
struct. encoding/json still promotes them, so PATCH /v1/agents/targets/:id works;
but its published schema has exactly one property, `id`, and openapi.yaml has
said so since it landed. Every SDK, the MCP tool and the CLI command for that op
offer only `id` — label, kind, status, capacity, host, spec and metrics are
invisible. The ops here flatten their fields rather than embed, so none of them
reproduces it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/guide had a route table and nothing else: no schema, no prose, no MCP tool,
no CLI command, no SDK method. Thirteen of its nineteen routes are typed ops now
and carry all five. The other six each name a wire fact the declaration cannot
yet make, and say so at the registration rather than being quietly skipped.
Typed: GET /v1/guide, /analytics, /profile, /strategies, /suggest, /curriculum,
/actions, /blueprint, /blueprint/versions; POST /chat, /steps/:id/skip,
/steps/:id/reset; DELETE /curriculum.
Left untyped, with the reason:
- PUT /curriculum and PUT /blueprint take a YAML-**or**-JSON document
(sigs.k8s.io/yaml). A typed In is decoded as JSON before the handler sees it,
so typing them would answer 400 to every YAML body they accept today.
- PATCH /blueprint/:collection/:id takes an opaque JSON merge-patch whose keys
are the patched item's own — not a declarable In.
- POST /steps/:id/start|done answer a blocked step with a STRUCTURED 409
({error, step, blockedBy}) written in-band. A typed op's only non-2xx is the
error it returns, whose envelope is a different shape. They convert with zip
multi-status (#78).
- POST /steps/:id/do also STREAMS SSE, and an op answers exactly one JSON value.
The gated/ungated split is the discriminator worth copying: skip and reset pass
gate=false, so the 409 branch is unreachable for them and their whole answer set
is expressible. One shared body (applyStep) serves both halves; the gate is a
parameter and the blocked case is a VALUE the untyped pair renders.
Wire preserved exactly, and checked rather than assumed:
- every response is the same JSON (maps became structs with the same keys);
- GET /strategies still binds category/stage/workload from the query, and now
DECLARES them;
- DELETE /curriculum takes no body, before and after;
- the SuperAdmin 403 on the blueprint plane is now one shared errNotSuperAdmin,
so the untyped wrapper and the typed ops cannot drift into two refusals;
- the URL stays the addressing authority on skip/reset — a body naming another
step cannot redirect the write (TestTypedStepOpsFailClosed pins it).
Two things typing surfaced that nothing else would have:
1. openapi.Weave REFUSED the whole fleet document: guide's Step and marketing's
Step are one schema name with two shapes, and a generated SDK would bind
whichever it read last. guide's type is JourneyStep now — marketing's is
already published and guide's was not, so guide yields. No wire change: the
JSON keys live on the fields. The comment on the type says why, so nobody
"simplifies" it back into the collision. LLM.md's failure mode 5 gains this
second instance and a shape-aware scan, since a name-only grep passes when two
apps legitimately agree.
2. zip cannot declare a bodyless POST (hasBody is unconditional), so skip and
reset publish a requestBody they never read. The wire is unharmed — bindURL
binds the path LAST, so the URL still names the target — but the document
asserts something false. apps/admin already ships two of these. Logged as
failure mode 7; same shape of gap as multi-status.
Three operationIds move (get_v1_guide_blueprint_, post_v1_guide_steps_id_skip,
..._reset). That is failure mode 6 and the house rule is explicit: TAKE the
rename, never pin it back with WithOperationID, which would make one app's ids a
special case. It is not the wire — no status, body or field name moves. The
registration says so where the next reader will look.
apps/guide gets its //go:generate zipdoc directive (it had none, so its prose
could never have reached the spec) and cloud.Bridge on its own subtree, so the
validated org reaches an op that receives only a context — never an In field,
which is caller-supplied.
TestRequestEscapeHatchIsPinned fired on this change, which is the gate working:
superAdminOK and ledgerOf are new cloud.Request sites, justified in the allowlist
rather than waved through — admin-ness lives in X-User-IsAdmin and the payer is
the SELECTED billing org, neither of which principal.OrgFrom carries, and both
fail closed off the HTTP path.
Green: apps/guide, openapi (the weave), the root package, manifest, cmd/cloud,
apps/marketing; go build ./...; zipdoc -check; openapi.yaml + plugin/guide
regenerated from source and clean under the drift gate.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Typing apps/crm surfaced a trap the playbook did not name, and it is already live
in the committed golden three times over: zip's structSchema skips every field
reflect says is unexported, which an EMBEDDED unexported type is — while
encoding/json promotes those same fields onto the wire. The route works, the test
passes, and the published schema is missing most of the payload. That is the one
direction nothing catches.
patchTargetIn (apps/agents/targets.go) -> publishes {id} alone, so
PATCH /v1/agents/targets/{id} documents none of label, kind,
status, capacity, host, spec, metrics — not in openapi.yaml,
not in a generated SDK, not in the MCP tool's inputSchema.
botView (apps/visor/bots.go) -> {agent,binding}, dropping the 14
machine fields it embeds.
clusterDetailView (apps/visor/k8s.go) -> {nodes} alone.
Exporting the embedded type is not the fix — zip then publishes a NESTED object
the wire does not have. Spell the fields at the top level, or teach structSchema
to flatten embedded structs the way the decoder does, which fixes the class.
Also records how zip picks the OpenAPI `summary` (first ". ", else the first
LINE), because a first sentence wrapped across two lines gets cut mid-sentence.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/crm was 20 raw fiber handlers: a route and nothing else, so /v1/crm/* had
no schema, no prose, no MCP tool, no CLI command and no SDK method. 19 of the 20
are now typed ops, which is ONE registry entry with N projections — the REST
route, the OpenAPI operation, the /mcp tool, the CLI command and every generated
client all follow from the same declaration.
The wire is unchanged, and that is the point of the exercise:
* 201 on the three creates, declared with zip.WithStatus(201) rather than set
per request, so the document keys its response on the code the route sends.
* 204 with no body on the three deletes — a typed op says that by returning a
nil Out.
* {"data": [...]} list envelopes, the {companies,contacts,opportunities}
summary, and every field name kept verbatim.
* ?limit=, ?companyId= and ?stage= now bind off the In instead of c.Query, and
still trim, case-fold and bound exactly as before.
* ids still bind from the PATH, which is the addressing authority — a body
cannot smuggle a different target past the org gate.
cloud.Bridge() is installed on the /v1/crm group, ahead of its leaves: a typed
op is handed only a context, so the VALIDATED org has to be parked there. It is
never an In field — an In field is caller-supplied, so a tenant key read from one
is a cross-tenant read the caller asserted for itself. Every op resolves its org
through principal.OrgFrom and fails closed off the HTTP path, so an MCP tools/call
or a CLI invocation with no principal gets the same 403 an anonymous REST call
gets. TestRed_NoPrincipalForgedOrgRefused still passes unchanged.
POST /v1/crm/applications stays a raw handler, deliberately: the public intake's
protections are an IP rate limit (HTTP middleware, which the MCP and CLI
projections do not run) and a 64 KiB body cap (which can only refuse BEFORE the
decode a typed op is handed). Typing it would publish an unmetered alias of a
deliberately metered public endpoint. It converts when zip can carry both.
One doc-truth trap the conversion surfaced, avoided here and live elsewhere:
zip's schema builder skips unexported struct fields, so an In that EMBEDS an
unexported request struct publishes a schema carrying only its own fields. Every
In here spells its fields at the top level, so all 15 published schemas are
complete.
Verified: apps/crm tests green (incl. a new TestTypedWire pinning 204-no-body and
the query bindings), the intake rate limiter's reach probed byte-identical on
HEAD and on this tree, plugin/crm/openapi.json + openapi.yaml regenerated from
source, and go test ./openapi green — 1006 paths, unchanged; 15 schemas and 19
descriptions added where there were none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/o11y/{logs,metrics,status}, the eight /v1/o11y/annotation-queues* routes and
POST /v1/o11y/ingestion are typed ops now, declared on the /v1/o11y group. Each
is ONE registry entry, so the OpenAPI schema, the MCP tool, the CLI command and
the SDK method all follow from it where there was a URL and nothing else. The
document gains 11 described operations and, for the first time, the request and
query SHAPE of this surface: `?product`, `?sinceNs`, `?window`, `?limit`,
`?range`, `?stepSec`, `?status`, `?page`, `?limit` are declared parameters with
types and examples, and the annotation-queue bodies have schemas.
WIRE PRESERVED. 1005 paths before, 1005 after — none added, none removed. The
clamps now take the decoded value instead of parsing the raw string, which lands
on the same branch (`?limit=abc` binds as 0, which IS "no limit given"). The two
creates keep their 201 via zip.WithStatus; the DELETE takes its id from the URL
and no body, as it always did. The app's own tests — 403 forged, 400 malformed
product, honest-empty, the full queue lifecycle with 201/200/404/409/400, org and
project isolation — pass unchanged.
Three latent defects surfaced, all fixed here:
1. NO cloud.Bridge IN THE o11y PROCESS. o11y runs as its own binary
(plugin/o11y/main.go builds a bare zip.App), and a context value does not
cross the host→plugin socket. cloud.Serve's app-wide Bridge parks the org in
the HOST, so every typed op in the child would have answered 403 to a caller
the host had already validated — a total outage of the surface, not a
degradation. MountO11y installs its own on the /v1/o11y group, first.
Pinned by TestTypedOpsResolveTheirOrgThroughTheBridge, which fails without it.
2. ONE NAME, TWO SHAPES: `usagePoint`. o11y's per-bucket {t,calls,tokens,
costCents} collided with admin's daily {date,requests,spendCents,tokens} in
the fleet document's single schema namespace — a generated SDK would bind
whichever it read last. o11y's renames to `usageBucket` (its name had never
been published; admin's has). Caught by the weave gate the moment the type
entered the document.
3. AN EMBEDDED STRUCT PUBLISHES A SCHEMA WITH HOLES. encoding/json flattens an
embedded struct; zip's schema builder skips it, so annQueueDetailView would
have published 3 of its 9 fields and the PATCH bodies would have published
none of theirs. The typed shapes are written FLAT — same bytes, and now the
schema says so. (apps/agents/targets.go has the same pattern: patchTargetIn
publishes only `id` today, so the whole patch body is missing from the
published spec and from every SDK. Not touched here — it is that app's diff.)
Eight routes stay untyped BECAUSE typing them would move the wire, and each is
named with its reason in apps/o11y/LLM.md: the two VM proxies return
VictoriaMetrics' own status and envelope verbatim; the two builder queries and
the sessions list are reverse proxies with no Go type for "whatever the runtime
answered"; the two alert routes are text/plain and the receiver deliberately
ACCEPTS an unparseable body (a body that will not parse still proves delivery,
and a 400 would make Alertmanager retry forever); /v1/sentry/* is a wildcard.
Also noted, not fixed: POST /v1/o11y/ingestion is registered only when a
datastore DSN is present, so it is absent from the published document — the
route exists in production and no generated client can reach it.
The three cloud.Request uses this needs (platform-sudo, validated-ness, project
scope — none of which principal.OrgFrom carries) are concentrated in one seam
file, apps/o11y/typed.go, added to allowedRequestUses with its justification.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
There were two `hanzo code`s. The real one is hanzoai/cli: it runs the agent
HEADLESS, parses its JSONL event stream, persists a resume handle and a
transcript pointer, relaunches with `--resume`, and answers pause/resume/stop/
message from the cloud control plane. This one only exec'd a binary and handed
over the terminal, so by construction it could never read a single event.
Its four unshared capabilities — the zen⇒carrier model map, a config home
separate from the user's `~/.claude`, that home's first-run seeding, and the
identity a carried model needs — now live in hanzoai/cli (v1.9.8), verified
against a live model. Nothing is lost, so this is a deletion and not a
deprecation: no shim, no alias.
Also gone, because they configured only this command: bare `hanzo` dropping
into an agent, the `code_tool` and `code_model` config keys, and their two
Config fields. `code_model` was already dead — it was settable and never once
read, even here.
The REST of this package stays. `cli/gpu.go` is a 2,364-line GPU link daemon
(heartbeat/claim/execute, nvidia-smi / rocm-smi / kfd-topology parsing, ComfyUI
supervision, systemd install) that has no Rust counterpart — `hanzo node join`
is a 90-line one-shot registration and says so itself. Until that is ported,
this code is the only specification of the work it does, so deleting it would
destroy the spec along with the duplicate.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A tenant's team page called /v1/iam/get-users. get-users is org-scoped but not
org-KEYED, so the edge's pin never reached it and the request was forwarded BARE
under cloud's ONE service credential. A bare read is not unscoped — IAM scopes it
to whatever org that credential resolves to — so every tenant was served the
credential's org: measured live as 262 rows, all owner=hanzo.
The same three-segment allowlist left add-user/update-user/delete-user without a
body check, so a tenant org-admin could name a foreign owner in a write body.
Both are now one rule: every gated segment that is not org METADATA (guarded by
NAME, its object owned by "admin") carries the caller's own org explicitly — in
the query, in the id, and in the write body. A super admin is still unpinned.
This is also the gate on IAM v1.33.31. Once Scope honours-or-refuses, a bare read
still answers from the credential's org, and the naive fix — giving the edge a
credential that can cross orgs — is strictly WORSE: IAM's listHandler calls
Scope(ctx, ""), which returns "" for a super principal, and then applies no
Owner filter at all. Unpinned + super = every tenant in one response. The pin is
what makes either credential safe, so it lands FIRST.
Found while collapsing the two IAM lineages: the edge is the boundary between
them, and it was trusting the far side to decide scope.
Tests: the leak and the write hole both reproduce red on the parent commit.
TestEdgePinRidesInTheId pins a break the first fix introduced — IAM's ReadTarget
only falls back to ?id= while ?owner= is empty, so a blanket owner pin would have
turned every id-addressed read into "id (owner/name) or name is required".
TestEdgeRefusesBareId records a real semantic divergence across the hop (cloud
reads a bare id as an OWNER, IAM as a NAME) and keeps the stricter side.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/account had eighteen routes and no typed op, so every one of them was a
route and nothing else: no schema, no prose, no MCP tool, no CLI command, no SDK
method. Nine are now typed ops — ONE registry entry each, which the REST route,
the OpenAPI operation, the MCP tool, the CLI command and every generated SDK
method are projections of:
GET /v1/csrf POST /v1/keys
GET /v1/keys POST /v1/iam/keys
GET /v1/iam/keys POST /v1/iam/onboard
GET /v1/embed-status POST /v1/commerce/topup/wallet
GET /v1/commerce/topup/rails
Measured: the account subset went from 0 described operations and 0 schemas to
9 and 12; the fleet golden gained 55 descriptions with 1006 paths / 1422
operations UNCHANGED and not one deleted line — the wire is the same, only the
document knows more about it.
LATENT DEFECT, fixed here: the subsystem installed no cloud.Bridge(). Serve
installs one app-wide so production was never broken, but the subsystem was not
self-sufficient — and its own tests mount it on a bare zip.New, where the first
typed op would have failed closed. Proven by removing the line: 20 tests go red.
It goes through Router.Use, which fans it over the prefixes the composition root
declared for account; account owns six top-level nouns and so has no single
group to hang it on.
WIRE PRESERVED, and nine routes left untyped BECAUSE of it:
- DELETE /v1/keys and DELETE /v1/iam/keys select the key class from `?type=`
and FALL BACK TO THE JSON BODY. A typed DELETE carries no body (zip's
hasBody), so typing them would revoke a caller's SECRET key when they named
the publishable one in the body. That is a wire change on a
credential-revocation route; they keep their raw handlers.
- the seven /v1/billing/* and /v1/commerce/* bridge routes are catch-alls: the
path is a wildcard remainder, the body is forwarded verbatim to another
service, and the answer is that service's bytes and status. There is no In
and no Out to name — opaque by construction, not by omission.
The gates that had to be satisfied, and did their job: the weave REFUSED the
first attempt because `keyList` already means git's SSH deploy keys — one name,
two shapes, which would have bound every generated SDK to whichever it read
last. Account's is now `apiKeyList`/`apiKey`, named for what it is.
Middleware is now zip.Middleware in one form each (requireCSRF, rateLimit,
deprecatedFor), composed through With so a typed op is gated exactly as the
untyped route beside it — a decorator that dropped the gate there would register
the op ungated. rateLimit also loses a parameter it never read.
cloud.Request is pinned; apps/account/account.go is added to allowedRequestUses
with its reason: account IS the caller's own account, and resolving them needs
the user id, the IAM username and validated-ness, none of which
principal.OrgFrom carries. ONE function (requestCaller) that every op asks, and
it fails closed off the HTTP path.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every route apps/ingress serves is now a typed op, so ONE registration feeds the
OpenAPI document, the MCP tool list, the CLI and the SDK. Before this the whole
edge control plane was 18 bare routes: no schema, no prose, no tool, no command,
no SDK method. It now contributes 18 described operations and 10 schemas.
Typing is a DESCRIPTION task, so the wire is preserved exactly, and that is
PROVEN rather than asserted: control_test.go drives the real router and pins the
status of every op — 200 on POST (not 201), 204 with an empty body on DELETE,
409 on a contested host, 403 for a validated non-admin, 404 across orgs, the
three list envelope keys, host normalisation, and hot-apply. The identical file
passes against the untyped handlers it replaces; that equality is the evidence.
- One receiver, `ops`, and every op a method value — the only bound form
cmd/zipdoc can lift prose from. The twelve CRUD ops share four generic
helpers (listOf/getOf/putOf/deleteOf) so a kind stays a parameter.
- Declared on the group, so each op's path is the prefix composed with its
leaf, which is the identity every projection keys on.
- cloud.Bridge() on the group, before the leaves: a typed op receives only a
context, so the request its SuperAdmin gate reads has to be parked there.
admin() now fails closed off the HTTP path — no request, no attested admin —
with no second gate to keep in sync. Pinned in typed_request_gate_test.go.
- DELETE takes its id from the URL and reads no body (zip v1.18+), and a body
naming another object can neither redirect a PUT nor smuggle a second delete.
Two things typing surfaced that were invisible while these routes were untyped:
- The fleet's OpenAPI schema namespace is FLAT. A typed op's Go type name IS
its schema name, and openapi.Weave refused this package outright because
`serviceList` is already apps/admin's launch board — one name, two shapes,
which every generated SDK would bind whichever it read last. An untyped route
contributes no schema, so the collision did not exist until now. The list
envelopes carry the product the namespace cannot (ingressRoutes, ...).
- plugin/websearch/openapi.json was stale on main: e83d7e90 moved the scrape to
/v1/scrape without re-emitting the subset, so the published spec named two
paths nobody serves and omitted the one that is. Regenerated here — the same
failure openapi-check exists to catch, second instance, found by running it.
Also: the status view claimed tlsHosts was a subset of liveHosts. It is not — an
extraHost owns no route, and a TLS route naming a missing service is skipped
while its host still wants a cert. Corrected, because that prose ships to the
document and the MCP tool description.
Gates: apps/ingress green (13 new tests), openapi green, root green, vet clean,
zipdoc -check clean, openapi-check regenerated from source. The 13 failures in
apps/{functions,platform,provisioning,storage} + plugin/kmsreseal are identical
at HEAD, before this change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Each converted route is now ONE registry entry: the REST route, the OpenAPI
operation with its schema and prose, the MCP tool, the CLI command and the
generated SDK method all follow from it. The 12 that stay untyped could not be
typed without changing what they accept or return, and each says why in the
code beside it.
Typed (7):
GET /v1/team/bots -> botRoster
POST /v1/team/bots/sync -> botSync
GET /v1/team/account/providers -> providerList
GET /v1/team/billing/plan -> planInfo
DELETE /v1/team/files/:workspace/:filename -> 204, URL-addressed
GET /v1/team/transactor/statistics -> statsOut
GET /v1/team/transactor/api/v1/statistics -> statsOut (the front's alias)
Left untyped, with the reason: the account JSON-RPC multiplexer (one POST, 20
verbs, a different result shape per verb); two OAuth redirects and two
cookie writers (302 + Set-Cookie is not an Out); the wallet page and the blob
download (asset BYTES under a per-response Content-Type); the multipart upload
(a form, not JSON, whose part filename IS the blob id); the collaborator RPC
(a second multiplexer); and the two WebSockets.
Latent defects this surfaced:
- team had NO cloud.Bridge. Serve installs one for the whole binary, so the
fused and plugin binaries were fine, but a bare Mount — the app's own test
harnesses, and any embedder that does not go through Serve — parked no
validated org at all. Installed on the /v1/team group, before the leaves.
- two test harnesses (billingApp, gateApp and its two siblings) built a
/v1/team group by hand with no Bridge, so they were exercising a wiring no
deployment has. They now build what Mount builds.
- the weave REFUSED the first attempt: team's botList/botView collided with
visor's — one schema name, two shapes (a workspace roster entry vs a bot
MACHINE), which would have bound every generated SDK to whichever it read
last. Renamed to botRoster/botMember.
- team had no //go:generate zipdoc directive, so no team prose could ever have
reached the document or the MCP tool list. Added in typed.go.
- GET /v1/team/billing/plan sets Cache-Control: no-store on per-tenant data and
nothing tested it; DELETE .../files/... answers an empty 204 and nothing
tested that either. Both are now pinned (typed_test.go), along with the
canonical /transactor/statistics path, which had no test at all — only its
/api/v1/ alias did.
The wire is byte-identical: same 19 routes, same statuses, same bodies, same
headers. The degraded (no SERVER_SECRET) 503 is preserved — a TypedHandler is
not a zip.Handler and cannot be wrapped by Mount's guard, so each typed op
returns the same refusal from its own first line, out of the one function that
states it. cloud.Request is confined to apps/team/typed.go (team authenticates
its billing/files planes with its OWN HS256 session token, which rides in a
header or an HttpOnly cookie that principal.OrgFrom cannot carry) and is pinned
with that reason.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The books surface was 25 routes and nothing else — no schema, no MCP tool, no CLI
command, no SDK method, and no `//go:generate zipdoc` directive in the package at
all, so its prose could not have reached the document even if a route had been
typed. Fourteen of them are now typed ops: one registry entry, four projections.
Typing is a DESCRIPTION task, so the wire is unchanged and that is now PROVEN
rather than asserted. wire_test.go pins all 25 routes — status, Cache-Control,
and the exact JSON envelope — and it was written against the untyped handlers,
run green there FIRST, and runs green against the typed ops with the same
literals. The package had no route test at all before this; every assertion in it
was over the store and the report engine.
Three things the typed signature drops, one home each:
- the VALIDATED org: principal.OrgFrom(ctx), parked by cloud.Bridge on the
/v1/books group. Never an In field — an In field is caller-supplied, and this
surface is a LEDGER, where that is another org's money.
- the ledger selector: `?sandbox` stays a STRING, because only the literal
"true" has ever selected the sandbox. Bool binding would additionally accept
"1" and a bare "?sandbox", handing a caller who asked for their live books the
sandbox's empty ones.
- Cache-Control: no-store. A typed op returns its Out and has no response to set
a header on, so it moves to the one place every books answer passes through —
noStore, on the group, on success only, exactly as booksJSON did.
A list route answers a bare JSON array, so its Out is a NAMED slice (accountList,
glList, bankTxnList): zip documents an anonymous type as no content at all, so the
name is what makes the array describable without wrapping it and moving the wire.
WHAT STAYED UNTYPED, and why — none of these is a wire change waiting to happen,
each is a wire change REFUSED:
- scan, inbox upload, bank/import take RAW document bytes (a PDF, an OFX/CSV
statement) as their body. There is no JSON input to name.
- ask, scan/book, vendors, rules, bank/sync read ?sandbox from the QUERY, and a
typed POST documents its whole input as a body. Typing them would move a
live/sandbox selector off the URL it lives on.
- metrics returns MetricsResponse, which EMBEDS Metrics. Go flattens an embedded
struct onto the wire; zip v1.18.3's schema walk does not. The published schema
would not match any answer this route has ever sent.
- bank/link-token and bank/exchange always answer 501. A typed op would publish
a success schema for a response neither has ever sent.
TWO LATENT DEFECTS SURFACED:
- `Vendor` meant two different things in books and admin — a vendor BOOK row
here, a vendor COST LINE there. The weave gate refuses it, correctly: every
generated SDK binds whichever it read last. books' side moved (VendorRow,
matching GLRow/BankTxnRow) because books' schema had never been published, so
the rename costs no caller anything; admin's would move a live SDK model.
admin's is the misnamed one — that is for the fleet dedup pass, not this one.
- zip drops or nests EMBEDDED struct fields when it builds a schema, and it has
already shipped: PATCH /v1/agents/targets/{id} publishes a body of `{id}` and
nothing else, because patchTargetIn embeds an unexported patchTargetReq. Every
generated SDK can send the id and none of the seven fields the patch exists
for. Same generator gap, already live, in the worked example.
The golden gained 818 lines and lost none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/company was 22 routes and nothing else: no schema, no prose, no MCP tool, no
CLI command, no SDK method. Twenty of them are typed ops now, so the whole
Stripe-Atlas-class incorporation flow — begin, structure, founders, KYC, docs,
esign, genesis, advance, skip, import, fundraise — carries a request/response
schema and a description into the document, the tool list and every generated
client. The app subset goes from 20 documented lines to 1149; openapi.yaml grows
907 lines and DELETES NONE, which is the document-level proof the wire did not
move.
Wire preserved exactly.
- begin answers 200 on the idempotent repeat and 201 on the first call. That is
correct REST and zip.WithStatus cannot express it (one declaration, one
status), so it is typed-but-shimmed: cloud.Created on the create branch only.
The conditional-status class, same as registerTarget.
- fundraise/round and fundraise/safe always answer 201, so they DECLARE it —
zip.WithStatus(201) — and the document now says 201 about routes that have
always sent 201.
- The 1 MiB JSON body cap was a line inside every handler's decode(). decode()
is gone (a typed op never sees the request), so the cap is now the ONE group
middleware limitBody, registered after the deck leaf and before every JSON
leaf. Same 413, same routes, one place instead of five.
TWO ROUTES ARE DELIBERATELY LEFT UNTYPED, and both are named at their
registration:
- POST /v1/company/payment. A billing denial answers the fleet-wide contract
cloud.DenyResource renders — 402 insufficient_balance / spend_cap_exceeded,
503 balance_unavailable, each {"error":{"code","message"}}. zip's HTTPError
renders {"status","code","error"}. Typing the route would silently reshape
that error for every metered client. This is not a company problem: it blocks
every metered create route in the fleet (~35 call sites) from typing until a
zip error can carry a body.
- POST /v1/company/fundraise/deck. The deck is the raw request BODY of any
content type, named by ?name=. A typed In would declare a JSON request the
route does not take.
Four things typing surfaced that nothing else would have:
- cloud.Bridge was not installed for this subsystem. Serve installs it app-wide,
so the monolith was fine and every app test — which mounts on a bare zip.App —
would have 403'd the moment an op became typed. Installed on the group, before
the leaves.
- openapi.Weave REFUSED the composition: schema "Summary" means a formation
register row here and campaign counters in apps/marketing. One name, two
shapes; every generated SDK would bind whichever it read last. marketing
already publishes its Summary, so company's — package-internal, no external
referent — becomes Registration. The untyped route hid this; the typed one
could not.
- apps/company had no //go:generate zipdoc directive, so its prose could never
have reached the spec or the tool list regardless.
- The surface root (POST/GET /v1/company) cannot be declared on the group: a
leaf of "" joins to "/v1/company/", a different path. It is declared on the
App with its whole path, and TestBodyCapCoversJSONNotTheDeck pins that the
group's middleware still reaches it — the fact the 201 and the 413 both
depend on.
TestRequestEscapeHatchIsPinned fired on my own change, which is the gate working.
apps/company/register.go is justified in the allowlist rather than waved through:
Hanzo forms the entity and carries the KYC/AML obligation, so the register and a
founder KYC decision are SuperAdmin operations that need X-User-IsAdmin and
X-User-Id for attribution, neither of which principal.OrgFrom carries. One
reviewer() in one file, failing closed off the HTTP path.
Gates: apps/company green (baseline was green), zipdoc -check clean, vet clean,
openapi weave green. plugin/kmsreseal's 6 failures are pre-existing — verified
identical with the change stashed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every /v1/framework route but the two document WRITES is now a typed op
(zip.Get[In, Out] and friends), declared on the group so each op's path is the
prefix composed with its leaf — the identity every projection keys on. One
registry entry, and the REST route, the OpenAPI operation, the MCP tool, the CLI
command and the generated SDK method all follow from it. Before this the whole
surface was route-only: 19 operations with no schema, no prose, no tool and no
client method.
The wire is unchanged — same paths, same statuses (201 on define/assign, 204 on
the three deletes, 200 elsewhere), same JSON, same percent-decoding of path
segments. The full pre-existing suite is green untouched, and the new
ops_projection_test.go pins the surface, the registry, the empty collection ([],
never null) and the summary body.
IDENTITY. A typed op receives only a context, so the engine Caller is assembled
from two carriers parked ahead of the leaves by g.Use(cloud.Bridge(),
bridgeFacts): the validated org from principal.OrgFrom, and the user id +
platform-admin bit from this package's own bridge. It is the same decision
caller() makes on the request — never an In field, which is caller-supplied and
would be a tenant key the caller asserted for itself. Off the HTTP path (an MCP
tools/call, a CLI LocalInvoke) neither bridge runs, both reads come back empty,
and the engine refuses 403 — the handler's own gate, no second gate to sync.
TWO ROUTES STAY RAW, with the reason recorded in Mount and checked by the test:
POST /v1/framework/:doctype and PUT /v1/framework/:doctype/:name take the
document's own field data as their body — an open object the DocType defines at
run time. A typed op's request schema is REFLECTED off its In type, and no Go
struct both accepts that body verbatim and describes it, so typing them would
publish a schema naming the two path segments and nothing else: an SDK method
that cannot send a document. They convert when zip can declare an open-object
input; a schema that lies is worse than the route-only entry they carry today.
WHAT TYPING SURFACED. The weave gate refused the first cut: engine.Summary as an
Out claims the fleet-global schema name "Summary", which apps/marketing already
owns — one name, two shapes, and every generated SDK binds whichever it read
last. Restated as a local summaryView. Re-exporting an external module's generic
type names into the fleet schema namespace is the general form of that hazard
(Role, Install and Document are the next candidates).
Also newly published: the ?filters / ?fields / ?order_by / ?limit parameters of
the document list, which existed on the wire and in no document.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The four routes that made a repo, an ssh key, a subscription and a mirror
target were the last /v1/git routes with a real JSON shape still registered as
raw handlers. They stayed raw for ONE reason, written in each of their doc
comments: they answer 201, and zip's typed registrar wrote 200 for a value and
204 for none with no seam to say otherwise. zip v1.18.2 closed that gap —
zip.WithStatus(201) declares the status ON the op, so the document's response
object is keyed on 201 and a generated client expects what the service sends.
That reason is now stale, so the routes convert:
POST /v1/git/repos createRepo -> repoView
POST /v1/git/keys registerKey -> keyView
POST /v1/git/repos/:name/subscriptions subscribe -> subscriptionView
POST /v1/git/repos/:name/mirrors addMirror -> mirrorTargetView
Each was a route and nothing else — no schema, no prose, no MCP tool, no CLI
command, no SDK method. Each is now one registry entry with all of them. The
wire is unchanged and the suite proves it rather than asserting it: 201, 400,
403, 404 and 409 on these four paths are pinned by existing tests (git_test,
ssh_test, lifecycle_test, hardening_test, public_repo_test, tenant_isolation_test)
and they pass untouched.
Registration moves onto the group. Every zip.<Verb> in routes() now takes `g`
rather than the *zip.App with an absolute path, so the /v1/git prefix lives in
exactly one place and each op's path is the prefix composed with its leaf — the
same composition the router does, and the identity every projection keys on.
cmd/zipdoc resolves it the same way since zip v1.18.3, so the prose reaches
both the document and the tool list.
The repo scope resolver collapses to one function. repoScope() existed only so
the raw creators could share the typed ops' preamble; with no raw creators left,
scoped() absorbs it and there is one way to turn a :name into a validated repo.
The principal keeps carrying the user. tenantFrom already read the validated
org and project off the request; it now reads c.User() there too, so registerKey
gets its owner from the bridge like everything else. An In field is
caller-supplied — a key written under a user read from one would let a caller
register a key in someone else's name.
Prose is product surface now, so it had to be TRUE, and one line was not:
addMirror's comment named git.hanzo.ai as an allowed mirror TARGET. It is
precisely the host mirrorOutHostAllowed excludes on purpose (Red MED-1 — an
internal SSRF that would make the server force-push with the shared token at an
arbitrary internal path). Typing lifts that sentence into the published
description and the MCP tool description, where it would have told SDK users and
agents to do the one thing the code refuses. Corrected to the real allowlist,
{github.com, gitlab.com}.
24 typed ops now, 24 raw. What is left raw has no JSON shape to type and says so
per route: the smart-HTTP pack protocol streams binary, the browser UI serves
HTML, the ZAP adapters answer a {status:"error", msg} envelope at a non-2xx that
a typed op cannot produce, and the webhook HMACs the RAW bytes and verifies
before it parses — typing it would invert that order and decode an
unauthenticated body.
Regenerated: apps/git/zipdoc_gen.go, plugin/git/openapi.json, openapi.yaml.
Described operations across the fleet: 170 -> 174.
hanzoai/cd renamed its module, so the engine is now published as
github.com/hanzoai/cd/gitops-engine. The old path stops at v0.7.2 — a tag can
never move — so an import pinned there sees no release ever again. This follows
it to v0.7.3, the first tag cut at the new path.
The version is the easy half. The engine at v0.7.3 builds against k8s 0.36.1 /
kubernetes 1.36.1, and replace directives in a dependency's go.mod are ignored,
so the pins that make it work inside hanzoai/cd do nothing here — this module
has to state them itself. It was pinning the staging tree to 0.35.3 while its
own requires already said 0.36.1, and k8s.io/api had no pin at all, so it
floated to 0.36.3 and lost the staging packages kubernetes 1.36.1 imports.
The whole tree now says 0.36.1, k8s.io/api included, and kubernetes 1.36.1.
controller-runtime follows to v0.24.1. v0.23.3 cannot compile against client-go
0.36: ResourceEventHandlerRegistration gained HasSyncedChecker, and its
handlerRegistration does not implement it. v0.24.1 targets client-go 0.36.
go vet ./... clean across the module, apps/deploy green, cmd/cloud links.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/integrations/github/repos/import and
/v1/integrations/github/repos/:repo/pages/builds were the only routes on this
surface with a real request and response shape still registered raw. The reason
was honest at the time: both answer 202 Accepted, zip wrote 200 (204 for a nil
Out) and cloud.Created only covered 201, so declaring 202 was not expressible —
and staying raw meant no schema, no prose, no MCP tool, no CLI command and no
SDK method for either one.
zip v1.18.3's WithStatus states that fact where every projection reads it, so
both are typed ops and the document says 202 because the op does. The wire is
byte-identical: same paths, same 202, same JSON keys (queued/repos,
repo/status/url), same 403/400 messages from the same two-step org gate. zip
binds body then query then PATH, so :repo still wins over anything a body
claims — the addressing authority is unchanged.
Latent defect this surfaced: the input type was named repoRef, and apps/git
already publishes a repoRef keyed by `name` (a repo hosted BY us) while this one
is keyed by `repo` (a repo GitHub grants our App). The OpenAPI schema namespace
is flat across the fleet and openapi.Weave refuses two apps that mean different
things by one name. The collision was invisible while every op taking this input
was bodyless — a GET/DELETE emits path params and no request schema — and the
first one with a body made the weave refuse. Renamed to githubRepoRef, matching
githubReposOut and githubPagesView, which already say what they are about.
Also: TestSpecCarriesProse told a failing reader to run
`go generate ./clients/integrations`. That path does not exist; the directive
lives in apps/integrations/ops.go. An error message that sends you nowhere is
worse than none, and only a reader who tried it would ever find out.
Raw routes left, both reasons properties of the wire rather than of effort:
thirteen 302 link/callback legs (WithStatus refuses a non-2xx at declaration, a
redirect is a Location header and not a JSON body, and the legs set signed
__Host- cookies), and six inbound webhooks whose auth reads bytes or headers a
typed op is never handed — Slack/GitHub HMAC over the raw body, Slack's
form-encoded /commands, Discord Ed25519, Teams' Bot Framework JWT and Telegram's
secret-token header. The webhooks also answer 200 with no body, which a nil Out
would turn into 204.
22 typed ops, 19 raw, each raw one carrying its reason in ops_projection_test.go
so the reason is checked and not merely written down.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
DESIGN ONLY — nothing wired. This marks the seam so the next change lands here
instead of adding a sixth object to every stateful pod.
Each replicated service currently carries four objects and a key: a replicate
container, a generated ConfigMap, a restore initContainer, and its own age
keypair. All of it exists because replicate is a separate binary watching a file
it does not own, so the file has to be described to it.
That arrangement produced four independent outages in one day (2026-07-29): a
misindented age stanza replicate refused, an age/plaintext mismatch between
config and bucket, a service whose data dir was not mounted, and a restore path
that had never once run. The last is the instructive one — restore only runs
-if-db-not-exists, so while the local file happened to exist it was never
exercised. The backups were configured, not current, and not restorable, and
nothing said so until a volume was lost.
Open is already 'the single way a cloud store opens its file' and Exists already
answers 'is there a store here' — which is the entire question the initContainer
shelled out to ask. Native, the lifecycle collapses into Open: hydrate if
absent, follow after. Restore stops being a lifecycle stage and becomes what
Open does; the ConfigMap, initContainer, second container and the ordering
between them all disappear.
ONE KEY. cek already holds CLOUD_KMS_MASTER_KEY_REF and refuses to open a store
unkeyed. The age identity is a SECOND key system encrypting the SAME data, with
no rotation story at all — an age identity cannot be rotated after the fact, so
losing it makes every replica under it unreadable. So the age keypair should not
be migrated to KMS; it should stop existing, and the replica should be encrypted
under the key the process already holds. That makes the KMSSecret file added to
universe today unnecessary rather than merely unarmed.
Three verbs, all about bytes at a path: Has, Hydrate, Follow. Follow returns
when following STARTS, not when caught up — a store that refuses to open until
its backup is current will not open during an S3 incident, trading a durability
risk for an availability one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/deploy/projects asked for appprojects at group argoproj.io. Nothing has
ever served that group here, so the request returns "the server doesn't have a
resource type" — and listAppProjects treats any error as "the CRD is absent"
and synthesizes a project set instead.
The cluster serves appprojects.apps.hanzo.ai and has real ones (default,
hanzo). The fallback was not covering for a missing CRD; it was covering for
asking the wrong question, so the operator's actual policy envelopes never
reached the dashboard.
The comment above the GVR said "this plane does not run argocd, so the CRD is
normally absent." True once. It stopped being true when Hanzo CD was installed,
and the code kept believing it.
CDApplications already named apps.hanzo.ai correctly one file over. The project
GVR was declared privately in projection.go and missed, so it moves to apps/k8s
beside its sibling — where GVRs live so the next one cannot drift alone.
Unchanged deliberately: the argoproj.io/v1alpha1 apiVersion strings the
projection EMITS are response shape the cd-ui SPA consumes, not a query. Those
need the UI checked before they move.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
research and deep were never two things. Same system prompt, same models, same
plan gate; only the dials differed (4 queries/12 sources/4 reads at 5c vs
6/16/6 at 10c). Two names made the product look like it offered a choice, and
hid what that choice cost behind an adjective.
research now always does the deeper pass and carries the price that pass actually
costs. 'deep' stays a retired NAME that folds into research in both IsMode and
resolveMode -- not a mode. Dropping it outright would have been worse than a
rename: an unrecognised mode falls through to search, so a client that had not
shipped the collapse would answer a deep request with ONE query. A wrong answer
is worse than an error.
Note search and news are now the closer pair -- identical but for newsBias.
Left alone: that one is a real editorial difference, not a duplicate.
Tests updated to the new contract and negative-controlled: removing the fold
turns IsModeAndResolve red.
It was mounted at /v1/websearch/v1/scrape -- a /v1 inside a /v1. That was not
a choice, it was fallout: the firecrawl client always builds
{apiUrl}/{version}/scrape, and firecrawlApiUrl pointed at the group.
Point firecrawlApiUrl at the API ROOT instead and the same client lands on a
clean top-level /v1/scrape. The path was freed by deleting ai's crawl-and-index
route of that name, which was a second door onto object.ScrapeAndIndex.
Requires FIRECRAWL_API_URL=https://api.hanzo.ai (was .../v1/websearch) in the
chat config. FIRECRAWL_VERSION stays v1.
Also drops the bare /v1/websearch/scrape duplicate -- two paths for one handler
is the thing this commit is removing.
zip v1.18.3 teaches cmd/zipdoc to resolve a group's prefix the way the router
composes it, so the ergonomic form is finally the correct one. The five target
ops move from spelling their whole path on the App to
`zip.Post(g, "/targets", …)` on the group the subsystem already has.
The published document is BYTE-IDENTICAL across the change — the drift gate
regenerated all 1006 paths and found nothing to write — which is the point: the
op's identity was always the composed path, and only the source ergonomics were
awkward.
Verified rather than inferred, because the whole lesson of this bug is that
extraction can silently disagree with what you believe you registered:
zipdoc_gen.go now keys all five under /v1/agents/..., and
TestTargetOpsProjectEverywhere asserts the handler's doc comment appears in BOTH
the OpenAPI description and the MCP tool description. It passes on the group
form.
The playbook drops the "spell the WHOLE path" workaround and tells the next
agent to declare on the group, with a note that an unresolvable router now fails
loudly naming the call rather than filing prose under a path that does not exist.
A doc that instructs people toward the awkward form is the same defect as a
comment promising a guarantee the code does not provide — it stops them looking.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestTargetOpsProjectEverywhere is the demonstration that makes the rest
mechanical: ONE typed op, asserted in all four places it now exists — the
OpenAPI operation (operationId, the handler's doc comment as description, a
$ref'd schema, the doc comment's own Example), the MCP tool (same description,
same fields in inputSchema), the CLI command (`agents targets-create`, flags off
the In type), and the same operation id addressing it through all of them. Plus
TestTargetDeleteIsURLOnly, which pins the v1.18 wire: no requestBody, one
required path parameter.
An untyped route has exactly ONE of those. That is the whole argument for
converting 986 of them, stated as a test that fails if any surface stops being
derived.
The playbook in LLM.md is the recipe as actually executed, with the parts that
are not obvious: a receiver rather than a closure (the only bound form zipdoc can
lift prose from), spell the WHOLE path because zipdoc keys on the path literal
and a group-declared op files under its leaf, Bridge on the group before the
leaves, identity never an In field, and doc comments written true because they
ship to both the reference and the tool list.
It carries the four failure modes found the hard way, because whoever takes a
partition will hit them and will not know to look: a gate comparing two derived
artifacts agrees with itself while both are wrong (how ingress lost 8 paths from
every SDK); the stale-tree pin walk-back, which is mechanical and will recur;
verify what CI actually invokes before trusting a gate you add to a make target;
and porcelain-not-diff, because a new app's subset is untracked and a diff cannot
see it. registerTarget is written up as the worked conditional-status example —
typed-but-shimmed until zip's multi-status responses land, so nobody invents a
third mechanism.
The partition list is six tranches over disjoint apps/<app>/ trees with the
re-measure command, since the counts move every few merges. Source does not
collide; two artifacts do — the regenerated golden and go.sum — and both resolve
by rebasing and regenerating rather than by hand, because the generator is
deterministic.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The first tranche of the typing migration, and the proof the chain works end to
end. /v1/agents/targets register|list|get|patch|delete are typed ops now, so the
machine-target API has a schema, prose, an MCP tool, a CLI command and a
by-name call target where it had a URL and nothing else.
Wire preserved EXACTLY. registerTarget still answers 200 on the idempotent
re-link and 201 on first registration — a distinction zip.WithStatus cannot
express, because it declares ONE status and this route legitimately has two. It
therefore keeps cloud.Created on the create branch only, and is the worked
example of the conditional-status class: typed-but-shimmed until zip grows
multi-status responses. Every other route answers what it always answered; the
app's own tests, which assert 201/200/400/403/404 across all five, pass
unchanged.
Two things typing surfaced that nothing else would have:
- cloud.Bridge was not installed for this subsystem. The untyped handlers read
identity straight off the request; a typed op receives only a context, so
the validated org has to be parked there. Installed on the group, the shape
apps/search and apps/integrations already use, before the leaves — fiber
runs middleware in registration order, so one installed after them never
runs.
- cmd/zipdoc keys prose on the path LITERAL in the registration call, so an op
declared as ("/targets") on a group files under "POST /targets" while its
real identity is "POST /v1/agents/targets". docFor never matches and every
doc comment is dropped from the document AND the tool list — silently, which
is the failure mode this whole effort exists to kill. zip gained group
registration in v1.18.0 and zipdoc did not catch up; that is mine to fix.
Until it does, a typed op spells its whole path (the apps/git shape) and the
group carries only the Bridge, which is prefix-matched and applies either
way. The comment at the registration says so, so the next person does not
rediscover it.
apps/agents gets its //go:generate zipdoc directive — it had none, so its prose
could never have reached the spec regardless.
TestRequestEscapeHatchIsPinned fired on my own change, which is the gate working:
targetOwns and targetCaller are new cloud.Request sites. They are justified in
the allowlist rather than waved through — ownership needs X-User-Id and
org-admin-ness, neither of which principal.OrgFrom carries, and both fail closed
off the HTTP path where there is no attested caller.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A plugin that will not start returned an error from zip.Load, and this host
escalated it to os.Exit(1). pubsub is Apps[0] and Eager, so when its child could
not open /var/lib/cloud/audit.db the process died with every other subsystem in
it — the API, IAM validation, billing, the team backend — and api.hanzo.ai and
cloud.hanzo.ai were 502/503 for 25 minutes:
cloud: zip: Add service 0: zip: Load(pubsub): exited before listening: exit status 1
zip returning an error is a library reporting the truth; turning that into a dead
process was policy, and the policy lived twice as `return err` in two loops. It is
now one function that both loops call:
required -> abort, naming the app. Nothing sets it; the argument is pinned.
otherwise -> ABSENT. The mount stands with no process, so the prefix answers
zip's own 503 instead of falling through to the console at "/".
Keeping the mount is load-bearing, not cosmetic. webui refuses only its
apiPrefixes list, so seven prefixes across five apps answer 200 text/html when
unregistered — including iam's /login/oauth, where an OAuth client would receive
the console shell instead of a redirect.
Absence is loud in three places, because a silently missing subsystem is the
failure this fleet keeps paying for: an error log, the reason on the host's
/healthz, and Running=false in zip's plugin table. /healthz stays 200 and
"status":"ok" — failing liveness for an optional plugin would recreate the outage
one layer up.
Required is a property of the app rather than of start order. Being first in a
list is not a claim on everyone else's availability.
Second half: why the child had no key. Every child this host spawns carries a
CREDZ_TOKEN, and credz refuses to fall back to a dev key once a token is present,
so a deployment with no broker is one where EVERY child resolves Unkeyed and dies
at its first store open. Two silences fixed: an --enable list that omits the
broker mounted zero brokers and said nothing (now refused, like a name the
manifest does not list), and a process holding the root key that declines to
broker now says which half it is missing rather than returning nil.
Reproduced end to end with the real binaries, then proven: host alive, /v1/flags
200 application/json, /v1/pubsub 503 JSON, "/" still the console, /healthz naming
both absent subsystems with their reasons. Four rows added to scripts/mutate.py;
4/4 KILLED.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
I had written that credz was a second protocol doing what the internal plane does,
and that its transport should collapse into kms methods. That is wrong, and the
correction matters: answer() calls parseIdent(call.Cap) and nothing checks it, so a
capability on the plane is whatever the caller wrote. credz proves the app name with
a launcher-minted token the caller cannot forge, which is precisely how a child gets
its own scoped bundle and not a sibling's.
Collapsing credz into the plane would have traded a proof for a claim. Recording what
each actually establishes, so the next person does not make the trade I nearly did —
and so nobody reads a plane capability as an authorization decision.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Five subsystems broke the same way in one day — a process without the store found
nothing and said nothing useful — and each was diagnosed from scratch because the
rule was not written down anywhere. It is now: a store has one owner, everyone else
asks, and the peer-absent/peer-answered-badly distinction is the part that decides
whether a fallback is correct or a 502 on a healthy deployment.
Also records what make build does NOT build, which cost an afternoon of 503s that
looked like a broken product.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/runner {release:true} demanded the machine build token ALONE. That
put a second auth system beside IAM: a SuperAdmin identity — which by
definition may do anything, and is trusted with KMS and every tenant's data —
was refused a release.
Every other privileged surface in cloud reads principal.IsSuperAdmin. Release
now reads the same predicate, or accepts the machine token CI runs under. An
org admin can still build and still cannot release, so the separation that
mattered is kept; what goes is the parallel authority.
IAM decides permission. A token is a transport, not an authority.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The only trigger was push-to-main, so recovering from a bad image meant landing
another commit and waiting — and POST .../workflows/cicd.yml/dispatches answered
500, which reads like a broken forge rather than a workflow that never opted in
to workflow_dispatch. Today that cost real time while production sat on a rolled-
back image.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.801.313 refused EVERY POST across the fleet with
"audit: persist: UNIQUE constraint failed: audit_log.seq" — ~94 a minute, not
self-healing, hitting tasks, integrations, visor and chat completions alike.
Because the audit gate fails CLOSED (correctly, AU-5), the entire write surface
was down while reads looked fine.
The cause is not the audit code, which is right: audit_log.seq is a gapless chain
position and every row's prev_hash seals the one before it, assigned under a
mutex, recovered from MAX(seq) at boot. That is correct for ONE writer. Every
process opened {DataDir}/audit.db, which was harmless while cloud was a single
binary and became a total outage the moment subsystems became plugin CHILD
PROCESSES: each child recovers its own in-memory nextSeq from the shared file and
then they all race for the same PRIMARY KEY.
Retrying the insert would not fix it. Two writers cannot share a hash chain, they
can only fork it — a retry would seal the new row against a prev_hash that
another process has already superseded, trading a loud constraint error for a
silently broken chain. So each process gets its OWN chain, which is exactly what
procName already exists for ("per-process resources ... instead of contending for
one global name"). The host keeps audit.db, so its history and the
/v1/admin/audit surface are untouched; children get audit-<app>.db.
Fleet-wide completeness is preserved by the OLAP mirror, which every Recorder
already writes to a SHARED datastore table — per-process files are the
tamper-evident authority, the mirror is the aggregate query surface.
audit_serve_chain_test.go pins the property that matters: distinct processes
never resolve to the same file. Collapse them back onto one name and the test
fails before the fleet does.
(./audit/... is unrunnable on macOS — it needs a RAM-backed scratch dir; it fails
identically on clean upstream, so it is untouched by this change and runs in CI.)
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
credits, usage and the ledger page are three projections of ONE list — the org's
entries — and all three answered 501 from a process that does not hold the ledger,
which is every process but commerce. financeTxns asks the owner now and falls back
to the configured commerce URL only when no peer serves it, the same split-deploy
shape balance already honours.
finance gained the read it was missing. ListUsage keeps only the usage debits by
design, but a credit and a welcome grant are transactions too when a customer is
looking at their account, so ListEntries returns the entries unfiltered and each
page decides which kinds it shows.
The amount crosses as its 18-DECIMAL INTEGER — money.AttoString, the storage and
on-chain form, parsed back by money.ParseInt. That pair is exact by construction:
no decimal point to misplace and no scale to agree on. It is flattened to cents at
the boundary where commerceTxn is already a cents-shaped view, so the day that view
stops being cents-shaped the precision is already on the wire waiting.
Two money packages exist and both are right: hanzoai/money is the general exact
value, and apps/money is cloud's USD carried at 18 decimals so an off-chain ledger
amount and an on-chain uint256 are THE SAME INTEGER. The ledger speaks the latter,
so the wire does too.
A missing peer stays INERT for the starter grant rather than becoming an error. A
deployment with no money plane at all is a real shape, and erroring would put a line
in the log on the first request of every wallet in it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The welcome grant never fired. StarterGrant is middleware on EVERY app's chain and
it bailed the moment it found no local ledger — which, once apps became their own
binaries, is every process but commerce. A new org reached tracker or billing, the
grant looked for a ledger one socket away, and returned silently. The org opened
broke, and the paywall then refused it correctly for a reason nobody had chosen.
Asking the owner is safe to do from anywhere: the grant's idempotency key is the
ACCOUNT and nothing else, and finance dedups on it inside the same transaction as
the insert, so two processes racing the same new wallet still grant once.
Usage had the same hole and answered 501 on a customer's own usage page. It carries
ROWS, not a rendered envelope: the ledger's owner knows what was debited, the HTTP
surface knows what its page looks like. Sending the envelope would have put one
app's response shape inside another app's process and required the renderer to live
with the ledger — which is the import cycle that shape implies, made visible
(billing already imports commerce; the reverse would close the loop).
This is what spec 136's exactly-once debit was waiting on. It has SKIPPED all day —
acme could never cover the fee, because acme was never funded — so the property the
whole prepaid suite exists to prove was the one thing it did not prove. It now runs:
17 passed, nothing skipped.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
CI did not run `make test`. It never has — there is no .github/workflows, and
hanzo.yml runs six discrete steps that name `go test` and `make -f mk/fleet.mk`
directly. So the gate added to `make test` protected local runs and nothing else,
and the constraint "CI runs make test, never test-fast" could not be satisfied by
splitting the make target alone. hanzo.yml had to change.
app-contract now calls mk/fleet.mk's openapi-check instead of restating half of
it inline. That is the fix, not just a refactor: the step used to regenerate the
per-app SUBSETS and check those, so openapi.yaml — the file the SDK repos
actually pull — was never checked against source by anything. openapi-composed
compares it to the subsets, and both are derived. Two gates, neither of them
looking at the routes, which is precisely how plugin/ingress lost eight paths
with everything green.
Calling the target also means one gate definition rather than two that drift, and
CI inherits the kafka exemption instead of dying on an app whose Mount is
fail-closed on a live broker.
The gate now checks `git status --porcelain`, not `git diff`. A NEW app produces
a NEW subset, which is untracked and therefore invisible to a diff — the failure
that matters most is the one a diff would miss. That reasoning was already in the
step it replaced; it belongs in the gate.
test-fast is the inner loop: everything `test` runs except the drift gate, which
rebuilds one binary per app and dominates the wall clock. It ANNOUNCES the skip
on every run, names what will fail in CI, and prints the command that checks
properly — the same reason the gate names its kafka exemption out loud. A skip
nobody sees is how a gate becomes decorative. It is documented nowhere as the
default, and CI does not reference it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`make test` claimed the weave caught a route added without regenerating. It did
not, and the comment saying so was the bug in one sentence.
openapi-weave compares two COMMITTED artifacts — the per-app subsets and the
golden they weave into. Both are derived, and nothing in that comparison forces
either back to the routes, so they agree with each other while both are wrong.
plugin/ingress proved it: eight paths (/v1/ingress/routes, /services,
/middlewares, /tls, /status and their :id forms) were added, the subset was never
regenerated, the golden was woven from that same stale subset, the gate stayed
green — and the entire ingress API was absent from openapi.yaml, and therefore
from every generated SDK. No Python, Go or TS caller could reach it at all.
openapi-check regenerates every subset and the fleet spec FROM SOURCE and fails
on any diff, printing what to run. It is in `make test`, expensive half and all,
because the cheap half is exactly the check that passed while the published
document was missing an API.
Both drift classes are PROVEN caught, not hoped for:
- a route added in source without regenerating: added one to apps/ingress, ran
the gate, watched it name the stale subset, reverted.
- a dependency walked back so the document can no longer be reproduced: reset
the pins to what bb10586e committed (commerce v1.49.30 -> v1.49.29, zip
v1.18.1 -> v1.17.6) and the gate went red with a 6045-line diff in
openapi.yaml plus four subsets. Nothing else on main detects that.
On that second one, since it will recur: bb10586e is not a bad merge. It is a
single-parent commit directly on top of 49f8eeec whose go.mod hunk downgrades
both pins outright — the signature of `go get`/`go mod tidy` run in a tree that
predated the bump and then committed wholesale. Any agent working from a stale
tree reproduces it, which is why the answer is a gate rather than a note asking
people to be careful.
kafka is exempt BY NAME and says so when it skips: its Mount is fail-closed on a
live pubsub broker, and a document is a projection of routes, not a reason to
need a running message bus. That is a defect in the app, not the gate. The
exemption is provably free — kafka's subset declares zero paths — and it goes
away when the adaptor moves to hanzoai/stream.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip v1.18.2 adds WithStatus, so the status a successful op answers with is
declared on the op and keyed into the document's responses object. cloud.Created
and cloud.Accepted are the reason it exists: they set 201/202 per request from
inside the handler, which works on the wire and nowhere else. The status is a
CONTRACT detail, and setting it there writes it into a side channel no
projection can read — the document says 200, every SDK generated from it says
200, and the route has always sent 201. Same failure as a query parameter's
required-ness being invisible: a contract detail that exists only at run time is
not a contract.
Both are now marked Deprecated, pointing at zip.WithStatus, and both still work.
They are NOT ripped out: 13 call sites depend on them today and 85 untyped
routes still return 201/202 and have not been converted. Converting those on top
of a workaround would have been knowingly writing debt, which is why zip got the
fix first; new ops declare WithStatus and the shims retire as the migration
reaches them.
cloud.Request is pinned. It is the escape hatch that hands a typed op its raw
request, so every use gives back some of what typing bought, and nothing in the
signature stops the next one. TestRequestEscapeHatchIsPinned asserts exactly the
four that exist and carries the reason for each — three identity gates that need
more of the validated principal than the org (admin-ness lives in a header
principal.OrgFrom does not carry) and one tenant-scoped proxy that FORWARDS the
caller's identity upstream. A fifth now has to edit the gate and write its
justification, which is a decision rather than a drift. Verified by adding a
fifth call site and watching it go red.
Also restores the pins bb10586e walked back — commerce v1.49.29 -> v1.49.30 and
zip v1.17.6 -> v1.18.2. Main was green either way, but it had my regenerated
documents committed against a zip that could not produce them, so the next
`make openapi` would have silently dropped the parameter examples, the derived
required-ness and the $ref sharing.
Regenerating also caught real staleness nobody had noticed: plugin/ingress was
missing 8 paths (/v1/ingress/routes, /services, /middlewares, /tls, /status and
their :id forms). Routes had been added without regenerating the subset, and
because openapi.yaml was woven from that same stale subset the two agreed with
each other while both omitted the surface — so the published spec the SDK repos
pull has been missing the whole ingress API.
Pre-existing failures unchanged (functions/provisioning/storage/kmsreseal need
billing config, S3 and a real KMS; graph is a DNS flake that passes on retry).
Host stays 401 packages with zero apps/* imports.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
isOrgAdmin matched only "admin", and IAM's coarse membership vocabulary is three
values: owner, admin, member (iam internal/store/membership.go). `owner` is the
one IAM assigns to whoever CREATES an org — self-service provisioning calls
EnsureMembership(..., RoleOwner) exactly so "a self-service org is born with
nobody on it" cannot happen (iam internal/oidc/provision.go:247).
So every org created through self-serve signup had a founder its own org-scoped
admin surface refused, across 25 files that consume principal.IsOrgAdmin. It is
the strictly worse version of the bug this function was WRITTEN to fix — the
comment above it describes the org-scoped admin surface "refusing its own owner
with 'admin required'" — because an admin could be granted by someone else, while
an owner has nobody above them to escalate from. It landed with self-serve org
creation, which is why it was not visible before.
This only restates a membership IAM already signed: the role is read from the
verified `orgs` claim, and a caller who is not in that set is admitted by nothing.
The org stays a VERBATIM compare (a fold would let a member of "acme" claim
"ACME"); the ROLE is folded, since it is a closed vocabulary IAM controls. IAM's
money path already treats the two as one (billingAccountFor admits
{RoleOwner, RoleAdmin}); this is the authz half of that same fact.
Test proven to FAIL on the old code before it passed on the new: owner, cased,
and whitespace-padded variants all returned false. Full package diffed against
baseline — identical failure set (21 pre-existing env failures needing tmpfs at
/dev/shm), zero introduced.
Closes the loop between the two moneys. cloud is what can see the vendor, so it
publishes; ai is where spend happens, so it enforces. The state is pushed from
the SAME numbers this board renders — credit remaining and average daily burn —
which is what stops the guard and the dashboard from disagreeing about whether we
are spending real money. OnCash is credit <= 0: the promo grant is gone.
The ceiling is CLOUD_DAILY_CASH_CEILING_CENTS and defaults to 0, which DISARMS
the breaker, so this is observational until an operator states a number. Every
ambiguous input — unset, blank, garbage, negative, and notably "200.00" (cents
written as dollars, the likely typo) — resolves to 0 rather than to some small
accidental ceiling. The failure direction is always "allow": this value gates all
paid inference, so a bad ConfigMap must cost a day of unguarded spend, never a
fleet-wide outage. cash_ceiling_test.go pins each of those.
Context: $250,573 of outstanding platform credit stood against $0.00 of DO promo
credit and ~$65/day of real burn, with nothing between them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip v1.18.x closes four gaps between what a typed op DOES and what its document
SAYS. Taking it regenerates the published spec — the file the SDK repos pull —
and unblocks the migration these routes were waiting on.
* A typed DELETE no longer reads a request body; its input is the URL. All 20
of cloud's DELETE ops keep working: 15 take only path params, and the other
5 take scalars that bind from the query exactly as the document already said
they would. DELETE /v1/marketing/suppressions is the one with no path param
at all, so its whole input is now ?channel=&address= — which is what a
client generated from openapi.yaml has always sent.
* A URL-borne field's `validate:"required"` reaches its parameter, so an
argument the handler refuses to run without stops being described as
optional.
* A bodyless op's example survives. openapi.Parameter had no name for it, so
the round-trip through Typed() dropped it — exactly the "dropped honestly,
if nothing here has a name for it" its own doc comment warned about. Every
GET and DELETE reached the published reference with no example at all;
adminDeleteSpendCap now carries example: cap_1 on the path and acme on the
query, from the one Example its doc comment already had.
* A path parameter is typed from the field it binds to, so ?sizeGiB= is an
integer in the document because it is an int in Go.
openapi.yaml is 245 lines SHORTER despite 128 new example/required/query lines:
a named struct is now one definition every op $refs instead of being inlined at
each use. 148 schemas across the admin subset alone.
probe/ is inverted, which is what it told whoever came next to do. It pinned two
zip limitations — a typed op cannot see its own URL, and a templated path is
emitted with no parameter object — and both are gone, so it now asserts the
capabilities: the whole URL binds, and /v1/agents/sessions/{id} declares its
parameter, typed. The 16 of 25 clients/agents routes that carry a path param
are unblocked. What stays pinned is the DEFAULT, not a framework limit: an op
with no Authorize installed answers an anonymous MCP caller.
commerce moves to v1.49.30 for the same reason — Mint decorates a zip.Router,
so it now answers for where a typed op lands, gate included.
Pre-existing failures unchanged (functions/provisioning/storage/kmsreseal need
billing config, S3 and a real KMS); the two probe failures are fixed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Enabled() had three answers where it needed two. A "staged" set was held back from
the mount-everything default and reachable only by being named — in CLOUD_ENABLE or
in the parallel CLOUD_ENABLE_STAGED — so "what runs here" was decided by two env
vars and a table, and a subsystem could be on by one lever while off by the other.
The membership rule was stated plainly: a subsystem is staged while its Mount can
ABORT STARTUP. Apps are their own processes now. A Mount that fails takes its own
child down, the host stays up, and its prefix answers 502 — which is what happened
when kms crashed and the rest of the fleet kept serving. The risk the exception
existed for is gone, so the exception goes: empty list mounts everything, a
non-empty list mounts exactly what it names, and there is no third case. The knob
appears in no manifest.
Also fixing main, which was red before any of this: TestOrgForKey asserted cloud
should read a key through /v1/iam/users/get. That route takes (owner, name) and
returns a bare user — it cannot answer "who owns this access key", so the
assertion named a door that could not open. The caller was right all along;
get-user?accessKey is what resolves a secret key, and IAM's own doc calls
resolve-key "the dual of get-user?accessKey". The test's four comments already
said so; only its want list disagreed.
go vet ./... clean; go test . ok; e2e 16 passed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Found by settling a question rather than assuming it: cmd/cloud's own doc says
it "serves the whole API by mounting every subsystem as its own process". So a
Register* seam — which resolves a package var inside ONE process — cannot
connect two apps at all. projects calls, git registers, and they are never the
same binary.
That means the visibility publisher I wired earlier tonight has never worked in
production. Public projects have not been getting their canonical repos, and
nothing said so: share() logged a warning only when the publisher RETURNED an
error, and a nil publisher returns nil. Silence is what the seam produced, and
silence is exactly what this plane exists to stop.
Now: projects packs cloud.Visibility and calls git.publish over git's socket;
git exposes it beside git.files. A missing git is an error naming the app, not
a no-op. The rest of the contract is unchanged and still lives on git's side —
the repo is created either way, visibility is applied on both hosts, and a
retraction is a flag flip rather than a delete.
community.go is deleted: RegisterPublisher/PublisherRegistered/Publish had no
callers left. Visibility itself moves to payloads.go, which is where it belongs
once it is a wire contract — the type and its codec in one file, so the two
halves cannot drift.
The projects tests now stand a real git peer on a real socket instead of
registering an in-process func. That matters more than it sounds: the old test
passed precisely BECAUSE it registered something, which is the one thing
production never did.
EIGHT more seams are broken the same way and are NOT fixed here, all failing
loud (ErrGitImporterUnavailable and friends) rather than silently, so they are
visibly-unavailable features rather than wrong data:
ImportGitRepo, InboundGitSync, GitRepoStatuses, EnsureGitMirror git ← integrations, sync
UpsertIssue tracker ← integrations
Sync sync ← integrations
OnGitPush, OnServiceRelease platform ← git, integrations, deploy
Four seams are genuinely in-process and stay: the commerce and KMS client
factories, the org-scope resolver, the lifecycle fanout and the trace sink.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Four stages, checked separately so a failure names the broken link.
Staged rather than one assertion because of what it currently reports: stage 1
passes (POST /v1/event -> 200, with a 404 control proving it is routed) and
stage 4 passes (insights, analytics, sentry all 200), while stage 3 fails —
traces 24h stale, logs 32h. The door accepts and the surfaces load, so a
shallower test reports green on a pipeline dead for over a day.
Stage 3 is load-bearing: accepted is not stored.
Stage 2 names the cause — neither 4317 nor 4318 is bound, because the traces
and logs receivers both bind 4317 and the collision kills the whole pipeline.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Picks up the two fixes for the class of outage that hid a broken backup
plane for eleven days while every dashboard read green.
v1.52.2 — the scheduler stops discarding errors. sweepSchedules' return
was `_ =` and per-schedule StartWorkflow failures hit a bare `continue`,
so a schedule that could never fire said nothing, forever.
v1.52.3 — the layer that ACTUALLY failed here becomes observable. Our
incident was one step below the scheduler: apps/cron fires a JobWorkflow
whose RunJobActivity re-reads the entry's ConfigMap at fire time, and the
ServiceAccount could not read ConfigMaps. StartWorkflow SUCCEEDED every
time — actionCount reached 4489 — so the scheduler was healthy and
correctly silent while the activity failed ~4500 times with no log, no
counter and no durable record. Activity/workflow failures now write a
durable per-(workflowType, activityType, taskQueue, scheduleId) streak
row, readable via View.FailureStreaks(ns), plus throttled WARN/ERROR/INFO
lines that distinguish "failed once, will retry" from "failing
persistently".
Keying on the recurring shape of the work rather than the run is the
load-bearing detail: every cron fire mints a fresh runId, so a run-keyed
counter would have reported "attempt 1 of 10" forty thousand times and
never once "dead for eleven days". Persistence is measured by AGE, not
count — a nightly backup reaches a count of 2 in two days and is plainly
broken, while a single run burning its 10 default attempts in ~5.5
minutes must not page anyone.
Retry and isolation semantics are unchanged; this is observability only.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The file credited an upstream identity server as the origin of Hanzo IAM.
That is wrong. github.com/hanzoai/iam is clean-room original work and its
own LICENSE says so; the derived identity server was iam-v1, which is
retired and ships in no current product. A NOTICE entry is a legal claim
about this codebase, so a wrong one gets removed, not reworded.
The Casibase attribution stays untouched: the Hanzo AI module genuinely
derives from it, and Apache-2.0 section 4 requires that notice.
The backend repo dropped its -go suffix once the TypeScript codebase holding the
name moved to hanzoai/team-v1, so the module path is github.com/hanzoai/team.
apps/team builds clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CLOUD_KMS_ZAP_ADDR selected clients.KMSRPCAt — a stub whose every method returned
"not yet wired (zapc-gen pending)". Setting it produced a KMS client that failed
every call while looking configured, which is worse than having none: a deployment
that sets the address gets silent, total secret failure and a config file that says
secrets are wired.
There is one way to reach the store now: the kms app over the internal plane. The
knob, its config field and the stub go with it, because a second path that has
never worked is not a fallback, it is a trap with a name.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
8a1b35d4 pointed the resolver at the right endpoints and left one expectation
behind, so main's own suite failed on a contract the code had already got
correct.
IAM has three doors and only one answers this question. get-user?accessKey
resolves a SECRET key (hk-/sk-) to its owning user behind CapKeyResolve, and
refuses a pk- by design; resolve-key is the publishable door, org-only;
users/get is the typed (owner, name) read, which carries no accessKey and
cannot answer at all. The test wanted users/get for a key lookup.
Fixed the test, not the code — the resolver was right. The comment now names
what each door is for, so the next reader does not have to re-derive it from
the IAM module to know which one belongs here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
"not a member of this room's workspace" describes a membership determination
this code never makes. meet has no members table, no store, and makes no call to
IAM — measured: zero lookups in the package. Membership was decided upstream at
the IAM login that minted the session, and is already signed into the token as
`workspace`.
What actually happens is narrower and worth naming: the room asked for must
belong to the workspace the token already names. It refuses to WIDEN an existing
decision; it does not make one.
The old wording reads as a second authorization system living in meet, which is
exactly how it was reported. The check is right; the sentence was wrong.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
platform.fleet landed reading CurrentFleet() — the package global whose nilness in
admin's binary is the bug the move was fixing. Keeping it meant the method observed
through fleetObserver.Observe, which scans scanOrder(): the three first-party
namespaces, hardcoded. /v1/platform/fleet scans discoverNamespaces() and confines the
result with scopedNamespaces. So the two boards did disagree about what the fleet is,
in the specific way nsClass was written to make impossible — every tenant-<org>
workload was on one board and missing from the other — and the plane had a second,
unconfined path to the estate held safe only by a blunt who.Admin gate at its mouth.
The method now binds to the service, which lets fleetRoutes register it and lets the
observer seam go: Fleet, PublishFleet, CurrentFleet and fleetObserver are deleted,
because a method that holds s has nothing left to ask a global for.
The scoping is one decision with two readers rather than two rules that drift.
fleetPrincipal is the caller reduced to the four facts this surface authorizes on;
requestPrincipal builds one from a zip.Ctx and capPrincipal from a delegated
capability; mayObserve is the whole role gate and scopeNamespaces the whole tenant
boundary, each in one expression the HTTP handler and the method both call.
principal.OrgOf came out of principal.Org the same way, so the org-key decision has
one definition whether the claims arrive on a request or in an envelope.
That also repairs the overview KPIs. AdmitScoped admits a white-label tenant's own
admin, and overview delegates that caller straight through to platform.fleet; against
a SuperAdmin-only gate it was refused, so those tenants read zero products, zero
active and zero drift with the fleet source merely marked degraded. It now sees its
own org's namespaces and nothing else — which is what the board it mirrors would have
shown it. Ident gained OrgAdmin to make that expressible: principal.go is explicit
that platform sudo and admin-of-one's-own-org must never be conflated, and carrying
only the first left a callee unable to tell an org admin from a plain member holding
an org. The bit rides in padding the four text pointers already round up to, so the
envelope does not grow and an older peer reads it as false.
An observer with no k8s client now refuses 503 with the init error instead of
answering an empty fleet. fleetReady tells every HTTP route on this board exactly
that, and "0 workloads, fleet ok" because no kubeconfig resolved is the same lie as
the nil seam, only narrower. A client that resolved and found nothing still answers an
empty list, because that is an observation.
The boundary is now proven rather than asserted. The tests drive a real request
through a real socket with real envelopes: acme's org admin observes tenant-acme and
neither tenant-initech nor the platform tier, an org owning no namespace gets an empty
board rather than a fallback, and an anonymous caller, a plain member, and an
unvalidated X-Org-Id are each refused 403. Dropping the confinement, widening the
gate, or calling observeFleet with the unscoped set each turns one of them red naming
the namespaces that crossed. plugin/admin holds at 982 packages and cmd/cloud at 401
with no apps/ package in its graph; PutApps also stops setting Registry twice.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
price.go, build.go and middleware_billing.go each told the reader that an
unpriced surface fails apps.TestPriceDeclared before it can ship. No such test
existed, and no such package existed either -- apps/ is a directory of packages,
not one. Five comments described enforcement that was never built, which is
worse than admitting there is none: it reads as covered, so nobody looks.
The test walks the 116 plugin composition roots and fails on any cloud.Plugin
that answers nothing about what its surface costs. Free and Metered are both
answers; only silence fails. It is a test rather than a runtime 402 because an
unpriced route is an unfinished composition root, not a customer's problem --
gate the deploy, never the request.
Negative-controlled: dropping Price from plugin/ask turns it red, restoring it
turns it green. Guarded against vacuity too -- the first draft matched
cloud.Plugin{} and the real source writes []cloud.Plugin{{}}, so it silently
checked nothing until the count assertion caught it.
MountSpec was a compound naming a struct after the mechanism that consumes it.
The thing it describes is one of the plugins the host loads: name, price, mount.
The directory is plugin/, the framework is the zip plugin framework, and every
doc comment already called them plugins in prose. So: Plugin.
Not App, which was the obvious first choice and is wrong twice over -- package
cloud already declares an App in payloads.go, and the struct itself carries an
App field for a subsystem that gates the whole binary. Either collision alone
would have made the name ambiguous at every use site.
Mechanical: 132 files, plus the parameter and loop variables that carried the
old noun (specs, spec, sp) to the noun they actually hold.
payloads.go states the rule — "one file, imported by BOTH ends of every method it
describes, so the two halves of a payload cannot drift… nothing is JSON" — and the
seven methods added for the app split all shipped JSON. Two ways to write a wire
contract is one too many, so they move onto the file that already had the answer.
Two of them were duplicates outright: finance.balance already had a codec, and a
scalar reply already had PutI64/I64 in rpc.go. Writing a second PutMoneyReq and a
third PutCents is exactly the drift the one-file rule exists to prevent, and it
only surfaced because the compiler refused the redeclaration.
MONEY IS NOT AN INT. hanzoai/money.Amount is a decimal.Decimal plus a Currency
because a minor-unit integer cannot represent everything this platform books —
HUSD carries 18 decimals, so "cents" is not even the smallest unit there. The wire
now carries the exact decimal text and the currency beside it, which Amount.String
and money.ParseAmount round-trip without loss. int64 cents would have been the
third representation of one value and the only lossy one.
Secrets travel as bytes for the same reason: base64 was an encoding tax JSON
imposed, and going native deletes that layer rather than porting it.
Fixed on the way, both mine: availableCents returned ok=true when the PEER was
unreachable, which made the socket the only path and 502'd the split deploy that
reads commerce over its configured URL — a deployment working exactly as designed.
Peer-absent is now "not resolved here" and the caller proxies; a peer that ANSWERS
unparseably still surfaces, because a corrupt reply rendered as zero is a funded
account shown as broke. And marketing's fail-closed test compared a sentinel by
identity when the roster now wraps it in the dial failure that names which store —
errors.Is, which is what it meant.
go vet ./... clean; e2e 16 passed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three more subsystems lost their data to the app split, all the same way: a store
has ONE owner, and a process that is not the owner used to find nothing and say
nothing useful about it.
IDENTITY. marketing read the roster through iamclient.DB(), which is nil outside
iam's process, so every audience resolved to "IAM unavailable" and a campaign
could reach nobody. iam now publishes the roster on the plane. The projection is
four fields — enough to address a person and to match them against a warehouse
cohort — because handing over model.User would put the whole identity record,
credential columns included, on the wire for an audience count.
SECRETS. clients.KMSRPCAt was a stub whose every method returned "not yet wired
(zapc-gen pending)", so pickKMSClient fell through to DisabledKMS and an app in its
own binary silently had no secrets: a mail provider stored through the KMS app read
back as "no email provider configured". KMSPeer is the real client. A ref names its
tenant ("orgs/<org>/…") or names none, and a ref that names one must match the org
the call acts for — that catches an app asking for one tenant while acting for
another. A ref that names none is the deployment's own material and is served to
any peer, because the socket already decided who may ask: 0600 and SO_PEERCRED, so
a caller is one of our own processes. A config-derived "platform org" would not add
a boundary, only a second spelling of one — and Ident.Admin, the real SuperAdmin
predicate, is minted from a validated token and cannot be asserted by a background
call at all.
DURABLE WORK. The engine bound a fixed port and a shared directory, so seven of
eight children lost the bind and ran with no engine — marketing's drip queue among
them, which is how a campaign resolved its audience and then mailed nobody. Port
and store are per process now, named for the app. The cluster-reachable listener is
the opposite case: exactly one process should expose it, so only the app that
serves Tasks does, and CLOUD_TASKS_GATED_PORT moves it — "two instances can never
coexist on one host" is right for a deployment and wrong for a workstation with two
stacks on it.
e2e: 16 passed, 0 failed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
fleetProducts called platform.CurrentFleet(), which resolves a package global.
admin is its own binary and does not mount platform, so that global is nil and
the board rendered an empty estate — while linking client-go, apimachinery and
their applyconfigurations to do it. Roughly 160 packages to print strings it
never received.
The board only ever reads what the operator already reconciled, so that is what
crosses: apps/platform exposes platform.fleet on the internal plane and admin
renders cloud.App. plugin/admin drops 1115 -> 982 packages and apps/platform
leaves its graph entirely.
The k8s that REMAINS is admin's own: apps/admin/infra imports client-go
directly to scan DOKS clusters, which is work admin genuinely does rather than
a number it wanted from somebody else. fleet.SafeRESTConfig stays an import for
the same reason — it is a pure kubeconfig validation gate, not a data read, and
moving a pure function to a socket would be worse in every respect.
Two honesty rules survive the move, on the side that can actually judge them:
an unmounted or unready observer is an EMPTY fleet with a nil error, because
"the operator has not observed yet" is a real state platform knows and admin
does not; a platform that cannot be REACHED is an error, because an unreachable
estate and an empty one must never look alike. Drift crosses as the severity
string the operator computed — the board's only question is whether it is "ok",
so carrying the enum's type would cost the caller the package that declares it
and buy nothing.
TestProductsAndSync_HonestShapes now stands a platform up answering an empty
fleet. Its subject is that an observer with nothing to report renders an empty
registry and fabricates no rows; without a platform to ask it was testing the
unreachable case instead, which is a different fact.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The internal plane had no entry in the seams section, so the rule that replaced
57 mounted package globals and 13 Register* seams was discoverable only by
reading dial.go. Records the calling convention, the one socket scheme both
halves derive, the capability-not-payload tenant rule, and where wire contracts
live — with the measured cost of getting the last one wrong.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The asset plane could create a Pages deployment but not drop the cache in
front of it, so a rebuilt site kept serving the old one and the only recourse
was the Cloudflare dashboard — a step outside the platform for something the
platform otherwise owns end to end.
Purge sits here rather than on /v1/dns or the integrations plane, which is the
same split those planes already draw: /v1/dns owns records, integrations owns
how an org connected, and this plane owns the zone-scoped resources. A cache
purge changes no record and no connection.
It gates on org admin, because purging sends every subsequent request to the
origin — on a site fronting a small origin that is a self-inflicted load spike,
which makes it a change rather than a look. And it requires exactly one
selector: Cloudflare answers 200 to a body with neither, having purged
nothing, which reads as success to a caller who purged nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
6e174404 carried the merged content but only one parent: investigating the
billing failures mid-merge meant a stash and a `git checkout origin/main -- .`,
which cleared MERGE_HEAD, so the commit that followed looked like ordinary
work. History said the twelve commits were unmerged while the tree said they
were. This is the same merge, recorded.
Both conflicts are additions of mine that main does not have yet — the
never-constructed guards in Gate and MeterUsage, and the loop test that proves
reachability rather than presence — so HEAD wins on each and nothing of theirs
is dropped.
Lesson worth keeping: do not go spelunking from inside a conflicted tree. Land
the merge, then investigate on top of it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
admin imported two graphs to read two numbers, and got neither.
apps/commerce came in for ONE call — BalanceCents, in the finance backfill. The
per-org prepaid ledger is single-writer SQLite, so the process that mounts
commerce is the only one that may open it; importing the package bought admin
1213 packages and still no ledger. Now the backfill ASKS, over the socket the
owner already serves, and a missing socket is an error the cutover reports
rather than a phantom zero it would silently carry as "nothing to migrate".
ai/object came in for EnsureCloudUsageTable, and that call could never succeed
here: it execs through a connection opened only inside aimod.Mount, and admin
does not link the ai module. So it always returned "datastore: not connected",
providerBurnCents always took its failure branch, and UsageFunding's query never
ran — two finance boards reporting zeros against a warehouse that was up. The
DDL admin needs is already in apps/datastore, on the connection this binary
actually holds. Same table, same idempotent statements, one import lighter.
plugin/admin 2261 -> 1115 packages
apps/billing 1246 -> 945 (it paid for commerce to name two structs)
THE ORG IS NOT IN THE PAYLOAD. finance.balance now carries (subject, currency)
and nothing else: the tenant rides the capability, so a request cannot ask for
another tenant's books. That is stronger than checking a field, because there is
no field left to forget to check — the codec cannot express the attack.
rpc_tenant_test.go proves it against the real transport: acme's capability with
initech in the payload returns acme's balance, an org-less call is refused 403,
and the decoder's signature is pinned so adding an org field breaks the build.
The BalanceRequest/BalanceReply structs are gone. They were the coupling — the
reason billing imported commerce at all — so the contract moved to
cloud/payloads.go beside git.files, where both ends read it from one definition
and neither imports the other. The reply is a bare scalar and rides PutI64.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
build_on_push.go enqueues against platform's /v1/arcd/enqueue. That surface is
retired — apps/platform/runner.go opens by saying /v1/runner replaces it — so
the reactor was aimed at a door that no longer opens. It already carries the
right credential (PLATFORM_BUILD_CALLBACK_TOKEN, the token /v1/runner checks),
which is what made the mismatch easy to miss.
The path stays dormant: it runs only when CLOUD_NATIVE_CICD_ENABLED is truthy,
and that variable is unset everywhere. This corrects the destination so the
code is true when someone turns it on.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both lines reached the same conclusion about the socket independently — one
server per path, and rpc.Listen is the owner — so the merge keeps that side
whole. The only thing carried over is the loop test, which now proves
REACHABILITY rather than presence: Expose, Listen, Call, and a real reply. A
socket that exists and answers nothing is precisely the failure it guards.
Then two real defects on main, from the ledger moving to its own writer:
NIL CRASHED. Gate and MeterUsage fall through to the peer path when !Enabled,
and the peer path dereferences rm — so a nil ResourceMeter segfaulted instead
of no-opping. TestResourceMeter_NilSafe caught it; it exists for exactly this.
The fix draws a line the code was missing:
rm == nil, or rm.m == nil never constructed. Serve always builds a meter
with a client, so this is a construction defect,
not a deployment shape: allow, never panic.
rm.m != nil, !Enabled() real. No ledger in THIS process because it lives
with commerce. Ask it.
UNCONFIGURED-IS-A-NO-OP was stale, not broken. !Enabled no longer means
"nothing bills" — allowing there would turn every priced act free the moment
an app is split out, silently. The test now asserts what the design says: no
local ledger and no reachable biller is UNKNOWN, and the error names commerce.
I started to add a CLOUD_BILLING_OFF switch for the "nobody bills here" case
and backed it out: resource_billing.go states three lines up that the gate
"never branches on env to bypass billing". An env kill-switch is the exact
thing that rule forbids, and every deployment has its own env's ledger, so the
state it would express does not exist. The rule was right and the code stays
without it.
The two AI tests gate through that same meter, so they were asserting the
absence of a biller rather than what they are named for; they now stand one up.
cloud, deploy, git, projects and admin suites pass. admin/audit and treasury
fail on a missing CLOUD_KMS_MASTER_KEY_REF in this shell — pre-existing on
baseline, unrelated.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
8d8f44f2/019a08af added the peer socket to listenOn so zip's app.Listen would
bind it too. rpc.Listen (Serve, since 56c1b003) already binds that exact path,
so the same socket had two servers — and peerSockets removed the path first to
dodge "address in use", which unlinked the LIVE rpc listener and left zip's
framing answering rpc's callers. The intent was right and load-bearing: what a
server binds must be what a caller resolves. The owner was wrong.
listenOn now returns the machine TCP and the HTTP edge (and, as a plugin, the
host's socket alone). The peer plane is rpc.Listen's, on both legs.
They cannot merge by accident, because they are not the same wire. rpc frames
are zaprpc.Call/Response read straight off the conn: the payload is opaque and
the caller's principal rides Call.Cap. zip's transport terminates fasthttp and
runs the /v1 middleware chain — including SanitizeIdentity, which DELETES every
X-Org-Id / X-User-Id / X-User-IsAdmin header on the way in, by design, because
a client must never assert identity. A peer call arriving there loses exactly
the delegation it was carrying and resolves anonymous. Folding the two is a
designed change (zip.Mount is built for it) that has to move identity into the
capability first — not a boot-order fix.
The tests keep the invariant and change what they ask. TestPeerSocketIsWhatDial
LooksFor now proves it end to end against the real binder: Listen, then Call,
and a real reply. The plugin tests assert the peer socket is ABSENT from
listenOn — the double-bind, pinned so it cannot return.
cloud, deploy, git and projects suites pass.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Fixes the ~17h/25h telemetry outage. Both receivers bound :4317; the second
bind failed, the failure killed the pipeline, and traces died with logs.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Telemetry has been dark ~17h (traces) / ~25h (logs). Root cause, from the
pod's own log:
zap log receiver: start on 0.0.0.0:4317:
bind: address already in use
-> OTLZ ingest collector exited
zapreceiver and zaplogreceiver each open their OWN listener and neither takes
a shared node, so both binding :4317 is a guaranteed collision. The second
bind fails, the failure kills the whole pipeline, and TRACES died with logs —
which is why :4317 appeared in the Service with nothing listening on it.
Logs now bind :4318, which is already exposed on the Service and bound by
nothing. No new port is introduced.
This is a workaround for a library limitation, and the comment says so: the
WIRE already multiplexes by path (/v1/traces, /v1/logs), so when zapreceiver
serves both signals from one node this collapses back to a single endpoint —
and under UDS (HIP-0120 §6) the collision cannot exist at all, because each
receiver owns a socket path.
go build + go vet ./apps/o11y/ exit 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A keyed deployment was running on two keys. The broker is KMS, and KMS was lazy
like any other app, so it came up only when a request reached /v1/kms — two
minutes into a run, after seven children had already asked for credentials, failed
to find it, and fallen back to the deterministic dev key. Whichever of them touched
a shared store first keyed it that way, and the real broker then could not read its
own data: KMS died on "unwrap DEK: message authentication failed", taking its
socket with it and guaranteeing the rest stayed on the dev key. The fleet ran, and
nothing said the keys had diverged.
Two changes, one property: a process either has the deployment's key or it has
none.
A child the host launched no longer falls back. A launch token means a broker was
promised, so a broker that has not answered YET is a race to wait out (pullUntil,
retrying the dial and not a refusal), and one that never answers leaves the child
Unkeyed — the first store open then refuses rather than succeeding against a key
nobody else holds. Inventing a second key is worse than not starting, because it
works.
And the broker starts first, eagerly. It was lazy AND in the middle of manifest
order, which made "KMS is up before anything needs it" a property of list position
rather than a stated dependency. Nothing else can start without it, so nothing else
should be able to start before it.
Result on a full local run: one ROOT and seven LEAF, zero unwrap failures, every
app's socket served. Previously seven DEV and two ROOT with KMS crash-looping.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/types.ParseWindow already resolves ?range/?start/?end here, but window()
cast the interval it returns back to a bare string, so query.go re-derived the
bucket width from that string in a local stepOf — a third copy of a rule that
already lives on types.Interval.Step(), after ai's cloudUsageStep and the one in
types itself. The two copies had already drifted: stepOf matched case-insensitively
while the toStartOf switch beside it compared exactly. Only ParseWindow produces
the value and it produces the lowercase constants, so nothing was reachable through
the gap, but two spellings of one rule is one too many.
The interval now stays typed from ParseWindow through to the series builder, which
asks it for its own step. stepOf is gone. types.Interval admits only Hour or Day,
so the closed set the toStartOf interpolation depends on is a property of the type
rather than of the comment that asserted it.
The wire is unchanged: Interval is a string type, so the response field marshals to
the same JSON it always did — asserted now rather than assumed, because nothing
covered that field before and a client charting the series switches on it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/types.ParseWindow already resolves ?range/?start/?end here, but window()
cast the interval it returns back to a bare string, so query.go re-derived the
bucket width from that string in a local stepOf — a third copy of a rule that
already lives on types.Interval.Step(), after ai's cloudUsageStep and the one in
types itself. The two copies had already drifted: stepOf matched case-insensitively
while the toStartOf switch beside it compared exactly. Only ParseWindow produces
the value and it produces the lowercase constants, so nothing was reachable through
the gap, but two spellings of one rule is one too many.
The interval now stays typed from ParseWindow through to the series builder, which
asks it for its own step. stepOf is gone. types.Interval admits only Hour or Day,
so the closed set the toStartOf interpolation depends on is a property of the type
rather than of the comment that asserted it.
The wire is unchanged: Interval is a string type, so the response field marshals to
the same JSON it always did — asserted now rather than assumed, because nothing
covered that field before and a client charting the series switches on it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Splitting apps into their own binaries took the money plane with it. The prepaid
ledger is per-org SQLite with a single writer, so only the process that mounts
commerce opens it — and every other app, finding no ledger and no commerce URL,
took the branch that means "this deployment does not bill". Reads answered 501 on
funded accounts and, worse, ResourceMeter.Gate ALLOWED: every priced create became
free, silently, without a line of billing code changing.
Neither was a configuration gap to fill in. Widening the enable set so each app
opens the ledger itself would put N writers on one file. The ledger is asked, not
opened: commerce exposes balance, the gate and the debit on the internal plane, and
availableCents / Gate / MeterUsage reach for them when this process has no ledger
of its own. The billed org rides the capability and never the payload — a caller
that could name the org could bill another tenant.
Unreachable is UNKNOWN, never allowed. The gate returns the same errors whichever
side answered, so DenyResource renders one contract: 402 insufficient_balance, 402
spend_cap_exceeded, and anything else fail-closed. The debit stays fire-and-forget
on a background context and logs a failure for reconciliation, because an unbilled
create is a number somebody has to find later.
serve.go no longer binds the app sockets. rpc.Listen already serves each one with
the internal plane's own framing; handing the same paths to zip as well put two
servers on one socket, and the loser's callers got a reply they could not parse —
"promise 0 for 1" on a balance read. One socket, one server, one protocol.
Plugins get 90s to listen (CLOUD_PLUGIN_START) rather than zip's 10s default.
commerce opens stores, migrates and seeds a catalog before it listens, misses 10s
on a cold volume, and the host then reports a prefix 502 for an app that was
merely still booting.
e2e builds the fleet, not just the host — with only the light host present every
app route 503s and the suite fails a product that is fine — waits on /healthz where
the host actually serves it, and waits for each app's socket, which rpc.Listen
creates only once the app is really up.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzo.events measured its two-year TTL from `timestamp`, which the caller sends.
A reduced principal posting a timestamp older than the retention window got a row
that inserted, fanned out to every destination in forward.go, and was already
TTL-eligible when the 200 came back — a write that acknowledges and then vanishes.
The schema had carried a server-stamped `ingested_at DateTime DEFAULT now()` all
along, absent from eventColumns and so unreachable from the wire; retention is
measured from that column now, and nothing on the wire can influence when a row
expires.
The same caller-chosen value led ORDER BY on a table with no PARTITION BY, so
every tenant shared one part range. A single small batch spanning 2019..now
produced a part intersecting every window any tenant would ever query: cheap to
write, whole-table to read, across tenants, and the reader is not the attacker.
Two independent bounds answer that. clampTS gains a past bound of seven days
beside the future bound it already had, which bounds the domain in code and so
takes effect on the live table with no migration. The DDL gains
PARTITION BY (tenant_id, toYYYYMM(timestamp)), the shape every other
tenant-scoped table in this warehouse already uses, which makes tenant isolation
a property of the storage layout rather than of a key range.
The DDL is a definition, not a migration: CREATE TABLE IF NOT EXISTS succeeds
without reading the existing table, so this reaches fresh deployments only.
ClickHouse has no ALTER for a partition key at all, so the live table needs a
swap; that one-time reconcile is recorded in universe LLM.md rather than run from
a boot latch on the ingest path.
Alongside: meet's stale comment claimed /v1/meet/health carries the failure
reason, which it deliberately does not on five public unauthenticated hosts, and
both meet leak tests called t.TempDir() inside the loop so they compared against
a fresh directory and could never fail. They assert the whole reason string now,
with a guard that fatals if the fixture is ever configured such that the test has
nothing to withhold. team's eight inline Extra["org"] reads become token.Org(),
the one spelling that trims.
The mutation harness proves these guards rather than asserting them: ten new
mutants, each reverting one property, all killed. Repointing the harness's eleven
anchor constants from clients/ to apps/ also revives the twenty-six pre-existing
mutants, which had been silently missing every anchor since the tree moved.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The 131 subsystems moved from clients/ to apps/, and the mutation harness names
its anchor files by path. Every one of the eleven constants still pointed at
clients/, so every anchor missed: strict scoring reports ANCHOR-MISS rather than
a kill, and a run that mutates nothing reports nothing killed instead of failing
loudly. The gate was dead, not passing.
Repointing the constants is the whole fix — the paths live in exactly one place,
so no mutant body changed. 36/36 KILLED with them corrected.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`TTL timestamp + INTERVAL 2 YEAR` measured retention from the CALLER's clock. A
reduced principal posting `timestamp < now-2y` got a row that inserted, fanned
out to every destination in forward.go, and was TTL-eligible before the 200
landed. The schema already carried `ingested_at DateTime DEFAULT now()`, absent
from eventColumns and so unreachable from the wire; retention is measured from
it now. Nothing else changes — no clamp is traded off, and batching and retry
are untouched.
The same caller-chosen value led ORDER BY on a table with no PARTITION BY, one
unpartitioned table for every tenant. A MergeTree part is skippable only when
its key range misses the query's, so ONE small batch spanning 2019..now yielded
a part intersecting every window any tenant would ever ask for: O(1) to write,
O(table) to read, cross-tenant, and the reader is not the attacker. Two changes,
because they answer different halves:
- clampTS gains a PAST bound (maxBackdate, 7 days) beside the future one it
already had. This bounds the domain, is code rather than schema, and so
takes effect on the LIVE table with no migration. It is also newly load-
bearing: with retention off `timestamp`, nothing else bounded it at all.
- the DDL gains PARTITION BY (tenant_id, toYYYYMM(timestamp)) — the shape
every other tenant-scoped table in this warehouse already uses, including
this table's own rollups. It makes tenant isolation a property of the
LAYOUT rather than of a key range, matches query.go's predicate exactly,
turns expiry into a partition drop, and makes the part range visible at all
(system.parts.min_time is only populated for a time-based partition key —
measured, every live part reports 1970).
THE DDL IS NOT A MIGRATION and hanzo.events exists on hanzo-k8s with 17,684
rows. `CREATE TABLE IF NOT EXISTS` returns success without reading the existing
table, so this reaches fresh deployments only. Measured against the live server,
not assumed: MODIFY TTL applies as metadata, and PARTITION BY is absent from
ClickHouse's ALTER grammar entirely, so the existing table needs a swap. Both
statements, in order, are in universe LLM.md as one owed step; a self-migrating
boot latch could only ever deliver half of it.
meet: the comment above /v1/meet/health said it carries the reason because
/v1/*/health is an operator surface. It does not, twenty lines below, and
deliberately — the route takes no credential on five public hosts. A stale
comment in the file where the posture WAS the finding invites restoring the leak.
meet tests: t.TempDir() in both leak sets asserted nothing — called inside the
loop it mints a FRESH directory, never the one holding keys.yaml, so the element
could not fail. Replaced by the whole reason string, which is the actual
property, plus a guard that fatals when the fixture is configured so the element
can never go vacuous again. Not cosmetic: the reworded-leak mutant SURVIVES on
45390bfff and is KILLED here.
team: eight inline t.Extra["org"] reads become token.Org(), the one spelling that
trims — which is what token.claim's own comment says it exists for. Three
redundant strings.TrimSpace checks go with them. token's own round-trip test
keeps reading Extra, because the wire is what that package owns.
10 new mutants, 36/36 KILLED. Two rows were wrong first: an anchor that occurred
twice scored AMBIGUOUS, and a tail-only edit orphaned a %s and scored NO-COMPILE.
Neither is a kill.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.832.0/1/2 were tagged on the UNMERGED origin/drop-genai branch, so they
carried higher version numbers than main while missing everything on it —
including the billing_account fix shipped as v1.831.13. go.mod had already been
bumped to v1.832.2, so the next cloud deploy would have silently reverted that
fix: hanzo.chat back to "your balance is $0.00" against a funded org pool, with
a green build and a larger version number the whole way. Nothing would have
looked wrong.
The running pod was still on v1.831.13 — read out of the binary, which is the
only place that answer is trustworthy — so production had not regressed yet.
v1.832.3 is cut from main with drop-genai merged in, so both lineages survive:
internal/iam/payer.go (billing) and internal/gemini (the native Gemini client)
are both present in the tag, verified before pinning.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two defects in the provider-credit board, both of which made an exhausted
promotional grant look alive while real cash was going out the door.
1. THE GRANT WAS A CONSTANT. providerGrantsCents said "do-ai": 2_600_000. On
2026-07-28 that number was wrong in three directions at once: the code said
$26,000, the operator believed $50,000, and DO's own ledger showed
$21,263.65 ever applied. The one nobody could check was the one the dashboard
rendered, and it rendered ~$22k of headroom that did not exist.
So stop declaring it. digitalocean.CreditIssued sums the `Credits` line items
across every invoice — DO knows this exactly, so ask DO. If the missing
tranche is ever applied, the board moves on its own and no one has to
remember. Two things it is careful about:
- it pages invoice items at 500. DO truncates them at 20 by default, and a
truncated page silently UNDER-reports credit, which is the direction that
invents headroom.
- it counts only product=="Credits", never wallet payments. This account's
$3.47 "remaining" is two Apple Pay top-ups totalling $4.00 minus a $0.53
invoice — not promo credit. Folding cash top-ups into a credit figure is
precisely what makes a spent grant look funded.
A failed read leaves the grant at 0 rather than substituting a guess:
headroom is the one quantity that must never be inferred.
2. IT CLASSIFIED ON THE WRONG FACT. HasCredit/IsPaidOnly were derived from
"was a grant ever issued", so do-ai kept reporting credit-funded after the
credit hit zero — while July accrued $1,824.45 of real cash. They now follow
REMAINING credit: an exhausted grant is paid-only from that moment, which is
the fact anything downstream actually needs.
do-ai is also named explicitly in the row set now. It carries a discovered grant
rather than a seeded one, so it would otherwise vanish from the board in a month
with no warehouse burn — and "the credit ran out" is exactly the state worth
showing.
The console module that renders this (hanzoai/console provider-billing, live on
admin.hanzo.ai) needs no change; it was already drawing remaining/burn/runway
and a paid-only badge. It was faithfully drawing a wrong constant.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ship built every app by recursing into `plugin` once per name. That is a loop
make cannot see into: it schedules nothing, so 112 links ran strictly one after
another however many cores were free.
Naming each binary as its own target hands make the graph instead. `make -j8 apps`
builds the fleet in 53s cold and 4.3s warm, against 22.5s warm for the serial
shape — and `make bin/tracker` builds exactly one. The recipe is written once and
`plugin` calls it, so there is still one way to build an app binary; ship is now
cloud plus apps rather than its own loop.
Incrementality is Go's, not make's, which is why these stay .PHONY. The build
cache is content-addressed: an unchanged app relinks from cache in ~40ms and
touching a file changes nothing, so letting make skip on mtime would only add a
second, weaker answer that goes stale when a dependency moves underneath it.
Each link keeps -p=2. N concurrent builds each spawning NPROC compilers is how a
parallel build becomes a thrash.
A misspelled APP is checked against the manifest — the actual list — rather than
against a directory that may exist for an app the manifest never declared.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
main did not compile its tests. The native ZAP-RPC transport landed and dropped
PeerSocket, while serve.go (which binds the socket) and two test files still
named it — so `go vet ./...` failed on dial_test.go and serve_plugin_test.go.
PeerSocket earns its place rather than being restored for the compiler: serve.go
binds a path per mounted app and Call resolves a path per name, and those must be
the same path or an app is up and unreachable. One definition is what makes that
true by construction. Without anything binding it there is no fallback now — the
transport is the socket, so a missing file is the whole failure.
The obsolete test went with the HTTP flavour it was written for; what remains
asserts the invariant that outlived it: what listenOn serves is what a caller
looks for.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The forge carried one commit GitHub did not. Merging rather than rebasing keeps
both lineages intact — neither side is a candidate for rewriting.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
dial.go resolved a co-located app by looking for {DataDir}/run/<app>.sock, and
nothing ever created that file. So Dial's local branch was unreachable in
practice: every peer call fell through to the remote leg and left the process to
reach a sibling on the same disk by going out to the public edge. The inner plane
existed in the caller and nowhere on disk.
listenOn now binds it, one per app this process mounts, at the same path
PeerSocket names — so a server and its callers cannot disagree about where an app
lives. A bare path is ZAP by zip's convention, so these carry ZAP frames: no port
to allocate, nothing on a network interface, and no clash between co-located apps
because each name is its own path.
Everything else keeps serving in parallel over the one route surface — :9653 ZAP
over TCP for cross-host, :8080 HTTP for the edge and for WS and SSE. The socket is
additive; it takes nothing from them.
The host's own socket stays FIRST in the plugin case. zip blocks in waitListening
on that one, and a peer socket bound ahead of it would let the host see "up"
before the address it actually waits on accepts.
A stale socket from a killed process would fail Bind with "address in use", so
each is removed before binding — safe precisely because the path is canonical and
per-app: this process owns that name and is about to serve it.
Proven against a live host: the child spawns, iam.sock and tracker.sock appear
under the data dir, HTTP keeps answering 200 alongside, and a real Dial("iam")
returns JWKS keys over ZAP frames on the socket.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two lines were written against the same intent from one working tree. The
remote line ported deploy back onto Peer.Get — "plain JSON over the socket",
zaphttp + fasthttpadaptor in the tests — reasoning from whatever dial.go it
saw committed. The local line is the directive as given: ZAP frames end to
end, payloads built once in wire layout, no JSON on the internal path at all.
This merge keeps the native line everywhere the two disagree: dial.go and its
tests (frames, capability identity, fault statuses), deploy's tree read
(git.files over the socket, bytes as bytes), git.go (exposeFiles + the
visibility publisher, which the remote line had silently dropped — an
unregistered seam is a no-op and public projects stop getting repos), and
browse.go (the /v1 face keeps its JSON, which is legitimate there: that route
answers browsers, not peers). go.mod tidied — the zaphttp/fasthttp test deps
left with the tests that needed them.
cloud, deploy, git and projects suites all pass on the result.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The previous commit called Peer.Call and encoded ZAP frames in its test. Neither
exists in committed code — both were transient edits from a concurrent refactor
of dial.go that has since been reverted, so HEAD did not build. Written against
the API that is actually in git: Get, and plain JSON over the socket.
The org is now passed explicitly ALONGSIDE the delegated principal. As() carries
the requesting user so git applies its own rules to the same caller; the org
names the tenant delivery acts for, which is what makes the background reconcile
work at all — it has no request to delegate from, and dial.go provides an
explicit org for exactly that case. The earlier version passed an empty org and
would have failed closed on the background path for no reason.
Lesson worth keeping: build against what is committed. A working tree shared
with other lanes is not an API.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
treeSource was correct and unreachable — the reconcile still constructed
gitSource directly. Both now sit behind a `source` interface the engine cannot
tell apart, because where a repo is hosted is not a fact the reconcile loop
should encode.
The choice is DERIVED from the repo value, not read from a mode flag beside it.
A reference carrying a scheme or an SSH host is somewhere else and gets cloned;
a bare `org/repo` is ours and gets read as a tree. A flag would be a second
place to say the same thing, and the failure when the two disagree is not an
error message — it is the wrong desired set, or an empty one, handed to a
reconcile that prunes.
Nothing changes for the current fleet: DEPLOY_ENGINE_REPO still defaults to
https://github.com/hanzoai/universe and still clones. Setting it to
hanzo/universe switches to the native read, with no other config touched.
The principal is delegated through the selection, so a native render reads as
the caller and git scopes the answer itself.
apps/deploy and apps/git green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The tree seam I added one commit ago is the exact pattern dial.go was written to
delete: "the Register* seams that only fire when two apps share a binary". Apps
are separate binaries now, so a registration that never happens leaves the
consumer reading a zero value — silently, which is how every cross-app read was
blank before Dial existed. Delivery would have rendered an EMPTY desired set and
handed it to a pruning reconcile.
So the seam is gone and treeSource calls cloud.Dial("git") like every other
cross-app read. Local resolves to a Unix socket where the kernel proves the peer
and there is no credential to mint or rotate; remote resolves to TLS; this file
never says which, so git can move hosts and nothing here changes.
The two git routes collapse into one. /paths returned the inventory and /files
would have returned the bytes, and delivery always wants both — two routes would
be two answers to one question, plus a request per file. GET
/v1/git/repos/:name/files?ref&glob returns the resolved revision, every selected
path, and its content, in one call.
A file past the read cap comes back Truncated with no content rather than being
dropped, and treeSource refuses the render by name when it sees one. A caller
assembling a desired set has to distinguish "this file is empty" from "this file
was not read"; silently omitting it deletes whatever the missing file declared.
A reply carrying no revision is likewise an error, never "nothing to deploy".
Identity is delegated with As(ctx), so git scopes the answer to the SAME
principal rather than trusting delivery to have scoped it. The background
reconcile loop has no request to delegate from, so that call reaches git
anonymously and git refuses it — failing closed, and exactly the gap dial.go
names: a background caller needs a service credential, not a forwarded header.
Tests drive the real transport — encode a frame, dial the socket, decode the
reply — rather than stubbing a function that would prove none of it.
apps/git and apps/deploy are green. The root package is red on dial_test.go
calling a Get that a concurrent rewrite of dial.go has removed; untouched here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The delivery engine's only source shallow-clones repo@ref with the git CLI.
That needs a writable POSIX workdir on every reconcile and a credential for
every source it reads — and it is the reason a repository cannot simply live on
S3. Neither is inherent to delivery: rendering needs the bytes of some files at
one commit, which is a tree read. treeSource asks for exactly that.
The seam is cloud.ReadTree, the same idiom as sync_seam.go and tracker_seam.go:
apps/git registers a reader at Mount, apps/deploy calls the package func, no
deploy⇄git import and no cycle. Co-mounted, that is a direct call — no
transport at all, which is the cheapest correct answer. Split across processes,
the registrant is a ZAP client against the same git procedures, and no caller
has to know which shape is deployed.
One call returns the pinned revision, the paths, and the bytes. A generator
that lists at `main` and then reads at `main` can straddle a push and assemble
half its inventory from one commit and half from the next; resolving once makes
the read consistent by construction rather than by everyone remembering to pin.
Three prune-safety properties are carried over from the clone path, because a
desired set that is quietly INCOMPLETE is how a reconcile deletes a fleet:
- a manifest listed but too large to read fails the render, naming the file,
rather than silently rendering without it;
- an unregistered reader is an error, so "the git plane is not mounted" can
never be mistaken for "the inventory is empty";
- subdirectories are included, so prune never reads a nested-but-present
object as removed.
What counts as a manifest is now ONE function both sources call. That rule
decides what a prune sees as absent, so the clone path and the tree path
answering it differently would make the same commit mean two desired sets. The
test for it caught this on the first run: the tree path was parsing README.md
and kustomization.yaml because it filtered nowhere.
apps/deploy, apps/git and the root package are green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
HEAD referenced cloud.FilesReq / cloud.File / cloud.PutFiles from
apps/git/files.go with no file defining them: the codec lived only in the
working tree, so 6709b73a did not build alone. This is that file.
payloads.go is the interim contract surface of the internal plane: the wire
shape of every structured method, in ONE file both ends import, so encode and
decode cannot drift apart. A payload is a ZAP message built in wire layout;
repeated records ride as inner frames — the same 4-byte framing the transport
itself uses, applied inside the payload. Nothing is JSON anywhere: file bytes
travel as bytes, which deletes the base64 detour that existed only because
JSON cannot carry binary. zapc replaces this file with generated codecs; the
shapes are deliberately those a schema would produce.
Peer.For(org) completes the caller side: it names the tenant a call acts FOR
when there is no request to delegate — the background reconcile case deploy
already relies on. Applied after As, the explicit tenant wins, which was the
internal-call contract before the plane existed. The subject rides in the
capability, so a request body can never widen the org it is answered for.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The docs still told people to `go install github.com/hanzoai/cloud/cmd/hanzo@latest`
(README twice, LLM.md once) and named `apps/apps.go:Wire()` as the composition root.
Both were deleted at 22f4fc64. An install line for a binary that cannot be built is
worse than no install line.
States the one way plainly: the `hanzo` CLI is the Rust binary in hanzoai/cli, this
module serves /v1 and ships plugins, and the CLI reaches it through a GENERATED
surface — genspec joins hanzoai/openapi with a live route table, genproduct emits
the commands. Because the registry can only refute an authored operation and never
add one, a missing verb is fixed by authoring the route in hanzoai/openapi, not by
hand-writing a command. Retires the `hanzo apps`/`hanzo deploy` names for the
`hanzo platform fleet ...` ones that now exist.
Also records what `cli/` IS, since nothing links it and that invites a delete: zero
importers, no main, no Makefile target, kept solely as the reference for the tools
not yet ported — the GPU worker daemon, `agent publish`s local half, `engine
install`. Rust node join is a one-shot registration, not that daemon. Deleting it
today would destroy the only spec for work that is owed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Second landing — the first (455fd82c) was erased by a force-move of main that
kept the surrounding restructure and dropped this diff, which is exactly the
dropped-fix hazard: the reaper resumed logging 'projects: get hanzo/index:
status 403' every cycle. The fix is unchanged: the machine grant admits only
GET on the org's own projects, IAM frames its single-project read as a POST,
so Get derives from List instead of tripping the wall it authenticated
through. The test fake now answers that POST 403 unconditionally and the
round-trip asserts zero POSTs — with the contract pinned, a third drop fails
CI instead of failing the reaper.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`git.files` — (repo, ref, glob) -> (rev, files) — over native ZAP on the unix
socket. Delivery now renders from it instead of shelling a clone, which is the
last thing on that path needing a writable POSIX workdir and a credential per
source, and the reason a repository could not simply live on object storage.
ONE core answers it. The REST route serves browsers and the CLI, the exposed
method serves delivery, and both call coreFiles — so the answer cannot differ by
who asked. The JSON surface still base64s bytes that are not valid UTF-8 because
JSON cannot carry binary; the internal plane carries them verbatim, which is the
layer going native deletes rather than ports.
The tenant rides as the CAPABILITY, never in the request body, so a body can
never widen the org it is answered for. A request being served delegates its own
principal; the background reconcile has none and names the org it acts for
instead of arriving anonymous and being refused. git re-applies its own rules to
whichever shows up.
Two things I wrote and then deleted, both worth recording:
- A tree_seam.go with RegisterTreeFunc. That is precisely the pattern dial.go
exists to remove: a registration that never fires when apps are separate
binaries, leaving the consumer reading a zero value in silence. For delivery
that is an EMPTY desired set handed to a reconcile that prunes.
- A second copy of this payload codec. A concurrent lane had already written
payloads.go with the same git.files contract, down to the same note about
base64 being a JSON artifact. Two codecs for one wire format is exactly the
drift a shared payload file prevents, so mine is gone and this uses theirs.
Tests drive the real server — Expose plus Listen on a real socket — so the
capability packing, both frame directions and the payload codec are all
exercised. A stub would have proved none of it, and the codec is where a silent
mistake becomes a wrong desired set. apps/git, apps/deploy and the root package
are green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
As must deliver the edge-minted principal to the callee byte-for-byte, and an
explicit org must override a delegated one — the background-job case where a
peer call inherited someone's identity but acts for a named tenant. Neither
direction had a test; a transport rewrite could silently drop the headers and
every delegated read would fail closed at the callee.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An app calling another app framed plain HTTP over the socket. The socket was
already the transport and already the authorization boundary — credz proves the
peer with SO_PEERCRED — but the bytes on it were HTTP, so the one machine wire
this deployment speaks stopped at the edge of the process.
Now a co-located call frames ZAP (zaphttp.Transport, the same transport the
gateway and ingress speak) over that socket. ZAP ops are zip handlers either
way, so there is still ONE router and one set of typed ops; only the framing
underneath changes, and the caller still never says which. do() builds one
request description and send() frames it — that is the only place the two legs
differ.
The remote leg stays TLS. It is not an inner call: it crosses the public
boundary, where TLS is the requirement and zaphttp carries no TLS. Inside the
deployment ZAP over a socket, outside it HTTPS — a real boundary rather than two
spellings of one hop.
A ZAP listener takes a PATH, not a port: nothing to allocate, no clash between
co-located apps, nothing on a network interface. PeerSocket names that path and
is the SAME one Dial looks for, so a server and its callers cannot disagree
about where an app lives. It goes alongside an app's existing HTTP address —
zip serves every address in parallel over one route surface, so ZAP, HTTPS, WS
and SSE all keep working.
The test serves the peer over real ZAP rather than a stub, so it exercises the
framing: pointing the same socket at an http.Server makes the client time out,
which is the proof the wire actually changed.
LLM.md's cek section said the per-principal binding was not done. It shipped —
Open takes the principal, Rebind carries a store to its owner, and the doc now
says so instead of the opposite.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Same-machine, same-binary services were paying TCP to talk to themselves.
rpc.go carries the internal plane over a unix socket — net.Listen("unix")
with a 200ms dial probe — and dial.go, serve.go and reserve.go move the
callers onto it.
Written on the box and never committed: rpc.go and native/ were untracked,
the rest uncommitted, on one disk with no remote copy. go build ./... is
green with the full set.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The previous commit called Peer.Call and encoded ZAP frames in its test. Neither
exists in committed code — both were transient edits from a concurrent refactor
of dial.go that has since been reverted, so HEAD did not build. Written against
the API that is actually in git: Get, and plain JSON over the socket.
The org is now passed explicitly ALONGSIDE the delegated principal. As() carries
the requesting user so git applies its own rules to the same caller; the org
names the tenant delivery acts for, which is what makes the background reconcile
work at all — it has no request to delegate from, and dial.go provides an
explicit org for exactly that case. The earlier version passed an empty org and
would have failed closed on the background path for no reason.
Lesson worth keeping: build against what is committed. A working tree shared
with other lanes is not an API.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Binding the owner into the derivation left every store already on a volume keyed the
old way, so a tenant store cannot be opened by the tenant that owns it until its
sidecar is rewrapped. Rebind is that move, and RebindOrgs is the walk an operator runs
over {DataDir}/orgs.
It is an OPERATION, not a fallback inside Open. A second derivation tried on failure
would mean every open silently accepts two answers forever — which is exactly what made
the old binding unenforceable. Running this once is a migration; leaving it in the open
path would be permanent ambiguity.
Only the sidecar changes. The DEK and fileID are read out under the source principal and
written straight back under the target, so no database page is rewritten and the file is
never opened — safe on a store too large to copy, and a failure cannot corrupt data. The
write is temp+rename, so an interrupted rebind leaves one whole sidecar or the other.
Already-bound is not an error. A walk over a live volume meets stores created after the
change, so those report ErrNotBound and are counted skipped: the walk converges on
"every store bound" rather than pretending to be a transaction. A sidecar that unwraps
under NEITHER principal is a real failure and is reported as one — an operator must not
read corruption as success.
Stores are found by SIDECAR, not by *.db: on a pure-Go build the codec envelope keys the
file out of band and the database may not exist at that path, so globbing *.db would
silently skip exactly those deployments.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
treeSource was correct and unreachable — the reconcile still constructed
gitSource directly. Both now sit behind a `source` interface the engine cannot
tell apart, because where a repo is hosted is not a fact the reconcile loop
should encode.
The choice is DERIVED from the repo value, not read from a mode flag beside it.
A reference carrying a scheme or an SSH host is somewhere else and gets cloned;
a bare `org/repo` is ours and gets read as a tree. A flag would be a second
place to say the same thing, and the failure when the two disagree is not an
error message — it is the wrong desired set, or an empty one, handed to a
reconcile that prunes.
Nothing changes for the current fleet: DEPLOY_ENGINE_REPO still defaults to
https://github.com/hanzoai/universe and still clones. Setting it to
hanzo/universe switches to the native read, with no other config touched.
The principal is delegated through the selection, so a native render reads as
the caller and git scopes the answer itself.
apps/deploy and apps/git green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The tree seam I added one commit ago is the exact pattern dial.go was written to
delete: "the Register* seams that only fire when two apps share a binary". Apps
are separate binaries now, so a registration that never happens leaves the
consumer reading a zero value — silently, which is how every cross-app read was
blank before Dial existed. Delivery would have rendered an EMPTY desired set and
handed it to a pruning reconcile.
So the seam is gone and treeSource calls cloud.Dial("git") like every other
cross-app read. Local resolves to a Unix socket where the kernel proves the peer
and there is no credential to mint or rotate; remote resolves to TLS; this file
never says which, so git can move hosts and nothing here changes.
The two git routes collapse into one. /paths returned the inventory and /files
would have returned the bytes, and delivery always wants both — two routes would
be two answers to one question, plus a request per file. GET
/v1/git/repos/:name/files?ref&glob returns the resolved revision, every selected
path, and its content, in one call.
A file past the read cap comes back Truncated with no content rather than being
dropped, and treeSource refuses the render by name when it sees one. A caller
assembling a desired set has to distinguish "this file is empty" from "this file
was not read"; silently omitting it deletes whatever the missing file declared.
A reply carrying no revision is likewise an error, never "nothing to deploy".
Identity is delegated with As(ctx), so git scopes the answer to the SAME
principal rather than trusting delivery to have scoped it. The background
reconcile loop has no request to delegate from, so that call reaches git
anonymously and git refuses it — failing closed, and exactly the gap dial.go
names: a background caller needs a service credential, not a forwarded header.
Tests drive the real transport — encode a frame, dial the socket, decode the
reply — rather than stubbing a function that would prove none of it.
apps/git and apps/deploy are green. The root package is red on dial_test.go
calling a Get that a concurrent rewrite of dial.go has removed; untouched here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Restructure to the canonical layout: cmd/host → cmd/cloud (the host IS the one
real binary; name it cloud), and every other cmd/<app> → plugin/<app>. `ls cmd/`
is `cloud/` alone; `ls plugin/` is the 116 per-app + tool dirs. gen-app-cmds
scaffolds into plugin/<app> and scans plugin/ for the bijection; the
Dockerfile / Makefile / mk / hanzo.yml / weave / controlplane-containment gate all
read the new paths. go build ./cmd/cloud links ~399 pkgs and zero subsystems.
credz KMS-key leak (#51 follow-up): zip builds each child's env as
append(os.Environ(), Plugin.Env...), so a host that keeps CLOUD_KMS_MASTER_KEY_REF
hands the root key to EVERY child — the Root posture credz exists to prevent, and
now the default entrypoint. cmd/cloud (stdlib credz/launch only — importing credz
would drag cek→sqlite and re-fatten the host) mints the launch secret, scrubs the
root key from its OWN environment, stamps each child a scoped CREDZ_TOKEN, and
re-injects the root key onto the kms broker child's Env ALONE. Every generic child
comes up with a token and no key and must ask the broker. Pinned by
cmd/cloud/main_test.go and proven by a live dns spawn.
helm flag forwarding: cmd/cloud accepts --brand/--domain/--data-dir/--iam-issuer
(the args the chart passes the entrypoint) and republishes each non-empty one as
its CLOUD_* env, which the per-app children read; an empty flag never clobbers a
value already pinned in the environment.
console at "/": nothing served the host root once mountConsole moved into the
per-app cloud.Serve. Extract the console into a light webui leaf (stdlib + embed +
a new strings-only brand leaf, both aliased back into package cloud so no call site
changes) so cmd/cloud — the front door — owns "/" and serves the white-labelled
SPA. The host stays ~399 packages and imports zero subsystems.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The delivery engine's only source shallow-clones repo@ref with the git CLI.
That needs a writable POSIX workdir on every reconcile and a credential for
every source it reads — and it is the reason a repository cannot simply live on
S3. Neither is inherent to delivery: rendering needs the bytes of some files at
one commit, which is a tree read. treeSource asks for exactly that.
The seam is cloud.ReadTree, the same idiom as sync_seam.go and tracker_seam.go:
apps/git registers a reader at Mount, apps/deploy calls the package func, no
deploy⇄git import and no cycle. Co-mounted, that is a direct call — no
transport at all, which is the cheapest correct answer. Split across processes,
the registrant is a ZAP client against the same git procedures, and no caller
has to know which shape is deployed.
One call returns the pinned revision, the paths, and the bytes. A generator
that lists at `main` and then reads at `main` can straddle a push and assemble
half its inventory from one commit and half from the next; resolving once makes
the read consistent by construction rather than by everyone remembering to pin.
Three prune-safety properties are carried over from the clone path, because a
desired set that is quietly INCOMPLETE is how a reconcile deletes a fleet:
- a manifest listed but too large to read fails the render, naming the file,
rather than silently rendering without it;
- an unregistered reader is an error, so "the git plane is not mounted" can
never be mistaken for "the inventory is empty";
- subdirectories are included, so prune never reads a nested-but-present
object as removed.
What counts as a manifest is now ONE function both sources call. That rule
decides what a prune sees as absent, so the clone path and the tree path
answering it differently would make the same commit mean two desired sets. The
test for it caught this on the first run: the tree path was parsing README.md
and kustomization.yaml because it filtered nowhere.
apps/deploy, apps/git and the root package are green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
First real call site for cloud.Dial. The reserve figure now comes from
GET /v1/admin/treasury — over the socket when treasury is co-located, over the
network when it is not — instead of from an import that could never work.
It could never work because admin is its own binary: treasury.ReserveCents
resolved against treasury's `mounted` global, which is nil there, so the board
printed 0 and the comment above it said "0, which is the truth: there is no
reserve on this deployment". That was not the truth. It was "we never asked".
Three things change:
- the number is real
- a failure is an ERROR, not a zero, so the board can say "could not reach
the treasury" rather than "you have no money" — a distinction the import
could not express, because a missing subsystem and an empty fund produced
identical output
- it joins the existing freshness rail (core.SrcOf), like every other remote
read admin already does
Peer.As(c) carries the principal core.Admit already validated, and treasury
re-checks IsAdmin on its own route: delegation, not escalation. Sound over the
socket because the kernel proves the peer is our own process (credz, 0600). NOT
sufficient over the network — the gateway mints identity from a JWT and ignores
inbound identity headers, so a remote peer call fails closed. That is the safe
direction and it is written down in dial.go as the remaining gap.
MEASUREMENT, HONESTLY: cmd/admin is still 2258 packages. Cutting one edge
changed nothing, because commerce, finance, platform, flags and admission each
pull treasury in anyway. The earlier "691 packages for one int64" framing was
wrong — 691 is treasury's closure SIZE, not its marginal cost to admin. The
graph only shrinks when ALL of admin's app imports go, and the remaining work
is now measured rather than guessed:
admin/*.go -> admission, commerce, flags, platform
admin/commerce-> commerce admin/finance -> commerce, finance
admin/core -> finance admin/infra -> fleet
32 symbols across 6 apps. commerce.Client/New/URL/Forward and finance.Client
are already client-shaped, so several are a type swap rather than a rewrite.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A delivery generator's whole question is "which files under this commit are the
inventory" — charts/app/values/*/*.yaml — and the answer is a LIST OF PATHS,
not a packfile. GET /v1/git/repos/:name/paths?ref&glob answers it in one call,
and the shared /zap plane makes it a ZAP procedure for free, so CD reaching it
needs no credential, no clone, and no second transport.
Serving delivery as a tree read is what lets a repository sit on S3. There is
no pack negotiation and no working copy, so nothing on this path needs POSIX —
and it does not re-open the pure-Go transport that buffered a whole pack in RAM
(a 3 GB clone costing 3 GB, which OOM-killed the pod).
MatchPaths is a FUNCTION over the Repository interface, not a method on it.
Every backend gets globbing the moment it can list a directory, a new backend
implements nothing extra, and the traversal policy lives once instead of once
per backend.
Matching is segment by segment, so `*` never crosses a `/`: values/*/*.yaml
selects values/hanzo/www.yaml and not values/a/b/c.yaml. That is load-bearing
rather than pedantic — a matcher that crossed a separator would silently widen
a fleet instead of failing. `**` spans whole segments, and as the final segment
takes everything beneath. Walking is prefix-pruned, so a specific glob reads a
handful of trees rather than the whole commit, and both the result count and
the directories descended are bounded.
The response carries the resolved revision alongside the paths. A generator
that lists at `main` and then reads files at `main` can straddle a push and
build from two different commits; pinning the rev makes the whole read
consistent.
A glob that matches nothing is an empty list, not a 404 — a generator pointed
at a path that does not exist yet must see "no services", not a failure it
retries forever.
Tests seed the real fleet shape and assert the separator boundary, both `**`
forms, the empty case, files-not-directories, the missing-glob 400, and cross-
org isolation. Full apps/git suite green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/flags decided flags by linking a Rust staticlib over cgo. That put a
second language and a second CI image in the release path for one pure
function — evaluate(definitions, context) — and made it a build-time
dependency of everything downstream: the image pulled a prebuilt
libhanzo_flags.a from ghcr.io/hanzoai/cloud-flags, staged it at the exact
${SRCDIR}-relative path the cgo directive named, and CI could not run
`go build ./...` at all because linking any cmd/ main needed that archive.
It also meant a !cgo build carried no evaluator, so the whole app had a
second, degraded personality that answered 503 and fell back to env values.
The evaluator now lives in github.com/hanzoai/flags/go — pure Go, zero
dependencies, importable by anyone. Its parity with the Rust implementation
is not assumed: 621 recorded cases covering every property operator, eight
rollout percentages against forty identities each, multivariate bucketing,
group aggregation, and the numeric and escaping edges are frozen in that
repo and asserted on every run.
So engineAvailable is gone rather than pinned true — the engine cannot be
absent now, and three tests that used to skip themselves when it was
(TestStoreBackedSwitchOverridesEnv, TestIntSwitchRidesThePayload,
TestProjectEvaluationRolloutAndVariants) now run.
Also removes what only existed to serve the archive: the FLAGS_IMAGE arg,
the flagslib stage and its COPY, the cloud-flags images: lane in hanzo.yml,
the native-flags cargo test step, `make native`, e2e's cargo bootstrap, and
libgcc — added in f49a6ac2 for the _Unwind_* symbols the Rust archive
referenced, and needed by nothing else in the runtime image.
The image still builds plugins with CGO_ENABLED=1 for SQLCipher; that is a
separate concern and is untouched. What changed is that the flags path no
longer needs it: CGO_ENABLED=0 go build ./cmd/flags now links clean.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
RED proved KMS org isolation holds and deliberately KEPT one capability:
a human SuperAdmin, holding a signed membership of the reserved admin
org, may switch into another tenant via X-Org-Id and read it. That is
platform sudo — unforgeable, human-only, machine-denied, and how every
other subsystem in the fleet already works.
The capability was never the problem. It left no trace.
A cross-org secret read is a plain 200 GET on a plain tenant route, so
isSecurityRelevant — mutations, /v1/admin/*, denials — classified the
single highest-value read in the fleet as request-log noise. And where
the trail DID fire (a cross-org write is a mutation), actorFromCtx
recorded c.Org(), the EFFECTIVE org, so an admin acting inside lux was
recorded as though they were lux. The impersonation fact was destroyed
at the moment of recording.
Two changes, both in the ONE place that already owns this decision:
- isSecurityRelevant gains a cross-org clause: an action taken inside
a tenant that is not the actor's own is audited, read or write,
success or failure.
- Actor gains Home, the actor's own org, recorded ONLY when it differs
from Org. A non-empty home therefore MEANS impersonation, and the
record finally names actor, home org, target org, path, and outcome.
Deliberately not scoped to secrets or to KMS: impersonation is a
property of the request, not of the route. KMS keeps knowing nothing
about audit and audit keeps knowing nothing about routes, so every
subsystem gains the coverage at once instead of KMS alone.
Home is persisted as its own column, not just a struct field. Verify
rehydrates each record from its columns and recomputes the hash, so a
hashed-but-unpersisted field would read back empty and report every
impersonation record as TAMPERED — a false alarm on the one control that
must never cry wolf. omitempty keeps existing records byte-identical
under canonicalization, so their stored hashes still verify, and the
column is added by an idempotent ALTER for trails that predate it.
Filter.Impersonated answers the question the control exists for: every
time an admin acted inside a tenant that was not their own.
blue_crossorg_audit_test pins all four properties — the capability still
works and is now recorded, the record carries the path and never the
value, a non-admin cross-org attempt is still refused and cannot
manufacture an impersonation record, and an ordinary same-org read stays
unaudited so the trail does not become a request log.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Apps are separate binaries, so a Go import cannot reach another app's data.
treasury.ReserveCents() compiles inside apps/admin and returns zero, because
treasury never mounts there — every cross-app read was silently blank, and
each one cost admin a whole dependency graph to be blank.
cloud.Dial(app) replaces all of it: the imports, the 57 `mounted` package
globals, and the 13 Register* seams that only fire when two apps share a
binary.
p := cloud.Dial("treasury")
p.Get(ctx, org, "/v1/treasury/reserve", &out)
LOCAL IS A SOCKET, REMOTE IS TLS, THE CALLER NEVER SAYS WHICH. Dial resolves
by whether {CLOUD_RUN_DIR}/<app>.sock exists, so a plugin can move hosts and
no call site changes. Resolution is per-Dial, not per-boot, so an app that
starts later is picked up without a restart.
The socket wins locally because the kernel proves who is calling — credz
already authenticates peers with SO_PEERCRED rather than a token, and 0600 is
the whole authorization boundary. Reaching a process on the same disk over TLS
would mean minting a credential, rotating it, terminating a handshake and
discovering a port, to cross a boundary the kernel enforces for free. The
socket is also not reachable from the network at all.
Plain HTTP over both, deliberately: ZAP ops are zip handlers and already speak
request/response, so swapping only net.Conn keeps ONE protocol, one router and
one set of typed ops. A second wire format for local calls would be a second
way to do the same thing.
The caller's org rides as X-Org-Id so the callee scopes the answer itself — a
peer call is never implicitly privileged. Non-2xx is returned verbatim,
because 402 vs 404 vs 503 is "unfunded" vs "no such thing" vs "that app is
down", and collapsing them would make every board lie the way the in-process
reads did.
Tests cover socket preference, network fallback, and the property that makes
this a correctness fix rather than a refactor: an unreachable peer ERRORS
instead of handing back a zero value.
Next: admin's boards call this instead of importing, then the seams and the
mounted globals go.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The names I added last night were compounds that repeated their own package:
communityMirrorURL, ensureGitHubRepo, RegisterCommunityPublisher,
publishCommunity, VisibilityPublic, ghVisibility. The package already says
which package it is, so the prefix was noise on every call site.
cloud.CommunityEvent -> cloud.Visibility
cloud.RegisterCommunityPublisher-> cloud.RegisterPublisher
cloud.OnCommunityPublish -> cloud.Publish
cloud.RegisterReserveReader -> cloud.RegisterReserve
cloud.ReserveCents -> cloud.Reserve
git.ensureGitHubRepo -> ensure
git.communityOrg / RepoName -> owner / name
git.ghDo / ghVisibility -> call / visibility
git.publishCommunity / mirror* -> publish / mirror
projects.VisibilityPublic -> Public
projects.visibilityFor -> resolve
Three collisions decided three names, and each is better for it:
- treasury.Reserve already exists, so its reader stays ReserveCents.
- projects has a `publish` test helper, so the visibility push is `share`
— which is what it does anyway.
- `token` shadows go/token, so the credential is `secret`.
Build clean. apps/git and apps/projects pass. (apps/treasury fails on
baseline too — pre-existing, unrelated.)
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two findings, one of them a live defect.
1. THE CROSS-APP READS ARE ALREADY DEAD. cmd/admin mounts admin.Mount and
nothing else, so every apps/<other> function admin calls resolves against
that package's `mounted` global — which is nil in admin's binary.
treasury.ReserveCents and flags.Board both return their zero value there.
The money board's reserve figure is not slow or stale; it is blank, and has
been since the fleet split into per-app binaries. Admin was linking 691
packages (treasury), 222 (commerce), ~160 (k8s, via platform) and 47
(go-git) to call functions that cannot work.
So this is not a build-size problem that happens to be ugly. The imports
are non-functional AND expensive.
2. The treasury edge is cut here as a first step: cloud.ReserveCents is a
registration seam like RegisterServiceReleaser, admin reads through it, and
the direct import is gone. Behaviour is unchanged (still unavailable in
admin's binary) — this removes the link cost and the false implication that
an import buys you data, nothing more.
It is explicitly NOT the end state. The end state is call-don't-import:
admin reaches /v1/treasury, /v1/commerce, /v1/iam over HTTP/unix socket and
carries a client, not the service packages. That is what makes the number
real again, and it is what collapses admin toward host-core + its own logic.
The `mounted` singleton is the pattern to retire fleet-wide: it lets a
cross-app read compile and then silently return nothing.
3. native/flags is built (cargo build --release). apps/flags cgo-links
libhanzo_flags.a, and without it EIGHT binaries and the apps/admin test
binary failed to LINK — an error that reads like a code fault and is not.
`go build ./...` is now clean, exit 0, no output.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The fold registered the board as Group("/v1/platform/fleet").Get(""), which
composes to the literal "/v1/platform/fleet/" — and that trailing slash is what
the OpenAPI emitter published. Fiber matches both forms, so nothing 404'd and
the bug was invisible from the router's side; the weave gate caught it because
the CONTRACT is not forgiving. Every SDK regenerated from openapi.yaml would
have called a path the manifest prefix does not name. Registered flat, like
every other platform route, so the served path and the published path are the
same string.
Carries the rest of the fold's tail:
- cli/ (the embedded `hanzo apps|deploy` client) still called /v1/paas/apps —
the last in-repo caller of the retired prefix. Repointed at /v1/platform/fleet.
- clients/admin/zipdoc_gen.go regenerated: the products board's lifted prose
named /v1/paas/apps as its source, which propagated into cmd/admin/openapi.json
and openapi.yaml.
- LLM.md's CLI↔control-plane contract updated, and it now states plainly that
/v1/paas does not exist and why: it was a second name for platform, and the
board is a SIBLING of /v1/platform/projects/:p/apps (the platform's own tier
vs a customer's apps), not a copy of it.
- openapi.yaml re-woven from the re-emitted platform + admin subsets.
Gates: build ./... clean; go test -tags sqlite_fts5 . ./manifest/... ./openapi/...
./clients/platform ./clients/admin ./cli — 6/6 ok (incl. TestFleetIsTheWeaveOfItsApps).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cmd/{campaign,integrations,guide}/seams.go arrived in the plugin-host commit
while the clients/ -> apps/ move was in flight, so they still named the old
path. Mechanical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
They were never clients. A "client" is something that CALLS a service; these
are the subsystems the cloud binary mounts and serves. The name said the
opposite of what the code does, and it sat one directory above a composition
root already named `apps` — so the tree read as though `apps/` and `clients/`
were different kinds of thing when one is simply the wiring for the other.
`apps` is a package, so its subsystems nest under it directly:
apps/apps.go package apps — Wire() returns the 118 MountSpecs
apps/git/ package git
apps/projects/ package projects
`apps.Wire()` composing `apps/git` and `apps/projects` needs no second noun.
WHAT DID NOT MOVE. `clients/*.go` (package clients — aihttp, rpc, s3vfs) is
genuinely a client package: HTTP/RPC/VFS handles the subsystems dial OUT with.
It keeps the name, because for those six files the name was always right. Only
the 131 misfiled subdirectories moved, and the import rewrite is scoped to
`hanzoai/cloud/clients/<x>` so the surviving package is untouched.
This is a MOVE, not a rewrite, and deliberately so. mk/plugin.mk already
derives every path from its own location precisely so that "an extracted
apps/<app> + cmd/<app> + mk/ keeps these paths intact" — the build contract was
written for this migration before the directory was renamed to match it. 31
hanzoai/* modules are already extracted and wired as external imports, 16 of
them with thin in-repo adapters that import their own module (verified: zero
duplicates, no forked implementations). This rename puts the remaining 115 in
the directory the extraction contract already names.
Mechanical: git mv per subdir, then `hanzoai/cloud/clients/` ->
`hanzoai/cloud/apps/` across 656 Go files, plus 23 docs/Makefiles/manifests
rewritten only where the path names a real moved app. Builds clean; projects,
git and catalog tests pass.
(cmd/admission and cmd/affiliates fail to LINK here, before and after: they
need native/flags/target/release/libhanzo_flags.a, a Rust artifact never built
in this checkout. Pre-existing and unrelated.)
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The fleet board (/v1/platform/fleet, folded in from /v1/paas) refused the
platform's own owner: GET /v1/paas/apps answered 403 "admin required" to
z@hanzo.ai in production. Verified against api.hanzo.ai with a real token.
Root cause is one layer down, at the identity boundary. IAM does not put a
normal org's adminness in the top-level `isAdmin` claim — that claim is for the
platform's own super-users, and IAM does not mint it for an org admin. It puts
adminness in the SIGNED membership set, as orgs[].role. SanitizeIdentity read
only `isAdmin`, so it minted X-User-IsOrgAdmin for nobody: every org admin was
silently demoted to a plain member, and every org-scoped admin surface
(principal.IsOrgAdmin — the fleet board, GuardScoped panels) refused its own
owner. Production token, verbatim: orgs:[{org:hanzo,role:admin}], no isAdmin.
The bit now derives from the same signed claim that already decides the
effective org — one claim, one parser, two questions (isMember asks the org
side, isOrgAdmin the role side). Keyed on effOrg, not the home org, so it
describes the org the request ACTS in: admin of your home org does not follow
you into an org you merely belong to. Machine principals stay excluded and the
header stays stripped-on-ingress, so this can only ever restate a membership
IAM signed — never widen one.
Reproduced and pinned by TestSanitizeIdentity_OrgAdminFromMembershipRole (5
cases: role-admin mints the bit, is not SuperAdmin, does not survive an org
switch to a member-only org, role-member mints nothing, a client-sent header
still never survives). Reverting the one-line predicate fails the first two.
Also drops a stale "after paas" comment in the Wire() freeze.
Gates: go build ./... clean; go test -tags sqlite_fts5 . ./apps/... ./manifest/...
./openapi/... ./cmd/cloud ./clients/platform — 6/6 ok.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
No `go build` links the fleet anymore. apps/apps.go (Wire, 111 imports), cmd/cloud
(the fused 3040-pkg monolith) and cmd/hanzo (3102) are gone. Max single build is now
2263 (admin), most apps ~600 — none is the fleet union.
- manifest/apps.go is the hand-authored SOURCE OF TRUTH. cmd/gen-app-cmds inverts:
it reads manifest.Apps (a value) and scaffolds/validates the per-app cmd/<app>,
never parsing a deleted Wire(). The frozen mount order moves to manifest/order_test.go.
- manifest.MultiCall + the multi-call rung are gone: a per-app binary is the ONLY
plugin source (on disk beside the host, or its own entry in the S3 index).
credz appOf collapses to the one dedicated-binary spawn shape.
- openapi golden regenerates from the WEAVE of per-app subsets (openapi/weave_test.go
-weave), not the deleted monolith; the woven doc trades the monolith's plugin
catch-alls (/v1/keys,/v1/o11y,/v1/sentry) for the apps' real routes. Gate byte-identical.
- cross-subsystem in-process seams (coding+automation, campaign paid/experiment, guide
signals) relocate from apps/wire_seams.go into the consuming per-app mains.
- Dockerfile: ENTRYPOINT=/host; builds every cmd/<app> into /plugins (CGO=1 + libsqlite3
+ sqlite_fts5, sqlcipher codec) beside the host; no cmd/cloud; modernc gate is now the
per-app union (./cmd/...). Makefile ship=host+plugins, monolith targets deleted.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The red team flagged a [LOW]: dbFor and orgFileExists compared the RAW
pre-slug org to reservedPlatformSlug, while the FILE is keyed by slug. The
sharper bug underneath, which a new test exposes: the per-org handle cache
also keyed on the raw org, so a tenant path "/orgs/_platform/..." and the
deployment facade (both raw "_platform") shared ONE cache slot — whichever
opened first served the other. No data crossed in production (validOrg
rejects "_" upstream and Seal binds the full path as AAD), but the store
layer itself aliased two distinct files.
Fix: route to PlatformDB on the facade BOOLEAN, and key the cache + the
existence check on the file-identity slug (SanitizeOrg for tenants, the
reserved slug for the facade). SanitizeOrg never emits "_platform", so a
tenant slug can never collide with the reserved one — the isolation is now
structural, not merely masked by the upstream gate. A tenant _platform path
reads a plain 404 (no oracle that _platform is special).
Deleted orgFileExists (dead after inlining the slug-keyed cek.Exists, and it
carried the same raw-compare bug). New TestDBFor_TenantCannotSpellReserved-
Partition proves the two stores never share a handle. Full kms suite green,
incl. the red team's isolation vectors.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The org left the URL and became a property of the validated principal, so every
pre-reshape isolation test was asserting a status on a path that no longer names
a tenant. Both tenants now spell one URL, which made `want 403` fire on 200s that
were the caller reading its OWN secret — 11 red tests describing nothing.
Adversarial proof first: red_orgscope_isolation_test.go attacks the boundary on
seven axes (same-name-other-org, forged X-Org-Id, admin traversal, aud
mismatch/absence, case-fold + unsafe-rune folding, write/list/delete, existence
oracle) with two orgs seeded at the IDENTICAL coordinate with distinct
plaintexts. No vector returns a foreign secret. No handler change was needed —
there was no leak, only stale oracles.
The assertions move from status to VALUE, because after the reshape a status
cannot distinguish a refusal from the caller being served its own record. That is
what let the old tests rot silently, and it is the one change that makes them bite
again. Verified by mutation: collapsing the org partition, dropping the
validated-principal gate, honoring an unsigned org selection, and re-granting
SuperAdmin to machine principals each turn these tests red with the leaked
plaintext quoted in the failure.
Cross-org is 404, not 403, and that is stronger: 403 conceded a resource existed
and merely refused it. The org is unspellable now, so the request resolves inside
the caller's own namespace and existence elsewhere is unobservable — the
enumeration oracle closes structurally instead of by handler ordering.
The aud axis is unchanged and not broken: audience was never an access gate
(trust is signature + issuer + expiry). Its tests were stale in ORACLE only —
"did this token get SuperAdmin?" used to be read off a cross-org URL. It is now
read off the claim-bound org-switch, which is what SuperAdmin actually confers.
Admin cross-org read was NOT deleted, it MOVED — from URL traversal to
SanitizeIdentity's org-switch. Pinned with the switched-into tenant's plaintext
asserted, and flagged in-line as a product decision rather than silently restored
or silently dropped.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Probed against the live hanzo-community org and caught a real defect in the
commit before this one. On an org repo:
PATCH {"private": true} -> 422, with an EMPTY errors list
PATCH {"visibility":"private"} -> 200
The two fields look interchangeable and are not. With `private`, CREATE worked
(that endpoint really does take a boolean) and every RETRACTION would have
failed — silently, since the 422 carries no error detail to log usefully. That
is the exact direction that must never fail: a project going private on
hanzo.app would have stayed public on GitHub.
So the mapping now lives in one place (ghVisibility) with the asymmetry
written down, because the next person will also assume these are the same
field. The test asserts the wire body carries `visibility` and explicitly
fails if `private` reappears on the PATCH path.
Verified live end to end: create -> 201 public, flip -> 200 private.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every store derived under the single tag "global" — platform and per-org alike — so a
key knew nothing about whose data it protected. Confidentiality between orgs held (each
file has its own random DEK under its own KEK), but nothing BOUND a file to its tenant:
a {db,.dek} pair carried into another org's directory opened there perfectly well.
cek's own header conceded this as a non-goal. It stops being one.
Open now takes the principal first, because it is the question a caller must answer
rather than one it may forget: cek.Global for a platform store, cek.Org(slug) for a
tenant's. The owner goes into the HKDF info and the GCM AAD, so a store carried across
a tenant boundary fails to unwrap instead of opening, and two orgs can never derive the
same key for the same file id. The path stays out of the derivation, so a store still
survives a move.
There is ONE derivation and no fallback. cek.Global keeps the exact formula every store
on disk was written under — type "global", id = hex(fileID) — which is asserted against
an independently written reference so the platform fleet cannot be silently orphaned;
only tenant stores gain an owner, and those reseal rather than dual-path.
OrgDB is where the owner comes from: orgDBPath already folded the org through
SanitizeOrg, so it returns that slug rather than making callers re-derive it — one
slugging per open. The reserved platform partition is not a tenant and keys as Global.
iam's legacy adoption goes with it. A store is at iam/global.db, full stop; no rename
path, no second name to reason about.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The broker named its peer by reading /proc/<pid>/cmdline through the pid
SO_PEERCRED gave it. SO_PEERCRED is sound; argv is not — execve takes argv
from the caller, so any same-uid process could exec itself as `billing`,
be handed billing's KMS scope *including the root key*, and be logged as a
legitimate grant. The launcher's decision and the child's self-report were
the same bytes at the socket.
Only the launcher knows which app it started as which process, so identity
comes from there now. New stdlib-only leaf credz/launch: the launcher mints
a secret, stamps CREDZ_TOKEN=<app>:<hex hmac-sha256(secret, app)> into that
ONE child's zip.Plugin.Env, and the broker opens it with launch.Open before
gating the result on manifest.Apps. Claim and proof are one variable, so
neither half recombines with another's.
Deleted, not deprecated: peerArgv (both platforms), appOf, enableFlag,
TestAppOf. The broker no longer reads anything the peer chose about itself.
SO_PEERCRED stays for the uid check and the pid in the audit line.
Two spawn sites, both per-plugin and never os.Environ() — which would hand
every child the same token and re-open the hole:
- cloud.PluginSpec: the launcher IS the broker in-process; credz mints and
memoizes the secret (LaunchSecret) and it never leaves the process.
- cmd/host: mints it, stamps every child, hands CREDZ_LAUNCH_SECRET to the
launch.Broker child alone. credz/launch is stdlib-only for this call
site — importing credz would drag cek → sqlite/sqlcipher into a build
whose reason to exist is being small. Host deps 398 → 399.
manifest stays pure; the stamp decorates the zip.Plugin it returns.
Protocol bumped credz/1 → credz/2: the request grew a second line.
Two caveats written where they belong, not in a new doc:
- The token is in the child's environ, same-uid readable via /proc. This
raises the bar from "assert any identity free" to "first steal a live
peer's token"; it is NOT a same-uid boundary. A real one needs the
socket itself as the credential (pre-connected fd as an ExtraFile),
which is a change to zip's spawn contract. (credz/launch, credz.go)
- cmd/host does not yet call credz.Boot, so it still passes the root key
down via os.Environ() and children resolve Root without asking the
broker — scoping bypassed there, not broken. The deployed entrypoint is
/cloud (fused), where the fix is live. (Dockerfile host-mode note)
Tests: pullAs split into launchAs (stamped) and forgeAs (must be refused).
The decisive one is TestArgvDoesNotDecideScope — a child named `billing`
presenting ai's token gets ai's scope, whole, proving argv is not consulted.
TestForgedIdentityIsRefused covers 7 forgeries incl. ai's mac under
billing's name and a token from another launcher. clients/kms bootAs stamps
too (end-to-end over the real sealed store) and asserts the token is
scrubbed from the child's environment after Boot.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
git.hanzo.ai is canonical and GitHub is the mirror — but a mirror nobody can
find is not marketing. A community project now gets an actual repo at
github.com/hanzo-community/<org>-<slug>: a link its author can hand out, star
and be found through.
We hold admin on the community org, so this CREATES the far-side repo instead
of assuming somebody provisioned it. That was load-bearing: mirror_out
force-pushes to a target that must already exist, so registering a mirror
without creating the repo would have failed on every project forever. The
previous commit's env-gated opt-out existed only to avoid shipping that
broken push, and is now deleted — appearing in the community is the opt-OUT
default, so the org name is a staging override, not an on/off switch.
Visibility is honoured on BOTH hosts from one switch. Public ⇒ public, private
⇒ private, in the same call that reconciles the canonical repo. Two choices
worth naming:
- The replica is PATCHed, never deleted, when a project goes private.
Deleting would destroy stars, forks and issue history for what the author
may have meant as a temporary change.
- The mirror registration stays enabled either way, because visibility lives
on the REPO. Deregistering would silently stop replicating, and the day it
went public again GitHub would be stale by however long it was private.
Created with the right visibility rather than created-then-patched, so a
private project's replica is never even briefly public. auto_init stays false:
the first mirror push carries the real history and an initial commit would
collide with it. Description is sent only at create, so an author who edits it
on GitHub keeps their edit. A 422 on create is treated as success — that is
GitHub's "name already exists", i.e. a concurrent publish reached the state we
wanted.
Credentials are the SAME KMS-injected GIT_MIRROR_TOKEN mirror_out already
pushes with, on the Authorization header only. No token ⇒ the whole replica is
a no-op and no mirror is registered, so dev and test run the entire publish
path without a network.
Registration still routes through gitMirrorController.EnsureMirror, the one
idempotent host-allowlisted path the sync engine and /mirror endpoint use —
one outbound target list, no second way to add to it.
Tests drive an httptest stand-in for api.github.com and cover create-on-404,
patch-on-exists, born-private, and unconfigured.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The zipdoc note already records the day api.hanzo.ai served every operation with
no description. The same bug had a second instance in the tool list and outlived
the fix, because a projection that reads the wrong field does not fail — it goes
quiet, and the spec next to it still looks right.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip's mcpTools read WithSummary alone, so every op documented the canonical
way (a doc comment lifted by cmd/zipdoc, no WithSummary) reached a model as an
empty description over a schema whose fields said nothing. The /v1/admin/plugins
surface is exactly that shape: 423, 621, 221 and 594 characters of prose in the
spec, and four nameless tools over /mcp.
zap-proto/zip@v1.17.6 reads the same docFor extraction openapi.go and cli.go
read and builds the tool inputSchema with schemaOfDoc, so the prose and the
per-field help land on the tool list too. No route, spec or count changes —
openapi.yaml is byte-identical and TestOpenAPIYAML passes untouched.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Visibility decided in clients/projects now reaches the source in
clients/git, so "public" means something a stranger can actually clone:
git.hanzo.ai/<org>/<slug>, world-readable exactly when the project is.
The seam is the same single-registrant inversion as RegisterServiceReleaser
and RegisterGitImporter — cloud.CommunityEvent carries one RESOLVED fact
(Listed = public AND not moderated), git subscribes in Mount, and projects
never imports git. The subscriber re-derives nothing, so the visibility rule
lives only in projects.Project.listed.
Fired on every create and every update rather than on detected transitions.
The asymmetry justifies the redundant write: a publish that fails to land
leaves a project un-browsable, which is annoying; a RETRACTION that fails to
land leaves a private or moderated project's source world-readable, which
cannot be taken back. So the subscriber is idempotent and the caller never
tries to be clever about what changed.
git's half reuses provision() — the one way a repo comes into being, the same
call the REST create handler makes — and reconciles an existing repo through
SetPublic. The repo is created on the first event whether or not the project
is public, so a private project still has somewhere for its code to live and
going public later is a flag flip rather than a migration. Name/Description
seed it only at creation: visibility is ours to enforce, an author's own repo
description is not.
Best-effort by design: a binary without the git plane co-resident, or a git
plane that is down, must not fail a publish. The project row is the source of
truth and the next update reconciles.
Tests cover both retraction paths (publisher goes private, platform
moderates), that a lifted moderation restores the publisher's own choice, and
that a create survives an unmounted git plane.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
paas and platform were the same product under two names: a duplicate
definition. clients/paas absorbed into clients/platform (fleet.go,
rollout.go, observer.go, drift.go), the /v1/paas/* prefix retired to
/v1/platform/fleet, the app removed from Wire()/manifest/cmd, and every
importer (admin products board, deploy actions) repointed at the platform
seam. One mount, one prefix, one product.
Two real de-dups the weave gate forced out in the process:
- schema repoView meant TWO shapes — git owns the repo RESOURCE
(id/org/name/…), platform had a git SOURCE pointer (url/branch/provider).
Renamed platform""s to gitSource; git keeps the canonical repoView.
Every generated SDK would otherwise bind whichever it read last.
- two /v1/*/health probes collapsed to one /v1/platform/health that
actually LISTs the operator CRD (Limit 1) instead of nil-checking.
Fixes the live 500: GET /v1/paas/apps returned 500 for a valid org-admin
when the dynamic k8s client was nil (no kubeconfig / unreachable apiserver).
listFleet now calls fleetReady first -> a clean 503, pinned by
TestFleetListWithoutK8sIs503Not500.
Gates green: build ./..., 8/8 gate packages, weave PASS (no collision, no
unrouted), golden byte-match.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Half-done: paas files renamed into platform, but apps.go still has 3 paas
refs, base is 5 commits behind main, health probes not yet merged, the
/v1/paas/apps 500 not yet fixed, consumers not updated. Checkpoint so the
resumable agent (session limit, resets 1:20pm PT) does not lose the rename.
DO NOT merge to main as-is — does not build.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`official` was an admin-gated boolean that restated what `org` already says,
and because it was gated it disagreed: the platform's own 74 demos were
published by a script holding an ordinary org token, so the gate refused them
and the cross-org catalog filed Hanzo's work as somebody else's. A patch had
pinned the badge back on from an embedded 75-slug manifest, which drifted out
of agreement with reality within days of the template rename — two places for
one fact, and the copy was the wrong one.
Authorship is the org that PAYS for a project. The tenancy boundary already
enforces it and no request can forge it, so there is nothing left for a badge
to say. The field, its gate, the manifest and the boot-time backfill are
deleted rather than deprecated.
What decides who appears is visibility, and it is the publisher's:
public (default) | private — one axis, owned by the publisher
hidden — moderation, admin-only, subtractive
Listed iff public AND NOT hidden, enforced in LiveSites' own query — the one
cross-org read — so a consumer that forgets to filter cannot leak anything.
The gate inverts. Publishing is UNGATED, because a community you must be
admitted to does not grow. Going private is the paid feature and rides the
same cloud.ResourceMeter funded-org gate as hosting, agents and functions, so
an unfunded org asking for it gets a 402 rather than being silently published
— quietly making somebody's private project public is the one failure here
that cannot be undone. Moderation is the only admin-gated field and is safe to
be one precisely because it only ever subtracts: the same shape as Apex's
reserved-host denylist, never an allowlist. It leaves Visibility untouched, so
lifting a moderation restores exactly what the publisher asked for.
Tests prove the new rules rather than the old ones: publishing without admin
or funding lands public, an unknown visibility is 400 rather than coerced, a
tenant's hidden:true is ignored while an admin's takes and lifts cleanly, and
private/moderated live sites leave no catalog row. The catalog asserts the
deleted field is gone from the wire, not merely unset.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
-check over ./... disagrees with what `go generate` writes: whole-module
loading extracts differently from single-package loading (which package a
run flags even varies between runs). A gate that can disagree with the
generator it polices is worse than no gate. Iterate the 15 directive
packages and check each exactly as generate produces it. Proven: all 15
clean; the ./... form flagged up to 4 of the same files as stale.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
clients/platform does not compile its tests on main:
vet: clients/platform/openapi_test.go:31:53: post.RequestBody.Content
undefined (type any has no field or method Content)
Operation.RequestBody, Operation.Responses and Components.Schemas are `any` on
purpose — two seams (Register and the typed fold) build different, JSON-identical
shapes, and the document model refuses to privilege one. The test reached through
them as if they were concrete, so it was never compiled against the model it
tests.
This test only exercises the REGISTER seam, because the platform surface declares
its bodies explicitly, so it narrows to Register's concrete types in one place at
the top and every assertion goes through that. A shape it did not build now fails
loudly, naming the type it got, instead of being a compile error in a file nobody
can run.
Found while verifying that /v1/agents/builds shipped: a package whose tests do
not build fails the gate that must pass before a release cuts, so this was
holding the whole trunk's rollout, not just its own package.
Hanzo-Session: 95715740-b8bb-4d96-8a9c-010a600ec9a6
Hanzo-Turn: 14878
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The lifted prose is committed because a bare `go build` cannot regenerate
it; mk/plugin.mk still claimed the opposite ("The file is NOT committed").
State the actual contract, and wire the drift detector the tool already
ships: -check writes nothing and goes red when a lift no longer matches
its source. It caught two stale lifts on its first run — admin and
finance had drifted through the merges — regenerated here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two operations landed between the two, which is itself the argument the
section makes: tag every number with the commit it was taken at and the
command that retakes it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
LLM.md had drifted at every seam the last few merges moved.
Corrected, against the tree:
* PluginSpec takes a Price, positionally, ahead of the variadic prefixes —
a plugin serves from another process, so nothing downstream can see what
the surface costs unless whoever mounts it says so.
* cloud.Global is deleted; MountSpec.App IS the grant. The doc still showed
the wrapper, and still showed Mount taking *zip.App rather than Router.
* "no checked-in spec file" was true and is not: openapi.yaml is a golden of
the live router, and the SDK repos pull it.
* "len(a.ops) == 0, so zip's own generator emits nothing here" — the typed
registry now carries 165 ops across 15 packages.
* "the untracked zipdoc_gen.go" — 15 of them are tracked, and the reason is
that the root build targets still do not regenerate them.
* "the 106 plugins weigh 5.3GB, that floor is what the image is waiting on"
— the multi-call binary answered it. Monolith and plugin are one artifact
under two invocations, `make plugins` is deleted, ship links two things.
Added: the resolution ladder (ADDR/BIN/sibling/multi-call/index) stated once,
where() over manifest.App.Plugin; the S3 plugin lane and its cache-invalidation
contract (fetch caches success forever — publishing cannot push, ReloadTo or
restart); the document pipeline as ONE registry with N projections, including
the Register reflection seam, zipdoc, per-app subsets, Weave's refusal, and the
golden; the typed migration and hanzoai/openapi's authored master, which shrinks
as ops go typed and must not be deleted first; and a section for the artifacts
that go stale silently.
Every count carries the command that re-measures it, and every claim a file:line.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The spec is a golden file the SDK repos pull, and TestOpenAPIYAML guards it
against the live router. It was red: the two agents-builds routes shipped
without a regen, so the published contract described 982 of 984 routes.
make openapi
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three native templates were live only because someone built them by hand, and
the pipeline that is supposed to own them would have broken all three on its
next run. Each failure hides behind a 200, which is why none of them showed up
as a failure.
framework() — the artifact's own filenames name it, so prep.py reports unity /
godot / unreal / static and deploy-templates.sh stops hardcoding "static".
That string is what clients/projects/sites.go turns into COOP/COEP: a Unity or
Godot export served without it loses SharedArrayBuffer and its multithreaded
wasm hangs forever on a page that answers 200. The deploy also PATCHes the
framework, because create is a no-op after the first run and every project born
before this keeps a stale one.
roots() — a page whose own same-origin scripts are not on disk cannot run,
whatever built it. Flutter's SOURCE web/index.html loads flutter_bootstrap.js,
a file that exists only in build/web, and web/ sits SHALLOWER so it outranked
the real build: the next batch would have packed a blank page over six working
Flutter demos. This is the general form of the %PUBLIC_URL% rule already here,
not a third framework special case.
build() — expo joins the static-build set. Its absence is the whole reason
android-expo-nativewind has no demo: prep.py returned before building, roots()
found nothing, and the slug 404s while its hand-built siblings are live. Flutter
gets the lane it needs too, gated on an SDK that Google publishes for Linux x64
only, so on arm64 it skips and the batch logs NO-SITE instead of shipping the
scaffold.
prep_test.py pins all of it — 12 checks, including that a wasm app which is not
a game stays un-isolated and that a CDN script tag is not ours to resolve.
__pycache__ stops being tracked; a committed .pyc conflicts on every run.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The account surface lists your sessions and links the published ones to their
public build page at /builds/:org/:project. It had the project and not the org,
so the browser either guessed the tenant or made a second call to learn a fact
the response it was already holding knew.
The org is the CALLER'S own — every read is org-scoped before a view is built,
so a row can only ever carry the tenant the caller authenticated as. Echoing it
discloses nothing and removes the guess.
Hanzo-Session: 95715740-b8bb-4d96-8a9c-010a600ec9a6
Hanzo-Turn: 14878
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hd-article-category is a category; the article part is carried by the module it
lives in, not by repeating it in the name. Same for hd-canned-response, and for
the ERP set where "entry", "order" and the "sales" qualifier on invoice were all
doing work the prefix already does:
hd-article-category -> hd-category
hd-canned-response -> hd-response
erp-journal-entry -> erp-journal
erp-journal-entry-account-> erp-journal-account
erp-payment-entry -> erp-payment
erp-purchase-order -> erp-purchase (+ -item)
erp-sales-order -> erp-sales (+ -item)
erp-sales-invoice -> erp-invoice (+ -item)
erp-stock-entry -> erp-stock (+ -item)
erp-stock-ledger-entry -> erp-stock-ledger
The child tables keep their parent prefix (erp-sales-item, erp-purchase-item)
because that distinction is real — a line on a sales order is not a line on a
purchase order — and collapsing them to erp-item would collide with the Item
master. erp-gl-entry keeps "entry" because "GL" alone names a ledger, not a row
in it.
Safe to rename outright rather than migrate: modules are installed per-org on
demand (POST /v1/framework/modules/erp/install) and no org has installed either
lane, so no stored document carries the old names. Verified the framework is
live and mounted first — /v1/framework/summary answers 403, not 404.
clients/erp and clients/help both build and test green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Git only treats the LAST paragraph of a commit message as trailers. Our own
global commit-msg hook appends a sign-off with a leading blank line, which starts
a new paragraph — so `git log --format=%(trailers)` and `git interpret-trailers
--parse` both report the sign-off and nothing above it, and a binding the agent
wrote is invisible to git's own tooling through no fault of the agent. Observed
on the two commits below this one: the trailers are in the message, and
interpret-trailers reports only the appended line.
ParseLinks scans the whole body line by line, so the fact is still read. Without
this test that robustness looks like an accident and the obvious "simplification"
to a last-paragraph parser would silently break every binding written under these
hooks. The test commits the exact shape the hook produces and fails if the parser
narrows.
Hanzo-Session: 95715740-b8bb-4d96-8a9c-010a600ec9a6
Hanzo-Turn: 14874
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`hanzo agent publish <project>` reads the harness's own JSONL — the log that
actually happened, never a summary written afterwards — guards it, pairs each
turn with the commit git says it produced, and posts it as the readable build.
A fabricated transcript is worthless the moment someone diffs it against the
commits, so the only thing this ships is the real log.
The guard runs LOCALLY first. The server refuses a leaking turn too and that is
the real boundary, but scanning here means the secret never crosses the wire and
the author is told which turn of their own transcript to fix. Running it on a
real 14,873-turn session is what surfaced the detect false positive fixed in the
previous commit.
--bind is the part that needed the most restraint. Commits made before the
trailer convention existed can only be bound after the fact, and the obvious
implementation — every commit authored inside the session's time span — bound
3,203 commits on the first real run: nearly the whole repository, most of it
other people's work and merges that merely overlapped in time. A clock cannot
tell "this turn produced that commit" from "that commit landed while this turn
was running", and a provenance record that over-claims is worse than none,
because it makes every honest link in the same ref suspect.
So binding now requires causal evidence: the turn must itself have run a commit
(the harness records the tool call, and the arguments are read for exactly that
one fact and never published), and the commit must land within fifteen minutes of
that turn starting — a turn is minutes of work, and an hours-long gap means the
operator walked away. The same session now binds 21 commits instead of 3,203,
and each note says `Hanzo-Bind: time` so a reader can always distinguish a
derived link from one a commit declares about itself in a trailer.
Notes, never rewritten history: adding a trailer to an existing commit changes
its sha and every sha after it, breaking every URL already pointing at them.
Verified on the real run — 21 notes written, HEAD unchanged.
--dry-run now writes nothing at all, git notes included. A preview that mutates
the repository is not a preview.
Hanzo-Session: 95715740-b8bb-4d96-8a9c-010a600ec9a6
Hanzo-Turn: 14873
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Publishing a real 14,873-turn agent session refused on one finding: turn 11245
mentions `--secret=id=GIT_AUTH_TOKEN`, and the generic-assigned-secret rule
reported the token's NAME as the token.
The value character class allowed `=` anywhere, so after matching the keyword
`secret` and its `=` separator the rule kept consuming and swallowed the next
`key=value` pair whole. `id=GIT_AUTH_TOKEN` clears sixteen characters and clears
the entropy gate, so it was reported with full confidence.
This is worse than missing a finding. Referencing a credential by name instead of
pasting its value is precisely the practice the whole secrets posture asks for —
secrets live in KMS, code and transcripts carry names — and a scanner that flags
the safe pattern is a scanner people learn to ignore. The one rule that exists to
catch carelessness was punishing care.
Base64 padding is the only reason `=` ever belongs in a credential, and it only
ever appears at the END, so that is where it is now allowed. Real assigned
secrets, padded or not, still fire; the four shapes of named reference in the
regression test no longer do.
Found by the transcript guard on a real session, which is the argument for
running one detection engine in both places: the transcript lane and the
code-at-rest lane share these rules, so this false positive was live in every
repository scan too.
clients/security gains the TestMain every other store-backed package has — cek
refuses to open security.db without a master key, so the whole package failed at
Mount and this engine change could not otherwise be proved against the suite that
exercises it.
Hanzo-Session: 95715740-b8bb-4d96-8a9c-010a600ec9a6
Hanzo-Turn: 14873
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A visitor can open a product and see what it is, but not how it came to be. The
demos read as finished artifacts with no account of the work, and the one thing
that would make them trustworthy — the actual session that produced them, with
the prompt, the reasoning, and the diff each turn caused — was nowhere. Worse,
the interesting claim ("this was built from template X by an agent") was exactly
the claim nobody could check.
The temptation is a build-log subsystem: a transcripts table, a commits table,
and a mapping between them. That is three new places for the same facts to
disagree. A session is ALREADY an ordered log of turns, and git ALREADY records
which commit came from which turn if you ask it to. So neither is rebuilt here.
Two columns carry the whole feature. `project` says which product a session
built; `published` is the author's decision to let the world read it. "The build
of project P" is then just the sessions tagged P, and GET /v1/agents/builds/:org/
:project can be anonymous precisely because the only rows it can reach are ones
an author published — publishing is the access rule, not a second permission
system. Everything else about a session is untouched, and every session that
already exists stays untagged and unpublished.
The turn⇄commit binding lives in git and only in git: a Hanzo-Session/Hanzo-Turn
trailer on a new commit, or a note under refs/notes/hanzo-provenance for history
that must not be rewritten (a trailer added retroactively changes every
downstream sha and breaks every URL pointing at it). A note body is just trailer
lines, so ONE parser reads both. There is deliberately no commit⇄turn table,
because a table is a second copy of a fact git holds, and a second copy can
disagree with the commits it claims to describe. Every build response carries the
exact `git log` that re-derives it, so no reader has to take our word for it.
ParseLinks is proved against real git output, not a fixture of what I believe git
emits — the fixture I wrote first was wrong about record separators, and only the
round-trip through a real repo caught it.
Secrets are refused, not redacted. Every event body is scanned by the engine the
code-security surface already uses (clients/security/detect, the in-binary port
of hanzoai/guard's redaction concept) BEFORE it is stored, and a hit returns 422
naming the rule, the line, a masked preview and the fingerprint. Silent redaction
would store a turn that reads clean while the secret is still live in whatever
log it was copied from, and the author would never learn to rotate it. Refusing
also gives publishing its safety: a stored transcript has never held a secret, so
making one public later cannot leak one. A transcript that names a KMS key
instead of pasting it passes untouched — the correct pattern is not punished.
The deploy seam clients/projects left open (observer.go, waiting since the
sessions lane landed) is now filled, so a site going live becomes the last turn
of the session that built it and the story ends where the product starts. A site
deployed by a script has no session, so nothing is narrated — an honest silence
rather than an invented build.
clients/projects gains the TestMain every other store-backed package has. Without
it cek refuses to open projects.db and the ENTIRE package fails at Mount, so none
of its 60+ tests could run at all; unrelated to this change, but it was masking
the one suite that had to keep passing next to it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
iam2.db carried a version number that stopped meaning anything when the Casdoor
iam-v1 embed was retired. A "2" whose only job is to not be a "1" is scar tissue, and
a version in a filename is a migration waiting to be mistaken for an identity.
The replacement says which PRINCIPAL PARTITION the file holds rather than repeating
the directory that already names the entity: iam/global.db. That is the same word the
key derivation uses — cek opens it under sqlitedrv.PrincipalGlobal, "the cross-org
global/platform database (certs, providers, the admin org catalog)", which is exactly
what this store contains. Name and key now agree, and the partitions that do not exist
yet have somewhere to go: a per-org or per-user IAM store derives under
PrincipalOrg/PrincipalUser and is named for that, so the split is visible on disk
instead of inferred.
Renaming an encrypted store is safe for a reason rather than by assumption: cek derives
the KEK from a random file id kept INSIDE the sidecar, never from the path, so a store
carried to a new name reopens under the same key. Moving the sidecar alongside the
database is the whole migration — no page rewritten, no key re-derived. adoptLegacyStore
does it once at boot and refuses, rather than guessing, if both names are present.
Presence is answered by cek.Exists, not os.Stat, and that distinction is the bug a live
boot caught: on a pure-Go build the codec envelope keys the file out of band, so the
sidecar can be the only thing on disk. Statting the database alone reported "nothing to
adopt", abandoned the real store and created an empty one beside it — the precise
failure this function exists to prevent.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Found by the tenant-readiness audit: the brand registry pinned zoo's IAMIssuer
to https://zoo.id, a host with no DNS, while the live IAM stamps every zoo
token iss=https://zoolabs.id (verified against both hosts'
/.well-known/openid-configuration; lux.id and pars.id answer 200 and stay).
BrandIssuers() feeds the ONE trusted-issuer set, so on a zoo deployment every
real token failed the issuer check — fail-secure, and completely broken.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.831.13 moves billing_account from the Claims envelope onto the User identity
and routes 17 account.Payer call sites through one (*User).Payer. Before it, the
claim was structurally unreachable from 27 of ai's 28 payer call sites — they
hold a *User, not a *Claims — so every one of them fell through to the shape
rule and billed the caller's personal wallet.
Live effect this fixes: admin hanzo/z was charged against a wallet holding $0.28
while the hanzo org pool sat funded at ~$149,918 and untouched, and hanzo.chat
answered "requires a positive balance. Your current balance is $0.00" to a
funded company. No layer could report it, because each was correctly answering
the question it was asked.
Module-only bump; nothing in cloud changes. ai's internal/iam is an INTERNAL
package, so cloud cannot reference the moved field even in principle — the
upgrade is source-compatible by construction, not by inspection.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Project.Official is the badge that says "Hanzo published this", and
createProject raises it only for a SuperAdmin — `body.Official &&
c.IsAdmin()`. That gate is right and it does not move here: a tenant that
asks for official:true must always get false, or the badge means nothing.
The gate was not the bug. The bug was that the platform's own example
catalogue was published by a SCRIPT holding an ordinary org-admin token,
so the one principal that could raise the badge never ran, and Official
was false on all 74. That is not cosmetic — the cross-org catalog
partitions rows by authorship, so apps we wrote and host were filed as
somebody else's work and the gallery rail labelled 117 first-party rows
"third-party". A directory that guesses authorship in its own DISfavour is
still a directory that guesses.
So stop asking. Which apps are ours is not a request parameter; it is a
fact the platform knows about itself. firstparty.json declares it in the
platform's own source — no request reaches it, no principal can assert it,
and a tenant cannot edit it — and migrate projects that declaration onto
the column on every boot, raise-only. Raise-only matters twice: a project
outside the manifest is untouched, and a badge a real SuperAdmin set by
hand is never revoked by a deploy.
One declaration: hanzoai/examples keys its product identities off the same
slugs and its publish step asserts each one came back official, so the two
halves fail loudly instead of drifting.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This workflow polled github.com every 10 minutes and fast-forwarded the forge
from it. It was written when which side was canonical was still open; it is not
open now, so a cron that reconciles two mains is a second answer to a settled
question. Removing it leaves exactly one way for code to move.
It was already inert by its own admission — the header notes it does nothing
while the GitHub repo is a pull mirror, because the forge overwrites main on its
own timer and rejects the push.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The audit that produced the iam fix also established what cek does NOT do: the KEK
derives from a random per-file id, never from the principal, so PrincipalOrg and
PrincipalUser are unused in cloud and a store file is not bound to the org that owns
it. Confidentiality between orgs holds; portability of a {db,.dek} pair between them
is the gap. Record the derivation, the legacy-unwrap-then-rewrap migration it needs,
and the one store still outside the envelope, so the next person starts from the
finding rather than the search.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cek's package doc says encryption at rest is "a property of the open path" because
every store goes through cek.Open. The identity store did not: it opened through
iamserver.OpenSQLite, which builds its own pool from a plain path, and orm's
SQLiteDBConfig carries no master key — so there was nothing to key it with.
The result on disk was `iam/iam2.db` beginning with the literal `SQLite format 3`
magic, with no wrapped-DEK sidecar: every identity, org membership and credential
record readable from a lifted PV snapshot or an in-cluster volume read. That is
exactly the exposure cek's threat model names, and this store was the counterexample
to its central claim. The credential hashes are argon2id, so a lifted file was never
a password disclosure — but the identity graph and every credential record were in
the clear.
No new crypto, and no second envelope: cek already mints the per-file DEK, wraps it,
migrates an existing plaintext file in place and shreds the plaintext copy once the
encrypted store is proven readable. orm's AdaptSQLDB exists for precisely this shape
and names this case in its own doc — the caller owns the file and its encryption, the
ORM only manages records inside it. So the fix is to stop opening the file twice over
and let the one gate own it.
One connection serves reads and writes, matching every per-org store (OrgDB pins
MaxOpenConns(1)). That is required rather than tidy: the pure-Go codec envelope is
single-writer, so a second pool over the same keyed file was never available.
Rollout note: the first codec-linked boot MIGRATES a live plaintext iam2.db in place
(verified copy, atomic rename, plaintext shredded only after the encrypted store
reads back). A pure-Go build cannot perform that conversion and refuses with the
reason rather than opening plaintext, so a dev with an existing store removes it and
starts fresh.
Tests assert the BYTES, not the call: a store whose first page is the unencrypted
SQLite magic fails, which is what the shipped code produced.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The projected document carried every operation's address and nothing about
payloads, so SDKs generated untyped bodies for the whole platform surface.
The router genuinely cannot derive them — a binding struct is a local
variable inside the handler — but what cannot be derived can be declared.
Register(path, method, req, resp) is that seam: a subsystem states the very
structs its handler binds, once, next to its route table, and the projector
reflects JSON Schema from their json tags into components. The drift-proof
property survives because the registry is one of BODIES, never of routes: a
declaration renders only when the live router carries the route, so the
document still cannot disagree with the router — schemas are additive
metadata on routes that exist. An orphan registration renders nothing; an
unregistered route renders exactly as before. From refuses loudly when two
different Go types claim one component name, the same posture as the
operationId guard, and the success body sits under the honest 2XX range
because the exact status code lives in the handler body.
The platform surface declares its bodies (createAppReq including storageGb,
appView, projectView, setEnvReq, runReq, runView) — the Goa design under
clients/platform/design is a stale parallel contract (no storageGb) and is
deliberately not the source; the handlers' own structs are.
Tests render the projection from the real mounted routes, so a typo'd
registration path fails instead of silently dropping schema; negatives pin
the orphan, the bare route, the duplicate Register, and the name collision.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cargo build in native/flags failed with "multiple workspace roots found in the
same workspace". Cargo walks up from the crate until it finds a workspace, and
above this checkout sits an untracked ~/work/hanzo/Cargo.toml that lists sibling
REPOS (cli, cloud-backend, crypto, guard, platform/pkg/zap) as members — while
cli declares a workspace of its own. Two roots, so cargo refuses.
That file is not ours and is not in any repo, but the crate should not be at the
mercy of whatever directory it is checked out under: clients/flags links its
staticlib directly, so it is always built standalone. An empty [workspace] table
is the documented way to declare that boundary, and it makes the build behave
the same in CI (where no ancestor exists) as on a dev box (where one does).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The production image ships every operation with no description. Measured
on api.hanzo.ai/v1/openapi.json: 1441 operations, ZERO with one.
Cause: zipdoc lifts a typed handler doc comment into zipdoc_gen.go, which
registers it with zip.Describe at init, and the generated file is compiled
INTO the binary. mk/plugin.mk makes that a prerequisite of the per-app
build; this Dockerfile never ran it, so the image was exactly the binary
mk/plugin.mk:45-48 warns about. The SDK repos and the CLI read that
document, so the prose reached none of them.
One step, placed before every build below it. Verified idempotent: running
it on a clean tree changes no tracked file.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Red-reviewed twice; cloud cleared to ship. 17/17 mutations killed.
/v1/event/collect carries the team SPA's analytics wire, which the canonical
decoder silently DROPPED — its keys are distinct_id/timestamp where the
canonical wire has distinctId/time, so identity and time vanished, Type ended up
empty, and admitPublic discarded the whole batch behind a 200
{accepted:0,dropped:N} the SPA's retry loop treats as success. Pinned by a test
of that old behaviour.
clients/meet replaces the team-love pod, minting LiveKit room tokens from the
same keys.yaml the LiveKit server validates against — read with yaml.v3 into
map[string]string, the library and target type LiveKit itself uses, because
sigs.k8s.io/yaml coerces 0123456789 to 1.2345679e+08 and yes to true.
The reduced principal is real: selectWorkspace had the invite role in hand and
was dropping it, so a guest got an owner-shaped token. It now signs extra.role,
token.Privileged() is the one predicate, and a guest writes into its OWN org
through the projection rather than being refused or filed under $public. Its
identity is the SIGNED account, not the body's distinct_id — a reduced principal
does not name the person; the token does.
error_message/error_type/error_stack now MOVE onto the typed Exception instead of
being copied, so a stack frame carrying a credential cannot reach the row or the
destinations fan-out around scrubException.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Test-only; the entire non-test diff is one comment, one Makefile target, and the
mutation harness. Closes three unpinned invariants Red found on v1.801.288, each
shown SURVIVED on a pristine baseline and KILLED on this branch:
- door.anon never stamped its own $source, the metric the sunset rule reads
- the write-path seam's defaults were unguarded: warehouseExec -> no-op meant
every write silently discarded behind a 200 receipt
- scrubProps had no call-site assertion, only direct unit tests
hostcarve_test.go's first-party-pin comment was false in a worse way than
reported: an unconditional panic in liveResolver.ResolveOrg left the suite green
because carveApp never set FirstPartyApex, so the resolver was never called. The
assertion now makes the pinned and unpinned lookups answer different orgs, which
is the only thing that makes the pin observable.
The harness is now scripts/mutate.py + make mutate, strict-scored
(ANCHOR-MISS / NO-COMPILE / AMBIGUOUS / VACUOUS / SURVIVED / KILLED / RED-by-PANIC).
It found two more unproven things on its first run, including a byte-identical
anchor that had been mutated in only one of its two occurrences.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
REVIEW THIS AS `git diff origin/main..HEAD` — 13 files. The merge itself is mechanical;
everything below is the delta on top of merged main.
THE MERGE. blue/ingest-one-door landed the `doors` table, which is where this door
belonged all along: /v1/event/collect is now one row {path, decodeTeam, sourceTeam} and
teamCollect is DELETED, because door.ingest already was that function. Route registration
and the site-host carve both derive from the row, so the door cannot be routed and
un-carved. apps.go picks up main's Price field — meet is cloud.Free like team, whose
product it is part of; placing a call mints one token, it is not a metered API call.
Two of main's tripwires fired, both correctly, and both were updated on purpose rather
than around:
- TestIngestSurfaceIsExactlyTheContract exists to fail when the surface changes.
wantDoors gains the row.
- TestAnonIdentity_RefusedAtEveryDoor posted two canonical-wire literals at every door,
which only worked while every door spoke that wire. The team door takes a bare ARRAY
and answers an object 400 — which satisfies that test's INTENT more strictly, since
nothing is stored either way, but "this kind is not writable anonymously" and "this
body is the wrong shape" are different facts and the test is about the first. It now
probes each door in its OWN wire, the way pageviewFor/commerceFor already did.
N-5 — the reduced lane no longer lets the caller NAME the person. The projection strips
revenue, personId, groupId and the event name, but distinct_id survives because it is the
join key that makes the lane useful. That was designed for an ANONYMOUS caller writing to
$public, where forged attribution is meaningless; aimed at a REAL org the same field
changes meaning, and the team SPA puts an account identifier there — so a guest could
attribute pageviews and errors to a named colleague inside the host org. The fix is not to
strip it but to stop taking it from the caller: admission carries the SIGNED account, and
attribute() stamps it over distinct_id (clearing anonymous_id, which exists to stitch an
anonymous session to a person and has nothing to stitch when the person is known). The two
genuinely anonymous callers are untouched — nobody signed for them, so there is no
identity to substitute. Timestamp is deliberately left alone and the back-dating window is
named in the comment: clamping the past would break the SPA's batching and retry queue.
Same principle in meet, as one change. LiveKit uses `sub` as the participant identity and
EJECTS on a duplicate, so a caller-supplied `_id` let any member kick a colleague out of a
call and impersonate them to the room. The identity is now the signed account; `_id` stays
on the wire because the published bundle sends it, and is ignored rather than checked —
verifying it would need the person<->account mapping from clients/team, whereas the token
already carries an identity the caller cannot choose.
N-3 — the PRODUCER side of the F1 fix had zero coverage, which is the round-1 defect one
layer up: every consumer test mints its own token with a role already set, so all of them
pass whether or not selectWorkspace signs one. TestSelectWorkspaceSignsGuestRole drives the
real RPC with a real invited guest and asserts the token it is handed reports itself as a
guest. Dropping the claim, hardcoding it, and misnaming it now all die.
N-4 — "into its own org" was never tested. The org assertion was on teamAdmission, a pure
function, and the end-to-end proof was a 503 that the absent warehouse produced either
way — so swapping handle's a.org for publicTenant survived, in the lane whose entire
rationale is "not $public". Using main's fakeWarehouse the tenant column is directly
observable, so it now binds to where the row actually lands.
Cleanups Red asked for: the orphaned duplicate "WHY A KEY REFUSES..." paragraph is gone
(it lives on presented()'s doc, attached to the function it describes), and
/v1/meet/health no longer returns State.reason or the api key — that endpoint takes no
credential and answers on five public hosts, and leaking the key-file path there while
deliberately withholding it from the getToken 503 was two postures in one file. ready:false
is the whole dashboard fact; the reason stays in the boot log at ERROR.
17 mutations this round, 17 killed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Adopting the strict scoring immediately paid for itself twice, on this repo's own
mutant table.
A panic is a kill, and it prints no --- FAIL summary because it takes the test
binary down first. The nil-carve-handler mutant lands exactly there: the test's
assertion fires, then the request nil-derefs. Scored on the exit code alone it
read as a build failure; scored on 'non-zero means killed' it read as a clean
assertion kill. It is neither, and now it says so.
An anchor that names two sites is a third silent-pass class, alongside the two
already known. Middleware dispatches the carve from two branches with a
byte-identical line — sites.go:310 for a slug host, :321 for a bound custom
domain — so the single anchor mutated only the first and reported KILLED while
the second stayed unproven. One row per branch, each anchored on the if above it.
Splitting them showed the custom-domain branch is guarded in clients/sites, which
owns host->org resolution, and not in analytics: the analytics test on that host
shape asserted a status code while its name claimed it forced the org. Renamed to
what it proves, pointed at where the org is actually pinned. Not duplicated here
— the argument-to-row tail is the same code for both host shapes and the slug
host already pins it end to end.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The fakeWarehouse seam made the written row readable, which is what these three
were waiting on.
$source is pinned on both of a door's handlers, not just ingest. anon is a
separate handler and stamped its tag independently, so a constant there survived
the whole suite — and $source is the sunset metric: a door is retired when its
volume reaches zero, so an anon lane that lies means deleting /v1/tracker while
site-host callers still beacon it.
The write path's seam defaults are asserted by code pointer. warehouseExec bound
to a no-op discards every INSERT behind a 200 {accepted:N} receipt and nothing
else notices; warehouseReady bound to true removes the gate that would make that
an honest 503. resolveKeyOrg is the same shape on the admission side.
The PII scrub's CALL SITE is pinned end-to-end. Every scrub test called
scrubProps directly, so the boundary was proven correct and proven nowhere in
particular; storing e.Properties raw kept the suite green.
The first-party org pin is now exercised, which the carve tests claimed and did
not do: liveResolver.ResolveOrg was never called by this package at all — an
unconditional panic in it left the suite green. It answers the pinned and
unpinned lookups with different orgs now, so resolving a first-party host by the
unique-slug-across-orgs fallback flips the tenant to a customer who published
the same project name, and says so.
scripts/mutate.py is that evidence made repeatable. Only KILLED counts, and an
anchor that drifted, a mutant that does not typecheck, and a -run that matched
nothing are each a named hard failure rather than a pass.
The /v1/insights/e gloss counted six spellings and said eight; the ingress
matches /e, /v1/e, /batch, /capture and each trailing-slash form.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both packages have typed ops and NO //go:generate zipdoc directive, so
nothing ever lifted their doc comments. The four plugin control-plane
ops — list, enable, disable, reload — shipped with empty descriptions in
the published spec, and every projection that reads it (OpenAPI, MCP,
the CLI) carried that emptiness for exactly the operations that can take
production down.
The prose was already written on the handlers. One directive per package
is the whole fix; the generated files are what the build compiles in.
Also in this merge: the analytics subset re-emitted (main dropped
POST /v1/ingest, so its committed subset claimed a route the monolith no
longer serves — the weave gate caught it) and openapi.yaml regenerated.
Verified: whole repo builds, 8/8 gate packages green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The forge carried four commits GitHub did not (the o11y Alertmanager receiver and the
Dockerfile toolchain floor), so the two halves of main had drifted apart. Merging rather
than rebasing keeps both lineages intact — 92 commits on one side and 4 on the other are
all real history, and neither is a candidate for rewriting.
# Conflicts:
# clients/o11y/o11y.go
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
F1 — the guest/readonly guards were INERT. The only two token.Generate sites mint
extra={org,orgs[,user]} and extra={org}; nothing has ever minted extra.guest or
extra.readonly, because upstream sets those on guest-LINK tokens and this port has no
such path. So the guards read as protection while every guest held an owner-shaped
token, and the tests passed by synthesizing a shape production never produces.
The real reduced principal is the workspace role, which selectWorkspace had in hand
from resolveWorkspace and DROPPED at mint. It now signs it, and token.Privileged is the
ONE predicate that reads it — replacing two string comparisons written twice that also
disagreed about whitespace (F7). Fail-closed: an ABSENT role is not privileged, and
guests are excluded by being outside the allowlist rather than by being named, so a role
added to the invite vocabulary tomorrow starts unprivileged.
A guest is now neither trusted nor exiled. handle grew a third level: a REDUCED
principal writes into its OWN org through the projection (publicIngest with the signed
org as tenant — the lane the site-host carve already uses). Its errors and pageviews
land where its org can read them; revenue, personId, groupId and arbitrary event kinds
do not. Being silently filed under $public would have been the same silent loss this
door exists to prevent.
F2 — presented() did not name the team bearer, so an expired token got 200 with rows
under $public. handle's own doc forbids exactly that. presented() now identifies a team
token STRUCTURALLY (the `account` claim, which an IAM token lacks), so it refuses 403
while a stale or foreign bearer keeps degrading to the anonymous lane — that asymmetry
is about what is decidable, and it is now written down where team.go cites it.
F3 — sigs.k8s.io/yaml was the wrong parser and the test that proved it was DELETED.
Measured: it turns 0123456789 into "1.2345679e+08", yes into "true", 0x1f into "31", and
silently keeps the LAST of a duplicated key. Now gopkg.in/yaml.v3 into map[string]string
— the exact library and target type the LiveKit server uses — so byte-exactness is by
construction. The duplicate-key case is restored, and the six coerced scalars are
asserted verbatim. Deleting a failing test and keeping a comment that claimed the
opposite converted a caught bug into a false assurance; that was the mistake, not the
test.
F3b — a key file is a map because a server may hold several keys. LIVEKIT_API_KEY now
names which one, so a multi-key file is a setting rather than a permanent 503. Ambiguity
with no selector is still refused, and both refusals name the available keys and the env
var to set.
F5/F6 — tests at the level that enforces the property, not one level down.
TestWorkspaceOfRoom pinned the parse in isolation and constrained nothing about how
admits USES it, so prefix-vs-exact-segment and the empty-workspace refusal were free
mutations. Capability had the same hole: the old proof batch was error+navigation, both
in publicKinds, so the anonymous lane produced an identical 503 and deleting the whole
teamTenant clause went unnoticed. The discriminator is a customEvent, whose kind the
projection drops.
Also: /v1/meet/health makes "the office is unconfigured" a probe fact rather than a grep
of a rotated boot log.
16 mutations this round, 16 killed — including all six Red found surviving (M4, M10,
M11, M12, M13, M14). clients/team's 80 failures are pre-existing and byte-identical at
194a77556 with timings stripped.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerce's per-tenant money stores open a concurrent read pool AND a serialized
write pool on the same file, which needs the live libsqlcipher codec — the pure-Go
codec envelope is single-writer and cannot serve that shape. cloud injected its
master key unconditionally, so on a pure-Go build (`make build`, CGO_ENABLED=0)
resolveDEK refused, Embed returned an error, and the whole money plane served a
fail-closed 503: no balance, no gate, no ledger, in every local build and every
`go test`.
Injecting regardless was not a stricter posture, it was the opposite. Mount never
reached transport.SetApp, so every S2S billing read fell through to the network and
DNS-resolved the in-process placeholder — the failure clients/commerce/transport
already documents, where a funded account reads as "Insufficient balance" and DNS
takes the blame.
commerceMasterKey gates on sqlitedrv.CodecLinked(), the same predicate commerce's own
resolveMasterKey and cek.EnsureDevKey use, so one posture decision holds across the
process instead of three that can disagree. A codec-linked build injects and commerce
encrypts (and still fails closed without a key, so production cannot write plaintext
money data); a pure-Go build hands over nothing and commerce opens its documented
zero-config dev store, which is the only encrypted-or-nothing choice such a build has.
The e2e harness now enables commerce and tracker, prices one gated kind, and refuses
to start if the embed failed — a fail-closed money plane must not read as a suite of
mysterious billing failures 200 log lines from the cause.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Red-reviewed (0 critical/high/medium), all four merge gates closed.
Three places independently answered 'what is an ingest door' — the route table,
sites.analyticsPaths, and a path if-chain with a silent fallthrough — and they
had drifted, so the same beacon was admitted on an API host and 405'd on a site
host. One doors declaration now answers it; sites holds zero path literal.
/v1/ingest deleted (zero callers, confirmed by two independent fleet sweeps).
The three named shims STAY — @hanzo/capture is a published npm package and the
python SDK ships /v1/analytics/batch, so retiring them in-repo would not retire
deployed bundles. 19/19 mutants killed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Measured live before this change: a client_credentials BEARER resolves its
principal from the SUBJECT's owner half — the app row's owner, "admin" — so
IAM's projects grant (own-org-only) rightly refused it, and prod's platform
surface 500'd on every non-default project the moment the canonical client
shipped. IAM's Basic path (app()) resolves Principal{App: name, Org: served
org} — exactly the shape the grant admits, verified live: own-org 200,
cross-org 403, write 403.
Basic also deletes the token dance entirely; what must be ONE per burst is
the identity resolution (a KMS read), bounded by a 60s cred cache and pinned
by TestCanonicalProjectsCredResolvedOnce.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Red's re-review: the surface and gate proofs held, but three mutants survived and
the carve could panic. All four are addressed here; the ingest model itself is
unchanged.
F4 — a present-but-nil carve handler was dispatched. The map crosses a package
boundary, so "the key exists" and "there is something to call" are two facts, and
treating the first as the second panics the request instead of serving it as
static. analyticsIngest now requires both, so the carve fails to the serve path,
never open. TestMiddlewareNilHandlerIsNotADoor covers it.
F2 — the site-host lane had no evidence behind it. Nothing observed WHICH org a
lane resolved, because without a warehouse the pipeline stops at the readiness
gate and every case answers 503: filing a customer's beacons under $public,
filing a stranger's payload under a customer's org, and doing it correctly were
byte-identical over HTTP.
So the write path's two datastore calls are now values — warehouseReady and
warehouseExec — on exactly the terms resolveKeyOrg already had in this package:
production is always the one datastore client, and a test substitutes them. That
makes tenant_id and properties.$source readable, which is what these need:
TestSiteHostLaneWritesTheResolvedSiteOrg every door on a live site host writes
under the RESOLVED Site.Org — not $public, not the caller's X-Org-Id.
TestSiteHostLaneNeverConsultsHandle raw X-User-Id/X-Org-Id on a site host
buy nothing. sites runs before the identity boundary, so those headers are
unvalidated there; if door.anon consulted handle they would resolve a
principal and a commerce payload would become a row. Asserted on the ROW, not
the status — with a warehouse present, "admitted" is a 200 too.
TestApiHostAnonymousLaneWritesThePublicTenant the pair: on an API host a
credential-less caller is $public whatever Host it used. One lane must be
$public and the other must not, so collapsing them fails on one side.
F3 — the contract pinned only the path, so a door could be rebound to the other
wire or relabelled with a different $source and stay green. wantDoors is now the
full path -> wire -> source triple (wire compared by code pointer, which is exact
for the package-level decoders doors binds), plus TestEveryDoorStampsItsOwnSource
so the declared tag is checked where it actually lands: the row.
F10 — /v1/insights/e now says where its traffic comes from. Eight PostHog SDK
spellings on insights.hanzo.ai (/e, /batch, /capture and each trailing-slash
form) are replacePath'd onto this one literal by insights-cloud-ingest-rewrite,
so almost nothing calls the path directly. Two consequences that were only
implicit: this door must NEVER be sunset on a $source count, because its callers
do not name it and $source='posthog' would not decay after they all moved; and if
that middleware is dropped or reordered below the catch-all, eight live ingest
paths break here with no change in this repo. Those are API-host requests, so
they reach the router, not the byte-exact site-host carve — the sunset paragraph
is now scoped to the three aliases whose callers do name their paths.
Also corrected two comments that pointed at things which do not exist: the write
core cited ai/object.DatastoreExec (it is clients/datastore), and routes claimed
/v1/ingest/keys mints a pk- (nothing registers that path; POST /v1/iam/keys in
clients/account is the one key-minting surface).
Mutations: 19/19 killed, including one per fix above — dispatch a nil handler,
pass publicTenant instead of the Site org, take the tenant from the caller's
header, consult handle on the site-host lane, rebind a door's wire, relabel a
door's source, drop the $source stamp, and give the credential-less lane a
brand-host tenant.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three places answered it independently — the route table, sites' analyticsPaths
literal, and a path switch inside the host carve — and they had drifted. Measured
on main: POST /v1/tracker and /v1/ingest were routed doors that sites did not
name, so the same anonymous beacon was admitted on an API host and 405'd on a
published-site host. Nothing decided that; two lists just disagreed.
doors (event.go) is now the only answer: a path bound to the WIRE it speaks.
routes registers exactly these paths; installHostCarve hands sites exactly these
paths already bound to their handlers, so sites holds no path literal of its own
and the map it looks a beacon up in IS the dispatch — membership and wire are one
decision. A door added or deleted moves both surfaces at once.
Capability is untouched: handle still decides it once for every door (bearer |
pk- | access key => full; presented-but-unresolvable => 403; nothing => the
anonymous projection). No door gains a capability and no gate is relaxed.
The carve stays byte-exact on the RAW request target, and is deliberately
stricter than the router: c.Path() is unescaped and unnormalized (resolveKey
documents this), so %65vent, a trailing slash or a dot segment misses and is
served as static even where Fiber would still route it. This carve derives a
tenant from a Host, so it admits only the exact strings it was handed.
Deleted POST /v1/ingest — a second door for the CANONICAL wire, which is the
alias shape this list exists to make unrepresentable. @hanzo/event 0.3.0 moved
pk- onto /v1/event and a fleet sweep found no remaining caller in any language,
SDK, ingress rewrite or config. Its dead scaffolding goes with it: the capture
handler, sourceIngest, the one-shot deprecation log (properties.$source already
records origin per row, which is the better sunset signal), and three orphaned
constants left from the retired self-minted pk_ HMAC codec.
/v1/analytics, /v1/analytics/batch and /v1/tracker STAY, as declared doors rather
than hand-written aliases. They still carry traffic: @hanzo/capture (a PUBLISHED
npm package, so retiring it in-repo does not retire deployed bundles) POSTs
/v1/analytics and beacons /v1/tracker from hanzoai-app and admin/apps/operator,
both pointed at api.hanzo.ai, which has a priority-1 catch-all to cloud with no
path carve; and /v1/analytics/batch is a published contract across openapi
analytics_batch, the python SDK and `hanzo analytics batch`.
/v1/insights/e stays for the same reason and is no longer described as a
deprecated shim: it is the PostHog WIRE, reached through the live
insights-cloud-ingest-rewrite on insights.hanzo.ai.
Posture delta, both surfaces:
/v1/ingest routed -> 404 everywhere (dead door removed)
/v1/tracker on a LIVE SITE HOST only: 405 -> the anonymous ingest every
other door already gets there (the drift, corrected)
everything else byte-identical per principal class
Tests quantify over doors instead of restating a path list, so a new door
inherits the whole contract; one hand-written list (wantDoors) anchors the
surface so a silent change fails instead of passing. Every gate has a paired
negative, verified by flipping the gate: 11 mutations, 11 killed. One of them
found a pre-existing hole — the GET-not-hijacked test GET'd a read lens, which
the path lookup refuses on its own, so deleting the carve's POST check left it
green. It now GETs a real door, where the method check is the only thing
refusing it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Red-cleared (ship). Closes the credential door at the backend:
/providers advertises one openid entry, and every session-establishing verb the
account client can send is refused with account:status:Unauthorized 'sign in at
hanzo.id'. Pinned by a tripwire that fails on any outbound issuer call, so a
resurrected handler cannot forge the refusal.
Necessary because HIDE_LOCAL_LOGIN is inert in front v0.7.404 and the SPA still
renders the password form; this is what actually shuts the door.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Red got past the previous guard twice, and both ways matter.
Restore passwordLogin but have its error paths return the door's own refusal —
statusUnauthorized(signInAtIssuer) instead of statusBadCredentials — and the
test passed while the door was open and POSTing credentials to production
hanzo.id. Asserting a reply SHAPE can only ever prove "looks like a refusal",
never "no handler ran", because a handler can produce that shape.
So the test now pins absence behaviorally: IAM_ENDPOINT points at a recording
httptest server and ANY request fails the case. Every handler that walks a
credential must call the issuer; the door never leaves the process. That is the
one thing a resurrected handler cannot forge, and it takes the credential POST
off production — the fixed test answers in 0.01s with zero outbound hops.
Five session-adjacent verbs were missing from the table. `confirm` is the sharp
one: upstream it is email confirmation returning a LoginInfo WITH a token, and a
handler minting a session plus planting the cookie left the whole package green.
Added with createAccessLink, checkJoin, checkAutoJoin and
refreshHanzoAssistantToken, so the tail of the client surface is covered by the
same door rather than by default:.
providerHint keeps its code and loses its false claim. I asserted twice that the
hint federates, having driven /login/oauth/authorize — which honours it. cloud
calls /v1/iam/oauth/authorize, which STRIPS it: the 302 Location is
byte-identical with and without the param, so /auth/google lands on the same SSO
page as /auth/openid. Not a hole, since every path runs IAM's authorize flow,
but it buys nothing today and the fix is param passthrough in IAM, not more code
here. The comment and the test now say that, and the test is scoped to what this
package actually controls: the param we emit and the callback we send.
Red's two new survivors both die:
A' (handler forges the refusal code) -> login reached the issuer ([POST /v1/iam/login])
H (confirm mints a session + cookie) -> caught
go test -race green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The engine and its metadata model now live in their own modules so the Go
CMS/ERP/CRM/Helpdesk apps can build on them without importing cloud:
github.com/hanzoai/doctype v0.1.0 the value layer
github.com/hanzoai/framework v0.1.0 the engine
clients/framework keeps the mounting and nothing else. It injects cloud's
storage policy (cek.Open) into the engine, turns a validated principal into
an engine Caller, binds each operation to its existing route, and maps the
engine's error Code to an HTTP status in one place.
Authorization is not reimplemented here — the engine enforces permissions
in its own operations, so there is one answer rather than two.
alias.go re-exports the engine and value vocabulary, so the eleven app
lanes (knowledge, cms, content, help, erp, guide, apps) compile unchanged;
framework.DocType is an alias of doctype.DocType, the same type, so a lane
can import the engine directly whenever it wants.
Routes, wire shapes, permission semantics and hook behaviour are unchanged
— proven by the existing HTTP suites (http_test, module_test, spacepath)
running green against the mounted engine.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Red found the guard, not the change, was broken — and it was the whole safety
margin, because the deployed front build still renders the password form.
TestPasswordRPCIsRefused was vacuous. mountTeam resolves iamEndpoint to
production hanzo.id, so restoring passwordLogin verbatim left the test GREEN:
the real issuer refused the invented credentials with a 401 carrying no token
and no cookie, satisfying every assertion. The test could not tell "handler
deleted" from "handler present, IAM refused THESE credentials", and it made a
live outbound credential POST to production on each run. My own mutation missed
it because the mutant I wrote — a handler minting an obviously fake token — was
easier to catch than the regression that matters.
The cause is that falling through to UnknownMethod makes absence and refusal
produce the same envelope, and absence is not a thing a test can assert. So the
door now states its answer: every credential verb the account client can send —
login, the OTP and signUp family, join, the password-reset trio, and the two
guest paths — answers account:status:Unauthorized "sign in at hanzo.id".
That fixes three faults at once. Mutant A dies by construction: a restored
handler cannot forge this code, and it now fails with "a handler answered
instead of the door". loginAsGuest was an untested refusal one line from the
door just closed — flipping it to mint a session left the whole package green;
it is in the table now. And an English user typing a correct password saw
literally "Unknown method: login" rather than a true sentence.
UnknownMethod also echoed the caller's method name into the reply, bounded only
by the 16MB gateway body limit; it is truncated, with a test.
statusBadCredentials is deleted — Red measured it at 0% coverage once the
password path went, and its platform:status:AccountNotFound code was exactly
what let Mutant A pass for a refusal.
providerHint STAYS. Red read local iam and found provider_hint unimplemented,
correctly flagging that deployed hanzo.id might differ. It does: driving the
live authorize endpoint with provider_hint=provider-google lands on
accounts.google.com. The mechanism is real, the comment is accurate, and
TestAuthStartProviderHint locks live behavior.
go test -race green. Red's two surviving mutants both die.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A public resource must not be spelled under /v1/iam (the edge routes that
prefix to IAM, so cloud never sees it) and a publishable key resolves at a
different IAM door than a secret one (get-user refuses a pk- by design).
Both cost a working feature the last time they were rediscovered.
/v1/ingest/keys was deleted when the bespoke pk_ HMAC family was, but two
comments still name it as where a publishable key comes from — so a reader
looking for the mint follows a dead pointer. It is POST /v1/keys with
{"type":"publishable"}: the same resource that mints a secret key,
because the type is a field on the key and not a second endpoint.
The env pair this targeted does not exist. Measured in-cluster: Secret
livekit-secrets has NO .data at all — its KMSSecret syncs keys:[] from a path
holding nothing — and the LiveKit server does not use env either; it mounts
Secret livekit-keys as a VOLUME and reads keys.yaml via --key-file. So
secretKeyRef'ing LIVEKIT_API_KEY/LIVEKIT_API_SECRET resolved to empty, and
crypto/hmac signs happily with an empty key. That is worse than a missing key:
it mints well-formed tokens that verify against nothing, so the office would
have been permanently broken in a token-shaped way.
Now clients/meet reads the SAME keys.yaml the LiveKit server validates against.
One representation of that material, so it cannot drift — a scalar copy in env
would have to be kept in step forever, and the day it diverged every token
would mint perfectly and be refused at the media edge.
keys.yaml is a YAML apiKey->apiSecret map, so EXACTLY one entry is required.
Zero is unconfigured. More than one is refused, not resolved: map iteration is
random, so "take the first" picks differently per process start and fails at
the media edge intermittently — the worst available failure shape.
Values are returned BYTE-EXACT. The TrimSpace in the first version was a bug
found by writing the test for it: the api key is LiveKit's `iss` and the secret
IS the HMAC key, so one stripped space mints tokens that verify nowhere.
Consistency with the other reader beats tidiness.
Failure is now LOUD, which is what let the empty Secret nearly ship. Absent or
ambiguous material logs an ERROR at boot naming the file and the Secret, and
every reason is actionable. The caller still gets a bare "the office is not
configured" — that 503 needs no credential, so it must not enumerate our secret
plumbing. And grant() refuses an empty key at the crypto boundary, so removing
the ready() gate still cannot produce an unverifiable token.
optional:true is kept, on the volume now: a non-optional secret volume that is
absent will not start the pod, which would take the whole ~60-subsystem binary
down because the office cannot mint tokens. Fail closed, fail small, fail loud.
11 mutations this round, all killed (36 across the change). No dependency edit:
sigs.k8s.io/yaml was already a direct require.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Following the price declaration to its one remaining exemption found a live
leak. cloud's inference decorator (metered_ai.go) is exempt on exactly one
condition — the empty string:
gate: if org == "" { return nil } // no balance check
record: if org == "" { log.Warn(...); return } // no debit
generateSite built its ChatRequest with neither Org nor BillingOrg, so POST
/v1/sites reserved its flat hosting fee against principal.Ledger(c) and then
spent the model for free, on the SAME request that had already resolved the
payer. The tokens are the expensive half. Nothing reported it: the mitigation
was a Warn line, which is the same silence the price work exists to remove —
free never errors.
Two changes, and they are different properties:
- org and payer are now positional arguments, so the COMPILER requires the
caller to name a ledger; buildSite passes the same principal.Ledger(c) that
gateHosting reserved against, because one request must not bill its two
halves to two accounts.
- naming NOBODY is refused, before the model is called. billedOrg falls back
to Org, so either one suffices; neither is the exempt case, and serving it
is giving inference away. buildSite already 403s without an org, so this
cannot fire for the live route — it is there to keep the exemption
unreachable through this function for whoever calls it next.
Proven by reverting both halves and watching TestGenerateSiteNamesThePayer go
red on all four assertions. The test asserts the billing ADDRESS on the request
the model sees, because that address is the entire input to the gate and debit.
The remaining 12 in-repo ChatCompletion call sites all name an org; this was the
only one that did not. The exemption itself still exists in metered_ai.go —
closing it means internal AI debiting an internal org, which needs that org
funded first or internal inference stops. Specified, not started.
Shape was already covered (one entry, variants as options). What was missing is
the part that was actually wrong on live data: an empty description propagates
into a customer's project list through fork.go, source has to be a repository
because fork.go clones it, and a demo has to be the template's own deploy or the
browse page shows a stranger's product.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
DefaultPrice's table ended in `return 0` — "metering is opt-in per path so a
new route never silently starts billing". The intent was to avoid over-billing.
The effect was the opposite asymmetry: over-billing is loud, under-billing is
silent and permanent, because free never errors. A surface nobody priced was
free for as long as it existed and no signal anywhere reported it.
So price stops being something the gate looks up and becomes part of declaring
the surface. MountSpec.Price sits beside Name in the composition root; its zero
value is Undeclared, and apps.TestPriceDeclared refuses it. A new subsystem
cannot reach main until someone answers what it costs, in the diff that adds
its routes, reviewed by whoever reviews the routes. There is no unpriced route
at runtime to catch, so there is no runtime gate to add and none was added.
Free IS a price: 91 surfaces say cloud.Free, 20 say cloud.Metered (a meter
downstream owns the charge — an edge charge on top double-bills). Both resolve
to 0 cents, so nothing about what any customer is charged changes today;
TestEdgeChargesNothingToday and TestDeletedSelfMeteredListStillPricesAtZero are
the equivalence proof, path by path, for the two prefix lists this deletes.
DELETED, not reconciled: selfMeteredPrefixes, and the /v1/agent special case
that shadowed it. Each was a second copy of a fact the spec now states once.
spend.go's meteredTrees stays — it answers a different question (does this need
standing) and rewiring the live paywall onto a boot-time index would fail OPEN
on a nil index, which is the one direction a money gate must never fail. It is
pinned against the declaration instead: TestFreeSurfacesAreNotGated fails if a
surface claims to cost nothing while the paywall gates it.
Declare (was indexSubsystems) is exported for one reason: the tests that ask
what the REAL composition root declares cannot boot 111 subsystems' encrypted
stores to find out, and a price test that cannot read the real declaration is a
price test that proves nothing.
Every zero here is falsifiable. The four price tests that previously passed
against a nil index — DefaultPrice, DefaultPriceExemptsIAM, the two auto-routing
ones — now declare a 1c/7c control surface first, so a dead index fails them
instead of satisfying them.
An IAM that predates the type field ignores an unknown query parameter
and answers with the sk- it has always minted. Handing that back as a
publishable key gives a caller who asked for something to embed in a
browser bundle a session-equivalent secret — a key's prefix is what
every downstream reader dispatches on, so this is a credential in the
wrong place, not a labelling nit.
Reachable without anyone making a mistake, purely by deploy order. So
the mint verifies the prefix it got and 502s otherwise: the worst case
becomes an honest error instead of a leaked class of credential, and
cloud may ship before IAM safely. Reverting the check reproduces the
exact bad response: {"type":"publishable","key":"sk-…"}.
One concept had four names — /v1/iam/mint-user-keys, /v1/iam/revoke-user-keys,
/v1/iam/keys, /v1/ingest/keys — and the only honest one 404'd. It is
/v1/keys now: POST creates, DELETE revokes, GET lists, and the key TYPE
(publishable | secret) is a FIELD. mint/issue/revoke are HTTP methods.
The name was load-bearing, not cosmetic. api.hanzo.ai routes /v1/iam/*
straight to the IAM service (ingress router api-hanzo-ai-iam-api), so
this handler was UNREACHABLE at the only host callers use: every request
landed on IAM's Guard and 401'd, which is why the console works around
it with its own Next route and a comment that cloud's keys endpoint
"501s on this deployment". A key surface spelled as if it belonged to
IAM was answered by IAM. Naming it for the resource makes it reachable.
/v1/iam/keys stays as a thin DEPRECATED alias — the same handlers, plus
RFC 8594 Deprecation + Link — because the go:embed console addresses it
on cloud's own origin, where the edge does not shadow it.
THE PUBLISHABLE KEY. Both halves of it were missing here. Nothing could
mint one — so every surface configured its own ingest credential and
error reporting kept a separate DSN — and nothing could RESOLVE one:
OrgForKey sent every prefix to get-user?accessKey, which refuses a pk-
by design, so a publishable key resolved to nobody and the ingest path
it exists for could never attribute a beacon. A pk- now resolves at
IAM's org-only door (resolve-key), which answers with the org and NO
principal. Two doors because they answer different questions, and their
cached values are different types so they cannot be confused.
GET reads the KEY ROWS. It read the USER row, while the mint writes a
key row — the "key never listed" bug in its second incarnation: the
read reported no key immediately after a successful POST. Reverting
that one line reproduces it exactly, as {"keys":[]}.
origin landed with four derived values and two new browse axes; the section
still described a corpus whose only questions were whose it is and whether you
may fork it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three of the four fields a browse row is made of were wrong on live data, and
each wrong one cost something concrete.
description was `""` on 43 of 66 rows. It is not decoration: fork.go copies it
onto the forked project, so an empty description propagated into every
customer's project list. Each row now carries one line, and where the template
has a live deploy that line describes what the deploy actually renders — read
off the page, not guessed from the slug. Rows with no live deploy are described
from their OWN facts (features/useCase) so a description can never be a claim
about a page nobody looked at.
source pointed at https://gallery.hanzo.ai/templates/<slug> on all 89 rows and
variants. Every one of those 404s — gallery.hanzo.ai is a static export whose
detail pages were never exported. That was not merely a dead link in the UI:
fork.go assigns Source to createReq.Repo.URL, so forking ANY gallery template
handed the builder an HTML error page as a git remote, and the provider came
out "git" because the host was not a forge. Source now names the repository the
template lives in (github.com/hanzo-templates/<repo>), which is the one thing a
build can use. A variant resolves to its own repo where it has one
(prism-react, cipher-html, cipher-react) and to the template's otherwise,
because a PAGE of a kit is not a repository.
demo was crossed on seven rows: Blocks advertised forge.hanzo.app, which
renders "Streamline"; Loop advertised blocks.hanzo.app, which renders Bento
v.3; Canvas advertised studio.hanzo.app, Studio advertised pixel.hanzo.app, and
Unity advertised loop.hanzo.app. Browsing a template showed a stranger's
product. The rule that ends that whole class: a template's demo is its OWN
deploy, <slug>.hanzo.app, or it has none. That re-points all seven, adds ten
that were live and unlisted (solo, forge, deploy, pixel and all six games), and
drops five whose own host is dead or serves an empty shell rather than keep
pointing them at someone else's page. 55 of 66 rows now have a demo, and all 55
answer 200 with the template's own content.
The facets the browse rails sort on were corrected against the same evidence:
Blocks/Deploy/Cipher are Bento Cards (that is literally what they render), Loop
is a crypto-card product site, Forge is a SaaS landing page, Soar is a web
studio, Serif is a magazine, Unity is creator admin. Features followed —
"Fitness, Booking, Classes" described a page Soar does not have.
preview: the ten @hanzo/ui rows carried none at all and nine mobile rows
pointed at a gallery path that never existed. Three are fixed by pointing at
screenshots that already exist (catalyst.png for Innovise, analytics-dashboard,
changelog); thirteen more were shot from the running demos and committed to
hanzoai/gallery, and go live with the next gallery release. swiftui and
swiftui-chat still have no image, because there is no web demo to shoot.
frameworks are deliberately untouched: the label is fork.go's build hint and
the REPO is its source of truth, which this pass did not read. Two rows look
wrong against their deploy (soar, forge) and are left for whoever reads the
repo.
catalog_test.go pins the four rules on the embedded data, so none of this can
rot back: every row describes itself in one line, every source is a repository
URL, every demo is the template's own host (deriving the `-template` suffix from
sites.IsReserved, the same predicate fork.go derives it from, so the two cannot
drift), and no slug repeats a word.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The team front talked to two pods that did one small thing each, and both
things are decisions the binary already had the keys to make.
analytics: POST /v1/event/collect is the team SPA's wire on the event plane.
It is a second WIRE, like /v1/insights/e, not an alias — the SPA POSTs a bare
array of {event, properties, timestamp:millis, distinct_id}, and the canonical
decoder accepts that and then drops it whole: `distinctId`/`time` are absent so
both come back empty, Type is empty, canonicalType("") is "event", and "event"
is not in publicKinds. The caller sees 200 with nothing stored, which is worse
than a 4xx because the SPA's retry loop discards on ok. decodeTeam names the
kind instead, so error and navigation land, and it re-homes the flat error_*
properties onto the typed Exception so foldException's scrubber actually runs
over the stack rather than a copy of it reaching the row and the destinations
fan-out unscrubbed.
The team session token joins eventTenant's trust order rather than the door.
It is a platform credential, so it belongs with the other three and works on
every door; a door that resolved its own tenant is the drift handle exists to
prevent. Verified fail-closed: signature, exp, nbf, a non-empty signed
extra.org, and a refusal of the public default key and of reduced (guest,
readonly) sessions.
meet: POST /v1/meet/getToken mints the LiveKit join token. Two keys, two roles
— SERVER_SECRET verifies the caller, LIVEKIT_API_SECRET signs the answer — and
the tenant boundary is the signed workspace claim against the room-name prefix,
because room names are client-chosen. The grant is roomJoin into one named
room and nothing else, for ten minutes. Media is untouched and stays external:
the browser still opens wss://live.hanzo.bot directly. Missing either key is a
503 on /v1/meet alone.
25 mutations across both, all killed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerce's `migrate encrypt` was wired in commerce 3d0c5407c and cut as
v1.49.29. Cloud pinned v1.49.27, so nothing that ships carried it: the
65 live tenant money stores under /var/lib/cloud/commerce/orgs are all
plaintext on disk (measured: 65 plaintext, 0 encrypted, 0 .dek sidecars,
plus one stray orgs/hanzo/data.db.encrypting.dek from a hand-run that
stopped).
v1.49.28 is NOT the tag to take — it sits on e02a57e91, which predates
the fix. v1.49.29 is main and the first tag that contains it.
The backfill is deliberately not on the boot path, so this bump does not
convert anything by itself. It puts the command in the image so the
one-time backfill can be run in a maintenance window with the daemon
stopped (the migration proves exclusivity with a verified TRUNCATE
checkpoint and refuses a busy file).
Also carried, because go.sum resolves it transitively: zap-proto/zip
v1.16.1 -> v1.17.2, which is the version commerce v1.49.29 requires.
Verified: `go build ./...` clean; `go test ./clients/commerce/...` clean;
the 15 failures in `go test .` and the one in ./clients/metering are
byte-identical on origin/main f015a3dbe without this change — all are the
same macOS "no RAM-backed scratch for the pure-Go SQLCipher codec"
limitation, not regressions.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/catalog returned 579 undifferentiated entries. In that list a curated
starter you fork FROM, a stranger's remix of one, somebody else's paid UI kit
and luxfi/node all rendered identically — same card, same shape, same implied
claim. That is not a labelling slip, it is the information architecture: the
corpus had no field for what a thing IS, so no surface could ask.
Origin is that field, and it takes four values because there are four nouns:
template our curated starter, the thing you fork FROM
community somebody BUILT this — forks, remixes, our own seeded examples
third-party somebody ELSE's work, carried only with its credit
product our own software: the fleet's repos
It is DERIVED, never typed: from which GitHub org a repo sits in (one table,
defaultOrgs, now saying both the brand and the lane), from whether a live slug
is one the curated gallery itself publishes (read forward through the three
shapes the fork flow derives a slug in, so a new template files its own demo),
and from what the owner declared — a recorded fork parent, an upstream credit.
A field an operator types is wrong by the second week; these cannot drift,
because they are the same facts the fork flow and the sites edge already run on.
Origin is deliberately NOT braided with Official. Origin says which lane; the
existing admin-gated marker says whose work it is. A single "official-example"
value would make them unaskable separately, and "show me community apps that
are NOT ours" is the entire point of having a community lane.
Third-party is now attributed or not listed. A fork holds somebody else's code
under one of our org headers, and GitHub's org listing omits `parent`, so the
sync spends one extra request per fork to name it and DROPS any fork it cannot
credit rather than showing it authorless. Against the live corpus: 437 repos,
40 third-party rows, every one carrying its real upstream (hanzo/ui →
frappe/ui, hanzo/BoatAttack → Unity-Technologies/BoatAttack). NOASSERTION is
GitHub failing to identify a licence, not a licence, so those rows state the
upstream and no terms.
Two axes fall out and both are faceted, because a facet nobody can act on is a
rail that lies: ?origin= cuts the corpus into the lanes, ?template= selects one
lineage ("everything built from folio") — which is what makes a community lane
readable rather than a pile.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1 is the .hanzo/workflows era. There is no v2 — one tag, forward only, no second
path kept alive for compatibility.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The two things a reader would otherwise have to reconstruct from sync.go: a
demo and its repo are one row (they collide on <org>/<name>, and the site
used to win by deleting the other), and forkable is a real axis with a
negative case rather than a label every row wears.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three live hanzo.app demos rendered wrong in a browser while answering 200, so
every status-code check passed them. Two separate causes, one shared symptom.
The symptom first. hanzo-team and edge both died on
Unexpected token '<', "<!doctype "... is not valid JSON
thrown from inside minified vendor code with no URL in the message. Neither app
was at fault for the message: each fetched a data file that is not in its
artifact (hanzo-team a /config.json the Huly front SERVER generates from env and
a static deploy therefore never has; edge a presets/wigglewobble.json that
sheryjs has never published), and the plane answered an honest 404 whose BODY
was an HTML page. Status right, media type wrong — and .json() on markup is the
least informative failure in the language. notFound now answers a missing DATA
asset as JSON, keyed off the same contentType() the 200 path uses, so the fetch
parses and the app is handed the path that is missing. A site's own 404.html is
for a human reading a page, so a data request never receives it.
The second cause was not in our bytes at all. savor rendered, then intermittently
collapsed to "Application error: a client-side exception has occurred" with
React #418/#425 → #423 on EVERY load. Diffing the served HTML against the
hydrated DOM found the edit: the hanzo.app zone had Cloudflare Email Obfuscation
on, rewriting example@gmail.com into a __cf_email__ placeholder plus a decoder
script. React compares the markup it receives against the markup it renders, so
that one substitution discards the whole server tree and re-renders the root on
the client; when the client render also throws, the error boundary is all you
see. Twelve of 163 live sites carried the rewrite. The setting is now off, and
AssertHTMLPassthrough keeps it that way: the site server holds this deployment's
zone credentials, so it is the one place that can state, in code, that the edge
must deliver our HTML byte-for-byte. It reads the rewriting settings at mount,
PATCHes only drift, and degrades to a Warn exactly as PurgeTags does — a token
that cannot read zone settings must never stop the plane from booting.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/gallery and hanzo-templates/gallery are both ours, both unstarred, so
the previous rule fell through and left the winner to map iteration again —
the row would change its repo link and description between syncs. Which one
wins does not matter; that ONE of them always wins does. Total order, closed
by the repo URL.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The published catalog admitted a live site on ONE question — whose org
published it. That is a question about a place, and a place cannot tell a
shipped app from a deploy probe, from a scaffolding placeholder, or from
the same page entered twice. So all three were live on the public catalog:
hanzo-tpl-probe 399 bytes of "<body>x"
hz-probe-01 480 bytes of "probe ok"
vite + next the same 824-byte "Template Placeholder" stub, twice,
byte-for-byte identical
android-expo-... a generic "Welcome to Project ACME" landing with
nothing Expo, NativeWind or Android about it
Whose it is and WHAT it is are now separate questions asked in separate
places. gate.go reads the document a visitor is actually served and
refuses on three grounds and no others: an unbuilt scaffolding
placeholder; a page under a kilobyte that is also inert; a body
byte-identical to one already admitted this pass.
Inert is load-bearing, not decoration. Size alone would have been a bug:
hanzo-team serves a 784-byte SPA shell and prism a 682-byte redirect, and
both are real apps. What has nothing to show is a small page that will
never fetch anything either — no first-party script, style or frame, no
redirect. Our own analytics tag and the edge's beacon are stapled onto
every page we serve, so a reference to another host does not count as the
site being alive, or every probe on the fleet would read as an app.
Refusal is DEMOTION, never deletion, and it never touches the repo row a
demo folds onto. A held site is still live and still its owner's, and it
lands in the platform's own corpus carrying Note — the reason it is not
public — so a demo that drops off the public lens can be explained instead
of just disappearing.
Fail open, twice over. A page we could not read is unjudged, so it is
admitted: an edge blip must never prune the catalog, the same reason
sync.go refuses to let a rejected GitHub token empty it. And a pass that
would hold MOST of the corpus has diagnosed the READER, not the sites, so
it is discarded whole rather than acted on.
Measured through corpus() against the 163 live pages the edge served on
2026-07-28: 158 published, 5 held — exactly the junk above, zero
collateral.
The gate does NOT catch two different BUILDS of one design (gleam and
prism-react serve the same XORA landing from a static and a Next.js build;
their bytes differ). That needs rendered comparison, which a reconcile
does not do, and is handled by removing the duplicate project.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/ui and hanzo-apps/ui are both "hanzo/ui" — the same one-id-two-rows
shape the site fold just fixed, one level up. sourceOrgs is a map, so the
survivor was Go's iteration order: hanzo/ui could report forkable:true on
one sync and forkable:false on the next, because hanzo-apps/ui is a fork of
a third-party upstream and hanzoai/ui is not. Nondeterminism was invisible
while every row said forkable:true; giving the axis meaning made it a
visible lie.
Two collisions exist today (hanzo/ui, hanzo/gallery), so the rule stays as
small as the problem: ours beats a vendored fork, and between two of ours
the one people actually use wins.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The split-brain, closed: platform read projects from cloud's EMBEDDED copy of
the iam store while the edge served /v1/iam from the standalone deployment —
two databases for one noun, so a project created at /v1/iam was invisible to
the PaaS and vice versa.
ProjectStore now has one selector: an external IAM named by IAM_URL is read
over HTTP; its absence means this binary IS the IAM and the embedded store
stays canonical (single-binary dev, 334 ms boot, unchanged). The HTTP client
authenticates AS THE ORG — each read runs on a client_credentials token for
that org's own "<org>-platform-kms" identity, the same credential the KMS
sync uses, minted on first need by kmsOrgIdentity and admitted by IAM's new
narrow grant (read-only, own-org-only, contract-named). One identity per
tenant serving both planes; tokens cached per org.
Failure is honest at every layer: IAM unreachable is an error (requireProject
503s; run proceeds under the implicit default with a warning); a nil identity
provider fails closed before anything reaches the wire. Each pinned by test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The projects store already gated the first-party marker correctly — only a
SuperAdmin can raise Official — and then clients/catalog/sync.go dropped it on
the floor. So /v1/catalog emitted 579 rows with no way to tell what we built
from what somebody else built, under a page headed "Every project, app and site
across Hanzo, Lux and Zoo". Nineteen of those rows are bought UI8 kits: the
served HTML of kinetic.hanzo.app still carries ui8.net/ui8/products/
fitness-pro-website-ui-kit and "Trusted by more than 2M users worldwide".
Omitting authorship there is not a neutral gap; in a first-party directory it
reads as a claim.
The gate was never the problem, so the gate is untouched. What changes is that
the answer travels: LiveSites selects it, fromSite copies it, Entry carries it,
the API emits it, and the browse rail can ask for it either way.
Authorship is two claims pointing opposite ways, and they get opposite rules:
Official "Hanzo made this" — a claim about US, so only we may make
it. Still admin-only, at create and at update, unchanged.
Upstream/Licence "somebody ELSE made this" — a claim that can only subtract
credit from the publisher, so it takes no gate at all. A
platform where claiming authorship is easier than disclaiming
it is a platform that launders provenance.
In fold, a declared credit is a VETO rather than a tie-break. The repo half
infers "ours" from the org a repo sits in; the site half is a human statement
that the work is not. An inference must never outrank a statement — otherwise
hosting a bought kit in our own template org launders it into a Hanzo example,
which is exactly how this went wrong. Forkable takes the same veto: a credited
kit is ours to SHOW, never ours to hand out.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two data gaps, one root cause.
A site row keys on "<org>/<slug>" and a repo row on "<brand>/<name>". For
anything we deployed from our own repo those are the SAME string, and the
index keys on it too — so the site row did not sit beside its repo row, it
OVERWROTE it. hanzo/kart-racer is live proof: hanzo-templates/kart-racer is
a public repo, and the published catalog held one row for it with repo:""
and no description, language or stars. 21 of the 163 live sites were eating
their own source that way. That is why no kind=site row could name where it
came from: the answer was there and got deleted an instant later.
So fold instead of clobber. A thing the fleet built has a source AND,
sometimes, a deployment; it was never two things. The site contributes what
is LIVE (url, human title, last deploy), the repo what is SOURCE (link,
description, language, stars), and nothing gets blanked. Row count and the
kind facet are unchanged — this is pure information recovered.
The authoritative path is now plumbed too: projects already stores repo_url
(what a project declared it was built from) and forked_from (the attribution
edge the fork path stamps, #76), and LiveSite carries both out, so a project
that DECLARES its source outranks any name match and lineage reaches
Entry.Template instead of dying in the store.
And forkable can finally say no. It was `true` on every row and read as
`c.Query("forkable") == "true"`, so the facet was {"true": 579}, the pill
filtered nothing and ?forkable=false silently meant "no filter" — a boolean
axis whose negative case is unaskable is a label, not a filter. It now means
one thing: we can hand you a public source for this.
- a repo that is itself a fork of a third-party upstream is NOT forkable;
its license and its lineage belong upstream and the honest fork button
points there
- a live demo with no public source is NOT forkable; there is nothing to
hand over, however good the screenshot
The query is tri-state (strconv.ParseBool: set-true, set-false, unasked) and
the facet counts both sides through the SAME loop as org/kind/language —
the special-cased block that could only ever accumulate "true" is gone.
Forkable also drops omitempty, because false is an answer and an absent
field is not.
One seam per source while I was here: fromOrgs joins liveSites as a package
var, so the assembly is testable without a live GitHub.
Tests fail against the old behavior — verified by reverting each line:
TestSiteCarriesItsSource prints the exact production row (Repo: empty),
TestForkableDiscriminates and TestForkableIsAskableBothWays both fail on
"the two sides must partition the corpus".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The login page advertised three doors and the account RPC opened a fourth.
/providers returned google, github and openid. Every entry did federate through
IAM, but the page then held its own copy of which identities hanzo.id accepts —
a second place for that answer, drifting the moment IAM gains or drops one.
It now returns the single openid door and lets the issuer answer for itself.
The account RPC accepted {"method":"login", email, password} and walked the IAM
password grant server-side. It authenticated correctly, and that was the
problem: a session minted there never passes through hanzo.id, so it skips the
identity check and the training-data consent that gate a first session. Both
handlers are gone; the method falls through to UnknownMethod.
This has to hold at the backend because the deployed front image still renders
the form: its bundle reads HIDE_LOCAL_LOGIN and stores the metadata, and the
compiled gate is present and correct, yet the flag arrives false at the
component while DISABLE_SIGNUP from the same config load arrives true. Hiding
the form is the front lane's fix; refusing the credential is this one's.
TestProvidersSurface locks the one-door list and names google/github as the
regression. TestPasswordRPCIsRefused proves a correct email and password get no
token and no session cookie. Both mutation-checked: resurrecting either door
turns its test red.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The 424 I shipped made /v1/run a dead end: the canonical project surface
(/v1/iam, the standalone deployment) and the store platform reads (the
embedded IAM) are DIFFERENT databases today, so no sanctioned door could
seed the row the refusal demanded. The dual-store fault line is its own
program; until it lands, refusing an org for a row nobody can write is not a
boundary, it is an outage.
The rule that honors both rulings, stated once and applied in the two places
that ask: the DEFAULT project is part of what an org IS, so run and the apps
tree under it proceed whether or not IAM has materialized the row — and
platform still never CREATES a project (nothing here writes the project
store). An explicit, non-default project must exist in IAM, exactly as
before. Pinned by TestRunWorksWithoutDefaultProjectRow.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
chore/telemetry-split does not build from a clean checkout, and has not
since 820bbca5. Three packages fail at HEAD, each for the same reason —
a commit landed code referencing symbols that exist only in the working
tree:
openapi weave.go wants sortedKeys / Components / uniqueOperationIDs
clients/plugin plugin.go wants core.Admit and core.OK-as-a-string, and
zip.PluginStatus, which no released zip defines
clients/search wants cloud.Bridge / cloud.Request from the root typed.go
Verified in a detached worktree: those three are the ONLY failures, and
clients/admin builds fine at HEAD, so core.Admit/OK are what plugin.go
was written against rather than a change it forces.
This is not new work. It is the missing half of commits that were pushed
without it, plus the zipdoc_gen.go projections and typed-op tests that
travel with them.
DELIBERATELY EXCLUDED — the templates redesign is genuinely half-done and
regresses. clients/projects + clients/templates rename brainwave→synapse
and collapse sibling slugs behind a `variant` selector; fork_test.go and
fork.go are updated for the new design but catalog.json still carries the
old slugs (77 entries: brainwave PRESENT, synapse/prism/saas-landing
ABSENT). TestFork* passes at HEAD and fails with those changes applied —
4 tests, all 404 "template not found". Left in the working tree so the
data migration lands with them.
Also excluded: LLM.md, whose working-tree copy reintroduces cloud.Global,
deleted in 62c7f52d.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
CacheControlFor gives a content-hashed name `public, max-age=31536000, immutable`
— but only for an extension in its list, and that list is a document-web list.
A WebGL build's payload is not: Unity ships <name>.data (5.7 MB in the template
we just published) and Godot ships <name>.pck, and both fell through to the
default `public, max-age=3600`. So the two files a player waits on longest were
re-fetched every hour while the 500 KB of JS beside them was cached for a year.
Measured before this change, on a live release:
GET /Build/reef-0881644a.data -> cache-control: public, max-age=3600
GET /orb-dd8b3278.wasm -> cache-control: public, max-age=31536000, immutable
Same name shape, same immutability guarantee, opposite policy. gameAssetType
right above already teaches this file that a site can be a WebGL build; the
cache policy has to know it too. Adds .data/.pck/.unityweb/.mem to the same
branch — no new rule, no second code path, and non-fingerprinted names keep the
conservative TTL exactly as before.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
f106e4b6 lowered the directive to 1.26.4 on the reading that `go mod tidy` had
bumped it by accident and nothing needed 1.26.5. That is right about our own
code and wrong about our dependencies:
$ curl https://proxy.golang.org/github.com/zap-proto/zip/@v/v1.16.1.mod
go 1.26.5
github.com/zap-proto/zip@v1.16.1 declares 1.26.5 itself, so go refuses on ITS
requirement regardless of ours.
Lowering it also had a second effect that is easy to miss: .hanzo/workflows/
cicd.yml runs `actions/setup-go@v5` with `go-version-file: go.mod`, so this
line chooses the CI runner's toolchain too. Dropping to 1.26.4 therefore moved
the failure from the Docker builder onto the runner —
`zap-proto/zip@v1.16.1 requires go >= 1.26.5 (running go 1.26.4)`, this time
with no GOTOOLCHAIN=local in sight, which makes it look like a different bug.
The reason 1.26.5 was unsatisfiable is now gone: the builder pin moved to
ghcr.io/hanzoai/mirror/golang:1.26.5-alpine@sha256:0178a641 in bd033d3d, after
the mirror itself was refreshed (it had been frozen at 1.26.4 with nothing
syncing it). With the directive back at 1.26.5 all three agree — go.mod, the
runner's setup-go, and the image builder — and the dependency floor is met.
1.26.5 is also current: go.dev lists it as the newest stable and golang:latest
is the same image.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
820bbca5 — a docs commit — swept in openapi/weave.go and left behind
the openapi.go it references. Since then the branch has not compiled:
sortedKeys, Components and uniqueOperationIDs exist only in the working
tree, so `go build ./openapi` fails at HEAD and takes every binary that
imports it down with it. Verified in a detached worktree at HEAD:
openapi is the ONLY package that fails; everything else builds.
Nothing here is new work. It is the already-referenced half of a commit
that shipped without it, and it was pushed in that state.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both lanes publish binaries.json and I claimed they were the same document, but
`name` disagreed: hanzoai/ci writes the `binaries:` entry's name (`cloud`), this
wrote the file (`cloud-linux-amd64`). A host resolves a plugin BY NAME —
`zip.Load(zip.Plugin{Name: "cloud"})` — so an index built by the platform
answered to a name no host asks for, and the same artifact had two identities
depending on which front door built it. The file was never missing information:
it is the tail of the url.
The build script carries the recipe name to the publisher in meta.txt (the same
per-file hand-off that already carries os/arch), so the publisher composes the
entry rather than inferring it from a filename.
Tested by running BOTH scripts back to back over a real repo with the PUT
stubbed — the meta.txt hand-off between two containers is exactly the seam a
unit test of either half alone would miss.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Circuit is input-heavy (continuous steering, brake, off-track grip loss) where
Orb Runner is physics-heavy, so the catalog now covers both the input and the
simulation paths of a browser game rather than a single sample.
Live and rendering at kart-racer.hanzo.app (60 fps, lap timer running, autopilot
until first key press).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
orm dropped its root xorm re-export, so cloud stops linking an engine it never
called. Six files import hanzoai/orm and use DB, Register and TxOptions; not one
used orm.Engine/Session/Rows/NewEngine. Go links whole packages, so the shim in
orm's root package was enough to pull ~5.9 MB of xorm into every build.
`go mod why -m github.com/hanzoai/xorm` now answers "main module does not need
module github.com/hanzoai/xorm", and go.mod carries zero xorm lines.
xorm is not gone from the estate — it is still the relational engine behind
hanzoai/orm/relational, which ~/work/hanzo/vm imports directly across 15 files.
It is simply no longer a dependency of every service that only wanted the
generics API.
go build ./... exits 0; clients/deploy, iam, platform, research all pass.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
f106e4b6 lowered `go 1.26.5` -> `1.26.4` on the reading that the directive was
an accidental `go mod tidy` bump and nothing needed it. The reasoning about our
own code is right, and the go.mod change is kept here. But the build still
fails, because the floor is not ours:
$ curl https://proxy.golang.org/github.com/zap-proto/zip/@v/v1.16.1.mod
go 1.26.5
github.com/zap-proto/zip@v1.16.1 declares 1.26.5 itself, so go refuses on the
DEPENDENCY's requirement no matter what our directive says — and with
GOTOOLCHAIN=local the builder cannot fetch one. Confirmed empirically: the cicd
run on f106e4b6 failed the same way.
So the builder has to move. The pin was never stale relative to the mirror — it
matched exactly; the MIRROR was frozen, still sha256:47d47cb5, pushed by hand
with nothing syncing it (its GHCR package reports repository:null). That is why
this reads as a dependency problem and is not one.
Refreshed through the Mirror workflow in hanzoai/universe, which now handles
base images: `skopeo copy --all`, verified byte-identical —
ghcr.io/hanzoai/mirror/golang:1.26.5-alpine is sha256:0178a641, the same digest
as docker.io/library/golang:1.26.5-alpine. Still digest-pinned, so it cannot
float, exactly as before.
The tag also leaves alpine3.22: upstream stopped building that variant on
2026-06-03 (Alpine is on 3.24), so no current 1.26.5-alpine3.22 exists to
mirror. 1.26.5-alpine is the maintained line.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every cloud release has been failing at `go mod download`:
go: go.mod requires go >= 1.26.5 (running go 1.26.4; GOTOOLCHAIN=local)
The directive moved to 1.26.5 in bf678adfd, a commit about the admin host view —
its go.mod diff is exactly `-go 1.26.4 / +go 1.26.5` and it carries no toolchain
directive. That is the signature of `go mod tidy` run on a workstation with a newer
toolchain, which rewrites the minimum silently. Nothing in the change needed it.
The builder is pinned by DIGEST (golang:1.26-alpine3.22@sha256:47d47cb…), so it
cannot float to a newer patch the way the tag would, and GOTOOLCHAIN=local stops it
downloading one — both deliberate, both the reason a pin is a pin. The build then
fails before compiling anything, which is why this reads as a dependency problem
and is not one.
Lowering the directive rather than re-pinning the builder: Go PATCH releases carry
no language or library changes, so 1.26.5 buys nothing here, and moving a pinned
toolchain digest to unblock an accident is a much larger change than undoing the
accident.
`apk add sqlcipher-dev=4.6.1-r0` now resolves to nothing — Alpine replaced the
r0 packaging revision with r1 and drops the old one from the index:
sqlcipher-dev-4.6.1-r1:
breaks: world[sqlcipher-dev=4.6.1-r0]
which is exactly the LOUD failure the pin exists to produce, and exactly the
remedy its own comment prescribes: bump the pin, then confirm the on-disk format
is unchanged before shipping. That confirmation is not a manual step here — the
image build runs cek's TestFrozenFixtureOpens INSIDE the image under the pinned
libsqlcipher, so an r1 that changed the format fails this build red rather than
bricking existing encrypted stores.
The pin stays EXACT (not `=~4.6.1`) on purpose: a packaging bump must remain a
decision someone makes, which is the whole point of freezing the cipher format.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every one of the 60 catalog entries was a document-shaped web app. A user
evaluating hanzo.app against Lovable/v0/Bolt therefore saw the category where we
are identical to them and none of the category where we are not: hanzo.app
serves published sites cross-origin-isolated (COOP/COEP) for a declared
unity|unreal|godot project, and pins the application/wasm and
application/octet-stream MIME types a WebGL engine loader needs. Nothing in the
product said so.
Five game/3D templates, each with a live demo that renders (verified in a real
browser, not by status code):
godot-arcade Godot 4.4 web export, playable 3D game godot-arcade.hanzo.app
unity-webgl real Unity WebGL build, COI + SAB true unity-webgl.hanzo.app
three-webgpu WebGPU-first three.js, WebGL2 fallback three-webgpu.hanzo.app
rapier-physics Rapier3D WASM rigid-body sandbox rapier-physics.hanzo.app
voxel-craft voxel world, face-culled chunk meshing voxel-craft.hanzo.app
`source` points at the real hanzo-templates repo rather than a gallery.hanzo.ai
path, because fork() feeds Source straight into the created project's repo URL
and the gallery paths the existing entries carry are 404. `preview` points at a
screenshot published by the demo itself, so the preview can never drift from
what the template actually renders.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two halves of the same table, found by attacking the boundary with a real
customer org. From `bluefin-studio` — an ordinary tenant, no admin — I claimed
`api.hanzo.ai` as a custom domain and got a 200 with a DNS challenge. Then org
`hanzo` was refused its own production API hostname:
409 domain api.hanzo.ai is already bound to another site
Nothing served: the serve gate (Server.customCandidate) already excludes every
self domain, so the claim was inert at read time. But site_hosts is FIRST-COME
and global, so writing the row was enough to deny the name permanently. That is
the failure mode reserved.go warns about in its own doc — "create AND bind both
reject, so site_hosts can NEVER hold a reserved host; the serve gate is a
backstop, not the sole guard" — except the self-domain rule only ever had the
backstop. Serve excluded all of hanzo.ai + hanzo.app; claim excluded only the
sites apex, so the brand domain walked through.
Fixed by giving the self-domain set the same shape the reserved set already has:
ONE package-level source in clients/sites, published by New from the SAME list
the serve gate is built with (SetSelfDomains), read by both gates (IsSelfHost).
customCandidate now delegates to it, so there is one implementation rather than
two that agreed by luck. Strictly wider than the old apex test, never narrower.
Second half: nothing could ever release a host. setDomains only adds (an empty
list is a 400, not a clear), delete-project unbinds the site's own bare slug and
nothing else, and there was no third writer — so a mistyped domain, a domain
moved to another provider, or a claim like the one above was held forever. That
is not ownership of a global namespace, it is a leak. DELETE
/v1/projects/:slug/domains/:host completes the surface over the UnbindHost
primitive that already existed: scoped to (host, org, slug) so it can only drop
this tenant's own row, and idempotent 204 so it cannot be used to probe which
hosts other tenants hold.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
71 templates and not one of them built a native application — every "ship me a
.dmg" question ended in the browser, and the one repo that looked like it might
help was empty. Eight desktop templates now exist in hanzo-templates: six Tauri
v2 (Rust core, system webview, single-digit MB installers) and two Electron,
used only where a Chromium-only API is the actual point (desktopCapturer).
Each row carries `demo` = the live web build at <slug>.hanzo.app. A native
window cannot be iframed, so the preview is a real build of the same UI: one
module (src/native.ts) decides whether a command goes to the Rust core or to a
web fallback, and nothing above it branches.
`preview` points at <slug>.hanzo.app/preview.png — the screenshot the template
itself serves — rather than gallery.hanzo.ai/screenshots/<slug>.png, which 404s
for anything added since that host was last built (mobile.png does today).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
It answered 404 when the catalog was cut, so the entry was left without a
demo rather than pointing at a dead host. It is live now and its page title
is the template's upstream name, which is the same corroboration every other
demo attachment used.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The build could not satisfy its own go.mod:
go: github.com/zap-proto/zip@v1.16.1: module requires go >= 1.26.5
(running go 1.26.4; GOTOOLCHAIN=local)
The pin was not stale relative to the MIRROR — it matched exactly. The mirror
itself was frozen: ghcr.io/hanzoai/mirror/golang:1.26-alpine3.22 was still
sha256:47d47cb5, pushed by hand with nothing syncing it (the GHCR package
reports repository:null). So a toolchain floor raised by a DEPENDENCY had no
way to be met.
Refreshed via the Mirror workflow in hanzoai/universe, which now knows base
images and copies them under mirror/ — `skopeo copy --all`, byte-identical:
ghcr.io/hanzoai/mirror/golang:1.26.5-alpine is sha256:0178a641, the same digest
as docker.io/library/golang:1.26.5-alpine.
Note the tag also moves OFF alpine3.22: upstream stopped building that variant
on 2026-06-03 (Alpine is on 3.24), so there is no current 1.26.5-alpine3.22 to
mirror. 1.26.5-alpine is the maintained line, and 1.26.5 is the newest Go —
golang:latest and golang:1.26.5 are the same image.
Still digest-pinned, per the immutability rule this file already states.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The defect: /v1/templates had no tenancy dimension at all. The catalog was ONE
embedded, global, read-only JSON — every org saw exactly the same list, and
there was no way for a customer to hold a starter kit that is THEIRS. Every
neighbouring surface already had the dimension (projects, sites, releases and
site_hosts all carry an org column keyed off principal.Org); templates was the
one catalog that did not, so "our internal portal scaffold" had nowhere to live
but a public gallery or a customer's own git.
Add the missing layer WITHOUT giving the public catalog a write route, because
the failure mode that matters is a private template leaking into what an
anonymous visitor browses on hanzo.app. So this is two containers, not one
visibility flag:
- public = the embedded catalog.json. Immutable, no writer, unchanged.
- private = rows in {DataDir}/templates.db keyed by (org, slug), org taken
from principal.Org — the gateway-minted, JWT-validated owner.
A body "org" is overwritten server-side, never trusted.
An anonymous GET never touches the store, so the public view cannot contain a
private row by CONSTRUCTION rather than by a filter each future reader has to
remember. Every private read/write binds the org column, so a cross-org GET,
PUT or DELETE is a 404 — not a hidden row, an unreachable one.
A slug stays single-valued across both layers: publishing over a public slug is
409, so slug -> template remains a function and no org can shadow the gallery.
Two different orgs may hold the same private slug; the key is (org, slug).
templates.Lookup(ctx, org, slug) replaces templates.Get as the ONE door other
subsystems read through — projects' fork resolves the caller org's own
templates first, then the gallery — so "which templates may this org use" is
answered in exactly one place. A fork of a private template records
owner-qualified lineage (acme/acme-portal); a public one keeps its bare slug,
which IS its global name. Variants are unchanged: a private template carries
the same Variants field and resolves through the same Template.Variant.
Proof, both directions: TestPrivateTemplateIsOrgOnly (owner sees it; another
org and anonymous do not, and the anonymous view equals the embedded gallery
entry-for-entry), TestWritesBindOrg, TestSlugStaysSingleValued, and
TestForkPrivateTemplateIsOwnerOnly end to end through the fork route.
templates now owns a store, so it owns a Shutdown; wire_test is refrozen for
that, and for "catalog", which landed earlier without refreezing the golden and
had left apps red on main.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/templates served 44 web starter kits and zero native ones, and the
`mobile` repo in hanzo-templates was an EMPTY placeholder — so "build an
app" had no entry point on the platform most people ship to. These eight
rows add the iOS lane: Expo/React Native and Flutter for cross-platform,
SwiftUI for native.
`demo` is set only where a demo can be real. Expo and Flutter both have a
genuine web target, so <slug>.hanzo.app serves the SAME source the iOS
simulator runs — all six verified in a browser: they load, paint, and
respond to a real tap or keystroke. SwiftUI has no web target, so those
two rows carry NO `demo` key rather than a screenshot dressed up as a
running app. The field is already `omitempty`; this is the first entry to
use that on purpose.
`source` points at github.com/hanzo-templates/<slug> instead of the
gallery.hanzo.ai fork URL the web rows use, because these are the repos
someone actually clones to open Xcode.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
THE DEFECT. /v1/runner is the platform's whole native build API and it had
exactly one output: launchDirectBuild -> BuildKit -> an OCI image pushed to a
registry. Anything a project actually produces that is not a container image — a
Go binary, a Rust binary, an npm package, a wheel — had NO in-cluster path, so
the platform's answer to "build my project" was "first write a Dockerfile". That
is the asymmetry: /v1/git has 27 routes and /v1/code has 7, and the build side
had one route that could build one thing.
THE REASON IT IS A DEFECT AND NOT A GAP. The declaration for those artifacts
already existed and was already load-bearing: hanzo.yml's `binaries:` + `bucket:`
blocks, which hanzoai/ci's reusable workflow has read since it was written, and
which THIS repo's own hanzo.yml uses to publish the 195MiB plugin binary that
108 apps resolve to. Only the GitHub lane implemented it. So the contract was
one and the implementations were one-and-a-half — a repo could declare an
artifact the platform was structurally unable to build.
WHAT THIS DOES. Implements the SAME contract natively (clients/platform/
artifact.go), the way `images:` is already built two ways — buildx on a runner,
BuildKit in-cluster — for one build path with two front doors:
POST /v1/runner {repo, sha, binaries:[...], bucket, tag}
launches ONE Job whose initContainers are one per recipe entry, each in that
entry's own toolchain image and sharing /w, followed by a publisher that hashes
what the recipe left in /w/dist, PUTs it to hanzoai/s3, and writes binaries.json
LAST. The published layout is byte-identical to the ci lane's
(<bucket>/<owner>/<repo>/<tag>/), so an artifact built by a GitHub push and one
built by the platform land at the same URL and a host reads ONE index.
THE FORMAT IS EXTENDED, NOT REPLACED. Two optional fields on a `binaries:`
entry: `run:` (the build command for any toolchain that is not Go) and `out:`
(the glob of what it produced). `main:` stays the zero-config Go lane, unchanged.
`image:` names the toolchain the platform runs `run:` in — the one field the
GitHub lane ignores, because there the toolchain IS the runner. There is no
second recipe: the JSON field names ARE the YAML field names, and `hanzo build`
with no --image reads the repo's own hanzo.yml rather than restating anything.
THE SECURITY LINE IS THE POD SPLIT, NOT A PACKAGING DETAIL. `run:` is arbitrary
shell BY DESIGN — it is a build command, the same trust as a Dockerfile RUN — so
it must never see a credential. The initContainers that run it carry no
object-store env and no service-account token; the publisher, which holds the
credential, runs a CONSTANT script. Every recipe value reaches both scripts as an
environment variable expanded in double quotes, never as interpolated script
text. The artifact lane inherits every bound the image lane has (isolated build
namespace, CI pool, per-org concurrency ceiling) and adds its own: the repo URL
goes through the same allowlisted-git-host validator, and on the IAM path the
forge owner must be one the caller's org owns — repoOwnerInOrg, reading the same
orgRegistryNamespaces map as imageInOrgRegistry, because a brand's registry
namespace and its forge owner are one name.
PROVEN END TO END on do-sfo3-hanzo-k8s against github.com/hanzoai/build-demo
(a recipe with no Dockerfile): golang builds hanzo-demo for linux/amd64 +
darwin/arm64, node:22 runs npm install && tsc && npm pack, and all three land at
https://s3.hanzo.ai/plugins/hanzoai/build-demo/9fc7653/binaries.json with
matching SHA-256s, fetched anonymously.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Collapsing the catalog onto the Hanzo names surfaced a name that cannot be a
project slug: `metrics` is a reserved subdomain (sites.IsReserved), so
POST /v1/projects/fork {"slug":"metrics"} answered 400 and the console's
one-click path had no way through — the caller would have had to know to pass
`target`. The DERIVED slug is a default, not the caller's choice, so it takes
the `-template` suffix its own live demo already carries
(metrics-template.hanzo.app) and the derived name matches the demo instead of
dead-ending.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The demo host a template was deployed to does not always carry the template's
name: catalyst.hanzo.app renders Innovise, studio.hanzo.app renders WebCanvas,
blocks.hanzo.app renders Bento v.3 (which is Loop). Matching slug-to-slug would
have attached eight demos to the WRONG template and left eight right ones dark,
so each is matched on the page's own <title> against the template's upstream
name — the same corroboration the first 23 used. 31 of 44 now carry a demo; the
remaining 13 are left empty rather than guessed.
innovise's screenshot (gallery.hanzo.ai/screenshots/innovise.png) is a 404 — the
only dead handoff URL of 58 — so it is blanked and the entry hands off to its
demo instead of a broken image.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Proved against production with three real customer orgs (northwind-labs,
bluefin-studio, kettle-river-co) created through the real IAM signup+onboard
path. Both northwind-labs and bluefin-studio created a project slugged
"journal" and deployed it. Both got:
status=live liveUrl=https://journal.hanzo.app
and https://journal.hanzo.app served northwind-labs' page. bluefin-studio was
told its site was live at a URL that renders another tenant's content — it
would have published that link to its own customers.
The store was never wrong: site_hosts is first-come and BindHost refuses a
foreign claim (TestSiteHostBindingIsFirstComeAndTenantSafe has always proved
that). What was wrong is that the ANSWER to "what is my URL" was computed from
the slug — siteURL(s, org, slug) — while the answer to "what does this host
serve" is computed from the binding. Two sources for one question, so they were
free to disagree, and for the loser of a global first-come race they always did.
Decomplected: ownership is the only thing that can answer both. onPublish
already performs the bind, so it now returns the URL that bind proves we own
(and a note when there is none), and all three publish paths — artifact deploy,
git/CI completion, release activation — stamp that instead of re-deriving it.
Only errHostTaken and errReservedHost blank the URL: those are definite proof
the host is not ours, while a transient bind error proves nothing and must not
blank a working site.
The deploy still succeeds, still bills, and the bytes are still live at the
org's own S3 prefix. What changes is that we stop lying about the address.
TestIdempotentRedeploy asserted the defect verbatim ("evil deploy url=%q, want
https://myapp.hanzo.app"); it now asserts the loser gets no URL.
CONTRACT.md said the tenant is the JWT `owner` claim — that claim is the
APPLICATION's org, not the user's, and believing it is how cross-tenant bugs
start. Corrected to the signed `orgs` membership claim, which is what the code
actually reads.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nothing built the artifacts manifest/release.go resolves. The host could read a
release index; no lane ever wrote one.
hanzo.yml now declares ONE binary, not 108. Measured on this tree: 108 dedicated
plugins are 4520MiB (mean 41, min 35 — that floor IS the core every one of them
links), against 195MiB for the multi-call binary, which serves any app via
--enable. 23x less to publish and fetch, and 1 link per platform instead of 108,
which matters because the Makefile already warns that linking those in parallel
OOMs a 128GiB box. The index names it once and every app resolves to it; an app
that ever needs its own cadence publishes a dedicated artifact and its entry
overrides the shared one, which remote() already prefers.
The image is untouched. This is a DIFFERENT artifact for a different consumer:
CGO_ENABLED=0 and static, because a plugin runs on whatever base its host is,
where the image's cgo+libsqlcipher /cloud would not start. release.go still owns
the image and the v* tags alone.
The v* trigger is what makes the lane fire at all — ci builds binaries on every
push and publishes only on a tag. That tag is release.go's receipt for a commit
that already built and smoked, so the gate is passed `tests: false` there rather
than re-running 108 links on a commit it already passed on main.
Also names the cache-invalidation contract in fetch(): success is cached for the
life of the process, so publishing a new index does not reach a live host. That
bounds anything built on top — per-org or on-demand upgrade has to restart the
host or drive zip.ReloadTo, which clients/plugin already exposes.
Verified against a real hanzoai/s3: both platforms cross-compiled from this
tree, published, and all 108 apps resolved from the served index to one artifact
with the right --enable on each.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
c.Path() is the RAW request target — zip returns Fiber's path verbatim and
nothing upstream unescapes it — but an object key is stored decoded. So any
path a browser had to encode could never match its own file.
Next.js names a dynamic route's chunk after the literal segment, and browsers
send that encoded, so on hanzo.app:
/_next/static/chunks/app/blog/[slug]/page-7bbb92f975391a19.js -> 200
/_next/static/chunks/app/blog/%5Bslug%5D/page-7bbb92f975391a19.js -> 404 (text/html)
Refused to execute script from '.../%5Bslug%5D/page-*.js' because its
MIME type ('text/html') is not executable
Every dynamic page on every deployed Next.js template served its HTML and then
never hydrated — a 200 that is not a working page. Same for any asset with a
space or a "+" in its name.
The decode goes in resolveKey, the one place a request path becomes an object
key, and it goes BEFORE path.Clean: that is also stricter, since "%2e%2e" is now
collapsed as the traversal it is instead of surviving as an opaque literal.
TestResolveKeyNeverEscapesPrefix passes unchanged. A malformed escape is kept
verbatim — a miss, never an error.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Retention is the first caller that hands purgePrefix a value READ FROM A ROW
rather than one computed on the spot. An empty or "/" prefix lists the whole
bucket, so a single bad row would turn a release GC into a bucket wipe. The one
primitive that deletes objects now refuses a prefix that does not scope anything,
instead of trusting every present and future caller to have computed a real one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
o11y is one subsystem serving two public prefixes — the read plane at
/v1/o11y/* and the Sentry ingest at /v1/sentry/*. The PluginSpec named
only the first, so the child process served /v1/sentry/* while the host
routed nothing to it, and the weave gate logged UNROUTED on every run.
The comment claiming zip.Load takes one prefix ("do not merge before
that is closed") was stale: Load has been variadic since v1.17.3 and
its own doc names this exact case.
manifest/apps.go regenerated from Wire(); openapi.yaml regenerated
because routing the prefix is what makes it publishable — the +101
lines are the two sentry paths and nothing else.
Both gates green on this tree: openapi weave PASS with no UNROUTED,
golden byte-compare PASS with no -update.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Live it returned a kind facet (repo 413 / site 119) that no parameter could act
on, so ?kind=site silently answered with everything. A rail that counts a
dimension it cannot filter is a rail that lies about being clickable. The test
now asserts the two sets are the same set, in both directions.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The catalog was spending a SLUG on every shape a template ships in, so a
browse row stopped meaning "a template". One portfolio kit (Folio) held 26
rows — folio-about, folio-grid-3-fluid, folio-masonry-4-columns … — Cipher
held 5, Prism 3, and matrix/mosaic/drive/loop/soar/hygge 2 each. Worse, the
catalog carried the SAME 72 templates TWICE under two naming generations
(hanzoai/gallery's `scripts/templates.json` upstream names — brainwave,
xora-react, bento-cards-v1-multipurpose-next-js — and `public/templates.json`
Hanzo names — synapse, prism-react, mosaic). They join 1:1 on the gallery id,
which is how 141 rows collapse to 44.
That defect had a visible cost, not just an aesthetic one: because the
"variant" of a multi-page kit IS its page-list index, two of those rows
deployed demos that rendered a bare column of links, and beta's and gallery's
were byte-identical.
So: slug identifies the TEMPLATE; format (-html/-react/-bootstrap), page
(folio's about/contact/grid-3-fluid) and theme are Variants inside it, and
the choice is made at fork time from what the user asks for —
POST /v1/projects/fork {"slug":"prism","variant":"react"}. Template.Variant
is the ONE resolution rule: no preference yields the default shape, a
template that ships one shape answers with itself, so callers never branch on
len(Variants). A non-default shape carries its id into the derived project
slug, so two shapes of one template coexist in an org; lineage still records
the template, because a shape is not a different parent.
Slugs are now the Hanzo names the demos and the gallery already use, so the
catalog, gallery.hanzo.ai and <slug>.hanzo.app finally say the same word for
the same thing. `demo` is attached only where the deployed page's own title
corroborates the template — 23 of 44; the rest are left empty rather than
guessed. Nine templates that only ever existed as deployed demos
(analytics-dashboard, changelog, markdown-editor …) enter the catalog with
the demo as their handoff, which is why preview is no longer unconditionally
required.
Dropped: beta-variant, whose upstream name is literally "Beta CRM
(Duplicate)" — a copy is not a variant.
TestVariantsAreOptionsNotSiblings pins the invariant: no variant id may also
be a catalog slug.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
deploy-templates.sh already calls "$(dirname $0)/prep.py"; the tool it calls
lived only in a scratch dir, so the thing that builds every hanzo.app demo was
untracked and unreviewable. It lands here, with the two shortcuts it was taking
removed.
1. heal_export os.remove()d every route.*, opengraph-image*, twitter-image*,
icon.* and apple-icon* under app/ the moment a build mentioned "Failed to
collect page data". Those are the author's social card and favicon, and
deleting them made the build pass while every demo shipped with no social
image at all — changelog.hanzo.app served zero og:image tags. A generated
metadata image DOES static-export; it just has to render at build time, so
it is now pinned (edge runtime dropped, force-static added) instead of
deleted. Only a route handler is still dropped, and only after the pin fails
to make it exportable — a static site cannot answer a POST.
2. FORCE injected eslint.ignoreDuringBuilds + typescript.ignoreBuildErrors into
every next.config, so "it builds" meant nothing: kanban-board's useState(null)
could never hold the task its detail dialog opens with, and that shipped
green. The knobs are gone. The first thing the honest build rejected was
prep.py's OWN config wrapper — a rest parameter is an implicit any in a
typechecked next.config.ts — so the merge is now a plain Object.assign.
3. Fallout the other two exposed: a failed Next build left the PREVIOUS run's
out/ on disk, root detection found it, and the artifact packed and reported
OK for a site that no longer builds. out/ is ours (FORCE asks for it), so it
is cleared before each build.
Verified live: changelog.hanzo.app now serves a 140,749-byte 1200x630
opengraph-image (404 before); kanban-board.hanzo.app renders and its task
dialog opens; 21 demos redeployed 200.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The file still told the next reader that releases are never garbage-collected and
that activate trusts the row. Both are now false, and the second one is the
subtle part: the bytes check is what makes retention safe, so it belongs in the
same paragraph as the retention rule, not left for someone to rediscover.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
openapi_yaml_test.go asserts ../../openapi.yaml matches the live
router, and both files were untracked on every branch — so the test
passed here and could not exist on a fresh clone, and openapi-composed
read a file CI never had. Verified before tracking: the test passes
against this spec (20414 lines, 18.4s).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Production ran with the durability fence disabled, on the strength of
durability disabled — object-store conditional-PUT atomicity NOT confirmed —
update-race admitted 2 writers (want 1) — If-Match not atomic
The store was fine. The probe was rewriting bytes that were already there.
probePayload returned one of four fixed values, cas-probe0..3, and BOTH race
phases drew from the same four. The create race leaves the object at one of them;
the update race then conditions on that ETag and writes the same four again. When
the update-race winner drew the create-race winner's index it stored identical
bytes — and an S3 ETag is derived from content, so the version did not move, so
every remaining racer's If-Match still held and every one of them committed. The
probe counted the winners and declared the store non-atomic. One collision in
four: 87/300 measured against s3 4.34.6, 56/300 against v1.0.4, 55/150 against a
local build. Binding the payload to the probe key (16 random bytes, fresh per
probe) and to the version being raced on makes an unchanged ETag impossible
unless the store really did reject the write. Same production gateway, 0/400.
The unit tests could not have caught this: fakeCondStore versions by a counter,
which advances even when a write stores identical bytes — the one behaviour that
mattered. So the regression is a fake that versions by CONTENT, the way a real
ETag does, and is atomic by construction; over it the probe must still find
exactly one winner. Restoring the old payloads fails it with the production
message verbatim.
Note for the next person down this path: the fault correlates almost perfectly
with the payload signature (minio-go streaming vs a plain-signed SDK) because the
probe is the only minio-go caller in the race. It is not the signature. Both
clients take the same routed filer transaction under the same exclusive lock, and
a post-commit re-read inside that lock returns the unchanged ETag.
promote() writes a new immutable prefix per distinct content and NOTHING ever
removed one. DeleteReleases only ran on project delete, so a site that publishes
on every commit accumulated every build it ever shipped in the object store,
forever, and the only way to free a byte was to delete the site. The rollback
menu needs depth, not history.
Retention goes where releases are BORN: every promote that mints a row reclaims
what fell past keepReleases (10 superseded, plus the live one). No sweeper, no
schedule, no second notion of "which releases exist" to drift from the rows.
A retention failure never fails the publish — the release the caller asked for
is already complete and recorded — it is logged and recomputed next publish.
That breaks an invariant activation was silently leaning on. ActivateRelease's
`WHERE EXISTS (release row)` was a sufficient guard ONLY because nothing could
prune bytes out from under a row. Now bytes can be reclaimed (by retention, an
operator purge, a bucket lifecycle rule), and flipping on the row alone would
point a LIVE site at an empty prefix and serve 404s with nothing in the row to
say so. So activate() now stats the release's index.html before the flip: row
first (404, no cross-tenant oracle), then bytes (410 GONE — that rollback target
is not coming back, publish again). Proven: with the stat removed, activating a
release whose bytes are gone returns 200 and takes the site down.
The live release is protected twice, and the second time is the one that holds
under concurrency: it is excluded from the prune scan (so it never even counts
against the depth — a site parked on an ancient release keeps it), and every
prune DELETE re-checks current_release AT DELETE TIME, so a rollback landing
after the scan makes that delete match zero rows. Rows go first and bytes after
— promote's ordering run backwards — so a row's existence keeps meaning "the
prefix is complete" and a crash in between leaks objects nothing points at,
never a live 404. ListReleases and PruneReleases now share one order
(created_at DESC, rowid DESC): created_at is second-granular, and the menu a
caller sees must be exactly the set retention keeps.
Tests, all four mutation-proven (each fails when its guard is removed):
retention bounds the set, the live release survives at any depth, activation
refuses reclaimed bytes with 410, and prune never deletes the row under a
concurrent activation (200 rounds, -race).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GH_PAT buys rate limit, not access — every repo the sync reads is public. So a
token that has expired or was minted with the wrong scope should cost throughput
and nothing else, but a 401 was being reported as a failed source, and a failed
source on the FIRST pass means an empty corpus. Retry that one case anonymously.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The loopback harness booted --enable=iam,base,kms,marketing,notify, so
/v1/billing/* was never mounted and a balance read answered an honest 404 —
nothing local could say whether the prepaid surface worked. billing joins the
list; its commerce upstream stays unconfigured on a throwaway instance, which is
the state the spec asserts against rather than around.
The seed also set enableSignUp:false on both applications, so signup answered
"the application does not allow to sign up new account" and self-service was
untestable locally. It is the flow under test, so the fixture enables it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The audit that fixed the tier table was itself handed "/v1/cloud is shipped with
29 live routes". It is shipped, but it is 5 routes. 29 is what a string-prefix
grep returns, because /v1/cloudflare (24 paths) shares the prefix and is an
unrelated product with an unrelated owner.
The product axis is the path SEGMENT after /v1/ (openapi.Product), so measuring
it as a string is off by a whole neighbouring plane — which is how a wrong
number reaches a doc while looking like it was measured. One line, so the next
person counts the same way twice.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The defect: three planes that are SHIPPED in production were documented as
unbuilt. /v1/cloud read "In flight (branch feat/cloud-account-connectors;
blue-held for red)", /v1/connectors "In flight (branch feat/connectors)", and
/v1/channels "Planned (branch feat/channels reserved; no transport code yet)".
All three branches are gone, clients/venue + clients/channels are on main and
wired into apps.Wire(), and api.hanzo.ai serves 5 / 8 / 8 operations for them.
The reason it rotted: the Tier column cited a BRANCH. A branch name is a place,
and the place stops existing at the moment the claim becomes true — so the row
degrades to "not built yet" exactly when it should read "call this". That is a
doc that actively causes the rebuild it exists to prevent. Tier now cites the
value instead: a route in GET /v1/openapi.json, which the deployment answers.
Also corrected against that live spec:
- /v1/compute/bots was attributed to clients/bots. clients/visor registers it
(visor.go); clients/bots owns /v1/bots. The table was making the exact merge
the file's own "Bot is three values" section exists to forbid — the one that
already shipped visor's machine list as the console's run list once.
- The compute row said "/v1/gpus + fleet | BYO GPU presence". clients/visor
serves six route families (machines, gpus, fleet, clusters, k8s, compute,
33 ops) and none of them are under /v1/visor.
- /v1/platform/* was documented as 500ing on a missing co-resident IAM store.
It is served, 29 paths, and answers the ordinary gate: 403 X-Org-Id required.
- The bijection was quoted at 983 operations / 692 paths / 109 products. Live
is 1467 / 1064 / 167. openapi_test.go LOGS that size and pins only the
bijection, so the number was never an invariant; the doc no longer quotes one.
- IAM's row carries no op count. /v1/iam/* is a catch-all proxy, so a count
there is a lie by omission.
Records the ONE way to fold a cloud account's k8s clusters into the fleet
(link -> POST /v1/cloud/{provider}/accounts/{label}/sync -> GET /v1/clusters,
discovery ending at fleet.Register where visor.attachCluster ends), venue's
per-org KMS custody path beside integrations' per-user one, and that visor is
an agent name rather than a query surface: the only live /v1/visor route is the
health probe serve.go auto-mounts for every subsystem without OwnsHealth.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The official marker was settable only at create, so the 60 examples already
live at <slug>.hanzo.app could never carry it — the label would have applied
only to apps published from here on, which is the wrong half. PATCH honors
official under the SAME one rule as create (SuperAdmin only; a tenant asking
for it on its own app is ignored), and un-badging is the same field, so a
mislabel is correctable rather than permanent.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
EnsureSystemAuthor INSERTed an approved system author and, on conflict,
returned whatever row was already there. On any deployment whose maintainer
org had once connected by hand — api.hanzo.ai is one; its hanzo author has
sat at status "connected" — that row stays CONNECTED, and a connected author
never accrues. "Pay ourselves" was therefore silently dead on exactly the
deployments that had used the manual flow first, with no error anywhere to
say so.
Ensure now means ensure: an existing CONNECTED row for the maintainer org is
promoted to approved. Only that one transition — a negotiated share_bps
stands, and a SUSPENDED author stays suspended, because un-suspending is an
operator decision and not a side effect of someone else's deploy.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A credential-less write could reach FULL CaptureEvent capability through two
doors that never consulted the anonymous lane:
1. captureTenant (capture.go) ended in a cloud.BrandForHostOK(Host) fallback,
so an unattested POST to /v1/analytics{,/batch}, /v1/tracker or
/v1/insights/e on a recognized brand Host was attributed to that brand's
REAL org — 'hanzo', 'lux', 'zoo' — with revenue, productId, quantity,
personId, groupId and the whole property bag, 500 per batch, unrated,
no DNT gate. Anyone on the internet could inject revenue and order rows
into a brand partition read by /v1/analytics/overview, /top,
/v1/analytics/campaign and the GTM funnel (clients/guide).
2. The published-site carve called the full-capability core with ZERO
credential and the site's real org, so the same injection worked against
any customer's org by setting a Host header.
Root cause: the "did this caller present a credential?" decision was written
once per DOOR instead of once per TRUST LEVEL, and ingestBody/eventWithOrg/
captureWithOrg/insightsWithOrg accepted full capability for any plain string
org — a value a door can synthesize from a header.
Fix: one decision in one place (handle). A door now supplies only its WIRE (a
decode) and its origin tag; capability is not a parameter, so no door can ask
for full capability and none can forget the decision. The four functions that
took an org and wrote unprojected are deleted, so the alternative is not a
check to skip — it is not expressible. ingestDecoded now has exactly two
callers: handle's credentialed branch and publicIngest.
admitPublic is decomplected into WHAT (the projection) and WHERE (the door's
org): it now takes no *zip.Ctx and no org at all. The published-site carve
calls publicIngest directly with the resolved Site.Org, which is the honest
shape — sites.Middleware runs BEFORE the identity boundary (serve.go: sites
241, IdentityMiddleware 267), so c.User()/c.Org() are raw client headers there
and nothing on a site host can be vouched for. A site's own pageviews still
land in the site's org, under the projection.
Also: isAnalyticsPath prefix-matched /v1/analytics, so it swallowed the
/overview, /timeseries, /top and /health read lenses and left the carve's POST
check as their only protection. It is an exact set now.
Preserved: a presented-but-unresolvable credential is still 403 on every door,
never downgraded (it was silently attributed to the brand org on the aliases).
A validated principal or resolvable key keeps full capability everywhere.
Tests: anon_capability_test.go proves the attack at every door. On origin/main
6 of its 12 tests fail; all pass here. Also fixes a pre-existing data race in
TestFanOut_PublicTenantNeverReachesDestinations (unguarded slice shared with
the fan-out goroutine), which failed under -race before this change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Global was stated TWICE. cloud.Global(fn) returned a MountFunc that asserted
Router was *zip.App and, when it was not, returned an error telling you to also
set Global: true. Wrapper and flag were the same fact — exactly 6 and 6 — and
the wrapper could not work without the flag.
Now the field IS the grant: MountSpec.App takes func(*zip.App, Deps) error, set
instead of Mount. `{Name: "authz", Mount: cloud.Global(authz.Mount), Global:
true}` becomes `{Name: "authz", App: authz.Mount}`. MountAll refuses a spec with
both, so scoped-or-global stays a decision someone made in writing.
The adapter, its type assertion and its error message are deleted, and with them
two tests: mounting a plugin on a scoped Router and forgetting the flag are no
longer failure modes to check, they are unrepresentable. gen-app-cmds reads App
as it reads Mount — the expression naming the registrar — so the 108 standalone
mains regenerate unchanged in meaning.
Also gone: `make plugins` and the TMPDIR it forced. That target linked 100+
binaries back to back, which is what exhausted a tmpfs /tmp and what OOM'd a
128GiB box — so it built sequentially at -p=2. The multi-call binary replaced
the set it built: `ship` links two things, not 108. Monolith and plugin are not
two artifacts to pick between, they are one artifact under two invocations
(direct, or as a child with --enable), which is why there is nothing to keep in
sync. Measured numbers replace the stale ones: 4.41GB across 108 dedicated vs
196MB for the one, 23x.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The first cut shipped PUT /v1/catalog behind principal.IsSuperAdmin. That gate is
correct and unusable: SuperAdmin is human-only here (no machine identity can hold
the admin org), so the only way to keep the catalog fresh would have been to mint
a second fabric credential and hand it to a cron. A capability nobody can
exercise is not a feature, and a new shared secret to make it exercisable is a
worse answer than not having the endpoint.
So there is no write door at all. The platform reconciles its own catalog from
the two places the truth already lives: the public repos of the source orgs (what
we built) and the live sites in the projects store (what is running). Both are
read in-process, on an hourly timer whose first pass is delayed so a boot never
waits on the network, and a failed source keeps the last good corpus rather than
pruning it to empty — a GitHub outage must not empty the catalog.
That collapses the tenancy question to one rule with no gate to misconfigure:
a public repo is public by definition, our OWN org's live sites are published
(they are the demos a visitor is meant to fork), and every other org's live sites
land in that org's corpus, which no other tenant ever queries. TestSyncRoutesSitesByOrg
asserts the routing directly, because that is where a customer's project would
leak, and TestNoOneCanPublish asserts every non-GET verb is unroutable.
Archetype is derived from the repo's own words, matched on whole words — "vm"
must not fire on "vmware". A wrong archetype is worse than none: it hides the row
from the browse rail instead of merely failing to file it.
projects gains LiveSites, its only cross-org read, returning the four facts a
catalog row needs and nothing that makes a project a tenant's business.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The creator loop had three holes that all reduce to the same thing — the
marketplace could not say WHO made something or WHERE the money went.
1. Fork only knew about the embedded catalog, so the examples people can
actually browse at <slug>.hanzo.app were not forkable. seedFrom now
resolves a catalog template FIRST (a curated slug is a stable public name
and must keep meaning what it means) and otherwise the unique live owner
of that slug — the SAME resolution the sites edge uses to serve it, so
what you can browse is what you can fork. The child inherits the parent's
repo and framework, never its deployed bytes: releases are per-tenant by
design, so a fork publishes its own.
2. A fork recorded no ancestry, so attribution had to be reconstructed by
guesswork. Project.ForkedFrom captures the parent the fork actually
resolved ("<org>/<slug>", or a template slug) at create time. It is
json:"-" on the request: lineage is a fact the server observed, never a
claim the caller makes.
3. First-party example apps were indistinguishable from community
submissions. Project.Official is the machine-readable half of the "Hanzo
Example" badge, and it is admin-only: a tenant asking for official:true
simply gets false. A fork never inherits it — a fork of our example is
the forker's app.
On the money side, the platform's own cut was implicit (10000 − shareBps)
and the destination of a payout was re-derived from the current
maintainerOrg. Payout.Settlement now RECORDS treasury/wallet/cash at
reservation time, and the basis publishes platformShareBps beside shareBps
plus settlesTo. That matters most for the seeded first-party creators: their
royalty settles into our own reserve, and the books have to say so rather
than let a house settlement read as an independent creator's earnings.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Everything the fleet has built was findable only if you already knew where to
look: hanzo repos on GitHub, lux repos in another org, deployed demos as rows in
the projects store, starter kits in an embedded JSON. Four places, four shapes,
no single question you could ask.
This adds the lens, not a search service. The corpus lives in clients/index —
the same store the Meilisearch dialect serves — so relevance, paging, encryption
at rest and persistence are the ones the platform already runs. catalog owns no
store. It owns the ONE thing a per-org index cannot express: a corpus that spans
orgs.
Cross-org is a SECOND corpus, not a weaker filter. The published catalog lives
under "~catalog", a name SanitizeIdentity can never mint (IAM org slugs begin
with an alphanumeric), so every authenticated caller reads it and no principal
can write it. A tenant's own entries live in their own org's `catalog` index and
are read with principal.Org and nothing else — a request field never selects an
org (HIP-0026). TestPrivateProjectNeverLeaks pins all three cases: the owner
sees their row, another tenant does not, an anonymous caller does not, and all
three still see the published corpus.
index gains Query's mirror: Reconcile, a full-corpus swap. A full swap rather
than incremental writes because the catalog's truth lives UPSTREAM — a re-run
must converge and a project deleted upstream must leave, the same prune-on-index
contract the code index already keeps. Store.PKs reads keys, not documents, so
the set difference never pulls a corpus into memory.
Facets ship on every response because browse and search are the same request: a
query with no q IS the browse.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
CLOUD_PUBSUB_ENABLED and CLOUD_KAFKA_ENABLED were the levers of a staged
cutover that is over. Production already sets both to true, the standalone nats
StatefulSet and the pubsub App are retired, and clients/webhooks (stream
COMMERCE) and the Kafka wire adaptor have nowhere else to go — so an unset env
did not mean "a deployment that opted out", it meant a cloud with no messaging
at all. Both gates and the subsystem_pubsub_active flag row are gone. Where the
plane listens stays configurable; whether it exists does not. A port collision
is answered by moving the port, never by running without the plane.
Making it unconditional put weight on a fail-closed promise the code could not
keep. psembed.Open calls ConfigureLogger, which installs the NATS logger whose
Fatalf exits the process, and the accept loop hits a bind error before
ReadyForConnections is consulted — so on the likeliest failure, something
already holding the port, Open never returns and Mount's error branch was
unreachable. The process just vanished: no cloud log naming the subsystem, no
shutdown of what had already mounted, and nothing a test could assert (the test
binary died with it). Mount now claims the address itself first and returns the
error it documents. That cannot close the race completely, so the library's
exit stays as the backstop rather than being papered over.
Tests match what the plane now promises unconditionally: a JetStream round trip
proves it carries a payload rather than merely opening a port, and the two ways
it can fail to serve — the port is taken, the store dir is unusable — each
assert Mount returns an error instead of continuing. Both were confirmed to
fail without the fix.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The staging gate runs org.ProbeCAS once. That is not enough to decide
anything, because the verdict is not stable: in one production pod, three
seconds apart, the host process logged
durability disabled — ... update-race admitted 2 writers (want 1) — If-Match not atomic
while the co-resident o11y plugin logged `durability enabled ... atomic_cas: true`.
The probes do not collide (probeKey mints a random key each time), so one of
those two verdicts is wrong about the store, and a single run cannot say which.
This runs the real ProbeCAS in a loop and reports a rate. Against production
s3 4.34.6 and against a local build of hanzoai/s3 main:
ProbeCAS ×300 against org-db: non-atomic verdicts=87 (29%)
ProbeCAS ×150 against a local build: non-atomic verdicts=55 (37%)
So the store genuinely fails, often, and the fence disabling itself is
correct. It also localises the fault: an S3-level race harness driving the
same gateway with aws-sdk-go-v2 stayed clean over 15,700 conditional PUTs.
The difference is the payload signature — minio-go over plain http sends
STREAMING-AWS4-HMAC-SHA256-PAYLOAD, and only that path over-admits. See
hanzoai/s3 test/s3/cas for the measurements from the other side.
sync.Once cached the FAILURE too. A lazy plugin first resolves on its first
request, which can be minutes after boot — so one unreachable-index moment
while the network was still coming up disabled every plugin for the life of the
process, and the symptom was 'no instance running', not a network error.
Observed doing exactly that in a live run.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Zero importers, not in the monolith's dep graph, and it mounts /v1/code/sessions
while the code app owns /v1/code and does not serve it. If that surface is
wanted it belongs inside the app that owns the prefix, not in a package nothing
links.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
clients/visor and clients/functions called ActivitiesPageForOrg,
CancelActivityForOrg and View.DescribeActivity — all on tasks main but never
tagged, so the module graph could not build them and 'make ship' died on the
monolith. Tagged tasks v1.52.1 and pinned it.
zip v1.17.4 is the one-Reload release; clients/plugin used both old names.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
remote() returned URL+Sum with no Args, so it could only ever name a dedicated
artifact — the multi-call path existed on disk and was unreachable over the
network. It now mirrors pluginIn exactly: dedicated wins, else the multi-call
binary with --enable=<app>.
Measured, 108 apps built today: dedicated total 4.41GB (min 36MB, median 37MB —
that floor is the core every plugin links), multi-call 196MB, host 18MB. One
artifact instead of 108 is 23x less to publish, fetch and cache, and the digest
is shared so the first app to start warms every other.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
PluginsEnv->Plugins, fromRelease->remote, loadIndex->fetch. manifest.Plugins
already says plugins. Comments cut to what the code does not say.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The resolution ladder ended on disk: ADDR, BIN, a dedicated binary beside the
host, the multi-call binary beside it. Every rung assumes the image already
carries the code. That is what makes "100% plugin" impossible to actually ship —
the plugins are in the image, so upgrading one means shipping a new image.
Adds the rung below those: CLOUD_PLUGINS names a release index and any app not
found on disk resolves to zip.Plugin{URL, Sum}. zip verifies the download
against Sum before the file is ever executable, and caches by that digest — so a
restart costs no download and a rollback to a previously run version is free and
offline.
The index format is hanzoai/ci's binaries.json verbatim
({repo,tag,binaries:[{name,os,arch,url,sha256}]}), not a second format invented
here. CI already writes it beside the artifacts it publishes, so the bits and the
digest that authorize them cannot come from different releases, and S3 or a
GitHub release are the same file at different URLs.
Placement is deliberate and purely additive: on-disk still wins, because a binary
shipped beside the host is what that host was built with, and an explicit
ADDR/BIN outranks everything because naming a binary and silently fetching a
different one would be a lie. An image that carries its plugins behaves exactly
as before; an image that carries none now works at all. A missing or unreachable
index is not an error — it falls through to the same failure as today, naming the
path a developer expects to have built.
The package still imports nothing but zip and the standard library, and the
index is fetched once per process: 108 apps must not become 108 requests.
Tests: 10 new (platform match, empty image, eager, single fetch under 25
resolutions, wrong platform, missing digest refused, no index, unreachable index,
BIN and ADDR both outranking it); 15/15 pass in the package.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The per-tenant secret sync needed a dedicated `<org>-platform-kms` IAM
application whose credential is sealed at orgs/<org>/kms-auth/* — and creating
it was a runbook: call IAM's bootstrap upsert, seal two fields, retry the
deploy. Executed by hand exactly once (org hanzo, live now); code from then on.
kmsOrgIdentity gains a minter. When the sealed credential is ABSENT and
IAM_SERVICE_TOKEN is configured, EnsureOrgIdentity calls the SAME idempotent
bootstrap upsert the K8s operator uses (self-authenticated by the unified
service token), verifies the response honors the audience contract
(clientId == "<org>-platform-kms" — a surprise identity is refused, never
sealed), seals both fields, and returns. Every later call is a pure read.
Unconfigured stays exactly today's fail-closed pending.
Two things learned live are pinned in tests: an app minted without a signing
cert exists but cannot SIGN (its tokens 500 at the token endpoint — the upsert
now always carries "cert-<brand>"), and a seal failure must surface as pending
so the idempotent re-mint converges rather than rotating.
De-dupe underneath: the in-cluster IAM base resolution (the split-horizon
policy — Cloudflare 403s server-side POSTs to the public issuer) was inlined
three times (ai M2M, the KMS login broker, and this would have been the
fourth). It is IAMBaseURL now, stated once; the endpoint-specific URL
overrides stay with their endpoints. And "<org>-platform-kms" is derived in
exactly one place, KMSMachineClientID, next to the recognition side that
reads it back.
Negative coverage: IAM down seals nothing; a wrong service token is refused;
a surprise clientId is refused; unconfigured provisions nothing; the second
call never touches IAM.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The section claimed identity came from the kernel and that billing could not ask
for /svc/ai. Neither is true while the app name is read from argv, which the peer
writes. State the real property, the demonstrated bypass, and where the fix has
to live.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
SO_PEERCRED is kernel-authenticated for pid and uid. argv is not: execve takes
argv from the caller, so a process names itself. Exec'ing any binary with
argv[0]="billing" makes peerArgv return billing and the broker hands over
billing's bundle — and the root key with it. Verified: a binary named spoof was
granted billing's and then ai's bundle, logged as legitimate grants both times.
The scope still partitions credentials against accident, which is most of the
sprawl. It does not partition against a process running code of its own, and the
package doc claimed it did. A wrong comment about a security property is worse
than no comment, because the next person builds on it.
Fixing it needs the spawner — only the launcher knows which app it started as
which pid — so the honest thing here is to say so and name the two shapes that
would close it, rather than harden argv parsing and look fixed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The comment said the gateway needed a matching allow-rule, "coordinated
separately". It has one, and the whole chain now demonstrably works — which was
only testable once the host stopped 404ing /v1/sentry a hop earlier.
Recorded as the three responses that distinguish the hops, because that is what
makes a future regression diagnosable in one curl:
404 -> host never routed it
403 {"msg":"no validated principal"} -> principal gate wrongly caught it
401 invalid ingest key (text/plain) -> correct: reached the DSN verifier
Reads still answer the 403, which is the half that must NOT change.
The store requires env on every write and credz reads "default"; a section that
gave the path but not the env sends an operator to a 400, or worse to a bucket
nothing reads.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A Leaf that installs its service credentials but gets no data-plane key still
boots, and then fails at the first store open with a cek error that names the
file and not the cause. The reason belongs where it is known. This package
exists because a boot announced a key it had not installed; the same silence in
the other direction is the same bug.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The socket path was stored on the broker and never used — Close unlinks through
the listener, which owns the file it bound. State that is written and never read
is a claim about the design that is not true.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
notify v1.6.18 declared "github.com/hanzoai/cloud v0.1.0 // indirect" at
go.mod:116, which closed a module cycle back into this repo. notify never
imported cloud itself — the requirement arrived transitively through
hanzoai/base v1.3.0, whose mount.go:33 imports "github.com/hanzoai/cloud"
for the HIP-0106 unified-binary Mount() entry point.
base inverted that dependency in v1.5.x and notify main has tracked
base v1.5.8 for a while, but the fix was never tagged: v1.6.18, v1.7.0 and
v1.7.1 all point at pre-history-rewrite commits. hanzoai/notify v1.7.2 tags
the current main, so the requirement is simply absent.
MVS pulls base v1.5.7 -> v1.5.8 and tasks v1.51.4 -> v1.52.0 with it.
go mod graph has no notify -> cloud and no base -> cloud edge;
./clients/notify builds and its tests pass.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/kms/orgs/:org/secrets let a request NAME the tenant it wanted, so guard()
existed to reconcile that name against ctx.Org() — two sources for one fact, and
the same caller-selectable-tenant shape auth_identity.go already documents as a
defect. Every other subsystem derives the org from the validated principal.
This one does now too: /v1/kms/secrets, org from the token, nothing to forge and
nothing to reconcile.
The admin bypass goes with it rather than moving. I built the cross-org route
first, then found nothing over HTTP needs one: kmsreseal authenticates AS the org
with a per-tenant machine credential, so its :org was redundant with its own
token exactly like everywhere else. Cross-org access remains only IN-PROCESS, for
the component that already holds the master key. Less surface, and "admin may
address any tenant" stops being a property of a route everyone shares.
The org stays folded into the STORE path as /orgs/<org>/… — that is the
isolation partition, the role the `org` column plays in every table, and it is
what keeps one tenant from reaching another's records.
The red-team suite moves with it. Where a test forged another org's :org, that
vector no longer exists to defend, so it now asserts the stronger property: no
URL may name an org at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
One credential — CLOUD_KMS_MASTER_KEY_REF — and the rule that follows from it:
the process holding it brokers, every other process asks for the credentials of
the app it is, and the scope is a path built from the peer's kernel-attested
identity rather than anything the peer said. Names the boundary honestly (pod
for the data plane, app for service credentials) and the ordering that makes
cek's memoized key a bug when it is violated.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Anthropic accepted only static credentials (API key, setup token), so a Claude
subscription could not be connected at all. Add the Adopt + Refresh legs so one
connector id now serves all three flavours.
Adopt proves the bundle LIVE with a read (GET /api/oauth/usage), not a
rotation. Anthropic invalidates the prior refresh token on every refresh, so
refreshing at intake — the way openai.go does — would kill the operator's
still-live local Claude Code session the moment they connected. The read also
proves the token carries user:profile, which the usage plane needs and a setup
token lacks.
The access token lands in Secrets[0] alongside the API key rather than a second
oauth-only slot: a subscription access token carries the same sk-ant-oat01-
prefix the use-time header rule already keys on, so the one rule covers the new
flavour with no second branch and fresh() needs no change.
Rotatability was gated on the provider (p.Refresh != nil), which now
over-approximates — anthropic can refresh a subscription but not an API key.
Move the gate to the fact that actually answers it: custody holding no refresh
token. That is the caller's 400, not a 502 inviting a retry that can never
succeed.
Live-verified end to end against api.anthropic.com with a real Claude Max
bundle; the local CLI session survived intake untouched.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
I reverted this on an incomplete diagnosis. I had truncated the vet output and
read the first failing package as the only one, so I attributed the whole break
to my commit and pulled it.
With the full output: clients/search (0ac9a91e, not mine) fails on the same
three symbols - cloud.ZipApp, cloud.Bridge, cloud.Request - so the branch is
red WITHOUT my code too. The revert restored nothing and only removed a tested
surface.
One shared cause, not several: the typed-op migration is still uncommitted in
the working tree, and every subsystem converted to typed ops on this branch
depends on it. It goes green when that migration commits, for search and this
together.
The control plane is a typed-op surface, so it needs core.Admit, cloud.Request
and cloud.ZipApp. All three are part of the typed-op migration that is still
uncommitted in the working tree - at HEAD, core.OK is the old envelope WRITER
rather than a status string, and the other two do not exist. So the package
compiled green in the tree it was written in and red at HEAD, which took
./apps and therefore the cloud binary with it.
That is my error: I built against the working tree and verified against the
working tree, and a shared branch has to build from its own commits.
The work is preserved on feat/plugin-control-plane, where it is green in a
tree carrying the migration. It re-lands unchanged once the typed-op migration
commits - nothing in it needs redesigning, it just has a dependency that is
not on this branch yet.
Every subsystem reads its secrets from the environment — CLOUD_AI_API_KEY,
IAM_CLIENT_SECRET, driverName, ~50 names across 108 apps. That was one
environment to fill when they were one binary. They are child processes now:
zip spawns each with os.Environ(), so whatever the launcher holds every child
holds, readable in each one's /proc/<pid>/environ. What was actually happening
is worse — nothing was filled in at all, so a lazily-spawned plugin booted with
no data-plane key and no provider credential and served 503.
One process holds the root key. Every other asks it over a unix socket for the
credentials of the app it is. The bundle is installed with os.Setenv, so all 108
apps keep reading os.Getenv and not one of them changes — and a value set after
execve never appears in /proc/<pid>/environ, which is the kernel's snapshot of
the argument page as it was passed. Same interface, none of the exposure. The
root key is scrubbed from the launcher's own environment, so no child inherits
it. A client sends no name and no token: the broker reads SO_PEERCRED off the
connection and resolves the peer's argv through /proc, and checks it against the
manifest — there is no credential to steal because there is no credential. Scope
is the path the app's own name spells, so provisioning is a write to the store
and the set of credentials is data rather than a table in code that drifts.
Ordering is the same bug 6193862 fixed for the dev key, generalized: cek
memoizes the master on first use and the first keyed open is edge.New inside
BuildDeps, so a key installed in Serve is installed after the once has cached
"no key" — every later open fails while the log reports a key was active. Boot
is sync.Once-guarded and called from both entry points that precede every open,
so calling it twice is free and calling it late is impossible.
Boot never fails. Each posture that cannot serve is a value plus a reason in
Err(), because the honest failure is the first store open refusing to run
unencrypted, not a boot that dies before the logger is up and takes the reason
with it. A production build with no key and no broker still fails closed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A project is IAM's resource — /v1/iam/projects has owned its CRUD all along, and
platform's store already persisted no project row. But platform still exposed a
second door to the same resource, and /v1/run quietly minted one, so "who owns a
project" had two answers depending on which surface you asked.
Now it has one. The lifecycle routes are gone (console called neither), and
ProjectStore itself is read-only — List, Get, Exists — so the constraint lives in
the type rather than in a convention. /v1/run resolves the org's default project
and fails with a pointer to IAM when it is absent, instead of creating it: a
default project is part of what an org IS, seeded when the org is provisioned,
not by whoever happens to run a container first.
The one project route that survives is a READ, and it is not a duplicate: it
answers a question IAM cannot, which is how many platform apps live under each
project.
Deleting a project used to cascade platform's app tree from platform's own DELETE
handler, which only worked when the project happened to be deleted THROUGH
platform — the wrong condition, since the project was never platform's to delete.
That cascade is now a reaper (orphans.go): it reaps apps whose IAM project is
gone, whoever deleted it and wherever. It fails SAFE — an unreachable project
store reaps nothing, because "IAM is down" and "the project was deleted" must
never be the same signal — asks once per (org,project) rather than once per app,
and leaves the app's VOLUME behind, so a deleted project never silently destroys
a tenant's data.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A terminating pod leaves the live membership while it is still serving, so a
read taken on that pod omitted the pod that answered it - and a rollout
started there skipped the host running it. Both read as "not deployed", which
is the one thing a fleet view must never say about a host that is up.
peers() now guarantees self is in the set, in the one place both the read and
the rollout walk it.
The platform had three ways to ask "what is relevant": /v1/kb/search (vector),
/v1/index/indexes/:uid/search (lexical), and /v1/search-docs/* (a proxy to the
same two). Each took a different request shape and returned a different score
scale, so a caller had to know which store held the answer before it could ask.
clients/search is the one entry point. It owns no store; it composes the legs
already running and returns a single ranked set. What it adds beyond composition:
- PROVENANCE. Every hit carries which backend matched it, at what rank, with
that backend's native score. A fused ranking without this cannot be
explained or debugged.
- AN HONEST DEGRADATION CONTRACT. Every response reports every leg with one of
four DISTINCT statuses -- ok / degraded / disabled / skipped -- because
"never provisioned" and "provisioned and broken" are different operational
facts. A leg that is down yields the survivors' results plus the error, never
a silent empty. That silent empty is exactly how a vector-store credential
drift stayed invisible for five days behind a fail-empty /v1/kb/search.
clients/search/rank is the ONE rank-fusion implementation, deliberately a leaf
that knows nothing about documents so both the cross-corpus surface and the
per-corpus tiers inside clients/code can share it instead of keeping two copies
of RRF. RRF over a weighted sum because the legs score on incomparable scales
(a term-match count and a cosine similarity); ranks are comparable by
construction, so there are no per-corpus weights to mis-tune and a leg dropping
out leaves the survivors correctly ordered.
Query is the typed op (it projects to OpenAPI/MCP/CLI from one registration);
ForOrg is the same composition for a caller that established its tenant another
way. Team's transactor is the first such caller: searchFulltext answered with a
hardcoded empty result, so the SPA's search box asked and was told "no matches"
forever. It now calls ForOrg in-process -- same binary, no HTTP hop -- scoped to
the transactor token's verified org.
Also corrects clients/index's package doc, which documented a /v1/search/* surface
while the code has always registered /v1/index/*.
NOT WIRED UP YET, deliberately: clients/provisioning already registers
POST/GET /v1/search as a resource-CRUD noun and mounts first, so registering the
query surface there today would be silently shadowed (verified: first
registration wins). Resolving that means moving provisioning's resource nouns
under /v1/provisioning/*, which breaks a wire and is not mine to decide.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
clients/plugin was a SECOND plugin registry: a CLOUD_PLUGINS JSON manifest
mounting wasm/goa modules and reverse proxies, with an untyped GET /v1/plugins
listing them. Nothing in this repo, in universe, or in any chart has ever set
CLOUD_PLUGINS, so in production it mounted nothing and published a route that
reported the empty set - a second answer to "what is a plugin here" that was
always empty, and invisible to OpenAPI, MCP and the CLI because it was untyped.
It is now the control plane for the plugins the host ACTUALLY runs: the
zip.Load children the generated manifest declares. The manifest stays
authoritative for which apps exist and what they answer; the artifact's
SHA-256 is authoritative for version, because it is the only identifier that
cannot drift from the bits serving. This package invents neither.
Four typed ops, so they land in zip's registry and the OpenAPI document, the
MCP tool list and the generated CLI are projections of it rather than three
hand-written clients:
GET /v1/admin/plugins what each host is running, and drift
POST /v1/admin/plugins/:name/reload pin a version, or roll one back
POST /v1/admin/plugins/:name/enable
POST /v1/admin/plugins/:name/disable
A version resolves through the origin's binaries.json - the index CI already
writes - so the mapping from version to digest has one author. A url must
carry its sum: zip verifies the download before the file is executable and
keys the cache by digest, so a rollback to bits this host has run needs no
network at all.
Fleet rollouts are sequential and halt on the first failure. That is what
makes the halt mean anything: zip refuses to move traffic onto a replacement
that is not listening, and a parallel rollout would have started a bad build
everywhere else before that error came back. One at a time, a bad build
reaches exactly one host, which keeps serving the old version.
Peers are asked with the CALLER'S replayed credential, so each applies the
same SuperAdmin gate to the same principal - a fan-out cannot become a
privilege escalation. Every mutation is appended to the hash-chained audit
trail, and a deployment with no audit store refuses to mutate at all, the same
refusal a credit grant makes before it moves money.
cloud.Members exposes the membership the shard router already routes on, so
the fleet the control plane rolls onto is the fleet serving traffic. Two views
that could disagree would be worse than one that is occasionally stale.
The fleet's biggest remaining cost line was invisible: the board modelled what
volumes are PROVISIONED and what they cost, and had no idea what is inside them.
184 volumes bill 7,706 GiB; 162 of them hold 559 GiB. That gap is $684.70/mo —
the delete lever the board already had is exhausted at $0.00 reclaimable, so all
of the remaining money is here.
Fill comes from each kubelet's stats/summary, which every cluster serves
unconditionally (metrics-server is absent from most of ours). It is read LAST in
scanOne and its errors are DROPPED: no safety verdict depends on fill, so a
kubelet that will not answer must cost a metric, never a mutation.
The distinction the whole change turns on: a reading exists ONLY for a volume a
running pod has mounted on a node that answered. Everything else is UNMEASURED,
which is not empty. Volume.HasUsage carries that, the totals carry a
measured/unmeasured split so the fleet figure can be read as the lower bound it
is, and a volume nobody measured contributes nothing to either. Rendering it as
0-used/100%-wasted would have put fabricated savings next to a delete button.
Cost.WastedMonthly is deliberately NOT Cost.ReclaimableMonthly. Reclaimable is
money a button here collects. Wasted is money locked inside volumes that are in
use holding live data, and DigitalOcean can only ever grow a volume — so
shrinkRecipe states the copy-and-swap migration exactly, including the
volumeClaimTemplates immutability that forces a StatefulSet to be recreated
around it, and says plainly that this board will not run it. The suggested size
carries the same honesty: it is arithmetic on ONE instantaneous reading, this
board keeps no history and cannot see a growth rate, and the recipe says so
rather than letting an operator squeeze a chain node into the outage the advice
was meant to prevent.
Expand ships as the mutation that IS safe, through the one mechanism that is
complete for each owner: a claimed volume is grown by patching its PVC, so the
CSI driver resizes the device and then the filesystem and nothing is left
declaring a stale capacity. Volume.ExpandTo mirrors NodePool.ScaleTo, and the
shared completeness gate still governs it.
The per-node fan-out also needed client-go's default 5 QPS limiter raised: a
17-node cluster was already spending over a second queued behind it, and a lost
reading fails safe (unmeasured) — which is precisely why it could not be left,
since it would quietly shrink the measured set as the fleet grows. Measured
coverage went from 157/178 to 162/184 and a full scan from 5.1s to 2.6s.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`make ship` builds the release layout — the host plus the one multi-call binary
that serves all 108 apps — and the Dockerfile now copies /host beside /cloud.
That costs 19MB and duplicates nothing. The host links no subsystem, and its
children ARE /cloud (`--enable=<name>`), so the image carries the core exactly
once whichever way it is run:
/cloud every app in ONE process (ENTRYPOINT, unchanged)
/host every app as its OWN process, started on the first request
Shipping 108 dedicated plugin binaries instead would be 4.5GB, and none of that
is symbols — -s -w has been the default LDFLAGS all along and the binaries are
already stripped. It is 108 copies of the same core.
The default stays /cloud, and that is a measurement rather than inertia. Five
apps in one process: 166MB PSS. The same five as host + children: 388MB, because
each child pays its own Go runtime and its own BuildDeps. Lazy start means an
idle app costs nothing, so the two curves cross at a handful of CONCURRENTLY HOT
apps — which makes process isolation something to buy deliberately for a
subsystem that needs it, not fleet-wide by default. Flipping a deployment is now
a command change on the same image, so that choice no longer needs a rebuild.
Two things a deployment must handle before switching, both silent if missed, and
both written down at the ENTRYPOINT: the host does not bind the :9090 ops port
(serve.go leaves it unbound for a plugin — N children cannot share one port), so
a 9090 scrape must move or go; and the children need a writable directory for
their sockets, which as uid 65532 on a read-only rootfs means mounting one.
No image was built here — CI builds images.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
108 dedicated plugins weigh 4.5GB in ./bin. ~35MB of each ~40MB binary is the
core every other plugin also links, so that is 108 copies of the same code — and
it is already stripped (-s -w has been the default LDFLAGS all along), so there
is no symbol-table win hiding in it. The size IS the duplication.
There was never a second mode to build. `cloud.Serve(specs, nil)` honours
cfg.Enable, so `cloud --enable=dns` already serves exactly dns, and since 48103214
every entrypoint sharing Serve honours ZIP_ADDR. A child started that way is the
same process a dedicated cmd/dns would be: same Serve, same middleware, same
socket contract. So this adds no flag and no dispatcher — inventing `--app` would
have been a second way to say a thing the binary already says.
What was missing is one rung on the resolution ladder. Plugin() now prefers a
dedicated binary beside the host and falls through to the multi-call one with
Args ["--enable=<name>"] (zip.Plugin.Args, which zip already had). Both rungs
stay, because they are the two LINK modes of one contract:
developer builds the single app being edited -> dedicated wins -> 1.3s loop
release ships host + cloud only -> multi-call -> 221MB
Preference order is deliberate: a dedicated binary on disk is someone's explicit
intent, and if the release binary won instead the fast loop would silently serve
stale code. CLOUD_<NAME>_BIN still wins over both and is now honoured exactly as
given rather than being second-guessed.
Measured, all 108 apps served:
dedicated 4.5GB (108 binaries)
multi-call 221MB (cloud 212MB + host 19MB) — 20x smaller, and smaller than
today's monolith-only image, which ships the same 212MB with no host.
Proof: host from a directory containing ONLY host + cloud, GET /v1/dns cold
255ms / warm 0.8ms, child cmdline `/…/ship/cloud --enable=dns`, answering the dns
app's own 403 "a validated principal is required" — the app, not a host 404.
Runtime cost is a wash and slightly better in aggregate: five concurrent
multi-call children measured 778MB summed RSS but 374MB summed PSS (~75MB each),
because every child maps the SAME text and the kernel charges it once. 108
dedicated binaries share nothing.
Plugin() grew a pluginIn(dir) seam so the on-disk half is testable: a test whose
answer comes from os.Executable() can only ever see its own directory.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
EnsureCloudUsage ran CREATE TABLE hanzo.cloud_usage against a database it never
created. That was faithful to the ai/object original it was moved from, and the
omission was safe THERE: ai's copy only ever runs where the ai router has already
made the database. Cloud's read path has no such guarantee — on a fresh warehouse
these seven readers are the first writer to touch it, and CREATE TABLE against a
missing database fails.
clients/sbom already had the answer (createDatabase, then createTable), so this
is the existing pattern rather than a new one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cek memoizes the resolved master on first use, and the first keyed open is
edge.New inside BuildDeps. EnsureDevKey ran in Serve, 46 lines later, so the
once had already cached "no key": a pure-Go build with no KMS key configured
failed every subsequent open — the audit store among them, which is fatal —
while logging that a dev key was active. `make host` then could not boot a
plugin without a key in the environment, which is precisely what EnsureDevKey
exists to avoid.
It self-gates on a configured key and on a codec-linked build, so production
reaches neither the key nor the warning and still fails closed as before.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cmd/{analytics,ask,evals,leaderboard,link,rollingcap,usage} each linked
github.com/hanzoai/ai/object, and it cost them 1310-1314 packages against a
671-package core floor. All seven are now 672-677, with ZERO ai packages.
The 638 packages were buying a guaranteed error. aiobject.EnsureCloudUsageTable
execs through object.DatastoreExec, whose connection is opened only by
object.InitDatastore, which runs only inside aimod.Mount — and none of these
seven link the ai module, so the call ALWAYS returned "datastore: not connected"
and every one of the nine call sites silently took its failure branch. The
symptom was an honest-empty dashboard against a warehouse that was up. The next
line at each site already queried clients/datastore, whose connection IS live in
exactly these binaries.
So the DDL moves there — clients/datastore/cloudusage.go, verbatim, guarded by
Ready() and latching only on success, following clients/sbom's ensureTable. It is
deliberately a SECOND copy: ai keeps its own for the write path. A func var the
host injects is the obvious alternative and it is a seam that can never be wired,
because the whole point of these binaries is that they do not link ai. Two copies
of idempotent DDL against one table converge; a nil hook does not.
Two call sites were not the table at all:
clients/answer reached aiobject.Crawl for page reads. clients/websearch already
has a native Crawl4AI client against the same service, and answer already
imports websearch — so Crawl is now exported there and crawl() became its len==1
case. One dial path, one auth path, one decode path, and a duplicate client
gone. Net new packages: zero.
clients/rollingcap read aiobject.TierReader(), and THAT WAS A LIVE BUG: the
rolling AI-spend cap has been dead in every deployment. aiobject's tier reader
is a COPY clients/ai installs at ai.Mount; cloud's is the source, set by
wireTierReader in BuildDeps before MountAll ever runs. cmd/rollingcap never
links clients/ai, and in the unified binary apps.Wire() mounts rollingcap
BEFORE ai — so the copy was nil either way and Mount took its no-op early-out
on every boot. It reads cloud.TierReader() now and the cap is live.
Its other half needed a real seam, so cloud.RollingCapReader joins the four in
ai.go. SetRollingCapReader is the one EXPORTED setter there and the deviation is
forced: the other four are written by build.go/durable.go inside package cloud,
but this producer is clients/rollingcap, which sits above the edge. clients/ai
installs a TRAMPOLINE rather than a snapshot — it resolves the reader per call
— so mount order cannot silence the cap a second time.
Not done here: clients/admin/finance has the same import and is mid-edit by
another change. It is also the one app that would not reach the floor (2184 ->
1612 measured), because it independently pulls part of ai/object's closure.
Measured, CGO_ENABLED=0 go list -deps ./cmd/<app>:
analytics 1311 -> 674 leaderboard 1311 -> 673
ask 1311 -> 675 link 1314 -> 676
evals 1311 -> 673 rollingcap 1310 -> 672
usage 1314 -> 677 ai packages 25 -> 0 (all seven)
Tests: cloud, apps, clients/{rollingcap,usage,analytics,answer,websearch,
leaderboard} all ok. vet clean on the touched set plus ./ and clients/ai.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The gate asserts on the import graph, which is right — but the line it echoes
ran without CGO_ENABLED, so CI printed 317 while `make host` printed 316. A
one-package disagreement with no visible cause is how a real regression gets
waved through as "probably the cgo thing". Pin it to CGO_ENABLED=0, the same
setting the Makefile builds with.
The stale comment above it went with it: 316, not ~317, against cmd/cloud's
3108, not 3105 — both re-measured on this tree.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Follows the Makefile flip. Two claims in LLM.md were stale the moment `build`
changed meaning, and stale build docs are how a reader ends up linking 3108
packages by accident:
- "Both entry points stay — `make build` and `make host`" is no longer true in
the way it reads. `make build` IS the host; the monolith is `make monolith`.
- `GOWORK=off go build ./...` was offered as the module-mode incantation. It is
also the single most expensive command in this repo — 100+ binaries at
~4.5GiB each — so it is now spelled with a named target and an explicit
warning, which is the rule everyone here already follows in practice.
Numbers re-measured on this tree rather than carried forward: the monolith is
3108 packages and 212MB, linking in 9.5s with a fully warm cache at 3.8GiB peak;
the host is 316 packages, 19MB, 0.6s, 13MB RSS.
Also records why the Dockerfile still ships the monolith, because that is the
question the flip invites: 106 plugins weigh 5.3GB in ./bin against the
monolith's 212MB, each statically re-linking the same ~650-package root. The
image gets ~25x bigger before host+plugins pays off, so the shipped artifact is
waiting on that floor and not on anything in the Makefile.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`make build` linked all 103 subsystems into one binary, so editing one app
relinked every other app with it. Measured on this tree with a fully warm cache
that is 9.5s and 3.8GiB of peak RSS for a 3108-package link; from cold it is
minutes. It was also the FIRST thing the Makefile offered, which made the
whole-fleet link the default answer to "how do I build this".
build now means the light host: 316 packages, 0.6s, 19MB. The loop it belongs to
is two commands, and neither grows as the fleet does —
make build # the router
make plugin APP=wallets # the ONE app you edited — 1.3s, recompile + relink
plugin declares no prerequisites, which is the property that matters: it cannot
drag the host or the monolith along behind it. Verified by timing a real
semantic edit (a comment-only edit is not a test — the compiler drops comments,
the object hashes identically and Go returns the cached link in 0.4s).
run follows: it builds the host plus EXACTLY the plugins in RUN_ENABLE rather
than the whole binary. The host resolves a plugin as a file beside itself, so a
name in that list with no binary in ./bin is the one way it fails, and building
that same list here is what keeps the two in step.
The old target survives as `monolith`, last in the file and last in `make help`,
marked SLOW FALLBACK — typing it is a choice and should look like one. It is not
dead: the Dockerfile still builds ./cmd/cloud, and host+plugins cannot replace it
yet because the 106 plugins weigh 5.3GB in ./bin against this binary's 212MB.
Each plugin statically re-links the same ~650-package core, so the image gets
~25x bigger before the model pays off. Flipping the shipped artifact waits on
cutting that floor, not on this target. It is also the reference the plugin set
is checked against, so the comment says to delete it when neither is true — and
not before.
No CI or image path changes: nothing invokes `make build`. The Dockerfile calls
`go build ./cmd/cloud` directly and still gets the monolith it expects.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The mount logged "archive: <deps.VFS != nil>". deps.VFS is guaranteed non-nil by
contract — R-7, so no consumer nil-derefs it — and resolves to a fail-closed STUB
when no object store is configured. The predicate therefore could never be false.
It printed archive:true just now with S3_ADMIN_* deliberately unset and the
corpus storing nothing, which is the worst moment for a field to read as
reassurance. Whether the archive can store is not knowable at mount without I/O
on the boot path, and that is the hazard this same release just fixed. So the
field is gone rather than made more elaborate: a signal that cannot be false says
nothing, and printing it says something false.
alert-sink was 30 lines of Python in a ConfigMap, on a stock
python:3.12-alpine image, behind its own Deployment, Service and
operator CR — to answer one question: did the page actually land?
That question belongs to the observability plane, and the
observability plane already runs. POST /v1/o11y/alerts/:receiver
records the delivery; GET /v1/o11y/alerts/last replays the ring.
The receiver name is a path parameter, so adding an Alertmanager
receiver stays a config change and never becomes a deploy.
Receipt lines are byte-for-byte identical to the receiver they
replace (differential-tested against the live sink.py), because the
line format IS the interface — it is what an operator greps.
Registered before the /v1/o11y/* wildcard so the in-order match
gives it precedence.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every cloud pod stalled at boot and crashlooped. No log line, because the next
one only prints after the call that hung. Both v1.801.266 and v1.801.271 stalled
at the IDENTICAL point — 9 lines, ending at "finance ledger wired" — so the
rollback between them could not have helped, and did not.
The call is s3vfs construction's bucket-ensure. It was
`_ = v.ensure(context.Background())`: error discarded because nothing depends on
it (ensure is idempotent and every Put/Get/Delete retries it), but UNBOUNDED, and
running on the boot path before the process listens. When the S3 gateway's
authenticated path stopped answering — it still completed TCP handshakes, so this
never failed, it just waited — the whole binary waited with it, past its liveness
budget, forever.
Discarding the error made it look optional. It was not: the WAIT was mandatory.
Best-effort has to be true of how long you wait, not just of whether you care
about the answer. Bounded at 5s, an unreachable store now costs a degraded start
instead of the process — ensured stays 0 and the first real op retries.
Membership.Start had the same shape one line later, and could not be fixed the
same way: it runs the first refresh synchronously AND hands the same ctx to the
refresh loop, so a deadline on it would stop membership refresh for the life of
the process — a drained pod elected owner forever, much worse than a slow boot.
The bound wraps only the synchronous first refresh; the loop keeps an
undeadlined ctx.
TestNewS3VFSDoesNotBlockBootOnAHangingStore reproduces the exact failure — a
listener that accepts and then never answers — and is not vacuous: unbounded it
FAILS at 30s, bounded it PASSES at 5s (verified both ways).
alert-sink was 30 lines of Python in a ConfigMap, on a stock
python:3.12-alpine image, behind its own Deployment, Service and
operator CR — to answer one question: did the page actually land?
That question belongs to the observability plane, and the
observability plane already runs. POST /v1/o11y/alerts/:receiver
records the delivery; GET /v1/o11y/alerts/last replays the ring.
The receiver name is a path parameter, so adding an Alertmanager
receiver stays a config change and never becomes a deploy.
Receipt lines are byte-for-byte identical to the receiver they
replace (differential-tested against the live sink.py), because the
line format IS the interface — it is what an operator greps.
Registered before the /v1/o11y/* wildcard so the in-order match
gives it precedence.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud is one of 13 non-mirror repos in the hanzoai org, and non-mirror is the
gap between the two mechanisms that keep git.hanzo.ai current:
hanzoai/mirrors reconcile.py makes qualifying GitHub repos pull-mirrors on a
6h schedule and the server syncs them — but it is idempotent and leaves
existing repos alone, so a repo already present as a NON-mirror is skipped
forever.
.hanzo/workflows/sync-from-github.yml is the per-repo pull for repos that are
native-canonical and therefore must not be mirrors.
A non-mirror without the file has neither. Measured across the 12 non-empty
non-mirror hanzoai repos: every one carrying the file sits 0-1 commits behind
GitHub (one cron tick); the two without it, cloud and ci, sat 10 and 3 behind.
None was ever AHEAD, so nothing is lost by pulling — GitHub is in practice the
write side for all of them.
Pull, not push: GitHub org secrets are inert on this Free-plan org (proven — a
public repo receives GIT_TOKEN, HANZO_GIT_TOKEN and DOCKERHUB_USERNAME all
empty), so a GitHub-side push has no working credential. The forge's own secret
store works, and already holds GH_PAT.
Fast-forward only. hanzoai/id is 163 ahead / 150 behind right now and that guard
is what stops a divergence being papered over.
LLM.md said the final step patches the operator CR. It does the opposite:
clients/paas.releaseService REFUSES to patch an App CR that Hanzo CD reconciles
with selfHeal, because the patch is reverted on the next sync and the release
would look applied then silently roll back. It validates first so the refusal is
specific, and names the remedy — commit the tag to the manifest.
The consequence is the part worth writing down, because it reads exactly like a
broken pipeline and is not: a GREEN release ends with "release tag minted
(receipt for a pushed, smoke-passed image)" immediately followed by "release
failed … reached: tagged". The image is real and proven; it just has no declared
state pointing at it. Production moves when someone bumps tag: in
universe/infra/k8s/operator/crs/cloud.yaml.
That is not hypothetical — .267 through .270 accumulated behind it while prod
sat healthy on .266, which is how this got noticed.
AdminRoute was mounted on storeV1 with no token gate, so only IAMTokenRequired
ran. That resolves a USER; the marker IsServiceToken reads is stamped by
TokenRequired's service-token branch alone. No scheduled run could authenticate
on the catalog sync — every attempt 403'd, and the model catalog is empty in
production as a result.
The standalone has always passed it (api.Route hands AdminRoute its
adminRequired); this is that call made the same way. Same reason the
auto-recharge poke carries its own TokenRequired instead of relying on the
group's chain.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The comment warned that /v1/sentry 404s until the o11y runtime carries the
routes. The pinned v1.5.33 does (o11yapiserver/sentry.go registers projects,
issues, discover and the DSN-ingest pair), so that caveat was stale — and it
pointed at the wrong hop anyway.
What actually 404'd was the hop ABOVE: once o11y moved out of process the host
mounted only /v1/o11y, so the request never reached this registration at all.
Now that both are true the chain closes, and the comment names all three hops
instead of one stale one.
Three things the code cannot tell you, one of which was actively wrong.
WRONG: the "four different things, four names" passage — which exists precisely
so nobody confuses them — still named `hanzoai/crawl` as what fetches the outside
world. That service is gone; `clients/crawl` does it in-binary. A stale name in
the disambiguation section is worse than no section.
ADDED, the plugin seam: a subsystem can run as its own binary with nothing about
it changing, prefixes is variadic because one plugin owns several subtrees, and
the image must actually contain the binary — each of those is a bug that already
happened, and none is visible from reading a subsystem.
ADDED, the crawl corpus: why Fetch and Read are separate doors, and the four
things the key scheme is load-bearing for (hash the url, scope from the verified
principal, key by REQUESTED not final url, digest because sanitising is lossy).
Also that the SSRF guard is in the DIALER — a hostname check is TOCTOU and
redirects re-enter the dialer anyway.
Documented next to the code, per the standing rule: LLM.md, not a new file.
Follow-on to the prefix fix, same root cause one layer over. Routing the second
subtree made it answer; it did not make anything KNOW about it.
indexSubsystems reads MountSpec.Prefixes and falls back to the /v1/<name>
convention when it is empty. PluginSpec never set it, so o11y claimed exactly
one subtree in the boot index — which feeds /v1/admin/subsystems and the
per-request subsystem attribution tracing hangs off. Every /v1/sentry request
would have served correctly and traced as belonging to no subsystem, and the
admin board would have understated what goes dark when o11y does.
So the prefixes are stated once at the composition root and reach both readers:
zip routes on them, the index reports them. Two lists that can drift is how the
first half of this bug happened.
Not a middleware change: MountAll only builds a scope when !Global, and a plugin
spec is always Global, so Prefixes here is read by the index alone.
Root/apps suites: failing set is byte-identical before and after (15 root, 8
apps) — all the known macOS "no RAM-backed scratch for the pure-Go SQLCipher
codec" failures, unrelated to this.
The PaaS could already give a tenant everything a per-tenant service needs —
its own namespace, RBAC, quota, App CR and ingress — except a volume, so only
stateless apps could be deployed through it. A tenant index, or any database,
had to be hand-written as a cluster CR instead.
An app now declares `storageGb`. Zero is the default and means stateless: no
claim, no volumes on the CR, rolling updates unchanged. Above zero the deploy
ensures a ReadWriteOnce claim `<slug>-data` and mounts it at /data.
Declaring storage also declares `strategy: Recreate`, because the two are one
fact rather than two settings. A ReadWriteOnce volume attaches to a single
node, so a rolling update asks the new pod to mount what the old has not
released and the CSI driver deadlocks on Multi-Attach — the new pod sits in
ContainerCreating indefinitely. That failure is already written into
crs/search.yaml as a comment; here it cannot be forgotten.
The claim is created once and never patched or deleted. Both edits that look
reasonable are the ones that lose data: a shrink is rejected by the driver and
wedges the app, and deleting on app-removal would destroy a database on what
reads as "undeploy". Resizing and reclaiming are deliberate and separate.
Also unifies the two scan paths. scanApp and scanRunningApp each spelled out
the field list for the same query prefix, so adding a column compiled fine and
failed at runtime with "expected 28 destination arguments in Scan, not 27" —
which is exactly how this column was added. One list now, appScanDest.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/sentry/* has been 404ing in prod since o11y moved out of the binary. The
route was never missing — cmd/o11y registers it, the same MountO11y it always
did — but the HOST only mounted /v1/o11y, so the request was refused a hop
before the process that serves it.
zip.Load has been variadic since v1.16.1, and its own doc comment names this
exact case. cloud.PluginSpec was the only thing still narrowing it to one, so
the fix is to stop narrowing: prefixes is variadic and passes straight through.
The signature moves the plugin ahead of the prefixes to match zip.Load's own
shape — one call site, and the two now read the same way.
No validation added here. zip already refuses an empty prefix list by name, and
restating that rule would put it in two places that can disagree.
Worst failure mode of the old shape, and why a test now pins it: everything
looks fine. The host mounts, starts, reports healthy, and dark-holes a whole
public subtree. TestPluginSpec_MountsEveryPrefix fails on exactly the
single-prefix regression (verified by reintroducing it).
/v1/chat/completions 404'd on the light host — the single most-used route in the
product — because ai, zen and commerce were wired through helpers defined IN
package apps (mountAI, mountZen, mountCommerce). cmd/gen-app-cmds can only emit
a package-qualified expression, so all three fell back to the stub that links
every subsystem and got no manifest row: no lean binary, nothing for the host to
route to.
None of them can move into package cloud — hanzoai/ai imports hanzoai/cloud, and
zen reaches ai/controllers — but a SIBLING that imports both is legal, and that
is all package apps ever was to them. So each helper moves verbatim into its own
wiring package: clients/ai, clients/zen, and commerce's into clients/commerce
beside the in-process client it already had (one subsystem, one package, one
cmd/<app>). Their tests move with them. Wire keeps all three entries at the same
positions; only the expression changes.
The composition root now also STATES what two of them answer, because a walk
reads them wrong: commerce opens an app.Group("/v1") for the store/catalog/plan
bundle, which reads as a claim on the whole version root, and ai registers
through a module so the walk finds nothing and falls back to /v1/ai. commerce
declares the list it already keeps (commerce.Prefixes, the same one the
fail-closed 503 and the error-scope guard use) and ai declares the /v1 catch-all
its Wire comment already describes. Both are Global, so neither binds a
middleware scope: this is the manifest's answer and nothing else.
zen gets no row, and that is the honest answer rather than a gap. It declares
/v1 to install a Claim there, serves the requests naming a zen* model, and
c.Next()s the rest to ai. In one binary that fall-through is a route lookup;
across a process boundary the request has already left the host and nothing
comes back to try the next candidate, so mounting zen would take every /v1
request ai serves and 404 all but zen*. The generator names that shape — an app
that registers no route of ITS own whose every prefix a later app answers — and
drops it, saying so on stderr. Both halves are load-bearing: registering nothing
alone would catch pubsub, and being shadowed alone would catch storage (/v1/s3)
and tracker (/v1/tracker), where the EARLIER app is the real owner and wins the
router's first match exactly as it does linked in. zen* models are not served by
the host until zen ships inside the ai binary.
Three fixes the host surfaced:
- team declares /collaborator. The Team front derives both the Y.js WebSocket
and the snapshot RPC from COLLABORATOR_URL, so it is a ROOT route — but the
walk cannot tell it from the group-relative /account, /bots and /billing/*
siblings and must assume a group, or an app claims /bots for the whole fleet.
- catalogsync is eager. It is a pure bus consumer with no HTTP surface; lazy,
its process waits for a request that never comes and the loop never runs.
(pubsub and kafka were already eager.)
- the generator was non-deterministic. pkgStrings resolved a value built from
one declared in another file only when Go's randomized map order happened to
visit the dependency first, so the SAME tree produced two different manifests
— deploy's /v1/deploy/{login,callback,logout} appeared about half the time.
It now resolves to a fixed point, which is what lets CI regenerate and diff.
And apps.go stops keeping its own copy of where a plugin binary lives: where()
delegates to manifest.App.Plugin, the rule the host itself reads. The copy had
already drifted — it left Plugin.Name empty and did not fold "-" to "_", so
CLOUD_ZERO_TRUST_ADDR was unreadable from this side.
Proven on the light host with all 108 apps mounted: POST /v1/chat/completions
lazy-started the ai child (225ms cold, 0.7ms warm) and reached its router, which
answered 503 "no DB configured" — the ai module's own fail-closed, not a 404. An
unowned path still gets the host's 404.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cmd/cloud links every subsystem to serve any of them: 3105 packages, ~570MB,
minutes to link, and a relink whenever any one of 103 apps changes. cmd/host
serves the same API and links zip plus the generated manifest — 316 packages,
19MB, sub-second. It knows three facts per app (name, prefixes, eager-or-lazy)
and nothing about what an app does, so a subsystem changing rebuilds itself and
leaves the router alone.
Lazy is what makes the set affordable. Mounted eagerly, 108 apps cost 108
processes and 108 startup times for a fleet that is mostly idle; mounted lazily
an app nobody calls costs a route entry and a struct, and the cost moves to the
first request that needs it. The ones whose work is not request-driven say so in
apps.go's `eager` map and start with the host.
Two CI gates keep it that way, because both properties are silent when they
break: generated-current re-runs the generator and fails on a dirty tree (a
Wire() edit without a regenerate leaves the host routing to binaries that are
not there), and host-is-light fails if cmd/host's import graph ever reaches apps
or a clients/* package — one stray import drags the whole graph back and the only
symptom would be a slow build.
make host builds it; make plugins builds the apps beside it, sequentially and at
-p=2 because each link peaks in the GiBs. TMPDIR defaults to disk: the Go linker
writes its temporaries there, and on a box where /tmp is tmpfs a hundred links
back to back exhaust RAM.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
git.hanzo.ai has 2364 active push webhooks aimed at
https://api.hanzo.ai/v1/git/webhook. Every one of them fails: the forge's own
hook_task rows record {"status":404} and the live cloud pod logs 61 x
`{"path":"/v1/git/webhook","status":404}` in a 30-minute window. So a push to
the host we call canonical builds nothing, and only the GitHub mirror releases —
the exact inversion of "canonical git, GitHub is the mirror".
The door was removed in #365 on the premise that "there is no external git
server. git.hanzo.ai IS this binary's /v1/git plane". That premise is false:
`kubectl get deploy -n hanzo` shows `hanzo-git` running ghcr.io/hanzoai/git
v1.26.26 behind Service hanzo-git, a SEPARATE process whose pushes never touch
our receive-pack. `fireBranchBuild` cannot see them, which is why the route
logged zero hits — the requests were 404ing before reaching any handler.
So this is a THIRD transport onto the existing single-registrant seam, not a
second pipeline: the handler verifies, maps, and calls cloud.OnGitPush, the same
one clients/git/smart_http.go and clients/integrations/github_webhook.go fire.
It carries the FULL ref and the forge's own clone_url and stops there;
clients/platform stays the only place that decides what a push MEANS.
It is a build trigger, so auth is fail-closed by construction: the HMAC-SHA256
over the raw body is compared with hmac.Equal, verified BEFORE the body is
parsed, and an unset GIT_WEBHOOK_SECRET refuses every delivery rather than
trusting it — same shape as the GitHub webhook. The secret is already KMS-synced
into the cloud CR env (secretKeyRef hanzo-git-secrets/webhook-secret); nothing
new to provision. Headers are the forge's own X-Git-* family, which v1.26.26
sends alongside the vendor aliases.
Bot exclusion moves to ONE predicate, cloud.IsBotActor, next to GitPushEvent
where both transports can reach it. It had to learn a second spelling anyway:
GitHub suffixes App logins with "[bot]", our forge attributes workflow pushes to
`hanzo-actions`. Without it a release's own commit triggers the next release.
Tests are proven non-vacuous by mutation: neutering hmac.Equal, dropping the bot
guard, and narrowing the ref filter to refs/heads/ each fail a distinct test
(bad-signature 401 -> 204, two no-op deliveries build, tag push vanishes).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
bot-gateway was a TypeScript service holding the sockets that bot nodes dial
into. A node runs on someone machine and attaches; that socket is the only way
to reach it, so a rendezvous cannot be spawned per request. This is that
rendezvous, in cloud, at /v1/bot.
Org is part of a node identity. The TypeScript keyed nodes by id alone, so
isolation was a property of deployment — one singleton, per-viewer bearers,
careful caching. Here a lookup without an org does not compile, and a foreign
node returns exactly what a nonexistent node returns, because telling them apart
leaks that another tenant has it.
One Registry, one constructor. NewRegistry() with no options IS the
single-replica registry; WithCluster adds presence and peer forwarding. An
earlier draft had a second ClusterRegistry type, which meant a caller wiring the
concrete *Registry would compile, pass every test, and silently never claim
presence — pre-port reachability restored invisibly, and only at replicas>1.
Removing the second type removed the trap.
A closure cannot cross a replica hop, so the frame travels with an empty
correlation id and the owning replica stamps the one it minted. That also closed
a seam where a peer could make a replica write bytes of its choosing into a
node socket.
Policy runs on the replica holding the session, because that is the only one
that knows what the node declared, and a denial survives the hop so a refusal
reads the same wherever the node is.
Verified: go build ./clients/... ./apps/... ., go test -race ./clients/bot/...
The apps TestStarterCredit failures are pre-existing — they reproduce with this
work stashed.
Three reactors ride the lifecycle fan-out and only one of them was durable.
indexOnPush enqueued onto the embedded tasks engine; notifyLifecycle and
mirrorOutbound did their work inline in the subscriber. So a deploy landing
during a rolling upgrade lost its Slack notice outright, with nothing to retry
it — on the plane we want to use as the deployment dashboard.
Notify now enqueues onto the SAME engine. There is still no second async
system: the queue seam every reactor shares moves to reactor.go, so a reactor
contributes only what differs — its queue name, workflow, activities, and the
key for one fact. indexOnPush moves onto it unchanged and sheds 56 lines,
which is what pays for the seam; mirrorOutbound follows next.
Delivery is one activity PER CHANNEL rather than one activity looping over
them. That is what makes retry safe: durable execution replays a completed
activity from history instead of re-running it, so a retry cannot re-post to a
channel that already received the message. Resolving the subscriber list is
its own activity, because a store read inside a workflow would not be
deterministic on replay.
The workflow id keys on the fact — org, repo, kind, and the pushed commit or
the deployment id — so a redelivery is one delivery. A fact carrying neither
discriminator cannot be deduped and is delivered inline instead, because
reusing a constant id there would collapse distinct events into one execution
and silently drop notices.
Fail-soft is unchanged: before the engine is wired, delivery happens inline,
so notify is never dark. That path is what the existing tests exercise, and
they pass untouched.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip.Load starts a plugin with a private unix socket in ZIP_ADDR and blocks
in waitListening until that socket accepts. Serve ignored it and bound
cfg's :9653/:8080/:9090 instead, so the socket never appeared: the host
timed out after 10s, killed the child, and every request to the mounted
prefix answered 503 — not at boot, where a test would see it, but on the
first request to a lazily-loaded prefix (apps.where sets Lazy). That made
all ~115 generated cmd/<app> binaries unusable as plugins; only the
hand-written cmd/o11y worked, because it calls zip.Addr itself.
listenOn is now the one place that decides where this process serves, and
it honours the contract for every entrypoint that shares Serve. The ops
port is deliberately left unbound in that mode: liveness for the fleet
belongs to the host, and N children sharing one cfg would otherwise fight
over one :9090.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The store derives a secret's partition from the first two ref segments:
clients/kms store.go fileOrg returns segs[1] only when segs[0]=="orgs", and
returns the shared "_platform" slug for EVERY other shape — silently, with no
error. Two callers never matched that shape, so their material was written to
one file shared by all tenants:
wallets keyRef built "wallets/<org>/..." — every tenant's sealed secp256k1
signing key, under a comment asserting "the org segment is the hard
isolation boundary". It was not.
provisioning fmt.Sprintf("org/%s/...") — singular "org", one character off,
so every tenant's datastore admin password.
Both now build "orgs/<org>/...". _platform/kms.db is 4096 bytes (one empty
SQLite page, untouched since Jul 17), so nothing was sealed at the old paths and
no migration is needed — the boundary was simply never exercised.
scope_test.go asserted the buggy shape, which is why this survived: the test
pinned the defect as expected behaviour. Updated, plus custody_partition_test.go
proving each scope narrowing resolves to the owning org and that two orgs never
share a partition.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Hanzo's L1 serves one HTTP prefix, /v1 — same as Lux's luxd
(node server/http/server.go baseURL). /ext/ is gone.
Measured 2026-07-27:
https://api.hanzo.network/v1/bc/C/rpc -> 200 eth_chainId 0x9063 (36963)
https://api.hanzo.network/ext/bc/C/rpc -> 404
control /v1/bc/ZZZNOPE/rpc -> 404
go test ./clients/treasury/ -run Anchor -> PASS. (The other tests in that
package fail without CLOUD_KMS_MASTER_KEY_REF, unrelated and pre-existing.)
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
grep exits 1 when it matches nothing, and whether `set -e` kills a bare assignment
on that is shell-dependent. Verified bash-in-posix-mode reaches the guard; the
builder runs Alpine's ash, which is the shell that matters and the one I cannot
check from here.
Swallowing the status keeps the FATAL message reachable in both, so a derivation
that stops matching says why instead of dying at an assignment with no explanation.
Cheap insurance on the path that just cost five stalled releases.
Every cmd/<app> stub called apps.ServeSingle, which reaches apps.Wire() — so
serving ONE subsystem linked EVERY subsystem: 3253 packages and ~4.5GiB of RSS
per link. That is what makes a whole-repo build OOM this box, and it is why the
working rule here is "named targets only".
cmd/gen-app-cmds now rebuilds each app's ONE Wire() entry from that entry's own
source text and imports only that app's package. Wire() stays the single source
of truth — a field added there appears in the stub on the next run — while the
link collapses to the app's own graph (mean 3253 -> 825; wallets and translate
sit on the 652-package floor).
An entry falls back to the fat stub when an identifier in it is apps-local,
because a standalone main cannot name it. Two adapters were freed that way:
ctxShutdown became cloud.CtxShutdown, and mountMetrics now becomes
cloud.MountMetrics — taking cmd/metrics from 3253 to 653 packages.
The other three adapters stay in package apps, and not for want of trying:
ai github.com/hanzoai/ai imports github.com/hanzoai/cloud. Moving
mountAI into cloud is an import cycle, not a weight question.
zen its meter reaches hanzoai/ai/controllers and hanzoai/ai/object
for the served-usage row and the family-routing ledger: +1354 and
+810 packages into the core, through the same module.
commerce +527 packages into the core, and its adapter also composes
clients/account and clients/finance, which import cloud.
metrics moved because it is nearly free: cloud's core already carries 398 of
metrics' 399 dependencies, so every binary grows by exactly one package —
hanzoai/metrics itself. Making 111 binaries heavier to make one lighter is the
opposite of the point, which is why the other three stayed.
No subsystem became a plugin, and the measurement says none should for size.
cmd/cloud is 3253 packages, of which 2536 are immovable: cloud's own root plus
hanzoai/{ai,commerce,zen}, which the three adapters above pin in place. The
remaining 717 are spread across 103 apps so thinly that converting EVERY entry
cloud.PluginSpec can express verbatim — 35 of them — frees 55 packages, 1.8%.
The two that would pay cannot go: clients/deploy (253) answers 503 on rollback
without the in-process release seam clients/paas installs, which is what
cloud.ServiceReleaserRegistered() asks; clients/plugin (53) takes its prefixes
from a runtime manifest, not a finite set. The lever on cmd/cloud is
hanzoai/ai, not the app list.
cmd/cloud does drop 3253 -> 3099 (-4.7%), all of it from the dependency bumps
here: o11y v1.5.34 sheds s2a-go, gax-go and grpc-gateway, and zip v1.10.0 ->
v1.17.2 makes zip.Load's prefixes variadic — which plugin_spec.go now passes
through, so a plugin can own more than one route subtree. That closes the gap
the o11y entry documents: it owns /v1/sentry as well as /v1/o11y, and a
single-prefix Load silently 404'd the second.
Also here: clients/websearch adapts through zip.AdaptNetHTTPFunc, and
cmd/{cron,gitops,sign,connectorruntime} are deleted — absent from Wire() and
referenced by nothing, so they could never have run.
Tests: apps ok (incl. TestWireOrderMatchesFrozen), cloud builds clean,
gen-app-cmds regenerates byte-identically.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Adding the second plugin should not require editing this file, and with a
hardcoded `go build -o /o11y ./cmd/o11y` it would — which is the same shape as the
bug that stalled five releases, just one step later. A hand-kept copy of "which
subsystems are not linked in" is a second place to forget.
cloud.PluginSpec("<name>", …) in apps/apps.go is what MAKES something a plugin, so
that is what the build reads. One line at the composition root now produces the
binary, ships it, and needs nothing here. The binaries land in /plugins and the
final stage copies the directory's contents beside /cloud, because that is where
zip.Load looks — dir(os.Executable())+"/<name>" — so the COPY is generic too.
Two ways to fail loudly rather than at a pod's first boot: an empty derivation
(the grep stopped matching) aborts the image build, and a plugin declared in Wire
with no cmd/<name> aborts it by name. The failure this replaces was a container
that built fine and then died at mount with `fork/exec /o11y: no such file`.
An observability plugin linked the entire Kubernetes API machinery to obtain a
struct type.
cmd/o11y imports github.com/hanzoai/cloud for cloud.Deps. The root package imports
writerpin. writerpin.resolve() referenced kubernetes.NewForConfig and NewLeasePin
directly — so client-go, 400 k8s.io packages, was in the graph of anything that
touched Deps. The election is opt-in at RUN time (CLOUD_WRITER_LEASE, off by
default, correct at replicas:1) and was mandatory at COMPILE time. That is the
whole defect: a runtime choice paid for as a link-time dependency.
The abstraction was already right and only the packaging was wrong. Pin is an
interface with three implementations and exactly one of them needs a cluster, so
that one moves out: writerpin keeps Pin/Held/SingleWriter/ConsensusPin and gains an
Elector seam; writerpin/lease keeps the client-go half; cmd/cloud — the binary that
actually runs in a cluster — registers lease.Elect before Serve. Explicit
registration at the composition root, not an init() registry, same rule apps.Wire
already follows.
Measured:
before after
github.com/hanzoai/cloud 1130 pkgs 400 k8s 644 pkgs 0 k8s
cmd/o11y (plugin) 3073 pkgs 400 k8s 2732 pkgs 71 k8s
cmd/cloud 3253 pkgs 3254 pkgs 612 k8s
cmd/cloud is deliberately unchanged: it needs the lease, and the point was never
to remove client-go from the binary that uses it. The 71 that remain in o11y come
from the hanzoai/o11y module itself (pkg/community, pkg/o11y, pkg/query-service) —
upstream's dependency, not something this repo can drop.
A binary that registers no elector and is asked for a lease now says so:
"CLOUD_WRITER_LEASE set but this binary registered no elector". It does not quietly
run as a single writer it never chose — two writers double-open the SQLite stores
and corrupt the audit chain, so a silent fallback here is the exact failure the pin
exists to prevent.
The two tests that exercised resolve() and truthy() moved back to writerpin, where
those live; a new test covers the seam in all three states — unregistered, elected,
and an elector that fails.
The git/CI deploy path can't stream its artifact through the API (a real docs
export is ~170 MB against a 16 MiB BodyLimit), so CI writes to S3 directly. What
it wrote with was the BUCKET'S OWN long-lived access key, handed to each repo as
a secret.
That is one credential for one shared bucket whose only tenant separation is the
key prefix (blob.go: sitePrefix = "<org>/<slug>"). Every repo holding it could
overwrite EVERY org's site, the blast radius grew with each site onboarded, and
the key never expired and lived in as many places as there were repos.
The deploy call already proves who you are, so it now hands back the grant:
202 { ..., "upload": { url, fields, prefix, expiresAt, maxBytes } }
A presigned POST policy that is prefix-scoped (starts-with $key "<org>/<slug>/"),
expires in 30m, and is size-bounded per object. S3 itself refuses a write outside
the prefix — a server-enforced condition, not a convention CI is trusted to keep.
No standing secret, nothing to rotate, nothing shared between tenants.
Presigning needs no STS: it is a signature computed with the key cloud already
holds. That matters here because s3.hanzo.ai does not expose an STS endpoint (an
AssumeRole POST answers 405), so scoped temporary credentials were not available
without an infra change. This needs none.
The grant is minted best-effort and returned ONLY on the 202 that creates the
deployment — never stored, never replayed on a later read, so it cannot be
re-fetched after the build it belongs to.
`aws s3 sync --delete` had to go with the credential, so deletion authority moved
server-side where it belonged: CI reports its manifest as `keys` on /complete and
cloud reconciles the prefix against it. Removing a file is now a decision cloud
makes, not something a build script can do to a prefix on its own.
Guards, each negative-tested by removing it and reproducing the exact failure:
- trailing slash on the grant's key condition — without it a grant for
"acme/site" also authorizes the sibling "acme/site-other".
- trailing slash on the reconcile's list prefix — without it reconciling
"acme/site" deletes "acme/site-two"'s objects.
- empty manifest deletes NOTHING — a build that reports no files has almost
certainly failed to enumerate its output, and honouring that literally would
delete the whole live site. Stale file: recoverable. Deleted site: not.
- reconcile runs only on a LIVE completion, so a failed build cannot prune the
site the last good build is still serving.
Verified on Linux (these store/S3 tests cannot run on macOS — cek needs
RAM-backed scratch and isRAMBacked is false off-Linux): go build ./... and
go vet ./... clean; clients/projects, clients/sites, clients/platform and
internal/fqdn suites green. The S3 paths run against the in-repo fakeS3 double,
per the repo convention that live round-trips are proven with a controlled
backend rather than the production store.
Smoke has been failing with
cloud: mount: mount o11y: zip: Load(o11y): start: fork/exec /o11y:
no such file or directory
SMOKE FAIL: never reached listening
o11y is the first subsystem that is NOT linked in — cloud fork/execs it as a
separate process at mount time — so unlike every other subsystem it has to
exist as a FILE in the image. The Dockerfile built /cloud and /smoke and
never mentioned o11y at all, so the mount could not succeed and boot died.
This was invisible until now only because the release died EARLIER, at
`go mod download`; with the build unblocked it advanced to `reached:"built"`
and the next gate immediately caught this. The gate worked — the binary was
simply never added when the app landed.
NOT VERIFIED LOCALLY: `go build ./cmd/o11y` cannot run on this machine right
now — a dependency (zap-proto/zip v1.16.1) requires go >= 1.26.5 while
go.mod's directive is 1.26.4 (lowered in 4b517e90 to match the pinned
builder). That conflict is real and separate from this change: the directive
now sits BELOW what a dependency demands, so the toolchain cannot satisfy
both. Whoever resolves it should pick one source of truth rather than
meeting in the middle.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Cloud has not booted since o11y stopped being linked in. Every release smoke dies
at the same line, before listening:
cloud: mount: mount o11y: zip: Load(o11y): start: fork/exec /o11y: no such file
The unlink itself is right: deleting the clients/o11y import is what keeps its
2.7k-package graph — otel-collector, prometheus, gonum — out of /cloud, and
apps.Wire now loads it at run time over a private socket. What did not ship with it
is the other half. cmd/o11y/main.go exists and apps.o11yPlugin resolves the binary
as filepath.Join(dir(os.Executable()), "o11y") — /o11y next to /cloud — but the
Dockerfile only ever built /cloud and /smoke, so the path it fork/execs was never
in the image.
Built CGO_ENABLED=0 like /smoke: cmd/o11y touches no sqlite, so it needs no libc
and a static binary is one less thing that can disagree with the runtime base.
This is a boot-time abort, not a degraded feature, so it took the whole binary down
rather than just observability — which is why four consecutive releases produced an
image and none of them shipped. The pipeline behaved correctly throughout: the
smoke gate caught it every time and refused to mint a tag, so nothing broken ever
reached a cluster. Prod has been sitting on v1.801.259 while main moved on.
STILL OPEN, and it is written in the Wire entry itself: o11y also owns /v1/sentry/*
(mountSentry), zip.Load takes ONE prefix, so /v1/sentry/* is not mounted on the host
and 404s. That comment ends "Do not merge this to main before that is closed" — it
is on main. This commit does not close it; it only gets the binary into the image so
cloud boots again. The sentry prefix needs zip.Plugin to name more than one prefix.
Sites and apps bound custom hostnames two different ways. Apps had the full
DNS-01 flow: claim pending, publish a TXT token, verify, then serve. Sites had
none of it — setDomains bound immediately and was gated to a SuperAdmin or the
platform-operator org, so any other org got a flat 403 and the only route onto
a custom domain was to ask an operator. clients/projects/domains.go said so
outright: verification "is the planned path for any org to self-bind".
This wires it, over the internal/fqdn primitives both paths now share:
- operator/admin → BindHost, verified immediately. The operator manages
customer DNS, so its bind is itself the vouch. Unchanged.
- any other org → ClaimHost, pending, and the response carries the exact
records to publish. POST .../domains/{host}/verify proves
control and promotes it.
site_hosts gains status/token/verified_at. status separates HOLDING a name from
SERVING it: a pending row takes the name against the PK — nobody else can claim
it — but never routes.
THE BOUNDARY is one line: ResolveHost filters status='verified'. A pending row
that routed would mean claiming yourco.com is enough to answer for it. That is
the only routing read of site_hosts (ResolveUniqueLiveSlug resolves bare slugs
out of `projects` and never sees this table), so gating it there gates
everything. claim_test.go asserts it, and the assertion has teeth: removing the
filter fails TestPendingClaimHoldsTheNameButNeverResolves with exactly that
hijack.
Migration defaults status to 'verified'. Every row that exists when it runs is
already serving, so defaulting to 'pending' would take every live site off the
air on upgrade — TestExistingRowsStayVerified inserts a row the pre-migration
way and proves it still resolves.
Other invariants pinned: verification is owner-scoped (another org's verify is
errNotFound and does not promote); a re-claim keeps the ORIGINAL token so it
cannot invalidate a record the customer already published; no path walks a
verified host back to pending; an operator bind promotes the project's own
pending claim rather than colliding with it. Hosts under the platform apex stay
operator-assigned — no customer can publish a TXT record in a zone we run, so
there is no proof to offer.
ListHostsForProject is now the ROUTING view (verified only) — reporting a name
the edge will not answer for as a "domain" is a lie the customer debugs for an
hour. ListHostClaims is the ownership view that shows both.
Verified on Linux (these store tests cannot run on macOS — cek needs RAM-backed
scratch and isRAMBacked is false off-Linux): go build ./... and go vet ./...
clean; internal/fqdn, clients/projects, clients/sites and clients/platform
suites all green, including the six new boundary tests.
Also folds in a stray gofmt fix to fork_test.go that predates this change, so
the package is gofmt-clean.
I was wrong, and the revert was worse than the bug.
The claim was that 1.26.5 arrived by accident and nothing needed it. The first half
is true — bf678adfd's go.mod diff really is a bare directive bump from an unrelated
commit. The second half is not: github.com/zap-proto/zip declares go 1.26.5, and
the module graph takes the MAXIMUM, so `go mod tidy` puts 1.26.5 straight back.
Lowering it therefore did not unblock the build, it broke a working one. With an
inconsistent go.mod every `go build` fails immediately:
go: updates to go.mod needed, disabled by -mod=readonly
which is a harder failure than the release-only one I was trying to fix — that at
least left local builds and tests working.
The real problem is unchanged and is NOT in this file: ghcr.io/hanzoai/mirror/golang
is pinned by digest to a Go 1.26.4 image, that mirror holds exactly one tag whose
current digest IS the pinned one, and GOTOOLCHAIN=local stops the builder fetching
a newer toolchain. The builder has to move, not the directive.
Every cloud release has been failing at `go mod download` with
go: go.mod requires go >= 1.26.5 (running go 1.26.4; GOTOOLCHAIN=local)
The golang image ships GOTOOLCHAIN=local, which makes the image's own Go
the authority and refuses a newer `go` directive. So the Go version lived
in TWO places that had to agree by hand — the digest-pinned builder here
and go.mod — and they drifted the moment the directive went to 1.26.5.
Four consecutive release builds died this way (pf-runner-reltvcmieenw,
rell76rgm83c, relcuuikwfud, relhtmsk6vwj); the last green one was 69
minutes earlier. The failure is at the BUILD step, so `reached: "none"` —
no tag was minted and nothing shipped broken. Releases just stopped, and
because the only signal is a log line nobody was watching, prod sat on
v1.801.259 while main moved on.
GOTOOLCHAIN=auto makes go.mod authoritative and the digest a floor. The
toolchain lands in GOMODCACHE, which the download step already mounts as a
shared cache, so it costs one fetch per cache generation. Bumping the `go`
directive is now a one-line change that cannot desync.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every cloud release has been failing at `go mod download`:
go: go.mod requires go >= 1.26.5 (running go 1.26.4; GOTOOLCHAIN=local)
The directive moved to 1.26.5 in bf678adfd, a commit about the admin host view —
its go.mod diff is exactly `-go 1.26.4 / +go 1.26.5` and it carries no toolchain
directive. That is the signature of `go mod tidy` run on a workstation with a newer
toolchain, which rewrites the minimum silently. Nothing in the change needed it.
The builder is pinned by DIGEST (golang:1.26-alpine3.22@sha256:47d47cb…), so it
cannot float to a newer patch the way the tag would, and GOTOOLCHAIN=local stops it
downloading one — both deliberate, both the reason a pin is a pin. The build then
fails before compiling anything, which is why this reads as a dependency problem
and is not one.
Lowering the directive rather than re-pinning the builder: Go PATCH releases carry
no language or library changes, so 1.26.5 buys nothing here, and moving a pinned
toolchain digest to unblock an accident is a much larger change than undoing the
accident.
Two things, one seam.
FIRST — research and deep answers were reading NOTHING. The answer loop's read
stage called ai/object.Crawl, which dials crawl.hanzo.svc.cluster.local:11235, the
service that does not exist. That stage is explicitly allowed to degrade to search
snippets, so every research answer quietly fell back to ~600 characters per source
and nothing ever looked broken. It now reads through the in-binary crawl.
The rebinding let the workaround go rather than move: the goroutine that existed to
make an uncancellable batch call cancellable, and the sync.Once that kept the
abandoned goroutine from blocking on send, are both gone, because Fetch takes a
ctx. The loop's 90s bound and a client disconnect now reach the socket instead of
merely ending the wait while the request ran on unattended. Per URL rather than per
batch, so one slow page no longer decides when the others arrive and a page that
panics the parser costs its own slot, not the set. The collector alone writes the
result slice, so rank order survives without a mutex.
SECOND — pages are kept. A crawl is now a corpus: re-readable without paying the
network again, and there to index and build against later. It rides the ONE object
seam the binary already has (types.VFSClient over the shared S3 gateway) — no
second client, no second bucket, no second credential to rotate.
Fetch and keep stay separate. Fetch is the pure network primitive, which is what
makes it testable with no store; Read is the door — archive first, network second,
keep what came back. One door, so no caller has to remember to persist and no two
callers can disagree about where pages live. /v1/crawl, websearch's /scrape and the
answer loop all go through it.
Scoped by the VERIFIED principal, org and project, never the body — the scope
selects the key prefix, and a caller-supplied prefix is a caller reading another
tenant's corpus. In the answer loop the scope is the DATA org, not the payer:
filing a tenant's pages under whoever funded the answer would put one org's
research in another's prefix.
Two bugs found while writing the tests, both mine, both in the isolation boundary:
• The archive keyed by Page.URL, which is where a page landed AFTER redirects,
while callers only ever ask by the url they know. Every redirecting page would
have been filed somewhere unreachable — a cache that silently never hits for a
large share of the real web. The requested url is now threaded through and is
what the key is built from.
• seg() was LOSSY: "a/b" and "a-b" both folded to "a-b", so two distinct orgs
could collide onto one corpus prefix and read each other's pages. Sanitising
alone cannot be the identity function. It now appends a digest of the RAW value,
so the readable half stays browsable (the point of keeping a corpus) while
identity is carried by something that cannot collide.
Failed fetches are never archived — caching a failure would serve it back forever.
A store that is absent, down or slow costs a cache hit and nothing else; crawling
must not start failing because bookkeeping is down.
go test ./clients/crawl/... ./clients/answer/... ./clients/websearch/... passes,
vet clean. The apps StarterCredit failures are pre-existing on unmodified main (no
/dev/shm for the pure-Go SQLCipher codec on macOS) — verified by stash-and-rerun.
Two subsystems bind customer hostnames — /v1/platform apps (clients/platform)
and PaaS sites (clients/projects) — and each carried its own copy of what a
hostname is. The copies had already drifted:
- both compiled the same hostname regexp independently;
- only the apps path stripped the trailing root dot, so `example.com.` was
canonical for an app and rejected as malformed for a site;
- only the apps path bounded the name at 253 bytes, so an over-long name was
refused by one endpoint and accepted by the other.
internal/fqdn is now the single implementation: Clean, Valid, Challenge,
Records, Token, Proven, Verify. Both callers delegate; the sites path picks up
the trailing-dot and length handling it never had.
The ownership check is also decomplected. It was one function braiding three
concerns — a DNS effect, the security rule, and a customer-facing explanation —
returning (bool, string). It is now:
Proven(txts, token) bool the RULE: a pure predicate over the DNS answer.
No context, no resolver, no clock. This one
expression is the boundary that stops a tenant
claiming a host it does not control, so it is
worth being readable and exhaustible by table test
without standing up a fake.
Verify(ctx, r, h, tok) the EFFECT: read the challenge name, apply Proven.
Returns nil when proven, *ErrUnproven otherwise —
the explanation is the error value.
Behaviour preserved on the apps path, with two hardenings that were previously
unreachable rather than intended: an empty token is never proven (a zone
publishing an empty TXT record can no longer satisfy a claim whose token was
never minted), and a resolver error stays a single "not proven" path that
cannot be mistaken for success.
Also narrows the resolver interface to the one method used: LookupCNAME was
declared but never called outside a test fake.
Verified: go vet ./... clean; internal/fqdn tests green (the rule table covers
substring, case, whitespace, empty-answer and empty-token); clients/platform
and clients/projects build and their test binaries compile. The store-backed
domain HTTP tests need Linux /dev/shm (LLM.md: cek's encrypted stores run only
in the CGO stage) and are left to CI.
The forge stopped being Gitea a long time ago; these were the last places the
name survived. gitea.com came out of the default source-host allowlist and the
provider enum, and the detection branch went with it — a gitea.com URL now falls
through to the generic "git" provider, which is what every other self-hosted
forge already resolves to, so nothing loses the ability to be a source.
The generated openapi is regenerated from the same design edit rather than
hand-patched, so the two cannot drift.
Zero occurrences of the string remain in the tree.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Every board under /v1/admin/* aggregates an upstream. GET /v1/admin/plugins
does not — it asks the process itself. zip v1.15 gave a host App.Plugins():
which subsystems it loaded out-of-binary, at which artifact digest, whether
they are still up, and what each costs from /proc. Nothing read it.
The gap it closes is not cosmetic. A deployment manifest answers what was
INTENDED. Only the process knows what is TRUE, and during a rolling upgrade
the two disagree by design. The version reported here is the artifact's
SHA-256 — the one version identifier that cannot drift from the bits
running, because it IS the bits.
Restarts is the field the board exists for. Reloads are deliberate (someone
swapped the binary); Restarts is the supervisor bringing a plugin back after
it died on its own. Nonzero Restarts is a crash, not a deploy, and a climbing
one is a crash loop. That verdict is computed once, server-side (Crashed), so
no reader re-derives the policy and no two readers can disagree about it.
/v1/admin/plugins, core.Guard, registered beside /o11y and /aimetrics in
routes() — SuperAdmin only, like every platform read. Artifact digests, pids
and RSS name what is deployed and where the memory went; that is not a
customer-visible fact.
Honest in the two ways that matter. It is THIS replica's answer, so the board
stamps Host (Deps.Self, the same id the durability ring elects on) — one
pod's restarts presented as the fleet's would be worse than no board. And a
host with no plugins returns an empty list, never an error: "everything is
linked in" is true and expected, since cloud composes a linked-in service and
a plugin as the same type.
Router gains Plugins() next to Fiber(): a second named, read-only hole, and
strictly the weaker of the two — it registers nothing and mutates nothing.
The alternative was making admin Global, granting app-wide middleware to buy
a status field.
Carried along, because the bump requires them:
zip v1.11.0 -> v1.16.1 (Load takes the Plugin first + variadic prefixes;
AdaptNetHTTPFunc is gone — http.HandlerFunc IS an http.Handler)
hanzoai/o11y v1.5.32 -> v1.5.33 (same adapter removal, already fixed there)
selfID(cfg) is now the ONE self-id resolver; durability and Deps.Self read
the same expression rather than two that could drift.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The provider was built inside clients/o11y and handed to the host through an
installer that package registered from init(). That worked only while o11y was
LINKED INTO the host. Once o11y became a plugin its init() ran in the CHILD,
telemetryInstaller stayed nil, and installTelemetry became exactly what its own
doc called it — "a clean no-op". Tracing went dark fleet-wide, including the
gen_ai spans it adopts into the ai module, and nothing failed.
Every request the host serves needs a span whether or not o11y is co-resident,
so the provider is a host concern and now lives there. What stayed in o11y is
what is genuinely o11y's: the collector, the datastore exporter, and the
SDK-span -> pdata conversion, which is coupled to the o11y_index_v3 schema and
to nothing in the host.
One Send, two deployments. cloud.RegisterTraceSink is a typed seam
(func(ctx, []sdktrace.ReadOnlySpan) error) over the ZAP router. Registration is
process-local, and that is the mechanism, not a detail: linked in, o11y
registers here and the host's Send takes the Cost-0 leg with the live batch;
as a plugin it registers in the child, this router has no route, and the
identical Send falls through to the ZAP wire. Neither the producer nor the
exporter branches on where o11y lives.
Not routed over /v1/o11y/*: TracingMiddleware traces /v1/*, so exporting spans
through the host's own HTTP plane emits spans about exporting spans, and would
put every batch through audit, rate limiting and the billing gate.
The ai adoption (aiobject.AdoptHostTracerProvider) moves to apps/install.go
beside the other aiobject.Set* calls, gated on cloud.TracerProviderInstalled()
so a disabled bootstrap never latches ai "ready" against the global no-op.
apps/ already links ai; the host must not.
Also fixed while owning the lifecycle: the meter provider was installed and
never shut down, so the last interval never flushed. And cmd/o11y now calls the
same bootstrap — a plugin is a host for its own requests, and it was serving
them with the no-op provider.
Cost to the host root graph: 644 -> 650 packages (the OTel trace SDK and
luxfi/trace). k8s.io, aws, grpc, hanzoai/ai and go.opentelemetry.io/collector
all remain at 0.
Proof, not assertion:
- TestTracing_EndToEnd_WithO11yLinkedIn (clients/o11y) emits a span on the
process-global tracer with o11y linked in and asserts it arrives at o11y's
sink as pdata carrying service.name and deployment.environment. Deleting
otel.SetTracerProvider makes it RED: "o11y's sink consumed 0 batches".
- cloud/telemetry_test.go pins the Cost table both ways: co-resident sink wins
over the wire, no sink falls back to the wire, neither surfaces ErrNoRoute.
Tests: cloud ok, apps ok, cmd/cloud ok (boots o11y as a real plugin child).
clients/o11y has 5 pre-existing TestAnnotationQueue* failures (cek requires
CLOUD_KMS_MASTER_KEY_REF); identical before and after, proved by stash.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
- platform: rolloutRelease had two writers (CR patch + a universe
repository_dispatch mirror) composed best-effort, so the step passed if
either landed and hid their disagreement. The mirror was never running
anyway -- UNIVERSE_DISPATCH_TOKEN is not set on the deployment -- so this
deletes a phantom, and a rollout with nowhere to write is now an error.
- cd+site: clients/projects already owns the versioned-release model for
sites (content-addressed rel_<digest> prefixes, atomic ActivateRelease,
servePrefix with legacy fallback, rollback routed). The parallel
clients/cd + clients/site lifecycle re-implemented it with a weaker
pointer and is deleted unmerged; its one real finding -- releases are
never garbage-collected -- is recorded in LLM.md.
# Conflicts:
# cmd/o11y/main.go
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The writer pin decides who may open the RWO stores. SingleWriter is correct only
because replicas:1 makes Kubernetes the elector; the moment a second pod exists
something has to choose, and be right, because two writers double-open the SQLite
stores and corrupt the audit chain's head.
LeasePin does that over coordination.k8s.io, wrapping client-go's leaderelection
rather than renewing a Lease by hand — the fencing rules (only an EXPIRED lease
may be taken, the holder stops the instant renewal fails, skew bounded by
RenewDeadline) are easy to write and hard to write correctly, and this is the
implementation Kubernetes itself runs on. ReleaseOnCancel makes a rolling restart
a handover instead of a LeaseDuration of downtime.
It is OPT-IN. At replicas:1 electing again over a Lease would add an API-server
dependency for no benefit — unreachable API, no writer, a real outage traded for
an imagined one. CLOUD_WRITER_LEASE=1 plus POD_NAME/POD_NAMESPACE arms it, and
every incomplete configuration falls back to SingleWriter and SAYS SO in the boot
log. A silent fallback is how a cluster ends up believing it elects when it does not.
Unsafe timings are refused at construction rather than in production:
RenewDeadline >= LeaseDuration is the classic split-brain window, and a shared
identity makes two holders indistinguishable on the lease.
The cross-cluster case stays open, and ConsensusPin now names the primitives that
actually solve it — luxfi/bft's LeaderForRound (deterministic, no election to get
wrong) and Quorum — instead of gesturing at "consensus". I tried the other way
first: a Raft-style vote over the peer set. It took four rounds of real safety
bugs (double-voting in a term, a leader that let its own lease go stale and voted
itself out, a candidate counting votes from a term the cluster had left) and STILL
produced two simultaneous holders under contention. That code is not here, because
shipping an election that is wrong 2 runs in 5 is worse than shipping none. The
lease passed the same suite 5/5 on the first attempt.
Ten tests, on a real fake API server: two candidates and only one leads, Release
hands the lease to the next writer, Release is idempotent, unsafe timings and a
missing identity are refused, a cancelled context yields no pin, and every
Resolve fallback explains itself.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Carries the catalog sync gate fix, so the scheduled model-catalog refresh can
authenticate as the internal service instead of 403ing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
deploy.listApplications — superseded in place. dashboard.go registers
dashAppList at the IDENTICAL path (/v1/deploy/applications); the pre-dashboard
handler kept compiling with no route and no caller. Its helpers
(observeApplication, syncStatus, runningVersions, listSiteApplications) are
live and stay.
deploy.engineResourceHealth — a one-line wrapper over
enginehealth.GetResourceHealth with zero callers, including from the unwired
handlers. The live health path is health.go's native resourceHealth, used by
applications/resource/tree.
kafka.order, pubsub.order, notify.subsystemOrder — order-ints from the
init()-registry that apps.Wire() replaced ("there is no init()-registry and no
order-int"; slice position IS the order). A stale order number is worse than
none: it reads as authoritative and binds nothing. The ordering CONSTRAINT each
one documented is real, so it stays as prose pointing at Wire().
gateway.mounted — a package-level singleton assigned in Mount and never read.
The Service it captured is already owned by the app; the assignment was
observable by nothing.
link.nowUnix — the routed counter's "default clock", never wired to the Meter
it was meant to be injectable through. Orphaned the "time" import.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
clients/session — a whole dark subsystem. Zero importers anywhere in the
workspace, no cmd/session, and absent from apps.Wire(), so its
/v1/code/sessions/* surface was never mounted. It is also a duplicate: the
live session registry is clients/agents (wired, richer — events, tree,
control) at /v1/agents/sessions, which is the path the Rust CLI actually
calls. It was already deleted once as dead in 70220647 with exactly this
reasoning and came back as collateral in 42f2ed8f, a commit that wired only
clients/domain into Wire().
clients/inprocess.go — eight identity functions (`return impl`) with zero
callers workspace-wide, including the two the file's own doc block advertises.
The type-system enforcement they were meant to provide is provided by
cloud.Deps' interface fields; nothing ever routed a value through the wrapper.
core.CHFloat64 — no callers. Its sibling CHInt64 is live in four packages;
the float variant never acquired one.
core.WarehouseReady — superseded by BillingEventsReady, the two-part gate
(warehouse connected AND commerce.events provisioned) that all three admin
fleet views actually call. The one-part version had no callers left.
sites.ReservedLabels — no callers; documented "exposed for diagnostics /
tests" but no test ever called it. Its removal orphaned the "sort" import.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/websearch/*/scrape proxied to a separate crawler at
crawl.hanzo.svc.cluster.local:11235. That name does not resolve — there is no such
deployment — so the surface answered 200 while every scrape inside it returned
{"success":false,"error":"...no such host"}. Chat's web_search could find pages and
never read one. Fixing the API key earlier today got the request AUTHENTICATED to a
backend that still wasn't there; this is the backend.
clients/crawl does the work in the binary, the same move already made for the
SEARCH half in websearch/search.go when the SearXNG pod was retired. Both halves of
web search are now in-process Go: nothing to deploy alongside, nothing to be down,
no hop for what is a fetch and a parse. One implementation behind two surfaces —
/v1/crawl natively, and websearch's firecrawl-shaped /scrape by calling Fetch
directly instead of speaking HTTP to itself.
Three orthogonal steps, each testable alone: Fetch (URL → bytes), extract (HTML →
readable subtree + metadata), render (node → markdown). Markdown rather than text
because the consumer is a model and structure is signal.
SSRF is the real cost of folding this in, and it is paid explicitly. The caller
chooses the URL and we now fetch it from INSIDE the cluster, next to service DNS
and a metadata endpoint at 169.254.169.254 that hands credentials to anyone who
asks. A separate pod got network policy for free by being separate; this package
carries the guard itself. The check is in the DIALER, not on the hostname, because
resolving to validate and then letting the transport resolve again is a TOCTOU gap
that DNS rebinding walks straight through — and redirects re-enter the same dialer,
so a public URL that 302s to the metadata endpoint is refused at the hop that
matters. public() is an allowlist of globally-routable unicast, not a blocklist:
a blocklist is one forgotten range (v4-mapped v6, CGNAT, 0/8) from being useless,
and the forgotten one is never noticed until it is used. 19 ranges are pinned in
tests, each with the reason it is there.
/v1/crawl is gated exactly like /v1/websearch/search — a validated principal OR the
shared service key, never neither, and an unset key 503s rather than defaulting
open. It reuses WEBSEARCH_API_KEY deliberately: same caller, same credential, and a
second key would be a second thing to mint and rotate with nothing to say which to
use when.
Found by probing real pages rather than only fixtures: Wikipedia's interlanguage
sidebar came back at the top of every article, because block scoring only judges a
container while it competes to BE the article — chrome nested inside the winner
rides along. Nested containers are now judged on their own link density, with prose
that merely contains links explicitly kept (a <p> of citations is a sentence, not a
link list). Both directions are pinned as tests.
Deleted with the dial: crawlRequest/crawlResponse/crawlResult, the
WEBSEARCH_CRAWL_ENDPOINT and WEBSEARCH_CRAWL_TOKEN accessors, and markdownField —
an UnmarshalJSON that existed only to absorb Crawl4AI's shape-polymorphic
`markdown` across versions. We control the shape now, so the polymorphism is gone
rather than accommodated. The four tests asserting that wire protocol went too;
they tested a conversation we no longer have.
go test ./clients/crawl/... ./clients/websearch/... ./apps/... passes. The eight
StarterCredit failures in ./apps are pre-existing on unmodified main (no /dev/shm
for the pure-Go SQLCipher codec on macOS) — verified by stashing this work and
re-running, not assumed.
team.collab.keepalive runs for the life of EVERY collab websocket, so the number
of these scales with connected users — a WriteControl racing a close is exactly
the fault that only appears under load, and one panic would take the binary down
for every connected tenant. collabService carries no logger, so this one is
contained but unlogged; a contained fault still beats a dead process.
projects.purgePrefix feeds a delete channel from a bucket listing. close(toDelete)
stays deferred INSIDE the work, so a panic still closes the channel and the
RemoveObjects range below drains and returns — otherwise containment would have
converted a crash into a permanent hang on a producer that is never coming back.
That is the end of the worthwhile sweep, and the reason matters: of the spawns
left, the great majority are boot-time listeners and daemon loops
(reader_proxy, ingress/edge, runner/daemon, the schedulers). A panic there is
immediate, loud, and happens before traffic — the opposite of the fire-and-forget
per-request case this helper exists for. The goja pair looked like the worst
offender and is not: those are watchdogs that only select on ctx and call
vm.Interrupt; the JavaScript runs on the CALLING goroutine, already covered.
go vet clean. clients/team and clients/projects fail identically on untouched
main here — same dev-env causes as the rest (no KMS master key ref, no /dev/shm).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/dns set the Authorization header only when CallerBearer returned non-empty and
forwarded regardless. CallerBearer returns "" for an API KEY (middleware_identity:
`tok == "" || isAPIKey(tok)`, prefixes pk-/sk-/hk-) because an API key is not a JWT
and the OIDC-gated DNS plane cannot validate one. So an API-key request left cloud
with NO credential at all.
Measured against the live plane (ghcr.io/hanzoai/dns:0.11.1, ready 1/1) with a real
API key: GET /v1/dns/zones came back
{"code":"unauthorized","message":"missing Authorization header"} — the DNS plane's
own error, which reads like the plane is broken when the real answer is that this
credential type cannot reach it. This head was written for the console, which
carries a session bearer, so the other case was simply never handled.
It is also the wrong place to fail. X-Org-Id is stamped from the server-validated
org, so a headerless forward arrived carrying a tenant claim with no proof of
identity — safe only for as long as the upstream keeps rejecting it. Cloud should
not lean on an upstream to refuse what it can refuse itself.
Now: no relayable bearer -> 401 from cloud, naming the actual problem, and the
request never leaves.
TESTS, and they fail without the change:
TestAPIKeyCallerIsRefusedAndNeverReachesUpstream — hk-/sk-/pk-, each asserting
401 AND upstream hits == 0. All three FAIL on the previous code (they forward
and get the upstream's opaque 401); all three pass now.
TestSessionBearerStillRelays — the console path is untouched: a session bearer
still reaches the plane verbatim and still gets 200.
Full clients/dns suite green.
Does NOT close the larger gap: binding a custom domain still creates no DNS record
(nothing in clients/projects or clients/sites calls clients/dns). This only makes
the existing relay fail honestly.
First piece of folding bot-gateway into cloud. A bot node runs on someone
s
machine and dials in, holding a long-lived socket; that socket is the only way
to reach it. You cannot spawn a rendezvous per request because the node is
already attached to one, so this is that rendezvous.
The TypeScript registry keyed nodes by id alone — nodesById: Map<string,
NodeSession> — with no tenant dimension anywhere in the type. Isolation was a
property of DEPLOYMENT: one shared singleton, per-viewer bearers, careful
caching. Get the caching wrong and one org sees another org
s nodes.
Here a node is addressed by (org, nodeID). Two orgs may use the same node id and
never meet, and a lookup without an org does not compile. That is the difference
between multi-tenant and single-tenant-with-care, and it is why this is a port
rather than a transpile.
Three behaviours are deliberate and pinned:
A foreign node returns exactly what a nonexistent node returns. Distinguishing
them would leak that another tenant has that node.
A disconnect fails every in-flight invoke at once rather than letting each wait
out its timeout on a question that can no longer be answered.
A reconnect replaces the stale session instead of closing it, so a node that
blinked is addressable immediately rather than stuck behind a dead entry.
The transport is deliberately absent: the registry takes a send func and is fed
answers by correlation id, so routing is testable without a network. The WS
layer comes next.
Both are fire-and-forget goroutines on hot paths, which is the shape that hides
this bug: nothing reads the result, so nothing notices until the panic takes the
whole binary down.
billing.record fires on EVERY billable request — the highest-frequency spawn in
the binary. A panic inside Record (a nil meter, a store fault, a malformed usage
row) killed the process for every tenant, on a path whose entire purpose is that
the caller does not wait for it.
webhooks.consume runs handlers over bus payloads and delivers to
customer-controlled endpoints — input we do not author on either side. Its inner
defer answers errc BEFORE the panic is recovered, so consume() returns instead of
blocking until ctx expires: a panicking consumer must look like a failed
consumer, not a hung one. It reports a distinct errStreamConsumerPanicked so
"died on bad input" is never silently read as "the bus closed".
Both take the logger already in scope (zip.Ctx.Log() and dispatcher.log), so a
contained panic is a line someone can act on rather than silence.
go vet clean. The webhooks tests fail identically on untouched main here —
CLOUD_KMS_MASTER_KEY_REF is unset on a dev laptop (16 failures before and after).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
middleware.Recover() wraps only the goroutine serving a request. A panic on any
goroutine a handler SPAWNS is unrecovered — and an unrecovered panic does not
fail that request, it takes the PROCESS down, and with it every tenant and every
subsystem in the shared binary.
The repo has 49 bare `go func()` outside tests. Four of them recover. The idiom
already existed, correct and well-argued, but it was private to
clients/integrations/bridge.go, so nothing else could reach it.
cloud.Go is that idiom, hoisted and exported, with Base.Go for callers that
already hold a Base (repeating the logger at every call site is how a helper
ends up skipped). It contains and logs; it deliberately does not wait, return an
error, or restart — a caller wanting a result uses a channel, and a caller
wanting bounds takes its limiter FIRST so the release defer survives the panic.
First conversion is the one most likely to fire: clients/answer/read.go spawns a
goroutine to parse pages we did not author, which is the likeliest panic site in
the binary and the least covered by tests. Its inner defer is registered first so
it runs first — the caller is answered "no pages" BEFORE the panic is recovered,
otherwise the answer loop would sit out its entire wall clock waiting for a
result that is never coming. The logger is threaded from the Engine rather than
passing nil, because a silently swallowed panic is barely better than a crash.
Tests assert containment directly: if the recover ever regresses the test does
not fail, it crashes the test binary — which is precisely the production symptom.
Note: `go build ./...` cannot LINK cmd/* in a fresh clone (the Rust
native/flags staticlib is not built); `go vet ./clients/... .` is clean and the
touched packages pass. The audit tests fail identically on untouched main here —
macOS has no /dev/shm for the SQLCipher codec.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Commerce could not parse KV_URL, so its embed failed, so every /v1/plans answered
"commerce unavailable". The whole revenue surface was dead for an unknown number
of days behind a green pod, a green deploy and a green test gate. Nothing in the
pipeline objected, and the reason is structural rather than an oversight.
A subsystem that mounts fail-closed serves an honest 503 on its own routes. From
outside that is indistinguishable from a subsystem which was never enabled here,
and cmd/smoke — the gate whose stated invariant is "a release must never ship if
a core endpoint is broken" — treats every 503 as "staged, tolerated". So the one
status code a dead plane returns is the one the gate is built to ignore.
Failure becomes state. cloud.Degraded(name, err) records it, /v1/health reports
it, and smoke hard-fails on it. Commerce is wired up first because it is the
plane that proved the hole; team's degraded mode is the obvious next caller.
/v1/health answers 200 EVEN WHEN DEGRADED, deliberately. It doubles as the
container probe, and evicting a pod because one plane of many is broken turns a
partial outage into a total one — the opposite of what fail-closed mounting is
for. Refusing belongs in the release gate, before an image ships, not in the
liveness path of one that already has.
Smoke also now probes /v1/health and /v1/plans. /v1/plans is on the paywall's
never-gate list precisely because a 402 tells the user to go there; it going 503
made the cure unreachable, and nothing tested it.
Verified end to end against two stub servers: a degraded /v1/health trips
"mounted fail-closed" and exits 1, a healthy one does not. Four tests pin the
registry semantics — first reason wins, nil/unnamed are not failures, and the
returned map is a copy. Gate 163 ok / 0 fail.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Carries the change that ends the second, hand-written table of what we pay
upstream: api/costs now resolves cost from the synced catalog and keeps the
curated table only as the fallback for models the sync does not cover.
commerce ships inside this binary, so this bump is how that reaches production.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
make e2e builds the rust staticlib + the binary, boots it on isolated ports
with a fresh data dir, seeds identity through the SAME operator upsert the
K8s operator reconciles against (argon2id, never plaintext), runs the
Playwright specs in universe/e2e, and tears down. Exits non-zero on any
failure. No cluster, no network, no KMS.
Three things it took to make the local stack self-consistent:
- CLOUD_JWKS_URL must point at the in-process IAM. The issuer identity is
left exactly as production; only key discovery moves. Without it cloud
fetches JWKS from the public hanzo.id, never finds this run's kid, drops
the principal, and every org-scoped route answers 'org scope required'.
- init_data.json seeds config only — it cannot create users. Users go
through /v1/iam/admin/users/upsert behind a service token.
- the tasks ports (19999/9999) are compile-time constants, so two
instances cannot coexist. Preflight refuses to start instead of leaving
the drip engine silently idle, and teardown escalates to SIGKILL rather
than leaving a listener behind for the next run.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ingress was missing from the list this was carried from. It is deployed raw
rather than through an operator CR, which is likely how it was missed — and
nothing else being reachable matters if the edge is not.
The rule for folded services is written down where it will be read. commerce and
the o11y runtime have no CR: they are compiled into this binary and reached
through api.hanzo.ai. Probing commerce.hanzo.svc would test a Service whose
selector matches no pod, reporting down for something that is up inside this
very process. Probing cloud covers them, because they ARE cloud.
iam and bot-gateway still carry CRs at replicas=1, so they are still separate
processes and stay probed. Their entries come out in the same change that
removes their CR — not before, or the fold reads as an outage.
Two things a reader keeps assuming and then building the wrong thing on: that a
repo or a ref is enrolled somewhere (neither is — no per-repo and no per-ref
config exists, and the per-repo plane that DOES exist is reactor config for
Slack notify and the outbound mirror), and that a second GitHub App pointed at
the same webhook adds redundancy.
It does the opposite. A connection is keyed (org, "github"), one row holding one
installation id, and the App identity is a single set of process values. Two Apps
contend for that row; the loser's deliveries resolve to no org and are acked 200
"unknown installation" so GitHub does not retry-storm — green deliveries, nothing
arriving. Records the runtime check for it: GET /v1/integrations already reports
`available` and `connection.externalId`, the installation id to compare against
the delivery, so no second surface is needed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A push event carried a short branch name, and the GitHub webhook dropped
anything that was not refs/heads/*. Tags never reached git.hanzo.ai, so a
release published by tag existed only on the mirror.
The event now carries the FULL ref end to end — webhook, sync engine, git
import, and push trigger — and the webhook filters on refs/ alone, so a tag
is carried like any other ref. A consumer that genuinely wants a branch name
cuts the prefix itself; that same cut is what keeps a tag from rebuilding an
app that tracks a branch.
Carrying tags needs no force: re-pointing an existing tag is not a
fast-forward, so the non-forcing refspec rejects it and the canonical copy
keeps the tag it published.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two pieces of cruft, both found by staticcheck rather than by eye.
cmd/connectorruntime, cmd/cron, cmd/gitops and cmd/sign call
apps.ServeSingle for apps that are not in apps.Wire(). Run any of them and you
get "unknown app" and exit 1 — they are leftovers from renames, gitops most
obviously (that surface became /v1/deploy). Nothing builds or ships them: no
Dockerfile, workflow, script or Makefile mentions any of the four. Verified by
running each one rather than assuming.
The generated stub's comment wrapped so that "go:generate" began a line, and Go
reads "// go:generate" as a typo'd compiler directive it then ignores — 86 files
carrying the same malformed line. Reworded the template in cmd/gen-app-cmds so
the phrase cannot land at the start of a comment, and regenerated. SA9009 goes
86 -> 0.
Deliberately NOT touched:
- 67 "unused" symbols. Every one is 18 days old or newer, in files with names
like "AttemptStore seam + runner" — that is work mid-wiring, not abandonment,
and deleting it would throw away someone's half-built feature.
- Six SA4000 "identical expressions" hits. They are determinism assertions —
f(x) != f(x) is how you catch map-iteration order, randomness or a clock
leaking into a supposedly pure function. The shape looks wrong; the intent is
right.
Gate: 163 ok, 0 fail.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Carrying the store FILE over was only half the rename. The tables inside it were
renamed in the same change (search_indexes/search_docs/search_terms ->
indexes/docs/terms), so migrate() created empty tables beside the populated ones
and the index read as empty while every document sat intact one identifier away.
Verified in prod: after the file moved, GET /v1/index/indexes returned nothing.
migrate() now adopts rows out of the previous names and drops them. Idempotent —
a missing table is skipped and an adopted one is dropped, so a later boot has
nothing to resurrect. INSERT OR IGNORE, so rows already written under the current
names win and the migration never overwrites live data with older rows.
The lesson is the same one the .dek taught an hour earlier: a store is the file,
its key, AND the names its rows live under. Move all three or lose the data.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The binary is one process mounting ~60 subsystems, so every request already
shares a service name and the trace warehouse cannot tell them apart. Give it
the one missing label instead of instrumenting 60 packages:
- MountAll indexes the composition root once at boot (name, prefixes, enabled);
TracingMiddleware stamps hanzo.subsystem on the span it ALREADY emits. No
second metrics path — the per-subsystem board reads the same o11y_traces
table, over the same datastore client, as /v1/admin/o11y.
- GET /v1/admin/subsystems fuses that process-local inventory (always truthful,
needs no warehouse) with RED signals + last-error from the warehouse, each
degrading independently behind core.SourceStatus.
- GET /v1/admin/money consolidates revenue, credits granted vs consumed, spend
by org, outstanding balance and infrastructure cost. It adds no arithmetic:
revenue.Compute / finance.Compute / customer.GrantRows were split out of
their handlers so the consolidated total cannot drift from the board it came
from. Money is money.Cents throughout.
Also fixes a live bug this work sat on top of: o11y.go queried durationNano and
serviceName against distributed_o11y_index_v3, which is snake_case and spells
resource attributes with $$. Those queries error, the caller's `if err == nil`
swallows it, and the trace half of the fleet board has been rendering
honest-looking zeros. Pinned as constants with a regression test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-27 09:47:43 -07:00
4005 changed files with 399466 additions and 167930 deletions
echo "::error::found a build/release invocation passing -tags controlplane — clients/controlplane's stub crypto must never enter a release/serve binary (see clients/controlplane/doc.go)"
echo "::error::found a build/release invocation passing -tags controlplane — apps/controlplane's stub crypto must never enter a release/serve binary (see apps/controlplane/doc.go)"
hits=1
fi
@@ -172,35 +216,30 @@ jobs:
fi
echo "OK: zen ${V} is at or above the streaming-fix floor ${FLOOR}"
- name:positive proof — clients/controlplane is unreachable from the default build
- name:positive proof — apps/controlplane is unreachable from the default build
run:|
set -euo pipefail
# Containment is an IMPORT-GRAPH property, so prove it with the import
# graph. This step used to lead with `go build ./...`, which compiles
# AND LINKS every cmd/ main — and linking needs
# native/flags/target/release/libhanzo_flags.a, the Rust staticlib
# clients/flags pulls in under cgo (clients/flags/engine.go). Nothing
# in THIS job builds it: `make native` runs in the gate job, in that
# job's own workspace. It passed on GitHub only because the arc
# runners reuse a workspace and target/ is gitignored, so an earlier
# gate job's leftover .a was still sitting there; on a container
# runner with a fresh volume every cmd/ main died at
# `ld: cannot find .../libhanzo_flags.a`. The compile was never the
# proof anyway — `go list -deps` reads the same untagged file set
# without linking, and the gate job is the ONE place that builds.
for m in $(go list ./cmd/...); do
if go list -deps "$m" | grep -qx 'github.com/hanzoai/cloud/clients/controlplane'; then
echo "::error::$m links clients/controlplane into a real binary — containment breach"
# AND LINKS every cmd/ main — work this proof does not need and this
# job should not be doing: `go list -deps` reads the same untagged
# file set without linking, in seconds, and the gate job is the ONE
# place that builds. A green compile here would also have proved
# nothing extra; the containment claim is entirely about which
# packages are reachable, which is what the graph answers.
for m in $(go list ./cmd/... ./plugin/...); do
if go list -deps "$m" | grep -qx 'github.com/hanzoai/cloud/apps/controlplane'; then
echo "::error::$m links apps/controlplane into a real binary — containment breach"
MODERNC="$(CGO_ENABLED=1 go list -tags "libsqlite3 sqlite_fts5" -deps ./cmd/cloud 2>/dev/null | grep -c 'modernc.org/sqlite'||true)";\
["$MODERNC"="0"]||{echo"SQLITE-GATE FAIL: cmd/cloud links modernc.org/sqlite ($MODERNC pkgs) under CGO=1 — double-registers \"sqlite\" with hanzoai/sqlite(mattn) and panics at init.";exit 1;}
MODERNC="$(CGO_ENABLED=1 go list -tags "libsqlite3 sqlite_fts5 sqlite_math_functions" -deps ./cmd/... ./plugin/... 2>/dev/null | grep -c 'modernc.org/sqlite'||true)";\
["$MODERNC"="0"]||{echo"SQLITE-GATE FAIL: a per-app binary links modernc.org/sqlite ($MODERNC pkgs) under CGO=1 — double-registers \"sqlite\" with hanzoai/sqlite(mattn) and panics at init. Find it: CGO_ENABLED=1 go list -tags 'libsqlite3 sqlite_fts5 sqlite_math_functions' -deps ./plugin/<app> | grep modernc";exit 1;}
# RED gate — ENCRYPTION PROOF + the cek.go GOLDEN-VECTOR KAT, under the SAME CGO +
# libsqlcipher build this image ships. TestEncryptionProof asserts real
# ciphertext-at-rest (SQLITE_REQUIRE_CODEC=1 makes a plaintext link FAIL → NO
@@ -136,7 +157,7 @@ RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
# encrypted stores stay readable, or NO image.
RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
@@ -63,43 +69,127 @@ webui: ## Build the real console static bundle into webui/dist (go:embed source)
cp -r "$(CONSOLE_DIR)/out/." webui/dist/
@echo ">> embedded real console bundle into webui/dist (index.html $$(wc -c < webui/dist/index.html) bytes)"
deploy-ui:## Build the monochrome ArgoCD dashboard bundle into clients/deploy/webui/dist (go:embed source). DEPLOY_DIR=<path to hanzoai/deploy>.
deploy-ui:## Build the monochrome ArgoCD dashboard bundle into apps/deploy/webui/dist (go:embed source). DEPLOY_DIR=<path to hanzoai/deploy>.
@command -v yarn >/dev/null 2>&1||{echo"yarn is required to build the deploy dashboard bundle";exit 1;}
@test -f "$(DEPLOY_DIR)/ui/package.json"||{echo"deploy checkout not found at $(DEPLOY_DIR) — set DEPLOY_DIR=<path to hanzoai/deploy on rebrand/hanzo-monochrome>";exit 1;}
agentskills:## Regenerate the FULL agent-skills catalog into clients/agentskills/catalog (go:embed source) from the openapi SOT. OPENAPI_DIR=<path to openapi>.
agentskills:## Regenerate the FULL agent-skills catalog into apps/agentskills/catalog (go:embed source) from the openapi SOT. OPENAPI_DIR=<path to openapi>.
@test -f "$(OPENAPI_DIR)/skills.py"||{echo"openapi checkout not found at $(OPENAPI_DIR) — set OPENAPI_DIR=<path> or clone hanzoai/openapi";exit 1;}
# skills.py rewrites the whole catalog dir; the .gitignore keeps only the tiny
# `ai` fallback tracked, so the full set is embedded at build but never committed.
smoke:## Build and run cmd/cloud-smoke (mount-time integration check).
$(GO) run ./cmd/cloud-smoke
plugin:## Build ONE app into ./bin: make plugin APP=wallets.
@test -n "$(APP)"||{echo"usage: make plugin APP=<name>";echo"apps: $(APPS)";exit 1;}
@echo "$(APPS)"| tr ' ''\n'| grep -qx "$(APP)"||{echo"no app named $(APP) — the manifest is the list; run 'make generate' after adding a row, or check the name against 'make plugin' with no APP";exit 1;}
@$(MAKE) --no-print-directory bin/$(APP)
# manifest/apps.go is the hand-authored source of truth for the subsystem set.
# This scaffolds a plugin/<app>/main.go for any manifest app that lacks one and
# validates the two are in bijection (every app has a command, every command is
# an app). Idempotent: a no-op run leaves the tree clean, which is what lets CI
# diff it. It does NOT rewrite existing mains — those are source.
generate:## Scaffold missing plugin/<app>/main.go and validate the manifest.Apps bijection.
$(GO) run ./plugin/gen-app-cmds
# NOTE: the shipped API is the light host plus one binary per app (there is no
# fused `cloud` binary anymore). The Go `hanzo` CLI (cmd/hanzo) is DELETED; the
# shipped `hanzo` is the Rust CLI (~/work/hanzo/cli, `curl hanzo.sh`), which talks
# to this API over HTTP via its OpenAPI-generated command surface. cli/ remains
# only as the reference for the still-to-port client-side tools and is not built
# here.
# Builds the host plus EXACTLY the plugins it is told to mount — not all 106.
# The host resolves a plugin as a file beside itself (manifest.App.Plugin), so a
# name in RUN_ENABLE with no binary in ./bin is the one way this fails; building
# that same list here is what keeps the two in step.
RUN_ENABLE?= iam,base,kms,gateway,o11y
run:cloud## Run the host with iam,base,kms,gateway,o11y (matches README quickstart); builds just those plugins.
@for a in $$(echo$(RUN_ENABLE)| tr ','' ');do$(MAKE) --no-print-directory plugin APP=$$a;done
./bin/cloud --enable=$(RUN_ENABLE)
smoke:## Build and run the smoke prober (mount-time integration check).
$(GO) run ./plugin/smoke
# The ONE end-to-end target: builds this repo's binary, boots it on isolated ports
# with a fresh data dir, seeds identity through the same operator upsert production
# uses, and drives it with the real Playwright suite (universe/e2e) — a real login,
# a real cross-tenant refusal, and a real SMTP delivery through the drip engine.
# Needs no cluster and no network. SUITE=<path> if universe is not a sibling.
e2e:## Boot the binary locally and run the Playwright e2e suite against it.
@E2E_ARGS="$(E2E_ARGS)" ./e2e/run.sh
# The console's IAM/cloud origins are NEXT_PUBLIC_* — inlined at BUILD time — so a
# bundle built for production points its login at hanzo.id and its reads at
# api.hanzo.ai. This rebuilds it against the loopback instance so the UI specs
# exercise the local binary end to end. It OVERWRITES webui/dist with a
# localhost-pinned bundle: run plain `make webui` before shipping anything.
E2E_ORIGIN?= http://127.0.0.1:18080
e2e-ui:## Rebuild the console pointed at the local instance, then run e2e.
# The release image builds with -tags "libsqlite3 sqlite_fts5" (see Dockerfile).
# libsqlite3 needs cgo and the C library, but sqlite_fts5 does not — and without it
# any store whose migration declares an FTS5 table fails to open, so a subsystem
# built on full-text search (clients/code) cannot be tested at all. Carry the tag
# built on full-text search (apps/code) cannot be tested at all. Carry the tag
# the shipped build carries, so the suite exercises the same schema surface.
TEST_TAGS:= sqlite_fts5
test:## Run unit + integration tests (pure-Go, with the FTS5 tag the image ships).
# The lifted prose is COMMITTED (zipdoc_gen.go) because bare `go build` cannot
# regenerate it; -check writes nothing and goes red when a lift no longer
# matches its source, which is the drift being committed makes possible.
# Per PACKAGE, not ./...: the checker must load exactly the way `go generate`
# does, one package at a time — whole-module loading extracts differently
# (zap-proto/zip zipdoc: single-vs-module load divergence) and a gate must
# never disagree with the generator it polices.
@set -e;for d in $$(grep -rl '^//go:generate go run github.com/zap-proto/zip/cmd/zipdoc' --include='*.go' clients cmd . 2>/dev/null | xargs -n1 dirname | sort -u);do(cd$$d &&$(GO) run github.com/zap-proto/zip/cmd/zipdoc -check)||{echo"$$d/zipdoc_gen.go is stale — run: go generate -run zipdoc ./$$d/...";exit 1;};done
test-cgo:## Prove the cgo build works too — forces the fork's pure-Go backend via -tags sqlite_purego so the embedded modernc importers don't double-register "sqlite".
Description:"RevokeKey revokes the caller's own API key of the requested class. The class is\nthe same field mint takes — `?type=publishable`, defaulting to secret — so\nrevoking the key that ships in a browser bundle does not sign its holder out of\ntheir own API: the other key keeps working.\n\nRevoking is how a key is replaced when it does not need replacing; minting the\nsame class again rotates it in one step. IAM drops the credential immediately,\nbut the gateway caches keys for a few minutes, so a request that beat the cache\nexpiry may still be served.\n\nFor callers written against the older shape, the class is also accepted in a JSON\nrequest body, read only when `?type=` is absent.",
Fields:map[string]string{
"keyTypeIn.type":"Type is the key class to act on: \"secret\" (sk-, session-equivalent, belongs\non a server) or \"publishable\" (pk-, org-identifying, safe in a browser\nbundle). Omitted means secret, which is what every existing caller means.",
"revokedKey.ok":"OK is true when the key was revoked. A failure is an error status, never a\nfalse here.",
"revokedKey.type":"Type is the key class that was revoked, resolved — so a caller that named\nnothing can see it revoked the secret key.",
Description:"RevokeKey revokes the caller's own API key of the requested class. The class is\nthe same field mint takes — `?type=publishable`, defaulting to secret — so\nrevoking the key that ships in a browser bundle does not sign its holder out of\ntheir own API: the other key keeps working.\n\nRevoking is how a key is replaced when it does not need replacing; minting the\nsame class again rotates it in one step. IAM drops the credential immediately,\nbut the gateway caches keys for a few minutes, so a request that beat the cache\nexpiry may still be served.\n\nFor callers written against the older shape, the class is also accepted in a JSON\nrequest body, read only when `?type=` is absent.",
Fields:map[string]string{
"keyTypeIn.type":"Type is the key class to act on: \"secret\" (sk-, session-equivalent, belongs\non a server) or \"publishable\" (pk-, org-identifying, safe in a browser\nbundle). Omitted means secret, which is what every existing caller means.",
"revokedKey.ok":"OK is true when the key was revoked. A failure is an error status, never a\nfalse here.",
"revokedKey.type":"Type is the key class that was revoked, resolved — so a caller that named\nnothing can see it revoked the secret key.",
Description:"TopupRails lists the accepted (chain, token, treasury) triples, so a browser can\nrender \"send USDC here\" without the addresses being baked into its bundle.\n\nThis exists because the console previously gated its top-up UI on\nNEXT_PUBLIC_HANZO_HUSD_ADDRESS/_TREASURY — build-time constants. Enabling a rail\ntherefore meant rebuilding and redeploying the frontend, and with them unset the\nUI reported \"not available yet\" no matter what the server could actually accept.\nServing the set at runtime keeps ONE source of truth (the server's config) and\nlets a rail be switched on without shipping a bundle.\n\nEverything here is public on-chain data; no secret is exposed, and the set is\nempty on a deployment that accepts no crypto rail.",
Fields:map[string]string{
"railList.rails":"Rails is every (chain, token, treasury) triple this deployment accepts.",
"railView.chain":"Chain is the human chain name, e.g. \"Base\".",
"railView.chainId":"ChainID is the EIP-155 chain id the wallet must be on.",
"railView.decimals":"Decimals is the token's decimal places — 6 for USDC, 18 for an 18-decimal\ntoken. Cents are derived per-rail from it.",
"railView.id":"ID is the stable rail id to name when submitting a transfer, e.g. \"base-usdc\".",
"railView.symbol":"Symbol is the display symbol, e.g. \"USDC\".",
"railView.token":"Token is the ERC-20 contract address to transfer.",
"railView.treasury":"Treasury is the address on this chain to send funds to.",
},
})
zip.Describe("GET /v1/csrf",zip.Doc{
Description:"IssueCSRFToken mints the anti-CSRF token a browser echoes as X-CSRF-Token on\nevery money write (mint/revoke a key, top up, onboard, and the billing/commerce\nwrite verbs). The token is bound to the caller's validated identity and expires,\nso one minted for one identity cannot authorize a write as another.\n\nIt is answered no-store, so it is never cached by a shared proxy. This is the\nsame-origin endpoint the embedded console reads — the Same-Origin Policy is what\nstops a cross-site page from reading the response and forging a write.",
Fields:map[string]string{
"csrfResp.csrfToken":"Token is the value to send back in the X-CSRF-Token header. It is bound to the\ncaller's identity, so it authorizes writes as them and as nobody else.",
"csrfResp.expiresIn":"ExpiresIn is the token's lifetime in seconds. Fetch a new one when it lapses;\na write with an expired token is refused.",
},
})
zip.Describe("GET /v1/embed-status",zip.Doc{
Description:"EmbedStatus reports whether one of this brand's shared embedded apps (cms, erp,\nhelp) may be framed by the caller and is actually running, so a console module\ncan choose between the embed and the provision panel.\n\nIt answers two questions the browser cannot answer for itself. ENTITLEMENT is\nserver-authoritative: each app is a single shared per-BRAND instance, so only a\nmember of the owning brand org — or a SuperAdmin — is given the embed URL; every\nother caller gets phase \"not-entitled\" and no URL. REACHABILITY is a probe of\nthat origin, which a cross-origin page cannot read for itself.\n\nThe probed host is always <app>.<this deployment's own brand domain>: no part of\nit comes from the request, so this can never be steered into probing an\narbitrary origin.",
Fields:map[string]string{
"embedStatusReq.app":"App is the embedded app to report on: cms (Content Studio), erp or help.",
"embedStatusResp.app":"App is the app this verdict is about.",
"embedStatusResp.embedUrl":"EmbedURL is the in-app landing URL to frame. Empty when the caller is not\nentitled — a non-entitled caller never receives it.",
"embedStatusResp.entitled":"Entitled is whether the caller's org may frame this brand-owned app.",
"embedStatusResp.origin":"Origin is the app's origin on this deployment's own brand domain.",
"embedStatusResp.phase":"Phase is the verdict in one word: not-entitled, not-provisioned or ready.",
"embedStatusResp.reachable":"Reachable is whether the app answered the liveness probe.",
},
Example:json.RawMessage(`{"app":"cms"}`),
})
zip.Describe("GET /v1/iam/keys",zip.Doc{
Description:"GetKey returns the caller's own API keys — every type they hold, read\nAUTHORITATIVELY from IAM rather than from the session claim, which lags a key\nminted moments ago. No secret material comes back: a secret key is represented\nby its prefix, and only a publishable key (public by construction) carries its\nfull value.\n\nA transient IAM read failure reports an empty set rather than a 5xx, so the\npage shows the honest empty state and never a fabricated key.",
Fields:map[string]string{
"apiKey.createdAt":"CreatedAt is when the key last changed, as IAM records it.",
"apiKey.key":"Key is the FULL value, and is present for a publishable key only: it is\npublic by construction and useless to its holder if it cannot be read back.",
"apiKey.prefix":"Prefix is the recognizable, non-secret head of the key — enough to tell two\nkeys apart, never enough to use one.",
"apiKey.type":"Type is the key class: secret (sk-) or publishable (pk-).",
"apiKeyList.keys":"Keys is every key the caller holds, at most one per type.",
},
})
zip.Describe("GET /v1/keys",zip.Doc{
Description:"GetKey returns the caller's own API keys — every type they hold, read\nAUTHORITATIVELY from IAM rather than from the session claim, which lags a key\nminted moments ago. No secret material comes back: a secret key is represented\nby its prefix, and only a publishable key (public by construction) carries its\nfull value.\n\nA transient IAM read failure reports an empty set rather than a 5xx, so the\npage shows the honest empty state and never a fabricated key.",
Fields:map[string]string{
"apiKey.createdAt":"CreatedAt is when the key last changed, as IAM records it.",
"apiKey.key":"Key is the FULL value, and is present for a publishable key only: it is\npublic by construction and useless to its holder if it cannot be read back.",
"apiKey.prefix":"Prefix is the recognizable, non-secret head of the key — enough to tell two\nkeys apart, never enough to use one.",
"apiKey.type":"Type is the key class: secret (sk-) or publishable (pk-).",
"apiKeyList.keys":"Keys is every key the caller holds, at most one per type.",
Description:"WalletTopup credits the caller's org for a stablecoin transfer they already sent\nto the treasury. It reads the receipt from that rail's chain, confirms a mined,\nsuccessful ERC-20 Transfer to the rail's treasury, derives USD cents from the\non-chain value using the token's own decimals, records the credit, and returns\nthe amount plus the new balance.\n\nThe credited amount is the ON-CHAIN value, never a number the caller sends, and\nthe credit lands on the caller's own validated org — there is no way to name a\nthird-party subject. Nothing is credited that the chain did not confirm: a\nmissing, failed or non-matching transaction is refused, and a deployment with no\npayment rail enabled says so rather than inventing a credit.",
Fields:map[string]string{
"walletTopupReq.fromAddress":"FromAddress is the wallet the transfer was sent from. Optional; when given it\nmust match the transfer's on-chain sender.",
"walletTopupReq.rail":"Which accepted rail the transfer was sent on, e.g. \"base-usdc\". The client\nnames it rather than the server guessing from the tx: the same address can\nexist on several chains, so inferring would risk crediting against the wrong\ntreasury. It may be omitted only while exactly one rail is enabled.",
"walletTopupReq.txHash":"TxHash is the hash of the ERC-20 transfer that was already sent to the rail's\ntreasury. The receipt is read from that chain; nothing is credited that the\nchain did not confirm.",
"walletTopupResp.balance":"Balance is the org's new USD-ledger balance in cents. Best-effort: a read\nfailure reports 0, and the credit has already landed either way.",
"walletTopupResp.creditedCents":"CreditedCents is the USD credit recorded, derived from the ON-CHAIN value\nusing the token's own decimals — never a client-supplied number.",
"walletTopupResp.status":"Status is how commerce recorded the payment.",
"walletTopupResp.txHash":"TxHash is the transfer that was credited.",
Description:"MintKey creates — or rotates — the caller's API key of the requested type and\nreturns it ONCE. A real IAM failure surfaces as 502, never a fabricated key.\n\nRotating is what creating means here: a user holds one key per type, so the\nendpoint is idempotent by (caller, type) and the superseded credential stops\nworking. Two live secrets for one user would make \"revoke my key\" a lie.",
Fields:map[string]string{
"keyTypeIn.type":"Type is the key class to act on: \"secret\" (sk-, session-equivalent, belongs\non a server) or \"publishable\" (pk-, org-identifying, safe in a browser\nbundle). Omitted means secret, which is what every existing caller means.",
"mintedKey.accessKey":"AccessKey is the same value under its predecessor name, carried so callers\nwritten against the older field keep working. One value, two names.",
"mintedKey.key":"Key is the credential, returned ONCE — a secret key is unreadable afterwards.",
"mintedKey.type":"Type is the class of key that was minted.",
Description:"Onboard creates the caller's organization. Two flows, keyed on whether the caller\nalready has a home org (mirrors app/onboard/route.ts):\n\n - FIRST-RUN (no owner): create + MOVE the user in as admin, so their next JWT\n carries the new owner and the cloud scopes everything to it.\n - ADDITIONAL (owner set): create the org but do NOT move the user — a move\n changes their IAM owner (stripping a SuperAdmin's status + orphaning their\n current org). They reach the new org via the OrgSwitcher, which re-scopes\n X-Org-Id without touching IAM membership. A personal-org request from someone\n who already has an org is meaningless → 409.",
Fields:map[string]string{
"onboardReq.name":"Name is the organization's display name. Ignored when personal is true, which\nderives the name from the caller's own username instead.",
"onboardReq.personal":"Personal asks for the caller's own workspace: the name is derived from their\nusername and the slug auto-suffixes to stay unique. Meaningless — and refused\n— for a caller who already has an organization.",
"onboardResp.additional":"Additional is true when the caller already had an organization and this one\nwas created WITHOUT moving them into it — they reach it via the org switcher.",
"onboardResp.displayName":"DisplayName is the organization's human name.",
"onboardResp.org":"Org is the created organization's slug, which is what X-Org-Id carries.",
},
Example:json.RawMessage(`{"name":"Acme"}`),
})
zip.Describe("POST /v1/keys",zip.Doc{
Description:"MintKey creates — or rotates — the caller's API key of the requested type and\nreturns it ONCE. A real IAM failure surfaces as 502, never a fabricated key.\n\nRotating is what creating means here: a user holds one key per type, so the\nendpoint is idempotent by (caller, type) and the superseded credential stops\nworking. Two live secrets for one user would make \"revoke my key\" a lie.",
Fields:map[string]string{
"keyTypeIn.type":"Type is the key class to act on: \"secret\" (sk-, session-equivalent, belongs\non a server) or \"publishable\" (pk-, org-identifying, safe in a browser\nbundle). Omitted means secret, which is what every existing caller means.",
"mintedKey.accessKey":"AccessKey is the same value under its predecessor name, carried so callers\nwritten against the older field keep working. One value, two names.",
"mintedKey.key":"Key is the credential, returned ONCE — a secret key is unreadable afterwards.",
"mintedKey.type":"Type is the class of key that was minted.",
Description:"Reads cloud's tamper-evident audit trail, newest first, with the chain's live\nintegrity attached so a listing can be badged as verified.\n\nWhen cloud has no local store configured it falls back to forwarding IAM's own\nget-records trail verbatim — a DIFFERENT trail, federated so the endpoint never\nregresses to an empty list. Those rows carry no integrity of ours, so the field is\nnull there.",
Fields:map[string]string{
"Integrity.brokenAt":"BrokenAt is the seq of the FIRST record that failed verification, or -1 when\nOK. Reason describes the break (recomputed-hash mismatch, prev-hash\ndiscontinuity, or a seq gap).",
"Integrity.count":"Count is the number of records walked.",
"Integrity.headHash":"HeadHash is the hash of the last record (or the genesis anchor for an empty\nchain). Pin this externally over time to detect tail-truncation.",
"Integrity.ok":"OK is true iff every record's stored hash equals the recomputed hash AND the\nchain links are continuous (each PrevHash == the prior record's Hash, seqs\ngapless from 0).",
"RecordsIn.action":"Action restricts it to one action name, e.g. \"admin.waitlist.grant\".",
"RecordsIn.org":"Org restricts the trail to one tenant.",
"RecordsIn.p":"Page is the 1-based page number, driving the offset.",
"RecordsIn.pageSize":"PageSize is rows per page, default 100.",
"RecordsIn.resource":"Resource restricts it to one resource kind, e.g. \"credit-grant\".",
"RecordsIn.resourceId":"ResourceID restricts it to one resource instance.",
"RecordsIn.result":"Result restricts it to \"success\" or \"error\".",
"RecordsIn.since":"Since is the inclusive lower time bound, RFC3339. An unparseable value is\nignored rather than refused — one malformed filter must not hide the trail.",
"RecordsIn.sub":"Sub restricts it to one actor (the validated subject that made the request).",
"RecordsIn.until":"Until is the upper time bound, RFC3339, with the same tolerance.",
Description:"Walks the WHOLE hash chain and reports whether it is intact: how many records\nwere checked, the head hash to pin externally against tail-truncation, and — when the\nchain is broken — the seq of the first bad record and why.\n\nbrokenAt is -1 exactly when ok is true. An unconfigured store is an honest failure\nhere rather than a fabricated pass.",
Fields:map[string]string{
"Integrity.brokenAt":"BrokenAt is the seq of the FIRST record that failed verification, or -1 when\nOK. Reason describes the break (recomputed-hash mismatch, prev-hash\ndiscontinuity, or a seq gap).",
"Integrity.count":"Count is the number of records walked.",
"Integrity.headHash":"HeadHash is the hash of the last record (or the genesis anchor for an empty\nchain). Pin this externally over time to detect tail-truncation.",
"Integrity.ok":"OK is true iff every record's stored hash equals the recomputed hash AND the\nchain links are continuous (each PrevHash == the prior record's Hash, seqs\ngapless from 0).",
// Validate the target is a REAL org (never mint an orphan wallet on a typo).
o,err:=FindOrg(s,ctx,cr,org)
iferr!=nil{
return&GrantOut{Status:Err,Msg:err.Error()},nil
}
ifo==nil{
c.Status(404)
return&GrantOut{Status:Err,Msg:"customer not found"},nil
}
// FAIL-CLOSED durability (SOC2 AU-2/AU-5): a credit grant moves REAL money and MUST
// leave a durable, tamper-evident record. If this deployment has no audit store to
// record into, REFUSE the grant BEFORE any money moves — never an unaudited money
// move. A nil store is not a production state: cloud requires a persistent data dir
// for the trail (audit_serve.go), and nil arises only from the explicit
// CLOUD_AUDIT_DISABLED dev opt-out, on which moving money is not a supported op.
ifs.State.AuditStore==nil{
c.Status(503)
return&GrantOut{Status:Err,Msg:"grant refused: no durable audit store is configured on this deployment; a credit grant must be recorded before money moves"},nil
}
// Resolve the ADDRESS the grant lands at — the same rule (account.Payer) the
// spend gate resolves the payer with, so the credit and the spend it funds name
// one wallet. Empty req.User is the org; a named member of a POOLED org still
// resolves to that org's pool, because that is the account their requests will
// be gated on. A refusal here means the name could not be turned into an address
// (a "/" in it would silently address something else), never a fallback.
w,addressed:=principal.WalletFor(org,req.User)
if!addressed{
return&GrantOut{Status:Err,Msg:"user must be a bare IAM username (no '/')"},nil
}
subject:=w.Account
tag,source:=grantTag(req.Source)
notes:=grantNote(c,req.Reason)
// ONE credit money-move: the co-resident native finance wallet is preferred (the ai
// prepaid gate + the edge meter read/debit THAT wallet, so a grant MUST land there),
// with the commerce HTTP deposit as the split-deploy fallback. Both return the
// pre-balance (recorded even on failure), the entry/transaction id, and the post-balance,
// so the audit + response below are one shape regardless of which path moved the money.
// w.Ledger, not the raw path/body org: both halves of the address come from ONE
// resolved Account, so the ledger a deposit opens can never disagree in case with
afterExact=bal.AttoString()// afterExact = the EXACT balance (sub-cent visible)
}
}
returnbefore,id,after,afterExact,nil
}
// Split deploy: no co-resident credit ledger → the commerce billing HTTP deposit, with
// its operator-nonce idempotency key so a retried grant dedupes at commerce. It is
// org-keyed, so a member-addressed grant has no destination here and is REFUSED —
// crediting the pool instead would put the money where that member cannot spend it.
ifsubject!=org{
return0,"",0,"",fmt.Errorf("this deployment has no co-resident ledger; only the org itself can be credited over the commerce HTTP path (asked for %q)",subject)
Description:"Lists every customer org at a glance, sorted by slug: owner email, plan,\nsuspend status, member count, balance, month-to-date spend and MRR.\n\nEach row costs one IAM read plus the org's money reads, fanned out under a fixed\nconcurrency ceiling so a large fleet cannot stampede the upstreams. Every read is\nbest-effort per row: an upstream miss degrades THAT field to its honest zero rather\nthan failing the fleet.",
"OrgIn.org":"Org is the tenant slug from the path.",
},
})
zip.Describe("GET /v1/admin/grants",zip.Doc{
Description:"Reads the credit-grant ledger across ALL orgs, newest first — who granted what\nto whom, when, and from which money bucket.\n\nIt is a PROJECTION of the tamper-evident audit trail, not a second store: every grant\nis written there as action \"admin.customer.credit\", so this view cannot drift from\nwhat actually happened, and FAILED grants appear too.\n\nA deployment with no local audit store has no history to project, and says so with an\nempty list and a msg rather than an error.",
Fields:map[string]string{
"GrantRow.actor":"staff email (or sub) who issued it",
"GrantRow.result":"success | error",
"GrantRow.source":"\"trial\" | \"prepaid\"",
"GrantsIn.limit":"Limit caps the rows returned. Default 200.",
"GrantsIn.org":"Org filters by the ACTOR's org (the staff org that issued the grant), which is\nrarely what a reader wants — the target org is a row field, not a filter.",
"GrantsIn.result":"Result filters by outcome: \"success\" or \"error\". Empty returns both, which is\nthe point of this view — a refused grant is as interesting as a granted one.",
Description:"Issues a staff credit grant to the org named in the path — a comp, refund\nor promo — through the ONE credit-write path core.ApplyGrant, which validates the\namount against the per-grant cap, checks the org exists, moves the money and records\nthe tamper-evident audit row.\n\nThe credit lands on the account account.Payer resolves, NOT necessarily the org: name\na member of a pooled org and the pool is credited. The receipt echoes the subject so\nthe caller can see which.",
Fields:map[string]string{
"GrantIn.amountCents":"AmountCents is the credit, in whole cents. Must be positive and within the\nper-grant cap.",
"GrantIn.currency":"Currency is the ISO code, lower-cased. Empty means usd.",
"GrantIn.org":"Org is the tenant to credit. Required.",
"GrantIn.reason":"Reason is the operator's justification, recorded on the audit row.",
"GrantIn.source":"Source is the money bucket: \"trial\" (default) for a non-cash comp that is never\nrefundable, or \"prepaid\" for real money. Anything unknown falls back to trial.",
"GrantIn.user":"User optionally names a MEMBER to credit, by bare IAM username. Empty credits\nthe org. Which of the two the money actually lands on is decided by\naccount.Payer, not here: a pooled org keeps one balance whatever is named.",
"GrantResult.balanceCents":"BalanceCents is the account balance AFTER the grant, in whole cents.",
"GrantResult.balanceExact":"BalanceExact is that same balance at full 18-decimal precision, so a sub-cent\ndebit is visible rather than rounded away.",
"GrantResult.currency":"Currency is the lower-cased ISO code the grant was denominated in.",
"GrantResult.grantedCents":"GrantedCents is the amount actually credited.",
"GrantResult.org":"Org is the tenant whose ledger was credited.",
"GrantResult.source":"Source is the money bucket: \"trial\" (non-cash comp) or \"prepaid\" (real money).",
"GrantResult.subject":"Subject is the ACCOUNT the credit landed on inside that ledger: the org slug for\na pooled org, \"<org>/<name>\" for a member of a per-member one. It is echoed\nbecause the operator does not choose it — account.Payer does — so naming a\nmember of a pooled org credits the pool and the receipt has to say so.",
"GrantResult.transactionId":"TransactionID is the ledger entry id, for reconciliation against commerce.",
Description:"Restores access for every member of the org, undoing a suspend. It\nreports the same per-user breakdown.",
Fields:map[string]string{
"AccessChange.affected":"Affected lists the usernames that were updated.",
"AccessChange.failed":"Failed lists the usernames that were NOT updated. Non-empty means the org is in\na mixed state and the action should be retried.",
"AccessChange.org":"Org is the tenant acted on.",
"AccessChange.suspended":"Suspended is the state applied: true for suspend, false for reactivate.",
"OrgIn.org":"Org is the tenant slug from the path.",
Description:"Cuts off every member of the org: IAM refuses a forbidden user at\nlogin AND at token issuance, so a suspended customer can neither sign in nor mint a\nfresh token. Fully reversible with ReactivateCustomer.\n\nThe result names every user updated and every user that was NOT — a partial failure\nleaves the org in a mixed state and says so instead of reporting a clean success.",
Fields:map[string]string{
"AccessChange.affected":"Affected lists the usernames that were updated.",
"AccessChange.failed":"Failed lists the usernames that were NOT updated. Non-empty means the org is in\na mixed state and the action should be retried.",
"AccessChange.org":"Org is the tenant acted on.",
"AccessChange.suspended":"Suspended is the state applied: true for suspend, false for reactivate.",
"OrgIn.org":"Org is the tenant slug from the path.",
Description:"Issues a credit grant to any org from the operator Grants view, with the\ntarget named in the body. It funnels through the SAME core.ApplyGrant that\nPOST /v1/admin/customers/:org/credit uses, so there is exactly ONE credit-write path\nand one audit trail behind both.",
Fields:map[string]string{
"GrantIn.amountCents":"AmountCents is the credit, in whole cents. Must be positive and within the\nper-grant cap.",
"GrantIn.currency":"Currency is the ISO code, lower-cased. Empty means usd.",
"GrantIn.org":"Org is the tenant to credit. Required.",
"GrantIn.reason":"Reason is the operator's justification, recorded on the audit row.",
"GrantIn.source":"Source is the money bucket: \"trial\" (default) for a non-cash comp that is never\nrefundable, or \"prepaid\" for real money. Anything unknown falls back to trial.",
"GrantIn.user":"User optionally names a MEMBER to credit, by bare IAM username. Empty credits\nthe org. Which of the two the money actually lands on is decided by\naccount.Payer, not here: a pooled org keeps one balance whatever is named.",
"GrantResult.balanceCents":"BalanceCents is the account balance AFTER the grant, in whole cents.",
"GrantResult.balanceExact":"BalanceExact is that same balance at full 18-decimal precision, so a sub-cent\ndebit is visible rather than rounded away.",
"GrantResult.currency":"Currency is the lower-cased ISO code the grant was denominated in.",
"GrantResult.grantedCents":"GrantedCents is the amount actually credited.",
"GrantResult.org":"Org is the tenant whose ledger was credited.",
"GrantResult.source":"Source is the money bucket: \"trial\" (non-cash comp) or \"prepaid\" (real money).",
"GrantResult.subject":"Subject is the ACCOUNT the credit landed on inside that ledger: the org slug for\na pooled org, \"<org>/<name>\" for a member of a per-member one. It is echoed\nbecause the operator does not choose it — account.Payer does — so naming a\nmember of a pooled org credits the pool and the receipt has to say so.",
"GrantResult.transactionId":"TransactionID is the ledger entry id, for reconciliation against commerce.",
Description:"Answers GET /v1/admin/finance. It reads the multi-vendor COGS from commerce\n/v1/costs, the DO promo-credit/burn-down treasury view, and the fleet commerce revenue,\nthen hands them to ComputeFinance. SuperAdmin only.",
Description:"Splits our upstream AI usage by how it was FUNDED: one row per (provider,\nmodel) over the window, tagged credit (provider grant still remaining), paid (grant\nexhausted) or paid_only (no grant at all).\n\nThe class is resolved at the PROVIDER level from the credit ledger, not per call — the\nper-call split, and the `byo` class, arrive when the metering write stamps a funding\ncolumn on cloud_usage and this can GROUP BY it directly. Until then a provider with\nremaining grant reports all of its usage as credit, which is right in aggregate and\napproximate at the boundary where a grant runs out mid-window.\n\nAn unparseable window falls back to the last 30 days rather than refusing: this is a\ndashboard read, and a typo in a date must not blank the board.",
Fields:map[string]string{
"UsageFundingIn.from":"From is the inclusive start of the window. Unparseable or absent, together with\nTo, falls back to the last 30 days.",
"UsageFundingIn.to":"To is the exclusive end of the window.",
Description:"Carries ONE org's current commerce prepaid balance into the native finance\nwallet — the one-time cutover between the two ledgers.\n\nIt is IDEMPOTENT: the deposit uses the fixed ref \"backfill:<org>\", so re-running it\ncredits the wallet at most once. Safe to retry.\n\nThe pre-migration balance is read from the CO-RESIDENT commerce ledger, not over HTTP:\nthe admin HTTP client dials an unroutable in-process address and would read $0, and a\nphantom zero would silently carry nothing while reporting success. When commerce is\nnot co-resident this fails rather than migrating nothing.",
Fields:map[string]string{
"BackfillIn.org":"Org is the tenant to migrate. Required — there is no fleet-wide form of this\ncutover, because each org must be reconciled on its own.",
"Backfilled.entryId":"EntryID is the finance ledger entry created, or \"\" when the balance was\nnon-positive and there was nothing to carry.",
"Backfilled.migratedCents":"MigratedCents is the balance carried across, read from commerce BEFORE the move.",
Description:"Destroys a droplet the board has just proven is NOT a DOKS node. There\nis no snapshot-first undo for a droplet the way there is for a volume: the local disk\ngoes with it.",
Fields:map[string]string{
"DropletIn.disk":"Disk requests a PERMANENT resize that grows the disk. DO can never resize such a\ndroplet down again, so it defaults false — a CPU/RAM-only change, reversible.",
"DropletIn.id":"ID is the DO droplet id, from the path. Numeric.",
"DropletIn.size":"Size is the target DigitalOcean size slug on resize, e.g. \"s-4vcpu-8gb\".",
Description:"Destroys a volume the board has just proven no PersistentVolume in any\ncluster references. Irreversible, so it snapshots first unless explicitly waived —\nthe snapshot IS the undo.",
Fields:map[string]string{
"VolumeIn.id":"ID is the DO volume id, from the path.",
"VolumeIn.name":"Name is the snapshot name on the snapshot action. Blank gets a deterministic\n\"<volume>-predelete-<unix>\" so the undo is findable in the DO console.",
"VolumeIn.sizeGiB":"SizeGiB is the target size on the resize action. A volume only ever grows —\nExpandTo is the verdict that refuses a shrink, so this is not validated here.",
"VolumeIn.snapshot":"Snapshot is the snapshot-first switch on DELETE. Anything other than the literal\n\"false\" snapshots before destroying — the snapshot IS the undo, so waiving it is\ndeliberate and explicit.",
Description:"Serves the whole DigitalOcean infrastructure board: droplets, volumes, DOKS\nclusters and load balancers, each cross-referenced against every cluster's live\nKubernetes state so the board can say what is safe to destroy and what is not.\n\nIt is cached for up to a minute because one read is a fan-out over the DO API plus a\nfull pod/PV listing per cluster. Staleness is never load-bearing: every MUTATION\nre-scans from scratch and ignores this cache.\n\nOnly an unusable DO account is a hard failure. A partial read still produces a board,\nwith the failing source named in sources[] — except for clusters and volumes, which\nthe safety verdict depends on; without those the analysis degrades rather than\nclassifying anything it cannot prove.",
Fields:map[string]string{
"Cost.wastedMonthly":"WastedMonthly is what the fleet pays every month for provisioned-but-empty space on\nthe volumes a kubelet actually measured.\n\nIt is NOT ReclaimableMonthly and must never be added to it. Reclaimable is money a\nbutton on this board collects, by deleting volumes proven to belong to no one.\nWasted is money locked inside volumes that are IN USE and holding live data:\nDigitalOcean can only ever grow a volume, so collecting it means copying a database\nonto a smaller one. See shrinkRecipe.\n\nIt is also a LOWER BOUND — unmeasured volumes contribute nothing.",
"LoadBalancer.service":"Service is the `namespace/name` of the live type=LoadBalancer Service that claims\nthis load balancer, proven from the cluster scan. Non-empty means IN USE.",
"Node.mutable":"Mutable reports whether this droplet may be changed DIRECTLY — deleted or resized.\nOne predicate covers both because one fact decides both: a DOKS node belongs to a\nnode pool, and the pool is the only thing allowed to change it.",
"ReadIn.refresh":"Refresh, when present, forces a full re-scan instead of serving the cached\nsnapshot. Every MUTATION re-scans regardless — this is only for the reader.",
"Totals.measuredVolumes":"Fill. MeasuredVolumes/UnmeasuredVolumes are the honesty denominator: UsedGiB and\nWastedGiB describe the measured set ONLY, so a board showing waste must show how\nmuch of the fleet the figure was computed from. Unmeasured capacity contributes\nnothing to either — it is not assumed empty, and it is not assumed full.",
"Volume.cluster":"Cluster/ClusterID are the PROVEN owner — resolved through a PV that names this\nvolume, never through the tag.",
"Volume.controller":"Controller is the workload owning the pod that mounts this volume\n(\"StatefulSet/luxd\"), or \"\" when nothing mounts it. It names who has to act.",
"Volume.expandable":"Expandable/ExpandBlockedReason are the GROW verdict, kept separate from Deletable\nbecause the two ask opposite questions: a volume is deletable when nothing uses it,\nand expandable when something uses it in a way this board can grow completely.",
"Volume.hasUsage":"HasUsage reports whether a kubelet actually MEASURED this volume's filesystem.\n\nFalse means NOT MEASURED. It does NOT mean empty, and the three fields below are\nmeaningless — not zero — when it is false. A reading exists only while a running pod\nhas the volume mounted on a node that answered; a detached, idle or unreferenced\nvolume has none. Rendering an unmeasured volume as \"0 used / 100% wasted\" would\ninvent the single most expensive lie this board could tell, so every consumer must\nbranch on this flag and show unknown.",
"Volume.tagCluster":"TagCluster is the `k8s:<uuid>` tag. ADVISORY ONLY: it outlives the cluster that\nset it. Shown so the operator can see tag-vs-truth disagree, never acted on.",
"Volume.usedBytes":"UsedBytes is the measured filesystem usage. BYTES, not GiB: the volumes this exists\nto catch hold a fraction of a GiB in 200, and rounding that to an integer GiB would\nprint the very 0 the flag above exists to prevent.",
"Volume.wastedGiB":"WastedGiB is provisioned minus measured, in the unit DigitalOcean BILLS: whole GiB\nof the volume's own size, never the filesystem's capacity — a 200 GiB volume carries\na 196 GiB filesystem after format overhead, and the invoice says 200.",
Description:"Sets a node pool's node count — the ONE correct way to change how many\nnodes a DOKS cluster has.\n\nThe response states what the board could NOT prove: DOKS picks which nodes a shrink\nremoves, so no particular pod is shown to survive one. See NodePool.ScaleTo.",
Fields:map[string]string{
"ScaleIn.count":"Count is the node count to set.",
"ScaleIn.id":"ID is the DOKS cluster id, from the path.",
"ScaleIn.pool":"Pool is the node pool, from the path. Its DO id or its name — both are unique\nwithin a cluster, and an operator reads the name off the board.",
Description:"Changes a droplet's plan. Same refusal as delete and for the same\nreason: a DOKS node's size is the node pool's to declare.\n\ndisk=true is a PERMANENT resize — the disk grows and DO can never resize the droplet\nDOWN again. disk=false (the default) changes CPU/RAM only and is reversible. DO\nrequires the droplet to be powered off and applies the change asynchronously, so the\nresponse carries the action to poll, not a completed change.",
Fields:map[string]string{
"DropletIn.disk":"Disk requests a PERMANENT resize that grows the disk. DO can never resize such a\ndroplet down again, so it defaults false — a CPU/RAM-only change, reversible.",
"DropletIn.id":"ID is the DO droplet id, from the path. Numeric.",
"DropletIn.size":"Size is the target DigitalOcean size slug on resize, e.g. \"s-4vcpu-8gb\".",
Description:"Marks one cluster node unschedulable — or schedulable again — and can drain\nthe pods already on it.\n\nIt is the ONE infra change that does not go through the run discipline, because there\nis no destructive verdict to check: cordoning is reversible and evicting respects the\ncluster's own PodDisruptionBudgets. It reads the cached board for the same reason.\nThe outcome is audited either way, and the result reports how many pods were evicted.",
Fields:map[string]string{
"CordonIn.cordon":"Cordon true marks the node unschedulable; false restores it.",
"CordonIn.drain":"Drain additionally evicts the pods already running there.",
"CordonIn.id":"ID is the node's droplet id, from the path.",
Description:"Grows a volume. GROW ONLY — see Volume.ExpandTo for why the other\ndirection is a data migration this board deliberately refuses to run.\n\nThe MECHANISM follows the volume's owner, because there is exactly one way to grow each\nkind completely. A volume a PVC claims is grown by patching the claim: the CSI driver\nthen resizes the DigitalOcean device AND grows the filesystem on it, leaving claim, PV,\ndevice and filesystem all agreeing. Calling DigitalOcean directly for that volume would\ngrow the device while the PV kept declaring the old capacity and the filesystem never\ngrew at all. One operation, one correct mechanism per owner — not two ways to do it.",
Fields:map[string]string{
"VolumeIn.id":"ID is the DO volume id, from the path.",
"VolumeIn.name":"Name is the snapshot name on the snapshot action. Blank gets a deterministic\n\"<volume>-predelete-<unix>\" so the undo is findable in the DO console.",
"VolumeIn.sizeGiB":"SizeGiB is the target size on the resize action. A volume only ever grows —\nExpandTo is the verdict that refuses a shrink, so this is not validated here.",
"VolumeIn.snapshot":"Snapshot is the snapshot-first switch on DELETE. Anything other than the literal\n\"false\" snapshots before destroying — the snapshot IS the undo, so waiving it is\ndeliberate and explicit.",
Description:"Takes a point-in-time snapshot of one volume — the undo a delete relies\non, available on its own so an operator can take one before any risky change.\n\nIt re-scans the board first (never the cache) so the volume it snapshots is one that\nexists right now, and audits the outcome either way.",
Fields:map[string]string{
"VolumeIn.id":"ID is the DO volume id, from the path.",
"VolumeIn.name":"Name is the snapshot name on the snapshot action. Blank gets a deterministic\n\"<volume>-predelete-<unix>\" so the undo is findable in the DO console.",
"VolumeIn.sizeGiB":"SizeGiB is the target size on the resize action. A volume only ever grows —\nExpandTo is the verdict that refuses a shrink, so this is not validated here.",
"VolumeIn.snapshot":"Snapshot is the snapshot-first switch on DELETE. Anything other than the literal\n\"false\" snapshots before destroying — the snapshot IS the undo, so waiving it is\ndeliberate and explicit.",
// TestFillHeadline proves the run-rate headline coercion (driver ints) + the
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
hanzo-dev
zandhanzo-dev