The hosted tag is now the per-site twin of track.js: on load it fetches
/v1/tags (dual-resolved per site by its key or host) and injects the
connected browser pixels (GA/Meta/TikTok/X) first-party, firing each event
to the native pixel — translated through the same taxonomy the server CAPI
uses and stamped with a shared event_id carried on the /v1/event row — so
the browser pixel and the server-side CAPI deduplicate. 11/11 behavioral
checks. The one-liner now does the whole loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
v1.50.28 is the one that matters: billing/allotment.Grant read "already granted
for this period?" and then wrote a transaction with a GENERATED id, so two
concurrent schedulers both read "no" and both created a row. The user received
the month's included credit TWICE, as real spendable balance. Three comments
asserted this was impossible, all resting on datastore.RunInTransaction — whose
body is "// For now, just run the function directly". There was no transaction.
Fixed by deriving the storage id from (user, period, mode), so concurrent writers
land on one row and the balance, a SUM over rows, moves once. Also fixed a second
bug the test found: the precheck filtered (DestinationId, Tags) but not Test,
while GrantedCents does — so a live grant silently suppressed the test-mode grant
for the same user and month.
Also riding along: v1.50.24 TON + XRPL deposit readers (XRPL matching a POOLED
address plus a per-intent destination tag, since XRPL's non-refundable reserve
makes one address per payer wasteful), and v1.50.27 XRPL tag minting, whose
uniqueness comes from a single INSERT … ON CONFLICT DO UPDATE … RETURNING —
measured against real Postgres to beat an explicit READ COMMITTED transaction.
Crypto remains gated: cryptoDepositsCanBeCredited is still a compile-time false
and no CRYPTO_DEPOSIT_* is configured, so the readers are inert.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every tool the Slack agent called answered "X-Org-Id required" or "sign in to
use", while tools/list and describe worked — and the plugin's own request log
said org=hanzo user=hanzo/z@hanzo.ai for the very request that refused. That is
what made it look impossible.
It is not. The run holds a principal it resolved SERVER-SIDE — the org a
workspace is installed in, the person the run is attributed to — and no bearer
to replay for it. The door forwarded that into the subsystem's EDGE door, where
SanitizeIdentity deletes every authority header and re-mints one only from a
verified credential. Right for a stranger, wrong for a sibling: the statement
was gone by the time the op read it, so principal.OrgOf refused the org that had
ridden along. The log was not disagreeing with the op — zip reports the caller
as the request BEGINS, so the log held an EARLIER fact than the handler did.
cloud.Door publishes the door on the internal Plane instead: the caller is read
off the request actually being served, and decided by the SAME predicate a
routed request gets — an org with no validated user is still refused, here as
there. What makes it safe is the ADDRESS, not an extra check: Plane listens on
the app's own socket and is never mounted on the edge router, so there is no
path to it from the internet, exactly as there is none to a route nobody
registered.
plugin/o11y/main.go needed the call spelled out because it is hand-written —
cloud.Listen adds it for every generated app main, and without it this app's
tools were the only ones in the fleet an inside caller could not reach.
door_test.go drives it from both directions against one subsystem.
Verified before landing: builds, and fleet + cmd/cloud + apps/o11y + the root
package all pass. The published surface does not move — openapi.yaml and
public.yaml re-weave to the same 1782/18.
Three commits added the per-site browser-tag surface — a `tags` map on the
project update and read, /v1/tags re-scoped from the ORG's pixels to the SITE's,
iam's enableCodeSignin — and none of them ran `make describe`. So the routes
moved and openapi.yaml did not, and app-contract has been red on main ever
since.
That gate is not decoration: openapi.yaml is the file the SDK repos pull, so a
route missing from it is a route no generated client can reach, and hanzoai/cli
derives its whole command tree from this document. `/v1/tags` was live, served
and reachable by nothing.
It is also what has been holding the release train. Every car declares needs: on
the gate, so nothing has shipped since — including the metrics tenant fix, which
is why production is still serving a build from before it.
No behaviour here. `make -f mk/fleet.mk check` regenerates every per-app subset
and weaves the golden; this is that output, committed.
Two tests pinned "no cloud.Bridge in front => 403", and that stopped being true
when principal.ValidatedFrom and OrgFrom gained their fallback to zip's caller.
They have been red on main ever since, and because every car of the release
train declares needs: on the test gate, nothing has shipped: the deployed build
is sha-35209c422c0b while main carries the metrics tenant fix three commits
later. A stale test is holding a security fix out of production.
What the 403 was pinning was an OUTAGE, not a boundary. A typed op takes a
context and its decoded input, so it sees its caller only if something parked
one — and the Bridge is not in front in two real places: o11y as its own binary
(a context value does not cross the host<->plugin socket) and MCP's tools/call,
which invokes an op directly so no route middleware runs at all. Both answered
403 to callers the host had already validated. The fallback fixed that.
So the tests now assert what is true and keep the teeth where the teeth belong:
a request carrying NO principal is still refused, on both shapes, and that
assertion is stated as the one that must never flip.
The fallback widens the door and not the trust because cloud installs
SanitizeIdentity app-wide and it DELETES every client-sent authority header
before a handler reads one. That reasoning covers cloud's own handlers and
stops there — it does NOT extend to a subsystem cloud proxies to, which is
exactly where the real defect was, and the comment says so rather than leaving
the next reader to over-generalise it.
hanzoai/metrics resolved its own tenant from X-Org-Id with a brand fallback and
no authentication, so /v1/{metrics,logs,traces}/* answered whatever org the
caller named. Verified live against api.hanzo.ai with no credential: a write
under an invented org returned {"written":1} and the matching query read it
back, while the same query without the header saw nothing — the header WAS the
boundary.
metrics cannot import cloud, which is why it grew its own resolver in the first
place. So the dependency inverts: metrics declares Deps.Org and cloud supplies
principal.Org, the one predicate that already requires a validated principal
beside the org claim. One rule, applied by the module that cannot see it.
Pin moves v1.110.2 -> v1.110.6, which also picks up the three intervening
releases (zip v1.18.1, go 1.26.5, CI removal) — deps and build only, no route
behaviour. The published contract is unchanged at 1782/18 paths: this is a
change in who may call, not in what is served.
Slack does not read Markdown, it reads mrkdwn, and the two share enough
punctuation to look identical until they render: **bold** keeps its
asterisks, ### prints a hash, [text](url) shows its brackets. Every model
writes Markdown — it is what they are trained on — so the translation
belongs at the edge rather than in a system-prompt rule a model drops by
the third turn.
Applied at the two seams model prose leaves through (the reply closure and
SendSlackAt), NOT inside slackChatPost: Block Kit callers here already
write mrkdwn by hand, and `*bold*` is mrkdwn bold and Markdown italic, so
a blanket pass at the bottom would rewrite every one into `_bold_`.
Code is exempt. Inside a fence or a span, `**x**` is text the reader is
meant to see, and translating a sample is worse than the bug being fixed.
Extends PATCH /v1/projects/:slug to accept `tags` (platform → non-secret
pixel id), sanitized + stored on Project.Tags, and returns them on the
project view. This is how a site's GA/Meta/TikTok/X pixel ids are set —
consumed by /v1/tags (client injection) and the server CAPI. The secret
(CAPI token) stays sealed via POST /v1/destinations; this is ids only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A project IS a site, so browser tags belong on it: adds Project.Tags
(platform → non-secret pixel id) to the projects store (additive column +
JSON codec). New projects.TagsFor(key, host) dual-resolves the site — by
the publishable key when it's a per-site project key (ResolveKey), else by
the request host (ResolveHost) for an org-level key. /v1/tags now reads
that instead of per-org config, so hanzo.ai and hanzo.chat inject different
pixels under one org. Client half; the fan-out goes per-site next.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Translate now keys the Conversion EventID on properties.event_id (which
track.js stamps into the event and fires on the browser pixel) when set,
falling back to messageId. This is what makes the browser pixel and the
server-side CAPI deduplicate instead of double-counting a conversion.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Built from hanzoai/admin@a4495ce, where the tracker stopped carrying 583
lines of git.hanzo.ai's own stylesheet and moved onto @hanzo/gui primitives
and @hanzogui/admin tokens — the substrate the console already paints with.
The chrome was always right; everything under it belonged to another product.
Also in these bundles: the brand registry now sends Zoo to zoolabs.id, and
useFetch bounds a read at 45s — longer than this binary's own 30s budget, on
purpose, so a client abort can never kill the request that warms the cache.
apps/tracker/ui and apps/meet/ui embed_test.go pin the base; the drift and
dist-integrity gates pass on the new bytes.
Re-reading the repository list for the 250-repo org is fifty requests to
the endpoint measured degrading from 13s to 22s under load. A one-minute
window put that on the forge every minute, per actor, to learn that nothing
had changed — a cache whose refresh is heavier than the thing it spares.
Nothing a board renders goes stale for it: issues are not cached, so cards,
columns and assignees stay the forge's current answer, and a new repository
that has work on it appears immediately because the list is built from
issues. Only an EMPTY repository created in the last few minutes waits.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The board sat on loading skeletons with an empty browser console. Two
independent defects produced that one symptom, and each hid the other.
THE ORG WAS WRONG. The IAM tenant is `hanzo`; its work lives on the forge
under `hanzoai`, the name the estate uses wherever a namespace is written
down. A near-empty `hanzo` org also exists, so asking by name answered 200
with an empty list — a wrong answer wearing a healthy one's clothes.
Measured on git.hanzo.ai: `hanzo` is 64 repos and 0 issues, `hanzoai` is
250 repos and the actual work; hanzo/cloud is 404 where hanzoai/cloud is
200. forgeOwners maps the two, as a declared value rather than a branch
inside the resolver, applied to the VALIDATED principal's org and never to
anything a caller sent. It does not touch the Sudo actor, so a wrong entry
can show an empty board but cannot widen what anyone may read.
NOTHING BOUNDED THE READ. The forge client's timeout bounds one request
while a board read makes many, so a slow forge was an unbounded wait: no
response, no error, and a page that looks like it is still working. Every
forge-backed op now runs under one budget, applied in one place, and a
timeout answers 504 with its reason — never an empty list, which renders as
"you have no work" and is indistinguishable from a correct answer.
The board list no longer waits on the repository inventory. That endpoint
charges per repository RETURNED — ~21s for one of five pages of the 250-repo
org — while issues-search answers the whole org in ~1.5s and carries the
repository, labels, milestone and assignees inline. The list is built from
issues; the inventory is read only when already warm, and completes the list
with the boards that have no work on them yet. A board addressed by name is
one direct repository read rather than a scan of the org.
Behind that, the two list endpoints are read through a per-(actor, org)
cache. The key carries the ACTOR because the forge answers as the sudoed
user: keyed by org alone it would serve one user another's visible set,
which is a disclosure dressed as a cache hit. Stale entries are served while
one refresh runs behind them, so only a genuinely cold cache blocks, and a
credential rotation no longer discards a warm list. Where the inventory is
still needed it pages CONCURRENTLY: the forge's per-repo cost serialises
inside a request but parallelises across requests, taking the same 64 repos
from 20.7s to 2.7s.
Tests: 74 across the two packages, forge under -race. The regression tests
are a wedged forge answering 504 rather than hanging, the board asking
`hanzoai` when the principal says `hanzo`, the list not blocking on a slow
inventory, and the cache refusing to serve one actor's repositories to
another. TestLiveOwner checks each mapping target actually holds work —
no stub can catch a namesake, because a stub answers what it was told.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
brand.go said zoo's issuer is https://zoolabs.id. Four other places said
https://zoo.id — a host with NO DNS RECORD, unregistered, that the live Zoo
IAM has never stamped:
- the tracker bundle embedded here and live at tracker.hanzo.ai
- the meet bundle, same
- apps/skills/catalog/zoo/index.json, which is the document an agent READS
to learn where to get a token for api.zoo.ngo
- that catalogue's SKILL.md, twice, in prose
Go was right and alone. Every Go test passed the entire time, because the
copies that were wrong are not Go: they are compiled into a bundle and
generated into a catalogue, and nothing compared them to the registry.
The severity is not the outage. A shipped artifact naming an identity host
the validator never vouched for is the precondition for cross-brand token
confusion — the set of origins a surface may send a credential to is meant
to be CLOSED by the registry, and a name anyone could register is a hole in
that closure.
The generator is fixed at the root (hanzoai/openapi skills.py): the issuer
was being read as "the brand's .id host", a rule that looks like it holds
and does not. It is a value per brand now, and the catalogue digests are
recomputed from the served bytes rather than typed.
TestShippedArtifactsNameOnlyRegisteredIssuers is the gate. It reads the
ARTIFACTS — the embedded dist bundles and the catalogue, the exact bytes
go:embed puts in the binary — and fails when any of them names an identity
host brand.Issuers() does not declare. A pin against a hand-written mirror
of the Go table would have passed all along; that is precisely what the TS
side had. Verified by reverting the fix: it fails on both bundles.
TestWhiteLabelByHost now DERIVES every expectation from brand.*, so the
disagreement it was meant to catch is unrepresentable rather than untested.
Both halves of its old table had failed silently: the `issuer` field was
declared and never asserted, and the cross-brand marker "zoo.id" was a
string the registry emits nowhere, so it could not fire. It asserts the
served issuer now, and mutation-testing confirms it fails without the fix.
dist_integrity_test.go takes the two facts about committed build artifacts
that no per-app embed_test.go owns: no git conflict marker (one reached
production, past every gate, because nothing in CI renders an SPA), and
every hashed chunk index.html names is actually present.
Grafted, IAM and notify share a process, so the org travels as an ARGUMENT --
which is the whole reason this exists beside the standalone iam's HTTP client.
notify's HTTP surface derives the sending tenant from the validated principal
and never from a header, so a service credential can only send as its own org;
that is wrong for a binary answering for every white-label identity host, and an
unpinned client would push lux and zoo codes through hanzo's Twilio while
reporting success. notify.Send takes the org explicitly, precisely for callers
carrying no request principal, so in-process delivery serves every tenant with
no credential to mint, hold or rotate and no cross-tenant hole in the public
surface.
Binding it lights four things at once, because email sign-in, SMS sign-in and
the email and SMS second factors all read one predicate that reports on the
bound sender. Until this line runs IAM correctly hides all four rather than
advertising a method it cannot finish.
Requires iam v1.34.29, which exports the seam: it lives in internal/oidc and was
therefore unreachable from a host that grafts IAM.
TestFleetIsTheWeaveOfItsApps was already red on forge/main before this merge,
and the drift was one line: /v1/websearch declares search_web in source while
the woven document still said post_v1_websearch. That identifier is what the
SDK cuts a method name from, so the golden was publishing a name the fleet had
stopped serving under.
`make describe` regenerates all 121 app subsets to no change — the subsets were
current the whole time. Only the weave lagged, which is exactly the gap the
gate exists to catch.
forge and inc2 had diverged 3/1 with no overlapping files — forge carried the
cicd credential message and the meet prose, inc2 the org fence and the build
that no longer needs the tags to be stated.
cloud pinned base v1.5.15, which had already deleted the compile-time guard, next
to sqlite v0.5.1, which still had the tag-gated cgo backend. That pairing is the
worst of the three: nothing refuses to build, and a cgo binary answers "no such
function: acos" at runtime instead. It has been safe only because the Dockerfile
passes -tags "libsqlite3 sqlite_fts5 sqlite_math_functions" by hand, so the tags
were load-bearing rather than incidental.
base v1.5.17 + sqlite v0.5.2 + csqlite v0.1.2 supply the capability itself: both
backends compile ENABLE_MATH_FUNCTIONS and ENABLE_FTS5 unconditionally, and
sqlite's TestBackendParity asserts they answer the same SQL under CGO_ENABLED=1
and 0. `go build ./...` is now clean here under BOTH, with no tags passed.
The ha rename came with base v1.5.17 (it requires ha v0.2.0, which renamed Fencer
-> Leases and StaticFencer -> StaticLeases). Followed upstream rather than keeping
a local synonym for one thing; the whole surface was two interface assertions and
the re-export block, and internal/org's tests pass under both modes. CASFencer
keeps its name — it is our CAS-backed implementation, not the interface.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The CAS moved to the forge, which answers 409 where github answered 422, and
the catch-all error still said "expected 201 or 422" — it would have printed
the wrong contract for the one status it exists to explain.
It also now separates the two ways a claim is refused. A 401/403 is not a
version collision, it is FORGE_TOKEN lacking write:repository, and this is
the first thing in this workflow to WRITE with that token — every prior use
is a go module fetch or an ls-remote. That makes it the most likely first
failure and the least self-evident, so it says which secret and which scope
rather than leaving a bare status code.
Two things pay.hanzo.ai has been waiting on ride in one bump, and neither is a
pay change: GET /v1/commerce/tenant answers logoUrl and primaryColor from
commerce's brand table, embedded here, so until this moves the checkout draws a
wordmark and a white accent no matter what the SPA does.
- logoUrl: the brand struct always had the field and no brand ever set it, so it
answered "" forever. Only hanzo carries one -- every other brand's CDN 404s or
522s, and lux.network's 200 is 112KB of text/html rather than an image.
- primaryColor #808000: olive has been in brandHanzo for a while and prod still
serves #ffffff, which is the whole reason it never appeared.
Verified against the resolved module rather than the source: the logoURL and the
olive are both on brandHanzo in GOMODCACHE at v1.50.25, and cloud builds clean
against it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/meet/zipdoc_gen.go was stale: /v1/meet/session and /v1/meet/getToken
were registered without their lift being regenerated, so zipdoc-check went
red and took the whole gate with it — and the gate is what the image job
needs, so no release could publish while this sat there.
Generated, not written. Note that both routes lift the doc of cloud.Handle
rather than of mint/session, which do carry good prose of their own: zipdoc
resolves the outermost call in `cloud.Handle(s, mint)` and stops there. That
is pre-existing and repo-wide — 24 of 108 zipdoc_gen.go files describe the
adapter — so it is not this commit's to fix, but those two descriptions are
wrong in openapi until the generator learns to unwrap.
MCP's tools/call invokes an op directly, so no route middleware runs and
nothing parked principal's org/validated slots. Every op gating on
ValidatedFrom was therefore unreachable as a tool, and every org-scoped op
resolved no tenant: websearch refused "sign in to search the web" for a
request that reached the plugin carrying org=hanzo and a validated user.
The gate had already been moved out of middleware and into the handler so
that every door would reach it — but the fact it read was still parked by
the one door tools/call does not pass through, so it was unsatisfiable
exactly where the move was meant to make it work.
OrgFrom and ValidatedFrom now fall back to zip's caller, which carries
across every door since zip's own caller_mcp_test. That is one more READER
of OrgOf, not a second rule: an org riding along without a validated user
is still refused, so the fallback widens the door and not the trust.
Also names the op search_web. `create_websearch` is what a POST to
/v1/websearch derives, it reads as "make a websearch", and a model picking
from an `op` enum reads the name before any description.
Two failures, one cause: this workflow asked the forge for values it does not
set, and read the answers as meaningful.
THE CONCURRENCY GROUP. It was cicd-${{ github.ref }}, and act_runner leaves
the ref empty — head_branch is null and there is no ref field on an ordinary
push to main. So the group rendered as the constant `cicd-` and every run in
the repository shared one slot. Gitea coalesces a concurrency queue to depth
one, which is correct on its own terms, so each arrival cancelled whatever was
queued: pull requests ate releases and releases ate pull requests. The runs
carry started_at 1970-01-01 — never given a runner. Two earlier passes blamed
cancel-in-progress and argued about its truthiness; the flag was never what
acted. Keyed on the event instead, releases queue in one slot and each pull
request gets its own.
THE CLAIM. It was made against github, so a step had to publish the commit
there first, and that step broke the lane twice — once pushing the private
tree to a public repo, once against a private repo the credential could not
see (run 38319: "Repository not found"). The forge answers 201 on create and
409 on collision, which is the same compare-and-swap for a register that now
lives in the git the code is already in. So the publish step is gone, along
with the mirror race it existed to close.
Receipts move with it: a smoke receipt is a push to refs/smoked/<digest>,
which is the same atomic create without needing an endpoint for a ref outside
refs/tags. Release notes likewise. The only github calls left are fanout's
dispatches to downstream SDK repos, which really are there.
A product reached the public SDK by being served. That is the wrong
default: the surface a customer can call was decided by whoever mounted a
route, and 2325 operations — admin, billing, commerce, o11y, treasury —
were one generator run away from a published client.
openapi.Public(path, method) is the third declaration seam, beside
Describe and Compat. It renders x-public: true, and the weave projects the
SAME composition twice: openapi.yaml is everything the fleet serves and is
what our own clients are cut from; public.yaml beside it is what declared
itself part of the contract. An operation that says nothing is internal, so
nothing reaches a customer by anyone forgetting.
The first declaration is inference and nothing else: 18 operations across
chat, completions, messages, embeddings, rerank, models, and the image,
video and audio doors. Each modality has its OWN door — publishing text
alone would ship a catalog naming zen-image and zen-video that no client
could call.
Two files, one command, one composition. `make check` now covers
public.yaml, so the two documents cannot come to describe two commits.
Also: name /v1/tags on the destinations row. It is destinations' browser
half, a bare noun, so the /v1 remainder answered it — 404 for a path the
document publishes and track.js fetches on every page load.
POST /v1/meet/getToken MINTS A CREDENTIAL to a live conversation, and it was
reachable with nothing but an ambient session cookie. callerToken accepts five
cookie names, validatedPrincipal mints from whatever it finds, and admits
selects the IAM lane on principal.Minted BEFORE it looks at an Authorization
header — so a signed-in tab could mint with no header at all. The deployment's
CORS policy reflects *.hanzo.ai with credentials, and that wildcard covers
hosts serving arbitrary user content, so a page there could mint a join token
for a room it was never admitted to.
meet had no route group at all, which is why there was nowhere for a gate to
live. It now has one, carrying account's RequireCSRF verbatim rather than a
second implementation of the same rule.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
touched fired only AFTER an exec returned, so LastUsedAt answered "when did a
call last FINISH". A sandbox in the middle of a forty-minute run therefore read
as forty minutes idle, and any idle window shorter than the longest legitimate
call would reap the work it was waiting for. The stamp now happens on entry too,
so it answers the question the reaper is actually asking.
This is what has to be true BEFORE the idle window can be shortened. The window
stays an hour here: a single call longer than the window still goes stale
mid-flight, and closing that needs the in-flight call to keep saying so rather
than one more constant.
Why the window WANTS to be short: a disconnected client whose agent has stopped
is holding a pod, and an hour of that per abandoned run is the difference
between how many tenants a node carries. That is the real cost, and it is worth
the in-flight heartbeat rather than a guess.
apps/destinations registers, serves and PUBLISHES GET /v1/tags (tags.go:60-61,
destinations.go:171) while its manifest row claimed only /v1/destinations. A
prefix owns its whole subtree, so /v1/tags fell through to ai's bare /v1
remainder and answered ai's 404. api.hanzo.ai/v1/tags returns 404 in production
right now, for an operation the document has been advertising to every SDK
generator — indistinguishable from an unmounted route from the outside, which is
how it survived.
Mounting is half an address: the app says what it will answer, the manifest row
says what the router may hand it. This is the other half.
Full suite at -p 4: ZERO failures, twice. The "baseline" of 3 was two missing
sqlite build tags; the residual 1 was apps/o11y flaking under the default
parallel sweep (passes 3/3 standalone). There is no baseline. It is green.
LastUsedAt already told us the difference between a run that is progressing and
one that is abandoned — it is stamped by every exec and every fs call, and the
idle rule already trusted it to KILL. Trusting it in one direction only was the
bug: activity could shorten a lease and never lengthen it, so a deep-research
run or a long build that was demonstrably alive still died at its 4-hour TTL.
That is the one case where reaping destroys the most work, and it fired on
exactly the runs worth keeping.
A busy sandbox — one within 10 minutes of expiry — now has its lease pushed to
now+1h, bounded by a 24-hour ceiling measured from CREATION. It stays a lease:
the ceiling is absolute, so no amount of activity turns a sandbox into a
permanent resident, and a job still working after a day wants a Deployment
rather than another hour.
The two rules compose rather than compete, and the idle rule still outranks:
untouched for an hour ends a sandbox whatever its lease says. Extension is only
reached by a sandbox that has been touched inside that window.
Store.Extend moves a lease FORWARD only — the `expires_at<?` guard means a stale
caller with an older timestamp cannot pull a lease in and kill a sandbox early.
That is what makes the sweep safe to run every minute, from more than one place.
apps/sandbox builds and its tests pass.
token gate — apps/destinations/x.go reached for crypto/hmac and was not named.
It is OAuth 1.0a request signing: HMAC-SHA1 over the method+URL+params base
string under consumerSecret&accessSecret, X's contract. It signs an OUTBOUND
call under credentials the tenant configured and mints nothing this deployment
would honour, which is the same shape venue/aws_sigv4.go is allowed under. Named
with the reason, because the gate exists so this list cannot grow in silence.
openapi — the host served 1773 and the document carried 1772. Regenerated from
the code: 1782 paths. Fourth time today git has resolved that GENERATED file to
one side while the routes merged to the union; it is not a merge artifact worth
resolving by hand, only by regenerating.
Verified with the tags the shipped build carries (sqlite_fts5 +
sqlite_math_functions): root, cmd/cloud and openapi all ok.
THE RUNNER. The seam had one implementation and it could not run: it POSTed to
bot's /v1/coding-tasks, which invokes the `docker` CLI, and bot-gateway has
neither that binary nor a socket. Every dispatch 503'd while the chain read as
configured.
The defect was never the ISOLATION — that path asked for --runtime=runsc or
kata-runtime, the same boundary a sandbox pod gets. It was asking through a CLI
that is not installed. apps/sandbox asks the apiserver for a Pod carrying
runtimeClassName instead: same boundary, a caller that exists. All three are
registered on the cluster (gvisor, kata-clh, kata-fc) and live pods run gvisor
today, so SANDBOX_RUNTIME_CLASS is a deployment field and never a fork in code.
That matters because a benchmark inverted the expected answer: Firecracker beat
gVisor on BOTH axes (git status 82ms vs 980ms, start 294ms vs 881ms, ~57 MiB
either way).
Every hop is plane.Ask — a typed op over ZAP/UDS. apps/bots' own doc says its
bytes "should move over ZAP" and that the swap is "a change to THIS FILE plus
each caller's one stub"; this skips the migration rather than performing it,
because a plane op IS ZAP. bots keeps its HTTP transport for bot traffic.
The five ops are the same ones the fleet door now publishes to agents
(lease_sandbox, run_in_sandbox, ...). One surface, whether a human drives a
sandbox from chat or a coding run drives one for a task.
COMPUTE IS NOT FREE, and it was. apps/sandbox had ZERO metering — no meter, no
debit, no usage — while a dev lease holds a pod for up to FOUR HOURS. An org
could lease one from Slack the moment sandboxes landed on the door.
A run now passes the prepaid gate BEFORE the lease, because a lease holds a pod
whether or not anyone can pay for it. It asks commerce (finance_authorize, "the
prepaid gate") rather than reading a balance and subtracting here: the gate
already knows about spend caps, which is why Verdict carries NoFunds and
CapSpent as separate bits — one says add money, the other says wait for the
period to roll. Arithmetic in this file would be a second pricing implementation
that can disagree with the first.
15 minutes of funded compute is the bar: long enough that no honest run is
refused for a rounding error, short enough that an unfunded account cannot open
a four-hour hole and walk away.
It fails CLOSED on an unreadable verdict, and plane.go says why in the type
itself: Reason "is an upstream failure the caller must treat as UNKNOWN and fail
closed on, and never read as permission."
The org is stated on a DETACHED context — commerce takes org from the caller's
identity, never an argument, and a Runner has no inbound request to carry one.
STILL MISSING, and it is the other half: the lease is GATED but not METERED.
Nothing debits as it runs. apps/agents already has the pattern
(cloud.NewResourceMeter, meterKind "agent") and a sandbox is a resource with a
duration. Also: maxLiveExec bounds exec at 16/org and dev/desktop have no
equivalent, so a per-query spawn path is still unbounded for one tenant.
go build ./... clean; apps/coding passes.
subScopeHeaders' doc grouped X-App-Id and X-Billing-Account-Id as "caller
attribution hints (no isolation boundary): forwarded as-is on the validated
path". That still describes X-App-Id and has not described the billing account
for some time: sanitizeSubScopes mints it from the validated billing_account
claim and deliberately does NOT restore the client copy, because Payer resolves
who pays from it -- a restored client value would be a caller naming its own
payer.
The function that enforces the rule explains it at length. The list above it said
the opposite, and that is the more dangerous of the two to read first: it
describes the header as deciding nothing, so the edit it invites is forwarding a
copy again.
Comment only, no behavior change.
act_runner does not populate the ref on these runs — the API returns
head_branch null and no ref field, on ordinary pushes to main, not only on
the API reruns an earlier note blamed. The condition allowed
ref=='refs/heads/main' or '' or null, which reads as though it covers that
and does not: an absent value satisfies none of the three, so the predicate
was false and the job was SKIPPED.
Skipped rather than failed is why this survived every attempt to fix it.
Run 38052 is the shape: gate success, containment success, then image,
rollout, reach, fanout and receipt all skipped — a green run that shipped
nothing and left no error to read. Work went into the gate, which was not
what was failing.
`on:` already decides what reaches this workflow, so the job no longer
re-derives it. It names the two things it must refuse — pull requests, and
tag pushes, since the tag is minted by the claim below and building on one
would publish a second image for a commit that already has one. An absent
ref is not a tag, which is the right answer rather than an accident.
/v1/audio/transcriptions has carried a full doc comment since it landed, but
comments do not survive compilation: routerdoc lifts each handler's sentence
into a generated table at generate time, and the commit that added the route
never re-ran it. openapi.Table refuses a document in which any served route
says nothing about itself, so describe/ai failed, the gate failed, and the
image job — which needs the gate — never started. Every cloud release since
has been blocked on one missing row.
Nothing about the endpoint changes; it simply publishes the sentence it
already had. plugin/ai/openapi.json is the regenerated projection.
A framing host that appends a query to a directory-shaped URL builds
/terminal/?ticket=…, and a host that joins path segments builds /terminal. The
page derives its SOCKET from its own path, so if only one spelling reached it the
other would load and then dial an address that is not there — a blank pane with
nothing in the log to explain it.
Both work already; this is the test that says so, because it is a property of the
router's configuration rather than of anything in this package, and a property
nothing measures is a property that changes without anyone noticing.
A socket is not a product. Every host that wants to show a shell — the console's
dock, tabs' panes, whatever is next — would otherwise carry its own emulator, its
own resize arithmetic and its own reconnect affordance, and the day one of them
is wrong is the day one product's terminal is quietly worse than another's. So
the terminal is SERVED, and a host embeds it.
POST /:id/terminal/ticket the credential {ticket, expiresIn, url}
GET /:id/terminal the terminal, as a page ?ticket=&arg=
GET /:id/terminal/ws the terminal, as a socket ?ticket=&arg=
The page and the socket are two addresses because a ticket is spent ONCE: a page
that redeemed it would hold a credential that no longer opens anything. So the
page is inert and ungated, and the socket is the gate — which is also why the
page needs no auth of its own to be safe.
THE PAGE IS A CONSTANT. Nothing is substituted into it; it reads its own
location, and the socket is this document's path plus /ws carrying this
document's query string unchanged — where the ticket and the session name already
are. A document with no substitution has no injection surface, so the whole class
of "a value from the URL reached the markup" does not exist. xterm ships inline
rather than from an address, because a terminal that fetches its emulator from
somewhere else stops working when the somewhere else does.
It posts {source:"hanzo-term", ready:true} to its parent when the terminal is up,
because a framing host has no other way to tell a live terminal from a page that
failed into a sign-in, and a host that guesses gets a blank pane it cannot
explain.
?arg NAMES A SESSION, which is what lets ONE sandbox hold MANY terminals: the
shell runs under `tmux new -A -s <arg>`, attaching if it exists and creating if
it does not, so a pane reframed finds what it left instead of a fresh empty
shell. The name reaches a command line, so what may be in it is an ALLOWLIST and
not an escape — `-` alone would be read by tmux as a flag — and it is quoted as
well, because one of the two being wrong later should not be enough on its own.
tmux is asked for, never required: an image without it gets the plain shell,
since a missing multiplexer should cost a caller its session names and not its
terminal. Measured on the live image (oci.hanzo.ai/hanzoai/sandbox): tmux, bash
and the hanzo CLI are all there.
WHO MAY FRAME IT is derived from the brand registry rather than listed, because a
list rots — tabs.hanzo.ai is one host of many that will want a terminal, and the
next is a subdomain added by somebody who never finds this file. brand.Domains()
is the enumeration beside ForHost's single-host answer, so adding a brand admits
its hosts everywhere at once. It is defence in depth against a clickjack; the
ticket is still the gate.
And the terminal now keeps its own sandbox alive. The reaper ends a sandbox that
has gone an hour untouched, attention is stamped by exec and fs calls, and a
terminal makes neither — so a session somebody was sitting in read as abandoned
and the pod would have been taken out from under it at the hour mark, mid
command. The heartbeat says the session is alive to both the client and the
reaper, because that is one fact.
TikTok now renders our line items into its native contents[] (content_id,
content_name, content_category, brand, price, quantity) + content_type,
alongside value/currency — Value-Based Optimization reads product detail,
not just a total. X now carries number_items from the line-item quantities.
Brings both to parity with Meta CAPI / GA4 items[] / Pinterest contents.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The client half of the one config: an org connects a destination once, and
that single row drives BOTH the server-side CAPI fan-out AND (for platforms
with a browser pixel: GA/Meta/TikTok/X) the tags the hosted tag injects, so
browser tag and server conversion share ids and never drift. Public,
pk--keyed like /v1/event(.js), non-secret ids only, fail-safe to an empty
set so a page never breaks on its tag config.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Google Ads PROPER, not GA4: uploads real conversions against a configured
conversion action via the Ads API, OAuth2-refreshed, attributed by gclid
and/or enhanced-conversion hashed identifiers (email/phone). Only
conversion-class events with a Google match key are uploaded; traffic and
keyless events are skipped. Composite JSON secret carries the four OAuth2
values; developer-token + login-customer-id headers per the Ads API.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes the X adapter: the payload builder was already done, this adds
the OAuth 1.0a HMAC-SHA1 request signer and wires Send. The four OAuth1
parts ride as one composite JSON secret (consumer key/secret + access
token/secret), because the fan-out resolves a single primary credential
per destination — no fan-out change. Signature cross-checked in tests by
independently recomputing the HMAC over the base string.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the server-side Pinterest conversion sink on the shared Destination
interface — closes the Pinterest gap in the fan-out. Bearer-token auth,
SHA-256 advanced matching (em/ph/external_id), the epik click id lifted
and deduped against the browser tag via the shared event_id, ecommerce
contents, value-as-string per Pinterest's v5 contract. Custom events
collapse to the "custom" enum value the API mandates.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/audio/transcriptions has carried a full doc comment since it landed, but
comments do not survive compilation: routerdoc lifts each handler's sentence
into a generated table at generate time, and the commit that added the route
never re-ran it. openapi.Table refuses a document in which any served route
says nothing about itself, so describe/ai failed, the gate failed, and the
image job — which needs the gate — never started. Every cloud release since
has been blocked on one missing row.
Nothing about the endpoint changes; it simply publishes the sentence it
already had. plugin/ai/openapi.json is the regenerated projection.
A framing host that appends a query to a directory-shaped URL builds
/terminal/?ticket=…, and a host that joins path segments builds /terminal. The
page derives its SOCKET from its own path, so if only one spelling reached it the
other would load and then dial an address that is not there — a blank pane with
nothing in the log to explain it.
Both work already; this is the test that says so, because it is a property of the
router's configuration rather than of anything in this package, and a property
nothing measures is a property that changes without anyone noticing.
A socket is not a product. Every host that wants to show a shell — the console's
dock, tabs' panes, whatever is next — would otherwise carry its own emulator, its
own resize arithmetic and its own reconnect affordance, and the day one of them
is wrong is the day one product's terminal is quietly worse than another's. So
the terminal is SERVED, and a host embeds it.
POST /:id/terminal/ticket the credential {ticket, expiresIn, url}
GET /:id/terminal the terminal, as a page ?ticket=&arg=
GET /:id/terminal/ws the terminal, as a socket ?ticket=&arg=
The page and the socket are two addresses because a ticket is spent ONCE: a page
that redeemed it would hold a credential that no longer opens anything. So the
page is inert and ungated, and the socket is the gate — which is also why the
page needs no auth of its own to be safe.
THE PAGE IS A CONSTANT. Nothing is substituted into it; it reads its own
location, and the socket is this document's path plus /ws carrying this
document's query string unchanged — where the ticket and the session name already
are. A document with no substitution has no injection surface, so the whole class
of "a value from the URL reached the markup" does not exist. xterm ships inline
rather than from an address, because a terminal that fetches its emulator from
somewhere else stops working when the somewhere else does.
It posts {source:"hanzo-term", ready:true} to its parent when the terminal is up,
because a framing host has no other way to tell a live terminal from a page that
failed into a sign-in, and a host that guesses gets a blank pane it cannot
explain.
?arg NAMES A SESSION, which is what lets ONE sandbox hold MANY terminals: the
shell runs under `tmux new -A -s <arg>`, attaching if it exists and creating if
it does not, so a pane reframed finds what it left instead of a fresh empty
shell. The name reaches a command line, so what may be in it is an ALLOWLIST and
not an escape — `-` alone would be read by tmux as a flag — and it is quoted as
well, because one of the two being wrong later should not be enough on its own.
tmux is asked for, never required: an image without it gets the plain shell,
since a missing multiplexer should cost a caller its session names and not its
terminal. Measured on the live image (oci.hanzo.ai/hanzoai/sandbox): tmux, bash
and the hanzo CLI are all there.
WHO MAY FRAME IT is derived from the brand registry rather than listed, because a
list rots — tabs.hanzo.ai is one host of many that will want a terminal, and the
next is a subdomain added by somebody who never finds this file. brand.Domains()
is the enumeration beside ForHost's single-host answer, so adding a brand admits
its hosts everywhere at once. It is defence in depth against a clickjack; the
ticket is still the gate.
And the terminal now keeps its own sandbox alive. The reaper ends a sandbox that
has gone an hour untouched, attention is stamped by exec and fs calls, and a
terminal makes neither — so a session somebody was sitting in read as abandoned
and the pod would have been taken out from under it at the hour mark, mid
command. The heartbeat says the session is alive to both the client and the
reaper, because that is one fact.
The image job published every release commit to github.com/hanzoai/cloud —
the PUBLIC repo, the one that carries the Apache-2.0 OSS core — as
refs/heads/forge-head, and claimed refs/tags/v<N> there too. So the register
of release names lived in a different repo from the bytes, and the full
private tree was world-readable at a ref nobody chose.
It surfaced as a stopped pipeline rather than as a leak. GitHub push
protection refused forge-head over two commits carrying deliberately
fake-shaped credentials — the fixtures that prove the secret-shape detector
works — and since those are ancestors of main they ride every push, so no
release has published an image since 21:22 UTC. The unblock URL in the log
would have re-opened the leak and taught the scanner to ignore the one shape
we most want it to catch.
Claiming against the private repo fixes both at once, and restores the
property the lane was missing: the name and the bytes it names come from the
same git. Numbering continues from v1.801.491; 487-490 stay holes, since a
version nothing published is not a release.
The image job published every release commit to github.com/hanzoai/cloud —
the PUBLIC repo, the one that carries the Apache-2.0 OSS core — as
refs/heads/forge-head, and claimed refs/tags/v<N> there too. So the register
of release names lived in a different repo from the bytes, and the full
private tree was world-readable at a ref nobody chose.
It surfaced as a stopped pipeline rather than as a leak. GitHub push
protection refused forge-head over two commits carrying deliberately
fake-shaped credentials — the fixtures that prove the secret-shape detector
works — and since those are ancestors of main they ride every push, so no
release has published an image since 21:22 UTC. The unblock URL in the log
would have re-opened the leak and taught the scanner to ignore the one shape
we most want it to catch.
Claiming against the private repo fixes both at once, and restores the
property the lane was missing: the name and the bytes it names come from the
same git. Numbering continues from v1.801.491; 487-490 stay holes, since a
version nothing published is not a release.
TikTok now renders our line items into its native contents[] (content_id,
content_name, content_category, brand, price, quantity) + content_type,
alongside value/currency — Value-Based Optimization reads product detail,
not just a total. X now carries number_items from the line-item quantities.
Brings both to parity with Meta CAPI / GA4 items[] / Pinterest contents.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The client half of the one config: an org connects a destination once, and
that single row drives BOTH the server-side CAPI fan-out AND (for platforms
with a browser pixel: GA/Meta/TikTok/X) the tags the hosted tag injects, so
browser tag and server conversion share ids and never drift. Public,
pk--keyed like /v1/event(.js), non-secret ids only, fail-safe to an empty
set so a page never breaks on its tag config.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Google Ads PROPER, not GA4: uploads real conversions against a configured
conversion action via the Ads API, OAuth2-refreshed, attributed by gclid
and/or enhanced-conversion hashed identifiers (email/phone). Only
conversion-class events with a Google match key are uploaded; traffic and
keyless events are skipped. Composite JSON secret carries the four OAuth2
values; developer-token + login-customer-id headers per the Ads API.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes the X adapter: the payload builder was already done, this adds
the OAuth 1.0a HMAC-SHA1 request signer and wires Send. The four OAuth1
parts ride as one composite JSON secret (consumer key/secret + access
token/secret), because the fan-out resolves a single primary credential
per destination — no fan-out change. Signature cross-checked in tests by
independently recomputing the HMAC over the base string.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the server-side Pinterest conversion sink on the shared Destination
interface — closes the Pinterest gap in the fan-out. Bearer-token auth,
SHA-256 advanced matching (em/ph/external_id), the epik click id lifted
and deduped against the browser tag via the shared event_id, ecommerce
contents, value-as-string per Pinterest's v5 contract. Custom events
collapse to the "custom" enum value the API mandates.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TEST_TAGS said `sqlite_fts5`. The release image builds AND tests with
`libsqlite3 sqlite_fts5 sqlite_math_functions` (Dockerfile:213), and the
Dockerfile's own comment says sqlite_math_functions "is not optional under cgo"
because hanzoai/base's search layer calls the math functions.
So a local `make test` linked a SQLite the shipped one is not, and apps/base,
apps/code and apps/commerce failed on `no such function: acos`. That was read as
"this box's SQLite is old" — including by me, all of today, in every regression
count I reported. It never was. The tag list here and the tag list in the
Dockerfile are two statements of one fact, and they disagreed.
Measured, same box, same commit:
default cgo FAIL x3
cgo + sqlite_math_functions + fts5 ok, ok, ok
CGO_ENABLED=0 (pure Go) ok, ok, ok
Both engines were fine. Only the tag list was wrong.
Full suite with the corrected tags: ONE failure, apps/meet/ui — a real defect
(the auth module ships twice, so the OIDC code is redeemed once per copy and the
second redemption destroys the session the first established). Its SPA source is
not in this repo; apps/meet/ui carries dist/ only, so the vite resolve.dedupe fix
belongs where the meet SPA is built. Left red on purpose: it is true.
The stale prose above the variable is corrected too — it claimed the image builds
with two tags while the image builds with three.
This package has already shipped handlers nothing could reach — Create, List,
Get and Delete existed as exported functions with no routes registered, and
nothing said so. The terminal adds two more doors, so the mounting is a fact with
a test rather than a line somebody remembered to write.
Both refusals happen before a pod or a store is addressed, which is what makes
them measurable anywhere: the ticket door answers the siblings' 403 without a
principal, and the socket door answers 401 for a missing, empty or forged ticket
— identically with a principal and without one, because a principal is not what
opens a terminal. It refuses BEFORE upgrading, since a socket that opens and then
closes tells a browser nothing.
The ticket is decided in memory and the wire is decided in a function, so both
are measured without a cluster. The pseudo-terminal is not: whether the apiserver
gives us a pty on that pod, and whether a shell comes up on it, is a question
only a cluster answers — and if the answer is no, the symptom is a socket that
opens, says nothing and closes.
So this drives the REAL argv through the REAL stream, guarded by SANDBOX_LIVE
like its siblings. It types `echo $((6*7))` and waits for 42, because a pty
echoes what is typed and a marker that appears in the command text would match
the echo rather than the output. It asks the shell, from inside, how wide it is,
and gets back the 100 columns the window sent. Then it types `exit` and requires
the stream to end — a terminal that outlives its shell holds a pod open until the
reaper takes it.
Measured against do-sfo3-hanzo-k8s, hanzo-sandboxes:
SHELL RUNS: echo $((6*7)) 42
WINDOW IS OURS: 100 columns, as sent
TERMINAL ENDED: <nil>
--- PASS: TestLiveTerminalIsARealShell (17.51s)
Every route on /v1/sandboxes was a request/response, so the one thing a sandbox
could not give anybody was a prompt. exec collects what a command produced; a
person at a keyboard needs the other shape of the same channel.
Two routes and one mechanism:
POST /v1/sandboxes/:id/terminal a ticket
GET /v1/sandboxes/:id/terminal/ws the terminal
The ticket exists because a browser's WebSocket constructor takes a URL and
nothing else. There is no header to put a bearer in, and the two usual answers
are both wrong: a bearer in the query string is a long-lived credential written
into every access log on the path, and trusting the session cookie makes the
socket a CSRF target, since no same-origin policy applies to a WebSocket. So the
credential is MINTED for the socket — crypto-random, bound to one org and one
sandbox, thirty seconds, spent on presentation and not on success. That is also
why the origin is not checked: the ticket IS the check, and a second gate beside
it would answer a question the first one already closed.
The wire, both directions, once. A text frame is stdin unless it is the one
control object {"resize":{"cols":N,"rows":M}}; a binary frame is always stdin.
Output comes back BINARY — a text frame must be valid UTF-8, a pty emits
arbitrary bytes cut at arbitrary offsets, and a frame carrying half a rune is one
the browser closes the connection over.
runtime.tty is a second shape of the exec channel, not a flag on exec: no
timeout that would insult whoever is typing, nothing buffered, one stream (a pty
merges stderr by construction and the apiserver refuses to open a second on a
TTY session). The shell is `bash -l` falling back to `sh -l`, so nothing on the
image is a precondition for getting a prompt — the hanzo CLI included.
The session ends once, from either end. The socket's read pump cancels the
stream when the far side goes, because EOF on stdin is a hint a pty may ignore;
the exec stream's end closes the socket with a reason instead of leaving it to
time out. The terminal is bounded by the sandbox's own lease, so it cannot
outlive what it is attached to even when the reaper is behind.
Tickets are held in memory and are replica-local — cloud-api runs one replica.
When that changes, this fails loudly with a 401 rather than quietly, which is
the property worth having in a gate.
The forge and github.com/hanzo-inc had diverged: six commits here that
github lacked (through commerce v1.50.23), four there that the forge
lacked (fleet verbs, sandbox leases) — and the LIVE image was built from
one of theirs. Shipping either tip alone would have regressed the other.
One lineage, so a release can only ever mean one thing again.
The fleet door served 98 tools and `sandboxes` was not among them, so @hanzo
could not lease a sandbox — while the run path underneath was built, deployed and
working. Real gVisor pods in ns hanzo-sandboxes, digest-pinned, hanzo-mcp
answering over stdio in-pod with 29 tools and a live fetch. All of it unnameable.
Cause: every /v1/sandboxes route is a RAW handler, and a raw route is invisible to
every projection zip derives from its typed registry — REST is the only one it
reaches (zip typed.go:6-9 says exactly this). The child answered tools/list
happily with an empty array, so `sandboxes` was absent from the tool list AND
absent from the outage list. Silent absence: the same shape as the stock node:22
that exited 0, and the callee that logged 200 after the caller had hung up.
The five typed ops already existed one file over. expose() registers them on
cloud.Plane() — a DIFFERENT zip.App on a DIFFERENT socket that the door never
asks (plane.go:29-33 states the rule; manifest/mcp.go:68-76 and webui/mcp.go:37-41
both record this exact failure happening to kms). Same handlers, same types, now
also on the server the door does ask. Nothing new is invented, so there is no
second implementation to drift.
Op names are verbs on the product noun, matching the surface that landed today:
lease_sandbox, run_in_sandbox, read_sandbox_file, write_sandbox_file,
end_sandbox. None trips refuse() — no bearer-secret noun, no mutating verb on an
authority object — so the gate is unchanged and the refused set stays 136.
This is the line between "@hanzo can run code" and "@hanzo can lease a computer".
apps/sandbox builds and its tests pass.
The embedded dist carried an unresolved merge (conflict markers in
index.html and .sync-stamp — the new invariants gate refused the tree,
correctly) and something worse underneath: vendor-BgRt8uZI.js was a
LINE-MERGED minified chunk importing BOTH generations' ui chunks — a
hashed filename over bytes no build ever produced. No deletion could
make that set coherent, so the whole dist is replaced by a clean build
from the stamped source (hanzoai/admin apps/tracker@8da44d7: tsc clean,
vitest 7/7, vite build) — six files, every relative import resolving
inside the set, verified. A hashed asset that does not match its hash
is worse than a conflict marker; only a rebuild tells the truth.
v1.50.21 adds a probe that drives the deposit watcher against a REAL chain,
which is the one thing 63 unit tests against fakes could not establish. Verified
against Base both ways: the correct USDC contract scans clean, and USDT's
contract labelled "usdc" is REFUSED quoting the on-chain symbol — which is also
the proof that decimals come off the contract rather than a constant.
v1.50.22 (another author) credits top-ups to the signed-identity payer rather
than a client-supplied subject.
v1.50.23 adds the Solana reader. SPL USDC is dollar-pegged, so it needs no price
oracle — which is why Solana came before BTC. The dedup key stopped being
EVM-shaped: chain:txHash:eventIndex, where eventIndex is the log index on EVM
and the token-BALANCE record index on Solana. It credits post − pre rather than
the instruction amount, because under a Token-2022 transfer fee those differ and
only the delta is what actually arrived.
Nothing here can take money: cryptoDepositsCanBeCredited is still false, and an
unconfigured watcher is disabled. GetCryptoOptions additionally intersects
"creditable" with "an address can be minted", so Solana stays out of the picker
until the custody fleet can mint an Ed25519 key.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
BYO images already worked — Spec.Image goes straight into the pod spec — and
nothing checked it. Our registry pull secret is attached to the `sandbox`
ServiceAccount and is FLEET-WIDE, so a caller naming
`oci.hanzo.ai/<someone-else>/private` had our credential fetch another tenant's
private image for them. Nothing in the request was forged; the field was simply
never read for what it implied.
The rule is the one the registry already uses. Our registry is org-namespaced as
<host>/<org>/<app>, so on OUR hosts the first path segment must BE the caller's
org — or `hanzoai`, which is the platform's own sandbox images and is the same
bytes imageFor would have chosen anyway. Anywhere else needs no credential of
ours, so it needs no permission from us: a public image is the caller's own
business, and the pod's securityContext contains it either way.
That makes BYO images a real feature rather than an accident. An org pushes to
its own namespace and names it on create.
RUNTIME IS PER SANDBOX NOW. It was one deployment-wide env var read at startup,
so the same task could not be run on two runtimes and compared without a
rollout, and a caller could not choose. Spec.RuntimeClass overrides it, against a
CLOSED set — an unknown runtimeClassName is a pod that never schedules, so a
typo would otherwise become a sandbox stuck Pending with nothing to read.
The row deliberately does NOT record which runtime a sandbox got: the pod is the
source of truth for what it is actually running, and a second copy could only go
stale.
The code interpreter tells the model to persist artifacts in /mnt/data. In a
deployed `exec` sandbox that directory was whatever the image shipped —
root:root 0755 — and the pod runs as runAsUser 1000, so every such write failed
with EACCES. The one directory the tool exists to fill was the one directory it
could not write.
Measured in a live sandbox rather than inferred:
$ kubectl -n hanzo-sandboxes exec m-643df8b317a6052579035359 -- sh -c '...'
id: uid=1000(sandbox) gid=1000(sandbox) groups=1000(sandbox)
drwxr-xr-x 1 root root 40 /mnt/data
mkdir: Permission denied
Only `dev` sandboxes ever got a mount at their workdir, because only they have a
project PVC. `exec` fell through the `if m.Volume != ""` and got nothing. It
survived because nothing checks: a run that prints its answer looks perfectly
successful, and only a run that SAVES something notices — as a traceback the
model apologises for rather than as an error anyone sees.
emptyDir, not a PVC: a code-interpreter session is exactly as long-lived as its
pod, which is what emptyDir already means. It is also what makes the mount
writable — the kubelet chowns an emptyDir to the pod's fsGroup, and the
securityContext already sets fsGroup 1000 — so the fix is the mount, not a chown
in the image. Bounded by SANDBOX_WORKDIR_SIZE (2Gi) because a container's
ephemeral-storage limit does not cover emptyDir usage on every runtime.
The re-embed at 86de6b94e copied the new bundle beside the old one
instead of over it, so dist carried two generations of the ui chunk —
and TestTheAuthModuleIsBundledOnce counted the auth module once per
generation and went red, holding every release since. The three files
the current index.html cannot reach (the old entry, its ui chunk, its
css — they reference only each other) are removed; the live five are
untouched.
The runtime already grew `tool`, an optional repo and `desktop`; this is the
cloud half of that contract.
`tool` (dev|claude|codex|python|node) and `desktop` are carried, not
interpreted. Cloud ships a NAME; the runtime owns the name→argv table, so
adding a tool is one edit over there and none here. `desktop` selects an image
variant — a tag — and nothing on this side branches on it.
THE REPO IS OPTIONAL, AND THE CREDENTIAL LIVES INSIDE IT. Every git field is
`omitempty` and `credential` is a POINTER: a value type always marshals, so a
repo-less run would still ship a blank `credential` object and the runtime could
not tell an absent grant from an empty one. A caller that supplies a credential
with no repo is REFUSED rather than quietly trimmed — silently dropping a secret
hides the bug that minted it, and the runtime says the same thing at its own
boundary, so both ends agree.
`Secret` is now true only when the body actually carries a credential. It was
unconditional, which meant the cleartext guard — a rule written to protect a git
token — refused every research and bare-exec run that has no token to protect.
SANDBOX_URL names where a run goes. A sandbox is not the bot: coding, deep
research and bare exec all want a computer to run something in, and none of them
wants the service that runs Slack channels. BOT_GATEWAY_URL stays correct for
bot traffic and is accepted here for one release, then deleted.
The transport learns none of this. `Call.Base` is a DESTINATION, which is a
transport concern; the caller resolves the address because resolving it inside
transport.go would mean that file learning what a run is — the exact line its
header draws. requireSecure now checks the base the call will ACTUALLY use;
checking the default while the bytes went elsewhere was a guard on the wrong hop.
The console overview's 'usage totals: code 184' card was ClickHouse
refusing WHERE timestamp beside any(timestamp) AS timestamp in the
dedup subquery. v1.832.36 moves the predicate to an inner plain scan.
The `projects` tool offered 37 operations spelled
`get_v1_projects_by_slug_deployments`, with no prose beside any of them.
Those are ROUTES — zip derives an operation id from method and path when
nobody declares one — so a model's first job was to reverse-engineer a URL
back into an intention, its second was to call describe to find out what the
intention took, and only its third was to act. Three round trips to use a
menu written in URL, and the assistant reads as stupid for the first two.
phrase() reads the route back as a verb on an object. The SHAPE decides it,
not a dictionary: a mutating method with a singular segment sitting after a
parameter is the author's own verb (deploy_project), a path that ends in a
parameter acts on one row (get_project, delete_project_domain), a GET of a
plural tail is a list and keeps it plural (list_project_deployments). PUT
and PATCH get different words — `set` replaces, `update` merges — because
they are different operations at one address. A DECLARED id is already a
verb on an object and is left alone, so o11y and iam are untouched.
get_v1_projects_by_slug_deployments list_project_deployments
post_v1_projects_by_slug_domains_by_host_verify verify_project_domain
post_v1_projects_by_slug_deploy deploy_project
delete_v1_projects_by_slug delete_project
THE REFUSED SET IS UNCHANGED, measured rather than reasoned. Naming happens
AFTER the gate and only there: refuse() judges the child's own id exactly as
it always did, fleet/surface.go is byte-identical, and the refused list is
the same element for element over the fleet's 2,440 declared operations —
211 — checked against the same rule compiled from HEAD and from this tree.
rank() reads the route for the same reason, so the order is unchanged too.
The mapping back is exact because offer() REFUSES to guess: a phrase two
operations would share, or one that is already some operation's own id, is
not used and both keep their ids. 182 of 2,229 do, nearly all of them the
fleet's own duplicates — one handler at /tasks and /v1/tasks, post_v1_agent
beside post_v1_agents in another subsystem. An operation's id also still
routes, and that is forced rather than kind: describe hands back the owning
subsystem's descriptor bytes verbatim, and those carry the owner's name.
And the enum now carries a line of the operation's own documentation beside
the product operations, which is the half that removes the describe round
trip for the calls an agent actually makes. It is rationed to productStems
because the bytes say so, over the 2,229 offered ops:
routes, as they shipped 63,370
verb phrases 50,151 a phrase is SHORTER than a route
+ a summary on the 143 ranked 64,081 +711 against the routes <- shipped
+ a summary on ALL of them 255,488 four times over
So the whole change costs 711 bytes: the naming pays for the prose. The
console tail stays bare and named well enough to recognise, with describe
one call away, because the point of grouping was 977 KB down to 71 and four
times the enum puts most of it back.
Proved on real children over real sockets under the real door:
TestACallByThePUBLISHEDNameReachesTheSameHandler stands up a child whose ids
zip DERIVES from its routes, then shows deploy_project and
post_v1_projects_by_slug_deploy reaching one handler with one reply, and
TestAColdDoorDispatchesAPublishedNameOnTheFirstCall covers the path that
caught a real bug in the first draft — a door that has answered no
tools/list holds neither table, so resolving before discovering answered
"unknown tool" for an operation it publishes. find() does both in one
lookup.
apps/agents/door.go stops requiring a descriptor's name to equal the op it
asked for. The door publishes the phrase and the descriptor carries the id,
so that check would have rejected 1,730 of 2,229 operations for being
correctly named; the seam's real guarantee — the model is offered exactly
what it will call — is kept by naming the offer with what came out of the
enum.
The door's tools shipped this morning as `hanzo_<app>` plus `hanzo_describe`.
The MCP server IS Hanzo — a client reaches these names through it and through
nothing else — so the prefix disambiguated nothing from nothing, and charged a
token for it on every one of 118 tools on every turn. Gone, in one change, with
no aliases: `hanzo_git` -> `git`, `hanzo_websearch` -> `websearch`,
`hanzo_describe` -> `describe`. Clients re-discover on connect, which is why
this is cheap now and would not have been in a week.
The prefix was also doing load-bearing work nobody had written down: composed()
read it to tell one of the door's own tools from an operation a child declared.
A convention holding up dispatch is a convention someone will break thinking it
is cosmetic — so that test is now an exact membership check against the app set
the door was mounted over (Door.composed). A child's operation id is
`<method>_<path>` or a declared PascalCase verb; an app name is neither, so
nothing a subsystem serves can be mistaken for an envelope.
Dropping the prefix puts the door's own tools in one namespace with the app
names, which creates exactly one new way to be wrong: an app called `describe`
would publish a second tool under that name and be silently unreachable.
TestNoSubsystemIsCalledDescribe reads the manifest and fails the build if one
ever appears — a fact about the app list, checked where it is decidable.
THE REFUSED SET IS UNCHANGED, and this was measured rather than reasoned:
refuse() reads CHILD operation ids and never saw the door's own names. Over the
fleet's own corpus (plugin/*/openapi.json, 117 subsystems, 2,440 ops) the
refused and kept lists are byte-for-byte identical before and after —
211 refused, 2,229 kept.
Measured through the real door, real children, real sockets:
the head of the surface describe ai agent agents code lsp product
provisioning git deploy exec projects …
118 tools, 106,282 bytes, 2,229 operations offered, 211 withheld
e2e/mcp-door.sh asked whether a tool name started with `hanzo_` to decide it was
a subsystem and not a flat operation. It now asks whether the tool CARRIES an
`op` enum, which is the thing itself rather than a convention about it.
/v1/ask is described as agentic research across all of our data. It had one
contributor, and that one could not work: the advisor ships as its own plugin
binary (plugin/ask/main.go mounts ask.Mount and nothing else), so the read it
made for its figures — an httptest replay against its OWN router — could only
ever reach /v1/ask. books runs in a different process. The replay 404'd, the
gather failed, and every money question in production was answered by the
"I can answer questions about your finances" fallback with an empty figures
array, while books sat healthy one socket away. Measured on api.hanzo.ai:
"what is my MRR and how long is my runway?" -> {"figures":[],"domain":""}.
So the seam was not merely empty, it was unreachable, and filling it with more
in-process replays would have produced more contributors that classify a
question and then answer nothing. Every domain is now a PLANE call, which is
the one thing that crosses a process boundary.
One shape for all of them: plane.FiguresOut, a domain's headline figures for
the caller's org, already formatted by the domain that owns the number. One op
each — books_figures, projects_figures, git_figures — because the ANSWER is
each app's own (only books knows what a dollar of revenue is) while the shape
is shared, so the advisor needs no per-domain branch and a new domain is a new
op and one line, never a router edit. apps/ask keeps ONE contributor type; the
three domains are values of it.
TENANCY. plane.FiguresIn is an empty struct. There is no org argument to
validate because there is no org argument: zip forwards the gateway's own
assertion off the in-flight request and each op reads cloud.Who(ctx).Org,
refusing anonymous rather than defaulting it. The advisor hands the peer
cloud.As(c, "") and not c.Context() — an UNTYPED handler is not given the
request on its context, so the peers would have seen "nobody is calling" and
correctly refused every question ever asked.
Verified in the production topology: four plugin binaries, four processes, four
sockets, real IAM-validated tokens. "how many repositories do I have and what
changed recently?" answers domain=git, sources=[git/figures], Repositories 3,
Code stored 412 B, most recently advisor-core — byte-identical to what the git
process reports for itself at /v1/git/usage. The same token switched to an org
with no repos answers 0, and a non-admin principal's forged X-Org-Id is ignored
entirely.
Skipped: o11y (its read needs ClickHouse and an allowlisted product slug — no
plane read op exists and I could not honestly verify one), knowledge (no plane
ops, needs a Qdrant this repo does not deploy) and index (org-scoped and ready,
but a query needs a UID and nothing indexes an org's code under a nameable one).
Two working domains beat five that classify and return nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzo-inc main was RED and had silently lost a mechanism. `go test ./apps/sandbox/`
failed two cases here and passed on the forge, which is the tell:
imageFor("dev") = "sandbox:dev-2026.6.7", want "sandbox:2026.6.7-dev"
a digest wins over any tag: got "sandbox:dev-2026.6.7", want "sandbox@sha256:2baf7ede"
b028ab0f rewrote imageFor from a base that predated dbcb9040, so it did not change
one line — it reinstated an older file over newer work and took two things out
with it:
the ORDER back to `<class>-<version>`, so it asks for `dev-2026.6.7` while
the registry holds `2026.6.7-dev`. Every version-pinned pull 404s.
digestFor DELETED. `SANDBOX_IMAGE_DIGEST_<CLASS>` stops being read at all.
The second one is the serious half and it is silent in the worst way. The live
deployment sets all three digests, and universe records why: they are the only
name that cannot be rewritten under a running fleet. A binary without digestFor
IGNORES them and falls through to SANDBOX_IMAGE_TAG=latest — a moving tag,
which is the exact class of name that a `crane copy` put stock node:22 onto in
the first place. The deployment would look correctly pinned and would not be.
Every signal stays green; the only tell is a `whoami`.
Its commit message says "apps/sandbox builds and its tests pass". On the tree it
was written against, that is true. On this one it is not, and that gap IS the
bug — the same shape as the defect it was fixing.
This restores apps/sandbox/runtime.go from forge/main, which CD reads, and
nothing else. One file, by path. The two mains now agree on this function.
That also settles the fallback: forge has `<class>-unset`, this had `<class>-latest`.
I had independently written `-latest` too and it is the weaker answer. `-latest`
is still a MOVING tag, so it fails OPEN into whatever was pushed last; `-unset`
is published by nothing and fails CLOSED. The argument for `-latest` rested on
the bare tags being unrepairable — but they were repaired: a pod requesting
`:dev-latest` today resolves to sha256:4e5deb07, the same digest universe pins,
and inside it whoami is `sandbox`, /etc/sandbox-version is 1.0.0 and the OS is
Ubuntu 26.04. Not node:22. For a component that runs untrusted code, a fallback
reached only by misconfiguration should refuse, not improvise.
TestImageForNeverComposesTheBareClassTag passes under BOTH answers, which is why
it was written as an invariant rather than a table row.
Merging inc2 onto forge surfaced four failures. All four were the gates working;
none was merge damage.
principal.ProjectFrom / WithProject — apps/crawl's scopeOf reached for
cloud.Request, the pinned escape hatch, to read two facts: org and project.
OrgFrom already existed; Project had no ctx counterpart, so the hatch was the
only way across the typed-op seam. It has one now, and crawl reads the context
like everything else.
WithProject parks UNCONDITIONALLY, which is the one asymmetry with WithOrg and
is deliberate: an org is an AUTHORITY, so an unvalidated request must park
nothing rather than an empty tenant a query would honour. A project is a
NARROWING — every consumer ANDs it with an org that already gates — so gating it
here would state the authority twice and let the two statements disagree.
agents/builtin_test — the test asserted the default assistant carries no tools;
the code now offers it the whole door. The CODE is right: an empty offer reads
to a model as "there is nothing here", and the assistant was reporting it could
not reach the cloud while the door served 88 tools one socket away. The test
sentence was the stale half, so it now says what is true.
floor tracker 10 -> 9 — GET and DELETE on /v1/tracker/projects/{key}/issues/{num}
stopped registering, and that is deliberate: tracker moved to reading the forge
(dd6cd6de, 7d42283f), so tracker.go registers the list route and no longer the
two by-number ones. A milestones op arrived, hence -1 net. Lowered here, next to
the reason, which is what the ratchet asks for when a deletion is real.
openapi.yaml regenerated: 1772 paths / 2442 operations, measured by the
generator rather than merged as text — the third time today git resolved the
generated document to one side while the routes merged to the union.
go build 0. Full suite 4 red: apps/base, apps/code, apps/commerce (the
environmental three — this box's SQLite has no acos/fts5) plus apps/o11y, which
passes standalone and is ok on main; it flakes under the parallel sweep.
Every sandbox in production ran docker.io/library/node:22. imageFor's default
returned the BARE class tag — <image>:dev / :exec / :desktop — and all three are
sha256:0557ac14, byte-identical to upstream node:22.
Forensics: a Job `sandbox-image-seed` ran `crane copy docker.io/library/node:22`
onto them at 20:08 UTC 2026-08-06, from a locally built +dirty crane rather than
CI's pinned 0.20.2. A second burst at 20:41 also clobbered exec-latest,
dev-latest and desktop-latest — those three were republished by the sha-8af2c8a
build at 21:39-21:42 and are correct again. The bare three are not, and never
will be: NO LANE EMITS A BARE CLASS TAG. hanzoai/ci and bot both publish
sha-<short>-amd64-<class>, <class>-latest and <version>-<class>. A tag nothing
can create is a tag nothing can repair.
The failure was SILENT, which is the expensive part: node:22 starts, reads EOF,
exits 0. A run that did nothing and reported success — the same shape as the
Slack bug that cost a day, where the callee logged 200 while the caller had
already hung up.
Default is now <class>-latest, a real published tag. It is not the strongest
form — SANDBOX_IMAGE_TAG_<CLASS> pins a version and a deployment that cares
should set it — but an unpinned tag that is CORRECT beats a pinned-looking one
that is a stock node image. The docstring's preference for a pinned version was
never satisfiable by a name carrying no version at all.
apps/sandbox builds and its tests pass, including the one that referenced the
bare tag — so nothing was pinning that behaviour.
Carries the sandbox image and network work onto this remote: the unset-tag
refusal (Red's -unset), the bare-tag invariant, and everything else already on
the forge. The two mains had diverged by one commit each way; the forge is the
one CD reads.
I arrived at this line independently and with a WEAKER answer: `<class>-latest`,
because that is a name hanzoai/ci actually publishes. Red got here first with
`-unset` and Red is right. `-latest` is still a MOVING tag, and a moving tag
someone can overwrite out of band is precisely what put stock node:22 under all
six of these names in the first place; resolving to it fails OPEN, into an image
that starts, reads EOF and exits 0 as root. `-unset` fails closed. Their line
stands unchanged.
What is added is the same rule as an invariant over every class and every tag
state, rather than the one `dev`-with-no-tag row that proves the fix. The bare
form is not one wrong answer among many — it is the ONE spelling in this
function that resolves to something in the registry which is not ours, and three
different roads reach it: an empty SANDBOX_IMAGE_TAG, an empty
SANDBOX_IMAGE_TAG_<CLASS>, or a later edit that reorders the concatenation and
drops a separator. A table covers the roads someone thought of.
bot's imageFor now asserts the identical invariant on its own side. Two
consumers, one rule, written twice because a Go service and a TypeScript one
share no code — only a registry.
The bare `exec`, `dev` and `desktop` tags all exist in the registry and all three
resolve to ONE digest: stock node:22. No toolchain, no agent, and User is unset,
so it runs as root. imageFor fell back to exactly that spelling whenever the tag
was empty — so one unset env var silently swapped a hardened sandbox for a root
shell, and every signal stayed green. The pod runs, the API answers, and the only
tell is `whoami`.
Nothing publishes `-unset`, so the pull now fails with a name that explains
itself. A loud stop beats a silent downgrade; the whole point of this subsystem
is that the thing running someone else's code is not root.
Found by RED, which noticed that fixing `{class}-latest` and leaving `{class}`
bare left the gun loaded rather than unloaded.
The verdict surface took a principal or nothing, so an SDK holding a project
key -- the credential the wire was designed around -- had no way in and read
'X-Org-Id required' for a header it cannot mint. A key now resolves its org
through the ONE IAM seam the event door already uses, fails CLOSED on an
unresolvable key rather than falling back to the host, and is refused at every
write: reading a verdict and changing what everyone reads are different powers.
Authorship keys on whether a validated identity named the tenant, not on the
actor email, which a principal need not carry.
Brings stt: an OpenAI-compatible /v1/audio/transcriptions provider, so
cloud can front the in-cluster speech service (whisper on CPU) the same
way it fronts every other model plane.
The Slack front door ran on a literal `enso` behind a BRIDGE_AGENT_MODEL knob,
justified in its own doc as "the auto-routing SKU that selects per query in the
gateway's own catalog". That is not what enso is.
enso is one fixed route entry: deepseek-v4-pro, 1M ctx, reasoning: medium
(zen-svc catalog-enso.yaml, and the live enso-catalog ConfigMap it is served
from). One rung, no ladder, no per-query selection, no escalation. Three code
comments in three repos and the Slack App Home menu all asserted otherwise, so
the front door's tier had never been chosen — it had been inherited from a
property nothing in the system has.
So I measured it instead, paired and interleaved against the live enso service so
upstream load drift cannot flatter either side. n=42 full turns, the real
assistant instructions, the real describe-then-call protocol:
enso p50 3489 ms p90 8000 ms 1.71 model round-trips/turn
enso-flash p50 3905 ms p90 17269 ms 1.71 model round-trips/turn
paired diff 1512 ms median in enso's favour, t=-3.29 — significant.
The tier is RIGHT and the stated reason was invented. Round count is identical;
the difference is generation rate. On a tool-shaped turn enso-flash emits at
11.7 tok/s against enso's 19.1 for the same ~75-token answer, so "flash" is
quicker only when the answer is short enough for terseness to beat rate — a
greeting, not a question about the fleet. "what did we deploy today?" ran
5,386 ms on enso and 15,122 ms on enso-flash.
I very nearly shipped the opposite. A single-call benchmark says flash wins
(p50 1648 vs 3250 ms, t=2.15) and it is the wrong benchmark: it measures the one
turn shape — a greeting — where terseness dominates, and the Slack assistant is
a tool-driving agent. Two runs of the full turn disagreed with each other before
n was large enough to separate the effect from the upstream's variance, which is
the real reason a turn is sometimes slow: p90 is 2-3x p50 for both tiers.
What changes:
- the tier and its evidence move to cloud.ChatModel, in the file that owns
model policy, beside DefaultModel and FallbackModel. It is deliberately NOT
DefaultModel: that one serves one-shot text, this one drives tools, and the
tiers do not rank the same on those jobs.
- BRIDGE_AGENT_MODEL is gone. No deployment ever set it, and a second place can
only disagree with the first.
- App Home stops selling `enso` as "Picks the right model for each message" and
stops calling enso-flash "Fastest, for quick questions" — it is not fastest on
the questions people ask. It now names the tiers and marks the default, read
from cloud.ChatModel rather than from the menu's own first row.
- builtin_test pinned "the default carries no tools" against code that had
already been given the whole door, so the package's tests were red.
The App Home pin still wins over the default, and is still limited to the three
SKUs the menu offers.
Carries one line of an in-flight fleet.Describe rename in onbehalf.go that landed
in the shared worktree mid-edit; reverting it would have dropped that work.
(cherry picked from commit fce13e455d7c15157d4c6f62b7184d34c917db13)
The meet client mints its LiveKit join token on a bearer-only host, where
cloud's session cookie never arrives, so POST /v1/meet/getToken has to carry
the Authorization header the session read already carries. Without it every
room join answers 401 and the lobby lists rooms nobody can enter.
The tracker client picks up the same-origin predicate, the token-resolved
org header, the hostname-derived brand registry and the tenant-keyed cache.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`COPY . .` was shipping 1087 MB. /bin/ was already excluded, but nothing that
mattered lived there:
hanzo 586 MB untracked
o11y 126 MB untracked
host 26 MB untracked
.claude ~92 MB gitignored — agent worktrees, i.e. second checkouts of this
same repository, cache-keyed into the layer so every agent
run invalidated it
All build litter; none of it tracked, so nothing in the build can want it.
Measured after: 152 MB.
NOT excluded, deliberately: `sandboxes` (50 MB), which IS tracked. COPY . . may
legitimately carry it, and a build output somebody committed is a question for
that commit — not something to drop from the image behind their back while
chasing a number.
The .claude/ line is the same defect the zipdoc gate hit, in a second place: an
agent worktree is a whole other checkout, and any tool that walks the tree
without excluding it reads our own copy as our own source.
One cloud binary answers every brand's API host, and its validator trusts EVERY
white-label issuer — trustedIssuers unions BrandIssuers, deliberately, because
api.hanzo.ai and api.lux.network are the same process. So a lux.id- or
zoo.ngo-issued token is genuinely valid on the hanzo deployment.
The forge, though, is resolved from the DEPLOYMENT's own domain (brand.Sibling
→ git.hanzo.ai), never from the principal's. So a token vouched by another
brand's IAM arrived with an attested org and an attested username, and both of
this surface's controls then did exactly what they were built to do: the org
scoped the query, and the username was Sudo'd against git.hanzo.ai. But "alice"
on this forge is a DIFFERENT HUMAN from lux.id's alice, and the forge answered
with that person's private issues.
Two individually-sound controls composed into a cross-brand private-repo read,
because neither of them asked WHO VOUCHED. The vouching brand must be this
deployment's own, so that is now checked — once, in the one resolver every read,
every write and the milestone rollup passes through, rather than six times on the
routes.
An absent vouching brand still passes: that is an hk-/sk- key minted by this
deployment's own IAM, which is by construction this brand, and principal.Brand
publishes "no brand" as "no second fact to compare" rather than as a brand.
Normalised on both sides, because a case difference must not decide a tenancy
question. Same pair, read the same way, as apps/tenant.
Tests: a lux-vouched principal is refused on all six routes and nothing reaches
the forge; the deployment's own brand passes, in any case; an unbranded
own-IAM principal passes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The assistant was asked what the weather was and had no path to the internet.
It was right: the fleet's MCP door carried 88 grouped tools and none of them
was websearch or crawl. Not because the capability was missing — apps/websearch
is a working keyless meta-search and apps/crawl a working fetch-and-extract,
both in-process, both serving over HTTP the whole time — but because ONLY TYPED
OPS PROJECT. A raw handler appends nothing to zip's op registry, and that
registry is the single value every projection reads: the route, the OpenAPI
operation, the SDK method, the CLI command and the MCP tool. A subsystem of raw
routes serves perfectly and is invisible to the agent.
websearch gains its NATIVE door. Its two existing routes are foreign-protocol
adapters — LibreChat's frozen searxng and firecrawl contracts — and neither can
be a typed op: one is registered with All and zip has no typed All, the other
answers 200 to a malformed body on purpose. Those are facts about the ADAPTERS,
and the package doc had read them as facts about web search itself. POST
/v1/websearch runs the SAME metaSearch over the SAME engines and answers the
SAME envelope, at an address the registry can hold.
crawl's POST /v1/crawl becomes a typed op with NOTHING moved on the wire. Its
two documented blockers were real and both were carried rather than dropped:
the 400 with `{"success":false,"error":"missing url"}` is stated by the ANSWER
through zip's StatusCoder, so the document publishes 400 with that schema; and
the 1 MiB bound and body-tolerance are facts about BYTES, which a typed op
never sees, so they are asked in middleware where the bytes still are.
TestCrawlWireSurvivedTyping is the measurement — same three inputs, same three
answers, byte for byte.
Both gates moved into the HANDLER, because a tools/call reaches a typed op with
no route and therefore no middleware: a gate that lived only in middleware would
be no gate at all for the two projections this exists to create.
fleet/reachable_test.go is the proof, and it stubs nothing: each subsystem
composed the way cloud.Serve composes a plugin child, on its own unix socket,
under the real composed door — so refuse() runs for real and an operation that
passes is one an agent can reach. post_v1_websearch, post_v1_crawl and
post_v1_exec all project. Their names are in surface_internal_test.go's
survivors table, which is the one place the rule is asserted; no second gate
site was added.
The work items on /v1/tracker were a SQLite table beside a github.com feeder,
while the estate files, labels and closes its issues on git.hanzo.ai. Two stores
under one prefix are two answers to what the state of a piece of work is, and
they disagreed the first time anyone touched the forge directly, which is every
day. The forge is now the store: every read is a read OF it, every write a write
TO it, and nothing here caches or mirrors a row.
A board is a repository, a column is a LABEL, and a card is an issue. Reading
the column off a label is what makes the board and the forge web UI the same
object seen twice — relabel in either and the card moves in both. So the
repository lifecycle is NOT on this surface: creating, renaming and deleting a
board are forge operations under forge permissions, and a second door onto them
here would be a weaker guard on the same object. Those three answer 405 naming
the forge, which is a different fact from 404.
Milestones are repo-scoped upstream and there is no org-level list, so the org
view is a server-side fan-out over the repositories the caller can see —
bounded, and failing whole rather than returning a partial rollup that reads as
complete.
TWO INDEPENDENT CONTROLS, because neither is trusted to be sufficient and this
forge really does host private orgs. The org comes from the validated principal
(principal.OrgFrom) and never from a path, query or body. Then every call is
made with Forgejo Sudo as the caller's own IAM username, which DROPS PRIVILEGE
to that user — measured against the live forge: the machine token reads
hanzo-private/patents (200), the same token sudoed as a non-member gets 404,
byte-identical to anonymous. So a bug in the first control cannot leak a private
repository on its own, and a write is attributed to the HUMAN rather than to a
shared bot.
One credential, held in KMS at orgs/hanzo/deploy/FORGE_TRACKER_TOKEN@prod —
never an env file, never a browser-side PAT. A separate secret from the universe
pin token on purpose: one credential per capability, so a compromise of the
tracker cannot deploy. Resolved lazily with a TTL so rotation is live without a
restart, invalidated when the forge rejects it, and fail-closed at every step —
an anonymous client would quietly serve public repos and read as "your board is
empty" rather than "this deployment is misconfigured". The forge host is
brand.Sibling of the deployment's own API host, so a white-labelled deployment
cannot read another brand's forge.
Also closes two tenancy defects found beside this work:
plane.AgentPRIn carried an Org the agent-PR seam read off the wire and passed
straight into the per-tenant store selector, so a caller on the plane could file
a work item onto ANOTHER tenant's board by naming it. Its sibling on the same
socket, IssueIn, has never had one. The field is gone, the handler reads
cloud.Who(ctx), the caller carries the org in the ENVELOPE (which is re-checked
by the same OrgOf rule as the HTTP boundary), and a reflection test now fails if
an org-shaped field returns to either input.
The audit trail's Home field means "a platform SuperAdmin acted inside another
tenant". Its predicate was home != effective, which WAS impersonation back when
a SuperAdmin was the only principal who could act outside their home org. Since
membership-based org switching, any ordinary member of two orgs trips it the
moment they work in their second one — telling auditors that routine work was an
admin impersonation, and burying the real events in volume. The predicate is now
the fact it always meant: home is the reserved admin org (authz.AdminOrg, the
issuer's constant).
Tests: 22 on the forge client (fail-closed with no actor, credential never in an
error, bounded fan-out, pagination that terminates against an endless forge, a
compile-time refusal of tenancy in the filter), 12 on the surface (cross-tenant
read, no-principal refusal, CSRF-refused writes never reaching the forge,
attribution of a move). The retired SQLite HTTP surface's tests go with it; the
two security properties they held — the CSRF gate and the per-IAM-project store
isolation, which still backs the plane doors — are re-pinned against what
survives. forge/live_test.go exercises the real forge, skipped unless
FORGE_LIVE_TOKEN is set, because a stub can only confirm we built what we
believed and not that what we believed is true.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TWO defects, both silent, both found by trying to start a real sandbox.
THE TAG ORDER WAS BACKWARDS. imageFor composed <class>-<version> and asked for
`dev-2026.6.7`; the registry holds `2026.6.7-dev`. So the default path 404'd on
an image sitting right there, and the only reason anything ever ran was a test
that overrode the image entirely. The publisher wins this argument: hanzoai/ci
appends a per-image tag-suffix to the version, so the whole fleet is
<version>-<suffix> and a consumer spelling it the other way is simply wrong.
A VERSION TAG IS NOT A PIN HERE. The sandbox image ships from hanzoai/bot under
BOT's package.json version, last bumped 2026-06-07. A rebuild today — commit
d1904514f4, "box: uv ships at the root" — republished `2026.6.7-dev` from source
two months newer. A tag that gets rewritten is not a pin, and one that LOOKS
pinned is worse than `latest`, which at least admits what it is.
So SANDBOX_IMAGE_DIGEST_<CLASS> is honoured ahead of any tag. `repo@sha256:…`
names bytes, and bytes do not change under a running fleet.
PER CLASS, not one variable. A single SANDBOX_IMAGE_DIGEST would have handed
every class the exec image while every log line still read correctly — the same
shape as the tag bug above. image_test.go pins that case specifically, along with
the order, because both of these are invisible until a pod fails to pull.
The SDK's leaf and the plane's root reach the same evaluator; the comment
said whose protocol it was rather than what the line does. The refusal said
'X-Org-Id required' to callers that can never send one — a project key does
not carry a tenant on this surface — so it now says what would satisfy it.
The image ends `USER sandbox` (uid 1000). A mounted volume — the emptyDir at the
exec class's workdir, or a project PVC — arrives owned root:root 0755, so uid
1000 takes EPERM on its first write. The pod is Running, the API answers, every
health signal is green, and the agent cannot save a file.
Reproduced before fixing, in both shapes:
uid=1000(node) gid=1000(node)
drwxr-xr-x 2 0 0 /mnt/data touch: Permission denied
drwxr-xr-x 3 0 0 /work touch: Permission denied
A Dockerfile `chown -R sandbox /work` cannot fix it — the mount happens after
the image layer and shadows it. fsGroup is the only mechanism that reaches a
volume: the kubelet chowns it to that GID and adds it as a supplemental group.
runAsUser/runAsGroup are stated rather than inherited from the image, because
the two have to agree and the one checkable from outside the image should be the
one that says so.
The image's own comments already asserted this existed — "the pod runs
runAsNonRoot with runAsUser 1000, so the uid is fixed by the securityContext."
Nothing fixed it. The live proof missed it because a stock node:22 runs as root,
so the one path exercised was the one that happened to survive. That is the
whole hazard of proving a mechanism with a substitute image.
The assistant reported it "can't query your projects directly" and named the
exact tools it would have used — hanzo_projects after hanzo_describe. It was
right, and the two halves of the system disagreed:
instructions "your tools are grouped hanzo_<subsystem>; call hanzo_describe"
door.go:93 if org == "" || len(want) == 0 { return nil }
want comes from callableTools(a), which iterates a.Tools — and the builtin
agent's Tools was empty, deliberately, from when the tool loop decided what to
offer. So the model was taught a protocol and handed nothing to practise it on,
while the door served 88 tools one socket away.
ToolsAll ("*") says "whatever the fleet serves", resolved per run. It has to be
STATED rather than implied: an agent that declares nothing still gets nothing,
because a user-defined agent's tool list is its authority. But the default
assistant cannot enumerate a surface discovered at runtime — the whole point of
grouping was that the set changes when a subsystem ships, so any list written
here would be stale by the next deploy.
The wildcard offers the door's tools AS GROUPED, not the ops flattened back out:
1,189 flat tools were 977 KB (~244k tokens) merely to list, and the same
operations grouped are 88 tools in 63 KB. Flattening here would hand back every
byte the grouping saved, and would contradict the prose the model is reading.
Also merges main (242 commits) into the security branch. plane.go's conflict was
two additive type blocks at one offset — both kept. The two generated files were
regenerated rather than hand-merged.
Red re-reviewed the chrome and found four client-side holes. All are fixed in
hanzoai/admin apps/tracker@577bb4c; this is the rebuilt bundle. No Go changed.
What is different in the bytes:
- ONE parser-based same-origin predicate gates the bearer. The old check
short-circuited on `startsWith('/')` without parsing, and seven root-relative
shapes — including ones carrying a raw TAB, LF or CR — fold to a foreign
authority under the parser fetch itself uses. Those shapes took the Bearer
and X-Org-Id off-origin. The base client attached the bearer with no check at
all.
- X-Org-Id is resolved against the token that carries it. The selection lives
in localStorage (permanent, shared by every user of a browser profile) and
the token in sessionStorage (dies with the tab); nothing reconciled them, so
the second person to use a machine sent the first person's org and got their
OWN rows back displayed under it. cloud already refused to honour it — this
stops it being sent, drawn, or kept.
- The IAM base comes from a brand registry (hanzo -> hanzo.id, lux -> lux.id,
zoo -> zoo.id) mirroring cloud's own, selected by brand rather than by an
arbitrary URL.
Audited on the shipped chunks: React only (0 hits for svelte/vue/solid/preact/
angular/htmx/alpine), 0 credential surface, 0 origin-derived IAM base.
Still not verified here: `go build` and `go test ./apps/tracker/...` — this box
has no Go toolchain. embed_test.go's assertions were checked by hand against the
new dist/ (base /tracker/, /v1/tracker baked in, every asset index.html names is
present, .sync-stamp still dot-prefixed).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
live_test.go proves a POD runs code. It calls r.exec, r.stop and r.purge
directly, so it never issues a request — delete every line of Routes() and it
still passes. What ships is not the runtime, it is the seven addresses in front
of it, and nothing was reading those.
So: the same proof, driven only through the routes.
CREATED over HTTP: id=m_53ce6be390c9eab1903222b7 class=exec status=running
WROTE FILE over HTTP: {"bytes":63,"path":"/mnt/data/answer.js"}
EDIT PERSISTS over HTTP: export const answer = 42; // written over .../fs
EDITED CODE RUNS over HTTP: ANSWER=42
FAILURE IS DATA over HTTP: exitCode=3
CROSS-ORG REFUSED over HTTP: GET as other-org -> 404
LIST CONTAINS IT over HTTP
DELETE /v1/sandboxes/m_53ce6be390c9eab1903222b7 -> 204
Three of those could not have been caught below the route. A non-zero exit is
DATA on POST /:id/exec — 200 with exitCode 3, not a 500 — and an agent that
cannot read a failing build is no use; that shape is a handler decision and
lives nowhere else. Org isolation is likewise a route fact: the unauthenticated
call is checked FIRST, so a later 201 cannot be explained away by an open door,
and a second org asking for the same id gets 404 rather than the row. The write
lands at /mnt/data because workdirFor answers the class, which is the same
confusion that made the git proof cd into a directory that was never there.
Guarded by SANDBOX_LIVE like its sibling, so it skips where there is no
cluster and runs where there is:
SANDBOX_LIVE=1 SANDBOX_NAMESPACE=hanzo-sandboxes go test ./apps/sandbox/ -run TestLiveHTTP -v
ledger.await gave a background debit 2 seconds to reach the ledger. Alone that
is ample; on the release gate the same process is running the whole fleet's
suite, and TestFeatures_IsPricedFromItsWindow failed there at "only 0 of 1
debits reached the ledger" while passing 20 of 20 runs in isolation.
That is a red money gate that proves nothing about the money — the metering it
exists to defend was never in question — and it is the shape of red that teaches
people to re-run a gate instead of reading it.
The question the helper asks is whether the debit arrives, which has a yes/no
answer that does not depend on the machine's load. Give it 30 seconds. The poll
is unchanged, so the happy path still returns in milliseconds and nothing gets
slower; only a genuine failure takes longer to declare.
Measured: apps/risk green over 6 consecutive runs under 12-way CPU contention,
which reproduced the failure before.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two concurrent transitions to `published` posted the item to its channels
twice. The fan-out was already leased; the erasure that defeated it was not.
Transition reads the document, then writes it back with the new status. That
write carries the WHOLE document — UpdateData replaces it, so external_ids
cannot be left out of the map without being deleted — and the snapshot it
writes is read before any fan-out. So: both callers read an empty skip-set, A
publishes and records its external_ids, B's stale snapshot writes that skip-set
back to empty, and B's fan-out (correctly leased, correctly re-reading) finds
nothing to skip and posts the item again. The lease inside Publish could not
see this, because the erasure happened outside it.
Widen the section to match the invariant it was defending: on the one edge that
distributes, Transition takes the item's publish lease across read, edge-check,
status write and fan-out. The loser then reads after the winner recorded — a
no-op edge (CanTransition is true for from==to), re-stamping the same status,
skipping every channel already on record. One post, both callers succeed.
Publish keeps its own acquisition for its own callers and delegates the fan-out
to publishHeld, which states the held lease as a precondition; the lease is not
reentrant, so Transition calls publishHeld directly. A contender that cannot win
the lease inside the wait window is refused with 409 rather than writing a
document it would corrupt.
Measured on apps/content: 2 of 30 runs failed before, 0 of 200 after.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The cleanup this file promises — "a leaked pod on a shared cluster is somebody
else's problem tomorrow" — was deferring purge, and purge deletes the VOLUME:
func (r *runtime) purge(ctx context.Context, m Sandbox) error {
if err := r.ready(); err != nil { return err }
if m.Volume == "" { return nil }
Neither live test declares a Volume, so purge returned at that second line every
time and deleted nothing. The pod was never anybody's job. Measured on
hanzo-k8s: four sandbox-live-* pods still Running, the oldest twelve minutes,
none carrying a deletionTimestamp — every pod both tests had ever started.
stop is the one that ends the pod, and it is separate from purge on purpose:
the volume holds the only copy of a checkout, so deleting it is opt-in. A test
that owns both wants both. Deferred in the order the product uses them.
Verified — pods gone after the run rather than left Running, and the proofs
still hold on the way out:
RUNS CODE: SANDBOX-RUNS-CODE v22.23.2
EDIT PERSISTS / EDITED CODE RUNS: ANSWER=42 / FAILURE IS DATA: exit=3
GIT PRESENT: git version 2.39.5
COMMIT MADE: 49523ec the agent committed this
ok github.com/hanzoai/cloud/apps/sandbox 79.993s
Pinning @v1.0.38 — which this file's own comments call for — made the forge
stop constructing a run entirely. No run row, nothing to inspect: the 'dead CI
is not red, it is absent' failure the same comments describe, reached from the
other direction. Absent is worse than red, so this goes back to @v1.
The lane remains broken and was before any of this: gate never gets a runner
while containment succeeds on the same run against an idle fleet, and no
release has minted since v1.801.490. Reverting restores a lane that at least
REPORTS its failure; it does not fix it, and the comment now says so instead of
implying the pin was the answer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestLiveSandboxDoesGit has been RED since exec's workdir split off from dev's.
The script opened with a bare `cd /work` — the `workdir` constant — against a
sandbox declared `Class: "exec"`, and workdirFor("exec") is /mnt/data. Kubernetes
creates a pod's workingDir and nothing else, so /work was never there:
live_test.go:163: git flow exit=2 stderr="sh: 2: cd: can't cd to /work\n"
So the one test that proves an agent can COMMIT was failing for a directory, and
"COMMIT MADE" has not actually been observed since.
TestLiveSandboxRunsRealCode hid the same mistake instead of failing on it: it
writes with `mkdir -p /work/src`, which CREATES the directory it then proves the
edit survives in. The edit was real and the persistence was real, but both were
happening in a directory the product does not read — confine() resolves every
caller path under workdirFor(class), so nothing served through /v1/sandboxes
would ever have seen that file. A proof that passes in the wrong directory is
weaker than a proof that fails.
Both now ask workdirFor(m.Class), which is the same question runtime.go asks when
it sets workingDir and confine() asks when it resolves a path — one source for
the path instead of a constant that is right for one class and wrong for the
other. Against hanzo-k8s, node:22, ns hanzo-sandboxes:
RUNS CODE: SANDBOX-RUNS-CODE v22.23.2
EDIT PERSISTS: export const answer = 42; // edited by the agent
EDITED CODE RUNS: ANSWER=42
FAILURE IS DATA: exit=3 stderr=to-stderr
GIT PRESENT: git version 2.39.5
COMMIT MADE: a9423dc the agent committed this
FORGE REACHABLE: exit=0
ok github.com/hanzoai/cloud/apps/sandbox 64.902s
The default was registry.hanzo.ai/hanzoai/sandbox. That host is a DEPRECATED
ALIAS — one Traefik file router serves both names from svc/registry:5000, so it
resolves, which is exactly what makes a stale name survive: nothing breaks, it
just spreads. oci.hanzo.ai is the canonical one.
Measured while confirming this: a probe pod in hanzo-sandboxes sits in
ImagePullBackOff on registry.hanzo.ai/hanzoai/sandbox:exec. The alias is not the
reason — the image does not exist under either name yet — but the next person to
read that error should be sent to the right host.
Two comments carried the old name too, including the one explaining why a
sandbox uses its own ServiceAccount rather than `default` (DOKS re-attaches its
own registry pull secrets to every namespace's default account, so a sandbox
inherited a credential for someone else's registry and asked ours anonymously —
a 401 that reads like a bad password and was no password at all).
The mirror moved while the ci pin was being fixed. Merged, not forced: the
forge is canonical but that does not make GitHub's commits disposable.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every cloud release since v1.801.490 has died in `gate`, and the rule that
prevents it is written in this file directly above the line that broke it:
pin the immutable patch tag, not the rolling `v1` alias.
`gate` is the ONLY job here that is a reusable-workflow call rather than a
literal `runs-on`, and it is the only one that never gets a runner —
runner_id 0, zero steps, ~32 minutes, then failure — while `containment` on
the same run succeeds against an idle 10/10 fleet. Runs 925 and 926 (at
7c50638e) and 928 (the forge merge) all died identically, so this predates
that merge; it is not what the merge introduced.
The alias is not just stale (v1 -> f098b39e vs v1.0.38 -> dfac7dc4) — it is
unpinnable by design: sync-from-github refuses to move a tag that already
exists, so `v1` on the forge can never catch up. Only a NEW immutable tag
syncs, which is exactly why the rule says to name one.
If this does not restore the lane, the next thing to read is why the forge
declines to schedule a called workflow's jobs while scheduling sibling jobs
in the same run.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
podSpec named no ServiceAccount, so every sandbox pod ran as `default` — and
DOKS's registry integration re-attaches its own DigitalOcean pull secrets to
every namespace's `default` account whenever it reconciles. A sandbox therefore
inherited a credential for a registry that is not ours and died asking
registry.hanzo.ai for its image anonymously: a 401 that reads like a bad
password and was in fact no password at all.
`sandbox` is ours and DOKS does not manage it. It is bound to no Role and
carries exactly one thing, the pull secret; the line above it still refuses the
pod a token, so naming an account grants nothing that omitting one withheld.
A constant and not an env var: which accounts exist in the sandbox namespace is
decided by the manifest that creates the namespace, so a second knob here could
only ever disagree with it. Declared in universe infra/k8s/sandboxes/registry.yaml.
Rebuilt from hanzoai/admin apps/tracker@e3d5672. The bundle changes what the
page does about identity and chrome; nothing in this package's Go changed.
ONE DOOR. The SPA signed in against `<origin>/v1/iam` — whatever IAM the host
serving it embeds. That is a second identity authority by construction, and
where the embedded store is not the one holding the accounts it fails as a login
screen nobody can get through. It now signs in at hanzo.id, which is the issuer
this binary validates against (brand registry: hanzo -> https://hanzo.id, JWKS
https://hanzo.id/v1/iam/.well-known/jwks). The bundle carries no form and no
password field: it asks IAM to authenticate somebody and holds the bearer.
That is the point of the cutover rather than a detail of it. tracker.hanzo.ai is
Huly today and Huly runs its own login form, so replacing it is only worth doing
if it REDUCES the number of places identity can be established.
TWO CONTROLS. The chrome is the org switcher (left) and the user menu (right)
over the board — no nav rail, no env/version/clock/theme chips.
THE SWITCHER RE-SCOPES. The selection rides as X-Org-Id on every /v1/tracker
call. It is a request, not a claim: SanitizeIdentity already deletes that header
and re-mints it, honouring the value only when the token's signed `orgs` claim
contains it — so the switcher lists exactly that claim, and cannot offer a
selection the server would silently swap for the home org. No change was needed
on this side; the mechanism was already here and tested.
embed.go and the README said the SPA "sends no tenancy of its own", which is now
half true and would read as a promise the page no longer keeps. They say what it
asserts (nothing) and what it asks for (a selection the server validates).
Verified against this bundle in a browser: sign-in offers one button and zero
inputs and leaves for https://hanzo.id/v1/iam/oauth/authorize with S256 PKCE and
client_id=hanzo-cloud; the board renders with exactly the two controls; choosing
acme sends X-Org-Id: acme and returns acme's board. Bundle audit: React only —
zero Svelte/Vue/other — and zero credential surface.
NOT verified here: `go build` and `go test ./apps/tracker/...`. This box has no
Go toolchain. The Go sources are unchanged, and embed_test.go's assertions were
checked by hand against the new dist/ (base /tracker/, /v1/tracker baked in,
every asset index.html names is present, .sync-stamp still dot-prefixed).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
openapi.yaml is GENERATED. git merged it as text, so the routes merged to the
union (1764) while the document took one side (1759) — five paths served and
unpublished, which every SDK generator would have missed. TestTheServedDocument
IsTheArtifact caught it and named the fix; this is the fix.
fleet/mcp.go kept both halves: Serve publishes the door at an internal address,
signpost answers the framework default with a 308. d.Serve IS on.Post(path,
d.serve), so the abstraction subsumed the raw line rather than racing it.
Every sweep in mk/fleet.mk was a `for` in a shell recipe, and a shell loop does
one thing at a time: 121 independent links, one after another, with the Go
toolchain itself pinned to two compilers. From an empty cache that shape takes
586s. The same builds scheduled across the box take 300s, and 25s warm.
They are static-pattern targets now — one per app, so the set is CLOSED and
`build/nosuchapp` names the app instead of quietly matching. (They are also
explicit rules, which is what makes them work at all: an implicit `build/%:` is
never searched for a target listed in .PHONY, so the phony declaration these
obviously wanted turned every one of them into "nothing to be done".)
`make -k` replaces `set -e`, and that is the part that was actually dangerous.
The old loop stopped at the first failure and never ran the apps behind it,
which reports nothing, and nothing is indistinguishable from passing — it is how
an unbalanced brace in apps/commerce hid ~45 apps that silently never
regenerated. make continues past every failure and names each one itself.
THE EXEMPTIONS WERE EXEMPTIONS FROM DESCRIBING, APPLIED TO BUILDING. kafka needs
a live broker and zen is coresident, so neither can MOUNT alone — and `describe`
was the only sweep that ever touched an app, so neither was ever COMPILED. Both
could have stopped linking on main with every gate green. `binaries` carries no
exemptions, because there is no such thing as an app that cannot be compiled by
itself; `describe` builds those two and skips only the projection. All 121
manifest apps link alone, verified as an exact bijection against ./bin.
`check` CALLS describe instead of restating it. They were one sweep written
twice with different skip messages and different error text, which is what let
them disagree — the bug its own exemption comment already had to be written
about once.
`dist` is that same recipe with the platform spelled into the name:
dist/<app>-<os>-<arch> for every app and every platform, 242 binaries, which is
the triple manifest/release.go resolves a plugin by. CGO_ENABLED=0 there is
load-bearing — a plugin fetched over the network runs on a box we did not build.
Two bugs found writing it: `VAR=x cmd1 && cmd2` sets the env for cmd1 only, so
the floor cross-compiled and everything behind it built for the host; and
`go generate` inherited the cross-target and tried to exec an amd64 zipdoc on
arm64. A generator runs where make runs, always.
SIZING IS ONE RULE, J apps at once with P compilers each, and J*P is what the
box sees. J is bounded by MEMORY read from the cgroup before /proc/meminfo,
because the git-runner pod is 26Gi on 6 CPU while nproc inside it reports the
NODE's cores — sizing off nproc there asks for a dozen concurrent links in a
cgroup that holds eight. The heaviest link measures 1.67 GB resident, not the
6.23 GB the pod's config still cites; that number predates the light host.
mk/go.mk stops imposing -p=2 on SOLO builds, which never had the problem it was
protecting against: one app from an empty cache is 57.6s at -p=2 and 41.4s with
the box. The runner's own injected GOFLAGS still wins, via `?=`.
Two negative results are recorded rather than dropped, because both are the
first thing the next person reaches for:
* Prebuilding the shared floor is SLOWER. J cold builds each compile the
587-package root, so warming it first is the obvious fix; at J=10 P=2 it went
300s -> 339s (the root) -> 327s (all 4149 packages). One process on a
dependency-shaped graph idles the box longer than the duplication costs.
* More parallelism is slower past a point: 431s at J=20 P=2 against 300s at
J=10 P=2, the same work with twice the actions.
hanzo.yml names the binaries: lane it cannot yet declare, and both blockers,
instead of tripping over them: ci's run:/out: lane indexes per RECIPE rather
than per FILE (unreadable to manifest/release.go), and `bucket:` needs an
S3_ADMIN_* credential that is in KMS for no org — declaring it would publish
nothing and red every tag build, which is the state ci's own site: lane shipped
in.
Two defects behind "it is SUPER SLOW and it seems stupid". Measured in
production, a turn is:
slack ack 0-6 ms (well under Slack's 3s retry threshold)
plane overhead ~25 ms (53,009 vs 52,985)
model completion 9,955 / 35,893 / 52,985 ms <- all of it
So the plumbing is not slow; the model takes ten to fifty seconds, and until now
the person saw an EMPTY THREAD for the whole of it. That is indistinguishable
from a dead bot, and it is what the "does nothing" reports actually were.
setStatus is Slack's own affordance and the only one available: there is no SSE
to a Slack client, so the honest vocabulary is a status then a message, never a
token stream. A second placeholder message would be worse — it occupies the
thread with something the reader must skip. Best-effort by construction: a
3-second budget, every error swallowed, because a courtesy that runs BEFORE the
work must never delay the answer it announces. Skipped when there is no thread
rather than faked.
THE MODEL DID NOT KNOW ITS TOOLS HAD A PROTOCOL, and this is why it looked
stupid rather than merely slow. The surface was collapsed from 1,189 flat tools
(977 KB, ~244k tokens just to LIST) to 88 grouped hanzo_<subsystem> tools whose
only argument is an `op` enum of bare names — the schemas are fetched on demand
through hanzo_describe. That trade is a WIN only if the model is told how to
make the fetch. It was not: builtinAgentInstructions was three sentences that
never mentioned tools at all. The model saw 88 tools it could not interpret and
answered from memory, which is exactly the observed behaviour.
The instructions now state the protocol — grouped by subsystem, choose an op
from the enum, call hanzo_describe for a shape you do not know — and say plainly
that answering from memory is wrong here, because the question is about THIS
org's live cloud and no training data contains it.
Kept short on purpose: every sentence is read on every turn and spends context
the user's actual question needs.
go build ./... clean; apps/integrations passes.
(cherry picked from commit 0d4093193e)
Two defects behind "it is SUPER SLOW and it seems stupid". Measured in
production, a turn is:
slack ack 0-6 ms (well under Slack's 3s retry threshold)
plane overhead ~25 ms (53,009 vs 52,985)
model completion 9,955 / 35,893 / 52,985 ms <- all of it
So the plumbing is not slow; the model takes ten to fifty seconds, and until now
the person saw an EMPTY THREAD for the whole of it. That is indistinguishable
from a dead bot, and it is what the "does nothing" reports actually were.
setStatus is Slack's own affordance and the only one available: there is no SSE
to a Slack client, so the honest vocabulary is a status then a message, never a
token stream. A second placeholder message would be worse — it occupies the
thread with something the reader must skip. Best-effort by construction: a
3-second budget, every error swallowed, because a courtesy that runs BEFORE the
work must never delay the answer it announces. Skipped when there is no thread
rather than faked.
THE MODEL DID NOT KNOW ITS TOOLS HAD A PROTOCOL, and this is why it looked
stupid rather than merely slow. The surface was collapsed from 1,189 flat tools
(977 KB, ~244k tokens just to LIST) to 88 grouped hanzo_<subsystem> tools whose
only argument is an `op` enum of bare names — the schemas are fetched on demand
through hanzo_describe. That trade is a WIN only if the model is told how to
make the fetch. It was not: builtinAgentInstructions was three sentences that
never mentioned tools at all. The model saw 88 tools it could not interpret and
answered from memory, which is exactly the observed behaviour.
The instructions now state the protocol — grouped by subsystem, choose an op
from the enum, call hanzo_describe for a shape you do not know — and say plainly
that answering from memory is wrong here, because the question is about THIS
org's live cloud and no training data contains it.
Kept short on purpose: every sentence is read on every turn and spends context
the user's actual question needs.
go build ./... clean; apps/integrations passes.
The paywall could not be switched on. A brand-new org's wallet is $0, the
automatic starter grant was deleted (41b23f12), and the "$5 free credit" the
catalog advertises was never provisioned -- so enforcing SpendGate would have
402'd every new signup on request #1. This adds the missing rung.
WHERE IT SITS. One call site, in standing()'s proven-unpaid branch, past the
enforcement check: the last point at which an account that has never been
funded can still be told apart from one that will not pay. Both authorities
have already answered, so the rung inherits "no subscription and no credit"
as a finding rather than re-reading it.
WHY THIS IS NOT THE MECHANISM THAT WAS DELETED. The old grant ran as app-wide
middleware on first credential contact and minted into any wallet that did not
exist yet. This one:
- is not money -- it posts the plan's own advertised term under the shared
starter-credit tag, which billing/bucket classifies non-cash: spendable on
metered usage, never refundable, never paid out;
- is not unbounded -- the amount is credit.StarterCreditCents, the constant
the catalog's entry plan advertises, and fund() takes no amount at all, so
no request body reaches it;
- is not repeatable -- starterRef keys the deposit on the address it credits,
with no time, nonce or request id, so finance dedups it inside the same
transaction as the insert;
- is not unscreened -- every grant is a PRIVILEGED Decide at StageSignup, so
a flagged signup receives nothing and a scorer that is present and silent
withholds. That is the bound on "$5 x as many fake signups as can be made";
- has no switch of its own -- it is unreachable while the paywall is dark, so
the mint and the refusal it cures are the same flag. That is the structural
answer to "a disabled money-mint is one flag away from an enabled one".
Two further guards keep one grant to one customer: the shared signup org is
excluded (a login is not an account), and only the caller's home org is funded,
because a founder is a member of every org they create and position -- orgs[0],
minted as X-User-Owner -- is what marks the one that is theirs.
Funding is not an authorization decision: a grant that cannot be posted leaves
the gate to refuse in its own words, with the actionable 402 body.
No switch is flipped here. SwitchPaywallEnforced and enableSignUp stay as they
are; this only makes the rung exist so the flip is a pricing decision.
Every mechanism above is mutation-proven: removing the rung, the ref dedup, the
screen, the Privileged bit, the lifetime-usage leg, the catalog amount, the
dark-gate guard, the home-org guard or the signup-org exclusion each turns a
test red.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The paywall could not be switched on. A brand-new org's wallet is $0, the
automatic starter grant was deleted (41b23f12), and the "$5 free credit" the
catalog advertises was never provisioned -- so enforcing SpendGate would have
402'd every new signup on request #1. This adds the missing rung.
WHERE IT SITS. One call site, in standing()'s proven-unpaid branch, past the
enforcement check: the last point at which an account that has never been
funded can still be told apart from one that will not pay. Both authorities
have already answered, so the rung inherits "no subscription and no credit"
as a finding rather than re-reading it.
WHY THIS IS NOT THE MECHANISM THAT WAS DELETED. The old grant ran as app-wide
middleware on first credential contact and minted into any wallet that did not
exist yet. This one:
- is not money -- it posts the plan's own advertised term under the shared
starter-credit tag, which billing/bucket classifies non-cash: spendable on
metered usage, never refundable, never paid out;
- is not unbounded -- the amount is credit.StarterCreditCents, the constant
the catalog's entry plan advertises, and fund() takes no amount at all, so
no request body reaches it;
- is not repeatable -- starterRef keys the deposit on the address it credits,
with no time, nonce or request id, so finance dedups it inside the same
transaction as the insert;
- is not unscreened -- every grant is a PRIVILEGED Decide at StageSignup, so
a flagged signup receives nothing and a scorer that is present and silent
withholds. That is the bound on "$5 x as many fake signups as can be made";
- has no switch of its own -- it is unreachable while the paywall is dark, so
the mint and the refusal it cures are the same flag. That is the structural
answer to "a disabled money-mint is one flag away from an enabled one".
Two further guards keep one grant to one customer: the shared signup org is
excluded (a login is not an account), and only the caller's home org is funded,
because a founder is a member of every org they create and position -- orgs[0],
minted as X-User-Owner -- is what marks the one that is theirs.
Funding is not an authorization decision: a grant that cannot be posted leaves
the gate to refuse in its own words, with the actionable 402 body.
No switch is flipped here. SwitchPaywallEnforced and enableSignUp stay as they
are; this only makes the rung exist so the flip is a pricing decision.
Every mechanism above is mutation-proven: removing the rung, the ref dedup, the
screen, the Privileged bit, the lifetime-usage leg, the catalog amount, the
dark-gate guard, the home-org guard or the signup-org exclusion each turns a
test red.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
podSpec named no ServiceAccount, so every sandbox pod ran as `default` — and
DOKS's registry integration re-attaches its own DigitalOcean pull secrets to
every namespace's `default` account whenever it reconciles. A sandbox therefore
inherited a credential for a registry that is not ours and died asking
registry.hanzo.ai for its image anonymously: a 401 that reads like a bad
password and was in fact no password at all.
`sandbox` is ours and DOKS does not manage it. It is bound to no Role and
carries exactly one thing, the pull secret; the line above it still refuses the
pod a token, so naming an account grants nothing that omitting one withheld.
A constant and not an env var: which accounts exist in the sandbox namespace is
decided by the manifest that creates the namespace, so a second knob here could
only ever disagree with it. Declared in universe infra/k8s/sandboxes/registry.yaml.
The drift gate stopped on commerce again, this time on the four reads mount.go
registered so the billing app's tabs would stop rendering empty: GET
/v1/billing/transactions, /credit-balance, /accounts and /accounts/{id}/members.
They were routed and described nowhere, and an operation that says nothing about
itself publishes an operationId and no sentence — an SDK method that cannot
explain itself and an MCP tool a model cannot pick.
They are raw by nature — the handlers live in the commerce module, so there is no
doc comment here for zipdoc to lift — so the prose is declared with
openapi.Describe beside the route table, which is where this file already keeps
the other seventy-five.
The prose is written from the handlers and from the chain mount.go gives them,
and it leads with the fact a reader is most likely to get wrong: these handlers
filter on a user or userId parameter, and PinBillingSubject OVERWRITES that key
with the caller's own account.Payer subject before the handler runs. So the
parameter is not the caller's to choose — naming another subject returns your own
rows, not theirs. Saying otherwise, or saying nothing, would leave a generated
client's author believing they had found a way to read another tenant's ledger.
accounts/{id}/members has no subject key to pin and guards itself instead, by
comparing the path segment against the resolved org and answering 403; that is
stated too, because it is the one member of the family whose refusal comes from
somewhere else. Both refusals are 401 unauthenticated rather than 403, because a
browser re-authenticates on the first and merely reports the second.
Regenerated from source: 1759 -> 1768 paths, 2430 -> 2440 operations, billing 39
-> 43. Every number rises; nothing shrank, so the floor moves up rather than
being lowered.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Follow-up to 7e34af0a, closing the two things it left open.
The terminal handler answered with `application/json; charset=utf-8` — this
file's usual spelling — while zip's mounted route answers `application/json`.
An agent could therefore tell WHICH adapter carried its answer, which is the
second door reappearing as a header. Unified to zip's media type, and
TestDoorlessPluginAnswersItsOwnDoor now compares the two paths' Content-Type
directly, so a future divergence fails rather than merely looking untidy.
TestCatchAllNeverShadowsTheDoor pins what was NOT the cause. Registration order
was the obvious suspect and it is innocent: zap-proto/fiber's insertRouteSorted
ranks a longer static literal ahead of the greedy `/*`, so a control route
registered late still sorts ahead of the console catch-all. Nor was it the
zip v1.25.1 -> v1.27.0 bump — generation.go is byte-identical across the two.
The cause was only ever installMCP declining to mount a route for an app with an
empty edge registry. Both dead ends are recorded in the test and in
manifest/mcp.go so the next reader does not re-run the investigation.
manifest/mcp.go's claim is corrected at its source. It said the host's framework
path "is claimed by nobody, which is exactly why the front door has to name it".
The host claims it now, from fleet.Mount, and a plugin answers there with its own
door — so the comment describes the mechanism that exists rather than the one
that was replaced.
v1.50.16 (already live) stopped the rail from handing out addresses nothing
could credit. These four carry the machinery that will eventually let it be
lifted, and all of it ships COLD:
- v1.50.18 the EVM deposit watcher. Exactly-once is a property of the KEY —
sha256(chain:txHash:logIndex) against a backend that upserts — so re-scans,
crash retries and N replicas all yield one credit with no leader election.
Disabled unless CRYPTO_DEPOSIT_* assets are configured, which they are not.
- v1.50.19 GET /v1/billing/crypto/options answers from the watcher's assets
instead of the MPC processor's mintable chains, so the picker cannot name an
asset nothing is watching.
- v1.50.20 GenerateAddress returns {Address, ID} and the intent records the
custody handle, so a credited deposit is one we can also sweep.
The gate (cryptoDepositsCanBeCredited) is still false and is a constant, not
config — nothing in this bump can take money.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Red re-reviewed the eight-door work and found the ref policy bypassed by the
code added to close its own lowest-severity finding. L2 was "shallow and signed
pushes fail to parse" — an availability nit. The push-certificate parser written
to fix it disagrees with real git about where a certificate's commands are, and
the disagreement is total.
git concatenates every cert pkt-line payload into one buffer and takes the
commands between the FIRST "\n\n" and the signature offset
(builtin/receive-pack.c, queue_commands_from_cert). This parser treats one
pkt-line as one logical line. Embed a "\n\n" inside a header payload and the two
part ways: we parse ZERO commands, git parses one and applies it.
Zero commands WAS the exploit. checkRefPolicy over an empty slice iterates
nothing and returns nil, because an empty command list satisfies every policy —
including a grant confined to one agent ref. The raw bytes are then forwarded to
git receive-pack, which executes the command we never saw.
Verified by Red against git 2.43: a 262-byte certificate whose `pusher` payload
carries "\n\n<old> <new> refs/heads/main\n" makes a grant scoped to
refs/heads/agent/<session> write refs/heads/main — which fires cloud.OnGitPush
and deploys a model-authored, unreviewed commit. No signature and no nonce are
needed; the forge sets no receive.certNonceSeed. Negative control: the same
trunk write as a PLAIN push under the same grant is correctly refused, so the
policy works and the hole is specifically the framing.
TWO GUARDS, either sufficient, so neither is load-bearing alone:
1. parseRefCommandsCaps refuses any pkt-line payload carrying an embedded
newline. This makes the parser's own one-line assumption TRUE rather than
assumed. Narrower and safer than teaching it git's concatenation: parity with
a second implementation must be re-proved every time either side changes,
whereas a payload with no embedded newline can only be read one way BY BOTH.
Nothing legitimate is lost — git's send-pack emits one line per pkt-line.
2. checkRefPolicy refuses a grant-bearing push that names no ref. A grant exists
to write one named ref, so naming none means the frame did not parse the way
we think it did. For a principal an empty push stays a harmless no-op.
Tests use Red's exact 262-byte exploit and FAIL without the guards — verified by
reverting them and re-running. The negative control (an ordinary single-line
push still parses and yields its one command) is what keeps the fix from being
worse than the bug.
Red's other findings are NOT closed here and are follow-ups, all lower:
writer 9 (corePush -> initBare sets HEAD to an agent branch on a NEW repo,
dead-ending checkHeadRef, principal-only), a check-then-set TOCTOU in corePush
(go-git SetReference is not CAS; the wire path is safe because git does CAS ref
locking), and zapface minting a socket for any credential string.
apps/git shows the same 3 pre-existing failures as baseline; go build ./... clean.
TestNoSecondMCPDoorInSource matched any string literal ending in /mcp, which made
`"github.com/zap-proto/mcp"` read as a second door. It is the opposite: webui's
terminal handler imports the PROTOCOL precisely so it can hand a frame to zip's
existing door rather than write an envelope of its own.
The pattern now requires a leading slash, which is what actually distinguishes
the two — a door is a route path, rooted and ending at /mcp; an import path is
neither. The gate still fails on a real second door, and apps/tasks' raw net/http
mux handler is still the reason it reads SOURCE rather than the document.
Every per-app plugin binary answered POST /mcp with 308 to /v1/mcp, and then 404
there. Measured on bin/kms and bin/tasks before this change; the door a subsystem
owns was unreachable over HTTP.
The console's terminal handler sent it. Its reasoning was that a plugin serving
its own door at the framework default matches a real route and never reaches a
terminal handler — so anything arriving there had to be a caller who guessed the
default on a host that moved it. That is false for a whole class of app: zip
mounts the /mcp route only when there is something to expose (installMCP), and a
plugin whose typed ops live on the internal plane has an EMPTY edge registry.
kms is the honest example — its four secret ops are on cloud.Plane() precisely so
no route runs from the edge to a secret — so kms has no route at its own door,
fell through to the console, and was told its door lived at an address only a
host serves.
The cost is not cosmetic. The fleet composes tools/list by asking every child at
FrameworkMCPPath and reading any non-2xx as an outage (fleet/fleet.go ask), so
such a child drops out of the composed list AND is reported down — for the crime
of having no tools. An empty list is the honest answer and it is a 200.
Two changes, each in the place that holds the fact:
The console SERVES the door instead of redirecting it. The door is not the route:
it is zip.App.MCP, a frame in and a frame out, which exists whether or not
anything was mounted over it. Serving zip's own door is not a second surface;
re-implementing one, or redirecting to a door this process does not have, is.
The signpost moves to fleet.Mount, which is the one place that KNOWS the door
moved, because it is the call that registers the new address — and it registers
the signpost only when the two differ. A host still tells a caller who guessed
/mcp where /v1/mcp is; a plugin, which moved nothing, no longer claims it did.
Verified against freshly built binaries, not by reading code:
bin/kms POST /mcp initialize -> 200, protocolVersion 2025-06-18, serverInfo kms
POST /mcp tools/list -> 200 {"tools":[]} (was 308 -> 404)
bin/blueprint POST /mcp tools/list -> 200, 2 tools, both carrying lifted prose
POST /mcp tools/call -> 200, get_v1_blueprint_health returned live data
git.hanzo.ai is canonical for every repo; GitHub is a mirror. These two had
diverged and BOTH sides held real work:
forge only (8): the platform deploy/git-lane series — b17f9e6a..26cfa8e5
hanzo-inc only (1): 827f3e66 plane: a call expires on the CALLER's deadline
Production was running 827f3e66 — a commit that existed ONLY on GitHub and not
on the canonical tree. That is the failure mode the one-remote rule exists to
prevent: the deployed sha was unreachable from the repo everything else reads.
Merged rather than force-pushed in either direction, so no commit is lost from
either side.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three conflicts, all mechanical: forge renamed Mount's router to zapp, moved
fleet.go's openapi.Describe calls out (so that import goes), and switched
runnerBuild to a plain ctx. Kept forge's shape in each and re-applied the
delivery wiring, the org attribution and the namespace import on top. Renamed
this package's test helper keysOf to mapKeys — forge added a keysOf of its own
with a different signature.
Verified by DIFF, not by commit count: this estate has a merge that kept a fix's
commit and dropped its diff, so every security-critical seam was re-checked in
the working tree — seal-by-default, the audit's caller-declared public set, the
tenant- reservation, owner() on both boards, checkFence, the rendered-patch
guard, tenantNamespace's single derivation, per-org build attribution, and the
deleted resource/logs handlers.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A typed op mints five things from one registration. Three defects here let an
operation reach all five carrying no sentence, and one hand-written file
described a surface nobody generated.
POST /v1/exec published a summary and no description. The handler had a doc
comment; the package had no zipdoc directive, so nothing lifted it — and the op
registered on the cloud.Router parameter, which zipdoc cannot follow to a prefix,
so it refuses to lift even with the directive. openapi.Complete accepts EITHER a
summary or a description, so every gate stayed green over the hole. The
registration moves to the *zip.App (the apps/meet and apps/blueprint pattern);
routes move, the credential middleware stays on the scoped router where the
prefix guard applies to it. Its 15 published fields carry prose now too.
docs/automations-openapi.yaml is deleted. 25 KB of hand-authored OpenAPI over 17
operations of /v1/automations, referenced by nothing and compared by no gate — so
it had drifted the way a second copy always does: it claimed two operations the
fleet does not serve (GET /v1/automations/health, POST /v1/automations/mcp) and
omitted three it does (connectors/{id}/run, flows/{id}/versions,
hooks/{source}/{event}). The document already describes that surface.
The field tranche closes 343 published properties across seven apps — authors
39->0, label 37->0, channels 34->0, prompts 29->0, leaderboard 40->0, campaign
42->6, affiliates 123->10 — and campaign's two result types stop being defined
types over another struct, which published all 30 of their properties bare; the
struct is declared under its published name and the domain name is an alias, so
there is still one shape.
The 16 that remain are not app defects and are recorded as such in LLM.md: zip
keys a promoted field's prose under the type that DECLARES it and looks it up
under the type that PROMOTES it, so no comment written here can reach them.
LLM.md records what the running deployment measures, taken by making the request:
the served document is the committed one at the revision the header names (1735
paths / 2474 ops, identical sets), /v1/commands answers 2448 commands under a
working ETag, POST /v1/mcp lists 88 tools and tools/call returns lifted prose —
and those tools reach only 1189 of 2422 operations, with 134 declared refusals
accounting for a tenth of the gap.
Red proved the classifier missed 8 of 8 real credential shapes, each for its own
reason: PGPASSWORD (libpq's own variable, one token to any splitter), *_PW (`pw`
absent), a symbol-rich password (the check returned false the moment it saw a
symbol, so a STRONGER password was MORE likely to be published), KUBECONFIG
(cluster-admin, base64 material, no PEM armour), a base32 MFA seed. Every fix
would have been another special case in a denylist over an unbounded set.
And the three gates were ONE gate: leakedSecret called the same mustSeal the
split called, so a miss passed both and reached git. An independent-looking
backstop that shares its single point of failure is worse than none, because it
is counted as defence.
THE FIX IS THE POLARITY, NOT A BETTER CLASSIFIER. This lane's output is a commit
in a repository replicated to every clone and impossible to unpublish, so what
matters is not how good the guess is but which way it fails. Every env value is
now a KMS reference UNLESS the caller explicitly marks it public. A miss means an
operator cannot read back a config value — a support ticket. The old miss meant a
password in git history — an incident with no rollback. It needs no list, so
there is no shape left to overlook, and it matches what the console already
sends. It also closes the over-seal complaint from the other side: GIT_COMMIT,
IMAGE_DIGEST and TENANT_ID are marked public and stay readable.
The audit now shares NO code with the decision it checks. It parses the RENDERED
bytes and asks a fact — "is this key in the set the caller marked public?" — not
a judgement. The public set is carried on the spec straight from the request and
is NOT derived from spec.Env: an audit that asks "is this key in Env?" of a
document rendered FROM Env answers yes by construction, which is the same
tautology in new clothing. A value reaching the render by any path the caller
did not authorise is refused, whatever it looks like.
V6 — the templatePatch guard scanned for the substring "project", which the very
engine that runs it defeats: `{{ "pro" }}ject:` and "\x70roject:" both set the
key and both evade a scan. It renders the patch with the generator's own funcs
and data and reads the RESULT as YAML. The real fleet patch (dig/syncPolicy)
still passes; anything resolving to a project is refused.
The operator lane keeps shape classification — it writes a database column it
can rewrite — but the named misses are closed there too, since they cost
nothing: substring matching for the names with no separator to tokenise,
`pw` added, and the symbol short-circuit inverted so a symbol-bearing,
space-free string of password length is password-SHAPED rather than exempt.
V2/V4 — evidence, and the containment, recorded at the emission site. The leading
slash is RIGHT: eleven live values files in universe use `secretsPath: /<path>`
and are healthy, and the chart schema requires it; the operator lane's no-slash
form is the outlier, flagged and not changed because it is live. Still
unverified: a CUSTOMER org's projectSlug resolving, since every live example is
platform-tier. Nothing from this lane can reach a pod yet — a declaration lands
on a branch the generator does not read, and modeCommit is refused by checkFence
until universe carries the companion rule. Both gates open deliberately, by a
human, and the first merge is where this resolves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A key-NAME regex running in a BROWSER decided which env values were sealed. Two
things are wrong with that and the second is the one that matters: the key is
not the secret, the VALUE is — GCP_SA_JSON names nothing credential-ish and
holds a private key, while STRIPE_SK, SK_LIVE, GH_PAT, PGPASS, SMTP_PASS, HMAC
and TLS_CERT are ordinary names that pattern missed — and a control the caller
can edit is not a control. secretshape.go decides on the SERVER now: issuer
prefixes and PEM armour, connection strings carrying a password, and a narrow
entropy tier for tokens with no recognisable prefix, OR'd with a server-side key
test so DB_PASS=hunter2 is caught too. Both signals, because they fail in
opposite directions. The client flag stays as UX and may only ADD secrecy.
Not default-seal-everything: PORT, NODE_ENV and LOG_LEVEL would become
write-only values an operator cannot read back, and every config line would take
a KMS round trip on the deploy path.
★ THE DECLARE LANE WAS WORSE THAN THE OPERATOR LANE. sealSecretEnv protects a
database column; a values file is committed to universe, replicated to every
clone, and cannot be unpublished — a credential written into one is cleartext in
git history forever. The split now happens BEFORE anything is rendered: secrets
are sealed into KMS at the coordinate the operator lane already uses, and the
file carries a `kmsSecrets` reference plus env valueFrom.secretKeyRef. That is
the chart's own rule, in its own words: "Secrets are REFERENCES, never values —
that is what makes this values file safe to publish." Fail-closed harder here
than in the operator lane: no KMS, no deploy. And a last-line guard re-asks the
question of the BYTES about to be committed, so the decision is proven to have
held all the way to the write rather than assumed.
V1 — reservation extended to the namespaces the cluster actually runs (adnexus,
bootnode, team-go, preview, registry, pars-*) plus `*-system` as a SUFFIX, so the
next operator installed is not claimable in the window before someone extends a
list. Deliberately STATIC rather than derived from the live namespace set: a
derivation is the stronger rule but fails OPEN exactly when the apiserver is
unreachable, which is the moment it is most needed. Wrong only by omission,
never by outage.
V2 — fenceOf saw only spec.template. templatePatch is rendered as TEXT and
merged OVER it, so a patch setting spec.project overrides the fence unseen;
evaluating the merge faithfully means reimplementing the generator, so a patch
that mentions `project` is refused instead. And `env`/`expandenv` are removed
from the function map: they read the PROCESS environment, so the same expression
would render from cloud's environment here and the controller's there — one
expression, two answers, and the deciding one is not ours.
tenantNamespace derives ONCE. It sanitized an input that tenant(s, c) had
already sanitized, and Sanitize is not idempotent — re-folding its own
<fold>-<hash> output appends a second hash. So for every org whose name is not
already a clean label, platform wrote App CRs into tenant-<double> while
apps/deploy and apps/provisioning both scanned tenant-<single>: fails closed,
nothing crosses a boundary, but the org's board is silently empty and its CRs
orphaned. It was the outlier of three copies; the other two already took the
slug. Non-label input now resolves to the inert "unknown" rather than rendering a
malformed namespace, so a caller that skips tenant(s, c) fails closed too.
Deleted the unrouted, unscoped appResource/appTree/appLogs. They resolved a
namespace with no org filter and no redaction, and the typed client already
ships tree/resource/logs methods — wiring them would have been a cross-tenant
unredacted read. resource.go and logs.go go entirely; tree.go keeps buildTree,
which dashboard and detail actually route.
Follow-ups flagged, not slipped in: a SanitizedOrg type in hanzoai/namespace
(26 call sites across three packages, a cross-repo release) is what would make
double application a COMPILE error instead of a guard test; and the legacy
tenant-* namespaces still want migrating onto the bare-org layout.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
R1 (HIGH, red) — THE RENAME'S OWN SHADOW. Dropping the `tenant-` prefix from
this API made `tenant-` an ordinary name, but the cluster's live fences are
still spelled with it: universe project-tenants.yaml declares AppProjects
tenant-hanzo|lux|zoo|zen|maxpower, each admitting namespace tenant-<org>. A
clean label, so Sanitize passes it through untouched — an IAM org named
`tenant-maxpower` would have claimed the REAL maxpower tenant's namespace under
maxpower's own fence and read its CD rows. Reserved as a FAMILY, so the next
tenant onboarded under the legacy layout is not claimable in the window before
someone extends a list. Red's PoC (7 tests) is kept and green.
That fix exposed a conflation in my own predicate, and splitting it is the real
change here. `reserved` was answering two questions at once:
reserved(dir) — may an org CLAIM this directory? The WRITE question. Wider
than platformOwned by exactly the tenant- family: those
directories are not the platform's, but they are already
somebody's.
owner(ns) — which org does this namespace BELONG to, "" for the
platform's own? The READ question. It decodes BOTH layouts,
because both are live.
Conflated, reserving tenant- would have blanked every legacy tenant's own board.
Two names, two questions, and the boards share the second one.
R3 (MEDIUM, LIVE) — /v1/platform/fleet asked nsOrg, which maps the brand
namespaces onto org "hanzo", so the ORG ADMIN of the brand org was handed the
platform tier — iam, kms, gateway — while the delivery board refused the
identical caller. A per-org isAdmin is never platform-privileged (HIP-0519);
this file's own deploy route already applied that reasoning to RESTARTING one of
these services, and observing them is the same class of act, only quieter, which
is why it survived. Both boards ask owner() now. Two tests asserted the old
behaviour in so many words ("an OrgAdmin of the platform org sees its own org's
whole board"); that expectation WAS the bug and they now assert the refusal.
R2 (MEDIUM) — checkFence refused one known-bad substring, so every OTHER unsafe
template passed: an unconditional `project: hanzo-platform`, a file with no
project, an empty file, garbage. A denylist of one is not a check. It now
asserts the POSITIVE — parses the ApplicationSet, evaluates its project template
for this org with the same engine and options the generator uses (text/template
+ sprig, missingkey=error), and compares to declareProject. Anything it cannot
answer is refused, including a template that reaches past the path for the fence
(the "thing being fenced chooses its fence" defect the ApplicationSet itself
rejects). Red's six templates are the refusal table.
R4 (LOW) — DECIDED: the per-org build ceiling stays SOFT, and the reasoning is
recorded where the check is. It is check-then-act; the only atom available is
the Job name, already spent on idempotency, and a counter object would be a
second source of truth about how many builds are running. The overrun is bounded
by requests in flight and confined to the caller's OWN org. The cluster bound
belongs where bounds are enforced atomically — a ResourceQuota on the build
namespace, applied at admission, which no race can widen.
Follow-up flagged, not done here: retire the legacy tenant-* namespaces and
AppProjects onto the bare-org layout. That is a migration, not a rename — it
moves live workloads — and the reservation holds until it lands.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
CTO naming rule: no prefix on an org. The values directory, the destination
namespace, the AppProject and the image path are all `<org>` —
charts/app/values/<org>/<app>.yaml, namespace <org>, project <org>,
ghcr.io/hanzoai/<org>/<app>. One value, four roles, nothing added to it.
The `tenant-` prefix was a naming convention doing a policy's job, and removing
it is only safe once the policy exists, because namespace.Sanitize is the
IDENTITY on a clean label: without a separator, customer directories and the
cluster's own namespaces share one name space, and an IAM org named
`kube-system` resolves to the real `kube-system`. RESERVATION replaces it —
one predicate, asked on read and on write, over the platform's namespace
family: the brands and their environments, the control and delivery planes,
kubernetes' own, and `admin`. A reserved directory is SuperAdmin-only even when
it is the caller's own org.
`tier` is gone with the prefix. An org is its name, so placement is not a field:
`org` is an ACT-AS, defaulting to the caller's own, and naming another requires
SuperAdmin. Both refusals refuse rather than downgrade, so an escape attempt is
never indistinguishable from a normal request.
The fence's real rule lives in universe's ApplicationSet, so it is VERIFIED, not
documented: checkFence reads the live template out of the clone the write
already makes and refuses to put a declaration on main while that template would
fence it wider than this API reports. It clears itself when universe lands the
reservation form — no flag, nothing to remember — and a branch write is exempt
because nothing is generated from a branch. A comment saying "land the companion
change first" is not a control, and neither is a test that fails on a
developer's machine and skips in CI.
RED FINDINGS, all fixed with red's PoC kept and green:
F1 (HIGH, cross-org read) — the write path derived a directory with
namespace.Sanitize(org) while the read path confined with the RAW owner claim.
Sanitize on ONE side of an authorization compare is a collision waiting to be
named: org "Acme" owns "acme-<hash>", so it never matched its own rows, and any
org whose raw name IS that literal string matched them instead — offline
-computable, since the slugger is public code. Both sides are canonicalised now,
in cd.go owns AND in fleet.go scopeNamespaces, which carried the identical
compare and is LIVE on /v1/platform/fleet. Sanitize is injective, so this is
collision-free and not merely symmetric. cdApps had no tests; it now has a table
over every shape of name Sanitize treats differently.
F3 (build DoS) — a declare build was charged to the constant "platform", so one
org looping deploys exhausted a shared ceiling of 3 and locked the fleet out of
building, with no attribution in the Job labels to see it by. launchDirectBuild
takes the org it is charged to; the ceiling is per-org, like /v1/runner's.
F4 — the default host interpolated the RAW owner claim, so any org without a
clean name got "web.Acme.hanzo.app", not a hostname at all. It is built from the
canonical org, the same value that is the directory.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every client retries. A branch write based on main committed the same tree at a
different timestamp on the second call, so a different sha, so a non-fast-forward
push and a raw git hint the caller could not act on. A branch write now BASES ON
ITS OWN BRANCH when that ref exists — the push is a fast-forward, and when the
declaration is already exactly this the whole call is a clean no-op reporting the
same ref and review URL.
The env comparison was the same defect one layer up: an existing declaration
refused ANY env, so a retry of a create-with-env could never succeed. It now
compares environments as sets of name=value and refuses only a genuine change.
The test that pins this sleeps a second between the two calls, and that sleep is
the point. Without it both commits land in the same second, git mints the
identical sha, the push is "everything up-to-date" and the bug is invisible —
which is exactly how the first version of this test passed while the bug was live.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A commit to main proves the image pullable, and the image a build launched by
the same call would produce does not exist until the Job finishes — so the
combination could only ever fail, after a privileged BuildKit Job had already
been spent on it. Refused at validation instead, naming the two-step flow the
caller wants: deploy (which returns the build's tag), then commit that tag once
the build is green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/platform/apps is the delivery surface for the ONE deploy plane. An app is a
values file in hanzoai/universe under charts/app/values/<namespace>/<name>.yaml;
the fleet ApplicationSet renders charts/app against it and cd.hanzo.ai reconciles
the result. Declaring the file is the whole of deploying the app, so the surface
is three moves: build the repository through the existing BuildKit lane, write
the declaration that names the image, and report the Application reconciling it.
Two things a caller may never choose, because both are fences.
The DIRECTORY. The generator derives the Application name, the destination
namespace, the Helm release name and — load-bearing — the AppProject from the
file's own path: tenant-<org> admits ONE namespace, no cluster scope and six
kinds, while hanzo-platform admits namespace * and ClusterRole/ClusterRoleBinding.
A caller that could name its directory could name its fence, so the directory is
derived from the validated owner claim and is not a request field. tier=platform
is the one lever over placement and it is SuperAdmin only.
The IMAGE REPOSITORY. A declaration is what the cluster pulls, so it is derived
per tenant on the same injective path the build lane already uses. There is no
image field to try.
A branch is not a deploy. The generator reads main, so the default mode pushes
deploy/<ns>/<name>/<tag> and deploys nothing; merging the review is the
deliberate act. mode=commit writes main and proves the image pullable first,
because a declaration naming an image the registry cannot serve is an
ImagePullBackOff with no rollback path.
The git seam is pin.go's, not a second one: the same shallow clone, the same KMS
-held token carried as an http.extraHeader rather than in argv, the same
fast-forward-only push retried by re-reading the tip. This composes with the pin
rather than duplicating it — declare ADDS a service deliberately, which is
exactly what resolvePinFile refuses to do, and the pin then moves its tag. An
update here moves that one scalar too and refuses anything that would rewrite a
hand-maintained declaration.
/v1/platform/cd reads the Applications in hanzo-cd. A cluster with no CD answers
an empty plane; a plane that cannot be READ answers 503 and says why, because
those are opposite facts. /v1/platform/ci answers 501 and names what is missing
rather than fabricating an empty run list — this deployment has no forge API
client. Static sites and bucket listing are NOT added: /v1/platform/sites and
/v1/s3/buckets already serve them.
Proved against the real chart, not a fixture: the generated declaration renders
through hanzoai/universe charts/app with its values.schema.json enforced, and a
key the chart does not declare is refused by that schema. A dry run against a
clone holding the real 105-file inventory wrote a scratch branch and left main
byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
refpolicy.go claimed receive-pack was "the point every path that changes a ref
passes through and no client can decline to". That was false, and it was not a
detail — it was the entire argument. It guarded ONE of eight ref writers into the
same bare repository, and the other seven took the credential the run already
held.
The cheapest was not the git wire protocol at all. One JSON POST to
/v1/git/repos/:name/push lands a FAST-FORWARD CHILD on the branch a human just
approved: worse than the force-push the wire door refused, because append-only
trips no "the branch was rewritten" signal anywhere. The reviewer approved A and
merges A+B. /repos/:name/mirror was worse still — `+refs/*:refs/*` with --prune
force-overwrites every ref from a source the caller names and DELETES any ref
that source omits. SSH receive-pack piped straight into git with no policy in
front of it, and any org member may enrol a key. Verified by test: before this,
a mirror from an attacker-chosen upstream deleted a reviewed agent branch and
created one of its own.
Underneath all seven was the real problem. The credential was an org-wide `agent`
key sealed in KMS — an ordinary IAM sk-, which IAM resolves to a USER, from which
cloud mints a full org principal, which cloud.Member admits everywhere. The one
process executing untrusted model output held something that opened
GET /v1/kms/secrets (every other secret the org has, including whatever posts to
its Slack) and every org-scoped API in the platform. Custody had not shrunk to a
safe place; it had shrunk to the worst place, holding a skeleton key.
So the credential stops being an identity. A run now asks the forge for a GRANT
(apps/git/grant.go): a bounded permission to drive the pack protocol against ONE
repository, creating ONE ref, until it expires. It authenticates nobody. It is
not a JWT and carries none of APIKeyPrefixes, so validatedPrincipal returns nil,
no X-User-Id is minted, principal.Validated is false, and every cloud.Guard and
every tenantOf refuses it BY DEFAULT — nothing had to be told to say no. The one
exception is resolvePackRepo, so the set of doors a grant opens is the set of
callers of that function, and a principal always wins because the grant is
consulted only where there is none. git owns refs, so git decides who may write
one; the orchestrator is no longer a credential custodian, and there is no
org-wide agent secret left in the world to seal or to leak. The constants that
named one are deleted, not deprecated: a constant naming a secret is an
instruction to seal one.
The policy then moved to all eight writers, enumerated in refpolicy.go beside the
rule so a ninth is a change to the list and not just a new function. Five state
their intent as the same refCommand value the wire door parses and call the same
function. Two cannot be judged command-by-command and are refused structurally
instead: the mirror takes a negative refspec so the machine namespace is outside
its refmap and therefore outside --prune, and HEAD — which no refspec reaches —
goes past checkHeadRef, including the first-push case where a client-less push
would otherwise make a run's branch the repository default, which is what the
deploy reactors gate on. One (tag fetch) was already structurally safe and now
says so.
Also, from the same review:
- Base was TrimSpace and nothing else while Repo and Org were shape-checked. It
reaches a `git clone -b <base>` argv on a CUSTOMER'S machine, where a leading
dash is not a branch but a flag, and --upload-pack= / --config=core.fsmonitor=
are each arbitrary execution there. BaseRE is git's own branch shape; the
load-bearing part is that a branch is alnum-led. Project is shaped too.
- The PR head is pinned to the cloud-issued BranchFor(sessionID). Adopting the
sandbox's self-reported branch let a compromised one answer `main`, which
passed VerifyRef (which only asks whether a ref EXISTS) and came out as
CreatePR{Head: "main"} — a pull request headed at the trunk, filed by us, for
a run that never had permission to write there. Same on the routed path,
where the reporter is a customer's machine.
- A refusal naming many refs was wrapped in ONE side-band packet. A pkt-line
length is four hex digits, so past ~65516 bytes it renders five and the client
desynchronises — the report the control exists to deliver became the
"RPC failed / unexpected disconnect" it exists to avoid. Now chunked, with
each line bounded independently.
- A push from a --depth clone (`shallow <oid>` lines) and `git push --signed`
(a push certificate wrapping the commands) both failed to parse and answered
400. Fail-closed, but closed on ordinary clients doing ordinary things, which
is how a control gets switched off. Both parse now, and a certificate's
commands are still judged — parsing one is not a bypass.
- A run's budget was unbounded, so a caller could hold one of its org's two pool
slots for a day and ask the forge to delegate a push for just as long. Capped,
and the grant's lifetime is now derived from it.
Proved with the real client against the real server, one test per door:
refwriters_wire_test.go drives the real git CLI over the real SSH listener and
over smart-HTTP, and refpolicy_framing_test.go replays wire bytes captured from
git 2.43. A grant clones its repository and pushes its one ref, and is refused —
with git's own report-status, in words the pusher can read — for the trunk, for
another run's branch, for a non-agent ref, for deleting the trunk, and for
appending to its own branch. The legitimate paths are asserted too: an agent
branch still creates, the builder's client-less push to main still lands, and an
ordinary SSH push to main still lands.
The sandbox remains INERT: BOT_GATEWAY_URL and HANZO_CODING_SANDBOX_IMAGE stay
unset until Red has re-reviewed this.
Every @hanzo turn in Slack answered "the agent hit an error handling that". With
the bridge's new non-silent branch the reason finally surfaced:
zip: call agents_run_on_behalf at /var/lib/cloud/run/agents.sock:
zaphttp: read response: i/o timeout
zap-proto/http@v0.3.1 sets readTimeout: 30 * time.Second (client.go:72) on every
dialled transport. An agent turn runs a real model completion and enso spends
40s+ on one — production measured 41,153ms, answered 200 by BOTH `ai` and
`agents`. The work succeeded. The caller had already hung up, and the reply was
written to a socket nobody was reading.
That is the worst shape a timeout can have: the expensive work is done and
billed, the callee logs success, and only the caller reports failure — so every
log you would naturally check says the system is healthy. It is why this
survived a day of looking at a healthy `ai`.
The context budget could not save it. The bridge bounds a turn at 110s on a
detached context; that governs the CALL and never reaches the transport's own
SetReadDeadline, which is wall-clock on the connection and shorter. Two
deadlines for one operation, and the smaller one — the one nobody chose — won.
The fix is a registration, not a patch. zip resolves a scheme through
RegisterTransport (transport.go:109) and its default zap Dial is one line, so
re-register the same scheme with the same dialler plus the knob zap-proto/http
exports for exactly this (SetReadTimeout, client.go:81). The stock Serve is
restated verbatim because re-registering replaces BOTH halves — omitting it
would stop every plugin listening.
15 minutes, matching the host's own plugin-start budget, so there is ONE answer
to "how long may an in-flight plane call take". It is a CEILING, never a floor:
every caller still bounds itself, and a caller that gives itself ten seconds
still gets ten. What changes is that it can no longer be cut off BELOW its own
budget by a default it never saw.
Tests pin the property rather than the wall clock (a unit test cannot hold a
socket for 40s): the ceiling must exceed the longest real caller budget,
re-registration must be idempotent, and networkOf must match zip's rule — it is
copied because zip keeps it unexported, and a drifted copy would hand a unix
path to a tcp dialler.
The two private lines had diverged 191/18 off a merge-base six hours old.
Production builds from inc2; every push went to forge; neither knew.
The 14 conflicting hunks, and why each resolved the way it did:
- bridgeReply loses its ctx parameter (inc2). Both lines fixed the same
production bug — the webhook's ctx made zip drop the stated org, so the
balance gate answered "no org on the call". forge passed a detached ctx in;
inc2 deleted the parameter so the webhook's cannot be passed at all, and
states the tenant with cloud.For. Unrepresentable beats discouraged. Its
body now calls forge's bridgeIdentity, so the link lookup exists once.
- commerce prefixes are the union: forge's /v1/cart and the /v1/billing/topup
stem, inc2's accounts, credit-balance and transactions. topup/token is
dropped — the stem owns its subtree. A prefix missing here never reaches
commerce; it falls to ai's bare /v1 and answers ai's 404.
- zipdoc-check keeps inc2's target (forge repeated the loop twice) with
forge's body: per-package, because whole-module load extracts differently
than the generator it polices, and skipping dot-dirs, because an agent
worktree is a second checkout and the walk read 203 packages where there
are 104.
- plane.go keeps both new sections — sandbox ops and the coding-run seams are
orthogonal, no shared names.
- plane_debit_harness_test.go takes forge's, which delegates to
internal/planetest instead of holding a second copy. Same API. The copy is
what hid a bind failure: a unix path is 108 bytes and t.TempDir() spells
the test's name into it.
- metering keeps Actor and the token counts; plane.Usage has the fields, and
a debit without them can be re-read but not re-derived.
- o11y v1.5.62 over v1.5.61 — upgrade, never downgrade.
go build ./... and go vet clean. Full suite: 3 red (base, code, commerce),
identical to both parents — this box's SQLite lacks acos and fts5.
Red returned do-not-ship on two CRITICALs that chain to unauthenticated
cross-tenant code execution. Both are proven by apps/exec/auth_test.go, and all
four assertions FAIL when the old behaviour is put back — mutation-checked, not
asserted.
1. THE TENANT CAME OFF A HEADER. callCtx preferred cloud.Who(ctx).Org, which is
zip.CallerOf, which reads the X-Org-Id REQUEST HEADER (zip caller.go:377) —
and for a request with no validated bearer, SanitizeIdentity deliberately
RESTORES the client's own header (middleware_identity.go:455). So
`X-Org-Id: victim-corp` made storeFor open the victim's SQLite file: the run
executed in their store and /v1/files + /v1/download read their artifacts
back out. principal.OrgFrom is the org a VALIDATED principal resolved to and
nothing else; every other app resolves through it and this one did not.
plane.go's own note — "an org in the argument is an org the caller chose" —
is the rule it was breaking.
2. THE CREDENTIAL CHECK WAS A LOWERCASE PREFIX LIST OVER c.Path(). fiber routes
case-insensitively; cloud.RoutePath exists in this repo for exactly that and
two other gates already use it. POST /V1/EXEC matched the route and missed
the list: no key at all ran code, a wrong key read another session's bytes,
and CODE_EXEC_API_KEY UNSET — the documented fail-closed 503 — still ran code.
Same root, so one fix: authorization stops being inferred from the SPELLING of a
request. The middleware normalizes with RoutePath AND parks two facts on the
context — principal.WithOrg (inherited by the typed op, typed.go:82) and an
unexported `admitted` marker. tenantOf is the ONE tenant decision and refuses a
context carrying neither. The prefix list may still drift; drift is now
fail-CLOSED — a missed path is a 403 on a route that should have worked, never a
route that works without a credential.
AND THE DOOR NO LIST COULD HAVE COVERED. A typed op is also an MCP tool and an
op-plane op, and tools/call invokes it DIRECTLY (zip typed.go:474) — no route, no
middleware. `POST /mcp name=post_v1_exec` with no key ran code. Typing /v1/exec
for its SDK value is what opened that door; the handler-side check closes it,
because those doors park no marker and carry no principal.
3. [HIGH] AN EXEC SANDBOX HAD NO CEILING. Single-attach bounds dev/desktop via
their project; an exec sandbox has none, and the code tool sends no
session_id, so every call mints a fresh pod on a 15-minute lease — 40 calls,
40 pods, each 250m/512Mi/2Gi. The reaper is the FLOOR, not the ceiling: it
ends leases that are over, bounding the steady state and never the burst.
maxLiveExec=16 is written in node capacity (8Gi, 4 cores) and refuses with
429, because the caller's correct response is to wait. Counted with a real
COUNT(*), not len(List) — List is LIMIT 200, which stops counting exactly
where refusing starts to matter.
4. [HIGH] TWO CLIENT-SHAPE BUGS, both silent. hanzo.chat primes an attachment as
{id, session_id, name} (Files/Code/process.js) while @hanzochat/agents spells
it storage_session_id (tools.d.ts) — reading only the second skipped every
attached file AND the "not available" note, so a user's CSV was invisible with
no error. CodeFile.Session() reads both, and a ref with neither is reported.
Artifacts were COLLECTED recursively (find) and LISTED top-level (ls -1A), so
a nested artifact appeared in the reply and was missing from /v1/files/{sid},
which the client reads as expired. One find answers both now.
Also: apps/functions.go's doc no longer cites the deleted CODE_EXEC_UPSTREAM.
Tests: 22 in apps/exec (14 + 5 auth + 3 client-shape), 6 in apps/sandbox. Full
suite 3 red — base, code, commerce — all pre-existing and environmental (this
box's SQLite lacks fts5 and acos). Zero new failures. make check green.
NOT FIXED, and it is a decision rather than a defect: the real chat client sends
no credential at all (EnvVar.CODE_API_KEY undefined, handleTools.js sends no auth
header, the chat pod has no LIBRECHAT_CODE_API_KEY), so the guard 401s every
legitimate request. Fixing the bypasses does not make the feature work; settling
what the client presents does, and the house rule says Hanzo IAM rather than a
second shared secret.
Every image built from main has failed, after a green gate, at one line:
#28 ERROR: process "/bin/sh -c CGO_ENABLED=0 go build -ldflags=\"$GO_LDFLAGS\"
-o /boxd ./cmd/boxd" did not complete successfully: exit code: 1
cmd/boxd was deleted with the design that needed it — a sandbox is a POD, and
commands reach it over the Kubernetes exec subresource, so there is no daemon
inside to talk to and nothing to ship into it. The Dockerfile stage that built it
outlived the source, and `go build` on a directory that is not there is not a
warning. So the lane could TEST a commit and could not BUILD one, which is the
whole reason a sandbox executor that is merged and proven has never been served:
no image has published since v1.801.476.
This deletes the stage and the COPY that carried its output. The consumer they
existed for — hanzo/bot's Dockerfile.box, `COPY --from=cloud:<pin> /boxd` — has
to move to a pod nothing needs installing into; leaving a stage that cannot
compile does not keep that consumer working, it only stops everything else from
shipping.
Verified against this tree, with the Dockerfile's own tags: ./cmd/cloud,
./plugin/smoke and ./plugin/sandboxes all build, all 121 manifest apps have a
plugin/<app> directory (the loop hard-fails otherwise), and no non-comment line
mentions boxd.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Start required a non-empty org and nothing more. The org is then interpolated
into the KMS reference for the agent credential (credRef) and into the git
namespace, so an org carrying a separator or a `..` segment would address
another tenant's secret.
It arrives already validated from the gateway or the plane. This is the second
lock, placed on the side that would actually be harmed if the first ever failed
— the same rule the git subsystem states for itself (git.go orgRE), spelled
where the concatenation happens.
apps/exec was a reverse proxy to code-exec.hanzo.svc.cluster.local:8000, and
that Service has had ZERO endpoints for 33 days — /v1/exec answered 503 in
production the whole time. It was not failing to reach the executor; there was
no executor. apps/functions read the same CODE_EXEC_UPSTREAM and was failing
against the same absence.
cloud already runs the one compute primitive. A LibreChat session IS a sandbox
("a code-exec call = a sandbox with a session lease", apps/sandbox's package
doc), so session_id is the sandbox id, upload/download/list are Write/Read/Read
on that sandbox, and exec holds no store, no session table and no lifetime of
its own. The lease ends on the reaper.
THE BLOCKER, AND THE DECOMPLECTION. Every sandbox operation existed ONLY as
`func X(s *Service, c *zip.Ctx) error` — the domain braided into the transport,
so nothing else in the process could use a sandbox. apps/sandbox/api.go is the
domain as a VALUE: Lease, Get, List, Run, Read, Write, End. The HTTP handlers
become adapters (bind, call, JSON) and apps/sandbox/plane.go is a SECOND adapter
over the same functions. Errors are zip errors in the core, so "not yours" is
404 once, decided where the fact is known.
(Functions taking *Service, not methods: `Service = cloud.Service[state]` is an
alias for a generic type in package cloud, and Go cannot define a method on a
non-local type. apps/git's files.go is the same shape for the same reason.)
cloud.Ask AND NOT A GO IMPORT, and that is correctness, not style. Apps ship as
separate plugin binaries (plugin/<app>/main.go "links only its own subsystem"),
so importing apps/sandbox would give exec a SECOND sandbox service — its own
OrgStore on the same per-org SQLite files, its own reaper racing the real one.
plane.Ask already collapses to an in-process dispatch where the fleet fuses two
apps (zip.Serving/zip.Here), so the ONE call is a function call when they are
co-resident and a socket hop when they are not.
THE CONTRACT IS MEASURED, from ~/work/hanzo/chat, not remembered. Download is
TWO segments — /download/{session_id}/{fileId} (crud.js) — not /download/{id}.
Upload answers {message:"success", session_id, files:[{fileId, filename}]} and
the client throws unless `message` is that literal. /files/{sid} answers a BARE
JSON ARRAY of {name, lastModified} whose name is that same two-segment
identifier. And the code tool tells the model to persist artifacts in /mnt/data,
so sandbox gains workdirFor(class): /mnt/data for exec, /work for dev. Listing
/work would have reported no files after every successful run.
/v1/exec/programmatic answers 501 rather than being routed into the interpreter.
It is a DIFFERENT protocol — a run suspended on each tool call and resumed from
a continuation token — and answering it with a Result its parser cannot read is
a wrong answer where a refusal is an actionable one.
TENANCY, and the trap. cloud.For states a caller only on a context with NO
request behind it: zip.CallerOf reads the request's headers first, deliberately,
so no handler can assert an org a caller did not arrive with. A typed op's ctx
HAS a request behind it, so cloud.For(typedCtx, org) silently states nothing and
the peer answers 403. exec.callCtx is the one rule — pass a ctx that already
carries a principal through unchanged (hanzo.chat forwards the user's IAM
bearer), otherwise detach to Background, state the deployment's brand org, and
put the request's cancellation back with context.AfterFunc.
apps/functions moves to the same door, because task 3 (deleting the orphan
Service) would otherwise turn its 502 into a DNS failure. It calls exec.Run —
one operation, one home — and ENDS the lease, because a function invoke is over
when it answers.
THE FLOOR IS LOWERED HERE, next to its reason. exec published 40 operations
because a proxy cannot describe a contract it does not own; it now implements
one, so it publishes 5 — POST /v1/exec is a TYPED op (CodeRun -> CodeResult),
which is the schema, the MCP tool, the CLI command and the SDK method a proxy
could never carry. The four that stay untyped have real reasons: multipart in,
bytes out, a bare array, and a protocol we do not serve. Numbers taken from the
ratchet's own report, not computed:
paths 1762 -> 1759 (-3)
operations 2465 -> 2430 (-35)
download 10 -> 1 · exec 10 -> 2 · files 10 -> 1 · upload 10 -> 1
Deleted: newProxy, defaultUpstream, CODE_EXEC_UPSTREAM, and the ledger of 40
untyped operations. CODE_EXEC_API_KEY stays — it is the credential the chat
presents, and the guard is now middleware because a typed op takes no handler
chain.
Tests: apps/exec 14 (a sandbox peer double on the REAL plane, so an op renamed
or a field moved fails here and not in production), apps/functions green
including TestInvoke_AllowsAndDebitsCallerOrg, which was RED before this change
— metering moved debits onto the plane and the test still had only an HTTP
double.
Measured against HEAD: 12 packages fail before and after; zero new; one fixed.
LLM.md also carries a concurrent agent's insights.hanzo.ai section, present in
the tree and committed rather than dropped.
A coding run could not complete, for four independent reasons, any one of which
was fatal:
1. The run was spawned on a bare context.Background(). Every seam it touches —
session, clone URL, ref verify, PR row, and the balance gate behind them —
authorizes on the CALLER's org and never on an argument, so each answered
`authorize: no org on the call`. The routed target lookup was worse: it ran
on the webhook's REQUEST context, where cloud.For is a silent no-op. This is
the same shape that killed every chat turn until bridgeRunContext fixed it,
and start.go's runContext is the other half of that fix.
2. There was no /v1/coding. The app had no way to start a run at all.
3. The chat surface assembled its own Dispatcher, so Slack and the app would
have been two engines with two pools and no shared address for a run.
4. (unchanged here, stated in the handoff) the sandbox hop and its image are
deployment configuration, and both currently refuse.
The engine now lives in one process — agents, which already holds the session
store, the durable engine and the routed mailbox — and the START travels instead,
exactly as AgentsRunOnBehalf made one brain reachable from every chat platform.
POST /v1/coding and the coding_start plane op are two doors onto the same
coding.Start: same pool, same validation, same credential custody, same detached
tenant-stated context. The chat adapter is an adapter again — parse, authorize,
dispatch, reply — and no longer holds the org's git credential at all, which the
engine now reads from KMS at the one moment it dispatches.
The forge gets the rule that holds when the rest has failed. A run executes
untrusted model output against a real checkout, and the model also READS the
repo, so a README can carry an instruction. Assume the run is hostile and holds
the credential: refs/heads/agent/* is CREATE-ONLY, and the default branch cannot
be deleted by a push. That refuses the bait-and-switch — open a clean PR, let a
human read it, force-push the payload before the merge — structurally, at the one
point every push must pass through, rather than by asking reviewers to be
vigilant. It refuses with git's own report-status, not an HTTP 403, because a
control that can only say "RPC failed" is a control someone switches off.
Proven with the real git CLI against the real server: an agent branch pushes and
re-clones; --force over it, deleting it, and deleting main all come back
`! [remote rejected]` with the reason; feature-branch force-push and trunk
updates are untouched.
The run id was minted at the end of executeRun, beside the row it fills in.
That reads naturally and made the run unobservable: every span the run
produced — the step, each tool dispatch, each model call — had already ended
and exported by the time the run had a name, so none of them could carry it,
and neither could the per-token debits the metering decorator makes one round
at a time. The id existed only on the record of a thing that was already over.
Mint it before the work instead. One value is now on the span, on the row and
on the money, which is the whole of "drill into this run":
- every span (agent.run / agent.step / agent.tool / the gen_ai client span)
carries hanzo.agent.run_id, so attribution never depends on walking a
parent chain that sampling or a truncated batch may have broken;
- the run row carries the trace id, which is the key the run history and the
span store had no version of — two accounts of one event that could not be
joined in either direction;
- types.ChatRequest.RunID rides to metering.Usage.RequestID, the field whose
stated job is exactly this, so a run's per-token cost is a SUM over ledger
rows rather than unanswerable. It is the correlation id and NOT Ref: a tool
loop settles once per round, and pinning the idempotency key to the run
would dedup every round after the first into the first one's debit.
WHO, not just which tenant. org answered "whose ledger" and nothing answered
"which person" — an actor reached the debit and the session row but was never
written to the run, so a scheduled or on-behalf run had no answer at all.
Runs now record it and the spans carry hanzo.user.
A tool dispatch is readable as a dispatch: the span names the tool, its call
id, the owning subsystem, the round it happened in, and its outcome — set on
every exit, including the happy one, because a status written only on failure
cannot tell "succeeded" from "never finished". So "it called six tools and
failed on the fourth" is a fact you can read instead of infer.
toolSubsystem derives the owner from the operation name rather than looking it
up: the fleet door spells ops <method>_v1_<subsystem>_<rest>, so it is a fact
the value already states, and it answers the same in the fused binary and in a
single-app plugin process — where cloud.SubsystemOf reads a boot-time mount
index that knows only that plugin's own routes and would answer "" for every
sibling's tool.
GET /v1/agents/runs is the org-wide feed the per-agent history could not be:
an operator asking what a tenant's agent plane is doing does not start out
knowing an agent ref, and listing the agents to page each one's history is N+1
round trips to rebuild an ordering the org index already has. The org comes
from the caller's identity; there is deliberately no org field to forge.
scanRun is one function because runCols was named once to stop the projection
drifting and both readers were spelling the column order out by hand anyway.
The tests run a real turn — real handler, real tool loop, real OpenAI-wire
client — and assert on spans that actually reached an exporter, because a span
that is created and never exported is the failure this is about. The provider
is installed once per process and only the sink swaps: OTel's global delegates
ONCE, so a tracer handle taken at package init binds to the first provider and
keeps it, and a test that installed its own and shut it down on cleanup left
every later span-asserting test in the binary seeing nothing.
The usage debit moved off HTTP onto the internal plane (ac09a86d), for a good
reason: metering.Usage.Ref is `json:"-"` so no request body can set the ledger's
idempotency key, and json.Marshal therefore dropped it — every split-deploy debit
reached the ledger anonymous and a sealed retry was charged twice.
Nine packages never moved with it. Each had hand-rolled a usage counter against
POST /v1/billing/usage, so each went on counting an endpoint nothing calls: 20
tests across apps/{agents,cloudflare,content,functions,ml,projects,provisioning,
risk,storage} reported "0 debits" for work that had in fact been billed. That is
the release lane's break, and it predates the sandbox work — bisected to
ac09a86d, which is a metering commit, not a sandbox one. Before it, the same
tests pass.
THE CROSSING WAS DROPPING FIELDS OF ITS OWN. plane.Usage had no Actor and no
token counts, and metering.Usage carries both — so the crossing lost WHO acted
and WHAT WORK the amount priced, and the receiver rebuilt a ledger row without
them. Exactly the failure the Ref fix was written for, one field over. That is
not a test problem and is fixed on the wire: plane.Usage gains Actor,
PromptTokens, CompletionTokens and TotalTokens, the sender sends them, and
commerce's record handler writes them. apps/agents already asserted an actor is
present; it now passes because the actor arrives, not because the assertion left.
ONE PEER, NOT TEN. internal/planetest is commerce's half of the money plane on a
real socket, following internal/iamtest: one fixture for one wire, because the
copies are what drifted. It also ends a bug all the copies shared — they bound
the socket under t.TempDir(), which spells the TEST'S NAME into the path, and a
unix address is a fixed 108-byte field. TestMeterUsage_OneActIsChargedOnceIn-
EitherTopology/split_deploy produced a 106-character address; the bind failed
inside a goroutine where nothing read the error, and the symptom was a debit
that never arrived. Whether it failed at all depended on how long $TMPDIR was,
which is to say on whose machine it ran. The shared peer uses a short directory
and refuses an address it cannot bind instead of failing silently.
Verified on the gates CI runs, with its tags and skips:
go vet -tags "sqlite_fts5 sqlite_math_functions" ./... clean
make -f mk/fleet.mk check openapi.yaml unchanged — 1762 paths
go test -tags "..." ./... the 9 packages green, root green
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Wire the embedded sentry runtime to the ONE binary-wide key resolver: a pk- key
posted to /v1/sentry resolves to its org through the SAME seam /v1/event uses
(sharedKeys), and the DSN's project is auto-provisioned under that org on first
ingest. So sentry.hanzo.ai accepts the exact key that already feeds analytics +
insights — no per-project DSN secret, dark no more.
- auth_apikey.go: export ResolvePublishableKeyOrg (the pk- -> org seam for
sibling packages that resolve a key out-of-band of the Authorization header).
- apps/o11y/embed.go: implsentry.SetIngestKeyResolver at the o11y mount.
- apps/o11y/o11y.go: migrate the removed o11y.SetHandler(gh) -> SetRuntime(Whole(gh))
(o11y v1.5.61's mount API), the drop-in "one door" replacement.
- go.mod: github.com/hanzoai/o11y v1.5.58 -> v1.5.61 (carries the sentry change).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every part of this already existed and nothing called it. ExpiresAt was written
on every create, LastUsedAt was stamped on every call, Store.Expired was
implemented, stop was implemented — and a sandbox, once created, ran forever.
Measured rather than inferred: a proof pod from a finished test was still
Running 64 minutes later and had to be deleted by hand.
That is one bug wearing two faces. A sandbox that never sleeps is a sandbox that
bills forever, so this is simultaneously the idle-sleep feature and the honest
meter. They are not separate components and building them separately would give
the fleet two clocks that disagree.
TWO TRIGGERS, ONE ACTION:
ExpiresAt the LEASE — the caller said how long they wanted it.
LastUsedAt ATTENTION — a sandbox nobody has touched for an hour is abandoned
even with hours of lease left, and holding a pod for a session
someone walked away from is the entire cost this exists to stop.
Both end the pod and both KEEP THE VOLUME, because purge is a separate opt-in
and ending a lease must never destroy what the tenant made. So the checkout and
the caches survive and the next call for that project gets a fresh pod against
the same disk — "resume is cheap" with no suspended state to define and no
resume route to serve.
A `status: suspended` was the obvious other design and it is worse: Status is
pending|running|error, nothing resumes, and a row in a fourth state that no
route can leave is a dead row wearing a live sandbox's clothes.
Idle is computed here rather than queried, because "untouched for an hour" is a
POLICY and the store holds facts. The constant that defines it lives beside the
code that applies it.
Started from Mount, not from a route and not from a caller. A reaper that
something has to remember to call is the bug it was written to fix.
An agent that can edit but cannot commit has done nothing durable, so the second
half needed its own proof. Against hanzo-k8s, 62s, stock node:22:
GIT PRESENT: git version 2.39.5
COMMIT MADE: 7fb283f the agent committed this
FORGE REACHABLE: exit=0
git needs no routes of its own — it is a command, and exec runs commands. The
predecessor had a /git/clone and a /git/push endpoint and a credential file
written to /tmp at 0600, which the code it was sandboxing could poll for during
the push window. None of that survives, because none of it was git.
Identity is set locally rather than globally: a sandbox is per-tenant, and a
global identity is one more thing to reset between leases.
The ls-remote is deliberately NOT a push. It exits 0, which proves the network
reaches the forge — and proves nothing about a credential. Those are different
gaps and conflating them sends someone to fix the wrong one. Pushing as a tenant
still needs a per-org token the sandbox does not yet have.
Stage 3 calls itself the load-bearing check and it could not pass. It read
o11y_traces.o11y_index_v3 and o11y_logs.logs_v2; both databases are gone, so
the query errored, `mins` came back empty, and the script printed "stale --
nothing is draining into the store" and TELEMETRY CHAIN: BROKEN on every run.
A check that always fails reports nothing. Worse, it spends the alarm it exists
to raise: the one run where telemetry really has stopped looks exactly like the
hundred before it.
Traces and logs are event.span and event.log now, and both date themselves with
the same `time` column, so the per-probe expression -- and the special case that
overrode it for traces -- has nothing left to vary and is gone. Measured while
writing this: both signals 0 minutes stale, 3.3M spans and 131.4M logs.
Every @hanzo turn answered "the agent hit an error handling that". The tenant
fix landed and was correct — the run reached agents with org=hanzo and passed
the balance gate — and then the model call died one hop lower:
Post "http://ai/v1/chat/completions": EOF
deps.AI dialed zip.SocketPath("ai") and spoke ordinary HTTP to it. That address
is wrong twice, and each half is independently fatal.
The WIRE is not HTTP. The socket is served by zaphttp.Server — ZAP, a framed
binary protocol. A cleartext request is not slow there, it is unintelligible:
the peer reads a malformed frame and closes. Measured on a healthy pod, every
request over that socket EOF'd — GET /v1/health, GET /v1/models, a bogus path,
POST /v1/chat/completions — and agents.sock and commerce.sock did the same, so
this was never about `ai`.
The SURFACE is not the app's. What binds there is the app's PLANE, the typed-op
door plane.Ask uses; the app's own routes are on a listener it knows nothing
about. Framed correctly as ZAP, ai.sock still answers 404 for /v1/models while
ai's own listener answers 200.
So `ai` never logged the request, because the request never arrived, and the
run came back error-status with a nil error — which is the branch that logged
nothing at all. Hence a day of looking at a healthy `ai`.
The address that does serve it is the fleet ROUTER's own listener: it owns the
route table that sends /v1/* to `ai`, and it owns starting a cold app. Entered
on 127.0.0.1 it never leaves the pod — no DNS, no Service hop, and no trip out
through Cloudflare and back to the pod's own public address, which is what the
configured base URL does. Only the port is read from CLOUD_LISTEN; the host half
a process binds is not an address a client may dial. Measured there:
/v1/models 200, bogus 404, /v1/chat/completions 401 for the Bearer the M2M
client already mints.
There is deliberately no second mechanism left. A raw route reached
process-to-process is not something this fleet offers; ops are. The custom
transport is deleted rather than repaired.
The non-"ok" branch in the bridge now says so, with the run id. It is the branch
a broken inference path lands in and it was the silent one.
Every metric the console reads through PromQL failed on the live pod with
"Database o11y_metrics does not exist" -- 96 times in 30 minutes, and each one
renders as an empty chart rather than an error, so service health, HTTP traffic,
ingest, memory, plane throughput and alert delivery all read as silence. The
data was never missing: event.metric holds 145.7M rows and was 3 seconds behind
real time throughout. Only the address was wrong.
o11y v1.5.62 points its PromQL client at event.series/event.metric via the
telemetrymetrics constants, which is where cloud's own gauge reader
(apps/o11y/metricsgauge.go) has been reading all along.
The bump also crosses o11y's runtime seam: SetHandler(http.Handler) became
SetRuntime(Runtime), which resolves a handler PER ADDRESS so a declared op that
the runtime does not serve is a nil rather than a 404 indistinguishable from a
typo. Both of cloud's installs are o11y.Whole -- the embedded runtime is one
router that matches the request's own path, and the proxy fallback has one door
with the far side selecting the route. Neither has anything to resolve per
address, and Whole is that shape stated honestly.
I moved this to hanzo-sandboxes on the reasoning that everything running
submitted code belongs in the one namespace whose policy denies it the cluster.
That reasoning is right and the change was wrong: there is no code-exec Service
in hanzo-sandboxes, so it renamed a 502 into a DNS failure. A peer caught it and
said so plainly — target ns is EMPTY, do not ship.
Moving the NAME before moving the SERVICE is the same mistake as a policy whose
selector matches no pod, and this file would have been the fourth this week.
The Service moves first; then the address follows it.
Neither namespace makes this path work today — the Service in hanzo has had no
endpoints for 33 days. The fix is not a better address, it is /v1/sandboxes,
where a caller gets a pod rather than a proxy hop to a workload nobody deployed.
sha-1fa6f964cb4d mounted GET /v1/billing/{transactions,credit-balance,
accounts} and shipped them — the route strings are provably in that
image's commerce binary — and all three still answered 404 in production.
An address has two halves. mount.go states what commerce will answer;
manifest.Apps states what the host may hand it, and the commerce row is
an explicit leaf allowlist, not a /v1/billing subtree ("Nobody claims the
bare /v1/billing REMAINDER"). A leaf absent there is never delivered: it
falls to the "/v1" remainder on ai's row and answers ai's bare 404. From
outside that is byte-identical to a route that was never mounted, which
is why the first fix verified correct at every layer anyone thought to
check — source, image bytes, deployed digest, running version — and
changed nothing a caller saw.
credit-balance is stated separately from credits on purpose: they are
sibling prefixes, so matching one does not match the other, and next to
an existing "credits" entry the missing one reads as already covered.
accounts covers its /:id/members child.
The test asserts the ROUTER's half specifically, because nothing else
can. apps/commerce's route tests cannot see this table, and a byte-level
check on the built image cannot either — the string was in the binary the
entire time it was returning 404.
The merge collapsed two manifest rows into one and left a name that matched
nothing. The app was called `sandbox`, it served `/v1/sandboxes`, and its
operations were tagged `sandboxes` because the tag comes from the path — one
thing under three spellings, and the Makefile named a fourth.
It is `sandboxes` now, which is what its path says and what every sibling
collection already does: bots, agents, tasks, books. The package stays
apps/sandbox and pkgOf carries that, which is the case pkgOf exists for.
The floor's `sandbox: 26` was the BOX POOL — a warm pool of pods, a daemon
inside each one, a shared pool-wide key, and a bind handshake so a box could
tell which box it was. All of it is deleted, and the row goes with it rather
than being pinned at 0: the product does not exist, and a floor entry for a
product that does not exist is a claim that something is missing.
That deletion is right and it is not mine to mourn. runtime.go says why in its
own words — the boundary is the pod's runtimeClassName, one field, and
"everything the predecessor built to approximate that boundary in Go is deleted
rather than kept for defence in depth, because a second half-boundary is a
second thing to keep true." Every problem the pool had, I spent tonight fixing:
the key that opened every box, the row whose address the CNI had reassigned, the
identity a warm pod could not learn. None of them exist without the daemon.
There is no key, and a pod is addressed by NAME through the apiserver, which
cannot go stale.
Four operations published an operationId and nothing else. These routes bind
through zip.Ctx rather than being typed ops, so zipdoc has no comment to lift
and openapi.Describe is the seam — the other three already used it. They say
what they do now, including the two facts a caller cannot guess: a 404 rather
than a 403 for another org's id, because a 403 confirms the id exists; and that
the volume SURVIVES a delete unless purge=1, which is the one part that cannot
be undone.
1758 paths, and the weave is green.
apps/lsp ran gopls, tsserver and the rest INSIDE a fleet pod: server.go,
workspace.go and langs.go checked repositories out, fetched their dependencies
and executed a third-party toolchain over untrusted bytes in the same process
that holds a principal, a ledger and a KMS-injected environment. That is the
wrong shape however carefully it is written, and hanzoai/lsp exists to be the
right one — a jailed daemon with gVisor, no egress but a module proxy, and no
git credential of any kind.
So this side becomes a PROXY over that daemon, and keeps the three things the
daemon must never hold:
TENANT the org is the validated principal's, never a body field, and it
is the daemon's isolation key.
REPOSITORY the revision and the tree come from git's own object plane, for
the caller's own org. The daemon cannot fetch a repository — a
credential that could reach every repository is exactly what must
not sit next to an unjailed compiler — so a cold revision is
/ask → 409 {"need":"tree"} → /root → ask again, ONCE.
LEDGER the gate runs at the prepare price before any work, the debit
after the answer. "prepare" and "query" are the two Models, and
spend.go's meteredApps entry still holds.
The surface moves from /v1/lsp to /v1/code/lsp, and from one door with a
`method` field to five typed ops — hover, locate, symbols, diagnostics,
complete. code and lsp are two reads of ONE repository, not two products: code
is the static index, lsp the live server that follows a symbol out of the
repository and into a dependency. One home means one place to look for it, in
the document and in the MCP tool list alike, and an agent picks a tool by its
name rather than by a union behind one. Nested static prefixes resolve by
specificity, so /v1/code/lsp beats /v1/code and both beat ai's bare /v1 — the
same relation storage's /v1/s3/buckets already has to provisioning's /v1/s3.
git gains ONE op, git_rev: the commit a ref names. It is separate from git_files
because the two questions cost differently — resolving is a ref lookup and runs
on every position query; reading is a walk of the whole tree and runs only when
the daemon says it holds no root. Folded together, a hover would drag a monorepo
across a socket. One resolve (coreRev) now backs both.
plane/agents was generated before agents_run_on_behalf existed; running the
generator carries it in.
The published surface GREW: 1762 → 1766 paths, 2480 → 2484 operations. The
floor's "lsp" product is gone because its five operations are tagged "code" now,
which is the move, and code rises 7 → 12.
The box vocabulary is gone. It named the same thing three ways — a box in the
Go, a machine in the types, a sandbox in the API — so a policy could select
hanzo.ai/box-class while the code set hanzo.ai/sandbox-class and both looked
right in isolation. That is not a style preference: it is how the containment
shipped selecting nothing.
apps/sandboxes -> apps/sandbox package sandbox, not sandboxes
Machine -> Sandbox the type is the noun the API serves
machines.go -> sandbox.go
MACHINE_* -> SANDBOX_*
/v1/sandboxes stays plural because a collection is plural. The PACKAGE is
singular because it is one thing.
DELETED, not deprecated: cmd/boxd (~2000 lines), the old apps/sandbox pool and
proxy, apps/sandbox/wire. The daemon existed to serve a filesystem and a shell
from inside the pod over HTTP; the Kubernetes exec subresource already does
that, so boxd was a second implementation of a thing the cluster ships. Deleting
it also deletes what it forced: a shared service key living in an environment
the submitted code could read out of /proc, a bind protocol so a recycled pod IP
could not serve another tenant's checkout, and an HTTP server that had to be
built into every sandbox image.
wire went with it, except for one string. It held the types boxd and the proxy
both marshalled, which is a real reason for a package — but the last survivor is
the /v1/exec path shared by apps/exec and apps/functions, and a package that
exists to hold one constant is not a package. It moves to exec.Path, in the app
that serves it. The reason it must have exactly one home is unchanged and worth
keeping: those two consumers disagreed about it in production, one asking for
/v1/exec and the other building upstream+"/exec", and because the executor did
not exist neither was ever wrong out loud.
Routes registered. Create, List, Get and Delete were exported functions no
request could reach — a caller could exec in a sandbox it had no way to create.
The comment said they were "shared with the compute surface", which was true
while this served /v1/machines (visor's, where a second registration is a
conflict) and false the moment it moved. Handlers present, routes absent,
nothing saying so: the same shape as the policy that selected no pod and the
installer that installed nothing.
Re-lands 2f9ffc2d, which 0d7011e2 reverted for want of evidence. The evidence
now exists, and it is a run whose gate had already passed.
Run 895 (41a0e5a3, event push, branch main):
gate success 09:07:35Z -> 09:46:45Z
cicd success 09:07:35Z -> 09:46:45Z
containment success 09:07:33Z -> 09:09:10Z
image cancelled never started -> 10:25:26Z
rollout cancelled never started -> 10:25:26Z
reach cancelled never started -> 10:25:26Z
fanout cancelled never started -> 10:25:26Z
receipt cancelled never started -> 10:25:26Z
The run's own conclusion is `cancelled`. On a push, `${{ github.event_name ==
'pull_request' }}` is supposed to be false and nothing should have cancelled
anything. The forge does not evaluate it to a boolean; a non-empty string is
truthy, so every push cancels the run before it.
The revert argued these runs "were superseded while still QUEUED, not killed
mid-gate, so cancel-in-progress was never the thing acting on them". That holds
for the runs it measured — 889 through 892 never received a runner — and cannot
hold for 895: a run that never started cannot contain three jobs that completed
successfully. Both failure modes are real. Only this one is a line in this file,
and it is the one that throws away work the gate already finished.
Runs 891-909, every one a push to main: 16 cancelled, 3 failure, 0 success. The
single run that earned a release lost it 39 minutes after going green, which is
the whole of "the gate is green and nothing ships" — and why a cutover that has
been on main all day is still 404 in production.
A literal false is read correctly and restores what the comment above already
described. It does not queue without bound: one run holds the group and the
rest collapse into a single pending run, so pushes during a release are
coalesced, not accumulated. The cost is that a stale PR run is no longer
cancelled by its own next push, because the setting can no longer ask what event
it is — cheap next to a release lane that cannot finish.
The other half of the diagnosis under the revert stands and is not addressed
here: ten runners of hanzo-build-linux-amd64 serve five orgs, and a run that
waits longer for a runner than the interval between pushes still starts late.
That is fleet sizing, and it belongs in universe, not in this file.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
MeterUsage documents a debit as exactly-once on the act's ref, and a surface that
already holds the act's server-assigned name sets it precisely so the work and its
charge are one thing under one name — apps/company mints a formation ref and hands
that same value to the debit.
The promise held only while the ledger was co-resident. In the shipped split
topology the debit leaves through meterPeer, which built its plane.Usage from
Project and Service alone: the ref never boarded. The receiver mints a fresh name
when an arrival carries none (apps/finance RecordUsage), so nothing could dedup and
a re-driven formation debit charged the customer twice for one company.
So the name crosses with the act, and the act is named ABOVE the topology branch.
Sealing below it was what let the two paths key the same act differently; sealing
above means neither can. Seal is idempotent, so a caller that already holds the act's
own name keeps it, and the co-resident path — where the metering client seals what it
is given — is unchanged by having been sealed one step earlier.
Two calls are still two acts: an unnamed usage is sealed with a fresh server-minted
name, so identical inferences bill twice and the fix cannot fold distinct work into
one charge.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The same act can hold BOTH usage keys at once — one entry under the empty program
and one under the wallet — whenever its ref was nameable by the client across the
change that scoped a usage ref to its wallet. An org that renewed the same domain
once before and once after writes `domain:renew:foo.com` into both program
namespaces.
Re-keying the legacy row onto the wallet then collides on
UNIQUE(kind, program, ref), and a plain UPDATE makes that abort the statement: the
migration fails, so Open fails, so storeFor fails, so the prepaid gate has no ledger
to read and fails closed — correctly — on EVERY paid request that org makes, for
good, on a store no retry can open. The repair runs at first open after a deploy, so
it was armed for every org not yet opened.
OR IGNORE skips the colliding row instead. That is the honest outcome rather than a
concession: the wallet row IS that act's entry, so the probe that asks whether the
act is already paid for finds it, and the legacy row's empty program can never fund a
second debit. It is also curative — a store that already aborted opens on the next
attempt, which is what un-bricks an org that has already hit this.
The repair also moves BELOW the DDL re-apply. On a legacy amount_cents file the
migration rebuilds treasury_postings without its indexes, and the repair reads each
candidate entry's postings twice as correlated subqueries — so running it first
scanned the whole postings table once per usage row, on the one open where the file
is largest and a customer is waiting on it.
And the kind is now the opener's to name. This is a generic journal; which of its
kinds are scoped by wallet rather than by the book is a fact only the app that writes
them holds, so finance passes its own KindUsage and the house book — whose accrual,
seed and payout kinds are all book-scoped — passes none. A literal here could be
renamed in finance and silently become a no-op that bills one act twice; the finance
tests now fail if the kind stops arriving.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Measured in-cluster on cloud's exact shape, same node pool, same buildkit,
fresh pod, run in both orders:
gzip / mode=max zstd / mode=min
exporting layers 194.6 245.5 256.1 s 39.2 58.1 53.3 s
cache export 212.1 s 2.0 s
image size 1,646,830,883 B 1,626,409,300 B
Derived on the real build: -219s cache export, -136s layer export = ~354s off
17m09s. Both scale WITH the image, so a smaller image compounds them.
WHY mode=max was wrong. The build stage holds the same 4.2 GB the final image
does, so mode=max gzips it a SECOND time and pushes a second 1.6 GB blob. What
it bought: two independent builds imported the cache and got exactly the same 8
CACHED steps — apk add / adduser / symlink / WORKDIR in each stage. Every
expensive step missed both times, and must: COPY . . sits in front of them, and
the compiles write into --mount=type=cache, which is worker-local and never
travels in an exported cache. The four build-stage records mode=min gives up
cost 4.6s cold. It was paying 221s per build to save 4.6s on the next one.
WHY 175s for one export. The plugin layer is 1,627,346,308 of 1,646,830,883
bytes — 98.8% of the image — and 4.15 GB at 175s is 23.7 MB/s: single-core
gzip, with seven of eight CPUs idle.
Three flags are load-bearing, not decoration:
- oci-mediatypes=true is REQUIRED; Docker schema2 has no media type that can
name a zstd layer.
- force-compression=true is REQUIRED. Without it buildkit publishes an OCI
manifest whose layers are still tar+gzip, reusing blobs it already has —
silent, and in the "still works, just slow" direction.
- the cache's compression must MATCH the image's, or mode=min re-compresses
what the image export just wrote and hands the saving back.
AND THE ONE THAT WOULD HAVE BITTEN. imagePullable in pin.go negotiates the
manifest by Accept, and its list omitted application/vnd.oci.image.manifest.v1
+json. Measured against ghcr with a real zstd image, with a bogus-tag control:
zstd image, old header -> 404 zstd image, new header -> 200
gzip image, old header -> 200 nonexistent tag, both -> 404
ghcr honours Accept strictly, so shipping zstd without this makes EVERY release
pin fail, insisting the registry does not have an image that is sitting there.
That list was coupled to the builder's compression setting without saying so;
the coupling is now written down.
SEQUENCING: this pin.go fix must be DEPLOYED before the build flags flip.
imagePullable runs in the deployed cloud binary and gates both v* and sha- pins.
NO LAYER SPLITTING. Every plugin embeds the per-build version string
(-X ...Version=${VERSION}, confirmed in 5/5 shipped binaries), so all 119 change
bytes on every commit. There is no stable subset to split off, and splitting
would add manifest entries while saving zero export work. Revisit only if the
version stamp moves out of the per-plugin binaries.
NOT VERIFIED: no full real cloud build with these flags yet — the numbers are a
same-shape synthetic whose gzip baseline reproduced production within 11%. zstd
is proven INSIDE this cluster (ghcr stores it; containerd 1.7.28 pulled and RAN
it on a node that had never seen the bytes). Anything pulling from outside — an
old Docker, an external mirror — is untested.
Four apps kept their own copy of the projection's method set and asserted it.
Dropping TRACE and OPTIONS from the document therefore surfaced as eight
failures that looked unrelated to it — bots and dns and o11y naming routes
"which this subsystem no longer serves", exec insisting on 56 operations where
40 are published.
Nothing about the WIRE changed. All() still binds every method and the executor
still answers them; exec's own subtest proves exactly that and is why it stayed.
What changed is what we OFFER, and the ledgers were describing the offer from a
second copy of the truth.
exec reads openapi.Methods() now, the same fix websearch took an hour ago and
for the same reason: one set, one place, both halves move together. The others
had literal entries, which are simply gone.
The counts move with it and say why in the prose beside them, so the next reader
gets the reason rather than a number that used to be right:
bots 10 -> 8 six product methods on the relay, plus two typed ops
exec 56 -> 40 8 paths x 5 published methods
SEPARATELY, and not caused by any of that: TestJWKSHasOneDerivation walks the
tree from ".", and git puts WORKTREES inside the working tree. It failed naming
.claude/worktrees/agent-…/apps/base/pool.go — a file from another commit of this
same repo, which this commit is not responsible for and cannot fix. It skips
.claude and .worktrees now. The gate could never fire in CI, which checks out
clean, so it only ever cost whoever had a worktree open — which, here, is
everyone running agents.
Verified against d2c44bab rather than asserted: the metering timeouts and the
frozen-app-order failure are pre-existing, and three of them could not even RUN
at that baseline because the duplicate Describe panicked their package at init.
Fixing that panic did not break them. It made them visible.
MeterUsage documents a debit as exactly-once on the act's ref, and a surface that
already holds the act's server-assigned name sets it precisely so the work and its
charge are one thing under one name — apps/company mints a formation ref and hands
that same value to the debit.
The promise held only while the ledger was co-resident. In the shipped split
topology the debit leaves through meterPeer, which built its plane.Usage from
Project and Service alone: the ref never boarded. The receiver mints a fresh name
when an arrival carries none (apps/finance RecordUsage), so nothing could dedup and
a re-driven formation debit charged the customer twice for one company.
So the name crosses with the act, and the act is named ABOVE the topology branch.
Sealing below it was what let the two paths key the same act differently; sealing
above means neither can. Seal is idempotent, so a caller that already holds the act's
own name keeps it, and the co-resident path — where the metering client seals what it
is given — is unchanged by having been sealed one step earlier.
Two calls are still two acts: an unnamed usage is sealed with a fresh server-minted
name, so identical inferences bill twice and the fix cannot fold distinct work into
one charge.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The same act can hold BOTH usage keys at once — one entry under the empty program
and one under the wallet — whenever its ref was nameable by the client across the
change that scoped a usage ref to its wallet. An org that renewed the same domain
once before and once after writes `domain:renew:foo.com` into both program
namespaces.
Re-keying the legacy row onto the wallet then collides on
UNIQUE(kind, program, ref), and a plain UPDATE makes that abort the statement: the
migration fails, so Open fails, so storeFor fails, so the prepaid gate has no ledger
to read and fails closed — correctly — on EVERY paid request that org makes, for
good, on a store no retry can open. The repair runs at first open after a deploy, so
it was armed for every org not yet opened.
OR IGNORE skips the colliding row instead. That is the honest outcome rather than a
concession: the wallet row IS that act's entry, so the probe that asks whether the
act is already paid for finds it, and the legacy row's empty program can never fund a
second debit. It is also curative — a store that already aborted opens on the next
attempt, which is what un-bricks an org that has already hit this.
The repair also moves BELOW the DDL re-apply. On a legacy amount_cents file the
migration rebuilds treasury_postings without its indexes, and the repair reads each
candidate entry's postings twice as correlated subqueries — so running it first
scanned the whole postings table once per usage row, on the one open where the file
is largest and a customer is waiting on it.
And the kind is now the opener's to name. This is a generic journal; which of its
kinds are scoped by wallet rather than by the book is a fact only the app that writes
them holds, so finance passes its own KindUsage and the house book — whose accrual,
seed and payout kinds are all book-scoped — passes none. A literal here could be
renamed in finance and silently become a no-op that bills one act twice; the finance
tests now fail if the kind stops arriving.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/zen was linked, copied into the image and pulled on every deploy to be executed
never. cmd/cloud's mount() returns at `if a.Coresident` BEFORE it can resolve a
path or spawn a child; zen's behaviour ships inside /ai, which links apps/zen and
mounts the Claim ahead of ai's catch-all.
The build list is still derived from manifest/apps.go: `names` (119) still guards
the plugin/<app> bijection, `spawned` (118) earns a binary. Flip
Coresident:false and the binary returns next build.
Checked rather than assumed: the other a.Plugin() call site (locate()) reads only
.Addr; with /zen absent resolve() falls to pluginIn() and fetch() returns nil,nil
because CLOUD_PLUGINS is unset — no stat error, no network. Nothing in helm,
compose, smoke or the workflows names a /zen binary.
MEASURED: 4225 -> 4060 MB uncompressed (-164.7 MB), 1.65 -> ~1.60 GB compressed.
While measuring, two things worth recording because they contradict the obvious
guesses:
- Sqlite/CGO is not the cost, it is a 3.29 MB SAVING per binary. CGO=1 with
libsqlite3 links the small mattn shim against system libsqlcipher; CGO=0
links pure-Go modernc. Dropping the tags for "non-storage" plugins would ADD
388 MB. The Dockerfile's uniform-tags decision is right.
- 100% of the delta is the duplicated library. plugin/dns carries 741 lines of
app source and is 32 BYTES larger than an empty app. 119 x 20.23 MB =
2,407 MB of 4,095 (58.8%) is the same request tier byte-for-byte, 119 times.
There is no Mount-style registry indirection to break here — manifest is 344
packages of stdlib+zip importing zero apps, and every plugin main imports only
cloud + its own app. The fat deps are honest (geth<-treasury, esbuild<-
connectorruntime, k8s<-ai/controllers). A perfect diet of the shared floor is
bounded at <=519 MB (12.7%), because the host is a real serving binary at
15.86 MB.
The big prize is a multi-call binary: 4,095 MB -> 336 MB (-92%), one file plus
118 symlinks, every app still its own process. Built and linked in 12.3s; NOT
shipped because argv[0] dispatch cannot be verified end-to-end without a
container, it needs gen-app-cmds to derive a fused main from 119 hand-authored
mains, and it reverses a documented decision in manifest/release.go. That is a
designed change with a test plan, not a slip-in.
Also rejected with numbers: -trimpath (-10 MB fleet, 0.25%, cold-busts the build
cache), reflect method-pruning (three independent triggers; closing two buys
+128 B), embeds (<=63 MB, 1.5%).
`go generate -run zipdoc ./...` was 355.9s of a 17-minute image build — 35% —
spent regenerating 99 zipdoc_gen.go files that are already committed. The image
now trusts them.
This is strictly stronger, not a trade. The build-time pass would happily build
a CORRECT image from a STALE commit and leave main wrong, silently. That is not
hypothetical: plane/plane.go documents RunOnBehalfIn.Model and HEAD's
apps/agents/zipdoc_gen.go carries not one word of it. Main was stale the whole
time that RUN was "doing its job" every build. Freshness is a property of the
COMMIT, so it is now enforced where commits are.
Two gates existed and neither could fire:
- make test has carried a zipdoc -check loop for a long time, but CI never
runs make test — hanzo.yml's test: block runs raw go steps. It has never
executed in CI, not once.
- app-contract → surface-check regenerates these files as a SIDE EFFECT
(mk/plugin.mk: describe: build, build: generate) and then scopes its
porcelain check to openapi.yaml / openapi/floor.json / plugin/. A stale lift
was silently repaired in the runner's tree and never reported.
So the new zipdoc-current step runs BEFORE app-contract. After it the tree is
already regenerated and the check asserts nothing.
The two inline -check loops in test and test-fast collapse into one zipdoc-check
target all three callers use — the same discipline app-contract already follows,
and exactly the drift that let the Makefile's own check end up policing nothing.
The old remediation rendered `././...` for d=.; the new one names a command that
works from a bare checkout.
The gate regenerates and diffs rather than asking -check for a second opinion: a
gate must never be able to disagree with the tool it polices. It uses
git status --porcelain, not git diff, so a NEW package's untracked zipdoc_gen.go
cannot slip through.
Also commits the live staleness in apps/agents/zipdoc_gen.go.
Measured: 40.4s warm on an 8-core box, 355.9s cold on the BuildKit runner.
go build ./... clean; TestSpecCarriesProse passes and is byte-unmodified — it
now proves the prose in the file that actually SHIPS is live.
Same shape as the bots entry: a ledger describing a surface that moved, and a
freeze that a new app walked past. No behaviour changes.
ceff43ac drew the line that the document publishes what was DECLARED rather
than whatever the router happened to bind under All(), which retired the
OPTIONS and TRACE entries every wildcard used to carry. Three ledgers still
named them, so each described operations its own document no longer has:
- apps/dns: five relay operations at the greedy wildcard, not seven.
- apps/o11y: the /v1/sentry catch-all, same two.
- apps/exec: derived rather than listed, so the fix is one line —
servedMethods is the five methods zip has a typed registrar for, and the
cross falls from 8x7=56 to 8x5=40. methodReason went with them; it existed
only to explain OPTIONS and TRACE.
Worth stating because the two facts disagree: OPTIONS and TRACE are still
SERVED. All() registers every method and the executor answers them, which
the proxy subtest asserts and which still passes. They are no longer
PUBLISHED. The ledger counts the document, so they leave it.
manifest: apps/sandbox joined manifest.Apps between exec and websearch without
joining the frozen sequence, so the freeze guarded 120 of 121 apps. Frozen at
the position it already mounts at, which is what the test compares.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Transactions, Credits, Team and Settings on billing.hanzo.ai were empty
because GET /v1/billing/{transactions,credit-balance,accounts} and
accounts/:id/members answered 404. They read as a stale deploy and were
not one: commerce declares all four on its api.Route() `user` group
(api/billing/handlers.go), but the co-resident embed registers on the
HOST's router and never compiles that table, so a commerce route reaches
production only if Mount names it. The handlers shipped in the pinned
module all along — v1.50.11 has every one of the four symbols — while no
binary in the fleet served them, which is why grepping the library found
them wired and the tabs stayed blank.
Chain is GetTier's, and each link earns its place on a money read:
IAMTokenRequired resolves the org but FALLS THROUGH when there is no
validated principal, so PinBillingSubject is both the gate and the IDOR
control — it fail-closes that fall-through as 401 (a browser
re-authenticates on 401 and only reports 403) and overwrites every
subject key {user,userId,customerId} with the caller's own account.Payer
subject. That is load-bearing, not decoration: ListTransactions filters
on ?user and GetCreditBalance on ?userId, both unpinned client values, so
an unmounted-but-naive mount would have returned every subject's rows in
the org namespace. Because the pin SETS the key rather than validating
it, the subject is exactly the one account.Payer debits — a read cannot
disagree with the wallet it describes. TokenRequired follows the pin so
the trusted S2S reader still resolves an org; IAMTokenRequired admits
only IAM principals and would leave GetOrganization nil, which panics.
The test asserts 401-not-404. Both are "no data" to a browser, but 404
means the gate never ran and 401 means it ran and refused — only the
second proves the route reached its middleware, and a library-side grep
cannot tell them apart.
The isolation boundary is the pod's RUNTIME — one field, SANDBOX_RUNTIME_CLASS,
holding gvisor or kata-fc or kata-clh or nothing. That is the whole of the
runtime decision: no fork in the code, no second implementation, and a swap is a
deployment change rather than a release. Empty means the node's default runtime,
which is honest rather than a hole — runsc has to be installed on the nodes
first, and installing it restarts containerd under 204 running pods, so that is
maintenance and not something a release does on its way past.
No daemon inside the pod. Commands go through the Kubernetes exec subresource,
so cloud talks to the apiserver rather than to a pod IP. That deletes three
problems the predecessor had to solve: a shared service key sitting in an
environment the submitted code could read out of /proc, a bind protocol to stop
a recycled pod IP serving another tenant's checkout, and an in-pod HTTP server
that had to be built into every image.
PROVEN LIVE, not asserted — go test ./apps/sandboxes -run TestLive against
hanzo-k8s, 27.7s, image node:22:
RUNS CODE: SANDBOX-RUNS-CODE v22.23.2
EDIT PERSISTS: export const answer = 42; // edited by the agent
EDITED CODE RUNS: ANSWER=42
FAILURE IS DATA: exit=3 stderr=to-stderr
The file is written through one exec and read back through a SEPARATE one, so
what is proven is a filesystem and not a buffer. The last line is the one an
agent depends on most: a command that exits 3 is a successful call carrying a
failed program, because a caller that cannot tell "your tests failed" from "the
sandbox is broken" will retry the wrong one forever.
The image is an env var, and that is why this could be proven before our own
image exists. A stock node:22 exercises the identical create/exec/edit path.
Named sandboxes, not machines: /v1/machines already exists and is visor's —
whole GPU and VPS instances, sized, quoted and billed. Two products cannot share
a noun. The namespace and pod labels follow the same word, which matters more
than it looks: universe's containment policy selects hanzo.ai/sandbox-class in
hanzo-sandboxes, and a policy naming a label the code does not set is the
enforce-nothing failure this fleet has shipped three times this week.
Four gates went red because two new apps arrived without the entries every app
owes, and because a change to what the document publishes was not carried into
the ledger that counts it. None is a behaviour change; each is the declaration
the gate exists to demand.
- spend.go: plugin/lsp declares Price: cloud.Metered, so lsp joins
meteredApps. Without the entry the surface is free the moment SpendGate
enforcement is switched on, which is precisely what
TestMeteredSurfacesRequireStanding refuses to let drift. apps/lsp/meter.go
already owns the debit — a cold checkout+index is billed, a warm query is
not.
- product_gate_test.go: apps/lsp and apps/sandbox mount no product and held
no pin. Neither has an upstream repo to mount, so both are `unextracted`,
the bucket that says the functionality lives only here and owes its
extraction at migration.
- typed_request_gate_test.go: apps/lsp/lsp.go takes cloud.Request for the
money gate in front of a query, which needs strictly more of the validated
principal than the org — the payer and the validated project sub-scope,
neither of which may become an In field without letting a caller name who
it bills. One call site, guarded by onHTTP so it fails closed off the HTTP
path.
- apps/bots/typed_wire_test.go: OPTIONS and TRACE on the relay wildcard left
plugin/bots/openapi.json in ceff43ac, which drew the line that the document
publishes what was DECLARED rather than whatever the router happened to
bind under All(). The ledger still named them, so it described a surface
that no longer exists and the partition summed to 10 against a served 8.
Five relay entries now, not seven.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/deploy/deploy.go:350 calls registerEngineRoutes, and the file that defined
it went away in 77f187f7 — a commit whose message is a commerce dependency bump
about crypto deposit addresses and which says nothing about apps/deploy. It took
engine_mount.go (172 lines) with it and renamed engine.go to source_git.go,
which reads like a dirty worktree committed alongside the bump rather than a
decision: the same file and the same call site are both still present on the
GitHub lineage the author works from.
The tree has not compiled since. `go build ./apps/deploy/...` fails on an
undefined symbol, so go-unit reports "[build failed]" for apps/deploy and
plugin/deploy, the gate cannot pass, and no run can reach the image car — which
is why main has published nothing while commits kept landing on top of it.
Restored verbatim from 77f187f7^. The engine stays gated by
DEPLOY_ENGINE_ENABLED (default off), so this brings back the wiring and no
behaviour.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
github.com/hanzoai/ai v1.832.30 -> v1.832.31, which removes the second debit
on the casibase chat answer path. There, recordCasibaseChatUsage and
AddTransactionForMessage both reached the usage recorder cloud installs, so
one completion charged the customer twice.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GPU is metered like any other resource through the machine launch, whose
balance gate and per-hour meter both live upstream in Visor. Two references to
the bespoke prepay charge outlived its deletion.
apps/billing/balance.go named gpu_charge.go as a second consumer comparing this
balance against a GPU's price. The point the paragraph makes — that `available`
is spent against rather than displayed — rests on its remaining example, the ai
balance gate that reads the field over the S2S HTTP path, so the clause goes and
the sentence keeps its subject. Comment-only: the FloorMinor rounding it
documents, and every caller that compares rather than debits, are untouched.
.hanzo/workflows/cicd.yml still spelled the drift incident with the two concrete
addresses that the same deletion generalized in cmd/reach/main.go and
openapi/fleet.go. It now describes that incident the way its siblings do — one
build serving a renamed route under its new name while still publishing the old
one — so the release train's own preamble stops naming routes the fleet no
longer serves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GPU is metered like any other resource through the machine launch, whose
balance gate and per-hour meter both live upstream in Visor. Two references to
the bespoke prepay charge outlived its deletion.
apps/billing/balance.go named gpu_charge.go as a second consumer comparing this
balance against a GPU's price. The point the paragraph makes — that `available`
is spent against rather than displayed — rests on its remaining example, the ai
balance gate that reads the field over the S2S HTTP path, so the clause goes and
the sentence keeps its subject. Comment-only: the FloorMinor rounding it
documents, and every caller that compares rather than debits, are untouched.
.hanzo/workflows/cicd.yml still spelled the drift incident with the two concrete
addresses that the same deletion generalized in cmd/reach/main.go and
openapi/fleet.go. It now describes that incident the way its siblings do — one
build serving a renamed route under its new name while still publishing the old
one — so the release train's own preamble stops naming routes the fleet no
longer serves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The sink commits one row per message, and insertSettings asks the store to
batch it (async_insert) while wait_for_async_insert holds the call open until
the block is durable. With a single puller those two deadlock: the one
in-flight row IS the whole batch, so every insert pays a full materialized-view
cascade to land a single fact, and the next message cannot start until it
finishes. Throughput collapses to 1/insert-latency — measured at ~24 rows/min
against a ~2.5s cascade — no matter how much is queued behind it.
The warehouse was ~3.6 hours behind live traffic and losing ground, so every
dashboard, heat map and rollup read empty: they filter on `time`, and `time`
had not advanced since the backlog formed. Nothing was lost — the door commits
to JetStream before it answers, and the stream holds 72h — but the window is
finite.
A JetStream pull consumer load-balances across every client bound to it, so
binding the durable N times is concurrency without a second cursor. Each
message is still delivered once and still acked only after ITS OWN row lands;
redelivery, MaxDeliver and the abandoned-fact advisory are untouched. That
gives async_insert real blocks to coalesce and lets the store overlap the
cascades, so the ceiling becomes the store's rather than the round-trip's.
Also drops insights.raw_sessions_v3_mv in the warehouse (done out-of-band):
854ms and 9.2MiB of every insert spent projecting a table with no readers.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nothing credits a crypto deposit. The intent never leaves Pending, no watcher
observes the deposit address, and GenerateAddress discards the wallet_id, so
money sent to a minted address is received and not credited — and not readily
recoverable either. The rail now refuses before any keygen.
Card, bank transfer and wire are unaffected.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two things, one seam.
boxd assembled its own request tier out of net/http: a ServeMux, an
http.Server, a hand-rolled guard wrapping a Handler, and three helpers
(writeJSON, writeErr, and a bind that re-applied a body ceiling per call) that
zip has had all along as c.JSON, zip.Errorf and c.Bind. It is a standalone
binary but it is not a different kind of program, and there is one web framework
here. It is zip now: one middleware seam, one error shape, one place a route is
declared. The METHOD moved onto the route where it belongs — three handlers
opened with the same six-line method switch precisely because a mux matches a
path and leaves the verb to whoever wrote the body, which is how PathFsDelete
came to share a handler with the fs root.
BodyLimit is stated rather than defaulted: zip's default is 4 MiB and a box
takes source, so leaving it implicit would have cut the ceiling 8x while every
test that writes a small file went on passing.
Two things the conversion had to get right rather than translate:
- A streamed body is written from inside c.SendStreamWriter, not to a captured
ResponseWriter. agentRun's terminal-frame contract is unchanged — every
return still ends the stream where it always did.
- A file handed to c.SendStream is read AFTER the handler returns, so it must
not be closed on the way out. It was, and the LibreChat artifact test caught
it as "file already closed" on a body nobody had written yet. Ownership
transfers to the stream; the one path that never reaches the stream closes
it itself.
zip has no multipart accessor, so the upload path parses the body zip already
read with the stdlib rather than reaching through c.Fiber() and putting the
framework we are hiding back into a handler. That gap is worth closing in zip —
this is the second consumer that needs it.
SECOND: the box now learns its identity, which is what makes X-Box-Id mean
anything. The header was being sent and compared against BOX_ID — an env var,
read at startup, set nowhere and unsettable in principle: boxes come from a WARM
POOL, the pods are already running when a tenant appears, and a claim is a label
patch on a live pod. There is no moment at which env could carry a per-box id,
so the guard compared against "" and skipped, forever.
Cloud binds the box at claim instead, over POST /v1/box/bind, before the address
is handed to anyone. A failed bind fails the claim and releases the pod rather
than returning an address that would 409 at its own owner. Binding twice to the
same id is fine (a retry must be able to finish); binding to a different one is
refused, because release() deletes a pod rather than recycling it and there is
no legitimate second tenant.
An UNBOUND box now refuses a named call too — that is the recycled-address case
exactly: a pod that died without release() leaves a row pointing at an IP the
CNI has reassigned, and the replacement is warm and bound to nobody, so serving
it would hand a stranger's request to a fresh checkout.
The two tenancy tests count box-bound requests, and the claim now legitimately
makes one. They assert a DELTA rather than an absolute, and the baseline is
itself asserted to be exactly 1 — so the setup cannot quietly absorb a leak.
Two halves of the same seam, entangled in the same files.
THE DOOR THE AGENT USES. An agent's tools now come from the fleet's own MCP
door, published on the plane socket every child already dials
(mcp.Serve(door, manifest.MCPPath) in serveWake). Not a second gather: refuse()
is applied inside gather(), where the routing table is written, so any second
projection would be a second policy site and the first time someone forgot it
the agent would see CreateServiceAccountKey while Slack did not. An agent is now
just another MCP client of the one door and cannot see a wider surface than an
external client -- there is no wider surface.
ONE TOOL PER SUBSYSTEM. tools/list published 1,189 flat tools in 977 KB --
roughly 244k tokens to merely enumerate. Clients truncate; Slack keeps 128. So
1,061 operations were unreachable no matter how well ordered. rank() fixed the
order and could not fix a hard cap.
Now: hanzo_<app> with inputSchema {op: enum[...names...], input: object}, plus
hanzo_describe returning one operation's own descriptor bytes. Names in the
enum, schemas on demand -- the schemas were the 977 KB. The grouping key is
gather()'s existing owner map, so there is no second lookup and no second
source. The envelope is a DECODING, not a route: call() unwraps {op,input} into
the (name, message) a direct call carries and takes the same hop with the same
headers, so a flat tools/call by op name still works byte-for-byte and Slack's
128 cached names keep functioning.
MEASURED against the fleet's own corpus (plugin/*/openapi.json, replayed through
real zip children over real sockets), baseline a real curl of api.hanzo.ai:
BEFORE 1189 tools 977636 bytes 822 B/op
AFTER 116 tools 106847 bytes 47 B/op
10.2x fewer tools, 9.1x fewer bytes, for 1.9x MORE operations (17x per op)
Like-for-like the baseline's own 1,189 ops would cost ~56 KB, ~14k tokens
instead of ~244k.
hanzo_describe is published FIRST, not last: it is what makes every other tool
usable, and a client that truncates and drops it holds 116 enums it cannot read.
HEADROOM: 116 of 128, against a 119-app manifest. This fits today and will not
fit forever. When it stops, group by productStems bucket (17) -- do not add a
second projection. Recorded in LLM.md.
SECURITY. TestARefusedOpIsInvisibleUncallableAndUndescribable: the child
genuinely serves CreateServiceAccountKey (asserted against MCPTools(), so
refusing it proves something), and the name appears nowhere in the tools/list
bytes, hanzo_console{op:...} returns -32602, and hanzo_describe{op:...} returns
-32602. Both messages are checked to stay non-oracular. All three paths read the
one set gather() produces. fleet/surface.go has ZERO diff -- refuse() is still
the only gate, unchanged.
describe re-asks gather() on every request rather than caching: a remembered
descriptor could describe an op the rule has since refused.
30 of 117 subsystems answered 308 to the door's tools/list and were reported as
outages. They were not down. Three links, each verified:
1. zip skips installing the MCP door when an app has no typed ops
(zip mcp.go:99). The deciding term is len(a.Registry())==0 on the SERVED app.
kms/billing/tasks/platform/index look like they have ops but register them on
cloud.Plane() -- a DIFFERENT *zip.App -- so their served app has none.
Census: all 30 dark apps have 0 typed ops on the served app; all 87 working
ones have >=1.
2. webui.Mount runs in every plugin process, not just the host (serve.go:413),
and registers the console catch-all.
3. So POST /mcp fell through to webui/mcp.go's signpost, which 308s to
manifest.MCPPath -- an address that is a 404 inside a child.
The signpost's own comment stated the assumption that made it safe: "a plugin
that serves its own door at FrameworkMCPPath matches a real route and never
reaches here." Link 1 made that false.
Following the redirect would mask it -- the hop lands on 404 in the child. It is
the front door's topology leaking into a process where it is untrue.
Fix is one line in cloud.App(), the constructor every Hanzo program reaches:
MCP.Source is now always non-nil (the declared Plugin.Door, else an empty
per-caller half), so hasCaller() holds and zip installs its own door. Still
exactly one MCP implementation, and zip returns the pre-rendered bytes verbatim
when the per-caller half is empty, so the memcpy stays.
MEASURED, and this is the honest part: 29 of 30 recover their door (kafka needs
a live broker to mount) and they contribute ZERO new tools. Every one has 0
typed ops -- which is why they were dark. This turns 30 phantom outages into 30
honest empty doors and makes hanzo.ai/unavailable mean what it says. It does not
conjure tools that were never registered.
The tool famine is a SEPARATE defect: MCP tools come only from typed ops, and
/v1/chat/completions and /v1/embeddings are raw routes. apps/ai declares exactly
one typed op, which is exactly the "one tool matching chat" the live door shows.
fleet/childdoor_test.go reproduces the 308 against a real apps/exec child and
asserts bytes, not status. Its sibling proves a genuinely-down app is still
reported, so this cannot be "fixed" by making every child look alive.
Every @hanzo turn failed in production with "authorize: no org on the call",
and the previous fix did not work because it stated the tenant in a place zip
deliberately ignores.
commerce takes the org from the CALLER's identity and never from an argument
(balance_rpc.go:36), so no caller can name the books it charges. The org has to
ride the caller. But zip reads a STATED caller only where there is NO request
behind the context (caller.go:352-356) — otherwise CallerOf reads the request's
own headers. That rule exists so that stating an identity can never override an
authenticated one.
The prior attempt called cloud.For inside the agents op, which is reached OVER
THE PLANE — a real request. The statement was silently discarded and the gate
still refused. It read exactly like the working background callers elsewhere in
the tree; the difference is invisible without the precedence rule.
So the org is now stated by the DISPATCHER, on a detached context, before the
hop: bridgeRunContext(org) = WithTimeout(cloud.For(context.Background(), org)).
Caller.headers renders it onto the wire (caller.go:302) and it rides onward for
free. This matches the existing convention in this package —
TestTheDetachedImportStatesItsTenant covers the git-import path the same way.
Detaching was independently required: the turn runs in bridgeSpawn's goroutine
after the webhook has already answered Slack 200, so on the request context the
model call was being cancelled the instant we replied.
bridgeReply loses its ctx parameter. An unused context argument here is an
invitation to pass the webhook's, which is precisely the bug; removing it makes
that unavailable rather than merely discouraged. The no-op in the agents op is
replaced by a comment stating why it must NOT be done there.
Also: the install URL asked Slack for 9 of the 13 bot scopes the app manifest
declares. Slack grants exactly what the consent URL requests, so anyone
installing through it got a token with no `commands` — /hanzo would fail at
first use, long after the install looked successful. Adds channels:read,
commands, team:read.
go build ./... clean. apps/integrations passes whole. apps/agents shows the
same 7 pre-existing metering-plane failures as baseline, no new ones.
@hanzo got past "agent not found" and then failed with
bridge: agent run err="authorize: no org on the call"
A run BILLS. The balance gate is a plane call to commerce, and commerce takes
the org from the CALLER's identity, never from an argument -- deliberately, so
no caller can name the books it charges (apps/commerce/balance_rpc.go:36, and
meter_rpc.go says it outright: "The org is the CALLER'S and can never be named
in the input"). A turn dispatched over the plane has no inbound request to carry
that identity, so the gate refused and every Slack message died after the agent
had already resolved.
Fixed with cloud.For, which is the function that exists for exactly this: it
states the tenant a BACKGROUND call acts for -- one with no request to forward
-- and it CANNOT launder an identity, because zip prefers a gateway assertion
over it whenever a request exists (plane/ask.go:236-242).
The org is trustworthy for the same reason every other Slack path trusts it:
the bridge resolved it from the Slack-verified team_id through the install-to-org
map, never from a payload field.
Tests: a plane run acts for a named tenant, and an empty org states nothing
rather than becoming a blank tenant that would bill an account named "".
Measured on a 23-minute image build, the four steps that cost it:
#27 422s the plugin loop
#24 396s go generate -run zipdoc ./...
#41 244s exporting cache to registry (mode=max)
#39 210s exporting + pushing layers
#27 was ~60 `go build` invocations run one at a time by a shell `for` loop, on
an 8-core runner. Each is a separate process, so Go's own intra-package
parallelism does nothing for the set: seven of eight cores idled for seven
minutes. They are independent binaries with no ordering between them, which is
the definition of embarrassingly parallel.
Now `xargs -P $(nproc)`. The existence check keeps its own pass so a missing
plugin/<name> still fails with the message that names the fix, before any
compile starts and while the output is still readable.
A failing build exits 255, which is xargs' stop-everything code — the whole
step fails fast rather than compiling 59 more plugins to report one error at
the end. It also prints WHICH plugin failed, because parallel output is
interleaved and the go error alone no longer says.
GO_LDFLAGS survives the subshell: it is ENV (Dockerfile:259), so it is in the
environment xargs' sh inherits, not a shell variable that would silently vanish
and leave every binary stamped "unknown". The existing strings|grep check on
the artifact still proves that on the bytes rather than on the flag string.
apps/integrations carried two openapi.Describe calls for GET
/v1/integrations/slack/install, written by different hands into different init()
funcs. Describe panics on a duplicate — correctly, because two descriptions of
one operation means one of them renders and nobody can tell which — and that
panic fires at init, so it took down every app that links integrations: ads,
automations, campaign, catalogsync, channels, cloudflare, company, content,
destinations, git, guide, integrations and sync all failed to describe.
The earlier one survives. It was already the superset: it has the attribution
constraint (Slack refuses a slack.com URL in that field, so the click has to
route through an address of ours to be counted) AND the tenant point (public, no
principal, binds no org, because minting an org for an anonymous click is the one
thing that would break isolation). The later one had a single fact the first did
not — 503 where the app is unconfigured, rather than a consent URL with an empty
client_id that Slack renders as its own dead end — so that sentence moved across
before the duplicate went.
The floor drops for the merge's own deletion too: /v1/billing/gpu/charge and
/v1/billing/gpu/eligibility are gone because GPU is metered like any other
resource now, and the bespoke prepay path with it. Checked rather than assumed —
a -1 that is not a multiple of two is not a TRACE/OPTIONS removal, and an
unexplained shrink is exactly what the ratchet is there to make someone look at.
1762 paths, 2480 operations, 185 products. Every one carries an operationId and a
summary; 51 still want a long description and 49 of those are hanzoai/ai, whose
prose belongs on its controllers, in that repo.
Two ways the same confusion showed up, and one build that could not run.
TRACE and OPTIONS were published — 33 each — because All() binds every method
at a path and the describe loops read that route table as a product surface.
CONNECT and HEAD were already excluded on exactly that principle. TRACE echoes
the request back; it is the Cross-Site Tracing verb and should not be reachable
on a public API at all, so advertising it in the contract is worse than merely
routing it, because the contract is what the SDKs, the CLI and the MCP tool list
are built from. OPTIONS is CORS preflight: a browser sends it, a person never
does, and `hanzo meet options` is not a command anyone wants. Between them, 66
operations — and the generated CLI already carried none of them, so this is the
document agreeing with the projection that was right first.
websearch held its own copy of that method list. It said seven, including the
two, and the day the document stopped publishing them the copy went on
describing operations that no longer existed — which openapi.Methods() exists to
prevent and which the describe gate caught by name. It reads the projection's own
set now.
The floor drops by exactly 66, in this commit, next to the reason.
Separately: every app Makefile claims "Generated by plugin/gen-app-cmds" and none
of them were. The generator wrote plugin/<app>/main.go and left a human to
remember the second file, so `sandbox` arrived with a main and no Makefile, so
`make describe` had no rule for it, so it published no subset, so `make openapi`
failed on an app that was otherwise finished. The generator writes both now, from
the same manifest row, taking the package from each app's own composition root
rather than a lookup table. It reproduced all 117 existing Makefiles with zero
APPS values changed, which is the evidence the rule is the one already in use.
External modules fall out structurally — no package directory here, no Makefile —
so mk/fleet.mk's EXTERNAL list is not copied.
The Home tab rendered Slack's own "this is still a work in progress"
placeholder -- what Slack shows for any app that enables home_tab_enabled and
never publishes a view. The choice was never Home vs no Home; it was our page
vs Slack's apology.
The page now carries what a person opening it actually needs:
Model enso (auto) / enso-flash / enso-ultra, THEIR choice pre-selected
Mode chat only, or chat + code
Connected who the turns run as, and the org they bill
Started DM, @mention, /hanzo, and a coding run
Both controls write the SAME userLink the chat path reads -- no second settings
store, no third source of truth for which model answers. The turn then carries
that model per-request (plane.RunOnBehalfIn.Model), because it is a preference
of the PERSON asking, not a property of the agent: two people in one workspace
can prefer different models of the same assistant.
Interactivity arrives at the SAME signed URL as events, form-encoded rather
than JSON, so it is sniffed and handled before routing -- that is where the two
encodings actually diverge, and no header separates them.
Three refusals, all deliberate:
- an UNLINKED user is shown how to connect and given NO controls; a model
chosen by an identity we cannot resolve is a control that does nothing
- a value not on our own menu is DROPPED, not stored: the options came from
us, so anything else is a stale client or a forged payload choosing what
this org pays for
- the per-turn model overrides only the BUILT-IN agent; a person's Slack
preference must not silently re-point an agent their org configured
Isolation is unchanged: the org comes only from the install-to-org map for the
Slack-verified team_id. The payload names which setting and which user, never
the tenant.
11 tests green. apps/integrations and plane suites pass; go build ./... clean.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/lsp — one door, because there is one value: a language server
rooted at a workspace, asked about a position. code (static index) and
lsp (live server) are two reads of the ONE checkout.
{repo, rev?, path, line, character, method} → hover | definition |
references | typeDefinition | implementation | documentSymbol |
completion | diagnostics
Positions are the LSP's and pass through untouched: 0-based line, 0-based
UTF-16 character. Re-basing them would corrupt every multi-byte line, and
the callers already speak LSP.
Isolation is structural, not checked. The org comes from the validated
principal and is BOTH the pool key and the OWNER SEGMENT of the git URL;
a caller supplies only a repo slug. There is no input from which one
tenant can name another tenant's repository.
Scripts-off by default. A dependency fetch that runs dependency-authored
code is RCE triggered by whatever the caller asked us to check out, and it
buys nothing — servers resolve from source. So: npm ci --ignore-scripts,
cargo fetch (not build), go mod download; the python fetch builds sdists
and therefore does not run. rust-analyzer is additionally told not to run
build.rs or expand proc macros, because otherwise the server does at load
time exactly what the fetch was chosen to avoid. One predicate, one place.
The deployed worker must still be sandboxed — see fetchable's comment.
Language table ported verbatim from hanzo-tools-lsp (same binaries, same
argv, same root markers); install_cmd deliberately dropped — a worker that
can npm install -g at request time is one an attacker can make write to
its own filesystem.
server.go is the testable core: Content-Length JSON-RPC with a SINGLE
reader goroutine demultiplexing responses, server→client requests and
notifications. The python tool reads inline from each call, which drops
every frame that is not the awaited response — which is why it cannot
report diagnostics, and why an unanswered client/registerCapability
deadlocks it. 24 tests drive a fake server over in-process pipes; no
toolchain, no network.
Two bugs the tests found, both real in production:
- Close() wrote a polite shutdown unconditionally, so a server that had
stopped reading its stdin blocked it forever — holding a pool slot and,
at Shutdown, the whole binary.
- rel() compared a symlink-resolved root against an unresolved target, so
any data dir with a symlink component (/var on a Mac, a mounted volume)
made every location "outside" the checkout and handed the caller the
worker's absolute path instead of a repo-relative one.
Metered, not Free: cost is the COLD start (checkout + fetch + first index),
not the query. Warm point queries are recorded and free, so the pricing
does not teach callers to re-key their workspace. Gate before the work.
apps/code has NO checkout to reuse — it indexes files POSTed to it and says
so. apps/deploy has the only working-tree checkout and is not importable
(it would link k8s into this binary). This is therefore a second one,
following deploy's invocation and hardened env exactly; the fix is hoisting
it into the root cloud package, not made here.
Mount order: apps.Wire() and its integers are gone. manifest.Apps slice
position IS the order, so lsp's row sits after code's, with order_test's
frozen sequence updated in the same commit.
apps/commerce would not run a single test. openapi.Describe PANICS on a
duplicate (openapi/register.go) and this package registers from an init(), so
eight re-described billing routes took the whole package down before any test
started:
wire, crypto/options, crypto/deposit, crypto/deposit/:id,
methods (GET and POST), methods/:id, portal/methods
Each already had a description at lines ~222-315; a later block at 444-552
described them again. Removing that block leaves 91 unique Describes and no
duplicates. The unique webhooks/:provider entry that followed it is untouched.
WHY THIS MATTERED MORE THAN A PANIC. Tests that fail to RUN are worse than tests
that fail, and this hid several: with the package dead at init, the billing
subject-pin guard added minutes earlier could never go red — a guard whose whole
job is to stay red until a route pins its subject. Removing the duplicates makes
it execute (14 subject-taking routes inspected).
It also uncovered six pre-existing failures the panic had been masking. All six
are the same environmental class, not defects: 42 occurrences of "no RAM-backed
scratch for the pure-Go SQLCipher codec (need tmpfs at /dev/shm or
HANZO_SQLITE_RAMFS_DIR)". macOS has neither, so the encrypted per-org store
refuses to open rather than decrypt to persistent storage. They run in CI.
The surviving descriptions are the earlier ones. The removed set was in places
more detailed — if that wording is wanted, replace the earlier text rather than
adding a second registration.
Two call sites had picked the wrong one of the three Minor variants, and both
failed silently in a way that looked like something else.
BACKFILL (blocked the finance cutover). apps/admin/finance/backfill.go read a
commerce balance with Minor(), which REFUSES anything finer than a cent. The
ledger keeps eighteen decimals and per-token charges are routinely finer than a
cent, so every org that has spent anything carries a sub-cent tail — and the
backfill returned "read commerce balance: amount … is finer than its minor unit"
and migrated ZERO. Fail-safe, and completely stuck. It now FLOORS, which is the
direction a migration must take: the amount moved must never exceed the amount
held, or the migration mints the difference. The dust stays behind and settles.
MONEY BOARD (display). apps/admin/moneyboard.go read the treasury reserve the
same way, so the same tail rendered as SrcOf("treasury", err) — "could not reach
the treasury" — for a treasury that was reachable and correct. It now ROUNDS.
Nothing is spent from that figure, and refusing to show it is worse than a half
cent.
The rule the two sites got wrong is now pinned in plane/money_test.go against the
real production balance, which distinguishes all three:
149913.078983985999994361 -> Minor() ERR Floor 14991307 Round 14991308
Minor() a DEBIT exact or refuse — rounding money you TAKE is theft
one way and a gift the other
FloorMinor() a COMPARISON never overstate — admitting spend a balance cannot
or MIGRATION cover leaves a negative balance nobody authorised
RoundMinor() a DISPLAY nearest — nothing is spent from it
RoundMinor also gains the coverage it never had: nearest in both signs, and it
still errors on overflow and on a malformed amount rather than wrapping or
rounding to zero.
Found by an adversarial review of the money path (its M1/M2), which asked the
question I had not: where should I have floored and did not.
The built-in default agent I added minutes ago took cloud.FallbackModel, which
is "best". That constant's own doc says what it is for:
"keeps a bot's reply landing when the flash tier is saturated; the
interactive chat path never uses it"
A Slack turn IS the interactive chat path, so I had wired the DEGRADED tier as
the default brain — I reached for the nearest available model constant instead
of asking what the chat model should be.
It is now enso, Hanzo's own auto-routing SKU that selects per query in the
gateway's catalog, with BRIDGE_AGENT_MODEL to override per deployment. Studio
names the same default for the same reason (STUDIO_CHAT_MODEL or "enso").
Forwarded verbatim and never validated here: the catalog resolves it, and a
check in cloud could only disagree with the thing that decides.
Tests pin both halves — the default is enso and is never "best", and the
override wins.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The forge already carried the front-door body-limit fix (internal/edge). These
seven were only on GitHub, so the next pipeline release would have shipped
without them:
84e26665 commerce: the 128 silent operations say what they do
702ecbf7 preflight admits the four headers the console actually sends
bd4115d3 o11y: the product scope is the fleet, sub-cent debit is not an outage
8f1f50e0 CORS: allowed origins derive from verified site hosts
3bfdeee0 slack: the agent turn crosses the plugin boundary over ZAP/UDS
c78fbe72 a console the customer owns can call the API it was forked from
Resolutions:
- apps/billing/usage_coresident.go — FloorMinor, not RoundMinor. Both fix the
same 502, but balance.go:126 already floors a row from the same ledger, and
two roundings on one ledger is how the next discrepancy starts.
- apps/framework — RoleAssignment/roleRef. Without the rename the weave refuses
the whole document: iam publishes a Role ENTITY and framework published a
(user, role) GRANT under the same name, and a generated SDK binds whichever it
read last. revokeRole still named the pre-rename type; it takes *roleRef now,
which is the URL-path form it actually addresses.
- manifest/apps.go — UNION, not a side. github adds /v1/event.js (the hosted
analytics tag, previously falling to ai's bare /v1 remainder, which a browser
reads as a broken script tag); the forge has /v1/replay, which github lacks.
Taking either side alone drops a live route. The superset check I wrote for
the last merge is what caught it — it asserted, and it fired.
- Generated artifacts (openapi.yaml, plugin/*/openapi.json, */zipdoc_gen.go) are
taken from github, not regenerated here: `make describe` on this tree still
fails for 14 apps, and github's are the set that was verified green at 1735
paths. A half-regenerated document is worse than a whole borrowed one.
go build ./... exits 0. TestTheServedDocumentIsTheArtifact and the door
body-limit tests both pass.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
I introduced this tonight. To make the S2S tier reader resolve an org I mounted
the route RequestContext -> IAMTokenRequired -> TokenRequired and stopped there.
TokenRequired AUTHENTICATES and does not PIN, which is the exact hazard the
comment on /v1/billing/methods already spells out: any authenticated browser can
then name another subject.
GetTier reads ?user= verbatim and answers with that subject's wallet —
prepaidAvailable, creditsRemaining, effectiveAvailable. Every one of the other
subject-taking billing routes carries accountclient.PinBillingSubject(); this one
did not.
It is CROSS-CUSTOMER, not merely cross-subject. Every self-serve signup lands in
the SAME org (account.SignupOrg = "hanzo") with a per-person subject, so the org
namespace was closed while the subject was open. Measured live against the running
pod before the fix — one caller, four wallets:
?user=hanzo/z prepaidAvailable 10966
?user=hanzo prepaidAvailable 6375
?user=hanzo/admin prepaidAvailable 10000
?user=hanzo/dev prepaidAvailable 0
The pin does not undo the fix it sits on top of: PinBillingSubject admits a
verified service-token caller that names its own org and leaves its query
UNTOUCHED (apps/account/billing_coresident.go), which is exactly what ai's rate
limiter and apps/metering send. TokenRequired stays after it so that caller still
resolves an org.
The guard is a CLASS guard because a one-route fix would not have found this one.
It walks the AST of the mount table and fails any /v1/billing route that takes a
caller-named subject without a pin.
Two things the guard had to survive to be worth committing:
- The first version used a regexp and SILENTLY SKIPPED the route it was written
for. The non-greedy chain match ended at the first `\n\t)` and the scan resumed
past several registrations, so it reported "14 routes checked", passed, and
would have passed with the leak present. It now asserts it can SEE
/v1/billing/tier, so a walk that stops reaching the table fails loudly instead
of reading as green.
- The alerts group is allowlisted only after checking it: billingSubject derives
from the RESOLVED ORG (org.Name, spend_alerts.go:62) and every read is
org.Namespaced, so there is no caller-named subject to pin. An unverified entry
in that list turns this guard into a rubber stamp for the leak it exists to catch.
Verified both ways: removing the pin fails the guard naming mount.go:603.
Found by an adversarial review of the money path, which ranked it the one thing
worth fixing before launch.
Closes the last two surfaces where a client could name the usage idempotency ref
(the ai answer surface and the domain register/renew/transfer refs) so Usage.Seal()
mints them, and backfills the program column on pre-existing finance.usage rows so a
ref straddling the deploy debits once. R7 asserts no plane op carries metering.Usage;
R8 routes the split-deploy debit over the plane, not the tag-stripped HTTP wire.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The span-signal slice of the analytics fan-out gets the sibling the error slice
already has. AddSpanSink mirrors AddErrorSink and filters on routeOf -- the
plane's ONE routing rule -- so the lens can never carry a fact the plane filed as
something else: an event with a span BODY that routes as an act stays an act.
apps/o11y writes the row, and it writes a ROW rather than calling a module
because that is what the read side is. llmobs.Module has no ingest surface; its
observations/traces/sessions/users are querier reads of event.span filtered by
`gen_ai.system EXISTS AND gen_ai.hanzo.org_id = <caller's org>`
(impllmobs.genAIFilter). So the only way a span becomes an observation is to BE
such a row, and the rows go through insertSpans, which owes event.span and the
event.trace partial the trace list resolves against -- a span written without its
partial is a span every trace read answers empty over.
Only gen_ai spans project. The marker is the reader's own, so the lens admits
exactly the spans the LLM views return; an ordinary span would be write
amplification no read can answer with, and a second copy of a span the ZAP door
already owns. An attribute that renders empty is dropped rather than stored
blank, because a key stored empty satisfies EXISTS while naming no provider.
The tenant is the SLUG, stamped last and unconditionally. gen_ai.hanzo.org_id is
the only org discriminator the span views have, and the handler binds it from the
validated X-Org-Id -- so the o11y org UUID the Sentry lens derives would be a row
every LLM view returns zero of. A wire-supplied org is overwritten, never read.
The span slice carries the scrubbed property copy the fact row stores, which is
where this seam differs from the destinations slice: that consumer FORWARDS and
must hash raw match keys, this one STORES, and scrubText's contract is that a
token in a property is redacted before storage. A projection must not store more
than the plane stores.
CLOUD_LLM_LENS turns it off; default on, the CLOUD_SENTRY_LENS posture. It
installs from mountPlaneIngest rather than mountRuntime, which is the same rule
against a different dependency -- a lens installs where the thing it writes
through becomes available, and mountPlaneIngest sets the plane sink after
mountRuntime has already run. ShutdownO11y detaches it first, beside the error
lens, so no in-flight ingest dispatches into a closing connection.
Mutation: widen the filter to admit any event carrying a span body and
TestFanOutSpansIsThePlanesRoute goes red with
want the 2 span-routed events, got 3: [... {Name:click Kind:track SpanID:x}]
-- the act the plane filed as an act, arriving on the span projection under its
own name and kind. Restored, green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A buyer who picked SOL reached the confirm screen and met 'the custody service
is not accepting new addresses'. It could never have worked: a Solana address is
an Ed25519 key and the fleet runs only the secp256k1 ceremony, so mpcd never
emits sol_address. The asset picker is rendered from the processor's currency
list, so listing it is what put the button there.
Every other chain mints — ethereum, bitcoin, polygon and lux all verified
returning addresses through our own MPC.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
THREE things, all measured against production tonight.
1. AGENT NOT FOUND. With the plugin-boundary fix deployed the turn finally
reached agents, which answered `agents: agent not found`. The bridges ask
for the conventional ref ("hanzo") and Store.Resolve is a plain row lookup
with NO seeding anywhere — so an org that connected Slack and did nothing
else had no agent, and @hanzo could never work out of the box in ANY
workspace. The conventional ref now resolves to a BUILT-IN default: not
persisted (a row would fork the definition per org and strand already-seeded
orgs on a later change), and a row the org DOES create still wins because
Resolve is tried first. An unknown ref stays a miss — silently substituting
the chat agent would make a typo in `code: repo` run the wrong thing and
look like it worked.
2. TOOL CALLING. executeRun did one completion and returned the text; Agent
.Tools was stored, updated and displayed but never read by a run, so the
agent could not reach anything. It now runs a bounded tool loop and takes
the actor, so every tool call is attributable to the org and user that
caused it.
3. TOOL SURFACE. fleet/mcp.go + fleet/surface.go curate and gate what a client
may see: the live server projected 1,323 internal ops with zero annotations,
and Slack saved the first 128 ALPHABETICALLY — a window containing zero
product tools and 36 credential/auth ops including CreateServiceAccountKey.
Cross-agent collisions found and fixed while integrating: tools.go's truncate
collided with an existing test helper (renamed truncateToolResult, which says
what it bounds), and executeRun gained an actor parameter that two test callers
had not been updated for.
go build ./... clean. fleet and plane suites green; apps/agents' one failure,
TestTargetOpsProjectEverywhere, is the known pre-existing op-id drift.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
413 lines of harness and tenancy tests referenced stateWarm, classDesktop and
pool.nodeTaint from an implementation that was replaced, so the package compiled
and its whole suite did not — zero coverage on the one component whose job is
keeping one tenant out of another's box.
The substitutions are both outside the implementation: client-go's fake dynamic
client for the pool, and an http.RoundTripper swapped into the package-level
boxClient for the box. forward() is therefore exercised byte for byte — headers,
query handling and all — against the production route table rather than a
paraphrase of it. Route tables are shared vars, so a route that moves breaks the
tests instead of leaving them testing a path nobody serves.
Six tests are skipped rather than deleted, each naming the mechanism it needs.
That is the point of the port: a skip that names a missing mechanism is a
finding, and deleting it is how the mechanism stays missing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/exec pointed at code-exec.hanzo.svc while apps/sandbox schedules boxes into
hanzo-boxes, so the two halves of one executor sat in different namespaces under
different containment — and the half in `hanzo` was the half sitting beside the
datastores. They run the same binary on the same submitted code; only the
lifetime differs, which is not a reason for two blast radii.
Also corrects two comments that described the containment as absent. It was:
the policy selected `app: code-exec` in namespace `hanzo` and matched no pod.
universe now declares one policy over hanzo.ai/box-class in hanzo-boxes, so
saying so is accurate rather than aspirational — which is the property those
comments were written to protect in the first place.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three things were true at once and each hid the next. cmd/boxd was written and
untracked, so no release could contain it. Cloud's Dockerfile built /cloud and
/smoke and nothing else, so committing boxd would still not have put it in an
image. And apps/sandbox had a Mount with no plugin/sandbox beside it, so the
package compiled, linked into nothing, and answered 404 in production while the
code to answer sat in the tree.
That is why `COPY --from=ghcr.io/hanzoai/cloud:<pin> /boxd` in hanzo/bot's
Dockerfile.box was red by construction — grep -c boxd Dockerfile was 0 — and why
every consumer already pointing at the executor (hanzo.app's ProjectFs,
apps/exec's upstream, apps/functions' invoke) had nothing behind it. The
33-day-old code-exec Service with <none> endpoints is the same fact seen from
the cluster.
boxd builds HERE rather than in hanzo/bot because its types ARE
apps/sandbox/wire's types: the scheduler and the daemon agree because they
compile against one declaration, not because two repos were kept in sync by
hand. It is copied into the FINAL layer, not just the build stage — a binary
that exists only in the builder is not in the published image, and the box image
reads the published one.
boxd is also the half that runs untrusted code, so it refuses to start in the
configuration where that code can steal the pool key: submitted work running as
boxd's own uid can read CODE_EXEC_API_KEY out of /proc/<pid>/environ, and that
key is shared across the pool, so the credential a box hands its own workload is
the credential that opens every other tenant's box. Reproduced end to end. Two
cheaper defenses do not work and pgroup_unix.go records why measurements, not
reasoning: scrubbing the child's env misses that the child reads boxd's, and
os.Unsetenv misses that the kernel serves /proc/environ from the exec-time stack
block rather than the live environment.
.gitignore takes boxd in both spellings. It is the one binary people build from
inside its own directory, since it runs standalone on a laptop, and that file
already carries what committing binaries cost this repo once.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Closes the last two surfaces where a client could name the usage idempotency ref
(the ai answer surface and the domain register/renew/transfer refs) so Usage.Seal()
mints them, and backfills the program column on pre-existing finance.usage rows so a
ref straddling the deploy debits once. R7 asserts no plane op carries metering.Usage;
R8 routes the split-deploy debit over the plane, not the tag-stripped HTTP wire.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`make describe` refuses to regenerate openapi.yaml while any operation is bare,
so the artifact could not be rebuilt AT ALL and had drifted: on clean main the
host served 1698 paths against the golden's 1696 and
cmd/cloud.TestTheServedDocumentIsTheArtifact was red.
128 of the bare operations were commerce's, and commerce registers its routes
from an embedded module, so there is no doc comment here for zipdoc to lift —
openapi.Describe beside the route is the seam. Nine are written out: the wire
top-up rail, the crypto custody rail, the saved-card family, and the tenant's
payment-rail toggle. The other 119 are seventeen merchant kinds behind ONE
generic REST scaffold, so the mechanics are written once and composed with the
kind. Seventeen hand-copied paragraphs describing one generator is the drift
DescribeRest already exists to prevent one level down.
Every sentence is written from the handlers, and what earns space is what a
caller gets wrong:
- PUT is a true REPLACEMENT — the body is decoded onto a FRESH entity, so a
field the body omits is written back as its zero value;
- POST /<kind>/{id} with NO override is a PARTIAL UPDATE, never a create;
- a wallet read renders the account's ENCRYPTED key blob and its salt, so
whoever may read one can attack it offline down to the owner's passphrase —
which is the reason the kind is admin-gated;
- a webhook's delivery consults neither `enabled` nor `live`, so enabled=false
does not stop delivery; deleting the row is what does;
- a discount is enabled by DEFAULT, so a bare create makes a live discount;
- the per-kind permission table covers 5 of the 17 kinds. On the other 12 the
scaffold logs that it is skipping the check and ALLOWS, so the route gate is
the whole authorization story. Each kind now says which it is.
THREE defects were hiding BEHIND that refusal, because describe-apps stops at
the first app that fails and commerce sorts early. Each is repaired, not
recorded:
- integrations: GET /v1/integrations/slack/install was bare. The handler
already carries the prose; it is a raw route, so zipdoc cannot lift it.
- the weave refused two schema names that meant two things. `Role` was iam's
role ENTITY and framework's (user, role) GRANT — the grant is now
RoleAssignment, converted at the handler boundary with the engine type
untouched. `Application` was iam's OAuth client and crm's startup-program
submission — the latter is now ProgramApplication.
- manifest: /v1/event.js, the hosted analytics tag, is served by analytics and
was routed to ai's bare "/v1". A prefix owns SEGMENTS, so "/v1/event" never
covered it, and a browser reads that 404 as a broken script tag rather than
as a routing mistake. Claimed on the analytics row.
All three predate this change and are provable on clean main: regenerating the
stale plugin/iam/openapi.json alone makes the weave fail the same way.
The regenerated zipdoc_gen.go files are the same class of staleness — prose that
was in the Go source and had never reached the artifact.
make describe exits 0. openapi.yaml carries 1735 paths, up from 1696, served
byte-identical to the golden. go build ./... is clean. The pre-existing red
tests (apps/commerce TestBalanceCents/TestInProcessClient, apps/framework's
three, apps/projects TestForkCreatesProjectFromTemplate) fail identically before
and after.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestADisposalIsIdempotent failed the release gate on a 409 — "a materialisation
of this dataset is running" — for a materialisation that had already finished.
The two facts land at different instants. plane.run publishes the version with
p.record and gives the tenant's scan slot back from a DEFERRED p.release, so
between the status going terminal and the slot clearing describe answers
status: ready, running: true. settled returned on the status alone, inside that
window, and dispose refuses a held slot — so the next call 409'd on a job that
was over.
The CI log says exactly this: the FAIL is printed BEFORE the 'dataset
materialised' line for the same dataset, and that line is logged AFTER
p.record — so the version was published, the test read it, disposed, and was
refused by a slot whose owner had not yet returned.
It is microseconds wide, which is why it passes on an idle machine and failed on
a loaded runner. running is already on the wire for this (typed.go view), so
settled now waits on the state the refusal is made from instead of a proxy for
it. No production behaviour changes: a client polling describe was always told
running: true, and always had to wait for it.
30 consecutive runs green under CPU contention.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every fact the plane files under the error signal fans out, detached and
fail-soft, to an error sink apps/o11y installs when its embedded runtime is
up: analytics.ErrorEvent -> the Sentry wire -> o11y's normalize+fingerprint
-> Modules.Sentry.Ingest, landing in o11y_sentry_events + the o11y_issues
lifecycle under the org's canonical project (get-or-create, cached; the org
UUID is the read side's own UUIDv5 formula, pinned by TestDeriveOrgUUID).
So an error a product beacons to /v1/event surfaces on sentry.hanzo.ai
beside the errors a Sentry SDK posts to /v1/event/{project}/envelope.
Re-expressed on the trunk where it moved past the branch:
- AddErrorSink returns a remover, the same plural-sink discipline as
AddSink; o11y detaches it first in ShutdownO11y.
- The filter is routeOf — the plane's ONE routing rule — so the lens can
never carry a fact the plane filed as something else; the branch's
bespoke isErrorEvent detector goes away.
- The envelope's first-class release/environment/service/site/trace
qualifiers win over the legacy property spellings; service_name tags
the wire's service when it named one, else the emitting surface.
- The branch's pk_ mint-door flow test is superseded: IAM mints pk- at
POST /v1/keys and capture_keyorg_test covers the resolver seam.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This reverts 2f9ffc2d. The diagnosis under it was wrong and the change bought
nothing, so it should not sit in the release lane implying otherwise.
What the evidence actually shows: runs 889, 890, 891 and 892 all report
started_at = 1970-01-01 — they never received a runner at all. They were
superseded while still QUEUED, not killed mid-gate, so cancel-in-progress was
never the thing acting on them and a literal false could not have saved them.
Superseding a queued run with a newer commit is moreover the RIGHT behaviour for
a release lane; the newest commit is the one worth shipping.
The real constraint is capacity against push rate: the ten runners of
hanzo-build-linux-amd64 serve five orgs and were 10/10 busy throughout, while
main took a push every few minutes. A run that waits longer for a runner than
the interval between pushes can never start. That is a fleet-sizing and
scheduling question, not a line in this file.
Reverted rather than reworded because the setting is only honest as the
expression that states the intent; a comment describing a mechanism I could not
reproduce is a worse landmine than the bug it claimed to fix.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
X-Actor-Id, X-Act-As-Project, X-Act-As-Org and X-CSRF-Token are stamped on every
signed-in console call (client.ts baseHeaders + applyCsrfToInit) and none of them
were in corsAllowHeaders. A header absent from that list fails PREFLIGHT: the
browser reports an opaque "TypeError: Failed to fetch" and the request never
reaches a server log, so there is nothing to find on our side. Measured from a
signed-in console.hanzo.ai page: adding any ONE of the four blocked the call,
the identical request without it returned 200.
Naming a header only lets the browser SEND it. Each stays exactly as trustworthy
as before — SanitizeIdentity still strips and re-mints client-supplied identity,
so an intent is validated, never believed.
Taken as a one-hunk cherry-pick rather than by merging PR #385: that branch is
443 files and 49,500 insertions off an ancient base, and merging it would drop
~17 commerce prefixes from manifest/apps.go and revert usage_coresident.go's
FloorMinor back to RoundMinor.
NOTE: this constant's own doc says it mirrors hanzoai/gateway routes.go
corsPreflightMiddleware so the browser contract is byte-identical through either
path. The gateway needs the same four or the mirror is broken.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
o11y probed visor.hanzo.svc:19000/health. visor serves its health as a typed op
at /v1/health, and everything outside /v1/ falls through to a static passthrough
that answers 200 with index.html. So the probe was green whatever the service was
doing -- it measured the file server, not visor.
This is the same bug found twice tonight in two repos. visor's OWN liveness and
readiness probes requested /api/health, also not a route, also served the SPA at
200, so a pod with a dead database stayed in the Service. One bug twice is a
pattern, and a pattern earns a gate rather than a second one-line fix.
The gate is narrow on purpose. It cannot know which path a given service serves
-- that is the service's business -- so it refuses only what is provably wrong:
a probe URL with no path, and a bare /health on a host known to answer under
/v1/. Anything it cannot prove wrong it permits, because a gate that guesses is a
gate someone disables.
Mutation: revert the probe to /health and it goes red with
visor: probes "/health" on visor.hanzo.svc, which serves under /v1/. Outside
/v1/ the request falls to the static passthrough and answers 200 with HTML, so
this probe is green whatever the service is doing.
Restored, green. The first run of that mutation reported "clean=1" and I nearly
read it as a pass -- the package was not compiling, because the test named the
target slice `probes` and it is `fleetTargets`. A package that fails to BUILD
reports zero test names, which is exactly why go vet answers "does this build"
and a test-name diff does not.
concurrency.cancel-in-progress was ${{ github.event_name == 'pull_request' }},
which on a push is supposed to be false. The forge does not evaluate it to a
boolean — it cancelled every push run regardless, so the setting written to keep
releases from being killed midway was itself killing them.
Measured on main this morning: runs 883-890 all CANCELLED, each by the next push
arriving seconds later, and 27 consecutive runs produced no release. The gate was
not merely red; even once green it could not have finished, because a repo pushed
more often than its gate takes to run never reaches the end of one. That is the
whole reason a feature merged yesterday is still 404 in production.
A literal false is read correctly and restores the behaviour the comment already
described: ONE release at a time, queued, never cancelled. The image is pushed
early and the tag and pin come last, so a killed run leaves an orphan image — the
ten between v1.801.335 and v1.801.350 are what that looks like.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The five branch commits join the trunk. Their event plane — the fact, the
stream, the writer, the /v1/event door — already lives here in evolved form
(one event.fact table discriminated by signal; the door table in event.go;
/v1/insights/e retired), so content conflicts resolve to the trunk
throughout, and the branch's two proofs land re-expressed against it:
- manifest/shadow_test.go: a prefix belongs to exactly ONE app. The router
merges identical patterns silently (mutation-checked: a duplicated
prefix passes TestAppsMountWithoutConflict), so the table is where the
invariant must hold. The branch's two recorded collisions are both
fixed on this side (storage's deeper /v1/s3 leaves; zen's Gates), so
the gate lands with no exception list.
- apps/analytics/signal_live_test.go: one batch, four signals, four rows —
act/error/log/span each land exactly one row under its own signal, with
the columns their reads sort and group by, sharing one trace.
The branch's fleet-wide zipdoc/openapi regeneration, its o11y receiver
ports, and its telemetry rungs are superseded by the trunk's plane sink,
typed ops and generated surfaces, and resolve to the trunk.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The image job skipped the smoke on `resumed`, and `resumed` was read from the
registry. So a run that PUSHED and then FAILED the smoke came back, found its own
bytes under its own version, declared itself resumed, and skipped the build AND
the proof — handing rollout an image nothing had ever booted, with every job
green. One flag was answering two questions and only one of them had an authority.
An artifact existing proves it was built. It never proves it was tested.
`built` stays the registry's answer and decides the build alone. `smoked` is a
separate fact with a separate authority: refs/smoked/sha256-<digest> at the
commit, created by the same compare-and-swap the version claim already uses — one
mechanism for recording a fact, used twice. Kept in git rather than beside the
image because the registry is the thing being attested and ns hanzo-build alone
had five ways to write it; an attestation stored inside what it attests can be
written by anyone who can write that.
Keyed by DIGEST, never by version. A receipt naming a version proves a version
was smoked and says nothing about which bytes that name resolves to now.
The smoke boots repo@digest instead of repo:tag for the same reason, and rollout
now refuses to ship when the digest pin.sh wrote is not the digest this run
smoked — a values file whose tag moved and whose digest did not deploys the old
image and reports success.
Measured on the real conditions, before vs after, at the state that caused it
(bytes exist, never smoked): before SKIPPED the smoke, after RUNS it. The two
agree on every other state.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The merge that brought the github lineage onto the forge one took the SOURCE of
several ops — agents' session count, affiliates' royaltyFailures among them —
without the artifacts derived from it, so the drift gate read the tree as a
surface that had changed and a document that had not.
Nothing here is hand-written: zipdoc_gen.go for seven apps, affiliates' subset
and the weave, all regenerated from the source already on main. A generated file
committed stale is the same defect as one never generated, and it fails the gate
in the same place for both.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two lineages each justified the same exemption, and the merge kept both entries.
A duplicate key in a map literal is a COMPILE error, so the root package's tests
did not run at all — go-unit reported "[build failed]" for the one package whose
job is to hold this gate, which is the failure mode the gate exists to prevent.
Kept the entry that states the mechanism (screen.op / seen, what comes off the
decoded In versus the request, and that a call with no request resolves no payer
and is screened as that state rather than exempted from it), and folded in the
one fact only the other carried: the request is parked by the app-wide Bridge,
which is why the same resolver works on the /mcp leg and the browser's alike.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/billing/topup was mounted on commerce and named nowhere in this table, so the
fleet published the path and handed it to ai's bare "/v1" remainder — the route
of last resort. The prepaid balance gate on that row would have made topping up
require the balance the top-up exists to create: the exact trap the /v1/cart
paragraph above describes, this time on the door that funds the account.
It was invisible until the operation was described, because router_test.go's
oracle compares PUBLISHED paths against what the router hands each app, and an
undocumented route publishes nothing to compare. The document and the table are
each other's check; a route missing from one hides a defect in the other.
The stem /v1/billing/topup replaces the leaf /v1/billing/topup/token rather than
joining it: commerce serves both top-up doors, a prefix owns its whole subtree,
and the subtree has no other claimant.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
@hanzo answered every message with "the agent hit an error handling that".
Cause, from the log: `cloud: app is not deployed here: agents`.
bridgeReply called agents.RunOnBehalf DIRECTLY, and that function gates on the
agents package's `mounted` global. A PLUGIN IS A PROCESS: a package global is
per-process, so `mounted` is nil on the bridge's side unless agents happens to
be in the same binary. The in-process seam was the ONLY door, which made
co-residency an undeclared requirement — and integrations and agents are
separate plugins in every real deployment, so every chat bridge failed
identically, for Slack, Discord, Teams and Telegram alike.
The turn now travels as a typed plane op over ZAP/UDS:
plane.AgentsRunOnBehalf + RunOnBehalfIn/RunOnBehalfOut (plane/plane.go)
POST /agents/run-on-behalf (apps/agents/onbehalf_rpc.go)
plane.Ask from the bridge (apps/integrations/bridge.go)
Both doors run the SAME runOnBehalf, so org isolation, linked-subject
attribution and billing are identical whichever way the call arrived —
sessions_rpc.go's "two doors, one teardown" shape, applied to the run.
The bridge no longer imports apps/agents at all: reaching into another plugin's
package WAS the coupling, and the dropped import is the proof it is gone. The
plane door lives in its own file so onbehalf.go keeps its promise to know
nothing of zip.Ctx or the wire. Status travels explicitly rather than being
inferred from a non-empty Output — "ran and had nothing to say" and "failed"
are different answers and a bridge must not post the second as the first. An
empty subject is refused, never defaulted to the org: a turn that lost its
caller must not bill the tenant for an unattributable act.
ALSO: a DM is answered INLINE. Threading every DM reply buried a one-line
answer behind a "1 reply" click; a channel still threads because the reply
shares the room, and a DM the user deliberately threaded is honoured.
Tests: DM inline, explicit DM thread honoured, channel mention still threads;
the existing route test now STATES the inline intent rather than the old
behaviour. apps/integrations and plane suites green. apps/agents'
TestTargetOpsProjectEverywhere fails identically on unmodified main (op-id
naming drift) — verified by stashing tracked AND untracked.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/billing/topup and /v1/integrations/slack/install were both registered on the
router and described nowhere, so the drift gate stopped: an operation that says
nothing about itself publishes an operationId and no sentence, which is an SDK
method that cannot explain itself and an MCP tool a model cannot pick. Neither
route was reachable from any generated client.
Both are raw by nature, so both state themselves with openapi.Describe rather
than a lifted doc comment: topup is charged and redirected, install answers a
302. The prose is written from the handlers — the gate each one gets, the tenant
scope, and the one rule a caller would otherwise get wrong: for topup that
paymentMethodId is NOT covered by the billing-subject pin and a card owned by
another subject answers 404 rather than 403, because 403 would be an ownership
oracle over other people's cards; for install that it is anonymous ON PURPOSE
and the org is resolved at the provider callback like every other install.
Regenerated: openapi.yaml 1751 -> 1753 paths, 2509 -> 2511 operations, and the
per-app subsets beside them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The forge is what builds cloud (the build job clones git.hanzo.ai/hanzoai/cloud;
hanzo-inc/cloud has no build workflow at all, only Dependabot). So work landing
on GitHub main was shipping nowhere. 133 forge commits vs 52 GitHub commits off
merge-base 1720eb20, 15 conflicts.
Resolutions, each decided rather than defaulted:
- Dockerfile: ARG CONSOLE_IMAGE deleted. Deleting it IS the change — the console
is a published site now, not an embedded image.
- apps/billing/gpu_charge.go: delete kept. The forge removed the bespoke prepay
charge path on purpose ("GPU is metered like any resource"); GitHub modified a
file whose reason for existing was gone.
- go.mod: higher version wins both ways (commerce v1.50.12, plans v1.4.14).
Forward, never backward.
- manifest/apps.go: forge kept, after PROVING it a superset — GitHub contributed
no prefix the forge lacks, the forge adds /v1 and /v1/cart.
- apps/commerce/mount.go: GitHub's comment, because it CORRECTS the forge's. The
forge said byte-identical patterns "merge silently, first wins"; measured
against the pinned zip they do not — the composer refuses the program and names
every conflicting pair. Plus the forge's POST /v1/billing/topup route, which
GitHub lacks.
- apps/affiliates: the forge's typed accrualsOut, given the field GitHub's
untyped map had and the typed shape did not. royaltyErrs was already counted
and then dropped on the floor — the same silence the typed leg exists to end.
- slack_events.go: the forge's fuller comment plus GitHub's app_home_opened arm.
- plane/plane.go: both sides had independently added an identical
ProjectsOwnership constant. Kept the one grouped with ProjectsResolveKey.
- go.sum regenerated by tidy, never hand-merged.
Two defects the merge exposed and this fixes: duplicate imports in commerce
mount.go, and affiliates calling s.Log.Warn where its two siblings in the same
loop call o.s.Log.Warn.
Carries the front-door body-limit fix (internal/edge): cmd/cloud built its app
with no BodyLimit, so the door enforced fasthttp's 4 MiB while the pod's
GATEWAY_BODY_LIMIT read 100 MiB. Measured on api.hanzo.ai: 4,194,304 bytes
passed, 4,194,305 answered 400.
go build ./... exits 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The bespoke prepay endpoints existed on both sides of this seam: cloud served
/v1/billing/gpu/{charge,eligibility}, and commerce served the handlers behind
them. Cloud's went in the previous commit; v1.50.13 is the commerce release that
deletes ChargeGPU and GPUChargeEligibility, so the module this binary links no
longer carries the path at all.
Nothing here referenced those two handlers — cloud names commerce's billing
handlers one by one and never called RegisterHandlers, which is why the route
was already gone from this binary. The bump is what makes that true of the
module graph as well, rather than leaving a deleted door still compiled in.
The ledger rule is untouched: billing/bucket still classifies a gpu-tagged
withdrawal as prepaid and never a grant, for every writer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two agents built apps/sandbox concurrently in the same working tree and each
overwrote the other's files. What survived was halves: proxy.go from one,
client.go from the other, both declaring boxPath with different signatures, and
the package did not build at all.
Resolved to ONE implementation. sandbox.go wires forward() from proxy.go and
nothing referenced client.go's httpBox transport, so client.go was the orphan
and is deleted rather than kept as a second way to reach a box.
proxy.go is now zip-native. It had been written against Fiber's context —
c.QueryString, c.BodyStream, c.Get, c.Set, c.Response — none of which exist on
zip.Ctx. zip offers two escape hatches that would have made it compile,
AdaptNetHTTP and c.Fiber(), and both are declined on purpose: either one makes a
second way to write a handler in a codebase whose whole premise is that there is
one. The real surface does the job — c.Query, c.Body, c.Header, c.SetHeader,
c.Status, c.SendStream.
The query forwarding is the part that changed shape rather than syntax. Copying a
raw query string blindly would make this an opaque tunnel that cannot tell a real
parameter from a smuggled one. wire.QueryParams now names the five the box
surface actually reads (path, depth, q, regex, limit), declared once where both
ends can see it, and the proxy forwards exactly those.
Response streaming is preserved and matters: SendStream rather than a buffered
read, because a `pnpm install` streams for minutes and the caller should watch it
arrive.
Also s.State.stores.Get -> .For; cloud.OrgStore has no Get.
NOT DONE, and it should not look done: harness_test.go and tenancy_test.go were
written against the OTHER implementation and reference stateWarm, classDesktop
and pool.nodeTaint, none of which exist here. The package BUILDS but `go test`
does not compile. Those two files need porting to this implementation — they are
kept rather than deleted because tenancy isolation is exactly the property worth
testing, and deleting the test would be the worst way to make the suite green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three console cards were dark for three unrelated reasons. Two of them are here.
METRICS — /v1/billing/usage answered 502 "billing upstream unreachable" with
nothing upstream involved. usage_coresident.go asked plane.Money.Minor() for
cents, and Minor() REFUSES an amount finer than a cent rather than round behind
the caller. A per-token AI charge is routinely finer than a cent, so one such row
failed the whole page; usage() tests err before coResident, so the refusal
surfaced as a dead upstream. This is the 2026-08-03 balance bug exactly —
balance.go and apps/ai were converted to FloorMinor then and this row was missed.
Measured on the live fleet, same org, same ledger, same process:
GET /v1/billing/balance -> 200 (FloorMinor)
GET /v1/billing/usage -> 502 (Minor)
Only the call differed. The row now also carries the EXACT amount through, so the
envelope's `decimal` is populated on this branch too — fixing the 502 without it
would have traded a visible error for a silently understated bill.
FLEET — knownServices, a hand-kept list of 26 k8s workloads, was the only door
into resolveService. manifest.Apps lists 119 routed apps. The overlap was TWELVE.
The other 107 answered honest-empty on status, product metrics and scoped logs —
not because their telemetry was missing (TracingMiddleware has been stamping
http.route on every one of their request spans into event.span all along) but
because this package had never heard of them. The set is now DERIVED from the
manifest the host already builds its router from, so a new plugin gets
metrics/logs/status the moment it has a row, which it must have to be routable at
all. 117 of 119 now scope. The two that do not are the two that cannot:
zen is Coresident — it routes no prefix of its own.
ai claims the API ROOT — it routes everything nobody else claimed.
Neither is a product boundary, for mirror-image reasons.
Routes come from the manifest rather than "/v1/"+name because for 48 apps that
convention names a subtree nobody serves (plan serves /v1/plans, storage serves
/v1/s3/buckets, account serves /v1/orgs and five more). Scoping RED to a path with
no spans returns zero, which reads as a healthy idle service — an answer, and
wrong.
Longest prefix wins, as the router already decided it. manifest.OwnerOf warns that
"anything deciding policy from a bare HasPrefix scan will attribute those paths to
the wrong app", and a RED query decides policy: admin owns /v1/admin and seven
apps live under it. Eleven apps need this; for the other hundred the exclusion set
is empty and costs nothing. Measured against six hours of live spans: a bounded
app with seven exclusions is 4.3s against a ~5s baseline, i.e. free — while
expressing ai's fallback as "/v1 minus 251 nested prefixes" costs 15.5s to compute
a number that would have counted KMS's 161,705 requests as inference. That is the
second reason the fallback has no route scope.
Addresses still come from probes.go: an app the fleet does not probe has no URL
and is not probed, which stays the honest outcome.
No new routes and no new Owns* flag. A health-shaped auto-registration loop would
have declared /v1/<name>/{metrics,logs,status} on top of 15 addresses 13 apps
already serve for unrelated business reasons, and zip refuses a twice-declared
address by crash-looping the binary.
Tests fail when they find nothing: the fleet gates guard their iteration source
with a magnitude floor, assert both directions, pin the two exemptions at exactly
one each, and refuse the vacuous run. Verified by reverting each fix and watching
them fail — the sub-cent test with the exact production error, the fleet gate with
106 unresolved apps, the attribution gate with ai swallowing /v1.
Also drops a comment citing query.go as the pin to follow. That file is gone, and
its removal is what broke the console's Logs card.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Visor typed its agent-binding routes (hanzoai/visor 575fc60), which drops the
casibase envelope with them: the answer IS the value, the status IS the outcome.
Nothing here could have kept working — an AgentBinding carries its own `status`
field, so the envelope reader takes "Pending" for an envelope status, decides
the upstream failed, and answers 502. A read of an unbound machine and every
unbind would have gone the same way, on a 204 with no body to decode.
So the client says which wire it is reading, per call site:
cl.call the casibase {status,msg,data} envelope at HTTP 200 (23 sites)
cl.op a typed op: the value, 204 for void, 404 for a miss (9 sites)
Not a fallback and neither sniffs — which wire an upstream op speaks is a
property of that op, and the two cannot be told apart by looking. They share
`do`, so there is still one request builder, one identity rule, one status map.
`call` shrinks to zero as visor finishes and goes with the last noun.
The addresses converge rather than translate. This package already published
/v1/machines/agents and /v1/machines/:id/agent to its own callers while spelling
vm's side three other ways; vm answers on ours now, so the translation is gone
instead of moved. The owning org travels once, as ?owner — vm derives the
binding's org from the same resolved principal, so the body field repeating it
is gone too.
An upstream 404 is a FACT about the machine (it runs no bot), not a fault:
notFound() recognises it so getAgent answers in its own words and messageBot
keeps its 400 — "no bound agent to message" tells a caller what to do, where a
relayed 404 says only that some lookup missed.
The fake vm in bots_http_test.go now speaks both wires, written to match what
visor's own controllers/agent_wire_test.go asserts against the real handlers —
a fake agreeing only with this client would prove the two agree with each other
and nothing about the service.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/billing/gpu/charge and /v1/billing/gpu/eligibility were a second,
product-specific way to move an org's money. They duplicated the general
metered path and nothing called them: no first-party caller exists for either
address, for gpuCharge/gpuEligibility, or for the commerce handlers behind them.
A machine is launched through /v1/machines, which fronts the compute provider's
resell endpoint. That path already owns both halves this one hardcoded: it
authorizes the first launch hour against the org's balance BEFORE provisioning
and refuses when the funds are not there, then records the usage keyed on the
machine id the provider minted, and an hourly sweep meters it from there. One
meter bills every resource, so a GPU needs no door of its own.
Deleted with it: proxyGPUCharge and the commerceProxy.post it was the only
caller of, pinSubjectBody, and the manifest prefixes, retired-name mappings and
published operations that named either address. The floor drops by the two
operations that left, so the reduction is reviewed next to its reason.
The ledger rule the endpoint was built to protect is untouched and lives where
it always did: a gpu-tagged withdrawal draws prepaid, never a credit grant,
because billing/bucket classifies it that way for every writer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/integrations/slack/install landed as a route but not as a DECLARATION,
and TestSurfaceIsRegistered compares the live router against the union of
typedOps and rawRoutes: 42 live, 41 declared. The gate did exactly its job —
a route nobody recorded a decision about is the thing it exists to catch.
It belongs in rawRoutes rather than typedOps for the reason already written at
the top of that map: a 302 is not a JSON body, so it cannot be a typed op. It
sits with the other redirect legs it is a sibling of.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzo.yml states the invariant as "main push -> build -> smoke -> tag -> pin ->
prove live", and every job in cicd.yml upholds its own link. Nothing upheld the
chain itself: each job can only see the release it is running, so an image
published by a lane that never enters the workflow is invisible to all of them.
That is how 82 versions came to be published with no tag in any repo. Seven of
them are recent — 478, 479, 480, 481, 482, 483, 484 — and 480 served
api.hanzo.ai while reporting {"revision":"unknown"} on its own health endpoint.
485 was published untagged while this was being written, which is the point: the
hole is open now, not in the log.
The lane is not a broken gate, it is a bypassed one. 484 and 485 came from
Jobs applied straight to ns hanzo-build (build-cloud-slackinstall,
build-cloud-slackfix) whose buildctl line reads
--opt=context=https://github.com/hanzo-inc/cloud.git#refs/heads/main
--opt=build-arg:VERSION=v1.801.485
--output=type=image,name=ghcr.io/hanzoai/cloud:v1.801.485,push=true
No build-arg:REVISION, so the Dockerfile's `unknown` default wins and the binary
can never name its commit. No version claim, so the number is taken rather than
owned. No tag, because nothing in that command mints one. A context of
refs/heads/main is not even a fixed input — it resolves at fetch time, so the
image is unreproducible as well as untraceable.
So the check is lane-agnostic on purpose. It does not ask who built an image or
whether some workflow succeeded; it asks the registry what is published and the
remotes what is tagged, and any published version that is neither tagged nor
recorded is red. A gate phrased in terms of a lane can only catch that lane, and
the lane that caused this was the one nobody thought to instrument.
It runs at the end of receipt rather than in rollout: a pre-existing orphan must
not be able to block a deploy, because a gate that wedges production on someone
else's old mess is switched off within a day. It is on the main path rather than
a cron for the same reason — a scheduled reconciliation can stop running and
look identical to a clean fleet.
Tags are read from BOTH git.hanzo.ai and github.com/hanzoai/cloud and the union
counts. cloud is canonical on the forge, but cicd.yml claims its version through
the GitHub refs API, so receipts genuinely live in two namespaces; asking only
one would paint every release red from the other. Two of the recorded entries
(319, 340) look tagged from a checkout and are not — the tag exists in one local
clone and was never pushed. Reading `git tag` instead of the remotes is how they
stayed invisible, and it is why the record is derived from the remotes.
None of the 82 were tagged retroactively. Fifteen carry a real commit in
org.opencontainers.image.revision and several name a commit on main, so they
could have been. Here a v* tag is a RECEIPT, minted only after build and smoke
pass; minting one now would assert that a pipeline proved something it never
ran, trading a gap everyone can see for a false claim nobody can detect later.
The image -> commit link those fifteen do have is recorded in column two, which
costs nothing and claims nothing. The other 67 report revision=unknown and no
mapping is guessed for them.
The tests are the mutation: same registry listing every time, and only the tag
list moves. Untagged is red with the refusal quoted verbatim, tagged is green,
recorded is green, and an unrecorded orphan beside a recorded one is still red.
An unreadable source exits 2 rather than 0 or 1, because "could not look" and
"nothing found" demand opposite handling and collapsing them is how a verifier
comes to pass by accident.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Upstream /v1/k8s/nodes is now a typed zip op, so it answers {"nodes":[...]} with
no {status,msg,data} around it. Both consumers move together: listK8sNodes and
managedMachines' third source.
client.call is split at the seam that actually exists. do issues the request —
URL, forwarded identity, credential, body limit, transport errors, one path for
everything. call reads an ENVELOPE (visor's untyped controller routes). op reads
a typed op's Out. Two readings because visor really serves two shapes right now;
call and envelope shrink as routes are typed and go with the last one.
The two readings must not be merged, and that is the trap this closes: decoding
an envelope into visorNodes does not fail — the keys are simply unknown — so
Nodes stays nil and an operator running eight clusters is told, with a 200, that
they have no worker nodes. The op always writes the key, so nil means "this Visor
does not serve this op": listK8sNodes answers 502 and the fleet fold drops that
one source and logs why, leaving the registry and live-droplet sources intact.
TestK8sNodesRefusesTheOldEnvelope and TestMachinesDropDOKSNodesOnSkew pin both
halves against a fake that speaks the pre-typed wire. Without the check the first
serves `200 {"nodes":[]}`.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cmd/cloud built its app as zip.New(zip.Config{AppName: "cloud", MCP: ...}) with
no ReadBufferSize and no BodyLimit. It is the FRONT DOOR -- every public request
reaches it before the program behind it -- so it terminated HTTP on fasthttp's
defaults: 4 KiB headers, 4 MiB bodies. cloud.App() set both correctly for the
program, and none of it could be reached past the door.
Measured on api.hanzo.ai: GATEWAY_BODY_LIMIT=104857600 in the running pod's
environment, and 4,194,304 bytes passed while 4,194,305 answered 400. That is
4 MiB exactly. In-cluster, straight to cloud.hanzo.svc:8000, the same 400 --
so ingress and gateway were never involved.
The consequence nobody saw: the 16 MiB default had ALSO never taken effect. Its
docstring says it exists so a 1M-token prompt (~4.3 MB of JSON) can reach the
1M-context models; at 4 MiB those models were unreachable, and fasthttp's wire
error is the opaque 400 "Error when parsing request", which reads like a
malformed payload rather than a size cap. Someone wrote that comment believing
it was fixed.
app.go already says an app is obtained one way and a program never builds a
zip.App itself. The door does, deliberately -- it links zip, manifest and webui
and nothing else, and importing the cloud root to reach a number would re-fuse
the monolith it replaced. So the numbers move to internal/edge, a leaf both can
import. One home per number; a second home is how this broke.
TestTheDoorAcceptsABodyLargerThanTheFrameworkDefault sends a real 4 MiB+1 KiB
body through a real door and fails on 400. It reads the wire, not the config,
because reading the config back is what the old code would also have passed:
the value was computed correctly and never reached the transport. Mutation-
tested -- hardcoding the framework defaults back turns both tests red.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
projectIsForeign answered false for a foreign project-id on every surface that
installed it, so the guard that keeps one org from asserting another's project
was a no-op fleet-wide. The registry it consults lives in the `projects` plugin
and is never mounted in the process where the edge middleware runs, so the
lookup could not succeed anywhere it mattered — measured false against a project
id belonging to another org.
The question "does this project belong to this org" cannot be answered inside
the identity boundary alone, so it becomes a plane op and is asked across the
boundary: plane.ProjectsOwnership, with the same fail-closed posture the rest of
the edge has. A caller that asserts a project it does not own is refused rather
than admitted on a lookup that silently returned false.
Cherry-picked onto the ship branch with its plane op, which is the 33 lines the
guard needs to reach the registry. Both were on the mirror main and not on the
integration main this image builds from, so production carries the hole today.
It ships with the tracker/meet cutover rather than behind it: a live cross-tenant
isolation gap outranks a UI host migration, and one image closes both.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Disabling a model made it invisible to the one caller who must still see it.
The read path delegates existence to the embedded @hanzo/pricing bundle, which
is a stale snapshot of first-party prices. The authority for what exists and
whether it is enabled is the OVERLAY, and commerce backs that. Those two
disagree in exactly one case — a model an admin has disabled — because the
bundle then stops serving it, and its 404 returned before the gate below could
decide. VisibleCatalog already keeps disabled entries for admins
(`if !isAdmin && !visible { continue }`); it simply never ran.
So a 404 from the bundle no longer ends the question: when the overlay knows the
id and the caller is an admin, the admin gets the overlay's answer, annotated
with the same _overlay state the list surface returns.
Measured before changing anything, because the shape invited two wrong guesses.
The op prints, for the admin request: bundle status=404 isAdmin=true. The
identity bridge was fine and the gate was fine — only their ORDER against the
bundle was wrong.
Not a regression this change introduced: apps/pricing was green at fe9bacb0 and
red after, and nothing in this cutover touches it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three defects, each of which fails silently rather than loudly, and all three
found by another agent reading hanzo.chat's client source instead of our own
docs. Verified against ~/work/hanzo/chat before changing anything.
1. POST /v1/upload omitted message:"success". crud.js:108 is
`if (result.message !== 'success') throw` — so EVERY upload failed, with
"Error uploading file: undefined" and nothing on our side to say why.
2. Upload elements were {name,id}. crud.js:112 builds the identifier as
`${result.session_id}/${result.files[0].fileId}`, and the client's own JSDoc
types the element as {fileId, filename}. Our shape produced
"<sid>/undefined": the upload looked fine and the file was unreachable.
3. GET /v1/files/{sid} returned {session_id, files}. Two readers assume a bare
array — ProgrammaticToolCalling guards with Array.isArray and returns [],
and process.js:294 calls response.data.find(...) directly. An object does not
error there; it makes every session read as permanently empty, which is much
harder to notice than a crash.
The three endpoints need three DIFFERENT shapes, which is why one struct could
not serve them: exec returns files:[{name}] (unchanged — newFiles is correct and
"fixing" it to match upload would break it), upload returns {fileId, filename},
and files/{sid} returns bare [{name:"<sid>/<fileId>", lastModified}] because
process.js matches on that prefix and reads that field.
librechat_contract_test.go pins all four shapes with the consumer line number
that reads each one, so a later tidy-up has to argue with hanzo.chat rather than
with a naming preference. Keep it. Credit for the findings goes to the agent
that read the client; only the file was lost to a concurrent-write collision.
cmd/boxd builds and its suite passes, including the four new conformance cases.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TWO defects, both measured in production.
1. EVERY Slack DM answered "Sorry — I couldn't reach your Hanzo account just
now." getUserLink already had the right branch:
if errors.Is(err, kms.ErrSecretNotFound) { return userLink{}, false, nil }
but the live error was the STRING `kms kms_get: kms.get: store: secret not
found`. A plugin is a PROCESS: an error crossing that wire is re-created from
its text, so the sentinel does not survive and errors.Is is false on the far
side even though the store said exactly that. An unlinked user therefore took
the BROKEN branch instead of "not linked yet", and the link prompt that
teaches them how to connect was unreachable — the feature could never be used.
Repaired in kmsGet, the ONE door onto the store, so the sentinel is whole for
every caller above it. Text matching is kept to the store's own phrase: a
broken store must never be laundered into a silent "unlinked".
2. The App Home tab rendered Slack's own "this is still a work in progress"
placeholder, which it shows for any app that enables home_tab_enabled and
never publishes a view. The choice was never Home vs no Home — it was our page
vs Slack's apology. app_home_opened now publishes a view answering what this
can do and what to type. A failed publish is logged and swallowed: a cosmetic
surface must not fail an event Slack would retry into an identical render.
The install route is DECLARED in ops_projection_test's rawRoutes rather than
bumping a count: that test asserts the surface is exactly what someone decided
on, and it caught the addition, which is it working.
Full apps/integrations suite green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every control plane for agentic execution exists; the thing they control does
not. This settles where it lives and what it looks like on the wire.
Four consumers, not three: apps/exec (proxy with no upstream),
apps/functions/invoke.go (same unset env), apps/coding (bot-gateway
/v1/coding-tasks, fail-closed on a docker daemon no pod has), and hanzo.app's
ProjectFs. hanzo.chat's execute_code resolves end to end onto zero endpoints
today.
Decisions: control plane is apps/sandbox in cloud (three of four consumers are
already in this binary; the standalone twin of that same pattern was deleted
last week for being a duplicate). Image is three layered tags in hanzoai/bot,
where the recipe already exists digest-pinned and non-root and has never been
built because bot has no CI at all. boxd is Go in cmd/boxd so its wire types
are one declaration shared with apps/sandbox and drift is not expressible.
Records a live defect proven with real output: apps/exec preserves the request
path (/v1/exec) while apps/functions appends /exec, so no value of
CODE_EXEC_UPSTREAM satisfies both consumers.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Retiring zen4 cost eight red gate runs on one line, because a test that
writes a catalog id down is making a claim about the snapshot and snapshots
retire models. The single-segment id started being read from the mounted
catalog; the SLASHED one did not, and the same literal was spelled a second
time in the enablement flow. The bundle already carries opus-4.7, 4.8 and 5,
so 4.6 ages out the way zen4 did — between two survivors, invisibly.
Both ids are now derived by SHAPE through one reader. A provider-qualified id
is what the greedy wildcard exists for and a single-segment one is what the
ordinary route serves; which id has that shape is the catalog's business.
The read is the PUBLIC view, which also makes the assertions mean more than
they did: the model is baseline-visible before it is disabled, so the
customer's 404 is the visibility gate refusing, not the lookup missing. Those
two answers reach the wire identically, which is why the failure pointed at
the admin gate instead of at the snapshot.
The precondition the route-shape tests rest on — the catalog serves both
shapes — is now stated by a test of its own instead of surfacing as a 404 far
from its cause.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Eight gate runs red on one line: the admin HTTP test hardcoded zen4 (and
zen5), and pricing v1.4.10's regenerated snapshot retired the zen4
generation — so the single-model section 404'd an id that no longer exists
and every push since the catalog bump failed the gate. A literal model id
in a test is a claim about the snapshot, and snapshots retire models; the
single-segment id is now READ from /v1/pricing/models on the mounted
subsystem, so the sections prove the ROUTE SHAPES (greedy wildcard vs
single segment, the visibility gate, the over-deep 400) against whatever
the catalog actually carries.
The allowed browser origins were a static list of domains we own, so the
shipped feature — fork hanzoai/console, deploy it on your own domain — could
not work: no list of our domains can name a customer's domain.
An origin is now admitted from either of two sources. DECLARED is the platform
allowlist an operator writes. PROVEN is a host whose site_hosts row is
verified, which is the DNS-01 proof that subsystem already takes: the claimant
publishes a 128-bit token as TXT at _hanzo-challenge.<host>, fqdn.Verify is
fail-closed, and ResolveHost's status='verified' filter is the existing
hostname-hijack boundary. CORS asks that boundary rather than growing a second
one, so a pending claim on a name grants nothing here either.
The Origin header is chosen by the caller, so the cheap total rules run first
and the store read runs last: exactly scheme://host reconstructed and compared,
https only, no port, and fqdn.Valid — which is also what keeps bare project
slugs out, since site_hosts holds every project's slug as a row that is always
verified and a bare label is not an FQDN. Answers are cached in BOTH
directions and the cache is bounded, because the attacker picks the key.
Two defects fixed on the way. Vary: Origin was assigned, not appended, so it
fought middleware_markdown's Vary: Accept and the loser's protection silently
vanished; and it was set only when the origin was allowed, though the answer
that carries no ACAO depends on Origin just as much. It is now appended on
every origin-dependent answer, including the denial.
And there was a second CORS authority. hanzoai/ai is mounted in-process and
inserts a filter ahead of every /v1 route that 403s any origin outside 21 apex
domains compiled into the module. The preflight short-circuits here and never
reaches it, so a customer console would have passed the preflight and failed
the call — allowed preflight, denied request. Its positive half is already
dead in production (setCorsHeaders returns early on X-Forwarded-Host, which
the ingress always sets), so only the refusal is live; clearing Origin on the
hop into it, for origins this edge already admitted, takes its own origin==""
no-op path. Denied origins are passed through untouched and still 403.
WebSocket upgrades keep their Origin: dev_bridge's CheckOrigin admits an empty
one for CLI clients, so clearing it there would fail open, and a socket's
origin check is a different mechanism from CORS.
The per-IP cap moves ahead of CORS. Resolving an unknown origin is a plane hop
in production, and a caller rotating a fresh hostname per request would defeat
the answer cache and turn each inbound request into an internal one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three parts, all required before Slack will OFFER @hanzo as an agent:
1. assistant:write bot scope (slack.go)
2. assistant_thread_started / _context_changed events (slack_events.go)
3. the Agents & AI Apps toggle in the app config (not code)
Without all three the app answers @mentions and DMs fine but never appears in
Slack's "Add Agents" picker, which is why only Claude was listed.
Both events merely ack, which the default arm already did, so the wire
behaviour is unchanged. They are named explicitly because a reader asking
'does this app support agents' must find the answer here; a silent default
cannot say yes.
GET /v1/integrations/slack/install -> 302 to Slack's consent URL. Slack REFUSES
a slack.com URL in the Direct install URL field and requires one of OURS that
302s to slack.com -- because the field is an ATTRIBUTION hook: routing the
Marketplace click through our address is what lets an install be counted, and
forcing the 302 stops the counter becoming a detour that never reaches consent.
A redirector and nothing more. Same destination every time, built by the SAME
slackAuthorize the console's Connect button uses. It carries no state and binds
no org: the org resolves where it always did, at the generic
/v1/integrations/:provider/callback. Minting a principal for an anonymous click
is what the isolation bar forbids.
Applied here because the BuildKit lane builds github.com/hanzo-inc/cloud#main,
which had NONE of these -- it is 49 commits ahead of the forge while the forge
is 124 ahead of it. Consolidating those is separate work; this is the change.
go build clean; go test -run Slack ok.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The fleet publishes /v1/commerce/{collection,product,wallet,webhook,...} — 17
resource families commerce genuinely serves — and routed every one of them to
`ai`, which owns the bare "/v1" prefix. commerce's manifest row enumerates 30
specific prefixes and none of them reached these, so a prefix deeper than the
sibling that currently wins was simply missing.
They were DARK: published in the document, therefore offered by every generated
SDK, the MCP tool list and the spec-derived CLI, and reaching a different app on
the wire. That is the exact defect the router oracle exists to catch, and it
could not see them until they had prose — an app that cannot project its own
document produces no paths to check, so this gap sat behind the describe failure
rather than behind a passing test.
Naming the 17 prefixes is the fix the oracle asks for, and it is the narrower one:
a bare "/v1/commerce" would capture the whole subtree including addresses other
apps may later claim, while a prefix per family owns exactly what commerce serves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/integrations/slack/install -> 302 to Slack's consent URL.
Slack REFUSES a slack.com URL in the Direct install URL field and requires one
of OURS that answers 302 to slack.com. That is not a hoop: the field is an
ATTRIBUTION hook. Routing the Marketplace / Add-to-Slack click through our own
address is what lets an install be COUNTED, and forcing the 302 is what stops
the counter becoming a detour that never reaches consent. Same destination
every time.
A redirector and nothing more. The consent URL is built by the SAME
slackAuthorize the console's Connect button uses, so there is one source of
truth for what we ask a workspace to grant however the click arrived.
It carries NO state and binds NO org, which is correct rather than a gap: the
org is resolved where it always was — the generic /v1/integrations/:provider/
callback, from the signed state a console connect minted or OrgForExternalID
for a workspace already connected. An install beginning here finishes there
under the same rules as every other install, and that callback already returns
the browser to the console on a labeled failure rather than a JSON dead end.
Minting a principal here would be inventing an org for an anonymous click,
which the isolation bar forbids.
Registered as a LITERAL before the /:provider wildcards (fiber resolves by
registration order) and Terminal like its siblings — the person clicking
Install in Slack's directory has no Hanzo session, so it must not 403.
Unconfigured answers an honest 503 rather than a consent URL with an empty
client_id, which Slack renders as its own error page with no way back.
Tests assert the contract Slack validates: 302 to slack.com/oauth/v2/authorize
carrying client_id, assistant:write and our callback as redirect_uri.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Same fix as billing's lane: the KMS read returns a flat {env, name,
value}, both pin steps indexed .secret.value, so the self-pin reported
UNIVERSE_PIN_TOKEN missing even once it existed. Tolerant read; the
automation identity (forge user 'deploy', write on hanzo/universe, token
sealed at deploy/UNIVERSE_PIN_TOKEN env prod) is live.
The account had four vaulted Square cards and $149k of balance, and no
customer-facing door could charge any card on file: subscribe/card demanded
a fresh nonce per attempt (vaulting one more identical row each time), and
POST /v1/billing/topup — the saved-card top-up commerce already ships, the
same chargeAndCredit core the auto-recharge cron uses — had no co-resident
route, so it fell to the account bridge and died in self-dispatch.
commerce v1.50.12 (this bump) makes subscribe/card accept paymentMethodId,
stamps brand/last4/expiry + fingerprint on vaulted rows, dedupes a re-saved
card into its existing row, heals legacy rows on list, and adds the
ownership guard Topup's body field always needed. The mount here routes
POST /v1/billing/topup with topup/token's exact chain — risk screen
included, both credit the spendable wallet.
TestPlaneWireCarriesNoFieldNames observed the bytes crossing the plane's socket
and proved they were ZAP rather than JSON -- field names absent, values present.
That was right, and zip v1.27.0 made it unobservable: Ask now asks zip.Serving
whether the peer is this process and hands the call to the op's own invoke seam,
so there are no bytes. The test read "no body was observed" and went red on a
change that was correct.
The encoding assertion did not need rescuing, it needed deleting: internal/zapenc
already pins it at the encoder, in TestNothingIsJSON and TestBytesAreAZAPMessage.
Cloud was keeping a second copy of a fact zip owns, at a level that could only
observe it by accident of the transport.
What is true HERE, and was not true before v1.27.0, is that the wire is gone. So
the middleware is still the instrument and an EMPTY observation is now the
result: if a body ever appears, cloud has started serialising a value, handing it
to the kernel, reading it back and parsing a fresh copy, to reach a function
pointer that was in memory the whole time.
Mutation: disable the zip.Serving short-circuit in plane/ask.go and the same call
puts 39 bytes on the socket --
"ZAP\x00\x01...\x00acmeusd"
-- which is both the failure and a demonstration that the frame is still ZAP
carrying values with no field names when a wire is genuinely involved.
zip v1.27 gave plane.Ask a co-residence short-circuit: zip.Serving(app) finds a
peer this process is already serving and zip.Here runs the op through the same
opByName and invoke the socket plane uses, with nothing encoded. That is right,
and it made this file's mutation test assert the wrong thing — it unlinked a
socket the call had stopped using and called the success a failure.
The two halves are now pinned separately, because they are two facts:
co-resident zip.Serving finds iam, the call runs in-process, and unlinking
the socket changes nothing because nothing was going to touch it
separate the socket is the whole dependency: unlink it with the peer alive
and the call fails, rebind and it succeeds
"Separate" is arranged the way zip.Serving actually decides it — the lookup keys
on SocketPath(name), so a caller resolving iam.sock in a different runtime
directory misses the local registration and dials for real. Same arrangement
planewire_probe_test.go uses, and the frames it captures are what says this path
is a wire at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The waitlist gate resolved approval with an HTTP GET to /v1/iam/get-account, and
because HTTP gave it no way to say WHO was asking, it said so by forwarding the
caller's own Cookie and Authorization header — cloud taking a user's raw
credential and presenting it to another service so that service would answer
about that user. It worked. It is the wrong shape twice: the credential is
handled where it need not be, and the answer is only as good as a bearer the
gateway had already validated once.
plane.IAMApproval carries the validated principal instead. iam reads the subject
with cloud.Who, resolves it with GetUserBySubject, and answers the recorded
approvalStatus — raw, judged by nobody. Whether "pending" gates a person stays
in admission next to the gate, so the identity store and the gate can never
disagree about who is on a waitlist.
EnforceConfig.IAMBase goes with it, and it was already dead: nothing in the tree
set it, so the resolver it built had an empty base and the lookup it installed
returned ok=false forever — a gate that had been failing open by configuration
accident rather than by decision. A peer is addressed by name, so there is no
base to supply and no way for one to be absent.
The availability rule is unchanged and now pinned against a real socket rather
than a JSON table: iam SAID pending → gated; iam SAID nothing → approved; iam
could not be ASKED → approved, and that verdict is NOT cached, so a recovered
iam re-gates on the very next request rather than a ttl later.
TestApprovals_NoCredentialCrossesTheWire proves the point on the bytes, not from
source. A recording relay sits at the socket the caller dials; the caller's
secret appears nowhere in the frames and the gateway-asserted subject does. The
structural half is that a plane handler has no header accessor at all, so a
forwarded credential would have nothing to read it.
TestApprovalStatusFromAccount is deleted rather than ported: it parsed IAM's
{status,data,properties} envelope in both its top-level and data-wrapped shapes,
and that envelope had no reader left.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The canonical project read was an HTTP GET to /v1/iam/projects. IAM is a peer —
the pod boots ~25 single-app processes and iam is one of them — so the only
thing the hop needed was a way to say which tenant was asking, and HTTP had
none. Platform built one: a per-org "<org>-platform-kms" IAM application, minted
on first need, sealed into KMS, cached for a minute, and admitted by IAM's
authorize as read-only/own-org-only.
The plane carries the tenant. cloud.For puts the org on the call, the callee
reads it with cloud.Who, and it is not a field a caller can set — so the whole
apparatus was standing in for something the transport now does. A minted secret,
a KMS seal, a cred cache and a grant, all deleted, and the narrowest grant is
the one that was never issued.
The workaround went with it. IAM frames its single-project read as a POST, which
the machine grant rightly refused, so Get was derived from List and the reaper
logged "projects: get hanzo/index: status 403" every cycle — a client tripping
the wall it came through. One op, one verb, nothing to route around.
iam declares the op because iam owns the store (apps/iam/projects_rpc.go), next
to the roster read that is there for the same reason. The projection is five
fields; the identity record stays home.
Proven on the wire, not by reading it: planewire_probe_test.go puts a recording
relay at the socket the caller dials and asserts the capture is ZAP —
ZAP magic, Content-Type: application/zap, addressed by name at
/.well-known/zip/op/iam_projects, X-Org-Id riding the call, and the reply
reading acmewebWebthe site2026-08-01T00:00:00Z
five values with ZERO field names, which is what zapenc's positional layout
means and what a JSON fallback could not look like. Plus the mutation: unlink
the socket with the peer alive and the call fails, rebind and it succeeds.
Co-residence is unchanged and still wins — no IAM_URL means this binary IS the
IAM and iamProjects reads the store directly. A Go call beats any transport.
ServePlane freezes the plane app, so every test here drops it on cleanup;
without that a later Mount panics on a frozen registry, which is what the two
new files sorting before pushbuild_e2e_test.go surfaced.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zz_probe_seam_test.go was a throwaway: it printed what zip.Op and cloud.Request
look like at each of the four doors so the design could be built on measurements
rather than on a reading of the framework. It answered its question -- op.Path is
the same value four times, the request is an envelope on three of them and absent
on the fourth -- and toll_test.go asserts that answer properly, in
TestOperationIsTheSameValueAtEveryDoor, where a regression fails a build instead
of printing a line into a log nobody reads.
It was swept into the tree by an add-all and never meant to land. Deleted rather
than kept "in case", because a test that asserts nothing is a test that goes stale
without anybody noticing it has.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Toll stands down for a request BillingGate has already answered, so an operation
reached over REST is charged once rather than twice. The first version worked out
whether that had happened: if the request's own path names a declared surface then
the edge must have answered for it. That is true, and its truth rests on a line in
a composition root two files away.
Unmount BillingGate and the inference still says yes. The toll still stands down,
nothing charges anything, and both gates are present in the source the whole time.
An invariant spread across two lines nobody reads together is an invariant nobody
is keeping -- which is the same shape as the bug this seam exists to close, where
the gate was real, the coverage was real, and the hole was real, all at once.
So the claim is made rather than deduced. BillingGate parks it at the point it
commits to the charge -- after every decision NOT to charge, because a request it
waved through is one the op seam must still weigh, and over MCP and the plane the
path it read was an envelope and not the operation at all. Toll reads the claim.
Unmount the edge gate and the claim stops being made, which is exactly what makes
the op seam take over.
TestTollTakesOverWhenTheEdgeIsGone is that property, and it is the test the
inference version would have passed. Composed without BillingGate, REST must still
be charged exactly once -- by the toll -- and refused at zero.
Four mutations, four distinct failures:
gate removed MCP + ZAP "balance settled at 25c, want 0c", CLI serves free
stand-down removed REST "books hold 1 deposits and 2 debits, want 1 and 1"
price the REQUEST MCP + ZAP unbilled again -- the original bug, reproduced
claim never made REST double-charges
flow and auto said in prose that cloud sets no zip authorizer and that no
transport but HTTP reaches a money gate anywhere in the fleet. Both were true when
written and neither is now, and an MCP tools/call was never the payer-less case
they grouped it with -- it arrives as POST /mcp and carries a request like any
other. Corrected where it is written, not left for the next reader to disbelieve.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Slack's "Add Agents" picker lists only apps that declare the Agents & AI Apps
surface. Ours declared none of it, so the workspace was offered Claude and
nothing else — not a bug in the bot, which answers @mentions and DMs fine, but
the app was never OFFERED as an agent.
The declaration is three parts and ALL are required:
1. the assistant:write bot scope <- added here (slack.go)
2. assistant_thread_started +
assistant_thread_context_changed events <- added here (slack_events.go)
3. the Agents & AI Apps toggle in the app
config at api.slack.com <- NOT code; owner action
Both events merely ack, which the default arm already did — so the wire
behaviour is unchanged. They are named explicitly because a reader asking
"does this app support agents" must be able to find the answer, and a silent
default cannot say yes; it also gives setSuggestedPrompts an obvious home later.
No reply is needed to open the thread: the user's first message arrives as a
normal message event with channel_type=="im", which the existing arm already
routes to the agent, so conversation works as soon as the app is listed.
Zero runtime risk on existing installs: Slack never grants new scopes to an
existing token, so this changes only the consent URL for a re-install.
Based on forge/main (70a67035) — github.com/hanzoai/cloud main is 4504 commits
behind and does not carry apps/integrations at all.
go build ./apps/integrations/ ok; go test -run Slack ok.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The next agent needs two facts this file did not hold. First, that most of the
paths apps/commerce/transport carries are not registered in this binary at all —
commerce's api.Route() bundle is behind //go:build cloud — so the rollup and the
transaction list dispatch to a 404 and the balance re-enters cloud's own customer
handler and 401s, while a split deploy leaves the base URL empty and every reader
answers a silent zero. Not a degraded path: one that has never worked in either
shape.
Second, the line between what converted and what did not, and WHY. books, usage
and payout (with referrals, affiliates and authors) ask the ledger by name, because
the ledger holds their answer. metering's tier and cap, admin's subscriptions and
costs, storefront's store reads and billing's remaining proxies do not, because
their answer lives in a hanzoai/commerce handler whose logic is in unexported
helpers — so those want the payments.go pattern, a value-taking core exported from
the module, and NOT a second implementation in cloud. Writing a spend verdict twice
is how a cap and a rate limit come to disagree about which requests they bind.
Until those land, maxDepth, the goroutine-id parsing and CLOUD_COMMERCE_HTTP_URL
stay. The file says so, so nobody reads a half-finished conversion as a finished
one.
I changed ListEntries to take the `test` selector every other read on that ledger
already took, and toll_test.go — which landed while I was rebasing — calls it.
The root package stopped COMPILING, and I read the suite by comparing failing
test NAMES against a baseline, which reported no regression: a package that fails
to build reports zero test names, so the comparison could not see it. The name
diff said clean while the package was not compiling at all.
go vet ./... compiles every test file and is what actually answers "does the tree
build". It is the check that belonged in front of the name comparison, and it is
how this was found.
coResidentUsage resolves ListUsage by INTERFACE ASSERTION and treats a miss as
"this ledger cannot list usage" — a fall-through, not an error. So a fake left on
the old signature does not fail to compile; it stops satisfying the interface,
the capability disappears, and the customer's usage view answers from nowhere.
Three tests said ok=false and one said the envelope was not JSON, which is what
that looks like from the outside.
Worth stating because the shape recurs: an optional capability resolved by
assertion turns a signature change into a silent behaviour change. The tests
caught it here. Nothing in the type system would have.
Two service-token GETs — /v1/billing/usage/rollup and /v1/billing/transactions —
sent through the commerce transport, which co-resident dispatches back into this
binary's own router BY PATH. Neither route is registered here (commerce's
api.Route() bundle is behind //go:build cloud), so both were 404s. Split into
per-app binaries the base URL was empty, the reader called itself "not
configured", and the whole spend block degraded to honest zeros. A customer's
usage page showed a blank month while their wallet was being debited all along.
Under that, isSpend matched "withdraw" — a word the ledger has never written; it
writes finance.usage. So even a row that arrived would not have counted, and the
category breakdown and the series would have summed to zero on live data. The
kind crosses as the ledger's own spelling and is parsed once by the one
recognizer, which makes that disagreement a compile error.
rollupWire is gone and rollup replaces it, with two fields where there were four.
There is ONE balance because the ledger has one number: what it calls available
IS the settled balance, and two names for one value is how a reader comes to
subtract one from the other. There is no overage because overage needs a plan
ALLOWANCE and none exists here — this is a prepaid wallet, there is nothing to
exceed, and the field it replaces was read out of a JSON body that never came.
The scope tests moved with the read. They proved isolation by asserting an
X-Org-Id header and a ?user parameter the reader itself set — which proves the
reader agrees with itself. The org rides the CALL now, so they assert what the
LEDGER acted for, and a forged ?user=victim&org=other is not a thing the request
can express.
referrals, affiliates and authors all qualify and accrue on ONE figure: what has
this org spent. They read it through payout.Client, which was an http.Client
aimed at GET /v1/billing/usage/rollup and sent through the commerce transport.
Co-resident, that transport dispatches back into this binary's own router BY
PATH, and /v1/billing/usage/rollup is registered nowhere here — commerce's
api.Route() bundle is behind //go:build cloud and is never compiled in. The read
was a 404 wearing an upstream failure's clothes. Split into per-app binaries it
failed differently and worse: the base URL is empty in every process but
commerce's, so Configured() was false and SpendCents returned a silent ZERO. A
program that pays a commission on spend, told the spend was zero, pays nothing
and reports nothing wrong. Referrals qualified nobody. Affiliates accrued
nothing. Authors were paid nothing.
It asks the ledger by name now. The client holds no address and no credential,
because reaching a peer takes neither — Configured() and ErrUnconfigured named
things that do not exist for a peer, and both are gone. What replaces them is
ErrNoLedger, which is the ROUTER'S word and the only absence a program may act
on; every other failure is an outage and is returned as one. The user argument is gone from
the read: the figure was always the org's, and the rollup's user parameter was
the org's own name being sent back to it.
The tests were green through all of it, because each stood up an httptest.Server
for the wire under test and proved the mock answers — which nobody doubted. They
drive a real per-org encrypted ledger and a real plane socket now, and the
no-mint proof got stronger for the move: the money-IN ops are registered live
beside the read, each failing the test if invoked. A caller cannot reach those by
half-matching a path. It has to name the op.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The depth cap it replaces was never a fix. maxDepth = 8, counted in a map keyed
on a goroutine id parsed out of the header of runtime.Stack, existed because a
co-resident caller entered the ROUTER: the transport dispatched an http.Request
into the whole shared fiber app, every edge middleware ran again, and a
middleware that itself reads commerce read again on a cache it had not yet
filled. A counter measures that recursion; it does not remove it.
A caller that enters the OP cannot recurse into a middleware, because it never
runs one. So the test does not count how deep the nesting goes. It unlinks the
socket the plane serves on, proves nothing can dial it, and shows the nested
read still answering — which no call carried by a transport could do.
The mutation is the other half: point the runtime dir at a directory this
process serves nothing in, so zip.Serving stops answering for commerce, and the
same call FAILS. Point it back and it answers again. Without that, the first
test is a coincidence with good manners.
Measured under strace, whole binary, connect() traced: 6 op handlers ran and
FOUR connect() syscalls occurred, all AF_UNIX to commerce.sock and all four the
test harness's own — three readiness probes waiting for the listener, one after
the unlink to prove the wire was really gone. None of the six calls opened
anything.
Two defects, one cause. books ingested commerce's money through an S2S
GET /v1/billing/transactions sent over the commerce transport, and that
transport dispatches co-resident requests back into this binary's own router by
PATH. /v1/billing/transactions is registered nowhere here — commerce's api.Route()
bundle is behind //go:build cloud and is never compiled in — so the read was a
404 wearing an upstream failure's clothes. Split into per-app binaries it was
worse: the base URL is empty in every process but commerce's, the reader called
itself 'not configured', and the general ledger ingested nothing at all.
Underneath that, the classifier matched 'deposit' and 'withdraw' — spellings the
ledger has never written; it writes finance.deposit and finance.usage. So even a
row that arrived would have classified as nothing and posted nothing. The tests
passed because the fake source spoke the same invented vocabulary. The kind now
crosses as the ledger's own and is parsed once with finance.ParseKind, which
makes that disagreement a compile error.
A refund is not a third kind. The ledger writes two, and a return of unspent
funds is a NEGATIVE deposit — the reversal rule already books it, against the
clearing account the cash arrived through. The refund arm and its Bank leg are
deleted with the word that reached them.
FinanceTxns takes an input now: which books, and how many rows. Sandbox money
and real money live in physically separate files, so the selector has to travel
— a reader that posts test rows into real revenue restates the company's income
and nothing downstream can tell. ListUsage and ListEntries take the same
selector every other read on that ledger already took; they arrived later and
had forgotten it, so a sandbox caller was silently answered from real money.
Six subsystems needed one number — what has this org consumed — and none of
them could get it. Each asked commerce for GET /v1/billing/usage/rollup over
the commerce transport, which co-resident dispatches back into this binary's
router BY PATH. That route is registered nowhere here: commerce's api.Route()
bundle is behind //go:build cloud and never compiled in. So every one of those
reads was a 404 wearing an upstream failure's clothes. Referrals qualified
nobody, affiliates accrued nothing, authors were paid nothing, the usage page
showed a blank month, and the admin fleet read broke — while the figure sat in
a SQLite file in whichever process mounts commerce.
finance_spend answers it there, from the ledger's OWN windowed sum: the same
source the rolling spend cap reads, so a program that qualifies an org on its
spend and the gate that stops that org spending cannot disagree about the
amount. One balance field, because the ledger has one number.
plane.Ask now runs an op that is registered in THIS process directly, through
zip.Serving + zip.Here (zip v1.26.1). A fused binary registers every app's
internal ops on one plane app, so commerce asking commerce for a balance was a
function call wearing an address; dialing our own socket for it encoded a value
we already held and parsed it back into a copy. The decision is made once, in
the one dispatcher — a caller names the op and never learns where it ran.
The three fleet targets were surface-check, describe-apps and openapi-weave:
compound words, and worse, words from the implementation's vocabulary rather
than from what you get. "weave" and "subsets" describe how the thing is built.
"openapi" and "describe" describe the thing.
surface-check -> check regenerate FROM SOURCE, fail on any diff
describe-apps -> describe every app describes itself, and runs zipdoc first
openapi-weave -> openapi compose every app's document into openapi.yaml
The root Makefile's own `describe` target is gone: it chained the generate, the
loop and the compose, which made two definitions of one word. mk/fleet.mk's
describe now absorbs the `go generate -run zipdoc ./...` step, so there is one.
`openapi` deliberately does NOT depend on describe -- it reads the committed
documents and stays a three-second command; `check` is the twelve-minute one
because regenerating is the whole of what makes it a gate.
55 references across 25 files, including hanzo.yml and .hanzo/workflows/cicd.yml,
so the gate CI runs is the gate a person runs by hand.
Also here: zip v1.27.0, which is where the toll's op.invoke seam and the
co-resident Here call come from. The working tree carried
`replace github.com/zap-proto/zip => /home/z/work/zap/zip` while that was
unpublished -- a path that exists on exactly one machine, and go.mod is not the
place to record which one.
Money was gated by HTTP middleware, and HTTP middleware reads the path the
TRANSPORT carried. For a plain REST call that path IS the operation, so the gate
looked right and tested green. It is not the operation anywhere else. One typed
op registered once is seen by the edge as four different things depending on the
door it arrived through, and PriceOf("/mcp") is Undeclared while
Billable("/.well-known/...") is false -- so an op reached over MCP or over the
call plane was free. Not by anyone's decision: the gate was asking about the
wrong value. Free never errors, so nothing said so.
That was survivable while everything spoke HTTP. It stops being survivable the
moment internal hops move to ZAP over UDS, because each converted call site is a
hop that leaves the only surface where the gate exists. The leak would have grown
with the migration rather than with traffic.
zip funnels every projection of a typed handler -- REST, MCP tools/call, the
by-name call plane, CLI LocalInvoke -- through one dispatcher, op.invoke, and
offers one hook inside it. Its argument is a zip.Op: {Method, Path, OperationID},
the operation as a value, identical on all four paths. So DefaultPrice and
Billable are asked there, about op.Method and op.Path, and answer the same thing
however the call arrived. The edge gates stay, because an untyped handler has no
op to ask about, and the toll stands down when the request's own path already
names a declared surface -- one operation, one answer.
Proven on four doors with both controls: an unpriced op moves nothing, and a read
of a priced surface moves nothing, each re-funded first so it cannot pass by
being broke. Neutering Toll fails MCP and ZAP at "balance settled at 25c, want
0c"; restoring it passes.
toll_wire_test.go closes the gap that proof leaves. The four-door test builds its
own app and installs the hook itself, so deleting both lines from serve.go leaves
every subtest passing -- measured, not supposed. Two claims, one gate: the toll
charges correctly, and the server actually mounts it. The second is now pinned,
and goes red naming which of the two lines went missing. A gate nothing mounts is
worth what no gate is worth, and worse, because the passing suite reads as
assurance.
pricing: the embedded catalog was a snapshot from 2026-03-14. /v1/pricing/models
was therefore selling 10 zen4 rows and 26 zen3 rows — both generations retired —
while the five zen5 rows that DO exist carried no price at all. 38 phantom rows
gone; the real SKUs now carry the numbers the gateway serves, derived from zen's
own cost x margin rather than hand-kept.
That regeneration only became correct today: the zen SERVICE was pinned at v1.4.1
while the library was at v1.4.11, so `npm run sync` reads ZEN_URL and would have
copied the same DigitalOcean-era prices straight back in. A sync is only as true
as the service it reads.
console: the playground's six starters suggested `zen-omni` or `zen-coder`, and
the gateway serves neither. Nothing threw — applying a starter falls back to the
selected model when the suggestion is absent — so the card advertised a model
that could never be the one that ran.
I installed this seam and wrote nothing that would notice its removal. ai's
builtin registry DECLARES web_search unconditionally and holds no backend; this
host supplies one. Delete that line and nothing fails to compile, no route 404s,
no test goes red — the tool just answers "not available in this deployment" to
every agent, forever. That is the same shape as every other wired-but-never-
consulted defect in this tree, and I had just re-created it.
installWebSearch takes the searcher as a PARAMETER rather than calling
websearch.Search directly, so the adapter — the truncation and the field mapping
— runs without a network round trip. Production passes the real one; the tests
pass their own.
What the tests hold:
- the seam is CLOSED after install (the deletion case)
- Content maps to Snippet. websearch.Result names the body Content and the tool
contract names it Snippet; lose that rename and every agent gets empty
snippets, which a model reads as "the web had nothing to say about this"
rather than as a bug
- the limit is honoured, and 0/negative mean UNSET rather than "truncate to
nothing"
- an empty upstream is an empty SLICE, not an error — the distinction the whole
tool layer rests on: an error means the capability is unavailable and the
model should say so, an empty list is a genuine finding
TestMountInstallsTheWebSearchSeam is a source assertion and says so. Mount needs
a real zip.App and cloud.Deps (it runs ai's Bootstrap), so no test in this
package constructs one; it pins the call site against deletion while the
behavioural tests pin the adapter. Naming the limit beats implying coverage that
is not there.
Three negative controls run, each failing only its own test: dropping the call
from Mount, blanking the Snippet mapping, and ignoring the limit.
webui/console.go's //go:embed all:dist and webui/dist/ are gone, as is the
CONSOLE_IMAGE Dockerfile stage and the Makefile webui target. cmd/cloud loads
the ACTIVE release of the hanzo-console site into RAM at boot and swaps it on an
atomic.Pointer, resolving it through sites.CurrentResolver() — the same path cd
uses, so there is one answer to 'which console is live'.
Publishing drops from an image build plus a Recreate rollout (a measured 2m15s
api.hanzo.ai outage to ship a CSS fix) to a sub-second release activate that
restarts nothing. Rollback likewise: two independent levers now, one for the
binary and one for the console.
Dockerfile conflict resolved toward this change — ARG CONSOLE_IMAGE is deleted
because deleting it is the point. Kept main's still-true warning that no host
routes to the standalone console Deployment, since rolling it still reaches
nobody.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The stub this removes was invisible for as long as it existed. It compiled. It
had test coverage -- two tests asserted the stub's error and passed. It logged
"deps.IAM -> ZAP RPC" at boot. Every angle said the wire was up except a running
fleet, and nothing in the repo looked from that angle.
So the gate is lexical and blunt, because the defect announces itself in the
words it uses: "not yet wired", "zapc-gen", "TODO" anywhere in clients/ is a
transport somebody is trusting and nobody has built. doc.go is exempt -- it
explains the deleted shape on purpose, and history is how the next reader learns
why the shape is what it is.
The second gate closes the visible half. A peer is addressed by NAME
(plane.Ask -> zip.SocketPath), so there is no endpoint for a deployment to
supply; the eight CLOUD_<X>_ZAP_ADDR knobs were what let a deployment believe it
had chosen a transport. It matches the SHAPE rather than a list of subsystem
names, because a list has to be kept in step and the ninth entry added is the one
that slips through. Note the digits in the suffix match: O11Y is exactly why a
[A-Z_]+ pattern would have missed one, and did, when I first swept for these.
Both were mutation-tested rather than merely observed to pass. Reintroducing the
stub in clients/relapse.go and a CLOUD_IAM_ZAP_ADDR read in config.go fails both
with the reason attached; removing them passes. A gate that has only ever been
green is a gate nobody has checked.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The mint has refused a payment whose two organisations are two organisations since
the split that fixed the receipt read. It refused from the wrong side of the money.
screen.record runs the ledger boundary AFTER next(c) — after commerce's card core has
taken the card — so the one caller the rule exists for, a SuperAdmin acting inside a
customer's org, got the worst of both halves: the charge cleared on the CUSTOMER's
merchant account, the credit had no address to land at, and the door answered 500 over
real money that was now permanently uncreditable. Worse, chargeable again: the retry
the answer invited replays into the same refusal, and a fresh idempotency key takes a
fresh card authorisation, every fifteen minutes for as long as somebody keeps trying.
Both names are already resolved by seen(), off a request that has moved nothing, so
the question is asked in screen.decide — one place, both doors, every projection of
the typed op — before the handler runs. It answers 409 rather than the screen's own
403 because it is not a risk verdict (nothing was scored, the scorer is not asked) and
an operator reading the door's refusals should never have to guess which of the two it
was, and the sentence names POST /v1/admin/grants: a platform operator meaning to fund
a customer is not doing anything wrong, they are at the wrong door. The predicate is
one method, payment.diverged, because the boundary keeps its own reading of the same
rule as a backstop — the pre-charge refusal is a property of the two doors the screen
is composed onto, not of the ledger, and a mint reached any other way still refuses.
Callers that pay for themselves are untouched: principal answers one string for both
names for an ordinary member, a service token and a SuperAdmin at home.
MigrateOrg keeps the exactly-once it documents over an input that MOVES. Its ref is
fixed per org and its amount is whatever commerce held when somebody ran it, so the
honest second run — a retried operator call over a balance the customer has spent from
since — collided with the deposit's conflict rule and came back an error. That rule is
right for a SETTLEMENT, where a ref is one payment and a differing amount is a
different payment wearing a taken key; a backfill ref names an ORG's one cutover and
nothing else may write it. So a conflict on its own ref is read back as "already
carried": the original entry id, no money moved, the same answer on every subsequent
run. depositByRef is untouched.
And a refusal now says WHETHER A RETRY CAN CLEAR IT. "The customer's own retry is the
recovery path" was true of a lost write and false of every refusal that is a fact about
the settlement itself — a charge that settled in another currency or for nothing, a
settlement the door could not identify, a reference already posted for another payment,
the two names that cannot be one name. Telling a customer to retry one of those is
worse than telling them nothing: the same key replays into the same refusal and a fresh
key charges the card again. Each branch answers through uncredited or stranded; the
RECONCILE line carries a terminal field to alert on and names what does settle it (the
admin grant credits the balance, the processor refunds the charge), and the customer is
told not to retry.
finance's deposit conflict is exported as ErrRefTaken, the twin of the usage plane's
ErrRefReused and for its reason: it is the one deposit failure no retry can clear, so a
caller that renders it as a transient fault sends a customer round a loop that cannot
end. The credit door reads it to classify its refusal; the backfill reads its own ref
back through it.
Note a response-shape change already in flight from the deposit-ref work: the ledger's
idempotency lookup returns the whole entry, so POST /v1/admin/treasury/seed's
seedData.Entry (apps/treasury/treasury.go) now carries `postings` on the REPLAY branch
too, where the created branch always had them. Additive, and the two branches agree now.
Tests are mutation-proven. The pre-charge refusal is asserted on a COUNTED charge
handler, because a status code cannot tell a refusal from a refusal that happened after
the money moved: leave the rule only at the boundary and the masquerade row runs the
handler once and answers 500; invert it and both org == ledger rows lose their 200 and
their money. The receipt-read discrimination moves to a direct settle call, which is
where that read still exists now no door reaches it. A structural check pins every
fin.Deposit call site in the package with the reason it may mint, so the next one is
added deliberately.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The mint has refused a payment whose two organisations are two organisations since
the split that fixed the receipt read. It refused from the wrong side of the money.
screen.record runs the ledger boundary AFTER next(c) — after commerce's card core has
taken the card — so the one caller the rule exists for, a SuperAdmin acting inside a
customer's org, got the worst of both halves: the charge cleared on the CUSTOMER's
merchant account, the credit had no address to land at, and the door answered 500 over
real money that was now permanently uncreditable. Worse, chargeable again: the retry
the answer invited replays into the same refusal, and a fresh idempotency key takes a
fresh card authorisation, every fifteen minutes for as long as somebody keeps trying.
Both names are already resolved by seen(), off a request that has moved nothing, so
the question is asked in screen.decide — one place, both doors, every projection of
the typed op — before the handler runs. It answers 409 rather than the screen's own
403 because it is not a risk verdict (nothing was scored, the scorer is not asked) and
an operator reading the door's refusals should never have to guess which of the two it
was, and the sentence names POST /v1/admin/grants: a platform operator meaning to fund
a customer is not doing anything wrong, they are at the wrong door. The predicate is
one method, payment.diverged, because the boundary keeps its own reading of the same
rule as a backstop — the pre-charge refusal is a property of the two doors the screen
is composed onto, not of the ledger, and a mint reached any other way still refuses.
Callers that pay for themselves are untouched: principal answers one string for both
names for an ordinary member, a service token and a SuperAdmin at home.
MigrateOrg keeps the exactly-once it documents over an input that MOVES. Its ref is
fixed per org and its amount is whatever commerce held when somebody ran it, so the
honest second run — a retried operator call over a balance the customer has spent from
since — collided with the deposit's conflict rule and came back an error. That rule is
right for a SETTLEMENT, where a ref is one payment and a differing amount is a
different payment wearing a taken key; a backfill ref names an ORG's one cutover and
nothing else may write it. So a conflict on its own ref is read back as "already
carried": the original entry id, no money moved, the same answer on every subsequent
run. depositByRef is untouched.
And a refusal now says WHETHER A RETRY CAN CLEAR IT. "The customer's own retry is the
recovery path" was true of a lost write and false of every refusal that is a fact about
the settlement itself — a charge that settled in another currency or for nothing, a
settlement the door could not identify, a reference already posted for another payment,
the two names that cannot be one name. Telling a customer to retry one of those is
worse than telling them nothing: the same key replays into the same refusal and a fresh
key charges the card again. Each branch answers through uncredited or stranded; the
RECONCILE line carries a terminal field to alert on and names what does settle it (the
admin grant credits the balance, the processor refunds the charge), and the customer is
told not to retry.
finance's deposit conflict is exported as ErrRefTaken, the twin of the usage plane's
ErrRefReused and for its reason: it is the one deposit failure no retry can clear, so a
caller that renders it as a transient fault sends a customer round a loop that cannot
end. The credit door reads it to classify its refusal; the backfill reads its own ref
back through it.
Note a response-shape change already in flight from the deposit-ref work: the ledger's
idempotency lookup returns the whole entry, so POST /v1/admin/treasury/seed's
seedData.Entry (apps/treasury/treasury.go) now carries `postings` on the REPLAY branch
too, where the created branch always had them. Additive, and the two branches agree now.
Tests are mutation-proven. The pre-charge refusal is asserted on a COUNTED charge
handler, because a status code cannot tell a refusal from a refusal that happened after
the money moved: leave the rule only at the boundary and the masquerade row runs the
handler once and answers 500; invert it and both org == ledger rows lose their 200 and
their money. The receipt-read discrimination moves to a direct settle call, which is
where that read still exists now no door reaches it. A structural check pins every
fin.Deposit call site in the package with the reason it may mint, so the next one is
added deliberately.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
clients/rpc.go declared ZAP RPC for nine subsystems -- iam, base, commerce,
ai, o11y, vfs, mq, payments, vault -- and every one of its thirty-odd methods
returned the same thing:
cloud: ZAP RPC client for iam@iam.hanzo.svc:9653 not yet wired
(zapc-gen pending) -- VerifyJWT
It was live wiring, not dead code. BuildDeps selected it whenever
CLOUD_<X>_ZAP_ADDR was set, and logged "deps.IAM -> ZAP RPC" when it did. So
the single signal an operator had said the wire was up, while nothing crossed
it. Setting CLOUD_VFS_ZAP_ADDR replaced a working S3-backed blob client with
one that failed every Put and Get. A missing client is diagnosed in seconds;
one that reports itself configured and then fails elsewhere is diagnosed in an
incident.
pickKMSClient already made this argument and already acted on it -- it dropped
CLOUD_KMS_ZAP_ADDR for the plane, with a comment saying a stub that looks
configured is worse than none. The argument was never specific to KMS. This
applies it to the other eight.
The peer plane is the transport, and it is the only one: plane.Ask over the
peer's own socket, addressed by NAME, with plane/gen emitting a typed client
per app. It takes no endpoint, which is why removing eight config fields and
eight env knobs removes nothing real -- no manifest in universe sets any of
them, so no deployment loses a path it had.
What each dependency does now is what it did: co-resident when mounted here,
and the disabled stub -- which says "enable the subsystem" -- when not. Payments
and vault stay non-nil because commerce nil-checks them, but they stop claiming
a cardholder-data transport that never carried a byte; nothing in the tree
invokes CreateIntent, ConfirmIntent, GetIntentStatus or Charge at all.
The two tests that covered this asserted the fake. One pinned that a configured
address yields a client which is NOT disabled -- true, and true only because it
failed in a transport-specific way instead. Both now assert the honest
property: a subsystem that is not here reports itself DISABLED, and no address
can say otherwise.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The comment above this mount said byte-identical patterns "merge silently,
first wins". Measured, they do not. zip refuses the whole program and names
every conflicting pair with file:line and the group path it came through:
zip: this program does not compose, so it has no projection:
zip: GET /v1/commerce/collection: declared by "/collection" at
rest/rest.go:133 (via root -> /v1/commerce -> /collection) and by
"/collection" at rest/rest.go:133 (...)
The correction matters because the false version makes an addition look free.
Commerce's api.Route already registers order, customergroup, promotion, region,
taxregion, pricelist, claim, role and the rest; the admin needs them at
/v1/commerce, so someone will reach for this leaf. Adding a kind here while
api.Route still registers it does not merge — it takes the STANDALONE down at
boot, in a different binary from the one that changed. One address, one
declaration.
WHERE the refusal lands differs by zip and both are before traffic: this tree
resolves v1.25.1, which accepts the second Route() call and refuses at
COMPOSITION; commerce standalone pins v1.24.2, which refuses inside Route().
That difference is not academic — the first version of this test only
registered, read "accepted", and would have shipped the claim that the rule was
gone. It now asks for the composition, and it is proven by construction: remove
the second Route() call and it fails.
Also drops the stale "two equal-specificity params with different names" framing
from the sibling guard's message, which described a narrower collision than the
one that actually fires.
console.hanzo.ai is this binary and stays this binary — same host, same
origin, /v1 still answered here, session cookie still first-party. What
moves is the SOURCE OF THE BYTES.
The console was //go:embed'd from webui/dist, overwritten at image build
from a pinned console-embed. That welded the frontend's lifecycle to the
backend's: a CSS fix cost a ~22-minute cloud build plus a Recreate
single-replica rollout — a measured 2m15s outage of api.hanzo.ai to ship
a stylesheet. It also needed a whole discipline to stay honest (never
:latest — v1.801.215 built 12 minutes early and silently shipped the
previous console, green — and never re-point a cut tag), a discipline
that existed only because the console's identity lived in a Dockerfile.
It is a site release now, through the site-release subsystem already in
this repo: the ACTIVE release of the hanzo-console site, read from the
same S3 the site edge reads, resolved through the same registry
cd.hanzo.ai resolves through (sites.CurrentResolver — one resolution
path, not a second). Publish is under a second, rollback faster, and
neither builds nor restarts anything.
webui keeps the whole handler — per-Host white-label <title>,
static-export route shells, precompressed negotiation,
cache policy, the API-namespace 404 — and takes the FS as
an argument. Still a leaf. The cached index.html is gone:
against a live release a cached shell names the previous
build's chunk hashes, so it is read per request now, via
the one HTML-shell path the route shells already used.
webui/release loads the release whole into RAM (68 files / 9.3 MB) and
swaps it atomically when the active pointer moves. The
steady state is one resolver call per poll; bytes are
re-read only when the release actually changed.
Required at the front door, reported in a child. cmd/cloud starts
projects (idempotent; bounded, so it refuses rather than hangs) and
fails with the reason if the release will not load. cloud.Listen cannot:
a child would have to ask projects, only the host can start projects,
and the broker every child needs boots first — so a child logs the
reason and answers 503 naming it. Never a blank shell.
Also: _next/static and assets/ are immutable by PATH at the site edge.
The basename rule needs a separator before the hash run, which Next does
not emit, so 5 of 44 _next/static objects were served max-age=3600 and
re-fetched hourly forever. The prefix stays narrow — _next/ also holds
_next/data and a build-id dir, and this path serves arbitrary tenant
sites, where a year of immutability on a service worker outlives every
deploy.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The 119 derived sentences were true about tenancy and silent about everything
else, and the seventeen families they cover are not uniform on the axis that
matters to a caller. Read off mount.go and commerce's api/resources:
- transfer, wallet and webhook are ADMIN-gated (TokenRequired(permission.Admin)).
A plain member's token is refused 403 there, on reads as much as writes.
- collection, discount, product, variant, saleschannel and stocklocation sit
behind paywall.Require: no active subscription, trial or redeemed invite is
402 subscription_required, and an unreadable billing store fails closed 503.
- collection, product, return, subscriber and variant additionally carry a
DefaultPermissions table that CheckPermissions enforces per method (403).
The other twelve have no table, and the miss logs and ALLOWS — so claiming a
scope check on those would be prose the server does not honour.
One sentence repeated seventeen times said none of it, which is worse than the
bare operationId it replaced: it told a reader that a plain token may write a
wallet and that the only refusal on a product is 404. A description nothing
downstream can check is believed. So the table now carries the gate — the only
thing that actually varies — and the sentence is generated from it.
Three corrections to the shape prose while reading the handlers:
POST on the item address is the METHOD-OVERRIDE door, not "an address that
accepts every method". With no override it runs r.Patch; `_method` (form or
query) or X-HTTP-Method-Override set to PUT, PATCH or DELETE runs THAT one, so a
POST here can DELETE the record. An invalid override is ignored, not 405'd —
IsValidMethodOverride only admits those three and the method stays POST.
LIST returns the Pagination envelope {page, display, count, models, facets}, not
a bare array, and it fails closed to an EMPTY PAGE with 200 when no org
namespace resolves — so empty `models` does not mean "no records exist".
PUT decodes onto an empty record carrying only the existing key, which is why
omitted fields are reset; PATCH decodes over the loaded record. Both 404 on an
unknown id. DELETE writes a copy aside under an internal deleted key first, so
the row stops answering but is not erased.
Also: a family that outgrows the table comes OUT of it rather than gaining a
Describe as well — Describe panics on a duplicate key, so doing both aborts the
binary at init. The old comment said the explicit one "wins", which would have
sent the next reader into that panic.
Regenerated: openapi.yaml and plugin/commerce/openapi.json. 119 operations, none
bare; regenerating from source reproduces these bytes exactly.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every chat and vision SKU (zen5-mini, zen5-flash, zen5, zen5-coder, zen5-pro,
zen-vl, zen-guard) now routes through openrouter. DigitalOcean cannot be billed
right now, and a provider we cannot pay is not a route.
It is cheaper on every rung, because openrouter publishes each upstream's own
rate: zen5 quotes 2.28/7.26 per MTok where it quoted 4.176/13.2. Margin is
untouched at 3x — the saving reaches the caller.
OPENROUTER_API_KEY must be in the environment for this image. zen's resolver
reads env first and KMS only as a fallback, so a missing key is not a degraded
route, it is a 401 on every chat. The universe values carry it in the same
commit that pins this image.
Four SKUs stay on DigitalOcean and nothing can move them: openrouter is a
chat-completions aggregator with no /embeddings, /images/generations,
/videos/generations or /rerank, so zen-embedding, zen-image, zen-video and
zen-rerank have no rung to move to.
A usage debit was idempotent on the caller's X-Request-Id. That header is the
CLIENT's: zip's RequestID middleware keeps an incoming one verbatim and the
gateway CORS-allows it from a browser, so the payer chose the key the ledger
deduped on. Pin one value and every call after the first replayed into the
first one's entry — the work ran, the wallet did not move, and the spend cap,
which sums that ledger, never rose. It reached finance from every metered
surface: the edge BillingGate, the net/http meter, ~25 ResourceMeter callers,
and zen, whose own field comment called the header "idempotency".
The key also carried no subject. (kind, program, ref) held program empty for
usage, so ONE namespace per org held every subject's refs and the first subject
to take a ref owned it for everyone: two people in one org sharing a ref meant
the second one's call debited nobody.
Both halves are the same mistake — the key named neither who paid nor which act
— and both are fixed where the key is formed rather than at the 27 places that
supply attribution:
- The scope is the WALLET. usageProgram is walletAcct, so the account a ref is
unique within IS the account the posting debits; the key and the money can
no longer name two wallets.
- The name is the SERVER's. metering.Usage.Ref carries it, Usage.Seal mints
it, and Record seals what it is given — one seam, so every caller is fixed
without touching one of them. Seal is idempotent, so a caller holding a
server-assigned act id (a message row's id, an x402 settlement id, a domain
registration ref) sets Ref instead and its own retry stays exactly-once.
- RequestID stays, as what it always was: correlation, on the log line and the
attribution wire, never the money.
- A ref names ONE charge. A hit whose amount differs is ErrRefReused, not the
first entry — the rule the deposit already states, so the GPU charge, the
one surface whose key a caller may name, answers 409 instead of handing back
work nobody paid for.
Exactly-once is preserved and proven both ways: three attempts at one sealed act
debit once; two acts identical in every field debit twice.
Mutation-proven: key the ledger on the header again and 20 pinned calls bill 1¢
instead of 20¢; drop the subject from the key and the second subject's wallet
never moves; make the mint non-stable and one act charges three times; drop the
amount check and a $5.00 charge under a $0.30 name is answered "ok".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A usage debit was idempotent on the caller's X-Request-Id. That header is the
CLIENT's: zip's RequestID middleware keeps an incoming one verbatim and the
gateway CORS-allows it from a browser, so the payer chose the key the ledger
deduped on. Pin one value and every call after the first replayed into the
first one's entry — the work ran, the wallet did not move, and the spend cap,
which sums that ledger, never rose. It reached finance from every metered
surface: the edge BillingGate, the net/http meter, ~25 ResourceMeter callers,
and zen, whose own field comment called the header "idempotency".
The key also carried no subject. (kind, program, ref) held program empty for
usage, so ONE namespace per org held every subject's refs and the first subject
to take a ref owned it for everyone: two people in one org sharing a ref meant
the second one's call debited nobody.
Both halves are the same mistake — the key named neither who paid nor which act
— and both are fixed where the key is formed rather than at the 27 places that
supply attribution:
- The scope is the WALLET. usageProgram is walletAcct, so the account a ref is
unique within IS the account the posting debits; the key and the money can
no longer name two wallets.
- The name is the SERVER's. metering.Usage.Ref carries it, Usage.Seal mints
it, and Record seals what it is given — one seam, so every caller is fixed
without touching one of them. Seal is idempotent, so a caller holding a
server-assigned act id (a message row's id, an x402 settlement id, a domain
registration ref) sets Ref instead and its own retry stays exactly-once.
- RequestID stays, as what it always was: correlation, on the log line and the
attribution wire, never the money.
- A ref names ONE charge. A hit whose amount differs is ErrRefReused, not the
first entry — the rule the deposit already states, so the GPU charge, the
one surface whose key a caller may name, answers 409 instead of handing back
work nobody paid for.
Exactly-once is preserved and proven both ways: three attempts at one sealed act
debit once; two acts identical in every field debit twice.
Mutation-proven: key the ledger on the header again and 20 pinned calls bill 1¢
instead of 20¢; drop the subject from the key and the second subject's wallet
never moves; make the mint non-stable and one act charges three times; drop the
amount check and a $5.00 charge under a $0.30 name is answered "ok".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Mounting the bundle was not enough and the last release proved it. The light
host routes by manifest.Apps: commerce's row names each leaf it serves, `ai`
holds the bare "/v1", and longest-prefix wins. A leaf nobody names goes to ai's
catch-all, so commerce sat there serving routes the host never handed it.
Production said so precisely — /v1/commerce/tenant (named) answered 200 from
commerce in the same second /v1/commerce/product (unnamed) answered 404 from ai.
So name them, which is the rule this file already states: nobody claims the bare
/v1/commerce remainder, every leaf is named deeper. Seventeen leaves, read off
the live router rather than guessed. Each is gated in Mount — token, admin token
and paywall — so this changes who is ROUTED, not who is allowed.
The guard is the real repair. Its first version asserted the bundle BINDS
/v1/commerce/product; it did, and production still 404'd, because binding says
nothing about the host's routing table. It now asserts every bound leaf is named
on commerce's manifest row, and it is proven by reverting the row: 17 failures,
one per leaf, then green.
That closes a gap the existing oracle cannot see. manifest/router_test.go
compares routing against plugin/commerce/openapi.json — a GENERATED file — so a
bundle whose spec has not been regenerated is invisible to it and it passes
vacuously, which is exactly what it did here. Reading the live router needs no
regeneration step, so an upstream kind fails the moment it is linked.
Separately and NOT fixed here: `describe` for commerce is already red without
this change. It refuses 128 operations that "say nothing about themselves", 9 of
which are pre-existing (/v1/billing/methods, /v1/billing/wire,
/v1/billing/crypto/*, /_/commerce/providers/{name}) and untouched by this commit.
So plugin/commerce/openapi.json is stale at 73 paths and cannot be regenerated
until those operations carry doc comments. Routing does not depend on it; the
published spec and the generated SDKs do.
console-embed 8.5.58 is the launch build: the sidebar down to chat, the builder,
models, keys, usage, billing and settings with everything else behind the beta
flag; a playground that offers only what the gateway routes and holds frontier
prices behind a plan; the org's own logo in the mark row; search across the whole
catalog. This pin named 8.5.48, so none of it was reachable.
The routing fact is the one worth writing down: hanzo-domains.yaml sends
console.hanzo.ai to `service: cloud`, and nothing routes to `service: console`
at all. The SPA a user loads is the one //go:embed bakes in here, so a console
release ships only when this line moves. console:v8.5.59 was built, pinned and
rolled to Ready today and served no one.
TWO THINGS STOOD BETWEEN THE FLEET DOCUMENT AND SOURCE, both revealed only once
licensing stopped failing first — surface-check stops at the first app that cannot
project, so everything behind it was invisible rather than passing.
commerce publishes 17 generated CRUD families over the same store, 7 operations
each: 119 sentences differing in one word. They are DERIVED now, for the reason
openapi.DescribeSPA derives the two SPA addresses — prose written 119 times drifts
119 ways, and the next family the module adds would arrive undescribed and stop the
gate again. Adding one is a line in `resources`. The vocabulary stays plain on
purpose: these are generic store resources, so each sentence states the shape (what
the address is, what the method does, what the id means) and stops. A resource that
earns its own prose keeps an explicit Describe above — Describe panics on a
duplicate, so the hand-written sentence and the loop cannot both claim an address.
The weave then refused for a different reason, and it was right to: schema
"Artifact" meant a RELEASE artifact in licensing (download_url, cosign_signature)
and a RESEARCH artifact in research (git_sha, run_id, retention_class). Schema names
are global in the woven document, so every generated SDK would have bound whichever
it read last — one name, two shapes, silently. research's is the one this repo owns,
so it becomes ResearchArtifact; licensing's stays, being the external module's.
Regenerated: openapi.yaml, the affected subsets and the floor.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two money bugs on the deposit path, one introduced and one waiting.
A card top-up names TWO organisations and the split that fixed the receipt read
left the MINT choosing between them. The charge clears on the EFFECTIVE org's
merchant account and commerce writes its receipt there; the credit is addressed
to principal.WalletOf. For every caller but a masquerading SuperAdmin those are
one string. For that one they are two, and the deposit took the second: a
customer's card funded a platform admin's spendable balance and the door
answered 200. Neither address is right — crediting the ledger moves a customer's
money into the admin's books, crediting the charged org has a masqueraded session
top up the org it is only inspecting — so the mint refuses a payment whose two
names are not one name, with the door's own 500 and a RECONCILE line naming both.
A SuperAdmin funding a customer uses the admin grant, which is a credit that
states whose it is. The refusal sits AFTER the receipt read, so the masquerade is
still the one caller that can tell payment.org from payment.ledger and the read
this replaces cannot quietly go back to the payer's namespace. Callers that pay
for themselves are untouched: org == ledger, so the guard cannot fire.
A deposit's idempotency key is (kind, program, ref) and carries neither subject
nor amount, so two DIFFERENT payments naming one ref in one org's books collide.
Both the in-transaction replay branch and the recovery read answered the second
of them with the FIRST one's entry id: alice's $1 posted, bob's $500 was told
SUCCESS, bob's wallet stayed at zero and nothing said a payment had been dropped.
A ref hit is a REPLAY only when it is the same money to the same wallet; anything
else is refused, because crediting anyway breaks the exactly-once the ref exists
to give and answering with the other payment's entry is the swallow. Nothing
reaches it today — the settlement ref is a Square payment id — but the webhook,
the backfill and the credit RPC all pick their own. Both callers now read the ref
through one function, so they cannot answer the question two ways.
The subject lives in the entry's legs, so the ledger's idempotency lookup returns
the whole entry the way its journal listing does rather than a header the caller
must complete; one postings read serves both, over a connection or inside a
transaction.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two money bugs on the deposit path, one introduced and one waiting.
A card top-up names TWO organisations and the split that fixed the receipt read
left the MINT choosing between them. The charge clears on the EFFECTIVE org's
merchant account and commerce writes its receipt there; the credit is addressed
to principal.WalletOf. For every caller but a masquerading SuperAdmin those are
one string. For that one they are two, and the deposit took the second: a
customer's card funded a platform admin's spendable balance and the door
answered 200. Neither address is right — crediting the ledger moves a customer's
money into the admin's books, crediting the charged org has a masqueraded session
top up the org it is only inspecting — so the mint refuses a payment whose two
names are not one name, with the door's own 500 and a RECONCILE line naming both.
A SuperAdmin funding a customer uses the admin grant, which is a credit that
states whose it is. The refusal sits AFTER the receipt read, so the masquerade is
still the one caller that can tell payment.org from payment.ledger and the read
this replaces cannot quietly go back to the payer's namespace. Callers that pay
for themselves are untouched: org == ledger, so the guard cannot fire.
A deposit's idempotency key is (kind, program, ref) and carries neither subject
nor amount, so two DIFFERENT payments naming one ref in one org's books collide.
Both the in-transaction replay branch and the recovery read answered the second
of them with the FIRST one's entry id: alice's $1 posted, bob's $500 was told
SUCCESS, bob's wallet stayed at zero and nothing said a payment had been dropped.
A ref hit is a REPLAY only when it is the same money to the same wallet; anything
else is refused, because crediting anyway breaks the exactly-once the ref exists
to give and answering with the other payment's entry is the swallow. Nothing
reaches it today — the settlement ref is a Square payment id — but the webhook,
the backfill and the credit RPC all pick their own. Both callers now read the ref
through one function, so they cannot answer the question two ways.
The subject lives in the entry's legs, so the ledger's idempotency lookup returns
the whole entry the way its journal listing does rather than a header the caller
must complete; one postings read serves both, over a connection or inside a
transaction.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
console-embed 8.5.58 is the launch build: the sidebar down to chat, the builder,
models, keys, usage, billing and settings with everything else behind the beta
flag; a playground that offers only what the gateway routes and holds frontier
prices behind a plan; the org's own logo in the mark row; search across the whole
catalog. This pin named 8.5.55, so none of it was reachable.
The routing fact is the one worth writing down: hanzo-domains.yaml sends
console.hanzo.ai to `service: cloud`, and nothing routes to `service: console`
at all. The SPA a user loads is the one //go:embed bakes in here, so a console
release ships only when this line moves. console:v8.5.59 was built, pinned and
rolled to Ready today and served no one.
hanzoai/licensing used to register ONE untyped wildcard — app.All("/v1/licensing/*")
over its own net/http mux — and an untyped route cannot carry prose, so its seven
method descriptions were declared here, in cloud, keyed to that pattern.
At v0.1.10 it types them instead: app.Group("/v1/licensing") with zip.Post(g,
"/issue", ...) and siblings. A typed op carries its sentence in the handler's own
doc comment and zipdoc lifts it, so the seven declarations in this file now name no
operation at all. That is worse than missing prose: they render nowhere while
reading, in the source, exactly as though they had landed.
Deleted with the route they described. The gate that caught it is the same one that
catches the opposite case — an app that cannot project its own document — which is
why it is worth having pointed at both directions.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nothing else in this package calls commerceresources.Route, so the two ways it
can fail were both invisible. Registration into a shared router panics on two
equal-specificity params with different names — a BOOT panic that passes every
build check. And the routes simply going missing is what 404'd every admin data
view in production.
Reads the route TABLE rather than driving a request: a request exercises
commerce's datastore and tenant resolution, and a panic in there reads as "the
route is missing" when the route is fine.
Trailing slashes are trimmed because the bundle binds /v1/commerce/product/ and
every client calls /v1/commerce/product. Those are one route only because
fiber's StrictRouting defaults false and zip never sets it, so that is pinned
too — defensively: zip.Config exposes no such field, so unlike the absence
assertion (proven to fire on a bogus kind) it cannot be falsified today.
api.hanzo.ai/v1/commerce is THE commerce endpoint, and commerce is a PLUGIN of
this binary — there is no commerce backend pod. So the merchant surface is
reachable through this binary or it is not reachable at all. It was not: every
admin data view 404'd in production while sign-in, catalog and billing answered
200. Honest, and total.
Mounts commerce's api/resources LEAF, not its api.Route. Route also binds an
index route, a permissive CORS policy and a wildcard OPTIONS onto whatever
router it is handed — right for a process that owns its tree, wrong in this
shared one, where byte-identical patterns merge silently and two
equal-specificity params with different names panic at registration. Importing
that package would also drag checkout, subscriptions and thirdparty/netlify,
which this binary deliberately does not carry.
commerce v1.50.7 -> v1.50.11 for the leaf.
Package tests unchanged: 40 failures before and 40 after, the identical set —
all pre-existing ledger/payments/settlement cases on this line.
Five checks were measuring something other than what ships.
captable pinned two MCP tool names in their pre-v1.26 spelling. zip v1.26.0
settled ONE rule — an id derives from the absolute path the occurrence answers at
— so the tools are patch_v1_captable_stakeholders_by_id and
post_v1_captable_rounds_by_id_close. bab535c2 taught the bots and agents
projections that rule and missed this file. The segments still name the same two
ops unambiguously and what the test proves is unchanged: the op is addressable
through its In alone, over a transport with no URL.
pricing declared /v1/pricing/datastore and /v1/pricing/services and proved
neither, so the byte-identity golden had quietly stopped covering two sections.
They are covered now. The scanner also skips fiber's "/" route, which is
middleware territory — every app.Use rides it as a handler chain, by design — and
never was a section address.
projects pinned the template repository at hanzo-templates/synapse. It is not
there: 4203c7f2 found 45 of 66 gallery entries pointing at repositories that do
not exist, verified every source against the GitHub API, and repointed the static
sites to hanzo-apps/template-<slug>, which is that org's convention. Only 21
templates (expo-*, flutter*, swiftui*, desktop-*) ever moved to the
hanzo-templates layout this line was written against — the catalog is mixed by
design, and the variant case beside it still pins hanzo-templates/prism-react.
The assertion's intent is untouched: a fork seeds the REPOSITORY, not the gallery
page that is an HTML 404, and the provider is github.
git's ZAP round trip installed its identity shim through app.Fiber().Use before
Mount. At zip v1.26 Fiber() materialises a DRAFT router and every later
registration invalidates it, so the next call builds a new *fiber.App and the shim
was written onto a throwaway. The handlers saw no X-Org-Id, principal.Org failed,
and createRepo answered a 403 that zapface rewrites to a bare UNAUTHORIZED naming
none of it. It is a zip component now, installed by the same compose the server
runs (f6c9605b, same class).
And apps/commerce/risk.go joins allowedRequestUses with its reason. The fraud
screen in front of the typed mint op reads the payer, the door actually reached
and the jurisdiction signals a credit decision is made on — an identity gate
reading strictly more than the org, which is one of the sanctioned reasons. The
amount comes off the decoded In and the settlement off the returned receipt, both
deliberately not read from the wire, so the request is consulted for exactly the
facts no projection can carry on a type.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/content/generate published an operationId and nothing else — no request
schema, no response schema, no MCP tool input, no CLI flags, an untyped `any` in
every generated SDK — for the ONE agentic call in the content loop. It stayed a
raw handler on a stated belief: that a studio-render billing denial answers the
platform's nested {"error":{"code","message"}} at 402/503, and a typed op can
answer only its Out schema or zip's flat {status,code,error}.
That belief is out of date. cloud.Denied carries the money wire's own status and
body as an ERROR — the one refusal channel a typed op has — and cloud.DenyEnvelope
writes those bytes back verbatim; apps/projects, apps/dataset and apps/risk
already pair them. So the op is typed and the 402 body is byte-for-byte what it
was, which TestGenerate402IsTheMoneyWireBody asserts by shape: exactly one key,
`error`, nested {code,message}, code insufficient_balance. Off the HTTP path
(MCP, CLI) deniedErr.Unwrap keeps the status and the sentence.
Two details the conversion turns on. 402 is DECLARED so a generated client knows
the refusal is a shape it can read; it is never the status this op RETURNS, since
statusFor takes Statuses[0] when the Out states none, so a success is still 201
and the 402 reaches the wire from the envelope. And every GenerateInput field is
`url:"-"`, because zip's binder fills an In field from the QUERY as well as the
body: without it the route silently starts accepting ?doctype= and ?source_media=,
inputs the raw handler's c.Bind never took — and a source_media reachable from a
query string is an SSRF surface reachable from a link. A wire widening no status
code shows, pinned by TestGenerateStillReadsTheBodyAndOnlyTheBody.
The prose moves from the openapi.Describe init onto the handler, which is where
zipdoc lifts it from for a typed op, so the description is declared once rather
than beside the wire fact that no longer holds. The regenerated document is the
point of the change: /v1/content/generate now carries a GenerateInput requestBody
and GenerateResult on 201.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three surfaces installed a gate on a group and then registered the routes it was
meant to cover somewhere else, which zip v1.26 does not forgive: a Use is scoped
to the group INSTANCE it was installed on and its children, and is never matched
by prefix across the app. Measured, not reasoned: a probe registering the same
prefix twice shows the second group's routes with none of the first's middleware,
and a route on the App with none of any group's.
books rebuilt `app.Group("/v1/books")` in scan.go and bank_api.go, so six reads —
bank/transactions, inbox, vendors, rules, transactions, bank/unreconciled — and
the vendor/rule upserts answered per-org money WITHOUT Cache-Control: no-store,
while /gl and /questions next door carried it. Each file now installs noStore on
the group it builds. It keeps building that group from a literal prefix because
cmd/zipdoc files a doc comment by the prefix it can READ in the file: passing the
group down as a parameter drops every leaf below out of the document and the MCP
tool list. The five RAW handlers stay on the App with their whole paths for the
same generator reason in reverse — a raw handler on a readable prefix is one
zipdoc describes FOR it, and the only doc comment at that call site is
cloud.Handle's "binds a Service-scoped handler to a route", which would ship as
what /v1/books/scan does.
company and validators declared their collection root on the App, outside the
group carrying the cap and the identity gate. An oversized body at /v1/company
answered 400 from the decode instead of 413, and an anonymous POST to
/v1/validators with a malformed body answered 400 where the surface has always
answered 403 — the refusal has to precede the decode, and it was not running at
all. Neither root can hang off its own group with an empty leaf, which joins to
"/v1/company/" and moves the published address. So the gate rides the ROUTER:
With() wraps the leaf it registers and installs nothing at "/v1". That distinction
is load-bearing — a Use there gates every other subsystem's /v1 routes, and
cloud's own ownership gate refuses it in as many words. .Group("/v1") is then only
an address, and one zipdoc can read, so both ops keep their prose. Same shape
apps/sync already uses.
Pinned by TestReadsAnswerTheirFrozenEnvelope, TestVendorAndRuleUpsertsEchoThe-
NormalizedRow, TestBodyCapCoversJSONNotTheDeck and TestIdentityRefusalStill-
PrecedesTheBody.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Commerce is a plugin of cloud, so its merchant surface is reachable through this
binary or it is not reachable at all. It was not. There is no commerce backend
pod; commerce-api.hanzo.ai routes here; and this embed carried no resource
bundle — so every admin data view 404'd in production while sign-in, catalog and
billing all answered 200. Honest, and total: /v1/product AND /v1/commerce/product
both 404'd, so no client-side prefix change could have reached anything.
Products, variants, collections, discounts, sales channels, stock locations,
subscribers and webhooks now answer at api.hanzo.ai/v1/commerce/<kind> — the
endpoint every Hanzo surface is told to use.
It mounts the resources LEAF (commerce v1.50.11, api/resources), not api.Route.
Route also binds an index route, a permissive CORS policy and a wildcard OPTIONS
onto whatever router it is handed: right for a process that owns its tree, wrong
in this shared one, where byte-identical patterns merge silently and two
equal-specificity params with different names panic at registration. Importing
that package would also drag checkout, subscriptions and thirdparty/netlify —
weight this binary deliberately does not carry, and the real reason the bundle
was skipped rather than a decision anyone made.
The leaf's dependency graph contains netlify zero times; api's still contains it
twice. That boundary is the whole mechanism.
productEvents is nil here: the storefront publish loop belongs to the
standalone's event bus, and the CRUD does not depend on it.
Package tests unchanged: 23 failures before this commit and 23 after, the same
set — all pre-existing ledger/payments/settlement cases on forge/main.
A card top-up names TWO organisations and the settlement held one value for both.
commerce writes the receipt under the EFFECTIVE org — the org the door is acting in,
resolved by iammiddleware for the browser route and by payingOrg for the typed op —
while the balance it funds is principal.WalletOf's, which for a platform SuperAdmin is
its own books. Those are the same string for every caller except a SuperAdmin
masquerading into a customer's org, and for that one the receipt read looked in the
admin's books for a row commerce had written in the customer's: not found, so the door
refused a charge that had already cleared. Permanently — the retry replays the same
receipt into the same absent namespace, and a fresh idempotency key charges the card
again. The payment now carries both names as principal names them, `org` for the data
namespace and `ledger` for the billing key: the receipt is read from org, the deposit
lands on (ledger, subject), and the screen and the accrual key on ledger as before.
A settled charge the credit refuses is still taught to the risk model. Teaching is
telemetry about a payment that HAPPENED, so gating it on the credit dropped real
settlements — a charge in a currency this ledger does not hold, a receipt it could not
read — out of the payer's velocity, and a caller who could provoke the refusal could
pay all day and accrue nothing. Both doors now run one value (screen.record) that holds
the order: the money first, because only it can refuse the door, then the model, always.
A deposit that fails on money ALREADY in the books answers with the money. finance
dedups a replay inside its insert transaction; it cannot cover a transaction that never
reached its own read — the write lost to a concurrent poster of the same Ref, or the
request context that died between the charge and the post. Both leave the ref credited
and the caller holding an error for it, which at a credit door is a 500 on a card that
cleared. A failed deposit now asks the only question it has left, on a context detached
from the caller's because the caller's is what may have just died.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
(cherry picked from commit de1e5c4341)
A card top-up names TWO organisations and the settlement held one value for both.
commerce writes the receipt under the EFFECTIVE org — the org the door is acting in,
resolved by iammiddleware for the browser route and by payingOrg for the typed op —
while the balance it funds is principal.WalletOf's, which for a platform SuperAdmin is
its own books. Those are the same string for every caller except a SuperAdmin
masquerading into a customer's org, and for that one the receipt read looked in the
admin's books for a row commerce had written in the customer's: not found, so the door
refused a charge that had already cleared. Permanently — the retry replays the same
receipt into the same absent namespace, and a fresh idempotency key charges the card
again. The payment now carries both names as principal names them, `org` for the data
namespace and `ledger` for the billing key: the receipt is read from org, the deposit
lands on (ledger, subject), and the screen and the accrual key on ledger as before.
A settled charge the credit refuses is still taught to the risk model. Teaching is
telemetry about a payment that HAPPENED, so gating it on the credit dropped real
settlements — a charge in a currency this ledger does not hold, a receipt it could not
read — out of the payer's velocity, and a caller who could provoke the refusal could
pay all day and accrue nothing. Both doors now run one value (screen.record) that holds
the order: the money first, because only it can refuse the door, then the model, always.
A deposit that fails on money ALREADY in the books answers with the money. finance
dedups a replay inside its insert transaction; it cannot cover a transaction that never
reached its own read — the write lost to a concurrent poster of the same Ref, or the
request context that died between the charge and the post. Both leave the ref credited
and the caller holding an error for it, which at a credit door is a 500 on a card that
cleared. A failed deposit now asks the only question it has left, on a context detached
from the caller's because the caller's is what may have just died.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
studio, bot and platform leave appProducts. No catalog version licenses them, and
nothing enforces them either — RequireProduct has zero production callers and the
three sites that would call it say so outright ("DEFERRED — DO NOT ENABLE YET ...
flip on once the catalog licenses it"). Listing them bought no strictness: it made
CheckEntitlement answer Active:false for every org on every tier, enterprise
included, so the projection reported all three LOCKED to every customer behind a
purchase that does not exist. When they launch they come back ATOMICALLY — this
list and the granting plans in one change, never half-wired.
A NOT-GATED PRODUCT IS NOT A LOCKED ONE, and that distinction is why the keys stay.
@hanzogui/shell maps over these names unconditionally, so dropping them from the
projection would break the console rather than correct it. shellApps now says what
must be REPORTED and appProducts says what is licence-CHECKED — the two stopped
being one list the moment cloud stopped claiming products nothing grants — and an
ungated app answers true, which is the truth, instead of a lock nobody can lift.
engine stays in the catalog and out of the console, deliberately. It is licensed by
a DIFFERENT authority: apps/plan/licence.go stamps it into a signed licence
("licensing.product:"+id) that a customer's own engine verifies offline. No console
surface reads it and none should, so the paid-for-nothing test is scoped to the
console's own vocabulary and names the two authorities where a reader will meet
them. engine-rocm resolved itself: the catalog dropped it at v1.4.14.
The test helper derives from shellApps instead of spelling the surface twice — the
count lived in a literal list AND a length check, two places that never mentioned
each other.
FOLLOW-UP, filed not fixed: @hanzogui/shell still advertises studio/bot/world/
platform to the browser as tier "pro", client-side, against a different endpoint.
That third vocabulary is the only place the product intent is written down, and
reconciling the console with the server is its own piece of work.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The console paywall asked the licence authority about "world", and no catalog
version has ever licensed that id. World access was conveyed by a tier-level
`bundles` field pointing at separate world-free/pro/team/enterprise PLANS;
@hanzo/plans carried that at v1.4.4 and deleted both the field and those plans at
v1.4.11, with nothing put in its place.
Asking about a product nothing grants is not a near miss — CheckEntitlement
resolves cleanly and answers Active:false for every org on every tier, enterprise
included, so GET /v1/entitlements reported world LOCKED to every customer
unconditionally. A lock no purchase can lift is worse than no claim at all, which
is what made this a contract failure rather than a missing feature.
apps/world already resolves its limits to the free floor when the plan does not
resolve (entitlement.go ResolveWorldLimits). That is now the whole answer rather
than a degraded one.
TestEntitlements_WorldTiers asserted world-pro/free/enterprise resolve through
the Go/goja path and cannot pass against a catalog that no longer sells them. The
seam it covered is worth keeping, so it becomes TestEntitlements_RetiredTierIsAnError:
a live tier still resolves and carries entitlements, and a retired id ERRORS
rather than answering an empty map — the distinction that matters, because an
empty map reads as "this tier grants nothing" and would silently strip a paying
subscriber of everything they bought.
Not touched: studio, bot and platform are asked about and granted by nothing
either, but unlike world they have never had a mechanism at all — that is a
product decision about whether they are sold, not a stale reference to remove.
engine-rocm resolved itself upstream: the catalog dropped it at v1.4.14.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`on:` declared workflow_dispatch twice - bare at the top, and again under the
comment explaining the resume entry point. YAML forbids a duplicate mapping
key, and the forge reads this trigger by DECODING the block: act/model
WorkflowDispatchConfig() decodes `on:` into map[string]yaml.Node and returns
nil the moment that fails. DispatchActionWorkflow reports nil as
`has no workflow_dispatch event trigger`, so every dispatch of this workflow
answered HTTP 422 - including the RESUME entry point the comment describes and
the call a sync lane makes after a fast-forward, which is the one door left
when a machine push has eaten the event.
Pushes still triggered runs throughout; only the dispatch door was shut.
Keep the documented occurrence, drop the bare one. The trigger set is
unchanged: push(main, tags v*), pull_request, workflow_dispatch.
engine-rocm was a SKU on the Enterprise plan, and nothing else. rocm is a cargo
feature in hanzo-cli/Cargo.toml beside cuda, metal, mkl and accelerate -- peers,
all built from one source tree -- so an AMD owner on Max was refused a binary an
NVIDIA owner on Max could download. That prices the plan by the customer's
hardware brand. Removed in plans v1.4.14, where the test that asserted the SKU
existed now asserts no plan sells an accelerator at all; licensing v0.1.10 says
the same thing where the distinction lives, on Entitlement.ProductID and
Release.Product. Nothing enforced it: no release carries Product "engine-rocm".
.gitignore line 62 was the bare word `tools`. A gitignore pattern with no slash
matches at ANY depth, so it covered apps/tools -- 25 tracked files, the /v1/tools
MCP catalogue -- and plugin/tools. Adding a NEW file to either was refused, and a
refusal to add is how work disappears with no diff to notice it. It sat directly
above a block that anchors every other root binary (/gateway /account /authz
/smoke /gen-app-cmds) and explains why. It is now /tools, in that block, proven
both ways: a new file under apps/tools is visible, a root ./tools binary is not.
Also here: iam v1.34.21 (13 more typed ops, 98 untyped -> 85), commerce's cart,
and the document regenerated from all of it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three defects, each of which alone made the meter charge nobody.
The edge gate billed reads. DefaultPrice priced by path and ignored method, so
declaring a surface at 25c charged 25c for GET /v1/<surface>/list. That is why
0 of 119 surfaces had ever declared a positive price: 95 chose Free, 24 chose
Metered, and DefaultPrice returned 0 for every path in production. Consumes(method)
in price.go is now the one rule, read by both DefaultPrice and spend.go's
Billable, so the charge question and the standing question cannot drift.
Two surfaces promised a meter and kept none. auto and flow declared Metered --
which makes the platform require standing and makes the edge charge nothing, on
the promise that a downstream meter owns it -- and had no meter. Every durable
run and every workflow run was free, invisibly. Both now gate before the work
and debit after. meteredWithoutAMeter is empty: 21 of 24 hold their own meter,
3 meter through the AI wrapper, 0 charge nothing.
The third was the launch blocker. The in-handler meter addressed money by bare
org; the customer's balance page, the edge gate and the paywall all address it
by principal.WalletOf. In a tenant org those coincide. In the shared signup org,
where a self-serve stranger lands, they do not -- so a stranger's top-up went to
hanzo/stranger while 21 products checked and debited hanzo, the platform's own
pool. Proven by mutation: revert the address and a customer who has just paid
gets 402 insufficient_balance. Fourth recurrence of a class apps/principal/wallet.go
already documents three instances of.
prepaid_e2e_test.go runs it over a real socket: real RSA key, real JWKS the
validator fetches, real RS256 bearer, real listener, real net/http client, real
double-entry ledger. Top up 100c, four 25c calls to 75/50/25/0, fifth refused
and the handler does not run; one balanced ledger entry per movement; an
unpriced op and a GET on the priced surface both move nothing (re-funded first,
so they cannot pass by being broke); another org's valid token cannot spend this
org's balance. Run against both addressing modes, because the address is half of
what is being proved.
Chargeable: 24 of 119 surfaces, and all 24 now actually charge. On the correct
payer address: 3 (auto, flow, tools). Migration is atomic per surface -- a
half-migrated surface gates one wallet and debits another, worse than
consistently wrong -- so the remaining 21 move one at a time: principal.Ledger(c)
-> principal.Payer(c) at both the Gate and the Meter in that surface, together.
booksOf makes it inert; a caller still passing a bare org gets the identical
value byte for byte.
Not closed: the money gates are HTTP-only. zip has a per-op authorizer at
op.invoke covering REST, MCP tools/call, CLI LocalInvoke and the ZAP call plane,
and cloud sets none, so a priced op reached over MCP or the plane bypasses
billing entirely.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The usage page 502'd on an honest ledger: per-token debits are
routinely finer than a cent (0.00589 USD is a live row), plane
Minor()'s exactness guard refuses those BY DESIGN — a rounded debit is
a ledger that drifts — and the co-resident usage read was making a
display go through a debit's gate. Sustained 'billing upstream
unreachable' on GET /v1/billing/usage, ~8/min, org hanzo, right now.
RoundMinor is the explicit display rounding that guard demands, made
once beside FloorMinor so every summary makes the same one:
half-away-from-zero to the currency's minor unit, for figures someone
reads — never a balance a gate spends against (FloorMinor: down is the
only safe direction) and never a debit (exact or refused, unchanged).
The test pins the 502's own row and that Minor() still refuses it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The first fully green gate pass this pipeline ever produced skipped its
image: the run was an API rerun, act_runner serves github.ref as null
on those, and the image guard read null != refs/heads/main as 'not
main'. The guard now accepts the rerun's null-ref shape (the workflow
only triggers from main pushes, tags and dispatch, and PRs are still
refused by event_name), and workflow_dispatch exists so a green head
can be re-driven without an empty commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The fleet sweep builds one binary per app — 120 of them, ~37 MB each — and kept
every one for the length of the run. Nothing reads them after `describe` has
written that app's subset, so 4.3 GB sat in bin/ purely because no line removed
it.
That is not tidiness. The gate runs on a shared runner whose docker-storage is
an EmptyDir capped at 38Gi, and the cap is enforced by EVICTION: the pod is
killed mid-job, the log simply stops, and the run reports failure with no error
in it. Measured twice today (git-runner-6 at 20:44:35, git-runner-5 at 21:15:31,
both "Usage of EmptyDir volume docker-storage exceeds the limit 38Gi"), each
landing inside app-contract.
Freeing as it goes takes bin/ from 4.3 GB to 74 MB across a full sweep, verified
green on the same tree: 1710 paths, regenerated from source and unchanged.
It does NOT make the gate fit on its own — the Go build cache from linking 120
binaries dominates, and a clean runner still climbs ~20 GB during this target.
This removes the part that was pure waste; the rest is a capacity question about
the runner, not about the sweep.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A customer's card was charged and the balance that decides whether an inference
request is served never moved. commerce's card core credits its OWN transaction
store; every spend gate reads apps/finance — build.go's balanceReader,
apps/metering's fetchAvailable, GET /v1/billing/balance are all fin.Balance — and
the two are different files on disk. The only door that reaches finance,
POST /v1/billing/credit, is not mounted. So a paid top-up funded nothing.
Both credit doors already compose one value onto their handler, which is where all
four projections of the typed op and the browser's raw route converge on "the
charge cleared". That is where the deposit is issued now, before the risk record
and, unlike it, synchronously: a credit that cannot be posted refuses the door with
a 500 and a RECONCILE line rather than answering success to a customer whose money
went nowhere. The customer's own retry is then the recovery — the money core
replays the receipt and the deposit completes the half that was missing.
The address is the payer's wallet, (payerOrg, principal.Subject), which is
byte-for-byte what principal.WalletOf hands the spend gate and apps/billing hands
the balance route. That also closes the split payments.go recorded: commerce's
typed door credits the org POOL, while for a member of the shared signup org — or
a credential carrying a signed person: billing_account claim — the wallet the gate
reads is the person's, so an agent's payment funded a balance nobody could spend.
The key is the settlement's own reference (firstRef: the processor's, else the
receipt), the same key the risk record is filed under, so a replay, a webhook and
both doors credit one payment once. The amount, the currency and the books come
off the RECEIPT, never off the request: commerce replays the first receipt for a
repeated idempotency key whatever amount the repeat carried, so sizing a credit
from the request would let a settled $5 charge be replayed as $5,000. A sandbox
charge credits the sandbox books, because the receipt states which bucket cleared
and finance keeps them in separate files.
The deposit goes through finance's own Deposit rather than commerce's injected
creditledger adapter for one reason: CreditInput has no test field, so that seam
cannot say which books to write.
Tests: a settled top-up at either door funds the wallet the gate reads, the
customer's /v1/billing/balance reports it, and a metered debit spends it; three
posts of one settlement credit once; one payment answered at both doors credits
once; a $42 settlement replayed under a $5,000 request credits $42; a sandbox
charge leaves the live books at zero; and every way the credit can fail refuses
the door instead of answering 200. Seven mutations — dropping the key, the credit
at either door, the test flag, keying on the receipt, sizing from the request, and
swallowing the refusal — are each caught by a named test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A customer's card was charged and the balance that decides whether an inference
request is served never moved. commerce's card core credits its OWN transaction
store; every spend gate reads apps/finance — build.go's balanceReader,
apps/metering's fetchAvailable, GET /v1/billing/balance are all fin.Balance — and
the two are different files on disk. The only door that reaches finance,
POST /v1/billing/credit, is not mounted. So a paid top-up funded nothing.
Both credit doors already compose one value onto their handler, which is where all
four projections of the typed op and the browser's raw route converge on "the
charge cleared". That is where the deposit is issued now, before the risk record
and, unlike it, synchronously: a credit that cannot be posted refuses the door with
a 500 and a RECONCILE line rather than answering success to a customer whose money
went nowhere. The customer's own retry is then the recovery — the money core
replays the receipt and the deposit completes the half that was missing.
The address is the payer's wallet, (payerOrg, principal.Subject), which is
byte-for-byte what principal.WalletOf hands the spend gate and apps/billing hands
the balance route. That also closes the split payments.go recorded: commerce's
typed door credits the org POOL, while for a member of the shared signup org — or
a credential carrying a signed person: billing_account claim — the wallet the gate
reads is the person's, so an agent's payment funded a balance nobody could spend.
The key is the settlement's own reference (firstRef: the processor's, else the
receipt), the same key the risk record is filed under, so a replay, a webhook and
both doors credit one payment once. The amount, the currency and the books come
off the RECEIPT, never off the request: commerce replays the first receipt for a
repeated idempotency key whatever amount the repeat carried, so sizing a credit
from the request would let a settled $5 charge be replayed as $5,000. A sandbox
charge credits the sandbox books, because the receipt states which bucket cleared
and finance keeps them in separate files.
The deposit goes through finance's own Deposit rather than commerce's injected
creditledger adapter for one reason: CreditInput has no test field, so that seam
cannot say which books to write.
Tests: a settled top-up at either door funds the wallet the gate reads, the
customer's /v1/billing/balance reports it, and a metered debit spends it; three
posts of one settlement credit once; one payment answered at both doors credits
once; a $42 settlement replayed under a $5,000 request credits $42; a sandbox
charge leaves the live books at zero; and every way the credit can fail refuses
the door instead of answering 200. Seven mutations — dropping the key, the credit
at either door, the test flag, keying on the receipt, sizing from the request, and
swallowing the refusal — are each caught by a named test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The gate compares cloud's appProducts against @hanzo/plans licensing.product_ids
because those are the two lists the server can reach. They are not the two lists
that decide what a customer sees.
@hanzogui/shell's APP_ENTITLEMENTS (hanzo-registry.ts) maps studio, bot, world and
platform to a minimum tier of "pro" and enforces it CLIENT-SIDE, against
GET /v1/billing/subscriptions — a different endpoint from the projection this
package serves. Nothing shipped reads GET /v1/entitlements at all: the shell hook
the projection's own doc comment names as its consumer reads the billing endpoint
instead, and Console reads the ENABLEMENT store. So the projection is a correct
answer to a question no client asks, and the only written-down statement that these
apps are paid lives in the client, where it is advice rather than enforcement.
That map also names the mechanism the catalog really used. Its world entry reads
"bundled via world-pro on pro/plus/max, world-team on team" — the tier-level
`bundles` field, which v1.4.4 carried for exactly those tiers and v1.4.11 deleted
along with the world-*/social-* ladders, with nothing put in its place. cloud never
read `bundles`. So World was granted by a key cloud does not consult, and is now
granted by no key at all.
Three vocabularies, then, not two, and the honest resolution collapses all three.
Recorded in the header and in the failing message so whoever picks it up starts
from the list that already has an answer instead of rediscovering it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nothing compared the two lists that decide whether a paying org may open a
console app, so they drifted until they shared one element out of eight.
ASKED apps/entitlements/require.go appProducts
= studio, bot, world, platform, team
GRANTED @hanzo/plans entitlements["licensing.product_ids"], union over
EVERY tier of v1.4.4, v1.4.11 and v1.4.12
= engine, engine-rocm, team
studio, bot, world and platform are licensed by no plan in any version, so
CheckEntitlement resolves cleanly and answers Active:false for every org on
every tier, enterprise included. That is not open-by-default: the licence leg
is dead, cloud.Stand falls to the wallet alone (LicenceNone + a readable zero
balance = Unpaid), and GET /v1/entitlements reports the app LOCKED to every
customer unconditionally, because the projection reads no kill switch. The
enforcement path is dark today only because paywall_enforced defaults off and
bots/world/platform still carry their RequireProduct markers commented out.
The other direction is the evidence for why. licensing.* is one coherent
block describing a signed ENGINE licence — app_ids, product_ids, seats,
engine_features — and apps/plan/licence.go consumes it exactly that way. When
the catalog gated a console product it used a different mechanism entirely:
World had its own tier ladder (category "world") and its own world.* keys,
which apps/world/entitlement.go reads, and 4633fe56 measured the same thing
from the other end ("the world-*/social-* lines carried no licensing block
even while on sale"). team overlaps only because team is genuinely both a
console app and a seat SKU. A spelling coincidence is not a shared vocabulary.
So this states the invariant instead of the answer, because which side gives
is a pricing decision and not the gate's to make. The gate is a COMPARISON,
not a pin: pinning today's five strings would have passed every day of the
drift, since both lists were internally consistent the whole time and wrong
only about each other. It re-derives both sides on every run and names both,
in both directions, and it FAILS today — truthfully, the way apps/plan's
TestEntitlements_WorldTiers does.
The probe carries a positive control. Reading product_ids at the top level
instead of nested under entitlements returns an empty set for every plan,
which is a clean and confident WRONG answer that would make the whole gate
pass vacuously; a probe that finds no grants anywhere now says its own query
is broken rather than reporting the catalog as empty.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
8.5.55 (console 12d061f57) moves the playground Response panel to a
full-width row directly under the surface tabs on every screen size,
lets an org upload its logo in Settings (compact data URL, one save
path), renders that logo in the context switcher's org slot, and drops
the expanded rail's separate brand row — the switcher carries the
identity, so the H above it said the same thing twice.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
CheckEntitlement asked @hanzo/plans which products a subscriber's tier licenses.
The catalog lists what is ON SALE TODAY, so a tier retired at 1.4.5 resolves to
404 -> found=false -> continue -> Active:false. That is a definitive "not
entitled", indistinguishable from a real refusal, for someone commerce is still
charging. On the team product it reaches cloud.Refuse(ReasonUnpaid): a paying
subscriber gets a 402 for a licence they hold.
This is the licensing half of the cut Paid already made for the paywall. Ask what
you bought, not what is for sale. The tier's authority row survives retirement —
that is what Status is for — and commerce v1.50.9 persists the licensing block on
it and backfills the rows archived before the field existed, so the row can now
answer.
So the resolver reads the row. Features are derived from the block it carries;
only the token spelling lives in Go, and a spelling is not a policy — the
plan->product decision is the row's.
Scope, measured rather than assumed: of the thirteen retired slugs only plus and
team-max ever licensed anything, both ["team"]. developer, custom and the
world-*/social-* lines carried no licensing block even while on sale, so nothing
regressed for them and nothing is invented for them here.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
969d755c added a cloud.Request call site in apps/commerce/risk.go without an
allowlist entry, so TestRequestEscapeHatchIsPinned failed on that commit ALONE
(proven on a clean tree). That test is car 0 of the release train and every
later car `needs:` it, so main has been unshippable since it landed.
The escape hatch is justified and the reason was already written at the call
site; this records it where the gate reads it. The credit screen serves BOTH the
browser's REST call and an agent's tools/call, and reads the PAYER, the billing
ADDRESS and the JURISDICTION off the request it is being served over — parked by
the app-wide Bridge, which runs for /mcp and the op path alike.
None of the three can be an In field, for exactly the reason apps/dataset is
already allowed here: a caller that could name its own payer would charge
another org. And on the MCP leg no HTTP response exists yet when this returns,
so a response reader would watch an agent's payment settle and learn nothing.
The screen fails closed when the request is absent.
The gate now passes. Nothing else changed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Under `set -e` this loop stopped at the first app that failed, and every app
after it never ran. An app that never ran reports nothing, and nothing is
indistinguishable from passing — so the gate was silently checking a prefix of
the fleet and reporting success.
On 2026-08-05 an unbalanced brace in apps/commerce/describe.go stopped the loop
there. Roughly 45 apps behind it never regenerated. Because `meet` was where the
run appeared to end, three separate people diagnosed a defect in `meet` — and
one really was there, so the misattribution survived scrutiny. Two independent
defects braided by one truncation, and the truncation is what made them hard to
separate.
Now the loop runs every app, collects the failures, names them, and exits
non-zero. A gate that hides what it did not check is worse than one that checks
nothing, because it is believed.
Proven by mutation: with a parse error injected into apps/eval, 117 apps still
ran, the summary named `eval`, and make exited non-zero. Removed, 117 ran and it
exited 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
No overlap — 969d755c is apps/commerce and its generated openapi
artifacts; this branch is the plane seams, the JWKS derivation and the
two silent zeros. Clean merge, no conflicts.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip v1.26.0 derives an operation id from the ABSOLUTE path with a parameter
written by_<name>, which is the rule cloud's openapi package already used. So
operationID and sanitize here were a SECOND implementation of one rule, kept in
step by nothing but attention. They are deleted; From calls zip.ID.
The proof that they were a true duplicate rather than merely a close one is that
`make describe` regenerates openapi.yaml BYTE-IDENTICAL across this change —
same sha256, 1710 paths, no diff in any of the 119 subsets. A refactor that
claims to remove a copy should be able to show the copy said nothing of its own,
and this one can.
The renames those two rules used to disagree about landed already in 0d602fc9;
nothing about the published document moves here.
The param-vs-literal test goes with the code it tested. It was a unit test of
this package's copy of the rule, and the rule now lives in zip, which grew its
own tests for it in zip a774624 — the collisions the encoding has to survive: a
parameter against a literal of the same name, the hyphen that must not fold into
the separator, the four spellings of one parameter that must reach one name, and
the '_' aliasing it deliberately CANNOT resolve.
Literals pinned in tests move because the RULE says so, not to go green:
apps/dataroom 10 dotted (v1.dataroom.get_datarooms_id -> get_v1_dataroom_datarooms_by_id)
apps/ingress 4 (put_v1_ingress_routes_id -> put_v1_ingress_routes_by_id)
apps/automations 1 (get_v1_automations_runs_id -> get_v1_automations_runs_by_id)
apps/notify 1 (v1.notify.post_send -> post_v1_notify_send)
apps/guide a comment that still described the dotted scheme as current
Suite: 183 ok, 0 build failures. Eleven tests go green — dataroom's MCP tool
list and its agent round trip, automations' and notify's by-name calls,
ingress's tool prose — and no test fails that did not already fail on a pristine
checkout of this base.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
mountedStore answers ErrNoPeer, and three sibling functions answered the
same fact — agents is not in this process — with a bare string a caller
cannot test for. RunOnBehalf, namespaceFor and eachStore now say it the
one way, so "not here" is routable wherever it is raised rather than only
where it was noticed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two seams returned a zero with no error, and both are seams whose zero is
read as an answer.
agents.StopSessions answered (0, nil) when the package was not in the
process. agents ships as its own binary and the revoke lives in link, so
that was every deployment: a credential revoke replied 200 with
{"sessionsStopped":0} having torn down nothing, while the sessions kept
running under the revoked account. It now errors with ErrNoPeer, and link
takes the plane leg — two doors onto the ONE StopSessions, so the actor
scoping that bounds a revoke to its own user's sessions holds across the
boundary. The org is the caller's plane identity and never an argument;
the actor is built on the answering side from the org the plane proved.
authors.AccrueForOrg had no error channel at all. Same split, same
consequence, on the money path: the royalty leg of every affiliate sweep
latched nothing and reported a completed accrual of zero. It returns
(int, error) now, absence is ErrNoPeer, and a lookup failure is returned
rather than logged-and-zeroed — a sweep that could not read the authors is
not a sweep that found none. The sweep reports royaltyFailures beside the
count.
An empty match stays (0, nil) in both: fail-closed is a real answer, and
turning it into a fault would break a revoke that legitimately matches
nothing.
projects.LiveSites, the third of the three, was already fixed on main —
Ready() plus the plane leg in apps/catalog serving(), with corpus()
returning the error so a failed read cannot prune the catalog.
plane/gen was stale: the ops the seam work added were declared but their
typed clients had never been regenerated, so plane/git, plane/platform
and plane/projects were missing them and plane/sync and plane/tracker did
not exist. Regenerated.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The session-replay write door landed as routes and typed ops but the document
was not re-projected, so /v1/replay existed in the binary and in no client: not
in openapi.yaml, not in analytics' subset, and therefore in none of the eight
generated SDKs, the MCP tool list or the spec-derived CLI.
Regenerated from source, which is the same fix surface-check names. The floor
ratchet follows the routes that now exist.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip v1.26.0 gave an operation ONE naming rule, and the bump landed without
re-projecting the document it renames. 758 operationIds across openapi.yaml and
55 plugin subsets still carried the old spellings — post_collaborator_rpc_documentId
and v1.admin.affiliates.post_id_approve where the rule now says
post_collaborator_rpc_by_documentid and post_v1_admin_affiliates_by_id_approve.
Nothing about the surface moved: same paths, same operations, same prose. An
operationId is the METHOD NAME in eight generated SDKs, the tool name in the MCP
list and the command in the spec-derived CLI, so leaving it stale means every
client offers a name the document no longer agrees with.
This is what surface-check exists to refuse, and it is why the gate has been red
on main: `make describe`, then commit what it wrote.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The release gate is car 0 of the train, and it stops at any app that cannot
project its own document. /meet and /tracker are bound with All(), so each
publishes every method the generator knows — fourteen operations apiece with no
handler to lift a sentence from, because a static bundle has no typed op. meet
died there and tracker would have next.
tasks had already solved it, in forty lines this file would have needed three
copies of. So the declaration is asked for rather than restated:
openapi.DescribeSPA(prefix, name) derives both addresses, the working methods
and the leftovers from the one fact that differs, and tasks now calls it too.
The copy tasks carried had ALREADY drifted — it promised a missing asset
"answers 200 with HTML rather than 404" after spa.Handler had begun answering
404 under assets/, so the published document described behaviour the binary no
longer had. The shared sentences state spa.Handler's actual policy, including
the assets/ exception and the 503 an unsynced bundle answers with.
Regenerated: openapi.yaml, plugin/{meet,tracker,tasks}/openapi.json, and the
floor ratchet (1700→1705 paths, 2354→2383 operations).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The door session recordings never had, plus the removal of the second token that
crept in with it. Cloud authenticates once against the IAM-issued publishable
pk- and puts the resolved ORG on the message as a routing fact; the ingester
files by that. No hi_ project token on the wire, and no org=token env map.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The snapshot message carried a `hi_` Insights project token, read from a
hand-maintained CLOUD_REPLAY_TOKENS env map of org=token. That was a second
token system — minted by Insights, not by IAM — made load-bearing on a path that
had ALREADY authenticated. IAM issues credentials here; nothing else does.
So the token is gone, and nothing replaces it. The door authenticates once,
against the IAM-issued publishable pk-, exactly as every other write on this
surface does. What the message now carries is the ORG that credential resolved
to — a routing fact, in one copy, in the header the consumer reads. The ingester
resolves it to the project the org owns and files the recording there.
Tenancy is not weakened, it is narrowed to one source: the org is whatever
eventTenant produced server-side, never a body field and never anything a caller
can set. The old fail-closed refusal of an unconfigured org goes with the map it
guarded — there is no longer a mapping that can be missing, so an authenticated
caller cannot be told its own org is "not configured for replay". Admission is
authentication and nothing else.
The envelope's `token` field goes too. The consumer never read it; it existed
because the fork's schema has one, and a tenant value that nothing reads is a
value that can only disagree with the one that does.
Also drops maxReplayEvents. Two caps bounded one thing — how much a single
request may carry — and the byte cap is the one derived from a real limit (the
1 MiB bus payload, with room for the double encoding). A 2000-event cap is an
arbitrary second answer, and a recorder that hit it got a 400 about a batch
whose only problem was its size.
Kept, and load-bearing: the 403 for a reduced principal (a screen recording has
no safe projection), the derived 413 (an honest bound a recorder can chunk
against), the synchronous produce (a receipt means durable), and the
manifest/apps.go prefix row (a path omitted there falls to commerce's bare /v1
catch-all and 405s silently).
The wire contract test is the contract — the consumer is another process in
another language — and it now asserts the org header, the absence of any token
header, and that no `hi_` reaches the message at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Session recordings listed and would not play, because nothing captured DOM and
nothing produced the snapshot stream. The rest of the pipeline was already there
and already working: an ingester consumes
`session_recording_snapshot_item_events`, writes a snappy block to object
storage, and lands the row carrying `block_urls` that the player reads back.
Only the door was missing.
So this writes no blob and invents no format. It renders an accepted batch as
the ONE message that consumer already parses — the partition key, the seven
headers, and the double-encoded envelope — and produces it. `snapshotRecord` is
pure, so the whole wire contract is asserted byte for byte with no broker and no
clock: the consumer is in another language and another repo, where nothing in
this build could catch a drifted header name.
The public wire stays ours — sessionId, windowId, distinctId, events — and the
$-prefixed vocabulary is translated in the one place that has to speak it. The
token travels in a header because that is the copy the consumer reads; it is
written from the same argument as the envelope's, so the two cannot disagree.
Admission is the existing one: eventTenant, the same publishable pk- and the
same refusals as every other write, and the tenant comes from the credential
rather than the body. An org with no configured team token fails closed instead
of answering 200 to a batch it would discard.
Verified end to end against the live pipeline: this door's own rendered bytes,
carrying a real 19-event browser recording, produced a block
(?range=bytes=0-1911) whose min_first_timestamp equals its first block
timestamp, and the recording plays in the insights UI with its inputs masked.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
CLOUD_JWKS_URL exists because the public issuer host is fronted by
Cloudflare, which 403s a server-side loopback — iamurl.go's own comment
root-caused that on 2026-07-04. Production pins the in-cluster address
against CLOUD_IAM_ISSUER=https://hanzo.id for exactly that reason.
jwksURLFor honoured the pin and was unexported, so the two planes that
could not reach it built the URL themselves: durable.go's gated ZAP
listener and apps/base's per-app pool each concatenated the suffix onto
the ISSUER. Both fetched signing keys from the host that refuses them
while the edge validator used the working one — one fleet verifying one
set of keys at the front door and a different set behind it.
It is exported as JWKSURLFor now and all three callers go through it.
TestJWKSHasOneDerivation pins the behaviour, and because the defect was
not calling it wrongly but not calling it at all, it also walks the
source: one file may spell the path as a URL, and naming durable.go is
what the check does when the inline rebuild comes back.
Also lands the rest of the deps-policy delta over 06c672b: the Deps.Brand
doc named "osage" and "any customer brand", neither of which the registry
has (brand.For folds an unknown id to hanzo silently); apps/platform's
`var apexOf = brand.Apex` alias is deleted with its call sites reading
brand.Apex directly, and apps/sites' unrelated apexOf — the published-site
ZONE, a different question with the same name one package away — is
siteZone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ActivePaidPlan classified a subscriber by looking their tier up in the
@hanzo/plans catalog. The catalog lists what is ON SALE TODAY; a
subscription records what was bought. plans v1.4.10 retired the
plus/team-max/custom ladder, so every org still paying for one of them
stopped resolving as paid — LicenceNone at the spend gate, a 402 to a
customer commerce was still charging.
Commerce goes out of its way to prevent exactly this: a retired plan is
ARCHIVED, never deleted, "so invoices and renewals that already reference
it still resolve — retiring a tier stops new sales; it never strands a
subscriber". The subscription carries that row itself, frozen at subscribe
time (StartSubscription: sub.Plan = *p), category and price included.
So classify on the row in hand and delete the second lookup. plan.Paid
becomes a pure predicate over a tier's money facts — an account category
that costs money — with no catalog read, no sync.Once and no error leg to
fail open through, because there is no longer a catalog that can be
unreadable. One authority (what the customer bought) instead of two that
drift apart the moment the price list changes.
Category still does the product-line work it always did: world/social/dns
are not cloud accounts and never clear the cloud paywall, retired or not.
Also re-points three plan tests that pinned the pre-v1.4.10 catalog
(the ladder canary now reads go $9 / dev $19 / pro $49 / max $99 /
team $25 per-seat, and the vocab floor no longer assumes the entitlement
namespace set only grows).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The screen was ROUTER middleware. zip records a typed op ONCE and four projections read
that one record — the REST route, the MCP tool, the by-name call plane and the CLI — and
all four dispatch to the op's HANDLER (registeredOp.invoke). Router middleware is composed
around the fiber handler the REST route is served through and around nothing else. So
POST /v1/payments was screened and `takePayment`, which is published in tools/list, was
not: an agent calling the tool reached commerce's one card money move with no decision in
front of it, and the settlement it produced taught the model nothing. The control was
present on the browser's door and absent on the agent's, and every HTTP test stayed green
because HTTP is the one plane it worked on.
The screen is composed onto the HANDLER of both doors now — screen.op for the typed op,
screen.route for the raw top-up — so it sits inside the value every projection invokes.
Only the composition point moved. The decision, the payer rule and the settlement key are
the landed ones, resolved once per payment into one `payment` value that the screen judges
and the record teaches, so the two halves cannot name different subjects.
Composing on the handler also stops the screen reading the wire, which is what lets it
work off the REST path at all. The amount is the DECODED input's, because over MCP the
request body is a JSON-RPC envelope with the payment inside `arguments` — a body reader
states no value on exactly the door an agent calls, leaving the sharpest axis a credit
door has blind there while it reads fine on the browser's. The settlement is the RETURNED
receipt, because over MCP the op's answer is wrapped in a tools/call result and no HTTP
response exists when the handler returns. The payer, the address and the jurisdiction
still come off the request, which the app-wide Bridge parks for /mcp and for the op plane
exactly as it does for a REST route, so all three resolve on every plane with a connection
behind it; a call with no request at all resolves no payer, is screened as that state, and
is refused by the handler's own gate rather than exempted from the screen.
The structural check reads WHERE THE VALUE WENT rather than which spellings appear. Every
credit door's handler expression must be built from an identifier that holds the screen,
and nothing anywhere may compose the screen as router middleware. The check it replaces
was satisfied by the exact registration that carried this bug.
Shadow is unchanged. No organisation is armed and defaultRegime is untouched; the screen
reaching two more planes widens the record, not the enforcement.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/store/{storeid}/authorize, /capture and /charge have been served by this
binary for as long as it has existed. The cart they operate on had no address at
all: hanzoai/commerce implements the whole noun — models/cart, api/cart, the
route table — and cloud never mounted it, so the documented flow began at step
two. The docs taught /v1/cart anyway, and it resolved to ai's bare /v1
remainder, whose prepaid balance gate would have made filling a basket require
the balance the basket exists to create.
Four typed ops close it, on the payments.go pattern rather than a passthrough:
openCart, getCart, setCartItem, discardCart. The RULES are not restated — the
module's own cart.Cart.SetItem still resolves a product or a variant into a line,
updates a quantity in place and drops the line at zero. What is added is the part
that was actually missing: an address, a declared input, a declared answer, and
prose, so the operation reaches the OpenAPI document, the MCP tool list, the SDKs
and the CLI instead of being a route and nothing else.
One way to change a line. There is no DELETE beside the set: quantity zero
removes, because that is what SetItem does and a second spelling of one act is a
second set of edge cases. Quantity is the RESULT and not a delta, so a retry is
safe and a double-submit cannot double an order.
Identity is not an input. The org comes from the validated principal cloud.Bridge
parks on the context, never from a field, so a cart is created, found and amended
only inside the caller's own namespaced store. Another tenant's id answers 404 on
both the read and the write — never 403, which would confirm it exists.
Exercised against a running binary behind a real JWKS issuer, not a fake: open
201, read 200, set refused 400 on each of its four bad shapes, discard 200 and
idempotent, unknown id 404, anonymous 403, and a second org refused acme's cart
id on both verbs.
The wider admin surface — products, orders, customers, promotions, inventory,
tax, regions, fulfillment — is implemented in the module and still unrouted here.
This adds the cart and claims nothing else.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The screen was ROUTER middleware. zip records a typed op ONCE and four projections read
that one record — the REST route, the MCP tool, the by-name call plane and the CLI — and
all four dispatch to the op's HANDLER (registeredOp.invoke). Router middleware is composed
around the fiber handler the REST route is served through and around nothing else. So
POST /v1/payments was screened and `takePayment`, which is published in tools/list, was
not: an agent calling the tool reached commerce's one card money move with no decision in
front of it, and the settlement it produced taught the model nothing. The control was
present on the browser's door and absent on the agent's, and every HTTP test stayed green
because HTTP is the one plane it worked on.
The screen is composed onto the HANDLER of both doors now — screen.op for the typed op,
screen.route for the raw top-up — so it sits inside the value every projection invokes.
Only the composition point moved. The decision, the payer rule and the settlement key are
the landed ones, resolved once per payment into one `payment` value that the screen judges
and the record teaches, so the two halves cannot name different subjects.
Composing on the handler also stops the screen reading the wire, which is what lets it
work off the REST path at all. The amount is the DECODED input's, because over MCP the
request body is a JSON-RPC envelope with the payment inside `arguments` — a body reader
states no value on exactly the door an agent calls, leaving the sharpest axis a credit
door has blind there while it reads fine on the browser's. The settlement is the RETURNED
receipt, because over MCP the op's answer is wrapped in a tools/call result and no HTTP
response exists when the handler returns. The payer, the address and the jurisdiction
still come off the request, which the app-wide Bridge parks for /mcp and for the op plane
exactly as it does for a REST route, so all three resolve on every plane with a connection
behind it; a call with no request at all resolves no payer, is screened as that state, and
is refused by the handler's own gate rather than exempted from the screen.
The structural check reads WHERE THE VALUE WENT rather than which spellings appear. Every
credit door's handler expression must be built from an identifier that holds the screen,
and nothing anywhere may compose the screen as router middleware. The check it replaces
was satisfied by the exact registration that carried this bug.
Shadow is unchanged. No organisation is armed and defaultRegime is untouched; the screen
reaching two more planes widens the record, not the enforcement.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Seven remote seams returned a zero value when their owning app was not in
the process: OnGitPush answered nil, so every push built nothing and
reported success; OnServiceRelease answered nil, so every release patched
no CR; projectIsForeign answered false, so the cross-org project guard was
off across the fleet. Absence now travels as ErrNoPeer over the existing
Ask plane rather than as a zero that reads like an answer.
Four genuinely-local seams stay local, each with its reason written down.
TestAllListed parses the package for Register* and fails the build when a
new one is added without declaring itself remote or local, so the class
cannot come back.
plane/plane.go conflicted because both sides appended types after Site and
each left its last struct open around a shared closing brace. The two sets
are disjoint — LiveSite* is the site directory, the branch's are the git,
sync, tracker, platform and projects call shapes — so both are kept and
LiveSite is closed on its own.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The gates' first full execution found the tail their absence had been
hiding: projection tests pinning path-derived operationIds while zip
spelled a Group-declared op dotted (v1.agents.post_targets). v1.26.0
settles it the published way — an id derives from the absolute path the
occurrence answers at — and the agents projections go green untouched.
Three harness truths follow the same stricter build: the bots fleet
harness mounted the relay twice (Mount already owns it) and zip now
refuses the duplicate it used to hand-detect, so the guard test pins
zip's own refusal; the route scanner skips fiber's '/' middleware
chains (stacked by design) and the derived HEAD rows; and the measured
partition reads 10 served — the document now carries every method of
the relay's one All() registration, each named in the ledger.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
mk/fleet.mk defines openapi-weave, describe-apps and surface-check. The root
Makefile had no include line, so all three were reachable only as
`make -f mk/fleet.mk <target>` — a path nobody would guess, mentioned nowhere,
and absent from `make help`. surface-check is the DRIFT GATE; it sat behind a
door with no handle. fleet.mk's own header has always said "included from the
repo root (`include mk/fleet.mk`)". It just never was.
Including it needed one real fix first. ROOT was:
ROOT := $(abspath $(dir $(firstword $(MAKEFILE_LIST)))..)
firstword is whichever makefile make STARTED with. Run as `make -f mk/fleet.mk`
that is fleet.mk and the root resolves correctly — but INCLUDED from the root
Makefile it is `Makefile`, so ROOT became the repo's PARENT and the `include
$(ROOT)/mk/go.mk` beneath it pointed outside the tree. lastword is this file in
both cases, which is what makes the header's claim actually true.
And pinning .DEFAULT_GOAL := help, because the include inserts three targets
ahead of `help` and make takes the first target it parses — without it, typing
`make` would silently have run the openapi weave. I introduced that regression
with the include and caught it by running bare `make`; positional defaults are
exactly the kind of thing that should be stated, not inferred.
Verified both doors and the neighbours: `make -f mk/fleet.mk -n openapi-weave`
resolves, `make -n openapi-weave` resolves, bare `make` prints help, and
`describe` and `compose` still parse. All three targets now appear in help.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Reconciles blue/meet-ui into main beside blue/tracker-ui, and settles the two
places the branches disagreed.
/v1/meet/health is main's TYPED op, not the branch's untyped route: zipdoc
lifts its prose from the op, so the branch's openapi.Describe for it would have
been a second copy of the same sentences drifting against the first. The
session route keeps its Describe — it is still untyped, and prose declared
beside the route table is the only way an untyped op reaches the document.
spa is now the ONE SPA handler, and it is tracker's rules rather than the
weaker set three apps shared:
- a miss under assets/ is 404, not the shell. That subtree is content-
addressed, so a name that is not there is a stale shell asking for a purged
chunk, never a client-side route; answering HTML hands a <script> tag a
document and reports "Unexpected token '<'" for what is a cache miss. It
stays 503 while the bundle has no shell at all, so an unsynced deploy is
still loud on every path rather than a plausible 404.
- a directory is not a page. http.FileServer lists one, so serving whatever
stat succeeds on published the whole asset manifest at /<app>/assets/.
- `//go:embed dist`, not `all:dist`, in all four ui packages. dist/.sync-stamp
names the source repo, branch and commit of each bundle; `all:` embedded it
and the handler serves anything it can stat, so /meet/.sync-stamp,
/tasks/.sync-stamp and /research/.sync-stamp published that. The stamp stays
in git and leaves the binary.
Proven by booting both plugins, not only by tests: /tracker/ and /meet/ answer
the shell no-cache, a deep link falls back to it, a hashed asset carries the
immutable hint, a purged chunk and a directory are 404, .sync-stamp is not
served, /v1/tracker/health is 200 and /v1/meet/session is 401 with no session.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Reconciles blue/tracker-ui into main: apps/tracker's typed surface, the
embedded SPA at /tracker, and internal/manifesttest — the gate that reads an
app's LIVE router declaration rather than its published document, which is the
only thing that can catch an untyped static prefix.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
8.5.53 (console 0f6cec331) resolves each catalog row's routing id
against the live set — the playground submitted the bundle's
anthropic/claude-haiku-4.5 while the gateway routes claude-haiku-4.5,
so picking Haiku 4.5 errored 'model not available'. It also stops
offering models live under no spelling, and gates frontier-priced
models behind a standing subscription (the gateway's 402 stays the
enforcement point; the picker stops promising what it would refuse).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Same fact as the guide commit, the rest of the estate. 9396df70 moved the
principal enrichment into cloud.Identify and updated 66 packages' harnesses; the
ones here were missed, so each built a bare zip.App, mounted a subsystem on it,
and then watched every typed op refuse a caller production serves — 82 tests
across automations, cms, company, entitlements, erp, framework, help, marketing,
marketplace, prefs and templates.
erp and cms read as a doctype bug and were not one: the module install 403s
first, so the doctype never exists and every later create honestly answers 404.
One cause, two sentences.
Two comments were describing the world before 9396df70 and now describe this
one: marketplace said the subsystem installs the bridge app-wide, and
automations' MCP harness said the plain harness does not install one. The
ordering claim both make is unchanged and still exactly right.
templates' TestTheBridgeIsInstalledAheadOfTheLeaves was pinning the OLD
arrangement out loud — "no app-wide bridge, exactly as this package's tests have
always mounted it". Both its teeth are untouched: a validated caller must get
201, an anonymous one must get 403. Only who installs the enrichment changed,
from the subsystem to the composer, which is what 9396df70 decided.
No gate was loosened anywhere. It cannot be: principal.OrgOf refuses when the
user claim is empty, so the bridge parks nothing for an anonymous request and
every anonymous-refusal test still refuses.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
9396df70 moved the principal enrichment out of all 67 subsystems and into the
one composer, cloud.Identify — correctly: a subsystem asserting identity for
itself repeats a claim it cannot check, and two of those copies sat on nodes
owning no routes, which is the outage it was fixing. It updated 66 packages'
test harnesses to install what the composer installs. guide's four were missed.
A harness that builds a bare zip.App and mounts guide on it therefore ran with
no cloud.Bridge, so principal.OrgFrom and cloud.Request read nothing and every
typed op answered 403 to a caller production serves — the exact failure
Identify's own comment predicts for the boundary installed alone. 27 tests.
The bridge cannot admit anyone: principal.OrgOf refuses when the user claim is
empty, so an anonymous request parks nothing and still refuses.
TestBlueprintAdminRequiresSuperAdmin passes unchanged — anon, a validated
non-admin and an org admin are all still 403, and only SuperAdmin reads the
blueprint. Nothing was loosened; the harness stopped lying about how the
program is built.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Pinned to @v1.0.17, which declares the reusable's `tests` input as
`type: boolean`. The forge substitutes a workflow_call input's DECLARED-TYPE
ZERO VALUE, so `inputs.tests` was always false and the test step was SKIPPED on
every single run. Measured on this repo: 20 skipped, 0 executed.
The most critical service in the fleet has been shipping with its suite never
having run. Not a flaky suite, not a slow one — never executed. The `tests:`
expression right below the pin has been decorative the whole time.
@v1 declares the input `type: string` with `default: 'true'` and guards on
`inputs.tests != 'false'`, so that expression finally means what it reads as.
VERIFIED BEFORE MOVING, because this pin was deliberate: the comment above it
records that v1.0.17 was taken to fix a `set -u` abort. v1 (8e43277) CONTAINS
v1.0.17 (81b3c18) — checked with merge-base against the REMOTE tags, since a
local clone here holds a stale annotated v1 pointing elsewhere. So this gains the
tests without losing the fix the pin was for.
Expect this to go red before it goes green. That is the point.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.50.7 accepted any non-empty signature on the mpc webhook and accepted
everything when no key was set, so a forged unauthenticated POST to
/v1/billing/webhooks/mpc minted credit with a caller-chosen tenant,
beneficiary and amount. v1.50.8 verifies the digest as hex HMAC-SHA256 over
the raw body with hmac.Equal and refuses every state it cannot verify.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`make test` finds the packages to check by grepping the tree for the
//go:generate directive. An agent worktree at .claude/worktrees/<id>/ is a whole
second checkout of this repository, so the walk found 203 packages where there
are 104 — every one of them twice — and went red on the COPY's apps/o11y while
nothing in this module had changed.
--exclude-dir='.?*' is the same rule the Go source-walking gates state
(typed_request_gate_test.go, orgns_test.go, iamurl_test.go, cmd/cloud/mount_test.go):
a dot-directory is not this module's source. The glob is .?* rather than .* so it
cannot match '.' or '..' and the search root survives.
Control-tested: with drift planted in apps/agents/zipdoc_gen.go the gate says
'stale' and exits 1; restored, all 104 packages pass.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestOnlyOrgnsBuildsANamespace walks the tree from the module root and skipped
.git, node_modules, webui and vendor by name. An agent worktree lives at
.claude/worktrees/<id>/ — a whole second checkout of this repository — and the
walk read it, found the SAME two doors (orgns.go and apps/finance/finance.go) at
paths that exist for nobody else, and failed. Nothing in the module had changed.
Every other source-walking gate here already states the rule and says why:
typed_request_gate_test.go ('a leftover working copy'), iamurl_test.go,
cmd/cloud/mount_test.go. One rule, one spelling — a dot-directory is not this
module's source. This was the only gate that had not learned it.
Control-tested both directions: with namespace.Parse planted in a real package
the gate names the file and fails; with it removed, and with two agent worktrees
present, it passes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerce holds ONE card money move and this binary opens two addresses onto it.
POST /v1/billing/topup/token was screened; POST /v1/payments ran the same
billing.TakePayment — the same bounds, the same idempotency guard, the same
mintauth-authorized deposit — with nothing in front of it. So the risk gate was
not a bound on the mint, it was a bound on one entrance, and the other entrance
is the one published as an MCP tool for an agent to call.
Both doors now hold the SAME screen, resolved once at the composition root and
handed to each registration. The gate becomes a zip.Middleware because a typed op
has no handler chain to sit in — zip composes middleware around it at
registration — so one gate reaches both forms; screenChain hands the raw route
its chain as `next`. The decision, the payer rule and the settlement key are the
landed ones, not copies.
Because both doors resolve the payer through payerOrg + principal.Subject and key
their settlement on the gateway's own payment id, a burst split across the two
accrues on ONE subject and one payment reached through both converges to one
observation. settlementOf learns the typed door's spelling of the ledger receipt
(`id`, where the browser answers `transactionId`) so a settlement whose processor
states no reference is still keyable there, and the settlement check reads the
whole 2xx band because the typed op declares 201.
The read op is not screened: it mints nothing, and a scorer that is present and
mute must not be able to withhold a customer's own receipt.
Shadow is unchanged. No organisation is armed and defaultRegime is untouched;
widening the gate widens the record, not the enforcement.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestRiskGate_ANotDeployedScorerDoesNotCloseTheCreditDoor and
TestScoreOverPlane_AnUndeployedPeerIsAbsentRatherThanAnOutage failed on every
Mac and passed in CI.
A unix socket address is capped near a hundred bytes — 104 on darwin — and
t.TempDir embeds the calling test's own name. The second of those names is 61
characters, so the sockaddr went over the cap and every dial answered
`connect: invalid argument`.
That is the part worth writing down: the failure looked exactly like the thing
under test. Both tests assert that an UNDEPLOYED peer is absent rather than an
outage, and an over-long address produces a dial error indistinguishable from a
peer that is not listening. A test whose environment failure mimics its own
subject cannot be read at a glance, and this one was read as a real regression.
shortRuntimeDir is the same fix, for the same reason, as servePeerLedger in
ledger_peer_test.go — an anonymous short-named dir keeps the address inside the
cap on every platform. Named and commented so the next test that needs a working
socket uses it instead of rediscovering the cap.
unusableRuntimeDir is untouched: it builds a deliberately over-long address to
assert the refusal path, which is the one case that WANTS to exceed the cap.
apps/commerce goes from 5 local failures to 3. The remaining three are
TestPeerLedger_*, which need a RAM-backed scratch (`/dev/shm` or
HANZO_SQLITE_RAMFS_DIR) for the pure-Go SQLCipher codec; macOS provides neither,
so they refuse to run rather than decrypt to persistent storage. Environmental,
not a defect, and not addressed here.
Verified both ways: reverting shortRuntimeDir fails exactly those two tests
again.
commerce holds ONE card money move and this binary opens two addresses onto it.
POST /v1/billing/topup/token was screened; POST /v1/payments ran the same
billing.TakePayment — the same bounds, the same idempotency guard, the same
mintauth-authorized deposit — with nothing in front of it. So the risk gate was
not a bound on the mint, it was a bound on one entrance, and the other entrance
is the one published as an MCP tool for an agent to call.
Both doors now hold the SAME screen, resolved once at the composition root and
handed to each registration. The gate becomes a zip.Middleware because a typed op
has no handler chain to sit in — zip composes middleware around it at
registration — so one gate reaches both forms; screenChain hands the raw route
its chain as `next`. The decision, the payer rule and the settlement key are the
landed ones, not copies.
Because both doors resolve the payer through payerOrg + principal.Subject and key
their settlement on the gateway's own payment id, a burst split across the two
accrues on ONE subject and one payment reached through both converges to one
observation. settlementOf learns the typed door's spelling of the ledger receipt
(`id`, where the browser answers `transactionId`) so a settlement whose processor
states no reference is still keyable there, and the settlement check reads the
whole 2xx band because the typed op declares 201.
The read op is not screened: it mints nothing, and a scorer that is present and
mute must not be able to withhold a customer's own receipt.
Shadow is unchanged. No organisation is armed and defaultRegime is untouched;
widening the gate widens the record, not the enforcement.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestRiskGate_ANotDeployedScorerDoesNotCloseTheCreditDoor and
TestScoreOverPlane_AnUndeployedPeerIsAbsentRatherThanAnOutage failed on every
Mac and passed in CI.
A unix socket address is capped near a hundred bytes — 104 on darwin — and
t.TempDir embeds the calling test's own name. The second of those names is 61
characters, so the sockaddr went over the cap and every dial answered
`connect: invalid argument`.
That is the part worth writing down: the failure looked exactly like the thing
under test. Both tests assert that an UNDEPLOYED peer is absent rather than an
outage, and an over-long address produces a dial error indistinguishable from a
peer that is not listening. A test whose environment failure mimics its own
subject cannot be read at a glance, and this one was read as a real regression.
shortRuntimeDir is the same fix, for the same reason, as servePeerLedger in
ledger_peer_test.go — an anonymous short-named dir keeps the address inside the
cap on every platform. Named and commented so the next test that needs a working
socket uses it instead of rediscovering the cap.
unusableRuntimeDir is untouched: it builds a deliberately over-long address to
assert the refusal path, which is the one case that WANTS to exceed the cap.
apps/commerce goes from 5 local failures to 3. The remaining three are
TestPeerLedger_*, which need a RAM-backed scratch (`/dev/shm` or
HANZO_SQLITE_RAMFS_DIR) for the pure-Go SQLCipher codec; macOS provides neither,
so they refuse to run rather than decrypt to persistent storage. Environmental,
not a defect, and not addressed here.
Verified both ways: reverting shortRuntimeDir fails exactly those two tests
again.
The bundle carried @hanzogui/admin twice, so AuthGate's one-code-one-redemption
guard existed twice and the OIDC code was redeemed once per copy — the second
failure destroying the session the first established. Deduped in the SPA's vite
config; TestTheAuthModuleIsBundledOnce asserts it here, against the bytes that
ship, because the defect is invisible in the source and lives only in how the
bundler resolved the graph.
Cold, against a strict single-use stub: /meet/login, one authorize, exactly one
authorization_code exchange, token persisted, lands in the lobby, reload adds
none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The org column on a risk_decided row is the bare slug, and an org name
is unique within an issuer rather than across issuers.
One warehouse serves every brand's deployments — there is no brand in
the datastore address, the database name, or anything else that reaches
it; two brands are two deployments of one artifact differing in a flag.
So two brands' identically named organisations are one value in that
column, in this table and in every other table on that plane.
The column stays what the plane's column is: filing risk's rows under a
qualified key would put them in a partition none of the organisation's
own lenses read, which is the whole point of stating decisions there.
What this app can do is refuse to add to the collision. The subject
column already cannot collide, because the digest covers the qualified
tenant, and the brand now travels as a server-minted attribute — so a
row states which issuer's organisation it describes and a read can bind
it. Qualifying rather than assuming costs one bounded attribute and is
correct whether or not a second brand ever shares the warehouse.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
severest and fuse both document that an action outside cloud's action
vocabulary ranks below allow and can never become the answer, and both
relied on the same predicate to hold it.
Asked as "not empty and not allow", that predicate answered YES for such
a string. A determination carrying one was therefore treated as a
finding, could lead the composition — it is only ever ranked against
other findings — and would then stand, contradicting the two comments
that promise it cannot.
It is the severity that answers now, because the vocabulary is a ranking
and the question is about rank: has this rule reached something stricter
than proceeding. Nothing in this package mints an unrecognised action
today; the guarantee is worth having from the predicate rather than from
the fact that nobody has broken it yet.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
"A score records nothing" is the invariant that keeps this plane from
measuring itself, and it held for exactly one hop.
Every decision is also stated onto the shared event plane — one
risk_decided row per decide, under product 'risk', with the decided
subject as its distinct id. The person rollup folded every signal='act'
fact with no product filter at all, so those rows came back as ACTIVITY:
a subject screened often enough became unusual for having been screened,
and the model learned from its own output one hop later.
The rollup that folds a plane now names the product it will not fold,
spelled from the same constant the emitter files rows under so the two
cannot drift to different words for one product. Only the person rollup
needs it — the session rollup requires a session id this app never
states, and the fault rollup reads a table it never writes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The rings' own claim is that they only move forward, and the guard that
made it true was measured against the wrong window.
velocity folds an event older than a window's span to that window's
leading edge — the right call for a compliance aggregate that must not
drop a record, and a detector evasion in reverse for a live rule, where
it means the event is counted as having just happened. placeable is that
guard, and it refused an event the WIDEST span the rings keep could not
hold. Thirty days is the one span nothing reaches: an event ninety
minutes old cleared it comfortably, and was then folded to now in the
one-hour ring — which is the ring the pace count and the pace accrual
are both read from. The burst window could be filled with last month's
history.
Placement is now measured against the narrowest window the rings keep,
which is the window the rules read. It is taken from the windows the
store is actually built with rather than written down as an hour, so a
deployment that changed them moves this with them instead of leaving a
rule reading a window nobody keeps, and a test holds the published value
against a real ring set's own answer.
What it costs is stated rather than hidden. velocity writes all four
windows in one call, so an event more than an hour behind the edge no
longer enters the wider rings either and the model loses a late arrival.
It still learns from the event itself. A wider aggregate missing one
late event is a slightly stale baseline; the narrowest one gaining it is
a rule freezing a payment for something that happened last month.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The reading a pace bound acts on was a statement about whenever the
rings were last taught something, not about now.
A velocity window sums the buckets in [lead-N, lead], and `lead` is the
newest event the rings received. So a subject that did sixty funded
things inside one hour last Tuesday read sixty events in "the last hour"
for as long as the residency lived. Nothing cleared it: the op that
reads the window records nothing, so the anchor never advanced and the
window never slid off. One busy hour, and every later payment by that
organisation was frozen by a burst that had finished a week earlier.
The reading is now believed only while the aggregates have been taught
something inside the span they claim to describe. Past that the two
aggregate halves fall silent for that axis and the event's own stated
facts are the whole rule — the same reading a subject with no history
gets, and the honest one, since a window with nothing in it is what
"has done nothing lately" means.
The edge is the tenant's and not the key's, and that is said out loud
rather than hidden: the engine's observation carries no per-key leading
edge, so the finest recency available here is "has this organisation
been taught anything inside the span". It clears the case that matters —
a self-serve organisation is its own payer, so its edge is its payer's
edge — and leaves one open, where a busy organisation's traffic keeps
the edge fresh for a quiet subject.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The credit door's pace and fan-out rules were structurally dead — not
mis-tuned, but unable to reach a correct answer for any input.
The door stated an amount, a currency, a country and an address, and no
aggregation axis at all. onFan reads a device and a counterparty, so it
could not fire for any top-up ever sent; onPace read one axis where it
reads three. A rule half nothing states an axis for is indistinguishable
from a rule half that found nothing, and the second reads as clean.
And nothing taught the model anything. A decide records nothing by
design, the published learn door is an organisation calling itself, and
at a self-serve credit door the organisation IS the payer — so the two
halves read an empty history for exactly the subject they exist for.
Five payments of eleven thousand dollars looked like five first
payments, every one under every stated bound by construction.
So a settlement now teaches, over a plane op of its own:
The server states it. The value is what settled, the moment is a
server clock, the subject is the one the credit landed on, and nothing
is stated unless the handler answered that money moved. A request that
fails, is declined or is refused upstream teaches nothing.
It is keyed on the settlement. Settlement is at-least-once, so the key
is the processor's own reference for the charge — the one identifier
the synchronous door and a replayed webhook share — falling back to
the ledger receipt where the processor states none. Velocity that
double-counted a retry would freeze a customer for paying once.
It cannot be pre-empted. Dedupe means the first writer of an id wins,
so the settlement id lands in a namespace the public learn door now
refuses outright. The same guard covers folded buckets, which could
already be claimed.
It cannot fail the payment. The money has moved by then, so the call
is detached, bounded, dropped at a ceiling and panic-guarded.
The door also states the address our own edge resolved as its
counterparty, which is what makes the fan-out reachable there. It states
no device, and that is declared at boot rather than left to read as a
rule that found nothing: no fingerprint reaches this binary from a
top-up, and a user-agent stated as one would summon a person for every
customer. Nothing is invented.
And the payer is now a subject kind of its own. An account's learned
history is its metered inference spend, so judging a top-up as an
account scored a payment against a distribution of money spent OUT —
and the windowed value bounds, which are a payments appetite, accrued on
that same key. A customer with a large inference bill was examined for
it, and no restatement of the number could fix it: one appetite over two
populations is two appetites.
Finally, a negative value is refused at the one door every bound is
applied at. The aggregates accrue a sum and the bounds are read off it,
so a negative amount does not describe a small event — it subtracts from
the finding, and one taught event cancels an hour of real payments. A
recorded row the door would now refuse is skipped and said rather than
abandoning the whole replay, which would blind a tenant for as long as
the row is retained.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The org column on a risk_decided row is the bare slug, and an org name
is unique within an issuer rather than across issuers.
One warehouse serves every brand's deployments — there is no brand in
the datastore address, the database name, or anything else that reaches
it; two brands are two deployments of one artifact differing in a flag.
So two brands' identically named organisations are one value in that
column, in this table and in every other table on that plane.
The column stays what the plane's column is: filing risk's rows under a
qualified key would put them in a partition none of the organisation's
own lenses read, which is the whole point of stating decisions there.
What this app can do is refuse to add to the collision. The subject
column already cannot collide, because the digest covers the qualified
tenant, and the brand now travels as a server-minted attribute — so a
row states which issuer's organisation it describes and a read can bind
it. Qualifying rather than assuming costs one bounded attribute and is
correct whether or not a second brand ever shares the warehouse.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
severest and fuse both document that an action outside cloud's action
vocabulary ranks below allow and can never become the answer, and both
relied on the same predicate to hold it.
Asked as "not empty and not allow", that predicate answered YES for such
a string. A determination carrying one was therefore treated as a
finding, could lead the composition — it is only ever ranked against
other findings — and would then stand, contradicting the two comments
that promise it cannot.
It is the severity that answers now, because the vocabulary is a ranking
and the question is about rank: has this rule reached something stricter
than proceeding. Nothing in this package mints an unrecognised action
today; the guarantee is worth having from the predicate rather than from
the fact that nobody has broken it yet.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
"A score records nothing" is the invariant that keeps this plane from
measuring itself, and it held for exactly one hop.
Every decision is also stated onto the shared event plane — one
risk_decided row per decide, under product 'risk', with the decided
subject as its distinct id. The person rollup folded every signal='act'
fact with no product filter at all, so those rows came back as ACTIVITY:
a subject screened often enough became unusual for having been screened,
and the model learned from its own output one hop later.
The rollup that folds a plane now names the product it will not fold,
spelled from the same constant the emitter files rows under so the two
cannot drift to different words for one product. Only the person rollup
needs it — the session rollup requires a session id this app never
states, and the fault rollup reads a table it never writes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The rings' own claim is that they only move forward, and the guard that
made it true was measured against the wrong window.
velocity folds an event older than a window's span to that window's
leading edge — the right call for a compliance aggregate that must not
drop a record, and a detector evasion in reverse for a live rule, where
it means the event is counted as having just happened. placeable is that
guard, and it refused an event the WIDEST span the rings keep could not
hold. Thirty days is the one span nothing reaches: an event ninety
minutes old cleared it comfortably, and was then folded to now in the
one-hour ring — which is the ring the pace count and the pace accrual
are both read from. The burst window could be filled with last month's
history.
Placement is now measured against the narrowest window the rings keep,
which is the window the rules read. It is taken from the windows the
store is actually built with rather than written down as an hour, so a
deployment that changed them moves this with them instead of leaving a
rule reading a window nobody keeps, and a test holds the published value
against a real ring set's own answer.
What it costs is stated rather than hidden. velocity writes all four
windows in one call, so an event more than an hour behind the edge no
longer enters the wider rings either and the model loses a late arrival.
It still learns from the event itself. A wider aggregate missing one
late event is a slightly stale baseline; the narrowest one gaining it is
a rule freezing a payment for something that happened last month.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The reading a pace bound acts on was a statement about whenever the
rings were last taught something, not about now.
A velocity window sums the buckets in [lead-N, lead], and `lead` is the
newest event the rings received. So a subject that did sixty funded
things inside one hour last Tuesday read sixty events in "the last hour"
for as long as the residency lived. Nothing cleared it: the op that
reads the window records nothing, so the anchor never advanced and the
window never slid off. One busy hour, and every later payment by that
organisation was frozen by a burst that had finished a week earlier.
The reading is now believed only while the aggregates have been taught
something inside the span they claim to describe. Past that the two
aggregate halves fall silent for that axis and the event's own stated
facts are the whole rule — the same reading a subject with no history
gets, and the honest one, since a window with nothing in it is what
"has done nothing lately" means.
The edge is the tenant's and not the key's, and that is said out loud
rather than hidden: the engine's observation carries no per-key leading
edge, so the finest recency available here is "has this organisation
been taught anything inside the span". It clears the case that matters —
a self-serve organisation is its own payer, so its edge is its payer's
edge — and leaves one open, where a busy organisation's traffic keeps
the edge fresh for a quiet subject.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The credit door's pace and fan-out rules were structurally dead — not
mis-tuned, but unable to reach a correct answer for any input.
The door stated an amount, a currency, a country and an address, and no
aggregation axis at all. onFan reads a device and a counterparty, so it
could not fire for any top-up ever sent; onPace read one axis where it
reads three. A rule half nothing states an axis for is indistinguishable
from a rule half that found nothing, and the second reads as clean.
And nothing taught the model anything. A decide records nothing by
design, the published learn door is an organisation calling itself, and
at a self-serve credit door the organisation IS the payer — so the two
halves read an empty history for exactly the subject they exist for.
Five payments of eleven thousand dollars looked like five first
payments, every one under every stated bound by construction.
So a settlement now teaches, over a plane op of its own:
The server states it. The value is what settled, the moment is a
server clock, the subject is the one the credit landed on, and nothing
is stated unless the handler answered that money moved. A request that
fails, is declined or is refused upstream teaches nothing.
It is keyed on the settlement. Settlement is at-least-once, so the key
is the processor's own reference for the charge — the one identifier
the synchronous door and a replayed webhook share — falling back to
the ledger receipt where the processor states none. Velocity that
double-counted a retry would freeze a customer for paying once.
It cannot be pre-empted. Dedupe means the first writer of an id wins,
so the settlement id lands in a namespace the public learn door now
refuses outright. The same guard covers folded buckets, which could
already be claimed.
It cannot fail the payment. The money has moved by then, so the call
is detached, bounded, dropped at a ceiling and panic-guarded.
The door also states the address our own edge resolved as its
counterparty, which is what makes the fan-out reachable there. It states
no device, and that is declared at boot rather than left to read as a
rule that found nothing: no fingerprint reaches this binary from a
top-up, and a user-agent stated as one would summon a person for every
customer. Nothing is invented.
And the payer is now a subject kind of its own. An account's learned
history is its metered inference spend, so judging a top-up as an
account scored a payment against a distribution of money spent OUT —
and the windowed value bounds, which are a payments appetite, accrued on
that same key. A customer with a large inference bill was examined for
it, and no restatement of the number could fix it: one appetite over two
populations is two appetites.
Finally, a negative value is refused at the one door every bound is
applied at. The aggregates accrue a sum and the bounds are read off it,
so a negative amount does not describe a small event — it subtracts from
the finding, and one taught event cancels an hour of real payments. A
recorded row the door would now refuse is skipped and said rather than
abandoning the whole replay, which would blind a tenant for as long as
the row is retained.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Re-syncs after the shared CallbackHandler's single-use guard learned to hold
across two instances of its own module.
The tracker did not have the defect. Measured against a stub that enforces
single use — proven by exchanging one code twice, 200 then 400 — a cold run
redeems EXACTLY ONCE, keeps the token, and lands in the board authenticated,
with the stub's own ledger agreeing: 1 exchanged, 0 rejected. A reload spends
nothing further.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Measured cold against the real binary and a STRICT IAM stub that 404s any
doubled or missing prefix: a zero-cookie visit to /meet/ is redirected by
AuthGate's render guard to /meet/login rather than mounting the lobby, and the
PKCE round trip resolves to exactly /v1/iam/oauth/authorize — the prefix once,
never twice.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The weave refuses one schema name with two shapes, and regenerating the
subsets for the converted surfaces surfaced exactly that: billing's series
bucket collided with admin's usagePoint; affiliates' directory tally, share
link row, payout family, sweep report and analytics summary collided with
referrals, link, authors, treasury and platform; and link's typed inputs
collided with agents, usage and visor. Each colliding type takes a name the
fleet does not already speak — sample, totals, tally, codeView, remittance,
settlement, accruals, enrollReq, readingReq, readingView, ingestReq,
ingestResp, boardResp — and no byte moves: a Go type's name never appears
in its JSON.
The subsets regenerate from the live routers (the commerce dedupe, the
declared Cache-Control response headers, notify's raw template object, the
renamed schemas) and openapi.yaml is rewoven from them — the document the
SDK repos pull.
The native availability road existed one step short of proof: the reporter's
gauge was read back through Gather, a test's convenience, not what the
framework ships. The exporter serializes the registry's text encoding, so the
evidence that a probe cycle lands the series where the exporter will find it
has to read that encoding — a real prober Start, both verdicts, and
hanzo_service_up present in EncodeText output under the same service label the
meter pipeline publishes. The meter path stays: two producers, one truth,
until every rule row is re-pointed with proof.
The four raw registrations that remain now state their reason where they are
made: the alert replay serves text/plain for an operator's tail, the receiver
speaks Alertmanager's webhook protocol and must accept a body that will not
parse, the sessions route relays the runtime's envelope byte-for-byte, and
the sentry wildcard carries Sentry's DSN-keyed frames.
/healthz and /readyz were the last plain-JSON raw routes across cmd/cloud,
apps/git and apps/integrations. They now register as typed ops, so the answer
the kubelet GETs is the same answer a native ZAP typed Call reads, and both
probes carry their shape in the registry every projection — OpenAPI, MCP, the
CLI, the SDKs — is built from. The bytes did not move: probeOut declares
absent ahead of status because the raw handlers marshalled a map and
encoding/json writes map keys sorted, and TestProbeAnswersAreByteStable pins
every body and status the old handlers answered, including the draining and
unfit 503s.
Every other raw registration in the three packages was audited and stays raw
for a reason now stated at its registration: the Slack, GitHub, Discord,
Teams and Telegram doors speak their platform's own webhook protocol over the
raw request; the link legs and the generic OAuth callback drive a browser
with a 302 or an HTML page; smart-HTTP speaks git's own pack protocol; the
git UI serves server-rendered HTML; and the five ZAP procedures answer the
bridge's envelope. git.go's webhook comment also catches up with webhook.go:
that door is a tombstone answering 410, not an HMAC verifier, and the comment
no longer describes a verification the handler stopped doing.
Every convertible raw route in notify, link, meet and analytics is now a typed
op — the one registration REST, OpenAPI, the MCP tool list, the CLI and the
by-name call plane all project from. An untyped route is invisible to four of
the five, which is why notify's send routes could never be reached as a typed
zip.Call by IAM's OTP sender.
notify: the three send routes type without moving a byte. The two-shape refusal
(one recipient answers the bare {message_id,status} outcome, several answer the
{items:[...]} envelope) is carried by the Out's own MarshalJSON, so every JSON
projection writes the exact bytes the raw handler always wrote while the call
plane reads one declared shape. The sync selector becomes a declared In field,
which is what makes it expressible in a by-name call's arguments at all. The
template values ride a raw JSON object rather than a map, because the call
plane computes an input's whole layout before reading any payload and refuses
a map field outright — a map anywhere on the In failed every by-name call,
including the OTP send that carries no template values at all. A socket test
drives the real call plane with template_vars populated, so the projection this
conversion exists for is proven, not presumed.
link: all eleven routes convert; the openapi.Describe ledger dies and its prose
moves onto the ops, where zipdoc lifts it. The (org, subject) gate stays the
one caller() both planes key.
meet and analytics: the two health probes' refusal — 200 and 503 carrying the
SAME body — went stale when zip learned to declare a non-2xx with a typed body.
Both now declare WithStatus(200, 503) and the report's own StatusCode picks,
so the degraded answer keeps its body and the refusal ledgers shrink.
The survivors each carry their reason in place: visor's polymorphic launches,
verbatim catalog passthroughs and the streaming bot verb; books' raw-bytes
uploads and the two 501 link-flow stubs; base's and tasks' verbatim relays of
another program's bytes; meet's text/plain token; analytics' ingest doors, tag
asset and Sentry relay; projects' archive deploys.
Two link tests that mount typed agents onto a bare app now install the
composer's Bridge — they were failing 403 at base for want of it.
The affiliate program's whole surface (17 routes) and billing's finance
projection plus the per-account breakdown (7 routes) become typed ops, so
the document, the MCP tool, the CLI command, the SDK method and the ZAP
call all project from each single registration. The contract does not
move: same paths, same field names, same statuses — apply and attribute
state their 200-vs-201 on the answer — bare arrays stay bare, the admin
envelope stays {status,msg,data}, and every write that refused a bodyless
request with c.Bind's 400 still refuses one, because zip's tolerant
decode would otherwise turn that refusal into a write of zero values
(requireBody replays it).
Refusal ORDER is part of the contract too, and zip decodes a body before
a handler runs, so the admin family rides a gated group: a non-admin
sending a body that will not parse is answered the 403 first, exactly as
the raw handlers ordered their refusals, and approve keeps its OPTIONAL
body through a tolerant decode rather than growing a 400 it never
answered. Body-only fields opt out of URL binding (url:"-"), so a money
parameter can never ride a query string into an access log and the query
can never override the body.
Per-tenant money answers declare Cache-Control: no-store on the contract
itself — zip.WithResponseHeader plus each Out's ResponseHeaders — so the
directive reaches the document, the SDKs and the tool schema instead of
riding a context slot no projection can see.
Four /v1/billing routes stay raw because they serve bytes: each serves
commerce's own body and status — a 402 decline keeps its reason —
enriched or envelope-built on the co-resident leg, a shape they
deliberately do not own. Each names that reason at its registration;
their prose stays on openapi.Describe, while the converted ops carry
theirs as doc comments zipdoc lifts. The saved-card trio is commerce's
co-resident surface now, so the package holds eleven routes: seven
typed, four raw.
The dual-shape reads keep BOTH shapes exactly through optional fields: an
enrolled zero — a rate of 0, no commission yet — still renders, and a
not-enrolled caller never grows a field. typed_compat tests pin those key
sets, the bodyless refusals, the bare-array kinds and the cache
discipline; typed_request_gate_test pins the two new request resolvers
(billing's payer/caller, affiliates' sudo/actor/requireBody).
A unix socket address is capped near a hundred bytes, and t.TempDir embeds
the test's own long name — on darwin the bind failed on a discarded goroutine
error and every dial refused, so the suite reported a phantom (socket never
began listening) instead of the truth. A short anonymous dir keeps the
address inside the cap on every platform; on a Mac without a mounted tmpfs
the tests now fail for the honest reason instead — the pure-Go SQLCipher
codec refuses to decrypt to persistent storage, which is the fail-closed
property it exists for.
The file also takes the name of what it proves — the peer path — because its
old name reached for a word this repo does not use for transport.
The probe was a closure marshalling a map — a route and nothing else, in none
of the five projections. It is now a typed op whose answer is byte-identical
to the map it replaces (the struct's field order mirrors the map's sorted
keys), pinned by test, and whose prose rides the handler's doc comment the
way every typed op's does — so its openapi.Describe entry is gone.
Every other raw registration now states its survival in one sentence. The
webhook intake speaks the provider's protocol (HMAC over the raw payload),
the invoice PDF serves bytes, the fail-closed wildcard must shadow every
method under every prefix — those are sanctioned reasons. The rest are not
raw by sanction at all: each is a JSON-shaped module route that is not typed
YET, because the typing is module work. The module-handler note above Mount
says so once, and every citing registration — the store/catalog/plan bundle
mounts included — is a conversion hanzoai/commerce still owes through the
payments.go exported-core pattern, never a route that cannot be typed.
The eight operations that said nothing about themselves — wire, the crypto
rail, and the saved-card family — now state their gate, their tenant scope
and what they fail closed on, which is what let the commerce subset
regenerate at all. That regeneration also repaired two stale artifacts the
committed subset carried: GET /v1/billing/methods claimed its portal
sibling's operationId (the duplicate the weave gate died on), and the two
saved-card POSTs were absent outright.
The fleet golden and the floor now agree with the module: the deposit proxy
and the webhook relay were removed upstream (deposits are commerce's own
rails; the real receiver is /v1/billing/webhooks/:provider), so the golden
drops those four operations, the commerce floor is hand-lowered to four, and
the dead prose goes with them.
Re-syncs after the shared seam learned to remember a refusal that arrives before
anyone is listening for it.
The tracker did not have that bug — verified cold, on a zero-cookie zero-storage
browser: it made NO /v1/tracker request at all before bouncing to sign-in,
because AuthGate returns null and navigates when there is no token, so its
children never mount to fetch. The fix is to the seam rather than to this app,
and it matters here anyway: nothing but the gate's render guard was stopping it.
The README's sync command now excludes .sync-stamp. The source dist has no copy,
so a plain --delete removed it, and no test can catch that: what embed_test.go
asserts is exactly that the stamp is absent from the binary. It went missing
twice — the warning belongs in the command, not in the file the command deletes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The bundle carrying AuthGate + the /meet/callback route. meet.hanzo.ai is its
own host and cloud's session cookie is host-only, so a first-time visitor had no
way to obtain a credential; the client now authenticates its own visitors
through the shared door rather than a login of its own.
The IAM base is ${origin}/v1/iam under the workspace's @hanzo/iam 0.9.4, pinned
by a test on the COMPOSED authorize URL because the wrong polarity does not
404 — cloud answers it 200 text/html from the SPA catch-all and sign-in stalls
with nothing to see.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The sync command in the README deletes it — the source dist has no copy — and
embed_test.go cannot notice, because what that test asserts is precisely that
the stamp is NOT in the binary. So the file now says so at the top of itself,
where whoever runs the sync will read it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Re-syncs the SPA after the shared gate learned to recover from an expired
session, to refuse an off-origin return-to, and to compose the IAM URL the
installed SDK actually wants. Verified against this binary with a STRICT IAM
stub that 404s any path but the five real endpoints — so a doubled or missing
/v1/iam prefix cannot pass unnoticed the way it would in production, where an
unknown path answers 200 text/html from the SPA catch-all.
The bundle also stops fetching webfonts from a third party: a signed-in admin
surface should not announce each of its visitors to Google, and a strict CSP
would block the request anyway. It wears the forge's own stack instead, which is
what the views were copied from.
And the schedule-horizon test now asserts ABSOLUTE instants (the years 2300 and
9999) rather than maxScheduleAt + 1. A case written in terms of the constant
moves with it, so loosening the horizon — the one regression that test exists to
catch — would have kept it green. The constant itself is pinned to
2200-01-01T00:00:00Z separately, so changing it is a deliberate edit rather than
a silent widening.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip projects every TYPED op — zip.Get[In, Out] and friends — into REST, OpenAPI,
MCP tools, the CLI and eight generated SDKs from ONE registry entry. A raw
app.Get(path, func(*zip.Ctx) error) serves the same bytes and reaches none of
them: it publishes an operationId and nothing else, so the SDK method cannot
explain itself and the CLI command has no help text. Every route on /v1/platform
was raw. All 32 are now ops, and NONE stays raw — not one of them streams,
serves bytes, redirects a browser, or speaks a third party's protocol, so not one
of them had a reason to.
The wire is unchanged, and that is the whole claim: same paths, same JSON field
names, same status codes, same refusal sentences. converted_test.go is the proof
rather than the assertion — it walks every converted route asserting status and
the exact set of JSON keys a client's decoder binds, and the SAME FILE passes
against the pre-conversion tree. The expectations were read off the old
c.JSON(status, …) calls, so it is a regression test and not a mirror.
WHAT THE CONTRACT GAINED. The subset went from 9 published schemas to 43: every
route's real input and output is now in the document instead of an unnamed body
a generator had to guess at. Three statuses the routes have always sent became
sayable — 201 on create, 202 on deploy/preview/promote/rollback/run/runner, 204
on the two deletes — where zip previously had no vocabulary for them and the
document said 200 about a route that has never sent one. Two routes answer with
their own status because the ANSWER states it: /v1/platform/health is 200 or 503
carrying the real reason as a typed body, and add-domain is 201 for a claim it
created or 200 for an idempotent re-add. Both declare their set, so a generated
client expects what the service sends.
THE PROSE MOVED, IT DID NOT CHANGE. ~340 lines of openapi.Register/Describe —
a SECOND source that had to be edited in lockstep with the router and, being
second, could not be — are gone. The same sentences are now doc comments on the
handlers they describe, lifted by zipdoc into zipdoc_gen.go, which is the only
road prose travels to the document, the tool list, the SDKs and the CLI.
FOUR NAMES YIELDED, because the fleet's schema namespace is FLAT and
single-valued: one name, one shape, wherever two apps meet, or every generated
SDK binds whichever it read last. buildView and pipelineView are published by
apps/agents and apps/world, healthView by apps/compliance, and Drift by
apps/plugins with an entirely different shape. Platform's were not published, so
platform's yield — the rule apps/templates followed when its Template became a
StarterKit — each to a word its own doc comment already used: the console
projections are rows on a board, and Drift is the verdict it says it is. The
JSON field is still `drift`; only the Go type is renamed. fqdn.Record collides
with apps/projects and was left alone: it is the same type, so the shapes are
identical and the weave accepts it.
THE GATE MOVED, IT DID NOT CHANGE. cloud.Guard wraps a zip.Handler, and a typed
op has none — it receives a context and its decoded In. So the fleet board's
cloud.Admin/cloud.Super gate is now the first line inside each op (board.admit),
applying the same Scope.Admits over the same AuthorityOf. gate.go gains ONE
exported method for it: Scope.Refusal, the 403 Guard itself now returns, so the
two forms of the one gate cannot answer differently and the refusal sentence
stays derived from the scope rather than written at a call site. It is a METHOD
and not a function taking a Scope because Refuse already names the 402 every
spend gate renders — two refusals that mean different things do not share a name.
An In field is never a tenant key: the org comes from the validated principal, so
cross-tenant identifiers remain structurally not inputs. The three cloud.Request
call sites are declared in allowedRequestUses with their reason — this plane
SPENDS the caller's identity (the run fee, the audit actor) rather than only
reading it, and /v1/runner authorizes on a shared credential no org can express.
No middleware is installed here. cloud.Bridge belongs to whoever composes the
app, and the tests compose their own.
Tests: the full apps/platform suite is green on Linux, where the SQLCipher codec
finds the RAM-backed scratch it fails closed without; the same suite cannot run
on darwin without a tmpfs, before this change or after.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The bundle behind RED F2 (a different room is a different component) and B2 (the
session read goes through @hanzogui/admin's useFetch, so meet inherits the shared
401 door instead of growing a login), plus the disconnect rule that keeps the URL
and the screen naming the same room.
Verified against this binary and a real LiveKit: from a LIVE standup, navigating
to retro lands on retro's join screen and mints nothing; both grants name the room
their URL names.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Re-syncs the SPA now that it authenticates its own visitors through AuthGate
(@hanzogui/admin + @hanzo/iam PKCE) rather than relying on a cookie some other
host set — which cloud never sends here, session cookies being host-only.
The bundle carries no credential and no tenancy of its own: it obtains a token
from IAM on this origin, and cloud mints the validated org from that token's
SIGNED membership claim. Verified against this binary: a fresh browser with no
cookie and no storage reaches the board through the OIDC redirect, and a raw
fetch without the bearer is still 403.
Registers one redirect URI: https://tracker.hanzo.ai/tracker/callback — under
the SPA's own base, which is the prefix the manifest routes to this app.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
RED F1. The machine-credential exclusion (p.Subject != "") in admits and spaces
was correct and UNTESTED, and mutation proved it: deleting it changed nothing the
suite could see. Every IAM-lane test stopped at an unanswerable ask — there is no
team peer in a unit test — so the machine got as far as the ask and was refused by
the missing peer rather than by the rule. A decision that only exists past a
boundary needs the boundary to be crossable in a test, or it is not tested.
roster is that boundary, named: the process that OWNS the membership rows, with
peer{} the real one (apps/team over the internal plane) and state.rows() defaulting
to it. Nil-safe by construction rather than by discipline — every load() failure
path returns a state carrying only a reason, and a lobby read on one of those must
answer like production instead of panicking. The org still rides the context; it is
not a parameter, so a caller cannot ask about another tenant by naming one.
With an authority that ANSWERS, the rules past it are reachable and now pinned:
a human is seated and a machine is refused WITHOUT THE AUTHORITY BEING CONSULTED
(refused, and refused for the right reason — only the second survives a mutation);
the authority's role decides, and offer and admit agree across every role on the
IAM lane too; no row is a refusal; an unreachable authority is a refusal on both
doors. Three mutants, three kills: neuter p.Subject!="" -> both machine tests fail;
drop the role filter -> the offer/grant test fails; drop the empty-workspace guard
-> the widening test fails.
It also corrected the suite's own story. With an authority that can answer, a room
name with no separator is visibly ASKED about (strings.Cut returns the whole string)
rather than refused up front — only a LEADING underscore is refused by meet itself.
Both are fail-closed, but the old comment described the wrong one, and a test that
flatters the program is not a test.
RED F4. plane.Spaces claimed "Account is empty exactly when there are no
workspaces". The implementation resolves the account BEFORE walking the rows, so it
never satisfied the biconditional. The one-way invariant is the one that matters
and the one now documented and tested: a non-empty Items implies a non-empty
Account, so every workspace offered has an identity to seat under. A doc that
overstates an invariant is worse than none — it is the one a caller writes an `if`
against.
RED F5. Said plainly why session answers outside ready(): same reason the bundle
is served outside it. A bad key file drops teamSecret with everything else, so such
a deploy answers a lobby read on the IAM lane (which never needed teamSecret) while
every mint is 503. Honest rather than contradictory — the workspaces someone
belongs to do not stop being true because this binary cannot sign.
RED F3. spa/ had three dependents and no tests of its own; each app proved the
policy against whichever bundle happened to be committed, and none could build the
tree that breaks it. The trees here are synthetic and hostile: an unsynced bundle
(503, naming the app), a chunk that is not there, seven traversal shapes that must
end inside the bundle or nowhere. Two behaviours are recorded rather than changed —
a missing chunk falls back to the shell (which is why the shell is no-cache), and
http.FileServer canonicalises /index.html to ./ with a 301.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Red's review of the native tracker. Five findings, all in the seams around the
code rather than in it.
ROUTED (blocker). /tracker was in no manifest row, so in the fleet — one process
per app behind a prefix router — the board's own host reached whoever owns "/"
and the visitor got the console's HTML shell with a 200. The prefix is now
claimed beside the API it reads.
Nothing could have caught that: manifest/router_test.go asks the real router
where each app's PUBLISHED paths land, and an embedded SPA is an untyped route
that appears in no openapi.json. So internal/manifesttest asks a different
question of a different source — it mounts the app and reads its LIVE router
declaration, which zip builds complete by construction, untyped routes included.
Every app can adopt it in five lines; tracker does. It fails without the manifest
fix and passes with it.
GATED (must-fix). Tracker writes ran with no anti-CSRF gate while the fleet
reflects *.hanzo.ai with credentials — a wildcard covering hosts that serve
arbitrary user content — so a page there could read and write another org's
boards with the visitor's own ambient session. The estate's ONE gate
(apps/account, minted at GET /v1/csrf) now runs on the group, discriminating by
METHOD: a gate written six times is a gate the seventh write forgets. Reads pass
untouched and a Bearer/gateway caller is unaffected, being un-CSRF-able.
BOUNDED (must-fix). checkSchedule accepted any non-negative int64, so
dueAt=2^63-1 was a legal write — and the timeline sizes its grid from the data,
so that one row made every member of the org who opened the view render an
unbounded number of ticks. A stored value that breaks the reader for everyone who
looks at it is the write's fault, so the horizon is enforced at the write. The
client clamps too, in the SPA; neither is load-bearing alone.
And three things the static handler gave away or got wrong: /tracker/assets/
returned a directory listing (fs.Stat succeeds on a directory and http.FileServer
lists it), /tracker/.sync-stamp published the source repo, branch and commit
(`all:dist` embeds dot-files; plain `dist` does not), and a missing hashed asset
answered the SPA shell — handing a <script> tag an HTML document, which reports
as a corrupt bundle rather than the cache miss it is.
Finally: `?scheduled=yes` bound FALSE and returned the whole board, because zip
parses a bool with ParseBool and leaves the zero value when it fails. Every other
filter on this route refuses its unknown values; so does this one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The registry the ⌘K bar reads is the registry Slack reads. `/hanzo platform
apps-get web` is now that operation, executed; anything that is not an operation
is prose and reaches the agent brain exactly as before. One branch in
slackSlashTurn, and the list it consults is zip.CommandsFromSpec over the fleet
document — the same projection GET /v1/commands serves, resolved in this process
rather than fetched, because a Slack turn should not depend on the host
answering a public GET about itself. 2,328 commands today.
TWO QUESTIONS, and a body has to answer both.
NAMING. Exact `<service> <operation>` first, whatever the method; failing that
an unambiguous prefix of a GET's name. A mutation is reachable only by its whole
name — structurally, because the fuzzy branch never looks at a non-GET, not by a
check someone can forget. An ambiguous abbreviation names nothing.
INVOKING. "audit log for last week" NAMES `audit log`, then hands it four words
it has nowhere to put; without the second question it became a CLI usage dump
instead of an answer. So zip's own runner parses it first, with an invoker that
refuses — the arity, the flag names and the required flags are checked by the ONE
parser, before any credential is touched and before anything is sent. Failing
either question is prose.
THREE THINGS THE FIRST CUT GOT WRONG, all found by review and all pinned:
THE ADDRESS. zip percent-encodes a path argument so it cannot become path
structure, and fasthttp decoded it again: `platform apps-get
../../../v1/iam/users` left as GET /v1/iam/users. One operation's ARGUMENT
addressing another operation, on the caller's own credential — which voids
every decision made above it, including "a mutation must be named". Fixed on
both sides: a positional must be one bare segment (no "/", "%", "..", control
rune, and no query or fragment rune), and zip 1.25.3 stops the transport
re-reading the address. Measured: the request-level flag ALONE does not hold,
because HostClient.doNonNilReqResp assigns it from the client's own field.
THE ORG. The call carried only Authorization, so middleware_identity fell back
to the token's HOME org — a person linked in org A's workspace ran mutations in
org B. The workspace org now rides as X-Org-Id, and membership is checked here
first through authz.Claims.EffectiveOrg (the same published predicate the front
door decides with), because the front door DISCARDS a selection outside the
signed set and continues rather than refusing. Right for a browser, wrong here.
THE REFUSAL. A spent or revoked sealed token said "try again shortly" forever —
a loop with no exit, for a credential that is never coming back. IAM refusing
the grant (4xx, or an OAuth error) is now told apart from IAM being
unreachable, and only one of them is worth waiting for; the other says the link
expired and carries the URL that fixes it.
AUTH is otherwise as designed: the refresh token slack_link sealed in KMS mints a
short-lived hanzo.id access token — the credential that whole flow exists to
establish, spent here for the first time — and the call goes through the front
door, so the command meets the same authorizer a REST client would. A rotated
refresh token is re-sealed, because IAM's are single-use and keeping the spent
one would work exactly once. Never a service identity.
Everything on this branch is EPHEMERAL, stated once at the branch: a command's
result is org data the caller asked for, and posting it to a whole channel is a
disclosure nobody chose. The prose answer keeps the in_channel delivery it had.
On the way out a payload passes audit.Redact — a chat message has left Hanzo's
custody, and some operations answer with provider config, so `api_key` would be
written into a workspace's history by someone who only asked what was configured.
The fleet's ONE credential-field denylist already knows those names. A REFUSAL
goes through it too: the status and the sentence survive, because a revoked key
and an outage need different actions, but the body they travel in is a payload
like any other. And every value is escaped for mrkdwn INCLUDING the backtick,
which would otherwise close the code block and let the rest render as markup.
The renderer has one rule: a scalar is read at a glance so it is printed, a
container is not so it is named by its size. A fact about the value's shape,
which holds for payloads nobody has seen — the alternative is a list of
interesting field names, wrong for every operation added after it was written.
Over the budget it says so and points at the console. It is NOT a second guard:
root scalars and the records in a list both print, so redaction is the only thing
standing there.
zip.Remote reads the context only before dialling and the https transport
declares no deadline, so the invoker applies the turn's deadline itself, where
the bridge pool slot is held. Without it one unanswered request would pin a slot
for the life of a socket and 32 would wedge the bridge for every org. The send is
left to finish into a buffered channel nobody reads — one goroutine and one
socket until the transport gives up, which is the honest limit of this side.
bridgeIdentity is lifted out of bridgeReply because two things now run on behalf
of a linked user and the link prompt carries a URL: a second copy of it is a
second chance to forget it is ephemeral.
The cost accepted: apps/integrations links plugin, so the binary carries the
4.5 MB of embedded subsets the host already carries. That is what reading the
whole fleet's registry without a network hop costs, paid at link time rather than
per request; the weave itself is lazy and once, and a fleet whose document will
not weave loses commands, not Slack.
Pins: the parse table (exact mutation, fuzzy GET, ambiguous, six shapes of prose
that open with a real command, help, an apostrophe, Slack's escaped entities,
quoted values, a quoted empty service — which is how the internal `/_/` plane is
addressed and the only way to type it); no prefix of a non-GET names it and every
non-GET names itself, over the fixture AND over all 2,328 real commands; ten
argument shapes that would retarget the path are refused and never leave the
process while an ordinary id goes through unchanged; the workspace org reaches
the front door; membership decides across six token shapes; a refusal is reported
in the service's own words and never as an outage; a credential is redacted at
the root and on a row; a payload cannot close the code fence; a cancelled context
returns rather than waiting on the socket; a refused grant is told from an outage
and says so; an unlinked turn produces the link prompt and writes nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`make describe` could not be run to completion and have its output committed,
which is the condition every stale artifact on main has grown behind. Not
because of the apps that refused it — those are fixed — but because the run
itself was not faithful: regenerating plugin/iam under v1.25.1 stripped the
`iam.` qualifier off all 95 of its schemas, and the bare `Role` that fell out
collided head-on with framework's. The weave refuses that, correctly, so the
one command told to repair the document produced a document that could not be
woven.
THE COMMITTED SUBSET WAS THE CORRECT SIDE, which is the opposite of the usual
direction and is the whole reason this is a dependency bump and not a
regenerate. `iam.Role` is the right name; bare `Role` was the defect.
zip qualifies a composed child's types by the child so that two children may
both honestly call a type Role, and it read that name off the ENCLOSING
definition — which for a route declared inside a group is a group, and
App.groupConfig strips a group's name on purpose, a group being a prefix and a
prefix not being an author. hanzoai/iam declares its whole surface in groups.
o11y is the control and it is what made this legible: it composes the same way,
registers on its child directly rather than through a group, and kept all 777 of
its o11y.* names through the very same regeneration that stripped iam's 95. Same
run, same binary, same toolchain, opposite answers — so the cause was in the
composition and never in the machine.
Fixed upstream in v1.25.2 (zap-proto/zip: "a group has no name, so it must not
answer for who wrote the op"), which carries the nearest NAMED ancestor down the
walk. A root's own groups are untouched, so a service that composes nothing
still names its own types exactly as it did.
Regenerated with the bump, and the delta is the proof: across all 119 subsets
the ONLY file that moves is plugin/iam, by 11 descriptions — no path, no
operationId and no schema name anywhere in the fleet. Those descriptions are
iam v1.34.5 → v1.34.20 catching up, and two of them are load-bearing: oauth/logout
now says it ACTUALLY ends the session (it spent a release answering
{"status":"ok"} without revoking anything), and authorize now documents
prompt=none|login|select_account.
woven 1700 paths / 2083 schemas / 180 tags from 119 apps — byte-identical
/v1/commands: 2328 commands over 2354 operations
go.sum also drops the entries for three superseded versions that a previous bump
left behind; `go mod tidy` is the only thing that touched it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Everything the risk plane writes down is its own. The observation record is
what a tenant TAUGHT its model; the policy record is what it STATED; both are
per-tenant files on this app's own shelf, reachable only through this app's own
ops. /risk/decide wrote neither — it is pure by contract — so a verdict existed
for the length of one plane call and then existed nowhere. Nothing joined any of
it to the traffic the organisation was already measuring, so the questions an
operator actually has could not be asked at all: did the gate start refusing the
hour the signup flow changed, is the shadow regime alerting on the campaign's
own traffic, which stage carries the refusals.
So each decision is ALSO stated on the plane the organisation already reads —
event.fact, filled by POST /v1/event, read by /v1/insights, grouped by every
product lens. One row per decide, in the same table as the pageview that led to
it, which is what makes "risk decisions beside product analytics" a join rather
than a project.
THE PLANE IS ASKED, NOT LINKED. analytics owns that table and the pod forks one
process per app, so its write core is in another pid — the shape this seam has
now met twice (cloud.SetRiskScorer, cloud.SetObsErrorIngest) and the answer is
the one both landed on: a plane op on the owning app's socket. analytics gains
event_capture, the WRITE side of the door obs_error_post already claims a slice
of, reaching the SAME write core every HTTP door reaches — one admission, one
normalizer, one storage projection. No new transport, no hop through the fleet's
front door, no second gate.
THE TENANT IS THE CALLER'S AT BOTH ENDS. The emitted row is filed for the tenant
the verdict was REACHED under, and the door mints its org from the plane
principal; the contract carries no field that could name one, so a peer can only
ever write into its own organisation's partition. An unidentified caller writes
nothing rather than defaulting to a tenant.
IT CANNOT FAIL THE DECISION. The gate that asks this scorer is the credit door,
so the emit happens after the verdict is computed, on a detached goroutine, with
its own budget, behind a ceiling that DROPS rather than queues. A queue defers
the loss instead of bounding it and makes this process's memory a function of
how fast the fleet is being screened; dropping says the true thing, which is
that a telemetry row is expendable and a decision is not. An absent peer, a
refusing one and a wedged one are the same fact here and none of them is the
verdict's.
WHAT TRAVELS IS BOUNDED, IN BOTH SENSES. The attributes are the stage, the
subject's kind, the action, the cause, the refusal, the model shape, the policy
version, the posture, and — on a scored answer only — the score and the cut it
was held against. Every value is server-minted, from a closed set the op already
refuses outside of, or narrowed before it travels: the country to alpha-2, and
the amount to which stated bound it was past rather than to itself. The subject
is a tenant-and-kind-scoped digest, so decisions about one subject still group
and the shared plane does not become a second copy of what the gate stated.
SHADOW IS EMITTED AND SAYS SO, which is the whole point of a shadow regime: what
the gate WOULD have done, on real traffic, changing nothing. No organisation is
armed and the default posture is untouched.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both gates in `make test` are red on main for reasons that predate this branch,
and both are the same failure: a derived artifact was not regenerated with the
source it derives from.
apps/projects/zipdoc_gen.go is missing the prose on POST /sites/live. The op was
added with its doc comment and Go drops comments at compile time, so the lift is
the only way that prose reaches the published document — the op ships described
by nothing.
openapi.yaml is missing the paragraph that says arming the risk model is an admin
act. It is present in plugin/risk/openapi.json, which is committed, so the golden
and the subset it is woven from disagree with each other in the tree. That
document is what the SDK repos pull.
Regenerated, not hand-edited: `go generate -run zipdoc ./apps/projects/...` and a
weave of the already-committed subsets. No source changed and no route moved.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Everything the risk plane writes down is its own. The observation record is
what a tenant TAUGHT its model; the policy record is what it STATED; both are
per-tenant files on this app's own shelf, reachable only through this app's own
ops. /risk/decide wrote neither — it is pure by contract — so a verdict existed
for the length of one plane call and then existed nowhere. Nothing joined any of
it to the traffic the organisation was already measuring, so the questions an
operator actually has could not be asked at all: did the gate start refusing the
hour the signup flow changed, is the shadow regime alerting on the campaign's
own traffic, which stage carries the refusals.
So each decision is ALSO stated on the plane the organisation already reads —
event.fact, filled by POST /v1/event, read by /v1/insights, grouped by every
product lens. One row per decide, in the same table as the pageview that led to
it, which is what makes "risk decisions beside product analytics" a join rather
than a project.
THE PLANE IS ASKED, NOT LINKED. analytics owns that table and the pod forks one
process per app, so its write core is in another pid — the shape this seam has
now met twice (cloud.SetRiskScorer, cloud.SetObsErrorIngest) and the answer is
the one both landed on: a plane op on the owning app's socket. analytics gains
event_capture, the WRITE side of the door obs_error_post already claims a slice
of, reaching the SAME write core every HTTP door reaches — one admission, one
normalizer, one storage projection. No new transport, no hop through the fleet's
front door, no second gate.
THE TENANT IS THE CALLER'S AT BOTH ENDS. The emitted row is filed for the tenant
the verdict was REACHED under, and the door mints its org from the plane
principal; the contract carries no field that could name one, so a peer can only
ever write into its own organisation's partition. An unidentified caller writes
nothing rather than defaulting to a tenant.
IT CANNOT FAIL THE DECISION. The gate that asks this scorer is the credit door,
so the emit happens after the verdict is computed, on a detached goroutine, with
its own budget, behind a ceiling that DROPS rather than queues. A queue defers
the loss instead of bounding it and makes this process's memory a function of
how fast the fleet is being screened; dropping says the true thing, which is
that a telemetry row is expendable and a decision is not. An absent peer, a
refusing one and a wedged one are the same fact here and none of them is the
verdict's.
WHAT TRAVELS IS BOUNDED, IN BOTH SENSES. The attributes are the stage, the
subject's kind, the action, the cause, the refusal, the model shape, the policy
version, the posture, and — on a scored answer only — the score and the cut it
was held against. Every value is server-minted, from a closed set the op already
refuses outside of, or narrowed before it travels: the country to alpha-2, and
the amount to which stated bound it was past rather than to itself. The subject
is a tenant-and-kind-scoped digest, so decisions about one subject still group
and the shared plane does not become a second copy of what the gate stated.
SHADOW IS EMITTED AND SAYS SO, which is the whole point of a shadow regime: what
the gate WOULD have done, on real traffic, changing nothing. No organisation is
armed and the default posture is untouched.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The comment blamed apps/answer for exposing no constructor. That is true and
it is not the reason.
Research carries an explicit 25-cent per-answer fee (apps/answer/mode.go),
charged through Bill.Gate on the request path where a payer has been resolved
and can be refused. A tool call has no payer, so a direct seam to the engine
would be an unbilled 25-cent operation an agent may invoke in a loop — free
inference, reached by the exact route this codebase keeps closing.
That the package makes it awkward is not an accident to route around: Params is
built from request-scoped billing context and Sink's methods are unexported, so
the money gate is structurally hard to bypass. Wiring it properly means an entry
that takes a payer and charges it — a billing decision, not an adapter.
web_search and fetch_url are different in KIND, not merely cheaper: their HTTP
routes gate on AUTHENTICATION, and the agent request reaching the tool was
already authenticated and metered at /v1/responses. In-process use matches how
they are reached over HTTP. deep_research does not.
The rule beside the model reads the event in front of it: where the payer
acted from, and how much moves. Both are bounds on ONE payment, so the
way past them is arithmetic anyone can do — five payments of eleven
thousand instead of one of fifty-five. Every piece is under the bound by
construction, which is what splitting it is for, so no point-in-time test
can ever see it. The same blindness in the other direction: twenty
accounts sharing one device are each unremarkable taken alone, and the
pattern exists only in what they share.
Two more halves now answer beside the first, and both read the
organisation's OWN aggregates rather than the event. PACE is what one of
the event's identifiers — its subject, its counterparty pair, its device
— had already done inside the aggregates' narrowest window. FAN-OUT is
how many distinct subjects one device or one counterparty already ties
together, counted from that organisation's own record.
The value bounds are not restated. What one payment may not move, an hour
of payments may not move either: the accrued value is judged against the
same examining and freeze thresholds a single event is, so there is one
statement of appetite and two readings of it. Only the count needed a
bound of its own, and the freeze is a conjunction exactly as geography's
is — a busy hour alone is not a determination that anything is wrong, and
a shared identifier never freezes anything at all, because a household,
an office and a farm are the same shape from a count.
The window is TAKEN and never named. A rule that read a misspelled window
name would read zero, which is indistinguishable from a quiet subject, so
the burst window is the narrowest one the aggregates actually keep and a
ring set that keeps none is refused rather than answered with a zero.
Both bounds are stated where the geography rule's are, and the count is
held above what a fold alone puts in that window: at or under it, the
rule would fire on this organisation's own history the moment a residency
rebuilt.
The reading is the tenant's own, on both halves. The rings belong to one
resident and the record query carries the qualified tenant as its leading
predicate, so two brands' identically named organisations — which share
one shelf file — cannot see each other's. A reading that cannot be taken
is an error and never an empty one, because empty means "this subject has
done nothing", which is the answer that would silently allow.
Reading the aggregates is a bounded range scan of the record's own
covering index, measured against a record filled to its ceiling at 6.3 ms
— four per cent of the door's budget, and the reason there is no second
index to widen the record's published per-tenant bound.
Shadow is preserved whole. The findings are computed and recorded, the
outcome is left exactly where the model left it, and the cause says what
would have happened. No organisation is armed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The rule beside the model reads the event in front of it: where the payer
acted from, and how much moves. Both are bounds on ONE payment, so the
way past them is arithmetic anyone can do — five payments of eleven
thousand instead of one of fifty-five. Every piece is under the bound by
construction, which is what splitting it is for, so no point-in-time test
can ever see it. The same blindness in the other direction: twenty
accounts sharing one device are each unremarkable taken alone, and the
pattern exists only in what they share.
Two more halves now answer beside the first, and both read the
organisation's OWN aggregates rather than the event. PACE is what one of
the event's identifiers — its subject, its counterparty pair, its device
— had already done inside the aggregates' narrowest window. FAN-OUT is
how many distinct subjects one device or one counterparty already ties
together, counted from that organisation's own record.
The value bounds are not restated. What one payment may not move, an hour
of payments may not move either: the accrued value is judged against the
same examining and freeze thresholds a single event is, so there is one
statement of appetite and two readings of it. Only the count needed a
bound of its own, and the freeze is a conjunction exactly as geography's
is — a busy hour alone is not a determination that anything is wrong, and
a shared identifier never freezes anything at all, because a household,
an office and a farm are the same shape from a count.
The window is TAKEN and never named. A rule that read a misspelled window
name would read zero, which is indistinguishable from a quiet subject, so
the burst window is the narrowest one the aggregates actually keep and a
ring set that keeps none is refused rather than answered with a zero.
Both bounds are stated where the geography rule's are, and the count is
held above what a fold alone puts in that window: at or under it, the
rule would fire on this organisation's own history the moment a residency
rebuilt.
The reading is the tenant's own, on both halves. The rings belong to one
resident and the record query carries the qualified tenant as its leading
predicate, so two brands' identically named organisations — which share
one shelf file — cannot see each other's. A reading that cannot be taken
is an error and never an empty one, because empty means "this subject has
done nothing", which is the answer that would silently allow.
Reading the aggregates is a bounded range scan of the record's own
covering index, measured against a record filled to its ceiling at 6.3 ms
— four per cent of the door's budget, and the reason there is no second
index to widen the record's published per-tenant bound.
Shadow is preserved whole. The findings are computed and recorded, the
outcome is left exactly where the model left it, and the cause says what
would have happened. No organisation is armed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/commerce installs the risk scorer; apps/gateway arms its fleet-wide
abuse gate on cloud.RiskScorerInstalled() and refuses mode=live while that
reads false. As composed, plugin/commerce links commerce alone and
plugin/gateway links gateway alone, so the gateway process sees no scorer
and the gate cannot be armed. That refusal is the fail-safe direction and it
is currently unconditional — which is what makes it fragile: nothing
anywhere fails when it stops being true.
One import into either root would make that predicate answer true in the
process that arms the gate, and the gate would be armed on a co-residency
accident rather than on the question arming asks — whether the risk plane
can answer for the FLEET, which no in-process global settles.
So the invariant is checked structurally, over every `package main` under
plugin/ and cmd/, following first-party imports transitively because linking
is transitive. Anti-vacuity assertions require the scan to actually reach
each app, so a renamed package or a moved root fails the gate instead of
silently passing it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The operator's listing won on MEMBERSHIP alone, so one stated without an
`as_of` went in force whole — and reference.Jurisdictions then refused every
country on it, correctly, because "not listed" from a listing of unknown
currency is not a fact. onGeography swallows that refusal by design, so the
ACTION tier became unreachable, the freeze the rule exists for silently
vanished, and what remained — review at or past the freeze value — looks
exactly like a rule that is working. Nothing logged it: the error that
mattered was consumed per country.
A date is now part of what makes an operator listing usable, checked where
the listing is chosen, and an unusable one loses to the dated compiled
default instead of disarming the half of the rule it was stated to arm.
Unset and malformed are unchanged — both already resolved to the empty
listing and fell to that same dated default.
Choosing is split from reading the environment: resolve is a rule over a
stated listing and can be exercised against every shape one can take, where
a value resolved once at first use cannot be. The reason travels out on
listing.Gap rather than being logged from inside the resolution, because a
control that switched itself off has to be visible from OUTSIDE the process
— Mount says it at startup and /v1/risk/health keeps saying it, beside the
listing's date and age.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Arming and tuning arrive at one address in one body, and only one of them is
self-service. PUT /v1/risk/policy was gated by the billing check and the
tenant guard, neither of which asks about authority, so any member of an
organisation could state {"live":true} and take its model out of shadow —
the decision that lets the model freeze that organisation's customers'
payments, taken by whoever held a token.
Setting `live` now requires cloud.Admin: an admin OF THAT ORG, with
SuperAdmin as the stated superset rather than the requirement, because the
organisation is arming ITSELF and requiring platform sudo would make every
customer's governance decision ours to take. Stating the appetite and the
sample stays self-service.
The check reads the platform's one predicate set and lives beside ops.gate
and caller, which is where this package is allowed to reach for the raw
request. It runs after the principal resolves and before anything is
written, so an unauthorised caller learns nothing about which appetites the
op accepts — the ordering ops.adopt already makes.
Disarming is deliberately left self-service: returning a model to shadow
cannot freeze a payment.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two facts reached the same answer at the self-serve credit door, and each
one reached the wrong one.
A scorer socket that cannot be DIALLED read as a scorer that is not there.
plane.Listening separates three facts — no listener, a listener, and a
socket present but unusable — and scorerUp folded the third into the first
by discarding the error. Absence is the one refusal the fail policy exempts,
so under fd exhaustion (EMFILE), a denied run directory (EACCES) or an
over-long one (ENAMETOOLONG) the probe answered "no scorer here" in
microseconds, against a scorer that was healthy, and a privileged top-up
settled unscored. The budget never caught it because nothing was slow.
scorerUp now returns the error and the seam refuses on it, so a genuine
absence still allows and an outage blocks.
A payment a RULE froze read as a scorer that had not answered. Since the
rule and the model were fused, Action and Refusal are independent: an armed
org with a warming model on a frozen payment answers {restrict, "warming"},
and a gate branching on the refusal alone called that a 503 "try again in a
moment" — inviting the retry that settles the payment it had just frozen.
The pair is the discriminator: riskUnavailable is the only producer of a
block that carries a refusal, and the scorer's own vocabulary tops out at
restrict, which TestActions_TheScorerNeverBlocks now holds closed
structurally.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/commerce installs the risk scorer; apps/gateway arms its fleet-wide
abuse gate on cloud.RiskScorerInstalled() and refuses mode=live while that
reads false. As composed, plugin/commerce links commerce alone and
plugin/gateway links gateway alone, so the gateway process sees no scorer
and the gate cannot be armed. That refusal is the fail-safe direction and it
is currently unconditional — which is what makes it fragile: nothing
anywhere fails when it stops being true.
One import into either root would make that predicate answer true in the
process that arms the gate, and the gate would be armed on a co-residency
accident rather than on the question arming asks — whether the risk plane
can answer for the FLEET, which no in-process global settles.
So the invariant is checked structurally, over every `package main` under
plugin/ and cmd/, following first-party imports transitively because linking
is transitive. Anti-vacuity assertions require the scan to actually reach
each app, so a renamed package or a moved root fails the gate instead of
silently passing it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The operator's listing won on MEMBERSHIP alone, so one stated without an
`as_of` went in force whole — and reference.Jurisdictions then refused every
country on it, correctly, because "not listed" from a listing of unknown
currency is not a fact. onGeography swallows that refusal by design, so the
ACTION tier became unreachable, the freeze the rule exists for silently
vanished, and what remained — review at or past the freeze value — looks
exactly like a rule that is working. Nothing logged it: the error that
mattered was consumed per country.
A date is now part of what makes an operator listing usable, checked where
the listing is chosen, and an unusable one loses to the dated compiled
default instead of disarming the half of the rule it was stated to arm.
Unset and malformed are unchanged — both already resolved to the empty
listing and fell to that same dated default.
Choosing is split from reading the environment: resolve is a rule over a
stated listing and can be exercised against every shape one can take, where
a value resolved once at first use cannot be. The reason travels out on
listing.Gap rather than being logged from inside the resolution, because a
control that switched itself off has to be visible from OUTSIDE the process
— Mount says it at startup and /v1/risk/health keeps saying it, beside the
listing's date and age.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Arming and tuning arrive at one address in one body, and only one of them is
self-service. PUT /v1/risk/policy was gated by the billing check and the
tenant guard, neither of which asks about authority, so any member of an
organisation could state {"live":true} and take its model out of shadow —
the decision that lets the model freeze that organisation's customers'
payments, taken by whoever held a token.
Setting `live` now requires cloud.Admin: an admin OF THAT ORG, with
SuperAdmin as the stated superset rather than the requirement, because the
organisation is arming ITSELF and requiring platform sudo would make every
customer's governance decision ours to take. Stating the appetite and the
sample stays self-service.
The check reads the platform's one predicate set and lives beside ops.gate
and caller, which is where this package is allowed to reach for the raw
request. It runs after the principal resolves and before anything is
written, so an unauthorised caller learns nothing about which appetites the
op accepts — the ordering ops.adopt already makes.
Disarming is deliberately left self-service: returning a model to shadow
cannot freeze a payment.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two facts reached the same answer at the self-serve credit door, and each
one reached the wrong one.
A scorer socket that cannot be DIALLED read as a scorer that is not there.
plane.Listening separates three facts — no listener, a listener, and a
socket present but unusable — and scorerUp folded the third into the first
by discarding the error. Absence is the one refusal the fail policy exempts,
so under fd exhaustion (EMFILE), a denied run directory (EACCES) or an
over-long one (ENAMETOOLONG) the probe answered "no scorer here" in
microseconds, against a scorer that was healthy, and a privileged top-up
settled unscored. The budget never caught it because nothing was slow.
scorerUp now returns the error and the seam refuses on it, so a genuine
absence still allows and an outage blocks.
A payment a RULE froze read as a scorer that had not answered. Since the
rule and the model were fused, Action and Refusal are independent: an armed
org with a warming model on a frozen payment answers {restrict, "warming"},
and a gate branching on the refusal alone called that a 503 "try again in a
moment" — inviting the retry that settles the payment it had just frozen.
The pair is the discriminator: riskUnavailable is the only producer of a
block that carries a refusal, and the scorer's own vocabulary tops out at
restrict, which TestActions_TheScorerNeverBlocks now holds closed
structurally.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A fresh account's first top-up is the one the model can never have an
opinion about: it has no history, so it is warming, so it declines, so
the credit door allowed it unscored. Ten million dollars from a
jurisdiction no anti-money-laundering supervision reaches is not
anomalous — it is unprecedented, which scores as nothing.
A second judge now answers beside the model, reading two stated facts:
the jurisdiction the payer acted from and the value moving. It does not
read the score, the cut or whether the model has warmed, so a refusing
model cannot soften it; the fused verdict is the severer of the two, and
a model with no opinion contributes an allow.
The jurisdiction listing is the one the screening engine already
maintains, in two tiers because the required response differs: a tier
called for countermeasures may freeze, a tier under monitoring may only
summon a person. A formal designation stays the AML plane's to make.
The gate states the jurisdiction our own edge resolved, under the same
trust rule the client address already uses — a direct caller's header is
the caller's own writing and is refused. The billing jurisdiction is the
one worth judging and no part of it reaches this process; the weaker
signal is documented as weak and replaced at one line when there is one.
Shadow is preserved whole. The finding is computed and recorded, the
outcome is left exactly where the model left it, and the cause says what
would have happened. No organisation is armed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A fresh account's first top-up is the one the model can never have an
opinion about: it has no history, so it is warming, so it declines, so
the credit door allowed it unscored. Ten million dollars from a
jurisdiction no anti-money-laundering supervision reaches is not
anomalous — it is unprecedented, which scores as nothing.
A second judge now answers beside the model, reading two stated facts:
the jurisdiction the payer acted from and the value moving. It does not
read the score, the cut or whether the model has warmed, so a refusing
model cannot soften it; the fused verdict is the severer of the two, and
a model with no opinion contributes an allow.
The jurisdiction listing is the one the screening engine already
maintains, in two tiers because the required response differs: a tier
called for countermeasures may freeze, a tier under monitoring may only
summon a person. A formal designation stays the AML plane's to make.
The gate states the jurisdiction our own edge resolved, under the same
trust rule the client address already uses — a direct caller's header is
the caller's own writing and is refused. The billing jurisdiction is the
one worth judging and no part of it reaches this process; the weaker
signal is documented as weak and replaced at one line when there is one.
Shadow is preserved whole. The finding is computed and recorded, the
outcome is left exactly where the model left it, and the cause says what
would have happened. No organisation is armed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`Deps` braided capabilities (a KMS client — a thing you USE) with policy (a
domain, a default model — a thing you DECIDE). Different owners, lifetimes and
failure modes: a client is wrong when it cannot connect; a policy is wrong when
it says the wrong thing, silently.
THE HEADLINE: `Domain` answered two questions. 18 readers want THIS PROCESS'S
API host; 5 want THE DEPLOYMENT'S APEX. The build outage was a reader in the
second group reading a field built for the first — selfGitHost = deps.Domain =
"api.hanzo.ai", so the forge at git.hanzo.ai (a SIBLING) was refused and every
native build fell back to GitHub.
dc84b46d fixed ONE derivation. There were THREE, with three algorithms:
platform.apexOf publicsuffix
sites.registrableDomain last-two-labels
git.defaultSSHHost TrimPrefix "api."
They diverge on 3 of 7 realistic hosts. defaultSSHHost("cloud.hanzo.ai") yields
git.cloud.hanzo.ai, which platform's allowlist then REFUSES — the identical
outage, still armed, needing only a different CLOUD_DOMAIN. And sites fed a bare
multi-label suffix (co.uk) into SelfDomains, claiming every domain under it.
All three now call brand.Apex / brand.Sibling — one implementation in the only
leaf all four consumers reach.
TWO MORE LIVE DEFECTS, measured:
IAMIssuer had TWO answers. With CLOUD_BRAND=lux and nothing pinned,
Config.IAMIssuer = "https://lux.id" while the package-level IAMIssuer()
read env raw with no brand fallback and returned "". Empty flows to IAMBase(),
which the sk- key resolver dials; unresolvable -> nil -> EVERY API-KEY REQUEST
AUTHENTICATES AS ANONYMOUS, silently, while JWT keeps working.
Domain was brand-blind where IAMIssuer was not: unpinned lux resolved
issuer=lux.id alongside domain=api.hanzo.ai.
DELETED, not migrated — no value gained a second home:
AIFallbackModel 1 reader, set by nobody, ever -> cloud.FallbackModel
AIDefaultModel -> cloud.DefaultModel; universe sets enso-flash and the
constant IS enso-flash, so behaviour is preserved. This made
agents.go's "model is required" branch provably dead.
Self ZERO readers, its whole life. selfID(cfg) is real and stays.
chart iamIssuer a value in NO brand row, rendered unconditionally, beating
the derivation on every chart deploy.
the "api.hanzo.ai" literal in apps/sites -> brand.APIHost, the same root
Config uses.
KEPT, each with a written justification in deps.go: Brand (whose deployment
this is, fixed at exec), Domain (this process's own API host — now documented as
NEVER the apex), IAMIssuer (the trust root; verifying identity precedes any
store you would read it from), Version/Env (boot facts).
deps_policy_test.go reflects over Deps and FAILS on any field without a
rationale, names the three evicted fields with why, and proves the env knobs are
inert — so a policy field cannot quietly re-enter through the dependency door.
build ./... 0, vet ./... 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerce v1.50.7 added PUT /_/commerce/providers/{name} — the verb behind "a payment
rail can be turned on and off" — and the dep bump landed without prose for it, so
`commerce describe` refused and the app could not project its own document. The
subset is fail-closed, so nothing was written and the drift stayed invisible until
the next regeneration asked.
It gets the sentence, and the same gate its GET twin states: the tenant comes from
the IAM owner claim and nowhere else, so a cross-tenant write is not expressible;
404 for an unknown provider is byte-identical to the cross-tenant probe's answer. The
one thing worth saying that the shape does not: this owns a single bit and never the
credential, which is why a rail can come back with the same stored secret.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
analytics' subset caught up in the regeneration and started publishing GET
/v1/event.js — an address the fleet then routed to ai, because a prefix owns a
SEGMENT subtree and "/v1/event" does not cover "/v1/event.js". It fell through to
ai's "/v1" and the router oracle said so: the fleet published a path it delivers
somewhere else.
The address is analytics' own — its binary projected it from its own router. So it
gets its own entry, which is what "deeper than the sibling that currently wins"
means for a path whose distinguishing character is a dot.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The weave was red on main and nothing downstream could move: openapi/weave_test.go
is the SOLE writer of openapi.yaml, so the served document, the CLI, every SDK, MCP
and the docs were all frozen behind
operationId "get_v1_billing_portal_methods" is claimed by both
"GET /v1/billing/methods" and "GET /v1/billing/portal/methods"
The derivation was never at fault. operationID(method, path) walks every segment and
yields two distinct ids for those two paths, and it has verified uniqueness inside
From since 2026-07-27. The subset that claimed otherwise was written on 2026-08-04 —
so no run of `commerce describe` at any version could have emitted it. It was
hand-edited: d7024e88 moved the three saved-card verbs to this app and wrote the two
new paths into plugin/commerce/openapi.json by copying the /v1/billing/portal/methods
block, operationId and prose together. The copied description then described itself
("the SERVICE-TOKEN face of the same list a customer reads at /v1/billing/methods",
published AT /v1/billing/methods).
It was hand-edited because it could not be generated: `commerce describe` refused,
and still refused here, with eight operations saying nothing about themselves. So the
fix is the prose, and the file follows from it.
- Eight operations get the sentence they owe a caller: the customer saved-card
family (GET/POST/DELETE /v1/billing/methods), its portal POST, and the top-up
rails nobody had described at all — wire instructions, crypto options, the
deposit mint and the deposit poll. Five of them were not in the published
document in any form.
- The two portal twins stop claiming a proxy that no longer exists. Both families
are served in this process; they are two addresses because they admit two
principals, not because either forwards to the other (apps/billing/billing.go
says the same at the spot the hop used to sit).
- mount.go said the opposite of the manifest — that /v1/billing/methods belongs to
billing and "a registration here is unreachable in the fleet", beside a live
registration of it. manifest.Apps gives commerce both prefixes and withholds them
from billing. Left alone, that comment invites deleting a route that works.
- /v1/commerce/{deposits,deposits/:id/confirm,deposits/:id/status,webhooks/:provider}
lose their prose. The broker-dealer proxy behind them was deleted and the manifest
stopped naming them; what was left was prose for routes nothing serves, which
renders nowhere and reads in source as though it were live. openapi/floor.json
drops commerce 8 → 4 in the same commit, which is where the ratchet asks for the
reason to be.
AND THE ASSUMPTION THAT LET IT LAND IS NOW A CHECK. Weave did refuse the document —
but a collision inside ONE part reaches it with no app attached, so it could name the
two addresses and nothing else, and which of 123 subsets shipped them was a search.
openapi.Subsets is holding the app's name when it decodes the bytes, so it asks there
whether the part is injective — the same uniqueOperationIDs, one statement of the
rule, asked once per part and once over the composition, because neither fact implies
the other. TestSubsetsRefusesAnAppWhoseOwnIDsCollide fails without it.
Regenerating also lands 13 subsets that had gone stale behind the red gate, and drops
zipdoc entries that were lifting commercemid.RequestContext's doc comment — a
paragraph about mint-gated context locals — as eight routes' descriptions.
plugin/iam is deliberately NOT regenerated: its current projection introduces a
schema "Role" that means something different from framework's ({role, user}), which
Weave refuses and which needs a rename in one of those two apps. That is a separate
defect this one merely uncovered.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Carries the P0 fix a live-browser pass proved: the assistant's streamed
completion posted with no Authorization header, was refused, and the card
then blamed the user's session. The release routes every self-reading
stream through the client's one authorized door, moves preferences onto
cloud's /v1/prefs, tells the truth on a 401, and mounts exactly one
composer per viewport.
The server halves for git_status and git_mirror, beside the import and inbound
ops already published here. Same rules as the others: the org is the caller's
plane identity, an anonymous caller is refused, an unmounted store answers 503.
A repo git does not know is simply absent from the reply, which is a real answer
and reads as not-imported.
It also closes a loop that was latent in the two existing handlers. planeInbound
and planeImport called cloud.InboundGitSync and cloud.ImportGitRepo, which now
fall through to the plane when the in-process importer is nil — so a handler
routing through them could dial its own socket and answer itself. It cannot
today, because Mount registers the importer before publishing the op, but that is
an ordering nobody is checking and exactly how the loop arrives later. Both now
call githubImporter directly, like the sync and tracker handlers.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ai v1.832.30 declares web_search / fetch_url / deep_research but holds no
backend for the two this host serves. agent/builtin_tool/web must stay a leaf
package (object imports agent, agent imports the registry), and websearch lives
here in any case — so this is where the seam closes, beside the balance and tier
readers, for the same reason they are here: this package links both sides and
the host does not.
In-process, never over api.hanzo.ai. The edge validates a CUSTOMER credential
and answers 401 to a service; routing our own calls back through it is what once
fail-closed every completion at 503 on a perfectly healthy pod.
deep_research is deliberately NOT installed yet. apps/answer builds its Params
from unexported fields and exposes no constructor, so wiring it means giving
that package an entry point rather than reaching into it from here. Until then
the tool reports that it is UNAVAILABLE IN THIS DEPLOYMENT — which is the honest
answer and specifically not an empty result: an agent told "no results"
concludes the web holds nothing on the subject and answers from memory in a
confident voice.
fetch_url needs nothing here — ai installs its own crawl at bootstrap.
Renames only. No behavior changes.
"Seam" named the thing badly. Every one of these is just one app calling
another, or a process calling itself, and saying so plainly is shorter than the
jargon was:
sync_seam.go -> sync.go
tracker_seam.go -> tracker.go
seam_absence_test.go -> absent_test.go
seamClassification -> kinds
crossApp / inProcess -> remote / local
absentSeamCalls -> calls
seamsInSource -> registers
clearSeams -> unregister
upsertIssueSeam -> upsertIssue
TestCrossAppSeamAbsenceIsAnError -> TestAbsentErrors
TestAbsenceIsDistinguishable -> TestAbsentIsNoPeer
TestEverySeamIsClassified -> TestAllListed
roundtrip_test.go is new and is the missing half of the argument. absent_test.go
shows a call errors when the app is gone, which on its own a call that always
failed would also satisfy. This stands the app up on a real socket and shows the
same call arrives with the caller's org and comes back with the app's answer —
through the public functions, not through Ask, so the op name, the wire types
and the org forwarding are all under test. Any of them drifting would leave
absent_test.go green and production broken.
The word survives in files this change does not own; it is house vocabulary
there and churning them would collide with work in flight.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/event.js resolved the anonymous id itself, out of localStorage alone. That is
ORIGIN-scoped, so it never saw the cookie @hanzo/event shares across *.hanzo.ai
and never saw the id hz.js had already left behind under its own key: an origin
carrying only this tag was a separate population, and one visitor was two or
three people depending on which snippet a surface happened to load.
anon.js is that chain, vendored BYTE-FOR-BYTE from @hanzo/event
(hanzoai/ui pkgs/event/src/anon.js), and tag.go now serves it with the tag as one
asset inside one wrapper — so the door holds no second implementation, and
neither half leaves a name on the page it is pasted into. tag.js calls hzAnonId
and no longer names an identity key at all.
Resolution is cookie · localStorage hz_anon_id · localStorage hz_id · in-memory ·
mint: every id already in a browser is adopted, and only a browser holding none
is given a new one.
TestTagBehavior now runs the COMPOSED asset rather than tag.js, because the chain
is half of what ships and the other half no longer runs alone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The last way the lobby and the mint could still disagree. mint refuses a token
carrying no account — the account IS the identity the seat is taken under — so a
lobby that offered a workspace off one would show a room, take the choice, and
then decline it. token.Generate cannot mint that shape, so this is not an
attacker's path; it is the invariant written down where the offer is made rather
than assumed from where it is granted.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The media server (LiveKit, live.hanzo.bot) and the join-token mint were already
native. The CLIENT was the office plugin inside a forked Team front — a whole
product's UI carrying one screen — so a Hanzo call was ours everywhere except
the part people look at. This embeds the native client in meet's own binary and
serves it at /meet, beside the /v1/meet it reads.
ONE ORIGIN is the point, not a convenience: the lobby's read is a credentialled
same-origin GET, so there is no CORS grant to make, no second host to hold a
session on, and no bundle carrying an API address it could be pointed away from.
Same shape apps/tasks already has.
THE LOBBY NEEDED A FACT NOBODY PUBLISHED. A room is bound to its tenant by the
leading segment of its name — apps/meet refuses to mint unless the caller holds a
privileged member row in exactly that workspace — so a client that does not know
its own workspace cannot compose a room name any lane would admit, and a uuid is
not something a person types. Two additions, and no new authority:
plane.TeamWorkspaces — memberOf asked the other way round, over the same rows
and the same three refusals, so the two can never disagree about who is a
member of what. It is a person's OWN memberships, never a workspace's roster.
GET /v1/meet/session — the identity a seat would be taken under, the media
address, and the workspaces this caller may open a room in, already narrowed by
the SAME predicate the mint admits on. The offer and the grant come from one
rule or they drift; a test holds them together across every role.
It is UNTYPED for a reason that is this app's alone: the gate is principal.Minted,
the boundary's own attestation, and a typed op holds a context rather than a
request. The context-side facts a typed op can read are header-derived, and in a
hand-written plugin main nothing strips those — which is exactly the forgeable
signal admits was fixed to stop selecting on.
An unreachable team is a REFUSAL, not an empty list: "you have no workspaces" is
a lie when the truth is "the authority is down", and it sends someone to ask for
an invite they already have.
LIVEKIT_WS is served rather than compiled in, because it is a deployment fact and
the bundle is a build artifact — baked in, a dev cluster's UI dials production.
Unset is an empty string the client acts on, not a 503: the published office
client supplies its own address, so this degrades the native UI and nothing else.
spa/ is the fifth copy of one SPA handler collapsed into the first. tasks and
research had it byte-identical and meet would have made three; the policy in it
(immutable hashed assets, index fallback for a deep link, 503 for an unsynced
bundle) is not obvious enough to be worth restating, and a policy restated drifts.
Both existing suites pass against it unchanged. webui stays its own handler — it
owns "/" and has to refuse the API namespaces and rewrite the title per brand.
manifest names /meet and the whole /v1/meet subtree. Listing each leaf was a list
that had to be edited every time a route was added, and an unnamed leaf falls to
whichever row holds the bare remainder.
The test helper `session` became `workspaceToken`, which is what it always made.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Embeds the Hanzo Tracker SPA (hanzoai/admin apps/admin-tracker) and serves it at
/tracker/*, beside the /v1/tracker surface it reads. One binary, one origin, one
deploy — which is what lets tracker.hanzo.ai answer with a native board and
retire the Huly tracker that answers it today.
Same origin is the design, not a convenience. The SPA sends NO tenancy of its
own: the composer's identity check mints the validated org from the IAM session
before either half runs, so the page carries a session cookie and nothing else.
A UI on a second host would have to hold a token and name an org, which is
precisely the client-supplied tenancy this surface refuses.
The routes stay untyped and always will: they answer HTML and hashed assets
under their own content types and cache hints, plus index.html for every path
the client router owns, and a typed op publishes JSON.
embed_test.go pins the two failures a bad sync makes SILENT rather than loud:
a bundle built for the wrong base resolves every chunk to a path nothing serves
(a blank page, not an error), and the API prefix is inlined at build time, so a
bundle built against a dev proxy renders and then talks to a surface this binary
does not answer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The bank details were stored and the rail still answered 'Wire transfer not
configured'. Both doors reach the same KMS store keyed by (path, name, env), so
a read at a path nothing writes is indistinguishable from a bank nobody entered
— it would have stayed silent indefinitely.
cloud writes every in-process secret under /orgs/{org} (apps/destinations,
apps/integrations, credz all build that shape, and the REST surface folds writes
under it from the validated org claim). commerce alone spelled it
/tenants/hanzo/wire. v1.50.7 adopts the host's convention and drops the
hardcoded org: the brand serving the page now decides whose bank is read.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The extension renders structured result widgets (comparison, steps, stats,
timeline, entity, definition) but nothing emitted them: /v1/ask's stream is
status|sources|text|follow_ups|done and the system prompts never mentioned the
format. The client was ready and the producer was missing.
widgetRule teaches the one format both modes share, so search and research
cannot drift apart. It says three things beyond the schema:
- Only when the question's SHAPE calls for one, at most two, most answers none.
A widget on every answer is a worse document, not a better one.
- The PROSE MUST STAND ALONE. The client validates every block and drops
anything malformed — a ragged table, an unknown kind, a field of the wrong
type — keeping the prose. An answer that leaned on a widget to be complete
would read as a hole exactly when validation refused one.
- DATA ONLY, NEVER HTML. This model reads the open web, so every page it
fetches is a potential injection source. Markup it authored would be a path
into the extension's origin, which holds the user's session. The renderer
owns the shapes and escapes every field; the model owns only content.
widget_rule_test.go is the closest thing to a shared type across the two repos.
The renderer lives in the extension, so a typo here fails nothing: it produces
answers whose widgets silently never appear, which looks exactly like a model
choosing not to emit one. The test parses every example out of the prompt and
holds it to the shape the client accepts — one example per kind, every required
field present, the comparison example rectangular (the client drops ragged
tables rather than padding them), and both modes carrying the rule.
Negative control run: renaming the steps example's field to "items" fails with
`kind "steps" example omits required field "steps"`.
Pre-existing and unrelated: the root package does not build right now
(middleware_spend.go, another session's in-flight work). apps/answer builds and
tests clean.
The board reads Status; a timeline reads dates, and the tracker had none. Adds
StartAt/DueAt (unix seconds, 0 = unset) as mutable board state on Issue, plus
one filter — Scheduled — selecting the rows a gantt has somewhere to draw.
No milestone table, because a milestone is not a second kind of thing: it is
this row with a due date and no start, an interval of zero length. A dated epic
is a phase and its children are already reachable by ExtRef. The timeline stays
what contract.go says every work-item surface is — a FILTER over the one table
— so the board and the timeline cannot drift.
The interval is validated at the boundary and refused rather than normalised: a
negative bound, or an end before its beginning, is 400. The PATCH validates the
RESULTING pair, so moving only the due date is still checked against the start
already stored.
Forward-added the way the polymorphic spine was: ADD COLUMN with defaults after
the base DDL, its index after the columns, so a live tracker.db migrates without
a crashloop and every existing row reads as unscheduled.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
No release has pinned since v1.801.454. .455 and .457 built, published, and
were never referenced by anything — the fleet has been serving .454 while
three versions sat in the registry.
The rollout car reads UNIVERSE_PIN_TOKEN from
/v1/kms/orgs/<org>/secrets/deploy/... and the broker has no such route. The
store root is derived from the validated claim, which is exactly what makes
another tenant's secret unnameable rather than merely refused, so there is no
org in a KMS path and never was. Measured against kms.hanzo.ai:
/v1/kms/auth/login 401 (route exists)
/v1/kms/orgs/hanzo/secrets/deploy/UNIVERSE_PIN_TOKEN 404
/v1/kms/secrets/deploy/UNIVERSE_PIN_TOKEN 403 (route exists)
fanout reads FLEET_DISPATCH_TOKEN through the same wrong shape and would have
failed the same way the moment rollout stopped failing first.
WHY IT WAS INVISIBLE, WHICH IS THE HALF WORTH FIXING. The step is
`set -euo pipefail` and the read is `TOKEN=$(curl -fsS ... | jq ...)`. curl
-f exits 22 on the 404, pipefail propagates it, set -e aborts the assignment —
so the authored `::error::UNIVERSE_PIN_TOKEN missing in KMS ...` is
unreachable code. Every one of these releases died with a bare `exitcode
'22': failure` and never printed the sentence written to explain it. Three
guards in this file were dead the same way; the two KMS logins are fixed here
too, and `receipt`'s create-or-update on a tag with no release yet is the
third (left alone — it is a different car and a different bug).
A guard that cannot run is worse than no guard: it reads as diagnosis
already handled.
KMS_ORG goes with the paths. The org is the credential's, so a variable that
named it was a knob that decided nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The corpus has two sources and both were in-process globals the catalog
process cannot see. The index write was one (previous commit). This is the
other, and it failed more quietly still.
projects.LiveSites answers nil when its package is unmounted, on the stated
reasoning that a deployment which hosts no sites is not an error. That is
true of a DEPLOYMENT and false of a PROCESS: catalog, index and projects are
three separate apps and therefore three separate processes, so in the
catalog process nil never meant "nothing is serving" — it meant "you asked
the wrong half of the fleet". nil and empty are the same answer, so the
corpus was published with no sites in it and nothing anywhere said so.
What that costs is the whole `site` kind: every demo URL, the lineage that
files a remix under community instead of leaving it looking like one of our
starters, and the deployed demos the template lane is mostly made of. The
repos alone would have made a catalog of source with nothing live in it.
plane sites_live, and it takes no org — the same shape as
sites_resolve beside it. This is THE cross-org read; the rule
that makes it safe (public, live, not hidden) is applied in the
query by the app that owns the store, so there is no tenant here
for a caller to widen into.
projects Ready(), so a caller can finally tell "nothing is serving" from
"ask the process that owns the store" — the distinction
LiveSites cannot make, because it answers nil for both.
catalog serving(), the third seam with the same two legs as lexical and
write.
Same test file, same reason: the split is what production runs, so the test
reaches a real peer over a real socket.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud.SetRiskScorer has held the seam and the whole fail policy since it was
written and has never had a producer. It could not have one: the model is
in-process mutable state, so exactly one binary may hold it, and the pod forks
one process per app. Every gate in the fleet therefore read a nil scorer, took
the absent exemption, and allowed unscored — fleet-wide, silently. The
observability plane's event door was the same shape and learned it the
expensive way; obsevents.go is gone and apps/o11y/obs_rpc.go replaced it.
So the scorer is REACHED rather than linked. apps/risk publishes risk_decide on
its own socket, commerce installs a plane client as the seam's first producer,
and POST /v1/billing/topup/token — the one self-serve credit door, whose mint
authority is a settled card charge — asks it before the charge.
The tenant is minted from the PLANE CALLER, never from the body. RiskDecideIn
carries no org and cannot: a model is trained on one organisation's own
behaviour, so naming which model answers would be the only cross-tenant read
this plane has to offer. The HTTP mint (tenantOf) reads a principal parked on a
request, which a plane call does not have, so it would fail closed on every
call; planeTenant qualifies cloud.Who(ctx).Org with the deployment's own brand
instead, and qualify() refuses an empty org.
The gate states Privileged explicitly. cloud.Privileged() matches IAM and KMS
paths and not this one, so the default would be the fail-OPEN branch — a scorer
outage waving through the one route that mints spendable balance. With the bit
set, a scorer that is present and cannot answer refuses and the caller retries;
a scorer that is NOT DEPLOYED still allows, which is the exemption cloud's fail
policy already carries and the plane client is careful to preserve: over a
socket "not deployed" would otherwise arrive as a failed call and take the
closed branch, so the socket is probed, ErrNoPeer is read as absence, and the
lazy child is woken off the request path rather than inside a 150ms budget.
A REFUSAL CARRIES NO SCORE. The engine assigns one before it checks whether the
model has warmed, so a declining model returns a populated, meaningless number;
publishing it would turn "no opinion" into "this is fine". And an alert is a
REVIEW, never a block: cloud's own vocabulary says a statistical judgement may
reach review and no further on its own, and this model is exactly that.
IT SHIPS IN SHADOW. No org is armed, no regime is changed, and a model nobody
has reviewed is in shadow — where alert is forced false however high the score.
So every legitimate top-up proceeds today and every decision is on the record
with the shape and policy version that produced it. The subject kinds move to
the call contract for the same reason the signal names are there: two spellings
of "account" would not read as a disagreement, they would namespace one subject
into two.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The published catalog has been empty since catalog and index became two
plugin rows, and not because a sync failed to run. It ran every hour, read
GitHub and the sites table correctly, assembled the whole corpus — and then
handed it to index.Reconcile, which serves out of the index's own
process-level global. In the catalog process that global is nil and always
will be. So the corpus was never written a single time.
The READ leg was given a plane op when the split happened; the write leg was
deliberately left in-process, on the reasoning that the store has one writer
and it lives where the file lives. That property is right and the conclusion
was wrong: the writer is still one and still the index's, whether the corpus
reaches it through a function call or a socket. Handing it over does not add
a second writer — a second process OPENING that SQLite would, and neither
leg does.
Fixing the read alone therefore could not have shown a row. It turned a
503 into 200 {"data":[],"total":0}: from a page that said it was broken to
one that said the fleet had built nothing. hanzo.app's /community and
/templates have rendered that empty ever since.
plane index_reconcile, the mirror of index_query, with the corpus
relayed verbatim for the same reason the read's rows are raw.
index publishes it beside the read; the swap still executes here,
in the process that owns the file. query_rpc.go -> rpc.go, the
name the other multi-op peers already use.
catalog write(), the mirror of lexical(): in-process first, then the
plane. Both legs now go through the GENERATED index client, so
the app name, the op and the In/Out pair are fixed to each
other by the compiler instead of at run time.
The suite could not have caught this: every existing catalog test mounts the
index and the lens on ONE app, which is the topology this deployment stopped
having — index.Ready() is true there, so the plane leg was never executed by
a test at all. split_test.go models the split instead and reaches a real peer
over a real socket. It fails on the old code with "index: not mounted".
A silent write failure outlives a loud read failure, and it is the harder one
to see: an error names itself, an empty page looks like an answer.
Also commits plane/team, generator output for an op added without a
regenerate in c8b7c31.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The provider list that decides which rails pay.hanzo.ai offers was readable and
not writable, so card, Apple Pay, Google Pay, Cash App, ACH, wire and crypto were
whatever happened to be in the tenant row and moving one meant editing the
database. v1.50.5 adds the write verb, one rail per call — a whole-list PUT built
from what the admin read projection exposes would write every provider back with
an empty KMS path and silently disconnect all of them from their credentials.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/commands serves zip.CommandsFromSpec over the rendered document —
the same function the CLI's tree is derived from, on the same bytes every
SDK is generated from. No new registry, no generator, no build step: a
route registered this morning is a command this afternoon.
serve() gains one line, so both document sources light up at once — Mount
for an app binary, MountFleet for the front door that answers api.hanzo.ai.
Total, and unfiltered by caller on purpose. zip has no per-op scope; it has
Authorizer, which decides on the DECODED INPUT of every op. Permission is a
fact about an input, so any list filtered here would be a second static
claim the Authorizer is free to contradict — wrong in the direction that
hides working functionality from people who have access.
Two things the design did not know, both measured here:
The size. The projection is 2,344,651 bytes, 454,881 gzipped — 1.32x
smaller than the document, not the 4.5x/107 KB the design quotes. That
figure was taken from hanzoai/cli's spec/products.json, a different
artifact with every description, summary and operationId stripped. This
one keeps the prose, and the prose is most of it (Description alone is
1,028,067 bytes). Recorded in command.go rather than fixed by forking
zip.Command into a trimmed wire shape.
The order. 41 of the 2,323 commands share zip's (Service, Name) sort key
— `mq streams-delete` is claimed by three — and sortCommands is unstable,
so the tie fell to a map walk. Two replicas weaving one document served
identical content under different ETags, which is a full re-download on
every conditional request that lands on a different pod. order() completes
the key with (Method, Path).
openapi.Door names what serve registers. Three gates needed exactly that
fact and each had written the one literal that was true when it was written
— Complete skipping a description no app can own, cmd/cloud exempting the
doors from a scoped deployment's surface, manifest refusing an app row that
claims one. The second door made all three wrong the same afternoon.
Pins: every served command is a route the document carries; the served
bytes are CommandsFromSpec of openapi.yaml exactly; two mounts of one
document agree on bytes AND ETag; a conditional request gets 304 and no
body; the projection is smaller than what it projects. Plus the door itself
answering on the host's own spec() mount.
TestFleetIsTheWeaveOfItsApps and two cmd/cloud gates are red on this tree
already, over a duplicate operationId between GET /v1/billing/methods and
GET /v1/billing/portal/methods. Not touched, not absorbed: openapi.yaml
therefore does not yet carry /v1/commands, and the full-fleet version of
the door test belongs in the commit that fixes the weave.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The image job reserves a version by creating refs/tags/v<N> AT THE COMMIT on
github.com. A ref can only point at an object that is there, so for a commit
github has never seen the claim answers 404 — 'Object does not exist' — and
refuses to build.
It routinely has not seen it. CI runs on git.hanzo.ai, which is canonical and
where the push lands; github is fed by a push mirror on an EIGHT-HOUR interval,
and the claim runs seconds after the push. The object the tag must name is
normally hours away, so the failure is not a permissions problem that looks like
a race — it is a race that looks like a permissions problem, and it is why four
days of releases stacked up behind one commit.
The commit is now published to github before the claim reads it, on a ref of its
own: main is the mirror's to move and the two lineages do diverge, while this
step's only job is to make the object exist.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
metric v1.10.0 moved the OTel bridge out of its root package, and zip took
the bump, so the framework and everything that inherits it now link no
OpenTelemetry SDK at all. Cloud's own meter pipeline still imports the SDK
directly and deliberately — it feeds the status page until zip's rows are
proven where the gauges read, and it is deleted with that proof, not before.
Moving /v1/billing/methods in-process (it had been forwarded through a base URL
this deployment never sets, so a signed-in customer got 401 listing their own
cards) quietly changed what an ANONYMOUS caller gets: the proxy answered 401 and
the co-resident chain answers 403, because PinBillingSubject refused both of its
cases with ErrForbidden.
Those two cases are different answers and must not share a status. A service
token is a credential: presenting one and omitting X-Org-Id is an authenticated
request that names no scope, and 403 is right. Presenting nothing is not signed
in. The difference is load-bearing on the customer path — a browser
re-authenticates on 401 and merely reports 403 — so an expired session on the
saved-cards screen showed a permission error instead of sending the customer to
sign in. The billing app it moved from answered 401 on purpose; its test said so
in as many words, and that test was deleted with the proxy.
So the tests move with the route, which is how this surfaced at all:
apps/commerce gains the assertion the billing app used to carry — anonymous
GET and POST answer 401, and a 404 fails loudly rather than passing as
"refused", since an unmounted route also declines every request.
apps/billing loses seven tests that pinned a proxy that no longer exists, the
three handlers behind them (paymentMethods, createPaymentMethod,
deletePaymentMethod — defined, registered nowhere, dead since the move), and
an openapi.Describe for DELETE /v1/billing/methods/{id}, an address this app
published and did not serve. That last one is the exact publish/serve
disagreement manifest.Apps exists to catch.
Also merges the GitHub lineage, which had diverged: 4 commits there (console
embed pinned by semver, platform forge sibling, the ai subscriber-scoring fix)
against 19 here. The version claim mints its tag on github.com, so a commit that
only ever reached the forge cannot be claimed — which is what the image job hit
after the containment fix let it run at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every team surface resolved its caller by decoding an HS256 token itself, so
"who is calling" was answered in six places against one signing key. This adds
identity (apps/team/account.go): ONE seam that resolves a caller, and the only
place a credential's algorithm is routed on. account, typed, files, billing,
collab and the transactor hold the seam instead of the secret, and three of them
stop importing the condemned package — account.go is the only reader left inside
apps/team.
Verification, tenancy and authorization are three questions, answered separately:
- VERIFICATION is cloud's own IAM validator, narrowed. A signature from a
trusted issuer says IAM minted the token, never that it was minted FOR this
surface — IAM's signer emits the same claims into an access token and an
id_token but for aud/tokenType/nonce. A session door must say which it
means, so the lane takes access tokens whose audience this deployment NAMES.
The boundary's no-audience-gate posture is right for an API door and wrong
here; the divergence is stated at the pin.
- TENANCY is the home org from the signed membership set, never `owner`.
`owner` carries the application's org, so it is chosen by whichever app the
caller authenticated through, and a lane reading it scopes every store query
to an org the caller selected. No membership set means no home, which is
also every machine credential — a team session is a person's.
- IDENTITY is the `sub` claim, resolved through the store. The canonical user
id falls back to preferred_username, and an account id derives from a UUID
verbatim, so a token with no sub whose username is a colleague's account uuid
resolved to the colleague. Subject-only, confirmed against the rows a login
created.
An IAM credential never leaves the seam: it is an estate-wide bearer held in an
HttpOnly cookie so page JS cannot read it, and the account RPC echoes a caller's
token back to page JS.
Workspace authorization on the IAM lane is the membership rows (admit) — the
server decides, the caller signs nothing. The transactor keeps its path-borne
workspace token and gains no ambient lane: a WebSocket is exempt from CORS, so a
cookie-borne credential would make the Origin list the data plane's only access
control, and that list no longer carries a wildcard either.
The existing credential answers first on every carrier, so a client that has one
behaves exactly as it did and the new lane serves only a browser holding nothing
else. meet decides a room join and does not own the workspace rows, so team
publishes them on the internal plane and meet asks, off the boundary's own
attestation rather than off headers a client can set. analytics is untouched:
its trust order already resolves a validated IAM bearer ahead of the team token.
The HS256 arm is deleted when login mints IAM-only and front/love/
analytics-collector verify IAM; getWorkspaceInfo is the one surface that still
needs a client change first.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Carries the converged console main: the Recent + Suggested strip on Models,
the topbar slimmed to navigation (environment, theme and alerts fold into
the account menu and drawer; production is the default environment), the
cross-app launcher restored, and the assistant defaulting to Enso. The pin
names the image GHCR serves — probed 200 with a negative control — and the
sha is the forge lineage's, which is the lineage that builds.
The framework boundary now exports a server span and a request record per
request to the o11y ears this binary already runs (planesink: 4317 spans,
4318 logs), shaped by zip against the collector's own receivers — no SDK,
nothing any plugin inherits. The address is the switch: O11Y_SPANS_ADDR and
O11Y_LOGS_ADDR turn each signal on, and O11Y_METRICS_ADDR — the name the
manifest already states — is now genuinely read (v1.24.x read a name the
manifest had stopped matching and reached the right ear by loopback
coincidence).
Every cloud release since 62590c3 — nine consecutive commits, four days — was
blocked in the `containment` job, and nothing about those commits caused it:
go: github.com/hanzoai/zen@v1.4.10: invalid version: git ls-remote ...
remote: Repository not found.
The token was valid and non-empty. hanzoai/zen simply has one collaborator where
its sibling modules — ai, commerce, orm, account — have fifteen, so the build
identity could read every private module in the graph except that one. GitHub
answers 404 for "private and not yours" and for "does not exist" alike, so the
error could not distinguish an ACL from a deletion. An ACL drifting beside the
code stopped the fleet, and the earlier green runs only hid it behind a warm
module cache.
So the build no longer asks GitHub for code that is ours. The module PATH stays
github.com/hanzoai/* — that is the package's name, not its address — while git
dials git.hanzo.ai, which is canonical anyway. go.sum is untouched and still
decides: the forge mirrors the same objects, the zip hashes to the committed h1:
line, and a forge serving different bytes would fail the build instead of
shipping them. Proved cold, with no GitHub credential present at all. GH_PAT
remains the fallback for anything the forge has not mirrored, and the four
modules it was missing (goauthorizenet, namespace, sendgrid-go, commerce) are
mirrored now.
Riding along, because it could not ship until the train moved: commerce v1.50.3
carries the wire rail reading the deployment's bank through the host's secret
plane, the payer reference on the wire memo, and the saved-card and tier guards.
Its work had landed on the lineage the history rewrite replaced, invisible to
the module path everything actually resolves; it is replayed onto the published
history, with thirdparty/square taken from the published side because fitKey()
is applied at all three Square boundaries there and carries tests.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The pin named a sha, then a digest. Both say which bytes built the bundle
but not which console RELEASE it is, so "what console is in cloud
v1.801.N" needed a second lookup, and the last six bumps read like
receipts for nothing.
The builder already publishes both: sha-<sha7>-amd64 on every main push,
and the bare semver on a cut v* tag. v8.5.37 is the tag for console
origin/main's tip.
This SUPERSEDES the digest pinned by #383 rather than reverting it: that
pin shipped the anonymous-visitor entry fix, and v8.5.37 contains that
commit (b1a76fff2a, verified ancestor) plus the toast render-loop fix --
the OAuth return stacked twelve identical cards and now raises one. It is
a strict forward move, not a swap.
Both images were pulled in-cluster with the fleet's own credentials before
this was written, because pinning an image the registry cannot serve has no
rollback path: v8.5.37 is 2383695 bytes, the superseded digest 2375128.
The discipline shifts rather than disappears. A digest cannot be re-pushed
to different bytes; a semver tag can, and :v8.4.118 was. So the rule is now
that a cut tag is never re-pointed -- cut the next patch instead.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The server halves for sync_run and tracker_upsert. Same callee pattern as
apps/git/files.go: the org is the CALLER's plane identity, an anonymous caller is
refused rather than defaulted, and an unmounted store answers 503 instead of
inventing a reply.
Each handler calls the LOCAL implementation directly — reconcileEvent and
upsertIssueSeam — never cloud.Sync or cloud.UpsertIssue. Those now fall through to
the plane when the in-process seam is nil, so a handler routing back through them
would let the process serving the op dial its own socket. The doc comments say so,
because the fall-through is what makes that the non-obvious failure mode.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The client halves landed in the previous commits; without these the calls reach a
live app and get "unknown op", which is louder than the silence but still not the
capability. These are the server halves.
projects_ownership the identity boundary asks who owns a claimed project
platform_push a landed push becomes a build
platform_release a proven image is patched onto its Service CR
Each follows the callee pattern already in the tree (apps/git/files.go,
apps/projects/key_rpc.go, apps/platform/rpc.go): the org comes from the CALLER's
plane identity, an anonymous caller is refused rather than defaulted, and an
unmounted store answers 503 instead of inventing a reply.
projects_ownership REFUSES when this process does not hold the project store,
rather than answering "nobody owns it". That distinction is the whole defect in
one line: a process that cannot look does not know, and the boundary reads
"not owned by another org" as permission to forward the claim.
exposePush reads the `mounted` global at call time instead of capturing the
service, matching the push builder registered directly above it — Shutdown sets
that global to nil and a captured pointer would keep building out of a torn-down
store.
plane.Built loses its speculative Matched field. Nothing consumes it and
buildFromPush does not return the count, so producing it would mean widening a
signature with six test call sites to compute a value no caller reads. ZAP fields
append at the end, so it can arrive the day something needs it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nothing caught this defect class, which is why it kept shipping. Two tests, one
to check the rule and one to keep it checked.
TestCrossAppSeamAbsenceIsAnError calls every cross-app seam with nothing
registered, no socket and no router — the state a process is in when the owning
app is simply not beside it — and fails any that returns nil. Before this change
OnGitPush and OnServiceRelease returned nil there; that is now a failing test
rather than a passing one.
TestAbsenceIsDistinguishable additionally requires the error to wrap ErrNoPeer,
because "this app is not deployed here" and "it is here and the call failed" need
opposite responses, and a fleet that cannot tell them apart has already shipped
that mistake in both directions.
TestEverySeamIsClassified is the part that outlives today. It parses this package
for `func Register*` and fails if one is missing from seamClassification, so a new
seam cannot be added without its author recording in writing whether it crosses a
process boundary. A seam classified crossApp must also appear in absentSeamCalls
or the test fails as well — a classification that nothing exercises is a claim,
not a guarantee. Adding a silent seam now requires deleting a test that says not
to.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The recipe handed each app 41000+index*10 and that was accidental complexity.
The question is "does this binary compose". Answering it does not require owning
a port namespace, a stride, or an index.
It also answered WRONG. A second run inside sixty seconds collided with the first
run's sockets in TIME_WAIT — which `ss -lnt` does not show, so the ports looked
free — and reported up to SIXTEEN healthy apps as DIED. A check that invents
failures gets ignored exactly as fast as one that misses them, and this check
exists because fifteen plugins reached production without one.
:0 on all four listeners deletes the bookkeeping, the stride, the TIME_WAIT
window and the concurrency cap in one move. The kernel already allocates ports
correctly; the recipe just had to stop doing it by hand.
Measured on this tree, same command, minutes apart:
before >> compose FAILED: 16 of 120 (15 of them "address already in use")
after >> compose FAILED: 1 of 120
The one is `kafka`, which fail-closes when no broker answers at
nats://127.0.0.1:4222. That is the app being honest about a missing dependency,
it reproduces identically on unmodified main, and it is not a composition fault.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.801.426 was pinned, rolled out and served traffic while the job meant to build
it sat Failed. The image had been built from OLDER source: two fixes reported as
shipped were not in the running binary, and nothing detected it, because there
was no way to ask the process what commit it was. /v1/health returned
{"status":"ok"} and nothing else, /v1/version was a 404, and no revision variable
existed in the Go source. Establishing the truth took exec-ing into the pod,
running the image's own /admin binary and reading its panic.
ONE fact, stamped ONCE, read through ONE function:
-X github.com/hanzoai/cloud.revision=<40-hex>
reported on the health payload every one of these processes already serves —
/v1/health on the product API, and /healthz, /readyz, /health on the ops listener
(CLOUD_HEALTH_LISTEN, the unauthenticated in-cluster read). No new route, no new
port, no new env var. The two surfaces built their bodies separately — one a map,
one a fixed byte string — which is exactly how a field comes to exist on one and
not the other, so both now build from healthBody and a literal can no longer miss
an addition.
MEASURED, on plugin/base linked with the flags the Dockerfile passes:
:18110 /v1/health {"revision":"d25b0f5e70f79bfb04ec60b1f535f20db9a62062",...}
:18112 /healthz {"revision":"d25b0f5e70f79bfb04ec60b1f535f20db9a62062",...}
:18112 /readyz {"revision":"d25b0f5e70f79bfb04ec60b1f535f20db9a62062",...}
and, linked with no -X at all, {"revision":"unknown"} on all three.
UNKNOWN IS A VALUE, NOT A BLANK. Only a full 40-hex lowercase object name is
reported; an unexpanded ${REVISION}, a branch name, "dev", a short sha and the
empty string all read "unknown" — measured on a real binary stamped `main`, which
serves "unknown". A value that is NEARLY a commit is worse than none, because
someone acts on it. That rule is cloud.IsCommit, and it is the rule the BUILDER
already applied before passing build-arg:REVISION, so apps/platform's private
isCommitSHA copy is deleted and calls it: the two ends of that wire cannot drift
into disagreeing about what they hand each other.
THE STAMP NOW REACHES THE STAGE THAT SHIPS. `ARG REVISION` existed already — in
the FINAL stage, feeding the OCI label, invisible to `go build`, because an ARG
is per-stage. The wire was connected at one end. Worse, the one -X that did exist
reaches nothing: cmd/cloud does not link the root package, so
`-X …cloud.Version=` on /cloud has always been dropped — measured, the flag is in
that binary's `go version -m` record and the value is nowhere in its bytes. The
PLUGINS serve /v1/health and they carried no -X whatsoever, so stamping only the
entrypoint would have left the answering process mute. Both facts now go into one
GO_LDFLAGS used by every binary in the image.
`-X` on a symbol that does not resolve is SILENTLY DROPPED, and under the rule
above a dropped stamp reads as the legitimate "unknown" — invisible, exactly like
the image revision LABEL that has read `unknown` in this fleet with nobody
noticing. Two things close it: the image greps its own linked binaries for the sha
(`go version -m` is not a witness — it echoes the flag that was REQUESTED, present
even when the symbol was never set), and version_test.go LINKS a real binary and
asks the process. The first draft of that test read the child's EXIT CODE, and a
deliberately mis-named symbol sailed through it green, because a test that skips
itself exits 0 like one that passes; it reads the child's OUTPUT now, and the
mutation fails it.
The Dockerfile's ARG sits as late as it can, below `COPY . .` — everything from
there down is already re-keyed by any source change, so a per-commit value costs
nothing, while the same value in scope above would re-key `go mod download` and
turn every build into a full one.
`make` builds report "unknown" on a dirty tree, correctly: REVISION comes from
`git describe --always --abbrev=40 --match='' --dirty`, and `-dirty` is not a
40-hex name, so an uncommitted tree cannot name a commit whose source is not what
was built — the same lie, locally.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three defects shipped today because the only thing that could see them was a
running binary and nothing ran one. Each is now the smallest program that
reproduces it.
scope_group_test.go — the group, which broke twice in one day, in opposite
directions:
- TestGroupWithLaterUseComposes. `g := app.Group(p); g.Use(mw)` while the
routes register through ZipApp on the ROOT. Group returning the raw router
hangs middleware on a node whose subtree is necessarily empty; zip refuses
that at boot, which is what crash-looped fifteen plugins. Asserted with
zip.App.Build — Listen minus the sockets, verdict returned rather than
thrown. RED against the raw-router Group with the exact production text.
- TestGroupPrefixesWhatRegistersThroughIt. The opposite kind: a program that
composes perfectly and answers somewhere else. A child Group must prefix
what registers through it down BOTH paths — the route methods and OpScope,
because `zip.Get(app.Group("/v1"), "/bots", h)` is a real idiom here — and
must leave an absolute path at the subsystem root alone. Asserted on the
composed route table, since no compose check can see a route that merely
MOVED. RED separately against each half of the fix.
- TestGroupUseOutsideThePrefixesFailsTheMount / ...IsAllowed. Confinement
through the door Group opened, and its limit: middleware at a prefix the
subsystem does not own installs nothing and fails the mount, while a BARE
group there is ordinary — a prefix is just a path.
plugin_surface_test.go — the two drifts, DERIVED from the specs the mains
declare (go/ast) and the routes the committed projections hold, so a subsystem
added tomorrow is checked tomorrow:
- TestHealthOwnershipMatchesWhatIsRegistered. OwnsHealth is a claim with two
halves. Claimed falsely, one address is declared twice and zip refuses the
program — reverting plugin/authz/main.go reproduces the crash verbatim:
`GET /v1/authz/health: declared by "authz" at serve.go:324 and by "authz" at
serve/mount.go:31`. Claimed while owning nothing, the address silently 404s.
The second half asks the manifest which health address is this app's rather
than assuming /v1/<name>/health — plan answers /v1/plans, storage /v1/s3.
- TestDeclaredPrefixesCoverTheSurface. A grant a main WRITES must cover the
surface it serves. One-directional on purpose: containment, never equality,
or deploy's 14-leaf row makes its own /v1/deploy bridge an escape. It checks
the half a document can answer; middleware has no address, so the other half
is `make compose`, which runs the binaries.
plugin/bot/main.go — bot declared OwnsHealth: true and registers no health
route anywhere, so the field's only effect was to suppress serve.go's generic
route and leave GET /v1/bot/health answering 404. Measured before: 404. After:
{"service":"bot","status":"ok"}. Nothing changes in the fleet, where /v1/bot/health
routes to `runtime`; this is the standalone binary's own liveness answer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The awk class was [a-zA-Z_-]+, which excludes digits, so `make help` silently
omitted e2e and e2e-ui — two targets that have existed all along and that nobody
browsing help could discover. One character.
Found by running `make help` in console and reading the output against the file,
which is the only way this shows up: the recipe succeeds either way.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The timeout is the cost and it is paid per app, so one-at-a-time made this most of
an hour for 120 apps. A check that takes an hour gets skipped, and a check that
gets skipped is how fifteen crash-looping plugins reached production in the first
place — the mechanism has to be cheap enough that it is actually used.
Each app already had its own data dir and its own port block, so nothing contends;
xargs -P just stops them queueing. Failures go to files rather than racing onto
stdout, and the summary names how many of how many failed. Measured: 120 apps in
minutes rather than ~50.
First full run on this tree: `>> compose: 120 apps boot`.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Aliases, not recipes: dev -> run, lint -> vet. Both targets already existed
and already did the right thing; only the names the rest of the fleet uses
were absent, so there is still exactly one recipe behind each.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip refuses to compose a program whose middleware could never run, and it
refuses at BOOT. So on v1.801.425/.426 fifteen plugins compiled, linked, vetted
and passed unit tests, then crash-looped in production. Nothing in this Makefile
could have caught that: the only thing that finds a compose panic is running the
binary.
SURVIVAL is the signal, and it is the only honest one. A compose panic is fatal,
so a process still alive when the timeout kills it (rc 124) composed. Grepping
the log for a success line does NOT work — `"message":"zip new"` is printed
BEFORE composition, and reading it as a pass is exactly how a broken build got
reported as shipped twice in one day.
Each app gets a writable data dir and its OWN four ports, because without them it
dies on `mkdir /var/lib/cloud/orgs` or on binding :8080/:9653/:9090/:8081 long
before it reaches the router — and an early death looks like silence, which reads
as a pass. That mistake is why a "16 binaries, 0 panics" check was worthless: the
binaries had exited before composing.
It reuses `apps`, so there is one way to build an app binary and no second
mechanism. `clean` takes the scratch dir with it.
Proven both directions on this commit:
GREEN make compose APPS="admin prefs" -> >> compose: 2 apps boot
RED inject `ZipApp(app).Group("/v1/prefs/zzz").Use(Bridge())` into prefs
-> PANIC prefs
zip: the group "/v1/prefs/zzz" declares middleware at
prefs/prefs.go:147 and no routes anywhere beneath it
-> >> compose FAILED, make exits 1
(The injection is needed because every real call site is already fixed — deploy's
was corrected by bd2c284e, so it can no longer reproduce the fault.)
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
projectIsForeign refused an X-Project-Id registered to ANOTHER org. It consulted
a registry list that `projects` fills at its Mount, and documented the empty case
as safe: "with no registry mounted, nothing is ever foreign, so the guard is a
no-op passthrough (an unmounted registry owns nothing)".
No registry is EVER mounted where this runs. The boundary is edge middleware in
every process; the registry belongs to projects, which runs as its own plugin. So
the list was empty everywhere, the guard concluded "not foreign" for every value,
and the cross-org project impersonation check was off fleet-wide. Measured with
projects absent: projectIsForeign(acme, "someone-elses-project") = false. Its own
tests passed because they exercised only the co-resident case.
Absence of an ANSWER may never read as permission. The registry is now ASKED over
the plane, and the outcomes stay distinct rather than collapsing into one bool:
mine keep provably the caller's own
other refuse a cross-org claim
neither, no error keep an unregistered within-org label, still free-form
ErrNoPeer keep no projects app exists anywhere in this fleet, so
no identifier is registered and none CAN be
foreign — the one case the old passthrough had
right, now proven instead of assumed
any other error REFUSE fail closed, as before
ProjectOwnership is the exported total form: co-resident registries answer
directly, otherwise projects is asked. It cannot decline to look — the only way
it yields no answer is by returning an error.
Cached per (org, project) on a detached context with a short TTL, the same shape
as the scope-rule read in middleware_ratelimit.go: this runs on every request
that asserts a project, and a client disconnect must not poison the cache.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
OnGitPush and OnServiceRelease returned a NIL ERROR when nothing was registered.
Not a wrong value — the absence of one. The push lands on git's embedded server
and the builder belongs to platform; the proving build and the Service CR control
plane are likewise different apps. Different apps are different processes, so
"nothing registered" was the state of every caller that ever ran, and every push
and every release reported success having done nothing. Measured, with the owner
absent: both returned nil.
A test asserted this. TestOnServiceReleaseNoop proved "the dispatch seam is a safe
no-op" and passed for as long as it existed; the no-op was never safe. It is
replaced by its opposite, plus a case proving the co-resident leg still wins so a
local releaser never pays a socket round trip to reach itself.
ServiceReleaserRegistered() is DELETED. A bool cannot carry "the app is
elsewhere": it read false both for a fleet with no control plane and for the
ordinary split fleet where platform is the next process over. apps/deploy turned
that false into a 503 that told operators the release plane did not exist while it
was up and reachable — a refusal whose stated cause was not the cause. Presence is
now answered by making the call, which is the only moment it is knowable at all,
since the router may start a lazy app on demand.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Four seams answered "not registered" for a capability running in the process next
door. Every plugin main mounts exactly ONE app and cmd/cloud mounts none, so the
registrant and the caller are never the same process: each seam was nil on
precisely the path that had work to do.
GitRepoStatuses the console repo list draws in integrations; the repos are
git's, so every repo rendered as never-imported
EnsureGitMirror the sync engine declares the target; git owns the repo that
pushes it, so configuring a mirror reconciled inbound forever
and pushed nothing back
Sync webhooks land on integrations and pushes on git; the engine is
the sync app, so no trigger could ever reach it
UpsertIssue integrations holds the GitHub App; the tracker owns the store,
so every mirrored issue and backfill row was refused
Each now takes the leg ImportGitRepo already took: the in-process call when the
owner is co-resident, the plane when it is not. No new mechanism — Ask is the one
mechanism and these are four more callers of it.
The Err*Unavailable sentinels stay for the case they always described honestly: a
registration this process was supposed to have and does not.
A map cannot cross ZAP, so the repo statuses travel as a slice whose rows carry
their own name.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/console d761fbc. Clicking "Sign in" on cloud.hanzo.ai appeared to do
nothing: console.hanzo.ai/ served a SECOND copy of the Hanzo Cloud marketing page
wearing the byte-identical @hanzogui/shell header, so the click landed on a page
indistinguishable from the one it left and read as a re-render. Reaching hanzo.id
took three clicks, two of them through pages that only asked "did you mean it?".
The console is the application; the marketing face is cloud.hanzo.ai. `/` is no
longer a special surface — an anonymous visitor STARTS the authorize hop.
Pinned by digest, not the usual sha-<sha7>-amd64 tag: that build published to
:latest, which is the moving target this file already warns about. The digest
names these bytes and no other.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The billing gate died of a peer call that was an HTTP URL. COMMERCE_URL defaulted
to the public api.hanzo.ai edge — which is THIS binary — so the /v1/billing/*
forwarder re-entered itself, and apps/commerce/transport still carries the scar
tissue: a whole app republished as an http.Handler, every edge middleware re-run
per peer read, and a goroutine-keyed depth counter (maxDepth = 8) to stop the
recursion it cannot otherwise prevent. A call by name cannot express that mistake,
because there is no address to point at the wrong thing.
An app already declares everything a caller needs:
zip.Post[plane.BalanceIn, plane.Balance](cloud.Plane(), "/finance/balance",
planeBalance, zip.WithOperationID(plane.FinanceBalance))
— the app, the op, the request type and the response type, in one expression.
Cloud already projects that registry as OpenAPI, a CLI, an MCP tool list and a
routing declaration. A typed Go client for a peer call is ONE MORE PROJECTION of
it, which is why it is generated here rather than hand-written once per caller.
commerce.FinanceBalance(ctx, &plane.BalanceIn{Currency: "usd"})
plane/gen emits one package per peer (14 apps, 28 ops) holding ONLY request and
response types and call stubs. What it buys is a check no care buys today: a
hand-written peer call is four independent facts that must agree at RUN time, and
nothing stops pairing commerce's op with iam's name, or BalanceIn with Txns. The
wrapper fixes all four to each other where they are declared.
THE CLIENT HALF MOVED TO THE LEAF, and that is what makes any of this possible.
package cloud is itself a caller — the edge rate-limiter reads finance_scope_rules
— so a client that imported cloud could never be imported BY cloud, and the one
call that most needed to stop being a URL is the one the mechanism could not have
expressed. Ask and everything under it now live in package plane; cloud keeps the
server half (Plane, ServePlane) because binding a socket reports itself to o11y.
cloud.Ask stays as a forwarder, so the 44 existing call sites do not move and
there is still exactly one implementation.
It does not drag the peer's tree: plane/commerce is 355 packages against
apps/commerce's 1231, and imports zero apps/ packages — one more than the leaf it
needs. An importable client that linked the implementation would have rebuilt the
problem with extra steps.
Generated FROM SOURCE, judged BY THE RUNNING REGISTRY. zipdoc already reads these
same call sites; reading source buys hermeticity a mount cannot (no store opened,
no boot order, no app that must come up before it can be described). zip's rule —
project from the live router, never the AST — is about a host discovering a plugin
it does not build, and it still binds: plane_registry_test.go mounts commerce and
asserts the generated surface IS the live plane registry, so the generator never
gets to be quietly wrong. Reading the AST's index expression alone had already
been quietly wrong once — treasury spells its registration with inferred type
arguments, so its only op was dropped; types.Info.Instances sees both spellings.
Proven against the real thing, not a fake. plane_client_test.go mounts commerce as
a plugin process does, binds its plane socket as Serve does, and calls the
generated function: commerce answers amount="0" currency="USD" over the socket. A
cold peer with no router answers ErrNoPeer naming the app — a named absence, never
a timeout a caller would have to guess at.
Three root-package call sites converted, including both money ops — the prepaid
gate and the meter now reach commerce as commerce.FinanceAuthorize and
commerce.FinanceRecord. Those are the imports that were structurally impossible
before, so they are the proof the direction is real.
Full suite: 133 failing test names / 26 packages, byte-identical to the same
measurement on origin/main. Regression set EMPTY. go vet ./... exit 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/graph was never a graph database. Its package doc says so: "chain data:
your block indexers and how far each has caught up, plus the on-chain price
feeds." It proxies luxfi/indexer (explorer REST) and luxfi/graph (GraphQL) --
"graph" here meant GraphQL, not a property graph.
Both upstreams already mount under /v1/explorer (client.go: graphd default
prefix), so the name follows the contract the app already speaks rather than
inventing one.
Wire unchanged: the app keeps /v1/indexers and /v1/oracles, keeps its frozen
mount position, and the woven document moves only x-app and the tag prose.
References to luxfi/graph -- GRAPH_URL, graphQLPath, the upstream log key --
stay, because those name the upstream, not this app.
/v1/graph is now free for the embedded per-org graph layer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The self-hosted-git allowance existed, was correct in intent, and could never
match. `selfGitHost` was `deps.Domain` verbatim — "api.hanzo.ai" — while
hostAllowed grants that host or a SUBDOMAIN of it. The forge is "git.hanzo.ai":
a sibling. So every native build was refused with
repo.url host "git.hanzo.ai" is not an allowed git provider
and the estate fell back to building from GitHub, which is exactly what broke
when hanzoai/cloud moved to hanzo-inc/cloud and the build credential lost access.
The code's own comment already said APEX; only the assignment disagreed.
selfGitHost is now the registrable apex ("hanzo.ai"), so every sibling the
deployment owns — git., ci., cd. — is a trusted build source by construction,
for hanzo.ai, lux.network, zoo.network and any white-label domain, with no
per-brand list to maintain. publicsuffix rather than "last two labels" so a
multi-label suffix (co.uk) yields the registrable domain and not the suffix
itself, which would trust every domain under it.
Proven both directions: TestSelfForgeIsAllowedFromTheApex admits
git/ci/cd.hanzo.ai and still REFUSES git.evil.com;
TestOldDomainVerbatimRefusedTheForge pins the old behaviour as the defect, so
this cannot silently regress.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/runtime was never a second product. It was the transport to @hanzo/bot —
the TS service that executes a run — plus a relay of that service's own ops
paths at /v1/bot/*. What separated it from apps/bots was a LANGUAGE boundary,
Go surface and TS executor, wearing the shape of a product boundary. Both
answer for the same thing: a bot doing your work on a real desktop.
So they merge. runtime.go becomes transport.go, ops.go becomes relay.go,
bots.Mount mounts both faces, and the manifest holds one row for the pair.
THE WIRE DOES NOT MOVE. Every path this fleet serves is the path it served
before — the whole diff to openapi.yaml is `x-app: runtime` -> `x-app: bots`
on the seven relayed operations. No CLI, SDK, MCP tool or doc regenerates to
a different address, because none of them has a different address to go to.
Two things the merge had to earn rather than assume:
- apps/coding also dispatches to that executor, so it followed the transport
from apps/runtime to apps/bots. That is not new coupling wearing a new
name: coding runs its tasks ON the bot runtime, which is what the import
now says out loud.
- the typed-or-named gate was TWO gates, one per old package, each blind to
half of what is now one surface. They are one gate over the whole product,
and mountWith mounts the relay so the gate can actually see it. Two gates
stapled together would have kept passing while measuring nothing.
/v1/bot is still shared with apps/bot, whose three deeper prefixes win on it by
specificity. That sharing is the remaining defect and it is not this commit's
to fix: apps/bot's product is connected machines, not a bot, and it is the one
that has to vacate.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip exported TWO things named Mount and they were unrelated:
zip/remote.go func Mount(prefix, addr string, decl ...Declaration) (*App, error)
112 subsystems func Mount(app cloud.Router, deps cloud.Deps) error
The second is this repo's registration contract, now enforced by the compiler
through cloud.MountFunc. The first was a leftover — one of five composition
verbs (Listen/Mount/Add/Graft/Use), the other four dissolved into Use, and this
one survived only by being the one that pointed at another process.
Renamed upstream to Proxy, which is what it BUILDS: every route it registers
runs one handler whose whole body is forward() to the given address. It is also
a noun, which a function returning a value should be. Not Remote — zip.Remote
already exists and is the CALLER's side; this is the SERVER's side, a stand-in
inside this program's own routing table.
Source-compatible here: cloud had no zip.Mount call site, only the one comment
in cmd/cloud, which now names the function that exists.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`type Store` is declared 40 times under apps/ and 35 of those are structurally
IDENTICAL — `struct { db *sql.DB }`. That number is a trap and I am not acting
on it: in Go, git.Store and bot.Store are different types because they are in
different packages, each carries its own app's schema in its own methods, and
"consolidating" a one-field handle wrapper would produce `type Store struct{
shared.DB }` — a rename that moves no code and removes no duplication. 35 of 40
share a shape; ~0 share a MEANING. Optimising that number would be pure damage.
The duplication the number was pointing at is one level down, in the
CONSTRUCTOR, and it is exact. 33 stores open themselves with byte-identical
code modulo their own name:
db, err := cek.Open(namespace.System(), "<app>", dir)
if err != nil { return nil, fmt.Errorf("open <app> store: %w", err) }
sqlpool.Single(db)
Those three lines are a PAIR that nothing paired. sqlpool.Single's own doc says
a two-statement read-modify-write (tracker's per-project issue number, agents'
MAX(seq)+1) is atomic ONLY because no second connection can interleave — so the
cap is a correctness requirement, and it was a separate call every caller had to
remember. 33 remembered.
apps/framework did not. It opens a cek database through an engine OpenDB
callback and never capped it, so its DocType store has been running with an
uncapped pool. That is the defect, and it is the one a 33-way copied prologue
exists to produce: the rule holds until someone writes the 34th store.
sqlpool.Open is the rule moved INSIDE the opener, so there is nothing left to
forget — the same argument as cloud.App being the only way to obtain an app.
36 call sites converted, framework included; the stores that open differently
(git takes a *sql.DB from the OrgStore cache, others carry extra migrations)
still do, because they are different and the point is not uniformity.
Measured, not assumed: shapes compared by AST field set, not by name. Of the
other collision families the brief named, Result (9 decls) and Config (10) have
ZERO structurally identical pairs, and Client has exactly one family of 3
(`base string; http *http.Client; token string`). Those are Go's package
qualifier working, and they were left alone.
Regression set EMPTY by name: all 15 failures in the touched packages
(TestDocTypeAndDocumentRoundTrip, TestForkCreatesProjectFromTemplate, the Redeem
family, …) fail identically on forge/main.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
All five diverged from the canonical Mount signature, and the divergence was
invisible because nothing checked it. Now MountFunc does (see the parent commit),
so these are what the compiler demanded.
agent, ai, commerce took *zip.App, so they had to be wired through the
old App field — which ALSO granted app-wide middleware.
They now take cloud.Router and reach the concrete app
through cloud.ZipApp, the named hole for it.
agent installs no middleware at all (hanzoai/agent
calls Use nowhere) and now mounts SCOPED: it had the
whole binary's gate to buy a typed-op registry.
ai and commerce genuinely do gate everything —
commerce wraps all of /v1 — and now DECLARE it as
Plugin.Global at their composition root, where a
capability belongs, instead of implying it with a
parameter type.
dataroom imported cloud as `hcloud`. The alias is cosmetic to
the compiler and fatal to every grep-based check —
which is how these five stayed invisible. One name.
treasury/anchor not a Mount, but the same disease one level down and
the cleanest Pike case in the repo: status() took
ledger.Backend — ELEVEN methods — to call Root and
nothing else. The parameter said the anchor could
accrue revenue, seed the reserve, debit a program and
rewrite the revenue-share policy. It can do none of
those. It now takes a one-method `rooted`, declared by
the consumer, which both backends satisfy without a
line of change.
RETRACTED: rollingcap's `func Mount(_ cloud.Router, _ cloud.Deps) error` was
reported to me as the fifth violation, possibly dead. It is neither. It installs
the rolling AI-spend cap reader — live, and load-bearing — and `_` is CORRECT Go
for parameters it genuinely does not read. It already IS a MountFunc; the
compiler says so. Naming those parameters to satisfy a text pattern would make
the code worse. Left exactly as it is.
Regression set EMPTY by name (148 failing tests before, the same 148 after).
apps/commerce's TestBalanceCents and TestInProcessClient fail identically on the
parent commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two names in this file each meant two things, and both cost the fleet something
real.
1. Plugin had TWO mount fields with TWO signatures
Mount MountFunc // func(Router, Deps) error
App func(*zip.App, Deps) error // "for a subsystem that gates everything"
App braided a POLICY question (may this subsystem install middleware over the
whole binary) into a TYPE question (what shape is its entry point). Because the
grant carried its own signature, a subsystem could take the grant just to get
the concrete *zip.App — and three did. agent, ai and commerce each held app-wide
middleware authority they never asked for, purely because their Mount named a
concrete type. hanzoai/agent calls Use nowhere; agent's grant bought it nothing
and risked everything after it in the mount order.
Nothing reported this. The field's own doc claimed "apps.TestWireFrozen fails on
a new one"; no such test exists anywhere in this repo — grep it. A contract
stated in a comment and violated five times means nothing is checking it.
Unbraided: App becomes `Global bool`, and every subsystem — scoped or global —
goes through spec.Mount. The grant now decides only WHICH Router arrives (the
bare app, or a scope bound to declared prefixes). One shape, so MountFunc is the
whole enforcement and it is the compiler: at 123 composition roots a divergent
Mount cannot be assigned and cannot link. It found a sixth violator I had missed
by grep on the first build — cloud.MountMetrics, whose doc says "adapts
hanzoai/metrics into a MountFunc" while its signature took *zip.App. It is one
now, and mounts SCOPED (hanzoai/metrics installs no middleware either).
The concrete app stays reachable through ZipApp, the named hole that already
existed for exactly this and reports nil rather than pretending.
2. Router restated zip.Router instead of embedding it
Ten method lines were COPIED here. Copying an interface makes cloud a second
place zip's routing surface is defined, and the two agree only while someone
keeps them agreeing. When zip v1.23 widened one signature (Use took Component,
not Handler), every implementor that had spelled the methods out had to move in
lockstep — which is what stalled v1.19+ adoption across the fleet.
Embedded, a zip routing change costs this file zero edits, and Fiber() +
Plugins() are visibly what cloud ADDS rather than being buried among ten lines
cloud merely echoes. It carries zip.OpTarget along, which is a gain, not a
widening: scope, *zip.App and commerce's mintRouter all already have OpScope,
and a Router that IS an OpTarget is one zip.Get[In, Out] accepts directly.
Regression set EMPTY, compared BY NAME against a pristine worktree at the same
commit: 148 failing tests across 29 packages before, the identical 148 after
(comm -13 on the sorted name sets is empty in both directions).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerceTierToLadder allow-listed only `starter`->trial and `pro`/`enterprise`
->paid, and folded EVERYTHING ELSE to free. The plans commerce actually sells
are go ($9), dev ($19), pro, max, team and business — so a subscriber on
go/dev/max/team/business was scored free, then refused every SKU carrying a
trial or paid floor AND throttled by the free-tier flash cap on top. We took
their money and shut the door.
v1.832.29 inverts the default: a plan is PAID unless it is explicitly free
("", "free", or a `*-free` suffix), with starter/trial the one middle rung.
That direction is safe because A TIER IS NOT A PAYMENT. filter_balance.go has
no exemptions and fails closed — "nothing runs on credit it has not been funded
for" — so an unrecognized plan reaching `paid` still cannot spend a cent it has
not been funded. The OLD default was the dangerous one: it silently cost
revenue every time anyone added a plan slug. Now, add a plan and it works.
Both tests had encoded the defect as intent ("developer"->free,
"mystery"->free) and were corrected to the real plan list. 22 packages ok,
0 FAIL at the module.
Also carries the zen seed fix (v1.832.24) and the object TestMain fix
(v1.832.27), which took that package from ZERO tests executing to 153 PASS —
TestMain called os.Exit(0) before m.Run(). That immediately surfaced a
crawl-storage default still naming the retired MinIO Service, a correction that
had been made twice and could never take effect.
Verified: `go build ./...` clean at this pin.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A signed-in customer got 401 listing their own cards, and the checkout's
prefill failed on every load: cloud's billing app forwarded /v1/billing/methods
to commerce over HTTP, and that proxy is unconfigured here
(CLOUD_COMMERCE_HTTP_URL is unset), so the customer address for saved cards has
never worked on this deployment.
An internal HTTP hop to a service compiled into the same binary is the wrong
shape whatever its config, so the three verbs move to the commerce app and are
served in-process on the same pinned-subject chain as their portal twins — the
gate that keeps a caller inside its own account whatever it sends. The prefix
moves with them, because the router must deliver where the handler lives.
Also drops /v1/commerce/deposits and /v1/commerce/webhooks from the manifest
and the published document: the broker-dealer proxy behind them is deleted, and
a manifest that claims an address nothing serves is how a path silently routes
to the wrong app.
The routing oracle is what found all of it — both halves, in both directions.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Per-org KMS hydration runs only under KMS_ENABLED, unset here, so the rail
answered "not configured" with every field stored correctly. v1.50.1 falls
back to deployment env, which universe now supplies from commerce-secrets.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`p.Status = "error"` on a non-live completion was UNCONDITIONAL, so "this project
is broken" was reachable from any deployment row the org could name — including
one that had already been superseded.
Found by triggering it on hanzo-ai, not by reading it. v5 was live with all 8401
objects correct in the bucket; completing v3 — an older probe deployment left
queued during the Sites-plane migration — as `error` flipped the PROJECT to
"error", and the sites edge stopped serving. Nothing was wrong with the site. The
bytes never moved. A status field on a superseded row took the host down, and
re-completing v5 as live brought it straight back.
The ordinary production shape is the same bug with worse timing: a rebuild that
fails while the PREVIOUS build is still live. The old content is still being
served and the site is up, yet the project gets marked broken — and the "report a
failed build" step that every CI workflow carries (so a dead build cannot leave a
deployment queued forever) is exactly the thing that would send it.
The rule is now named rather than inlined: failureOwnsProject(currentDeploy,
deployID). Error propagates to the project only when the deployment IS the one the
project points at, or when it points at nothing — a first deploy that fails leaves
a project that has genuinely never served, and that one should read as error. The
deployment row is still error in every case and LifecycleDeployFailed still fires,
so the failure stays visible where it belongs: on the deployment, not on the health
of a site that is up.
The `live` branch already recorded p.CurrentDeploy, so the information needed to
make this distinction was there the whole time and simply was not consulted.
TestFailureOwnsProject pins all four cases. Mutation-checked: restoring the old
`return true` fails exactly the two that matter — the superseded deployment and
the failed rebuild — so the test cannot pass against the behaviour it exists to
prevent.
(TestDeleteStays204WithNoBody fails on darwin both with and without this change:
the SQLCipher codec refuses to decrypt without tmpfs, and macOS has no /dev/shm.
Pre-existing and unrelated.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A plan-refused caller was told to request limited-preview access — a
door that cannot open when the plan is the blocker. Each family gate
now speaks its own sentence: upgrade for the subscription floor, paid
capacity for the funding floor, request-access only where a grant is
truly what is missing.
The edge resolved an extension-less path to two candidates — the exact key and
`<rel>/index.html` — and Next.js `output: export` writes neither for a normal
route. Without `trailingSlash` it emits the FLAT file `pricing.html`, so the
homepage served and every other route 404'd.
Measured on the hanzo.ai export (759 pages, 8402 objects) published to
hanzo-ai.hanzo.app:
/ 200 /pricing 404 (/pricing.html 200)
/zen 404 /zen.html 200
/zen/models 404 /zen/models.html 200
That is the shape of a site that reports `status: live`, serves its homepage,
and is unusable — the failure is invisible from the deploy and from the root URL.
`<rel>.html` is added as a middle candidate. Both spellings are legitimate and
both are now tried: Hugo, Jekyll, Vite MPA and `trailingSlash: true` emit the
directory-index form, Next's default emits the flat form. The alternative was
setting `trailingSlash: true` in every repo, which pushes a server limitation
onto each site and rewrites every canonical URL to work around it — this is one
extra HEAD miss on the paths using the other convention, in one place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The embedded family catalog no longer gates any enso SKU, so discovery
stops advertising a waitlist and ai serves every caller without a grant.
The app builder's default model refused everyone outside the launch org;
now it answers.
The envelope's RAM-backed guard now recognises a native macOS tmpfs
(sudo mount_tmpfs <dir>, then HANZO_SQLITE_RAMFS_DIR=<dir>), so the
cek-backed suites can run on a Mac instead of failing on every commit.
Fail-closed is unchanged: anything statfs cannot verify as tmpfs is
still refused.
Record.NeedCommit carried a xorm-style `db:"index"`, and dbx reads that tag
as the column NAME — so the column was called `index` while
ScanNeedCommitRecords queries `need_commit`. Live in this pod: 'no such
column: need_commit' every five minutes, and the record-chain commit task had
never committed a record.
Also carries a guard for the class, placed in routers rather than object
because object's TestMain os.Exit(0)s that package without a seeded database —
a test there reports ok without ever running.
hanzo_service_up is now recorded a second time, through Metrics() — the
registry the framework exports natively — under the exact series name and
label the meter pipeline has always published. Two producers, one probe, one
truth: every rule and reader sees an uninterrupted series whichever road the
sample travelled, which is what lets the old road be retired without a gap
once every reader is confirmed on this one.
The verdict is recorded where it is decided: the probe client's transport,
which already tells each target's reason on change. A nil registry skips
recording — a test that mounts probes without one measures the probing, not
the export.
zip moves to v1.24.6 for Metrics(), which also brings the native span and log
export and the convention that finds a collector with no configuration.
hanzoai/ai is a library this binary links, not a service of its own, so the
crawl4ai leg it dropped stays live in production until this pin moves. v1.832.25
points /v1/crawl at apps/crawl.Fetch — the same guarded dialer the answer
engine's read stage uses, refusing non-public addresses on every hop including
redirects — instead of dialling crawl.hanzo.svc.cluster.local:11235, a name that
does not resolve and returned success:false for every request.
The survey shipped with the loop steerable by the pages it read. Its decision
prompt carries titles and URLs crawled from hosts we do not control, so the move
that comes back is partly authored by them — and nothing checked it. Three
consequences, all reachable by getting one page into the ranked set:
a `read` URL was fetched whether or not the survey had ever gathered it, so a
page could point the cluster's egress at any address and carry the user's
question there in a query string, then have the answer filed in the tenant's
corpus;
a round's `queries` list was unbounded — mode.maxQueries bounded round zero and
nothing else — so one move could issue hundreds of serial meta-searches from a
shared egress that is already bot-challenged;
page text was spliced into the numbered source block unfenced, so a body
containing `[9] Title\nURL\nbody` became a source indistinguishable from a real
one, and nothing validated that the URLs the answer cited were sources at all.
CONTAINMENT, NOT INSTRUCTION. A prompt is advice to a model. These are properties
of the text:
`read` is intersected with the pool the survey itself gathered, and `queries`
and `read` are clipped where the untrusted value crosses into the loop;
ground.go fences each source with a per-request nonce the page cannot know,
because it was written before the request existed;
every markdown link in the answer is checked against the gathered set — on the
stream and in `done`, through the same function — so a citation always points at
a page THIS request fetched. An ungrounded link keeps its text and loses its
target: the sentence still reads.
Two bounds that could not bind now do. tokenCeiling measured the survey's own
subtotal while the plan and the synthesis — the expensive calls — sat outside it,
so research's 400k ceiling was ~50x the reachable total; it now takes the
request's running total. And plan, survey and synthesis shared one 300s clock
with no reservation, so a survey that spent it handed a full corpus to a
completion that could not start and the caller got "the model is unavailable"
after five minutes; the gather now gets 70% and synthesis keeps the rest.
Also fixed, and each one a bug on its own:
unread MARKED every URL a move named while read FETCHED only the first six, so
a move naming ten pages blacklisted four without ever fetching one — and the
round then tripped saturation and ended the survey early, losing evidence on
exactly the runs working hardest. The cap now comes before the marking.
Source.Text carries the fetched page and Source.Snippet stays the search
summary. Reading no longer overwrites what the wire shows, so a research answer
stops re-sending up to a megabyte of duplicate page text across twelve
snapshots, and one frame per round is now the whole story.
A round's queries run concurrently. Serially, at 12s each, a six-query round
spent most of the answer's wall clock waiting.
An unreadable move gets ONE stricter reprompt. A model that opens with "Sure!
Let me look at the JVM next" collapsed research into a single pass, and nothing
downstream could tell that from a model that judged the evidence complete.
A client that hung up ends the run: one research answer costs five minutes,
three dozen fetches and up to eight completions, and a closed tab bought all of
it. The plan's topic headings now reach the client as a planning detail, which
is what makes a three-minute wait legible. Crawl and completion failures are
logged — a degradation nobody can see is a degradation nobody can fix.
The package doc claimed METERED ONCE. It is not true: build.go hands this engine
a metered AI plane, so every internal completion also debits the payer per token
alongside the flat fee. Which layer should price /v1/ask is an open decision,
recorded rather than claimed away.
The SearchEvent union is unchanged — same variants, same keys, same order.
v1.832.24 stops seeding an admin/zen provider row that pointed at do-ai's base
https://inference.do-ai.run/v1. familyProvider reads that row as an operator
override of ZEN_URL, and the family paths append their own /v1, so every zen
catalog refresh hit .../v1/v1/models and 404'd — no zen SKU was listed or
served on api.hanzo.ai, once a minute, silently. Sibling enso was never seeded
and never broke.
The seed also self-heals ProviderUrl on every boot and re-creates a deleted
row, so this was not fixable in the database; v1.832.24 drops the seed entry
AND prunes the existing row, scoped to the exact stale shape.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The rail's "could not generate a deposit address" was a 502, and Cloudflare
replaces an origin 502 with its own HTML interstitial — so the customer read
"request failed" while a clear JSON message sat at the origin, unreachable.
v1.49.68 answers 503 (which passes through, as the wire rail's own 503
already proves) with a sentence naming the rail, saying it is temporary, and
pointing at the alternatives.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The plane had no proof that event.log and event.span were reachable at all — the
reads had been repointed onto them while nothing was shown to fill them.
TestLiveEverySignalLandsInItsOwnTable drives one fact of each signal through the
SAME door and asserts each landed in exactly one table, with the columns that
table's ORDER BY is built on: service on the log, trace_id on the span, and a
shared trace across both, which is the correlation the identical envelope buys.
Two things it found:
The analytics manifest row had lost /v1/event. The row is the fleet router now,
so a missing prefix is not a tidier list — it is every beacon in the fleet
falling through to a bare /v1 catch-all that does not serve it. Restored with the
ingest doors named, and the comment says why they are load-bearing.
The round-trip readback waited for NINE groups and then asserted TEN rows. The
poll returned the moment the ninth distinct name landed and the tenth row was
simply not there yet — reproducible under load, invisible when the drain was
fast. Waiting for a smaller number than the assertion checks is a race, not a
shortcut; it now waits for what it asserts. The signal test had the same shape
and now awaits each table before counting it, because landing is a consumer and
a count taken the instant a sibling landed asserts the synchronous insert this
design removed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An event is the FACT; a message is the container it travels in. The schema now
says so: one database named for what it holds, one table per KIND of fact, and
the subject a fact travels on carries the same name as the table it lands in.
event.event what someone did (kind = track | page | identify | group)
event.error what broke, grouped for issue lists
event.log what a service said
event.span what a service did, and how long
event.metric a sample; event.series the dimension it resolves through
The first fifteen columns are identical across event/error/log/span, so a
cross-kind read is a UNION ALL and not a translation layer. org leads every sort
key, so a single-tenant read is a primary-key seek. This retires three schemes at
once: a brand (hanzo.*), a numeronym (o11y_*), and two product names used as a
storage layer (insights, analytics).
Ingest is a bus, not a handler. The door accepts, normalizes, authorizes and
enriches, then publishes; fan-out happens on JetStream so adding a consumer never
touches the ingest path. Limits retention, one durable pull consumer per sink,
commit-the-sink-then-ack, idempotent on event.id because an unacked message is
redelivered. The subject carries taxonomy, never a tenant.
Two writes that could not land:
SETTINGS rendered AFTER VALUES in every warehouse INSERT. The Values input format
reads everything after VALUES as data, so a trailing SETTINGS is not a setting —
it is a row the parser cannot read, and the store answered
CANNOT_PARSE_INPUT_ASSERTION_FAILED for all four tables while the door still
returned 200 and the shape test still passed. Silent total loss behind a green
receipt. Rendered before VALUES now, with a test that pins the ordering rather
than the prefix, because a prefix and a placeholder count are both satisfied by a
statement the store refuses.
The Guide's funnel and marketing's cohorts still named the old table. They would
have gone on reading a table that no longer takes writes the moment the writer
moved — the same failure that left the observability reads pointed at databases
that had been dropped. Both now name apps/datastore's constant, so the plane has
one home for its names and a rename cannot leave a reader behind.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The unified telemetry plane had a schema and no writer. event.log and event.span
have sat at 0 rows since the databases were created, because nothing in any
shipped binary knew how to fill them — these three files existed only as
untracked working-tree state and had never been committed, built, or run.
fact.go the fact. ONE envelope (org, time, id, name, kind, product,
session/distinct/anonymous/person, url, path, attributes, el)
plus what each signal adds. kind is a COLUMN, not a table:
event.event carries track|page|identify|group, so a caller verb
never becomes a schema decision.
bus.go publish to the EVENT stream, subject per signal (event.<kind>).
LimitsPolicy, not WorkQueue — warehouse, alerts and replay are
independent consumers and each needs its own copy.
warehouse.go durable pull consumer, one per table, ACK only AFTER the insert
commits, idempotent on event.id, so a redelivery cannot double
count and a crash cannot silently drop.
Org rides in the signed envelope, never in the subject, so wildcard
subscriptions stay stable and tenancy stays an authorization concern.
Committed alone, out of a tree carrying 551 files of other in-flight work.
Builds and vets clean on its own package.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The prior comment said provisioning 'registered nothing there'. That is wrong:
provisioning.go:431-434 registers POST /v1/s3, GET /v1/s3, and GET+DELETE
/v1/s3/:name as the s3 slice of its 7-kind loop. Adversarial verification of the
doc sweep caught it.
Behaviour is unchanged either way — storage also claims bare /v1/s3 and won the
duplicate-pattern merge, so those four were already unreachable when declared.
But the comment asserted a fact about the code that is not true, and the four
dead routes are a real open question (provisioning creates S3 instances, storage
performs S3 operations) rather than a non-issue.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The router registers All(prefix)+All(prefix/*) per app and MERGES identical
patterns, appending the later handler behind the earlier one. The earlier is a
proxy that never calls Next(), so a duplicated prefix does not conflict loudly —
the later app just never runs. Two shipped that way, out of 241 prefixes:
/v1/s3 provisioning behind storage — pure shadow, provisioning registered
nothing under it. Removed.
/v1 zen behind commerce. Not a dead endpoint: zen mounts
Group("/v1", z.Claim()) — spend/claim MIDDLEWARE (apps/zen/zen.go:84)
— so the metering gate on the inference path is inert. Recorded in
knownShadowed rather than flipped, because turning it on turns
metering on and that needs a billing owner, not a refactor.
manifest_test.go asserted only that mounting does not panic, which is why both
shipped. TestNoShadowedPrefix asserts ownership, fails on any NEW collision, and
fails again if a knownShadowed entry stops colliding so the list cannot rot.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Add an error/Sentry sink to the canonical event fan-out (forward.go): every
type:'error' event is handed, o11y-free, to clients/o11y, which normalizes it
through the reused errortracking engine and ingests it via the embedded
Modules.Sentry into o11y_sentry_events + the o11y_issues lifecycle, so
/v1/event errors surface on sentry.hanzo.ai alongside the errors a Sentry SDK
posts to /v1/sentry directly.
The seam is the twin of the existing destinations SetSink: analytics imports
neither subsystem, each installs its own sink at Mount, and every sink is
detached + fail-soft + additive (a projection failure never touches the
hanzo.events write or the receipt). The org UUID is derived from the tenant
slug by the same UUIDv5 the o11y read side uses (iamidentn), so an error lands
under the exact org the console resolves; each org gets one canonical Sentry
project (get-or-create, cached). Default-on, gated by CLOUD_SENTRY_LENS;
active only when the in-process o11y runtime is up (Modules.Sentry).
Reuses the ONE datastore path already open (o11y's telemetrystore for the
events plane, its sqlstore for the issue lifecycle) via Modules.Sentry.Ingest;
opens no new connection.
Tests: error fan-out detection/extraction across every wire shape (folded
*Exception, native $exception map, bare error-typed); org-UUID pinned against
the o11y formula; ErrorEvent -> normalize -> fingerprinted occurrence; project
get-or-create + cache; and the pk_ ingest-key mint->use flow end-to-end
(POST /v1/ingest/keys -> POST /v1/event accepted, no 403), so anonymous site
ingest stops 403ing.
2026-07-23 02:52:36 -07:00
1116 changed files with 120967 additions and 20497 deletions
[ -s "$work/images"]||{echo"orphans: no published versions found for ${IMAGE_PATH} — that is not a clean run, it is a read that returned nothing" >&2;exit 2;}
# ── what is tagged ───────────────────────────────────────────────────────────
# `git ls-remote` failing and a repo having no tags produce the same empty list,
# so each remote is checked for the READ succeeding, separately from what it
# returned. At least one remote must answer: a network that answered nothing
# would otherwise make every published image look untagged and turn a broken
comm -23 "$work/images""$work/tags" > "$work/untagged"
comm -23 "$work/untagged""$work/accepted" > "$work/new"
if[ -s "$work/new"];then
whileread -r v;do
echo"::error::${v} is published but no git tag names it — an image no commit can be traced to is not a release. Tag it at the commit it was built from, or record it in .hanzo/orphans.txt with the reason it cannot be."
done < "$work/new"
echo"orphans: $(wc -l < "$work/new") published image(s) with no receipt; $(wc -l < "$work/accepted") previously recorded" >&2
exit1
fi
echo"orphans: $(wc -l < "$work/images") published, $(wc -l < "$work/tags") tagged, $(wc -l < "$work/accepted") recorded — every published image has a receipt"
# TWO DIFFERENT FAILURES SHARE THIS STATUS, and treating them alike
# would turn one of them into a ten-attempt loop that ends in the
# wrong diagnosis. "Reference already exists" is the collision this
@@ -407,14 +541,14 @@ jobs:
# a name to skip past; it is a repo that has not been published, and
# the next number would fail identically.
WHY=$(jq -r '.message // empty' /tmp/claim.json)
if [ "$WHY" != "Reference already exists" ]; then
echo "::error::claiming v${CAND} was refused with: ${WHY:-unknown}. If this is 'Object does not exist', commit ${SHA} is on the forge but not on github.com — the release lane claims versions against GitHub, so the mirror must carry the commit first."
exit 1
fi
case "$WHY" in *"already exists"*) : ;; *)
echo "::error::claiming v${CAND} was refused with: ${WHY:-unknown}"
exit 1 ;;
esac
# Taken. By us, or by somebody else? The distinction is the whole
# difference between a resume and a collision, and it is one GET.
echo "v${CAND} is already claimed at our sha — this release, resumed"
CLAIMED="$CAND"; break
@@ -422,8 +556,17 @@ jobs:
echo "v${CAND} is held by ${HAVE:-another ref} — trying the next number"
continue
fi
if [ "$CODE" = "401" ] || [ "$CODE" = "403" ]; then
# Say which credential and which scope, because the symptom
# otherwise reads as "the claim is broken" and the cause is one
# secret. FORGE_TOKEN has only ever been READ with in this
# workflow — go module fetches and ls-remote — so the first
# release to write with it is the first test of its scope.
echo "::error::the forge refused the claim for v${CAND} with $CODE. FORGE_TOKEN must carry write:repository on hanzoai/cloud — it creates refs/tags/v<N> and the refs/smoked/<digest> receipt. Re-issue it with that scope and set it as the hanzoai org actions secret; nothing else here needs changing."
cat /tmp/claim.json; exit 1
fi
if [ "$CODE" != "201" ]; then
echo "::error::claiming v${CAND} returned $CODE (expected 201 or 422) — refusing to build a version this run cannot prove it owns"
echo "::error::claiming v${CAND} returned $CODE (expected 201 created or 409 taken) — refusing to build a version this run cannot prove it owns"
cat /tmp/claim.json; exit 1
fi
echo "claimed v${CAND} at ${SHA}"
@@ -434,18 +577,31 @@ jobs:
# WE OWN THE NAME — so anything already published under it is either
# our own earlier attempt or a lane that had no right to it, and those
# two need opposite handling. `resumed` is therefore derived from the
# two need opposite handling. `built` is therefore derived from the
# IMAGE, not from the tag: since the claim now precedes the build, a
# tag at our sha no longer implies bytes exist.
RESUMED=0
#
# IT ANSWERS ONE QUESTION AND IT USED TO ANSWER TWO. This flag was
# called `resumed`, and the smoke step was skipped on it — so a run that
# PUSHED and then FAILED smoke came back, found its own bytes, declared
# itself resumed, and skipped both the build and the proof. rollout then
# pinned an image nothing had ever booted. The registry is the authority
# on whether bytes exist and it is the authority on nothing else: AN
# ARTIFACT EXISTING PROVES IT WAS BUILT, NEVER THAT IT WAS TESTED.
#
# So the second question is asked separately, of a separate authority,
# further down — and it is asked about the DIGEST, because a receipt
# that named only a version would prove a version was smoked and say
# nothing about which bytes are behind that name today.
echo "v$NEXT is already published from our sha — skipping the build"
RESUMED=1
echo "v$NEXT is already published from our sha — skipping the build (whether it was SMOKED is a different question, asked below)"
BUILT=1
else
echo "::error::v$NEXT is a version this run OWNS (tag at ${SHA}) but the registry already serves bytes built from '${REV:-an unlabelled commit}'. Another lane pushed onto a name it did not hold. Nothing here may overwrite it — publish the intended bytes under a new number and delete the foreign image."
exit 1
@@ -456,22 +612,37 @@ jobs:
fi
echo "version=$NEXT" >> "$GITHUB_OUTPUT"
echo "resumed=$RESUMED" >> "$GITHUB_OUTPUT"
echo "built=$BUILT" >> "$GITHUB_OUTPUT"
echo "highest seen v$LAST (registry + git tags) -> v$NEXT, claimed at ${SHA} (document sha256:$SPEC_SHA)"
| tr -d '\r' | sed -n 's/^[Dd]ocker-[Cc]ontent-[Dd]igest: *//p' | head -1)
case "$DIGEST" in
sha256:*) : ;;
*) echo "::error::${img} resolves but the registry named no Docker-Content-Digest — these bytes have no name, so nothing may attest to them"; exit 1 ;;
esac
echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"
echo "$img is $DIGEST"
# The receipt's address, derived ONCE. A ref name cannot hold a colon,
# and three places needing "the digest with the colon swapped" is three
"refs/${{ steps.img.outputs.receipt }}" >/dev/null 2>&1; then
echo "already recorded by another run — same digest, same statement"
else
echo "::error::could not record the smoke receipt for ${{ steps.img.outputs.digest }}. These bytes booted and nothing can prove it, so a resume would smoke them again at best and skip them at worst — refusing to leave that state."
cat /tmp/receipt.err; exit 1
fi
# NOTHING LEAVES THIS JOB UNSMOKED, re-asked of the authority rather than
# remembered from a step output. Cheap, and it is the assertion `rollout`
# would otherwise have to trust: every later car declares `needs: image`, so
# this is the last place the question can still be answered before an
|| { echo "::error::no smoke receipt for ${{ steps.img.outputs.digest }} — these bytes have never been proven to boot. Nothing downstream may pin them."; exit 1; }
echo "${{ steps.img.outputs.digest }} is smoked"
# ══ CAR 2 ══ ROLL OUT, AND PROVE IT IS LIVE.
#
# The train used to stop at "pin pushed". A pin is not a release — the RUNNING
@@ -600,12 +879,12 @@ jobs:
# because the one thing worse than a moved tag is a moved tag that shipped.
/tmp/universe/charts/app/values/hanzo/cloud.yaml | head -1)
if [ "$PINNED" != "${{ needs.image.outputs.digest }}" ]; then
echo "::error::pinned cloud.yaml names ${PINNED:-no digest}, but the bytes this run smoked are ${{ needs.image.outputs.digest }}. v${VERSION} resolved to a different image between the smoke and the pin — refusing to deploy an image nothing proved."
exit 1
fi
echo "pin names ${PINNED} — the digest this run smoked"
if git -C /tmp/universe push --quiet origin HEAD:main 2>/dev/null; then
echo "pinned cloud to v${VERSION}"; exit 0
echo "pinned cloud to v${VERSION} @ ${PINNED}"; exit 0
fi
echo "universe moved under us (attempt ${attempt}/5) — re-applying on the new tip"
@@ -683,7 +982,7 @@ jobs:
# ══ CAR 3 ══ THE DOCUMENT TELLS THE TRUTH ABOUT PRODUCTION.
#
# `surface-check` proves the document equals the code. Nothing proved the
# `check` proves the document equals the code. Nothing proved the
# address is REACHABLE, and three things break that independently: a stale
# subset, a prefix missing from manifest/apps.go, and an edge worker that
# intercepts before the origin. Every address here is a method in eight SDKs, a
@@ -706,13 +1005,19 @@ jobs:
with:
go-version-file:go.mod
# Same private-module contract as the containment job: github.com/hanzoai/*
# is private, so it must resolve direct+authenticated and skip a sumdb that
# cannot see it, while everything else stays on the public proxy.
# Same private-module contract as the containment job, including the forge
# substitution: our own modules resolve from git.hanzo.ai (the canonical
# address) with GitHub as the fallback, so one drifted GitHub ACL cannot
# stop a release. go.sum still decides whether the bytes were right.
echo "::error::FLEET_DISPATCH_TOKEN missing in KMS at ${KMS_ORG}/deploy (env ${KMS_SECRET_ENV}). Create it with contents:write + metadata:read on hanzoai/{python-sdk,js-sdk,java-sdk,cli} hanzo-go/sdk hanzo-rs/sdk hanzo-kotlin/sdk hanzo-cpp/sdk hanzo-docs/docs — six owners, so a fine-grained PAT cannot carry it; use a classic PAT with repo scope or a GitHub App installed on all six. Until it exists, every cloud release ships a document no client is regenerated from — which is the failure this car was built to end, so it fails rather than skipping."
echo "::error::FLEET_DISPATCH_TOKEN missing in KMS at deploy/ (env ${KMS_SECRET_ENV}, org from the KMS credential). Create it with contents:write + metadata:read on hanzoai/{python-sdk,js-sdk,java-sdk,cli} hanzo-go/sdk hanzo-rs/sdk hanzo-kotlin/sdk hanzo-cpp/sdk hanzo-docs/docs — six owners, so a fine-grained PAT cannot carry it; use a classic PAT with repo scope or a GitHub App installed on all six. Until it exists, every cloud release ships a document no client is regenerated from — which is the failure this car was built to end, so it fails rather than skipping."
exit 1
fi
echo "::add-mask::${TOKEN}"
@@ -933,9 +1238,13 @@ jobs:
runs-on:[hanzo-build-linux-amd64]
timeout-minutes:15
steps:
# This job used to need no source. It does now: the invariant re-checked at
# the end of it lives in .hanzo/scripts/orphans.sh, beside the list of
["${n:-0}" -gt 1]||{echo"SKILLS-GATE FAIL: apps/skills/catalog holds the ${n:-0}-skill fallback — the overlay missed the //go:embed path";exit 1;};\
j="$(find /src/webui/dist -type f -name '*.js'| wc -l)";\
["$j" -gt 0]||{echo"CONSOLE-GATE FAIL: webui/dist carries no JavaScript — the overlay missed the //go:embed path and the image would ship the fallback shell";exit 1;};\
# THE STAMP LANDED — asked of the ARTIFACT, not of the flag string.
#
# `-X` naming a path or symbol the linker cannot resolve is not an error: it is
# dropped, the build succeeds, and every binary then reports the entirely
# legitimate-looking "unknown" forever. A renamed package or variable would fail
# in exactly the one way nobody looks at, which is how this started.
#
# `go version -m` is NOT a witness — it echoes the -ldflags string that was
# REQUESTED, and that string is present even when the symbol was never set
# (measured on /cloud, whose Version stamp has been dropped all along). Only the
# linked bytes answer.
#
# strings|grep rather than a bare grep: grep treats binary input as non-text and
# its exit status there is not portable across implementations, so a plain
# `grep -qF` can report no match on a binary that demonstrably contains the sha.
# strings normalises to text lines first; binutils is already installed above.
#
# An image built with no REVISION is not a failure — it is a build that cannot
# name its commit, and it says so here and on every health response it serves.
RUNset -eu;\
if["$REVISION"="unknown"];then\
echo">> no REVISION build-arg: this image cannot name its commit, and every health response it serves will report revision=unknown";\
else\
strings -a /plugins/base | grep -qF "$REVISION"||{echo"FATAL: -X did not reach /plugins/base — github.com/hanzoai/cloud.revision was not resolved, so it was dropped and every health response would report 'unknown'";exit 1;};\
echo">> revision $REVISION linked into the plugins";\
fi
# Prove a SHIPPED sqlite-backed plugin binds sqlite3_* to libsqlcipher, not a
# plaintext libsqlite3. /plugins/base opens per-org stores under the SAME CGO=1 +
# libsqlite3 build every plugin above got, so it is a real witness for the set.
# the shipped build carries, so the suite exercises the same schema surface.
TEST_TAGS:= sqlite_fts5
test:## Run unit + integration tests (pure-Go, with the FTS5 tag the image ships).
# The lifted prose is COMMITTED (zipdoc_gen.go) because bare `go build` cannot
# regenerate it; -check writes nothing and goes red when a lift no longer
# matches its source, which is the drift being committed makes possible.
# Go drops comments at compile time, so cmd/zipdoc is the ONLY path from a typed
# handler's prose to /v1/openapi.json — the document the SDK repos and the CLI
# read. Its output is COMMITTED, and that is what lets a bare `go build` (and the
# release image) produce a binary that still describes itself without anyone
# paying to lift the prose again. The image used to pay: `go generate -run zipdoc
# ./...` ran on the build's critical path for 355.9s of a 17-minute build, to
# reproduce 99 files that were already in the tree.
#
# Committed means it can go STALE, so exactly one thing has to stay true:
# regenerating from source changes nothing. This asserts it by running THE
# GENERATOR and diffing, rather than asking a -check mode for a second opinion —
# a gate must never be able to disagree with the tool it polices. It is also the
# only form that catches the case below.
#
# `git status --porcelain`, not `git diff`: a NEW package's zipdoc_gen.go is
# untracked and therefore invisible to a diff, which is the failure that matters
# most. The pathspec scopes it to the generator's own files, so an unrelated
# dirty tree neither hides a stale lift nor invents one.
zipdoc-check:## Regenerate the lifted prose FROM SOURCE and fail on any diff.
# Per PACKAGE, not ./...: the checker must load exactly the way `go generate`
# does, one package at a time — whole-module loading extracts differently
# (zap-proto/zip zipdoc: single-vs-module load divergence) and a gate must
# never disagree with the generator it polices.
@set -e;for d in $$(grep -rl '^//go:generate go run github.com/zap-proto/zip/cmd/zipdoc' --include='*.go' clients cmd . 2>/dev/null | xargs -n1 dirname | sort -u);do(cd$$d &&$(GO) run github.com/zap-proto/zip/cmd/zipdoc -check)||{echo"$$d/zipdoc_gen.go is stale — run: go generate -run zipdoc ./$$d/...";exit 1;};done
# A dot-directory is not this module's source. An agent worktree at
# .claude/worktrees/<id>/ is a whole second checkout of this repository, and
# the walk read it: 203 packages where there are 104, and it went red on a
# copy's o11y while nothing here had changed.
@set -e;for d in $$(grep -rl '^//go:generate go run github.com/zap-proto/zip/cmd/zipdoc' --include='*.go' --exclude-dir='.?*' clients cmd . 2>/dev/null | xargs -n1 dirname | sort -u);do\
(cd$$d &&$(GO) run github.com/zap-proto/zip/cmd/zipdoc -check)||{echo"$$d/zipdoc_gen.go is stale — run: go generate -run zipdoc ./$$d/...";exit 1;};\
done
test:## Run unit + integration tests (pure-Go, with the FTS5 tag the image ships).
@echo ">> openapi.yaml — $$(grep -c '^ /' openapi.yaml) paths. The MCP tool list is NOT an artifact: POST /v1/mcp asks every subsystem."
test-cgo:## Prove the cgo build works too — forces the fork's pure-Go backend via -tags sqlite_purego so the embedded modernc importers don't double-register "sqlite".
Description:"Lists every affiliate across the fleet with its ORG exposed, plus a\nfleet summary of lifetime accrued, still-pending and paid commission in\ninteger cents.\n\nPLATFORM SUDO ONLY, and a non-admin is refused outright. This is the\ncross-tenant view and it names orgs — exactly what the partner-facing\nleaderboard refuses to do. There is deliberately no org-scoped variant of this\nread; a partner sees its own standing through its own dashboard. Bounded per\nrequest.",
Fields:map[string]string{
"adminAffiliateView.accruedCents":"AccruedCents is lifetime commission accrued, in cents. It only grows — a\npayout moves paidCents, never this.",
"adminAffiliateView.approvedAt":"ApprovedAt is when staff approved, Unix seconds UTC. 0 means never approved.",
"adminAffiliateView.code":"Code is the minted referral code, the slug the ?aff link carries. Empty until\napproval mints it. Codes are one global namespace across all affiliates.",
"adminAffiliateView.createdAt":"CreatedAt is when the org applied, Unix seconds UTC.",
"adminAffiliateView.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed — the id the\napprove, suspend, rate and payout routes address.",
"adminAffiliateView.org":"Org is the partner's own org slug. It appears ONLY on this cross-tenant admin\nview; no partner-facing read ever names another org.",
"adminAffiliateView.paidCents":"PaidCents is lifetime commission already paid out, in cents — credits grants\nand record-only cash disbursements alike.",
"adminAffiliateView.pendingCents":"PendingCents is accrued minus paid, in cents: what is still owed, and the hard\nceiling the next payout is reserved against. Never negative.",
"adminAffiliateView.rateBps":"RateBps is this affiliate's DIRECT (level 1) commission rate in basis points\nOF Hanzo's margin (2000 = 20% of margin, never of the customer's bill). Levels\n2 and 3 are platform-wide switches and are not carried per affiliate.",
"adminAffiliateView.referredCount":"ReferredCount is how many orgs this affiliate is the DIRECT referrer of,\ncounted from the attribution edges. It is 0 on the single-affiliate answers\n(approve, suspend, rate, payout), which do not run the count.",
"adminAffiliateView.requestedCode":"RequestedCode is the vanity code the applicant asked for. A request, not an\nallocation: approval mints a different slug if this one was taken. Absent when\nnone was asked for.",
"adminAffiliateView.status":"Status is \"applied\", \"approved\" or \"suspended\". Only \"approved\" resolves for\nattribution and accrues; \"suspended\" stops future earning and claws nothing\nback.",
"adminAffiliateView.suspendedAt":"SuspendedAt is when staff suspended, Unix seconds UTC. 0 means never\nsuspended; it is not cleared by a later re-approval.",
"directoryData.affiliates":"Affiliates is one row per affiliate across the whole fleet, ORG EXPOSED,\noldest first and bounded by the request's limit.",
"directoryData.summary":"Summary tallies exactly the rows above — not the whole table — so a limit that\ntruncates the page truncates the tally with it.",
"directoryOut.data":"Data is the affiliate directory and its tally.",
"envelope.msg":"Msg is an operator-facing note. Empty on every success here — it exists\nbecause the console's admin unwrapper reads the shape cloud.OK writes.",
"envelope.status":"Status is \"ok\" on every 2xx from this surface; a failure is an HTTP error with\nzip's error body, not this envelope carrying a different word.",
"page.limit":"Limit caps the rows returned. Absent or non-positive means the default of\n500; anything above 1000 is clamped to 1000.",
"totals.accruedCents":"AccruedCents is lifetime commission accrued summed over those rows, in cents.",
"totals.applied":"Applied is how many of those rows are still awaiting approval — no code, no\naccrual yet.",
"totals.approved":"Approved is how many are approved: the only rows whose code resolves for\nattribution and whose balance can still grow.",
"totals.paidCents":"PaidCents is lifetime commission already paid out summed over those rows, in\ncents.",
"totals.pendingCents":"PendingCents is accrued minus paid summed over those rows, in cents — the\noutstanding liability across the page.",
"totals.suspended":"Suspended is how many were suspended. What they already accrued stays accrued\nand stays payable.",
"totals.total":"Total is how many affiliate rows this page covered, at every status. It is the\npage, not the table: a limit that truncates truncates this too.",
},
})
zip.Describe("GET /v1/admin/referrals",zip.Doc{
Description:"Answers the referral board: the top referrers by lifetime\ncommission, the funnel conversion rate (referred orgs that have actually\nproduced commission, over all referred orgs), and the accrual LIABILITY the\nplatform owes, broken out by upline level.\n\nRead the liability figure carefully — it is commission accrued and NOT yet\npaid, so it is money owed, not money spent, and the per-level split says how\nmuch of it comes from direct referrals versus the second and third levels.\n\nPLATFORM SUDO ONLY, cross-tenant, and it names orgs. It reads the SAME single\nattribution spine the accrual itself walks, so the board and the ledger cannot\ndisagree. Amounts are integer cents.",
Fields:map[string]string{
"envelope.msg":"Msg is an operator-facing note. Empty on every success here — it exists\nbecause the console's admin unwrapper reads the shape cloud.OK writes.",
"envelope.status":"Status is \"ok\" on every 2xx from this surface; a failure is an HTTP error with\nzip's error body, not this envelope carrying a different word.",
"funnel.convertedOrgs":"ConvertedOrgs is how many distinct referred orgs have produced positive\ncommission at least once — a referral that actually spent.",
"funnel.ratePct":"RatePct is convertedOrgs over referredOrgs as a PERCENTAGE, 0–100, and the one\nnon-integer figure on this board. It is 0 when nothing has been referred yet,\nnot undefined.",
"funnel.referredOrgs":"ReferredOrgs is how many attribution edges exist fleet-wide — one per referred\norg, first-touch, so it is also the count of distinct referred orgs.",
"levelSplit.l1Cents":"L1Cents is lifetime commission accrued to DIRECT referrers, in cents.",
"levelSplit.l2Cents":"L2Cents is lifetime commission accrued one step above the direct referrer, in\ncents, at the platform-wide level-2 rate.",
"levelSplit.l3Cents":"L3Cents is lifetime commission accrued two steps above, in cents. Nothing\naccrues past level 3, so l1+l2+l3 is the whole accrual.",
"referralBoard.accrualByLevel":"AccrualByLevel splits the lifetime accrual across the three upline levels —\nhow much of the liability comes from direct referrals versus the chain above.",
"referralBoard.conversion":"Conversion is the funnel: referred orgs against those that actually earned.",
"referralBoard.summary":"Summary is the fleet tally — population by status, and lifetime accrued, paid\nand still-owed commission.",
"referralBoard.topReferrers":"TopReferrers is the 25 affiliates with the most lifetime accrued commission,\ndescending, orgs named.",
"referralsOut.data":"Data is the referral board: leaders, funnel, tally and per-level liability.",
"referrerRow.accruedCents":"AccruedCents is lifetime commission accrued, in cents. The board is sorted by\nthis, descending.",
"referrerRow.code":"Code is that affiliate's minted referral code; empty if it is not approved.",
"referrerRow.org":"Org is the partner's own org slug. Named only here, on the SuperAdmin board —\nthe partner-facing leaderboard shows an opt-in handle and never an org.",
"referrerRow.pendingCents":"PendingCents is accrued minus paid, in cents — what is still owed to this\naffiliate. Never negative.",
"referrerRow.referredCount":"ReferredCount is how many orgs this affiliate is the DIRECT referrer of —\nits level-1 downline, not the whole three-level chain.",
"referrerRow.status":"Status is \"applied\", \"approved\" or \"suspended\".",
"tally.accruedLifetimeCents":"AccruedLifetimeCents is all commission ever accrued, summed across every\naffiliate, in cents. It only grows; a payout does not reduce it.",
"tally.affiliates":"Affiliates is how many affiliate rows the board read, at every status. The\nread is bounded at 1000 rows, so a larger fleet reports the bound.",
"tally.approved":"Approved is how many of those rows are approved — the only ones whose code\nresolves for attribution and whose balance can grow.",
"tally.paidLifetimeCents":"PaidLifetimeCents is all commission ever paid out, in cents: credits grants\nplus record-only cash disbursements.",
"tally.pendingLiabilityCents":"PendingLiabilityCents is accrued minus paid across every affiliate, in cents.\nRead it as money OWED and not yet disbursed — a liability, not spend.",
},
})
zip.Describe("GET /v1/affiliates",zip.Doc{
Description:"Answers the caller org's OWN affiliate standing: status, referral\ncode and share link, commission rate, how many orgs it has referred, and its\nlifetime accrued, still-pending and already-paid commission in integer cents,\nwith its payout history.\n\nAn org that never applied gets an honest `isAffiliate:false` and the default\nrate rather than a 404 — the console renders the apply form off that answer.\n\nThe affiliate is resolved from the VALIDATED org, never from a field, so this\ncan only ever read the caller's own row; without a principal it is refused. It\nis a PURE READ: nothing accrues until the sweep runs. Commission is earned on\nHanzo's MARGIN, never on the referred customer's bill, so nothing here changes\nwhat that customer pays.",
Fields:map[string]string{
"affiliateStanding.accruedCents":"AccruedCents is lifetime commission accrued, in cents.",
"affiliateStanding.code":"Code is the minted referral code; empty until staff approve.",
"affiliateStanding.defaultRateBps":"DefaultRateBps is the direct rate a new affiliate would get, answered only\nto a caller that has not applied.",
"affiliateStanding.handle":"Handle is the opt-in public leaderboard name; empty means opted out.",
"affiliateStanding.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed — what staff\napprove, suspend, re-rate and pay against. Absent until the org applies.",
"affiliateStanding.isAffiliate":"IsAffiliate says whether the caller org has an affiliate record at all. It is\nthe ONE field an org that never applied gets besides defaultRateBps: on false,\nread nothing else here — every other field is absent, not zero.",
"affiliateStanding.link":"Link is the shareable ?aff URL; empty until a code is minted.",
"affiliateStanding.marginBps":"MarginBps is the platform gross-margin fraction commission is a rate OF.",
"affiliateStanding.paidCents":"PaidCents is lifetime commission already paid out, in cents.",
"affiliateStanding.payouts":"Payouts is the payout history, newest rows bounded.",
"affiliateStanding.pendingCents":"PendingCents is accrued minus paid — what the platform still owes.",
"affiliateStanding.rateBps":"RateBps is the affiliate's own direct commission rate, in basis points.",
"affiliateStanding.referredCount":"ReferredCount is how many orgs this affiliate has referred.",
"affiliateStanding.requestedCode":"RequestedCode is the vanity code asked for at apply time — a request, not an\nallocation. Approval mints `code`, which may be a different slug if this one\nwas already taken.",
"affiliateStanding.status":"Status is \"applied\", \"approved\" or \"suspended\". Only an approved affiliate has\na code that resolves for attribution and accrues commission; suspended keeps\nwhat it already earned but stops earning more.",
"remittance.amountCents":"AmountCents is the amount disbursed, in cents. It was reserved against pending\ncommission atomically when recorded, so it never exceeds what was owed.",
"remittance.createdAt":"CreatedAt is when the payout was recorded, Unix seconds UTC — when the balance\nmoved, not necessarily when the cash landed.",
"remittance.id":"ID is the payout row's server-minted handle, \"apo_\"-prefixed.",
"remittance.method":"Method is how it was settled. \"credits\" issued a commerce grant into the\naffiliate org's own wallet; any other value (wire, paypal, check, …) is a\nRECORD of cash a human moved out of band.",
"remittance.reference":"Reference is the operator's settlement note — a bank id, a ledger ref. Free\ntext, absent when none was given.",
"remittance.txn":"Txn is the commerce ledger transaction id, set ONLY where a \"credits\" payout\nactually issued the grant. Absent for cash methods, which write no ledger row.",
Description:"Answers the top affiliates by lifetime accrued commission, shown by\nOPT-IN HANDLE with aggregate figures only, plus the caller's own exact rank.\n\nIt never discloses an org identity and never a referred org's usage. An\naffiliate that has set no handle still OCCUPIES its rank but is not listed —\nso opting out hides the name, not the position, and the visible board must not\nbe read as a complete roster.\n\nThe caller's own row carries its exact GLOBAL rank, computed over the whole\napproved set rather than over the page, so it is right well outside the top of\nthe board. Only an approved affiliate has a rank. Requires a validated\nprincipal; a signed-in non-affiliate may read the board but gets no personal\nrow.",
Fields:map[string]string{
"affiliateBoard.leaders":"Leaders are the top opt-in affiliates, by handle and aggregate figures only.",
"affiliateBoard.total":"Total is the approved population where it is known; omitted where the top\npage truncated and the caller has no rank to derive it from.",
"affiliateBoard.you":"You is the caller's own row with its exact global rank; only an approved\naffiliate has one.",
"leaderboardRow.accruedCents":"AccruedCents is that affiliate's lifetime commission accrued, in cents, and\nwhat the board is ordered by. An aggregate: no per-customer figure is exposed.",
"leaderboardRow.handle":"Handle is the affiliate's self-chosen display name — the only identity the\nboard ever carries. The org behind it is never disclosed.",
"leaderboardRow.isYou":"IsYou marks the caller's own row, so a client can highlight it without\nmatching on a handle. Absent on every other row.",
"leaderboardRow.rank":"Rank is the position in the GLOBAL approved set ordered by lifetime accrued\ncommission, 1-based. Affiliates that set no handle still occupy their rank and\nare simply not listed, so the visible ranks have gaps and the board is not a\ncomplete roster. On the caller's own row the rank is computed over the whole\nset, so it is exact well outside the top page.",
"leaderboardRow.referredCount":"ReferredCount is how many orgs that affiliate directly referred — a count\nonly, never which orgs.",
},
})
zip.Describe("GET /v1/affiliates/me",zip.Doc{
Description:"Answers the richer self-view: the same lifetime accrued, pending and paid\ncommission and payout history, plus the caller's downline broken out by upline\nLEVEL — direct, second, third — each with the rate paid at that level and how\nmany orgs sit there.\n\nCommission is MULTI-LEVEL: a referred org's spend pays up its referral chain,\nthree levels deep and no further. The direct level is the affiliate's own\nnegotiated rate; the second and third are platform-wide switches, read live,\nso the schedule shown is the one actually in force rather than one compiled\nin. A caller that has not applied still gets that schedule alongside\n`isAffiliate:false`, so the console can show what it would earn.\n\nScoped to the validated org and nothing else, and refused without a\nprincipal. A PURE READ — it reports the downline but accrues nothing.",
Fields:map[string]string{
"affiliateSelf.accruedCents":"AccruedCents is lifetime commission accrued, in cents. It only grows — a\npayout is recorded against paidCents and never reduces this.",
"affiliateSelf.code":"Code is the minted referral code, the slug the ?aff link carries. Absent until\nstaff approve; codes live in ONE global namespace across all affiliates.",
"affiliateSelf.defaultRateBps":"DefaultRateBps is the direct rate a new affiliate starts at, in basis points\nof margin (2000 = 20%). Answered ONLY to a caller that has not applied, as the\nquote beside `schedule`.",
"affiliateSelf.downlineTotal":"DownlineTotal counts every org in the caller's downline across the levels.",
"affiliateSelf.handle":"Handle is the opt-in public leaderboard name. Empty means opted out: the\ncaller keeps its rank and still sees its own row, it is just not listed.",
"affiliateSelf.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed. Absent until the\norg applies.",
"affiliateSelf.isAffiliate":"IsAffiliate says whether the caller org has an affiliate record. On false the\nanswer carries the rate SCHEDULE and the default rate instead of a downline,\nso the console can show what the caller would earn.",
"affiliateSelf.levels":"Levels is the caller's downline per upline level, with the rate paid there.",
"affiliateSelf.link":"Link is the shareable ?aff URL built from the code. Empty until a code is\nminted, since there is nothing to share before approval.",
"affiliateSelf.marginBps":"MarginBps is the platform gross-margin fraction, in basis points, that every\nrate here is a rate OF. Read live per request, so it is the value in force\nnow, not the one that applied to commission already accrued.",
"affiliateSelf.paidCents":"PaidCents is lifetime commission already paid out, in cents — credits grants\nand record-only cash disbursements alike.",
"affiliateSelf.payouts":"Payouts is the payout history, newest first, bounded to the last 100 rows.",
"affiliateSelf.pendingCents":"PendingCents is accrued minus paid, in cents — what the platform still owes\nand the ceiling on the next payout. Never negative.",
"affiliateSelf.rateBps":"RateBps is the caller's OWN direct (level 1) commission rate, in basis points\nof margin. Levels 2 and 3 are platform-wide and appear in `levels`.",
"affiliateSelf.schedule":"Schedule is the rate schedule quoted to a caller that has not applied.",
"affiliateSelf.status":"Status is \"applied\", \"approved\" or \"suspended\"; absent for a caller that never\napplied. Only \"approved\" mints links and accrues.",
"levelView.downlineCount":"DownlineCount is how many orgs sit exactly this many hops below the caller. It\nis 0 in the schedule quoted to a caller that has not applied, which has no\ndownline to count.",
"levelView.level":"Level is the upline distance from the org whose spend is being shared: 1 is\nthe direct referrer, 2 and 3 the referrers above it. Nothing accrues past 3.",
"levelView.rateBps":"RateBps is the commission paid at this level, in basis points OF Hanzo's\nmargin (2000 = 20% of margin, never of the customer's bill). Level 1 is the\naffiliate's own negotiated rate; 2 and 3 are platform switches read live, so\nthis is the schedule actually in force, not one compiled in.",
"remittance.amountCents":"AmountCents is the amount disbursed, in cents. It was reserved against pending\ncommission atomically when recorded, so it never exceeds what was owed.",
"remittance.createdAt":"CreatedAt is when the payout was recorded, Unix seconds UTC — when the balance\nmoved, not necessarily when the cash landed.",
"remittance.id":"ID is the payout row's server-minted handle, \"apo_\"-prefixed.",
"remittance.method":"Method is how it was settled. \"credits\" issued a commerce grant into the\naffiliate org's own wallet; any other value (wire, paypal, check, …) is a\nRECORD of cash a human moved out of band.",
"remittance.reference":"Reference is the operator's settlement note — a bank id, a ledger ref. Free\ntext, absent when none was given.",
"remittance.txn":"Txn is the commerce ledger transaction id, set ONLY where a \"credits\" payout\nactually issued the grant. Absent for cash methods, which write no ledger row.",
Description:"Answers the caller's own commission ledger: per period, the margin it\nearned against and the commission taken from that margin; and per referred\norg, that referral's aggregate contribution. Integer cents throughout.\n\nThe per-org view deliberately carries the affiliate's OWN earned share and NOT\nthe referred org's spend or margin. An affiliate is entitled to what it\nearned, not to a restatement of its customer's usage — the period view is\nwhere the margin base appears, aggregated across every referral.\n\nScoped server-side to the validated caller's affiliate; a caller that is not\none gets `isAffiliate:false`.",
Fields:map[string]string{
"affiliateEarnings.accruedCents":"AccruedCents is lifetime commission accrued, in cents.",
"affiliateEarnings.byPeriod":"ByPeriod is the per-period ledger: the margin earned against and the\ncommission taken from it.",
"affiliateEarnings.byReferredOrg":"ByReferredOrg is each referral's aggregate contribution — the affiliate's\nOWN share, never the referred org's spend.",
"affiliateEarnings.isAffiliate":"IsAffiliate says whether the caller org has an affiliate record. On false it\nis the ONLY field present — there is no ledger to report, and the zeros you\nmight expect are absent rather than reported as earnings of nothing.",
"affiliateEarnings.marginBps":"MarginBps is the platform gross-margin fraction commission is a rate OF.",
"affiliateEarnings.paidCents":"PaidCents is lifetime commission already paid out, in cents.",
"affiliateEarnings.pendingCents":"PendingCents is accrued minus paid — what the platform still owes.",
"orgEarningView.commissionCents":"CommissionCents is what the caller earned from that org across ALL periods, in\ncents. Deliberately the caller's own share and nothing else: that org's spend\nand the margin on it are not restated here.",
"orgEarningView.referredOrg":"ReferredOrg is the org slug this contribution came from — one the caller\nreferred, directly or up to three levels down.",
"periodEarningView.commissionCents":"CommissionCents is what the caller earned that period, in cents: the sum over\neach referred org and upline level of margin × that level's rate. Always ≤\nmarginCents, by construction.",
"periodEarningView.marginCents":"MarginCents is the margin Hanzo earned in that period on the spend of every\norg the caller referred, in cents — the base commission is a rate OF. It is\nthe aggregate base, never any one customer's bill.",
"periodEarningView.period":"Period is the accrual bucket: the UTC year-month, \"YYYY-MM\". Commission is\nlatched at most once per referred org per period, so one row is one month.",
Description:"Answers the caller's share links, each with its URL and its funnel:\nclicks tracked, signups — orgs attributed with that code — and conversions,\nmeaning how many of those signups have actually produced commission.\n\nSignups and conversions are DERIVED from the commission ledger and never\nstored, so they cannot drift from the money. Clicks are the one stored counter\nand the one that is pure vanity.\n\nAny pending public click pings are folded into the store before the read, in\none batch — which is how the counters stay current without a database write\nper click. Scoped to the validated caller's own affiliate; a non-affiliate\ngets `isAffiliate:false` and the link cap.",
Fields:map[string]string{
"affiliateLinks.isAffiliate":"IsAffiliate says whether the caller org has an affiliate record. On false only\nmaxLinks comes back — there are no links, and there is no link to mint until\nthe org applies and is approved.",
"affiliateLinks.links":"Links is the caller's share links, each with its URL and funnel.",
"affiliateLinks.maxLinks":"MaxLinks is how many share links one affiliate may hold.",
"affiliateLinks.status":"Status is the caller's affiliate status: \"applied\", \"approved\" or\n\"suspended\"; absent for a non-affiliate. Minting a link requires \"approved\",\nbecause a link that cannot accrue quietly loses the referral.",
"codeView.clicks":"Clicks is how many pings this code has taken. The one STORED counter here and\npure vanity: no accrual or payout reads it, pings are coalesced in memory and\nflushed in batches, and a dropped tally is accepted rather than contending\nwith the money write path. Do not reconcile it against anything.",
"codeView.code":"Code is the link's slug — 3–32 chars of a–z, 0–9 and hyphen — unique across\nthe WHOLE directory, so any affiliate's code resolves an attribution.",
"codeView.conversions":"Conversions is how many of those signups have actually produced positive\ncommission for the caller. Also derived, from the accrual rows, so it is\n≤ signups and lags a referral until the first sweep after it spends.",
"codeView.createdAt":"CreatedAt is when the link was minted, Unix seconds UTC.",
"codeView.label":"Label is the caller's own note for the link (\"twitter\", \"newsletter\").\nCosmetic: trimmed, stripped of control characters, capped at 48 bytes, and\nnever part of the code. \"primary\" on the link mirrored at approval.",
"codeView.signups":"Signups is how many orgs were attributed with this code — DERIVED by counting\nattribution edges, never stored, so it cannot drift from the ledger.",
"codeView.url":"URL is the full shareable link, the brand host plus ?aff=<code>. The host is\nthe deployment's own brand, so a Lux or Zoo install never mints a hanzo.ai\nlink.",
Description:"Approves an affiliate and MINTS its referral code — the moment\nthe partner has a working share link and starts accruing.\n\nThe code is taken from the body if one is given, else the vanity code the\napplicant requested, else a slug derived for them. Codes are ONE global\nnamespace, so a taken code is a 409 and nothing is approved. The minted code\nis also mirrored as a link row so click tracking is uniform across every code\nthe affiliate holds; that mirror is best-effort and its failure never fails\nthe approval.\n\nApproval is what makes an affiliate eligible: before it, attribution against\nits code does not resolve and no sweep accrues to it. PLATFORM SUDO ONLY.\nAudited.",
Fields:map[string]string{
"adminAffiliateView.accruedCents":"AccruedCents is lifetime commission accrued, in cents. It only grows — a\npayout moves paidCents, never this.",
"adminAffiliateView.approvedAt":"ApprovedAt is when staff approved, Unix seconds UTC. 0 means never approved.",
"adminAffiliateView.code":"Code is the minted referral code, the slug the ?aff link carries. Empty until\napproval mints it. Codes are one global namespace across all affiliates.",
"adminAffiliateView.createdAt":"CreatedAt is when the org applied, Unix seconds UTC.",
"adminAffiliateView.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed — the id the\napprove, suspend, rate and payout routes address.",
"adminAffiliateView.org":"Org is the partner's own org slug. It appears ONLY on this cross-tenant admin\nview; no partner-facing read ever names another org.",
"adminAffiliateView.paidCents":"PaidCents is lifetime commission already paid out, in cents — credits grants\nand record-only cash disbursements alike.",
"adminAffiliateView.pendingCents":"PendingCents is accrued minus paid, in cents: what is still owed, and the hard\nceiling the next payout is reserved against. Never negative.",
"adminAffiliateView.rateBps":"RateBps is this affiliate's DIRECT (level 1) commission rate in basis points\nOF Hanzo's margin (2000 = 20% of margin, never of the customer's bill). Levels\n2 and 3 are platform-wide switches and are not carried per affiliate.",
"adminAffiliateView.referredCount":"ReferredCount is how many orgs this affiliate is the DIRECT referrer of,\ncounted from the attribution edges. It is 0 on the single-affiliate answers\n(approve, suspend, rate, payout), which do not run the count.",
"adminAffiliateView.requestedCode":"RequestedCode is the vanity code the applicant asked for. A request, not an\nallocation: approval mints a different slug if this one was taken. Absent when\nnone was asked for.",
"adminAffiliateView.status":"Status is \"applied\", \"approved\" or \"suspended\". Only \"approved\" resolves for\nattribution and accrues; \"suspended\" stops future earning and claws nothing\nback.",
"adminAffiliateView.suspendedAt":"SuspendedAt is when staff suspended, Unix seconds UTC. 0 means never\nsuspended; it is not cleared by a later re-approval.",
"affiliateData.affiliate":"Affiliate is the row as it stands AFTER the action that returned it. Its\nreferredCount is 0 here: these single-affiliate answers do not run the count.",
"affiliateOut.data":"Data carries the affiliate row the action just wrote.",
"approval.code":"Code overrides the minted code; else the requested vanity code, else a\nderived slug.",
"approval.id":"ID is the affiliate to approve, from the path.",
"envelope.msg":"Msg is an operator-facing note. Empty on every success here — it exists\nbecause the console's admin unwrapper reads the shape cloud.OK writes.",
"envelope.status":"Status is \"ok\" on every 2xx from this surface; a failure is an HTTP error with\nzip's error body, not this envelope carrying a different word.",
Description:"Pays out accrued commission and answers the payout row with the\naffiliate's updated balances.\n\nThe amount is reserved atomically against the affiliate's PENDING commission —\naccrued minus paid — so a payout can never exceed what is owed. The METHOD\ndecides whether money actually moves: `credits` issues a commerce grant into\nthe affiliate ORG's own wallet, tagged so the ledger can tell an affiliate\npayout apart from an admin or referral grant; every other method — wire,\npaypal and the rest — is RECORD-ONLY: the payout row and the balances move,\nthe cash is disbursed out of band.\n\nThe amount is integer cents and must be positive. PLATFORM SUDO ONLY.\nAudited.",
Fields:map[string]string{
"adminAffiliateView.accruedCents":"AccruedCents is lifetime commission accrued, in cents. It only grows — a\npayout moves paidCents, never this.",
"adminAffiliateView.approvedAt":"ApprovedAt is when staff approved, Unix seconds UTC. 0 means never approved.",
"adminAffiliateView.code":"Code is the minted referral code, the slug the ?aff link carries. Empty until\napproval mints it. Codes are one global namespace across all affiliates.",
"adminAffiliateView.createdAt":"CreatedAt is when the org applied, Unix seconds UTC.",
"adminAffiliateView.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed — the id the\napprove, suspend, rate and payout routes address.",
"adminAffiliateView.org":"Org is the partner's own org slug. It appears ONLY on this cross-tenant admin\nview; no partner-facing read ever names another org.",
"adminAffiliateView.paidCents":"PaidCents is lifetime commission already paid out, in cents — credits grants\nand record-only cash disbursements alike.",
"adminAffiliateView.pendingCents":"PendingCents is accrued minus paid, in cents: what is still owed, and the hard\nceiling the next payout is reserved against. Never negative.",
"adminAffiliateView.rateBps":"RateBps is this affiliate's DIRECT (level 1) commission rate in basis points\nOF Hanzo's margin (2000 = 20% of margin, never of the customer's bill). Levels\n2 and 3 are platform-wide switches and are not carried per affiliate.",
"adminAffiliateView.referredCount":"ReferredCount is how many orgs this affiliate is the DIRECT referrer of,\ncounted from the attribution edges. It is 0 on the single-affiliate answers\n(approve, suspend, rate, payout), which do not run the count.",
"adminAffiliateView.requestedCode":"RequestedCode is the vanity code the applicant asked for. A request, not an\nallocation: approval mints a different slug if this one was taken. Absent when\nnone was asked for.",
"adminAffiliateView.status":"Status is \"applied\", \"approved\" or \"suspended\". Only \"approved\" resolves for\nattribution and accrues; \"suspended\" stops future earning and claws nothing\nback.",
"adminAffiliateView.suspendedAt":"SuspendedAt is when staff suspended, Unix seconds UTC. 0 means never\nsuspended; it is not cleared by a later re-approval.",
"disbursal.amountCents":"AmountCents is the payout, integer cents; it must be positive and can\nnever exceed the affiliate's pending commission. Body-only (`url:\"-\"`,\nlike every money field here): a payout must never ride the URL into\naccess logs, and the raw handler read only the body.",
"disbursal.id":"ID is the affiliate to pay, from the path.",
"disbursal.method":"Method decides whether money moves: `credits` issues a commerce grant,\nevery other method (wire, paypal, …) is record-only.",
"disbursal.reference":"Reference is the operator's settlement note (a bank id, a ledger ref).",
"envelope.msg":"Msg is an operator-facing note. Empty on every success here — it exists\nbecause the console's admin unwrapper reads the shape cloud.OK writes.",
"envelope.status":"Status is \"ok\" on every 2xx from this surface; a failure is an HTTP error with\nzip's error body, not this envelope carrying a different word.",
"payoutOut.data":"Data is the recorded payout and the balances it left behind.",
"remittance.amountCents":"AmountCents is the amount disbursed, in cents. It was reserved against pending\ncommission atomically when recorded, so it never exceeds what was owed.",
"remittance.createdAt":"CreatedAt is when the payout was recorded, Unix seconds UTC — when the balance\nmoved, not necessarily when the cash landed.",
"remittance.id":"ID is the payout row's server-minted handle, \"apo_\"-prefixed.",
"remittance.method":"Method is how it was settled. \"credits\" issued a commerce grant into the\naffiliate org's own wallet; any other value (wire, paypal, check, …) is a\nRECORD of cash a human moved out of band.",
"remittance.reference":"Reference is the operator's settlement note — a bank id, a ledger ref. Free\ntext, absent when none was given.",
"remittance.txn":"Txn is the commerce ledger transaction id, set ONLY where a \"credits\" payout\nactually issued the grant. Absent for cash methods, which write no ledger row.",
"settlement.affiliate":"Affiliate is the row re-read AFTER the payout, so its paidCents and\npendingCents already account for the row beside it.",
"settlement.payout":"Payout is the payout row just recorded.",
Description:"Sets one affiliate's DIRECT commission rate, in basis points of\nHanzo's margin.\n\nThe rate is CAPPED so that the direct rate plus the platform-wide second- and\nthird-level rates can never exceed the whole margin — the structural guarantee\nthat everything paid on one source event stays inside the margin actually\nearned. The cap is resolved from the rates in force at the moment of the call\nand quoted in the refusal, because those switches move; a hardcoded bound\nwould start lying the moment somebody edits the schedule.\n\nOnly the direct level is per-affiliate. The second and third levels are\nplatform switches and are not settable here. The change applies to FUTURE\naccruals — commission already latched for a period is not recomputed. PLATFORM\nSUDO ONLY. Audited.",
Fields:map[string]string{
"adminAffiliateView.accruedCents":"AccruedCents is lifetime commission accrued, in cents. It only grows — a\npayout moves paidCents, never this.",
"adminAffiliateView.approvedAt":"ApprovedAt is when staff approved, Unix seconds UTC. 0 means never approved.",
"adminAffiliateView.code":"Code is the minted referral code, the slug the ?aff link carries. Empty until\napproval mints it. Codes are one global namespace across all affiliates.",
"adminAffiliateView.createdAt":"CreatedAt is when the org applied, Unix seconds UTC.",
"adminAffiliateView.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed — the id the\napprove, suspend, rate and payout routes address.",
"adminAffiliateView.org":"Org is the partner's own org slug. It appears ONLY on this cross-tenant admin\nview; no partner-facing read ever names another org.",
"adminAffiliateView.paidCents":"PaidCents is lifetime commission already paid out, in cents — credits grants\nand record-only cash disbursements alike.",
"adminAffiliateView.pendingCents":"PendingCents is accrued minus paid, in cents: what is still owed, and the hard\nceiling the next payout is reserved against. Never negative.",
"adminAffiliateView.rateBps":"RateBps is this affiliate's DIRECT (level 1) commission rate in basis points\nOF Hanzo's margin (2000 = 20% of margin, never of the customer's bill). Levels\n2 and 3 are platform-wide switches and are not carried per affiliate.",
"adminAffiliateView.referredCount":"ReferredCount is how many orgs this affiliate is the DIRECT referrer of,\ncounted from the attribution edges. It is 0 on the single-affiliate answers\n(approve, suspend, rate, payout), which do not run the count.",
"adminAffiliateView.requestedCode":"RequestedCode is the vanity code the applicant asked for. A request, not an\nallocation: approval mints a different slug if this one was taken. Absent when\nnone was asked for.",
"adminAffiliateView.status":"Status is \"applied\", \"approved\" or \"suspended\". Only \"approved\" resolves for\nattribution and accrues; \"suspended\" stops future earning and claws nothing\nback.",
"adminAffiliateView.suspendedAt":"SuspendedAt is when staff suspended, Unix seconds UTC. 0 means never\nsuspended; it is not cleared by a later re-approval.",
"affiliateData.affiliate":"Affiliate is the row as it stands AFTER the action that returned it. Its\nreferredCount is 0 here: these single-affiliate answers do not run the count.",
"affiliateOut.data":"Data carries the affiliate row the action just wrote.",
"envelope.msg":"Msg is an operator-facing note. Empty on every success here — it exists\nbecause the console's admin unwrapper reads the shape cloud.OK writes.",
"envelope.status":"Status is \"ok\" on every 2xx from this surface; a failure is an HTTP error with\nzip's error body, not this envelope carrying a different word.",
"rateSet.id":"ID is the affiliate whose direct rate moves, from the path.",
"rateSet.rateBps":"RateBps is the direct commission rate, in basis points of Hanzo's margin;\ncapped so the whole L1+L2+L3 schedule never exceeds the margin. Body-only\n(`url:\"-\"`): a money parameter must never ride the URL into access logs.",
Description:"Suspends an affiliate: it stops accruing on the next sweep, and\nits code stops resolving for new attributions.\n\nIt CLAWS NOTHING BACK. Commission already accrued stays accrued and stays\npayable, and existing attribution edges are left standing — suspension ends\nearning, it does not unwind history. PLATFORM SUDO ONLY. Audited.",
Fields:map[string]string{
"adminAffiliateView.accruedCents":"AccruedCents is lifetime commission accrued, in cents. It only grows — a\npayout moves paidCents, never this.",
"adminAffiliateView.approvedAt":"ApprovedAt is when staff approved, Unix seconds UTC. 0 means never approved.",
"adminAffiliateView.code":"Code is the minted referral code, the slug the ?aff link carries. Empty until\napproval mints it. Codes are one global namespace across all affiliates.",
"adminAffiliateView.createdAt":"CreatedAt is when the org applied, Unix seconds UTC.",
"adminAffiliateView.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed — the id the\napprove, suspend, rate and payout routes address.",
"adminAffiliateView.org":"Org is the partner's own org slug. It appears ONLY on this cross-tenant admin\nview; no partner-facing read ever names another org.",
"adminAffiliateView.paidCents":"PaidCents is lifetime commission already paid out, in cents — credits grants\nand record-only cash disbursements alike.",
"adminAffiliateView.pendingCents":"PendingCents is accrued minus paid, in cents: what is still owed, and the hard\nceiling the next payout is reserved against. Never negative.",
"adminAffiliateView.rateBps":"RateBps is this affiliate's DIRECT (level 1) commission rate in basis points\nOF Hanzo's margin (2000 = 20% of margin, never of the customer's bill). Levels\n2 and 3 are platform-wide switches and are not carried per affiliate.",
"adminAffiliateView.referredCount":"ReferredCount is how many orgs this affiliate is the DIRECT referrer of,\ncounted from the attribution edges. It is 0 on the single-affiliate answers\n(approve, suspend, rate, payout), which do not run the count.",
"adminAffiliateView.requestedCode":"RequestedCode is the vanity code the applicant asked for. A request, not an\nallocation: approval mints a different slug if this one was taken. Absent when\nnone was asked for.",
"adminAffiliateView.status":"Status is \"applied\", \"approved\" or \"suspended\". Only \"approved\" resolves for\nattribution and accrues; \"suspended\" stops future earning and claws nothing\nback.",
"adminAffiliateView.suspendedAt":"SuspendedAt is when staff suspended, Unix seconds UTC. 0 means never\nsuspended; it is not cleared by a later re-approval.",
"affiliateData.affiliate":"Affiliate is the row as it stands AFTER the action that returned it. Its\nreferredCount is 0 here: these single-affiliate answers do not run the count.",
"affiliateOut.data":"Data carries the affiliate row the action just wrote.",
"affiliateRef.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed.",
"envelope.msg":"Msg is an operator-facing note. Empty on every success here — it exists\nbecause the console's admin unwrapper reads the shape cloud.OK writes.",
"envelope.status":"Status is \"ok\" on every 2xx from this surface; a failure is an HTTP error with\nzip's error body, not this envelope carrying a different word.",
Description:"Runs the accrual: for each referred org it reads that org's metered\nspend for the current period and accrues commission to every affiliate up its\nreferral chain, then answers how many sources were swept and how many NEW\naccruals landed.\n\nThis is the cron path, and it is LATCHED at most once per affiliate, source\norg and period — so re-running it inside the same period accrues nothing\nfurther. Safe to retry, and safe to run by hand beside the schedule.\n\nCommission is a rate of Hanzo's MARGIN on that spend, never of the customer's\ngross bill, so every level's share summed over one source event stays within\nthe margin actually earned and the customer's charge is untouched. Nothing\naccrues past the third upline level, and only an APPROVED affiliate accrues at\nall.\n\nThe same spend read drives the OSS author royalty — one read, both programs —\nso the answer reports royalties accrued alongside. PLATFORM SUDO ONLY. Bounded\nper run; a source whose spend cannot be read is skipped and picked up next\ntime, never half-accrued.",
Fields:map[string]string{
"accruals.accrued":"Accrued is how many NEW commission accruals this run created, counted across\nevery upline level. The accrual is latched at most once per (affiliate, source\norg, period), so a re-run inside the same month reports 0 having changed\nnothing — 0 means \"already accrued\", not \"failed\".",
"accruals.royaltiesAccrued":"RoyaltiesAccrued is how many OSS-author royalty accruals the SAME spend read\nproduced in the sibling authors program. One read drives both.",
"accruals.royaltyFailures":"RoyaltyFailures is reported, not swallowed: a sweep that could not reach\nthe royalty store must not read as one that found nothing owed. The count\nwas already computed and then dropped on the floor, which is the same\nsilence the typed leg was added to end.",
"accruals.swept":"Swept is how many source (referred) orgs the run visited, bounded at 500 per\nrun. A source with no spend this period, or one whose spend could not be read,\nstill counts as swept.",
"accrualsOut.data":"Data is what the run did: sources visited, new accruals, royalties alongside.",
"envelope.msg":"Msg is an operator-facing note. Empty on every success here — it exists\nbecause the console's admin unwrapper reads the shape cloud.OK writes.",
"envelope.status":"Status is \"ok\" on every 2xx from this surface; a failure is an HTTP error with\nzip's error body, not this envelope carrying a different word.",
},
})
zip.Describe("POST /v1/affiliates/apply",zip.Doc{
Description:"Enrolls the caller's OWN org as an affiliate at status `applied`,\noptionally requesting a vanity code, and answers the record — 201 on the first\napply, 200 with `created:false` afterwards.\n\nIDEMPOTENT, first apply wins: one affiliate per org, so re-applying never\ncreates a second row and never resets an existing approval. Applying is not\njoining — no code is minted and nothing accrues until staff approve, which is\nwhere both the code and the commission rate come from.\n\nThe org is the validated caller's, never a field. A malformed vanity code is\nrefused up front; the code is only REQUESTED here, and approval may mint a\ndifferent one if the requested code is taken.",
Fields:map[string]string{
"application.code":"Code is the minted referral code. Empty on a first apply — applying does not\nmint a code, approval does; a re-apply echoes whatever the row already holds.",
"application.created":"Created says whether THIS call made the row. false means the org had already\napplied and nothing changed — no second row, no reset of an existing approval.\nThe HTTP status states the same fact: 201 when true, 200 when false.",
"application.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed — the id staff\napprove, suspend, re-rate and pay against.",
"application.rateBps":"RateBps is the direct (level 1) commission rate the row carries, in basis\npoints OF Hanzo's margin (2000 = 20% of margin, never of the customer's bill).",
"application.requestedCode":"RequestedCode echoes the vanity code asked for, normalized to lower case. It\nis a request only: approval mints a different slug if this one is taken.",
"application.status":"Status is \"applied\" for a row this call created. A re-apply echoes the\nexisting row's status, which may already be \"approved\" or \"suspended\".",
"applyRequest.requestedCode":"RequestedCode is the vanity code the applicant asks for; approval may mint\na different one if it is taken. Body-only: the URL cannot supply it.",
Description:"Records the first-touch edge every later commission is computed\nfrom: the caller's org was referred by the affiliate that owns this code.\n\nThe REFERRED org is the validated caller, never a field. A caller that could\nname the referred org could attach itself to somebody else's revenue. The\naffiliate is resolved from the code, and only an APPROVED affiliate's code\nresolves.\n\nFIRST TOUCH WINS, set once: one affiliate per referred org, so a re-post\nanswers the existing edge with `created:false` rather than moving the\nattribution. Self-attribution is refused, and so is a code that would make a\ncycle in the upline chain. An unknown code is a 404, deliberately: an\naffiliate code IS a public shareable link, so whether one is real is public by\ndesign, and the caller legitimately needs to know its link resolved.\n\nA user-level mirror of the edge is written best-effort; a conflict there never\nfails the org attribution, which is the money-bearing one.",
Fields:map[string]string{
"attributeRequest.code":"Code is the affiliate code the referred org arrived with. Body-only: the\nURL cannot supply it.",
"attribution.code":"Code is the affiliate code the edge was recorded under, normalized to lower\ncase. On a re-post it is the code of the STANDING edge, which may differ from\nthe one just sent — first touch wins.",
"attribution.created":"Created says whether THIS call made the edge. false means the caller org was\nalready attributed and nothing moved. The HTTP status says the same: 201 when\ntrue, 200 when false.",
"attribution.createdAt":"CreatedAt is when the edge was FIRST recorded, Unix seconds UTC. On a re-post\nit is the original time, not now.",
"attribution.id":"ID is the attribution edge's server-minted handle, \"afr_\"-prefixed.",
},
Example:json.RawMessage(`{"code":"acme"}`),
})
zip.Describe("POST /v1/affiliates/click",zip.Doc{
Description:"Counts a click on a share link. PUBLIC — it takes no principal, because\na visitor clicking a shareable link has no session yet.\n\nThe ping folds into an in-memory buffer and NEVER writes the money database\nsynchronously, so a click flood cannot contend with the accrual and payout\nwrite path; tallies are flushed in one batch on the next authenticated links\nread and at shutdown. Clicks are a vanity metric: no accrual and no payout\never reads them — those key on real metered spend — so click inflation cannot\nmove money.\n\nAny well-formed code is accepted WITHOUT checking that it exists,\ndeliberately: this is not a code-existence oracle. `counted` reports that the\nbuffer took the ping, not that the code is real; an unknown code simply no-ops\nat flush time.",
Fields:map[string]string{
"clickCount.counted":"Counted says the in-memory buffer took the ping. It does NOT say the code\nexists — this is deliberately not a code-existence oracle, and an unknown code\nsimply no-ops at flush time. false means the buffer was full and the ping was\ndropped, which is harmless: clicks are vanity and move no money.",
"clickRequest.code":"Code is the share-link code that was clicked. Body-only: the URL cannot\nsupply it.",
Description:"Sets the caller's public leaderboard display name, or clears it.\n\nThe handle IS the opt-in. An empty handle opts out: the affiliate keeps its\nrank and can still see its own row, it simply stops being listed to anyone\nelse. That is the whole privacy control — there is no separate visibility\nflag, and no way to be listed without choosing a name.\n\nRequires a validated principal and an existing affiliate record; apply first.\nThe handle is bounded and restricted to letters, digits, space, hyphen,\nunderscore and dot.",
Fields:map[string]string{
"handleRequest.handle":"Handle is the public leaderboard display name; empty opts out. Body-only:\nthe URL cannot supply it.",
"handleSet.handle":"Handle is the display name as STORED, echoed back after trimming. Empty means\nthe caller opted out: it keeps its rank and still sees its own row, it is just\nno longer listed to anyone else.",
Description:"Mints a new share link for the caller's own affiliate and answers it\nwith its full URL, 201.\n\nAPPROVAL IS REQUIRED: an org that has applied but is not approved is refused,\nbecause a link that cannot accrue is a link that quietly loses the referral. A\nrequested vanity code must be valid and free across the WHOLE directory —\ncodes are one global namespace, so a taken code is a 409 rather than a silent\nalias. Omit the code and a random one is minted.\n\nBounded per affiliate. The label is cosmetic: it is trimmed, stripped of\ncontrol characters and capped, and it is never part of a code.",
Fields:map[string]string{
"codeView.clicks":"Clicks is how many pings this code has taken. The one STORED counter here and\npure vanity: no accrual or payout reads it, pings are coalesced in memory and\nflushed in batches, and a dropped tally is accepted rather than contending\nwith the money write path. Do not reconcile it against anything.",
"codeView.code":"Code is the link's slug — 3–32 chars of a–z, 0–9 and hyphen — unique across\nthe WHOLE directory, so any affiliate's code resolves an attribution.",
"codeView.conversions":"Conversions is how many of those signups have actually produced positive\ncommission for the caller. Also derived, from the accrual rows, so it is\n≤ signups and lags a referral until the first sweep after it spends.",
"codeView.createdAt":"CreatedAt is when the link was minted, Unix seconds UTC.",
"codeView.label":"Label is the caller's own note for the link (\"twitter\", \"newsletter\").\nCosmetic: trimmed, stripped of control characters, capped at 48 bytes, and\nnever part of the code. \"primary\" on the link mirrored at approval.",
"codeView.signups":"Signups is how many orgs were attributed with this code — DERIVED by counting\nattribution edges, never stored, so it cannot drift from the ledger.",
"codeView.url":"URL is the full shareable link, the brand host plus ?aff=<code>. The host is\nthe deployment's own brand, so a Lux or Zoo install never mints a hanzo.ai\nlink.",
"createLinkRequest.code":"Code is an optional vanity code; it must be free across the whole\ndirectory, and omitting it mints a random one. Body-only.",
"createLinkRequest.label":"Label is cosmetic — trimmed, stripped of control characters, capped — and\nnever part of a code. Body-only: the URL cannot supply it.",
"linkMint.link":"Link is the link just minted, with its full shareable URL. Its funnel counters\nall start at zero — nothing has clicked or signed up through it yet.",
Description:"Returns one agent with its system prompt and its 20 most recent runs.\nThe ref is the agent's public id or its org-unique name — a created agent is\nimmediately gettable by whatever create handed back.",
Fields:map[string]string{
"agentRef.ref":"Ref is the agent's public id (the agent_… handle create and list return) or\nits org-unique name, from the path. Either resolves the same agent.",
"agentRef.ref":"Ref is the agent's public id (the agent_… handle create and list return) or\nits org-unique name, from the path. Either resolves the same agent.",
"agentRunView.agent":"What an operator needs to answer \"what ran, for whom, and what did it do\" —\nand, through traceId, to leave this record for the waterfall of the very\nsame run rather than a search that hopefully lands near it.\n\nAgent is on the row because the org-wide feed lists runs across agents, and\na run that cannot name its agent is an orphan in exactly the view built to\nmake sense of many of them. Every field is omitempty: a run recorded before\nthese columns existed reports absence rather than a zero it never measured.",
},
Example:json.RawMessage(`{"ref":"helper"}`),
})
zip.Describe("GET /v1/agents/:ref/runs",zip.Doc{
Description:"Returns one agent's execution history, newest first — each run's\ninput, its output or its error, and how long it took. Every row is a run that\nactually happened.",
Fields:map[string]string{
"runList.runs":"Runs is the agent's executions, newest first.",
"runsQuery.limit":"Limit caps how many runs come back, newest first. Absent, zero or out of\nrange (1..200) reads as 50.",
"runsQuery.ref":"Ref is the agent's public id or its org-unique name, from the path.",
"agentRunView.agent":"What an operator needs to answer \"what ran, for whom, and what did it do\" —\nand, through traceId, to leave this record for the waterfall of the very\nsame run rather than a search that hopefully lands near it.\n\nAgent is on the row because the org-wide feed lists runs across agents, and\na run that cannot name its agent is an orphan in exactly the view built to\nmake sense of many of them. Every field is omitempty: a run recorded before\nthese columns existed reports absence rather than a zero it never measured.",
"runList.runs":"Runs is the agent's executions, newest first.",
"runsQuery.limit":"Limit caps how many runs come back, newest first. Absent, zero or out of\nrange (1..200) reads as 50.",
"runsQuery.ref":"Ref is the agent's public id or its org-unique name, from the path.",
Description:"Returns the org's agent runs across EVERY agent, newest first —\nwhat ran here, for whom, on which model, how long it took, and why it failed.\n\nIt is the feed the per-agent history could not be: an operator asking \"what is\nthis tenant's agent plane doing\" does not start out knowing an agent ref, and\nanswering by listing the agents and then paging each one's history is N+1 round\ntrips to reconstruct one ordering the database already has (RunsSince, ordered\nby created_at over the org index).\n\nThe org is the CALLER's, resolved from identity by tenantStore — never a\nparameter. There is deliberately no org field on orgRunsQuery to forge: run\nhistory is the tenant's own record, and the only tenant this can answer for is\nthe one asking.",
Fields:map[string]string{
"agentRunView.agent":"What an operator needs to answer \"what ran, for whom, and what did it do\" —\nand, through traceId, to leave this record for the waterfall of the very\nsame run rather than a search that hopefully lands near it.\n\nAgent is on the row because the org-wide feed lists runs across agents, and\na run that cannot name its agent is an orphan in exactly the view built to\nmake sense of many of them. Every field is omitempty: a run recorded before\nthese columns existed reports absence rather than a zero it never measured.",
"orgRunsQuery.limit":"Limit caps how many runs come back, newest first. Absent, zero or out of\nrange (1..200) reads as 50.",
"orgRunsQuery.status":"Status keeps only runs with this outcome (\"ok\" or \"error\"). Empty keeps\nboth. It is the filter an operator reaches for first — \"show me what broke\"\n— and answering it here rather than by paging the whole history client-side\nis the difference between a usable feed and a download.",
"runList.runs":"Runs is the agent's executions, newest first.",
Description:"Returns the caller org's live sessions, newest first — each with\nits event count, its direct-child count and a one-line preview of its latest\nevent. With no filter it returns ROOT sessions only, so a dashboard shows one\nrow per flow rather than one per subagent; ?root= or ?parent= descends.",
Description:"Answers a bridge's turn.\n\nUnlike the session ops, the org travels IN the request rather than being taken\nfrom the caller's plane identity: the tenant here is the one that connected the\nSlack workspace, resolved by the bridge from the signed team_id, and the bridge\nplugin's own identity is not it. That is safe because this op only SPENDS the\nnamed org's own balance under its own agent — it reads nothing across tenants —\nand because the subject must be a link the bridge already proved.\n\nAn empty subject is refused rather than defaulted. A turn that lost its caller\nmust not run AS THE ORG: that would bill the tenant for an unattributable act\nand hand an unlinked user the org's agent.",
Fields:map[string]string{
"RunOnBehalfIn.input":"Input is the user's message, already stripped of the leading @mention.",
"RunOnBehalfIn.model":"Model is the ASKER's own choice, empty when they have not made one. It is a\npreference of the person, not a property of the agent, which is why it rides\nthe turn instead of being written into an agent row: two people in one\nworkspace can prefer different models of the same assistant.",
"RunOnBehalfIn.org":"Org is the isolation gate, the tenant, and the balance the run bills.",
"RunOnBehalfIn.ref":"Ref names the agent to run.",
"RunOnBehalfIn.subject":"Subject is the caller's LINKED Hanzo identity, unqualified. Attribution and\nauthorization both hang off it, so a turn can never run as nobody: the\nanswering side refuses an empty subject rather than falling back to the org.",
Description:"Tears down every live session of the CALLER's org matching\nthe revoking subject, and reports how many it stopped.\n\nThe org is the caller's plane identity and never the argument — plane\n.SessionMatchIn has no org field, deliberately, because this op STOPS things\nand a caller able to state the org could stop a co-tenant's work. Anonymous is\nrefused rather than defaulted: a teardown arriving with no principal must\nfail, not pick a tenant.\n\nThe actor is built HERE, from the org the plane proved and the subject the\ncaller names, so the HIGH-1 actor scoping (a revoke stops only that user's own\nsessions) is enforced by the side that owns the store rather than trusted from\nthe wire.\n\nA named handler, not a closure, so zipdoc can lift this prose into the registry.",
Fields:map[string]string{
"SessionMatchIn.host":"Host/Provider/Account narrow WITHIN the actor's own sessions; empty is any.",
"SessionMatchIn.subject":"Subject is the revoking user, unqualified. The answering side qualifies it.",
},
})
zip.Describe("POST /v1/agents",zip.Doc{
Description:"Defines an agent in the caller's org: a model, a system prompt\n(instructions) and a set of tool names. The name must be unique in the org and\nmatch ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$. An omitted model takes the\ndeployment's configured default; a named one is checked against the gateway's\nserved catalog, so a model this deployment never serves is refused here rather\nthan failing at run time. A long-running agent must carry a 5-field cron\nschedule (the scheduler would otherwise never fire it) and counts against a\nper-org cap on scheduled agents.",
Description:"Health reports whether the event plane can take a write and the warehouse can\nanswer a read.\n\nIt reports the analytics subsystem's own liveness in BOTH directions: plane is\nthe event plane it WRITES (the bus and the JetStream stream every accepted\nevent is published to, both named in the report), and datastore is the\nwarehouse it READS, with each read lens's table reported as it is provisioned\n(the LLM usage ledger and the product-event table).\n\nEITHER ONE DOWN IS A 503, and the report says WHICH — they are probed\nindependently and never collapse into a single bit. This endpoint used to\nreport the read half only, and answered 200/ok while every POST /v1/event\nfailed on a stream that could not bind: a total ingest outage behind a green\nprobe. A readiness gate here now gates on the write path too.\n\nplane.ready IS A REAL PROBE and walks the ingest path itself — the same\nconnection and the same stream a publish uses — so it cannot answer ready while\na publish would 503. plane.reason carries the plane's own error text when it is\nfalse.\n\ndatastore IS NOT PROBED WITH A QUERY. It is the state of the process's own\nshared client — established, and not since closed — so a warehouse accepting\nconnections and failing reads still reports true. Degraded CARRIES the report\n(status, the failing half, reason) as its body rather than an error envelope,\nso a gate reads the cause off the same object it got at 200.\n\nA MISSING LENS TABLE IS NOT A FAILURE and never moves the status: a lens\nreported available:false answers honest-empty rather than erroring, so a fresh\ndeployment whose collector has not emitted yet is legitimately 200 with the\nproduct-event lens unavailable. The lens block is reported whenever the\nwarehouse is REACHABLE — including on a report degraded by the plane, where the\ntables genuinely were probed — and is absent only when the warehouse is not,\nhaving nothing to say about tables it could not reach.\n\nUnauthenticated on purpose — liveness has to be probe-able — and it reads NO\ntenant data: table existence and stream presence only, never a row and never an\nevent.",
Fields:map[string]string{
"healthLens.available":"Available reports whether that table exists in the warehouse right now.",
"healthLens.table":"Table is the fully-qualified warehouse table the lens reads.",
"healthLenses.events":"Events is the web/commerce lens (event.fact, signal='act'), honest-empty until the\ncollector emits.",
"healthLenses.llm":"LLM is the live per-org usage ledger lens (hanzo.cloud_usage).",
"healthPlane.bus":"Bus is the address this process reaches the plane at.",
"healthPlane.ready":"Ready reports whether an ingest would succeed right now. False is a 503.",
"healthPlane.reason":"Reason is the plane's own failure text, present only when Ready is false.",
"healthPlane.stream":"Stream is the JetStream stream every signal lands on.",
"healthReport.datastore":"Datastore reports whether the shared warehouse client has a live connection.\nIt is load-bearing for the READ path: false is one of the two ways this\nanswers 503.",
"healthReport.lenses":"Lenses is per-lens table availability, probed only when connected — so it is\nabsent from a degraded report, which has nothing to say about tables it could\nnot reach.",
"healthReport.lost":"Lost is the count of facts the sink irrecoverably dropped since boot\n(warehouse.go). It is reported on the DEGRADED report too, and deliberately: a\nwarehouse that is unreachable is exactly when facts start failing their\ndeliveries, so suppressing the number here would hide it precisely when it\nmoves. ANY NON-ZERO VALUE IS AN ALARM — it counts data the door already\nanswered 200 for.",
"healthReport.plane":"Plane reports the event plane — the bus and the stream every accepted event is\npublished to BEFORE any of it reaches the warehouse. It is load-bearing for the\nWRITE path, and it is here because its absence was a real outage: this endpoint\nanswered 200/ok on warehouse connectivity alone while every POST /v1/event 503'd\non a stream that could not bind, so 100% ingest loss was invisible to monitoring.\nA probe that cannot see the write path cannot report the write path.",
"healthReport.reason":"Reason is the human-readable cause, present only on a degraded report.",
"healthReport.service":"Service names the subsystem answering, so a probe aggregating several health\nendpoints can attribute a degraded one.",
"healthReport.status":"Status is ok or degraded. Degraded is the 503 and means EITHER load-bearing\ndependency is down — the warehouse this subsystem reads, or the event plane it\nwrites. It is not moved by a missing lens table, which is honest-empty.",
"healthReport.warehouse":"Warehouse names the datastore database every lens reads.",
"loss.exhausted":"Exhausted counts facts the bus abandoned after maxDeliver failed inserts.",
"loss.undecodable":"Undecodable counts messages acked without landing because they did not parse.",
Description:"Overview returns the caller org's analytics KPIs for one time window. Three lenses\nover one warehouse: llm is the live per-org LLM usage ledger (requests, tokens,\nspend, models, providers, errors) and is always real; web (pageviews, visitors,\nsessions) and commerce (orders, revenue, AOV) read the product-event table and\nreport available=false rather than fabricating zeros when it holds nothing yet.\n\nThe org is the validated principal's — never a parameter — so a caller can only\never read its own tenant. 403 without a validated bearer, 400 on an unknown range,\n503 when the warehouse is unreachable.",
Fields:map[string]string{
@@ -162,4 +185,16 @@ func init() {
"insightsStatus.surface":"Surface is the path prefix this status covers: /v1/insights.",
},
})
zip.Describe("POST /event/capture",zip.Doc{
Description:"Captures ONE occurrence onto the calling organisation's own event plane — the\nsame plane the HTTP door fills and /v1/insights reads back, through the SAME\nwrite core.\n\nThe organisation is the CALLER's, minted from the plane principal and never\nfrom this body, which carries no field that could name one. A peer that states\nno principal writes nothing: an unidentified caller has no partition, and\ndefaulting one would be a shared store with a tenant anybody can reach.\n\nA name is REQUIRED and is refused rather than defaulted. Every other route into\nthis core has a server-chosen default name for the signal it carries (a page\nview, an error, a span); a peer's occurrence is a tracked act, and a tracked\nact with no name is unroutable by the plane's own rule — so the refusal is the\nnormalizer's, said at the boundary where the caller can still be told.\n\nA named handler, not a closure, so zipdoc can lift this prose into the registry.",
Fields:map[string]string{
"EventCaptured.accepted":"Accepted is how many occurrences were admitted and published.",
"EventCaptured.dropped":"Dropped is how many were not, because nothing about them named a landable\nrow.",
"EventIn.at":"At is when it happened, RFC 3339. Empty means now, decided by the app that\nowns the plane (which is also the one that clamps a stated time).",
"EventIn.attributes":"Attributes are the occurrence's own facts, as [Signal] — the plane's ONE\nname/value pair — because a map cannot cross this plane at all (see\n[Header], which learned it the expensive way). They land in the row's\nattributes map, so a new fact is a new value and never a schema change.",
"EventIn.name":"Name is what happened, in the emitter's own verb-object vocabulary\n(risk_decided, secret_rotated). It is the column every lens groups by, so it\nmust come from a CLOSED set the emitter owns — never from a string a caller\nof the emitter chose, which is unbounded cardinality in the one column the\nplane indexes.",
"EventIn.product":"Product is the emitting SURFACE — the app's own name. Empty attributes the\nrow to no surface, which is honest and unhelpful.",
"EventIn.subject":"Subject is whom the occurrence concerned, as the value the emitter is\nwilling to have stored: the row's distinct id. An identifying value belongs\nhere only in an opaque form — the plane is a SHARED store read by every lens\nthe organisation has, not the emitter's private record.",
"Lead with the answer; be concise, factual, and well structured (short paragraphs, bullets where they help). "+
"Cite inline as Markdown links [source title](url) immediately after the claim each source supports, and cite generously. "+
"Do NOT add a References or Sources section, footnote markers, or bare URLs — citations are inline links only. "+
"If the sources conflict or are insufficient, say so plainly and answer from general knowledge while noting the uncertainty. Never fabricate facts or URLs."
"If the sources conflict or are insufficient, say so plainly and answer from general knowledge while noting the uncertainty. Never fabricate facts or URLs."+
widgetRule
// widgetRule teaches the model the ONE structured-result format the answer
// surfaces render. It is shared by every mode so the shapes cannot drift apart
// between search and research.
//
// A widget is an ENHANCEMENT, and the instruction says so explicitly: the
// client validates every block and DROPS anything malformed — a ragged table,
// an unknown kind, a field of the wrong type — keeping the prose. So an answer
// whose prose depends on a widget to make sense would read as a hole whenever
// validation refused one. The prose must stand alone; the widget makes it
// faster to read.
//
// The model supplies DATA, never markup: the renderer holds the shapes and
// escapes every field. That is deliberate — this model reads the open web, so
// any page it fetches is a potential injection source, and markup it authored
// would be a path into the extension's origin.
widgetRule="\n\nWhen the question's SHAPE calls for one — a comparison, a procedure, "+
"key figures, a chronology, a single entity, or a term to define — ALSO emit exactly one "+
"structured result block, fenced as ```hanzo-widget containing only JSON. Use at most two "+
"per answer, and only when the shape genuinely fits; most answers need none. "+
"The prose must stand on its own without the block. Emit DATA only, never HTML. "+
if!strings.Contains(widgetRule,"stand on its own"){
t.Error("the rule does not tell the model the prose must stand alone; a dropped "+
"widget would then leave the answer incomplete")
}
if!strings.Contains(widgetRule,"never HTML"){
t.Error("the rule does not forbid HTML; the model reads the open web, so markup it "+
"authored is an injection path into the client's origin")
}
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
z